Index: /trunk/gsdl3/docs/manual/manual.tex
===================================================================
--- /trunk/gsdl3/docs/manual/manual.tex	(revision 3710)
+++ /trunk/gsdl3/docs/manual/manual.tex	(revision 3711)
@@ -22,15 +22,4 @@
 {\end{list}}
 
-\noindent
-{\em \tiny This is intended to turn into a multipurpose document that
-\begin{bulletedlist}
-\item forms the basis of a JCDL paper submission
-\item fulfills our NERF pledge to produce a ``design document for Greenstone3''
-by December 2002 ...
-\item ... and a ``definition of internal and external interfaces for all major
-components (including API for external clients)'' by July 2003
-\item turns into a proper manual for Greenstone3
-\end{bulletedlist}
-}
 
 \noindent
@@ -58,246 +47,11 @@
 Native Interface) will be used to communicate with these.
 
+
 \section{Architecture}
 
-A typical basic Greenstone3 digital library system is made up of a ``back
-end,'' which we call a digital library {\em site\/}, coupled to a ``front end''
-that provides the user interface. Figure 1 shows a simple stand-alone digital library with a web-based front end which communicates with a single site. In this simple example,  the entire system is compiled together as a single executable. The point of contact with the back end is the MessageRouter (MR) module---all communication with the site occurs through this module. 
-
-The digital library back end in Figure 1 contains two collections, {\em demo}
-and {\em myfiles\/}, and a cluster of collection-formation services.  All
-functions of the digital library are called ``services.''  For example,
-AddDocument is a service that adds a document to a collection; ImportCollection
-imports into the Greenstone system all documents associated with a collection,
-converting them as necessary from their original form; BuildCollection builds
-all indexes and browsing structures that are associated with collection;
-ActivateCollection makes a newly-built collection active, so that it can be
-seen by digital library users.  These particular services are related: they are
-all concerned with creating a digital library collection.  Related
-services may be grouped together into a  ``service cluster'': all these services are provided by
-the CollectionFormation ServiceCluster module in Figure 1.
-
-A collection, which as far as the digital library user is concerned is a
-focused group of documents with a uniform means of access, is a type of service
-cluster that groups a set of services that are related by the set of data they
-work on.  For example, the {\em demo} collection in Figure 1 contains four
-services. These provide text searching, metadata searching,
-document retrieval, and browsing services to the user.  
-
-The Web-based front end in Figure 1 centers around the
-Receptionist, which is the point of contact for the interface generator.  A
-servlet takes HTTP commands (in the form of URLs and arguments) and translates
-them into XML form for the Receptionist.  This is capable of executing various
-different Actions, each of which involve one or (usually) many calls to the
-digital library's MessageRouter.
-
-Figure 1 shows a very simple example of a digital library structure.
-In practice, there may be many digital library sites, possibly involving
-distributed computers.  Each site will have a structure similar to that of the back end in Figure
-1.  Different sites may know about each other and can gain access to each other's
-collections by forwarding requests.  There may also be different user
-interfaces to the library.  Figure 1 shows a simple web-based interface, but
-other interfaces may exist, ranging from applets that display documents in
-different ways to alert services that note when new information becomes
-available in one of the collections and formulate email to users.  Although in
-the simplest case the front and back ends are compiled together into
-one executable process, in general different MessageRouters will communicate
-amongst themselves, and with Receptionists, using a protocol.
-
-The following subsections elaborate on this structure.
-
-\subsection{Modular structure}
-
-Greenstone3 is made up of independent modules that communicate via a single
-method call:
-\begin{quote}
-	XMLout =  process(XMLin);
-\end{quote}
-Both input and output are expressed in XML.  This decision shifts attention
-from the design of an Applications Programming Interface (API) to the design of XML
-forms that encode the equivalent information.  The advantage is modularization:
-the XML specifications can be modified locally and communication will proceed
-effectively according to the new scheme provided only that all affected modules
-are altered appropriately.  Conversely, if an API is changed then all modules
-usually have to be recompiled to reflect the update.
-
-Modules are thought of as ``agents'' that have, or have access to, certain
-functionality.  A module may respond to a message by processing it itself, or
-forwarding it to another module, or a combination of the two.\footnote{Francois
-used some nice words to tie up modules and agents.  Kathy, can you remember
-what he was saying?}
-
-If modules are on different computers, the communication will take place using
-SOAP (Simple Object Access Protocol) (although other protocols are possible).  Figure 2 shows a Greenstone system where the local site has no collections or services of its own. Instead, the MessageRouter (1 in the diagram) talks to two other sites using SOAP.  The local MR  has two Communicator modules
- that enable it to make SOAP requests; the two remote sites each have a SOAP server which
-listens for such requests and fulfills them.  
-
-A potential downside of expressing the programming interface structure in XML
-is execution efficiency.  The input and output XMLin and XMLout in the above
-statement can be either a serialized String representation, which is the
-primary representation method, or a Document Object Model (DOM), which is a
-tree that represents the parsed XML string.  Two versions of the processing
-operation will be provided, string to string and tree to tree.
-
-\subsection{Dynamic configurability}
-
-Digital libraries need to be dynamic.  It must be possible to routinely add new
-collections, or new user interfaces, or completely new kinds of service, to a
-running digital library without having to bring it down and restart it.
-
-The digital library back end is built around a central MessageRouter module
-that provides a way of gaining access to any collection or service.  When new
-collections come up, they register with the MessageRouter in order to make
-themselves visible throughout the system.  When users make requests, they are
-passed to the MessageRouter, which forwards them to the appropriate module for
-processing.  Requests are synchronous; the requesting process is blocked until
-the result is received.  (An asynchronous-to-synchronous buffering module is
-envisaged if this should become necessary for certain purposes.)
-
-The most basic request, which any module will respond to, is
-``describe-yourself''.  (In fact, the ability to respond to
-``describe-yourself'' is really what defines a ``module.'')  The MessageRouter
-responds with an XML document which typically specifies some collections that
-are available locally, and some other Greenstone sites (their own collections
-may also be listed).  Its response may also describe service clusters or single  services provided by the
-MessageRouter itself, for example, cross-collection searching, or collection formation capability.
-
-A plain ``describe-yourself'' request will return a complete description.  A
-``describe-yourself'' message sent to a collection returns collection-specific
-metadata, and a list of services that the collection provides.  It is possible
-to add a qualifier to the request which asks for a particular facet of the
-complete description instead, thereby achieving communication economy.
-
-Using these facilities, it is possible for a user interface module to ask a
-MessageRouter for a list of local collections, remote sites and their
-collections, and for each collection a list of the services available.  The XML
-documents containing this information could be amalgamated and presented to the
-user as an XML form that actually implements the services that are represented.
-
-\subsection{Interacting with the user}
-
-The MessageRouter, together with the services it provides access to, forms the
-core of the Greenstone digital library system.  Clients could be written that
-call in a variety of ways upon the services that Greenstone provides.
-
-A very important form of client is one that implements user interaction with
-Greenstone3 through a Web browser, which is the standard way of communicating
-with the digital library system.  The user makes a request by clicking a URL or
-submitting a Web form.  This request is intercepted by a servlet which invokes
-a Greenstone module called a Receptionist.  The Receptionist represents the
-user's normal point of contact with the system: based on the input, it creates  XML messages which it passes i into the Greenstone system through the
-MessageRouter. The responses are gathered together and translated it into the form of
-a Web page for presentation to the user.
-
-The Receptionist receives from the servlet an XML representation of
-the arguments in the URL (``CGI arguments'', though we do not use the
-CGI mechanism).  One of these arguments is the Action, which, along
-with the Subaction argument determines what information must be
-requested from the MessageRouter to fulfill the request.  Table 1 shows
-a list of the actions that are understood by Greenstone2; Greenstone
-3 will have similar functionality.
-
-The Receptionist includes a Java class for each action.  These classes do not
-know anything about the collections, services, or other sites that are
-available in the Greenstone system.  Instead, they decode the other arguments in
-the URL to determine what information must be requested, and send it through
-the MessageRouter.  A single action often generates several different requests:
-for example, to generate the traditional Greenstone home page, the PageAction must query the MessageRouter for a list of its collections. Then, for each collection, collection metadata such as the collection image and collection Title must be retrieved.  The XML results returned by these requests are put together
-into one large XML tree, to which is appended system configuration and
-translation information.  The resulting XML structure is converted, using XSLT
-files appropriate to that particular action, to an HTML page for presentation
-to the user.
-
-Other types of client which do not use HTML may interact with the Receptionist. An output type specifier is included in each request to the Receptionist: using XSLT modes, different output formats may be generated such as XML or WML.
-
-\subsection{Digital library services}
-
-A digital library consists of several different ``collections,'' each
-represented by a collection module.  For each collection, a set of ``services''
-is provided.  Examples of services are
-\begin{bulletedlist}
-     \item	full-text query
-     \item	fielded query
-     \item	music query
-     \item document retrieval
-     \item	metadata retrieval
-     \item browsing classifier
-     \item	hierarchical phrase browsing.
-\end{bulletedlist}
-
-Services are provided by modules called ``service modules'', which each
-implement a group of related operations.  For example, one service is MGPPGDBM,
-which implements four operations: full-text and fielded queries, and document
-and metadata retrieval.  MGPPGDBM operates on collections that are in the
-format of standard Greenstone2 collections, and provides these four services
-for such collections.  Another service is GSDL2Classifier, which provides
-operations that correspond to a browsing classifier.  Together these two
-classes allow a Greenstone2 collection to be used, completely unchanged, within
-Greenstone3 (provided an appropriate configuration file is created).
-
-Service modules are self-describing modules: that is, they respond to the
-``describe-yourself'' message.  As noted above, collections are also
-self-describing modules: they respond to ``describe-yourself'' by returning
-collection-specific metadata, and a list of services that the collection
-provides---which can then be queried individually using ``describe-yourself''
-messages.  Thus a collection may be viewed as a cluster of services.
-Greenstone3 uses service clusters to represent other things than collections.
-For example, all the operations associated with building a particular kind of
-collection may be grouped together into a service cluster.
-
-\subsection{Data in the system}\footnote{I haven't discussed this with anyone yet, however I like it :-)  actually now Rob likes it too. NOTE: if we keep this document-resource idea, need to change all the resource refs in this paper to document!!}
-
-Data in the system consists of 'documents' and 'resources'. A document is an XML document\footnote{whats a better word for a generic document, not a greenstone document ??} that exists independently in the system. You could delete all other documents and it would still be valid (although links to other documents may become invalid). A resource is something that is associated with a document, and doesn't exist outside of that document's context.
-
-For example, a book that has been added to a collection will be represented by an XML document. The document contains metadata associated with the book, for example Title, Source Author etc. It has xlinks to associated resources or other documents. Any images in the book would be resources belonging to that document. The original representation of the book, eg the pdf file, would also be a resource of the document. There may be associated documents, such as the same book but translated into a different language. This translation is a document in its own right, but is linked to by the original document.
-
-Documents are indexed, but resources are not. This means that documents can be discovered through searching and browsing. Resources, on the other hand, can only be found via the containing document. Both can be retrieved. Documents are identified by a system id eg HASHxxx. Resources are identified by a unique identifier. This is likely to be a file path---this could be appended to an HTTP address to enable retrieval of the document via HTTP, or could be used as an identifier to request the resource from the site via XML messages. 
-
-The content of the document need not be stored with the document---it may live in the compressed data files. The documents themselves may be stored compressed or in a database. Currently, in Greenstone2, the equivalent information is stored in a gdbm database.
-
-Documents don't just have to be books and text files. A collection could contain images---each image would have a document, and the content of the document would point to the image file.
-A document could be a sequence of other documents eg a powerpoint show of individual slides.
-A classifier is a document - a hierarchical ordering by metadata of a set of documents into lists or categories.
-
-\subsection{Getting off the ground}
-
-We have described in broad terms the basic components of Greenstone3.  It is a
-highly configurable system that allows new modules to be added while it is
-running---dynamic configuration.  However, in order to get it off the ground,
-configuration files are used to define an initial configuration.
-
-A single computer system may have several different Greenstone systems
-or ``sites'' running simultaneously, each of which typically serve
-different collections.  For example, a single user may have a public
-Greenstone site which offers collections to external users over the
-web, as well as a private site that offers personal collections (like
-email) that cannot be accessed externally.  Or in a multiuser research
-environment, each user may have one or more sites reflecting
-Greenstone collections, or additional facilities, in different stages
-of development.
-
-The computer system will have just one Greenstone directory structure,
-though this structure may support several different sites.  Each site
-has a home directory in the Greenstone structure, inside which is a
-``collect'' directory that contains the collections offered by that site.
-
-The sites can be ``served'' in different ways. A servlet can be started up, which invokes a
-Receptionist and a MessageRouter.   One of the arguments to
-the servlet is the site's home directory. This configuration has a client and server compiled together. The information in this site can then be accessed via the web. Alternatively, a SOAPServer could be started up, which just invokes a MessageRouter. Other Greenstone systems or clients can communicate with this site via SOAP. Greenstone is not limited to SOAP communication---any protocol which can transmit XML may be used to communicate between sites, or between clients and servers.
-
-For each site there is a configuration file that specifies the URI for the site
-(localSiteName), and a list of external sites that the site connects to.  It
-may also specify any services or service clusters provided by the site that are not connected with
-a collection---for example, a language translation service.  Collections are
-not specified in this configuration file; instead they are determined by the
-contents of the ``collect'' directory for the site.  This allows new
-collections to be added dynamically by placing them in that directory.
+This section is covered by the paper: An agent based architecture for dynamic digital library construction and configuration. Either cut and paste it in here, or link to the text?? or have two separate docs. dont want to have to maintain two separate versions of the same thing.
 
 \section{Greenstone Implementation}
 \label{sec:impl}
-
-
-\subsection{classes etc??}
-
-In general, a Greenstone module corresponds to a Java class. The Receptionist, Action, MessageRouter, Collection, ServiceCluster modules are all Java classes. The exception is the service. Many services share operations, for example, access to the MGPP index files. For this reason, several services may be implemented by a single class---we call this a  ServicesImpl class. For example, MGPPGDBMServices is subclass of ServicesImpl which provides services that use the MGPP files and GDBM databases of a Greenstone 2 collection: TextQuery, DocumentRetrieve and MetadataRetrieve. MGGDBMServices provides the same services, but uses MG and GDBM files from a Greenstone 2 collection.
 
 \subsection{Configuring Greenstone}
@@ -312,6 +66,5 @@
 instructions on how the collection is to be built.  The second is produced by
 the build-time process and includes any metadata that can be determined
-automatically.\footnote{Currently it is produced by hand, because collections must
-be built with Greenstone2.}
+automatically.\footnote{Currently only the buildConfig.xml file is used - collections are built using gs2 style building and therefore use the old collect.cfg.}
 
 \subsubsection{Site configuration file}
@@ -319,5 +72,5 @@
 The file {\em siteConfig.xml} specifies the URI for the site ({\em
 localSiteName\/}), any services or service clusters provided by the site that are not connected
-with a particular collection (for example, translation services), and a list of
+with a particular collection (for example, translation services, or collection building), and a list of
 known external sites to connect to.  Collections are not specified in the site
 configuration file, instead they are determined by the contents of the site's
@@ -325,22 +78,31 @@
 
 Here is a configuration file for a rudimentary site with no site-wide services,
-which does not connect to any external sites.
+which does not connect to any external sites.\footnote{should the code be tolerant of missing elements? or do we require empty elements?}
 \begin{quote}\begin{footnotesize}\begin{verbatim}
 <config>
   <localSiteName value="org.greenstone.localsite"/>
   <serviceClusterList/>
-  <servicesImplList/>
+  <serviceRackList/>
   <siteList/>
 </config>
 \end{verbatim}\end{footnotesize}\end{quote}
-The following configuration file is for a site with one site-wide service, a
-translation service.  It connects to the previous site using SOAP.
+The following configuration file is for a site with one site-wide service cluster - a collection building cluster.  It also connects to the previous site using SOAP.
 \begin{quote}\begin{footnotesize}\begin{verbatim}
 <config>
   <localSiteName value="org.greenstone.gsdl1"/>
-  <servicesImplList>
+  <serviceRackList/>
     <servicesImpl name="TranslationServices"/>
   </servicesImplList>
-  <serviceClusterList/>
+  <serviceClusterList>	
+    <serviceCluster name="build">
+      <metadataList>
+        <metadata name="Title">Collection builder</metadata>
+        <metadata name="Description">Builds collections in a gsdl2-style manner</metadata>
+      </metadataList>
+      <serviceRackList>
+        <serviceRack name="GS2Construct"/>
+      </serviceRackList>
+    </serviceCluster>
+  </serviceClusterList>
   <siteList>
     <site name="org.greenstone.localsite"
@@ -351,24 +113,34 @@
 \end{verbatim}\end{footnotesize}\end{quote}
 
+These two sites are running on the same machine. For site1 to talk to localsite, a SOAP server must be run for localsite. The address of the SOAP server, in this case, is "http://localhost:8080/soap/servlet/rpcrouter"
+
 \subsubsection{Building configuration file}
 
-The file {\em buildConfig.xml} contains all metadata about the collection that can
+The file {\em buildConfig.xml} contains all metadata and other information about the collection that can
 be determined automatically when building the collection, such as the number of
-documents it contains.  It also includes a list of servicesImpl classes that are
+documents it contains.  It also includes a list of serviceRack classes that are
 required at runtime to provide the services that have been built into the
-collection.  The servicesImpl names are Java classes that are loaded
-dynamically at runtime. Any information inside the servicesImpl element is
+collection.  The serviceRack names are Java classes that are loaded
+dynamically at runtime. Any information inside the serviceRack element is
 specific to that service---there is no set format.  Here is an example:
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<buildConfiguration>
+
+<buildConfig>
   <metadataList>
-    <metadata name="iconCollection">mgppdemo.gif</metadata>
-    <metadata name="colName">mgpp demo</metadata>
-    <metadata name="numDocs">5</metadata>
-    <metadata name="numSections">189</metadata>
+    <metadata name="numDocs">11</metadata>
+    <metadata name="colIcon">mgppdemo.gif</metadata>
+    <metadata name="colName">Greenstone demo collection</metadata>
+    <metadata name="colDescription">This is a demonstration collection for the Greenstone digital library software. It contains a small subset  of the Humanitarian and Development Libraries.</metadata>
   </metadataList>
-  <servicesImplList>
-    <servicesImpl name="MGPPGDBMServices">
+  <serviceRackList>
+    <serviceRack name="GS2MGPPRetrieve">
+      <defaultLevel name="Section"/>
+	<!-- something list this should be used to advertise what metadata the collection has available to be retrieved - however, it is not used yet -->
+      <metadataList> 
+        <element name="Title"/><element name="Subject"/><element name="Organization"/><element name="URL"/>
+      </metadataList>
+    </serviceRack>
+    <serviceRack name="GS2MGPPSearch">
       <defaultIndex name="tt"/>
       <defaultLevel name="Section"/>
@@ -380,31 +152,22 @@
         <index name="tt"/>
         <index name="t0"/>
-      </indexList>  
-      <metadataList>
-        <element name="Title"/>
-        <element name="Subject"/>
-        <element name="Organization"/>
-        <element name="URL"/>
-      </metadataList>
-    </servicesImpl>
-    <servicesImpl name="PhindServices"/> 
-    <servicesImpl name="GSDL2ClassifierServices">
+      </indexList>
+      <fieldList>
+        <field name="TX"/><field name="SU"/><field name="TI"/>
+      </fieldList>
+    </serviceRack>
+    <serviceRack name="PhindPhraseBrowse"/>
+    <serviceRack name="GS2Browse">
       <classifierList>
-        <classifier name="CL1">
-          <metadataList>
-            <metadata name="Title">Subject</metadata>
-          </metadataList>
-        </classifier>
-        <classifier name="CL2" >
-          <metadataList>
-            <metadata name="Title">Title</metadata>
-          </metadataList>
-        </classifier>
+        <classifier name="CL1"><metadataList><metadata name="Title">Subject</metadata></metadataList></classifier>
+        <classifier name="CL2" ><metadataList><metadata name="Title">Title</metadata></metadataList></classifier>
+	<classifier name="CL4"><metadataList><metadata name="Title">Organization</metadata></metadataList></classifier>
+	<classifier name="CL5" ><metadataList><metadata name="Title">Keyword</metadata></metadataList></classifier>
       </classifierList>
-    </servicesImpl>
-  </servicesImplList>
+    </serviceRack>
+  </serviceRackList>
 </buildConfig>   
 \end{verbatim}\end{footnotesize}\end{quote}
-Note: because {\em collectionConfig.xml} is not used yet, the {\em iconCollection}
+Note: because {\em collectionConfig.xml} is not used yet, the {\em colIcon}, {\em colDescription}
 and {\em colName} metadata elements have been specified here.
 
@@ -431,5 +194,5 @@
 
 The MessageRouter reads in its site configuration file {\em siteConfig.xml}. This
-lists the ServicesImpl classes that need to be loaded, and lists any sites that need
+lists the ServiceRack classes that need to be loaded, and lists any sites that need
 to be connected to.  It looks inside the {\em collect} directory which contains
 all the site's collections and loads up a Collection object for each valid
@@ -437,7 +200,6 @@
 
 The Collection object reads its {\em buildConfig.xml} and {\em collectionConfig.xml}
-files, determines the metadata, and loads ServicesImpl classes based on the 
-names specified in {\em buildConfig.xml\/}. The {\footnotesize \verb#<ServicesImpl>#} XML element is passed to the object to be used in  configuration.\footnote{Kathy, I don't
-understand this sentence.}
+files, determines the metadata, and loads ServiceRack classes based on the 
+names specified in {\em buildConfig.xml\/}. The {\footnotesize \verb#<ServiceRack>#} XML element is passed to the object to be used in  configuration.
 
 \section{System messages}
@@ -450,19 +212,13 @@
 All messages are enclosed in
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='xx'>
-\end{verbatim}\end{footnotesize}\end{quote}
+<message>
+\end{verbatim}\end{footnotesize}\end{quote}
+Messages contain either {\em <request>\/} or {\em <response>\/} elements--- a single message may contain multiple requests. Each {\em <request>\/} (and {\em <response>\/}?) has a language attribute, of the form ``lang='xx'''.
 The language attribute is used by the XSLT to determine the language currently
 being used by the user interface.  Virtually all messages contain text strings,
 and services use this attribute to return strings in the appropriate language.
-Requests are called {\em <request>\/}, responses are called {\em <response>\/}.
-A single message can hold several requests or responses.
-
-There are two different types of message, explained in the two subsections
-below.  The first is a simple representation of the arguments in a Greenstone
-URL.  It is a rudimentary message passed into the digital library system from
-outside. The response is a page of data, typically in HTML.  All other messages
-are internal Greenstone messages, and have the same basic format.\footnote{We
-format names in lower case with the first letter of internal words capitalized,
-like 'matchDocs'.} They typically request one service or one action, and the response contains either the data requested, or a status message.
+
+There are two different styles of messaging, explained in the two subsections
+below.  The first is the communication between the servlet (or other external agent) and the Greenstone system (via the Receptionist). The request contains a simple representation of the arguments in a Greenstone URL, and has the same format as any request in the system.  The response is a page of data, typically in HTML.  The second style of messaging is the internal Greenstone communication. Requests and responses follow a basic format, and both are in XML.\footnote{We format names in lower case with the first letter of internal words capitalized, like 'matchDocs'.} They typically request one service or one action, and the response contains either the data requested, or a status message.
 
 This section describes the two message formats. The following section looks at how the front-end (Receptionist plus Actions) responds to the URL-type messages, and creates internal xxx-type\footnote{are there good names to distinguish the two types of messages?} messages to pass into the system.
@@ -480,6 +236,6 @@
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<request type='action' action='a-arg-value' subaction='sa-arg-value'
-         output='html'>
+<request type='cgi' action='a-arg-value' subaction='sa-arg-value'
+         lang='en' output='html'>
   <paramList>
     <param name='xx' value=''yyy'/>
@@ -497,46 +253,34 @@
 Receptionist or directly with the MessageRouter. If they communicate with the Receptionist they must use the cgi-args type of request, asking for predefined pages of information.  If they communicate with the MessageRouter directly, they must use the internal message format described in the next section---this is more powerful, but involves more work by the client. Individual services are requested---the results need to be put together by the client.
 
-The arguments used currently are shown in Table~\ref{tab:args}a.
-Other arguments can be specified by the particular service. For example, the
-TextQuery service that the MGPPGDBMService module provides uses the additional
-arguments shown in Table~\ref{tab:args}b.
+The cgi arguments used currently are shown in Table~\ref{tab:args}.
+Other arguments can be specified by  particular actions.. For example, when the query action recieves a list of parameters from the TextQuery service, it creates short names for them and adds them to the global list of cgi-args. 
 
 \begin{table}
 \center{\footnotesize
 \begin{tabular}{llll}
-\cline{2-4}
-(a) & \bf Action & \bf Argument & \bf Typical value \\
-\cline{2-4}
-& p (page) & sa & home, about \\
-& & c (collection) & demo, mgppdemo, ... \\
-& q (query) & sa & text, field, music\\
-& & c  & demo, mgppdemo, ... \\
-& & q (query) & the \\
-& r (resource) & sa & (not used yet) \\
-& & c  & demo, mgppdemo, ... \\
-& & r (resource) & HASH01af33...\\
-& a (applet) & sa & d (display), r (request) \\
-& & c  & demo, mgppdemo, ... \\
-\cline{2-4}\\
-\cline{2-4}
-(b) & \bf Argument & \bf Values \\
-\cline{2-4}
-& s (stem) & 0, 1 \\
-& k (casefold) & 0, 1 \\
-& mm (matchMode) & all, some \\
-& sb (sortBy) & rank, natural \\
-& ql (queryLevel) & \multicolumn{2}{l}{Document, Section, Paragraph} \\
-& md (matchDocs) & 10, 20, ... \\
-\cline{2-4}
+\hline
+\bf Argument & \bf Meaning &\bf Typical values \\
+\hline
+a & action & a (applet), q (query), b (browse), p (page), pr (process) \\
+sa & subaction & home, about (page action)\\
+c & collection or service cluster & demo, build \\
+s & service name & TextQuery, ImportCollection \\
+rt & request type & d (display), r (request), s (status) \\
+ro & request only & 0 or 1 - if set to one, the request is carried out but no processing of the results is done \\
+o & output type & xml, html, wml \\
+l & language & en, fr, zh \\
+d & document id & HASHxxx \\
+r & resource id & ???\\
+id & process handle & an integer identifying a particular process request \\
+\hline
 \end{tabular}}
 \label{tab:args}
-\caption{Arguments that can appear in a Greenstone URL: (a) generic;
-(b) additional arguments for the TextQuery service}
+\caption{Generic rguments that can appear in a Greenstone URL}
 \end{table}
 
 Here is an example message that retrieves the home page in French:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='fr'>
-  <request type='action' action='p' subaction='home' output='html'/>
+<message>
+  <request lang='fr' type='cgi' action='p' subaction='home' output='html'/>
 </message>
 \end{verbatim}\end{footnotesize}\end{quote}
@@ -544,16 +288,20 @@
 This message represents a text query:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request type='action' page='q/text' output='html'>
+<message>
+  <request  lang='en' type='cgi' action='q'  output='html'>
   <paramList>
-    <param name='k' value='0'/>
-    <param name='s' value='1'/>
-    <param name='md' value='10'/>
+    <param name='s' value='TextQuery'/>
     <param name='c' value='demo'/>
-    <param name='q' value='the'/>
+    <param name='rt' value='r'/>
+    <!-- the rest are the service specific params -->
+    <param name='ca' value='0'/> <!-- casefold -->
+    <param name='st' value='1'/> <!-- stem -->
+    <param name='m' value='10'/> <!-- maxdocs -->
+    <param name='q' value='snail'/> <!-- query string -->
   </paramList>
 </message>
 \end{verbatim}\end{footnotesize}\end{quote}
 
+**** UP TO HERE **************
 \subsubsection{Module to module messages}
 
@@ -571,6 +319,6 @@
 The most basic message is ``describe-yourself'', which can be sent to any module in the system. The module responds with a predefined piece of XML, making these requests very efficient.
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request type='describe' to=''/>
+<message>
+  <request lang='en' type='describe' to=''/>
 </message>
 \end{verbatim}\end{footnotesize}\end{quote}
@@ -578,6 +326,6 @@
 An example response from a MessageRouter might look like this:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <response type='describe'>
+<message>
+  <response lang='en' type='describe'>
     <serviceList>
       <service name='CrossCollectionSearch' type='query' />
@@ -625,6 +373,6 @@
 </message>
 
-<message lang='en'>
-  <response type='describe' from='demo' >
+<message>
+  <response lang='en' type='describe' from='demo' >
     <collection name='demo'>
       <serviceList>
@@ -649,11 +397,15 @@
 Parameters have the following format:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<param name='xxx' type='integer|boolean|string|input' default='yyy'/>
-<param name='xxx' type='enum' default='aa'/>
+<param name='xxx' type='integer|boolean|string' default='yyy'/>
+<param name='xxx' type='enum_single|enum_multi' default='aa'/>
   <option name='aa'/><option name='bb'/>...
 </param>
+<param name='xxx' type='multi' occurs='4'>
+ <param .../>
+ <param .../>
+</param>
 \end{verbatim}\end{footnotesize}\end{quote}
 If no default is specified, the parameter is assumed to be mandatory.
-Here are three examples of parameters:
+Here are some examples of parameters:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
 <param name='Case' type='boolean' default='0'/>
@@ -666,13 +418,22 @@
   <option name='stx'/>
 <param>
+
+<!-- this one is for the text box and field list for the simple field query-->
+<param name='simple' type='multi' occurs='4'>
+  <param name='fqv' type='string'/>
+  <param name='fqf' type='enum_single'>
+    <option name='TI'/><option name='AU'/><option name='OR'/>
+  </param>
+</param>
+
 \end{verbatim}\end{footnotesize}\end{quote}
 Here is a message, along with a sample response.
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request type='describe' to='demo/TextQuery'/>
-</message>
-
-<message lang='en'>
-  <response type='describe' from='demo/TextQuery' >
+<message>
+  <request lang='en'  type='describe' to='demo/TextQuery'/>
+</message>
+
+<message>
+  <response lang='en' type='describe' from='demo/TextQuery' >
     <service name='TextQuery' type='query'>
     <paramList>
@@ -704,10 +465,10 @@
 
 <message><request type='configure' to=''>
-<configure action='activate' type='servicesImpl' 
+<configure action='activate' type='serviceRack' 
            name='TranslationServices'/>
 </request></message>
 \end{verbatim}\end{footnotesize}\end{quote}
 
-The first request is used to remove a collection from the running system once it has been physically deleted. The Collection module is removed from the module list, and information about the collection is removed from the collection list XML. The second request is used when the demo collection has either been modified, or has been newly created. The MessageRouter first checks whether a Collection module of that name already exists, and if so deactivates it, as described above.  Then a new Collection module is created and configured, and information added into the XML tree. The final request (re)activates the services provided by the servicesImpl class TranslationServices. The site config file is re-read, and the appropriate element used for configuration of the new servicesImpl object. As for collections, if one already exists, it is deactivated first.
+The first request is used to remove a collection from the running system once it has been physically deleted. The Collection module is removed from the module list, and information about the collection is removed from the collection list XML. The second request is used when the demo collection has either been modified, or has been newly created. The MessageRouter first checks whether a Collection module of that name already exists, and if so deactivates it, as described above.  Then a new Collection module is created and configured, and information added into the XML tree. The final request (re)activates the services provided by the serviceRack class TranslationServices. The site config file is re-read, and the appropriate element used for configuration of the new serviceRack object. As for collections, if one already exists, it is deactivated first.
 
 The response to a configure request is a status or an error message. No data is sent back, just success or error. An example is:
@@ -721,11 +482,12 @@
 Configure requests are only answered by the MessageRouter at this stage. It is possible that other modules may need to respond to these requests also. 
 
-The main type of requests in the system are for services. There are different types of services: query, build\footnote{need new name?}, transform, enrich, extract, accrete. The two most common ones are build and query. Build is for collection formation, query is for the typical use of those collections---querying, browsing, retrieving documents. The other types of service generally enhance the functionality of the first two. They may be used during collection formation: 'accrete' documents by adding them to a collection, 'transform' the documents into a different format, 'extract' information or acronyms from the documents, 'enrich' those documents with the information extracted or by adding new information. They may also be used during querying: 'transform' a query before using it to query a collection, or 'transform' the documents you get back into an appropriate form.
-
-'Query' requests are the most used requests in the system. They are requests for data of some kind, for example, a list of the documents matching a certain criteria, the Title and Author metadata for some specified documents, the text for a specified document, and so on. Each request has a content, and some parameters that specify modifications to the way the query is carried out. So the basic form of a query request is as follows:
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request type='query' to='demo/TextQuery'>
+The main type of requests in the system are for services. There are different types of services: query, browse, retrieve, process, applet. Query services do some kind of search and return a list of documents. Retrieve services can return those documents, metadata about the documents, or other resources. Browse is for browsing lists or hierarchies of documents. process type services are those where the request is for a command to be run. A status code will be returned immediately, and then if the command has not finished, an update of the status can be requested. Applet services are those that run an applet.
+
+  Other possibilities include  transform, enrich, extract, accrete. These types of service generally enhance the functionality of the first set. They may be used during collection formation: 'accrete' documents by adding them to a collection, 'transform' the documents into a different format, 'extract' information or acronyms from the documents, 'enrich' those documents with the information extracted or by adding new information. They may also be used during querying: 'transform' a query before using it to query a collection, or 'transform' the documents you get back into an appropriate form.
+
+The basic structure of a service request is as follows:
+\begin{quote}\begin{footnotesize}\begin{verbatim}
+<message>
+  <request lang='en'  type='query' to='demo/TextQuery'>
     <paramList/>
     <content/>
@@ -734,6 +496,5 @@
 \end{verbatim}\end{footnotesize}\end{quote}
 
-The parameters are name value pairs corresponding to parameters that were specified in the service description sent in response to a describe request. The value of the parameter can be an attribute, or the content of the parameter.
-Attributes can be used for simple strings.
+The parameters are name value pairs corresponding to parameters that were specified in the service description sent in response to a describe request. 
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
@@ -742,14 +503,9 @@
 <param name='index' value='dtx'/>
 \end{verbatim}\end{footnotesize}\end{quote}
-or
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<param name='case'>1</param>
-<param name='maxDocs'>34</param>
-<param name='index'>dtx</param>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-The content of the query is the actual query itself---for a text query, this is the query string. For an image or music query, it would be the image file or music clip. For document retrieval, the identifier of the document is the content.
-
-Responses to query requests contain a content, which is the actual result, along with some metadata about the query\footnote{is this called metadata or something else?}. For instance, a text query on 'snail farming', with the parameter 'maxDocs=10' might return the first 10 documents, and one of the query metadata items would be the total number of documents that matched the query.
+
+Some requests have a content---for document retrieval, the content is the list of documents to retrieve. For metadata retrieval, teh content is the list of documents, and a list of metadata to retrieve for each document.
+
+Responses vary depending on the type of request. 
+Responses to query requests contain a content, which is the actual result, along with some metadata about the query\footnote{is this called metadata or something else?}. For instance, a text query on 'snail farming', with the parameter 'maxDocs=10' might return the first 10 documents, and one of the query metadata items would be the total number of documents that matched the query.\footnote{no metadata about the query result is returned yet.}
 
 The following shows some example query requests and their responses.
@@ -757,6 +513,6 @@
 Find at most 10 Sections containing the word snail (stemmed), returning the results in unsorted order:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request to="mgppdemo/TextQuery" type="query">
+<message>
+  <request lang='en'  to="mgppdemo/TextQuery" type="query">
     <paramList>
       <param name="maxDocs" value="10"/>
@@ -774,12 +530,12 @@
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <response from="mgppdemo/TextQuery" type="query">
+<message>
+  <response lang='en' from="mgppdemo/TextQuery" type="query">
     <content>
-      <resourceList>
-        <resource name="HASH010f073f22033181e206d3b7"/>
-        <resource name="HASH010f073f22033181e206d3b7.2"/>
-        <resource name="HASHac0a04dd14571c60d7fbfd"/>
-      </resourceList>
+      <documentList>
+        <document name="HASH010f073f22033181e206d3b7"/>
+        <document name="HASH010f073f22033181e206d3b7.2"/>
+        <document name="HASHac0a04dd14571c60d7fbfd"/>
+      </documentList>
     </content>
   </response>
@@ -789,12 +545,12 @@
 Give me the Title metadata for these documents:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request to="mgppdemo/MetadataRetrieve" type="query">
+<message>
+  <request lang='en'  to="mgppdemo/MetadataRetrieve" type="retrieve">
     <content>
-      <resourceList>
-        <resource name="HASH010f073f22033181e206d3b7"/>
-        <resource name="HASH010f073f22033181e206d3b7.2"/>
-        <resource name="HASHac0a04dd14571c60d7fbfd"/>
-      </resourceList>
+      <documentList>
+        <document name="HASH010f073f22033181e206d3b7"/>
+        <document name="HASH010f073f22033181e206d3b7.2"/>
+        <document name="HASHac0a04dd14571c60d7fbfd"/>
+      </documentList>
       <metadataList>
         <metadata name="Title"/>
@@ -806,9 +562,9 @@
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <response from="mgppdemo/MetadataRetrieve" type="query">
+<message>
+  <response lang='en' from="mgppdemo/MetadataRetrieve" type="retrieve">
     <content>
-      <resourceList>
-        <resource name="HASH010f073f22033181e206d3b7">
+      <documentList>
+        <document name="HASH010f073f22033181e206d3b7">
           <metadataList>
             <metadata name="Title">Farming snails 1: 
@@ -816,11 +572,11 @@
             </metadata>
           </metadataList>
-        </resource>
-        <resource name="HASH010f073f22033181e206d3b7.2">
+        </document>
+        <document name="HASH010f073f22033181e206d3b7.2">
           <metadataList>
             <metadata name="Title">Learning about snails</metadata>
           </metadataList>
-        </resource>
-        <resource name="HASHac0a04dd14571c60d7fbfd">
+        </document>
+        <document name="HASHac0a04dd14571c60d7fbfd">
           <metadataList>
             <metadata name="Title">Farming snails 2: 
@@ -828,6 +584,6 @@
             </metadata>
           </metadataList>
-        </resource>
-      </resourceList>
+        </document>
+      </documentList>
     </content>
   </response>
@@ -837,10 +593,10 @@
 Give me the text for this document:
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request  to="mgppdemo/ResourceRetrieve" type="query">
+<message>
+  <request lang='en'   to="mgppdemo/DocumentRetrieve" type="retrieve">
     <content>
-      <resourceList>
-        <resource name="HASH010f073f22033181e206d3b7.2"/>
-      </resourceList>
+      <documentList>
+        <document name="HASH010f073f22033181e206d3b7.2"/>
+      </documentList>
     </content>
   </request>
@@ -849,8 +605,8 @@
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <response from="mgppdemo/ResourceRetrieve" type="query">
+<message>
+  <response lang='en' from="mgppdemo/DocumentRetrieve" type="retrieve">
     <content>
-      <resource name="HASH010f073f22033181e206d3b7.2">
+      <document name="HASH010f073f22033181e206d3b7.2">
         <content> 
 &lt;/B&gt;&lt;P ALIGN=&quot;JUSTIFY&quot;&gt;&lt;/P&gt;
@@ -863,5 +619,5 @@
 &lt;/P&gt;....
         </content>
-      </resource>
+      </document>
     </content>
   </response>
@@ -876,6 +632,6 @@
 
 \begin{quote}\begin{footnotesize}\begin{verbatim}
-<message lang='en'>
-  <request type='build' to='build/NewCollection'>
+<message>
+  <request lang='en'  type='process' to='build/NewCollection'>
     <paramList>
       <param name='creator' value='me@home.com'/>
@@ -886,6 +642,6 @@
 </message>
 
-<message lang='en'>
-  <request type='build' to='build/ImportCollection'>
+<message>
+  <request lang='en'  type='process' to='build/ImportCollection'>
     <paramList>
       <param name='collection' value='demo'/>
@@ -907,5 +663,5 @@
 <page>
  <config/>
- <translate/>
+ <display/>
  <request/>
  <response/>
@@ -913,7 +669,7 @@
 \end{verbatim}\end{footnotesize}\end{quote}
 
-There are four main elements in the page: config, translate, request, response. The request is the original request that came into the Receptionist---this is included so that any parameters  can be preset to their previous values, for example, the query options on the query form. The response contains all the data that has been gathered from the system by the action. The other two elements contain extra information needed by XSLT. Config contains run-time variables such as the location of the gsdl home directory, the current site name, the name of the executable that is running (eg library)---these are needed to allow the XSLT to generate correct HTML URLs. Translate contains some of the text strings needed in the interface---these are separate from the XSLT to allow for internationalization.
-
-The following subsections outline, for each action, what data is needed and what requests are generated to send to the system. Following that, Section~\ref{subsec:xslt} describes the config and translate information, and the xslt files.
+There are four main elements in the page: config, translate, request, response. The request is the original request that came into the Receptionist---this is included so that any parameters  can be preset to their previous values, for example, the query options on the query form.\footnote{this should be saved instead in some sort of state saving - if you leave a page and go back you want your parameters to be the same as well}. The response contains all the data that has been gathered from the system by the action. The other two elements contain extra information needed by XSLT. Config contains run-time variables such as the location of the gsdl home directory, the current site name, the name of the executable that is running (eg library)---these are needed to allow the XSLT to generate correct HTML URLs. Display contains some of the text strings needed in the interface---these are separate from the XSLT to allow for internationalization.
+
+The following subsections outline, for each action, what data is needed and what requests are generated to send to the system. Following that, Section~\ref{subsec:xslt} describes the config and display information, and the xslt files.
 
 \subsubsection{Page action}
@@ -926,12 +682,10 @@
 \subsubsection{Query action}
 
-Currently, only text query has been implemented.
-For each page, the service description is requested from the TextQuery service  or the current collection (via a describe request).  This is done every time the query page is
-displayed.\footnote{This information should be cached.} The description includes a list of the parameters available for the query, such as case/stem, max num docs to return, etc. If there is no query
-string specified in the URL, only this information is needed---the request was for the blank query page. 
-If there is a query string specified, i.e. the user has entered a query, a query request to the TextQuery service is sent. This has the query string as content, and all the parameters from the URL in the parameter list. A list of document identifiers
+There are three query services which have been implemented: TextQuery, SimpleFieldQuery, and AdvancedFieldQuery. These are all handled in the same way by query action.
+For each page, the service description is requested from the  service  of the current collection (via a describe request).  This is done every time the query page is
+displayed.\footnote{This information should be cached.} The description includes a list of the parameters available for the query, such as case/stem, max num docs to return, etc. If the request type (rt) parameter is set to d for display, the action only needs to display the form, and this is the only request to the service. Otherwise, the submit button has been pressed, and a query request to the TextQuery service is sent. This has  all the parameters from the URL put into the parameter list. A list of document identifiers
 is returned. A followup query is sent to the MetadataRetrieve service of the collection: the content includes the list of
-documents, with a request for their {\em Title} metadata.  The result is
-transformed using {\em textquery.xsl\/}.
+documents, with a request for their {\em Title} metadata.  The service description and query result are combined into a page of xml, which is
+transformed using {\em basicquery.xsl\/} to produce the html page.
 
 \subsubsection{Applet action}
@@ -1046,7 +800,7 @@
 current library servlet as its value.
 
-\subsubsection{Resource action}
-
-ResourceAction sends a query to the ResourceRetrieve service of the collection requesting the text of the specified document.  At this stage no additional information is obtained, but in future stuff like Title and
+\subsubsection{Document action}
+
+DocumentAction sends a query to the DocumentRetrieve service of the collection requesting the text of the specified document.  At this stage no additional information is obtained, but in future stuff like Title and
 table of contents would be needed to make the display nicer.
 
@@ -1057,175 +811,56 @@
 can override these files by having their own copy of the appropriate
 files. New interfaces have their own directory inside interfaces/. Sites and collections can have a transform directory containing XSLT files. The order in which the XSLT files are looked for is collection, site, current
-interface, default interface. 
+interface, default interface.\footnote{this currently breaks down for remote sites - need to rethink it a bit.}
 
 \subsection{Internationalization}
 
-Internationalization is a bit part of Greenstone3. Language specific text strings are separated out from the rest of the system to allow for easy incorporation of new languages.
-
-At the moment:\footnote{this may change soon, so I haven't 'nice'd this text yet}
-
-Language specific text strings are specified as xml files, named by
-the language code, eg en.xml, fr.xml.
-
-They are located in interfaces/translate. This assumes one set of
-language files per system set up. (or should they be site/interface
-specific??)
-
-A Translate class is used to hold the xml for the languages. The
-Receptionist has a Translate object. It sets the default language to
-be 'en', the current language is whatever a message lang attribute
-specifies.
-
-The translation object internally holds DOM trees for the languages it
-has loaded. It has a mapping between language name and the tree. When
-the default language is set, the appropriate xml file is read in and
-parsed into a DOM tree.
-
-A call to getLanguageTree(lang) returns a DOM element of the form:
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<translate>
-<current><text>.. the actual text elems...</text></current>
-<default><text>.. the actual text elems...</text></default>
-<translate>
-\end{verbatim}\end{footnotesize}\end{quote}
-If the specified lang has not been loaded yet, it will be read into
-memory. Only languages which have been asked for are loaded into
-memory. But once loaded, they stay there. Will need to see how much
-memory this requires once we use full language files.---may need to
-limit the number of cached languages? or maybe only hold two in
-memory, and read them in from file again when a new one is asked for.
-
-The xml files start with the {\em <text>\/} element. The elements are
-organized hierarchically. An example is the following.
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<text>
-<common>
-<nzdl>New Zealand Digital Library</nzdl>
-<aboutpage>about page</aboutpage>
-<search>Search</search>
-<browse>Browse</browse>
-<applet>Applet</applet>
-<home>HOME</home>
-<on>on</on>
-<off>off</off>
-</common>
-<query>
-<queryoptions>Query Options:</queryoptions>
-<params><case><name>Case differences:</name>
-<on>ignore case differences</on>
-<off>upper/lower case must match</off></case>
-<stem><name>Word endings:</name>
-<on>ignore word endings</on>
-<off>whole word must match</off></stem>
-<sortBy><name>Sort results by:</name>
-<rank>rank</rank><natural>none</natural></sortBy>
-<maxDocs><name>Maximum number of documents to return:</name></maxDocs>
-<matchMode><name>Match mode:</name>
-<all>all</all><some>some</some></matchMode>
-<queryLevel><name>Level:</name><Section>Section</Section>
-<Document>Document</Document></queryLevel></params>
-<beginsearch>Begin Search</beginsearch>
-</query>
-</text>
-\end{verbatim}\end{footnotesize}\end{quote}
-Most of the text strings will be specified by the main xml files, but
-some will come from the services/collections. In this case, the lang
-attribute of the message will indicate which language text to return.
-
-Text strings can added to the HTML output in two ways. In the XSLT, we
-know which text strings are needed, eg 'home' for the home link. home
-is in common/home, so we get the text by calling the text template
-with common/home as a param:
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<xsl:call-template name='text'>
-<xsl:with-param name='key'>common/home</xsl:with-param>
-</xsl:call-template>
-\end{verbatim}\end{footnotesize}\end{quote}
-If we want to specify text strings in the xml result (rather than the
-XSLT---would we want to do this?), we can use
-{\footnotesize \verb#<text key='common/home'/>#}.
-{\footnotesize \verb#<xsl:apply-templates select='text'/>#} must then be used when
-processing the parent node.
-
-The template is shown below. Basically, it looks for an appropriate
-element in the current language tree, and if its not found, it looks
-in the default language tree.
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<xsl:template name='text' match='text'>
-<xsl:param name='key'><xsl:value-of select='@key'/></xsl:param>
-<!-- try the current language -->
-<xsl:variable name='path1'>
-ancestor::page/translate/current/text/<xsl:value-of select='$key'/>
-</xsl:variable>
-<xsl:variable name='string1'><xsl:value-of
-select='java:org.apache.xalan.lib.Extensions.evaluate($path1)'/>
-</xsl:variable>
-<xsl:choose><xsl:when test='boolean(string($string1))'>
-<xsl:value-of select='$string1'/></xsl:when>
-<xsl:otherwise>
-<!-- try the default language -->
-<xsl:variable name='path2'>
-ancestor::page/translate/default/text/<xsl:value-of select='$key'/>
-</xsl:variable>
-<xsl:value-of select=
-'java:org.apache.xalan.lib.Extensions.evaluate($path2)'/>
-</xsl:otherwise>
-</xsl:choose>
-</xsl:template>
-\end{verbatim}\end{footnotesize}\end{quote}
-
+Internationalization is a big part of Greenstone3. Language specific text strings are separated out from the rest of the system to allow for easy incorporation of new languages.
+
+Language specific text strings are specified in resource bundle property files. These live in resources/java.
+
+There is a properties file per class, and one per interface. At the moment, we have
+
+GS2MGPPSearch.properties
+GS2MGPPRetrieve.properties etc - the service classes
+
+interface_default.properties. - for the default interface
+
+To add other languages, create eg GS2MGPPSearch_fr.properties.
+
+The interface ones are treated differently from the other ones. The action doesn't know which text strings are needed by a particular transform, so it gets them all out of the properties file, and puts them into an xml $<$display$>$ element - the xslt can get the ones it needs from there.
+xslt could perhaps get the stuff from the properties bundle on the fly using java extension elements - would this be better?
+
+All other class specific text strings are just retrieved one by one as they are needed and added into the xml - for example, the names for query params are retrieved when the service description is created.
 
 \subsection{Collection formation}
 
-
-There is no facility to create collections in GSDL3 yet. There are three
-working servicesImpl classes: MGPPGDBMServices, GSDL2ClassifierServices and PhindServices---these use
-standard collections built with MGPP and gdbm from GSDL2. For
-PhindService, you need to add 'classify phind' to the collect.cfg file
-during building. For the GSDL2ClassifierServices you need to have any other classifiers specified.
-
-To use a collection in GSDL3, build using mgpp in the old greenstone
-(see mgpp\_in\_greenstone.txt in the mgpp/docs directory of either
-gsdl).
-
-Then copy the collection over into the appropriate collect directory,
-and create index/buildConfig.xml (see \ref{subsec:config}). The basic info
-that you need is shown below. Substitute the appropriate values for
-your collection. Only put the phind service one in if you have a phind
-classifier.
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<buildConfiguration>
-  <metadataList>
-    <metadata name="iconCollection">mgppdemo.gif</metadata>
-    <metadata name="colName">mgpp demo</metadata>
-  </metadataList>
-  <servicesImplList>
-    <servicesImpl name="MGPPGDBMServices">
-      <defaultIndex name="tt"/>
-      <defaultLevel name='Section'/>
-    </servicesImpl>
-    <servicesImpl name="PhindServices"/>
-    <servicesImpl name="GSDL2ClassifierServices">
-      <classifierList>
-        <classifier name="CL1">
-          <metadataList>
-            <metadata name="Title">Subject</metadata>
-          </metadataList>
-        </classifier>
-        <classifier name="CL2" >
-          <metadataList>
-            <metadata name="Title">Title</metadata>
-          </metadataList>
-        </classifier>
-      </classifierList>
-    </servicesImpl>	
-  </servicesImplList>
-</buildConfiguration>
-\end{verbatim}\end{footnotesize}\end{quote}
+Greenstone 2 compatible building has been implemented in gsdl3. so far only mgpp collections will work.
+
+Collection construction can be done through the web, using the build servicecluster in localsite. Just sequence through the steps needed. So far, addDocument does not work, so documents need to be manually added to teh import directory.
+
+You need to carry out the following services:
+NewCollection
+- add docs to import directory
+ImportCollection
+BuildCollection
+ActivateCollection
+
+If you want anything other than the default for the config file, you need to add it by hand - there is currently no ConfigureCollection service which would enable you to do this.
+
+Collection building can also be done on the command line:
+
+ConstructCollection -site <site-path> -mode new|import|build|activate [options] <coll-name>
+
+eg
+
+ConstructCollection -site /research/kjdon/home/gsdl3/sites/localsite -mode new -creator kjdon@cs.waikato.ac.nz testcol
+
+the options get passed to the underlying script, - there is no good help message yet.
+
+import and build use gs2 import.pl and buildcol.pl so you can specify any of their options if you like.
+
+Building stuff is in src/java/org/greenstone/gsdl3/build.
+
+CollectionConstructor is the base class for building control. GS2PerlConstructor is the implementation that uses greenstone 2 perl scripts. The building process sends events (ConstructionEvent) to any listeners (ConstructionListener) as important stages happen. You can add one or more listeners to the constructor which will get notified of events.
 
 \section{Details}
@@ -1257,7 +892,9 @@
   & Utility classes \\
 gsdl3/src/java/org/greenstone/gsdl3/collection 
-  & Collection class\\
+  & ServiceCluster and Collection classes\\
 gsdl3/src/java/org/greenstone/gsdl3/comms 
   & Communicator classes, eg SOAP\\
+gsdl3/src/java/org/greenstone/gsdl3/build 
+  & stuff for collection building \\
 gsdl3/src/java/org/greenstone/gsdl3/action 
   & Action classes used by the Receptionist---do the work of displaying the pages\\
@@ -1276,4 +913,14 @@
 gsdl3/lib/java
   & Java jar files\\
+gsdl3/resources 
+ & any resources that may be needed\\
+gsdl3/resources/java
+ & properties files for java resource bundles - used to handle all the language specific text\\
+gsdl3/bin
+  & executable stuff lives here\\
+gsdl3/bin/script
+  & some perl building scripts\\
+gsdl3/bin/linux
+  & linux executables for eg mgpp\\
 gsdl3/comms
   & Put some stuff here for want of a better place---things to do with servers and communication. eg soap stuff, and tomcat servlet container\\
@@ -1305,6 +952,4 @@
 gsdl3/interfaces/default/transforms
   & The XSLT files\\
-gsdl3/interfaces/translate
-  & Language specific stuff---language xml files containing all the text strings go here\\
 \hline
 \end{tabular}}
@@ -1317,5 +962,5 @@
 \newcommand{\gsdlhome}{\begin{footnotesize}{\em \$GSDL3HOME}\end{footnotesize}}
 
-Cuurently, greenstone3 is only available through CVS. The installation procedure has not been automated. Eventually, all that will be needed (hopefully) will be a {\footnotesize \verb#configure, make, make install#} sequence. But for now, all the steps must be done by hand.
+Cuurently, greenstone3 is only available through CVS. The installation procedure has  been automated. 
 
 \subsubsection{Get the source}
@@ -1340,60 +985,34 @@
 
 \noindent If you need it, the password for anonymous CVS access is {\footnotesize \verb#anonymous#}.
-\\
-\\
-\noindent You also need to download the mgpp code - it comes in a separate CVS module.
-
-\noindent I once added a directory for mgpp in gsdl3/packages in cvs---now I can't get
-rid of it, so you need to delete it before you start.
+
+\subsubsection{Compile and install greenstone}\label{subsec:compile}
+
+An install.sh script has been constructed (thanks, Stuart) to compile and install greenstone 3. What you nee to do is:
 
 \begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/packages\\
-rm -r mgpp\\
-cvs co mgpp\\
-\end{tt}\end{footnotesize}
-
-\subsubsection{Compile and install greenstone}\label{subsec:compile}
-
-\noindent From here on, \gsdlhome\  is the absolute path to the top-level directory of the gsdl3 checkout.
-For example, /research/kjdon/gsdl3.
-\\
-\\
-\noindent First, set up your classpath:\\
-\begin{footnotesize}\begin{tt}
-cd \gsdlhome\\
+cd gsdl3
+source setup.bash
+install.bash
 source setup.bash
 \end{tt}\end{footnotesize}
 
-\noindent Note: this step needs to be done once in any xterm window before doing a make or running tomcat. setup.bash sets the environment variables {\footnotesize \verb#CLASSPATH#, \verb#PATH#, \verb#JAVA_HOME#} etc.
-\\
-\\
-\noindent Compile mgpp:\\
+If you want to do greenstone2 compatible building (currently the only type) you need to have greenstone 2 installed, 'source setup.bash' in the top level greenstone 2 directory, then re-'source setup.bash' for greenstone 3. This is to set GSDLHOME for tomcat.
+
+\noindent Note: 'source setup.bash' needs to be done once in any xterm window before doing a make or running tomcat. setup.bash sets the environment variables {\footnotesize \verb#CLASSPATH#, \verb#PATH#, \verb#JAVA_HOME#} etc.
+
+If you want to use SOAP to talk to remote sites, you also need to do the following:
+
 \begin{footnotesize}\begin{tt}
-cd \gsdlhome/packages/mgpp\\
-./configure --prefix \gsdlhome\\
-make\\
-make install\\
+install-soap.bash  
 \end{tt}\end{footnotesize}
 
-\noindent Note: you need to use \gsdlhome\  as the prefix for mgpp's configure at this stage---mgpp has been set up properly, but gsdl3 hasn't.
-
-\noindent Next you need to compile greenstone.
-
-\noindent A jar file is used from tomcat during compilation, so this must be unpacked first.
-\begin{footnotesize}\begin{tt}
-cd \gsdlhome/comms/tomcat/\\
-tar xzvf jakarta-tomcat-4.0.1.tar.gz \\
-\end{tt}\end{footnotesize}
-\\
-\\
-\noindent Do a \verb#make#, then a \verb#make install# in each of the following directories:\\
-\begin{footnotesize}\begin{tt}
-\gsdlhome/src/java/org/greenstone/gdbm\\
-\gsdlhome/src/java/org/greenstone/testing\\
-\gsdlhome/src/java/org/greenstone/gsdl3\\
-\gsdlhome/src/java/org/greenstone/applet/phind
-\end{tt}\end{footnotesize}
-
-\subsubsection{Set up the sample sites}
+Thats it. 
+
+You dont want to run install.bash twice - it adds stuff into files
+
+To update your installation, you can run update.bash - this remakes all the java stuff.
+
+
+\subsubsection{The sample sites}
 
 \noindent There are two greenstone ``sites'' that come with the checkout: localsite, and site1. localsite has several collections, only two of which have any actual data. The third is a dummy collection. site1 has one dummy collection. Each site has a configuration file which specifies the site name,  site-wide services if any, and a list of remote sites to connect to.
@@ -1402,14 +1021,6 @@
 \noindent The collections which do not have data can be looked at but you cant do any queries on them.
 
-\noindent The data comes in tar files, which need to be unpacked:
-
-\begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/sites/localsite/collect/mgppdemo/index/\\
-tar xzvf mgpp-indexfiles.tar.gz\\
-cd ../../chinesedemo/index\\
-tar xzvf chinese-index-files.tar.gz\\
-\end{tt}\end{footnotesize}
-
-\subsubsection{Set up tomcat}
+
+\subsubsection{Tomcat}
 
 \noindent Tomcat is a servlet container. It is used to serve a greenstone site using a servlet.
@@ -1428,48 +1039,5 @@
 \end{verbatim}\end{footnotesize}
 
-\noindent You need to replace {\footnotesize \verb#/research/kjdon/home/gsdl3#} with the correct path for \gsdlhome, in both library servlet entries.
-\\
-\\
-\noindent Next, symbolic links to the sites, interfaces and lib directories need to be set up---this enables tomcat to 'see' files in these directories.
-
-\begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/web\\
-ln -s ../interfaces\\
-ln -s ../sites\\
-ln -s ../lib
-\end{tt}\end{footnotesize}
-
-\noindent The test servlet needs to be compiled: (you need to set up your {\footnotesize CLASSPATH} if you haven't already, see \ref{subsec:compile})\\
-\begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/web/WEB-INF/classes\\
-javac TestServlet.java
-\end{tt}\end{footnotesize}
-\\
-\\
-\noindent Next, one of the scripts that runs tomcat needs to be altered to use our {\footnotesize CLASSPATH}.
-
-\begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/comms/tomcat/jakarta-tomcat-4.0.1
-\end{tt}\end{footnotesize}
-\\
-\\
-\noindent edit {\footnotesize \verb#bin/catalina.sh#}:
-
-\noindent on line 89 add {\footnotesize \$CLASSPATH} to the {\footnotesize CP="...."} line ie. {\footnotesize CP="\$CLASSPATH:..."}---this
-sets up the classpath properly
-\\
-\\
-\noindent Now you need to tell tomcat about the greenstone context:
-\\
-\\
-\noindent edit {\footnotesize \verb#conf/server.xml#}:
-
-\noindent you need to add a context for gsdl servlets---there are other context elements in the xml---this one goes at the same level as those ones.\\
-add the following (putting the correct path for \gsdlhome)
-
-\begin{footnotesize}\begin{tt}
-\noindent <!-- GSDL3 Service -->\\
-<Context path="/gsdl3" docBase="\gsdlhome/web" debug="1" reloadable="true"/>
-\end{tt}\end{footnotesize}
+The file \gsdlhome/comms/tomcat/jakarta/conf/server.xml is the tomcat configuration file. setup.bash adds a context for gsdl servlets - this tells tomcat where to find the web.xml file, and what url (eg /gsdl3) to give it.
 
 \noindent Note: tomcat runs on port 8080 - you can change that if you wish in this file
@@ -1507,32 +1075,7 @@
 \\
 \noindent The SOAP server we use is actually run as a servlet in tomcat. You need to set up SOAP, set up the SOAP server class which will be your service, and then deploy that service.
-\\
-\\
-\noindent Set up SOAP:
-\\
-\\
-\begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/comms/soap\\
-tar xzvf soap-bin-2.2.tar.gz
-\end{tt}\end{footnotesize}
-\\
-\\
-\noindent The context for the SOAP servlet needs to be added to the tomcat server.xml file in the same way that you added the context for gsdl3:
-
-\noindent edit \begin{footnotesize}{\tt \gsdlhome/comms/tomcat/jakarta-tomcat-4.0.1/conf/server.xml}\end{footnotesize}
-
-\noindent add the following (put the proper path for \gsdlhome)
-
-\begin{footnotesize}\begin{tt}
-\noindent <!-- SOAP Service -->\\
-<Context path="/soap" docBase="\gsdlhome/comms/soap/soap-2\_2/webapps/soap"\\
-debug="1" reloadable="true"/>
-\end{tt}\end{footnotesize}
-\\
-\\
-\noindent Next, the class SOAPServer must be altered---the constructor is not allowed any arguments, so it has a path hard coded in it. This is the address of the site that is to be served. In \begin{footnotesize}{\tt \gsdlhome/src/java/org/greenstone/gsdl3/SOAPServer.java}\end{footnotesize}, you need to change the {\footnotesize \verb#site_home#} variable to \begin{footnotesize}{\tt \gsdlhome/sites/localsite}\end{footnotesize} (using the absolute path).
-\\
-\\
-\noindent The SOAPServer service now needs to be deployed. If tomcat is not running, start it up (see \ref{subsec:runtomcat}).
+
+this is done by install-soap.bash.
+You can also deploy a service through the website.  If tomcat is not running, start it up (see \ref{subsec:runtomcat}).
 
 \noindent The SOAP servlet can be accessed at \begin{footnotesize}{\tt http://localhost:8080/soap}\end{footnotesize}. You should see a welcome page. Click on ``Run the admin client''. This enables you to list, deploy and undeploy SOAP services.
