Index: /trunk/gsdl3/docs/manual/manual.tex
===================================================================
--- /trunk/gsdl3/docs/manual/manual.tex	(revision 4161)
+++ /trunk/gsdl3/docs/manual/manual.tex	(revision 4162)
@@ -3,4 +3,9 @@
 \hyphenation{Message-Router Text-Query}
 
+\newenvironment{gsc}% Greenstone text bits
+{\begin{footnotesize}\begin{tt}}%
+{\end{tt}\end{footnotesize}}
+ 
+\newcommand{\gst}[1]{{\footnotesize \tt #1} }
 \begin{document}
 
@@ -47,51 +52,74 @@
 Native Interface) will be used to communicate with these.
 
-
-\section{Architecture}
-
-This section is covered by the paper: An agent based architecture for dynamic digital library construction and configuration. Either cut and paste it in here, or link to the text?? or have two separate docs. dont want to have to maintain two separate versions of the same thing.
-
-\section{Greenstone Implementation}
-\label{sec:impl}
-
-\subsection{Configuring Greenstone}
-\label{subsec:config}
-
-Greenstone3 involves several different kinds of configuration files, all
-expressed in XML. Each site has a configuration file that binds parameters for
-the site, {\em siteConfig.xml}.  Each collection has two configuration files, {\em collectionConfig.xml} and {\em buildConfig.xml\/}, that give metadata for the
-collection.\footnote{These replace {\em collect.cfg} and {\em build.cfg} in
+A description of the general design and architecture of Greenstone3 is covered by the document ``The design of Greenstone3: An agent based dynamic digital library'' (design-2002.ps, in the gsdl3/docs/manual directory).
+
+\section{System modules}\label{sec:modules}
+
+A Greenstone3 'library' system consists of many components... Figure~\ref{fig:local} shows they fit together in a stand-alone system.
+
+\begin{figure}[t]
+  \centering
+  \includegraphics[width=4in]{local} %5.8
+  \caption{A simple stand-alone site.}
+  \label{fig:local}
+\end{figure}
+
+
+{\em MessageRouter}: this is the central module for a site. It controls the site, loading up all the collections, clusters, communicators needed. All messages pass through the MessageRouter. Communication between remote sites is always done between MessageRouters, one for each site.
+
+{\em Collection and ServiceCluster}: these are very similar. They both provide some metadata about the collection/cluster, and a list of services. The services are provided by ServiceRack objects that the collection/cluster loads up. A Collection is a specific type of ServiceCluster. A ServiceCluster groups services that are related conceptually, eg all the building services may be part of a cluster. What is part of a cluster is specified by the site config file. A Collection's services are grouped by the fact that they all operate on some common data---the documents in the collection.
+Functionally Collection and ServiceCluster are very similar, but conceptually, and to the user, they are quite different.
+
+{\em ServiceRack}: these provide one or more services - they are grouped into a single class purely for code reuse, or to avoid instantiating the same objects several times. For example, MGPP searching services all need to have the index loaded into memory.
+
+{\em Communicator/Server}: these facilitate communication between remote modules. For example, if you want MR1 to talk to MR2, you need a Communicator-Server pair. The Server sits on top of MR2, and MR1 talks to the Communicator. Each communication type needs a new pair. So far we have only been using SOAP, so we have a SOAPCommunicator and a SOAPServer.
+
+{\em Receptionist}: this is the point of contact for the 'front end'. It is pretty much a router to actions, but it also handles anything that is common to all pages, such as creating some XML data for the pages.
+
+{\em Actions}: these do the job of creating the 'pages'. There is a different action for each type of page, for example PageAction handles semi-static pages, QueryAction handles queries, DocumentAction displays documents. They know a little bit about specific service types. Based on the 'cgi' arguments passed in to them, they construct requests for the system, and put together the responses into data for the page. This data is transformed (currently into HTML) using XSLT. The various actions are described in  more detail in Section~\ref{sec:pagegen}.
+
+
+\section{Configuration}\label{sec:config}
+
+Initial Greenstone3 system configuration  is determined by a set of configuration files, all expressed in XML. Each site has a configuration file that binds parameters for
+the site, \gst{siteConfig.xml}.  Each collection has two configuration files, \gst{collectionConfig.xml} and \gst{buildConfig.xml}, that give metadata and other information for the
+collection.\footnote{\gst{siteConfig.xml} is new for Greenstone3, while \gst{collectionConfig.xml} and \gst{buildConfig.xml} replace \gst{collect.cfg} and \gst{build.cfg} in
 Greenstone2.}  The first includes user-defined metadata for the collection,
 such as its name and the {\em About this collection} text; and also gives
 instructions on how the collection is to be built.  The second is produced by
 the build-time process and includes any metadata that can be determined
-automatically.\footnote{Currently only the buildConfig.xml file is used - collections are built using gs2 style building and therefore use the old collect.cfg.}
-
-\subsubsection{Site configuration file}
-
-The file {\em siteConfig.xml} specifies the URI for the site ({\em
-localSiteName\/}), any services or service clusters provided by the site that are not connected
-with a particular collection (for example, translation services, or collection building), and a list of
+automatically. It also includes configuration information for any serviceRacks needed by the collection.
+
+The configuration files are read in when the system is initialised, and their contents are cached in memory. This means that changes made to these files once the system is running will have no effect. There are a series of cgi-type commands that can be sent to the library to induce reconfiguration of different modules, including reloading the whole site. This removes the need to shutdown and restart the system to reflect these changes. These commands are described in Section~\ref{sec:runtime-config}.
+
+\subsection{Site configuration file}\label{sec:siteconfig}
+
+The file \gst{siteConfig.xml} specifies the URI for the site (\gst{localSiteName}), the HTTP address for site resources (\gst{httpAddress}), any ServiceClusters that the site provides (for example, collection building), any ServiceRacks that do not belong to a cluster or collection, and a list of
 known external sites to connect to.  Collections are not specified in the site
 configuration file, instead they are determined by the contents of the site's
 collections directory.
 
-Here is a configuration file for a rudimentary site with no site-wide services,
-which does not connect to any external sites.\footnote{should the code be tolerant of missing elements? or do we require empty elements?}
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<config>
+The HTTP address is used for retrieving resources from a site outside the XML protocol. Because a site is HTTP accessible, any files (e.g. images) belonging to that site or to its collections can be specified in the HTML of a page by a URL. This avoids having to retrieve these files from a remote site via the XML protocol\footnote{Currently, sites live inside the Tomcat gsdl3 root context, and therefore all their content is accessible over HTTP via the Tomcat address. We need to see if parts can be restricted. Also, if we use a different protocol, then resources from remote sites may need to come through the XML. Also, if we are running locally without using Tomcat, we may want to get them via file:// rather than http://.}.
+ 
+The first example in Figure~\ref{fig:siteconfig} shows a site configuration file for a rudimentary site with no site-wide services,
+which does not connect to any external sites. The second example is for a site with one site-wide service cluster - a collection building cluster.  It also connects to the first site using SOAP.
+These two sites are running on the same machine. For site gsdl1 to talk to site localsite, a SOAP server must be run for localsite. The address of the SOAP server, in this case, is \gst{http://localhost:8090/soap/servlet/rpcrouter}.
+
+
+\begin{figure}
+\begin{gsc}\begin{verbatim}
+<siteConfig>
   <localSiteName value="org.greenstone.localsite"/>
+  <httpAddress value="http://localhost:8090/gsdl3/sites/localsite"/>
   <serviceClusterList/>
   <serviceRackList/>
   <siteList/>
-</config>
-\end{verbatim}\end{footnotesize}\end{quote}
-The following configuration file is for a site with one site-wide service cluster - a collection building cluster.  It also connects to the previous site using SOAP.
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<config>
+</siteConfig>
+\end{verbatim}\end{gsc}
+
+\begin{gsc}\begin{verbatim}
+<siteConfig>
   <localSiteName value="org.greenstone.gsdl1"/>
-  <serviceRackList/>
-    <servicesImpl name="TranslationServices"/>
-  </servicesImplList>
+  <httpAddress value="http://localhost:8090/gsdl3/sites/gsdl1"/>
   <serviceClusterList>	
     <serviceCluster name="build">
@@ -108,15 +136,89 @@
   <siteList>
     <site name="org.greenstone.localsite"
-      address="http://localhost:8080/soap/servlet/rpcrouter"
+      address="http://localhost:8090/soap/servlet/rpcrouter"
       type="soap"/>
   </siteList>
-</config>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-These two sites are running on the same machine. For site1 to talk to localsite, a SOAP server must be run for localsite. The address of the SOAP server, in this case, is "http://localhost:8080/soap/servlet/rpcrouter"
-
-\subsubsection{Building configuration file}
-
-The file {\em buildConfig.xml} contains all metadata and other information about the collection that can
+</siteConfig>
+\end{verbatim}\end{gsc}
+\caption{Two sample site config files}
+\label{fig:siteconfig}
+\end{figure}
+
+
+
+\subsection{Collection configuration file}\label{sec:collconfig}
+
+The collection configuration file is where the collection designer (eg a librarian) decides what form the collection should take. This includes the collection metadata such as title and description, and also includes what indexes and browsing structures should be built. The format of \gst{collectionConfig.xml} is still under consideration. However, Figure~\ref{fig:collconfig}
+here is an example as it is at present.
+
+\begin{figure}
+\begin{gsc}\begin{verbatim}
+<collectionConfig xmlns:gsf="http://www.greenstone.org/
+                         configformat">	
+  <metadataList>	
+    <metadata name="colName" lang="en">greenstone mgpp demo
+    </metadata>
+    <metadata name="colDescription" lang="en">This is a 
+      demonstration collection for the Greenstone digital 
+      library software. It contains a small subset (11 books) 
+      of the Humanity Development Library.</metadata>
+    <metadata name="colDescription" lang="fr">C'est une 
+      collection pour demonstration du logiciel Greenstone.
+      Elle contient une petite partie du projet de bibliotheques 
+      humanitaires et de developpement (11 livres).</metadata>
+    <metadata name="colIcon">mgppdemo.gif</metadata>
+  </metadataList>
+  <search type='mgpp'>
+    <index name="tt" content="text,metadata" 
+			level="Document,Section">
+      <displayName lang="en">books</displayName>
+    </index>
+    <format>	
+      <gsf:template match="documentNode">
+	<td><gsf:link><gsf:metadata name="Title"/>(<gsf:metadata 
+		name="Source"/>)</gsf:link></td>
+      </gsf:template>
+    </format>
+  </search>
+  <browse>
+    <classifier name="CL1" type="Hierarchy" content="Subject" 
+	level="Document">
+      <option name="hfile" value="sub.txt"/>
+      <option name="sort" value="Title"/>
+    </classifier>
+    <classifier name="CL2" type="AZList" content="Title" 
+	level="Document">
+      <displayName lang='en'>all titles</displayName>
+      <format>
+        <gsf:template match="classifierNode">
+	  <td><gsf:link type="classifier"><gsf:metadata name="Title"/>
+	    </gsf:link></td>
+ 	</gsf:template>
+      </format>
+    </classifier>
+    <classifier name="CL3" type="List" content="Keyword" 
+	level="Document">
+      <format>
+	<gsf:template match="documentNode"><td><gsf:link>
+	<gsf:metadata name="Keyword"/></gsf:link></td></gsf:template>
+      </format>
+    </classifier>
+    <classifier type="Phind" content="text" level="Section"/>      
+  </browse>
+</collectionConfig>
+\end{verbatim}\end{gsc}
+\caption{Sample collectionConfig.xml file}
+\label{fig:collconfig}
+\end{figure}
+
+The \gst{<metadataList>} element specifies some collection metadata, such as name and description. These metadata elements can be specified in different languages. The configuration file should be encoded in utf-8. 
+The \gst{<search>} element specifies what type of indexer to use, and what indexes to build. A \gst{<format>} element is used to customize what each document entry in a results list suold look like.
+The \gst{<browse>} element specifies what browsing structures should be created over the documents. Again, \gst{<format>} elements are used to customize items in teh hierarchy, both classifier nodes, and document entries. Section~\ref{sec:colldesign} looks at the collection configuration file in more detail.
+
+There is also a need for a descripiton of how documents should be displayed. For example, whether a table of contents is needed, what metadata to display, and whether or not the text should be displayed. This will probably be in an element such as \gst{<documentDisplay>}. 
+
+\subsection{Building configuration file}\label{sec:buildconfig}
+
+The file \gst{buildConfig.xml} contains the metadata and other information about the collection that can
 be determined automatically when building the collection, such as the number of
 documents it contains.  It also includes a list of serviceRack classes that are
@@ -124,28 +226,33 @@
 collection.  The serviceRack names are Java classes that are loaded
 dynamically at runtime. Any information inside the serviceRack element is
-specific to that service---there is no set format.  Here is an example:
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-
-<buildConfig>
+specific to that service---there is no set format. Figure~\ref{fig:buildconfig} shows an example. This config file specifies that the collection should load up 3 ServiceRacks: GS2MGPPRetrieve,  GS2MGPPSearch, and PhindPhraseBrowse. The contents of each \gst{<serviceRack>} element are passed to the appropriate ServiceRack objects for configuration.
+
+
+\begin{figure}
+\begin{gsc}\begin{verbatim}
+<buildConfig xmlns:gsf="www.greenstone.org/format" >
   <metadataList>
     <metadata name="numDocs">11</metadata>
-    <metadata name="colIcon">mgppdemo.gif</metadata>
-    <metadata name="colName">Greenstone demo collection</metadata>
-    <metadata name="colDescription">This is a demonstration 
-collection for the Greenstone digital library software. It 
-contains a small subset  of the Humanitarian and Development 
-Libraries.</metadata>
+    <metadata name="documentMetadata"><element name="Title"/>
+        <element name="Subject"/><element name="Organization"/>
+        <element name="URL"/></metadata>
   </metadataList>
   <serviceRackList>
     <serviceRack name="GS2MGPPRetrieve">
       <defaultLevel name="Section"/>
-	<!-- something list this should be used to advertise 
-what metadata the collection has available to be retrieved - 
-however, it is not used yet -->
-      <metadataList> 
-        <element name="Title"/><element name="Subject"/>
-	<element name="Organization"/><element name="URL"/>
-      </metadataList>
+      <levelList>
+        <level name="Document"/>
+        <level name="Section"/>
+      </levelList>
+      <classifierList>
+        <classifier name="CL1" content="Subject" 
+          documentInterleave="true" orientation='vertical'/>
+        <classifier name="CL2" content="Title" 
+          documentInterleave="false" orientation='horizontal'/>
+        <classifier name="CL4" content="Organisation" 
+          documentInterleave="true" orientation='vertical'/>
+        <classifier name="CL5" content="Keyword" 
+          documentInterleave="true" orientation='vertical'/>
+      </classifierList>
     </serviceRack>
     <serviceRack name="GS2MGPPSearch">
@@ -161,73 +268,79 @@
       </indexList>
       <fieldList>
-        <field name="TX"/><field name="SU"/><field name="TI"/>
+        <field shortname="TX" name="TextOnly"/>
+        <field shortname="SU" name="Subject"/>
+        <field shortname="TI" name="Title"/>
       </fieldList>
     </serviceRack>
     <serviceRack name="PhindPhraseBrowse"/>
-    <serviceRack name="GS2Browse">
-      <classifierList>
-        <classifier name="CL1"><metadataList>
-	   <metadata name="Title">Subject</metadata>
-	</metadataList></classifier>
-        <classifier name="CL2" ><metadataList>
-	   <metadata name="Title">Title</metadata>
-        </metadataList></classifier>
-	<classifier name="CL4"><metadataList>
-          <metadata name="Title">Organization</metadata>
-        </metadataList></classifier>
-	<classifier name="CL5" ><metadataList>
-          <metadata name="Title">Keyword</metadata>
-        </metadataList></classifier>
-      </classifierList>
-    </serviceRack>
   </serviceRackList>
-</buildConfig>   
-\end{verbatim}\end{footnotesize}\end{quote}
-Note: because {\em collectionConfig.xml} is not used yet, the {\em colIcon}, {\em colDescription}
-and {\em colName} metadata elements have been specified here.
-
-\subsubsection{Collection configuration file}
-
-The format of {\em collectionConfig.xml} has not yet been defined.
-
-\subsubsection{Starting up}
+</buildConfig>
+\end{verbatim}\end{gsc}
+\caption{Sample buildConfig.xml file}
+\label{fig:buildconfig}
+\end{figure}
+
+ 
+\subsection{Start up configuration}\label{sec:startup-config}
 
 We use the Tomcat web server, which operates either stand-alone in a test mode
 or in conjunction with the Apache web server.  The Greenstone LibraryServlet
-class is loaded by Tomcat  and the servlet's {\em init()} method is called.  Each time a
-{\em get\/}/{\em put\/}/{\em post} (etc.) is used, a new thread is started and
-{\em doGet()\/}/{\em doPut()\/}/{\em doPost()} (etc.) is called.
-
-The {\em init()} method creates a new Receptionist and a new instance of the
+class is loaded by Tomcat  and the servlet's \gst{init()} method is called.  Each time a
+\gst{get/put/post} (etc.) is used, a new thread is started and
+\gst{doGet()/doPut()/doPost()} (etc.) is called.
+
+The \gst{init()} method creates a new Receptionist and a new 
 MessageRouter. The appropriate system variables are set in each (interface
-name, site name, etc.) and then {\em configure()} is called. A MessageRouter
+name, site name, etc.) and then \gst{configure()} is called. A MessageRouter
 reference is given to the Receptionist. The servlet then communicates only with
 the Receptionist, not with the MessageRouter.
 
 The Receptionist loads up all the different Action classes. A
-static list is used initially, and other Actions may be loaded on the fly as needed.
-
-The MessageRouter reads in its site configuration file {\em siteConfig.xml}. This
-lists the ServiceRack classes that need to be loaded, and lists any sites that need
-to be connected to.  It looks inside the {\em collect} directory which contains
-all the site's collections and loads up a Collection object for each valid
-collection found.
-
-The Collection object reads its {\em buildConfig.xml} and {\em collectionConfig.xml}
+static list is used initially, and other Actions may be loaded on the fly as needed. Actions are added to a map, with shortnames for keys. Eg the QueryAction is added with key 'q'. The Actions are passed the MessageRouter reference too.
+
+The MessageRouter reads in its site configuration file \gst{siteConfig.xml}. This
+lists the ServiceRack and ServiceCluster classes that need to be loaded and  any sites that need
+to be connected to.  
+It has a module map that maps names to objects. This is used for routing the messages. It also keeps small chunks of XML---serviceList, collectionList, clusterList and siteList. These are what get returned in response to a describe request (see Section~\ref{sec:describe}.).
+Each ServiceRack specified in the config file is created, then queried for its list of services. Each service name is added to the map, pointing to the ServiceRack object. Each service is added to the serviceList. After this stage, ServiceRacks are transparent to the system, and each service is treated as a separate module.
+ServiceClusters are created and passed the \gst{<serviceCluster>} element for configuration. They are added to the map as is, with the cluster name as a key. A serviceCluster is also added to the serviceClusterList.
+For each site specified, the MessageRouter creates an appropriate type Communicator object. Then is tries to get the site description. If teh server for teh remote site is up and running, this should  be successful. The site will be added to the map with its site name as a key. The sites collections, services and clusters will also be added into the static lists. 
+
+The MessageRouter also looks inside the site's \gst{collect} directory loads up a Collection object for each valid collection found.
+
+The Collection object reads its \gst{buildConfig.xml} and \gst{collectionConfig.xml}
 files, determines the metadata, and loads ServiceRack classes based on the 
-names specified in {\em buildConfig.xml\/}. The {\footnotesize \verb#<ServiceRack>#} XML element is passed to the object to be used in  configuration.
-
-\section{System messages}
+names specified in \gst{buildConfig.xml\/}. The \gst{<ServiceRack>} XML element is passed to the object to be used in configuration. The collectionConfig.xml contents are also passed in to the ServiceRacks. Any format or display information that the services need must be extracted from the collection config file.
+Collection objects are added to teh module map with their name as a key, and also a collection element is added into teh collectionList xml.
+
+\subsection{Run-time (re)configuration}\label{sec:runtime-config}
+
+The startup configuration reads in teh various config files and loads up quite a lot of XML into memory. This avoids having to read in files all the time. However, this means that any changes to these files will have no effect in the system. So some run-time reconfiguration options are provided.
+
+Currently there are commands to reconfigure the entire site---i.e. the MessageRouter repeats the whole of its startup initialisation. 
+
+***TODO***
+whats available, whats not. show URLS, refer to system messages in next section
+
+\section{System messages}\label{sec:messages}
+
+for each type of message, show the basic elements, then some example messages.
+Lists must only have the same elements in them.
 
 Once the system is up and running (the configuration
-process described in Section~\ref{subsec:config} has been carried out), it is passing messages back and forth. All modules communicate via message passing.
- First, we examine the basic message
-formats, then how the system creates and responds to the messages.
+process described in Section~\ref{sec:startup-config} has been carried out), it is passing messages back and forth. All modules communicate via message passing.
+
+First, we look at how messages originate, and how they flow in the system. Then, we examine the basic message
+format, and look at the different types of messages. 
+
+\subsection{Message flow}
+
+\subsection{Basic format}
 
 All messages are enclosed in
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<message>
-\end{verbatim}\end{footnotesize}\end{quote}
-Messages contain either {\em <request>\/} or {\em <response>\/} elements--- a single message may contain multiple requests. Each {\em <request>\/} (and {\em <response>\/}?) has a language attribute, of the form ``lang='xx'''.
+\begin{quote}\begin{gsc}\begin{verbatim}
+<message>
+\end{verbatim}\end{gsc}\end{quote}
+Messages contain either \gst{<request>} or \gst{<response>} elements--- a single message may contain multiple requests. Each \gst{<request>} (and \gst{<response>}?) has a language attribute, of the form \gst{lang='xx'}.
 The language attribute is used by the XSLT to determine the language currently
 being used by the user interface.  Virtually all messages contain text strings,
@@ -239,24 +352,24 @@
 This section describes the two message formats. The following section looks at how the front-end (Receptionist plus Actions) responds to the URL-type messages, and creates internal xxx-type\footnote{are there good names to distinguish the two types of messages?} messages to pass into the system.
 
-\subsubsection{Servlet to Receptionist messages}\label{subsec:url-type}
+\subsection{cgi-type messages}\label{sec:cgi}
 
 Servlet to Receptionist messages are requests for a 'page' of data---for example, the home page for a site; the query page for a collection; the text of a document. They contain, in XML, a representation of the arguments in a
-Greenstone URL.  The two main arguments are {\em a} (action) and {\em sa}
-(subaction).\footnote{The {\em sa} replaces Greenstone's old {\em p} arg for
+Greenstone URL.  The two main arguments are \gst{a} (action) and \gst{sa}
+(subaction).\footnote{The \gst{sa} replaces Greenstone's old \gst{p} arg for
 the page action, and is new for other actions.  For example, a text query could
-be encoded as {\em a=q \& sa=text\/}.}  All other arguments are treated as
+be encoded as \gst{a=q \& sa=text\/}.}  All other arguments are treated as
 parameters.
 
 Here is the XML representation of the arguments:
 
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <request type='cgi' action='a-arg-value' subaction='sa-arg-value'
          lang='en' output='html'>
   <paramList>
-    <param name='xx' value=''yyy'/>
+    <param name='xx' value='yyy'/>
     <param name=...
   </paramList>
 </request>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 The receptionist routes the message to the appropriate action.  The output
 field is used to indicate what type of output to return. The actions do not
@@ -278,4 +391,5 @@
 \hline
 a & action & a (applet), q (query), b (browse), p (page), pr (process) \\
+& & s (system)\\
 sa & subaction & home, about (page action)\\
 c & collection or  & demo, build \\
@@ -285,25 +399,26 @@
 ro & request only & 0 or 1 - if set to one, the request is carried out \\
 & & but no processing of the results is done \\
+& & currently only used in process actions \\
 o & output type & xml, html, wml \\
-l & language & en, fr, zh \\
+l & language & en, fr, zh ...\\
 d & document id & HASHxxx \\
 r & resource id & ???\\
-id & process handle & an integer identifying a particular process request \\
+pid & process handle & an integer identifying a particular process request \\
 \hline
 \end{tabular}}
+\caption{Generic arguments that can appear in a Greenstone URL}
 \label{tab:args}
-\caption{Generic rguments that can appear in a Greenstone URL}
 \end{table}
 
 Here is an example message that retrieves the home page in French:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='fr' type='cgi' action='p' subaction='home' 
 	output='html'/>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 This message represents a text query:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request  lang='en' type='cgi' action='q'  output='html'>
@@ -319,5 +434,5 @@
   </paramList>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 \subsubsection{Module to module messages}
@@ -326,21 +441,22 @@
 information from one module to another, for example from an Action to the
 MessageRouter module, and from that module to a service module.  Requests have
-a {\em to} attribute and responses have {\em from\/}.  These are addresses used
-by routing modules.  For example {\em to='site1/site2/demo/TextQuery'} routes a
-message to a MessageRouter ({\em site1\/}), from there to another MessageRouter
-({\em site2\/}), from there to a collection ({\em demo\/}), and from there to a
-particular service ({\em TextQuery\/}).
+a \gst{to} attribute and responses have \gst{from}.  These are addresses used
+by routing modules.  For example \gst{to='site1/site2/demo/TextQuery'} routes a
+message to a MessageRouter (\gst{site1}), from there to another MessageRouter
+(\gst{site2}), from there to a collection (\gst{demo}), and from there to a
+particular service (\gst{TextQuery}).
 
 Each request asks for a description of a single module, or requests a particular service. Unlike the first type of message which requests pre-defined types of pages, these internal requests can ask for any functionality available in the system.
 
+\subsection{'describe'-type messages}\label{sec:describe}
 The most basic message is ``describe-yourself'', which can be sent to any module in the system. The module responds with a predefined piece of XML, making these requests very efficient.
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='en' type='describe' to=''/>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-If the {\em to} field is empty, the request is answered by the first module that it is passed to.
+\end{verbatim}\end{gsc}\end{quote}
+If the \gst{to} field is empty, the request is answered by the first module that it is passed to.
 An example response from a MessageRouter might look like this:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <response lang='en' type='describe'>
@@ -362,8 +478,8 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 This MessageRouter has one site-wide service, a cross-collection searching service. It
-communicates with one site, {\em org.greenstone.gsdl1\/}.  It is aware of four
-collections.  One of these, {\em myfiles\/}, belongs to it; the other three are
+communicates with one site, \gst{org.greenstone.gsdl1}.  It is aware of four
+collections.  One of these, \gst{myfiles}, belongs to it; the other three are
 available through the external site.  One of those collections is actually from
 a further external site.
@@ -371,6 +487,6 @@
 It is possible to ask just for a specific part of the information provided by a
 describe request, rather than the whole message.  For example, these two
-messages get the {\em collectionList} and the {\em siteList} respectively:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+messages get the \gst{collectionList} and the \gst{siteList} respectively:
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message lang='en'>
   <request type='describe' to='' info='collectionList'/>
@@ -380,10 +496,10 @@
   <request type='describe' to='' info='siteList'/>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 When a collection is asked to describe itself, what is returned is all of the 
 collection specific metadata and a list of services.  For example, here is such
 a message, along with a sample response.
 
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message lang='en'>
   <request type='describe' to='demo'/>
@@ -408,11 +524,11 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-A {\em describe} request sent to a service returns a list of parameters that
+\end{verbatim}\end{gsc}\end{quote}
+A \gst{describe} request sent to a service returns a list of parameters that
 the service accepts, and describes the content type for the request and
 response.
 
 Parameters have the following format:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <param name='xxx' type='integer|boolean|string' default='yyy'/>
 <param name='xxx' type='enum_single|enum_multi' default='aa'/>
@@ -423,8 +539,8 @@
  <param .../>
 </param>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 If no default is specified, the parameter is assumed to be mandatory.
 Here are some examples of parameters:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <param name='Case' type='boolean' default='0'/>
 
@@ -446,7 +562,7 @@
 </param>
 
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 Here is a message, along with a sample response.
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='en'  type='describe' to='demo/TextQuery'/>
@@ -466,13 +582,14 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 So far, we have only looked at ``describe'' requests. These can be asked of any module. Other requests are ``configure'' requests, and requests for services.
 
-``Configure'' requests are used to tell the MessageRouter to update its cached information and activate or deactivate other modules. For example, the MessageRouter has a set of Collection modules that it can talk to. It also holds some XML information about those collections---this is returned when a request for a collection list comes in. If a collection is deleted or modified, or a new one created, this information may need to change, and the list of available modules may also change.
-
-So far, we have {\em activate} and {\em deactivate} configure requests.
+\subsection{'system'-type messages}
+``System'' requests are used to tell the MessageRouter or a Collection or ServiceCluster to update its cached information and activate or deactivate other modules. For example, the MessageRouter has a set of Collection modules that it can talk to. It also holds some XML information about those collections---this is returned when a request for a collection list comes in. If a collection is deleted or modified, or a new one created, this information may need to change, and the list of available modules may also change.
+
+So far, we have \gst{activate} and \gst{deactivate} configure requests.
 Some examples are as follows.
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message><request type='configure' to=''>
 <configure action='deactivate' type='collection' name='demo'/>
@@ -487,18 +604,23 @@
            name='TranslationServices'/>
 </request></message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 The first request is used to remove a collection from the running system once it has been physically deleted. The Collection module is removed from the module list, and information about the collection is removed from the collection list XML. The second request is used when the demo collection has either been modified, or has been newly created. The MessageRouter first checks whether a Collection module of that name already exists, and if so deactivates it, as described above.  Then a new Collection module is created and configured, and information added into the XML tree. The final request (re)activates the services provided by the serviceRack class TranslationServices. The site config file is re-read, and the appropriate element used for configuration of the new serviceRack object. As for collections, if one already exists, it is deactivated first.
 
 The response to a configure request is a status or an error message. No data is sent back, just success or error. An example is:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message><response from='' type='configure'>
   <status>demo collection activated</status>
 </response></message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 \footnote{this format not properly defined yet}
 
 Configure requests are only answered by the MessageRouter at this stage. It is possible that other modules may need to respond to these requests also. 
 
+\subsection{'process'-type messages}
+
+divide this up into service types: query, retrieve (metadata, structure, content), process, applet, enrich, browse...
+show basic structure, then more detailed format for each subtype
+
 The main type of requests in the system are for services. There are different types of services: query, browse, retrieve, process, applet. Query services do some kind of search and return a list of documents. Retrieve services can return those documents, metadata about the documents, or other resources. Browse is for browsing lists or hierarchies of documents. process type services are those where the request is for a command to be run. A status code will be returned immediately, and then if the command has not finished, an update of the status can be requested. Applet services are those that run an applet.
 
@@ -506,24 +628,26 @@
 
 The basic structure of a service request is as follows:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='en'  type='query' to='demo/TextQuery'>
     <paramList/>
-    <content/>
+    other elements...
   </request>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 The parameters are name value pairs corresponding to parameters that were specified in the service description sent in response to a describe request. 
 
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <param name='case' value='1'/>
 <param name='maxDocs' value='34'/>
 <param name='index' value='dtx'/>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-Some requests have a content---for document retrieval, the content is the list of documents to retrieve. For metadata retrieval, teh content is the list of documents, and a list of metadata to retrieve for each document.
+\end{verbatim}\end{gsc}\end{quote}
+
+Some requests have other content---for document retrieval, this would be a list of documents to retrieve. For metadata retrieval, the content is the list of documents, and a list of metadata to retrieve for each document.
 
 Responses vary depending on the type of request. 
+
+\subsubsection{'query'-type services}
 Responses to query requests contain a content, which is the actual result, along with some metadata about the query\footnote{is this called metadata or something else?}. For instance, a text query on 'snail farming', with the parameter 'maxDocs=10' might return the first 10 documents, and one of the query metadata items would be the total number of documents that matched the query.\footnote{no metadata about the query result is returned yet.}
 
@@ -531,7 +655,7 @@
 
 Find at most 10 Sections containing the word snail (stemmed), returning the results in unsorted order:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
-<message>
-  <request lang='en'  to="mgppdemo/TextQuery" type="query">
+\begin{quote}\begin{gsc}\begin{verbatim}
+<message>
+  <request lang='en'  to="mgppdemo/TextQuery" type="process">
     <paramList>
       <param name="maxDocs" value="10"/>
@@ -542,30 +666,28 @@
       <param name="index" value="t0"/>
       <param name="case" value="0"/>
+      <param name="query" value="snail"/>
     </paramList>
-    <content>snail</content>
   </request>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\end{verbatim}\end{gsc}\end{quote}
+
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <response lang='en' from="mgppdemo/TextQuery" type="query">
-    <content>
-      <documentList>
-        <document name="HASH010f073f22033181e206d3b7"/>
-        <document name="HASH010f073f22033181e206d3b7.2"/>
-        <document name="HASHac0a04dd14571c60d7fbfd"/>
-      </documentList>
-    </content>
+    <documentList>
+      <document name="HASH010f073f22033181e206d3b7"/>
+      <document name="HASH010f073f22033181e206d3b7.2"/>
+      <document name="HASHac0a04dd14571c60d7fbfd"/>
+    </documentList>
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
+\end{verbatim}\end{gsc}\end{quote}
+
+\subsubsection{'retrieve'-type services}
 Give me the Title metadata for these documents:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='en'  to="mgppdemo/MetadataRetrieve" 
 	type="retrieve">
-    <content>
       <documentList>
         <document name="HASH010f073f22033181e206d3b7"/>
@@ -579,7 +701,7 @@
   </request>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\end{verbatim}\end{gsc}\end{quote}
+
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <response lang='en' from="mgppdemo/MetadataRetrieve" 
@@ -611,8 +733,8 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 Give me the text for this document:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='en'   to="mgppdemo/DocumentRetrieve" 
@@ -625,7 +747,7 @@
   </request>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\end{verbatim}\end{gsc}\end{quote}
+
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <response lang='en' from="mgppdemo/DocumentRetrieve" 
@@ -647,6 +769,9 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
+\end{verbatim}\end{gsc}\end{quote}
+
+\subsubsection{'browse'-type services}
+
+\subsubsection{'process'-type services}
 Build requests are not a request for data---they are a request for some action to be carried out, for example, create or import or build or activate a collection. The response is a status or an error message. The import and build commands may take a long time to complete, so a message is sent back after a successful start of the command. The status may be polled by the requester to see how the process is going. 
 
@@ -655,5 +780,5 @@
 Some example requests (note that the build services are grouped into a service cluster called 'build', hence the addresses all begin with 'build/'):
 
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request lang='en'  type='process' to='build/NewCollection'>
@@ -673,10 +798,18 @@
   </request>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-
-\subsection{Generating the pages}
-
-URL-style requests are received by the Receptionist. Based on the arguments, a page of data must be returned to the servlet. As described in Section~\ref{subsec:url-type}, the requests are XML representations of Greenstone URLs. One of the arguments is action (a). This tells the Receptionist which Action module to pass the request to. Action modules decode the rest of the cgi-arguments to determine what requests need to be made to the system. 
+\end{verbatim}\end{gsc}\end{quote}
+
+\subsubsection{'enrich]-type services}
+
+\subsection{'status'-type messages}
+
+
+\subsection{'format'-type messages}
+
+\subsection{'applet'-type services}
+
+\section{Page generation}\label{sec:pagegen}
+
+URL-style requests are received by the Receptionist. Based on the arguments, a page of data must be returned to the servlet. As described in Section~\ref{sec:cgi}, the requests are XML representations of Greenstone URLs. One of the arguments is action (a). This tells the Receptionist which Action module to pass the request to. Action modules decode the rest of the cgi-arguments to determine what requests need to be made to the system. 
 System requests are received by the MessageRouter, which answers them one by one, either itself or by passing them on to the appropriate module. 
 
@@ -684,41 +817,71 @@
 
 The basic  page format  is:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <page>
- <config/>
- <display/>
- <request/>
- <response/>
+  <pageExtra>
+    <config/>
+    <display/>
+  </pageExtra>
+  <pageRequest/>
+  <pageResponse/>
 </page>
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 
 There are four main elements in the page: config, translate, request, response. The request is the original request that came into the Receptionist---this is included so that any parameters  can be preset to their previous values, for example, the query options on the query form.\footnote{this should be saved instead in some sort of state saving - if you leave a page and go back you want your parameters to be the same as well}. The response contains all the data that has been gathered from the system by the action. The other two elements contain extra information needed by XSLT. Config contains run-time variables such as the location of the gsdl home directory, the current site name, the name of the executable that is running (eg library)---these are needed to allow the XSLT to generate correct HTML URLs. Display contains some of the text strings needed in the interface---these are separate from the XSLT to allow for internationalization.
 
-The following subsections outline, for each action, what data is needed and what requests are generated to send to the system. Following that, Section~\ref{subsec:xslt} describes the config and display information, and the xslt files.
-
-\subsubsection{Page action}
+The following subsections outline, for each action, what data is needed and what requests are generated to send to the system. 
+
+
+Once the xml page has been put together, the page to return to the user is created by transforming the XML using XSLT. The output is HTML at this stage, but it will be possible to generate alternative outputs, such as XML, WML etc. A set of XSLT files defines an 'interface'. Different users can change the look of their web pages by creating new XSLT files for a new 'interface'. Just as we have a sites directory where different sites 'live' (ie where their configuration file and collections are located), we have an interfaces directory where the different interfaces 'live' (ie their transforms and images are located there). The default XSLT files are
+located in interfaces/default/transforms. Collections, sites and other interfaces
+can override these files by having their own copy of the appropriate
+files. New interfaces have their own directory inside interfaces/. Sites and collections can have a transform directory containing XSLT files. The order in which the XSLT files are looked for is collection, site, current
+interface, default interface.\footnote{this currently breaks down for remote sites - need to rethink it a bit.}
+***TODO*** describe a bit more??
+
+\subsection{Internationalization}
+
+Internationalization is a big part of Greenstone3. Language specific text strings are separated out from the rest of the system to allow for easy incorporation of new languages.
+
+Language specific text strings are specified in resource bundle property files. These live in resources/java.
+
+There is a properties file per class, and one per interface. At the moment, we have
+
+GS2MGPPSearch.properties
+GS2MGPPRetrieve.properties etc - the service classes
+
+interface\_default.properties. - for the default interface
+
+To add other languages, create eg GS2MGPPSearch\_fr.properties.
+
+The interface ones are treated differently from the other ones. The action doesn't know which text strings are needed by a particular transform, so it gets them all out of the properties file, and puts them into an xml \gst{<display>} element - the xslt can get the ones it needs from there.
+xslt could perhaps get the stuff from the properties bundle on the fly using java extension elements - would this be better?
+
+All other class specific text strings are just retrieved one by one as they are needed and added into the xml - for example, the names for query params are retrieved when the service description is created.
+
+\subsection{Page action}
 
 Depending on the subaction argument, different pages can be generated. For the 'home' page, a 'describe' request is sent to the MessageRouter---this returns a list of all the collections, services, serviceClusters and sites known about. For each collection, its metadata is retrieved via a 'describe' request. This metadata is added into the previous result, which is then added into the page.  The page is
-transformed using {\em home.xsl\/}.  For the 'about' page, a {\em
-describe} request is sent to the module that the about page is about: this may be a collection or a service cluster.  This returns a list of metadata
-and a list of services, and the result is transformed using {\em about.xsl\/}.
-
-\subsubsection{Query action}
-
-There are three query services which have been implemented: TextQuery, SimpleFieldQuery, and AdvancedFieldQuery. These are all handled in the same way by query action.
+transformed using \gst{home.xsl}.  For the 'about' page, a \gst{describe} request is sent to the module that the about page is about: this may be a collection or a service cluster.  This returns a list of metadata
+and a list of services, and the result is transformed using \gst{about.xsl}.
+
+
+\subsection{Query action}
+
+There are three query services which have been implemented: TextQuery, FieldQuery, and AdvancedFieldQuery. These are all handled in the same way by query action.
 For each page, the service description is requested from the  service  of the current collection (via a describe request).  This is done every time the query page is
 displayed.\footnote{This information should be cached.} The description includes a list of the parameters available for the query, such as case/stem, max num docs to return, etc. If the request type (rt) parameter is set to d for display, the action only needs to display the form, and this is the only request to the service. Otherwise, the submit button has been pressed, and a query request to the TextQuery service is sent. This has  all the parameters from the URL put into the parameter list. A list of document identifiers
 is returned. A followup query is sent to the MetadataRetrieve service of the collection: the content includes the list of
-documents, with a request for their {\em Title} metadata.  The service description and query result are combined into a page of xml, which is
-transformed using {\em basicquery.xsl\/} to produce the html page.
-
-\subsubsection{Applet action}
-
-There are two types of request to the applet action: {\em a=a \& sa=d\/} and
-{\em a=a \& sa=r\/}.  The value {\em sa=d\/} means ``display the applet.'' A
-{\em describe} request is sent to the service, which returns the {\footnotesize \verb#<applet>#} HTML element.  The transformation file {\em applet.xsl} embeds this
+documents, with a request for their \gst{Title} metadata.  The service description and query result are combined into a page of xml, which is
+transformed using \gst{basicquery.xsl} to produce the html page.
+
+\subsection{Applet action}
+
+There are two types of request to the applet action: \gst{a=a \& sa=d\/} and
+\gst{a=a \& sa=r\/}.  The value \gst{sa=d\/} means ``display the applet.'' A
+\gst{describe} request is sent to the service, which returns the \gst{<applet>} HTML element.  The transformation file \gst{applet.xsl} embeds this
 into the page, and the servlet returns the HTML.
 
-The value {\em sa=r} signals a request from the applet.  The result is returned
+The value \gst{sa=r} signals a request from the applet.  The result is returned
 directly to the applet code, in XML.  The other parameters are sent to the
 service untransformed, and the result is passed directly back to the applet.
@@ -728,9 +891,9 @@
 Here are two examples of requests generated by the Applet action, along with their corresponding responses.
 
-The first request corresponds to the URL arguments {\em a=a \&
+The first request corresponds to the URL arguments \gst{a=a \&
 sa=d \& sn=Phind \& c=mgppdemo\/}, which translate to ``display the Phind
 applet for the mgppdemo collection''.
 
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request type='describe' to='mgppdemo/PhindApplet'/>
@@ -761,13 +924,13 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-The second request corresponds to the  arguments {\em a=a \& sa=r \& sn=Phind \& c=mgppdemo \& pc=1 \& pptext=health \& pfe=0 \& ple=10 \& pfd=0 \& pld=10 \& pfl=0 \& pll=10}---this 
+\end{verbatim}\end{gsc}\end{quote}
+
+The second request corresponds to the  arguments \gst{a=a \& sa=r \& sn=Phind \& c=mgppdemo \& pc=1 \& pptext=health \& pfe=0 \& ple=10 \& pfd=0 \& pld=10 \& pfl=0 \& pll=10}---this 
 indicates a request to the service itself. The extra arguments (not a, sa, sn, c)  are simply copied into the
 request as parameters. The response is in a form suitable for the applet, placed inside
-{\footnotesize \verb#<appletData>#} in a standard Greenstone message.  AppletAction returns the
+\gst{<appletData>} in a standard Greenstone message.  AppletAction returns the
 contents of appletData to the browser, i.e. to the applet itself.
 
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <message>
   <request type='query' to='mgppdemo/PhindApplet'>
@@ -812,52 +975,27 @@
   </response>
 </message>
-\end{verbatim}\end{footnotesize}\end{quote}
-
-Note that the applet HTML may need to know the name of the {\em library}
+\end{verbatim}\end{gsc}\end{quote}
+
+Note that the applet HTML may need to know the name of the \gst{library}
 program.  However, that name is chosen by the person who installed the software
 and will not necessarily be ``library''.  To get around this, the applet can
 put a parameter called ``library'' into the applet data with a null value:
-\begin{quote}\begin{footnotesize}\begin{verbatim}
+\begin{quote}\begin{gsc}\begin{verbatim}
 <PARAM NAME='library' VALUE=''/>\/}
-\end{verbatim}\end{footnotesize}\end{quote}
+\end{verbatim}\end{gsc}\end{quote}
 When the Applet action encounters this parameter it inserts the name of the
 current library servlet as its value.
 
-\subsubsection{Document action}
+\subsection{Document action}
 
 DocumentAction sends a query to the DocumentRetrieve service of the collection requesting the text of the specified document.  At this stage no additional information is obtained, but in future stuff like Title and
 table of contents would be needed to make the display nicer.
 
-\subsubsection{Formatting the page using XSLT}\label{subsec:xslt}
-
-Once the xml page has been put together, the page to return to the user is created by transforming the XML using XSLT. The output is HTML at this stage, but it will be possible to generate alternative outputs, such as XML, WML etc. A set of XSLT files defines an 'interface'. Different users can change the look of their web pages by creating new XSLT files for a new 'interface'. Just as we have a sites directory where different sites 'live' (ie where their configuration file and collections are located), we have an interfaces directory where the different interfaces 'live' (ie their transforms and images are located there). The default XSLT files are
-located in interfaces/default/transforms. Collections, sites and other interfaces
-can override these files by having their own copy of the appropriate
-files. New interfaces have their own directory inside interfaces/. Sites and collections can have a transform directory containing XSLT files. The order in which the XSLT files are looked for is collection, site, current
-interface, default interface.\footnote{this currently breaks down for remote sites - need to rethink it a bit.}
-
-\subsection{Internationalization}
-
-Internationalization is a big part of Greenstone3. Language specific text strings are separated out from the rest of the system to allow for easy incorporation of new languages.
-
-Language specific text strings are specified in resource bundle property files. These live in resources/java.
-
-There is a properties file per class, and one per interface. At the moment, we have
-
-GS2MGPPSearch.properties
-GS2MGPPRetrieve.properties etc - the service classes
-
-interface\_default.properties. - for the default interface
-
-To add other languages, create eg GS2MGPPSearch\_fr.properties.
-
-The interface ones are treated differently from the other ones. The action doesn't know which text strings are needed by a particular transform, so it gets them all out of the properties file, and puts them into an xml $<$display$>$ element - the xslt can get the ones it needs from there.
-xslt could perhaps get the stuff from the properties bundle on the fly using java extension elements - would this be better?
-
-All other class specific text strings are just retrieved one by one as they are needed and added into the xml - for example, the names for query params are retrieved when the service description is created.
-
-\subsection{Collection formation}
-
-Greenstone 2 compatible building has been implemented in gsdl3. so far only mgpp collections will work.
+
+
+\section{Collection formation}
+
+
+Greenstone 2 compatible building has been implemented in gsdl3. 
 
 Collection construction can be done through the web, using the build servicecluster in localsite. Just sequence through the steps needed. So far, addDocument does not work, so documents need to be manually added to teh import directory.
@@ -874,9 +1012,9 @@
 Collection building can also be done on the command line:
 
-ConstructCollection -site <site-path> -mode new|import|build|activate [options] <coll-name>
+\gst{ConstructCollection -site <site-path> -mode new|import|build|activate [options] <coll-name>}
 
 eg
 
-ConstructCollection -site /research/kjdon/home/gsdl3/sites/localsite -mode new -creator kjdon@cs.waikato.ac.nz testcol
+\gst{ConstructCollection -site /research/kjdon/home/gsdl3/sites/localsite -mode new -creator kjdon@cs.waikato.ac.nz testcol}
 
 the options get passed to the underlying script, - there is no good help message yet.
@@ -888,5 +1026,7 @@
 CollectionConstructor is the base class for building control. GS2PerlConstructor is the implementation that uses greenstone 2 perl scripts. The building process sends events (ConstructionEvent) to any listeners (ConstructionListener) as important stages happen. You can add one or more listeners to the constructor which will get notified of events.
 
-\section{Details}
+\subsection{Collection design}\label{sec:colldesign}
+
+\section{Installation details}
 
 This section describes the directory structure of the Greenstone source, and provides an installation guide to installing Greenstone from CVS.
@@ -900,4 +1040,6 @@
 
 \begin{table}
+\caption{The Greenstone directory structure}
+\label{tab:dirs}
 \center{\footnotesize
 \begin{tabular}{l p{7cm}}
@@ -929,4 +1071,8 @@
 gsdl3/src/java/org/greenstone/testing
   & Junit scaffolding for unit testing.\\
+gsdl3/src/java/org/greenstone/applet
+ & where the code for applets goes \\
+gsdl3/src/java/org/greenstone/applet/phind
+  & the phind applet (phrase browsing) \\
 gsdl3/src/cpp/
   & Place for any cpp source code---none yet \\
@@ -940,5 +1086,7 @@
  & any resources that may be needed\\
 gsdl3/resources/java
- & properties files for java resource bundles - used to handle all the language specific text\\
+ & properties files for java resource bundles - used to handle all the language specific text This directory is on the classpath, so any other Java resources can be placed here \\
+gsdl3/resources/soap
+ & soap service description files \\
 gsdl3/bin
   & executable stuff lives here\\
@@ -951,119 +1099,126 @@
 gsdl3/docs
   & Documentation :-)\\
+\hline
 gsdl3/web
-  & The place to put any web stuff that the servlet needs. html files go here\\
+  & This is where the web site is defined. Any static html files can go here. This directory is the Tomcat root directory.\\
 gsdl3/web/WEB-INF
-  & The web.xml file lives here (configuration information for tomcat)\\
+  & The web.xml file lives here (servlet configuration information for tomcat)\\
 gsdl3/web/WEB-INF/classes
   & Servlet classes go in here\\
-\hline
-gsdl3/sites
+gsdl3/web/sites
   & Contains directories for different sites---a site is a set of collections and services served by a single MessageRouter (MR). The MR may have connections (eg soap) to other sites\\
-gsdl3/sites/localsite 
-  & One site\\
-gsdl3/sites/localsite/collect 
+gsdl3/web/sites/localsite 
+  & One site - the site configuration file lives here\\
+gsdl3/web/sites/localsite/collect 
   & The collections directory \\
-gsdl3/sites/localsite/images 
+gsdl3/web/sites/localsite/images 
   & Site specific images \\
-gsdl3/sites/localsite/transforms 
+gsdl3/web/sites/localsite/transforms 
   & Site specific transforms \\
-gsdl3/interfaces
-  & Contains all interface specific stuff (eg images and XSLT transforms\\
-gsdl3/interfaces/default
+gsdl3/web/interfaces
+  & Contains directories for different interfaces - an interface is defined by its images and xslt files \\
+gsdl3/web/interfaces/default
   & The default interface\\
-gsdl3/interfaces/default/images
-  & The images\\
-gsdl3/interfaces/default/transforms
-  & The XSLT files\\
+gsdl3/web/interfaces/default/images
+  & The images for the default interface\\
+gsdl3/web/interfaces/default/transforms
+  & The XSLT files for the default interface\\
 \hline
 \end{tabular}}
-\label{tab:dirs}
-\caption{The Greenstone directory structure}
 \end{table}
 
 \subsection{Installation guide}
 
-\newcommand{\gsdlhome}{\begin{footnotesize}{\em \$GSDL3HOME}\end{footnotesize}}
-
-Cuurently, greenstone3 is only available through CVS. The installation procedure has  been automated. 
+\newcommand{\gsdlhome}{\$GSDL3HOME}
+\newcommand{\gshome}{\$GSDLHOME}
+
+Cuurently, Greenstone3 is only available through CVS. The installation procedure has been semi-automated. Note, these instructions are for installation on linux. If you want to use Greenstone3 on Windows, download it using CVS, then follow the instructions in \gst{http://www.cs.waikato.ac.nz/~mdewsnip/GSDL3Windows.html}.
 
 \subsubsection{Get the source}
 
-\noindent If you have a greenstone\_cvs account, you can use the following:
-
-\begin{footnotesize}\begin{tt}
-\noindent export CVSROOT=:ext:{\em your-username}@cvs.scms.waikato.ac.nz:\\
-\indent /usr/local/global-cvs/gsdl-src\\
-export CVS\_RSH=ssh\\
-cvs co gsdl3\\
-\end{tt}\end{footnotesize}
-
-\noindent Otherwise, you can get it through anonymous access:
-
-\begin{footnotesize}\begin{tt}
-\noindent export CVSROOT=:pserver:cvs\_anon@cvs.scms.waikato.ac.nz:2402\\
-\indent /usr/local/global-cvs/gsdl-src\\
-export CVS\_RSH=ssh\\
-cvs co gsdl3\\
-\end{tt}\end{footnotesize}
-
-\noindent If you need it, the password for anonymous CVS access is {\footnotesize \verb#anonymous#}.
+If you have a greenstone\_cvs account, you can use the following:
+
+\begin{quote}\begin{gsc}\begin{verbatim}
+export CVS_RSH=ssh
+cvs -d :ext:@cvs.scms.waikato.ac.nz:/usr/local/global-cvs/
+                   gsdl-src co gsdl3
+\end{verbatim}\end{gsc}\end{quote}
+
+Otherwise, you can get it through anonymous access:
+
+\begin{quote}\begin{gsc}\begin{verbatim}
+cvs -d :pserver:cvs\_anon@cvs.scms.waikato.ac.nz:2402/usr/local/
+           global-cvs/gsdl-src co gsdl3
+\end{verbatim}\end{gsc}\end{quote}
+
+If you need it, the password for anonymous CVS access is \gst{anonymous}. Note that some versions of CVS have trouble accessing this repository. We are using version 1.11.1p1.
 
 \subsubsection{Compile and install greenstone}\label{subsec:compile}
 
-An install.sh script has been constructed (thanks, Stuart) to compile and install greenstone 3. What you nee to do is:
-
-\begin{footnotesize}\begin{tt}
-cd gsdl3
-source setup.bash
-install.bash
-source setup.bash
-\end{tt}\end{footnotesize}
-
-If you want to do greenstone2 compatible building (currently the only type) you need to have greenstone 2 installed, 'source setup.bash' in the top level greenstone 2 directory, then re-'source setup.bash' for greenstone 3. This is to set GSDLHOME for tomcat.
-
-\noindent Note: 'source setup.bash' needs to be done once in any xterm window before doing a make or running tomcat. setup.bash sets the environment variables {\footnotesize \verb#CLASSPATH#, \verb#PATH#, \verb#JAVA_HOME#} etc.
+An install.sh script has been constructed to compile and install Greenstone3. What you need to do is:
+
+\begin{quote}\begin{gsc}
+cd gsdl3\\
+source setup.bash\\
+install.bash\\
+source setup.bash\\
+\end{gsc}\end{quote}
+
+If you want to do Greenstone2 compatible building (currently the only type) you need to have Greenstone2 installed, \gst{source setup.bash} in the top level Greenstone2 directory, then re-\gst{source setup.bash} for Greenstone3. This is to set \gst{\gshome} for tomcat.
+
+\noindent Note: \gst{source setup.bash} needs to be done once in any xterm window before doing a make or running tomcat. setup.bash sets the environment variables \gst{CLASSPATH, PATH, JAVA\_HOME} etc.
 
 If you want to use SOAP to talk to remote sites, you also need to do the following:
 
-\begin{footnotesize}\begin{tt}
+\begin{quote}\begin{gsc}
 install-soap.bash  
-\end{tt}\end{footnotesize}
-
-Thats it. 
-
-You dont want to run install.bash twice - it adds stuff into files
-
-To update your installation, you can run update.bash - this remakes all the java stuff.
+\end{gsc}\end{quote}
+
+There is one java command that sometimes doesn't work under bash, so you may need to cut and paste it into the terminal to get it to work. See the output from the bash-script for details.
+
+To shutdown or startup tomcat, the commands are:
+\begin{quote}\begin{gsc}
+\gsdlhome/comms/tomcat/jakarta/bin/shutdown.sh\\
+\gsdlhome/comms/tomcat/jakarta/bin/startup.sh\\
+\end{gsc}\end{quote}
+
+You dont want to run install.bash twice - it adds stuff into files.
+To update your installation, you can run update.bash - this updates your code form cvs, and remakes all the java stuff.
 
 
 \subsubsection{The sample sites}
 
-\noindent There are two greenstone ``sites'' that come with the checkout: localsite, and site1. localsite has several collections, only two of which have any actual data. The third is a dummy collection. site1 has one dummy collection. Each site has a configuration file which specifies the site name,  site-wide services if any, and a list of remote sites to connect to.
-localsite does not connect to any other sites. site1 specifies a SOAP connection to localsite.
-
-\noindent The collections which do not have data can be looked at but you cant do any queries on them.
-
+\noindent There are two greenstone {\em sites} that come with the checkout: localsite, and soapsite. localsite has three collections, while soapsite has none. Each site has a configuration file which specifies the site name, site-wide services if any, and a list of remote sites to connect to.
+localsite does not connect to any other sites. soapsite specifies a SOAP connection to localsite.
 
 \subsubsection{Tomcat}
 
 \noindent Tomcat is a servlet container. It is used to serve a greenstone site using a servlet.
-\\
-\\
-\noindent The file \begin{footnotesize}{\tt \gsdlhome/web/WEB-INF/web.xml}\end{footnotesize} contains the setup information for tomcat---tells it what servlets to load, what initial paramaters to pass them, and what web names map to the servlets.
-There are three servlets specified in web.xml: one is a test servlet that just prints ``hello greenstone'' to a web page. This is useful if you are having trouble getting tomcat set up. The other two are greenstone library servlets, ``library'', which serves localsite, and ``library1'' which serves site1.
-\\
-\\
-\noindent One initialisation parameter for the library servlets is {\footnotesize \verb#gsdl3home#}.
-\begin{footnotesize}\begin{verbatim}
-<init-param>
-  <param-name>gsdl3home</param-name>
-  <param-value>/research/kjdon/home/gsdl3</param-value>
-</init-param>
-\end{verbatim}\end{footnotesize}
-
-The file \gsdlhome/comms/tomcat/jakarta/conf/server.xml is the tomcat configuration file. setup.bash adds a context for gsdl servlets - this tells tomcat where to find the web.xml file, and what url (eg /gsdl3) to give it.
-
-\noindent Note: tomcat runs on port 8080 - you can change that if you wish in this file
+
+The file \gst{\gsdlhome/web/WEB-INF/web.xml} contains the setup information for tomcat---tells it what servlets to load, what initial paramaters to pass them, and what web names map to the servlets.
+There are three servlets specified in web.xml: one is a test servlet that just prints ``hello greenstone'' to a web page. This is useful if you are having trouble getting tomcat set up. The other two are greenstone library servlets, {\em library}, which serves localsite, and {\em library1} which serves soapsite.
+
+The initialisation parameters used by the library servlets are as follows:
+
+\begin{tabular}{lll}
+\bf name & \bf sample value & \bf description \\
+\hline
+gsdl3home & /research/kjdon/gsdl3 & the base directory of the gsdl3 installation \\
+sitename & localsite & the site to use \\
+interfacename & default & the interface to use\\
+libraryname & library & the name of the library program \\
+defaultlang & en & the default language for the interface\\
+receptionist & NZDLReceptionist & (optional) specifies an alternative Receptionist to use\\
+messagerouter & NewMessageRouter & (optional) specifies an alternative MessageRouter to use\\
+\hline
+\end{tabular}
+
+It is possible to run several servlets at once, with different combinations of sites and/or interfaces.
+
+The file \gst{\gsdlhome/comms/tomcat/jakarta/conf/server.xml} is the tomcat configuration file. The installation process adds a context for greenstone3 servlets (\gst{\gsdlhome/web})---this tells tomcat where to find the web.xml file, and what url (\gst{/gsdl3}) to give it. Anything inside the context directory is accessible via tomcat\footnote{can we use .htaccess files to restrict access??}. For example, the index.html file that lives in \gst{\gsdlhome/web} can be accessed through the URL \gst{localhost:8080/gsdl3/index.html}. The demo collection's images can be accessed through \gst{localhost:8080/gsdl3/sites/localsite/collect/demo/images/}~.
+
+
+Tomcat runs by default on port 8080---this can be changed in server.xml. The siteConfig files also need changing if Tomcat's port is changed: \gst{<httpAddress>} for the site, and \gst{<address>} for a remote site both use this.
+
 
 \subsubsection{Serving your site using tomcat}\label{subsec:runtomcat}
@@ -1071,37 +1226,38 @@
 \noindent To run tomcat, you need to have sourced {\footnotesize \verb#setup.bash#} in \gsdlhome\  to set up {\footnotesize \$CLASSPATH} (see \ref{subsec:compile}). Then, 
 
-\begin{footnotesize}\begin{tt}
-\noindent cd \gsdlhome/comms/tomcat/jakarta-tomcat-4.0.1/bin\\
+\begin{gsc}\begin{tt}
+\noindent cd \gsdlhome/comms/tomcat/jakarta/bin\\
 ./startup.sh
-\end{tt}\end{footnotesize}
+\end{tt}\end{gsc}
 
 \noindent ({\footnotesize \verb#./shutdown.sh#} shuts down tomcat)
 \\
 \\
-\noindent The tomcat server can be accessed on the web at {\footnotesize \verb#http://localhost:8080#}---this gets you to a welcome page.
-The greenstone stuff is at {\footnotesize \verb#http://localhost:8080/gsdl3#}---this displays {\footnotesize \gsdlhome/web/index.html}. You should be able to run the test servlet and both library servlets from this page.
+\noindent The tomcat server can be accessed on the web at \gst{http://localhost:8080}---this gets you to a welcome page.
+The greenstone stuff is at \gst{http://localhost:8080/gsdl3}---this displays \gst{\gsdlhome/web/index.html}. You should be able to run the test servlet and both library servlets from this page.
 
 \noindent Note: tomcat must be shutdown and restarted any time you make changes in the following for those changes to take effect:\\
 \begin{bulletedlist}
-\begin{footnotesize}\begin{tt}
+\begin{gsc}
 \item \gsdlhome/web/WEB-INF/web.xml
 \item \gsdlhome/comms/tomcat/jakarta-tomcat-4.0.1/conf/server.xml
-\end{tt}\end{footnotesize}
+\end{gsc}
 \item any classes or jar files used by the servlets
 \end{bulletedlist}
 \noindent Note: stdin and stdout for the servlets both go to\\
-\begin{footnotesize}{\tt \gsdlhome/comms/tomcat/jakarta-tomcat-4.0.1/logs/catalina.out}\end{footnotesize}
+\gst{\gsdlhome/comms/tomcat/jakarta/logs/catalina.out}
+
+On startup, the servlet loads in its collections and services. If the site or collection configuration files are changed, these changes will not take effect until the site/collection is reloaded. This can be done through the reconfiguration messages (see Section~\ref{sec:runtime-config}, or by restarting tomcat.
 
 \subsubsection{Using SOAP to talk to a remote site}
 
-\noindent The previous installation stuff is fine if you only want to talk to local sites. However, if you want to connect using SOAP to a remote site, some more stuff needs to be done. site1 specifies a SOAP connection to localsite. If you run site1 without connecting to localsite, you can only see the local  collections, eg the dummy collection myfiles. However, if you connect to localsite, you can see all of {\em its} collections as well.
+\noindent The previous installation stuff is fine if you only want to talk to local sites. However, if you want to connect using SOAP to a remote site, some more stuff needs to be done. soapsite specifies a SOAP connection to localsite. If you run soapsite without connecting to localsite, you don't get any collections. However, if you connect to localsite, you can see all of {\em its} collections.
 \\
 \\
-\noindent The SOAP server we use is actually run as a servlet in tomcat. You need to set up SOAP, set up the SOAP server class which will be your service, and then deploy that service.
-
-this is done by install-soap.bash.
+\noindent The SOAP server we use is actually run as a servlet in tomcat. You need to set up SOAP, set up the SOAP server class which will be your SOAP web service, and then deploy that service.
+This is done by install-soap.bash.
 You can also deploy a service through the website.  If tomcat is not running, start it up (see \ref{subsec:runtomcat}).
 
-\noindent The SOAP servlet can be accessed at \begin{footnotesize}{\tt http://localhost:8080/soap}\end{footnotesize}. You should see a welcome page. Click on ``Run the admin client''. This enables you to list, deploy and undeploy SOAP services.
+\noindent The SOAP servlet can be accessed at \begin{gsc}{\tt http://localhost:8080/soap}\end{gsc}. You should see a welcome page. Click on ``Run the admin client''. This enables you to list, deploy and undeploy SOAP services.
 
 \noindent To deploy the SOAPServer for localsite:
@@ -1124,24 +1280,294 @@
 \subsubsection{Debugging SOAP}
 
-\noindent If you need to debug the SOAP stuff for some reason, or just want to look at the SOAP messages that are being passed back and forth, there is a program called TcpTunnelGui. This intercepts messages coming in to one port, displays them, and passes them to another port.
-
-\noindent To run it:
-
-\noindent {\footnotesize \verb#java org.apache.soap.util.net.TcpTunnelGui 8070 localhost 8080#}
-
-\noindent tomcat uses port 8080 - you need to modify greenstone to talk to port 8070 instead of 8080. - this is specified in the {\footnotesize \verb#site#} element of the site configuration file. 
-\\
-\\
-\noindent eg, in \begin{footnotesize}{\tt \gsdlhome/sites/site1/siteConfig.xml}\end{footnotesize}:
-\begin{footnotesize}\begin{verbatim}
+If you need to debug the SOAP stuff for some reason, or just want to look at the SOAP messages that are being passed back and forth, use a program called TcpTunnelGui. This intercepts messages coming in to one port, displays them, and passes them to another port.
+To run it, type:
+
+\begin{quote}\gst{java org.apache.soap.util.net.TcpTunnelGui 8070 localhost 8080}
+\end{quote}
+
+8070 is the port that TcpTunnelGui listens on, and 8080 is the port that it sends the messages onto---the port that Tomcat is using. You need to modify Greenstone to talk to port 8070 when it wants to talk to Tomcat, so that the messages go through TcpTunnelGui. This is specified in the \gst{<site>} element of the soapsite site configuration file (\gst{\gsdlhome/web/sites/soapsite/siteConfig.xml}). 
+\begin{quote}\begin{gsc}\begin{verbatim}
 <site name="org.greenstone.localsite" 
       address="http://localhost:8080/soap/servlet/rpcrouter" 
       type="soap"/>
-\end{verbatim}\end{footnotesize}
-
-\noindent You can replace the 8080 with 8070 if you want to run TcpTunnelGui.
-
-\noindent Note that \begin{footnotesize}{\tt http://localhost:8080/soap/servlet/rpcrouter}\end{footnotesize} is the
+\end{verbatim}\end{gsc}\end{quote}
+
+Note that \gst{http://localhost:8080/soap/servlet/rpcrouter} is the
 address for talking to the tomcat SOAP servlet services.
+
+\section{Developer's notes}
+
+Here are some random notes for developers who want to modify the source code.
+\subsection{Greenstone utility classes}
+
+These are found in \gst{gsdl3/src/java/org/greenstone/gsdl3/util} and provide a variety of useful functions. Table~\ref{tab:utils} gives a brief description of the various classes.
+
+\begin{table}
+\caption{The utility classes in org.greenstone.gsdl3.util}
+\label{tab:utils}
+\center{\footnotesize
+\begin{tabular}{lp{3.75in}}
+\hline
+\bf Utility class & \bf Description\\
+ConfigVars & holds the servlet startup variables, including library name, site name, interface name, default language\\
+Dictionary & wrapper around a ResourceBundle, providing strings with parameter\\
+GSCGI & class to map between short name cgi args and long name request parameters \\
+GSFile & class to create all greenstone file paths eg used to locate configuration files, xslt files and collection data. \\
+GSHTML & provides convenience methods for dealing with HTML, eg making strings HTML safe\\
+GSPath & used to create, examine and modify message address paths\\
+GSStatus & some static codes for status messages\\ 
+GSXML & lots of methods for extracting information out of greenstone XML, and creating some common types of elements. Also has static Strings for element and attribute names used by greenstone.\\
+GSXSLT & some manipulation functions for greenstone XSLT\\
+Misc & miscellaneous functions\\
+OID & class to handle greenstone (2) OIDs\\
+XMLConverter & provides methods to create new Documents, parse Strings or Files into Documents, and convert Nodes to Strings\\
+XMLTransformer & methods to transform XML using XSLT \\
+XSLTUtil & contains static methods to be called from within XSLT \\
+\hline
+\end{tabular}
+}
+\end{table}
+
+\subsection{Creating new services}
+
+a browse type service must also implement servicenameMetadataRetrieve service.
+\subsection{Working with XML}
+
+We use the DOM model for handling XML. This involves Documents, Nodes, Elements etc. Node is the basic thing in the tree, all others inherit from this. A Document represents a whole document, and is a kind of container for all the nodes. Elements and Nodes are not supposed to exist outside of the context of a document, so you have to have a document to create them. The document is not the top level node in the tree, to get this, use Document.getDocumentElement(). If you create nodes etc but dont append them to something already in the document tree, they will be separate - but they still know who their owner document is.
+
+To create new Documents, and convert Strings or Files to Documents, use XMLConverter.
+eg:
+\begin{quote}\begin{gsc}
+XMLConverter converter = new XMLConverter();\\
+Document doc = converter.newDOM();\\
+
+File stylesheet = new File(``query.xsl'');\\
+Document style = converter.getDOM(stylesheet);\\
+
+String message = ``<message><request type='cgi'/></message>'';\\
+Document m = converter.getDOM(message);\\
+\end{gsc}\end{quote}
+
+To output a document as a String, use \gst{converter.getString(doc);}
+
+To add nodes and stuff to an empty document - create them, then append to the tree:
+\begin{quote}\begin{gsc}
+Document doc = converter.newDOM();\\
+Element e = doc.createElement(``message'');\\
+doc.appendChild(e);\\
+\end{gsc}\end{quote}
+
+Note that you can only append one node to a document---this will become the toplevel node. After that, you can append nodes to child nodes as you like, but a document is only allowed one top level node.
+
+Nodes can only be created by a Document. Document has creation methods for all types of Nodes, for example \gst{createElement(element\_name)}, \gst{createAttribute(attr\_name)},  \gst{createTextNode(text\_data)} etc.
+
+DOM006 Hierarchy request error: happens if you have more than one root node in your document
+
+\subsection{Greenstone XML}
+
+Greenstone format namespace: (at the moment)
+xmlns:gsf="http://www.greenstone.org/configformat"
+
+
+no DTDs or Schema defined yet. Until there are, try and keep to teh following rules:
+
+\begin{bulletedlist}
+
+\item always return expected elements even if empty, eg \gst{<paramList/>}.
+
+\item If you get the whole documetn it is called \gst{<document>}. However if you are returned a list of pointers to parts of the documetns, they are \gst{<documentNode>}s.
+
+\item insiode a list you can only have elements of the same name as the list. For example, a \gst{<paramList>} should only have \gst{<param>} elements inside it.
+
+\end{bulletedlist}
+\subsection{Working with XSLT}
+
+\begin{bulletedlist}
+\item {\em adding html to an xml doc:}
+
+eg I have a text node with html inside it inside a resource element
+to add that to a new XML doc, I use
+\gst{<xsl:value-of select='resource'>}
+
+if the output mode is xml or html, this will escape any special characters
+ie $<$ and $>$ etc
+
+use
+\gst{<xsl:value-of disable-output-escaping="yes" select='resource'>}
+instead.
+
+\item {\em including an xml doc into a stylesheet:}
+
+\gst{<xsl:variable name='import' select='document(``newdoc.xml'')'/>}
+
+then can use the info:
+
+\gst{<xsl:value-of select='\$import/element'/>}
+
+\item {\em selecting an ancestor:}
+
+ the ancestor axis contains the parent of the context node, and its
+ parent and so on. to pick one node among these:
+ ancestor::elem-name. I dont know how this works if there are two
+ nodes with the same name in the axis.
+
+\item {\em basic XSLT elements:}
+\begin{quote}\begin{footnotesize}\begin{verbatim}
+<xsl:template match='xxx' name='yyy'/>
+
+<xsl:apply-templates select='xxx'/>
+<xsl:call-templates name='yyy'/>
+
+<xsl:variable name='doc' select='document("layout.xml")'/>
+
+<xsl:value-of select='$doc/chapter1'/> $
+\end{verbatim}\end{footnotesize}\end{quote}
+
+\item {\em using namespaces:}
+If you are using the same namespace in more than one file, eg in the source xml and in the stylesheet, make sure that the URI for the xmlns:xxx thingy is the same in both cases---otherwise the names dont match. This includes http:// on the front.
+
+\item I dont think \gst{<xsl:with-param name='xxx' select='true'/>} is
+the same as \gst{<xsl:with-param name='xxx'>true</xsl:with-param>}.
+Use the second one.
+
+\item to select a node from a list based on an attribute value: for example
+\begin{quote}\begin{footnotesize}\begin{verbatim}
+<xsl:variable name='name'>CL1</xsl:variable>
+
+<xsl:value-of select="classifier[@name=\$name]/@content"/>
+\end{verbatim}\end{footnotesize}\end{quote}
+
+
+\end{bulletedlist}
+\subsubsection{What can I do to speed up XSL transformations?}
+
+This information taken from the Xalan FAQS page.
+
+\begin{bulletedlist}
+
+\item Use a Templates object (with a different Transformers for each
+transformation) to perform multiple transformations with the same set
+of stylesheet instructions.
+
+\item Set up your stylesheets to function efficiently.
+
+\item Don't use "//" (descendant axes) patterns near the root of a
+large document.
+
+\item Use xsl:key elements and the key() function as an efficient way
+to retrieve node sets.
+
+\item Where possible, use pattern matching rather than xsl:if or
+xsl:when statements.
+
+\item xsl:for-each is fast because it does not require pattern matching.
+
+\item Keep in mind that xsl:sort prevents incremental processing.
+
+\item When you create variables,\\
+\gst{<xsl:variable name="fooElem" select="foo"/>} is usually faster
+than \\
+\gst{<xsl:variable name="fooElem"><xsl:value-of-select="foo"/></xsl:variable>}.
+
+\item Be careful using the last() function.
+
+\item The use of index predicates within match patterns can be expensive.
+
+\item Decoding and encoding is expensive.
+
+\item For the ultimate in server-side scalability, perform transform
+operations on the client. 
+
+\end{bulletedlist}
+
+\subsection{Java gdbm}
+
+To talk to gdbm, a jni wrapper called java-gdbm is used. It was
+obtained from:\\ \gst{http://aurora.rg.iupui.edu/~schadow/dbm-java/pip/gdbm/}
+
+It uses packing objects to convert to and from an array of bytes (in
+gdbm file) from and to java objects. In my GDBMWrapper class I use
+StringPacking - uses UTF-8 encoding. but some stuff came out funny. so
+I had to changes the from\_bytes method in StringPacking.java to use
+new String(raw, "UTF-8") instead of new String(raw). this seems to
+work.
+
+Note---if we use this gdbm stuff to create the file too, may need to
+alter the to-bytes method.
+
+The makefile in j-gdbm is crap---it tries to get stuff from its
+original CVS tree.  I have created a new Makefile---in my-j-gdbm
+directory.  this stuff needs to go into cvs probably.
+
+
+
+\subsection{Resources}
+
+This is a list of some useful resources that we have come across during development of gsdl3. 
+
+Contents for 'The Java Native Interface Programmer's Guide and
+Specification' on-line\\ 
+\gst{http://java.sun.com/docs/books/jni/html/jniTOC.html}
+
+Java Native Interface Specification\\
+\gst{http://java.sun.com/j2se/1.4/docs/guide/jni/spec/jniTOC.doc.html}
+
+JNI Documentation Contents\\
+\gst{http://java.sun.com/j2se/1.4/docs/guide/jni/index.html}
+
+another JNI page\\
+\gst{http://mindprod.com/jni.html}
+
+Java 1.4 api index\\
+\gst{http://java.sun.com/j2se/1.4/docs/api/index.html}
+
+Java tutorial index\\
+\gst{http://java.sun.com/docs/books/tutorial/index.html}
+
+Safari books online - has java, XML, XSLT, etc books\\
+\gst{http://proquest.safaribooksonline.com/mainhom.asp?home}
+
+Java 1.4 i18n FAQ\\
+\gst{http://www.sun.com/developers/gadc/faq/java/java1.4.html}
+
+Java and XSLT page\\
+\gst{http://www.javaolympus.com/java/Java\%20and\%20XSLT.html}
+
+Xalan-Java overview\\
+\gst{http://xml.apache.org/xalan-j/overview.html}
+
+Tomcat documentation index\\
+\gst{http://jakarta.apache.org/tomcat/tomcat-4.0-doc/index.html}
+
+Servlet and JSP tutorial\\
+\gst{http://www.apl.jhu.edu/~hall/java/Servlet-Tutorial/}
+
+Core Servlets and JavaServer Pages, book by Marty Hall. download the
+pdf from here (try before you buy link)\\ 
+\gst{http://www.coreservlets.com/}
+
+J-gdbm page\\
+\gst{http://aurora.rg.iupui.edu/~schadow/dbm-java/pip/gdbm/}
+
+Stuarts page of links\\
+\gst{http://www.cs.waikato.ac.nz/~nzdl/gsdl3/}
+
+a good basic xslt tutorial\\
+\gst{http://www.zvon.org/xxl/XSLTutorial/Books/Output/contents.html}
+
+JAXP (java api for xml processing) package overview\\
+\gst{http://java.sun.com/xml/jaxp/dist/1.1/docs/api/overview-summary.html}
+
+DeveloperWorks, xml zone\\
+\gst{http://www-106.ibm.com/developerworks/xml/}
+
+xslt.com\\
+\gst{http://www.xslt.com/}
+
+jeni tennison's xslt pages\\
+\gst{http://www.jenitennison.com/xslt/}
+
+apaches xml tools\\
+\gst{http://xml.apache.org/}
 
 
