source: trunk/gsdl3/docs/manual/manual.tex@ 10775

Last change on this file since 10775 was 10775, checked in by kjdon, 18 years ago

small changes for installation, and url (now /greenstone3 not /gsdl3)

  • Property svn:keywords set to Author Date Id Revision
File size: 165.6 KB
Line 
1\documentclass[a4paper,11pt]{article}
2\usepackage{isolatin1,times,epsfig}
3\hyphenation{Message-Router Text-Query}
4
5\newenvironment{gsc}% Greenstone text bits
6{\begin{footnotesize}\begin{tt}}%
7{\end{tt}\end{footnotesize}}
8
9\newcommand{\gst}[1]{{\footnotesize \tt #1}}
10\newcommand{\gsdlhome}{\$GSDL3HOME}
11
12\newcommand{\gsii}{Greenstone 2}
13\newcommand{\gsiii}{Greenstone 3}
14\newcommand{\gs}{Greenstone}
15
16\begin{document}
17
18\title{\gsiii\ : A modular digital library.}
19
20% if you work on this manual, add your name here
21\author{Katherine Don, George Buchanan and Ian H. Witten \\[1ex]
22 Department of Computer Science \\
23 University of Waikato \\ Hamilton, New Zealand \\
24 \{kjdon, grbuchan, ihw\}@cs.waikato.ac.nz}
25
26\date{}
27
28\maketitle
29
30\newenvironment{bulletedlist}%
31{\begin{list}{$\bullet$}{\setlength{\itemsep}{0pt}\setlength{\parsep}{0pt}}}%
32{\end{list}}
33
34\noindent
35Greenstone Digital Library Version 3 is a complete redesign and
36reimplementation of the \gs\ digital library software. The current
37version (\gsii) enjoys considerable success and is being widely used.
38\gsiii \ will capitalise on this success, and in addition it will
39\begin{bulletedlist}
40\item improve flexibility, modularity, and extensibility
41\item lower the bar for ``getting into'' the \gs\ code with a view to
42 understanding and extending it
43\item use XML where possible internally to improve the amount of
44 self-documentation
45\item make full use of existing XML-related standards and software
46\item provide improved internationalisation, particularly in terms of sort order,
47 information browsing, etc.
48\item include new features that facilitate additional ``content management''
49 operations
50\item operate on a scale ranging from personal desktop to corporate library
51\item easily permit the incorporation of text mining operations
52\item use Java, to encourage multilinguality, X-compatibility, and to permit
53 easier inclusion of existing Java code (such as for text mining).
54\end{bulletedlist}
55Parts of \gs\ will remain in other languages (e.g. MG, MGPP); JNI (Java
56Native Interface) will be used to communicate with these.
57
58A description of the general design and architecture of \gsiii\ is covered by the document {\em The design of Greenstone3: An agent based dynamic digital library} (design-2002.ps, in the docs/manual directory).
59
60This documentation consists of several parts. Section~\ref{sec:install} is for administrators, and covers \gsiii\ installation, how to access the library, and some administration issues. Section~\ref{sec:user} is for users of the software, and looks at using the sample collections, creating new collections, and how to make small customisations to the interface. The remaining sections are aimed towards the \gs\ developer. Section~\ref{sec:develop-runtime} describes the run-time system, including the structure of the software, and the message format, while Section~\ref{sec:develop-build} describes the collection building process. Section~\ref{sec:new-features} describes how to add new features to \gs, such as how to add new services, new page types, new plugins for different document formats. Section~\ref{sec:distributed} describes how to make \gs\ run in a distributed fashion, using SOAP as an example communications protocol. Finally, there are several appendices, including how to install \gs\ from CVS, some notes on Tomcat and SOAP, and a comparison of \gsii\ and \gsiii\ format statements.
61\newpage
62\tableofcontents
63\newpage
64\section{\gs\ installation and administration}\label{sec:install}
65
66This section covers where to get \gsiii\ from, how to install it and how to run it. The standard method of running \gsiii\ is as a Java servlet. We provide the Tomcat servlet container to run the servlet. Standard web servers may be able to be configured to provide servlet support, and thereby remove the need to use Tomcat. Please see your web server documentation for this. This documentation assumes that you are using Tomcat. To access \gsiii, Tomcat must be started up, and then it can be accessed via a web browser.
67
68Ant (Java's XML based build tool) is used for compilation, installation and running Greenstone. The build.xml file is the configuration file for the Greenstone project, and build.properties contains parameters that can be altered by the user.
69
70\subsection{Get and install \gs\ }
71
72\gsiii\ is available for download from Sourceforge:\\
73 \gst{https://sourceforge.net/projects/greenstone3}. There are Windows, Linux and Mac OS X releases. They consist of a ZIP/TAR file which should be unpacked. Please check and edit (if necessary) the installation properties in build.properties, then run 'ant install' in the greenstone3 directory. Please read the file README.txt for more detailed (and up to date) instructions.
74
75Greenstone 3 can be started by running 'ant start', and will be available at \gst{http://localhost:8080/greenstone3}\\
76(or \gst{http://your-computer-name:your-chosen-port/greenstone3}). \\
77This gets you to a welcome page containing links to four servlets: the \gst{test} servlet (this allows you to check that Tomcat is running properly); the standard \gst{library} servlet which serves \gst{localsite} site with the \gst{default} interface; the \gst{classic} servlet which serves \gst{localsite} using the \gst{classic} or \gsii-style interface; the \gst{gateway} servlet, which serves \gst{gateway} site with the \gst{default} interface. The \gst{gateway} site uses a SOAP connection to communicate with \gst{localsite}, and demonstrates the library working in a distributed fashion.
78
79\gsiii\ is also available through CVS (Concurrent Versioning System). This provides the latest development version, and is not guaranteed to be stable. Appendix~\ref{app:cvs} describes how to download and install \gsiii\ from CVS.
80
81\subsection{How the library works}
82
83The standard library program is a Java servlet. We use the Tomcat servlet container to present the servlets over the web. Tomcat takes CGI-style URLs and passes the arguments to the servlet, which processes these and returns a page of HTML. As far as an end-user is concerned, a servlet is a Java version of a CGI program. The interaction is similar: access is via a web browser, using arguments in a URL.
84
85Other types of interfaces can be used, such as Java GUI programs. See Section~\ref{sec:new-interfaces} for details about how to make these.
86
87\subsubsection{Restarting the library}
88
89The library program (actually Tomcat and MYSQL) can be restarted by running \gst{ant restart} in the greenstone3 directory.
90
91Tomcat must be restarted any time you make changes in the following for those changes to take effect:\\
92\begin{bulletedlist}
93\begin{gsc}
94\item \gsdlhome/web/WEB-INF/web.xml
95\item \gsdlhome/comms/jakarta/tomcat/conf/server.xml
96\end{gsc}
97\item any classes or jar files used by the servlets
98\end{bulletedlist}
99\noindent Note: stdout and stderr for the servlets (on Linux and Mac OS X) both go to\\
100\gst{\gsdlhome/comms/jakarta/tomcat/logs/catalina.out}
101
102
103\subsection{Directory structure}
104
105Table~\ref{tab:dirs} shows the file hierarchy for \gsiii\ .
106The first part shows the common stuff which can be shared between
107\gs\ users---the source, libraries etc. The second part shows the file hierarchy for the greenstone3/web directory, which comprises the greenstone3 context for Tomcat, and is accessible via Tomcat. The main directories are for sites and interfaces: there can be several sites and interfaces per installation, and they are described in the following section.
108
109
110\begin{table}
111\caption{The \gs\ directory structure}
112\label{tab:dirs}
113{\footnotesize
114\begin{tabular}{l p{8cm}}
115\hline
116\bf directory & \bf description \\
117\hline
118greenstone3
119 & The main installation directory---gsdl3home can be changed to something more standard\\
120greenstone3/src
121 & Source code lives here \\
122greenstone3/src/java/
123 & java source code \\
124greenstone3/packages
125 & Imported packages from other systems e.g. MG, MGPP \\
126greenstone3/lib
127 & Shared library files\\
128greenstone3/lib/java
129 & Java jar files\\
130greenstone3/resources
131 & any resources that may be needed\\
132greenstone3/resources/java
133 & properties files for java resource bundles - used to handle all the language specific text This directory is on the class path, so any other Java resources can be placed here \\
134greenstone3/resources/soap
135 & soap service description files \\
136greenstone3/resources/dtd
137 & \gsiii\ has trouble locating DTD files sometimes. They can go here\\
138greenstone3/bin
139 & executable stuff lives here\\
140greenstone3/bin/script
141 & some Perl and/or shell building scripts\\
142greenstone3/comms
143 & Communication packages: Tomcat and SOAP\\
144greenstone3/docs
145 & Documentation\\
146\hline
147greenstone3/web
148 & This is where the web site is defined. Any static HTML files can go here. This directory is the Tomcat root directory.\\
149greenstone3/web/WEB-INF
150 & The web.xml file lives here (servlet configuration information for Tomcat)\\
151greenstone3/web/WEB-INF/classes
152 & Servlet classes go in here\\
153greenstone3/web/sites
154 & Contains directories for different sites---a site is a set of collections and services served by a single MessageRouter (MR). The MR may have connections (e.g. soap) to other sites\\
155greenstone3/web/sites/localsite
156 & An example site - the site configuration file lives here\\
157greenstone3/web/sites/localsite/collect
158 & The collections directory \\
159greenstone3/web/sites/localsite/images
160 & Site specific images \\
161greenstone3/web/sites/localsite/transforms
162 & Site specific transforms \\
163greenstone3/web/interfaces
164 & Contains directories for different interfaces - an interface is defined by its images and XSLT files \\
165greenstone3/web/interfaces/default
166 & The default interface\\
167greenstone3/web/interfaces/default/images
168 & The images for the default interface\\
169greenstone3/web/interfaces/default/transforms
170 & The XSLT files for the default interface\\
171\hline
172\end{tabular}}
173\end{table}
174
175
176\subsection{Sites and interfaces}\label{sec:sites-and-ints}
177
178[local gs stuff (sites and interfaces) vs installed stuff (code)\\
179where they live, whats the difference, what each contains.]\\
180Sites and interfaces contain the content and presentation information, respectively, for the digital library.
181A site is comprised of a set of collections and possibly some site-wide services. An interface (in this web-based servlet context) is a set of images along with a set of XSLT files used for translating xml output from the library into an appropriate form---HTML in general.
182
183One \gsiii\ installation can have many sites and interfaces, and these can be paired in different combinations. One instantiation of a servlet uses one site and one interface, so every specified pairing results in a new servlet instance. For example, a single site might be served with two different interfaces. This provides different modes of access to the same content. e.g. HTML vs WML, or perhaps providing a completely different look and feel for different audiences. Alternatively, a standard interface may be used with many different sites---providing a consistent mode of access to a lot of different content.
184
185Collections live in the \gst{collect} directory of a site. Any collections that are found in this directory when the servlet is initialised will be loaded up and presented to the user. Collections require valid configuration files, but apart from this, nothing needs to be done to the site to use new collections. Collections added while Tomcat is running will not be noticed automatically. Either the server needs to be restarted, or a configuration request may be sent to the library, triggering a (re)load of the collection (this is described in Section~\ref{sec:runtime-config}).
186
187There are two sites that come with the distribution: \gst{localsite}, and \gst{gateway}. \gst{localsite} has several demo collections, while \gst{gateway} has none. \gst{gateway} specifies that a SOAP connection should be made to \gst{localsite}. Getting this to work involves setting up a soap server for localsite: see Section~\ref{sec:distributed} for details.
188There are also two interfaces provided in the distribution: \gst{default} and \gst{classic}. The default interface is a generic \gsiii\ interface, while the \gst{classic} interface aims to look like the old \gsii\ interface.
189
190Each site and interface has a configuration file which specifies parameters for the site or interface---these are described in Section~\ref{sec:config}.
191
192\subsection{Configuring Tomcat}\label{sec:tomcat-config}
193
194The file \gst{\gsdlhome/web/WEB-INF/web.xml} contains the configuration information for Tomcat. It tells Tomcat what servlets to load, what initial parameters to pass them, and what web names map to the servlets.
195There are four servlets specified in web.xml (these correspond to the four servlet links in the welcome page for \gsiii): one is a test servlet that just prints ``hello greenstone'' to a web page. This is useful if you are having trouble getting Tomcat set up. The other three are the \gs\ library servlets described in Section~\ref{sec:browser-access}, \gst{library}, \gst{classic} and \gst{gateway}. Each servlet must specify which site and which interface to use. Having multiple servlets provides a way of serving different sites, or the same site with a different style of presentation. Site\_name and interface\_name are just two examples of initialisation parameters used by the library servlets. The full list is shown in Table~\ref{tab:serv-init}.
196
197For more details about Tomcat see Appendix~\ref{app:tomcat}.
198
199\begin{table}
200\caption{\gs\ servlet initialisation parameters}
201\label{tab:serv-init}
202{\footnotesize
203\begin{tabular}{llp{5cm}}
204\hline
205\bf name & \bf sample value & \bf description \\
206\hline
207gsdl3\_home & /research/kjdon/greenstone3 & the base directory of the greenstone3 installation \\
208site\_name & localsite & the name of the site to use \\
209interface\_name & default & the name of the interface to use\\
210library\_name & library & the web name of the servlet \\
211default\_lang & en & the default language for the interface\\
212receptionist\_class & NZDLReceptionist & (optional) specifies an alternative Receptionist to use\\
213messagerouter\_class & NewMessageRouter & (optional) specifies an alternative MessageRouter to use\\
214params\_class & NZDLParams & (optional) specifies an alternative GSParams class to use \\
215\hline
216\end{tabular}}
217\end{table}
218
219\subsection{Configuring a \gs\ library}\label{sec:config}
220
221Initial \gsiii\ system configuration is determined by a set of configuration files, all expressed in XML. Each site has a configuration file that binds parameters for the site, \gst{siteConfig.xml}. Each interface has a configuration file, \gst{interfaceConfig.xml}, that specifies Actions for the interface. Collections also have several configuration files; these are discussed in Section~\ref{sec:collconfig}.
222The configuration files are read in when the system is initialised, and their contents are cached in memory. This means that changes made to these files once the system is running will not take immediate effect. Tomcat needs to be restarted for changes to the interface configuration file to take effect. However, changes to the site configuration file can be incorporated sending a system command to the library. There are a series of system commands that can be sent to the library to induce reconfiguration of different modules, including reloading the whole site. This removes the need to restart the system to reflect these changes. These commands are described in Section~\ref{sec:runtime-config}.
223
224\subsubsection{Site configuration file}\label{sec:siteconfig}
225
226The file \gst{siteConfig.xml} specifies the URI for the site (\gst{localSiteName}), the HTTP address for site resources (\gst{httpAddress}), any ServiceClusters that the site provides (for example, collection building), any ServiceRacks that do not belong to a cluster or collection, and a list of
227known external sites to connect to. Collections are not specified in the site
228configuration file, but are determined by the contents of the site's
229collections directory.
230
231The HTTP address is used for retrieving resources from a site outside the XML protocol. Because a site is HTTP accessible through Tomcat, any files (e.g. images) belonging to that site or to its collections can be specified in the HTML of a page by a URL. This avoids having to retrieve these files from a remote site via the XML protocol\footnote{Currently, sites live inside the Tomcat greenstone3 root context, and therefore all their content is accessible over HTTP via the Tomcat address. We need to see if parts can be restricted. Also, if we use a different protocol, then resources from remote sites may need to come through the XML. Also, if we are running locally without using Tomcat, we may want to get them via file:// rather than http://.}.
232
233Figure~\ref{fig:siteconfig} shows two example site configuration files. The first example is for a rudimentary site with no site-wide services,
234which does not connect to any external sites. The second example is for a site with one site-wide service cluster - a collection building cluster. It also connects to the first site using SOAP.
235These two sites happen to be running on the same machine, which is why they can use \gst{localhost} in the address. For site \gst{gsdl1} to talk to site \gst{localsite}, a SOAP server must be run for \gst{localsite}. The address of the SOAP server, in this case, is \gst{http://localhost:8080/soap/servlet/rpcrouter}.
236
237
238\begin{figure}
239\begin{gsc}\begin{verbatim}
240<siteConfig>
241 <localSiteName value="org.greenstone.localsite"/>
242 <httpAddress value="http://localhost:8080/greenstone3/sites/localsite"/>
243 <serviceClusterList/>
244 <serviceRackList/>
245 <siteList/>
246</siteConfig>
247\end{verbatim}\end{gsc}
248
249\begin{gsc}\begin{verbatim}
250<siteConfig>
251 <localSiteName value="org.greenstone.gsdl1"/>
252 <httpAddress value="http://localhost:8080/greenstone3/sites/gsdl1"/>
253 <serviceClusterList>
254 <serviceCluster name="build">
255 <metadataList>
256 <metadata name="Title">Collection builder</metadata>
257 <metadata name="Description">Builds collections in a
258 gsdl2-style manner</metadata>
259 </metadataList>
260 <serviceRackList>
261 <serviceRack name="GS2Construct"/>
262 </serviceRackList>
263 </serviceCluster>
264 </serviceClusterList>
265 <siteList>
266 <site name="org.greenstone.localsite"
267 address="http://localhost:8080/soap/servlet/rpcrouter"
268 type="soap"/>
269 </siteList>
270</siteConfig>
271\end{verbatim}\end{gsc}
272\caption{Two sample site configuration files}
273\label{fig:siteconfig}
274\end{figure}
275
276\subsubsection{Interface configuration file}\label{sec:interfaceconfig}
277
278The interface configuration file \gst{interfaceConfig.xml} lists all the actions that the interface knows about at the start (other ones can be loaded dynamically). Actions create the web pages for the library: there is generally one Action per type of page. For example, a query action produces the pages for searching, while a document action displays the documents. The configuration file specifies what short name each action maps to (this is used in library URLs for the a (action) parameter) e.g. QueryAction should use a=q. If the interface uses XSLT, it specifies what XSLT file should be used for each action and possibly each subaction. This makes it easy for developers to implement and use different actions and/or XSLT files without recompilation. The server must be restarted, however.
279
280It also lists all the languages that the interface text files have been translated into. These have a \gst{name} attribute, which is the ISO code for the language, and a \gst{displayElement} which gives the language name in that language (note that this file should be encoded in UTF-8). This language list is used on the Preferences page to allow the user to change the interface language. Details on how to add a new language to a \gsiii\ library are shown in Section~\ref{sec:interface-customise}.
281
282\begin{figure}
283\begin{gsc}\begin{verbatim}
284<interfaceConfig>
285 <actionList>
286 <action name='p' class='PageAction'>
287 <subaction name='home' xslt='home.xsl'/>
288 <subaction name='about' xslt='about.xsl'/>
289 <subaction name='help' xslt='help.xsl'/>
290 <subaction name='pref' xslt='pref.xsl'/>
291 </action>
292 <action name='q' class='QueryAction' xslt='basicquery.xsl'/>
293 <action name='b' class='GS2BrowseAction' xslt='classifier.xsl'/>
294 <action name='a' class='AppletAction' xslt='applet.xsl'/>
295 <action name='d' class='DocumentAction' xslt='document.xsl'/>
296 <action name='xd' class='XMLDocumentAction'>
297 <subaction name='toc' xslt='document-toc.xsl'/>
298 <subaction name='text' xslt='document-content.xsl'/>
299 </action>
300 <action name='pr' class='ProcessAction' xslt='process.xsl'/>
301 <action name='s' class='SystemAction' xslt='system.xsl'/>
302 </actionList>
303 <languageList>
304 <language name="en">
305 <displayItem name='name'>English</displayItem>
306 </language>
307 <language name="fr">
308 <displayItem name='name'>Français</displayItem>
309 </language>
310 <language name='es'>
311 <displayItem name='name'>Español</displayItem>
312 </language>
313 </languageList>
314</interfaceConfig>
315\end{verbatim}\end{gsc}
316\caption{Default interface configuration file}
317\label{fig:ifaceconfig}
318\end{figure}
319
320
321\subsection{Run-time re-initialisation}\label{sec:runtime-config}
322
323When Tomcat is started up, the site and interface configuration files are read in, and actions/services/collections loaded as necessary. The configuration is then static unless Tomcat is restarted, or re-configuration commands issued.
324
325There are several commands that can be issued to Tomcat to avoid having to restart the server. These can reload the entire site, or just individual collections. Unfortunately at present there are no commands to reconfigure the interface, so if the interface configuration file has changed, Tomcat must be restarted for those changes to take effect. Similarly, if the Java classes are modified, Tomcat must be restarted then too.
326
327Currently, the runtime configuration commands can only be accessed by typing arguments into the URL; there is no nice web form yet to do this.
328
329The arguments are entered after the \gst{library?} part of the URL. There are three types of commands: configure, activate, deactivate\footnote{There is no security for these commands yet in \gs, so the deactivate/delete command is disabled}. These are specified by \gst{a=s\&sa=c}, \gst{a=s\&sa=a}, and \gst{a=s\&sa=d}, respectively (\gst{a} is action, \gst{sa} is subaction). By default, the requests are sent to the MessageRouter, but they can be sent to a collection/cluster by the addition of \gst{sc=xxx}, where \gst{xxx} is the name of the collection or cluster. Table~\ref{tab:run-time config} describes the commands and arguments in a bit more detail.
330
331\begin{table}
332\caption{Example run-time configuration arguments.}
333\label{tab:run-time config}
334{\footnotesize
335\begin{tabular}{lp{8cm}}
336\hline
337\gst{a=s\&sa=c} & reconfigures the whole site. Reads in siteConfig.xml, reloads all the collections. Just part of this can be specified with another argument \gst{ss} (system subset). The valid values are \gst{collectionList}, \gst{siteList}, \gst{serviceList}, \gst{clusterList}. \\
338\gst{a=s\&sa=c\&sc=XXX} & reconfigures the XXX collection or cluster. \gst{ss} can also be used here, valid values are \gst{metadataList} and \gst{serviceList}. \\
339\gst{a=s\&sa=a} & (re)activate a specific module. Modules are specified using two arguments, \gst{st} (system module type) and \gst{sn} (system module name). Valid types are \gst{collection}, \gst{cluster} \gst{site}.\\
340\gst{a=s\&sa=d} & deactivate a module. \gst{st} and \gst{sn} can be used here too. Valid types are \gst{collection}, \gst{cluster}, \gst{site}, \gst{service}. Modules are removed from the current configuration, but will reappear if Tomcat is restarted.\\
341\gst{a=s\&sa=d\&sc=XXX} & deactivate a module belonging to the XXX collection or cluster. \gst{st} and \gst{sn} can be used here too. Valid types are \gst{service}. \\
342\hline
343\end{tabular}}
344\end{table}
345\newpage
346\section{Using \gsiii\ }\label{sec:user}
347
348Once \gsiii\ is installed, the sample collections can be accessed. The installation comes with several example collections, and Section~\ref{sec:usecolls} describes these collections and how to use them. Section~\ref{sec:buildcol} describes how to build new collections.
349
350\subsection{Using a collection}\label{sec:usecolls}
351
352A collection typically consists of a set of documents, which could be text, HTML, word, PDF, images, bibliographic records etc, along with some access methods, or ``services''. Typical access methods include searching or browsing for document identifiers, and retrieval of content or metadata for those identifiers.
353Searching involves entering words or phrases and getting back lists of documents that contain those words. The search terms may be restricted to particular fields of the document.
354
355Browsing involves navigating pre-defined hierarchies of documents, following links of interest to find documents. The hierarchies may be constructed on different metadata fields, for example, alphabetical lists of Titles, or a hierarchy of Subject classifications. Clicking on a bookshelf icon takes you to a lower level in the hierarchy, while clicking on a book or page icon takes you to a document.
356
357In the standard interface that comes with \gsiii\ \footnote{of course, this is all customisable}, collections in a digital library are presented in the following manner. The 'home' page of the library shows a list of all the public collections in that library. Clicking on a collection link takes you to the home page for the collection, which we call the collection's 'about' page. The standard page banner looks something like that shown in Figure~\ref{fig:page-banner}.
358
359\begin{figure}[h]
360 \centering
361 \includegraphics[width=4in]{pagebanner.ps} %5.8
362 \caption{A sample collection page banner}
363 \label{fig:page-banner}
364\end{figure}
365
366The image at the top left is a link to the collection's home page. The top right has buttons to link to the library home page, help and preferences pages. All the available services are arrayed along a navigation bar, along the bottom of the banner. Clicking on a name accesses that service.
367
368Search type services generally provide a form to fill in, with parameters including what field or granularity to search, and the query itself. Clicking the search button carries out the search, and a list of matching documents will be displayed. Clicking on the icons in the result list takes you to the document itself.
369
370Once you are looking at a document, clicking the open book icon at the top of the document, underneath the navigation bar, will take you back to the service page that you accessed the document from.
371
372\subsection{Building a collection}\label{sec:buildcol}
373
374There are three ways to get a new collection into \gsiii. The first is to build it using the \gsiii\ command line building process. The second way is to use the Greenstone Librarian Interface to build a new collection. This creates a collection in a \gsiii\ context, but uses the \gsii\ Perl collection building process. The third way is to import a pre-built \gsii\ collection.
375
376Collections live in the collect directory of a site. As described in Section~\ref{sec:sites-and-ints}, there can be several sites per \gsiii\ installation. The collect directory is at \gst{\$GSDL3HOME/web/sites/site-name/collect}, where site-name is the name of the site you want your new collection to belong to.
377
378The following three sections describe how to create a collection from scratch, using command line and GLI building, and how to import a \gsii\ collection. Once a collection has been built (and is located in the collect directory), the library server needs to be notified that there is a new collection. This can be accomplished in two ways\footnote{and eventually there will also probably be automatic polling for new collections}. If you are the library administrator, you can restart Tomcat. The library servlet will then be created afresh, and will discover the new collection when it scans the collect directory for the collection list. Alternatively, an activate collection command can be issued to the servlet, using the arguments \gst{a=s\&sa=a\&st=collection\&sn=collname}, where \gst{collname} should be replaced with the collection name---this tells the library program to (re)load the \gst{collname} collection.
379
380
381\subsubsection{Creating a collection from scratch}
382
383To create the director
384Building native \gsiii\ collections is done using the \gst{gs3-build.sh/bat} script, with the \gst{collectionConfig.xml} file controlling how the building is done. There are a number of considerations in building a collection: what documents appear in the collection, how they are indexed for searching, which classifications are used for browsing, etc.
385
386Firstly, the documents that comprise the collection should be placed in the import subdirectory. At present, only documents in this directory will appear in the collection. Documents can be organised into sub folders inside the import directory.
387[TODO: describe the kinds of documents that can be added, something about METS files?]
388
389Metadata for documents can be added using metadata.xml files. These files have already been used in \gsii, and the format is the same in \gsiii. A metadata.xml file has a root element of \gst{<DirectoryMetadata>}. This encloses a series of \gst{<FileSet>} items. Neither of these tags has any attributes. Each \gst{<FileSet>} item includes two parts: firstly, one or more \gst{<FileName>} tags, each of which encloses a regular expression to identify the files which are to be assigned the metadata. Only files in the same directory as the metadata.xml, or in one of its child directories, will be selected. The filename tag encloses the regular expression as text, e.g.:
390
391\begin{gsc}\begin{verbatim}
392<FileName>example</FileName>
393\end{verbatim}\end{gsc}
394
395This would match any file containing the text 'example' in its name. The second part of the \gst{<FileSet>} item is a \gst{<Description>} item. The \gst{<Description>} tag has no attributes, but encloses one or more \gst{<Metadata>} tags. Each \gst{<Metadata>} tag contains one metadata item, i.e. a label to describe the metadata and a corresponding value. The \gst{<Metadata>} tag has one compulsory attribute: ``name''. This attribute gives the metadata label to add to the document. Each \gst{<Metadata>} tag also has an optional attribute: ``mode''. If this attribute is set to ``accumulate'' then the value is added to the document, and any existing values for that metadata item are retained. If the attribute is set to ``set'' or is omitted, then any existing value of the metadata item will be deleted.
396
397\begin{figure}
398\begin{gsc}\begin{verbatim}
399<?xml version="1.0" encoding="UTF-8"?>
400<!DOCTYPE DirectoryMetadata SYSTEM "http://greenstone.org/dtd/DirectoryMetadata
401 /1.0/DirectoryMetadata.dtd">
402<DirectoryMetadata>
403 <FileSet>
404 <FileName>ec160e</FileName>
405 <Description>
406 <Metadata name="Title">The Courier - No.160 - Nov - Dec 1996 -
407 Dossier Habitat - Country reports: Fiji , Tonga (ec160e)</Metadata>
408 <Metadata mode="accumulate" name="Language">English</Metadata>
409 <Metadata mode="accumulate" name="Subject">Settlements and housing:
410 general works incl. low- cost housing, planning techniques, surveying,
411 etc.</Metadata>
412 <Metadata mode="accumulate" name="Subject">The Courier ACP 1990 - 1996
413 Africa-Caribbean-Pacific - European Union</Metadata>
414 <Metadata mode="accumulate" name="Organization">EC Courier</Metadata>
415 <Metadata mode="accumulate" name="AZList">T.1</Metadata>
416 </Description>
417 </FileSet>
418 <FileSet>
419 <FileName>b22bue</FileName>
420 <Description>
421 <Metadata name="Title">Butterfly Farming in Papua New Guinea
422 (b22bue)</Metadata>
423 <Metadata mode="accumulate" name="Language">English</Metadata>
424 <Metadata mode="accumulate" name="Subject">Other animals (micro-
425 livestock, little known animals, silkworms, reptiles, frogs,
426 snails, game, etc.)</Metadata>
427 <Metadata mode="accumulate" name="Organization">BOSTID</Metadata>
428 <Metadata mode="accumulate" name="AZList">T.1</Metadata>
429 <Metadata mode="accumulate" name="Keyword">start a butterfly farm
430 </Metadata>
431 </Description>
432 </FileSet>
433</DirectoryMetadata>
434\end{verbatim}\end{gsc}
435\caption{Sample metadata.xml file}
436\label{fig:metadatafile}
437\end{figure}
438
439Figure~\ref{fig:metadatafile} shows an example metadata.xml file.
440Here, only one file pattern is found in each file set. However, the \gst{Description} tag contains a number of separate metadata items. Note that the \gst{Title} metadata does not have the \gst{mode=accumulate} attribute. This means that when the title is assigned to a document, its existing \gst{Title} information will be lost.
441
442The basic means of finding documents in \gs\ is search. Options for building the search indexes include which indexer to use, what granularity to use for the indexes (e.g. whether to index documents as a whole, or sections of documents), what content the index should have (the whole text of the document or one or many metadata fields). Section-level indexes allow a reader to recall part of a document (for instance, a chapter) rather than the entire document. However, \gsiii\ must be able to identify the internal structure of the document to achieve this. The degree to which structure can be found varies from file format to file format.
443
444An alternative means of finding documents is through browsing. Greenstone can create pre-defined browsing hierarchies based on document metadata. Each browsing structure is called a classifier. Options for building classifiers include what type of classifier to use (linear list or multi-level hierarchy), what metadata to build the classifier on, e.g. Title, Author etc.
445
446The collectionConfig.xml file controls the all of these options for collection building, and the format is described in Section~\ref{sec:collconfig}.
447
448To build a collection, place the source documents and optional metadata.xml file(s) in the import directory, place the \gst{collectionConfig.xml} file in the etc directory, and execute \gst{gs3build.sh/bat sitename collectionname}. The process will run, placing the new indexes in the \gst{building} subdirectory of the collection's directory. You must have mysql running before you start building---running \gst{ant start} will start up the MySQL server as well as tomcat.
449
450Once the build process is complete, the building directory should be renamed to index (after deleting or renaming the existing index directory, if any), and Tomcat prompted to reload the collection---either by restarting the server, or by sending an activate collection command to the library servlet.
451
452\subsubsection{Using the Librarian Interface}
453
454The Greenstone Librarian Interface (GLI) can be used to create \gsii\ style collections for \gsiii. It can be started under Windows by selecting Greenstone Librarian Interface from the Greenstone 3 Digital Library menu in the Program Files section of the Start menu. On Linux, run \gst{./gli4gs3.sh} from the \gst{greenstone3/gli} directory.
455
456Currently, the GLI works almost exactly the same as for \gsii\footnote{Eventually the GLI will be modified to use native \gsiii\ config files and collection building}. Collection configuration is done in a \gsii\ manner. The main difference is that \gsiii\ has different sites and interfaces and servlets, whereas \gsii\ has a single collect directory, and a single runtime cgi program.
457
458The GLI for \gsiii\ has a couple of new configuration parameters: site and servlet. It operates within a single site---you can edit, delete, create new collections within this site. A servlet is also specified for that site---this is used when previewing a collection. While you are working in one site, you cannot edit collections from another site. However, you can base a collection on one from another site. To change the working site and/or servlet, go to Preferences-$>$Connection in the File menu. By default, the GLI will use site \gst{localsite}, and servlet \gst{library}.
459
460Collection building using the GLI will use the \gsii\ Perl scripts and plugins. At the conclusion of the \gsii\ build process, a conversion script will be run to create the \gsiii\ configuration files. This means that format statements are no longer 'live'---changing these will require changes to the \gsiii\ config files. You can either rebuild the collection through the GLI (may take a while), or run the conversion script directly (see following section).
461
462Detailed instructions about using the GLI can be found in Sections 3.1 and 3.2 of the Greenstone 2 User's Guide (\gst{GS2-User-en.pdf}. This can be found in your \gsii\ installation, or in the greenstone3/docs/manual directory if you have installed \gsiii\ from a distribution.
463
464
465\subsubsection{Importing a \gsii\ collection}
466
467
468Pre-built \gsii\ collections can also be used in \gsiii\footnote{For information about the \gsii\ software, and how to build collections using it, visit \gst{www.greenstone.org}}. The collection folder should be copied to the collect directory of the site it is to appear in (or a symbolic link may be used if possible).
469The \gsiii\ run time system requires different configuration files for a collection, so you need to run a conversion script. All this does is create the new collectionConfig.xml and buildConfig.xml from the old collect.cfg and build.cfg files. It does not change the collection in any way, so it can still be used by \gsii\ software.
470
471The conversion script is \gst{convert\_coll\_from\_gs2.pl}. To run it, make sure you have run \gst{source setup.bash} (or \gst{setup} in Windows) in your top-level gsdl directory of the \gsii\ installation (as well as running the standard \gst{gs3-setup} command). Then you need to specify the path to the collect directory and the collection name as parameters to the conversion script. For example,
472
473\begin{gsc}
474\begin{verbatim}
475convert_coll_from_gs2.pl -collectdir
476 $GSDL3HOME/web/sites/localsite/collect demo
477\end{verbatim}
478\end{gsc}
479%$
480The script attempts to create \gsiii\ format statements from the old \gsii\ ones. The conversion may not always work properly, so if the collection looks a bit strange under \gsiii\ , you should check the format statements. Format statements are described in Section~\ref{sec:formatstmt}.
481
482Once again, to have the collection recognised by the library servlet, you can either restart Tomcat, or load it dynamically.
483
484\subsection{Collection configuration files}\label{sec:collconfig}
485
486Each collection has two, or possibly three, configuration files, \gst{collectionConfig.xml} and \gst{buildConfig.xml}, and optionally \gst{collectionInit.xml}, that give metadata, display and other information for the
487collection.\footnote{For collections imported from \gsii, \gst{collectionConfig.xml} and \gst{buildConfig.xml}are generated from \gst{collect.cfg} and \gst{build.cfg}.} The first includes user-defined presentation metadata for the collection,
488such as its name and the {\em About this collection} text; gives formatting information for the collection display; and also gives
489instructions on how the collection is to be built. The second is produced by
490the build-time process and includes any metadata that can be determined
491automatically. It also includes configuration information for any ServiceRacks needed by the collection.
492
493All the configuration files should be encoded using UTF-8.
494
495\subsubsection{collectionInit.xml}
496
497This optional file is only used for non-standard, customised collections. It specifies the class name of the non-standard collection class. The only syntax so far is the class name:
498
499\begin{gsc}\begin{verbatim}
500<collectionInit class="XMLCollection"/>
501\end{verbatim}\end{gsc}
502
503Section~\ref{sec:new-coll-types} describes an example collection where this file is used. Depending on the type of collection that this is used for, one or both of the other config files may not be needed.
504
505\subsubsection{collectionConfig.xml}
506
507The collection configuration file is where the collection designer (e.g. a librarian) decides what form the collection should take. This includes the collection metadata such as title and description, and also includes what indexes and browsing structures should be built. The format of \gst{collectionConfig.xml} is still under consideration. However, Figure~\ref{fig:collconfig} shows the parts of it that have been defined so far.
508
509Display elements for a collection or metadata for a document can be entered in any language---use lang='en' attributes to metadata elements to specify which language they are in.
510
511\begin{figure}
512\begin{gsc}\begin{verbatim}
513<collectionConfig xmlns:gsf="http://www.greenstone.org/configformat"
514 xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
515 <metadataList>
516 <metadata name="creator">[email protected]</metadata>
517 </metadataList>
518 <displayItemList>
519 <displayItem name="name" lang="en">Greenstone 3 demo</displayItem>
520 <displayItem name="icon" lang="en">gs3demo.gif</displayItem>
521 <displayItem name="smallicon" lang="en">gs3demosm.gif</displayItem>
522 <displayItem name="description" lang="fr">Il s'agit d'une collection
523 de démonstration pour le logiciel Greenstone. Elle contient
524 seulement un petit échantillon des Bibliothèques humanitaires
525 pour le Développement (11 documents).</displayItem>
526 <displayItem name="description" lang="en">This is a demonstration
527 collection for the Greenstone digital library software. It contains
528 a small subset (11 books) of the Humanity Development Library. It
529 is built with mg using Greenstone 3 native building.</displayItem>
530 </displayItemList>
531 <search type='mg'>
532 <index name="i1">
533 <field>text</field>
534 <level>document</level>
535 <displayItem name='name' lang="en">entire documents</displayItem>
536 <displayItem name='name' lang="fr">documents entiers</displayItem>
537 <displayItem name='name' lang="es">documentos enteros</displayItem>
538 </index>
539 <index name="i2">
540 <field>text</field>
541 <level>section</level>
542 <displayItem name='name' lang="en">chapters</displayItem>
543 <displayItem name='name' lang="fr">chapitres</displayItem>
544 <displayItem name='name' lang="es">capítulos</displayItem>
545 </index>
546 <format>
547 <gsf:template match="documentNode">
548 <td valign='top'><gsf:link><gsf:icon/></gsf:link></td>
549 <td><gsf:metadata name='Title' /></td>
550 </gsf:template>
551 </format>
552 </search>
553 <browse>
554 <classifier name="CL1" type="AZList" horizontalAtTop='true'>
555 <field>Title</field>
556 <sort>Title</sort>
557 <displayItem name='name' lang='en'>Titles</displayItem>
558 </classifier>
559 <classifier name="CL2" type="Hierarchy">
560 <field>Organization</field>
561 <sort>Title</sort>
562 <displayItem name='name' lang='en'>Organizations</displayItem>
563 <file URL="/research/kjdon/home/greenstone3/web/sites/localsite/collect/
564 gs3test/etc/org.xml"/>
565 </classifier>
566 <classifier name="CLKeyword" type="Hierarchy">
567 <field>Keyword</field>
568 <sort>Title</sort>
569 <displayItem name='name' lang='en'>HowTo</displayItem>
570 <file URL="/research/kjdon/home/greenstone3/web/sites/localsite/collect/
571 gs3test/etc/keyword.xml"/>
572 <format>
573 <gsf:template match="documentNode">
574 <br /><gsf:link><gsf:metadata name='Keyword' />
575 </gsf:link></gsf:template>
576 </format>
577 </classifier>
578 </browse>
579 <display/>
580</collectionConfig>
581\end{verbatim}\end{gsc}
582[TODO: add in building instructions for the classifiers]
583\caption{Sample collectionConfig.xml file (gs3demo collection)}
584\label{fig:collconfig}
585\end{figure}
586
587The \gst{<metadataList>} element specifies some collection metadata, such as creator. The \gst{<displayItemList>} specifies some language dependent information that is used for collection display, such as collection name and short description. These displayItem elements can be specified in different languages.
588
589The \gst{<search>} element specifies what indexes should be built, and provides some display and formatting information for each one. Search has an attribute, \gst{type}, which specifies which indexer to be used for indexing. Currently, \gst{mg} and \gst{mgpp}[??] are available. If type is not specified, mg is used. Multiple search elements may be specified, if more than one indexer is to be used. (Note, this is not yet recognised by the run-time system.)
590
591Search indexes appear as individual \gst{<index>} elements within the \gst{<search>} element. Some choices for the index are made using attributes of the element itself, and some through child elements.
592
593Each index must have a unique name, which is used to identify it within \gsiii\ The name is given as an attribute of the \gst{<index>} element.
594
595The other choices are described using child elements of \gst{<index>}. The \gst{<level>} tag indicates the index level and the \gst{<field>} tag the text to be used. The \gst{<level>} tag can contain one of document, section or paragraph, while the \gst{<field>} tag can contain ``text'' or the name of a metadata field. If the \gst{<level>} tag is omitted, the default setting is to index by document, and if the \gst{<field>} tag is omitted, the default setting is to index the document text.
596
597Example index specifications include:
598
599[NOTE: I think we shouldn't have default level and field and that it must be specified--kjdon]
600
601To index only the title of each separate document in the collection:
602\begin{gsc}\begin{verbatim}
603<index name="dtt">
604 <level>document</level>
605 <field>dc:title</field>
606</index>
607\end{verbatim}\end{gsc}
608...in this case the \gst{<field>} tag refers to the ``title'' metadata item, found in the Dublin Core namespace. The mg search engine would be used on this index.
609
610Alternatively, to index the full document texts by section:
611\begin{gsc}\begin{verbatim}
612<index name="stx">
613 <level>section</level>
614</index>
615\end{verbatim}\end{gsc}
616...or...
617\begin{gsc}\begin{verbatim}
618<index name="stx">
619 <level>section</level>
620 <field>text</field>
621</index>
622\end{verbatim}\end{gsc}
623...in the first example, the \gst{<field>} tag is not explicitly defined, and would default to 'text', whereas it is explicitly set to 'text' in the second example. As they are of the same name, they should not appear in the same \gst{collectionConfig.xml} file.
624
625Moving onto \gst{<classifier>} items, the format is broadly similar to \gst{<index>} items, but with a couple of different choices. Firstly, each classifier should have ``name'' and ``type'' attributes. In the case of \gst{<classifier>} items the ``type'' attribute identifies the type of classifier it is. At present, this should either be ``Hierarchy'' or ``AZList''.
626
627The remaining choices for the classifier should follow as child elements of the \gst{<classifier>} element. The \gst{<file>} element should contain the name of the file that describes the classifier as its ``URL'' attribute. The format of this file varies from classifier type to classifier type. The \gst{<field>} element identifies the name of the field to index. More than one \gst{<field>} element may appear if two or more metadata fields are to be used with the classifier. Finally, the \gst{<sort>} item identifies another metadata field which the items within one classifier node are to be ordered. Unlike the \gst{<index>} element, the \gst{<classifier>} element does not have default, assumed values for its children.
628
629Figure~\ref{fig:hierarchyfile} shows the format of the file for a Hierarchy classifier. [TODO add a description]
630\begin{figure}
631\begin{gsc}\begin{verbatim}
632<Hierarchy>
633 <Classification>
634 <Name>ACCU</Name>
635 <Path>1</Path>
636 <Description>ACCU</Description>
637 </Classification>
638 <Classification>
639 <Name>Agenda 21</Name>
640 <Path>2</Path>
641 <Description>Agenda 21</Description>
642 </Classification>
643 <Classification>
644 <Name>FAO</Name>
645 <Path>3</Path>
646 <Description>FAO</Description>
647 <Children>
648 <Classification>
649 <Name>FAO Better Farming series</Name>
650 <Path>3.1</Path>
651 <Description>FAO Better Farming Series</Description>
652 </Classification>
653 </Children>
654 </Classification>
655</Hierarchy>
656\end{verbatim}\end{gsc}
657\caption{Sample Hierarchy classifier file}
658\label{fig:hierarchyfile}
659\end{figure}
660
661Inside the \gst{<search>} and \gst{<browse>} elements, \gst{<displayItem>} elements are used to provide titles for the indexes or classifiers, while \gst{<format>} elements provide formatting instructions, typically for a document or classifier node in a list of results. Placing the \gst{<format>} instructions at the top level in the search or browse element will apply the format to all the indexes or classifiers, while placing it inside an individual index or classifier element will restrict that formatting instruction to that item.
662
663The \gst{<display>} element contains optional formatting information for the display of documents. Templates that can be specified here include \gst{documentHeading}, \gst{DocumentContent}. Other formatting options may also be specified here, such as whether to display a table of contents and/or cover image for the documents.
664
665Format elements are described in Section~\ref{sec:formatstmt}.
666
667An optional \gst{<replaceList>} element can be included at the top level. This contains a list of strings and their replacements. This is particularly useful for Greenstone 2 collections that use macros.
668
669The format is like the following:
670\begin{gsc}\begin{verbatim}
671<replaceList>
672<replace scope='text' macro="xxx" text="yyy"/>
673<replace scope='metadata' macro="xxx" bundle="yyy" key="zzz"/>
674<replace scope='all' macro='xxx' metadata='yyy'/>
675</replaceList>
676\end{verbatim}\end{gsc}
677
678Scope determines on what text the replacements are carried out: text, metadata, or both (all). An empty scope attribute is equivalent to scope=all. Each replace type can be used with all scope values. Replacing uses Java's 'String.replaceAll' functionality, so macro and replacement text are actually regular expressions. The first example is a straight textual replacement. The second example uses dictionary lookups. xxx will be replaced with the (language-dependent) value for key zzz in resource bundle yyy. The third example uses metadata: xxx will be replaced by the value of the yyy metadata for that document.
679
680Appendix~\ref{app:gs2replace} gives some examples that have been used for Greenstone 2 collections.
681
682\subsubsection{buildConfig.xml}\label{sec:buildconfig}
683
684The file \gst{buildConfig.xml} is produced by the collection building process. Generally it is not necessary to look at this file, but it can be useful in determining what went wrong if the collection doesn't appear quite the way it was planned.
685
686It contains metadata and other information about the collection that can
687be determined automatically, such as the number of
688documents it contains. It also includes a list of ServiceRack classes that are
689required to provide the services that have been built into the
690collection. The serviceRack names are Java classes that are loaded
691dynamically at runtime. Any information inside the serviceRack element is
692specific to that service---there is no set format. Figure~\ref{fig:buildconfig} shows an example. This configuration file specifies that the collection should load up 3 ServiceRacks: \gst{GS2MGPPRetrieve}, \gst{GS2MGPPSearch}, and \gst{PhindPhraseBrowse}. The contents of each \gst{<serviceRack>} element are passed to the appropriate ServiceRack objects for configuration. The \gst{collectionConfig.xml} file content is also passed to the ServiceRack objects at configure time---the \gst{format} and \gst{displayItem} information is used directly from the \gst{collectionConfig.xml} file rather than added into \gst{buildConfig.xml} during building. This enables formatting and metadata changes in \gst{collectionConfig.xml} to take effect in the collection without rebuilding being necessary. However, as these files are cached, the collection needs to be reloaded for the changes to appear in the library.
693
694
695\begin{figure}
696\begin{gsc}\begin{verbatim}
697<buildConfig xmlns:gsf="http://www.greenstone.org/configformat">
698 <metadataList>
699 <metadata name="numDocs">11</metadata>
700 </metadataList>
701 <serviceRackList>
702 <serviceRack name="GS2MGPPRetrieve">
703 <defaultLevel name="Sec" />
704 <classifierList>
705 <classifier name="CL1" content="Subject" />
706 <classifier name="CL2" content="Title" horizontalAtTop="true" />
707 <classifier name="CL3" content="Organization" />
708 <classifier name="CL4" content="Keyword" />
709 </classifierList>
710 </serviceRack>
711 <serviceRack name="PhindPhraseBrowse" />
712 <serviceRack name="GS2MGPPSearch">
713 <defaultLevel name="Sec" />
714 <levelList>
715 <level name="Doc" />
716 <level name="Sec" />
717 <level name="Para" />
718 </levelList>
719 <fieldList>
720 <field shortname="ZZ" name="allfields" />
721 <field shortname="TX" name="text" />
722 <field shortname="TI" name="Title" />
723 <field shortname="SU" name="Subject" />
724 <field shortname="ORG" name="Organization" />
725 <field shortname="SO" name="Source" />
726 </fieldList>
727 <searchTypeList>
728 <searchType name="plain" />
729 <searchType name="form" />
730 </searchTypeList>
731 <defaultIndex name="idx" />
732 <indexList>
733 <index name="idx" />
734 </indexList>
735 </serviceRack>
736 </serviceRackList>
737</buildConfig>
738\end{verbatim}\end{gsc}
739\caption{Sample buildConfig.xml file (mgppdemo collection)}
740\label{fig:buildconfig}
741\end{figure}
742
743\subsection{Formatting the collection}\label{sec:formatstmt}
744
745Part of collection design involves deciding how the collection should look. \gsiii\ has a default 'look' for a collection, so this is optional. However, the default may not suit the purposes of some collections, so many parts to the look of a collection can be determined by the collection designer.
746
747In standard \gsiii\ , the library is served to a web browser by a servlet, and the HTML is generated using XSLT. XSLT templates are used to format all the parts of the pages. These templates can be overridden by including them in the \gst{collectionConfig.xml} file. Some commonly overridden templates are those for formatting lists: search results list, classifier browsing hierarchies, and for parts of the document display.
748
749Real XSLT templates for formatting search results or classifier lists are quite complicated, and not at all easy for a new user to write. For example, the following is a sample template for formatting a classifier list, to show Keyword metadata as a link to the document.
750
751\begin{gsc}\begin{verbatim}
752<xsl:template match="documentNode" priority="2"
753 xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
754 <xsl:param name="collName"/>
755 <td><a href="{\$library_name}?a=d&amp;c={\$collName}&amp;
756 d={@nodeID}&amp;dt={@docType}"><xsl:value-of
757 select="metadataList/metadata[@name='Keyword']"/></a>
758 </td>
759</xsl:template>
760 \end{verbatim}\end{gsc}
761
762To write this, the user would need to know that:
763\begin{bulletedlist}
764\item the variable \gst{\$library\_name} exists,
765\item the collection name is passed in as a parameter called \gst{collName}
766\item metadata for a document is found in a \gst{<metadataList>} and that its form is \gst{<metadata name="Keyword">the value</metadata>}
767\item the arguments needed for the link to the document are \gst{a, sa, c, d, a, dt}.
768\end{bulletedlist}
769
770Since XSLT is written in XML, we can use XSLT to transform XML into XSLT. \gsiii\ provides a simplified set of formatting commands, written in XML, which will be transformed into proper XSLT. The user specifies a \gst{<gsf:template>} for what they want to format---these typically match \gst{documentNode} or \gst{classifierNode} (for node in a classification hierarchy).
771
772The template at the start of this section can be represented as:
773
774\begin{gsc}\begin{verbatim}
775<gsf:template match='documentNode'>
776 <td><gsf:link><gsf:metadata name='Keyword'/></gsf:link></td>
777</gsf:template>
778\end{verbatim}\end{gsc}
779
780Table~\ref{tab:gsf-format} shows the set of 'gsf' (Greenstone Format) elements. If you have come from a \gsii\ background, Appendix~\ref{app:gs2format} shows \gsii\ format elements and their equivalents in \gsiii\ .
781
782\begin{table}
783\caption{Format elements for GSF format language}
784\label{tab:gsf-format}
785{\footnotesize
786\begin{tabular}{p{6.5cm}p{6.5cm}}
787\hline
788\bf Element & \bf Description \\
789\hline
790\gst{<gsf:text/>} & The document's text\\
791\hline
792\gst{<gsf:link>...</gsf:link>} & The HTML link to the document itself \\
793\gst{<gsf:link type='document'>...
794</gsf:link>} & Same as above\\
795\gst{<gsf:link type='classifier'>...
796</gsf:link>} & A link to a classification node (use in classifierNode templates)\\
797\gst{<gsf:link type='source'>...
798</gsf:link>} & The HTML link to the original file---set for documents that have been converted from e.g. Word, PDF, PS \\
799\hline
800\gst{<gsf:icon/>} & An appropriate icon\\
801\gst{<gsf:icon type='document'/>} & same as above\\
802\gst{<gsf:icon type='classifier'/>} & bookshelf icon for classification nodes\\
803\gst{<gsf:icon type='source'/>} & An appropriate icon for the original file e.g. Word, PDF icon\\
804\hline
805\gst{<gsf:metadata name='Title'/>} & The value of a metadata element for the current document or section, in this case, Title\\
806\gst{<gsf:metadata name='Title' select='select-type' [separator='y' multiple='true']/>} & A more extended selection of metadata values. The select field can be one of those shown in Table~\ref{tab:gsf-select-types}. There are two optional attributes: separator gives a String that will be used to separate the fields, default is ``, ``, and if multiple is set to true, looks for multiple values at each section.\\
807\gst{<gsf:metadata name='Date' format='formatDate'/>} & The value of a metadata element for the current document, formatted in some way. Current formatting options available are formatDate: turns '20040201' into '01 February 2004', and formatLanguage: turns 'en' into 'English', both in a language dependent manner. \\
808\hline
809\gst{<gsf:choose-metadata>
810 <gsf:metadata name='metaA'/>
811 <gsf:metadata name='metaB'/>
812 <gsf:metadata name='metaC'/>
813</gsf:choose-metadata>}
814 & A choice of metadata. Will select the first existing one. the metadata elements can have the select, separator and multiple attributes like normal.\\
815\hline
816\gst{<gsf:switch preprocess=
817'preprocess-type'>
818<gsf:metadata name='Title'/>
819<gsf:when test='test-type'
820test-value='xxx'>...</gsf:when>
821<gsf:when test='test-type'
822test-value='yyy'>...</gsf:when>
823<gsf:otherwise>...</gsf:otherwise>
824</gsf:switch>} & switch on the value of a particular metadata - the metadata is specified in gsf:metadata, has the same attributes as normal.\\
825\hline
826\end{tabular}}
827\end{table}
828
829The \gst{<gsf:metadata>} elements are used to output metadata values. The simplest case is \gst{<gsf:metadata name='Title'/>}---this outputs the Title metadata for the current document or section. Namespaces are important here: if the Title metadata is in the Dublin Core (dc) namespace, then the element should look like \gst{<gsf:metadata name='dc.Title'/>}. There are three other attributes for this element. The attribute \gst{multiple} is used when there may be more than one value for the selected metadata.
830For instance, one document may fall into several classification categories, and therefore may have multiple Subject metadata values. Adding \gst{multiple='true'} to the \gst{<gsf:metadata>} element will retrieve all values, not just the first one. Multiple values are separated by commas by default. The \gst{separator} attribute is used to change the separating string. For example, adding \gst{separator=':~'} to the element will separate all values by a colon and a space.
831
832Sometimes you may want to display metadata values for sections other than the current one. For example, in the mgppdemo collection, in a search list we display the Titles of all the enclosing sections, followed by the Title of the current section, all separated by semi-colons. The display ends up looking something like:
833\emph{Farming snails 2; Starting out; Selecting your snails}
834where \emph{Selecting your snails} is the Title of the section in the results list, and \emph{Farming snails 2} and \emph{Starting out} are the Titles of the enclosing sections. The \gst{select} attribute is used to display metadata for sections other than the current one. Table~\ref{tab:gsf-select-types} shows the options available for this attribute. The \gst{separator} attribute is used here also, to specify the separating text.
835
836To get the previous metadata, the format statement would have the following in it:
837
838\begin{gsc}
839\begin{verbatim}
840<gsf:metadata name='Title' select='ancestors' separator='; '/>;
841 <gsf:metadata name='Title'/>
842\end{verbatim}
843\end{gsc}
844
845\begin{table}
846\caption{Select types for metadata format elements}
847\label{tab:gsf-select-types}
848{\footnotesize
849\begin{tabular}{ll}
850\hline
851\bf Select Type & \bf Description\\
852\hline
853current & The current section \\
854parent & The immediate parent section\\
855ancestors & All the parents back to the root (topmost) section\\
856root & The root or topmost section \\
857siblings & All the sibling sections\\
858children & The immediate child sections of the current section\\
859descendents & All the descendent sections\\
860\hline
861\end{tabular}}
862\end{table}
863
864The \gst{<gsf:choose-metadata} element selects the first available metadata value from the list of options.
865\begin{gsc}
866\begin{verbatim}
867<gsf:choose-metadata>
868 <gsf:metadata name='dc.Title'/>
869 <gsf:metadata name='dls.Title'/>
870 <gsf:metadata name='Title'/>
871</gsf:choose-metadata>
872\end{verbatim}
873\end{gsc}
874
875This will display the dls.Title metadata if available, otherwise it will use the dc.Title metadata if available, otherwise it will use the Title metadata. If there are no values for any of these metadata elements, then nothing will be displayed.
876
877The \gst{<gsf:switch>} element allows different formatting depending on the value of a specified metadata element. For example, the following switch statement could be used to display a different icon for each document in a list depending on which organisation it came from.
878
879\begin{gsc}
880\begin{verbatim}
881<gsf:switch preprocess='toLower;stripSpace'>
882 <gsf:metadata name='Organization'/>
883 <gsf:when test='equals' test-value='bostid'>
884 <!-- output BOSTID image --></gsf:when>
885 <gsf:when test='equals' test-value='worldbank'>
886 <!-- output world bank image --></gsf:when>
887 <gsf:otherwise><!-- output default image--></gsf:otherwise>
888</gsf:switch>
889\end{verbatim}
890\end{gsc}
891
892Preprocessing of the metadata value is optional. The preprocess types are \gst{toLower} (make the value lowercase), \gst{toUpper} (make the value uppercase), \gst{stripSpace} (removes any whitespace from the value). These operations are carried out on the value of the selected metadata before the test is carried out. Multiple processing types can be specified, separated by ; and they will be applied in the order specified (from left to right).
893
894Each option specifies a test and a test value. Test values are just text. Tests include \gst{startsWith}, \gst{contains}, \gst{exists}, \gst{equals}, \gst{endsWith}. Exists doesn't need a test value. Having an otherwise option ensures that something will be displayed even when none of the tests match.
895
896If none of the gsf elements meets your needs for formatting, XSLT can be entered directly into the format element, giving the collection designer full flexibility over how the collection appears.
897
898The collection specific templates are added into the configuration file \gst{collectionConfig.xml}. Any templates found in the XSLT files can be overridden.
899The important part to adding templates into the configuration file is determining where to put them. Formatting templates cannot go just anywhere---there are standard places for them. Figure~\ref{fig:format-places} shows the positions that templates can occur.
900
901\begin{figure}
902\begin{gsc}\begin{verbatim}
903<collectionConfig>
904 <metadataList/>
905 <displayItemList/>
906 <search>
907 <format> <!--Put here templates related to searching and
908 the query page. The common one is the documentNode
909 template -->
910 <gsf:template match='documentNode'>...</gsf:template>
911 </format>
912 </search>
913 <browse>
914 <classifier name='xx'>
915 <format><!-- put here templates related to formating a
916 particular classifier page. Common ones are documentNode
917 and classifierNode templates-->
918 <gsf:template match='documentNode'>...</gsf:template>
919 <gsf:template match='classifierNode'>...</gsf:template>
920 <gsf:template match='classifierNode' mode='horizontal'>...
921 </gsf:template>
922 </format>
923 </classifier>
924 <classifier>...</classifier>
925 <format><!-- formatting for all the classifiers. these will
926 be overridden by any classifier specific formatting
927 instructions --></format>
928 </browse>
929 <display>
930 <format><!-- here goes any formatting relating to the display
931 of the documents. These are generally named templates,
932 and format options -->
933 <gsf:template name='documentContent'>...</gsf:template>
934 <gsf:option name='TOC' value='true'/>
935 </format>
936 </display>
937</collectionConfig>
938\end{verbatim}\end{gsc}
939\caption{Places for format statements}
940\label{fig:format-places}
941\end{figure}
942
943
944There are also formatting instructions that are not templates but are options.
945These are described in Table~\ref{tab:format_options}. They are entered into the configuration file like \gst{<gsf:option name='coverImages' value='false'/>}
946
947\begin{table}
948\caption{Formatting options}
949\label{tab:format_options}
950{\footnotesize
951\begin{tabular}{llp{5cm}}
952\hline
953\bf option name & \bf values & \bf description \\
954\hline
955coverImages & true, false & whether or not to display cover images for documents \\
956TOC & true, false & whether or not to display the table of contents for the document\\
957\hline
958\end{tabular}}
959\end{table}
960
961Note, format templates are added into the XSLT files before transforming, while the options are added into the page source, and used in tests in the XSLT.
962\subsubsection{Changing the service text strings}
963
964Each collection has a set of services which are the access points for the information in the collection. Each service has a set of text strings which are used to display it. These include name, description, the text on the submit button, and names and descriptions of all the parameters to the service.
965
966These text strings are found in .properties files, in greenstone3/resources/java. The names of the files are based on class names. Subclasses can defined their own properties, or can use their parent class ones. For example, AbstractSearch defines strings for the TextQuery service, in AbstractSearch.properties. GS2MGSearch just uses these default ones, so doesn't need its own property file.
967
968A particular collection can override the properties for any service. For example, if a collection uses the GS2MGSearch service rack (look in the buildConfig.xml file for a list of service racks used), and the collection builder wants to change the text associated with this service, they can put a GS2MGSearch.properties file in the resources directory of the collection.
969This will be used in preference to one in the default resources directory.
970Note that while changes in the default properties files seem to require a tomcat restart to take effect, changes in the colleciton specific properties files take effect immediately.
971
972\subsection{Customising the interface}\label{sec:interface-customise}
973
974Format statements in the collection configuration files provide a way to change small parts of the collection display. For large scale customisations to a collection, or ones that apply to a site as a whole, a second mechanism is available. The interface is defined by a set of XSLT files that transform the page data into HTML. Any of these files can be overridden to provide specialised display, on a site or collection basis.
975
976The first section looks at customizing the existing interface, while the second section looks at defining a whole new interface. The last section describes how to add a new language translation of an interface.
977
978\subsubsection{Modifying an existing interface}
979
980Most of an interface is defined by XSLT files, which are stored in \gst{\$GSDL3HOME/\-web/\-interfaces/\-interface-name/\-transform}. These can be changed and the changes will take effect straight away. If changes only apply to certain collections or sites, not everything that uses the interface, you can override some of the files by putting new ones in a different place. XSLT files are looked for in the following order: collection, site, interface, default interface. (This currently only apples to sites, and therefore collections, that reside in the same \gs\ installation as the interface.)
981
982Sites and collections can have a transform directory, which is where customised XSLT files should go. Any XSLT files in here will be used in preference to the interface files when using this collection. For example, if you want to have a completely different layout for the about page of a collection, you can put a new \gst{about.xsl} file into the collection's \gst{transform} directory, and this will be used instead. This is what we do for the Gutenberg sample collection.
983
984This also applies to files that are included from other XSLT files. For example the query.xsl for the query pages includes a file called querytools.xsl. To have a particular site show a different query interface either of these files may need to be modified. Creating a new version of either of these and putting it in the site transform directory will work. Either the new query.xsl will include the default querytools, or the default query.xsl will include the new querytools.xsl. The xsl:include directives are preprocessed by the java code and full paths added based on availability of the files, so that the correct one is used.
985
986Note that you cannot include a file with the same name as the including file. For example query.xsl cannot include query.xsl (it is tempting to want to do this if you just want to change one template for a particular file, and then include the default. but you cant).
987
988\subsubsection{Defining a new interface}
989
990A new interface may be needed if different instantiations of the library require different interfaces, or different developers want their own look and feel. Creating a new interface will allow modifications to be made while leaving the original one intact.
991
992A new interface needs a directory in \gst{\$GSDL3HOME/web/interfaces}, the name of this directory becomes the interface name. Inside, it needs images and transform directories, and an interfaceConfig.xml file. Any XSLT may be overridden for a new interface by putting the replacement in the new transform directory. If the appropriate XSLT file is not there, the one from the default interface will be used - this enables just overriding a few XSLT files as needed.
993
994To use a new interface, the Tomcat web.xml must be edited: either change the interface that a current servlet instance is using, or add another servlet instantiation to the file (see Section~\ref{sec:sites-and-ints} or Appendix~\ref{app:tomcat}). The Tomcat server must be restarted for this to take effect.
995
996\subsubsection{Changing the interface language}
997
998The interface language can be changed by going to the preferences page, and choosing a language from the list, which includes all languages into which the interface has been translated.
999
1000It is easy to add a new interface language to \gs\ . Language specific text strings are separated out from the rest of the system to allow for easy incorporation of new languages. These text strings are contained in Java resource bundle properties files. These are plain text files consisting of key-value pairs, located in \gst{resources/java}. Each interface has one named \gst{interface\_name.properties} (where `name' is the interface name). Each service class has one with the same name as the class (e.g. \gst{GS2Search.properties}). To add another language all of the base .properties files must be translated. The translated files keep the same names, but with a language extension added. For example, a French version of \gst{interface\_default.properties} would be named \gst{interface\_default\_fr.properties}.
1001
1002Keys will be looked up in the properties file closest to the specified language. For example, if language \gst{fr\_CA} was specified (French language, country Canada), and the default locale was \gst{en\_GB}, Java would look at properties files in the following order, until it found the key: \gst{XXX\_fr\_CA.properties}, \gst{XXX\_fr.properties}, \gst{XXX\_en\_GB.properties}, then \gst{XXX\_en.properties}, and finally the default \gst{XXX.properties}.
1003
1004These new files are available straight away---to use the new language, add e.g. \gst{l=fr} to the arguments in the URL. To get \gs\ to add it in to the list of languages on the preferences page, an entry needs to be added into the languages list in the \gst{interfaceConfig.xml} file (see Section~\ref{sec:interfaceconfig}). Modification of this file requires a restart of the Tomcat server for the changes to be recognised.
1005
1006\newpage
1007\section{Developing \gsiii : Run-time system}\label{sec:develop-runtime}
1008
1009[TODO: rewrite this!!]
1010runtime object structure diagram. describe the modules.\\
1011class hierarchy,\\
1012directory structure and where everything lives\\
1013message format.\\
1014overall description of message passing sequence.\\
1015configuration process - start up and runtime\\
1016\\
1017page generation\\
1018accessing the javadoc\\
1019
1020\subsection{Overview of modules??}
1021
1022A \gsiii\ 'library' system consists of many components: MessageRouter, Receptionist, Actions, Collections, ServiceRacks etc. Figure~\ref{fig:local} shows how they fit together in a stand-alone system. The top left part is concerned with displaying the data, while the bottom right part is the collection data serving part. The two sides communicate through the MessageRouter. There is a one-to-one correspondence between modules and Java classes, with the exception of services: for coding and/or run-time efficiency reasons, several Service modules may be grouped together into one ServiceRack class.
1023
1024\begin{figure}[t]
1025 \centering
1026 \includegraphics[width=4in]{local} %5.8
1027 \caption{A simple stand-alone site.}
1028 \label{fig:local}
1029\end{figure}
1030
1031
1032{\em MessageRouter}: this is the central module for a site. It controls the site, loading up all the collections, clusters, communicators needed. All messages pass through the MessageRouter. Communication between remote sites is always done between MessageRouters, one for each site.
1033
1034{\em Collection and ServiceCluster}: these are very similar, and group a set of services into a conceptual group.. They both provide some metadata about the collection/cluster, and a list of services. The services are provided by ServiceRack objects that the collection/cluster loads up. A Collection is a specific type of ServiceCluster. A ServiceCluster groups services that are related conceptually, e.g. all the building services may be part of a cluster. What is part of a cluster is specified by the site configuration file. A Collection's services are grouped by the fact that they all operate on some common data---the documents in the collection.
1035Functionally Collection and ServiceCluster are very similar, but conceptually, and to the user, they are quite different.
1036
1037{\em Service}: these provide the core functionality of the system e.g. searching, retrieving documents, building collections etc. One or more may be grouped into a single Java class (ServiceRack) for code reuse, or to avoid instantiating the same objects several times. For example, MGPP searching services all need to have the index loaded into memory.
1038
1039{\em Communicator/Server}: these facilitate communication between remote modules. For example, if you want MR1 to talk to MR2, you need a Communicator-Server pair. The Server sits on top of MR2, and MR1 talks to the Communicator. Each communication type needs a new pair. So far we have only been using SOAP, so we have a SOAPCommunicator and a SOAPServer.
1040
1041{\em Receptionist}: this is the point of contact for the 'front end'. Its core functionality involves routing requests to the Actions, but it may do more than that. For example, a Receptionist may: modify the request in some way before sending it to the appropriate Action; add some data to the page responses that is common to all pages; transform the response into another form using XSLT. There is a hierarchy of different Receptionist types, which is described in Section~\ref{sec:recepts}.
1042
1043{\em Actions}: these do the job of creating the 'pages'. There is a different action for each type of page, for example PageAction handles semi-static pages, QueryAction handles queries, DocumentAction displays documents. They know a little bit about specific service types. Based on the 'CGI' arguments passed in to them, they construct requests for the system, and put together the responses into data for the page. This data is returned to the Receptionist, which may transform it to HTML. The various actions are described in more detail in Section~\ref{sec:pagegen}.
1044
1045
1046\subsection{Start up configuration}\label{sec:startup-config}
1047
1048We use the Tomcat web server, which operates either stand-alone in a test mode
1049or in conjunction with the Apache web server. The \gs\ LibraryServlet
1050class is loaded by Tomcat and the servlet's \gst{init()} method is called. Each time a
1051\gst{get/put/post} (etc.) is used, a new thread is started and
1052\gst{doGet()/doPut()/doPost()} (etc.) is called.
1053
1054The \gst{init()} method creates a new Receptionist and a new
1055MessageRouter. Default classes (DefaultReceptionist, MessageRouter) are used unless subclasses have been specified in the servlet initiation parameters (see Section~\ref{sec:sites-and-ints}). The appropriate system variables are set for each object (interface
1056name, site name, etc.) and then \gst{configure()} is called on both. The MessageRouter handle
1057is passed to the Receptionist. The servlet then communicates only with
1058the Receptionist, not with the MessageRouter.
1059
1060The Receptionist reads in the \gst{interfaceConfig.xml} file (see Section~\ref{sec:interfaceconfig}), and loads up all the different Action classes. Other Actions may be loaded on the fly as needed. Actions are added to a map, with shortnames for keys. Eg the QueryAction is added with key 'q'. The Actions are passed the MessageRouter reference too.
1061If the Receptionist is a TransformingReceptionist, a mapping between shortnames and XSLT file names is also created.
1062
1063The MessageRouter reads in its site configuration file \gst{siteConfig.xml} (see Section~\ref{sec:siteconfig}). It creates a module map that maps names to objects. This is used for routing the messages. It also keeps small chunks of XML---serviceList, collectionList, clusterList and siteList. These are part of what get returned in response to a describe request (see Section~\ref{sec:describe}.).
1064
1065Each ServiceRack specified in the configuration file is created, then queried for its list of services. Each service name is added to the map, pointing to the ServiceRack object. Each service is also added to the serviceList. After this stage, ServiceRacks are transparent to the system, and each service is treated as a separate module.
1066
1067ServiceClusters are created and passed the \gst{<serviceCluster>} element for configuration. They are added to the map as is, with the cluster name as a key. A serviceCluster is also added to the serviceClusterList.
1068
1069For each site specified, the MessageRouter creates an appropriate type of Communicator object. Then it tries to get the site description. If the server for the remote site is up and running, this should be successful. The site will be added to the mapping with its site name as a key. The site's collections, services and clusters will also be added into the static xml lists. If the server for the remote site is not running, the site will not be included in the siteList or module map. To try again to access the site, either Tomcat must be restarted, or a run-time reconfigure-site command must be sent (see Section~\ref{sec:runtime-config}).
1070
1071The MessageRouter also looks inside the site's \gst{collect} directory, and loads up a Collection object for each valid collection found. If a \gst{collectionInit.xml} file is present, a subclass of Collection may be used.
1072The Collection object reads its \gst{buildConfig.xml} and \gst{collectionConfig.xml}
1073files, determines the metadata, and loads ServiceRack classes based on the
1074names specified in \gst{buildConfig.xml\/}. The \gst{<serviceRack>} XML element is passed to the object to be used in configuration. The \gst{collectionConfig.xml} contents are also passed in to the ServiceRacks. Any format or display information that the services need must be extracted from the collection configuration file.
1075Collection objects are added to the module map with their name as a key, and also a collection element is added into the collectionList XML.
1076
1077\subsection{Message passing}
1078
1079There are two types of messages used by the system: external and internal messages. All messages have an enclosing \gst{<message>} element, which contains either one or more requests, or one or more responses. In the following descriptions, the message element is not shown, but is assumed to be present.
1080Action in \gsiii\ is originated by a request coming in from the outside. In the standard web-based \gs, this comes from a servlet and is passed into the Receptionist. This ``external'' type request is a request for a page of data, and contains a representation of the CGI style arguments. A page of XML is returned, which can be in HTML format or other depending on the output parameter of the request.
1081
1082Messages inside the system (``internal'' messages) all follow the same basic format: message elements contain multiple request elements, or multiple response elements. Messaging is all synchronous. The same number of responses as requests will be returned. Currently all requests are independent, so any requests can be combined into the same message, and they will be answered separately, with their responses being sent back in a single message.
1083
1084When a page request (external request) comes in to the Receptionist, it looks at the action attribute and passes the request to the appropriate Action module. The Action will fire one or more internal requests to the MessageRouter, based on the arguments. The data is gathered into a response, which is returned to the Receptionist. The page that the receptionist returns contains the original request, the response from the action and other info as needed (depends on the type of Receptionist). The data may be transformed in some way --- for the \gs\ servlet we transform using XSLT to generate html pages.
1085
1086Actions send internal style messages to the MessageRouter. Some can be answered by it, others are passed on to collections, and maybe on to services. Internal requests are for simple actions, such as search, retrieve metadata, retrieve document text
1087There are different internal request types: describe, process, system, format, status. Process requests do the actual work of the system, while the other types get auxiliary information. The format of the requests and responses for each internal request type are described in the following sections. External style requests, and their page responses are described in the Section about page generation (Section~\ref{sec:pagegen}).
1088
1089\subsection{'describe'-type messages}\label{sec:describe}
1090
1091The most basic of the internal standard requests is ``describe-yourself'', which can be sent to any module in the system. The module responds with a semi-predefined piece of XML, making these requests very efficient. The response is predefined apart from any language-specific text strings, which are put together as each request comes in, based on the language attribute of the request.
1092\begin{quote}\begin{gsc}\begin{verbatim}
1093<request lang='en' type='describe' to=''/>
1094\end{verbatim}\end{gsc}\end{quote}
1095If the \gst{to} field is empty, a request is answered by the MessageRouter.
1096An example response from a MessageRouter might look like this:
1097\begin{quote}\begin{gsc}\begin{verbatim}
1098<response lang='en' type='describe'>
1099 <serviceList/>
1100 <siteList>
1101 <site name='org.greenstone.gsdl1'
1102 address='http://localhost:8080/soap/servlet/rpcrouter'
1103 type='soap' />
1104 </siteList>
1105 <serviceClusterList>
1106 <serviceCluster name="build" />
1107 </serviceClusterList>
1108 <collectionList>
1109 <collection name='org.greenstone.gsdl1/
1110 org.greenstone.gsdl2/fao' />
1111 <collection name='org.greenstone.gsdl1/demo' />
1112 <collection name='org.greenstone.gsdl1/fao' />
1113 <collection name='myfiles' />
1114 </collectionList>
1115</response>
1116\end{verbatim}\end{gsc}\end{quote}
1117This MessageRouter has no individual site-wide services (an empty \gst{<serviceList>}), but has a service cluster called build (which provides collection importing and building functionality). It
1118communicates with one site, \gst{org.greenstone.gsdl1}. It is aware of four
1119collections. One of these, \gst{myfiles}, belongs to it; the other three are
1120available through the external site. One of those collections is actually from
1121a further external site.
1122
1123It is possible to ask just for a specific part of the information provided by a
1124describe request, rather than the whole thing. For example, these two
1125messages get the \gst{collectionList} and the \gst{siteList} respectively:
1126\begin{quote}\begin{gsc}\begin{verbatim}
1127<request lang='en' type='describe' to=''>
1128 <paramList>
1129 <param name='subset' value='collectionList'/>
1130 </paramList>
1131</request>
1132
1133<request lang='en' type='describe' to=''>
1134 <paramList>
1135 <param name='subset' value='siteList'/>
1136 </paramList>
1137</request>
1138\end{verbatim}\end{gsc}\end{quote}
1139
1140Subset options for the MessageRouter include \gst{collectionList}, \gst{serviceClusterList}, \gst{serviceList}, \gst{siteList}.
1141
1142When a collection or service cluster is asked to describe itself, what is returned is a list of metadata, some display elements, and a list of services. For example, here is such a message, along with a sample response.
1143
1144\begin{quote}\begin{gsc}\begin{verbatim}
1145<request lang='en' type='describe' to='mgppdemo'/>
1146
1147<response from="mgppdemo" type="describe">
1148 <collection name="mgppdemo">
1149 <displayItem lang="en" name="name">greenstone mgpp demo
1150 </displayItem>
1151 <displayItem lang="en" name="description">This is a
1152 demonstration collection for the Greenstone digital
1153 library software. It contains a small subset (11 books)
1154 of the Humanity Development Library. It is built with
1155 mgpp.</displayItem>
1156 <displayItem lang="en" name="icon">mgppdemo.gif</displayItem>
1157 <serviceList>
1158 <service name="DocumentStructureRetrieve" type="retrieve" />
1159 <service name="DocumentMetadataRetrieve" type="retrieve" />
1160 <service name="DocumentContentRetrieve" type="retrieve" />
1161 <service name="ClassifierBrowse" type="browse" />
1162 <service name="ClassifierBrowseMetadataRetrieve"
1163 type="retrieve" />
1164 <service name="TextQuery" type="query" />
1165 <service name="FieldQuery" type="query" />
1166 <service name="AdvancedFieldQuery" type="query" />
1167 <service name="PhindApplet" type="applet" />
1168 </serviceList>
1169 <metadataList>
1170 <metadata name="creator">[email protected]</metadata>
1171 <metadata name="numDocs">11</metadata>
1172 <metadata name="buildType">mgpp</metadata>
1173 <metadata name="httpPath">http://kanuka:8090/greenstone3/sites/
1174 localsite/collect/mgppdemo</metadata>
1175 </metadataList>
1176 </collection>
1177</response>
1178\end{verbatim}\end{gsc}\end{quote}
1179
1180Subset options for a collection or serviceCluster include \gst{metadataList}, \gst{serviceList}, and \gst{displayItemList}.
1181
1182This collection provides many typical services. Notice how this response lists the services available, while the collection configuration file for this collection (Figure~\ref{fig:collconfig}) described serviceRacks. Once the service racks have been configured, they become transparent in the system, and only services are referred to.
1183There are three document retrieval services, for structural information, metadata, and content. The Classifier services retrieve classification structure and metadata. These five services were all provided by the GS2MGPPRetrieve ServiceRack. The three query services were provided by GS2MGPPSearch serviceRack, and provide different kinds of query interface. The last service, PhindApplet, is provided by the PhindPhraseBrowse serviceRack and is an applet service.
1184
1185A \gst{describe} request sent to a service returns a list of parameters that
1186the service accepts and some display information, (and in future may describe the content type for the request and response). Subset options for the request include \gst{paramList} and \gst{displayItemList}.
1187
1188Parameters can be in the following formats:
1189\begin{quote}\begin{gsc}\begin{verbatim}
1190<param name='xxx' type='integer|boolean|string|invisible' default='yyy'/>
1191<param name='xxx' type='enum_single|enum_multi' default='aa'/>
1192 <option name='aa'/><option name='bb'/>...
1193</param>
1194<param name='xxx' type='multi' occurs='4'>
1195 <param .../>
1196 <param .../>
1197</param>
1198\end{verbatim}\end{gsc}\end{quote}
1199
1200If no default is specified, the parameter is assumed to be mandatory.
1201Here are some examples of parameters:
1202\begin{quote}\begin{gsc}\begin{verbatim}
1203<param name='case' type='boolean' default='0'/>
1204
1205<param name='maxDocs' type='integer' default='50'/>
1206
1207<param name='index' type='enum' default='dtx'>
1208 <option name='dtx'/>
1209 <option name='stt'/>
1210 <option name='stx'/>
1211<param>
1212
1213<!-- this one is for the text box and field list for the
1214simple field query-->
1215<param name='simpleField' type='multi' occurs='4'>
1216 <param name='fqv' type='string'/>
1217 <param name='fqf' type='enum_single'>
1218 <option name='TI'/><option name='AU'/><option name='OR'/>
1219 </param>
1220</param>
1221
1222\end{verbatim}\end{gsc}\end{quote}
1223The type attribute is used to determine how to display the parameters on a web page or interface. For example, a string parameter may result in a text entry box, a boolean an on/off button, enum\_single/enum\_multi a drop-down menu, where one or many items, respectively, can be selected.
1224A multi-type parameter indicates that two or more parameters are associated, and should be displayed appropriately. For example, in a field query, the text box and field list should be associated. The occurs attribute specifies how many times the parameter should be displayed on the page.
1225Parameters also come with display information: all the text strings needed to present them to the user. These include the name of the parameter and the display values for any options. These are included in the above parameter descriptions in the form of \gst{<displayItem>} elements.
1226
1227A service description also contains some display information---this includes the name of the service, and the text for the submit button.
1228
1229Here is a sample describe request to the FieldQuery service of collection mgppdemo, along with its response. The parameters in this example include their display information. Figure~\ref{fig:query-display} shows an example html search form that may be generated from this describe response.
1230
1231\begin{quote}\begin{gsc}\begin{verbatim}
1232<request lang="en" to="mgppdemo/FieldQuery" type="describe" />
1233
1234<response from="mgppdemo/FieldQuery" type="describe">
1235 <service name="FieldQuery" type="query">
1236 <displayItem name="name">Form Query</displayItem>
1237 <displayItem name="submit">Search</displayItem>
1238 <paramList>
1239 <param default="Doc" name="level" type="enum_single">
1240 <displayItem name="name">Granularity to search at</displayItem>
1241 <option name="Doc">
1242 <displayItem name="name">Document</displayItem>
1243 </option>
1244 <option name="Sec">
1245 <displayItem name="name">Section</displayItem>
1246 </option>
1247 <option name="Para">
1248 <displayItem name="name">Paragraph</displayItem>
1249 </option>
1250 </param>
1251 <param default="1" name="case" type="boolean">
1252 <displayItem name="name">Turn casefolding </displayItem>
1253 <option name="0">
1254 <displayItem name="name">off</displayItem>
1255 </option>
1256 <option name="1">
1257 <displayItem name="name">on</displayItem>
1258 </option>
1259 </param>
1260 <param default="1" name="stem" type="boolean">
1261 <displayItem name="name">Turn stemming </displayItem>
1262 <option name="0">
1263 <displayItem name="name">off</displayItem>
1264 </option>
1265 <option name="1">
1266 <displayItem name="name">on</displayItem>
1267 </option>
1268 </param>
1269 <param default="10" name="maxDocs" type="integer">
1270 <displayItem name="name">Maximum documents to return
1271 </displayItem>
1272 </param>
1273 <param name="simpleField" occurs="4" type="multi">
1274 <displayItem name="name"></displayItem>
1275 <param name="fqv" type="string">
1276 <displayItem name="name">Word or phrase </displayItem>
1277 </param>
1278 <param default="ZZ" name="fqf" type="enum_single">
1279 <displayItem name="name">in field</displayItem>
1280 <option name="ZZ">
1281 <displayItem name="name">allfields</displayItem>
1282 </option>
1283 <option name="TX">
1284 <displayItem name="name">text</displayItem>
1285 </option>
1286 <option name="TI">
1287 <displayItem name="name">Title</displayItem>
1288 </option>
1289 <option name="SU">
1290 <displayItem name="name">Subject</displayItem>
1291 </option>
1292 <option name="ORG">
1293 <displayItem name="name">Organization</displayItem>
1294 </option>
1295 <option name="SO">
1296 <displayItem name="name">Source</displayItem>
1297 </option>
1298 </param>
1299 </param>
1300 </paramList>
1301 </service>
1302</response>
1303\end{verbatim}\end{gsc}\end{quote}
1304
1305\begin{figure}[t]
1306 \centering
1307 \includegraphics[width=3.5in]{query2.ps}
1308 \caption{The previous query service describe response as displayed on the search page.}
1309 \label{fig:query-display}
1310\end{figure}
1311
1312A describe request to an applet type service returns the applet html element: this will be embedded into a web page to run the applet.
1313\begin{quote}\begin{gsc}\begin{verbatim}
1314<request type='describe' to='mgppdemo/PhindApplet'/>
1315
1316<response type='describe'>
1317 <service name='PhindApplet' type='query'>
1318 <applet ARCHIVE='phind.jar, xercesImpl.jar, gsdl3.jar,
1319 jaxp.jar, xml-apis.jar'
1320 CODE='org.greenstone.applet.phind.Phind.class'
1321 CODEBASE='lib/java'
1322 HEIGHT='400' WIDTH='500'>
1323 <PARAM NAME='library' VALUE=''/>
1324 <PARAM NAME='phindcgi' VALUE='?a=a&amp;sa=r&amp;sn=Phind'/>
1325 <PARAM NAME='collection' VALUE='mgppdemo' />
1326 <PARAM NAME='classifier' VALUE='1' />
1327 <PARAM NAME='orientation' VALUE='vertical' />
1328 <PARAM NAME='depth' VALUE='2' />
1329 <PARAM NAME='resultorder' VALUE='L,l,E,e,D,d' />
1330 <PARAM NAME='backdrop' VALUE='interfaces/default/>
1331 images/phindbg1.jpg'/>
1332 <PARAM NAME='fontsize' VALUE='10' />
1333 <PARAM NAME='blocksize' VALUE='10' />
1334 The Phind java applet.
1335 </applet>
1336 <displayItem name="name">Browse phrase hierarchies</displayItem>
1337 </service>
1338</response>
1339\end{verbatim}\end{gsc}\end{quote}
1340
1341Note that the library parameter has been left blank. This is because library refers to the current servlet that is running and the name is not necessarily known in advance. So either the applet action or the Receptionist must fill in this parameter before displaying the html.
1342
1343\subsection{'system'-type messages}\label{sec:system}
1344
1345``System'' requests are used to tell a MessageRouter, Collection or ServiceCluster to update its cached information and activate or deactivate other modules. For example, the MessageRouter has a set of Collection modules that it can talk to. It also holds some XML information about those collections---this is returned when a request for a collection list comes in. If a collection is deleted or modified, or a new one created, this information may need to change, and the list of available modules may also change. Currently these requests are initiated by particular CGI requests (see Section~\ref{sec:runtime-config}).
1346
1347The basic format of a system request is as follows:
1348
1349\begin{quote}\begin{gsc}\begin{verbatim}
1350<request type='system' to=''>
1351 <system .../>
1352</request>
1353\end{verbatim}\end{gsc}\end{quote}
1354
1355One or more actual requests are specified in system elements. The following are examples:
1356\begin{quote}\begin{gsc}\begin{verbatim}
1357<system type='configure' subset=''/>
1358<system type='configure' subset='collectionList'/>
1359<system type='activate' moduleType='collection' moduleName='demo'/>
1360<system type='deactivate' moduleType='site' moduleName='site1'/>
1361\end{verbatim}\end{gsc}\end{quote}
1362
1363The first request reconfigures the whole site---the MessageRouter goes through its whole configure process again. The second request just reconfigures the collectionList---the MessageRouter will delete all its collection information, and re-look through the collect directory and reload all the collections again.
1364The third request is to activate collection demo. This could be a new collection, or a reactivation of an old one. If a collection module already exists, it will be deleted, and a new one loaded. The final request deactivates the site site1---this removes the site from the siteList and module map, and also removes any of that sites collections/services from the static lists.
1365
1366A response just contains a status message\footnote{TODO: add in error/status codes}, for example:
1367\begin{quote}\begin{gsc}\begin{verbatim}
1368<status>MessageRouter reconfigured successfully</status>
1369<status>Error on reconfiguring collectionList</status>
1370<status>collection:demo activated</status>
1371<status>site:site1 deactivated</status>
1372\end{verbatim}\end{gsc}\end{quote}
1373
1374System requests are mainly answered by the MessageRouter. However, Collections and ServiceClusters will respond to a subset of these requests.
1375
1376\subsection{'format'-type messages}\label{sec:format}
1377
1378Collection designers are able to specify how their collection looks to a certain degree. They can specify format statements for display that will apply to the results of a search, the display of a document, entries in a classification hierarchy, for example. This info is generally service specific. All services respond to a format request, where they return any service specific formatting information. A typical request and response looks like this:
1379\begin{quote}\begin{gsc}\begin{verbatim}
1380<request lang="en" to="mgppdemo/FieldQuery" type="format" />
1381
1382<response from="mgppdemo/FieldQuery" type="format">
1383 <format>
1384 <gsf:template match="documentNode"><td><gsf:link>
1385 <gsf:metadata name="Title" />(<gsf:metadata name="Source" />)
1386 </gsf:link></td>
1387 </gsf:template>
1388 </format>
1389</response>
1390\end{verbatim}\end{gsc}\end{quote}
1391
1392The actual format statements are described in Section~\ref{sec:formatstmt}. They are templates written directly in XSLT, or in GSF (GreenStone Format) which is a simple XML representation of the more complicated XSLT templates.
1393GSF-style format statements need to be converted to proper XSLT. This is currently done by the Receptionist (but may be moved to an ActionHelper): the format XML is transformed to XSLT using XSLT with the config\_format.xsl stylesheet.
1394
1395\subsection{'status'-type messages}\label{sec:status}
1396
1397These are only used with process-type services, which are those where a request is sent to start some type of process (see Section~\ref{sec:process}). An initial 'process' request to a 'process' service generates a response which states whether the process had successfully started, and whether its still continuing. If the process is not finished, status requests can be sent repeatedly to the service to poll the status, using the pid to identify the process. Status codes are used to identify the state of a process. The values used at the moment are listed in Table~\ref{tab:status codes}\footnote{A more standard set of codes should probably be used, for example, the HTTP codes}.
1398
1399\begin{table}
1400\caption{Status codes currently used in \gsiii\ }
1401\label{tab:status codes}
1402{\footnotesize
1403\begin{tabular}{llp{8cm}}
1404\hline
1405\bf code name & \bf code & \bf meaning \\
1406& \bf value & \\
1407\hline
1408SUCCESS & 1 & the request was accepted, and the process was completed \\
1409ACCEPTED & 2 & the request was accepted, and the process has been started, but it is not completed yet \\
1410ERROR & 3 & there was an error and the process was stopped \\
1411CONTINUING & 10 & the process is still continuing \\
1412COMPLETED & 11 & the process has finished \\
1413HALTED & 12 & the process has stopped \\
1414INFO & 20 & just an info message that doesn't imply anything \\
1415\hline
1416\end{tabular}}
1417\end{table}
1418
1419 The following shows an example status request, along with two responses, the first a 'OK but continuing' response, and the second a 'successfully completed' response. The content of the status elements in the two responses is the output from the process since the last status update was sent back.
1420
1421\begin{quote}\begin{gsc}\begin{verbatim}
1422<request lang="en" to="build/ImportCollection" type="status">
1423 <paramList>
1424 <param name="pid" value="2" />
1425 </paramList>
1426</request>
1427
1428<response from="build/ImportCollection">
1429 <status code="2" pid="2">Collection construction: import collection.
1430command = import.pl -collectdir /research/kjdon/home/greenstone3/web/sites/
1431 localsite/collect test1
1432starting
1433 </status>
1434</response>
1435
1436<response from="build/ImportCollection">
1437 <status code="11" pid="2">RecPlug: getting directory
1438/research/kjdon/home/greenstone3/web/sites/localsite/collect/test1/import
1439WARNING - no plugin could process /.keepme
1440
1441*********************************************
1442Import Complete
1443*********************************************
1444* 1 document was considered for processing
1445* 0 were processed and included in the collection
1446* 1 was rejected. See /research/kjdon/home/greenstone3/web/sites/
1447 localsite/collect/test1/etc/fail.log for a list of rejected documents
1448Success
1449 </status>
1450</response>
1451\end{verbatim}\end{gsc}\end{quote}
1452
1453\subsection{'process'-type messages}
1454
1455Process requests and responses provide the major functionality of the system---these are the ones that do the actual work. The format depends on the service they are for, so I'll describe these by service.
1456
1457Query type services TextQuery, FieldQuery, AdvancedFieldQuery (GS2MGSearch, GS2MGPPSearch), TextQuery (LuceneSearch)
1458The main type of requests in the system are for services. There are different types of services, currently: \gst{query}, \gst{browse}, \gst{retrieve}, \gst{process}, \gst{applet}, \gst{enrich}. Query services do some kind of search and return a list of document identifiers. Retrieve services can return the content of those documents, metadata about the documents, or other resources. Browse is for browsing lists or hierarchies of documents. Process type services are those where the request is for a command to be run. A status code will be returned immediately, and then if the command has not finished, an update of the status can be requested. Applet services are those that run an applet. Enrich services take a document and return the document with some extra markup added.
1459
1460 Other possibilities include transform, extract, accrete. These types of service generally enhance the functionality of the first set. They may be used during collection formation: 'accrete' documents by adding them to a collection, 'transform' the documents into a different format, 'extract' information or acronyms from the documents, 'enrich' those documents with the information extracted or by adding new information. They may also be used during querying: 'transform' a query before using it to query a collection, or 'transform' the documents you get back into an appropriate form.
1461
1462The basic structure of a service 'process' request is as follows:
1463\begin{quote}\begin{gsc}\begin{verbatim}
1464
1465<request lang='en' type='process' to='demo/TextQuery'>
1466 <paramList/>
1467 other elements...
1468</request>
1469
1470\end{verbatim}\end{gsc}\end{quote}
1471
1472The parameters are name-value pairs corresponding to parameters that were specified in the service description sent in response to a describe request.
1473
1474\begin{quote}\begin{gsc}\begin{verbatim}
1475<param name='case' value='1'/>
1476<param name='maxDocs' value='34'/>
1477<param name='index' value='dtx'/>
1478\end{verbatim}\end{gsc}\end{quote}
1479
1480Some requests have other content---for document retrieval, this would be a list of document identifiers to retrieve. For metadata retrieval, the content is the list of documents to retrieve metadata for.
1481
1482Responses vary depending on the type of request. The following sections look at the process type requests and responses for each type of service.
1483
1484\subsubsection{'query'-type services}
1485Responses to query requests contain a list of document identifiers, along with some other information, dependent on the query type. For a text query, this includes term frequency information, and some metadata about the result. For instance, a text query on 'snail farming', with the parameter 'maxDocs=10' might return the first 10 documents, and one of the query metadata items would be the total number of documents that matched the query.\footnote{no metadata about the query result is returned yet.}
1486
1487The following shows an example query request and its response.
1488
1489Find at most 10 Sections in the mgppdemo collection, containing the word snail (stemmed), returning the results in ranked order:
1490\begin{quote}\begin{gsc}\begin{verbatim}
1491<request lang='en' to="mgppdemo/TextQuery" type="process">
1492 <paramList>
1493 <param name="maxDocs" value="10"/>
1494 <param name="queryLevel" value="Section"/>
1495 <param name="stem" value="1"/>
1496 <param name="matchMode" value="some"/>
1497 <param name="sortBy" value="1"/>
1498 <param name="index" value="t0"/>
1499 <param name="case" value="0"/>
1500 <param name="query" value="snail"/>
1501 </paramList>
1502</request>
1503
1504<response from="mgppdemo/TextQuery" type="process">
1505 <metadataList>
1506 <metadata name="numDocsMatched" value="59" />
1507 </metadataList>
1508 <documentNodeList>
1509 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2"
1510 docType='hierarchy' nodeType="leaf" />
1511 <documentNode nodeID="HASH010f073f22033181e206d3b7.2.12"
1512 docType='hierarchy' nodeType="leaf" />
1513 <documentNode nodeID="HASH010f073f22033181e206d3b7.1"
1514 docType='hierarchy' nodeType="interior" />
1515 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.2.2"
1516 docType='hierarchy' nodeType="leaf" />
1517 ...
1518 </documentNodeList>
1519 <termList>
1520 <term field="" freq="454" name="snail" numDocsMatch="58" stem="3">
1521 <equivTermList>
1522 <term freq="" name="Snail" numDocsMatch="" />
1523 <term freq="" name="snail" numDocsMatch="" />
1524 <term freq="" name="Snails" numDocsMatch="" />
1525 <term freq="" name="snails" numDocsMatch="" />
1526 </equivTermList>
1527 </term>
1528 </termList>
1529</response>
1530\end{verbatim}\end{gsc}\end{quote}
1531
1532The list of document identifiers includes some information about document type and node type. Currently, document types include \gst{simple}, \gst{paged} and \gst{hierarchy}. \gst{simple} is for single section documents, i.e. ones with no sub-structure. \gst{paged} is documents that have a single list of sections, while \gst{hierarchy} type documents have a hierarchy of nested sections. For \gst{paged} and \gst{hierarchy} type documents, the node type identifies whether a section is the root of the document, an internal section, or a leaf.
1533
1534The term list identifies, for each term in the query, what its frequency in the collection is, how many documents contained that term, and a list of its equivalent terms (if stemming or casefolding was used).
1535
1536\subsubsection{'browse'-type services}
1537
1538Browse type services are used for classification browsing. The request consists of a list of classifier identifiers, and some structure parameters listing what structure to retrieve.
1539
1540\begin{quote}\begin{gsc}\begin{verbatim}
1541<request lang="en" to="mgppdemo/ClassifierBrowse" type="process">
1542 <paramList>
1543 <param name="structure" value="ancestors" />
1544 <param name="structure" value="children" />
1545 </paramList>
1546 <classifierNodeList>
1547 <classifierNode nodeID="CL1.2" />
1548 </classifierNodeList>
1549</request>
1550
1551<response from="mgppdemo/ClassifierBrowse" type="process">
1552 <classifierNodeList>
1553 <classifierNode nodeID="CL1">
1554 <nodeStructure>
1555 <classifierNode nodeID="CL1">
1556 <classifierNode nodeID="CL1.2">
1557 <classifierNode nodeID="CL1.2.1" />
1558 <classifierNode nodeID="CL1.2.2" />
1559 <classifierNode nodeID="CL1.2.3" />
1560 <classifierNode nodeID="CL1.2.4" />
1561 <classifierNode nodeID="CL1.2.5" />
1562 </classifierNode>
1563 </classifierNode>
1564 </nodeStructure>
1565 </classifierNode>
1566 </classifierNodeList>
1567</response>
1568\end{verbatim}\end{gsc}\end{quote}
1569
1570Possible values for structure parameters are \gst{ancestors}, \gst{parent}, \gst{siblings}, \gst{children}, \gst{descendents}. The response gives, for each identifier in the request, a \gst{<nodeStructure>} element with all the requested structure put together into a hierarchy. The structure may include classifier and document nodes.
1571
1572
1573\subsubsection{'retrieve'-type services}
1574
1575Retrieval services are special in that requests are not explicitly initiated by a user from a form on a web page, but are called from actions in response to other things. This means that their names are hard-coded into the Actions. DocumentContentRetrieve, DocumentStructureRetrieve and DocumentMetadataRetrieve are the standard names for retrieval services for content, structure, and metadata of documents. Requests to each of these include a list of document identifiers. Because these generally refer to parts of documents, the elements are called \gst{<documentNode>}. For the content, that is all that is required. For the metadata retrieval service, the request also needs parameters specifying what metadata is required. For structure retrieval services, requests need parameters specifying what structure or structural info is required.
1576
1577Some example requests and responses follow.
1578
1579Give me the Title metadata for these documents:
1580\begin{quote}\begin{gsc}\begin{verbatim}
1581
1582<request lang="en" to="mgppdemo/DocumentMetadataRetrieve" type="process">
1583 <paramList>
1584 <param name="metadata" value="Title" />
1585 </paramList>
1586 <documentNodeList>
1587 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2"/>
1588 <documentNode nodeID="HASH010f073f22033181e206d3b7.2.12"/>
1589 <documentNode nodeID="HASH010f073f22033181e206d3b7.1"/>
1590 ...
1591 </documentNodeList>
1592</request>
1593
1594<response from="mgppdemo/DocumentMetadataRetrieve" type="process">
1595 <documentNodeList>
1596 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2">
1597 <metadataList>
1598 <metadata name="Title">Putting snails in your second pen</metadata>
1599 </metadataList>
1600 </documentNode>
1601 <documentNode nodeID="HASH010f073f22033181e206d3b7.2.12">
1602 <metadataList>
1603 <metadata name="Title">Now you must decide</metadata>
1604 </metadataList>
1605 </documentNode>
1606 <documentNode nodeID="HASH010f073f22033181e206d3b7.1">
1607 <metadataList>
1608 <metadata name="Title">Introduction</metadata>
1609 </metadataList>
1610 </documentNode>
1611 </documentNodeList>
1612</response>
1613\end{verbatim}\end{gsc}\end{quote}
1614
1615One or more parameters specifying metadata may be included in a request. Also, ametadata value of \gst{all} will retrieve all the metadata for each document.
1616
1617Any browse-type service must also implement a metadata retrieval service to provide metadata for the nodes in the classification hierarchy. The name of it is the browse service name plus \gst{MetadataRetrieve}. For example, the ClassifierBrowse service described in the previous section should also have a ClassifierBrowseMetadataRetrieve service. The request and response format is exactly the same as for the DocumentMetadataRetrieve service, except that \gst{<documentNode>} elements are replaced by \gst{<classifierNode>} elements (and the corresponding list element is also changed).
1618
1619Give me the text (content) of this document:
1620\begin{quote}\begin{gsc}\begin{verbatim}
1621<request lang="en" to="mgppdemo/DocumentContentRetrieve" type="process">
1622 <paramList />
1623 <documentNodeList>
1624 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2" />
1625 </documentNodeList>
1626</request>
1627
1628<response from="mgppdemo/DocumentContentRetrieve" type="process">
1629 <documentNodeList>
1630 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2">
1631 <nodeContent>&lt;Section&gt;
1632 &lt;/B&gt;&lt;P ALIGN=&quot;JUSTIFY&quot;&gt;&lt;/P&gt;
1633 &lt;P ALIGN=&quot;JUSTIFY&quot;&gt;190. When the plants in
1634 your second pen have grown big enough to provide food and
1635 shelter, you can put in the snails.&lt;/P&gt;
1636 </nodeContent>
1637 </documentNode>
1638 </documentNodeList>
1639</response>
1640\end{verbatim}\end{gsc}\end{quote}
1641
1642The content of a node is returned in a \gst{<nodeContent>} element. In this case it is escaped HTML.
1643
1644Give me the ancestors and children of the specified node, along with the number of siblings it has:
1645\begin{quote}\begin{gsc}\begin{verbatim}
1646<request lang="en" to="mgppdemo/DocumentStructureRetrieve" type="process">
1647 <paramList>
1648 <param name="structure" value="ancestors" />
1649 <param name="structure" value="children" />
1650 <param name="info" value="numSiblings" />
1651 </paramList>
1652 <documentNodeList>
1653 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2" />
1654 </documentNodeList>
1655</request>
1656
1657<response from="mgppdemo/DocumentStructureRetrieve" type="process">
1658 <documentNodeList>
1659 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2">
1660 <nodeStructureInfo>
1661 <info name="numSiblings" value="2" />
1662 </nodeStructureInfo>
1663 <nodeStructure>
1664 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd"
1665 docType='hierarchy' nodeType="root">
1666 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4"
1667 docType='hierarchy' nodeType="interior">
1668 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd.4.2"
1669 docType='hierarchy' nodeType="leaf" />
1670 </documentNode>
1671 </documentNode>
1672 </nodeStructure>
1673 </documentNode>
1674 </documentNodeList>
1675</response>
1676\end{verbatim}\end{gsc}\end{quote}
1677
1678Structure is returned inside a \gst{<nodeStructure>} element, while structural info is returned in a \gst{<nodeStructureInfo>} element. Possible values for structure parameters are as for browse services: \gst{ancestors}, \gst{parent}, \gst{siblings}, \gst{children}, \gst{descendents}, \gst{entire}. Possible values for info parameters are \gst{numSiblings}, \gst{siblingPosition}, \gst{numChildren}.
1679
1680\subsubsection{'process'-type services}\label{sec:process}
1681Requests to process-type services are not requests for data---they request some action to be carried out, for example, create a new collection, or import a collection. The response is a status or an error message. The import and build commands may take a long time to complete, so a response is sent back after a successful start to the command. The status may be polled by the requester to see how the process is going.
1682
1683Process requests generally contain just a parameter list. Like for any service, the parameters used by a process-type service can be obtained by a describe request to that service.
1684
1685Here are two example requests for process-services that are part of the build service cluster (hence the addresses all begin with 'build/'), followed by an example response:
1686
1687\begin{quote}\begin{gsc}\begin{verbatim}
1688<request lang='en' type='process' to='build/NewCollection'>
1689 <paramList>
1690 <param name='creator' value='[email protected]'/>
1691 <param name='collName' value='the demo collection'/>
1692 <param name='collShortName' value='demo'/>
1693 </paramlist>
1694</request>
1695
1696<request lang='en' type='process' to='build/ImportCollection'>
1697 <paramList>
1698 <param name='collection' value='demo'/>
1699 </paramlist>
1700</request>
1701
1702<response from="build/ImportCollection">
1703 <status code="2" pid="2">Starting process...</status>
1704</response>
1705\end{verbatim}\end{gsc}\end{quote}
1706
1707The \gst{code} attribute in the response specifies whether the command has been successfully stated, whether its still going, etc (see Table~\ref{tab:status codes} for a list of currently used codes). The pid attribute specifies a process id number that can be used when querying the status of this process. The content of the status element is (currently) just the output from the process so far. Status messages, which were described in Section~\ref{sec:status}, are used to find out how the process is going, and whether it has finished or not.
1708
1709\subsubsection{'applet'-type services}
1710
1711Applet-type services are those that process the data for an applet. A request consists only of a list of parameters, and the response contains an \gst{<appletData>} element that contains the XML data to be returned to the applet. The format of this is entirely specific to the applet---there is no set format to the applet data.
1712
1713Here is an example request and response, used by the Phind applet:
1714\begin{quote}\begin{gsc}\begin{verbatim}
1715 <request type='query' to='mgppdemo/PhindApplet'>
1716 <paramList>
1717 <param name='pc' value='1'/>
1718 <param name='pptext' value='health'/>
1719 <param name='pfe' value='0'/>
1720 <param name='ple' value='10'/>
1721 <param name='pfd' value='0'/>
1722 <param name='pld' value='10'/>
1723 <param name='pfl' value='0'/>
1724 <param name='pll' value='10'/>
1725 </paramList>
1726 </request>
1727
1728 <response type='query' from='mgppdemo/PhindApplet'>
1729 <appletData>
1730 <phindData df='9' ef='46' id='933' lf='15' tf='296'>
1731 <expansionList end='10' length='46' start='0'>
1732 <expansion df='4' id='8880' num='0' tf='59'>
1733 <suffix> CARE</suffix>
1734 </expansion>
1735 ...
1736 </expansionList>
1737 <documentList end='10' length='9' start='0'>
1738 <document freq='78' hash='HASH4632a8a51d33c47a75c559' num='0'>
1739 <title>The Courier - N??159 - Sept- Oct 1996 Dossier Investing
1740 in People Country Reports: Mali ; Western Samoa
1741 </title>
1742 </document>
1743 ...
1744 </documentList>
1745 <thesaurusList end='10' length='15' start='0'>
1746 <thesaurus df='7' id='12387' tf='15' type='RT'>
1747 <phrase>PUBLIC HEALTH</phrase>
1748 </thesaurus>...
1749 </thesaurusList>
1750 </phindData>
1751 </appletData>
1752 </response>
1753
1754\end{verbatim}\end{gsc}\end{quote}
1755
1756\subsubsection{'enrich'-type services}
1757
1758Enrich services typically take some text of documents (inside \gst{<nodeContent>} tags) and returns the text marked up in some way. One example of this is the GatePOSTag service: this identifies Dates, Locations, People and Organizations in the text, and annotates the text with the labels. In the following example, the request is for Location and Dates to be identified.
1759
1760\begin{quote}\begin{gsc}\begin{verbatim}
1761<request lang="en" to="GatePOSTag" type="process">
1762 <paramList>
1763 <param name="annotationType" value="Date,Location" />
1764 </paramList>
1765 <documentNodeList>
1766 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd">
1767 <nodeContent>
1768 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS
1769 Rome 1986
1770 P-69
1771 ISBN 92-5-102397-2
1772 FAO 1986
1773 </nodeContent>
1774 </documentNode>
1775 </documentNodeList>
1776</request>
1777
1778<response from="GatePOSTag" type="process">
1779 <documentNodeList>
1780 <documentNode nodeID="HASHac0a04dd14571c60d7fbfd">
1781 <nodeContent>
1782 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS
1783 <annotation type="Location">Rome</annotation>
1784 <annotation type="Date">1986</annotation>
1785 P-69
1786 ISBN 92-5-102397-2
1787 FAO <annotation type="Date">1986</annotation>
1788 </nodeContent>
1789 </documentNode>
1790 </documentNodeList>
1791</response>
1792\end{verbatim}\end{gsc}\end{quote}
1793
1794\subsection{Page generation}\label{sec:pagegen}
1795
1796A 'page' is some XML or HTML (or other?) data returned in response to an
1797external 'page'-type request. These requests originate from outside \gs\ , for example from a servlet, or java application, and are received by the Receptionist. As described below in Section~\ref{sec:page-requests}, the requests are XML representations of \gs\ URLs. One of the arguments is action (a). This tells the Receptionist which Action module to pass the request to.
1798
1799Action modules decode the rest of the arguments to determine what requests need to be made to the system. One or more internal requests may be made to the MessageRouter. A request for format information from the Collection/Service may also be made. The resulting data is gathered together into a single XML response, \gst{<page>}, and returned to the Receptionist.
1800
1801The page format is described in Section~\ref{sec:page-format}. The XML may be returned as is, or may be modified by the Receptionist. The various Receptionists are described in Section~\ref{sec:recepts}. The default receptionist used by a servlet transforms the XML into HTML using XSL stylesheets. Section~\ref{sec:collformat} looks at collection specific formatting, in particular for HTML output.
1802Sections~\ref{sec:pageaction} to \ref{sec:systemaction} look at the various actions and what kind of data they gather.
1803
1804\subsubsection{'page'-type requests and their arguments}\label{sec:page-requests}
1805
1806These are requests for a 'page' of data---for example, the home page for a site; the query page for a collection; the text of a document. They contain, in XML, a list of arguments specifying what type of page is required. If the external context is a servlet, the arguments represent the 'CGI' arguments in a \gs\ URL. The two main arguments are \gst{a} (action) and \gst{sa} (subaction). All other arguments are encoded as parameters.
1807
1808Here are some examples of requests\footnote{In a servlet context, these correspond to the arguments \gst{a=p\&sa=about\&c=demo\&l=fr}, and \gst{a=q\&l=en\&s=TextQuery\&c=demo\&rt=r\&ca=0\&st=1\&m=10\&q=snail}.}:
1809
1810\begin{quote}\begin{gsc}\begin{verbatim}
1811<request type='page' action='p' subaction='about'
1812 lang='fr' output='html'>
1813 <paramList>
1814 <param name='c' value='demo'/>
1815 </paramList>
1816</request>
1817\end{verbatim}\end{gsc}\end{quote}
1818
1819\begin{quote}\begin{gsc}\begin{verbatim}
1820<request type='page' action='q' lang='en' output='html'>
1821 <paramList>
1822 <param name='s' value='TextQuery'/>
1823 <param name='c' value='demo'/>
1824 <param name='rt' value='r'/>
1825 <!-- the rest are the service specific params -->
1826 <param name='ca' value='0'/> <!-- casefold -->
1827 <param name='st' value='1'/> <!-- stem -->
1828 <param name='m' value='10'/> <!-- maxdocs -->
1829 <param name='q' value='snail'/> <!-- query string -->
1830 </paramList>
1831</request>
1832\end{verbatim}\end{gsc}\end{quote}
1833
1834There are some standard arguments used in Greenstone, and they are described in Table~\ref{tab:args}. These are used by Receptionists and Actions. The GSParams class specifies all the general basic arguments, and whether they should be saved or not (Some arguments need to be saved during a session, and this needs to be implemented outside \gs\ proper --- currently we do this in the servlet, using servlet session handling). The servlet has an init parameter \gst{params\_class} which specifies which params class to use: GSParams can be subclassed if necessary. The Receptionist and Actions must not have conflicting argument names.
1835
1836Other arguments are used dynamically and come from the Services. Service arguments must always be saved during a session. Services may be created by different people, and may reside on a different site. There is no guarantee that there is no conflict with argument names between services and actions. Therefore service parameters are namespaced when they are put on the page, whereas interface (receptionist and action) parameters have no namespace. The default namespace is s1 (service1) --- any parameters that are for the service will be prefixed by this. For example, the case parameter for a search will be put in the page as s1.case, and the resulting argument in a search URL will be s1.case. When actions are deciding which parameters need to be sent in a request to a service, they can use the namespace information.
1837
1838If there are two or more services combined on a page with a single submit button, they will use namespaces s1, s2, s3 etc as needed. The s (service) parameter will end up with a list of services. For example, \gst{s=TextQuery,MusicQuery,} and the order of these determines the mapping order of the namespaces, i.e. s1 will map to TextQuery, s2 to MusicQuery.
1839
1840\begin{table}
1841{\footnotesize
1842\begin{tabular}{lll}
1843\hline
1844\bf Argument & \bf Meaning &\bf Typical values \\
1845\hline
1846a & action & a (applet), q (query), b (browse), p (page), pr (process) \\
1847& & s (system)\\
1848sa & subaction & home, about (page action)\\
1849c & collection or & demo, build \\
1850& service cluster \\
1851s & service name & TextQuery, ImportCollection \\
1852rt & request type & d (display), r (request), s (status) \\
1853ro & response only & 0 or 1 - if set to one, the request is carried out \\
1854& & but no processing of the results is done \\
1855& & currently only used in process actions \\
1856o & output type & XML, html, WML \\
1857l & language & en, fr, zh ...\\
1858d & document id & HASHxxx \\
1859r & resource id & ???\\
1860pid & process handle & an integer identifying a particular process request \\
1861\hline
1862\end{tabular}}
1863\caption{Generic arguments that can appear in a \gs\ URL}
1864\label{tab:args}
1865\end{table}
1866
1867\subsubsection{page format}\label{sec:page-format}
1868
1869The basic page format is:
1870\begin{quote}\begin{gsc}\begin{verbatim}
1871<page lang='en'>
1872 <pageRequest/>
1873 <pageResponse/>
1874</page>
1875\end{verbatim}\end{gsc}\end{quote}
1876
1877* show configuration and describe whats its used for
1878
1879There are two main elements in the page: pageRequest, pageResponse. The pageRequest is the original request that came into the Receptionist---this is included so that any parameters can be preset to their previous values, for example, the query options on the query form. The pageResponse contains all the data that has been gathered from the system by the action. The other two elements contain extra information needed by XSLT. Config contains run-time variables such as the location of the gsdl home directory, the current site name, the name of the executable that is running (e.g. library)---these are needed to allow the XSLT to generate correct HTML URLs. Display contains some of the text strings needed in the interface---these are separate from the XSLT to allow for internationalization.
1880
1881The following subsections outline, for each action, what data is needed and what requests are generated to send to the system.
1882
1883
1884Once the XML page has been put together, the page to return to the user is created by transforming the XML using XSLT. The output is HTML at this stage, but it will be possible to generate alternative outputs, such as XML, WML etc. A set of XSLT files defines an 'interface'. Different users can change the look of their web pages by creating new XSLT files for a new 'interface'. Just as we have a sites directory where different sites 'live' (ie where their configuration file and collections are located), we have an interfaces directory where the different interfaces 'live' (ie their transforms and images are located there). The default XSLT files are
1885located in interfaces/default/transforms. Collections, sites and other interfaces
1886can override these files by having their own copy of the appropriate
1887files. New interfaces have their own directory inside interfaces/. Sites and collections can have a transform directory containing XSLT files. The order in which the XSLT files are looked for is collection, site, current
1888interface, default interface.\footnote{this currently breaks down for remote sites - need to rethink it a bit.}
1889***TODO*** describe a bit more?? currently only can get this locally
1890
1891\subsubsection{Receptionists}\label{sec:recepts}
1892
1893The receptionist is the controlling module for the page generation part of \gs\ . It has the job of loading up all the actions, and it knows about the message router it and the actions are supposed to talk to. It routes messages received to the appropriate action (page-type messages) or directly to the message router (all other types). Receptionists also do other things, for example, adding to the page received back from the action any information that is common to all pages.
1894
1895There are different ways of providing an interface to \gs\ , from web based CGI style (using servlets) to Java GUI applications. These different interfaces require slightly different responses from a receptionist, so we provide several standard types of receptionist.
1896
1897Receptionist: This is the most basic receptionist. The page it returns consists of the original request, and the response from the action it was sent to. Methods preProcessRequest, and postProcessPage are called on the request and page, respectively, but in this basic receptionist, they don't do anything.
1898
1899TransformingReceptionist: This extends Receptionist, and overwrites postProcessPage to transform the page using XSLT. An XSLT is listed for each action in the receptionists configuration file, and this is used to transform the page. First, some display information, and configuration information is added to the page. Then it is transformed using the specified XSLT for the action, and returned.
1900
1901WebReceptionist: The WebReceptionist extends TransformingReceptionist. It doesn't do much else except some argument conversion. To keep the URLs short, parameters from the services are given shortnames, and these are used in the web pages.
1902
1903DefaultReceptionist: This extends WebReceptionist, and is the default one for \gsiii\ servlets. Due to the page design, some extra information is needed for each page: some metadata about the current collection. The receptionist sends a describe request to the collection to get this, and appends it to the page before transformation using XSLT.
1904
1905NZDLReceptionist: (do we want to talk about this?) This is an example of a custom receptionist. For a look-alike nzdl.org system, even more information is needed for each page, namely the list of classifiers available from the ClassifierBrowse service.
1906
1907By default, the LibraryServlet uses DefaultReceptionist. However, there is a servlet init-param called \gst{receptionist} which can be set to make the servlet use a different one.
1908
1909\subsubsection{Collection specific formatting}\label{sec:collformat}
1910get format info, transform gsf->xsl. transfrom xml->html
1911
1912config params are passed in to the transformation
1913\subsubsection{CGI arguments}
1914
1915
1916\subsubsection{Page action}\label{sec:pageaction}
1917
1918PageAction is responsible for displaying kinds of information pages, such as the home page of the library, or the home page of a collection, or the help and preferenecs pages. These pages are not associated with specific services like the other page types. In general, the data comes from describe requests to various modules.
1919The different pages are requested using the subaction argument. For the 'home' page, a 'describe' request is sent to the MessageRouter---this returns a list of all the collections, services, serviceClusters and sites known about. For each collection, its metadata is retrieved via a 'describe' request. This metadata is added into the previous result, which is then added into the page. For the 'about' page, a \gst{describe} request is sent to the module that the about page is about: this may be a collection or a service cluster. This returns a list of metadata
1920and a list of services.
1921
1922
1923\subsubsection{Query action}\label{sec:queryaction}
1924
1925The basic URL is \gst{a=q\&s=TextQuery\&c=demo\&rt=d/r}.
1926There are three query services which have been implemented: TextQuery, FieldQuery, and AdvancedFieldQuery. These are all handled in the same way by query action.
1927For each page, the service description is requested from the service of the current collection (via a describe request). This is currently done every time the query page is
1928displayed, but should be cached. The description includes a list of the parameters available for the query, such as case/stem, max num docs to return, etc. If the request type (rt) parameter is set to d for display, the action only needs to display the form, and this is the only request to the service. Otherwise, the submit button has been pressed, and a query request to the TextQuery service is sent. This has all the parameters from the URL put into the parameter list. A list of document identifiers
1929is returned. A followup query is sent to the MetadataRetrieve service of the collection: the content includes the list of
1930documents, with a request for some of their metadata. Which metadata to retrieve is determined by looking through the XSLT that will be used to transform the page. The service description and query result are combined into a page of XML, which is returned to the Receptionist.
1931
1932\subsubsection{Applet action}\label{sec:appletaction}
1933
1934There are two types of request to the applet action: \gst{a=a \& rt=d\/} and
1935\gst{a=a \& rt=r\/}. The value \gst{rt=d\/} means ``display the applet.'' A
1936\gst{describe} request is sent to the service, which returns the \gst{<applet>} HTML element. The transformation file \gst{applet.xsl} embeds this
1937into the page, and the servlet returns the HTML.
1938
1939The value \gst{rt=r} signals a request from the applet. A process request containing all the parameters is sent to the applet service. The result contains an appletData element, which contains a single element - this element is returned
1940directly to the applet, in XML. No transformation is done.
1941Because the AppletAction doesn't know or care anything about the applet data, it can work with any applet-service pair.
1942
1943Note that the applet HTML may need to know the name of the \gst{library}
1944program. However, that name is chosen by the person who installed the software
1945and will not necessarily be ``library''. To get around this, the applet can
1946put a parameter called ``library'' into the applet data with a null value:
1947\begin{quote}\begin{gsc}\begin{verbatim}
1948<PARAM NAME='library' VALUE=''/>
1949\end{verbatim}\end{gsc}\end{quote}
1950When the AppletAction encounters this parameter it inserts the name of the
1951current library servlet as its value.
1952
1953\subsubsection{Document action}\label{sec:documentaction}
1954
1955DocumentAction is responsible for displaying a document to the user. The display might involve some metadata and/or text for a document or part of a document. For hierarchical documents, a table of contents may be shown, while for paged documents (those with a single linear list of sections), next and previous page buttons may be shown. These different display types require different information about the document. Depending on the arguments, DocumentAction will send requests to several services: DocumentMetadataRetrieve, DocumentStructureRetrieve and DocumentContentRetrieve.
1956
1957A basic display, for example, Title and text, involves a metadata request to get the Title, and a content request to get the text. Hierarchical table of contents display requires a structure request. If the entire contents is to be displayed, the parameter \gst{structure=entire} would be sent in the request. Otherwise, parameters \gst{structure=ancestors}, \gst{structure=children} and possibly \gst{structure=siblings} may be used, depending in the position of the current node in the document. These return a hierarchical structure of nodes, containing ancestor nodes, child nodes and sibling nodes, respectively.
1958For paged display, the structure is not actually needed. A structure request is still sent, but this time it requests some information, rather the structure itself. The information requested includes the number of siblings and the current position of the current node, or the number of children (if the current node is the root of the document).
1959
1960Metadata may be requested for the current node, or for any nodes in the structure, and content also. The metadata and content are added into the appropriate nodes in the structure hierarchy, and this is returned as the page data.
1961
1962\subsubsection{XML Document action}\label{sec:xmldocumentaction}
1963
1964XMLDOcumentAction is a little different to the standard DocumentAction. It operates in two modes, \gst{text} and \gst{toc}. In \gst{text} mode, it will retrieve the content of the current document node using a DocumentContentRetrieve request. In \gst{toc} mode, it retrieves the entire table of contents for the document using a DocumentStructureRetrieve request. Either mode may also retrieve metadata for the current section or each section in the table of contents.
1965
1966\subsubsection{GS2Browse action}\label{sec:browseaction}
1967
1968GS2BrowseAction is for displaying Greenstone 2 style classifiers.
1969\subsubsection{System action}\label{sec:systemaction}
1970
1971SystemAction allows for manual reconfiguration of various components at run-time. There is no interactive web-page displaying the options, it merely turns a set of CGI arguments into an XML system request. The response from a system request is a message which is displayed to the user.
1972
1973\begin{table}
1974\caption{Configure CGI arguments}
1975\label{tab:system-cgi}
1976{\footnotesize
1977\begin{tabular}{ll}
1978\hline
1979\bf arg & \bf description\\
1980\hline
1981a=s & system action\\
1982sa=c$|$a$|$d & type of system request: c (configure), a (add/activate), \\
1983& d (delete/deactivate) \\
1984c=demo & the request will go to this collection/servicecluster \\
1985& instead of the message router\\
1986ss=collectionList & subset for configure: only reconfigure this part.\\
1987& For the MessageRouter, can be serviceClusterList, serviceList, \\
1988& collectionList, siteList.\\
1989& For a collection/cluster, can be metadataList or serviceList.\\
1990sn=demo & \\
1991st=collection& \\
1992\hline
1993\end{tabular}}
1994\end{table}
1995
1996
1997\subsection{Other code information}
1998
1999Greenstone has a set of Utility classes, which are briefly described in Table~\ref{tab:utils}.
2000
2001\begin{table}[h]
2002\caption{The utility classes in org.greenstone.gsdl3.util}
2003\label{tab:utils}
2004{\footnotesize
2005\begin{tabular}{lp{3.75in}}
2006\hline
2007\bf Utility class & \bf Description\\
2008\hline
2009Dictionary & wrapper around a Resource Bundle, providing strings with parameters\\
2010GSConstants & holds some constants used for servlet arguments and configuration variables\\
2011GSEntityResolver & an EntityResolver which can be used to find resources such as DTDs\\
2012GSFile & class to create all \gs\ file paths e.g. used to locate configuration files, XSLT files and collection data. \\
2013GSHTML & provides convenience methods for dealing with HTML, e.g. making strings HTML safe\\
2014GSParams & contains names and default values for interface parameters\\
2015NZDLParams & a subclass of GSParams which holds default service parameters too, necessary for the classic style interface.\\
2016GSPath & used to create, examine and modify message address paths\\
2017GSSQL & contains static strings for all the SQL table/field names\\
2018GSStatus & some static codes for status messages\\
2019GSXML & lots of methods for extracting information out of \gs\ XML, and creating some common types of elements. Also has static Strings for element and attribute names used by \gs\ .\\
2020GSXSLT & some manipulation functions for \gs\ XSLT\\
2021Misc & miscellaneous functions\\
2022OID & class to handle \gs\ (2) OIDs\\
2023GS3OID & subclass of OID to handle \gsiii\ OIDs\\
2024SQLQuery & contains a connection to a SQL database, along with some methods for accessing the data, such as converting MG numbers to and from Greenstone OIDs.\\
2025XMLConverter & provides methods to create new Documents, parse Strings or Files into Documents, and convert Nodes to Strings\\
2026XMLTransformer & methods to transform XML using XSLT \\
2027XSLTUtil & contains static methods to be called from within XSLT \\
2028\hline
2029\end{tabular}}
2030\end{table}
2031
2032\newpage
2033\section{Collection building architecture}\label{sec:develop-build}
2034**** GEORGE ****
2035how building actually works\\
2036the building structure/architecture\\
2037modules API\\
2038
2039\newpage
2040\section{Developing \gsiii\ : Adding new features}\label{sec:new-features}
2041
2042\subsection{Creating new services}\label{sec:new-services}
2043
2044*inherit from ServiceRack - abstract base class. this handles the main process method, determines the service name and request type. if request type is describe, and to is empty, it returns a list of services (short\_service\_info) which is initialised in the configure method. a describe request to a particular service results in getServiceDescription being called, which must be supplied by the subclass.
2045other request types (process) get sent to processXXX methods, where XXX is the service name.
2046
2047* what methods are expected
2048
2049*service type responses expected
2050
2051*a browse type service must also implement servicenameMetadataRetrieve service.
2052
2053* should a metadata retrieval service advertise what metadata is available??
2054
2055standard service type vs new service type - standard needs some xml response syntax.
2056
2057\subsection{creating new actions/pages}\label{sec:new-pages}
2058
2059\subsection{new interfaces}\label{sec:new-interfaces}
2060
2061It is easy to create new interfaces to \gsiii. Here we are talking about interfaces other than those to display in typical browser.
2062
2063Handheld devices: Use the standard servlet setup, but with a different set of XSLT files to format the pages for small screens, or use WML.
2064
2065Java GUI Interface: There are couple of alternatives. Depending on what you want to display in the GUI, you could talk to either a Receptionist or a MessageRouter. The library classes can be set up and compiled into the GUI program.
2066Talking to a Receptionist will give you access to pages of XML. It is likely that the standard Receptionist class would be used - this doesn't transform the data to HTML. Queries such as ``give me the home page of a collection'' and ``do the following search'' can be issued. All teh data needed for the result view is returned. Queries are quite simple, but are limited to what kinds of Actions are available in the library.
2067Talking to a MessageRouter requires a bit more effort on the part of the GUI program, but results in greater flexibility. The kinds of queries that can be issued are individual units of action, such as ``describe yourself'', ``search'', ``retrieve the content for this document''. More than one request may need to be made for a particular feature of the GUI. However you can ask for any combination of data available in the system, you are not relying on Actions. What you will implemenet though, may be a lot like the Action code in terms of request sequences.
2068
2069Interfaces in other programming languages: Because the communication is all XML based, other interfaces can talk to the Java library if a communication protocol is set up. This could be done using SOAP for example. LIke for Java GUI interfaces, the program could talk to a Receptionist or to a MessageRouter.
2070e.g. java interface. where you can interface to. MR vs Receptionist. diff receptionists. egs, handheld - using servlet, transforming recpt, but new set of XSLT java program other program - talk to recpt but just get back XML data for pages. java gui - just talk to MR, do all processing itself.
2071
2072Remote interfaces: remote interfaces can be set up in the same way as above, using a communication protocol between the interface, and the library program.
2073
2074\subsection{Adding new classifiers}\label{sec:new-classifiers}
2075*** GEORGE ***
2076\subsection{Adding new plugins}\label{sec:new-plugins}
2077*** GEORGE ***
2078
2079\subsection{New types of collections}\label{sec:new-coll-types}
2080
2081There are two types of standard \gs\ collections: collections built with the \gsiii\ building system, and collections that are imported from \gsii\ . There are many options to collection building but it is conceivable that these options don't meet the needs of all collection builders. \gsiii\ has an ability to use any type of collection you can come up with, assuming some java code is provided.
2082
2083There are four levels of customisation that may be needed with new collections: service, collection, interface XSLT, and action levels. We will use the example collections that come with \gs\ to describe these different levels.
2084
2085Firstly, new service classes need to be written to provide the functionality to search/browse/whatever the collection. If the services have similar interfaces and functionality to the standard services, this may be all that is needed. For example, the \gsii\ MGPP collections were the first to be served in \gsiii\ . When we came to do \gsii\ MG collections, all we had to do was write some new service classes that interacted with MG instead of MGPP. Because these collections used the same type of services, this was all we had to do. The format of the configuration files was similar, they just specified MG serviceRack classes rather than MGPP ones.
2086
2087The XML Sample Texts (gberg) collection, however, was done quite differently to the standard collections. New services were provided to search the database (built with Lucene) and to provide the documents and parts of documents (using XSLT to transform the raw XML files). The collectionConfig file had some extra information in it: a list of the documents in the collection along with their Titles. Because the standard collection class has no notion of document lists, a new class was created (org.greenstone.gsdl3.collection.XMLCollection). This class is basically the same as a standard collection class except that it looks for and stores in memory the documentList from the collectionConfig file.
2088
2089To tell \gs\ to load up a different type of collection class, we use another configuration file: etc/collectionInit.xml. This specifies the name of the collection class to use.
2090Currently, this is all that is specified in that file, but you may want to add parameters for the class etc.
2091
2092\gst{<collectionInit class="XMLCollection"/>}
2093
2094The display for the collection is also quite different. The home page for the collection displays the list of documents. To achieve this, the describe response from the collection had to include the list, and a new XSLT was written for the collection that displayed this. Collection XSLT should be put in the transform directory of the collection\footnote{These are currently only used when running \gs\ in a non-distributed fashion, but it will be added in properly at some stage}.
2095
2096Document display is significantly different to standard \gs\ . There are two modes of display: table of contents mode, and content mode. Clicking on a document link from the collection home page takes the user to the table of contents for the collection. Clicking on one of the sections in the table of contents takes them to a display of that section. To facilitate this, not only do we need new XSLT files , we also needed a new action. XMLDocumentAction was created, that used two subactions, toc and text, for the different modes of display.
2097
2098The Receptionist was told about this new action by the addition of the following element to the interfaceConfig.xml file:
2099
2100\begin{gsc}\begin{verbatim}
2101<action name='xd' class='XMLDocumentAction'>
2102 <subaction name='toc' xslt='document-toc.xsl'/>
2103 <subaction name='text' xslt='document-content.xsl'/>
2104</action>
2105\end{verbatim}\end{gsc}
2106
2107XSLT files are linked to subactions rather than the action as a whole. The collection supplies the two XSLT files written appropriately for the data it contains.
2108
2109All links that link to the documents have to be changed to use the xd action rather than the standard d action. These include the links from the home page, and the links from query results.
2110
2111Querying of the collection is almost the same as usual. The query service provides a list of parameters, does the query and then sends back a list of document identifiers. The standard query action was fine for this collection. The change occurs in the way that the results are displayed---this is accomplished using a format statement supplied in the collectionConfig file inside the search node.
2112
2113\begin{gsc}\begin{verbatim}
2114<search>
2115 <format>
2116 <gsf:template match="documentNode">
2117 <xsl:param name="collName"/>
2118 <xsl:param name="serviceName"/>
2119 <td>
2120 <b><a href="{$library_name}?a=xd&amp;sa=text&amp;c={$collName}&
2121 amp;d={@nodeID}&amp;p.a=q&amp;p.s={$serviceName}">
2122 <xsl:choose>
2123 <xsl:when test="metadataList/metadata[@name='Title']">
2124 <gsf:metadata name="Title"/>
2125 </xsl:when>
2126 <xsl:otherwise>(section)</xsl:otherwise>
2127 </xsl:choose>
2128 </a>
2129 </b> from <b><a href="{$library_name}?a=xd&amp;sa=toc&amp;
2130 c={$collName}&amp;d={@nodeID}.rt&amp;p.a=q&amp;p.s={$serviceName}">
2131 <gsf:metadata name="Title" select="root"/></a></b>
2132 </td>
2133 </gsf:template>
2134 </format>
2135</search>
2136\end{verbatim}\end{gsc}
2137
2138Instead of displaying an icon and the Title, it displays the Title of the section and the title of the document. Both of these are linked to the document: the section title to the content of that section, the document title to the table of contents for the document. Because these require non-standard arguments to the library, these parts of the template are written in XSLT not \gs\ format language. As is shown here it is perfectly feasible to write a format statement that includes XSLT mixed in with \gs\ format elements.
2139
2140The document display uses CSS to format the output---these are kept in the collection and specified in the collections XSLT files. The documents also specify DTD files. Due to the way we read in the XML files, Tomcat sometimes has trouble locating the DTDs. One option is to make all the links absolute links to files in the collection folder, the other option is to put them in \gs\ 's DTD folder greenstone3/resources/dtd.
2141
2142\subsection{The Classic Interface}
2143
2144The library seen at \gst{http://www.greenstone.org/greenstone3/nzdl} is like a mirror to \gst{http://www.nzdl.org}---it aims to present the same collections, in the same way but using \gsiii\ instead of \gsii\ . It uses a new site (nzdl) with the classic interface. The web.xml file had a new servlet entry in it to specify the combination of nzdl site and classic interface.
2145
2146The site was created by making a directory called nzdl in the sites folder. A siteConfig file was created. Because it is running on Linux, we were able to link to all the collections in the old \gs\ installation. The convert\_coll\_from\_gs2.pl script was run over all the collections to produce the new XML configuration files.
2147
2148The classic interface was created to be used by this site (and is now a standard part of Greenstone).
2149In many cases, creating a new interface just requires the new images and XSLT to be added to the new directory(see Sections~\ref{sec:sites-and-ints} and \ref{sec:interface-customise}). This classic interface required a bit more customisation.
2150
2151The standard \gsiii\ navigation bar lists all the services available for the collection. In \gsii\ , the navigation bar provides the search option, and the different classifiers. This is not service specific, but hard coded to the search and classifiers. The XSLT that produces the navigation bar needed to be altered to produce this. But also, a new Receptionist was needed.
2152The standard receptionist (DefaultReceptionist) gathers a little bit of extra information for each page of XML before transforming it: this is the list of services for the collection and their display information, allowing the services to be listed along the navigation bar. This is information that is needed by every page (except for the library home page) and therefore is obtained by the receptionist instead of by each action. The nzdl interface needed a bit more information than this: for the ClassifierBrowse service, if there was one, the list of classifiers and their display elements must be obtained. So a new Receptionist (NZDLReceptionist) was written that inherited from DefaultReceptionist, and added this new info into the page.
2153
2154One of the servlet initialisation parameters is the receptionist class: this was added to the servlet definition in the web.xml file so that the LibraryServlet would load up the right receptionist class.
2155
2156
2157\newpage
2158\section{Distributed \gs\ }\label{sec:distributed}
2159
2160\gs\ is designed to run in a distributed fashion. One \gs\ installation can talk to several sites on different computers. This requires some sort of communication protocol. Any protocol can be used, currently we have a simple SOAP protocol.
2161
2162more explanation..
2163
2164\begin{figure}[h]
2165 \centering
2166 \includegraphics[width=4in]{remote} %5.8
2167 \caption{A distributed digital library configuration running over several servers}
2168 \label{fig:remote}
2169\end{figure}
2170
2171We have used Apache Axis SOAP implementation. This is run as a servlet in Tomcat. Axis is setup during installation of Greenstone. For more details about SOAP in Greenstone, see Appendix~\ref{app:soap}. Debugging soap is described in Appendix~\ref{app:soap-debug}.
2172
2173\subsection{Serving a site using soap}
2174
2175A webs service for localsite comes predeployed, but if you want to setup a service for another site, run \gst{ant soap-deploy-site}. This will prompt you for the sitename (its directory name), and a siteuri - a unique identifier for the web service. Tomcat needs to be running for this to work.
2176
2177The ant target deploys the service for the site specified. A resource file (\gst{<sitename>.wsdd}) is created which is used to specify the service. It can be found in \gst{greenstone3/resources/soap}, and is generated from \gst{site.wsdd.template}.
2178
2179To get siteA to talk to siteB, you need to deploy a SOAP server on siteB, then add a \gst{<site>} element to the \gst{<siteList>} of siteA's \gst{siteConfig.xml} file (in \gst{greenstone3/web/sites/siteA/siteConfig.xml}).
2180
2181In the \gst{<siteList>} element, add the following (substituting the chosen site uri for siteBuri):
2182
2183\begin{gsc}\begin{verbatim}
2184<site name="siteBuri"
2185 address="http://localhost:8080/greenstone3/services/siteBuri"
2186 type="soap"/>
2187\end{verbatim}\end{gsc}
2188
2189(Note that localhost and 8080 should be changed to the values you entered when installing \gsiii).
2190
2191The servlet for siteA needs to be reconfigured \\
2192(e.g. \gst{http://localhost:8080/greenstone3/library?a=s\&sa=c}).
2193\appendix
2194
2195\newpage
2196\section{Using \gsiii\ from CVS}\label{app:cvs}
2197
2198[TODO: need to make sure building stuff is in here]
2199
2200\gsiii\ is also available via CVS. You can download the latest version of the code. This is not guaranteed to be stable, in fact it is likely to be unstable. The advantage of using CVS is that you can update the code and get the latest fixes.
2201
2202Note that you will need the Java 2 SDK, version 1.4.0 or higher, and Ant (Apache's Java based build tool, http://ant.apache.org) installed.
2203
2204To check out the \gs\ code, use:
2205
2206\begin{quote}\begin{gsc}\begin{verbatim}
2207cvs -d :pserver:cvs\[email protected]:2402/usr/local/
2208 global-cvs/gsdl-src co -P gsdl3
2209\end{verbatim}\end{gsc}\end{quote}
2210
2211If you need it, the password for anonymous CVS access is \gst{anonymous}. Note that some older versions of CVS have trouble accessing this repository due to the port number being present. We are using version 1.11.1p1.
2212
2213Greenstone is built and installed using Ant (Apache's Java based build tool,
2214http://ant.apache.org). You will need a Java Development
2215Environment (1.4 or higher), and Ant installed to use Greenstone. You can download Ant from http://ant.apache.org/bindownload.cgi.
2216
2217In the gsdl3 directory, you can run 'ant' which will give you a help message.
2218Running 'ant -projecthelp' gives a list of the targets that you can run - these
2219do various things like compile the source code, startup the server etc.
2220
2221The README.txt file has up-to-date instructions for installing from CVS. Briefly, for a first time install, run 'ant prepare install'.
2222
2223The file build.properties contains various parameters that can be set by the user. Please check these settings before running the installation process. The install process will ask you if you accept the properties before starting.
2224For a non-interactive version of the install, run
2225ant -Dproperties.accepted=yes install
2226
2227To log the output in build.log, run
2228ant -Dproperties.accepted=yes -logfile build.log install
2229
2230Under Linux, Java and C/C++ compilation is carried out. For windows, since Visual Studio is not a standard component, only Java compilation is carried out. Pre-compiled binaries are provided for the C/C++ components (packages and Greenstone 2 style building). If you have Visual Studio installed (version 6), you can run the compile-windows-c++ targets to compile the code locally. (Don't forget to setup the Visual Studio environment first, by running, e.g. C:/Program Files/Microsoft Visual Studio/VC98/Bin/VCVARS32.BAT or equivalent.)
2231
2232
2233Note: \gst{gs3-setup} sets the environment variables \gst{CLASSPATH, PATH, JAVA\_HOME} and needs to be done in a shell before doing collection building etc.
2234
2235To startup or shutdown the library (includes the Tomcat server and MYSQL server), the commands are (run from the gsdl3 directory):
2236
2237\begin{quote}\begin{gsc}
2238ant start \\
2239ant stop
2240\end{gsc}\end{quote}
2241
2242If you want to restart only Tomcat, run \gst{ant restart-tomcat}.
2243
2244\newpage
2245\section{Tomcat}\label{app:tomcat}
2246
2247Tomcat is a servlet container, and Greenstone 3 runs as a servlet inside it.
2248
2249The file \gst{\gsdlhome/comms/jakarta/tomcat/conf/server.xml} is the Tomcat configuration file. The installation process adds a context for \gsiii\ servlets (\gst{\gsdlhome/web})---this tells Tomcat where to find the web.xml file, and what URL (\gst{/greenstone3}) to give it. Anything inside the context directory is accessible via Tomcat\footnote{can we use .htaccess files to restrict access??}. For example, the index.html file that lives in \gst{\gsdlhome/web} can be accessed through the URL \gst{localhost:8080/greenstone3/index.html}. The demo collection's images can be accessed through \\
2250\gst{localhost:8080/greenstone3/sites/localsite/collect/demo/images/}.
2251
2252
2253Grenstone sets up Tomcat to run on port 8080 by default. To change this, you can edit the tomcat.port property in build.properties. If you do this before installing Greenstone, then running 'ant install' will use the new port number. If you want to change it later on, shutdown tomcat, run 'ant reconfigure-server-settings', then when you restart tomcat it will use the new port.
2254
2255Note: Tomcat must be shutdown and restarted any time you make changes in the following for those changes to take effect:
2256\begin{bulletedlist}
2257\begin{gsc}
2258\item \gsdlhome/web/WEB-INF/web.xml
2259\item \gsdlhome/comms/jakarta/tomcat/conf/server.xml
2260\end{gsc}
2261\item any classes or jar files used by the servlets
2262\end{bulletedlist}
2263\noindent Note: stdin and stdout for the servlets (on linux) both go to\\
2264\gst{\gsdlhome/comms/jakarta/tomcat/logs/catalina.out}
2265
2266On startup, the servlet loads in its collections and services. If the site or collection configuration files are changed, these changes will not take effect until the site/collection is reloaded. This can be done through the reconfiguration messages (see Section~\ref{sec:runtime-config}), or by restarting Tomcat.
2267
2268We have set up Tomcat to follow symlinks. To disable this feature, remove the \gst{<Resources>} element from the greenstone3 context in \\\gst{\$GSDL3HOME/comms/jakarta/tomcat/conf/server.xml}:
2269
2270\begin{quote}\begin{gsc}
2271<Context path="/greenstone3" docBase="\$GSDL3HOME/web" debug="1" \\
2272reloadable="true">\\
2273 <Resources allowLinking='true'/>\\
2274</Context>\\
2275\end{gsc}\end{quote}
2276
2277By default, Tomcat allows directory listings. To disable this, change the 'listings' paramter to false in the default servlet definition, in Tomcat's web.xml file (\gst{\$GSDL3HOME/comms/jakarta/tomcat/conf/web.xml}):
2278
2279We have set the greenstone context to be reloadable. This means that if a class or resource file in web/WEB-INF/lib or web/WEB-INF/classes changes, the servlet will be reloaded. This is useful for development, but should be turned off for production mode (set the reloadable attribute to false).
2280
2281Tomcat uses a Manager to handle HTTP session information. This may be stored between restarts if possible. To use a persistent session handling manager, uncomment the \gst{<Manager>} element in \\
2282\gst{\$GSDL3HOME/comms/jakarta/tomcat/conf/server.xml}. For the default manager, session information is stored in the work directory:\\
2283\gst{\$GSDL3HOME/comms/jakarta/tomcat/work/Standalone/localhost/greenstone3/SESSIONS.ser}. Delete this file to clear the cached session info. Note that Tomcat needs to be shutdown to delete this file.
2284
2285\subsection{Proxying Tomcat with apache}
2286
2287Instead of incorporating servlet support into your existing web server, an easy alternative is to proxy Tomcat. The \gst{http://www.greenstone.org/greenstone3} site uses apache to proxy Tomcat. ProxyPass and ProxyPassReverse directives need to be added to the Virtualhost description for the www.greenstone.org server.
2288
2289\begin{quote}\begin{gsc}
2290<VirtualHost xx.xx.xx.xx>\\
2291ServerName www.greenstone.org\\
2292...\\
2293ProxyPass /greenstone3 http://puka.cs.waikato.ac.nz:8080/greenstone3\\
2294ProxyPassReverse /greenstone3 http://puka.cs.waikato.ac.nz:8080/greenstone3\\
2295</VirtualHost>\\
2296\end{gsc}\end{quote}
2297
2298In our example, the \gsiii\ servlet can be accessed at \\
2299\gst{http://www.greenstone.org/greenstone3/library}, instead of at \\
2300\gst{http://puka.cs.waikato.ac.nz:8080/greenstone3/library}, which is not publically accessible.
2301
2302\subsection{Running Tomcat behind a proxy}
2303
2304Almost everything works fine when Tomcat is running behind a proxy. The only time this causes trouble is if the servlet itself needs to make external http connections. We do this in the infomine demo collection for example. One of the service classes sends http requests to the infomine database at riverside. Since this is going through the proxy, a username and password is needed. It is not sufficient to prompt the user for a password because they are unlikely to have a password for the particular proxy that Tomcat is using. What we have done at present is to put a proxy element in the siteConfig.xml file. Here you have to enter a suitable username and password for the proxy server. Unfortunately these are entered in plain text. And the file is viewable via the servlet. So we need a better solution.
2305
2306\newpage
2307\section{SOAP}\label{app:soap}
2308
2309Grenstone uses the Apache Axis SOAP implementation for distributed communications. Axis runs as a servlet inside Tomcat, and SOAP web services can be deployed by this Axis servlet. The Greenstone installation process sets up Axis for Tomcat, and predeploys the localsite web service.
2310
2311To deploy a SOAP service for other sites, run \gst{ant soap-deploy-site}
2312
2313This will prompt you for the sitename (the site's directory name), and a unique URI for the site. It creates a new SOAPServer class for the site \\(\gst{\$GSDL3HOME/src/java/org/greenstone/gsdl3/SOAPServer<sitename>.java}), creates a resource file for deployment (\gst{\$GSDL3HOME/resources/soap/<sitename>.wsdd}), and then tries to deploy the service.
2314
2315Information about deployed services is maintained between Tomcat sessions---you only need to deploy something once. To undeploy a site, use \gst{ant undeploy-soap-site}.
2316
2317The axis services can be accessed at \gst{localhost:8080/greenstone3/index.jsp}.
2318
2319\subsection{Debugging SOAP}\label{app:soap-debug}
2320
2321If you need to debug the SOAP stuff for some reason, or just want to look at the SOAP messages that are being passed back and forth, you can use the TCP monitor. This intercepts messages coming in to one port, displays them, and passes them to another port.
2322To run it, type:
2323
2324\begin{quote}\gst{java -cp <path to greenstone3>/comms/soap/axis/lib/axis.jar \\
2325org.apache.axis.utils.tcpmon}
2326\end{quote}
2327
2328The listen port is the port that you want the monitor to be listening on. It should 'act as' a Listener, with target hostname 127.0.0.1 (localhost), and target port the port that Tomcat is running on (8080). You need to modify the address used to talk to the SOAP service. For example, if you want to monitor traffic between the gateway site and the localsite SOAP server, you will need to edit gateway's siteConfig.xml file and change the port number (in the site element) to whatever you have chosen as the listen port.
2329
2330\newpage
2331\section{Tidying up the formatting for imported Greenstone 2 collections}\label{app:gs2tidy}
2332
2333\subsection{Format statements: \gsii\ vs \gsiii\ }\label{app:gs2format}
2334The following table shows the \gsii\ format elements, and their equivalents in \gsiii\
2335\begin{table}[h]
2336\caption{\gsiii\ equivalents of \gsii\ format statements}
2337{\footnotesize
2338\begin{tabular}{ll}
2339\hline
2340\bf \gsii\ & \bf \gsiii\ \\
2341\hline
2342\gst{[Text]} & \gst{<gsf:text/>} \\
2343\gst{[num]} & \gst{<gsf:metadata name='docnum'/>}\\
2344\gst{[link][/link]} & \gst{<gsf:link></gsf:link>} or \\
2345& \gst{<gsf:link type='document'></gsf:link>}\\
2346\gst{[srclink][/srclink]} & \gst{<gsf:link type='source'></gsf:link>}\\
2347\gst{[icon]} & \gst{<gsf:icon/>} or \\
2348& \gst{<gsf:icon type='document'/>}\\
2349\gst{[srcicon]} & \gst{<gsf:icon type='source'/>}\\
2350\gst{[Title]} (metadata) & \gst{<gsf:metadata name='Title'/>} or \\
2351& \gst{<gsf:metadata name='Title' select='current'/>}\\
2352\gst{[parent:Title]} & \gst{<gsf:metadata name='Title' select='parent' />}\\
2353\gst{[parent(All):Title]} & \gst{<gsf:metadata name='Title' select='ancestors'/>}\\
2354\gst{[parent(Top):Title]} & \gst{<gsf:metadata name='Title' select='root' />}\\
2355\gst{[parent(All': '):Title]} & \gst{<gsf:metadata name='Title' select='ancestors'}\\
2356& \gst{ separator=': ' />}\\
2357\gst{[sibling(All': '):Title]} & \gst{<gsf:metadata name='Title' multiple='true'} \\
2358& \gst{ separator=': ' />}\\
2359\gst{\{Or\}\{[dc.Title],} & \gst{<gsf:choose-metadata>}\\
2360\gst{ [dls.Title], [Title]\}}& \gst{ <gsf:metadata name='dc.Title'/>}\\
2361& \gst{ <gsf:metadata name='dls.Title'/>}\\
2362& \gst{ <gsf:metadata name='Title'/>}\\
2363& \gst{</gsf:choose-metadata>}\\
2364\gst{\{If\}\{[parent:Title],} & \gst{<gsf:choose-metadata>}\\
2365\gst{ [parent:Title], [Title]\}}& \gst{ <gsf:metadata name='Title' select='parent'/>}\\
2366& \gst{ <gsf:metadata name='Title'/>}\\
2367& \gst{</gsf:choose-metadata>}\\
2368\gst{\{If\}\{[Subject],} & \gst{<gsf:switch>}\\
2369\gst{ <td>[Subject]</td>\}}& \gst{ <gsf:metadata name='Subject'/>}\\
2370& \gst{ <gsf:when test='exists'>} \\
2371& \gst{ <td><gsf:metadata name='Subject'/></td>}\\
2372& \gst{ </gsf:when></gsf:switch>}\\
2373\hline
2374\end{tabular}}
2375\end{table}
2376\subsection{Cleaning up macros}\label{app:gs2replace}
2377
2378Here we show some of the replace items that have been used for Greenstone 2 collections.
2379
2380Getting rid of silly backslashes:
2381\begin{gsc}\begin{verbatim}
2382<replace scope='text' macro="\\?\\\(" text="\("/>
2383\end{verbatim}\end{gsc}
2384
2385Macro resolving using resource bundles and metadata:
2386\begin{gsc}\begin{verbatim}
2387<replace scope='metadata' macro="_magazines_" bundle="NZDLMacros"
2388 key="Magazines"/>
2389<replace scope='all' macro='_thisOID_' metadata='archivedir'/>
2390<replace macro="_httpcollimg_"
2391 text="sites/localsite/collect/folktale/index/assoc"/>
2392\end{verbatim}\end{gsc}
2393
2394Fixing up broken external links:
2395\begin{gsc}\begin{verbatim}
2396<replace macro="_httpextlink_&amp;rl=1&amp;href="
2397 text="?a=d&amp;c=folktale&amp;s0.ext=1&amp;d="/>
2398<replace macro="_httpextlink_&amp;rl=0&amp;href="
2399 text="?a=p&amp;sa=html&amp;c=folktale&amp;url="/>
2400\end{verbatim}\end{gsc}
2401
2402These two examples show how to deal with Greenstone 2's external link macros. The first one is for a 'relative' external link. In this case, the links are like URL's but they actually refer to Greenstone internal documents. So the Greensotne 3 link is to the document, but with parameter s0.ext signifying that the d argument will need translating before retrieving the content.
2403The second example is a truly external link. This is translated into a html type page action, where the url is presented as a frame along with the collection header in a separate frame.
2404
2405Sometimes we need to add in macros to be resolved in a second step:
2406\begin{gsc}\begin{verbatim}
2407<replace macro="_iconpdf_" scope="metadata"
2408 text="&lt;img title='_texticonpdf_' src='interfaces/default/images/ipdf.gif'/&gt;"/>
2409<replace macro="_texticonpdf_" scope="metadata" bundle="interface_classic"
2410 key="texticonpdf"/>
2411\end{verbatim}\end{gsc}
2412
2413\end{document}
Note: See TracBrowser for help on using the repository browser.