root/other-projects/nightly-tasks/diffcol/trunk/model-collect/Word-PDF-Formatting/archives/HASH1a9c.dir/doc.xml @ 29405

Revision 29405, 53.6 KB (checked in by ak19, 5 years ago)

Trying to rebuild the Word-PDF-Formatting collection with unique dc.Title metadata for docs that have identical final names and with the 2nd browsing classifier sorted on dc.Title, in order to produce a consistent order for browse classifiers' children (a consistent presentation order of the files under the browsing classifiers). This is necessary for perl 5.18/5.17 and later, since they randomise the order of children of unsorted classifiers and for those children with identical filenames. Changes made particularly to collect.cfg and import/metadata.xml

Line 
1<?xml version="1.0" encoding="utf-8" standalone="no"?>
2<!DOCTYPE Archive SYSTEM "http://greenstone.org/dtd/Archive/1.0/Archive.dtd">
3<Archive>
4<Section>
5  <Description>
6    <Metadata name="gsdldoctype">indexed_doc</Metadata>
7    <Metadata name="Language">en</Metadata>
8    <Metadata name="Encoding">utf8</Metadata>
9    <Metadata name="Author">Bronwyn</Metadata>
10    <Metadata name="Title">Greenstone: A Comprehensive Open-Source Digital Library Software...</Metadata>
11    <Metadata name="URL">http://Scratch/ak19/gs2-svn-22Oct2014/collect/Word-PDF-Formatting/tmp/1414470426_1/pdf01.html</Metadata>
12    <Metadata name="UTF8URL">http://Scratch/ak19/gs2-svn-22Oct2014/collect/Word-PDF-Formatting/tmp/1414470426_1/pdf01.html</Metadata>
13    <Metadata name="gsdlsourcefilename">import/pdf01.pdf</Metadata>
14    <Metadata name="gsdlconvertedfilename">tmp/1414470426_1/pdf01.html</Metadata>
15    <Metadata name="OrigSource">pdf01.html</Metadata>
16    <Metadata name="Source">pdf01.pdf</Metadata>
17    <Metadata name="SourceFile">pdf01.pdf</Metadata>
18    <Metadata name="Plugin">PDFPlugin</Metadata>
19    <Metadata name="FileSize">269487</Metadata>
20    <Metadata name="FilenameRoot">pdf01</Metadata>
21    <Metadata name="FileFormat">PDF</Metadata>
22    <Metadata name="srcicon">_iconpdf_</Metadata>
23    <Metadata name="srclink_file">doc.pdf</Metadata>
24    <Metadata name="srclinkFile">doc.pdf</Metadata>
25    <Metadata name="NumPages">9</Metadata>
26    <Metadata name="dc.Creator">Ian H. Witten</Metadata>
27    <Metadata name="dc.Creator">Rodger J. McNab</Metadata>
28    <Metadata name="dc.Creator">Stefan J. Boddie</Metadata>
29    <Metadata name="dc.Creator">David Bainbridge</Metadata>
30    <Metadata name="dc.Title">Greenstone: A comprehensive open-source digital library software system - pdf</Metadata>
31    <Metadata name="ex.ExifTool.ExifToolVersion">8.57</Metadata>
32    <Metadata name="ex.File.Directory">/Scratch/ak19/gs2-svn-22Oct2014/collect/Word-PDF-Formatting/import</Metadata>
33    <Metadata name="ex.File.FileModifyDate">2014:10:28 17:27:05+13:00</Metadata>
34    <Metadata name="ex.File.FileName">pdf01.pdf</Metadata>
35    <Metadata name="ex.File.FilePermissions">664</Metadata>
36    <Metadata name="ex.File.FileSize">269487</Metadata>
37    <Metadata name="ex.File.FileType">PDF</Metadata>
38    <Metadata name="ex.File.MIMEType">application/pdf</Metadata>
39    <Metadata name="ex.PDF.Author">Bronwyn</Metadata>
40    <Metadata name="ex.PDF.CreateDate">2000:03:02 15:21:24</Metadata>
41    <Metadata name="ex.PDF.Creator">Microsoft Word</Metadata>
42    <Metadata name="ex.PDF.Linearized">false</Metadata>
43    <Metadata name="ex.PDF.PDFVersion">1.2</Metadata>
44    <Metadata name="ex.PDF.PageCount">9</Metadata>
45    <Metadata name="ex.PDF.Producer">Acrobat PDFWriter 4.0 for Power Macintosh</Metadata>
46    <Metadata name="Identifier">HASH1a9cea0f239f754007681b</Metadata>
47    <Metadata name="lastmodified">1414470425</Metadata>
48    <Metadata name="lastmodifieddate">20141028</Metadata>
49    <Metadata name="oailastmodified">1414470426</Metadata>
50    <Metadata name="oailastmodifieddate">20141028</Metadata>
51    <Metadata name="assocfilepath">HASH1a9c.dir</Metadata>
52    <Metadata name="gsdlassocfile">pdf01-2_1.jpg:image/jpeg:</Metadata>
53    <Metadata name="gsdlassocfile">pdf01-3_1.jpg:image/jpeg:</Metadata>
54    <Metadata name="gsdlassocfile">pdf01-4_1.jpg:image/jpeg:</Metadata>
55    <Metadata name="gsdlassocfile">pdf01-5_1.jpg:image/jpeg:</Metadata>
56    <Metadata name="gsdlassocfile">pdf01-7_1.jpg:image/jpeg:</Metadata>
57    <Metadata name="gsdlassocfile">pdf01-8_1.jpg:image/jpeg:</Metadata>
58    <Metadata name="gsdlassocfile">doc.pdf:application/pdf:</Metadata>
59  </Description>
60  <Content>
61&lt;A name=1&gt;&lt;/a&gt;&lt;b&gt;Greenstone:  A Comprehensive Open-Source&lt;/b&gt;&lt;br&gt;
62&lt;b&gt;Digital Library Software System&lt;/b&gt;&lt;br&gt;
63&lt;i&gt;Ian H. Witten,* Rodger J. McNab,† Stefan J. Boddie,* David Bainbridge*&lt;/i&gt;&lt;br&gt;
64* Dept of Computer Science&lt;br&gt;
65† Digilib Systems&lt;br&gt;
66University of Waikato, New Zealand&lt;br&gt;
67Hamilton, New Zealand&lt;br&gt;
68E-mail:  {ihw, sjboddie, davidb}@cs.waikato.ac.nz&lt;br&gt;
69E-mail:  rodger@digilibs.com&lt;br&gt;
70&lt;b&gt;ABSTRACT&lt;/b&gt;&lt;br&gt;
71multilingual information retrieval to distributed computing&lt;br&gt;protocols, from interoperability to search engine&lt;br&gt;
72This paper describes the Greenstone digital library&lt;br&gt;
73technology, from metadata standards to multiformat&lt;br&gt;
74software, a comprehensive, open-source system for the&lt;br&gt;
75document parsing, from multimedia to multiple operating&lt;br&gt;
76construction and presentation of information collections.&lt;br&gt;
77systems, from Web browsers to plug-and-play DVDs.&lt;br&gt;
78Collections built with Greenstone offer effective full-text&lt;br&gt;searching and metadata-based browsing facilities that are&lt;br&gt;
79The Greenstone Digital Library Software from the New&lt;br&gt;
80attractive and easy to use. Moreover, they are easily&lt;br&gt;
81Zealand Digital Library (NZDL) project tackles this issue&lt;br&gt;
82maintainable and can be augmented and rebuilt entirely&lt;br&gt;
83by providing a new way of organizing information and&lt;br&gt;
84automatically. The system is extensible: software&lt;br&gt;
85making it available over the Internet. A &lt;i&gt;collection&lt;/i&gt; of&lt;br&gt;
86“plugins” accommodate different document and metadata&lt;br&gt;
87information comprises several (typically several thousand,&lt;br&gt;
88types.&lt;br&gt;
89or several million) &lt;i&gt;documents&lt;/i&gt;, and a uniform interface is&lt;br&gt;provided to all documents in a collection. A library may&lt;br&gt;
90&lt;b&gt;INTRODUCTION&lt;/b&gt;&lt;br&gt;
91include many different collections, each organized&lt;br&gt;differently—though there is a strong family resemblance in&lt;br&gt;
92Notwithstanding intense research activity in the digital&lt;br&gt;
93how collections are presented.&lt;br&gt;
94library field during the second half of the 1990s,&lt;br&gt;comprehensive software systems for creating digital&lt;br&gt;
95Making information available using this system is far more&lt;br&gt;
96libraries are not widely available. In fact, the usual solution&lt;br&gt;
97than “just putting it on the Web.” The collection becomes&lt;br&gt;
98when creating a digital library is also the most&lt;br&gt;
99maintainable, searchable, and browsable. Each collection,&lt;br&gt;
100obvious—just put it on the Web. But consider how much&lt;br&gt;
101prior to presentation, undergoes a “building” process that,&lt;br&gt;
102effort is involved in constructing a Web site for a digital&lt;br&gt;
103once established, is completely automatic. This process&lt;br&gt;
104library. To be effective it needs to be visually attractive&lt;br&gt;
105creates all the structures that are used at run-time for&lt;br&gt;
106and ergonomically easy to use, incorporate convenient and&lt;br&gt;
107accessing the collection. Searching is based on various&lt;br&gt;
108powerful searching capabilities, and offer rich and natural&lt;br&gt;
109indexes, while browsing is based on various metadata;&lt;br&gt;
110browsing facilities. Above all it must be easy to maintain&lt;br&gt;
111support structures for both are created during the building&lt;br&gt;
112and augment, which presents a significant challenge if any&lt;br&gt;
113operation. When new material appears it can be fully&lt;br&gt;
114manual organization is involved.&lt;br&gt;
115incorporated into the collection by rebuilding.&lt;br&gt;
116The alternative is to automate these activities through&lt;br&gt;
117To address the exceptionally broad demands of digital&lt;br&gt;
118software tools. But the broad scope of digital library&lt;br&gt;
119libraries, the system is public and extensible. It is issued&lt;br&gt;
120requirements makes this a daunting prospect. Ideally the&lt;br&gt;
121under the Gnu public license and, in the spirit of open-&lt;br&gt;
122software should incorporate facilities ranging from&lt;br&gt;
123source software, users are invited to contribute&lt;br&gt;modifications and enhancements. Only through an&lt;br&gt;international cooperative effort will digital library software&lt;br&gt;become sufficiently comprehensive to meet the world’s&lt;br&gt;needs. Currently the Greenstone software is used at sites in&lt;br&gt;Canada, Germany, New Zealand, Romania, UK, and the&lt;br&gt;US, and collections range from newspaper articles to&lt;br&gt;technical documents, from educational journals to oral&lt;br&gt;history, from visual art to folksongs. The software has&lt;br&gt;been used for collections in many different languages, and&lt;br&gt;for CD-ROMs that have been published by the United&lt;br&gt;Nations and other humanitarian agencies in Belgium,&lt;br&gt;France, Japan, and the US for distribution in developing&lt;br&gt;countries (Humanity Libraries, 1998; PAHO, 1999;&lt;br&gt;UNESCO, 1999; UNU, 1998). Further details can be&lt;br&gt;obtained from &lt;i&gt;www.nzdl.org&lt;/i&gt;.&lt;br&gt;
124&lt;hr&gt;
125&lt;A name=2&gt;&lt;/a&gt;&lt;IMG src=&quot;_httpdocimg_/pdf01-2_1.jpg&quot;&gt;&lt;br&gt;
126become a first-class component of the library. And what&lt;br&gt;permits it to be integrated into existing searching and&lt;br&gt;browsing structures without any manual intervention is&lt;br&gt;&lt;i&gt;metadata&lt;/i&gt;. This provides sufficient focus to the concept of&lt;br&gt;“digital library” to support the development of a&lt;br&gt;construction kit.&lt;br&gt;
127&lt;b&gt;OVERVIEW OF GREENSTONE&lt;/b&gt;&lt;br&gt;
128 &lt;br&gt;Information collections built by Greenstone combine&lt;br&gt;extensive full-text search facilities with browsing indexes&lt;br&gt;based on different metadata types. There are several ways&lt;br&gt;for users to find information, although they differ between&lt;br&gt;collections depending on the metadata available and the&lt;br&gt;collection design. Typically you can &lt;i&gt;search for particular&lt;br&gt;words&lt;/i&gt; that appear in the text, or within a section of a&lt;br&gt;document, or within a title or section heading. You can&lt;br&gt;&lt;i&gt;browse documents by title&lt;/i&gt;: just click on the displayed book&lt;br&gt;icon to read it. You can &lt;i&gt;browse documents by subject&lt;/i&gt;.&lt;br&gt;Subjects are represented by bookshelves: just click on a&lt;br&gt;shelf to see the books. Where appropriate, documents&lt;br&gt;
129&lt;b&gt;Figure 1: Searching the HDL collection&lt;/b&gt;&lt;br&gt;
130come complete with a table of contents (constructed&lt;br&gt;automatically): you can click on a chapter or subsection to&lt;br&gt;
131This paper sets the scene with a brief discussion of what a&lt;br&gt;
132open it, expand the full table of contents, or expand the full&lt;br&gt;
133digital library is. We then give an overview of the facilities&lt;br&gt;
134document.&lt;br&gt;
135offered by Greenstone and show how end users find&lt;br&gt;information in collections. Next we describe the files and&lt;br&gt;
136 &lt;br&gt;An example of searching is shown in Figure 1 where&lt;br&gt;
137directories involved in a collection, and then discuss the&lt;br&gt;
138documents in the Global Help Project’s Humanity&lt;br&gt;
139processes of updating existing collections and creating new&lt;br&gt;
140Development Library (HDL) are being searched for&lt;br&gt;
141ones, including extending the software to provide new&lt;br&gt;
142chapters matching the word &lt;i&gt;butterfly&lt;/i&gt;. In Figure 2 the same&lt;br&gt;
143facilities. We conclude with an overview of related work.&lt;br&gt;
144collection is being browsed by subject: by clicking on the&lt;br&gt;bookshelf icons the user has discovered an item under&lt;br&gt;
145&lt;b&gt;WHAT IS A DIGITAL LIBRARY?&lt;/b&gt;&lt;br&gt;
146Section 16, Animal Husbandry. Pursuing an interest in&lt;br&gt;butterfly farming, the user selects a book by clicking on its&lt;br&gt;
147 &lt;br&gt;Ten definitions of the term “digital library” have been&lt;br&gt;
148book icon. In Figure 3 the front cover of the book is&lt;br&gt;
149culled from the literature by Fox (1998), and their spirit is&lt;br&gt;
150displayed as a graphic on the left, and the automatically&lt;br&gt;
151captured in the following brief characterization:&lt;br&gt;
152constructed table of contents appears at the start of the&lt;br&gt;
153 &lt;br&gt;
154document. The current focus, &lt;i&gt;Introduction and Summary&lt;/i&gt;,&lt;br&gt;
155&lt;i&gt;A collection of digital objects, including text,&lt;/i&gt;&lt;br&gt;
156is shown in bold in the table of contents with its text&lt;br&gt;
157&lt;i&gt;video, and audio, along with methods for access&lt;/i&gt;&lt;br&gt;
158starting further down the page.&lt;br&gt;
159&lt;i&gt;and retrieval, and for selection, organization&lt;br&gt;and maintenance of the collection&lt;/i&gt;&lt;br&gt;
160 &lt;br&gt;In accordance with Lesk’s advice, a statement of purpose&lt;br&gt;
161 &lt;br&gt;
162and coverage accompanies each collection, along with an&lt;br&gt;
163(Akscyn and Witten, 1998). Lesk (1998) views digital&lt;br&gt;
164explanation of how it is organized (Figure 1 shows the&lt;br&gt;
165libraries as “organized collections of digital information,”&lt;br&gt;
166start of this). A distinction is made between &lt;i&gt;searching&lt;/i&gt; and&lt;br&gt;
167and wisely recommends that they articulate the principles&lt;br&gt;
168&lt;i&gt;browsing&lt;/i&gt;. Searching is full-text, and—depending on the&lt;br&gt;
169governing what is included and how the collection is&lt;br&gt;
170collection’s design—the user can choose between indexes&lt;br&gt;
171organized.&lt;br&gt;
172built from different parts of the documents, or from&lt;br&gt;
173 &lt;br&gt;Digital libraries are generally distinguished from the&lt;br&gt;
174different metadata. Some collections have an index of full&lt;br&gt;
175World-Wide Web, the essential difference being in&lt;br&gt;
176documents, an index of sections, an index of paragraphs,&lt;br&gt;
177selection and organization. But they are not generally&lt;br&gt;
178an index of titles, and an index of section headings, each of&lt;br&gt;
179distinguished from a web &lt;i&gt;site&lt;/i&gt;: indeed, virtually all extant&lt;br&gt;
180which can be searched for particular words or phrases.&lt;br&gt;
181digital libraries manifest themselves as a web site. Hence&lt;br&gt;
182Browsing involves data structures created from metadata&lt;br&gt;
183the obvious question: to make a digital library, why not&lt;br&gt;
184that the user can examine: lists of authors, lists of titles,&lt;br&gt;
185just put the information on the Web?&lt;br&gt;
186lists of dates, hierarchical classification structures, and so&lt;br&gt;
187 &lt;br&gt;
188on. Data structures for both browsing and searching are&lt;br&gt;
189But we make a distinction between a digital library and a&lt;br&gt;
190built according to instructions in a configuration file,&lt;br&gt;
191web site that lies at the heart of our software design: one&lt;br&gt;
192which controls both building and serving the collection.&lt;br&gt;
193should easily be able to add new material to a library&lt;br&gt;
194Sample configuration files are discussed below.&lt;br&gt;
195without having to integrate it manually or edit its content&lt;br&gt;in any way. Once added, new material should immediately&lt;br&gt;
196&lt;hr&gt;
197&lt;A name=3&gt;&lt;/a&gt;&lt;IMG src=&quot;_httpdocimg_/pdf01-3_1.jpg&quot;&gt;&lt;br&gt;
198matter of specifying all the necessary plugins. In order to&lt;br&gt;build browsing indexes from metadata, an analogous&lt;br&gt;scheme of “classifiers” is used: classifiers create indexes&lt;br&gt;of various kinds based on metadata. Source documents are&lt;br&gt;brought into the Greenstone system through a process&lt;br&gt;called &lt;i&gt;importing&lt;/i&gt;, which uses the plugins and classifiers&lt;br&gt;specified in the collection configuration file.&lt;br&gt;
199 &lt;br&gt;The international Unicode character set is used throughout,&lt;br&gt;so documents—and interfaces—can be written in any&lt;br&gt;language. Collections have so far been produced in&lt;br&gt;English, French, Spanish, German, Maori, Chinese, and&lt;br&gt;Arabic. The NZDL Web site provides numerous examples.&lt;br&gt;Collections can contain text, pictures, and even audio and&lt;br&gt;video clips; a text-only version of the interface is also&lt;br&gt;provided to accommodate visually impaired users.&lt;br&gt;Compression technology is used to ensure best use of&lt;br&gt;storage (Witten &lt;i&gt;et al &lt;/i&gt;., 1999). Most non-textual material is&lt;br&gt;either linked to textual documents or accompanied by&lt;br&gt;textual descriptions (such as photo captions) to allow full-&lt;br&gt;text searching and browsing. However, the architecture&lt;br&gt;
200&lt;b&gt;Figure 2: Browsing the HDL collection by subject&lt;/b&gt;&lt;br&gt;
201permits the implementation of plugins and classifiers even&lt;br&gt;for non-textual data.&lt;br&gt;
202 &lt;br&gt;Rich browsing facilities can be provided by manually&lt;br&gt;
203 &lt;br&gt;
204linking parts of documents together and building explicit&lt;br&gt;
205The system includes an “administrative” function whereby&lt;br&gt;
206indexes and tables of contents. However, manually-created&lt;br&gt;
207specified users can examine the composition of all&lt;br&gt;
208linking becomes difficult to maintain, and often falls into&lt;br&gt;
209collections, protect documents so that they can only be&lt;br&gt;
210disrepair when a collection expands. The Greenstone&lt;br&gt;
211accessed by registered users on presentation of a password,&lt;br&gt;
212software takes a different tack: it facilitates &lt;i&gt;maintainability&lt;/i&gt;&lt;br&gt;
213and so on. Logs of user activity are kept that record all&lt;br&gt;
214by creating all searching and browsing structures&lt;br&gt;
215queries made to every Greenstone collection (though this&lt;br&gt;
216automatically from the documents themselves. No links&lt;br&gt;
217facility can be disabled).&lt;br&gt;
218are inserted by hand. This means that when new&lt;br&gt;
219 &lt;br&gt;Although primarily designed for Internet access over the&lt;br&gt;
220documents in the same format become available, they can&lt;br&gt;
221World-Wide Web, collections can be made available, in&lt;br&gt;
222be added automatically. Indeed, for some collections this is&lt;br&gt;
223precisely the same form, on CD-ROM. In either case they&lt;br&gt;
224done by processes that wake up regularly, scout for new&lt;br&gt;
225are accessed through any Web browser. Greenstone CD-&lt;br&gt;
226material, and rebuild the indexes—all without manual&lt;br&gt;
227ROMs operate on a standalone PC under Windows 3.X,&lt;br&gt;
228intervention.&lt;br&gt;
22995, 98, and NT, and the interaction is identical to accessing&lt;br&gt;
230Collections comprise many documents: thousands, tens of&lt;br&gt;
231the collection on the Web—except that response is faster&lt;br&gt;
232thousands, or even millions. Each document may be&lt;br&gt;
233and more predictable. The requirement to operate on early&lt;br&gt;
234hierarchically organized into &lt;i&gt;sections&lt;/i&gt; (subsections, sub-&lt;br&gt;
235Windows systems is one that plagues the software design,&lt;br&gt;
236subsections, and so on). Each section comprises one or&lt;br&gt;
237but is crucial for many users—particularly those in&lt;br&gt;
238more &lt;i&gt;paragraphs&lt;/i&gt;. Metadata such as author, title, date,&lt;br&gt;
239underdeveloped countries seeking access to humanitarian&lt;br&gt;
240keywords, and so on, may be associated with documents,&lt;br&gt;
241aid collections. If the PC is connected to a network&lt;br&gt;
242or with individual sections of documents. This is the raw&lt;br&gt;
243(intranet or Internet), a custom-built Web server provided&lt;br&gt;
244material for indexes. It must either be provided explicitly&lt;br&gt;
245on each CD makes exactly the same information available&lt;br&gt;
246for each document and section (for example, in an&lt;br&gt;
247to others through their standard Web browser. The use of&lt;br&gt;
248accompanying spreadsheet) or be derivable automatically&lt;br&gt;
249compression ensures that the greatest possible volume of&lt;br&gt;
250from the source documents. Metadata is converted to&lt;br&gt;
251information can be packed on to a CD-ROM.&lt;br&gt;
252Dublin Core and stored with the document for internal use.&lt;br&gt;
253 &lt;br&gt;The collection-serving software operates under Unix and&lt;br&gt;
254 &lt;br&gt;In order to accommodate different kinds of source&lt;br&gt;
255Windows NT, and works with standard Web servers. A&lt;br&gt;
256documents, the software is organized so that “plugins” can&lt;br&gt;
257flexible process structure allows different collections to be&lt;br&gt;
258be written for new document types. Plugins exist for plain&lt;br&gt;
259served by different computers, yet be presented to the user&lt;br&gt;
260text documents, HTML documents, email documents, and&lt;br&gt;
261in the same way, on the same Web page, as part of the&lt;br&gt;
262bibliographic formats. Word documents are handled by&lt;br&gt;
263same digital library, even as part of the same collection&lt;br&gt;
264saving them as HTML; PostScript ones by applying a&lt;br&gt;
265(McNab and Witten, 1998). Existing collections can be&lt;br&gt;
266preprocessor (Nevill-Manning &lt;i&gt;et al&lt;/i&gt;., 1998). Specially&lt;br&gt;
267updated and new ones brought on-line at any time, without&lt;br&gt;
268written plugins also exist for proprietary formats such as&lt;br&gt;
269bringing the system down; the process responsible for the&lt;br&gt;
270that used by the BBC archives department. A collection&lt;br&gt;
271user interface will notice (through periodic polling) when&lt;br&gt;
272may have source documents in different forms: it is just a&lt;br&gt;
273new collections appear and add them to the list presented&lt;br&gt;to the user.&lt;br&gt;
274&lt;hr&gt;
275&lt;A name=4&gt;&lt;/a&gt;&lt;IMG src=&quot;_httpdocimg_/pdf01-4_1.jpg&quot;&gt;&lt;br&gt;
276&lt;b&gt;FILES IN A COLLECTION&lt;/b&gt;&lt;br&gt;
277 &lt;br&gt;When a new collection is created or material is added to an&lt;br&gt;existing one, the original source documents are first&lt;br&gt;brought into the system through a process known as&lt;br&gt;“importing.” This involves converting documents into a&lt;br&gt;simple HTML-like format known as GML (for&lt;br&gt;“Greenstone Markup Language”), which includes any&lt;br&gt;metadata associated with the document. Documents are&lt;br&gt;assumed to be in the Unicode UTF-8 code (of which the&lt;br&gt;ASCII characters form a subset).&lt;br&gt;
278 &lt;br&gt;&lt;b&gt;Files and directories&lt;/b&gt;&lt;br&gt;
279 &lt;br&gt;There is a separate directory for each collection, which&lt;br&gt;contains five subdirectories: the original raw material&lt;br&gt;(&lt;i&gt;import&lt;/i&gt;), the GML files created from this (&lt;i&gt;archives&lt;/i&gt;), the&lt;br&gt;final collection as it is served to users (&lt;i&gt;index&lt;/i&gt;), a directory&lt;br&gt;for use during the building process (&lt;i&gt;building&lt;/i&gt;), and one for&lt;br&gt;any supporting files (&lt;i&gt;etc&lt;/i&gt;)—including the configuration file&lt;br&gt;
280&lt;b&gt;Figure 3: Reading a book in the HDL&lt;/b&gt;&lt;br&gt;
281that controls the collection creation procedure. Additional&lt;br&gt;files might be required: for example, building a hierarchy&lt;br&gt;of classifications requires a data file of sub-classifications.&lt;br&gt;
282&lt;b&gt;FINDING INFORMATION&lt;/b&gt;&lt;br&gt;
283 &lt;br&gt;Greenstone digital library systems generally include&lt;br&gt;
284 &lt;br&gt;
285several separate collections. A home page allows you to&lt;br&gt;
286&lt;b&gt;The imported documents&lt;/b&gt;&lt;br&gt;
287select a collection; in addition, each collection has its own&lt;br&gt;
288 &lt;br&gt;In order to identify documents internally, a unique object&lt;br&gt;
289“about” page that gives you information about how the&lt;br&gt;
290identifier or OID is assigned to each original source&lt;br&gt;
291collection is organized and the principles governing what&lt;br&gt;
292document when it is imported (formed by hashing the&lt;br&gt;
293is included.&lt;br&gt;
294content, to overcome file duplication effects caused by&lt;br&gt;
295 &lt;br&gt;All icons in the screenshots of Figures 1–4 are clickable.&lt;br&gt;
296mirroring) and stored as metadata within that document. It&lt;br&gt;
297Those icons at the top of the page return to the home page,&lt;br&gt;
298is important that OIDs persist throughout the index-&lt;br&gt;
299provide help text, and allow you to set user interface and&lt;br&gt;
300building process—so that a user’s search history is&lt;br&gt;
301searching preferences. The navigation bar underneath&lt;br&gt;
302unaffected by rebuilding the collection. OIDs are assigned&lt;br&gt;
303gives access to the searching and browsing facilities,&lt;br&gt;
304by hashing the contents of the original source document.&lt;br&gt;
305which differ from one collection to another.&lt;br&gt;
306 &lt;br&gt;Once imported, each document is stored in its own&lt;br&gt;
307 &lt;br&gt;Each of the five buttons provides a different way to find&lt;br&gt;
308subdirectory of &lt;i&gt;archives&lt;/i&gt;, along with any associated&lt;br&gt;
309information. You can &lt;i&gt;search for particular words&lt;/i&gt; that&lt;br&gt;
310files—for example, images. To ensure compatibility with&lt;br&gt;
311appear in the text from the “search” page (or from the&lt;br&gt;
312Windows 3.0, only eight characters are used in directory&lt;br&gt;
313“about” page of Figure 1). This collection contains indexes&lt;br&gt;
314and file names, which causes annoying but essentially&lt;br&gt;
315of chapters, section titles, and entire books. The default&lt;br&gt;
316trivial complications.&lt;br&gt;
317search interface is a simple one, suitable for casual users;&lt;br&gt;advanced searching—which allows full Boolean&lt;br&gt;
318 &lt;br&gt;&lt;b&gt;Inside the documents&lt;/b&gt;&lt;br&gt;
319expressions, phrase searching, case and stemming&lt;br&gt;control—can be enabled from the &lt;i&gt;Preferences&lt;/i&gt; page.&lt;br&gt;
320 &lt;br&gt;The GML format imposes a limited amount of structure on&lt;br&gt;
321 &lt;br&gt;
322documents. Documents are divided into paragraphs. They&lt;br&gt;
323This collection has four browsable metadata indexes. You&lt;br&gt;
324can be split hierarchically into sections and subsections.&lt;br&gt;
325can &lt;i&gt;access publications by subject&lt;/i&gt; by clicking the &lt;i&gt;subjects&lt;/i&gt;&lt;br&gt;
326OIDs are extended to identify these components by&lt;br&gt;
327button, which brings up a list of subjects, represented by&lt;br&gt;
328appending numbers, separated by periods, to a document’s&lt;br&gt;
329bookshelves (Figure 2). You can &lt;i&gt;access publications by&lt;/i&gt;&lt;br&gt;
330OID. When a book is read, its section hierarchy is visible&lt;br&gt;
331&lt;i&gt;title&lt;/i&gt; by clicking &lt;i&gt;titles a-z&lt;/i&gt; (Figure 4), which brings up a list&lt;br&gt;
332as the table of contents (Figure 3). Chapters, sections,&lt;br&gt;
333of books in alphabetic order. You can &lt;i&gt;access publications&lt;/i&gt;&lt;br&gt;
334subsections, and pages are all implemented simply as&lt;br&gt;
335&lt;i&gt;by organization&lt;/i&gt; (i.e. Dublin Core “publisher”), bringing up&lt;br&gt;
336“sections” within the document. In some collections&lt;br&gt;
337a list of organizations. You can &lt;i&gt;access publications by&lt;/i&gt;&lt;br&gt;
338documents do not have a hierarchical subsection structure,&lt;br&gt;
339&lt;i&gt;“how to” listing&lt;/i&gt;, yielding a list of hints defined by the&lt;br&gt;
340but are split into pages to permit browsing within a&lt;br&gt;
341collection’s editors. We use the Dublin Core as a base and&lt;br&gt;
342retrieved document.&lt;br&gt;
343extend it in an &lt;i&gt;ad hoc&lt;/i&gt; manner to accommodate the&lt;br&gt;individual requirements of collection designers.&lt;br&gt;
344 &lt;br&gt;The document structure is used for searchable indexes.&lt;br&gt;There are three levels of index: &lt;i&gt;documents&lt;/i&gt;, &lt;i&gt;sections&lt;/i&gt;, and&lt;br&gt;
345&lt;hr&gt;
346&lt;A name=5&gt;&lt;/a&gt;&lt;IMG src=&quot;_httpdocimg_/pdf01-5_1.jpg&quot;&gt;&lt;br&gt;
347the &lt;i&gt;import&lt;/i&gt; process is invoked, which converts the files into&lt;br&gt;GML using the specified plugins. Old material for which&lt;br&gt;GML files have previously been created is not re-imported.&lt;br&gt;Then the &lt;i&gt;build&lt;/i&gt; process is invoked to build the requisite&lt;br&gt;indexes for the collection. Finally, the contents of the&lt;br&gt;&lt;i&gt;building&lt;/i&gt; directory are moved into the &lt;i&gt;index&lt;/i&gt; directory, and&lt;br&gt;the new version of the collection automatically becomes&lt;br&gt;live.&lt;br&gt;
348 &lt;br&gt;This procedure may seem cumbersome. But all the steps&lt;br&gt;are necessary for efficient operation with large collections.&lt;br&gt;The &lt;i&gt;import&lt;/i&gt; process could be performed on the fly during&lt;br&gt;the building operation—but because building indexes is a&lt;br&gt;multipass operation, the often lengthy importing would be&lt;br&gt;repeated several times. The &lt;i&gt;build&lt;/i&gt; process can take&lt;br&gt;considerable time—a day or two, for very large&lt;br&gt;collections. Consequently, the results are placed in the&lt;br&gt;&lt;i&gt;building&lt;/i&gt; directory so that, if the collection already exists, it&lt;br&gt;will continue to be served to users in its old form&lt;br&gt;throughout the building operation.&lt;br&gt;
349 &lt;br&gt;Active users of the collection will not be disturbed when&lt;br&gt;the new version becomes live—they will probably not&lt;br&gt;
350&lt;b&gt;Figure 4: Browsing titles in the HDL&lt;/b&gt;&lt;br&gt;
351even notice. The persistent OIDs ensure that interactions&lt;br&gt;remain coherent—users who are examining the results of a&lt;br&gt;query or browse operation will still retrieve the expected&lt;br&gt;
352&lt;i&gt;paragraphs&lt;/i&gt;, corresponding to the distinctions that GML&lt;br&gt;
353documents—and if a search is actually in progress when&lt;br&gt;
354makes—the hierarchical structure is flattened for the&lt;br&gt;
355the change takes place the program detects the resulting&lt;br&gt;
356purposes of creating these indexes. Indexes can be of text,&lt;br&gt;
357file-structure inconsistency and automatically and&lt;br&gt;
358or metadata, or any combination. Thus you can create a&lt;br&gt;
359transparently re-executes the query, this time on the new&lt;br&gt;
360searchable index of section titles, and/or authors, and/or&lt;br&gt;
361version of the collection.&lt;br&gt;
362document descriptions, as well as the document text.&lt;br&gt;
363&lt;b&gt;UPDATING EXISTING COLLECTIONS&lt;/b&gt;&lt;br&gt;
364 &lt;br&gt;&lt;b&gt;How it works&lt;/b&gt;&lt;br&gt;
365 &lt;br&gt;Updating an existing collection with new files in the same&lt;br&gt;
366 &lt;br&gt;The original material in the &lt;i&gt;import&lt;/i&gt; directory may be in any&lt;br&gt;
367format is easy. For example, the raw material for the HDL&lt;br&gt;
368format, and plugins are required to process each format&lt;br&gt;
369is supplied in the form of HTML files marked up with&lt;br&gt;
370type. The plugins that a collection uses must be specified&lt;br&gt;
371&amp;lt;&amp;lt;TOC&amp;gt;&amp;gt; tags to split books into sections and&lt;br&gt;
372in the collection configuration file. The &lt;i&gt;import&lt;/i&gt; program&lt;br&gt;
373subsections, and &amp;lt;&amp;lt;I&amp;gt;&amp;gt; tags to indicate where an image is&lt;br&gt;
374reads the list of plugins and passes each document to each&lt;br&gt;
375to be inserted. For each book in the library there is a&lt;br&gt;
376plugin in order until it finds one that can process it. When&lt;br&gt;
377directory that contains a single HTML file representing the&lt;br&gt;
378updating an existing collection, all plugins necessary to&lt;br&gt;
379book, and separate files containing the associated images.&lt;br&gt;
380process new material should already have been specified in&lt;br&gt;
381An accompanying spreadsheet file contains the&lt;br&gt;
382the configuration file.&lt;br&gt;
383classification hierarchy; this is converted to a simple file&lt;br&gt;format (using Excel’s &lt;i&gt;Save As&lt;/i&gt; command).&lt;br&gt;
384 &lt;br&gt;The building step creates the indexes for both searching&lt;br&gt;and browsing. The MG software is generally used to do the&lt;br&gt;
385 &lt;br&gt;Since the collection exists, its directory is already set up&lt;br&gt;
386searching (Witten  &lt;i&gt;et al.&lt;/i&gt;, 1999), and the &lt;i&gt;mgbuild&lt;/i&gt; module is&lt;br&gt;
387with subdirectories &lt;i&gt;import&lt;/i&gt;, &lt;i&gt;archives&lt;/i&gt;, &lt;i&gt;building&lt;/i&gt;, &lt;i&gt;index&lt;/i&gt;, and&lt;br&gt;
388automatically invoked to create each of the indexes that is&lt;br&gt;
389&lt;i&gt;etc&lt;/i&gt;, and the &lt;i&gt;etc&lt;/i&gt; directory will contain a suitable collection&lt;br&gt;
390required. For example, the Humanity Development Library&lt;br&gt;
391configuration file.&lt;br&gt;
392has three indexes, one for entire books, one for chapters,&lt;br&gt;and one for section titles. Subdirectories of the &lt;i&gt;index&lt;/i&gt;&lt;br&gt;
393 &lt;br&gt;
394directory are created for each of these indexes.&lt;br&gt;
395&lt;b&gt;The updating procedure&lt;/b&gt;&lt;br&gt;
396 &lt;br&gt;To update a collection, the new raw material is placed in&lt;br&gt;the &lt;i&gt;import&lt;/i&gt; directory, in whatever form it is available. Then&lt;br&gt;
397&lt;hr&gt;
398&lt;A name=6&gt;&lt;/a&gt;creator&lt;br&gt;
399davidb@cs.waikato.ac.nz&lt;br&gt;
4001&lt;br&gt;
401maintainer&lt;br&gt;
402davidb@cs.waikato.ac.nz&lt;br&gt;
4032&lt;br&gt;
404public&lt;br&gt;
405True&lt;br&gt;
4063&lt;br&gt;4&lt;br&gt;
407indexes&lt;br&gt;
408document:text&lt;br&gt;
4095&lt;br&gt;
410defaultindex&lt;br&gt;
411document:text&lt;br&gt;
4126&lt;br&gt;
413plugins&lt;br&gt;
414GMLPlug TEXTPlug ArcPlug RecPlug&lt;br&gt;
4157&lt;br&gt;8&lt;br&gt;
416classify&lt;br&gt;
417AZList metadata=Title&lt;br&gt;
4189&lt;br&gt;10&lt;br&gt;
419collectionmeta&lt;br&gt;
420collectionname    &amp;quot;generic text collection&amp;quot;&lt;br&gt;
42111&lt;br&gt;
422(a)&lt;br&gt;
423collectionmeta&lt;br&gt;
424.document:text    &amp;quot;documents&amp;quot;&lt;br&gt;
42512&lt;br&gt;
426creator&lt;br&gt;
427davidb@cs.waikato.ac.nz&lt;br&gt;
4281&lt;br&gt;
429maintainer&lt;br&gt;
430davidb@cs.waikato.ac.nz&lt;br&gt;
4312&lt;br&gt;
432public&lt;br&gt;
433True&lt;br&gt;
4343&lt;br&gt;4&lt;br&gt;
435indexes&lt;br&gt;
436document:text document:From&lt;br&gt;
4375&lt;br&gt;
438defaultindex&lt;br&gt;
439document:text&lt;br&gt;
4406&lt;br&gt;
441plugins&lt;br&gt;
442GMLPlug EMAILPlug ArcPlug RecPlug&lt;br&gt;
4437&lt;br&gt;8&lt;br&gt;
444classify&lt;br&gt;
445AZList metadata=Title&lt;br&gt;
4469&lt;br&gt;
447classify&lt;br&gt;
448DateList&lt;br&gt;
44910&lt;br&gt;11&lt;br&gt;
450collectionmeta&lt;br&gt;
451collectionname    &amp;quot;Email messages&amp;quot;&lt;br&gt;
45212&lt;br&gt;
453collectionmeta&lt;br&gt;
454.document:text    &amp;quot;documents&amp;quot;&lt;br&gt;
45513&lt;br&gt;
456collectionmeta&lt;br&gt;
457.document:From    &amp;quot;email senders&amp;quot;&lt;br&gt;
45814&lt;br&gt;15&lt;br&gt;
459format&lt;br&gt;
460QueryResults \\\\&lt;br&gt;
46116&lt;br&gt;
462(b)&lt;br&gt;
463&amp;lt;td&amp;gt;[link][icon][/link]&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;[Title]&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;[Author]&amp;lt;/td&amp;gt;&lt;br&gt;
46417&lt;br&gt;
465&lt;b&gt;Figure 5: Collection configuration files (a) generic, (b) for an email collection&lt;/b&gt;&lt;br&gt;
466 &lt;br&gt;MG also compresses the text of the collection; and the&lt;br&gt;
467certain circumstances, however, it might be preferable to&lt;br&gt;
468image files are linked into the &lt;i&gt;index&lt;/i&gt; subdirectory. Now&lt;br&gt;
469use a standardized format such as XML. This is&lt;br&gt;
470none of the material in the &lt;i&gt;import&lt;/i&gt; and &lt;i&gt;archives&lt;/i&gt; directories&lt;br&gt;
471straightforward to implementjust write an XML&lt;br&gt;
472is needed to run the collection and can be removed from&lt;br&gt;
473pluginalthough we have not done so ourselves. Given&lt;br&gt;
474the file system (though they would be needed if the&lt;br&gt;
475the transitory nature of the imported data, to date, we have&lt;br&gt;
476collection were rebuilt).&lt;br&gt;
477found GML a satisfactory and beneficial format.&lt;br&gt;
478 &lt;br&gt;Associated with each collection is a database stored in&lt;br&gt;
479&lt;b&gt;CREATING NEW COLLECTIONS&lt;/b&gt;&lt;br&gt;
480GDBM (Gnu database manager) format. This contains an&lt;br&gt;entry for each document, giving its OID, its internal MG&lt;br&gt;
481 &lt;br&gt;Building new collections from scratch is only slightly&lt;br&gt;
482document number, and metadata such as title. Information&lt;br&gt;
483different from updating an existing collection. The key&lt;br&gt;
484for each of the browsing indexes, which appear as buttons&lt;br&gt;
485new requirement is creating a collection configuration file,&lt;br&gt;
486on the Greenstone search/browse bar, is also extracted&lt;br&gt;
487and a software utility is provided to help. Two pieces of&lt;br&gt;
488during the building process and stored in the database. A&lt;br&gt;
489information are required for this: the name of the directory&lt;br&gt;
490“classifier” program is required for each browsing index to&lt;br&gt;
491that the collection will use (into which the source data and&lt;br&gt;
492extract the appropriate information from GML documents.&lt;br&gt;
493other files will eventually be placed), and a contact e-mail&lt;br&gt;
494Like plugins, classifiers are written on an &lt;i&gt;ad hoc&lt;/i&gt; basis for&lt;br&gt;
495address for use if any problems are encountered by the&lt;br&gt;
496the particular information required, and where possible&lt;br&gt;
497software once the collection is up and running. The utility&lt;br&gt;
498reused from one collection to another.&lt;br&gt;
499creates files and directories within the newly-named&lt;br&gt;
500 &lt;br&gt;
501directory to support a generic collection of plain text&lt;br&gt;
502The building program creates the indexes based on&lt;br&gt;
503documents. With suitable data placed in the &lt;i&gt;import&lt;/i&gt;&lt;br&gt;
504whatever appears in the &lt;i&gt;archives&lt;/i&gt; directory. The first plugin&lt;br&gt;
505directory, building the collection at this point will yield a&lt;br&gt;
506specified by all collections is one that processes GML&lt;br&gt;
507document-level searchable index of all the text and a&lt;br&gt;
508files, and so if &lt;i&gt;archives&lt;/i&gt; contains imported files they will be&lt;br&gt;
509browsable list of “titles” (defined in this case to be the&lt;br&gt;
510processed correctly. If it contains material in the original&lt;br&gt;
511document filenames).&lt;br&gt;
512format, that will be converted using the appropriate plugin.&lt;br&gt;Thus the import process is optional.&lt;br&gt;
513 &lt;br&gt;To enhance the functionality and presentation— something&lt;br&gt;
514 &lt;br&gt;
515anything but the most trivial collection will require—the&lt;br&gt;
516GML is designed to be fast and easy to parse, an important&lt;br&gt;
517configuration file must be edited. For a collection sourced&lt;br&gt;
518requirement when millions of documents are to be&lt;br&gt;
519from documents in an already supported data format,&lt;br&gt;
520processed. Something as simple as requiring tags to be&lt;br&gt;
521presented in a similar fashion to an existing collection, the&lt;br&gt;
522lower-case, for example, yields a substantial speed-up. In&lt;br&gt;
523&lt;hr&gt;
524&lt;A name=7&gt;&lt;/a&gt;&lt;IMG src=&quot;_httpdocimg_/pdf01-7_1.jpg&quot;&gt;&lt;br&gt;
525 &lt;br&gt;These are modules of code that can be slotted into the&lt;br&gt;system to enhance its capabilities. Plugins parse&lt;br&gt;documents, extracting the text and metadata to be indexed.&lt;br&gt;Classifiers control how metadata is brought together to&lt;br&gt;form browsable data structures. Both are specified in an&lt;br&gt;object-oriented framework using inheritance to minimize&lt;br&gt;the amount of code written.&lt;br&gt;
526 &lt;br&gt;A plugin must specify three things: what file formats it can&lt;br&gt;handle, how they should be parsed, and whether the plugin&lt;br&gt;is recursive. File formats are normally determined using&lt;br&gt;regular expression matching on the filename. For example,&lt;br&gt;the HTML plugin accepts all files that end in &lt;i&gt;.htm&lt;/i&gt;, . &lt;i&gt;html&lt;/i&gt;,&lt;br&gt;&lt;i&gt;.HTM&lt;/i&gt;, or &lt;i&gt;.HTML&lt;/i&gt;. (It is quite possible, however, to write&lt;br&gt;plugins that “look inside” the file as well.) For other files,&lt;br&gt;the plugin returns &lt;i&gt;undefined&lt;/i&gt; and the file is passed to the&lt;br&gt;next plugin in the collection’s configuration file (e.g.&lt;br&gt;Figure 5 line 7). If it can, the plugin parses the file and&lt;br&gt;returns the number of documents processed. This involves&lt;br&gt;extracting text and metadata and adding it to the library’s&lt;br&gt;content through calls to &lt;i&gt;add text&lt;/i&gt; and &lt;i&gt;add metadata&lt;/i&gt;.&lt;br&gt;
527 &lt;br&gt;Some plugins (“recursive” ones) add extra files into the&lt;br&gt;
528&lt;b&gt;Figure 6: Searching bookmarked Web pages&lt;/b&gt;&lt;br&gt;
529stream of data processed during the building phase by&lt;br&gt;artificially reactivating the list of plugins. This is how&lt;br&gt;directory hierarchies are traversed.&lt;br&gt;
530amount of editing is minimal. Importing new data formats&lt;br&gt;and browsing metadata in ways not currently supported are&lt;br&gt;
531 &lt;br&gt;Plugins are small modules of code that are easy to write.&lt;br&gt;
532more complex activities that require programming skills.&lt;br&gt;
533We monitored the time it took to develop a new one that&lt;br&gt;was different to any we had produced so far. We chose to&lt;br&gt;make as an example a collection of HTML bookmark files,&lt;br&gt;
534 &lt;br&gt;&lt;b&gt;Modifying the configuration file&lt;/b&gt;&lt;br&gt;
535the motivation being to produce a convenient way of&lt;br&gt;
536 &lt;br&gt;
537searching and browsing one’s bookmarked Web pages.&lt;br&gt;
538Figure 5b shows simple alterations to the generic&lt;br&gt;
539Figure 6 shows a user searching for bookmarked pages&lt;br&gt;
540configuration file in Figure 5a that was generated by the&lt;br&gt;
541about &lt;i&gt;music&lt;/i&gt;. The new plugin took under an hour to write,&lt;br&gt;
542new-collection utility. &lt;i&gt;TEXTPlug&lt;/i&gt; is replaced with&lt;br&gt;
543and was 160 lines long (ignoring blank lines and&lt;br&gt;
544&lt;i&gt;EMAILPlug&lt;/i&gt; (line 7) which reads email files and extracts&lt;br&gt;
545comments)—about the average length of existing plugins.&lt;br&gt;
546metadata (&lt;i&gt;From&lt;/i&gt;, &lt;i&gt;To&lt;/i&gt;, &lt;i&gt;Date&lt;/i&gt;, &lt;i&gt;Subject&lt;/i&gt;) from them. A classifier&lt;br&gt;for dates is added (line 10) to make the collection&lt;br&gt;
547 &lt;br&gt;Classifiers are more general than plugins because they&lt;br&gt;
548browsable chronologically. The default presentation of&lt;br&gt;
549work on GML-format data. For example, any plugin that&lt;br&gt;
550search results is overridden (line 17) to display both the&lt;br&gt;
551generates date metadata in accordance with the Dublin&lt;br&gt;
552title of the message (i.e. Dublin Core &lt;i&gt;Title&lt;/i&gt;) and its sender&lt;br&gt;
553core can request the collection to be browsable&lt;br&gt;
554(i.e. Dublin Core &lt;i&gt;Author&lt;/i&gt;). Elements in square brackets,&lt;br&gt;
555chronologically by specifying the &lt;i&gt;DateList&lt;/i&gt; classifier in the&lt;br&gt;
556such as &lt;i&gt;[Title]&lt;/i&gt;, are replaced by the metadata associated&lt;br&gt;
557collection’s configuration file (Figure 7). Classifiers are&lt;br&gt;
558with a particular document. The built-in term &lt;i&gt;[icon]&lt;/i&gt;&lt;br&gt;
559more elaborate than most plugins, but new ones are seldom&lt;br&gt;
560produces a suitable image that represents the document&lt;br&gt;
561required. The average length of existing classifiers is 230&lt;br&gt;
562(such as a book icon or page icon), and the &lt;i&gt;[link]
[/link]&lt;/i&gt;&lt;br&gt;
563lines.&lt;br&gt;
564construct forms a hyperlink to the complete document.&lt;br&gt;
565 &lt;br&gt;
566Anything else in the format statement, which in this case is&lt;br&gt;
567Classifiers must specify three things: an initialization&lt;br&gt;
568solely table-cell tags in HTML, is passed through to the&lt;br&gt;
569routine, how individual documents are classified, and the&lt;br&gt;
570page being displayed.&lt;br&gt;
571final browsable data structure. Initialization takes care of&lt;br&gt;any options specified in the configuration file (such as&lt;br&gt;
572As this example shows, creating a new collection that stays&lt;br&gt;
573&lt;i&gt;metadata=Title &lt;/i&gt;on line 9 of Figure 5b). Classifying&lt;br&gt;
574within the bounds of the library’s established capabilities&lt;br&gt;
575individual documents is an iterative process: for each one,&lt;br&gt;
576falls within the capability of many computer users—for&lt;br&gt;
577a call to &lt;i&gt;document-classify&lt;/i&gt; is made. On presentation of the&lt;br&gt;
578instance, computer-trained librarians. Extending&lt;br&gt;
579document’s OID, the necessary metadata is located and&lt;br&gt;
580Greenstone to handle new document formats and browse&lt;br&gt;
581used to control where the document is added to the&lt;br&gt;
582metadata in new ways is more challenging.&lt;br&gt;
583browsable data structure being constructed.&lt;br&gt;
584 &lt;br&gt;Once all documents have been added, a request is made for&lt;br&gt;
585 &lt;br&gt;&lt;b&gt;Writing new plugins and classifiers&lt;/b&gt;&lt;br&gt;
586the completed data structure. Some classifiers return the&lt;br&gt;data structure directly; others transform the data structure&lt;br&gt;
587 &lt;br&gt;Extensibility  is  obtained through  plugins  and  classifiers.&lt;br&gt;
588before it is returned. For example, the &lt;i&gt;AZList&lt;/i&gt; classifier&lt;br&gt;
589&lt;hr&gt;
590&lt;A name=8&gt;&lt;/a&gt;&lt;IMG src=&quot;_httpdocimg_/pdf01-8_1.jpg&quot;&gt;&lt;br&gt;
591a page number, next and previous page buttons, and&lt;br&gt;displaying a particular page at different resolutions. A text&lt;br&gt;version of the page is also available upon which a&lt;br&gt;searching option is also provided.&lt;br&gt;
592Started in 1994, Harvest is also a long-running research&lt;br&gt;project. It provides an efficient means of gathering source&lt;br&gt;data from the Internet and distributing indexing&lt;br&gt;information over the Internet. This is accomplished&lt;br&gt;through five components: &lt;i&gt;gatherer&lt;/i&gt;, &lt;i&gt;broker&lt;/i&gt;, &lt;i&gt;indexer&lt;/i&gt;,&lt;br&gt;&lt;i&gt;replicator&lt;/i&gt; and &lt;i&gt;cache&lt;/i&gt;. The first three are central to creating,&lt;br&gt;updating and searching a collection; the last two help to&lt;br&gt;improve performance over the Internet through transparent&lt;br&gt;mirroring and caching techniques.&lt;br&gt;
593The system is configurable and customizable. While&lt;br&gt;searching is most commonly implemented using Glimpse&lt;br&gt;(&lt;i&gt;glimpse.cs.arizona.edu&lt;/i&gt;), in principle any search engine&lt;br&gt;that supports incremental updates and Boolean&lt;br&gt;combinations of attribute-based queries can be used. It is&lt;br&gt;possible to control what type of documents are gathered&lt;br&gt;during creation and updating, and how the query interface&lt;br&gt;
594&lt;b&gt;Figure 7: Browsing a newspaper collection by date&lt;/b&gt;&lt;br&gt;
595looks and is laid out.&lt;br&gt;
596Sample collections cited by the developers include 21,000&lt;br&gt;
597divides the alphabetically sorted list of metadata into&lt;br&gt;
598computer science technical reports and 7,000 home pages.&lt;br&gt;
599separate pages of about the same size and returns the&lt;br&gt;
600Other examples include a sizable collection of agriculture-&lt;br&gt;
601alphabetic ranges for each one (Figure 4).&lt;br&gt;
602related electronic journals and magazines called “tomato-&lt;br&gt;juice” (accessed through &lt;i&gt;hegel.lib.ncsu.edu&lt;/i&gt;) and a full-text&lt;br&gt;
603&lt;b&gt;OVERVIEW OF RELATED WORK&lt;/b&gt;&lt;br&gt;
604index of library-related electronic serials&lt;br&gt;
605Two projects that provide substantial open source digital&lt;br&gt;
606(&lt;i&gt;sunsite.berkeley.edu/IndexMorganagus&lt;/i&gt;). Harvest is also&lt;br&gt;
607library software are Dienst (Lagoze and Fielding, 1998)&lt;br&gt;
608often used to index Web sites (for example&lt;br&gt;
609and Harvest (Bowman &lt;i&gt;et al.&lt;/i&gt;, 1994). The origins of Dienst&lt;br&gt;
610&lt;i&gt;www.middlebury.edu&lt;/i&gt;).&lt;br&gt;
611(&lt;i&gt;www.cs.cornell.edu/cdlrg&lt;/i&gt;) stretch back to 1992. The term&lt;br&gt;
612Comparing Greenstone with Dienst and Harvest, there are&lt;br&gt;
613has come to represent three entities: a conceptual&lt;br&gt;
614both similarities and differences. All provide substantial&lt;br&gt;
615architecture for distributed digital libraries; an open&lt;br&gt;
616digital library systems, hence common themes recur, but&lt;br&gt;
617protocol for service communication; and a software&lt;br&gt;
618they are driven by projects with different aims. Harvest,&lt;br&gt;
619system that implements the protocol. To date, five sample&lt;br&gt;
620for instance, was not conceived as a digital library project&lt;br&gt;
621digital libraries have been built using this technology.&lt;br&gt;
622at all, but by virtue of its selective document gathering&lt;br&gt;
623They manifest themselves in two forms: technical reports&lt;br&gt;
624process it can be classed (and is used) as one. While it&lt;br&gt;
625and primary source documents.&lt;br&gt;
626provides sophisticated search options, it lacks the&lt;br&gt;
627Best known is NCSTRL, the Networked Computer&lt;br&gt;
628complementary service of browsing. Furthermore it adds&lt;br&gt;
629Science Technical Reference Library project&lt;br&gt;
630no structure or order to the documents collected, relying&lt;br&gt;
631(&lt;i&gt;www.ncstrl.org&lt;/i&gt;). This collection facilitates searching by&lt;br&gt;
632on whatever structures are present in the site that they&lt;br&gt;
633title, author and abstract, and browsing by year and author,&lt;br&gt;
634were gathered from. A proven strength of the design is its&lt;br&gt;
635across a distributed network of document repositories.&lt;br&gt;
636flexibility through configuration and customizationan&lt;br&gt;
637Documents can (where supported) be delivered in various&lt;br&gt;
638element also present in Greenstone.&lt;br&gt;
639formats such as PostScript, a thumbnail overview of the&lt;br&gt;
640Dienstbest exemplified through the NCSTRL&lt;br&gt;
641pages, and a GIF image of a particular page.&lt;br&gt;
642worksupports searching and browsing, like Greenstone.&lt;br&gt;
643The &lt;i&gt;Making of America&lt;/i&gt; resource is an example of a&lt;br&gt;
644Both use open protocols. Differences include a high&lt;br&gt;
645collection based around primary sourcesin this case&lt;br&gt;
646reliance in Dienst on user-supplied information when a&lt;br&gt;
647American social history, 1830−1900. It has a different&lt;br&gt;
648document is added, and a smaller range of document types&lt;br&gt;
649“look and feel” to NCSTRL, being strongly oriented&lt;br&gt;
650supported—although Dienst does include a document&lt;br&gt;
651toward browsing rather than searching. A user navigates&lt;br&gt;
652model that should, over time, allow this to expand with&lt;br&gt;
653their way through a hierarchical structure of hyperlinks to&lt;br&gt;
654relative ease.&lt;br&gt;
655reach a book of interest. The book itself is a series of&lt;br&gt;
656There are also commercial systems that provide similar&lt;br&gt;
657scanned images: delivery options include going directly to&lt;br&gt;
658digital library services to those described. However, since&lt;br&gt;
659&lt;hr&gt;
660&lt;A name=9&gt;&lt;/a&gt;corporate culture instills proprietary attitudes there is little&lt;br&gt;
661&lt;b&gt;REFERENCES&lt;/b&gt;&lt;br&gt;
662opportunity for advancement through a shared&lt;br&gt;
6631.  Akscyn, R.M. and Witten, I.H. (1998) “Report on First&lt;br&gt;
664collaborative effort. Consequently they are not reviewed&lt;br&gt;
665Summit on International Cooperation on Digital&lt;br&gt;
666here.&lt;br&gt;
667Libraries.” ks.com/idla-wp-oct98.&lt;br&gt;
6682.  Bowman, C.M., Danzig, P.B., Manber, U., and&lt;br&gt;
669&lt;b&gt;CONCLUSIONS&lt;/b&gt;&lt;br&gt;
670Schwartz, M.F. “Scalable Internet resource discovery:&lt;br&gt;
671Greenstone is a comprehensive software system for&lt;br&gt;
672Research problems and approaches” &lt;i&gt;Communications&lt;/i&gt;&lt;br&gt;
673creating digital library collections. It builds data structures&lt;br&gt;
674&lt;i&gt;of the ACM,&lt;/i&gt; Vol. 37, No. 8, pp. 98−107, 1994.&lt;br&gt;
675for searching and browsing from the material provided,&lt;br&gt;
6763.  Fox, E. (1998) “Digital library definitions.”&lt;br&gt;
677rather than relying on any hand-crafting. The process is&lt;br&gt;
678ei.cs.vt.edu/~fox/dlib/def.html.&lt;br&gt;
679controlled by a configuration file, and once a collection&lt;br&gt;exists new material can be added completely&lt;br&gt;
6804.  Humanity Libraries (1998) &lt;i&gt;Humanity Development&lt;/i&gt;&lt;br&gt;
681automatically. Browsing is based on Dublin Core&lt;br&gt;
682&lt;i&gt;Library&lt;/i&gt;. CD-ROM produced by the Global Help&lt;br&gt;
683metadata.&lt;br&gt;
684Project, Antwerp, Belgium.&lt;br&gt;
685New collections can be developed easily, particularly if&lt;br&gt;
6865.  Lagoze, C. and Fielding, D “Defining Collections in&lt;br&gt;
687they resemble existing ones. Extensibility is achieved&lt;br&gt;
688Distributed Digital Libraries” &lt;i&gt;D-Lib Magazine&lt;/i&gt;, Nov.&lt;br&gt;
689through software “plugins” that can be written to&lt;br&gt;
6901998.&lt;br&gt;
691accommodate documents, and metadata, in different&lt;br&gt;
6926.  PAHO (1999) &lt;i&gt;Virtual Disaster Library&lt;/i&gt;. CD-ROM&lt;br&gt;
693formats. Standard plugins exist for many document types;&lt;br&gt;
694produced by the Pan-American Health Organization,&lt;br&gt;
695new ones are easily written. Browsing is controlled by&lt;br&gt;
696Washington DC, USA.&lt;br&gt;
697“classifiers” that process metadata into browsing structures&lt;br&gt;
6987.  McNab, R.J., Witten, I.H. and Boddie, S.J. (1998) “A&lt;br&gt;
699(by date, alphabetical, hierarchical, etc).&lt;br&gt;
700distributed digital library architecture incorporating&lt;br&gt;
701However, the most powerful support for extensibility is&lt;br&gt;
702different index styles.” &lt;i&gt;Proc IEEE Advances in Digital&lt;/i&gt;&lt;br&gt;
703achieved not by technical means but by making the source&lt;br&gt;
704&lt;i&gt;Libraries&lt;/i&gt;, Santa Barbara, CA, pp. 36–45.&lt;br&gt;
705code freely available under the Gnu public license. Only&lt;br&gt;
7068.  Nevill-Manning, C.G., Reed, T., and Witten, I.H.&lt;br&gt;
707through an international cooperative effort will digital&lt;br&gt;
708(1998) “Extracting text from PostScript”&lt;br&gt;
709library software become sufficiently comprehensive to&lt;br&gt;
710&lt;i&gt;Software—Practice and Experience&lt;/i&gt;, Vol. 28, No. 5, pp.&lt;br&gt;
711meet the world’s needs with the richness and flexibility&lt;br&gt;
712481–491; April.&lt;br&gt;
713that users deserve.&lt;br&gt;
7149.  UNESCO (1999) &lt;i&gt;SAHEL point DOC: Anthologie du&lt;/i&gt;&lt;br&gt;
715&lt;b&gt;ACKNOWLEDGMENTS&lt;/b&gt;&lt;br&gt;
716&lt;i&gt;développement au Sahel&lt;/i&gt;. CD-ROM produced by&lt;br&gt;UNESCO, Paris, France.&lt;br&gt;
717We gratefully acknowledge all those who have worked on&lt;br&gt;the Greenstone software, and all members of the New&lt;br&gt;
71810. UNU (1998) &lt;i&gt;Collection on critical global issues.&lt;/i&gt; CD-&lt;br&gt;
719Zealand Digital Library project for their enthusiasm and&lt;br&gt;
720ROM produced by the United Nations University&lt;br&gt;
721ideas.&lt;br&gt;
722Press, Tokyo, Japan.&lt;br&gt;
72311. Witten, I.H., Moffat, A. and Bell, T. (1999) &lt;i&gt;Managing&lt;/i&gt;&lt;br&gt;
724&lt;i&gt;Gigabytes: compressing and indexing documents and&lt;br&gt;images&lt;/i&gt;, Morgan Kaufmann, second edition.&lt;br&gt;
725&lt;hr&gt;
726
727
728</Content>
729</Section>
730</Archive>
Note: See TracBrowser for help on using the browser.