source: trunk/greenorg/docs/compiling.html@ 10030

Last change on this file since 10030 was 10030, checked in by kjdon, 19 years ago

added some stuff about 64 bit processors

  • Property svn:mime-type set to application/octet-stream
File size: 16.1 KB
Line 
1<html>
2<head>
3<title>Compiling Greenstone</title>
4<h1>Compiling Greenstone</h1>
5<h3>Greenstone Software, New Zealand Digital Library Project</h3>
6</head>
7
8<body bgcolor="white">
9<p>These notes were written for gsdl-2.39, but probably apply to other
10versions as well. You might find a more up-to-date version on-line at
11<a href="http://www.greenstone.org/docs/compiling.html"
12>http://www.greenstone.org/docs/compiling.html</a>.
13
14Greenstone version 2.39 was released on the 10th March, 2003.
15
16</p>
17
18<h3>Table of contents:</h3>
19<ul>
20<li><a href="#platforms">Tested Platforms</a></li>
21<li><a href="#unix">Generic Unix Notes</a></li>
22<li>Specific Unix Notes
23 <ul>
24 <li><a href="#cygwin">Cygwin Notes</a></li>
25 <li><a href="#darwin">Darwin Mac OS X Notes</a></li>
26 <li><a href="#linux">GNU/Linux Notes</a></li>
27 <li><a href="#freebsd">FreeBSD Notes</a></li>
28 <li><a href="#solaris">Solaris Notes</a></li>
29 </ul>
30</li>
31<li><a href="#visualc">MS Visual C++ Notes</a></li>
32<li><a href="#64bit">64 bit Notes</a></li>
33<li><a href="#contact">Our Contact Details</a></li>
34</ul>
35
36
37<a name="platforms"><h1>Platforms</h1></a>
38We have tested Greenstone on the following platforms:
39<ul>
40<li>GNU/Linux:
41 <ul>
42 <li>Debian 3.0 (potato) (i386), gcc 2.95.4</li>
43 <li>Debian sid/unstable (i386), gcc 3.2</li>
44 <li>Debian 2.2 (potato) (i386 and ppc rs/6000) (gsdl 2.38)</li>
45 <li>Red Hat 7.3 (i386), gcc 2.96</li>
46 <li>Slackware 8.0.0 (i386), gcc 2.95.3</li>
47 <li>Slackware 7.1.0 (i386) (gsdl 2.38)</li>
48 </ul>
49</li>
50<li>Solaris 2.8 (sparc), gcc 2.95.2 and gmake.</li>
51<li>Solaris 2.6 (sparc) using gcc and gmake. (gsdl 2.38)</li>
52<li>FreeBSD 4.2 (i386) (gsdl 2.38)</li>
53<li>Cygwin (minor fiddling needed)</li>
54<li>Darwin / Mac OS X (G4/ppc7400)</li>
55<li>Microsoft Visual C++ 4.2 and 6.0</li>
56</ul>
57If you would like to add other platforms to this list, or inform us of
58any portability changes required, send mail to
59<a href="mailto:[email protected]">[email protected]</a>.
60
61
62
63
64<a name="unix"><h1>Unix Compilation notes</h1></a>
65<p>(we include cygwin here).</p>
66
67The "standard" commands of
68<pre>
69$ ./configure
70$ make all
71$ make install
72</pre>
73should (hopefully) be all that is required. Currently, Greenstone does
74not honour the <tt>`--prefix'</tt> flag, but the directories are
75self-contained, so should be able to be moved after installation.
76
77<p>You will probably need to use GNU make.</p>
78
79<p>Version 2.38 of Greenstone compiles with gcc version 3.0 (as well
80as earlier versions of gcc). Earlier versions of Greenstone will
81require earlier versions of the compiler -- we have successfully used
82versions egcs-2.91.66 and gcc-2.95.
83</p>
84<p>The Greenstone Librarian Interface (GLI) code is written in Java, and
85compiling it requires a suitable
86version of the Java Software Development Kit (version 1.4.0 or newer).
87To compile this source code, run <tt>makegli.sh</tt> from the
88<tt>gsdl/gli</tt> directory.
89</p>
90<p>
91<hr>
92<strong>Note!</strong> - version 2.41 of Greenstone (released in December 2003)
93requires slight modification to compile cleanly with gcc version 3.x.
94The "Isis" package in greenstone's "packages" directory requires 3 files
95to be changed.
96You can download the 3 files from:<br>
97 <a href="http://www.greenstone.org/tmp/isis-gdl-fixes.zip">
98http://www.greenstone.org/tmp/isis-gdl-fixes.zip</a><br>
99
100which contains:
101<pre>
102 isis-gdl/CRC32.cpp
103 isis-gdl/IsisTypes.h
104 isis-gdl/Master.cpp
105</pre>
106
107If you copy these over the 3 files with the same name in gsdl/packages/isis-gdl
108then it should all compile ok with gcc 3.
109<hr>
110</p>
111
112<p>The GDBM library and headers are needed - Linux distributions typically
113come with the library, but may or may not come standard with the header file.
114Darwin does not come with either. If it is not installed in
115<tt>/usr</tt> or <tt>/usr/local</tt> then you will have to specify where it
116is by using the configure flag <tt>--with-gdbm=/path/to/gdbm/dir</tt>.</p>
117
118The GDBM library can be downloaded from
119 ftp://ftp.gnu.org/pub/gnu/gdbm/gdbm-1.8.3.tar.gz or a closer mirror.
120
121<p>The following configure flags add extra functionality to Greenstone:
122<table>
123<tr valign=top><td>--enable-corba</td>
124 <td>Creates a CORBA server as well as the .cgi server. This is currently
125 still developmental, and compilation hasn't been tested on many
126 platforms. A java CORBA client is available through anonymous CVS
127 (password <tt>anonymous</tt>) -
128 <pre>
129export CVSROOT=:pserver:[email protected]:2402/usr/local/global-cvs/gsdl-src
130cvs login
131cvs export -r HEAD java-client
132</pre>
133</td></tr>
134<tr valign=top><td>--with-micodir</td>
135 <td>Use an existing MICO compiler for the CORBA server instead of
136 compiling our included version.</td></tr>
137<tr valign=top><td>--enable-z3950</td>
138 <td>Enable rudimentary Z39.50 client support in the .cgi server.</td></tr>
139</table>
140</p>
141
142
143
144<h1>Platform-Specific Issues</h1>
145
146<a name="cygwin"><h2>Cygwin</h2></a>
147Greenstone does not currently compile under cygwin "out-of-the-box".
148We had to manually edit some of the Makefiles. More specifically,
149the packages/mg subtree needed "-ansi" in the CFLAGS, while some parts of the
150src/mgpp subtree fails with "-ansi", as their version of the standard
151header files don't include any (eg) POSIX or XPG/OPEN functions that
152aren't ANSI if -ansi is supplied, even if flags like "_XOPEN_SOURCE"
153are defined.
154
155<p>Also some of the third-party packages required some manual attention.</p>
156
157<p>Make sure you have the gdbm package installed.</p>
158
159
160<a name="darwin"><h2>Darwin/Mac OS X</h2></a>
161<p>All compilation is currently done through a terminal.</p>
162
163<p>Note that the default filesystem is case-insensitive.<p>
164
165<p>Mac OS X uses a compiler based on gcc version 3, so read the above
166section on <a href="#unix">Unix Compilation Notes</a> for changes required
167to the source code for greenstone version 2.41.</p>
168<p>darwin doesn't come with gdbm, so:
169<ol>
170<li>Download the gdbm source code from ftp.gnu.org into your home folder, as
171mentioned previously.</li>
172<li>Unpack it using the command <tt>"tar -zxf gdbm-1.8.3.tar.gz"</tt> (without quotes)</li>
173<li>If you used the older gdbm-1.8.0 (instead of 1.8.3), you will need
174 to update some files that can't figure out what the system type is. Eg:
175 <tt>"cp&nbsp;/usr/libexec/config.sub&nbsp;/usr/libexec/config.guess&nbsp;gdbm-1.8.3"</tt></li>
176<li>Make (and install) the library:<br>
177 <tt>cd gdbm-1.8.3 && make all install</tt><br>
178 (When we did this, we did not have permission to install in a system
179 directory)</li>
180</ol>
181
182If the "make install" command fails to install libgdbm into the
183default "/usr/local" folder due to permissions:
184<ol>
185<li>Remove the dynamic libraries, so that the Greenstone files are
186only linked using the "static" gdbm libraries using the following
187command (without the quotes):<br>
188<tt>"rm ~/gdbm-1.8.3/.libs/*.dylib"</tt></li>
189<li>when you configure Greenstone you should add
190<tt>--with-gdbm=/Users/&lt;username&gt;/gdbm-1.8.3</tt> (or wherever you
191installed it to) to the command. Note that this means that the compile
192option in the CD-ROM distribution Install script will fail, since it will
193only look in the default locations (/lib, /usr/lib/ and /usr/local/lib)
194</li>
195</ol>
196
197Also see the
198<a href="http://www.greenstone.org/docs/Install-Notes-osx.html">OSX
199Install Notes</a> page.
200</p>
201
202
203
204
205<h3>Issues</h3>
206<p>
207<ul><li>Due to "upgrades" of the config.guess and config.sub files,
208 version 2.37 of Greenstone might not recognise Mac OS X during the
209 configure. You can either overwrite these files (as above) from
210 /usr/libexec, or try to fake it by adding
211 <tt>--host=powerpc-apple-machten</tt> (or similar) to the configure
212 line. This has been resolved in gsdl-2.38.</li></ul>
213</p>
214<p>
215<ul><li>If the make fails when compiling a file called
216"display.cpp", you need to work around a compiler bug. This only seems
217to occur when "-O2" is part of the compiler flags, so changing this to
218"-O1" or removing it from the flags will work. This seems to have been
219fixed with version 10.1 and later of the developer tools.</li></ul>
220</p>
221<p>
222<ul><li> "There is a known bug in the version of gcc shipped with
223MacOS X 10.2." Version 2.38 of greenstone fails to build the pdftohtml
224converter, with the error message
225<pre>FontFile.h:27: storage size of `_ZTI8FontFile' isn't known
226FontFile.h:46: storage size of `_ZTI13Type1FontFile' isn't known
227FontFile.h:67: storage size of `_ZTI14Type1CFontFile' isn't known
228FontFile.h:144: storage size of `_ZTI16TrueTypeFontFile' isn't known
229</pre>
230The work-around is to remove all lines with "#pragma" from the
231source-code of the pdftohtml package. See <a
232href="http://sourceforge.net/tracker/index.php?func=detail&aid=631593&group_id=45839&atid=444239">the
233bug report</a> on pdftohtml's site. This will be fixed in Greenstone
234version 2.39 and later.</li></ul>
235</p>
236
237
238<a name="linux"><h2>GNU/Linux</h2></a>
239<p>As mentioned above, version 2.38 of Greenstone compiles with gcc3.
240If you are using an earlier version of Greenstone, you will need to
241make sure you are using an older version of gcc, and not gcc2.96 or
242gcc3. Red Hat >= 7.0 comes with the newer versions of gcc by default.
243</p>
244
245<p>You will need the gdbm.h header file. If you don't already have it, it is in
246the libgdbmg1-dev (debian) or gdbm-devel-1.8.0 (rpm) package.</p>
247
248<p>We have had a couple of reports from SuSE users that one of our third-party
249packages (wget) fails to configure as it uses GNU msgfmt for translation
250catalogues - this is resolved by installing the gettext package. This seems
251to be already installed on other distributions.</p>
252
253<p><strong>Alpha architectures</strong>:<br>
254Well, the good news is that it compiles OK (for versions of gsdl &gt;= 2.37
255- earlier versions need updated config.sub files),
256the bad news is that it fails to build collections. One possibility is that
257mg (the backend code) doesn't like the 64-bit ints.
258</p>
259
260
261
262<a name="freebsd"><h2>FreeBSD</h2></a>
263Everything should go smoothly... you might need to install lib gdbm if it
264is not already installed.
265
266
267
268<a name="solaris"><h2>Solaris</h2></a>
269<p>We no longer have solaris machines running in our department.
270However, Greenstone built and ran the last time I could log on (about version
2712.33?), and we have reports of people getting current versions working,
272with minor changes.</p>
273
274<p>Greenstone includes a perl module from CPAN, XML::Parser (+Expat),
275which has caused problems on some machines during the make. This is
276because perl uses it's own config file to get compiler settings,
277etc. We think we have worked around this. If you get compilation problems,
278you can do one of the following:
279<ol>
280<li> After doing the toplevel ./configure, edit
281gsdl/packages/cpan/XML-Parser-2.27/Makefile and XML-Parser-2.27/Expat
282and change <tt>CC = cc</tt> to <tt>CC = gcc</tt>. (Assuming you are
283using gcc). <strong>Or:</strong></li>
284
285<li> You could install the perl XML::Parser module manually, and
286comment out or remove any mention of the cpan/XML-Parser-2.27
287directory from gsdl/packages/Makefile(.in). </li>
288</ol>
289
290</p>
291
292<p>You might need to manually install the gdbm library, and you probably
293need to use GNU's make. try setting the MAKE variable when
294you run the configure script, such as:<br>
295<tt>$ MAKE=gmake ./configure [options]</tt><br>
296or
297<pre>
298$ ./configure [options]
299...
300$ gmake all
301</pre>
302</p>
303<a name="visualc"><h2>Windows (Visual C++)</h2></a>
304<p>We have tested with versions 4.2 and 6.0. Our distribution includes a
305cut-down port of the gdbm library for windows.</p>
306
307<p>We compile Greenstone from the Windows command prompt.
308You must manually unzip files in the <tt>gsdl\packages\crypt</tt> and
309<tt>gsdl\packages\gdbm</tt> directories.
310After setting up the VC++ environment, simply type <tt>nmake /f win32.mak</tt>
311and everything should compile OK. For us, setting up the environment requires
312running the <tt>vcvars32.bat</tt> script, found in the VC++ installation
313directory (for us this is in
314<tt>C:\Program Files\Microsoft Visual Studio\VC98\bin</tt>.)</p>
315
316<p>The third-party packages (pdftohtml, wvWare, rtftohtml, xlhtml, etc) were
317compiled using cygwin.</p>
318
319<p>The Greenstone Librarian Interface (GLI) code is written in Java, and
320compiling it requires a suitable
321version of the Java Software Development Kit (version 1.4.0 or newer).
322To compile this source code, run <tt>makegli.bat</tt> from the
323<tt>gsdl\gli</tt> directory.
324
325<a name="64bit"><h2>64 bit processors</h2></a>
326<p>John has tried once to compile Greenstone on a 64 bit processor (Opteron, Amd64) running 64 Ubuntu. He reported several problems which are listed below. These have yet to be resolved.
327
328<p>1. When trying to build anything using mg (i.e. mg_passes) I'm suffering a seg fault somewhere deep in the malloc code (mallopt). From what I can tell the mg stuff uses some library wrapper (gsdl/packages/mg/lib/memlib) which has some prototypes which override the stdlib.h ones. If I let them do this, I believe malloc might be using 32 bit ints to hold pointers (which are 64 bit under X86_64 architecture). If I prevent memlib overriding by setting the STD_MEM precompiler flag I get a bazillion warnings - and it still seg faults in the same place (again implying a pointer != int problem).
329
330<p>Below is a dump of the error messages I encountered when trying to debug the malloc problem.
331
332<pre>
333(gdb) file /var/www/projects/john/gsdl/bin/linux/mg_passes
334
335(gdb) run -D
336-f /var/www/projects/john/gsdl/collect/demo/building/text/demo -b 12000
337-T1 -M 4 -d / < /tmp/output.txt
338
339(gdb) backtrace
340#0 0x0000002a95c7bab6 in mallopt () from /lib/libc.so.6
341#1 0x0000002a95c7aa63 in malloc () from /lib/libc.so.6
342#2 0x0000000000402bf5 in process_text_1 ()
343#3 0x0000000000401fb1 in driver (in_fd=0, Trace=0x0,
344file_name=0x7fbffffb5d
345"/var/www/projects/john/gsdl/collect/demo/building/text/demo") at
346mg_passes.c:336
347#4 0x0000000000402617 in main (argc=11, argv=0x7fbffff918) at
348mg_passes.c:626
349</pre>
350
351<p>2. When trying to search a collection which was imported/built using mgpp I get this cool error message:
352<br/><b>Couldn't load index information for "idx"</b>
353<br/>and no results. Strangely enough browsing works just fine. I tracked the problem back to the loading of the index files in (gsdl/src/mgpp/text/IndexData.c) and in particular these lines
354
355<pre>
356 // blocked dictionary
357 ...
358 fseek (dictFile, bdh.wblk_idx_start, SEEK_SET);
359 if (!ReadBlockIdx (dictFile, biWords)) { UnloadData (); return false; }
360 ...
361</pre>
362This always returns false. So the fseek to the start of the dictionary is failing. Weird eh? Any ideas? Currently I suspect a bogus pointer to int cast somewhere in the mgpp code during import/building is resulting in an equally bogus index file.
363
364<p>3. So... believing I didn't have quite enough problems to sort out I attempted to fix the second problem and thus created another wonderful problem. Whenever I browse to my custom collection I get the "Opps! An error occurred..." page - but with no error message. However if I run the binary from the cgi-bin it works just peachy. It sounds like a permissions problem - but it isn't. I've triple checked all the permissions. Running the plain-jane greenstone library, then going to the custom collection works just fine (bar the searching problem outlined above).
365
366<a name="contact"><h1>Contact Us</h1></a>
367<p>If Greenstone doesn't work for you, please let us know! If you add
368information such as operating system and software tool versions to your
369request then it will help us to fix the problem.</p>
370
371<p>If you have any updates or corrections to this documentation, then
372don't hesitate to send them to us.</p>
373
374<p>
375Send mail to us at:
376<a href="mailto:greenstone&#64;cs.waikato.ac.nz">greenstone&#64;cs.waikato.ac.nz</a>.
377</p>
378
379<p>We also have 2 mailing lists, one aimed more
380at users and the other aimed more for developmental issues. You can
381subscribe by going to
382<a href="https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users"
383>https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-users</a> and/or
384<a href="https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-devel"
385>https://list.scms.waikato.ac.nz/mailman/listinfo/greenstone-devel</a>.
386Archives are available (using Greenstone of course!) at <a
387href="http://www.nzdl.org/cgi-bin/library?c=gsarch&p=about">http://www.nzdl.org/cgi-bin/library?c=gsarch&p=about</a>.
388</p>
389
390<address>Last modified 20 April 2004 by <a href="http://www.cs.waikato.ac.nz/~jrm21/">John McPherson</a></address>
391</body>
392</html>
393
Note: See TracBrowser for help on using the repository browser.