Index: /trunk/indexers/mgpp/COPYING
===================================================================
--- /trunk/indexers/mgpp/COPYING	(revision 3365)
+++ /trunk/indexers/mgpp/COPYING	(revision 3365)
@@ -0,0 +1,344 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+                          675 Mass Ave, Cambridge, MA 02139, USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) 19yy  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) 19yy name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Library General
+Public License instead of this License.
Index: /trunk/indexers/mgpp/Makefile.in
===================================================================
--- /trunk/indexers/mgpp/Makefile.in	(revision 3365)
+++ /trunk/indexers/mgpp/Makefile.in	(revision 3365)
@@ -0,0 +1,111 @@
+# Main Makefile for GNU tar.
+# Copyright (C) 1994, 1995 Free Software Foundation, Inc.
+ 
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+ 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+ 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ 
+PACKAGE = @PACKAGE@
+VERSION = @VERSION@
+ 
+@SET_MAKE@
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+srcdir = @srcdir@
+ 
+
+CC = @CC@
+CPP = @CXX@
+CFLAGS = @CFLAGS@ -ansi
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS = @LIBS@  
+TOUCH = echo timestamp >
+ 
+bindir = $(exec_prefix)/bin
+infodir = $(prefix)/info
+libexecdir = $(exec_prefix)/libexec
+ 
+MDEFINES = 
+#CC='$(CC)' CFLAGS='$(CFLAGS)' LDFLAGS='$(LDFLAGS)' LIBS='$(LIBS)' \
+#prefix='$(prefix)' exec_prefix='$(exec_prefix)' \
+#bindir='$(bindir)' libexecdir='$(libexecdir)' \
+#infodir='$(infodir)' infodir='$(infodir)' libexecdir='$(libexecdir)'
+
+SUBDIRS = lib text java/org/greenstone/mgpp jni
+
+
+.SUFFIXES:
+
+all install uninstall execinstall execuninstall maninstall manuninstall:
+#config.h
+	for subdir in $(SUBDIRS); do \
+	  echo making $@ in $$subdir; \
+	  (cd $$subdir && $(MAKE) $(MDEFINES) $@) || exit 1; \
+	done
+
+# Have not written any texinfo files for the documentation yet
+#info dvi:
+#	cd doc && $(MAKE) $@
+
+mostlyclean: mostlyclean-recursive mostlyclean-local
+
+clean: clean-recursive clean-local
+
+distclean: distclean-recursive distclean-local
+	rm config.status
+
+maintainer-clean: maintainer-clean-recursive maintainer-clean-local
+	rm config.status
+
+mostlyclean-recursive clean-recursive distclean-recursive \
+maintainer-clean-recursive:
+	for subdir in $(SUBDIRS); do \
+	  target=`echo $@ | sed 's/-recursive//'`; \
+	  echo making $$target in $$subdir; \
+	  (cd $$subdir && $(MAKE) $$target) || exit 1; \
+	done
+
+mostlyclean-local:
+
+clean-local: mostlyclean-local
+
+distclean-local: clean-local
+	rm -f Makefile config.cache config.h config.log stamp-h
+
+maintainer-clean-local: distclean-local
+	@echo "This command is intended only for maintainers to use;"
+	@echo "rebuilding the deleted files may require special tools."
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+distdir = $(PACKAGE)-$(VERSION)
+
+distname = $(distdir)
+
+DISTFILES = \
+INSTALL.mg INSTALL README README.port COPYING MODIFICATIONS about_mg.html \
+Makefile.in acconfig.h aclocal.m4 configure.in install-sh mkinstalldirs \
+config.guess config.sub \
+config.h.in configure stamp-h.in
+
+#-----------------------------------------------------------------
+
+# Tell versions [3.59,3.63) of GNU make not to export all variables.
+# Otherwise a system limit (for SysV at least) may be exceeded.
+.NOEXPORT:
+
Index: /trunk/indexers/mgpp/acconfig.h
===================================================================
--- /trunk/indexers/mgpp/acconfig.h	(revision 3365)
+++ /trunk/indexers/mgpp/acconfig.h	(revision 3365)
@@ -0,0 +1,41 @@
+/* Special definitions for mg, processed by autoheader.
+   Created by [TS:Aug/95] 
+   by grepping for AC_DEFINE on configure.in and using the appropriate defs 
+   of acconfig.h used by GNU tar.
+*/
+
+/* Define to the name of the distribution.  */
+#undef PACKAGE
+
+/* Define to the version of the distribution.  */
+#undef VERSION
+
+/* Define if `union wait' is the type of the first arg to wait functions.  */
+#undef HAVE_UNION_WAIT
+
+/* Define to 1 if you have the valloc function.  */
+#undef HAVE_VALLOC
+
+/* Define to 1 if ANSI function prototypes are usable.  */
+#undef PROTOTYPES
+ 
+/* Define to 1 for better use of the debugging malloc library.  See
+   site ftp.antaire.com in antaire/src, file dmalloc/dmalloc.tar.gz.  */
+#undef WITH_DMALLOC
+ 
+/* Define to 1 if GNU regex should be used instead of GNU rx.  */
+#undef WITH_REGEX
+
+/* Define to 1 if GNU readline should be used. */
+#undef WITH_GNU_READLINE
+
+/* Some braindead header files do not have the requ'd function declarations */
+#undef HAVE_FGETC_DECL
+#undef HAVE_FREAD_DECL
+
+/* See if have this field in the prstatus_t structure */
+/* It stores the size of the process heap */
+#undef HAVE_PR_BRKSIZE
+
+/* Define to 1 to override the automatic selection of "long long" */
+#undef DISABLE_LONG_LONG
Index: /trunk/indexers/mgpp/aclocal.m4
===================================================================
--- /trunk/indexers/mgpp/aclocal.m4	(revision 3365)
+++ /trunk/indexers/mgpp/aclocal.m4	(revision 3365)
@@ -0,0 +1,225 @@
+# Local additions to Autoconf macros.
+# Copyright (C) 1992, 1994, 1995 Free Software Foundation, Inc.
+# François Pinard <pinard@iro.umontreal.ca>, 1992.
+
+# [TS:Aug/95] - chopped out the NLS stuff which I don't support
+
+## ------------------------------- ##
+## Check for function prototypes.  ##
+## ------------------------------- ##
+
+AC_DEFUN(fp_C_PROTOTYPES,
+[AC_REQUIRE([fp_PROG_CC_STDC])
+AC_MSG_CHECKING([for function prototypes])
+if test "$ac_cv_prog_cc_stdc" != no; then
+  AC_MSG_RESULT(yes)
+  AC_DEFINE(PROTOTYPES)
+  U= ANSI2KNR=
+else
+  AC_MSG_RESULT(no)
+  U=_ ANSI2KNR=ansi2knr
+fi
+AC_SUBST(U)dnl
+AC_SUBST(ANSI2KNR)dnl
+])
+
+## ----------------------------------------- ##
+## ANSIfy the C compiler whenever possible.  ##
+## ----------------------------------------- ##
+
+# @defmac AC_PROG_CC_STDC
+# @maindex PROG_CC_STDC
+# @ovindex CC
+# If the C compiler in not in ANSI C mode by default, try to add an option
+# to output variable @code{CC} to make it so.  This macro tries various
+# options that select ANSI C on some system or another.  It considers the
+# compiler to be in ANSI C mode if it defines @code{__STDC__} to 1 and
+# handles function prototypes correctly.
+#
+# If you use this macro, you should check after calling it whether the C
+# compiler has been set to accept ANSI C; if not, the shell variable
+# @code{ac_cv_prog_cc_stdc} is set to @samp{no}.  If you wrote your source
+# code in ANSI C, you can make an un-ANSIfied copy of it by using the
+# program @code{ansi2knr}, which comes with Ghostscript.
+# @end defmac
+
+AC_DEFUN(fp_PROG_CC_STDC,
+[AC_MSG_CHECKING(for ${CC-cc} option to accept ANSI C)
+AC_CACHE_VAL(ac_cv_prog_cc_stdc,
+[ac_cv_prog_cc_stdc=no
+ac_save_CFLAGS="$CFLAGS"
+# Don't try gcc -ansi; that turns off useful extensions and
+# breaks some systems' header files.
+# AIX			-qlanglvl=ansi
+# Ultrix and OSF/1	-std1
+# HP-UX			-Aa -D_HPUX_SOURCE
+# SVR4			-Xc
+for ac_arg in "" -qlanglvl=ansi -std1 "-Aa -D_HPUX_SOURCE" -Xc
+do
+  CFLAGS="$ac_save_CFLAGS $ac_arg"
+  AC_TRY_COMPILE(
+[#if !defined(__STDC__) || __STDC__ != 1
+choke me
+#endif	
+], [int test (int i, double x);
+struct s1 {int (*f) (int a);};
+struct s2 {int (*f) (double a);};],
+[ac_cv_prog_cc_stdc="$ac_arg"; break])
+done
+CFLAGS="$ac_save_CFLAGS"
+])
+AC_MSG_RESULT($ac_cv_prog_cc_stdc)
+case "x$ac_cv_prog_cc_stdc" in
+  x|xno) ;;
+  *) CC="$CC $ac_cv_prog_cc_stdc" ;;
+esac
+])
+
+## --------------------------------------------------------- ##
+## Use AC_PROG_INSTALL, supplementing it with INSTALL_SCRIPT ##
+## substitution.					     ##
+## --------------------------------------------------------- ##
+
+AC_DEFUN(fp_PROG_INSTALL,
+[AC_PROG_INSTALL
+test -z "$INSTALL_SCRIPT" && INSTALL_SCRIPT='${INSTALL} -m 755'
+AC_SUBST(INSTALL_SCRIPT)dnl
+])
+
+## ----------------------------------- ##
+## Check if --with-dmalloc was given.  ##
+## ----------------------------------- ##
+
+# I just checked, and GNU rx seems to work fine with a slightly
+# modified GNU m4.  So, I put out the test below in my aclocal.m4,
+# and will try to use it in my things.  The idea is to distribute
+# rx.[hc] and regex.[hc] together, for a while.  The WITH_REGEX symbol
+# (which should also be documented in acconfig.h) is used to decide
+# which of regex.h or rx.h should be included in the application.
+#
+# If `./configure --with-regex' is given, the package will use
+# the older regex.  Else, a check is made to see if rx is already
+# installed, as with newer Linux'es.  If not found, the package will
+# use the rx from the distribution.  If found, the package will use
+# the system's rx which, on Linux at least, will result in a smaller
+# executable file.
+
+AC_DEFUN(fp_WITH_DMALLOC,
+[AC_MSG_CHECKING(if malloc debugging is wanted)
+AC_ARG_WITH(dmalloc,
+[  --with-dmalloc          use dmalloc, as in
+                          ftp://ftp.letters.com/src/dmalloc/dmalloc.tar.gz],
+[if test "$withval" = yes; then
+  AC_MSG_RESULT(yes)
+  AC_DEFINE(WITH_DMALLOC)
+  LIBS="$LIBS -ldmalloc"
+  LDFLAGS="$LDFLAGS -g"
+else
+  AC_MSG_RESULT(no)
+fi], [AC_MSG_RESULT(no)])
+])
+
+## --------------------------------- ##
+## Check if --with-regex was given.  ##
+## --------------------------------- ##
+
+AC_DEFUN(fp_WITH_REGEX,
+[AC_MSG_CHECKING(which of rx or regex is wanted)
+AC_ARG_WITH(regex,
+[  --with-regex            use older regex in lieu of GNU rx for matching],
+[if test "$withval" = yes; then
+  ac_with_regex=1
+  AC_MSG_RESULT(regex)
+  AC_DEFINE(WITH_REGEX)
+  LIBOBJS="$LIBOBJS regex.o"
+fi])
+if test -z "$ac_with_regex"; then
+  AC_MSG_RESULT(rx)
+  AC_CHECK_FUNC(re_rx_search, , 
+   [AC_CHECK_SIZEOF(unsigned char *, unsigned char *)  
+    if test "$ac_cv_sizeof_unsigned_char_p" = 8
+    then
+      AC_MSG_WARN(I'm forcing you to use regex because I can't 
+        find a local rx library and the one included with this 
+        distribution doesn't work on 64-bit machines like yours)
+      [LIBOBJS="$LIBOBJS regex.o"]
+    else
+      [LIBOBJS="$LIBOBJS rx.o"]
+    fi]
+  )
+fi
+AC_SUBST(LIBOBJS)dnl
+])
+
+
+## --------------------------------------- ##
+## Check if --with-gnu-readline was given. ##
+## --------------------------------------- ##
+
+AC_DEFUN(fp_WITH_GNU_READLINE,
+[AC_MSG_CHECKING(whether GNU readline requested)
+  AC_ARG_WITH(gnu_readline,
+    [  --with-gnu-readline     compile with GNU readline support],
+    [if test "$withval" = yes; then
+      AC_MSG_RESULT(yes)
+      ac_with_gnu_readline=1
+    else
+      AC_MSG_RESULT(no)
+    fi],
+    AC_MSG_RESULT(no))
+
+  if test -n "$ac_with_gnu_readline"; then
+    AC_CHECK_HEADER(readline/readline.h, ac_mg_readline_header_found=1,
+	AC_MSG_WARN(Can't find GNU readline headers; configuring without \
+GNU readline support))
+    if test -n "$ac_mg_readline_header_found"; then
+      # first check whether we can find the readline library itself
+      AC_CHECK_LIB(readline, main, 
+        [ac_mg_readline_lib_found=1
+         AC_DEFINE(WITH_GNU_READLINE)
+         LIBS="$LIBS -lreadline"],
+        AC_MSG_WARN(Can't find GNU readline library; configuring without \
+GNU readline support))
+      if test -n "$ac_mg_readline_lib_found"; then
+        # on some systems, readline needs curses.  It is difficult to 
+        #  determine whether this is necessary on the current system,
+        #  since other undefined symbols can be turned up by an
+        #  AC_CHECK_LIB(readline, readline) test that are actually defined
+	#  by mg itself (e.g. xmalloc, xrealloc).  So, if we find libcurses,
+	#  we just bung it on and hope for the best.
+	AC_CHECK_LIB(curses, main,
+	  LIBS="$LIBS -lcurses")
+      fi
+    fi
+  fi
+])	
+
+
+
+#AC_DEFUN(fp_WITH_GNU_READLINE,
+#[AC_MSG_CHECKING(whether GNU readline requested)
+#  AC_ARG_WITH(gnu-readline,
+#    [  --with-gnu-readline   compile with GNU readline support],
+#    [if test "$withval" = yes; then
+#      AC_MSG_RESULT(yes)
+#      ac_with_gnu_readline=1
+#    else
+#      AC_MSG_RESULT(no)
+#    fi],
+#    AC_MSG_RESULT(no))
+#  if test -n "$ac_with_gnu_readline"; then
+#    AC_CHECK_LIB(readline, main, 
+#      AC_CHECK_HEADER(readline/readline.h,
+#	[AC_DEFINE(WITH_GNU_READLINE)
+## curses required on some systems (DEC OSF/1), but not on others (GNU/Linux,
+##  Solaris 2.6).  Forcing its inclusion is unsophisticated; if it causes 
+##  any actual problems on any UNIX platforms, let the maintainer know.
+#	LIBS="$LIBS -lreadline -lcurses"],
+#        AC_MSG_WARN(Can't find GNU readline headers: configuring \
+#without GNU readline support)
+#      ),
+#      AC_MSG_WARN(Can't find GNU readline library: configuring \
+#without GNU readline support)
+#    )
+#  fi 
+#])
Index: /trunk/indexers/mgpp/config.guess
===================================================================
--- /trunk/indexers/mgpp/config.guess	(revision 3365)
+++ /trunk/indexers/mgpp/config.guess	(revision 3365)
@@ -0,0 +1,470 @@
+#!/bin/sh
+# Attempt to guess a canonical system name.
+#   Copyright (C) 1992, 1993, 1994 Free Software Foundation, Inc.
+#
+# This file is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+# As a special exception to the GNU General Public License, if you
+# distribute this file as part of a program that contains a
+# configuration script generated by Autoconf, you may include it under
+# the same distribution terms that you use for the rest of that program.
+
+# Written by Per Bothner <bothner@cygnus.com>.
+# The master version of this file is at the FSF in /home/gd/gnu/lib.
+#
+# This script attempts to guess a canonical system name similar to
+# config.sub.  If it succeeds, it prints the system name on stdout, and
+# exits with 0.  Otherwise, it exits with 1.
+#
+# The plan is that this can be called by configure scripts if you
+# don't specify an explicit system type (host/target name).
+#
+# Only a few systems have been added to this list; please add others
+# (but try to keep the structure clean).
+#
+
+# This is needed to find uname on a Pyramid OSx when run in the BSD universe.
+# (ghazi@noc.rutgers.edu 8/24/94.)
+if (test -f /.attbin/uname) >/dev/null 2>&1 ; then
+	PATH=$PATH:/.attbin ; export PATH
+fi
+
+UNAME_MACHINE=`(uname -m) 2>/dev/null` || UNAME_MACHINE=unknown
+UNAME_RELEASE=`(uname -r) 2>/dev/null` || UNAME_RELEASE=unknown
+UNAME_SYSTEM=`(uname -s) 2>/dev/null` || UNAME_SYSTEM=unknown
+UNAME_VERSION=`(uname -v) 2>/dev/null` || UNAME_VERSION=unknown
+
+trap 'rm -f dummy.c dummy.o dummy; exit 1' 1 2 15
+
+# Note: order is significant - the case branches are not exclusive.
+
+case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
+    alpha:OSF1:V*:*)
+	# After 1.2, OSF1 uses "V1.3" for uname -r.
+	echo alpha-dec-osf`echo ${UNAME_RELEASE} | sed -e 's/^V//'`
+	exit 0 ;;
+    alpha:OSF1:*:*)
+	# 1.2 uses "1.2" for uname -r.
+	echo alpha-dec-osf${UNAME_RELEASE}
+        exit 0 ;;
+    arm:RISC*:1.[012]*:*|arm:riscix:1.[012]*:*)
+	echo arm-acorn-riscix${UNAME_RELEASE}
+	exit 0;;
+    Pyramid*:OSx*:*:*)
+	if test "`(/bin/universe) 2>/dev/null`" = att ; then
+		echo pyramid-pyramid-sysv3
+	else
+		echo pyramid-pyramid-bsd
+	fi
+	exit 0 ;;
+    sun4*:SunOS:5.*:*)
+	echo sparc-sun-solaris2`echo ${UNAME_RELEASE}|sed -e 's/[^.]*//'`
+	exit 0 ;;
+    sun4*:SunOS:6*:*)
+	# According to config.sub, this is the proper way to canonicalize
+	# SunOS6.  Hard to guess exactly what SunOS6 will be like, but
+	# it's likely to be more like Solaris than SunOS4.
+	echo sparc-sun-solaris3`echo ${UNAME_RELEASE}|sed -e 's/[^.]*//'`
+	exit 0 ;;
+    sun4*:SunOS:*:*)
+	# Japanese Language versions have a version number like `4.1.3-JL'.
+	echo sparc-sun-sunos`echo ${UNAME_RELEASE}|sed -e 's/-/_/'`
+	exit 0 ;;
+    sun3*:SunOS:*:*)
+	echo m68k-sun-sunos${UNAME_RELEASE}
+	exit 0 ;;
+    RISC*:ULTRIX:*:*)
+	echo mips-dec-ultrix${UNAME_RELEASE}
+	exit 0 ;;
+    VAX*:ULTRIX*:*:*)
+	echo vax-dec-ultrix${UNAME_RELEASE}
+	exit 0 ;;
+    mips:*:5*:RISCos)
+	echo mips-mips-riscos${UNAME_RELEASE}
+	exit 0 ;;
+    m88k:CX/UX:7*:*)
+	echo m88k-harris-cxux7
+	exit 0 ;;
+    m88k:*:4*:R4*)
+	echo m88k-motorola-sysv4
+	exit 0 ;;
+    m88k:*:3*:R3*)
+	echo m88k-motorola-sysv3
+	exit 0 ;;
+    AViiON:dgux:*:*)
+	if [ ${TARGET_BINARY_INTERFACE}x = m88kdguxelfx \
+	     -o ${TARGET_BINARY_INTERFACE}x = x ] ; then
+		echo m88k-dg-dgux${UNAME_RELEASE}
+	else
+		echo m88k-dg-dguxbcs${UNAME_RELEASE}
+	fi
+ 	exit 0 ;;
+    M88*:DolphinOS:*:*)	# DolphinOS (SVR3)
+	echo m88k-dolphin-sysv3
+	exit 0 ;;
+    M88*:*:R3*:*)
+	# Delta 88k system running SVR3
+	echo m88k-motorola-sysv3
+	exit 0 ;;
+    XD88*:*:*:*) # Tektronix XD88 system running UTekV (SVR3)
+	echo m88k-tektronix-sysv3
+	exit 0 ;;
+    Tek43[0-9][0-9]:UTek:*:*) # Tektronix 4300 system running UTek (BSD)
+	echo m68k-tektronix-bsd
+	exit 0 ;;
+    *:IRIX:*:*)
+	echo mips-sgi-irix${UNAME_RELEASE}
+	exit 0 ;;
+    i[34]86:AIX:*:*)
+	echo i386-ibm-aix
+	exit 0 ;;
+    *:AIX:2:3)
+	if grep bos325 /usr/include/stdio.h >/dev/null 2>&1; then
+		sed 's/^		//' << EOF >dummy.c
+		#include <sys/systemcfg.h>
+
+		main()
+			{
+			if (!__power_pc())
+				exit(1);
+			puts("powerpc-ibm-aix3.2.5");
+			exit(0);
+			}
+EOF
+		${CC-cc} dummy.c -o dummy && ./dummy && rm dummy.c dummy && exit 0
+		rm -f dummy.c dummy
+		echo rs6000-ibm-aix3.2.5
+	elif grep bos324 /usr/include/stdio.h >/dev/null 2>&1; then
+		echo rs6000-ibm-aix3.2.4
+	else
+		echo rs6000-ibm-aix3.2
+	fi
+	exit 0 ;;
+    *:AIX:*:4)
+	if /usr/sbin/lsattr -EHl proc0 | grep POWER >/dev/null 2>&1; then
+		IBM_ARCH=rs6000
+	else
+		IBM_ARCH=powerpc
+	fi
+	if grep bos410 /usr/include/stdio.h >/dev/null 2>&1; then
+		IBM_REV=4.1
+	elif grep bos411 /usr/include/stdio.h >/dev/null 2>&1; then
+		IBM_REV=4.1.1
+	else
+		IBM_REV=4.${UNAME_RELEASE}
+	fi
+	echo ${IBM_ARCH}-ibm-aix${IBM_REV}
+	exit 0 ;;
+    *:AIX:*:*)
+	echo rs6000-ibm-aix
+	exit 0 ;;
+    *:BOSX:*:*)
+	echo rs6000-bull-bosx
+	exit 0 ;;
+    DPX/2?00:B.O.S.:*:*)
+	echo m68k-bull-sysv3
+	exit 0 ;;
+    9000/[34]??:4.3bsd:1.*:*)
+	echo m68k-hp-bsd
+	exit 0 ;;
+    hp300:4.4BSD:*:* | 9000/[34]??:4.3bsd:2.*:*)
+	echo m68k-hp-bsd4.4
+	exit 0 ;;
+    9000/[3478]??:HP-UX:*:*)
+	case "${UNAME_MACHINE}" in
+	    9000/31? )            HP_ARCH=m68000 ;;
+	    9000/[34]?? )         HP_ARCH=m68k ;;
+	    9000/7?? | 9000/8?7 ) HP_ARCH=hppa1.1 ;;
+	    9000/8?? )            HP_ARCH=hppa1.0 ;;
+	esac
+	HPUX_REV=`echo ${UNAME_RELEASE}|sed -e 's/[^.]*.[0B]*//'`
+	echo ${HP_ARCH}-hp-hpux${HPUX_REV}
+	exit 0 ;;
+    3050*:HI-UX:*:*)
+	sed 's/^	//' << EOF >dummy.c
+	#include <unistd.h>
+	int
+	main ()
+	{
+	  long cpu = sysconf (_SC_CPU_VERSION);
+	  /* The order matters, because CPU_IS_HP_MC68K erroneously returns
+	     true for CPU_PA_RISC1_0.  CPU_IS_PA_RISC returns correct
+	     results, however.  */
+	  if (CPU_IS_PA_RISC (cpu))
+	    {
+	      switch (cpu)
+		{
+		  case CPU_PA_RISC1_0: puts ("hppa1.0-hitachi-hiuxwe2"); break;
+		  case CPU_PA_RISC1_1: puts ("hppa1.1-hitachi-hiuxwe2"); break;
+		  case CPU_PA_RISC2_0: puts ("hppa2.0-hitachi-hiuxwe2"); break;
+		  default: puts ("hppa-hitachi-hiuxwe2"); break;
+		}
+	    }
+	  else if (CPU_IS_HP_MC68K (cpu))
+	    puts ("m68k-hitachi-hiuxwe2");
+	  else puts ("unknown-hitachi-hiuxwe2");
+	  exit (0);
+	}
+EOF
+	${CC-cc} dummy.c -o dummy && ./dummy && rm dummy.c dummy && exit 0
+	rm -f dummy.c dummy
+	echo unknown-hitachi-hiuxwe2
+	exit 0 ;;
+    9000/7??:4.3bsd:*:* | 9000/8?7:4.3bsd:*:* )
+	echo hppa1.1-hp-bsd
+	exit 0 ;;
+    9000/8??:4.3bsd:*:*)
+	echo hppa1.0-hp-bsd
+	exit 0 ;;
+    hp7??:OSF1:*:* | hp8?7:OSF1:*:* )
+	echo hppa1.1-hp-osf
+	exit 0 ;;
+    hp8??:OSF1:*:*)
+	echo hppa1.0-hp-osf
+	exit 0 ;;
+    C1*:ConvexOS:*:* | convex:ConvexOS:C1*:*)
+	echo c1-convex-bsd
+        exit 0 ;;
+    C2*:ConvexOS:*:* | convex:ConvexOS:C2*:*)
+	if getsysinfo -f scalar_acc
+	then echo c32-convex-bsd
+	else echo c2-convex-bsd
+	fi
+        exit 0 ;;
+    C34*:ConvexOS:*:* | convex:ConvexOS:C34*:*)
+	echo c34-convex-bsd
+        exit 0 ;;
+    C38*:ConvexOS:*:* | convex:ConvexOS:C38*:*)
+	echo c38-convex-bsd
+        exit 0 ;;
+    C4*:ConvexOS:*:* | convex:ConvexOS:C4*:*)
+	echo c4-convex-bsd
+        exit 0 ;;
+    CRAY*X-MP:UNICOS:*:*)
+	echo xmp-cray-unicos
+        exit 0 ;;
+    CRAY*Y-MP:UNICOS:*:*)
+	echo ymp-cray-unicos
+        exit 0 ;;
+    CRAY-2:UNICOS:*:*)
+	echo cray2-cray-unicos
+        exit 0 ;;
+    hp3[0-9][05]:NetBSD:*:*)
+	echo m68k-hp-netbsd${UNAME_RELEASE}
+	exit 0 ;;
+    i[34]86:BSD/386:*:*)
+	echo ${UNAME_MACHINE}-unknown-bsdi${UNAME_RELEASE}
+	exit 0 ;;
+    *:FreeBSD:*:*)
+	echo ${UNAME_MACHINE}-unknown-freebsd`echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`
+	exit 0 ;;
+    *:NetBSD:*:*)
+	echo ${UNAME_MACHINE}-unknown-netbsd`echo ${UNAME_RELEASE}|sed -e 's/[-_].*/\./'`
+	exit 0 ;;
+    *:GNU:*:*)
+	echo `echo ${UNAME_MACHINE}|sed -e 's,/.*$,,'`-unknown-gnu`echo ${UNAME_RELEASE}|sed -e 's,/.*$,,'`
+	exit 0 ;;
+    *:Linux:*:*)
+	echo ${UNAME_MACHINE}-unknown-linux
+	exit 0 ;;
+# ptx 4.0 does uname -s correctly, with DYNIX/ptx in there.  earlier versions
+# are messed up and put the nodename in both sysname and nodename.
+    i[34]86:DYNIX/ptx:4*:*)
+	echo i386-sequent-sysv4
+	exit 0 ;;
+    i[34]86:*:4.*:* | i[34]86:SYSTEM_V:4.*:*)
+	if grep Novell /usr/include/link.h >/dev/null 2>/dev/null; then
+		echo ${UNAME_MACHINE}-univel-sysv${UNAME_RELEASE}
+	else
+		echo ${UNAME_MACHINE}-unknown-sysv${UNAME_RELEASE}
+	fi
+	exit 0 ;;
+    i[34]86:*:3.2:*)
+	if /bin/uname -X 2>/dev/null >/dev/null ; then
+		UNAME_REL=`(/bin/uname -X|egrep Release|sed -e 's/.*= //')`
+		(/bin/uname -X|egrep i80486 >/dev/null) && UNAME_MACHINE=i486
+		echo ${UNAME_MACHINE}-unknown-sco$UNAME_REL
+	elif test -f /usr/options/cb.name; then
+		UNAME_REL=`sed -n 's/.*Version //p' </usr/options/cb.name`
+		echo ${UNAME_MACHINE}-unknown-isc$UNAME_REL
+	else
+		echo ${UNAME_MACHINE}-unknown-sysv32
+	fi
+	exit 0 ;;
+    Intel:Mach:3*:*)
+	echo i386-unknown-mach3
+	exit 0 ;;
+    i860:*:4.*:*) # i860-SVR4
+	if grep Stardent /usr/include/sys/uadmin.h >/dev/null 2>&1 ; then
+	  echo i860-stardent-sysv${UNAME_RELEASE} # Stardent Vistra i860-SVR4
+	else # Add other i860-SVR4 vendors below as they are discovered.
+	  echo i860-unknown-sysv${UNAME_RELEASE}  # Unknown i860-SVR4
+	fi
+	exit 0 ;;
+    mini*:CTIX:SYS*5:*)
+	# "miniframe"
+	echo m68010-convergent-sysv
+	exit 0 ;;
+    M680[234]0:*:R3V[567]*:*)
+	test -r /sysV68 && echo 'm68k-motorola-sysv' && exit 0 ;;
+    3[34]??:*:4.0:3.0 | 3[34]??,*:*:4.0:3.0)
+        uname -p 2>/dev/null | grep 86 >/dev/null \
+          && echo i486-ncr-sysv4.3 && exit 0 ;;
+    3[34]??:*:4.0:* | 3[34]??,*:*:4.0:*)
+        uname -p 2>/dev/null | grep 86 >/dev/null \
+          && echo i486-ncr-sysv4 && exit 0 ;;
+    m680[234]0:LynxOS:2.2*:*)
+	echo m68k-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    mc68030:UNIX_System_V:4.*:*)
+	echo m68k-atari-sysv4
+	exit 0 ;;
+    i[34]86:LynxOS:2.2*:*)
+	echo i386-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    TSUNAMI:LynxOS:2.2*:*)
+	echo sparc-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    rs6000:LynxOS:2.2*:*)
+	echo rs6000-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    RM*:SINIX-*:*:*)
+	echo mips-sni-sysv4
+	exit 0 ;;
+    *:SINIX-*:*:*)
+	if uname -p 2>/dev/null >/dev/null ; then
+		UNAME_MACHINE=`(uname -p) 2>/dev/null`
+		echo ${UNAME_MACHINE}-sni-sysv4
+	else
+		echo ns32k-sni-sysv
+	fi
+	exit 0 ;;
+esac
+
+#echo '(No uname command or uname output not recognized.)' 1>&2
+#echo "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" 1>&2
+
+cat >dummy.c <<EOF
+main ()
+{
+#if defined (sony)
+#if defined (MIPSEB)
+  /* BFD wants "bsd" instead of "newsos".  Perhaps BFD should be changed,
+     I don't know....  */
+  printf ("mips-sony-bsd\n"); exit (0);
+#else
+  printf ("m68k-sony-newsos\n"); exit (0);
+#endif
+#endif
+
+#if defined (__arm) && defined (__acorn) && defined (__unix)
+  printf ("arm-acorn-riscix"); exit (0);
+#endif
+
+#if defined (hp300) && !defined (hpux)
+  printf ("m68k-hp-bsd\n"); exit (0);
+#endif
+
+#if defined (NeXT)
+#if !defined (__ARCHITECTURE__)
+#define __ARCHITECTURE__ "m68k"
+#endif
+  int version;
+  version=`(hostinfo | sed -n 's/.*NeXT Mach \([0-9]*\).*/\1/p') 2>/dev/null`;
+  printf ("%s-next-nextstep%s\n", __ARCHITECTURE__,  version==2 ? "2" : "3");
+  exit (0);
+#endif
+
+#if defined (MULTIMAX) || defined (n16)
+#if defined (UMAXV)
+  printf ("ns32k-encore-sysv\n"); exit (0);
+#else
+#if defined (CMU)
+  printf ("ns32k-encore-mach\n"); exit (0);
+#else
+  printf ("ns32k-encore-bsd\n"); exit (0);
+#endif
+#endif
+#endif
+
+#if defined (__386BSD__)
+  printf ("i386-unknown-bsd\n"); exit (0);
+#endif
+
+#if defined (sequent)
+#if defined (i386)
+  printf ("i386-sequent-dynix\n"); exit (0);
+#endif
+#if defined (ns32000)
+  printf ("ns32k-sequent-dynix\n"); exit (0);
+#endif
+#endif
+
+#if defined (_SEQUENT_)
+  printf ("i386-sequent-ptx\n"); exit (0);
+#endif
+
+#if defined (vax)
+#if !defined (ultrix)
+  printf ("vax-dec-bsd\n"); exit (0);
+#else
+  printf ("vax-dec-ultrix\n"); exit (0);
+#endif
+#endif
+
+#if defined (alliant) && defined (i860)
+  printf ("i860-alliant-bsd\n"); exit (0);
+#endif
+
+  exit (1);
+}
+EOF
+
+${CC-cc} dummy.c -o dummy 2>/dev/null && ./dummy && rm dummy.c dummy && exit 0
+rm -f dummy.c dummy
+
+# Apollos put the system type in the environment.
+
+test -d /usr/apollo && { echo ${ISP}-apollo-${SYSTYPE}; exit 0; }
+
+# Convex versions that predate uname can use getsysinfo(1)
+
+if [ -x /usr/convex/getsysinfo ]
+then
+    case `getsysinfo -f cpu_type` in
+    c1*)
+	echo c1-convex-bsd
+	exit 0 ;;
+    c2*)
+	if getsysinfo -f scalar_acc
+	then echo c32-convex-bsd
+	else echo c2-convex-bsd
+	fi
+	exit 0 ;;
+    c34*)
+	echo c34-convex-bsd
+	exit 0 ;;
+    c38*)
+	echo c38-convex-bsd
+	exit 0 ;;
+    c4*)
+	echo c4-convex-bsd
+	exit 0 ;;
+    esac
+fi
+
+#echo '(Unable to guess system type)' 1>&2
+
+exit 1
Index: /trunk/indexers/mgpp/config.h.in
===================================================================
--- /trunk/indexers/mgpp/config.h.in	(revision 3365)
+++ /trunk/indexers/mgpp/config.h.in	(revision 3365)
@@ -0,0 +1,183 @@
+/* config.h.in.  Generated automatically from configure.in by autoheader.  */
+
+/* Define if on AIX 3.
+   System headers sometimes define this.
+   We just want to avoid a redefinition error message.  */
+#ifndef _ALL_SOURCE
+#undef _ALL_SOURCE
+#endif
+
+/* Define if using alloca.c.  */
+#undef C_ALLOCA
+
+/* Define to empty if the keyword does not work.  */
+#undef const
+
+/* Define to one of _getb67, GETB67, getb67 for Cray-2 and Cray-YMP systems.
+   This function is required for alloca.c support on those systems.  */
+#undef CRAY_STACKSEG_END
+
+/* Define if you have alloca, as a function or macro.  */
+#undef HAVE_ALLOCA
+
+/* Define if you have <alloca.h> and it should be used (not on Ultrix).  */
+#undef HAVE_ALLOCA_H
+
+/* Define if you don't have vprintf but do have _doprnt.  */
+#undef HAVE_DOPRNT
+
+/* Define if you have the vprintf function.  */
+#undef HAVE_VPRINTF
+
+/* Define if on MINIX.  */
+#undef _MINIX
+
+/* Define to `long' if <sys/types.h> doesn't define.  */
+#undef off_t
+
+/* Define if the system does not provide POSIX.1 features except
+   with this defined.  */
+#undef _POSIX_1_SOURCE
+
+/* Define if you need to in order for stat and other things to work.  */
+#undef _POSIX_SOURCE
+
+/* Define as the return type of signal handlers (int or void).  */
+#undef RETSIGTYPE
+
+/* Define to `unsigned' if <sys/types.h> doesn't define.  */
+#undef size_t
+
+/* If using the C implementation of alloca, define if you know the
+   direction of stack growth for your system; otherwise it will be
+   automatically deduced at run-time.
+ STACK_DIRECTION > 0 => grows toward higher addresses
+ STACK_DIRECTION < 0 => grows toward lower addresses
+ STACK_DIRECTION = 0 => direction of growth unknown
+ */
+#undef STACK_DIRECTION
+
+/* Define if you have the ANSI C header files.  */
+#undef STDC_HEADERS
+
+/* Define if you can safely include both <sys/time.h> and <time.h>.  */
+#undef TIME_WITH_SYS_TIME
+
+/* Define if your <sys/time.h> declares struct tm.  */
+#undef TM_IN_SYS_TIME
+
+/* Define to the name of the distribution.  */
+#undef PACKAGE
+
+/* Define to the version of the distribution.  */
+#undef VERSION
+
+/* Define to 1 if you have the valloc function.  */
+#undef HAVE_VALLOC
+
+/* Define to 1 if ANSI function prototypes are usable.  */
+#undef PROTOTYPES
+
+ 
+/* Define to 1 for better use of the debugging malloc library.  See
+   site ftp.antaire.com in antaire/src, file dmalloc/dmalloc.tar.gz.  */
+#undef WITH_DMALLOC
+
+ 
+/* Define to 1 if GNU regex should be used instead of GNU rx.  */
+#undef WITH_REGEX
+
+/* Define to 1 if GNU readline should be used. */
+#undef WITH_GNU_READLINE
+
+/* Some braindead header files do not have the requ'd function declarations */
+#undef HAVE_FGETC_DECL
+#undef HAVE_FREAD_DECL
+
+/* See if have this field in the prstatus_t structure */
+/* It stores the size of the process heap */
+#undef HAVE_PR_BRKSIZE
+
+/* Define to 1 to override the automatic selection of "long long" */
+#undef DISABLE_LONG_LONG
+
+/* The number of bytes in a unsigned char *.  */
+#undef SIZEOF_UNSIGNED_CHAR_P
+
+/* Define if you have the ftime function.  */
+#undef HAVE_FTIME
+
+/* Define if you have the ftruncate function.  */
+#undef HAVE_FTRUNCATE
+
+/* Define if you have the getpagesize function.  */
+#undef HAVE_GETPAGESIZE
+
+/* Define if you have the getrusage function.  */
+#undef HAVE_GETRUSAGE
+
+/* Define if you have the mallinfo function.  */
+#undef HAVE_MALLINFO
+
+/* Define if you have the select function.  */
+#undef HAVE_SELECT
+
+/* Define if you have the setbuffer function.  */
+#undef HAVE_SETBUFFER
+
+/* Define if you have the strcasecmp function.  */
+#undef HAVE_STRCASECMP
+
+/* Define if you have the strerror function.  */
+#undef HAVE_STRERROR
+
+/* Define if you have the strftime function.  */
+#undef HAVE_STRFTIME
+
+/* Define if you have the strstr function.  */
+#undef HAVE_STRSTR
+
+/* Define if you have the strtol function.  */
+#undef HAVE_STRTOL
+
+/* Define if you have the times function.  */
+#undef HAVE_TIMES
+
+/* Define if you have the valloc function.  */
+#undef HAVE_VALLOC
+
+/* Define if you have the <dirent.h> header file.  */
+#undef HAVE_DIRENT_H
+
+/* Define if you have the <fcntl.h> header file.  */
+#undef HAVE_FCNTL_H
+
+/* Define if you have the <limits.h> header file.  */
+#undef HAVE_LIMITS_H
+
+/* Define if you have the <memory.h> header file.  */
+#undef HAVE_MEMORY_H
+
+/* Define if you have the <ndir.h> header file.  */
+#undef HAVE_NDIR_H
+
+/* Define if you have the <string.h> header file.  */
+#undef HAVE_STRING_H
+
+/* Define if you have the <sys/dir.h> header file.  */
+#undef HAVE_SYS_DIR_H
+
+/* Define if you have the <sys/ndir.h> header file.  */
+#undef HAVE_SYS_NDIR_H
+
+/* Define if you have the <sys/procfs.h> header file.  */
+#undef HAVE_SYS_PROCFS_H
+
+/* Define if you have the <sys/time.h> header file.  */
+#undef HAVE_SYS_TIME_H
+
+/* Define if you have the <unistd.h> header file.  */
+#undef HAVE_UNISTD_H
+
+/* Define if you have the m library (-lm).  */
+#undef HAVE_LIBM
Index: /trunk/indexers/mgpp/config.sub
===================================================================
--- /trunk/indexers/mgpp/config.sub	(revision 3365)
+++ /trunk/indexers/mgpp/config.sub	(revision 3365)
@@ -0,0 +1,793 @@
+#!/bin/sh
+# Configuration validation subroutine script, version 1.1.
+#   Copyright (C) 1991, 1992, 1993, 1994 Free Software Foundation, Inc.
+# This file is (in principle) common to ALL GNU software.
+# The presence of a machine in this file suggests that SOME GNU software
+# can handle that machine.  It does not imply ALL GNU software can. 
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+# As a special exception to the GNU General Public License, if you
+# distribute this file as part of a program that contains a
+# configuration script generated by Autoconf, you may include it under
+# the same distribution terms that you use for the rest of that program.
+
+# Configuration subroutine to validate and canonicalize a configuration type.
+# Supply the specified configuration type as an argument.
+# If it is invalid, we print an error message on stderr and exit with code 1.
+# Otherwise, we print the canonical config type on stdout and succeed.
+
+# This file is supposed to be the same for all GNU packages
+# and recognize all the CPU types, system types and aliases
+# that are meaningful with *any* GNU software.
+# Each package is responsible for reporting which valid configurations
+# it does not support.  The user should be able to distinguish
+# a failure to support a valid configuration from a meaningless
+# configuration.
+
+# The goal of this file is to map all the various variations of a given
+# machine specification into a single specification in the form:
+#	CPU_TYPE-MANUFACTURER-OPERATING_SYSTEM
+# It is wrong to echo any other type of specification.
+
+# First pass through any local machine types.
+case $1 in
+	*local*)
+		echo $1
+		exit 0
+		;;
+	*)
+	;;
+esac
+
+# Separate what the user gave into CPU-COMPANY and OS (if any).
+basic_machine=`echo $1 | sed 's/-[^-]*$//'`
+if [ $basic_machine != $1 ]
+then os=`echo $1 | sed 's/.*-/-/'`
+else os=; fi
+
+### Let's recognize common machines as not being operating systems so
+### that things like config.sub decstation-3100 work.  We also
+### recognize some manufacturers as not being operating systems, so we
+### can provide default operating systems below.
+case $os in
+	-sun*os*)
+		# Prevent following clause from handling this invalid input.
+		;;
+	-dec* | -mips* | -sequent* | -encore* | -pc532* | -sgi* | -sony* | \
+	-att* | -7300* | -3300* | -delta* | -motorola* | -sun[234]* | \
+	-unicom* | -ibm* | -next | -hp | -isi* | -apollo | -altos* | \
+	-convergent* | -ncr* | -news | -32* | -3600* | -3100* | -hitachi* |\
+	-c[123]* | -convex* | -sun | -crds | -omron* | -dg | -ultra | -tti* | \
+	-harris | -dolphin | -highlevel | -gould | -cbm | -ns | -masscomp )
+		os=
+		basic_machine=$1
+		;;
+	-hiux*)
+		os=-hiuxwe2
+		;;
+	-sco4)
+		os=-sco3.2v4
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-sco3.2.[4-9]*)
+		os=`echo $os | sed -e 's/sco3.2./sco3.2v/'`
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-sco3.2v[4-9]*)
+		# Don't forget version if it is 3.2v4 or newer.
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-sco*)
+		os=-sco3.2v2
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-isc)
+		os=-isc2.2
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-clix*)
+		basic_machine=clipper-intergraph
+		;;
+	-isc*)
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-lynx)
+		os=-lynxos
+		;;
+	-ptx*)
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-sequent/'`
+		;;
+	-windowsnt*)
+		os=`echo $os | sed -e 's/windowsnt/winnt/'`
+		;;
+esac
+
+# Decode aliases for certain CPU-COMPANY combinations.
+case $basic_machine in
+	# Recognize the basic CPU types without company name.
+	# Some are omitted here because they have special meanings below.
+	tahoe | i[345]86 | i860 | m68k | m68000 | m88k | ns32k | arm | pyramid \
+		| tron | a29k | 580 | i960 | h8300 | hppa1.0 | hppa1.1 \
+		| alpha | we32k | ns16k | clipper | sparclite | i370 | sh \
+		| powerpc | sparc64 | 1750a | dsp16xx | mips64 | mipsel \
+		| pdp11 | mips64el | mips64orion | mips64orionel )
+		basic_machine=$basic_machine-unknown
+		;;
+	# Object if more than one company name word.
+	*-*-*)
+		echo Invalid configuration \`$1\': machine \`$basic_machine\' not recognized 1>&2
+		exit 1
+		;;
+	# Recognize the basic CPU types with company name.
+	vax-* | tahoe-* | i[345]86-* | i860-* | m68k-* | m68000-* | m88k-* \
+	      | sparc-* | ns32k-* | fx80-* | arm-* | c[123]* \
+	      | mips-* | pyramid-* | tron-* | a29k-* | romp-* | rs6000-* \
+	      | none-* | 580-* | cray2-* | h8300-* | i960-* | xmp-* | ymp-* \
+	      | hppa1.0-* | hppa1.1-* | alpha-* | we32k-* | cydra-* | ns16k-* \
+	      | pn-* | np1-* | xps100-* | clipper-* | orion-* | sparclite-* \
+	      | pdp11-* | sh-* | powerpc-* | sparc64-* | mips64-* | mipsel-* \
+	      | mips64el-* | mips64orion-* | mips64orionel-* )
+		;;
+	# Recognize the various machine names and aliases which stand
+	# for a CPU type and a company and sometimes even an OS.
+	3b1 | 7300 | 7300-att | att-7300 | pc7300 | safari | unixpc)
+		basic_machine=m68000-att
+		;;
+	3b*)
+		basic_machine=we32k-att
+		;;
+	alliant | fx80)
+		basic_machine=fx80-alliant
+		;;
+	altos | altos3068)
+		basic_machine=m68k-altos
+		;;
+	am29k)
+		basic_machine=a29k-none
+		os=-bsd
+		;;
+	amdahl)
+		basic_machine=580-amdahl
+		os=-sysv
+		;;
+	amiga | amiga-*)
+		basic_machine=m68k-cbm
+		;;
+	amigados)
+		basic_machine=m68k-cbm
+		os=-amigados
+		;;
+	amigaunix | amix)
+		basic_machine=m68k-cbm
+		os=-sysv4
+		;;
+	apollo68)
+		basic_machine=m68k-apollo
+		os=-sysv
+		;;
+	balance)
+		basic_machine=ns32k-sequent
+		os=-dynix
+		;;
+	convex-c1)
+		basic_machine=c1-convex
+		os=-bsd
+		;;
+	convex-c2)
+		basic_machine=c2-convex
+		os=-bsd
+		;;
+	convex-c32)
+		basic_machine=c32-convex
+		os=-bsd
+		;;
+	convex-c34)
+		basic_machine=c34-convex
+		os=-bsd
+		;;
+	convex-c38)
+		basic_machine=c38-convex
+		os=-bsd
+		;;
+	cray | ymp)
+		basic_machine=ymp-cray
+		os=-unicos
+		;;
+	cray2)
+		basic_machine=cray2-cray
+		os=-unicos
+		;;
+	crds | unos)
+		basic_machine=m68k-crds
+		;;
+	da30 | da30-*)
+		basic_machine=m68k-da30
+		;;
+	decstation | decstation-3100 | pmax | pmax-* | pmin | dec3100 | decstatn)
+		basic_machine=mips-dec
+		;;
+	delta | 3300 | motorola-3300 | motorola-delta \
+	      | 3300-motorola | delta-motorola)
+		basic_machine=m68k-motorola
+		;;
+	delta88)
+		basic_machine=m88k-motorola
+		os=-sysv3
+		;;
+	dpx20 | dpx20-*)
+		basic_machine=rs6000-bull
+		os=-bosx
+		;;
+	dpx2* | dpx2*-bull)
+		basic_machine=m68k-bull
+		os=-sysv3
+		;;
+	ebmon29k)
+		basic_machine=a29k-amd
+		os=-ebmon
+		;;
+	elxsi)
+		basic_machine=elxsi-elxsi
+		os=-bsd
+		;;
+	encore | umax | mmax)
+		basic_machine=ns32k-encore
+		;;
+	fx2800)
+		basic_machine=i860-alliant
+		;;
+	genix)
+		basic_machine=ns32k-ns
+		;;
+	gmicro)
+		basic_machine=tron-gmicro
+		os=-sysv
+		;;
+	h3050r* | hiux*)
+		basic_machine=hppa1.1-hitachi
+		os=-hiuxwe2
+		;;
+	h8300hms)
+		basic_machine=h8300-hitachi
+		os=-hms
+		;;
+	harris)
+		basic_machine=m88k-harris
+		os=-sysv3
+		;;
+	hp300-*)
+		basic_machine=m68k-hp
+		;;
+	hp300bsd)
+		basic_machine=m68k-hp
+		os=-bsd
+		;;
+	hp300hpux)
+		basic_machine=m68k-hp
+		os=-hpux
+		;;
+	hp9k2[0-9][0-9] | hp9k31[0-9])
+		basic_machine=m68000-hp
+		;;
+	hp9k3[2-9][0-9])
+		basic_machine=m68k-hp
+		;;
+	hp9k7[0-9][0-9] | hp7[0-9][0-9] | hp9k8[0-9]7 | hp8[0-9]7)
+		basic_machine=hppa1.1-hp
+		;;
+	hp9k8[0-9][0-9] | hp8[0-9][0-9])
+		basic_machine=hppa1.0-hp
+		;;
+	i370-ibm* | ibm*)
+		basic_machine=i370-ibm
+		os=-mvs
+		;;
+# I'm not sure what "Sysv32" means.  Should this be sysv3.2?
+	i[345]86v32)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-sysv32
+		;;
+	i[345]86v4*)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-sysv4
+		;;
+	i[345]86v)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-sysv
+		;;
+	i[345]86sol2)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-solaris2
+		;;
+	iris | iris4d)
+		basic_machine=mips-sgi
+		case $os in
+		    -irix*)
+			;;
+		    *)
+			os=-irix4
+			;;
+		esac
+		;;
+	isi68 | isi)
+		basic_machine=m68k-isi
+		os=-sysv
+		;;
+	m88k-omron*)
+		basic_machine=m88k-omron
+		;;
+	magnum | m3230)
+		basic_machine=mips-mips
+		os=-sysv
+		;;
+	merlin)
+		basic_machine=ns32k-utek
+		os=-sysv
+		;;
+	miniframe)
+		basic_machine=m68000-convergent
+		;;
+	mips3*-*)
+		basic_machine=`echo $basic_machine | sed -e 's/mips3/mips64/'`
+		;;
+	mips3*)
+		basic_machine=`echo $basic_machine | sed -e 's/mips3/mips64/'`-unknown
+		;;
+	ncr3000)
+		basic_machine=i486-ncr
+		os=-sysv4
+		;;
+	news | news700 | news800 | news900)
+		basic_machine=m68k-sony
+		os=-newsos
+		;;
+	news1000)
+		basic_machine=m68030-sony
+		os=-newsos
+		;;
+	news-3600 | risc-news)
+		basic_machine=mips-sony
+		os=-newsos
+		;;
+	next | m*-next )
+		basic_machine=m68k-next
+		case $os in
+		    -nextstep* )
+			;;
+		    -ns2*)
+		      os=-nextstep2
+			;;
+		    *)
+		      os=-nextstep3
+			;;
+		esac
+		;;
+	nh3000)
+		basic_machine=m68k-harris
+		os=-cxux
+		;;
+	nh[45]000)
+		basic_machine=m88k-harris
+		os=-cxux
+		;;
+	nindy960)
+		basic_machine=i960-intel
+		os=-nindy
+		;;
+	np1)
+		basic_machine=np1-gould
+		;;
+	pa-hitachi)
+		basic_machine=hppa1.1-hitachi
+		os=-hiuxwe2
+		;;
+	paragon)
+		basic_machine=i860-intel
+		os=-osf
+		;;
+	pbd)
+		basic_machine=sparc-tti
+		;;
+	pbb)
+		basic_machine=m68k-tti
+		;;
+        pc532 | pc532-*)
+		basic_machine=ns32k-pc532
+		;;
+	pentium-*)
+		# We will change tis to say i586 once there has been
+		# time for various packages to start to recognize that.
+		basic_machine=i486-`echo $basic_machine | sed 's/^[^-]*-//'`
+		;;
+	pn)
+		basic_machine=pn-gould
+		;;
+	ps2)
+		basic_machine=i386-ibm
+		;;
+	rtpc | rtpc-*)
+		basic_machine=romp-ibm
+		;;
+	sequent)
+		basic_machine=i386-sequent
+		;;
+	sh)
+		basic_machine=sh-hitachi
+		os=-hms
+		;;
+	sps7)
+		basic_machine=m68k-bull
+		os=-sysv2
+		;;
+	spur)
+		basic_machine=spur-unknown
+		;;
+	sun2)
+		basic_machine=m68000-sun
+		;;
+	sun2os3)
+		basic_machine=m68000-sun
+		os=-sunos3
+		;;
+	sun2os4)
+		basic_machine=m68000-sun
+		os=-sunos4
+		;;
+	sun3os3)
+		basic_machine=m68k-sun
+		os=-sunos3
+		;;
+	sun3os4)
+		basic_machine=m68k-sun
+		os=-sunos4
+		;;
+	sun4os3)
+		basic_machine=sparc-sun
+		os=-sunos3
+		;;
+	sun4os4)
+		basic_machine=sparc-sun
+		os=-sunos4
+		;;
+	sun3 | sun3-*)
+		basic_machine=m68k-sun
+		;;
+	sun4)
+		basic_machine=sparc-sun
+		;;
+	sun386 | sun386i | roadrunner)
+		basic_machine=i386-sun
+		;;
+	symmetry)
+		basic_machine=i386-sequent
+		os=-dynix
+		;;
+	tower | tower-32)
+		basic_machine=m68k-ncr
+		;;
+	ultra3)
+		basic_machine=a29k-nyu
+		os=-sym1
+		;;
+	vaxv)
+		basic_machine=vax-dec
+		os=-sysv
+		;;
+	vms)
+		basic_machine=vax-dec
+		os=-vms
+		;;
+	vxworks960)
+		basic_machine=i960-wrs
+		os=-vxworks
+		;;
+	vxworks68)
+		basic_machine=m68k-wrs
+		os=-vxworks
+		;;
+	xmp)
+		basic_machine=xmp-cray
+		os=-unicos
+		;;
+        xps | xps100)
+		basic_machine=xps100-honeywell
+		;;
+	none)
+		basic_machine=none-none
+		os=-none
+		;;
+
+# Here we handle the default manufacturer of certain CPU types.  It is in
+# some cases the only manufacturer, in others, it is the most popular.
+	mips)
+		basic_machine=mips-mips
+		;;
+	romp)
+		basic_machine=romp-ibm
+		;;
+	rs6000)
+		basic_machine=rs6000-ibm
+		;;
+	vax)
+		basic_machine=vax-dec
+		;;
+	pdp11)
+		basic_machine=pdp11-dec
+		;;
+	we32k)
+		basic_machine=we32k-att
+		;;
+	sparc)
+		basic_machine=sparc-sun
+		;;
+        cydra)
+		basic_machine=cydra-cydrome
+		;;
+	orion)
+		basic_machine=orion-highlevel
+		;;
+	orion105)
+		basic_machine=clipper-highlevel
+		;;
+	*)
+		echo Invalid configuration \`$1\': machine \`$basic_machine\' not recognized 1>&2
+		exit 1
+		;;
+esac
+
+# Here we canonicalize certain aliases for manufacturers.
+case $basic_machine in
+	*-digital*)
+		basic_machine=`echo $basic_machine | sed 's/digital.*/dec/'`
+		;;
+	*-commodore*)
+		basic_machine=`echo $basic_machine | sed 's/commodore.*/cbm/'`
+		;;
+	*)
+		;;
+esac
+
+# Decode manufacturer-specific aliases for certain operating systems.
+
+if [ x"$os" != x"" ]
+then
+case $os in
+	# -solaris* is a basic system type, with this one exception.
+	-solaris1 | -solaris1.*)
+		os=`echo $os | sed -e 's|solaris1|sunos4|'`
+		;;
+	-solaris)
+		os=-solaris2
+		;;
+	-gnu/linux*)
+		os=`echo $os | sed -e 's|gnu/linux|linux|'`
+		;;
+	# First accept the basic system types.
+	# The portable systems comes first.
+	# Each alternative must end in a *, to match a version number.
+	# -sysv* is not here because it comes later, after sysvr4.
+	-gnu* | -bsd* | -mach* | -minix* | -genix* | -ultrix* | -irix* \
+	      | -vms* | -sco* | -esix* | -isc* | -aix* | -sunos | -sunos[345]* \
+	      | -hpux* | -unos* | -osf* | -luna* | -dgux* | -solaris* | -sym* \
+	      | -amigados* | -msdos* | -newsos* | -unicos* | -aos* \
+	      | -nindy* | -vxworks* | -ebmon* | -hms* | -mvs* | -clix* \
+	      | -riscos* | -linux* | -uniplus* | -iris* | -rtu* | -xenix* \
+	      | -hiux* | -386bsd* | -netbsd* | -freebsd* | -riscix* \
+	      | -lynxos* | -bosx* | -nextstep* | -cxux* | -aout* | -elf* \
+	      | -ptx* | -coff* | -winnt*)
+		;;
+	-sunos5*)
+		os=`echo $os | sed -e 's|sunos5|solaris2|'`
+		;;
+	-sunos6*)
+		os=`echo $os | sed -e 's|sunos6|solaris3|'`
+		;;
+	-osfrose*)
+		os=-osfrose
+		;;
+	-osf*)
+		os=-osf
+		;;
+	-utek*)
+		os=-bsd
+		;;
+	-dynix*)
+		os=-bsd
+		;;
+	-acis*)
+		os=-aos
+		;;
+	-ctix* | -uts*)
+		os=-sysv
+		;;
+	-triton*)
+		os=-sysv3
+		;;
+	-oss*)
+		os=-sysv3
+		;;
+	-svr4)
+		os=-sysv4
+		;;
+	-svr3)
+		os=-sysv3
+		;;
+	-sysvr4)
+		os=-sysv4
+		;;
+	# This must come after -sysvr4.
+	-sysv*)
+		;;
+	-xenix)
+		os=-xenix
+		;;
+	-none)
+		;;
+	*)
+		# Get rid of the `-' at the beginning of $os.
+		os=`echo $os | sed 's/[^-]*-//'`
+		echo Invalid configuration \`$1\': system \`$os\' not recognized 1>&2
+		exit 1
+		;;
+esac
+else
+
+# Here we handle the default operating systems that come with various machines.
+# The value should be what the vendor currently ships out the door with their
+# machine or put another way, the most popular os provided with the machine.
+
+# Note that if you're going to try to match "-MANUFACTURER" here (say,
+# "-sun"), then you have to tell the case statement up towards the top
+# that MANUFACTURER isn't an operating system.  Otherwise, code above
+# will signal an error saying that MANUFACTURER isn't an operating
+# system, and we'll never get to this point.
+
+case $basic_machine in
+	*-acorn)
+		os=-riscix1.2
+		;;
+        pdp11-*)
+		os=-none
+		;;
+	*-dec | vax-*)
+		os=-ultrix4.2
+		;;
+	i386-sun)
+		os=-sunos4.0.2
+		;;
+	m68000-sun)
+		os=-sunos3
+		# This also exists in the configure program, but was not the
+		# default.
+		# os=-sunos4
+		;;
+	*-tti)	# must be before sparc entry or we get the wrong os.
+		os=-sysv3
+		;;
+	sparc-* | *-sun)
+		os=-sunos4.1.1
+		;;
+	*-ibm)
+		os=-aix
+		;;
+	*-hp)
+		os=-hpux
+		;;
+	*-hitachi)
+		os=-hiux
+		;;
+	i860-* | *-att | *-ncr | *-altos | *-motorola | *-convergent)
+		os=-sysv
+		;;
+	*-cbm)
+		os=-amigados
+		;;
+	*-dg)
+		os=-dgux
+		;;
+	*-dolphin)
+		os=-sysv3
+		;;
+	m68k-ccur)
+		os=-rtu
+		;;
+	m88k-omron*)
+		os=-luna
+		;;
+	*-sequent)
+		os=-ptx
+		;;
+	*-crds)
+		os=-unos
+		;;
+	*-ns)
+		os=-genix
+		;;
+	i370-*)
+		os=-mvs
+		;;
+	*-next)
+		os=-nextstep3
+		;;
+        *-gould)
+		os=-sysv
+		;;
+        *-highlevel)
+		os=-bsd
+		;;
+	*-encore)
+		os=-bsd
+		;;
+        *-sgi)
+		os=-irix
+		;;
+	*-masscomp)
+		os=-rtu
+		;;
+	*)
+		os=-none
+		;;
+esac
+fi
+
+# Here we handle the case where we know the os, and the CPU type, but not the
+# manufacturer.  We pick the logical manufacturer.
+vendor=unknown
+case $basic_machine in
+	*-unknown)
+		case $os in
+			-riscix*)
+				vendor=acorn
+				;;
+			-sunos*)
+				vendor=sun
+				;;
+			-lynxos*)
+				vendor=lynx
+				;;
+			-aix*)
+				vendor=ibm
+				;;
+			-hpux*)
+				vendor=hp
+				;;
+			-hiux*)
+				vendor=hitachi
+				;;
+			-unos*)
+				vendor=crds
+				;;
+			-dgux*)
+				vendor=dg
+				;;
+			-luna*)
+				vendor=omron
+				;;
+			-genix*)
+				vendor=ns
+				;;
+			-mvs*)
+				vendor=ibm
+				;;
+			-ptx*)
+				vendor=sequent
+				;;
+		esac
+		basic_machine=`echo $basic_machine | sed "s/unknown/$vendor/"`
+		;;
+esac
+
+echo $basic_machine$os
Index: /trunk/indexers/mgpp/configure
===================================================================
--- /trunk/indexers/mgpp/configure	(revision 3365)
+++ /trunk/indexers/mgpp/configure	(revision 3365)
@@ -0,0 +1,5792 @@
+#! /bin/sh
+# Guess values for system-dependent variables and create Makefiles.
+# Generated by Autoconf 2.50.
+#
+# Copyright 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001
+# Free Software Foundation, Inc.
+# This configure script is free software; the Free Software Foundation
+# gives unlimited permission to copy, distribute and modify it.
+
+# Be Bourne compatible
+if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then
+  emulate sh
+  NULLCMD=:
+elif test -n "${BASH_VERSION+set}" && (set -o posix) >/dev/null 2>&1; then
+  set -o posix
+fi
+
+# Name of the executable.
+as_me=`echo "$0" |sed 's,.*[\\/],,'`
+
+if expr a : '\(a\)' >/dev/null 2>&1; then
+  as_expr=expr
+else
+  as_expr=false
+fi
+
+rm -f conf$$ conf$$.exe conf$$.file
+echo >conf$$.file
+if ln -s conf$$.file conf$$ 2>/dev/null; then
+  # We could just check for DJGPP; but this test a) works b) is more generic
+  # and c) will remain valid once DJGPP supports symlinks (DJGPP 2.04).
+  if test -f conf$$.exe; then
+    # Don't use ln at all; we don't have any links
+    as_ln_s='cp -p'
+  else
+    as_ln_s='ln -s'
+  fi
+elif ln conf$$.file conf$$ 2>/dev/null; then
+  as_ln_s=ln
+else
+  as_ln_s='cp -p'
+fi
+rm -f conf$$ conf$$.exe conf$$.file
+
+as_executable_p="test -f"
+
+# Support unset when possible.
+if (FOO=FOO; unset FOO) >/dev/null 2>&1; then
+  as_unset=unset
+else
+  as_unset=false
+fi
+
+# NLS nuisances.
+$as_unset LANG || test "${LANG+set}" != set || { LANG=C; export LANG; }
+$as_unset LC_ALL || test "${LC_ALL+set}" != set || { LC_ALL=C; export LC_ALL; }
+$as_unset LC_TIME || test "${LC_TIME+set}" != set || { LC_TIME=C; export LC_TIME; }
+$as_unset LC_CTYPE || test "${LC_CTYPE+set}" != set || { LC_CTYPE=C; export LC_CTYPE; }
+$as_unset LANGUAGE || test "${LANGUAGE+set}" != set || { LANGUAGE=C; export LANGUAGE; }
+$as_unset LC_COLLATE || test "${LC_COLLATE+set}" != set || { LC_COLLATE=C; export LC_COLLATE; }
+$as_unset LC_NUMERIC || test "${LC_NUMERIC+set}" != set || { LC_NUMERIC=C; export LC_NUMERIC; }
+$as_unset LC_MESSAGES || test "${LC_MESSAGES+set}" != set || { LC_MESSAGES=C; export LC_MESSAGES; }
+
+# IFS
+# We need space, tab and new line, in precisely that order.
+as_nl='
+'
+IFS=" 	$as_nl"
+
+# CDPATH.
+$as_unset CDPATH || test "${CDPATH+set}" != set || { CDPATH=:; export CDPATH; }
+
+# Name of the host.
+# hostname on some systems (SVR3.2, Linux) returns a bogus exit status,
+# so uname gets run too.
+ac_hostname=`(hostname || uname -n) 2>/dev/null | sed 1q`
+
+exec 6>&1
+
+#
+# Initializations.
+#
+ac_default_prefix=/usr/local
+cross_compiling=no
+subdirs=
+MFLAGS= MAKEFLAGS=
+SHELL=${CONFIG_SHELL-/bin/sh}
+
+# Maximum number of lines to put in a shell here document.
+# This variable seems obsolete.  It should probably be removed, and
+# only ac_max_sed_lines should be used.
+: ${ac_max_here_lines=38}
+
+# Avoid depending upon Character Ranges.
+ac_cr_az='abcdefghijklmnopqrstuvwxyz'
+ac_cr_AZ='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+ac_cr_09='0123456789'
+ac_cr_alnum=$ac_cr_az$ac_cr_AZ$ac_cr_09
+
+# Sed expression to map a string onto a valid sh and CPP variable names.
+ac_tr_sh="sed y%*+%pp%;s%[^_$ac_cr_alnum]%_%g"
+ac_tr_cpp="sed y%*$ac_cr_az%P$ac_cr_AZ%;s%[^_$ac_cr_alnum]%_%g"
+
+ac_unique_file="text/MGQuery.cpp"
+# Factoring default headers for most tests.
+ac_includes_default="\
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#if STDC_HEADERS
+# include <stdlib.h>
+# include <stddef.h>
+#else
+# if HAVE_STDLIB_H
+#  include <stdlib.h>
+# endif
+#endif
+#if HAVE_STRING_H
+# if !STDC_HEADERS && HAVE_MEMORY_H
+#  include <memory.h>
+# endif
+# include <string.h>
+#else
+# if HAVE_STRINGS_H
+#  include <strings.h>
+# endif
+#endif
+#if HAVE_INTTYPES_H
+# include <inttypes.h>
+#endif
+#if HAVE_UNISTD_H
+# include <unistd.h>
+#endif"
+
+# Initialize some variables set by options.
+ac_init_help=
+ac_init_version=false
+# The variables have the same names as the options, with
+# dashes changed to underlines.
+cache_file=/dev/null
+exec_prefix=NONE
+no_create=
+no_recursion=
+prefix=NONE
+program_prefix=NONE
+program_suffix=NONE
+program_transform_name=s,x,x,
+silent=
+site=
+srcdir=
+verbose=
+x_includes=NONE
+x_libraries=NONE
+
+# Installation directory options.
+# These are left unexpanded so users can "make install exec_prefix=/foo"
+# and all the variables that are supposed to be based on exec_prefix
+# by default will actually change.
+# Use braces instead of parens because sh, perl, etc. also accept them.
+bindir='${exec_prefix}/bin'
+sbindir='${exec_prefix}/sbin'
+libexecdir='${exec_prefix}/libexec'
+datadir='${prefix}/share'
+sysconfdir='${prefix}/etc'
+sharedstatedir='${prefix}/com'
+localstatedir='${prefix}/var'
+libdir='${exec_prefix}/lib'
+includedir='${prefix}/include'
+oldincludedir='/usr/include'
+infodir='${prefix}/info'
+mandir='${prefix}/man'
+
+# Identity of this package.
+PACKAGE_NAME=
+PACKAGE_TARNAME=
+PACKAGE_VERSION=
+PACKAGE_STRING=
+PACKAGE_BUGREPORT=
+
+ac_prev=
+for ac_option
+do
+  # If the previous option needs an argument, assign it.
+  if test -n "$ac_prev"; then
+    eval "$ac_prev=\$ac_option"
+    ac_prev=
+    continue
+  fi
+
+  ac_optarg=`expr "x$ac_option" : 'x[^=]*=\(.*\)'`
+
+  # Accept the important Cygnus configure options, so we can diagnose typos.
+
+  case $ac_option in
+
+  -bindir | --bindir | --bindi | --bind | --bin | --bi)
+    ac_prev=bindir ;;
+  -bindir=* | --bindir=* | --bindi=* | --bind=* | --bin=* | --bi=*)
+    bindir=$ac_optarg ;;
+
+  -build | --build | --buil | --bui | --bu)
+    ac_prev=build_alias ;;
+  -build=* | --build=* | --buil=* | --bui=* | --bu=*)
+    build_alias=$ac_optarg ;;
+
+  -cache-file | --cache-file | --cache-fil | --cache-fi \
+  | --cache-f | --cache- | --cache | --cach | --cac | --ca | --c)
+    ac_prev=cache_file ;;
+  -cache-file=* | --cache-file=* | --cache-fil=* | --cache-fi=* \
+  | --cache-f=* | --cache-=* | --cache=* | --cach=* | --cac=* | --ca=* | --c=*)
+    cache_file=$ac_optarg ;;
+
+  --config-cache | -C)
+    cache_file=config.cache ;;
+
+  -datadir | --datadir | --datadi | --datad | --data | --dat | --da)
+    ac_prev=datadir ;;
+  -datadir=* | --datadir=* | --datadi=* | --datad=* | --data=* | --dat=* \
+  | --da=*)
+    datadir=$ac_optarg ;;
+
+  -disable-* | --disable-*)
+    ac_feature=`expr "x$ac_option" : 'x-*disable-\(.*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_feature" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid feature name: $ac_feature" >&2
+   { (exit 1); exit 1; }; }
+    ac_feature=`echo $ac_feature | sed 's/-/_/g'`
+    eval "enable_$ac_feature=no" ;;
+
+  -enable-* | --enable-*)
+    ac_feature=`expr "x$ac_option" : 'x-*enable-\([^=]*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_feature" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid feature name: $ac_feature" >&2
+   { (exit 1); exit 1; }; }
+    ac_feature=`echo $ac_feature | sed 's/-/_/g'`
+    case $ac_option in
+      *=*) ac_optarg=`echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"`;;
+      *) ac_optarg=yes ;;
+    esac
+    eval "enable_$ac_feature='$ac_optarg'" ;;
+
+  -exec-prefix | --exec_prefix | --exec-prefix | --exec-prefi \
+  | --exec-pref | --exec-pre | --exec-pr | --exec-p | --exec- \
+  | --exec | --exe | --ex)
+    ac_prev=exec_prefix ;;
+  -exec-prefix=* | --exec_prefix=* | --exec-prefix=* | --exec-prefi=* \
+  | --exec-pref=* | --exec-pre=* | --exec-pr=* | --exec-p=* | --exec-=* \
+  | --exec=* | --exe=* | --ex=*)
+    exec_prefix=$ac_optarg ;;
+
+  -gas | --gas | --ga | --g)
+    # Obsolete; use --with-gas.
+    with_gas=yes ;;
+
+  -help | --help | --hel | --he | -h)
+    ac_init_help=long ;;
+  -help=r* | --help=r* | --hel=r* | --he=r* | -hr*)
+    ac_init_help=recursive ;;
+  -help=s* | --help=s* | --hel=s* | --he=s* | -hs*)
+    ac_init_help=short ;;
+
+  -host | --host | --hos | --ho)
+    ac_prev=host_alias ;;
+  -host=* | --host=* | --hos=* | --ho=*)
+    host_alias=$ac_optarg ;;
+
+  -includedir | --includedir | --includedi | --included | --include \
+  | --includ | --inclu | --incl | --inc)
+    ac_prev=includedir ;;
+  -includedir=* | --includedir=* | --includedi=* | --included=* | --include=* \
+  | --includ=* | --inclu=* | --incl=* | --inc=*)
+    includedir=$ac_optarg ;;
+
+  -infodir | --infodir | --infodi | --infod | --info | --inf)
+    ac_prev=infodir ;;
+  -infodir=* | --infodir=* | --infodi=* | --infod=* | --info=* | --inf=*)
+    infodir=$ac_optarg ;;
+
+  -libdir | --libdir | --libdi | --libd)
+    ac_prev=libdir ;;
+  -libdir=* | --libdir=* | --libdi=* | --libd=*)
+    libdir=$ac_optarg ;;
+
+  -libexecdir | --libexecdir | --libexecdi | --libexecd | --libexec \
+  | --libexe | --libex | --libe)
+    ac_prev=libexecdir ;;
+  -libexecdir=* | --libexecdir=* | --libexecdi=* | --libexecd=* | --libexec=* \
+  | --libexe=* | --libex=* | --libe=*)
+    libexecdir=$ac_optarg ;;
+
+  -localstatedir | --localstatedir | --localstatedi | --localstated \
+  | --localstate | --localstat | --localsta | --localst \
+  | --locals | --local | --loca | --loc | --lo)
+    ac_prev=localstatedir ;;
+  -localstatedir=* | --localstatedir=* | --localstatedi=* | --localstated=* \
+  | --localstate=* | --localstat=* | --localsta=* | --localst=* \
+  | --locals=* | --local=* | --loca=* | --loc=* | --lo=*)
+    localstatedir=$ac_optarg ;;
+
+  -mandir | --mandir | --mandi | --mand | --man | --ma | --m)
+    ac_prev=mandir ;;
+  -mandir=* | --mandir=* | --mandi=* | --mand=* | --man=* | --ma=* | --m=*)
+    mandir=$ac_optarg ;;
+
+  -nfp | --nfp | --nf)
+    # Obsolete; use --without-fp.
+    with_fp=no ;;
+
+  -no-create | --no-create | --no-creat | --no-crea | --no-cre \
+  | --no-cr | --no-c)
+    no_create=yes ;;
+
+  -no-recursion | --no-recursion | --no-recursio | --no-recursi \
+  | --no-recurs | --no-recur | --no-recu | --no-rec | --no-re | --no-r)
+    no_recursion=yes ;;
+
+  -oldincludedir | --oldincludedir | --oldincludedi | --oldincluded \
+  | --oldinclude | --oldinclud | --oldinclu | --oldincl | --oldinc \
+  | --oldin | --oldi | --old | --ol | --o)
+    ac_prev=oldincludedir ;;
+  -oldincludedir=* | --oldincludedir=* | --oldincludedi=* | --oldincluded=* \
+  | --oldinclude=* | --oldinclud=* | --oldinclu=* | --oldincl=* | --oldinc=* \
+  | --oldin=* | --oldi=* | --old=* | --ol=* | --o=*)
+    oldincludedir=$ac_optarg ;;
+
+  -prefix | --prefix | --prefi | --pref | --pre | --pr | --p)
+    ac_prev=prefix ;;
+  -prefix=* | --prefix=* | --prefi=* | --pref=* | --pre=* | --pr=* | --p=*)
+    prefix=$ac_optarg ;;
+
+  -program-prefix | --program-prefix | --program-prefi | --program-pref \
+  | --program-pre | --program-pr | --program-p)
+    ac_prev=program_prefix ;;
+  -program-prefix=* | --program-prefix=* | --program-prefi=* \
+  | --program-pref=* | --program-pre=* | --program-pr=* | --program-p=*)
+    program_prefix=$ac_optarg ;;
+
+  -program-suffix | --program-suffix | --program-suffi | --program-suff \
+  | --program-suf | --program-su | --program-s)
+    ac_prev=program_suffix ;;
+  -program-suffix=* | --program-suffix=* | --program-suffi=* \
+  | --program-suff=* | --program-suf=* | --program-su=* | --program-s=*)
+    program_suffix=$ac_optarg ;;
+
+  -program-transform-name | --program-transform-name \
+  | --program-transform-nam | --program-transform-na \
+  | --program-transform-n | --program-transform- \
+  | --program-transform | --program-transfor \
+  | --program-transfo | --program-transf \
+  | --program-trans | --program-tran \
+  | --progr-tra | --program-tr | --program-t)
+    ac_prev=program_transform_name ;;
+  -program-transform-name=* | --program-transform-name=* \
+  | --program-transform-nam=* | --program-transform-na=* \
+  | --program-transform-n=* | --program-transform-=* \
+  | --program-transform=* | --program-transfor=* \
+  | --program-transfo=* | --program-transf=* \
+  | --program-trans=* | --program-tran=* \
+  | --progr-tra=* | --program-tr=* | --program-t=*)
+    program_transform_name=$ac_optarg ;;
+
+  -q | -quiet | --quiet | --quie | --qui | --qu | --q \
+  | -silent | --silent | --silen | --sile | --sil)
+    silent=yes ;;
+
+  -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
+    ac_prev=sbindir ;;
+  -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
+  | --sbi=* | --sb=*)
+    sbindir=$ac_optarg ;;
+
+  -sharedstatedir | --sharedstatedir | --sharedstatedi \
+  | --sharedstated | --sharedstate | --sharedstat | --sharedsta \
+  | --sharedst | --shareds | --shared | --share | --shar \
+  | --sha | --sh)
+    ac_prev=sharedstatedir ;;
+  -sharedstatedir=* | --sharedstatedir=* | --sharedstatedi=* \
+  | --sharedstated=* | --sharedstate=* | --sharedstat=* | --sharedsta=* \
+  | --sharedst=* | --shareds=* | --shared=* | --share=* | --shar=* \
+  | --sha=* | --sh=*)
+    sharedstatedir=$ac_optarg ;;
+
+  -site | --site | --sit)
+    ac_prev=site ;;
+  -site=* | --site=* | --sit=*)
+    site=$ac_optarg ;;
+
+  -srcdir | --srcdir | --srcdi | --srcd | --src | --sr)
+    ac_prev=srcdir ;;
+  -srcdir=* | --srcdir=* | --srcdi=* | --srcd=* | --src=* | --sr=*)
+    srcdir=$ac_optarg ;;
+
+  -sysconfdir | --sysconfdir | --sysconfdi | --sysconfd | --sysconf \
+  | --syscon | --sysco | --sysc | --sys | --sy)
+    ac_prev=sysconfdir ;;
+  -sysconfdir=* | --sysconfdir=* | --sysconfdi=* | --sysconfd=* | --sysconf=* \
+  | --syscon=* | --sysco=* | --sysc=* | --sys=* | --sy=*)
+    sysconfdir=$ac_optarg ;;
+
+  -target | --target | --targe | --targ | --tar | --ta | --t)
+    ac_prev=target_alias ;;
+  -target=* | --target=* | --targe=* | --targ=* | --tar=* | --ta=* | --t=*)
+    target_alias=$ac_optarg ;;
+
+  -v | -verbose | --verbose | --verbos | --verbo | --verb)
+    verbose=yes ;;
+
+  -version | --version | --versio | --versi | --vers | -V)
+    ac_init_version=: ;;
+
+  -with-* | --with-*)
+    ac_package=`expr "x$ac_option" : 'x-*with-\([^=]*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_package" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid package name: $ac_package" >&2
+   { (exit 1); exit 1; }; }
+    ac_package=`echo $ac_package| sed 's/-/_/g'`
+    case $ac_option in
+      *=*) ac_optarg=`echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"`;;
+      *) ac_optarg=yes ;;
+    esac
+    eval "with_$ac_package='$ac_optarg'" ;;
+
+  -without-* | --without-*)
+    ac_package=`expr "x$ac_option" : 'x-*without-\(.*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_package" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid package name: $ac_package" >&2
+   { (exit 1); exit 1; }; }
+    ac_package=`echo $ac_package | sed 's/-/_/g'`
+    eval "with_$ac_package=no" ;;
+
+  --x)
+    # Obsolete; use --with-x.
+    with_x=yes ;;
+
+  -x-includes | --x-includes | --x-include | --x-includ | --x-inclu \
+  | --x-incl | --x-inc | --x-in | --x-i)
+    ac_prev=x_includes ;;
+  -x-includes=* | --x-includes=* | --x-include=* | --x-includ=* | --x-inclu=* \
+  | --x-incl=* | --x-inc=* | --x-in=* | --x-i=*)
+    x_includes=$ac_optarg ;;
+
+  -x-libraries | --x-libraries | --x-librarie | --x-librari \
+  | --x-librar | --x-libra | --x-libr | --x-lib | --x-li | --x-l)
+    ac_prev=x_libraries ;;
+  -x-libraries=* | --x-libraries=* | --x-librarie=* | --x-librari=* \
+  | --x-librar=* | --x-libra=* | --x-libr=* | --x-lib=* | --x-li=* | --x-l=*)
+    x_libraries=$ac_optarg ;;
+
+  -*) { echo "$as_me: error: unrecognized option: $ac_option
+Try \`$0 --help' for more information." >&2
+   { (exit 1); exit 1; }; }
+    ;;
+
+  *=*)
+    ac_envvar=`expr "x$ac_option" : 'x\([^=]*\)='`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_envvar" : ".*[^_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid variable name: $ac_envvar" >&2
+   { (exit 1); exit 1; }; }
+    ac_optarg=`echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"`
+    eval "$ac_envvar='$ac_optarg'"
+    export $ac_envvar ;;
+
+  *)
+    # FIXME: should be removed in autoconf 3.0.
+    echo "$as_me: WARNING: you should use --build, --host, --target" >&2
+    expr "x$ac_option" : ".*[^-._$ac_cr_alnum]" >/dev/null &&
+      echo "$as_me: WARNING: invalid host type: $ac_option" >&2
+    : ${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}
+    ;;
+
+  esac
+done
+
+if test -n "$ac_prev"; then
+  ac_option=--`echo $ac_prev | sed 's/_/-/g'`
+  { echo "$as_me: error: missing argument to $ac_option" >&2
+   { (exit 1); exit 1; }; }
+fi
+
+# Be sure to have absolute paths.
+for ac_var in bindir sbindir libexecdir datadir sysconfdir sharedstatedir \
+              localstatedir libdir includedir oldincludedir infodir mandir \
+              exec_prefix prefix
+do
+  eval ac_val=$`echo $ac_var`
+  case $ac_val in
+    [\\/$]* | ?:[\\/]* ) ;;
+    NONE ) ;;
+    *)  { echo "$as_me: error: expected an absolute path for --$ac_var: $ac_val" >&2
+   { (exit 1); exit 1; }; };;
+  esac
+done
+
+# There might be people who depend on the old broken behavior: `$host'
+# used to hold the argument of --host etc.
+build=$build_alias
+host=$host_alias
+target=$target_alias
+
+# FIXME: should be removed in autoconf 3.0.
+if test "x$host_alias" != x; then
+  if test "x$build_alias" = x; then
+    cross_compiling=maybe
+    echo "$as_me: WARNING: If you wanted to set the --build type, don't use --host.
+    If a cross compiler is detected then cross compile mode will be used." >&2
+  elif test "x$build_alias" != "x$host_alias"; then
+    cross_compiling=yes
+  fi
+fi
+
+ac_tool_prefix=
+test -n "$host_alias" && ac_tool_prefix=$host_alias-
+
+test "$silent" = yes && exec 6>/dev/null
+
+# Find the source files, if location was not specified.
+if test -z "$srcdir"; then
+  ac_srcdir_defaulted=yes
+  # Try the directory containing this script, then its parent.
+  ac_prog=$0
+  ac_confdir=`echo "$ac_prog" | sed 's%[\\/][^\\/][^\\/]*$%%'`
+  test "x$ac_confdir" = "x$ac_prog" && ac_confdir=.
+  srcdir=$ac_confdir
+  if test ! -r $srcdir/$ac_unique_file; then
+    srcdir=..
+  fi
+else
+  ac_srcdir_defaulted=no
+fi
+if test ! -r $srcdir/$ac_unique_file; then
+  if test "$ac_srcdir_defaulted" = yes; then
+    { echo "$as_me: error: cannot find sources in $ac_confdir or .." >&2
+   { (exit 1); exit 1; }; }
+  else
+    { echo "$as_me: error: cannot find sources in $srcdir" >&2
+   { (exit 1); exit 1; }; }
+  fi
+fi
+srcdir=`echo "$srcdir" | sed 's%\([^\\/]\)[\\/]*$%\1%'`
+ac_env_build_alias_set=${build_alias+set}
+ac_env_build_alias_value=$build_alias
+ac_cv_env_build_alias_set=${build_alias+set}
+ac_cv_env_build_alias_value=$build_alias
+ac_env_host_alias_set=${host_alias+set}
+ac_env_host_alias_value=$host_alias
+ac_cv_env_host_alias_set=${host_alias+set}
+ac_cv_env_host_alias_value=$host_alias
+ac_env_target_alias_set=${target_alias+set}
+ac_env_target_alias_value=$target_alias
+ac_cv_env_target_alias_set=${target_alias+set}
+ac_cv_env_target_alias_value=$target_alias
+ac_env_CXX_set=${CXX+set}
+ac_env_CXX_value=$CXX
+ac_cv_env_CXX_set=${CXX+set}
+ac_cv_env_CXX_value=$CXX
+ac_env_CXXFLAGS_set=${CXXFLAGS+set}
+ac_env_CXXFLAGS_value=$CXXFLAGS
+ac_cv_env_CXXFLAGS_set=${CXXFLAGS+set}
+ac_cv_env_CXXFLAGS_value=$CXXFLAGS
+ac_env_LDFLAGS_set=${LDFLAGS+set}
+ac_env_LDFLAGS_value=$LDFLAGS
+ac_cv_env_LDFLAGS_set=${LDFLAGS+set}
+ac_cv_env_LDFLAGS_value=$LDFLAGS
+ac_env_CPPFLAGS_set=${CPPFLAGS+set}
+ac_env_CPPFLAGS_value=$CPPFLAGS
+ac_cv_env_CPPFLAGS_set=${CPPFLAGS+set}
+ac_cv_env_CPPFLAGS_value=$CPPFLAGS
+ac_env_CC_set=${CC+set}
+ac_env_CC_value=$CC
+ac_cv_env_CC_set=${CC+set}
+ac_cv_env_CC_value=$CC
+ac_env_CFLAGS_set=${CFLAGS+set}
+ac_env_CFLAGS_value=$CFLAGS
+ac_cv_env_CFLAGS_set=${CFLAGS+set}
+ac_cv_env_CFLAGS_value=$CFLAGS
+ac_env_CPP_set=${CPP+set}
+ac_env_CPP_value=$CPP
+ac_cv_env_CPP_set=${CPP+set}
+ac_cv_env_CPP_value=$CPP
+
+#
+# Report the --help message.
+#
+if test "$ac_init_help" = "long"; then
+  # Omit some internal or obsolete options to make the list less imposing.
+  # This message is too long to be a string in the A/UX 3.1 sh.
+  cat <<EOF
+\`configure' configures this package to adapt to many kinds of systems.
+
+Usage: $0 [OPTION]... [VAR=VALUE]...
+
+To assign environment variables (e.g., CC, CFLAGS...), specify them as
+VAR=VALUE.  See below for descriptions of some of the useful variables.
+
+Defaults for the options are specified in brackets.
+
+Configuration:
+  -h, --help              display this help and exit
+      --help=short        display options specific to this package
+      --help=recursive    display the short help of all the included packages
+  -V, --version           display version information and exit
+  -q, --quiet, --silent   do not print \`checking...' messages
+      --cache-file=FILE   cache test results in FILE [disabled]
+  -C, --config-cache      alias for \`--cache-file=config.cache'
+  -n, --no-create         do not create output files
+      --srcdir=DIR        find the sources in DIR [configure dir or \`..']
+
+EOF
+
+  cat <<EOF
+Installation directories:
+  --prefix=PREFIX         install architecture-independent files in PREFIX
+                          [$ac_default_prefix]
+  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
+                          [PREFIX]
+
+By default, \`make install' will install all the files in
+\`$ac_default_prefix/bin', \`$ac_default_prefix/lib' etc.  You can specify
+an installation prefix other than \`$ac_default_prefix' using \`--prefix',
+for instance \`--prefix=\$HOME'.
+
+For better control, use the options below.
+
+Fine tuning of the installation directories:
+  --bindir=DIR           user executables [EPREFIX/bin]
+  --sbindir=DIR          system admin executables [EPREFIX/sbin]
+  --libexecdir=DIR       program executables [EPREFIX/libexec]
+  --datadir=DIR          read-only architecture-independent data [PREFIX/share]
+  --sysconfdir=DIR       read-only single-machine data [PREFIX/etc]
+  --sharedstatedir=DIR   modifiable architecture-independent data [PREFIX/com]
+  --localstatedir=DIR    modifiable single-machine data [PREFIX/var]
+  --libdir=DIR           object code libraries [EPREFIX/lib]
+  --includedir=DIR       C header files [PREFIX/include]
+  --oldincludedir=DIR    C header files for non-gcc [/usr/include]
+  --infodir=DIR          info documentation [PREFIX/info]
+  --mandir=DIR           man documentation [PREFIX/man]
+EOF
+
+  cat <<\EOF
+
+Program names:
+  --program-prefix=PREFIX            prepend PREFIX to installed program names
+  --program-suffix=SUFFIX            append SUFFIX to installed program names
+  --program-transform-name=PROGRAM   run sed PROGRAM on installed program names
+EOF
+fi
+
+if test -n "$ac_init_help"; then
+
+  cat <<\EOF
+
+Optional Features:
+  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
+  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
+  --enable-override-longlong
+                          overide use of long long with GCC
+
+Optional Packages:
+  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
+  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
+  --with-dmalloc          use dmalloc, as in
+                          ftp://ftp.letters.com/src/dmalloc/dmalloc.tar.gz
+  --with-regex            use older regex in lieu of GNU rx for matching
+  --with-gnu-readline     compile with GNU readline support
+
+Some influential environment variables:
+  CXX         C++ compiler command
+  CXXFLAGS    C++ compiler flags
+  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
+              nonstandard directory <lib dir>
+  CPPFLAGS    C/C++ preprocessor flags, e.g. -I<include dir> if you have
+              headers in a nonstandard directory <include dir>
+  CC          C compiler command
+  CFLAGS      C compiler flags
+  CPP         C preprocessor
+
+Use these variables to override the choices made by `configure' or to help
+it to find libraries and programs with nonstandard names/locations.
+
+EOF
+fi
+
+if test "$ac_init_help" = "recursive"; then
+  # If there are subdirs, report their specific --help.
+  ac_popdir=`pwd`
+  for ac_subdir in : $ac_subdirs_all; do test "x$ac_subdir" = x: && continue
+    cd $ac_subdir
+    # A "../" for each directory in /$ac_subdir.
+    ac_dots=`echo $ac_subdir |
+             sed 's,^\./,,;s,[^/]$,&/,;s,[^/]*/,../,g'`
+
+    case $srcdir in
+    .) # No --srcdir option.  We are building in place.
+      ac_sub_srcdir=$srcdir ;;
+    [\\/]* | ?:[\\/]* ) # Absolute path.
+      ac_sub_srcdir=$srcdir/$ac_subdir ;;
+    *) # Relative path.
+      ac_sub_srcdir=$ac_dots$srcdir/$ac_subdir ;;
+    esac
+
+    # Check for guested configure; otherwise get Cygnus style configure.
+    if test -f $ac_sub_srcdir/configure.gnu; then
+      echo
+      $SHELL $ac_sub_srcdir/configure.gnu  --help=recursive
+    elif test -f $ac_sub_srcdir/configure; then
+      echo
+      $SHELL $ac_sub_srcdir/configure  --help=recursive
+    elif test -f $ac_sub_srcdir/configure.ac ||
+           test -f $ac_sub_srcdir/configure.in; then
+      echo
+      $ac_configure --help
+    else
+      echo "$as_me: WARNING: no configuration information is in $ac_subdir" >&2
+    fi
+    cd $ac_popdir
+  done
+fi
+
+test -n "$ac_init_help" && exit 0
+if $ac_init_version; then
+  cat <<\EOF
+
+Copyright 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001
+Free Software Foundation, Inc.
+This configure script is free software; the Free Software Foundation
+gives unlimited permission to copy, distribute and modify it.
+EOF
+  exit 0
+fi
+exec 5>config.log
+cat >&5 <<EOF
+This file contains any messages produced by compilers while
+running configure, to aid debugging if configure makes a mistake.
+
+It was created by $as_me, which was
+generated by GNU Autoconf 2.50.  Invocation command line was
+
+  $ $0 $@
+
+EOF
+{
+cat <<_ASUNAME
+## ---------- ##
+## Platform.  ##
+## ---------- ##
+
+hostname = `(hostname || uname -n) 2>/dev/null | sed 1q`
+uname -m = `(uname -m) 2>/dev/null || echo unknown`
+uname -r = `(uname -r) 2>/dev/null || echo unknown`
+uname -s = `(uname -s) 2>/dev/null || echo unknown`
+uname -v = `(uname -v) 2>/dev/null || echo unknown`
+
+/usr/bin/uname -p = `(/usr/bin/uname -p) 2>/dev/null || echo unknown`
+/bin/uname -X     = `(/bin/uname -X) 2>/dev/null     || echo unknown`
+
+/bin/arch              = `(/bin/arch) 2>/dev/null              || echo unknown`
+/usr/bin/arch -k       = `(/usr/bin/arch -k) 2>/dev/null       || echo unknown`
+/usr/convex/getsysinfo = `(/usr/convex/getsysinfo) 2>/dev/null || echo unknown`
+hostinfo               = `(hostinfo) 2>/dev/null               || echo unknown`
+/bin/machine           = `(/bin/machine) 2>/dev/null           || echo unknown`
+/usr/bin/oslevel       = `(/usr/bin/oslevel) 2>/dev/null       || echo unknown`
+/bin/universe          = `(/bin/universe) 2>/dev/null          || echo unknown`
+
+PATH = $PATH
+
+_ASUNAME
+} >&5
+
+cat >&5 <<EOF
+## ------------ ##
+## Core tests.  ##
+## ------------ ##
+
+EOF
+
+# Keep a trace of the command line.
+# Strip out --no-create and --no-recursion so they do not pile up.
+# Also quote any args containing shell meta-characters.
+ac_configure_args=
+ac_sep=
+for ac_arg
+do
+  case $ac_arg in
+  -no-create | --no-create | --no-creat | --no-crea | --no-cre \
+  | --no-cr | --no-c) ;;
+  -no-recursion | --no-recursion | --no-recursio | --no-recursi \
+  | --no-recurs | --no-recur | --no-recu | --no-rec | --no-re | --no-r) ;;
+  *" "*|*"	"*|*[\[\]\~\#\$\^\&\*\(\)\{\}\\\|\;\<\>\?\"\']*)
+    ac_arg=`echo "$ac_arg" | sed "s/'/'\\\\\\\\''/g"`
+    ac_configure_args="$ac_configure_args$ac_sep'$ac_arg'"
+    ac_sep=" " ;;
+  *) ac_configure_args="$ac_configure_args$ac_sep$ac_arg"
+     ac_sep=" " ;;
+  esac
+  # Get rid of the leading space.
+done
+
+# When interrupted or exit'd, cleanup temporary files, and complete
+# config.log.  We remove comments because anyway the quotes in there
+# would cause problems or look ugly.
+trap 'exit_status=$?
+  # Save into config.log some information that might help in debugging.
+  echo >&5
+  echo "## ----------------- ##" >&5
+  echo "## Cache variables.  ##" >&5
+  echo "## ----------------- ##" >&5
+  echo >&5
+  # The following way of writing the cache mishandles newlines in values,
+{
+  (set) 2>&1 |
+    case `(ac_space='"'"' '"'"'; set | grep ac_space) 2>&1` in
+    *ac_space=\ *)
+      sed -n \
+        "s/'"'"'/'"'"'\\\\'"'"''"'"'/g;
+    	  s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1='"'"'\\2'"'"'/p"
+      ;;
+    *)
+      sed -n \
+        "s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1=\\2/p"
+      ;;
+    esac;
+} >&5
+  sed "/^$/d" confdefs.h >conftest.log
+  if test -s conftest.log; then
+    echo >&5
+    echo "## ------------ ##" >&5
+    echo "## confdefs.h.  ##" >&5
+    echo "## ------------ ##" >&5
+    echo >&5
+    cat conftest.log >&5
+  fi
+  (echo; echo) >&5
+  test "$ac_signal" != 0 &&
+    echo "$as_me: caught signal $ac_signal" >&5
+  echo "$as_me: exit $exit_status" >&5
+  rm -rf conftest* confdefs* core core.* *.core conf$$* $ac_clean_files &&
+    exit $exit_status
+     ' 0
+for ac_signal in 1 2 13 15; do
+  trap 'ac_status=$?; ac_signal='$ac_signal'; { (exit $ac_status); exit $ac_status; }' $ac_signal
+done
+ac_signal=0
+
+# confdefs.h avoids OS command line length limits that DEFS can exceed.
+rm -rf conftest* confdefs.h
+# AIX cpp loses on an empty file, so make sure it contains at least a newline.
+echo >confdefs.h
+
+# Let the site file select an alternate cache file if it wants to.
+# Prefer explicitly selected file to automatically selected ones.
+if test -z "$CONFIG_SITE"; then
+  if test "x$prefix" != xNONE; then
+    CONFIG_SITE="$prefix/share/config.site $prefix/etc/config.site"
+  else
+    CONFIG_SITE="$ac_default_prefix/share/config.site $ac_default_prefix/etc/config.site"
+  fi
+fi
+for ac_site_file in $CONFIG_SITE; do
+  if test -r "$ac_site_file"; then
+    { echo "$as_me:865: loading site script $ac_site_file" >&5
+echo "$as_me: loading site script $ac_site_file" >&6;}
+    cat "$ac_site_file" >&5
+    . "$ac_site_file"
+  fi
+done
+
+if test -r "$cache_file"; then
+  # Some versions of bash will fail to source /dev/null (special
+  # files actually), so we avoid doing that.
+  if test -f "$cache_file"; then
+    { echo "$as_me:876: loading cache $cache_file" >&5
+echo "$as_me: loading cache $cache_file" >&6;}
+    case $cache_file in
+      [\\/]* | ?:[\\/]* ) . $cache_file;;
+      *)                      . ./$cache_file;;
+    esac
+  fi
+else
+  { echo "$as_me:884: creating cache $cache_file" >&5
+echo "$as_me: creating cache $cache_file" >&6;}
+  >$cache_file
+fi
+
+# Check that the precious variables saved in the cache have kept the same
+# value.
+ac_suggest_removing_cache=false
+for ac_var in `(set) 2>&1 |
+               sed -n 's/^ac_env_\([a-zA-Z_0-9]*\)_set=.*/\1/p'`; do
+  eval ac_old_set=\$ac_cv_env_${ac_var}_set
+  eval ac_new_set=\$ac_env_${ac_var}_set
+  eval ac_old_val="\$ac_cv_env_${ac_var}_value"
+  eval ac_new_val="\$ac_env_${ac_var}_value"
+  case $ac_old_set,$ac_new_set in
+    set,)
+      { echo "$as_me:900: WARNING: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&5
+echo "$as_me: WARNING: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&2;}
+      ac_suggest_removing_cache=: ;;
+    ,set)
+      { echo "$as_me:904: WARNING: \`$ac_var' was not set in the previous run" >&5
+echo "$as_me: WARNING: \`$ac_var' was not set in the previous run" >&2;}
+      ac_suggest_removing_cache=: ;;
+    ,);;
+    *)
+      if test "x$ac_old_val" != "x$ac_new_val"; then
+        { echo "$as_me:910: WARNING: \`$ac_var' has changed since the previous run:" >&5
+echo "$as_me: WARNING: \`$ac_var' has changed since the previous run:" >&2;}
+        { echo "$as_me:912: WARNING:   former value:  $ac_old_val" >&5
+echo "$as_me: WARNING:   former value:  $ac_old_val" >&2;}
+        { echo "$as_me:914: WARNING:   current value: $ac_new_val" >&5
+echo "$as_me: WARNING:   current value: $ac_new_val" >&2;}
+        ac_suggest_removing_cache=:
+      fi;;
+  esac
+done
+if $ac_suggest_removing_cache; then
+  { echo "$as_me:921: WARNING: changes in the environment can compromise the build" >&5
+echo "$as_me: WARNING: changes in the environment can compromise the build" >&2;}
+  { echo "$as_me:923: WARNING: consider removing $cache_file and starting over" >&5
+echo "$as_me: WARNING: consider removing $cache_file and starting over" >&2;}
+fi
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+case `echo "testing\c"; echo 1,2,3`,`echo -n testing; echo 1,2,3` in
+  *c*,-n*) ECHO_N= ECHO_C='
+' ECHO_T='	' ;;
+  *c*,*  ) ECHO_N=-n ECHO_C= ECHO_T= ;;
+  *)      ECHO_N= ECHO_C='\c' ECHO_T= ;;
+esac
+echo "#! $SHELL" >conftest.sh
+echo  "exit 0"   >>conftest.sh
+chmod +x conftest.sh
+if { (echo "$as_me:942: PATH=\".;.\"; conftest.sh") >&5
+  (PATH=".;."; conftest.sh) 2>&5
+  ac_status=$?
+  echo "$as_me:945: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  ac_path_separator=';'
+else
+  ac_path_separator=:
+fi
+PATH_SEPARATOR="$ac_path_separator"
+rm -f conftest.sh
+
+ac_config_headers="$ac_config_headers config.h"
+
+if test "$program_transform_name" = s,x,x,; then
+  program_transform_name=
+else
+  # Double any \ or $.  echo might interpret backslashes.
+  cat <<\EOF >conftest.sed
+s,\\,\\\\,g; s,\$,$$,g
+EOF
+  program_transform_name=`echo $program_transform_name | sed -f conftest.sed`
+  rm -f conftest.sed
+fi
+test "$program_prefix" != NONE &&
+  program_transform_name="s,^,${program_prefix},;$program_transform_name"
+# Use a double $ so make ignores it.
+test "$program_suffix" != NONE &&
+  program_transform_name="s,\$\$,${program_suffix},;$program_transform_name"
+
+# sed with no file args requires a program.
+test -z "$program_transform_name" && program_transform_name="s,x,x,"
+
+PACKAGE=mgpp
+VERSION=1.3
+
+cat >>confdefs.h <<EOF
+#define PACKAGE "$PACKAGE"
+EOF
+
+cat >>confdefs.h <<EOF
+#define VERSION "$VERSION"
+EOF
+
+ac_ext=cc
+ac_cpp='$CXXCPP $CPPFLAGS'
+ac_compile='$CXX -c $CXXFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CXX -o conftest$ac_exeext $CXXFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_cxx_compiler_gnu
+if test -n "$ac_tool_prefix"; then
+  for ac_prog in $CCC g++ c++ gpp aCC CC cxx cc++ cl KCC RCC xlC_r xlC
+  do
+    # Extract the first word of "$ac_tool_prefix$ac_prog", so it can be a program name with args.
+set dummy $ac_tool_prefix$ac_prog; ac_word=$2
+echo "$as_me:996: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CXX+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CXX"; then
+  ac_cv_prog_CXX="$CXX" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CXX="$ac_tool_prefix$ac_prog"
+echo "$as_me:1011: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CXX=$ac_cv_prog_CXX
+if test -n "$CXX"; then
+  echo "$as_me:1019: result: $CXX" >&5
+echo "${ECHO_T}$CXX" >&6
+else
+  echo "$as_me:1022: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+    test -n "$CXX" && break
+  done
+fi
+if test -z "$CXX"; then
+  ac_ct_CXX=$CXX
+  for ac_prog in $CCC g++ c++ gpp aCC CC cxx cc++ cl KCC RCC xlC_r xlC
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1035: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CXX+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CXX"; then
+  ac_cv_prog_ac_ct_CXX="$ac_ct_CXX" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CXX="$ac_prog"
+echo "$as_me:1050: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CXX=$ac_cv_prog_ac_ct_CXX
+if test -n "$ac_ct_CXX"; then
+  echo "$as_me:1058: result: $ac_ct_CXX" >&5
+echo "${ECHO_T}$ac_ct_CXX" >&6
+else
+  echo "$as_me:1061: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$ac_ct_CXX" && break
+done
+test -n "$ac_ct_CXX" || ac_ct_CXX="g++"
+
+  CXX=$ac_ct_CXX
+fi
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 1073 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+ac_clean_files_save=$ac_clean_files
+ac_clean_files="$ac_clean_files a.out a.exe"
+# Try to create an executable without -o first, disregard a.out.
+# It will help us diagnose broken compiler, and finding out an intuition
+# of exeext.
+echo "$as_me:1089: checking for C++ compiler default output" >&5
+echo $ECHO_N "checking for C++ compiler default output... $ECHO_C" >&6
+ac_link_default=`echo "$ac_link" | sed 's/ -o *conftest[^ ]*//'`
+if { (eval echo "$as_me:1092: \"$ac_link_default\"") >&5
+  (eval $ac_link_default) 2>&5
+  ac_status=$?
+  echo "$as_me:1095: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  for ac_file in `ls a.exe conftest.exe a.* conftest conftest.* 2>/dev/null`; do
+  case $ac_file in
+    *.$ac_ext | *.o | *.obj | *.xcoff | *.tds | *.d | *.pdb ) ;;
+    a.out ) # We found the default executable, but exeext='' is most
+            # certainly right.
+            break;;
+    *.* ) ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'`
+          export ac_cv_exeext
+          break;;
+    * ) break;;
+  esac
+done
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+{ { echo "$as_me:1112: error: C++ compiler cannot create executables" >&5
+echo "$as_me: error: C++ compiler cannot create executables" >&2;}
+   { (exit 77); exit 77; }; }
+fi
+
+ac_exeext=$ac_cv_exeext
+echo "$as_me:1118: result: $ac_file" >&5
+echo "${ECHO_T}$ac_file" >&6
+
+# Check the compiler produces executables we can run.  If not, either
+# the compiler is broken, or we cross compile.
+echo "$as_me:1123: checking whether the C++ compiler works" >&5
+echo $ECHO_N "checking whether the C++ compiler works... $ECHO_C" >&6
+# FIXME: These cross compiler hacks should be removed for Autoconf 3.0
+# If not cross compiling, check that we can run a simple program.
+if test "$cross_compiling" != yes; then
+  if { ac_try='./$ac_file'
+  { (eval echo "$as_me:1129: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1132: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+    cross_compiling=no
+  else
+    if test "$cross_compiling" = maybe; then
+	cross_compiling=yes
+    else
+	{ { echo "$as_me:1139: error: cannot run C++ compiled programs.
+If you meant to cross compile, use \`--host'." >&5
+echo "$as_me: error: cannot run C++ compiled programs.
+If you meant to cross compile, use \`--host'." >&2;}
+   { (exit 1); exit 1; }; }
+    fi
+  fi
+fi
+echo "$as_me:1147: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+
+rm -f a.out a.exe conftest$ac_cv_exeext
+ac_clean_files=$ac_clean_files_save
+# Check the compiler produces executables we can run.  If not, either
+# the compiler is broken, or we cross compile.
+echo "$as_me:1154: checking whether we are cross compiling" >&5
+echo $ECHO_N "checking whether we are cross compiling... $ECHO_C" >&6
+echo "$as_me:1156: result: $cross_compiling" >&5
+echo "${ECHO_T}$cross_compiling" >&6
+
+echo "$as_me:1159: checking for executable suffix" >&5
+echo $ECHO_N "checking for executable suffix... $ECHO_C" >&6
+if { (eval echo "$as_me:1161: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:1164: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  # If both `conftest.exe' and `conftest' are `present' (well, observable)
+# catch `conftest.exe'.  For instance with Cygwin, `ls conftest' will
+# work properly (i.e., refer to `conftest.exe'), while it won't with
+# `rm'.
+for ac_file in `(ls conftest.exe; ls conftest; ls conftest.*) 2>/dev/null`; do
+  case $ac_file in
+    *.$ac_ext | *.o | *.obj | *.xcoff | *.tds | *.d | *.pdb ) ;;
+    *.* ) ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'`
+          export ac_cv_exeext
+          break;;
+    * ) break;;
+  esac
+done
+else
+  { { echo "$as_me:1180: error: cannot compute EXEEXT: cannot compile and link" >&5
+echo "$as_me: error: cannot compute EXEEXT: cannot compile and link" >&2;}
+   { (exit 1); exit 1; }; }
+fi
+
+rm -f conftest$ac_cv_exeext
+echo "$as_me:1186: result: $ac_cv_exeext" >&5
+echo "${ECHO_T}$ac_cv_exeext" >&6
+
+rm -f conftest.$ac_ext
+EXEEXT=$ac_cv_exeext
+ac_exeext=$EXEEXT
+echo "$as_me:1192: checking for object suffix" >&5
+echo $ECHO_N "checking for object suffix... $ECHO_C" >&6
+if test "${ac_cv_objext+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1198 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.o conftest.obj
+if { (eval echo "$as_me:1210: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1213: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  for ac_file in `(ls conftest.o conftest.obj; ls conftest.*) 2>/dev/null`; do
+  case $ac_file in
+    *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb ) ;;
+    *) ac_cv_objext=`expr "$ac_file" : '.*\.\(.*\)'`
+       break;;
+  esac
+done
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+{ { echo "$as_me:1225: error: cannot compute OBJEXT: cannot compile" >&5
+echo "$as_me: error: cannot compute OBJEXT: cannot compile" >&2;}
+   { (exit 1); exit 1; }; }
+fi
+
+rm -f conftest.$ac_cv_objext conftest.$ac_ext
+fi
+echo "$as_me:1232: result: $ac_cv_objext" >&5
+echo "${ECHO_T}$ac_cv_objext" >&6
+OBJEXT=$ac_cv_objext
+ac_objext=$OBJEXT
+echo "$as_me:1236: checking whether we are using the GNU C++ compiler" >&5
+echo $ECHO_N "checking whether we are using the GNU C++ compiler... $ECHO_C" >&6
+if test "${ac_cv_cxx_compiler_gnu+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1242 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+#ifndef __GNUC__
+       choke me
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1257: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1260: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1263: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1266: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_compiler_gnu=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_compiler_gnu=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+ac_cv_cxx_compiler_gnu=$ac_compiler_gnu
+
+fi
+echo "$as_me:1278: result: $ac_cv_cxx_compiler_gnu" >&5
+echo "${ECHO_T}$ac_cv_cxx_compiler_gnu" >&6
+GXX=`test $ac_compiler_gnu = yes && echo yes`
+ac_test_CXXFLAGS=${CXXFLAGS+set}
+ac_save_CXXFLAGS=$CXXFLAGS
+CXXFLAGS="-g"
+echo "$as_me:1284: checking whether $CXX accepts -g" >&5
+echo $ECHO_N "checking whether $CXX accepts -g... $ECHO_C" >&6
+if test "${ac_cv_prog_cxx_g+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1290 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1302: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1305: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1308: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1311: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cxx_g=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_prog_cxx_g=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:1321: result: $ac_cv_prog_cxx_g" >&5
+echo "${ECHO_T}$ac_cv_prog_cxx_g" >&6
+if test "$ac_test_CXXFLAGS" = set; then
+  CXXFLAGS=$ac_save_CXXFLAGS
+elif test $ac_cv_prog_cxx_g = yes; then
+  if test "$GXX" = yes; then
+    CXXFLAGS="-g -O2"
+  else
+    CXXFLAGS="-g"
+  fi
+else
+  if test "$GXX" = yes; then
+    CXXFLAGS="-O2"
+  else
+    CXXFLAGS=
+  fi
+fi
+for ac_declaration in \
+   ''\
+   '#include <stdlib.h>' \
+   'extern "C" void std::exit (int) throw (); using std::exit;' \
+   'extern "C" void std::exit (int); using std::exit;' \
+   'extern "C" void exit (int) throw ();' \
+   'extern "C" void exit (int);' \
+   'void exit (int);'
+do
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1348 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1361: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1364: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1367: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1370: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+continue
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1380 "configure"
+#include "confdefs.h"
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1392: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1395: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1398: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1401: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+echo '#ifdef __cplusplus' >>confdefs.h
+echo $ac_declaration      >>confdefs.h
+echo '#endif'             >>confdefs.h
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+for ac_prog in mawk gawk nawk awk
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1424: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_AWK+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$AWK"; then
+  ac_cv_prog_AWK="$AWK" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_AWK="$ac_prog"
+echo "$as_me:1439: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+AWK=$ac_cv_prog_AWK
+if test -n "$AWK"; then
+  echo "$as_me:1447: result: $AWK" >&5
+echo "${ECHO_T}$AWK" >&6
+else
+  echo "$as_me:1450: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$AWK" && break
+done
+
+for ac_prog in 'bison -y' byacc
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1461: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_YACC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$YACC"; then
+  ac_cv_prog_YACC="$YACC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_YACC="$ac_prog"
+echo "$as_me:1476: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+YACC=$ac_cv_prog_YACC
+if test -n "$YACC"; then
+  echo "$as_me:1484: result: $YACC" >&5
+echo "${ECHO_T}$YACC" >&6
+else
+  echo "$as_me:1487: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$YACC" && break
+done
+test -n "$YACC" || YACC="yacc"
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}gcc", so it can be a program name with args.
+set dummy ${ac_tool_prefix}gcc; ac_word=$2
+echo "$as_me:1503: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CC="${ac_tool_prefix}gcc"
+echo "$as_me:1518: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1526: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1529: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$ac_cv_prog_CC"; then
+  ac_ct_CC=$CC
+  # Extract the first word of "gcc", so it can be a program name with args.
+set dummy gcc; ac_word=$2
+echo "$as_me:1538: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CC"; then
+  ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CC="gcc"
+echo "$as_me:1553: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CC=$ac_cv_prog_ac_ct_CC
+if test -n "$ac_ct_CC"; then
+  echo "$as_me:1561: result: $ac_ct_CC" >&5
+echo "${ECHO_T}$ac_ct_CC" >&6
+else
+  echo "$as_me:1564: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  CC=$ac_ct_CC
+else
+  CC="$ac_cv_prog_CC"
+fi
+
+if test -z "$CC"; then
+  if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}cc", so it can be a program name with args.
+set dummy ${ac_tool_prefix}cc; ac_word=$2
+echo "$as_me:1577: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CC="${ac_tool_prefix}cc"
+echo "$as_me:1592: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1600: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1603: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$ac_cv_prog_CC"; then
+  ac_ct_CC=$CC
+  # Extract the first word of "cc", so it can be a program name with args.
+set dummy cc; ac_word=$2
+echo "$as_me:1612: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CC"; then
+  ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CC="cc"
+echo "$as_me:1627: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CC=$ac_cv_prog_ac_ct_CC
+if test -n "$ac_ct_CC"; then
+  echo "$as_me:1635: result: $ac_ct_CC" >&5
+echo "${ECHO_T}$ac_ct_CC" >&6
+else
+  echo "$as_me:1638: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  CC=$ac_ct_CC
+else
+  CC="$ac_cv_prog_CC"
+fi
+
+fi
+if test -z "$CC"; then
+  # Extract the first word of "cc", so it can be a program name with args.
+set dummy cc; ac_word=$2
+echo "$as_me:1651: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_prog_rejected=no
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+if test "$ac_dir/$ac_word" = "/usr/ucb/cc"; then
+  ac_prog_rejected=yes
+  continue
+fi
+ac_cv_prog_CC="cc"
+echo "$as_me:1671: found $ac_dir/$ac_word" >&5
+break
+done
+
+if test $ac_prog_rejected = yes; then
+  # We found a bogon in the path, so make sure we never use it.
+  set dummy $ac_cv_prog_CC
+  shift
+  if test $# != 0; then
+    # We chose a different compiler from the bogus one.
+    # However, it has the same basename, so the bogon will be chosen
+    # first if we set CC to just the basename; use the full file name.
+    shift
+    set dummy "$ac_dir/$ac_word" ${1+"$@"}
+    shift
+    ac_cv_prog_CC="$@"
+  fi
+fi
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1693: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1696: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$CC"; then
+  if test -n "$ac_tool_prefix"; then
+  for ac_prog in cl
+  do
+    # Extract the first word of "$ac_tool_prefix$ac_prog", so it can be a program name with args.
+set dummy $ac_tool_prefix$ac_prog; ac_word=$2
+echo "$as_me:1707: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CC="$ac_tool_prefix$ac_prog"
+echo "$as_me:1722: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1730: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1733: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+    test -n "$CC" && break
+  done
+fi
+if test -z "$CC"; then
+  ac_ct_CC=$CC
+  for ac_prog in cl
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1746: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CC"; then
+  ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CC="$ac_prog"
+echo "$as_me:1761: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CC=$ac_cv_prog_ac_ct_CC
+if test -n "$ac_ct_CC"; then
+  echo "$as_me:1769: result: $ac_ct_CC" >&5
+echo "${ECHO_T}$ac_ct_CC" >&6
+else
+  echo "$as_me:1772: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$ac_ct_CC" && break
+done
+
+  CC=$ac_ct_CC
+fi
+
+fi
+
+test -z "$CC" && { { echo "$as_me:1784: error: no acceptable cc found in \$PATH" >&5
+echo "$as_me: error: no acceptable cc found in \$PATH" >&2;}
+   { (exit 1); exit 1; }; }
+
+echo "$as_me:1788: checking whether we are using the GNU C compiler" >&5
+echo $ECHO_N "checking whether we are using the GNU C compiler... $ECHO_C" >&6
+if test "${ac_cv_c_compiler_gnu+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1794 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+#ifndef __GNUC__
+       choke me
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1809: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1812: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1815: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1818: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_compiler_gnu=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_compiler_gnu=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+ac_cv_c_compiler_gnu=$ac_compiler_gnu
+
+fi
+echo "$as_me:1830: result: $ac_cv_c_compiler_gnu" >&5
+echo "${ECHO_T}$ac_cv_c_compiler_gnu" >&6
+GCC=`test $ac_compiler_gnu = yes && echo yes`
+ac_test_CFLAGS=${CFLAGS+set}
+ac_save_CFLAGS=$CFLAGS
+CFLAGS="-g"
+echo "$as_me:1836: checking whether $CC accepts -g" >&5
+echo $ECHO_N "checking whether $CC accepts -g... $ECHO_C" >&6
+if test "${ac_cv_prog_cc_g+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1842 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1854: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1857: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1860: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1863: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cc_g=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_prog_cc_g=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:1873: result: $ac_cv_prog_cc_g" >&5
+echo "${ECHO_T}$ac_cv_prog_cc_g" >&6
+if test "$ac_test_CFLAGS" = set; then
+  CFLAGS=$ac_save_CFLAGS
+elif test $ac_cv_prog_cc_g = yes; then
+  if test "$GCC" = yes; then
+    CFLAGS="-g -O2"
+  else
+    CFLAGS="-g"
+  fi
+else
+  if test "$GCC" = yes; then
+    CFLAGS="-O2"
+  else
+    CFLAGS=
+  fi
+fi
+# Some people use a C++ compiler to compile C.  Since we use `exit',
+# in C++ we need to declare it.  In case someone uses the same compiler
+# for both compiling C and C++ we need to have the C++ compiler decide
+# the declaration of exit, since it's the most demanding environment.
+cat >conftest.$ac_ext <<_ACEOF
+#ifndef __cplusplus
+  choke me
+#endif
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1900: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1903: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1906: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1909: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  for ac_declaration in \
+   ''\
+   '#include <stdlib.h>' \
+   'extern "C" void std::exit (int) throw (); using std::exit;' \
+   'extern "C" void std::exit (int); using std::exit;' \
+   'extern "C" void exit (int) throw ();' \
+   'extern "C" void exit (int);' \
+   'void exit (int);'
+do
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1921 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1934: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1937: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1940: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1943: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+continue
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1953 "configure"
+#include "confdefs.h"
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1965: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1968: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1971: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1974: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+echo '#ifdef __cplusplus' >>confdefs.h
+echo $ac_declaration      >>confdefs.h
+echo '#endif'             >>confdefs.h
+
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+ac_aux_dir=
+for ac_dir in $srcdir $srcdir/.. $srcdir/../..; do
+  if test -f $ac_dir/install-sh; then
+    ac_aux_dir=$ac_dir
+    ac_install_sh="$ac_aux_dir/install-sh -c"
+    break
+  elif test -f $ac_dir/install.sh; then
+    ac_aux_dir=$ac_dir
+    ac_install_sh="$ac_aux_dir/install.sh -c"
+    break
+  elif test -f $ac_dir/shtool; then
+    ac_aux_dir=$ac_dir
+    ac_install_sh="$ac_aux_dir/shtool install -c"
+    break
+  fi
+done
+if test -z "$ac_aux_dir"; then
+  { { echo "$as_me:2015: error: cannot find install-sh or install.sh in $srcdir $srcdir/.. $srcdir/../.." >&5
+echo "$as_me: error: cannot find install-sh or install.sh in $srcdir $srcdir/.. $srcdir/../.." >&2;}
+   { (exit 1); exit 1; }; }
+fi
+ac_config_guess="$SHELL $ac_aux_dir/config.guess"
+ac_config_sub="$SHELL $ac_aux_dir/config.sub"
+ac_configure="$SHELL $ac_aux_dir/configure" # This should be Cygnus configure.
+
+# Find a good install program.  We prefer a C program (faster),
+# so one script is as good as another.  But avoid the broken or
+# incompatible versions:
+# SysV /etc/install, /usr/sbin/install
+# SunOS /usr/etc/install
+# IRIX /sbin/install
+# AIX /bin/install
+# AmigaOS /C/install, which installs bootblocks on floppy discs
+# AIX 4 /usr/bin/installbsd, which doesn't work without a -g flag
+# AFS /usr/afsws/bin/install, which mishandles nonexistent args
+# SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff"
+# ./install, which can be erroneously created by make from ./install.sh.
+echo "$as_me:2035: checking for a BSD compatible install" >&5
+echo $ECHO_N "checking for a BSD compatible install... $ECHO_C" >&6
+if test -z "$INSTALL"; then
+if test "${ac_cv_path_install+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+    ac_save_IFS=$IFS; IFS=$ac_path_separator
+  for ac_dir in $PATH; do
+    IFS=$ac_save_IFS
+    # Account for people who put trailing slashes in PATH elements.
+    case $ac_dir/ in
+    / | ./ | .// | /cC/* \
+    | /etc/* | /usr/sbin/* | /usr/etc/* | /sbin/* | /usr/afsws/bin/* \
+    | /usr/ucb/* ) ;;
+    *)
+      # OSF1 and SCO ODT 3.0 have their own names for install.
+      # Don't use installbsd from OSF since it installs stuff as root
+      # by default.
+      for ac_prog in ginstall scoinst install; do
+        if $as_executable_p "$ac_dir/$ac_prog"; then
+	  if test $ac_prog = install &&
+            grep dspmsg "$ac_dir/$ac_prog" >/dev/null 2>&1; then
+	    # AIX install.  It has an incompatible calling convention.
+	    :
+	  elif test $ac_prog = install &&
+	    grep pwplus "$ac_dir/$ac_prog" >/dev/null 2>&1; then
+	    # program-specific install script used by HP pwplus--don't use.
+	    :
+	  else
+	    ac_cv_path_install="$ac_dir/$ac_prog -c"
+	    break 2
+	  fi
+	fi
+      done
+      ;;
+    esac
+  done
+
+fi
+  if test "${ac_cv_path_install+set}" = set; then
+    INSTALL=$ac_cv_path_install
+  else
+    # As a last resort, use the slow shell script.  We don't cache a
+    # path for INSTALL within a source directory, because that will
+    # break other packages using the cache if that directory is
+    # removed, or if the path is relative.
+    INSTALL=$ac_install_sh
+  fi
+fi
+echo "$as_me:2084: result: $INSTALL" >&5
+echo "${ECHO_T}$INSTALL" >&6
+
+# Use test -z because SunOS4 sh mishandles braces in ${var-val}.
+# It thinks the first close brace ends the variable substitution.
+test -z "$INSTALL_PROGRAM" && INSTALL_PROGRAM='${INSTALL}'
+
+test -z "$INSTALL_SCRIPT" && INSTALL_SCRIPT='${INSTALL}'
+
+test -z "$INSTALL_DATA" && INSTALL_DATA='${INSTALL} -m 644'
+
+echo "$as_me:2095: checking whether ln -s works" >&5
+echo $ECHO_N "checking whether ln -s works... $ECHO_C" >&6
+LN_S=$as_ln_s
+if test "$LN_S" = "ln -s"; then
+  echo "$as_me:2099: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+else
+  echo "$as_me:2102: result: no, using $LN_S" >&5
+echo "${ECHO_T}no, using $LN_S" >&6
+fi
+
+echo "$as_me:2106: checking whether ${MAKE-make} sets \${MAKE}" >&5
+echo $ECHO_N "checking whether ${MAKE-make} sets \${MAKE}... $ECHO_C" >&6
+set dummy ${MAKE-make}; ac_make=`echo "$2" | sed 'y,./+-,__p_,'`
+if eval "test \"\${ac_cv_prog_make_${ac_make}_set+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.make <<\EOF
+all:
+	@echo 'ac_maketemp="${MAKE}"'
+EOF
+# GNU make sometimes prints "make[1]: Entering...", which would confuse us.
+eval `${MAKE-make} -f conftest.make 2>/dev/null | grep temp=`
+if test -n "$ac_maketemp"; then
+  eval ac_cv_prog_make_${ac_make}_set=yes
+else
+  eval ac_cv_prog_make_${ac_make}_set=no
+fi
+rm -f conftest.make
+fi
+if eval "test \"`echo '$ac_cv_prog_make_'${ac_make}_set`\" = yes"; then
+  echo "$as_me:2126: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  SET_MAKE=
+else
+  echo "$as_me:2130: result: no" >&5
+echo "${ECHO_T}no" >&6
+  SET_MAKE="MAKE=${MAKE-make}"
+fi
+
+if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}ranlib", so it can be a program name with args.
+set dummy ${ac_tool_prefix}ranlib; ac_word=$2
+echo "$as_me:2138: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_RANLIB+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$RANLIB"; then
+  ac_cv_prog_RANLIB="$RANLIB" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_RANLIB="${ac_tool_prefix}ranlib"
+echo "$as_me:2153: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+RANLIB=$ac_cv_prog_RANLIB
+if test -n "$RANLIB"; then
+  echo "$as_me:2161: result: $RANLIB" >&5
+echo "${ECHO_T}$RANLIB" >&6
+else
+  echo "$as_me:2164: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$ac_cv_prog_RANLIB"; then
+  ac_ct_RANLIB=$RANLIB
+  # Extract the first word of "ranlib", so it can be a program name with args.
+set dummy ranlib; ac_word=$2
+echo "$as_me:2173: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_RANLIB+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_RANLIB"; then
+  ac_cv_prog_ac_ct_RANLIB="$ac_ct_RANLIB" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_RANLIB="ranlib"
+echo "$as_me:2188: found $ac_dir/$ac_word" >&5
+break
+done
+
+  test -z "$ac_cv_prog_ac_ct_RANLIB" && ac_cv_prog_ac_ct_RANLIB=":"
+fi
+fi
+ac_ct_RANLIB=$ac_cv_prog_ac_ct_RANLIB
+if test -n "$ac_ct_RANLIB"; then
+  echo "$as_me:2197: result: $ac_ct_RANLIB" >&5
+echo "${ECHO_T}$ac_ct_RANLIB" >&6
+else
+  echo "$as_me:2200: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  RANLIB=$ac_ct_RANLIB
+else
+  RANLIB="$ac_cv_prog_RANLIB"
+fi
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+echo "$as_me:2214: checking how to run the C preprocessor" >&5
+echo $ECHO_N "checking how to run the C preprocessor... $ECHO_C" >&6
+# On Suns, sometimes $CPP names a directory.
+if test -n "$CPP" && test -d "$CPP"; then
+  CPP=
+fi
+if test -z "$CPP"; then
+  if test "${ac_cv_prog_CPP+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+      # Double quotes because CPP needs to be expanded
+    for CPP in "$CC -E" "$CC -E -traditional-cpp" "/lib/cpp"
+    do
+      # break 2 since there is a loop in there.
+      ac_preproc_ok=false
+for ac_c_preproc_warn_flag in '' yes
+do
+  # Use a header file that comes with gcc, so configuring glibc
+  # with a fresh cross-compiler works.
+  # On the NeXT, cc -E runs the code through the compiler's parser,
+  # not just through cpp. "Syntax error" is here to catch this case.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2236 "configure"
+#include "confdefs.h"
+#include <assert.h>
+                     Syntax error
+_ACEOF
+if { (eval echo "$as_me:2241: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2247: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Broken: fails on valid input.
+continue
+fi
+rm -f conftest.err conftest.$ac_ext
+
+  # OK, works on sane cases.  Now check whether non-existent headers
+  # can be detected and how.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2270 "configure"
+#include "confdefs.h"
+#include <ac_nonexistent.h>
+_ACEOF
+if { (eval echo "$as_me:2274: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2280: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  # Broken: success on invalid input.
+continue
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Passes both tests.
+ac_preproc_ok=:
+break
+fi
+rm -f conftest.err conftest.$ac_ext
+
+done
+# Because of `break', _AC_PREPROC_IFELSE's cleaning code was skipped.
+rm -f conftest.err conftest.$ac_ext
+if $ac_preproc_ok; then
+  break 2
+fi
+
+    done
+    ac_cv_prog_CPP=$CPP
+
+fi
+  CPP=$ac_cv_prog_CPP
+else
+  ac_cv_prog_CPP=$CPP
+fi
+echo "$as_me:2317: result: $CPP" >&5
+echo "${ECHO_T}$CPP" >&6
+ac_preproc_ok=false
+for ac_c_preproc_warn_flag in '' yes
+do
+  # Use a header file that comes with gcc, so configuring glibc
+  # with a fresh cross-compiler works.
+  # On the NeXT, cc -E runs the code through the compiler's parser,
+  # not just through cpp. "Syntax error" is here to catch this case.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2327 "configure"
+#include "confdefs.h"
+#include <assert.h>
+                     Syntax error
+_ACEOF
+if { (eval echo "$as_me:2332: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2338: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Broken: fails on valid input.
+continue
+fi
+rm -f conftest.err conftest.$ac_ext
+
+  # OK, works on sane cases.  Now check whether non-existent headers
+  # can be detected and how.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2361 "configure"
+#include "confdefs.h"
+#include <ac_nonexistent.h>
+_ACEOF
+if { (eval echo "$as_me:2365: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2371: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  # Broken: success on invalid input.
+continue
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Passes both tests.
+ac_preproc_ok=:
+break
+fi
+rm -f conftest.err conftest.$ac_ext
+
+done
+# Because of `break', _AC_PREPROC_IFELSE's cleaning code was skipped.
+rm -f conftest.err conftest.$ac_ext
+if $ac_preproc_ok; then
+  :
+else
+  { { echo "$as_me:2399: error: C preprocessor \"$CPP\" fails sanity check" >&5
+echo "$as_me: error: C preprocessor \"$CPP\" fails sanity check" >&2;}
+   { (exit 1); exit 1; }; }
+fi
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+echo "$as_me:2410: checking for AIX" >&5
+echo $ECHO_N "checking for AIX... $ECHO_C" >&6
+cat >conftest.$ac_ext <<_ACEOF
+#line 2413 "configure"
+#include "confdefs.h"
+#ifdef _AIX
+  yes
+#endif
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "yes" >/dev/null 2>&1; then
+  echo "$as_me:2422: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+cat >>confdefs.h <<\EOF
+#define _ALL_SOURCE 1
+EOF
+
+else
+  echo "$as_me:2429: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+rm -f conftest*
+
+echo "$as_me:2434: checking for POSIXized ISC" >&5
+echo $ECHO_N "checking for POSIXized ISC... $ECHO_C" >&6
+if test -d /etc/conf/kconfig.d &&
+   grep _POSIX_VERSION /usr/include/sys/unistd.h >/dev/null 2>&1
+then
+  echo "$as_me:2439: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  ISC=yes # If later tests want to check for ISC.
+
+cat >>confdefs.h <<\EOF
+#define _POSIX_SOURCE 1
+EOF
+
+  if test "$GCC" = yes; then
+    CC="$CC -posix"
+  else
+    CC="$CC -Xp"
+  fi
+else
+  echo "$as_me:2453: result: no" >&5
+echo "${ECHO_T}no" >&6
+  ISC=
+fi
+
+echo "$as_me:2458: checking for minix/config.h" >&5
+echo $ECHO_N "checking for minix/config.h... $ECHO_C" >&6
+if test "${ac_cv_header_minix_config_h+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2464 "configure"
+#include "confdefs.h"
+#include <minix/config.h>
+_ACEOF
+if { (eval echo "$as_me:2468: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2474: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_minix_config_h=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_minix_config_h=no
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:2493: result: $ac_cv_header_minix_config_h" >&5
+echo "${ECHO_T}$ac_cv_header_minix_config_h" >&6
+if test $ac_cv_header_minix_config_h = yes; then
+  MINIX=yes
+else
+  MINIX=
+fi
+
+if test "$MINIX" = yes; then
+
+cat >>confdefs.h <<\EOF
+#define _POSIX_SOURCE 1
+EOF
+
+cat >>confdefs.h <<\EOF
+#define _POSIX_1_SOURCE 2
+EOF
+
+cat >>confdefs.h <<\EOF
+#define _MINIX 1
+EOF
+
+fi
+
+echo "$as_me:2517: checking for ${CC-cc} option to accept ANSI C" >&5
+echo $ECHO_N "checking for ${CC-cc} option to accept ANSI C... $ECHO_C" >&6
+if test "${ac_cv_prog_cc_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_cv_prog_cc_stdc=no
+ac_save_CFLAGS="$CFLAGS"
+# Don't try gcc -ansi; that turns off useful extensions and
+# breaks some systems' header files.
+# AIX			-qlanglvl=ansi
+# Ultrix and OSF/1	-std1
+# HP-UX			-Aa -D_HPUX_SOURCE
+# SVR4			-Xc
+for ac_arg in "" -qlanglvl=ansi -std1 "-Aa -D_HPUX_SOURCE" -Xc
+do
+  CFLAGS="$ac_save_CFLAGS $ac_arg"
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2534 "configure"
+#include "confdefs.h"
+#if !defined(__STDC__) || __STDC__ != 1
+choke me
+#endif
+
+int
+main ()
+{
+int test (int i, double x);
+struct s1 {int (*f) (int a);};
+struct s2 {int (*f) (double a);};
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2551: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2554: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2557: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2560: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cc_stdc="$ac_arg"; break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+CFLAGS="$ac_save_CFLAGS"
+
+fi
+
+echo "$as_me:2573: result: $ac_cv_prog_cc_stdc" >&5
+echo "${ECHO_T}$ac_cv_prog_cc_stdc" >&6
+case "x$ac_cv_prog_cc_stdc" in
+  x|xno) ;;
+  *) CC="$CC $ac_cv_prog_cc_stdc" ;;
+esac
+
+echo "$as_me:2580: checking for function prototypes" >&5
+echo $ECHO_N "checking for function prototypes... $ECHO_C" >&6
+if test "$ac_cv_prog_cc_stdc" != no; then
+  echo "$as_me:2583: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  cat >>confdefs.h <<\EOF
+#define PROTOTYPES 1
+EOF
+
+  U= ANSI2KNR=
+else
+  echo "$as_me:2591: result: no" >&5
+echo "${ECHO_T}no" >&6
+  U=_ ANSI2KNR=ansi2knr
+fi
+
+echo "$as_me:2596: checking for $CC option to accept ANSI C" >&5
+echo $ECHO_N "checking for $CC option to accept ANSI C... $ECHO_C" >&6
+if test "${ac_cv_prog_cc_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_cv_prog_cc_stdc=no
+ac_save_CC=$CC
+cat >conftest.$ac_ext <<_ACEOF
+#line 2604 "configure"
+#include "confdefs.h"
+#include <stdarg.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+/* Most of the following tests are stolen from RCS 5.7's src/conf.sh.  */
+struct buf { int x; };
+FILE * (*rcsopen) (struct buf *, struct stat *, int);
+static char *e (p, i)
+     char **p;
+     int i;
+{
+  return p[i];
+}
+static char *f (char * (*g) (char **, int), char **p, ...)
+{
+  char *s;
+  va_list v;
+  va_start (v,p);
+  s = g (p, va_arg (v,int));
+  va_end (v);
+  return s;
+}
+int test (int i, double x);
+struct s1 {int (*f) (int a);};
+struct s2 {int (*f) (double a);};
+int pairnames (int, char **, FILE *(*)(struct buf *, struct stat *, int), int, int);
+int argc;
+char **argv;
+int
+main ()
+{
+return f (e, argv, 0) != argv[0]  ||  f (e, argv, 1) != argv[1];
+  ;
+  return 0;
+}
+_ACEOF
+# Don't try gcc -ansi; that turns off useful extensions and
+# breaks some systems' header files.
+# AIX			-qlanglvl=ansi
+# Ultrix and OSF/1	-std1
+# HP-UX 10.20 and later	-Ae
+# HP-UX older versions	-Aa -D_HPUX_SOURCE
+# SVR4			-Xc -D__EXTENSIONS__
+for ac_arg in "" -qlanglvl=ansi -std1 -Ae "-Aa -D_HPUX_SOURCE" "-Xc -D__EXTENSIONS__"
+do
+  CC="$ac_save_CC $ac_arg"
+  rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2653: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2656: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2659: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2662: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cc_stdc=$ac_arg
+break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext
+done
+rm -f conftest.$ac_ext conftest.$ac_objext
+CC=$ac_save_CC
+
+fi
+
+case "x$ac_cv_prog_cc_stdc" in
+  x|xno)
+    echo "$as_me:2679: result: none needed" >&5
+echo "${ECHO_T}none needed" >&6 ;;
+  *)
+    echo "$as_me:2682: result: $ac_cv_prog_cc_stdc" >&5
+echo "${ECHO_T}$ac_cv_prog_cc_stdc" >&6
+    CC="$CC $ac_cv_prog_cc_stdc" ;;
+esac
+
+echo "$as_me:2687: checking for an ANSI C-conforming const" >&5
+echo $ECHO_N "checking for an ANSI C-conforming const... $ECHO_C" >&6
+if test "${ac_cv_c_const+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2693 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+/* FIXME: Include the comments suggested by Paul. */
+#ifndef __cplusplus
+  /* Ultrix mips cc rejects this.  */
+  typedef int charset[2];
+  const charset x;
+  /* SunOS 4.1.1 cc rejects this.  */
+  char const *const *ccp;
+  char **p;
+  /* NEC SVR4.0.2 mips cc rejects this.  */
+  struct point {int x, y;};
+  static struct point const zero = {0,0};
+  /* AIX XL C 1.02.0.0 rejects this.
+     It does not let you subtract one const X* pointer from another in
+     an arm of an if-expression whose if-part is not a constant
+     expression */
+  const char *g = "string";
+  ccp = &g + (g ? g-g : 0);
+  /* HPUX 7.0 cc rejects these. */
+  ++ccp;
+  p = (char**) ccp;
+  ccp = (char const *const *) p;
+  { /* SCO 3.2v4 cc rejects this.  */
+    char *t;
+    char const *s = 0 ? (char *) 0 : (char const *) 0;
+
+    *t++ = 0;
+  }
+  { /* Someone thinks the Sun supposedly-ANSI compiler will reject this.  */
+    int x[] = {25, 17};
+    const int *foo = &x[0];
+    ++foo;
+  }
+  { /* Sun SC1.0 ANSI compiler rejects this -- but not the above. */
+    typedef const int *iptr;
+    iptr p = 0;
+    ++p;
+  }
+  { /* AIX XL C 1.02.0.0 rejects this saying
+       "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue. */
+    struct s { int j; const int *ap[3]; };
+    struct s *b; b->j = 5;
+  }
+  { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */
+    const int foo = 10;
+  }
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2751: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2754: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2757: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2760: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_c_const=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_c_const=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:2770: result: $ac_cv_c_const" >&5
+echo "${ECHO_T}$ac_cv_c_const" >&6
+if test $ac_cv_c_const = no; then
+
+cat >>confdefs.h <<\EOF
+#define const
+EOF
+
+fi
+
+echo "$as_me:2780: checking for ANSI C header files" >&5
+echo $ECHO_N "checking for ANSI C header files... $ECHO_C" >&6
+if test "${ac_cv_header_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2786 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+#include <stdarg.h>
+#include <string.h>
+#include <float.h>
+
+_ACEOF
+if { (eval echo "$as_me:2794: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2800: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_stdc=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_stdc=no
+fi
+rm -f conftest.err conftest.$ac_ext
+
+if test $ac_cv_header_stdc = yes; then
+  # SunOS 4.x string.h does not declare mem*, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2822 "configure"
+#include "confdefs.h"
+#include <string.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "memchr" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # ISC 2.0.2 stdlib.h does not declare free, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2840 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "free" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # /bin/cc in Irix-4.0.5 gets non-ANSI ctype macros unless using -ansi.
+  if test "$cross_compiling" = yes; then
+  :
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2861 "configure"
+#include "confdefs.h"
+#include <ctype.h>
+#if ((' ' & 0x0FF) == 0x020)
+# define ISLOWER(c) ('a' <= (c) && (c) <= 'z')
+# define TOUPPER(c) (ISLOWER(c) ? 'A' + ((c) - 'a') : (c))
+#else
+# define ISLOWER(c) (('a' <= (c) && (c) <= 'i') \
+                     || ('j' <= (c) && (c) <= 'r') \
+                     || ('s' <= (c) && (c) <= 'z'))
+# define TOUPPER(c) (ISLOWER(c) ? ((c) | 0x40) : (c))
+#endif
+
+#define XOR(e, f) (((e) && !(f)) || (!(e) && (f)))
+int
+main ()
+{
+  int i;
+  for (i = 0; i < 256; i++)
+    if (XOR (islower (i), ISLOWER (i))
+        || toupper (i) != TOUPPER (i))
+      exit(2);
+  exit (0);
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:2887: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:2890: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:2892: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2895: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_header_stdc=no
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+fi
+echo "$as_me:2908: result: $ac_cv_header_stdc" >&5
+echo "${ECHO_T}$ac_cv_header_stdc" >&6
+if test $ac_cv_header_stdc = yes; then
+
+cat >>confdefs.h <<\EOF
+#define STDC_HEADERS 1
+EOF
+
+fi
+
+for ac_header in stdlib.h string.h memory.h strings.h inttypes.h unistd.h
+do
+ac_ac_Header=`echo "ac_cv_header_$ac_header" | $ac_tr_sh`
+echo "$as_me:2921: checking for $ac_header" >&5
+echo $ECHO_N "checking for $ac_header... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_Header+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2927 "configure"
+#include "confdefs.h"
+#include <$ac_header>
+_ACEOF
+if { (eval echo "$as_me:2931: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2937: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  eval "$ac_ac_Header=yes"
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  eval "$ac_ac_Header=no"
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:2956: result: `eval echo '${'$ac_ac_Header'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_Header'}'`" >&6
+if test `eval echo '${'$ac_ac_Header'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_header" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+echo "$as_me:2966: checking for off_t" >&5
+echo $ECHO_N "checking for off_t... $ECHO_C" >&6
+if test "${ac_cv_type_off_t+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2972 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+if ((off_t *) 0)
+  return 0;
+if (sizeof (off_t))
+  return 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2987: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2990: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2993: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2996: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_off_t=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_off_t=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3006: result: $ac_cv_type_off_t" >&5
+echo "${ECHO_T}$ac_cv_type_off_t" >&6
+if test $ac_cv_type_off_t = yes; then
+  :
+else
+
+cat >>confdefs.h <<EOF
+#define off_t long
+EOF
+
+fi
+
+echo "$as_me:3018: checking for size_t" >&5
+echo $ECHO_N "checking for size_t... $ECHO_C" >&6
+if test "${ac_cv_type_size_t+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3024 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+if ((size_t *) 0)
+  return 0;
+if (sizeof (size_t))
+  return 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3039: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3042: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3045: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3048: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_size_t=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_size_t=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3058: result: $ac_cv_type_size_t" >&5
+echo "${ECHO_T}$ac_cv_type_size_t" >&6
+if test $ac_cv_type_size_t = yes; then
+  :
+else
+
+cat >>confdefs.h <<EOF
+#define size_t unsigned
+EOF
+
+fi
+
+echo "$as_me:3070: checking whether time.h and sys/time.h may both be included" >&5
+echo $ECHO_N "checking whether time.h and sys/time.h may both be included... $ECHO_C" >&6
+if test "${ac_cv_header_time+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3076 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <sys/time.h>
+#include <time.h>
+
+int
+main ()
+{
+struct tm *tp;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3091: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3094: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3097: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3100: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_header_time=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_header_time=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3110: result: $ac_cv_header_time" >&5
+echo "${ECHO_T}$ac_cv_header_time" >&6
+if test $ac_cv_header_time = yes; then
+
+cat >>confdefs.h <<\EOF
+#define TIME_WITH_SYS_TIME 1
+EOF
+
+fi
+
+echo "$as_me:3120: checking whether struct tm is in sys/time.h or time.h" >&5
+echo $ECHO_N "checking whether struct tm is in sys/time.h or time.h... $ECHO_C" >&6
+if test "${ac_cv_struct_tm+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3126 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <time.h>
+
+int
+main ()
+{
+struct tm *tp; tp->tm_sec;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3140: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3143: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3146: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3149: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_struct_tm=time.h
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_struct_tm=sys/time.h
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3159: result: $ac_cv_struct_tm" >&5
+echo "${ECHO_T}$ac_cv_struct_tm" >&6
+if test $ac_cv_struct_tm = sys/time.h; then
+
+cat >>confdefs.h <<\EOF
+#define TM_IN_SYS_TIME 1
+EOF
+
+fi
+
+if test "$ac_cv_prog_cc_stdc" = '-Xc'; then
+cat >conftest.$ac_ext <<_ACEOF
+#line 3171 "configure"
+#include "confdefs.h"
+#include <stdio.h>
+
+int
+main ()
+{
+ FILE *rubbish = popen("yes", "r");
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3184: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3187: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3190: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3193: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+CC="`echo $CC | sed 's/-Xc/-Xa/'`"    ac_cv_prog_cc_stdc='-Xa'
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+
+echo "$as_me:3204: checking for main in -lm" >&5
+echo $ECHO_N "checking for main in -lm... $ECHO_C" >&6
+if test "${ac_cv_lib_m_main+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lm  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 3212 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+main ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3224: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3227: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3230: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3233: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_m_main=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_m_main=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:3244: result: $ac_cv_lib_m_main" >&5
+echo "${ECHO_T}$ac_cv_lib_m_main" >&6
+if test $ac_cv_lib_m_main = yes; then
+  cat >>confdefs.h <<EOF
+#define HAVE_LIBM 1
+EOF
+
+  LIBS="-lm $LIBS"
+
+fi
+
+ac_header_dirent=no
+for ac_hdr in dirent.h sys/ndir.h sys/dir.h ndir.h; do
+  ac_ac_Header=`echo "ac_cv_header_dirent_$ac_hdr" | $ac_tr_sh`
+echo "$as_me:3258: checking for $ac_hdr that defines DIR" >&5
+echo $ECHO_N "checking for $ac_hdr that defines DIR... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_Header+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3264 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <$ac_hdr>
+
+int
+main ()
+{
+DIR *dirp = 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3278: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3281: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3284: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3287: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_Header=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_Header=no"
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3297: result: `eval echo '${'$ac_ac_Header'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_Header'}'`" >&6
+if test `eval echo '${'$ac_ac_Header'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_hdr" | $ac_tr_cpp` 1
+EOF
+
+ac_header_dirent=$ac_hdr; break
+fi
+
+done
+# Two versions of opendir et al. are in -ldir and -lx on SCO Xenix.
+if test $ac_header_dirent = dirent.h; then
+  echo "$as_me:3310: checking for opendir in -ldir" >&5
+echo $ECHO_N "checking for opendir in -ldir... $ECHO_C" >&6
+if test "${ac_cv_lib_dir_opendir+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-ldir  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 3318 "configure"
+#include "confdefs.h"
+
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char opendir ();
+int
+main ()
+{
+opendir ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3337: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3340: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3343: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3346: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_dir_opendir=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_dir_opendir=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:3357: result: $ac_cv_lib_dir_opendir" >&5
+echo "${ECHO_T}$ac_cv_lib_dir_opendir" >&6
+if test $ac_cv_lib_dir_opendir = yes; then
+  LIBS="$LIBS -ldir"
+fi
+
+else
+  echo "$as_me:3364: checking for opendir in -lx" >&5
+echo $ECHO_N "checking for opendir in -lx... $ECHO_C" >&6
+if test "${ac_cv_lib_x_opendir+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lx  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 3372 "configure"
+#include "confdefs.h"
+
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char opendir ();
+int
+main ()
+{
+opendir ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3391: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3394: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3397: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3400: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_x_opendir=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_x_opendir=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:3411: result: $ac_cv_lib_x_opendir" >&5
+echo "${ECHO_T}$ac_cv_lib_x_opendir" >&6
+if test $ac_cv_lib_x_opendir = yes; then
+  LIBS="$LIBS -lx"
+fi
+
+fi
+
+echo "$as_me:3419: checking for ANSI C header files" >&5
+echo $ECHO_N "checking for ANSI C header files... $ECHO_C" >&6
+if test "${ac_cv_header_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3425 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+#include <stdarg.h>
+#include <string.h>
+#include <float.h>
+
+_ACEOF
+if { (eval echo "$as_me:3433: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:3439: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_stdc=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_stdc=no
+fi
+rm -f conftest.err conftest.$ac_ext
+
+if test $ac_cv_header_stdc = yes; then
+  # SunOS 4.x string.h does not declare mem*, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3461 "configure"
+#include "confdefs.h"
+#include <string.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "memchr" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # ISC 2.0.2 stdlib.h does not declare free, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3479 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "free" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # /bin/cc in Irix-4.0.5 gets non-ANSI ctype macros unless using -ansi.
+  if test "$cross_compiling" = yes; then
+  :
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3500 "configure"
+#include "confdefs.h"
+#include <ctype.h>
+#if ((' ' & 0x0FF) == 0x020)
+# define ISLOWER(c) ('a' <= (c) && (c) <= 'z')
+# define TOUPPER(c) (ISLOWER(c) ? 'A' + ((c) - 'a') : (c))
+#else
+# define ISLOWER(c) (('a' <= (c) && (c) <= 'i') \
+                     || ('j' <= (c) && (c) <= 'r') \
+                     || ('s' <= (c) && (c) <= 'z'))
+# define TOUPPER(c) (ISLOWER(c) ? ((c) | 0x40) : (c))
+#endif
+
+#define XOR(e, f) (((e) && !(f)) || (!(e) && (f)))
+int
+main ()
+{
+  int i;
+  for (i = 0; i < 256; i++)
+    if (XOR (islower (i), ISLOWER (i))
+        || toupper (i) != TOUPPER (i))
+      exit(2);
+  exit (0);
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:3526: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3529: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:3531: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3534: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_header_stdc=no
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+fi
+echo "$as_me:3547: result: $ac_cv_header_stdc" >&5
+echo "${ECHO_T}$ac_cv_header_stdc" >&6
+if test $ac_cv_header_stdc = yes; then
+
+cat >>confdefs.h <<\EOF
+#define STDC_HEADERS 1
+EOF
+
+fi
+
+for ac_header in fcntl.h limits.h sys/time.h unistd.h string.h memory.h sys/procfs.h
+do
+ac_ac_Header=`echo "ac_cv_header_$ac_header" | $ac_tr_sh`
+echo "$as_me:3560: checking for $ac_header" >&5
+echo $ECHO_N "checking for $ac_header... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_Header+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3566 "configure"
+#include "confdefs.h"
+#include <$ac_header>
+_ACEOF
+if { (eval echo "$as_me:3570: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:3576: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  eval "$ac_ac_Header=yes"
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  eval "$ac_ac_Header=no"
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:3595: result: `eval echo '${'$ac_ac_Header'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_Header'}'`" >&6
+if test `eval echo '${'$ac_ac_Header'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_header" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 3606 "configure"
+#include "confdefs.h"
+#include <stdio.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "fread" >/dev/null 2>&1; then
+  cat >>confdefs.h <<\EOF
+#define HAVE_FREAD_DECL 1
+EOF
+
+fi
+rm -f conftest*
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 3621 "configure"
+#include "confdefs.h"
+#include <stdio.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "fgetc" >/dev/null 2>&1; then
+  cat >>confdefs.h <<\EOF
+#define HAVE_FGETC_DECL 1
+EOF
+
+fi
+rm -f conftest*
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 3636 "configure"
+#include "confdefs.h"
+#include <sys/procfs.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "pr_brksize" >/dev/null 2>&1; then
+  cat >>confdefs.h <<\EOF
+#define HAVE_PR_BRKSIZE 1
+EOF
+
+fi
+rm -f conftest*
+
+# The Ultrix 4.2 mips builtin alloca declared by alloca.h only works
+# for constant arguments.  Useless!
+echo "$as_me:3652: checking for working alloca.h" >&5
+echo $ECHO_N "checking for working alloca.h... $ECHO_C" >&6
+if test "${ac_cv_working_alloca_h+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3658 "configure"
+#include "confdefs.h"
+#include <alloca.h>
+int
+main ()
+{
+char *p = (char *) alloca (2 * sizeof (int));
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3670: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3673: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3676: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3679: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_working_alloca_h=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_working_alloca_h=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:3689: result: $ac_cv_working_alloca_h" >&5
+echo "${ECHO_T}$ac_cv_working_alloca_h" >&6
+if test $ac_cv_working_alloca_h = yes; then
+
+cat >>confdefs.h <<\EOF
+#define HAVE_ALLOCA_H 1
+EOF
+
+fi
+
+echo "$as_me:3699: checking for alloca" >&5
+echo $ECHO_N "checking for alloca... $ECHO_C" >&6
+if test "${ac_cv_func_alloca_works+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3705 "configure"
+#include "confdefs.h"
+#ifdef __GNUC__
+# define alloca __builtin_alloca
+#else
+# ifdef _MSC_VER
+#  include <malloc.h>
+#  define alloca _alloca
+# else
+#  if HAVE_ALLOCA_H
+#   include <alloca.h>
+#  else
+#   ifdef _AIX
+ #pragma alloca
+#   else
+#    ifndef alloca /* predefined by HP cc +Olibcalls */
+char *alloca ();
+#    endif
+#   endif
+#  endif
+# endif
+#endif
+
+int
+main ()
+{
+char *p = (char *) alloca (1);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3737: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3740: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3743: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3746: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_func_alloca_works=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_func_alloca_works=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:3756: result: $ac_cv_func_alloca_works" >&5
+echo "${ECHO_T}$ac_cv_func_alloca_works" >&6
+
+if test $ac_cv_func_alloca_works = yes; then
+
+cat >>confdefs.h <<\EOF
+#define HAVE_ALLOCA 1
+EOF
+
+else
+  # The SVR3 libPW and SVR4 libucb both contain incompatible functions
+# that cause trouble.  Some versions do not even contain alloca or
+# contain a buggy version.  If you still want to use their alloca,
+# use ar to extract alloca.o from them instead of compiling alloca.c.
+
+ALLOCA=alloca.$ac_objext
+
+cat >>confdefs.h <<\EOF
+#define C_ALLOCA 1
+EOF
+
+echo "$as_me:3777: checking whether \`alloca.c' needs Cray hooks" >&5
+echo $ECHO_N "checking whether \`alloca.c' needs Cray hooks... $ECHO_C" >&6
+if test "${ac_cv_os_cray+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3783 "configure"
+#include "confdefs.h"
+#if defined(CRAY) && ! defined(CRAY2)
+webecray
+#else
+wenotbecray
+#endif
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "webecray" >/dev/null 2>&1; then
+  ac_cv_os_cray=yes
+else
+  ac_cv_os_cray=no
+fi
+rm -f conftest*
+
+fi
+echo "$as_me:3801: result: $ac_cv_os_cray" >&5
+echo "${ECHO_T}$ac_cv_os_cray" >&6
+if test $ac_cv_os_cray = yes; then
+  for ac_func in _getb67 GETB67 getb67; do
+    ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:3806: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3812 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3843: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3846: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3849: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3852: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:3862: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+
+cat >>confdefs.h <<EOF
+#define CRAY_STACKSEG_END $ac_func
+EOF
+
+    break
+fi
+
+  done
+fi
+
+echo "$as_me:3876: checking stack direction for C alloca" >&5
+echo $ECHO_N "checking stack direction for C alloca... $ECHO_C" >&6
+if test "${ac_cv_c_stack_direction+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test "$cross_compiling" = yes; then
+  ac_cv_c_stack_direction=0
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3885 "configure"
+#include "confdefs.h"
+int
+find_stack_direction ()
+{
+  static char *addr = 0;
+  auto char dummy;
+  if (addr == 0)
+    {
+      addr = &dummy;
+      return find_stack_direction ();
+    }
+  else
+    return (&dummy > addr) ? 1 : -1;
+}
+
+int
+main ()
+{
+  exit (find_stack_direction () < 0);
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:3908: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3911: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:3913: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3916: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_c_stack_direction=1
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_c_stack_direction=-1
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+echo "$as_me:3928: result: $ac_cv_c_stack_direction" >&5
+echo "${ECHO_T}$ac_cv_c_stack_direction" >&6
+
+cat >>confdefs.h <<EOF
+#define STACK_DIRECTION $ac_cv_c_stack_direction
+EOF
+
+fi
+
+if test $ac_cv_c_compiler_gnu = yes; then
+    echo "$as_me:3938: checking whether $CC needs -traditional" >&5
+echo $ECHO_N "checking whether $CC needs -traditional... $ECHO_C" >&6
+if test "${ac_cv_prog_gcc_traditional+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+    ac_pattern="Autoconf.*'x'"
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3945 "configure"
+#include "confdefs.h"
+#include <sgtty.h>
+Autoconf TIOCGETP
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "$ac_pattern" >/dev/null 2>&1; then
+  ac_cv_prog_gcc_traditional=yes
+else
+  ac_cv_prog_gcc_traditional=no
+fi
+rm -f conftest*
+
+  if test $ac_cv_prog_gcc_traditional = no; then
+    cat >conftest.$ac_ext <<_ACEOF
+#line 3960 "configure"
+#include "confdefs.h"
+#include <termio.h>
+Autoconf TCGETA
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "$ac_pattern" >/dev/null 2>&1; then
+  ac_cv_prog_gcc_traditional=yes
+fi
+rm -f conftest*
+
+  fi
+fi
+echo "$as_me:3973: result: $ac_cv_prog_gcc_traditional" >&5
+echo "${ECHO_T}$ac_cv_prog_gcc_traditional" >&6
+  if test $ac_cv_prog_gcc_traditional = yes; then
+    CC="$CC -traditional"
+  fi
+fi
+
+echo "$as_me:3980: checking return type of signal handlers" >&5
+echo $ECHO_N "checking return type of signal handlers... $ECHO_C" >&6
+if test "${ac_cv_type_signal+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3986 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <signal.h>
+#ifdef signal
+# undef signal
+#endif
+#ifdef __cplusplus
+extern "C" void (*signal (int, void (*)(int)))(int);
+#else
+void (*signal ()) ();
+#endif
+
+int
+main ()
+{
+int i;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4008: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4011: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4014: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4017: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_signal=void
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_signal=int
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:4027: result: $ac_cv_type_signal" >&5
+echo "${ECHO_T}$ac_cv_type_signal" >&6
+
+cat >>confdefs.h <<EOF
+#define RETSIGTYPE $ac_cv_type_signal
+EOF
+
+for ac_func in vprintf
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4037: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4043 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4074: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4077: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4080: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4083: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4093: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+echo "$as_me:4100: checking for _doprnt" >&5
+echo $ECHO_N "checking for _doprnt... $ECHO_C" >&6
+if test "${ac_cv_func__doprnt+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4106 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char _doprnt (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char _doprnt ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub__doprnt) || defined (__stub____doprnt)
+choke me
+#else
+f = _doprnt;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4137: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4140: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4143: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4146: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_func__doprnt=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_func__doprnt=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4156: result: $ac_cv_func__doprnt" >&5
+echo "${ECHO_T}$ac_cv_func__doprnt" >&6
+if test $ac_cv_func__doprnt = yes; then
+
+cat >>confdefs.h <<\EOF
+#define HAVE_DOPRNT 1
+EOF
+
+fi
+
+fi
+done
+
+for ac_func in ftime select strftime strtol getrusage times mallinfo setbuffer getpagesize strerror
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4172: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4178 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4209: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4212: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4215: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4218: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4228: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+for ac_func in ftruncate strstr strcasecmp
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4241: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4247 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4278: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4281: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4284: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4287: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4297: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+else
+  LIBOBJS="$LIBOBJS $ac_func.$ac_objext"
+fi
+done
+
+# *** Custom checking (based on GNU tar configure.in) ***
+# ---------------------------------------------------------------------------
+echo "$as_me:4311: checking for HP-UX needing gmalloc" >&5
+echo $ECHO_N "checking for HP-UX needing gmalloc... $ECHO_C" >&6
+if test "`(uname -s) 2> /dev/null`" = 'HP-UX'; then
+  echo "$as_me:4314: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  LIBOBJS="$LIBOBJS gmalloc.o"
+  cat >>confdefs.h <<\EOF
+#define HAVE_VALLOC 1
+EOF
+
+else
+  echo "$as_me:4322: result: no" >&5
+echo "${ECHO_T}no" >&6
+
+for ac_func in valloc
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4328: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4334 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4365: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4368: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4371: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4374: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4384: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+fi
+
+echo "$as_me:4396: checking if malloc debugging is wanted" >&5
+echo $ECHO_N "checking if malloc debugging is wanted... $ECHO_C" >&6
+
+# Check whether --with-dmalloc or --without-dmalloc was given.
+if test "${with_dmalloc+set}" = set; then
+  withval="$with_dmalloc"
+  if test "$withval" = yes; then
+  echo "$as_me:4403: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  cat >>confdefs.h <<\EOF
+#define WITH_DMALLOC 1
+EOF
+
+  LIBS="$LIBS -ldmalloc"
+  LDFLAGS="$LDFLAGS -g"
+else
+  echo "$as_me:4412: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+else
+  echo "$as_me:4416: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi;
+
+echo "$as_me:4420: checking which of rx or regex is wanted" >&5
+echo $ECHO_N "checking which of rx or regex is wanted... $ECHO_C" >&6
+
+# Check whether --with-regex or --without-regex was given.
+if test "${with_regex+set}" = set; then
+  withval="$with_regex"
+  if test "$withval" = yes; then
+  ac_with_regex=1
+  echo "$as_me:4428: result: regex" >&5
+echo "${ECHO_T}regex" >&6
+  cat >>confdefs.h <<\EOF
+#define WITH_REGEX 1
+EOF
+
+  LIBOBJS="$LIBOBJS regex.o"
+fi
+fi;
+if test -z "$ac_with_regex"; then
+  echo "$as_me:4438: result: rx" >&5
+echo "${ECHO_T}rx" >&6
+  echo "$as_me:4440: checking for re_rx_search" >&5
+echo $ECHO_N "checking for re_rx_search... $ECHO_C" >&6
+if test "${ac_cv_func_re_rx_search+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4446 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char re_rx_search (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char re_rx_search ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_re_rx_search) || defined (__stub___re_rx_search)
+choke me
+#else
+f = re_rx_search;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4477: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4480: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4483: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4486: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_func_re_rx_search=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_func_re_rx_search=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4496: result: $ac_cv_func_re_rx_search" >&5
+echo "${ECHO_T}$ac_cv_func_re_rx_search" >&6
+if test $ac_cv_func_re_rx_search = yes; then
+  :
+else
+  echo "$as_me:4501: checking for unsigned char *" >&5
+echo $ECHO_N "checking for unsigned char *... $ECHO_C" >&6
+if test "${ac_cv_type_unsigned_char_p+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4507 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+if ((unsigned char * *) 0)
+  return 0;
+if (sizeof (unsigned char *))
+  return 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4522: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4525: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4528: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4531: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_unsigned_char_p=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_unsigned_char_p=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:4541: result: $ac_cv_type_unsigned_char_p" >&5
+echo "${ECHO_T}$ac_cv_type_unsigned_char_p" >&6
+
+echo "$as_me:4544: checking size of unsigned char *" >&5
+echo $ECHO_N "checking size of unsigned char *... $ECHO_C" >&6
+if test "${ac_cv_sizeof_unsigned_char_p+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test "$ac_cv_type_unsigned_char_p" = yes; then
+  if test "$cross_compiling" = yes; then
+  # Depending upon the size, compute the lo and hi bounds.
+cat >conftest.$ac_ext <<_ACEOF
+#line 4553 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) >= 0)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4565: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4568: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4571: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4574: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_lo=0 ac_mid=0
+  while :; do
+    cat >conftest.$ac_ext <<_ACEOF
+#line 4579 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) <= $ac_mid)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4591: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4594: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4597: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4600: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_hi=$ac_mid; break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_lo=`expr $ac_mid + 1`; ac_mid=`expr 2 '*' $ac_mid + 1`
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  done
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_hi=-1 ac_mid=-1
+  while :; do
+    cat >conftest.$ac_ext <<_ACEOF
+#line 4616 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) >= $ac_mid)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4628: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4631: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4634: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4637: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_lo=$ac_mid; break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_hi=`expr $ac_mid - 1`; ac_mid=`expr 2 '*' $ac_mid`
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  done
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+# Binary search between lo and hi bounds.
+while test "x$ac_lo" != "x$ac_hi"; do
+  ac_mid=`expr '(' $ac_hi - $ac_lo ')' / 2 + $ac_lo`
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4653 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) <= $ac_mid)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4665: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4668: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4671: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4674: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_hi=$ac_mid
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_lo=`expr $ac_mid + 1`
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+ac_cv_sizeof_unsigned_char_p=$ac_lo
+else
+  if test "$cross_compiling" = yes; then
+  { { echo "$as_me:4687: error: cannot run test program while cross compiling" >&5
+echo "$as_me: error: cannot run test program while cross compiling" >&2;}
+   { (exit 1); exit 1; }; }
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4692 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+FILE *f = fopen ("conftest.val", "w");
+if (!f)
+  exit (1);
+fprintf (f, "%d", (sizeof (unsigned char *)));
+fclose (f);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:4708: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4711: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:4713: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4716: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_sizeof_unsigned_char_p=`cat conftest.val`
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+rm -f conftest.val
+else
+  ac_cv_sizeof_unsigned_char_p=0
+fi
+fi
+echo "$as_me:4732: result: $ac_cv_sizeof_unsigned_char_p" >&5
+echo "${ECHO_T}$ac_cv_sizeof_unsigned_char_p" >&6
+cat >>confdefs.h <<EOF
+#define SIZEOF_UNSIGNED_CHAR_P $ac_cv_sizeof_unsigned_char_p
+EOF
+
+    if test "$ac_cv_sizeof_unsigned_char_p" = 8
+    then
+      { echo "$as_me:4740: WARNING: I'm forcing you to use regex because I can't
+        find a local rx library and the one included with this
+        distribution doesn't work on 64-bit machines like yours" >&5
+echo "$as_me: WARNING: I'm forcing you to use regex because I can't
+        find a local rx library and the one included with this
+        distribution doesn't work on 64-bit machines like yours" >&2;}
+      LIBOBJS="$LIBOBJS regex.o"
+    else
+      LIBOBJS="$LIBOBJS rx.o"
+    fi
+
+fi
+
+fi
+
+echo "$as_me:4755: checking whether GNU readline requested" >&5
+echo $ECHO_N "checking whether GNU readline requested... $ECHO_C" >&6
+
+# Check whether --with-gnu_readline or --without-gnu_readline was given.
+if test "${with_gnu_readline+set}" = set; then
+  withval="$with_gnu_readline"
+  if test "$withval" = yes; then
+      echo "$as_me:4762: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+      ac_with_gnu_readline=1
+    else
+      echo "$as_me:4766: result: no" >&5
+echo "${ECHO_T}no" >&6
+    fi
+else
+  echo "$as_me:4770: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi;
+
+  if test -n "$ac_with_gnu_readline"; then
+    echo "$as_me:4775: checking for readline/readline.h" >&5
+echo $ECHO_N "checking for readline/readline.h... $ECHO_C" >&6
+if test "${ac_cv_header_readline_readline_h+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4781 "configure"
+#include "confdefs.h"
+#include <readline/readline.h>
+_ACEOF
+if { (eval echo "$as_me:4785: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:4791: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_readline_readline_h=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_readline_readline_h=no
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:4810: result: $ac_cv_header_readline_readline_h" >&5
+echo "${ECHO_T}$ac_cv_header_readline_readline_h" >&6
+if test $ac_cv_header_readline_readline_h = yes; then
+  ac_mg_readline_header_found=1
+else
+  { echo "$as_me:4815: WARNING: Can't find GNU readline headers; configuring without \
+GNU readline support" >&5
+echo "$as_me: WARNING: Can't find GNU readline headers; configuring without \
+GNU readline support" >&2;}
+fi
+
+    if test -n "$ac_mg_readline_header_found"; then
+      # first check whether we can find the readline library itself
+      echo "$as_me:4823: checking for main in -lreadline" >&5
+echo $ECHO_N "checking for main in -lreadline... $ECHO_C" >&6
+if test "${ac_cv_lib_readline_main+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lreadline  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 4831 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+main ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4843: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4846: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4849: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4852: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_readline_main=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_readline_main=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:4863: result: $ac_cv_lib_readline_main" >&5
+echo "${ECHO_T}$ac_cv_lib_readline_main" >&6
+if test $ac_cv_lib_readline_main = yes; then
+  ac_mg_readline_lib_found=1
+         cat >>confdefs.h <<\EOF
+#define WITH_GNU_READLINE 1
+EOF
+
+         LIBS="$LIBS -lreadline"
+else
+  { echo "$as_me:4873: WARNING: Can't find GNU readline library; configuring without \
+GNU readline support" >&5
+echo "$as_me: WARNING: Can't find GNU readline library; configuring without \
+GNU readline support" >&2;}
+fi
+
+      if test -n "$ac_mg_readline_lib_found"; then
+        # on some systems, readline needs curses.  It is difficult to
+        #  determine whether this is necessary on the current system,
+        #  since other undefined symbols can be turned up by an
+        #  AC_CHECK_LIB(readline, readline) test that are actually defined
+	#  by mg itself (e.g. xmalloc, xrealloc).  So, if we find libcurses,
+	#  we just bung it on and hope for the best.
+	echo "$as_me:4886: checking for main in -lcurses" >&5
+echo $ECHO_N "checking for main in -lcurses... $ECHO_C" >&6
+if test "${ac_cv_lib_curses_main+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lcurses  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 4894 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+main ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4906: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4909: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4912: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4915: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_curses_main=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_curses_main=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:4926: result: $ac_cv_lib_curses_main" >&5
+echo "${ECHO_T}$ac_cv_lib_curses_main" >&6
+if test $ac_cv_lib_curses_main = yes; then
+  LIBS="$LIBS -lcurses"
+fi
+
+      fi
+    fi
+  fi
+
+# ---------------------------------------------------------------------------
+if test "$ac_cv_func_alloca" = 'no'; then
+  LIBOBJS="$LIBOBJS xmalloc.o error.o"
+fi
+
+# ---------------------------------------------------------------------------
+# see if the user wants to override use of long long
+# Check whether --enable-override-longlong or --disable-override-longlong was given.
+if test "${enable_override_longlong+set}" = set; then
+  enableval="$enable_override_longlong"
+  cat >>confdefs.h <<\EOF
+#define DISABLE_LONG_LONG 1
+EOF
+
+fi;
+
+# ---------------------------------------------------------------------------
+
+ac_config_files="$ac_config_files Makefile text/Makefile lib/Makefile jni/Makefile java/org/greenstone/mgpp/Makefile"
+ac_config_commands="$ac_config_commands default"
+cat >confcache <<\_ACEOF
+# This file is a shell script that caches the results of configure
+# tests run on this system so they can be shared between configure
+# scripts and configure runs, see configure's option --config-cache.
+# It is not useful on other systems.  If it contains results you don't
+# want to keep, you may remove or edit it.
+#
+# config.status only pays attention to the cache file if you give it
+# the --recheck option to rerun configure.
+#
+# `ac_cv_env_foo' variables (set or unset) will be overriden when
+# loading this file, other *unset* `ac_cv_foo' will be assigned the
+# following values.
+
+_ACEOF
+
+# The following way of writing the cache mishandles newlines in values,
+# but we know of no workaround that is simple, portable, and efficient.
+# So, don't put newlines in cache variables' values.
+# Ultrix sh set writes to stderr and can't be redirected directly,
+# and sets the high bit in the cache file unless we assign to the vars.
+{
+  (set) 2>&1 |
+    case `(ac_space=' '; set | grep ac_space) 2>&1` in
+    *ac_space=\ *)
+      # `set' does not quote correctly, so add quotes (double-quote
+      # substitution turns \\\\ into \\, and sed turns \\ into \).
+      sed -n \
+        "s/'/'\\\\''/g;
+    	  s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1='\\2'/p"
+      ;;
+    *)
+      # `set' quotes correctly as required by POSIX, so do not add quotes.
+      sed -n \
+        "s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1=\\2/p"
+      ;;
+    esac;
+} |
+  sed '
+     t clear
+     : clear
+     s/^\([^=]*\)=\(.*[{}].*\)$/test "${\1+set}" = set || &/
+     t end
+     /^ac_cv_env/!s/^\([^=]*\)=\(.*\)$/\1=${\1=\2}/
+     : end' >>confcache
+if cmp -s $cache_file confcache; then :; else
+  if test -w $cache_file; then
+    test "x$cache_file" != "x/dev/null" && echo "updating cache $cache_file"
+    cat confcache >$cache_file
+  else
+    echo "not updating unwritable cache $cache_file"
+  fi
+fi
+rm -f confcache
+
+test "x$prefix" = xNONE && prefix=$ac_default_prefix
+# Let make expand exec_prefix.
+test "x$exec_prefix" = xNONE && exec_prefix='${prefix}'
+
+# VPATH may cause trouble with some makes, so we remove $(srcdir),
+# ${srcdir} and @srcdir@ from VPATH if srcdir is ".", strip leading and
+# trailing colons and then remove the whole line if VPATH becomes empty
+# (actually we leave an empty line to preserve line numbers).
+if test "x$srcdir" = x.; then
+  ac_vpsub='/^[ 	]*VPATH[ 	]*=/{
+s/:*\$(srcdir):*/:/;
+s/:*\${srcdir}:*/:/;
+s/:*@srcdir@:*/:/;
+s/^\([^=]*=[ 	]*\):*/\1/;
+s/:*$//;
+s/^[^=]*=[ 	]*$//;
+}'
+fi
+
+DEFS=-DHAVE_CONFIG_H
+
+: ${CONFIG_STATUS=./config.status}
+ac_clean_files_save=$ac_clean_files
+ac_clean_files="$ac_clean_files $CONFIG_STATUS"
+{ echo "$as_me:5035: creating $CONFIG_STATUS" >&5
+echo "$as_me: creating $CONFIG_STATUS" >&6;}
+cat >$CONFIG_STATUS <<_ACEOF
+#! $SHELL
+# Generated automatically by configure.
+# Run this file to recreate the current configuration.
+# Compiler output produced by configure, useful for debugging
+# configure, is in config.log if it exists.
+
+debug=false
+SHELL=\${CONFIG_SHELL-$SHELL}
+ac_cs_invocation="\$0 \$@"
+
+_ACEOF
+
+cat >>$CONFIG_STATUS <<\_ACEOF
+# Be Bourne compatible
+if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then
+  emulate sh
+  NULLCMD=:
+elif test -n "${BASH_VERSION+set}" && (set -o posix) >/dev/null 2>&1; then
+  set -o posix
+fi
+
+# Name of the executable.
+as_me=`echo "$0" |sed 's,.*[\\/],,'`
+
+if expr a : '\(a\)' >/dev/null 2>&1; then
+  as_expr=expr
+else
+  as_expr=false
+fi
+
+rm -f conf$$ conf$$.exe conf$$.file
+echo >conf$$.file
+if ln -s conf$$.file conf$$ 2>/dev/null; then
+  # We could just check for DJGPP; but this test a) works b) is more generic
+  # and c) will remain valid once DJGPP supports symlinks (DJGPP 2.04).
+  if test -f conf$$.exe; then
+    # Don't use ln at all; we don't have any links
+    as_ln_s='cp -p'
+  else
+    as_ln_s='ln -s'
+  fi
+elif ln conf$$.file conf$$ 2>/dev/null; then
+  as_ln_s=ln
+else
+  as_ln_s='cp -p'
+fi
+rm -f conf$$ conf$$.exe conf$$.file
+
+as_executable_p="test -f"
+
+# Support unset when possible.
+if (FOO=FOO; unset FOO) >/dev/null 2>&1; then
+  as_unset=unset
+else
+  as_unset=false
+fi
+
+# NLS nuisances.
+$as_unset LANG || test "${LANG+set}" != set || { LANG=C; export LANG; }
+$as_unset LC_ALL || test "${LC_ALL+set}" != set || { LC_ALL=C; export LC_ALL; }
+$as_unset LC_TIME || test "${LC_TIME+set}" != set || { LC_TIME=C; export LC_TIME; }
+$as_unset LC_CTYPE || test "${LC_CTYPE+set}" != set || { LC_CTYPE=C; export LC_CTYPE; }
+$as_unset LANGUAGE || test "${LANGUAGE+set}" != set || { LANGUAGE=C; export LANGUAGE; }
+$as_unset LC_COLLATE || test "${LC_COLLATE+set}" != set || { LC_COLLATE=C; export LC_COLLATE; }
+$as_unset LC_NUMERIC || test "${LC_NUMERIC+set}" != set || { LC_NUMERIC=C; export LC_NUMERIC; }
+$as_unset LC_MESSAGES || test "${LC_MESSAGES+set}" != set || { LC_MESSAGES=C; export LC_MESSAGES; }
+
+# IFS
+# We need space, tab and new line, in precisely that order.
+as_nl='
+'
+IFS=" 	$as_nl"
+
+# CDPATH.
+$as_unset CDPATH || test "${CDPATH+set}" != set || { CDPATH=:; export CDPATH; }
+
+exec 6>&1
+
+_ACEOF
+
+# Files that config.status was made for.
+if test -n "$ac_config_files"; then
+  echo "config_files=\"$ac_config_files\"" >>$CONFIG_STATUS
+fi
+
+if test -n "$ac_config_headers"; then
+  echo "config_headers=\"$ac_config_headers\"" >>$CONFIG_STATUS
+fi
+
+if test -n "$ac_config_links"; then
+  echo "config_links=\"$ac_config_links\"" >>$CONFIG_STATUS
+fi
+
+if test -n "$ac_config_commands"; then
+  echo "config_commands=\"$ac_config_commands\"" >>$CONFIG_STATUS
+fi
+
+cat >>$CONFIG_STATUS <<\EOF
+
+ac_cs_usage="\
+\`$as_me' instantiates files from templates according to the
+current configuration.
+
+Usage: $0 [OPTIONS] [FILE]...
+
+  -h, --help       print this help, then exit
+  -V, --version    print version number, then exit
+  -d, --debug      don't remove temporary files
+      --recheck    update $as_me by reconfiguring in the same conditions
+  --file=FILE[:TEMPLATE]
+                   instantiate the configuration file FILE
+  --header=FILE[:TEMPLATE]
+                   instantiate the configuration header FILE
+
+Configuration files:
+$config_files
+
+Configuration headers:
+$config_headers
+
+Configuration commands:
+$config_commands
+
+Report bugs to <bug-autoconf@gnu.org>."
+EOF
+
+cat >>$CONFIG_STATUS <<EOF
+ac_cs_version="\\
+config.status
+configured by $0, generated by GNU Autoconf 2.50,
+  with options \\"`echo "$ac_configure_args" | sed 's/[\\""\`\$]/\\\\&/g'`\\"
+
+Copyright 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001
+Free Software Foundation, Inc.
+This config.status script is free software; the Free Software Foundation
+gives unlimited permission to copy, distribute and modify it."
+srcdir=$srcdir
+INSTALL="$INSTALL"
+EOF
+
+cat >>$CONFIG_STATUS <<\EOF
+# If no file are specified by the user, then we need to provide default
+# value.  By we need to know if files were specified by the user.
+ac_need_defaults=:
+while test $# != 0
+do
+  case $1 in
+  --*=*)
+    ac_option=`expr "x$1" : 'x\([^=]*\)='`
+    ac_optarg=`expr "x$1" : 'x[^=]*=\(.*\)'`
+    shift
+    set dummy "$ac_option" "$ac_optarg" ${1+"$@"}
+    shift
+    ;;
+  -*);;
+  *) # This is not an option, so the user has probably given explicit
+     # arguments.
+     ac_need_defaults=false;;
+  esac
+
+  case $1 in
+  # Handling of the options.
+EOF
+cat >>$CONFIG_STATUS <<EOF
+  -recheck | --recheck | --rechec | --reche | --rech | --rec | --re | --r)
+    echo "running $SHELL $0 " $ac_configure_args " --no-create --no-recursion"
+    exec $SHELL $0 $ac_configure_args --no-create --no-recursion ;;
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+  --version | --vers* | -V )
+    echo "$ac_cs_version"; exit 0 ;;
+  --he | --h)
+    # Conflict between --help and --header
+    { { echo "$as_me:5211: error: ambiguous option: $1
+Try \`$0 --help' for more information." >&5
+echo "$as_me: error: ambiguous option: $1
+Try \`$0 --help' for more information." >&2;}
+   { (exit 1); exit 1; }; };;
+  --help | --hel | -h )
+    echo "$ac_cs_usage"; exit 0 ;;
+  --debug | --d* | -d )
+    debug=: ;;
+  --file | --fil | --fi | --f )
+    shift
+    CONFIG_FILES="$CONFIG_FILES $1"
+    ac_need_defaults=false;;
+  --header | --heade | --head | --hea )
+    shift
+    CONFIG_HEADERS="$CONFIG_HEADERS $1"
+    ac_need_defaults=false;;
+
+  # Handling of arguments.
+  'Makefile' ) CONFIG_FILES="$CONFIG_FILES Makefile" ;;
+  'text/Makefile' ) CONFIG_FILES="$CONFIG_FILES text/Makefile" ;;
+  'lib/Makefile' ) CONFIG_FILES="$CONFIG_FILES lib/Makefile" ;;
+  'jni/Makefile' ) CONFIG_FILES="$CONFIG_FILES jni/Makefile" ;;
+  'java/org/greenstone/mgpp/Makefile' ) CONFIG_FILES="$CONFIG_FILES java/org/greenstone/mgpp/Makefile" ;;
+  'default' ) CONFIG_COMMANDS="$CONFIG_COMMANDS default" ;;
+  'config.h' ) CONFIG_HEADERS="$CONFIG_HEADERS config.h" ;;
+
+  # This is an error.
+  -*) { { echo "$as_me:5239: error: unrecognized option: $1
+Try \`$0 --help' for more information." >&5
+echo "$as_me: error: unrecognized option: $1
+Try \`$0 --help' for more information." >&2;}
+   { (exit 1); exit 1; }; } ;;
+  *) { { echo "$as_me:5244: error: invalid argument: $1" >&5
+echo "$as_me: error: invalid argument: $1" >&2;}
+   { (exit 1); exit 1; }; };;
+  esac
+  shift
+done
+
+exec 5>>config.log
+cat >&5 << _ACEOF
+
+## ----------------------- ##
+## Running config.status.  ##
+## ----------------------- ##
+
+This file was extended by $as_me 2.50, executed with
+  > $ac_cs_invocation
+on `(hostname || uname -n) 2>/dev/null | sed 1q`
+
+_ACEOF
+EOF
+
+cat >>$CONFIG_STATUS <<\EOF
+# If the user did not use the arguments to specify the items to instantiate,
+# then the envvar interface is used.  Set only those that are not.
+# We use the long form for the default assignment because of an extremely
+# bizarre bug on SunOS 4.1.3.
+if $ac_need_defaults; then
+  test "${CONFIG_FILES+set}" = set || CONFIG_FILES=$config_files
+  test "${CONFIG_HEADERS+set}" = set || CONFIG_HEADERS=$config_headers
+  test "${CONFIG_COMMANDS+set}" = set || CONFIG_COMMANDS=$config_commands
+fi
+
+# Create a temporary directory, and hook for its removal unless debugging.
+$debug ||
+{
+  trap 'exit_status=$?; rm -rf $tmp && exit $exit_status' 0
+  trap '{ (exit $?); exit $?; }' 1 2 13 15
+}
+
+# Create a (secure) tmp directory for tmp files.
+: ${TMPDIR=/tmp}
+{
+  tmp=`(umask 077 && mktemp -d -q "$TMPDIR/csXXXXXX") 2>/dev/null` &&
+  test -n "$tmp" && test -d "$tmp"
+}  ||
+{
+  tmp=$TMPDIR/cs$$-$RANDOM
+  (umask 077 && mkdir $tmp)
+} ||
+{
+   echo "$me: cannot create a temporary directory in $TMPDIR" >&2
+   { (exit 1); exit 1; }
+}
+
+EOF
+
+cat >>$CONFIG_STATUS <<EOF
+
+#
+# CONFIG_FILES section.
+#
+
+# No need to generate the scripts if there are no CONFIG_FILES.
+# This happens for instance when ./config.status config.h
+if test -n "\$CONFIG_FILES"; then
+  # Protect against being on the right side of a sed subst in config.status.
+  sed 's/,@/@@/; s/@,/@@/; s/,;t t\$/@;t t/; /@;t t\$/s/[\\\\&,]/\\\\&/g;
+   s/@@/,@/; s/@@/@,/; s/@;t t\$/,;t t/' >\$tmp/subs.sed <<\\CEOF
+s,@SHELL@,$SHELL,;t t
+s,@exec_prefix@,$exec_prefix,;t t
+s,@prefix@,$prefix,;t t
+s,@program_transform_name@,$program_transform_name,;t t
+s,@bindir@,$bindir,;t t
+s,@sbindir@,$sbindir,;t t
+s,@libexecdir@,$libexecdir,;t t
+s,@datadir@,$datadir,;t t
+s,@sysconfdir@,$sysconfdir,;t t
+s,@sharedstatedir@,$sharedstatedir,;t t
+s,@localstatedir@,$localstatedir,;t t
+s,@libdir@,$libdir,;t t
+s,@includedir@,$includedir,;t t
+s,@oldincludedir@,$oldincludedir,;t t
+s,@infodir@,$infodir,;t t
+s,@mandir@,$mandir,;t t
+s,@PACKAGE_NAME@,$PACKAGE_NAME,;t t
+s,@PACKAGE_TARNAME@,$PACKAGE_TARNAME,;t t
+s,@PACKAGE_VERSION@,$PACKAGE_VERSION,;t t
+s,@PACKAGE_STRING@,$PACKAGE_STRING,;t t
+s,@PACKAGE_BUGREPORT@,$PACKAGE_BUGREPORT,;t t
+s,@ECHO_C@,$ECHO_C,;t t
+s,@ECHO_N@,$ECHO_N,;t t
+s,@ECHO_T@,$ECHO_T,;t t
+s,@PATH_SEPARATOR@,$PATH_SEPARATOR,;t t
+s,@DEFS@,$DEFS,;t t
+s,@LIBS@,$LIBS,;t t
+s,@PACKAGE@,$PACKAGE,;t t
+s,@VERSION@,$VERSION,;t t
+s,@CXX@,$CXX,;t t
+s,@CXXFLAGS@,$CXXFLAGS,;t t
+s,@LDFLAGS@,$LDFLAGS,;t t
+s,@CPPFLAGS@,$CPPFLAGS,;t t
+s,@ac_ct_CXX@,$ac_ct_CXX,;t t
+s,@EXEEXT@,$EXEEXT,;t t
+s,@OBJEXT@,$OBJEXT,;t t
+s,@AWK@,$AWK,;t t
+s,@YACC@,$YACC,;t t
+s,@CC@,$CC,;t t
+s,@CFLAGS@,$CFLAGS,;t t
+s,@ac_ct_CC@,$ac_ct_CC,;t t
+s,@INSTALL_PROGRAM@,$INSTALL_PROGRAM,;t t
+s,@INSTALL_SCRIPT@,$INSTALL_SCRIPT,;t t
+s,@INSTALL_DATA@,$INSTALL_DATA,;t t
+s,@LN_S@,$LN_S,;t t
+s,@SET_MAKE@,$SET_MAKE,;t t
+s,@RANLIB@,$RANLIB,;t t
+s,@ac_ct_RANLIB@,$ac_ct_RANLIB,;t t
+s,@CPP@,$CPP,;t t
+s,@U@,$U,;t t
+s,@ANSI2KNR@,$ANSI2KNR,;t t
+s,@ALLOCA@,$ALLOCA,;t t
+s,@LIBOBJS@,$LIBOBJS,;t t
+CEOF
+
+EOF
+
+  cat >>$CONFIG_STATUS <<\EOF
+  # Split the substitutions into bite-sized pieces for seds with
+  # small command number limits, like on Digital OSF/1 and HP-UX.
+  ac_max_sed_lines=48
+  ac_sed_frag=1 # Number of current file.
+  ac_beg=1 # First line for current file.
+  ac_end=$ac_max_sed_lines # Line after last line for current file.
+  ac_more_lines=:
+  ac_sed_cmds=
+  while $ac_more_lines; do
+    if test $ac_beg -gt 1; then
+      sed "1,${ac_beg}d; ${ac_end}q" $tmp/subs.sed >$tmp/subs.frag
+    else
+      sed "${ac_end}q" $tmp/subs.sed >$tmp/subs.frag
+    fi
+    if test ! -s $tmp/subs.frag; then
+      ac_more_lines=false
+    else
+      # The purpose of the label and of the branching condition is to
+      # speed up the sed processing (if there are no `@' at all, there
+      # is no need to browse any of the substitutions).
+      # These are the two extra sed commands mentioned above.
+      (echo ':t
+  /@[a-zA-Z_][a-zA-Z_0-9]*@/!b' && cat $tmp/subs.frag) >$tmp/subs-$ac_sed_frag.sed
+      if test -z "$ac_sed_cmds"; then
+  	ac_sed_cmds="sed -f $tmp/subs-$ac_sed_frag.sed"
+      else
+  	ac_sed_cmds="$ac_sed_cmds | sed -f $tmp/subs-$ac_sed_frag.sed"
+      fi
+      ac_sed_frag=`expr $ac_sed_frag + 1`
+      ac_beg=$ac_end
+      ac_end=`expr $ac_end + $ac_max_sed_lines`
+    fi
+  done
+  if test -z "$ac_sed_cmds"; then
+    ac_sed_cmds=cat
+  fi
+fi # test -n "$CONFIG_FILES"
+
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+for ac_file in : $CONFIG_FILES; do test "x$ac_file" = x: && continue
+  # Support "outfile[:infile[:infile...]]", defaulting infile="outfile.in".
+  case $ac_file in
+  - | *:- | *:-:* ) # input from stdin
+        cat >$tmp/stdin
+        ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  *:* ) ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  * )   ac_file_in=$ac_file.in ;;
+  esac
+
+  # Compute @srcdir@, @top_srcdir@, and @INSTALL@ for subdirectories.
+  ac_dir=`$as_expr X"$ac_file" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
+         X"$ac_file" : 'X\(//\)[^/]' \| \
+         X"$ac_file" : 'X\(//\)$' \| \
+         X"$ac_file" : 'X\(/\)' \| \
+         .     : '\(.\)' 2>/dev/null ||
+echo X"$ac_file" |
+    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/; q; }
+  	  /^X\(\/\/\)[^/].*/{ s//\1/; q; }
+  	  /^X\(\/\/\)$/{ s//\1/; q; }
+  	  /^X\(\/\).*/{ s//\1/; q; }
+  	  s/.*/./; q'`
+  if test "$ac_dir" != "$ac_file" && test "$ac_dir" != .; then
+    { case "$ac_dir" in
+  [\\/]* | ?:[\\/]* ) as_incr_dir=;;
+  *)                      as_incr_dir=.;;
+esac
+as_dummy="$ac_dir"
+for as_mkdir_dir in `IFS='/\\'; set X $as_dummy; shift; echo "$@"`; do
+  case $as_mkdir_dir in
+    # Skip DOS drivespec
+    ?:) as_incr_dir=$as_mkdir_dir ;;
+    *)
+      as_incr_dir=$as_incr_dir/$as_mkdir_dir
+      test -d "$as_incr_dir" || mkdir "$as_incr_dir"
+    ;;
+  esac
+done; }
+
+    ac_dir_suffix="/`echo $ac_dir|sed 's,^\./,,'`"
+    # A "../" for each directory in $ac_dir_suffix.
+    ac_dots=`echo "$ac_dir_suffix" | sed 's,/[^/]*,../,g'`
+  else
+    ac_dir_suffix= ac_dots=
+  fi
+
+  case $srcdir in
+  .)  ac_srcdir=.
+      if test -z "$ac_dots"; then
+         ac_top_srcdir=.
+      else
+         ac_top_srcdir=`echo $ac_dots | sed 's,/$,,'`
+      fi ;;
+  [\\/]* | ?:[\\/]* )
+      ac_srcdir=$srcdir$ac_dir_suffix;
+      ac_top_srcdir=$srcdir ;;
+  *) # Relative path.
+    ac_srcdir=$ac_dots$srcdir$ac_dir_suffix
+    ac_top_srcdir=$ac_dots$srcdir ;;
+  esac
+
+  case $INSTALL in
+  [\\/$]* | ?:[\\/]* ) ac_INSTALL=$INSTALL ;;
+  *) ac_INSTALL=$ac_dots$INSTALL ;;
+  esac
+
+  if test x"$ac_file" != x-; then
+    { echo "$as_me:5479: creating $ac_file" >&5
+echo "$as_me: creating $ac_file" >&6;}
+    rm -f "$ac_file"
+  fi
+  # Let's still pretend it is `configure' which instantiates (i.e., don't
+  # use $as_me), people would be surprised to read:
+  #    /* config.h.  Generated automatically by config.status.  */
+  configure_input="Generated automatically from `echo $ac_file_in |
+                                                 sed 's,.*/,,'` by configure."
+
+  # First look for the input files in the build tree, otherwise in the
+  # src tree.
+  ac_file_inputs=`IFS=:
+    for f in $ac_file_in; do
+      case $f in
+      -) echo $tmp/stdin ;;
+      [\\/$]*)
+         # Absolute (can't be DOS-style, as IFS=:)
+         test -f "$f" || { { echo "$as_me:5497: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         echo $f;;
+      *) # Relative
+         if test -f "$f"; then
+           # Build tree
+           echo $f
+         elif test -f "$srcdir/$f"; then
+           # Source tree
+           echo $srcdir/$f
+         else
+           # /dev/null tree
+           { { echo "$as_me:5510: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         fi;;
+      esac
+    done` || { (exit 1); exit 1; }
+EOF
+cat >>$CONFIG_STATUS <<EOF
+  sed "$ac_vpsub
+$extrasub
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+:t
+/@[a-zA-Z_][a-zA-Z_0-9]*@/!b
+s,@configure_input@,$configure_input,;t t
+s,@srcdir@,$ac_srcdir,;t t
+s,@top_srcdir@,$ac_top_srcdir,;t t
+s,@INSTALL@,$ac_INSTALL,;t t
+" $ac_file_inputs | (eval "$ac_sed_cmds") >$tmp/out
+  rm -f $tmp/stdin
+  if test x"$ac_file" != x-; then
+    mv $tmp/out $ac_file
+  else
+    cat $tmp/out
+    rm -f $tmp/out
+  fi
+
+done
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+
+#
+# CONFIG_HEADER section.
+#
+
+# These sed commands are passed to sed as "A NAME B NAME C VALUE D", where
+# NAME is the cpp macro being defined and VALUE is the value it is being given.
+#
+# ac_d sets the value in "#define NAME VALUE" lines.
+ac_dA='s,^\([ 	]*\)#\([ 	]*define[ 	][ 	]*\)'
+ac_dB='[ 	].*$,\1#\2'
+ac_dC=' '
+ac_dD=',;t'
+# ac_u turns "#undef NAME" without trailing blanks into "#define NAME VALUE".
+ac_uA='s,^\([ 	]*\)#\([ 	]*\)undef\([ 	][ 	]*\)'
+ac_uB='$,\1#\2define\3'
+ac_uC=' '
+ac_uD=',;t'
+
+for ac_file in : $CONFIG_HEADERS; do test "x$ac_file" = x: && continue
+  # Support "outfile[:infile[:infile...]]", defaulting infile="outfile.in".
+  case $ac_file in
+  - | *:- | *:-:* ) # input from stdin
+        cat >$tmp/stdin
+        ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  *:* ) ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  * )   ac_file_in=$ac_file.in ;;
+  esac
+
+  test x"$ac_file" != x- && { echo "$as_me:5571: creating $ac_file" >&5
+echo "$as_me: creating $ac_file" >&6;}
+
+  # First look for the input files in the build tree, otherwise in the
+  # src tree.
+  ac_file_inputs=`IFS=:
+    for f in $ac_file_in; do
+      case $f in
+      -) echo $tmp/stdin ;;
+      [\\/$]*)
+         # Absolute (can't be DOS-style, as IFS=:)
+         test -f "$f" || { { echo "$as_me:5582: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         echo $f;;
+      *) # Relative
+         if test -f "$f"; then
+           # Build tree
+           echo $f
+         elif test -f "$srcdir/$f"; then
+           # Source tree
+           echo $srcdir/$f
+         else
+           # /dev/null tree
+           { { echo "$as_me:5595: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         fi;;
+      esac
+    done` || { (exit 1); exit 1; }
+  # Remove the trailing spaces.
+  sed 's/[ 	]*$//' $ac_file_inputs >$tmp/in
+
+EOF
+
+# Transform confdefs.h into two sed scripts, `conftest.defines' and
+# `conftest.undefs', that substitutes the proper values into
+# config.h.in to produce config.h.  The first handles `#define'
+# templates, and the second `#undef' templates.
+# And first: Protect against being on the right side of a sed subst in
+# config.status.  Protect against being in an unquoted here document
+# in config.status.
+rm -f conftest.defines conftest.undefs
+# Using a here document instead of a string reduces the quoting nightmare.
+# Putting comments in sed scripts is not portable.
+#
+# `end' is used to avoid that the second main sed command (meant for
+# 0-ary CPP macros) applies to n-ary macro definitions.
+# See the Autoconf documentation for `clear'.
+cat >confdef2sed.sed <<\EOF
+s/[\\&,]/\\&/g
+s,[\\$`],\\&,g
+t clear
+: clear
+s,^[ 	]*#[ 	]*define[ 	][ 	]*\(\([^ 	(][^ 	(]*\)([^)]*)\)[ 	]*\(.*\)$,${ac_dA}\2${ac_dB}\1${ac_dC}\3${ac_dD},gp
+t end
+s,^[ 	]*#[ 	]*define[ 	][ 	]*\([^ 	][^ 	]*\)[ 	]*\(.*\)$,${ac_dA}\1${ac_dB}\1${ac_dC}\2${ac_dD},gp
+: end
+EOF
+# If some macros were called several times there might be several times
+# the same #defines, which is useless.  Nevertheless, we may not want to
+# sort them, since we want the *last* AC-DEFINE to be honored.
+uniq confdefs.h | sed -n -f confdef2sed.sed >conftest.defines
+sed 's/ac_d/ac_u/g' conftest.defines >conftest.undefs
+rm -f confdef2sed.sed
+
+# This sed command replaces #undef with comments.  This is necessary, for
+# example, in the case of _POSIX_SOURCE, which is predefined and required
+# on some systems where configure will not decide to define it.
+cat >>conftest.undefs <<\EOF
+s,^[ 	]*#[ 	]*undef[ 	][ 	]*[a-zA-Z_][a-zA-Z_0-9]*,/* & */,
+EOF
+
+# Break up conftest.defines because some shells have a limit on the size
+# of here documents, and old seds have small limits too (100 cmds).
+echo '  # Handle all the #define templates only if necessary.' >>$CONFIG_STATUS
+echo '  if egrep "^[ 	]*#[ 	]*define" $tmp/in >/dev/null; then' >>$CONFIG_STATUS
+echo '  # If there are no defines, we may have an empty if/fi' >>$CONFIG_STATUS
+echo '  :' >>$CONFIG_STATUS
+rm -f conftest.tail
+while grep . conftest.defines >/dev/null
+do
+  # Write a limited-size here document to $tmp/defines.sed.
+  echo '  cat >$tmp/defines.sed <<CEOF' >>$CONFIG_STATUS
+  # Speed up: don't consider the non `#define' lines.
+  echo '/^[ 	]*#[ 	]*define/!b' >>$CONFIG_STATUS
+  # Work around the forget-to-reset-the-flag bug.
+  echo 't clr' >>$CONFIG_STATUS
+  echo ': clr' >>$CONFIG_STATUS
+  sed ${ac_max_here_lines}q conftest.defines >>$CONFIG_STATUS
+  echo 'CEOF
+  sed -f $tmp/defines.sed $tmp/in >$tmp/out
+  rm -f $tmp/in
+  mv $tmp/out $tmp/in
+' >>$CONFIG_STATUS
+  sed 1,${ac_max_here_lines}d conftest.defines >conftest.tail
+  rm -f conftest.defines
+  mv conftest.tail conftest.defines
+done
+rm -f conftest.defines
+echo '  fi # egrep' >>$CONFIG_STATUS
+echo >>$CONFIG_STATUS
+
+# Break up conftest.undefs because some shells have a limit on the size
+# of here documents, and old seds have small limits too (100 cmds).
+echo '  # Handle all the #undef templates' >>$CONFIG_STATUS
+rm -f conftest.tail
+while grep . conftest.undefs >/dev/null
+do
+  # Write a limited-size here document to $tmp/undefs.sed.
+  echo '  cat >$tmp/undefs.sed <<CEOF' >>$CONFIG_STATUS
+  # Speed up: don't consider the non `#undef'
+  echo '/^[ 	]*#[ 	]*undef/!b' >>$CONFIG_STATUS
+  # Work around the forget-to-reset-the-flag bug.
+  echo 't clr' >>$CONFIG_STATUS
+  echo ': clr' >>$CONFIG_STATUS
+  sed ${ac_max_here_lines}q conftest.undefs >>$CONFIG_STATUS
+  echo 'CEOF
+  sed -f $tmp/undefs.sed $tmp/in >$tmp/out
+  rm -f $tmp/in
+  mv $tmp/out $tmp/in
+' >>$CONFIG_STATUS
+  sed 1,${ac_max_here_lines}d conftest.undefs >conftest.tail
+  rm -f conftest.undefs
+  mv conftest.tail conftest.undefs
+done
+rm -f conftest.undefs
+
+cat >>$CONFIG_STATUS <<\EOF
+  # Let's still pretend it is `configure' which instantiates (i.e., don't
+  # use $as_me), people would be surprised to read:
+  #    /* config.h.  Generated automatically by config.status.  */
+  if test x"$ac_file" = x-; then
+    echo "/* Generated automatically by configure.  */" >$tmp/config.h
+  else
+    echo "/* $ac_file.  Generated automatically by configure.  */" >$tmp/config.h
+  fi
+  cat $tmp/in >>$tmp/config.h
+  rm -f $tmp/in
+  if test x"$ac_file" != x-; then
+    if cmp -s $ac_file $tmp/config.h 2>/dev/null; then
+      { echo "$as_me:5712: $ac_file is unchanged" >&5
+echo "$as_me: $ac_file is unchanged" >&6;}
+    else
+      ac_dir=`$as_expr X"$ac_file" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
+         X"$ac_file" : 'X\(//\)[^/]' \| \
+         X"$ac_file" : 'X\(//\)$' \| \
+         X"$ac_file" : 'X\(/\)' \| \
+         .     : '\(.\)' 2>/dev/null ||
+echo X"$ac_file" |
+    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/; q; }
+  	  /^X\(\/\/\)[^/].*/{ s//\1/; q; }
+  	  /^X\(\/\/\)$/{ s//\1/; q; }
+  	  /^X\(\/\).*/{ s//\1/; q; }
+  	  s/.*/./; q'`
+      if test "$ac_dir" != "$ac_file" && test "$ac_dir" != .; then
+        { case "$ac_dir" in
+  [\\/]* | ?:[\\/]* ) as_incr_dir=;;
+  *)                      as_incr_dir=.;;
+esac
+as_dummy="$ac_dir"
+for as_mkdir_dir in `IFS='/\\'; set X $as_dummy; shift; echo "$@"`; do
+  case $as_mkdir_dir in
+    # Skip DOS drivespec
+    ?:) as_incr_dir=$as_mkdir_dir ;;
+    *)
+      as_incr_dir=$as_incr_dir/$as_mkdir_dir
+      test -d "$as_incr_dir" || mkdir "$as_incr_dir"
+    ;;
+  esac
+done; }
+
+      fi
+      rm -f $ac_file
+      mv $tmp/config.h $ac_file
+    fi
+  else
+    cat $tmp/config.h
+    rm -f $tmp/config.h
+  fi
+done
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+
+#
+# CONFIG_COMMANDS section.
+#
+for ac_file in : $CONFIG_COMMANDS; do test "x$ac_file" = x: && continue
+  ac_dest=`echo "$ac_file" | sed 's,:.*,,'`
+  ac_source=`echo "$ac_file" | sed 's,[^:]*:,,'`
+
+  case $ac_dest in
+    default ) test -z "$CONFIG_HEADERS" || echo timestamp > stamp-h ;;
+  esac
+done
+EOF
+
+cat >>$CONFIG_STATUS <<\EOF
+
+{ (exit 0); exit 0; }
+EOF
+chmod +x $CONFIG_STATUS
+ac_clean_files=$ac_clean_files_save
+
+# configure is writing to config.log, and then calls config.status.
+# config.status does its own redirection, appending to config.log.
+# Unfortunately, on DOS this fails, as config.log is still kept open
+# by configure, so config.status won't be able to write to it; its
+# output is simply discarded.  So we exec the FD to /dev/null,
+# effectively closing config.log, so it can be properly (re)opened and
+# appended to by config.status.  When coming back to configure, we
+# need to make the FD available again.
+if test "$no_create" != yes; then
+  ac_cs_success=:
+  exec 5>/dev/null
+  $SHELL $CONFIG_STATUS || ac_cs_success=false
+  exec 5>>config.log
+  # Use ||, not &&, to avoid exiting from the if with $? = 1, which
+  # would make configure fail if this is the last instruction.
+  $ac_cs_success || { (exit 1); exit 1; }
+fi
+
Index: /trunk/indexers/mgpp/configure.in
===================================================================
--- /trunk/indexers/mgpp/configure.in	(revision 3365)
+++ /trunk/indexers/mgpp/configure.in	(revision 3365)
@@ -0,0 +1,106 @@
+dnl Process this file with autoconf to produce a configure script.
+AC_INIT(text/MGQuery.cpp)
+AC_CONFIG_HEADER(config.h)
+AC_ARG_PROGRAM
+
+dnl
+PACKAGE=mgpp
+VERSION=1.3
+
+AC_DEFINE_UNQUOTED(PACKAGE, "$PACKAGE")
+AC_DEFINE_UNQUOTED(VERSION, "$VERSION")
+AC_SUBST(PACKAGE)
+AC_SUBST(VERSION)
+
+dnl Checks for programs.
+AC_PROG_CXX
+AC_PROG_AWK
+AC_PROG_YACC
+AC_PROG_CC
+AC_PROG_INSTALL
+AC_PROG_LN_S
+AC_PROG_MAKE_SET
+AC_PROG_RANLIB
+
+dnl Checks for typedefs, structures, and compiler characteristics.
+AC_AIX
+AC_ISC_POSIX
+AC_MINIX
+fp_C_PROTOTYPES
+AC_C_CONST
+AC_TYPE_OFF_T
+AC_TYPE_SIZE_T
+AC_HEADER_TIME
+AC_STRUCT_TM
+
+dnl --- This should be in the stdc checking -----------------
+if test "$ac_cv_prog_cc_stdc" = '-Xc'; then
+dnl The Pyramids running SMP_DC.OSx on MIPS r3000 chips with
+dnl /usr/opt/cc3.11/bin/cc treat -Xc as full ANSI mode - no POSIX or
+dnl XOPEN extensions (like popen).
+dnl Unfortunately -D_{XOPEN,POSIX}_SOURCE can't provide all that mg
+dnl needs. So drop back to -Xa
+AC_TRY_COMPILE(
+[#include <stdio.h>
+], [ FILE *rubbish = popen("yes", "r"); ],,
+[CC="`echo $CC | sed 's/-Xc/-Xa/'`"    ac_cv_prog_cc_stdc='-Xa'])
+fi
+dnl ----------------------
+
+
+dnl Checks for libraries.
+dnl Replace `main' with a function in -lm:
+AC_CHECK_LIB(m, main)
+
+dnl Checks for header files.
+AC_HEADER_DIRENT
+AC_HEADER_STDC
+AC_CHECK_HEADERS(fcntl.h limits.h sys/time.h unistd.h string.h memory.h sys/procfs.h)
+AC_EGREP_HEADER(fread, stdio.h, AC_DEFINE(HAVE_FREAD_DECL))
+AC_EGREP_HEADER(fgetc, stdio.h, AC_DEFINE(HAVE_FGETC_DECL))
+AC_EGREP_HEADER(pr_brksize, sys/procfs.h, AC_DEFINE(HAVE_PR_BRKSIZE))
+
+dnl Checks for library functions.
+AC_FUNC_ALLOCA
+AC_PROG_GCC_TRADITIONAL
+AC_TYPE_SIGNAL
+AC_FUNC_VPRINTF
+AC_CHECK_FUNCS(ftime select strftime strtol getrusage times mallinfo setbuffer getpagesize strerror)
+AC_REPLACE_FUNCS(ftruncate strstr strcasecmp)
+
+
+# *** Custom checking (based on GNU tar configure.in) ***
+# ---------------------------------------------------------------------------
+AC_MSG_CHECKING(for HP-UX needing gmalloc)
+if test "`(uname -s) 2> /dev/null`" = 'HP-UX'; then
+  AC_MSG_RESULT(yes)
+  LIBOBJS="$LIBOBJS gmalloc.o"
+  AC_DEFINE(HAVE_VALLOC)
+else
+  AC_MSG_RESULT(no)
+  AC_CHECK_FUNCS(valloc)
+fi
+
+fp_WITH_DMALLOC
+fp_WITH_REGEX
+fp_WITH_GNU_READLINE
+
+# ---------------------------------------------------------------------------
+if test "$ac_cv_func_alloca" = 'no'; then
+dnl    support functions for alloca.c
+  LIBOBJS="$LIBOBJS xmalloc.o error.o"
+fi
+
+
+# ---------------------------------------------------------------------------
+# see if the user wants to override use of long long
+AC_ARG_ENABLE(override-longlong,
+[  --enable-override-longlong
+                          overide use of long long with GCC],
+  AC_DEFINE(DISABLE_LONG_LONG))
+
+# ---------------------------------------------------------------------------
+
+AC_OUTPUT([Makefile text/Makefile lib/Makefile 
+   jni/Makefile java/org/greenstone/mgpp/Makefile], 
+[test -z "$CONFIG_HEADERS" || echo timestamp > stamp-h])
Index: /trunk/indexers/mgpp/docs/Queryer_help.txt
===================================================================
--- /trunk/indexers/mgpp/docs/Queryer_help.txt	(revision 3365)
+++ /trunk/indexers/mgpp/docs/Queryer_help.txt	(revision 3365)
@@ -0,0 +1,67 @@
+Queryer is a standalone program enabling search and retrieval of documents
+in an mgpp collection. Even when using greenstone, it is useful for testing
+your collections.
+
+Queryer can be run like
+
+Queryer -f <index files> -t <text files>
+
+index and text files is the path to these files down to the collection name. 
+eg
+if your text files are demo/index/text/demo.t, demo/index/text/demo.ti etc, 
+then the text path is demo/index/text/demo
+similary for the index files.
+
+Commands available in the Queryer are:
+
+.q              quit
+.h              print the help message
+.i              change the search level (enter the new level at the prompt)
+.l              change the result level ( ""   ""  )
+.b              full text browse (enter a word or fragment at the prompt)
+.r0/.r1         ranking off/on
+.t0/.t1         query type some/all
+.c0/.c1         casefolding off/on
+.s0/.s1         stemming off/on
+.o0/.o1         short output off/on
+ 
+.p              print a document (enter the docnum at the prompt)
+otherwise just enter a query
+
+
+The current query syntax involves the following:
+
+boolean operators:  & AND | OR ! NOT, with () for precedence
+
+term modifiers: #icus /x - this is stemming, casefolding and weighting like 
+in gsdl
+
+     #i = case insensitive, #c = case sensitive
+     #u = unstemmed, #s = stemmed
+     /x = term weight (default = 1).
+
+eg computer#is/10  is computer, stemmed and casefolded, with a weight of 10
+compared to other terms in the same query
+
+Proximity searching:  NEARx
+this is used to specify the maximum distance apart two words must be to match
+eg dog NEAR4 cat - cat must be within 4 words either side of dog.
+NEAR by itself defaults to 20(??).
+
+fielded searching: [ terms]:Field
+
+eg [Witten]:Creator
+[Witten]:CR
+
+the field names need to be the names of the metadata elements in your 
+collection. If the collection was built with greenstone, these names are the 
+two letter codes found in the build.cfg file. 
+
+Multiple terms inside the [] are ANDed together.
+
+Different fields can be combined using normal boolean stuff, eg
+
+[Witten]:CR & [Gigabytes]:TI
+
+Term modifiers can be included inside the [].
+
Index: /trunk/indexers/mgpp/docs/mgpp_in_greenstone.txt
===================================================================
--- /trunk/indexers/mgpp/docs/mgpp_in_greenstone.txt	(revision 3365)
+++ /trunk/indexers/mgpp/docs/mgpp_in_greenstone.txt	(revision 3365)
@@ -0,0 +1,116 @@
+BUILDING
+
+Greenstone can build collections using mg or mgpp. The default is mg, but you
+can use mgpp by editing the collection configuration file.
+
+First, add the line 'buildtype	 mgpp'
+
+Second, the way indexes are described is different.
+
+mg uses a line like:
+
+indexes   document:text  section:text,Title
+
+This builds two indexes, one of all the text, at document level, the second one
+of all the text and Title metadata, at section level.
+
+The document and section tags determine the granularity of the results of a 
+search. The first index returns document numbers, while the second index 
+returns section numbers.
+
+mgpp does things differently. By default it builds a word level index. Then 
+you specify levels at which you want results returned. For example, in the 
+one index, you might want to be able to retrieve whole documents, and sections.
+
+The greenstone building code builds a word level index, with Document level
+granularity. To add other levels (Section and Paragraph are permitted), you add
+a line like
+
+levels   Section  Paragraph
+
+Note that Paragraph level indexes can be used for searching, but you cant 
+retrieve Paragraph level documents, only Section and Document.
+
+To specify what goes into the index, we use an indexes line, similar to mg but 
+without the level information (it is specified separately by the levels info).
+eg:
+
+indexes	  text
+
+This will index all the text at word level.
+
+To add metadata fields to the index, you can say
+
+indexes   text,Title,Subject       for example, or
+indexes	  text,metadata
+
+The first one builds one index, with tagged entries for Title and Subject 
+metadata. Unlike levels, metadata names can be anything - obviously they 
+should match the names in your documents though.
+
+The second one builds one index with tagged entries for all the metadata it 
+finds - this is useful if you dont know in advance what metadata are available,
+or want all of it indexed anyway.
+
+After the building has finished, the build.cfg file in the building directory
+has a list of what metadata it has found and indexed, for example
+
+indexfields     Subject TextOnly        Title
+indexfieldmap   TextOnly->TX    Subject->SU     Title->TI
+
+The metadata names are  passed to mgpp during building as two letter codes -
+indexfieldmap specifies what codes were used.
+
+By default, only the text is compressed, not the metadata. To change this, you
+ can add a line to the config file like
+
+textcompress	text,Title
+
+this will add Title metadata to the text that gets passed to the compressor.
+
+QUERYING
+
+A collection built with mgpp can be searched in the usual way through 
+greenstone. Search terms can be combined with & and |, phrases are specified using "". Because it uses a word level index, it has some extended searching capability over mg. If metadata has been specified in the index, fielded search can also be done.
+
+The current query syntax involves the following:
+
+boolean operators:  & AND | OR ! NOT, with () for precedence
+
+term modifiers: #icus /x - this is stemming, casefolding and weighting like 
+in gsdl
+
+     #i = case insensitive, #c = case sensitive
+     #u = unstemmed, #s = stemmed
+     /x = term weight (default = 1).
+
+eg computer#is/10  is computer, stemmed and casefolded, with a weight of 10
+compared to other terms in the same query
+
+Proximity searching:  NEARx
+this is used to specify the maximum distance apart two words must be to match
+eg dog NEAR4 cat - cat must be within 4 words either side of dog.
+NEAR by itself defaults to 20(??).
+
+fielded searching: [ terms]:Field
+
+eg [Witten]:CR
+
+the field names need to be the names of the metadata elements in your 
+collection. If the collection was built with greenstone, these names are the two letter codes found in the build.cfg file. 
+
+Multiple terms inside the [] are ANDed together.
+
+Different fields can be combined using normal boolean stuff, eg
+
+[Witten]:CR & [Gigabytes]:TI
+
+Term modifiers can be included inside the [].
+
+
+This syntax can be entered into the standard greenstone search box. For mgpp 
+collections, however, there are additional query pages using forms. These can 
+be accessed through the preferences page - select form query, then simple/
+advanced.
+hopefully the forms are fairly self explanatory.
+
Index: /trunk/indexers/mgpp/docs/standalone_building.txt
===================================================================
--- /trunk/indexers/mgpp/docs/standalone_building.txt	(revision 3365)
+++ /trunk/indexers/mgpp/docs/standalone_building.txt	(revision 3365)
@@ -0,0 +1,99 @@
+BUILDING COLLECTIONS OUTSIDE OF GREENSTONE
+
+
+The programs for building are like mgpp_passes, mgpp_invf_dict etc. they have 
+man pages with them, like mgpp_passes.1 etc. mgpp_passes is the main
+program used for building and the man page has a simple script that gives an 
+example of how to build a collection.
+
+Here is a simple bash script that can be used to build a collection, 
+  
+#! /bin/bash 
+
+
+# The arguments on the command line specify the
+#   source of the text
+source=$@
+
+# This is the name of the collection
+text=demo
+
+echo $source
+
+# Create *.text.stats, *.invf.dict, *.invf.level
+#   *.invf.chunk and *.invf.chunks.trans
+cat ${source} | mgpp_passes -T1 -I1 -f ${text}
+
+# Create *.text.dict
+mgpp_compression_dict -f ${text}
+
+# Create *.invf.dict.hash
+mgpp_perf_hash_build -f ${text}
+
+# Create *.text, *.text.idx, *.text.level
+#   *.invf and *.invf.idx
+cat ${source} | mgpp_passes -T2 -I2 -f ${text}
+
+# Create *.text.weight and *.weight.approx
+mgpp_weights_build -f ${text}
+
+# Create *.invf.dict.blocked
+mgpp_invf_dict -f ${text}
+
+# Create *.invf.dict.blocked.1
+mgpp_stem_idx -s 1 -f ${text}
+
+# Create *.invf.dict.blocked.2
+mgpp_stem_idx -s 2 -f ${text}
+
+# Create *.invf.dict.blocked.3
+mgpp_stem_idx -s 3 -f ${text}
+
+
+This builds a basic collection, using 'Document' as the document level tag, 
+with a word level index.
+
+Format of documents:
+
+There must be a document level tag thats starts (and optionally ends) each 
+document. The default is 'Document'. Text outside these tags is ignored. To 
+change the document tag add '-J <tagname>' to the mgpp_passes commands. There
+can only be one -J option.
+
+Smaller granularity is added using level tags. These must enclose all the
+ text enclosed by the document tags. They can be specified to mgpp_passes by 
+'-K <level tag>'. There can be many -K options.
+
+The smallest level of granularity (default is word level) can be changed by 
+adding '-L <index level>'. This can be no larger than the smallest level.
+
+Metadata or tagged fields are specified like
+<Title>the text of the title </Title>
+
+these are all automatically indexed and can be searched as fields. 
+
+The output files are placed in the current directory. To change where they go,
+use '-d <directory name> ' for all the commands.
+
+The man pages describe all the different options to the various programs.
+
+
+There are four stages to mgpp_passes, specified by T1 and T2 (for text 
+compression) and I1 and I2 (for indexing). T1 and I1 must be done before T2 
+and I2, respectively. As shown above, T1/I1 and T2/I2 can be run together, 
+reducing the number of passes through the documents to 2. But all output files
+get placed in the same directory, and the same text is passed to the indexer
+and the compressor.
+
+In greenstone, we pass only the text to the compressor, but metadata and text
+to the indexer, and text files and index files are put in separate directories.
+
+So mgpp_passes is run four times in this case, like:
+
+cat <text src> | mgpp_passes -d <text_dir> -T1 ...
+cat <text src> | mgpp_passes -d <text_dir> -T2 ...
+cat <index src> | mgpp_passes -d <index_dir> -I1 ...
+cat <index src> | mgpp_passes -d <index_dir> -I2 ...
+
+
+
Index: /trunk/indexers/mgpp/install-sh
===================================================================
--- /trunk/indexers/mgpp/install-sh	(revision 3365)
+++ /trunk/indexers/mgpp/install-sh	(revision 3365)
@@ -0,0 +1,238 @@
+#! /bin/sh
+#
+# install - install a program, script, or datafile
+# This comes from X11R5.
+#
+# Calling this script install-sh is preferred over install.sh, to prevent
+# `make' implicit rules from creating a file called install from it
+# when there is no Makefile.
+#
+# This script is compatible with the BSD install script, but was written
+# from scratch.
+#
+
+
+# set DOITPROG to echo to test this script
+
+# Don't use :- since 4.3BSD and earlier shells don't like it.
+doit="${DOITPROG-}"
+
+
+# put in absolute paths if you don't have them in your path; or use env. vars.
+
+mvprog="${MVPROG-mv}"
+cpprog="${CPPROG-cp}"
+chmodprog="${CHMODPROG-chmod}"
+chownprog="${CHOWNPROG-chown}"
+chgrpprog="${CHGRPPROG-chgrp}"
+stripprog="${STRIPPROG-strip}"
+rmprog="${RMPROG-rm}"
+mkdirprog="${MKDIRPROG-mkdir}"
+
+tranformbasename=""
+transform_arg=""
+instcmd="$mvprog"
+chmodcmd="$chmodprog 0755"
+chowncmd=""
+chgrpcmd=""
+stripcmd=""
+rmcmd="$rmprog -f"
+mvcmd="$mvprog"
+src=""
+dst=""
+dir_arg=""
+
+while [ x"$1" != x ]; do
+    case $1 in
+	-c) instcmd="$cpprog"
+	    shift
+	    continue;;
+
+	-d) dir_arg=true
+	    shift
+	    continue;;
+
+	-m) chmodcmd="$chmodprog $2"
+	    shift
+	    shift
+	    continue;;
+
+	-o) chowncmd="$chownprog $2"
+	    shift
+	    shift
+	    continue;;
+
+	-g) chgrpcmd="$chgrpprog $2"
+	    shift
+	    shift
+	    continue;;
+
+	-s) stripcmd="$stripprog"
+	    shift
+	    continue;;
+
+	-t=*) transformarg=`echo $1 | sed 's/-t=//'`
+	    shift
+	    continue;;
+
+	-b=*) transformbasename=`echo $1 | sed 's/-b=//'`
+	    shift
+	    continue;;
+
+	*)  if [ x"$src" = x ]
+	    then
+		src=$1
+	    else
+		# this colon is to work around a 386BSD /bin/sh bug
+		:
+		dst=$1
+	    fi
+	    shift
+	    continue;;
+    esac
+done
+
+if [ x"$src" = x ]
+then
+	echo "install:	no input file specified"
+	exit 1
+else
+	true
+fi
+
+if [ x"$dir_arg" != x ]; then
+	dst=$src
+	src=""
+	
+	if [ -d $dst ]; then
+		instcmd=:
+	else
+		instcmd=mkdir
+	fi
+else
+
+# Waiting for this to be detected by the "$instcmd $src $dsttmp" command
+# might cause directories to be created, which would be especially bad 
+# if $src (and thus $dsttmp) contains '*'.
+
+	if [ -f $src -o -d $src ]
+	then
+		true
+	else
+		echo "install:  $src does not exist"
+		exit 1
+	fi
+	
+	if [ x"$dst" = x ]
+	then
+		echo "install:	no destination specified"
+		exit 1
+	else
+		true
+	fi
+
+# If destination is a directory, append the input filename; if your system
+# does not like double slashes in filenames, you may need to add some logic
+
+	if [ -d $dst ]
+	then
+		dst="$dst"/`basename $src`
+	else
+		true
+	fi
+fi
+
+## this sed command emulates the dirname command
+dstdir=`echo $dst | sed -e 's,[^/]*$,,;s,/$,,;s,^$,.,'`
+
+# Make sure that the destination directory exists.
+#  this part is taken from Noah Friedman's mkinstalldirs script
+
+# Skip lots of stat calls in the usual case.
+if [ ! -d "$dstdir" ]; then
+defaultIFS='	
+'
+IFS="${IFS-${defaultIFS}}"
+
+oIFS="${IFS}"
+# Some sh's can't handle IFS=/ for some reason.
+IFS='%'
+set - `echo ${dstdir} | sed -e 's@/@%@g' -e 's@^%@/@'`
+IFS="${oIFS}"
+
+pathcomp=''
+
+while [ $# -ne 0 ] ; do
+	pathcomp="${pathcomp}${1}"
+	shift
+
+	if [ ! -d "${pathcomp}" ] ;
+        then
+		$mkdirprog "${pathcomp}"
+	else
+		true
+	fi
+
+	pathcomp="${pathcomp}/"
+done
+fi
+
+if [ x"$dir_arg" != x ]
+then
+	$doit $instcmd $dst &&
+
+	if [ x"$chowncmd" != x ]; then $doit $chowncmd $dst; else true ; fi &&
+	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd $dst; else true ; fi &&
+	if [ x"$stripcmd" != x ]; then $doit $stripcmd $dst; else true ; fi &&
+	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd $dst; else true ; fi
+else
+
+# If we're going to rename the final executable, determine the name now.
+
+	if [ x"$transformarg" = x ] 
+	then
+		dstfile=`basename $dst`
+	else
+		dstfile=`basename $dst $transformbasename | 
+			sed $transformarg`$transformbasename
+	fi
+
+# don't allow the sed command to completely eliminate the filename
+
+	if [ x"$dstfile" = x ] 
+	then
+		dstfile=`basename $dst`
+	else
+		true
+	fi
+
+# Make a temp file name in the proper directory.
+
+	dsttmp=$dstdir/#inst.$$#
+
+# Move or copy the file name to the temp name
+
+	$doit $instcmd $src $dsttmp &&
+
+	trap "rm -f ${dsttmp}" 0 &&
+
+# and set any options; do chmod last to preserve setuid bits
+
+# If any of these fail, we abort the whole thing.  If we want to
+# ignore errors from any of these, just make sure not to ignore
+# errors from the above "$doit $instcmd $src $dsttmp" command.
+
+	if [ x"$chowncmd" != x ]; then $doit $chowncmd $dsttmp; else true;fi &&
+	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd $dsttmp; else true;fi &&
+	if [ x"$stripcmd" != x ]; then $doit $stripcmd $dsttmp; else true;fi &&
+	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd $dsttmp; else true;fi &&
+
+# Now rename the file to the real destination.
+
+	$doit $rmcmd -f $dstdir/$dstfile &&
+	$doit $mvcmd $dsttmp $dstdir/$dstfile 
+
+fi &&
+
+
+exit 0
Index: /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPDocInfo.java
===================================================================
--- /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPDocInfo.java	(revision 3365)
+++ /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPDocInfo.java	(revision 3365)
@@ -0,0 +1,20 @@
+package org.greenstone.mgpp;
+
+/** DocInfo 'struct' class for a search result from mgpp
+ * used by MGPPQueryResult
+ *
+ * @See MGPPQueryResult
+ */
+public class MGPPDocInfo {
+
+    public long num_=0;
+    public float rank_=0;
+
+    public MGPPDocInfo(long doc, float rank) {
+	num_=doc;
+	rank_=rank;
+    }
+    public String toString(){
+	return ""+num_+"("+rank_+")";
+    }
+}
Index: /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPQueryResult.java
===================================================================
--- /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPQueryResult.java	(revision 3365)
+++ /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPQueryResult.java	(revision 3365)
@@ -0,0 +1,97 @@
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+
+/** a java version of QueryResult for mgpp
+ *
+ * contains get methods for the java side, and simple
+ * set methods to be called from c++ side
+ *
+ */
+public class MGPPQueryResult {
+
+    /** the list of MGPPDocInfo */
+    protected Vector docs_=null;
+    /** the list of MGPPTermInfo */
+    protected Vector terms_=null;
+    /** the total number of docs found - not likely to be the size of docs_*/
+    protected long total_num_docs_=0;
+    
+    MGPPQueryResult() {
+	docs_ = new Vector();
+	terms_ = new Vector();
+    }
+    /** clear the info from the last query - should be called before setting any new docs/terms */
+    public void clear() {
+	total_num_docs_=0;
+	docs_.clear();
+	terms_.clear();
+    }
+    /** returns the result as a String - useful for printing out results */
+    public String toString() {
+	String result = "";
+	result += "docs (ranks): ";
+	for (int i=0; i<docs_.size(); i++) {
+	    result += ((MGPPDocInfo)docs_.elementAt(i)).toString()+", ";
+	}
+	result += "\nterms: ";
+	for (int i=0; i<terms_.size(); i++) {
+	    result += ((MGPPTermInfo)terms_.elementAt(i)).toString()+", ";
+	}
+	result += "\nactual number of docs found = "+total_num_docs_;
+
+	return result;
+    }
+    /** a shorter representation - just terms and total docs - not the 
+	individual docnums and ranks */
+    public String toShortString() {
+	String result = "";
+	result += "\nterms: ";
+	for (int i=0; i<terms_.size(); i++) {
+	    result += ((MGPPTermInfo)terms_.elementAt(i)).toString()+", ";
+	}
+	result += "\nactual number of docs found = "+total_num_docs_;
+	return result;
+    }
+
+    // set methods used by c++ code
+    
+    public void setTotalDocs(long num) {
+	total_num_docs_=num;
+    }
+
+    public void addDoc(long doc, float rank) {
+	MGPPDocInfo doc_info = new MGPPDocInfo(doc, rank);
+	docs_.add(doc_info);
+    }
+
+    public void addTerm(String term, String tag, int stem, long match, long freq, String[] equivs) {
+	MGPPTermInfo ti = new MGPPTermInfo();
+	ti.term_=term;
+	ti.tag_=tag;
+	ti.stem_method_=stem;
+	ti.match_docs_=match;
+	ti.term_freq_=freq;
+	if (equivs!=null) {
+	  for (int i=0; i<equivs.length; i++) {
+	    ti.equiv_terms_.add(equivs[i]);
+	  }
+	}
+	terms_.add(ti);
+    }
+
+    // get methods for the java side
+    public Vector getDocs() {
+	return docs_;
+    }
+
+    public Vector getTerms() {
+	return terms_;
+    }
+
+    public long getTotalDocs() {
+	return total_num_docs_;
+    }
+       
+}
+
Index: /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPTermInfo.java
===================================================================
--- /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPTermInfo.java	(revision 3365)
+++ /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPTermInfo.java	(revision 3365)
@@ -0,0 +1,46 @@
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+
+/** TermInfo 'struct' class for a search result from mgpp
+ * used by MGPPQueryResult
+ * 
+ *
+ *@see MGPPQueryResult
+ */
+public class MGPPTermInfo {
+
+    /** the term itself */
+    public String term_=null;
+    /** the tag - level or metadata - for which the query was done */
+    public String tag_=null;
+    /** what stem and casefold method was used
+	0 = none
+	1 = casefold only
+	2 = stem only
+	3 = casefold and stem */
+    public int stem_method_=0;
+    /** the number of documents containing this term (where document is defined by the tag data */
+    public long match_docs_=0;
+    /** overall term freq  - word level*/
+    public long term_freq_=0;
+    /** list of stemmed and casefolded equivalent terms - if stem_method_ is non-zero 
+      - Vector of Strings */
+    public Vector equiv_terms_=null;
+
+    public MGPPTermInfo() {
+	equiv_terms_ = new Vector();
+    }
+	
+    /** output the class as a string */
+    public String toString() {
+	String result="";
+	result +="<"+tag_+">\""+term_+"\"stem("+stem_method_+")docs("+match_docs_;
+	result +=")freq("+term_freq_+")equiv terms(";
+	for (int i=0; i<equiv_terms_.size(); i++) {
+	    result += (String)equiv_terms_.elementAt(i)+", ";
+	}
+	result += ")";
+	return result;
+    }
+}
Index: /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPWrapper.java
===================================================================
--- /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPWrapper.java	(revision 3365)
+++ /trunk/indexers/mgpp/java/org/greenstone/mgpp/MGPPWrapper.java	(revision 3365)
@@ -0,0 +1,105 @@
+
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+
+/** java wrapper class for access to mgpp in C++
+ *
+ * the native side implemented in MGPPWrapperImpl.cpp
+ * uses MGPPQueryResult to hold the result of a query. the 
+ * result of a getDocument is a String
+ * uses the jni
+ *
+ *@see MGPPQueryResult
+ */
+public class MGPPWrapper {
+  
+    /** the query result
+     * will be filled in by runQuery
+     */
+    protected MGPPQueryResult mgpp_query_result_=null; 
+    /** pointer to c++ MGPPWrapperData class - cached indexData and queryInfo
+     */
+    protected long mgpp_data_ptr_=0;
+
+    static {
+	System.loadLibrary("mgppjni");
+	initIDs();
+    }
+    
+    public MGPPWrapper() {
+	mgpp_query_result_ = new MGPPQueryResult();
+	initCppSide();
+    }
+
+    /** initialises field and method IDs for java side to enable access on
+	C++ side */
+    private static native void initIDs();
+
+    /** initialises the mgpp_data_ptr_ */
+    private native boolean initCppSide();
+
+    /** returns a document: number docnum at level level 
+     * the base_dir and text_path paths should join together to provide
+     * the absolute location of the mgpp text files eg ..../index/text/demo
+     * returns teh doc in utf-8
+     */
+    public native String getDocument(String base_dir,
+				     String text_path,
+				     String level,
+				     long docnum );
+
+    /** the public method - converts between utf-8 and unicode
+     */
+    // public String getDocument(String base_dir,
+    //		      String text_path,
+    //		      String level,
+    //		      long docnum ) {
+
+    //String utf8_doc = getUtF8Document(base_dir, text_path, level, docnum);
+	
+    /** load up the data structure for index 
+     * - maintains state between requests as can be slow  
+     * base_dir and text_path should join together to provide
+     * the absolute location of the mgpp index files eg ..../index/tt/demo 
+     */
+    public native boolean loadIndexData(String base_dir,
+					String index_path);
+    /** unloads the data */
+    public native boolean unloadIndexData();
+
+    // query param methods
+
+    /** if on=true, sets default stemming on  - its off by default*/
+    public native void setStem(boolean on);
+    /** if on=true, sets default casefolding on - its off by default */
+    public native void setCase(boolean on);
+    /** default is 50 */
+    public native void setMaxDocs(int num);
+    /** if on=true, sorts by rank, otherwise returns in build order - 
+	default is on */
+    public native void setSortByRank(boolean on);
+    /** if on=true, a query returns term freq info - default is on */
+    public native void setReturnTerms(boolean on);
+    /** sets the granularity of the  query  - default 'Section' */
+    public native void setQueryLevel(String level);
+    /** sets the granularity of the result - default 'Section' */
+    public native void setReturnLevel(String level);
+    /** sets the default boolean operator - AND(=1)/OR(=0) */
+    public native void setMatchMode(int mode);
+
+    /** returns a string with all the current query param settings */
+    public native String getQueryParams();
+
+    /** actually carry out the query. Use the set methods to set any query
+	params 
+    * writes the result to query_result
+    */
+    public native void runQuery(String query_string); 
+
+    /** get the result out of the wrapper */
+    public MGPPQueryResult getQueryResult() {
+	return mgpp_query_result_;
+    }
+    
+}
Index: /trunk/indexers/mgpp/java/org/greenstone/mgpp/Makefile.in
===================================================================
--- /trunk/indexers/mgpp/java/org/greenstone/mgpp/Makefile.in	(revision 3365)
+++ /trunk/indexers/mgpp/java/org/greenstone/mgpp/Makefile.in	(revision 3365)
@@ -0,0 +1,64 @@
+
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+bindir = @bindir@
+srcdir = @srcdir@
+libdir = @libdir@
+subdir = java/org/greenstone/mgpp
+top_srcdir = @top_srcdir@
+transform = @program_transform_name@
+
+AR = ar
+CC = @CC@
+CXX = @CXX@
+JAVAC = javac
+JAVAH = javah
+JAR = jar
+CFLAGS = @CFLAGS@ -ansi -DSILENT -DSHORT_SUFFIX
+CXXFLAGS = @CXXFLAGS@ -DSILENT -DSHORT_SUFFIX
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS =  -lstdc++ @LIBS@
+TOUCH = echo timestamp >
+AWK = @AWK@
+
+MGPPHOME=../../../..
+JAVALIBDIR=$(libdir)/java
+JAVACLASSDIR=./classes
+JAVASRCDIR=../../..
+JAVACOPTIONS= -deprecation -g -O
+JAVACLASSPATH=$(JAVACLASSDIR):
+
+datadir = $(prefix)/share
+libexecdir = $(exec_prefix)/libexec
+localedir = $(datadir)/locale
+mandir = $(prefix)/man/man1
+
+
+all: setup compile
+
+setup:
+	mkdir -p $(JAVACLASSDIR)
+
+compile: setup
+	$(JAVAC) -d $(JAVACLASSDIR) -sourcepath $(JAVASRCDIR) $(JAVACOPTIONS) *.java
+	$(JAVAH) -classpath $(CLASSPATH) -d $(MGPPHOME)/jni org.greenstone.mgpp.MGPPWrapper
+	$(JAR) cf mgpp.jar -C $(JAVACLASSDIR) org
+
+clean:
+	rm -rf $(JAVACLASSDIR) mgpp.jar $(MGPPHOME)/jni/org_greenstone_mgpp_MGPPWrapper.h
+
+install:
+	$(MGPPHOME)/mkinstalldirs $(JAVALIBDIR); \
+	cp mgpp.jar $(JAVALIBDIR)/
+
+distclean: clean
+	rm -f Makefile
+
+
Index: /trunk/indexers/mgpp/java/org/greenstone/mgpp/Queryer.java
===================================================================
--- /trunk/indexers/mgpp/java/org/greenstone/mgpp/Queryer.java	(revision 3365)
+++ /trunk/indexers/mgpp/java/org/greenstone/mgpp/Queryer.java	(revision 3365)
@@ -0,0 +1,176 @@
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+
+/** Queryer - java port of c++ Queryer 
+ * - uses MGPPWrapper to access mgpp
+ * only has get document and search - full text browse not implemented
+ *
+ * run as:
+ * java org.greenstone.mgpp.Queryer <basedir> <indexdir> <textdir>
+ *@see MGPPWrapper
+ */
+public class Queryer {
+  
+  /** outputs to std out a help message describing the commands used by 
+   * Queryer 
+   */
+  public void printHelp() {
+	
+	System.out.println( "commands available are:\n"+
+			    "\t.q\t\tquit\n"+
+			    "\t.h\t\tprint this help message\n"+
+			    "\t.d\t\tprint the current query parameter settings\n" +
+			    "\t.i<level>\tchange the search level to level\n"+
+			    "\t.l<level>\tchange the result level to level\n"+
+			    "\t.r0/.r1\t\tranking off/on\n"+
+			    "\t.t0/.t1\t\tquery type some/all\n"+
+			    "\t.c0/.c1\t\tcasefolding off/on\n"+
+			    "\t.s0/.s1\t\tstemming off/on\n"+
+			    "\t.o0/.o1\t\tshort output off/on\n"+
+			    "\t.m<num>\t\tset max docs to return to num\n\n"+
+			    "\t.p<docnum>\tprint document docnum\n"+
+			    "\t<query string>\t do a query\n");
+    }
+
+   
+    public static void main(String[] args) {
+	if (args.length != 3) {
+	    System.out.println("Usage: java org.greenstone.mgpp.Queryer <basedir> <indexdir> <textdir>");
+	    return;
+	}
+	
+	Queryer self = new Queryer();
+	
+	String base_dir = args[0];
+	String text_path = args[2];
+	String index_path = args[1];
+
+	// the jni class to access mgpp stuff
+	MGPPWrapper wrapper = new MGPPWrapper();
+	wrapper.loadIndexData(base_dir, index_path);
+	
+	// the return level
+	String level = "Section";
+
+	System.out.println("Welcome to Java Queryer :-)");
+	self.printHelp();
+
+	BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
+	boolean shortOutput=false;
+
+	String command; // user input
+	char x; // the command letter
+	String data=null; // any auxiliary input to a command
+	while (true) {
+	  System.out.print(">"); // the prompt
+            try {
+                command = br.readLine();
+                command=command.trim();
+		if (command.startsWith(".")) {
+		    // a system command
+		    x = command.charAt(1);
+		    if (command.length() > 2) {
+			data = command.substring(2);
+			data = data.trim();
+			
+		    }
+		    switch (x) {
+		    case 'q': // clean up and exit
+			wrapper.unloadIndexData();
+			return;  
+		    case 'h': // print help message
+			self.printHelp();
+			break;
+		    case 'd': // print query param settings
+			String info = wrapper.getQueryParams();
+			System.out.println(info);
+			break;
+		    case 'p': // print doc
+			int docnum = Integer.parseInt(data);
+			String doc = wrapper.getDocument(base_dir, text_path, 
+							 level, docnum);
+			System.out.println(doc);
+			break;
+		    case 'm': //match docs
+			int match = Integer.parseInt(data);
+			wrapper.setMaxDocs(match);
+			break;
+		    case 's': // set stem on/off
+			int stem = Integer.parseInt(data);
+			if (stem==0 ){
+			    wrapper.setStem(false);
+			} else if(stem==1) {
+			    wrapper.setStem(true);
+			} else {
+			    System.err.println("Error: stem should be 0 or 1");
+			}
+			break;
+		    case 'c': // set case on/off
+			int casef = Integer.parseInt(data);
+			if (casef==0) {
+			    wrapper.setCase(false);
+			} else if (casef==1) {
+			    wrapper.setCase(true);
+			} else {
+			    System.err.println("Error: case should be 0 or 1");
+			}
+			break;
+		    case 'i': // set search level
+			wrapper.setQueryLevel(data);
+			break;
+		    case 'l': // set result level
+			wrapper.setReturnLevel(data);
+			break; 
+		    case 'r': // set rank on/off
+			int rank = Integer.parseInt(data);
+			if (rank==0) {
+			    wrapper.setSortByRank(false);
+			} else if (rank==1) {
+			    wrapper.setSortByRank(true);
+			} else {
+			    System.err.println("Error: rank should be 0 or 1");
+			}
+			break;
+		    case 't': // set query type some/all
+			int type = Integer.parseInt(data);
+			if (type==0 || type==1) {
+			    wrapper.setMatchMode(type);
+			} else {
+			    System.err.println("Error: type should be 0 (some) or 1 (all)");
+			}
+			break;
+		    case 'o': // set output short/long
+			int output = Integer.parseInt(data);
+			if (output==0) {
+			    shortOutput = false;
+			} else if (output==1) {
+			    shortOutput = true;
+			}else {
+			    System.err.println("Error: output should be 0 or 1");
+			}
+			break;
+		    }
+		}
+		else {
+		    // a query
+		    wrapper.runQuery(command);
+		    MGPPQueryResult res = wrapper.getQueryResult();
+		    if (shortOutput) {
+		      System.out.println(res.toShortString());
+		    } else {
+		      System.out.println(res.toString());
+		    }
+		}
+		    
+	    } catch (Exception e) {
+		System.out.println("Queryer error: "+e.getClass() + " "+e.getMessage());
+	    }
+
+	}
+    }
+
+
+}
Index: /trunk/indexers/mgpp/jni/MGPPWrapperImpl.cpp
===================================================================
--- /trunk/indexers/mgpp/jni/MGPPWrapperImpl.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/jni/MGPPWrapperImpl.cpp	(revision 3365)
@@ -0,0 +1,486 @@
+#include <jni.h>
+#include "org_greenstone_mgpp_MGPPWrapper.h"
+#include "MGPPWrapperImpl.h"
+#include "TextGet.h"
+#include "GSDLQueryParser.h"
+#include "MGQuery.h"
+#include <iostream.h>
+#include <strstream.h>
+  
+MGPPWrapperData::MGPPWrapperData() {
+  indexData = new IndexData();
+  queryInfo = new QueryInfo();
+
+  if (queryInfo==NULL) {
+    cerr<<"couldn't allocate new query info\n";
+    if (indexData!=NULL) {
+      delete indexData;
+    }
+  }
+
+  // set all the default params
+  SetCStr(queryInfo->docLevel, "Section"); // the level to search at
+  queryInfo->maxDocs = 50;
+  queryInfo->sortByRank = true;
+  queryInfo->exactWeights = false;
+  queryInfo->needRankInfo = true;
+  queryInfo->needTermFreqs = true;
+
+  UCArrayClear(level);
+  SetCStr(level, "Section"); // the level to return docs at
+  defaultStemMethod=0;
+  defaultBoolCombine=0;
+  
+}
+
+MGPPWrapperData::~MGPPWrapperData() { 
+  if (indexData !=NULL) {
+    delete indexData;
+  }
+  if (queryInfo !=NULL) {
+    delete queryInfo;
+  }
+}
+
+// ********************************************
+// initialisation stuff
+// ********************************************
+
+// cached ids for java stuff 
+jfieldID FID_mgpp_data = NULL; // MGPPWrapperData
+jfieldID FID_query_result = NULL; // MGPPQueryResult
+jmethodID MID_addDoc=NULL; // MGPPQueryResult.addDoc()
+jmethodID MID_addTerm=NULL; // MGPPQueryResult.addTerm()
+jmethodID MID_setTotalDocs=NULL; // MGPPQueryResult.setTotalDocs()
+jmethodID MID_clearResult=NULL; //MGPPQueryResult.clear()
+jclass CID_String=NULL; // class ID of String
+
+/* to access objects and methods on java side, need their field/method ids - 
+ this initialises them at the start to avoid recalculating them each time they
+ are needed 
+Note: the descriptors need to be exactly right, otherwise you get an error 
+saying "no such field" but no reference to the fact that it has the right 
+name but the wrong type.
+Note: apparently the jclass is a local ref and should only work 
+in the method that created it. It seems to work ok, but I'll make it
+ global cos the book said I should, and it may avoid future hassles.
+*/
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_initIDs (JNIEnv *j_env, jclass j_cls) {
+  
+  FID_mgpp_data = j_env->GetFieldID(j_cls, "mgpp_data_ptr_", "J"); //a long-"J"
+  if (FID_mgpp_data==NULL) {
+      cerr <<"field mgpp_data_ptr_ not found"<<endl;
+  } 
+
+  FID_query_result = j_env->GetFieldID(j_cls, "mgpp_query_result_", "Lorg/greenstone/mgpp/MGPPQueryResult;"); // an object -"L<class name>;"
+  if (FID_query_result==NULL) {
+      cerr <<"field mgpp_query_result_ not found"<<endl;
+  }
+  // the methods we want to use
+
+  // addDoc(long doc, float rank)
+  jclass JC_MGPPQueryResult = j_env->FindClass("org/greenstone/mgpp/MGPPQueryResult");
+  MID_addDoc = j_env->GetMethodID(JC_MGPPQueryResult, "addDoc", "(JF)V");
+  if (MID_addDoc==NULL) {
+      cerr <<"method not found"<<endl;
+  }
+  // addTerm(String term, String tag, int stem_method, long match_docs, 
+  // long term_freq, String[] equiv_terms)
+  MID_addTerm = j_env->GetMethodID(JC_MGPPQueryResult, "addTerm", "(Ljava/lang/String;Ljava/lang/String;IJJ[Ljava/lang/String;)V");
+  if (MID_addTerm==NULL) {
+      cerr <<"method addTerm not found"<<endl;
+  }
+  
+  // setTotalDocs(long)
+  MID_setTotalDocs = j_env->GetMethodID(JC_MGPPQueryResult, "setTotalDocs", "(J)V");
+  if (MID_setTotalDocs==NULL) {
+      cerr <<"method setTotalDocs not found"<<endl;
+  }
+  
+  MID_clearResult = j_env->GetMethodID(JC_MGPPQueryResult, "clear", "()V");
+  if (MID_clearResult==NULL) {
+      cerr <<"method clear not found"<<endl;
+  }
+
+  // get the class for String to use in NewObjectArray in runQuery()
+  // FindClass returns a local reference - have to convert it to a global one
+  jclass local_CID_String = j_env->FindClass("java/lang/String");
+  if (local_CID_String==NULL) {
+    cerr <<"java String class not found"<<endl;
+  } else { 
+    /* create a global ref */
+    CID_String = (jclass)j_env->NewGlobalRef(local_CID_String);
+    /* The local reference is no longer useful */
+    j_env->DeleteLocalRef(local_CID_String);
+ 
+    /* Is the global reference created successfully? */
+    if (CID_String == NULL) {
+      return; /* out of memory exception thrown */
+    }
+  }
+  
+}
+
+/* the java side MGPPWrapper has a pointer to a C++ object - MGPPWrapperData
+   initialise this and set the pointer 
+*/
+JNIEXPORT jboolean JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_initCppSide (JNIEnv *j_env, jobject j_obj){
+
+  MGPPWrapperData * data = new MGPPWrapperData();
+  j_env->SetIntField(j_obj, FID_mgpp_data, (long)data);
+
+  return true;
+
+}
+
+//****************************************************
+// retrieve a document
+//****************************************************
+
+/* returns a document from mgpp as a string
+Note: TextData isn't cached - just reloaded each time
+*/
+JNIEXPORT jstring JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_getDocument (JNIEnv *j_env, 
+ jobject j_obj, jstring j_base_dir, 
+ jstring j_text_name, jstring j_level, jlong j_docnum) {
+  
+  
+  const char * base_dir = j_env->GetStringUTFChars(j_base_dir, NULL);
+  if (base_dir==NULL) {
+    return NULL; // exception already thrown
+  }
+  const char * text_name = j_env->GetStringUTFChars(j_text_name, NULL);
+  
+  if (text_name==NULL) {
+   j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+   return NULL;
+  }
+
+  const char * level = j_env->GetStringUTFChars( j_level, NULL);
+  if (level==NULL) {
+   j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+   j_env->ReleaseStringUTFChars(j_text_name, text_name);
+   return NULL;
+  }
+    
+  // does this work alright? j_docnum is a long (64 bit)
+  unsigned long docnum = j_docnum;
+  TextData td;
+
+  // cast to char* otherwise complains about const
+  td.LoadData((char *)base_dir, (char *)text_name);
+
+  UCArray mg_level;
+  SetCStr(mg_level, level);
+  UCArray docText;
+  docText.clear();
+  // get the actual text
+  if (!GetDocText(td, mg_level, docnum, docText)) {
+    cerr <<"couldn't retrieve doc text"<<endl;
+  }
+
+  td.UnloadData();
+  
+  char * doc = GetCStr(docText); // do I need to free this char *??
+  jstring result = j_env->NewStringUTF(doc);
+  // release any gets
+  j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+  j_env->ReleaseStringUTFChars(j_text_name, text_name);
+  j_env->ReleaseStringUTFChars(j_level, level);
+
+  // free any C++ stuff
+  delete doc;
+
+  return result;
+ 
+}
+
+//******************************************
+// do a query
+// ****************************************
+
+/* load the IndexData - cached for querying
+ */
+JNIEXPORT jboolean JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_loadIndexData (JNIEnv *j_env, jobject j_obj, jstring j_base_dir, jstring j_index_name) {
+
+  jint data_ptr = j_env->GetIntField(j_obj, FID_mgpp_data);
+  MGPPWrapperData * data = (MGPPWrapperData *)data_ptr;
+
+  const char * base_dir = j_env->GetStringUTFChars( j_base_dir, NULL);
+  if (base_dir==NULL) {
+    return false; // exception already thrown
+  }
+
+  const char * index_name = j_env->GetStringUTFChars( j_index_name, NULL);
+  if (index_name==NULL) {
+    j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+    return false;
+  }
+  
+  jboolean j_result=false;
+  
+  // why doesn't this complain about const??
+  if (data->indexData->LoadData(base_dir, index_name)) {
+    j_result=true;
+  }
+  
+  // release any gets
+  j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+  j_env->ReleaseStringUTFChars(j_index_name, index_name);
+  
+  return j_result;
+}
+
+/* unload the data 
+ */
+JNIEXPORT jboolean JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_unloadIndexData (JNIEnv *j_env, jobject j_obj) {
+
+  jint data_ptr = j_env->GetIntField(j_obj, FID_mgpp_data);
+  MGPPWrapperData * data = (MGPPWrapperData *)data_ptr;
+
+  data->indexData->UnloadData();
+  return true;
+
+}
+
+/* do the actual query - the results are written to query_result held on the 
+   java side */
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_runQuery (JNIEnv *j_env, jobject j_obj, jstring j_query){
+  
+  jthrowable exc; // an exception - check if something funny has happened
+  const char *query = j_env->GetStringUTFChars(j_query, NULL);
+  if (query==NULL) {
+    return; // exception already thrown
+  }
+  // turn to UCArray for mgpp and then release the string
+  UCArray queryArray;
+  SetCStr(queryArray, query);
+  j_env->ReleaseStringUTFChars(j_query, query);
+  
+  // the query data
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  
+  // the result to write to
+  jobject result_ptr = j_env->GetObjectField(j_obj, FID_query_result);
+  if (result_ptr==NULL) {
+    cerr <<"couldn't access the result to write to"<<endl;
+    return;
+  }
+  
+  // clear the result
+  j_env->CallVoidMethod(result_ptr, MID_clearResult);
+  exc = j_env->ExceptionOccurred(); // this catches teh exception I think - it
+  //wont be thrown any further
+  if (exc) {
+    j_env->ExceptionDescribe();
+    return;
+  }
+  // the mgpp QueryResult that we will use
+  ExtQueryResult queryResult;
+
+  QueryNode * queryTree = NULL;
+  // parse the query string into a tree structure
+  queryTree = ParseQuery(queryArray, data->defaultBoolCombine, 
+			 data->defaultStemMethod);
+  // print the query
+  PrintNode (cout, queryTree);
+  // finally, do the query
+  MGQuery(*(data->indexData), *(data->queryInfo), queryTree, queryResult, data->level);
+
+  delete queryTree;
+
+  // convert queryResult to the java side version
+  // use levels rather than docs of ExtQueryResult
+  // CallVoidMethod(obj, method id, args to method)
+  for (int i=0; i<queryResult.levels.size(); i++) {
+    jlong doc = queryResult.levels[i];
+    jfloat rank = queryResult.ranks[i];
+    j_env->CallVoidMethod(result_ptr, MID_addDoc, doc, rank);
+    exc = j_env->ExceptionOccurred();
+    if (exc) {
+      j_env->ExceptionDescribe();
+      return;
+    }
+    
+  }
+
+  // actual num of docs
+  jlong total = queryResult.actualNumDocs;
+  j_env->CallVoidMethod(result_ptr, MID_setTotalDocs, total);
+  exc = j_env->ExceptionOccurred();
+  if (exc) {
+    j_env->ExceptionDescribe();
+    return;
+  }
+
+  // the terms
+  for (int j=0; j<queryResult.termFreqs.size(); j++) {
+    
+    TermFreqData tf = queryResult.termFreqs[j];
+    jstring term = j_env->NewStringUTF(GetCStr(tf.term));
+    jstring tag = j_env->NewStringUTF(GetCStr(tf.tag));
+    jint stem = tf.stemMethod;
+    jlong match = tf.matchDocs;
+    jlong freq = tf.termFreq;
+    
+    jobjectArray equivs=NULL;
+    jstring empty = j_env->NewStringUTF(""); // the initial object to fill the array
+    jint num_equivs = tf.equivTerms.size();
+    equivs = j_env->NewObjectArray(num_equivs, CID_String, empty);
+    if (equivs==NULL) {
+      cerr<<"couldn't create object array"<<endl;
+      
+    } else {
+      for (int k=0; k<num_equivs;k++) {
+	jstring equiv = j_env->NewStringUTF(GetCStr(tf.equivTerms[k]));
+	j_env->SetObjectArrayElement(equivs, k, equiv);
+      }
+      
+      
+      j_env->CallVoidMethod(result_ptr, MID_addTerm, term, tag, stem, match, freq, equivs);
+      exc = j_env->ExceptionOccurred();
+      if (exc) {
+	j_env->ExceptionDescribe();
+	return;
+      }
+    }
+  }
+
+
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setStem (JNIEnv *j_env, 
+					      jobject j_obj, 
+					      jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  if (j_on) {
+    data->defaultStemMethod |= 2;
+  } else {
+    data->defaultStemMethod &= 0xd;
+  }
+
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setCase (JNIEnv *j_env, 
+					      jobject j_obj, 
+					      jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+
+  if (j_on) {
+    data->defaultStemMethod |= 1;
+  } else {
+    data->defaultStemMethod &= 0xe;
+  }
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setMaxDocs (JNIEnv *j_env, 
+						 jobject j_obj, 
+						 jint j_max) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  data->queryInfo->maxDocs=j_max;
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setSortByRank (JNIEnv *j_env, 
+						    jobject j_obj, 
+						    jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  
+  data->queryInfo->sortByRank=j_on;
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setReturnTerms(JNIEnv *j_env, 
+						    jobject j_obj, 
+						    jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  data->queryInfo->needTermFreqs = j_on;
+ 
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setQueryLevel(JNIEnv *j_env, 
+						   jobject j_obj, 
+						   jstring j_level){
+  
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+
+  const char * level = j_env->GetStringUTFChars(j_level, NULL);
+  if (level==NULL) {
+    return; // exception already thrown
+  }
+  
+  data->queryInfo->docLevel.clear();
+  SetCStr(data->queryInfo->docLevel, level);
+  
+  // release the java stuff
+  j_env->ReleaseStringUTFChars(j_level, level);
+ 
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setReturnLevel(JNIEnv *j_env, 
+						    jobject j_obj, 
+						    jstring j_level){
+
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+
+  const char * level = j_env->GetStringUTFChars(j_level, NULL);
+  if (level==NULL) {
+    return; // exception already thrown
+  }
+  
+  data->level.clear();
+  SetCStr(data->level, level);
+  
+  // release the java stuff
+  j_env->ReleaseStringUTFChars(j_level, level);
+    
+
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setMatchMode (JNIEnv *j_env,
+						   jobject j_obj,
+						   jint j_mode){
+
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  data->defaultBoolCombine=j_mode;
+
+}
+
+JNIEXPORT jstring JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_getQueryParams (JNIEnv *j_env,
+						     jobject j_obj){
+
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  
+  // print the data to an output string stream, then convert to char*, then to 
+  //jstring
+
+  ostrstream output;
+  output << "Query params:"<<endl
+    // need to change this to use platform specific separator for niceness
+	 << "index\t\t"<<data->indexData->basePath<<"/"<<data->indexData->filename<<endl
+	 <<"search level\t"<<GetCStr(data->queryInfo->docLevel)<<endl
+	 <<"result level\t"<<GetCStr(data->level)<<endl
+	 <<"casefold\t"<<(data->defaultStemMethod&1)<<endl
+	 <<"stem\t\t"<<(data->defaultStemMethod&2)<<endl
+	 <<"order by rank\t"<<data->queryInfo->sortByRank<<endl
+	 <<"query type\t"<<(data->defaultBoolCombine==1?"all":"some")<<endl
+	 <<"max docs\t"<<data->queryInfo->maxDocs<<endl<<ends;
+
+  char *result = output.str();
+  jstring j_result = j_env->NewStringUTF(result);
+  delete result;
+  return j_result;
+}
Index: /trunk/indexers/mgpp/jni/MGPPWrapperImpl.h
===================================================================
--- /trunk/indexers/mgpp/jni/MGPPWrapperImpl.h	(revision 3365)
+++ /trunk/indexers/mgpp/jni/MGPPWrapperImpl.h	(revision 3365)
@@ -0,0 +1,15 @@
+#include "MGQuery.h"
+
+// the global stuff we need to do a query
+class MGPPWrapperData {
+ public:
+  IndexData * indexData;
+  QueryInfo * queryInfo;
+  int defaultStemMethod;
+  int defaultBoolCombine;
+  UCArray level; // the return level
+
+  MGPPWrapperData();
+  ~MGPPWrapperData();
+
+};
Index: /trunk/indexers/mgpp/jni/Makefile.in
===================================================================
--- /trunk/indexers/mgpp/jni/Makefile.in	(revision 3365)
+++ /trunk/indexers/mgpp/jni/Makefile.in	(revision 3365)
@@ -0,0 +1,59 @@
+
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+bindir = @bindir@
+srcdir = @srcdir@
+libdir = @libdir@
+subdir = jni
+top_srcdir = @top_srcdir@
+transform = @program_transform_name@
+
+AR = ar
+CC = @CC@
+CXX = @CXX@
+CFLAGS = @CFLAGS@ -ansi -DSILENT -DSHORT_SUFFIX
+CXXFLAGS = @CXXFLAGS@ -DSILENT -DSHORT_SUFFIX
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS = ../lib/libmg.a ../text/libtextin.a -lstdc++ @LIBS@
+TOUCH = echo timestamp >
+AWK = @AWK@
+
+datadir = $(prefix)/share
+libexecdir = $(exec_prefix)/libexec
+localedir = $(datadir)/locale
+mandir = $(prefix)/man/man1
+
+CXXCOMPILE = $(CXX) -c $(DEFS) $(INCLUDES) $(CXXFLAGS)
+
+.SUFFIXES:
+
+.SUFFIXES: .cpp .o
+.cpp.o:
+	$(CXXCOMPILE) $<
+
+
+INCLUDES = -I.. -I../lib -I../text -I/usr/local/share/j2sdk1.4.0/include -I/usr/local/share/j2sdk1.4.0/include/linux
+
+all: compile link
+
+compile:
+	$(CXXCOMPILE) MGPPWrapperImpl.cpp
+link:
+	$(CXX) -shared -o libmgppjni.so MGPPWrapperImpl.o $(LIBS)
+
+clean:
+	rm -rf *.o libmgppjni.so 
+
+distclean: clean
+	rm -f Makefile
+
+install:
+	cp libmgppjni.so  $(libdir)/
+
Index: /trunk/indexers/mgpp/lib/Makefile.in
===================================================================
--- /trunk/indexers/mgpp/lib/Makefile.in	(revision 3365)
+++ /trunk/indexers/mgpp/lib/Makefile.in	(revision 3365)
@@ -0,0 +1,154 @@
+###########################################################################
+#
+# Makefile.in -- Makefile for the general purpose library
+# Copyright (C) 1994  Neil Sharman; (C) 1995 Tim Shimmin
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+#       @(#)Makefile.in	1.3 16 Mar 1994
+#
+###########################################################################
+
+
+PACKAGE = @PACKAGE@
+VERSION = @VERSION@
+ 
+SHELL = /bin/sh
+srcdir = @srcdir@
+subdir = lib
+VPATH = @srcdir@
+ 
+AR = ar
+CC = @CC@
+CXX = @CXX@
+ifneq ($(NZDL), 1)
+CFLAGS = @CFLAGS@ -ansi
+else 
+CFLAGS = @CFLAGS@ -ansi -DSILENT
+endif
+CXXFLAGS = @CXXFLAGS@
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+
+prefix = @prefix@
+datadir = $(prefix)/share
+localedir = $(datadir)/locale
+ 
+COMPILE = $(CC) -c $(DEFS) $(INCLUDES)  $(CFLAGS)
+CXXCOMPILE = $(CXX) -c $(DEFS) $(INCLUDES) $(CXXFLAGS)
+ 
+.SUFFIXES:
+.SUFFIXES: .c .o
+.c.o:
+	$(COMPILE) $<
+
+.SUFFIXES: .cpp .o
+.cpp.o:
+	$(COMPILE) -D__cplusplus $<
+
+ANSI2KNR = @ANSI2KNR@
+o = .@U@o
+ 
+.SUFFIXES: ._c ._o
+.c._c:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $@
+._c._o:
+	@echo $(COMPILE) $<
+	@rm -f _$*.c
+	@ln $< _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+.c._o:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $*._c
+	@echo $(COMPILE) $*._c
+	@rm -f _$*.c
+	@ln $*._c _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+
+INCLUDES = -I.. -I$(srcdir)
+ 
+HEADERS = \
+    bitio_m_random.h   local_strings.h    lovinstem.h \
+    bitio_m_stdio.h    memlib.h           netorder.h \
+    bitio_gen.h        mgheap.h           messages.h \
+    bitio_m.h          huffman.h  \
+    bitio_m_mem.h      perf_hash.h \
+    bitio_m_mems.h      \
+    pathmax.h          getpagesize.h      sysfuncs.h         bitio_m_abstract.h  \
+    random.h           simplefrenchstem.h unitool.h	     longlong.h \
+    non_ansi.h
+
+
+SOURCES = \
+    mgheap.cpp         \
+    huffman.cpp       messages.cpp \
+    bitio_gen.cpp     perf_hash.cpp \
+    local_strings.cpp lovinstem.cpp \
+    bitio_m_mem.cpp   bitio_m_stdio.cpp bitio_m_mems.cpp \
+    alloca.c          xmalloc.c         strstr.cpp \
+    ftruncate.cpp     strcasecmp.cpp    random.cpp \
+    simplefrenchstem.cpp                unitool.cpp       bitio_m_random.cpp \
+    bitio_m_abstract.cpp
+# memlib.cpp removed
+
+OBJECTS =  mgheap$o   \
+    huffman$o         messages$o \
+    bitio_gen$o       perf_hash$o \
+     local_strings$o   lovinstem$o \
+    bitio_m_mem$o     bitio_m_stdio$o   bitio_m_mems$o \
+    xmalloc$o         bitio_m_random$o  bitio_m_abstract$o \
+     random$o          simplefrenchstem$o \
+    unitool$o
+
+DISTFILES = Makefile.in $(HEADERS) $(SOURCES)
+
+all : libmg.a
+
+libmg.a : $(OBJECTS)
+	rm -f libmg.a
+	$(AR) cru libmg.a $(OBJECTS)
+	$(RANLIB) libmg.a
+
+install: 
+installdirs:
+uninstall:
+execinstall:
+execuninstall:
+maninstall:
+manuninstall:
+ 
+dist: $(DISTFILES)
+	for file in $(DISTFILES); do \
+	  ln $$file $(distdir) 2> /dev/null \
+	  || cp -p $$file $(distdir); \
+	done
+
+mostlyclean:
+	rm -f *.o *_.o
+ 
+clean: mostlyclean
+	rm -f libmg.a
+ 
+distclean: clean
+	rm -f Makefile
+ 
+maintainer-clean: distclean
+	@echo "This command is intended only for maintainers to use;"
+	@echo "rebuilding the deleted files may require special tools."
+
+# breaks make distclean (jrm21)
+#Makefile: Makefile.in ../../../config.status
+#	cd ../../.. && CONFIG_FILES=$(subdir)/$@ CONFIG_HEADERS= ./config.status
+ 
+# Tell versions [3.59,3.63) of GNU make not to export all variables.
+# Otherwise a system limit (for SysV at least) may be exceeded.
+.NOEXPORT:
Index: /trunk/indexers/mgpp/lib/WIN32.MAK
===================================================================
--- /trunk/indexers/mgpp/lib/WIN32.MAK	(revision 3365)
+++ /trunk/indexers/mgpp/lib/WIN32.MAK	(revision 3365)
@@ -0,0 +1,98 @@
+###########################################################################
+#
+# win32 makefile -- mgpp\text
+# A component of the Greenstone digital library software
+# from the New Zealand Digital Library Project at the
+# University of Waikato, New Zealand.
+#
+# Copyright (C) 2000  The New Zealand Digital Library Project
+#
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+###########################################################################
+
+GSDLHOME = ..\..\..
+USE_STLPORT = 0
+
+STLPATH = "$(GSDLHOME)\packages\windows\stlport\stlport"
+
+CC = cl
+CPPFLAGS =
+DEFS = -DSHORT_SUFFIX -DSILENT -DHAVE_CONFIG_H -D__WIN32__
+INCLUDES = -I"$(GSDLHOME)"
+!IF $(USE_STLPORT)
+DEFS = $(DEFS) -DGSDL_NAMESPACE_BROKEN -DGSDL_USE_IOS_H -D__STL_NO_NEW_IOSTREAMS
+INCLUDES = $(INCLUDES) -I$(STLPATH)
+!ENDIF
+
+LDFLAGS =
+
+LIBS = 
+
+COMPILE = $(CC) -c $(DEFS) $(INCLUDES)
+LINK = $(CC) $(LDFLAGS)
+
+.SUFFIXES:
+.SUFFIXES: .c .obj .cpp
+.c.obj:
+	$(COMPILE) $<
+.cpp.obj:
+	$(COMPILE) $<
+
+o = .obj
+
+HEADERS = \
+    bitio_m_random.h   local_strings.h    lovinstem.h \
+    bitio_m_stdio.h    memlib.h           netorder.h \
+    bitio_gen.h        mgheap.h           messages.h \
+    bitio_m.h          huffman.h  \
+    bitio_m_mem.h      perf_hash.h \
+    bitio_m_mems.h     getopt_old.h \
+    pathmax.h          getpagesize.h      sysfuncs.h         bitio_m_abstract.h  \
+    random.h           simplefrenchstem.h unitool.h	     longlong.h \
+    non_ansi.h
+
+SOURCES = \
+    mgheap.cpp         \
+    huffman.cpp       messages.cpp \
+    bitio_gen.cpp     perf_hash.cpp \
+    local_strings.cpp lovinstem.cpp \
+    bitio_m_mem.cpp   bitio_m_stdio.cpp bitio_m_mems.cpp \
+    alloca.c          xmalloc.c         strstr.cpp \
+    ftruncate.cpp     strcasecmp.cpp    random.cpp \
+    simplefrenchstem.cpp                unitool.cpp       bitio_m_random.cpp \
+    bitio_m_abstract.cpp getopt_old.c
+
+OBJECTS =  mgheap$o   \
+    huffman$o         messages$o \
+    bitio_gen$o       perf_hash$o \
+     local_strings$o   lovinstem$o \
+    bitio_m_mem$o     bitio_m_stdio$o   bitio_m_mems$o \
+    xmalloc$o         bitio_m_random$o  bitio_m_abstract$o \
+     random$o          simplefrenchstem$o \
+    unitool$o  win32in$o getopt_old$o
+
+all : libmg.lib
+
+libmg.lib: $(OBJECTS)
+	if EXIST libmg.lib del libmg.lib
+	lib /out:libmg.lib $(OBJECTS)
+
+clean:
+	del *$o 
+	del libmg.lib
+
+install:
Index: /trunk/indexers/mgpp/lib/alloca.c
===================================================================
--- /trunk/indexers/mgpp/lib/alloca.c	(revision 3365)
+++ /trunk/indexers/mgpp/lib/alloca.c	(revision 3365)
@@ -0,0 +1,496 @@
+/* alloca.c -- allocate automatically reclaimed memory
+   (Mostly) portable public-domain implementation -- D A Gwyn
+
+   This implementation of the PWB library alloca function,
+   which is used to allocate space off the run-time stack so
+   that it is automatically reclaimed upon procedure exit,
+   was inspired by discussions with J. Q. Johnson of Cornell.
+   J.Otto Tennant <jot@cray.com> contributed the Cray support.
+
+   There are some preprocessor constants that can
+   be defined when compiling for your specific system, for
+   improved efficiency; however, the defaults should be okay.
+
+   The general concept of this implementation is to keep
+   track of all alloca-allocated blocks, and reclaim any
+   that are found to be deeper in the stack than the current
+   invocation.  This heuristic does not reclaim storage as
+   soon as it becomes invalid, but it will do so eventually.
+
+   As a special case, alloca(0) reclaims storage without
+   allocating any.  It is a good idea to use alloca(0) in
+   your main control loop, etc. to force garbage collection.  */
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__  /* [RPAP - Feb 97: WIN32 Port] */
+#  include <win32cfg.h>
+# else
+#  include <sysfuncs.h>
+# endif
+#endif
+
+#ifdef emacs
+#include "blockinput.h"
+#endif
+
+/* If compiling with GCC 2, this file's not needed.  */
+#if !defined (__GNUC__) || __GNUC__ < 2
+
+/* If someone has defined alloca as a macro,
+   there must be some other way alloca is supposed to work.  */
+#ifndef alloca
+
+#ifdef emacs
+#ifdef static
+/* actually, only want this if static is defined as ""
+   -- this is for usg, in which emacs must undefine static
+   in order to make unexec workable
+   */
+#ifndef STACK_DIRECTION
+you
+lose
+-- must know STACK_DIRECTION at compile-time
+#endif /* STACK_DIRECTION undefined */
+#endif /* static */
+#endif /* emacs */
+
+/* If your stack is a linked list of frames, you have to
+   provide an "address metric" ADDRESS_FUNCTION macro.  */
+
+#if defined (CRAY) && defined (CRAY_STACKSEG_END)
+long i00afunc ();
+#define ADDRESS_FUNCTION(arg) (char *) i00afunc (&(arg))
+#else
+#define ADDRESS_FUNCTION(arg) &(arg)
+#endif
+
+#if __STDC__
+typedef void *pointer;
+#else
+typedef char *pointer;
+#endif
+
+#define	NULL	0
+
+/* Different portions of Emacs need to call different versions of
+   malloc.  The Emacs executable needs alloca to call xmalloc, because
+   ordinary malloc isn't protected from input signals.  On the other
+   hand, the utilities in lib-src need alloca to call malloc; some of
+   them are very simple, and don't have an xmalloc routine.
+
+   Non-Emacs programs expect this to call use xmalloc.
+
+   Callers below should use malloc.  */
+
+#ifndef emacs
+#define malloc xmalloc
+#endif
+extern pointer malloc ();
+
+/* Define STACK_DIRECTION if you know the direction of stack
+   growth for your system; otherwise it will be automatically
+   deduced at run-time.
+
+   STACK_DIRECTION > 0 => grows toward higher addresses
+   STACK_DIRECTION < 0 => grows toward lower addresses
+   STACK_DIRECTION = 0 => direction of growth unknown  */
+
+#ifndef STACK_DIRECTION
+#define	STACK_DIRECTION	0	/* Direction unknown.  */
+#endif
+
+#if STACK_DIRECTION != 0
+
+#define	STACK_DIR	STACK_DIRECTION	/* Known at compile-time.  */
+
+#else /* STACK_DIRECTION == 0; need run-time code.  */
+
+static int stack_dir;		/* 1 or -1 once known.  */
+#define	STACK_DIR	stack_dir
+
+static void
+find_stack_direction ()
+{
+  static char *addr = NULL;	/* Address of first `dummy', once known.  */
+  auto char dummy;		/* To get stack address.  */
+
+  if (addr == NULL)
+    {				/* Initial entry.  */
+      addr = ADDRESS_FUNCTION (dummy);
+
+      find_stack_direction ();	/* Recurse once.  */
+    }
+  else
+    {
+      /* Second entry.  */
+      if (ADDRESS_FUNCTION (dummy) > addr)
+	stack_dir = 1;		/* Stack grew upward.  */
+      else
+	stack_dir = -1;		/* Stack grew downward.  */
+    }
+}
+
+#endif /* STACK_DIRECTION == 0 */
+
+/* An "alloca header" is used to:
+   (a) chain together all alloca'ed blocks;
+   (b) keep track of stack depth.
+
+   It is very important that sizeof(header) agree with malloc
+   alignment chunk size.  The following default should work okay.  */
+
+#ifndef	ALIGN_SIZE
+#define	ALIGN_SIZE	sizeof(double)
+#endif
+
+typedef union hdr
+{
+  char align[ALIGN_SIZE];	/* To force sizeof(header).  */
+  struct
+    {
+      union hdr *next;		/* For chaining headers.  */
+      char *deep;		/* For stack depth measure.  */
+    } h;
+} header;
+
+static header *last_alloca_header = NULL;	/* -> last alloca header.  */
+
+/* Return a pointer to at least SIZE bytes of storage,
+   which will be automatically reclaimed upon exit from
+   the procedure that called alloca.  Originally, this space
+   was supposed to be taken from the current stack frame of the
+   caller, but that method cannot be made to work for some
+   implementations of C, for example under Gould's UTX/32.  */
+
+pointer
+alloca (size)
+     unsigned size;
+{
+  auto char probe;		/* Probes stack depth: */
+  register char *depth = ADDRESS_FUNCTION (probe);
+
+#if STACK_DIRECTION == 0
+  if (STACK_DIR == 0)		/* Unknown growth direction.  */
+    find_stack_direction ();
+#endif
+
+  /* Reclaim garbage, defined as all alloca'd storage that
+     was allocated from deeper in the stack than currently. */
+
+  {
+    register header *hp;	/* Traverses linked list.  */
+
+#ifdef emacs
+    BLOCK_INPUT;
+#endif
+
+    for (hp = last_alloca_header; hp != NULL;)
+      if ((STACK_DIR > 0 && hp->h.deep > depth)
+	  || (STACK_DIR < 0 && hp->h.deep < depth))
+	{
+	  register header *np = hp->h.next;
+
+	  free ((pointer) hp);	/* Collect garbage.  */
+
+	  hp = np;		/* -> next header.  */
+	}
+      else
+	break;			/* Rest are not deeper.  */
+
+    last_alloca_header = hp;	/* -> last valid storage.  */
+
+#ifdef emacs
+    UNBLOCK_INPUT;
+#endif
+  }
+
+  if (size == 0)
+    return NULL;		/* No allocation required.  */
+
+  /* Allocate combined header + user data storage.  */
+
+  {
+    register pointer new = malloc (sizeof (header) + size);
+    /* Address of header.  */
+
+    ((header *) new)->h.next = last_alloca_header;
+    ((header *) new)->h.deep = depth;
+
+    last_alloca_header = (header *) new;
+
+    /* User storage begins just after header.  */
+
+    return (pointer) ((char *) new + sizeof (header));
+  }
+}
+
+#if defined (CRAY) && defined (CRAY_STACKSEG_END)
+
+#ifdef DEBUG_I00AFUNC
+#include <stdio.h>
+#endif
+
+#ifndef CRAY_STACK
+#define CRAY_STACK
+#ifndef CRAY2
+/* Stack structures for CRAY-1, CRAY X-MP, and CRAY Y-MP */
+struct stack_control_header
+  {
+    long shgrow:32;		/* Number of times stack has grown.  */
+    long shaseg:32;		/* Size of increments to stack.  */
+    long shhwm:32;		/* High water mark of stack.  */
+    long shsize:32;		/* Current size of stack (all segments).  */
+  };
+
+/* The stack segment linkage control information occurs at
+   the high-address end of a stack segment.  (The stack
+   grows from low addresses to high addresses.)  The initial
+   part of the stack segment linkage control information is
+   0200 (octal) words.  This provides for register storage
+   for the routine which overflows the stack.  */
+
+struct stack_segment_linkage
+  {
+    long ss[0200];		/* 0200 overflow words.  */
+    long sssize:32;		/* Number of words in this segment.  */
+    long ssbase:32;		/* Offset to stack base.  */
+    long:32;
+    long sspseg:32;		/* Offset to linkage control of previous
+				   segment of stack.  */
+    long:32;
+    long sstcpt:32;		/* Pointer to task common address block.  */
+    long sscsnm;		/* Private control structure number for
+				   microtasking.  */
+    long ssusr1;		/* Reserved for user.  */
+    long ssusr2;		/* Reserved for user.  */
+    long sstpid;		/* Process ID for pid based multi-tasking.  */
+    long ssgvup;		/* Pointer to multitasking thread giveup.  */
+    long sscray[7];		/* Reserved for Cray Research.  */
+    long ssa0;
+    long ssa1;
+    long ssa2;
+    long ssa3;
+    long ssa4;
+    long ssa5;
+    long ssa6;
+    long ssa7;
+    long sss0;
+    long sss1;
+    long sss2;
+    long sss3;
+    long sss4;
+    long sss5;
+    long sss6;
+    long sss7;
+  };
+
+#else /* CRAY2 */
+/* The following structure defines the vector of words
+   returned by the STKSTAT library routine.  */
+struct stk_stat
+  {
+    long now;			/* Current total stack size.  */
+    long maxc;			/* Amount of contiguous space which would
+				   be required to satisfy the maximum
+				   stack demand to date.  */
+    long high_water;		/* Stack high-water mark.  */
+    long overflows;		/* Number of stack overflow ($STKOFEN) calls.  */
+    long hits;			/* Number of internal buffer hits.  */
+    long extends;		/* Number of block extensions.  */
+    long stko_mallocs;		/* Block allocations by $STKOFEN.  */
+    long underflows;		/* Number of stack underflow calls ($STKRETN).  */
+    long stko_free;		/* Number of deallocations by $STKRETN.  */
+    long stkm_free;		/* Number of deallocations by $STKMRET.  */
+    long segments;		/* Current number of stack segments.  */
+    long maxs;			/* Maximum number of stack segments so far.  */
+    long pad_size;		/* Stack pad size.  */
+    long current_address;	/* Current stack segment address.  */
+    long current_size;		/* Current stack segment size.  This
+				   number is actually corrupted by STKSTAT to
+				   include the fifteen word trailer area.  */
+    long initial_address;	/* Address of initial segment.  */
+    long initial_size;		/* Size of initial segment.  */
+  };
+
+/* The following structure describes the data structure which trails
+   any stack segment.  I think that the description in 'asdef' is
+   out of date.  I only describe the parts that I am sure about.  */
+
+struct stk_trailer
+  {
+    long this_address;		/* Address of this block.  */
+    long this_size;		/* Size of this block (does not include
+				   this trailer).  */
+    long unknown2;
+    long unknown3;
+    long link;			/* Address of trailer block of previous
+				   segment.  */
+    long unknown5;
+    long unknown6;
+    long unknown7;
+    long unknown8;
+    long unknown9;
+    long unknown10;
+    long unknown11;
+    long unknown12;
+    long unknown13;
+    long unknown14;
+  };
+
+#endif /* CRAY2 */
+#endif /* not CRAY_STACK */
+
+#ifdef CRAY2
+/* Determine a "stack measure" for an arbitrary ADDRESS.
+   I doubt that "lint" will like this much. */
+
+static long
+i00afunc (long *address)
+{
+  struct stk_stat status;
+  struct stk_trailer *trailer;
+  long *block, size;
+  long result = 0;
+
+  /* We want to iterate through all of the segments.  The first
+     step is to get the stack status structure.  We could do this
+     more quickly and more directly, perhaps, by referencing the
+     $LM00 common block, but I know that this works.  */
+
+  STKSTAT (&status);
+
+  /* Set up the iteration.  */
+
+  trailer = (struct stk_trailer *) (status.current_address
+				    + status.current_size
+				    - 15);
+
+  /* There must be at least one stack segment.  Therefore it is
+     a fatal error if "trailer" is null.  */
+
+  if (trailer == 0)
+    abort ();
+
+  /* Discard segments that do not contain our argument address.  */
+
+  while (trailer != 0)
+    {
+      block = (long *) trailer->this_address;
+      size = trailer->this_size;
+      if (block == 0 || size == 0)
+	abort ();
+      trailer = (struct stk_trailer *) trailer->link;
+      if ((block <= address) && (address < (block + size)))
+	break;
+    }
+
+  /* Set the result to the offset in this segment and add the sizes
+     of all predecessor segments.  */
+
+  result = address - block;
+
+  if (trailer == 0)
+    {
+      return result;
+    }
+
+  do
+    {
+      if (trailer->this_size <= 0)
+	abort ();
+      result += trailer->this_size;
+      trailer = (struct stk_trailer *) trailer->link;
+    }
+  while (trailer != 0);
+
+  /* We are done.  Note that if you present a bogus address (one
+     not in any segment), you will get a different number back, formed
+     from subtracting the address of the first block.  This is probably
+     not what you want.  */
+
+  return (result);
+}
+
+#else /* not CRAY2 */
+/* Stack address function for a CRAY-1, CRAY X-MP, or CRAY Y-MP.
+   Determine the number of the cell within the stack,
+   given the address of the cell.  The purpose of this
+   routine is to linearize, in some sense, stack addresses
+   for alloca.  */
+
+static long
+i00afunc (long address)
+{
+  long stkl = 0;
+
+  long size, pseg, this_segment, stack;
+  long result = 0;
+
+  struct stack_segment_linkage *ssptr;
+
+  /* Register B67 contains the address of the end of the
+     current stack segment.  If you (as a subprogram) store
+     your registers on the stack and find that you are past
+     the contents of B67, you have overflowed the segment.
+
+     B67 also points to the stack segment linkage control
+     area, which is what we are really interested in.  */
+
+  stkl = CRAY_STACKSEG_END ();
+  ssptr = (struct stack_segment_linkage *) stkl;
+
+  /* If one subtracts 'size' from the end of the segment,
+     one has the address of the first word of the segment.
+
+     If this is not the first segment, 'pseg' will be
+     nonzero.  */
+
+  pseg = ssptr->sspseg;
+  size = ssptr->sssize;
+
+  this_segment = stkl - size;
+
+  /* It is possible that calling this routine itself caused
+     a stack overflow.  Discard stack segments which do not
+     contain the target address.  */
+
+  while (!(this_segment <= address && address <= stkl))
+    {
+#ifdef DEBUG_I00AFUNC
+      fprintf (stderr, "%011o %011o %011o\n", this_segment, address, stkl);
+#endif
+      if (pseg == 0)
+	break;
+      stkl = stkl - pseg;
+      ssptr = (struct stack_segment_linkage *) stkl;
+      size = ssptr->sssize;
+      pseg = ssptr->sspseg;
+      this_segment = stkl - size;
+    }
+
+  result = address - this_segment;
+
+  /* If you subtract pseg from the current end of the stack,
+     you get the address of the previous stack segment's end.
+     This seems a little convoluted to me, but I'll bet you save
+     a cycle somewhere.  */
+
+  while (pseg != 0)
+    {
+#ifdef DEBUG_I00AFUNC
+      fprintf (stderr, "%011o %011o\n", pseg, size);
+#endif
+      stkl = stkl - pseg;
+      ssptr = (struct stack_segment_linkage *) stkl;
+      size = ssptr->sssize;
+      pseg = ssptr->sspseg;
+      result += size;
+    }
+  return (result);
+}
+
+#endif /* not CRAY2 */
+#endif /* CRAY */
+
+#endif /* no alloca */
+#endif /* not GCC version 2 */
Index: /trunk/indexers/mgpp/lib/bitio_gen.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_gen.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_gen.cpp	(revision 3365)
@@ -0,0 +1,134 @@
+/**************************************************************************
+ *
+ * bitio_gen.c -- General supoport routines for bitio
+ * Copyright (C) 1994  Neil Sharman and Alistair Moffat
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "bitio_m_abstract.h"
+
+int fprintf (FILE *, const char *,...);
+
+unsigned long 
+BIO_Unary_Length (unsigned long val)
+{
+  register unsigned long num;
+  num = bitio_buffer::unary_length(val);
+  //  UNARY_LENGTH (val, num);
+  return (num);
+}
+
+
+unsigned long 
+BIO_Binary_Length (unsigned long val, unsigned long b)
+{
+  register unsigned long num;
+  num = bitio_buffer::binary_length (val, b);
+  return (num);
+}
+
+
+unsigned long BIO_Gamma_Length (unsigned long val) {
+  register unsigned long num;
+  num = bitio_buffer::gamma_length(val);
+  return (num);
+}
+
+
+unsigned long 
+BIO_Delta_Length (unsigned long val)
+{
+  register unsigned long num;
+  num = bitio_buffer::delta_length(val);
+  return (num);
+}
+
+
+unsigned long 
+BIO_Elias_Length (unsigned long val, unsigned long b, double s)
+{
+  register unsigned long num;
+  num = bitio_buffer::elias_length(val, b, s);
+  return (num);
+}
+
+unsigned long 
+BIO_Bblock_Length (unsigned long val, unsigned long b)
+{
+  register unsigned long num;
+  num = bitio_buffer::bblock_length (val, b);
+  return (num);
+}
+
+
+int 
+BIO_Bblock_Init (int N, int p)
+{
+  int b;
+  b = (int) (0.5 + 0.6931471 * N / p);
+  return (b ? b : 1);
+}
+
+
+int 
+BIO_Bblock_Init_W (int N, int p)
+{
+  int logb;
+  logb = bitio_buffer::floor_log2 ((N - p) / p);
+  return (logb < 0 ? 1 : (1 << logb));
+}
+
+int 
+BIO_Bblock_Bound_b (int N, int p, int b)
+{
+  int clogb;
+  clogb = bitio_buffer::ceil_log2 (b);
+  return (p * (1 + clogb) + (N - p * ((1 << clogb) - b + 1)) / b);
+}
+
+int 
+BIO_Bblock_Bound (int N, int p)
+{
+  int b;
+  b = BIO_Bblock_Init_W (N, p);
+  return (BIO_Bblock_Bound_b (N, p, b));
+}
+
+/* adjustment is a hack to overcome inaccuracies in floating point,
+   where (int)(ln4/ln2) can equal 1, not 2. */
+int 
+BIO_Gamma_Bound (int N, int p)
+{
+  return ((int) (p * (2 * log2 (((double) N * 1.0001) / p) + 1)));
+}
+
+int 
+floorlog_2 (int b)
+{
+  int logb;
+  logb = bitio_buffer::floor_log2 (b);
+  return logb;
+}
+
+int 
+ceillog_2 (int b)
+{
+  int logb;
+  logb = bitio_buffer::ceil_log2 (b);
+  return logb;
+}
Index: /trunk/indexers/mgpp/lib/bitio_gen.h
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_gen.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_gen.h	(revision 3365)
@@ -0,0 +1,61 @@
+/**************************************************************************
+ *
+ * bitio_gen.h -- General supoport routines for bitio
+ * Copyright (C) 1994  Neil Sharman and Alistair Moffat
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains function definitions for doing general bitio operations.
+ *
+ *  This file defines functions for calculation the length in bits of a number
+ *  coded unary, binary, delta, gamma, elias, and  bblock coding. It defines 
+ *  routines for calculating the parameter and bounds for bblock coding. 
+ *  Routines for calculating ceil and floor lg are also defined.
+ *
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_GEN
+#define H_BITIO_GEN
+
+unsigned long BIO_Unary_Length (unsigned long val);
+
+unsigned long BIO_Binary_Length (unsigned long val, unsigned long b);
+
+unsigned long BIO_Gamma_Length (unsigned long val);
+
+unsigned long BIO_Delta_Length (unsigned long val);
+
+unsigned long BIO_Elias_Length (unsigned long val, unsigned long b, double s);
+
+unsigned long BIO_Bblock_Length (unsigned long val, unsigned long b);
+
+int BIO_Bblock_Init (int N, int p);
+
+int BIO_Bblock_Init_W (int N, int p);
+
+int BIO_Bblock_Bound_b (int N, int p, int b);
+
+int BIO_Bblock_Bound (int N, int p);
+
+int BIO_Gamma_Bound (int N, int p);
+
+int floorlog_2 (int b);
+
+int ceillog_2 (int b);
+
+#endif
Index: /trunk/indexers/mgpp/lib/bitio_m_abstract.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_abstract.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_abstract.cpp	(revision 3365)
@@ -0,0 +1,9 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_abstract.h"
+
+
+
+
+
Index: /trunk/indexers/mgpp/lib/bitio_m_abstract.h
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_abstract.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_abstract.h	(revision 3365)
@@ -0,0 +1,340 @@
+#ifndef H_BITIO_ABSTRACT
+#define H_BITIO_ABSTRACT
+
+class bitio_buffer {
+public:
+  virtual int bit() = 0;
+  virtual void encodeBit(int bit) = 0;
+  virtual void seek(unsigned long topos) = 0;
+  
+  static inline void check_positive(char *type, unsigned long x) {
+    if (x <= 0) {
+      fprintf(stderr, "Error: Cannot %s encode %lu\n", type, x);
+      exit(1);
+    }
+  }
+
+  static inline int floor_log2(unsigned long x) {
+    unsigned long  _B_x = x;
+    int reply;
+    
+    reply = -1;
+    while (_B_x != 0) {
+      _B_x >>= 1;
+      reply ++;
+    }
+    return reply;
+  }
+  
+  static inline int ceil_log2(unsigned long x) {
+    register int _B_x;
+    unsigned long reply;
+    
+    _B_x = x - 1;
+    reply = 0;
+    while (_B_x != 0) {
+      _B_x >>= 1;
+      reply ++;
+    }
+    return reply;
+  }
+
+  inline unsigned long unary_decode(unsigned long *count) {
+    unsigned long x;
+    
+    x = 1;
+    while (!this->bit()) x++;
+    if (count != NULL) {
+      *count += x;
+    }
+    return x;
+  }
+
+  inline void unary_encode(unsigned long x, unsigned long *count) {
+    register unsigned long _B_x = x;
+    check_positive("unary", _B_x);
+    if (count != NULL) {
+      *count += _B_x;
+    }
+    while(--_B_x) 
+      encodeBit(0);
+    encodeBit(1);
+  }
+
+
+  inline unsigned long binary_decode(unsigned long b, unsigned long *count) {
+    register unsigned long _B_x = 0;
+    register unsigned long _B_b = b;
+    register int _B_i, _B_logofb, _B_thresh;
+    register unsigned long reply;
+    
+    if (_B_b != 1) {					
+      _B_logofb = ceil_log2(_B_b);
+      _B_thresh = (1<<_B_logofb) - _B_b;
+      _B_logofb--;
+      
+      if (count != NULL) {
+	*count += _B_logofb;
+      }
+      
+      for (_B_i=0; _B_i < _B_logofb; _B_i++) {
+	_B_x += _B_x + this->bit();
+      }
+
+      if ((int)_B_x >= _B_thresh) {
+	_B_x += _B_x + this->bit();
+	_B_x -= _B_thresh;
+	if (count != NULL) {
+	  *count = *count + 1;
+	}
+      }
+      reply = _B_x+1;
+      
+    } else reply = 1;
+    
+    return reply;
+  }
+
+  inline void binary_encode(unsigned long x, unsigned long b, unsigned long *count) {
+    register unsigned long _B_x = (x);
+    register unsigned long _B_b = (b);
+    register int _B_nbits, _B_logofb, _B_thresh;
+    
+    check_positive("binary", _B_x);
+    _B_logofb = ceil_log2(_B_b);
+    
+    _B_thresh = (1<<_B_logofb) - _B_b;
+    if ((int)(--_B_x) < _B_thresh) _B_nbits = _B_logofb-1;
+    else {
+      _B_nbits = _B_logofb;
+      _B_x += _B_thresh;
+    }
+    if (count != NULL) {
+      *count += _B_nbits;
+    }
+    while (--_B_nbits>=0)
+      encodeBit((_B_x>>_B_nbits) & 0x1);
+  }
+
+  inline unsigned long bblock_decode(unsigned long b, unsigned long *count) {
+    register unsigned long _B_x1, _B_xx = 0;
+    register unsigned long _B_bb = (b);
+    register int _B_xdivb;
+    
+    _B_xdivb = this->unary_decode(count);
+    _B_xdivb--;				       
+    while (_B_xdivb--) {
+      _B_xx += _B_bb;
+    }
+    _B_x1 = this->binary_decode(_B_bb, count);
+    return (_B_xx + _B_x1);
+  }
+  
+  inline void bblock_encode(unsigned long x, unsigned long b, unsigned long *count) {
+    register unsigned long _B_xx = (x);
+    register unsigned long _B_bb = (b);
+    register int _B_xdivb = 0;
+    check_positive("bblock", _B_xx);
+    _B_xx--;
+    while (_B_xx >= _B_bb) {
+      _B_xdivb++;
+      _B_xx -= _B_bb;
+    }
+    unary_encode(_B_xdivb+1, count);
+    binary_encode(_B_xx+1, _B_bb, count);
+  }
+
+  inline unsigned long elias_decode(unsigned long b, double s, unsigned long *count) {
+    register unsigned long _B_xx;
+    register unsigned long _B_b = b;
+    register double _B_s = s;
+    register int _B_lower=1, _B_upper=1;
+    register int _B_k;
+    register double _B_pow=1.0;
+    
+    _B_k = this->unary_decode(count);
+    _B_k--;
+    while (_B_k--) {
+      _B_lower = _B_upper+1;
+      _B_pow *= _B_s;
+      _B_upper += (int) (_B_b*_B_pow);
+    }
+    _B_xx = this->binary_decode(_B_upper-_B_lower+1, count);
+    _B_xx--;
+    check_positive("gamma", _B_xx+_B_lower);
+    return _B_xx+_B_lower;
+  }
+  
+  inline void elias_encode(unsigned long x, unsigned long b, double s, unsigned long *count) {
+    register unsigned long _B_xx = x;
+    register unsigned long _B_b = b;
+    register double _B_s = s;
+    register int _B_lower=1, _B_upper=1;
+    register int _B_k=0;
+    register double _B_pow=1.0;
+    
+    check_positive("elias", _B_xx);
+    
+    while ((int)_B_xx>_B_upper) {
+      _B_k++;
+      _B_lower = _B_upper+1;
+      _B_pow *= _B_s;
+      _B_upper += (int) (_B_b*_B_pow);
+    }
+    unary_encode(_B_k+1, count);
+    binary_encode(_B_xx-_B_lower+1, _B_upper-_B_lower+1, count);
+  }
+  
+  inline unsigned long gamma_decode(unsigned long *count) {
+    register unsigned long _B_xx = 1;
+    register int _B_nbits = 0;
+    
+    while(!this->bit()) _B_nbits++;
+    if (count != NULL) {
+      *count += _B_nbits*2+1;
+    }
+    while (_B_nbits-- > 0)
+      _B_xx += _B_xx + this->bit();
+    return _B_xx;
+  }
+
+  inline void gamma_encode(unsigned long x, unsigned long *count) {
+    register unsigned long _B_xx = x;
+    register int _B_logofb;
+    register int _B_nbits;
+    
+    check_positive("gamma", _B_xx);
+    _B_logofb = floor_log2(_B_xx);
+    _B_nbits = _B_logofb+1;
+    
+    if (count != NULL) { 
+      *count += _B_logofb*2+1;
+    }
+    while(_B_logofb--) encodeBit(0);
+    while (--_B_nbits>=0) encodeBit((_B_xx>>_B_nbits) & 0x1);
+  }
+  
+  inline unsigned long delta_decode(unsigned long *count) {
+    register unsigned long _B_xxx;
+    register int _B_logx;
+    _B_logx = gamma_decode(count);
+    _B_logx--;
+    _B_xxx = binary_decode(1<<_B_logx, count); 
+    _B_xxx--;
+    return (_B_xxx + (1<<_B_logx));
+  }
+
+  inline void delta_encode(unsigned long x, unsigned long *count) {
+    register unsigned long _B_xxx = x;
+    register int _B_logx;
+    _B_logx = floor_log2(_B_xxx);
+    gamma_encode(_B_logx+1, count);
+    binary_encode(_B_xxx+1, 1<<_B_logx, count);
+  }
+
+  inline unsigned long huff_decode(unsigned long *mcodes, unsigned long **values,
+				   unsigned long *count = NULL) {
+    register unsigned long *__min_code = mcodes;
+    register unsigned long *__mclen    = mcodes;
+    register unsigned long **__values  = values;
+    register unsigned long __code = 0;
+    do {
+      __code += __code + bit();
+      if (count != NULL) (*count) ++;
+    } while (__code < *++__mclen);
+    
+    return __values[__mclen - __min_code][__code - *__mclen];
+  }
+
+  inline void huff_encode(unsigned long x, unsigned long *codes, char *lens,
+			  unsigned long *count = NULL) {
+    register int __i;
+    register int __clen = lens[x];
+    register unsigned long __code = codes[x];
+    for (__i=__clen-1; __i>=0; __i--) {
+      encodeBit((__code >> __i) & 1);
+    }
+    if (count != NULL) {
+      *count += lens[x];
+    }
+  }
+
+  static inline unsigned long unary_length(unsigned long x) {
+    check_positive("unary", x);
+    return x;
+  }
+
+  static inline int binary_length(unsigned long x, unsigned long b) {
+    register unsigned long _B_x = (x);
+    register unsigned long _B_b = (b);
+    register int _B_nbits, _B_logofb, _B_thresh;
+    
+    check_positive("binary", _B_x);
+    _B_logofb = ceil_log2(_B_b);
+    _B_thresh = (1<<_B_logofb) - _B_b;
+    if ((int)(--_B_x) < _B_thresh) _B_nbits = _B_logofb-1;
+    else _B_nbits = _B_logofb;
+    return _B_nbits;
+  }
+
+  static inline unsigned long bblock_length(unsigned long x, unsigned long b) {
+    register unsigned long _B_bcount, count, _B_xx = (x);
+    register unsigned long _B_bb = (b);
+    register int _B_xdivb = 0;
+    
+    check_positive("bblock", _B_xx);
+    _B_xx--;
+    while (_B_xx >= _B_bb) {
+      _B_xdivb++;
+      _B_xx -= _B_bb;
+    }
+    count = unary_length(_B_xdivb+1);
+    _B_bcount = binary_length(_B_xx+1, _B_bb);
+    count += _B_bcount;
+    return count;
+  }
+  
+  static inline unsigned long elias_length(unsigned long x, unsigned long b, 
+					   double s) {
+    register unsigned long _B_xx = x;
+    register unsigned long _B_b = b;
+    register double _B_s = s;
+    register int _B_lower=1, _B_upper=1;
+    register int _B_k=0, _B_ecount;
+    register double _B_pow=1.0;
+    register unsigned long count;
+    check_positive("gamma", _B_xx);
+    
+    while ((int)(_B_xx)>_B_upper) {
+      _B_k++;
+      _B_lower = _B_upper+1;
+      _B_pow *= _B_s;
+      _B_upper += (int) (_B_b*_B_pow);
+    }
+    count = unary_length(_B_k+1);
+    _B_ecount = binary_length(_B_xx-_B_lower+1, _B_upper-_B_lower+1);
+    count += _B_ecount;
+    return count;
+  }
+  
+  static inline unsigned long gamma_length(unsigned long x) {
+    register unsigned long _B_xx = x;
+    register int _B_logofb;
+    check_positive("gamma", _B_xx);
+    _B_logofb = floor_log2(_B_xx);
+    return _B_logofb*2+1;
+  }
+
+  static inline unsigned long delta_length(unsigned long x) {
+    register unsigned long _B_xxx = x;
+    register int _B_logx, _B_dcount;
+    register unsigned long count;
+    _B_logx = floor_log2 (_B_xxx);
+    count = gamma_length(_B_logx+1);
+    _B_dcount = binary_length (_B_xxx+1, 1<<_B_logx);
+    count += _B_dcount;
+    return count;
+  }
+};
+
+#endif
Index: /trunk/indexers/mgpp/lib/bitio_m_mem.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_mem.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_mem.cpp	(revision 3365)
@@ -0,0 +1,128 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_abstract.h"
+#include "bitio_m_mem.h"
+
+mem_bitio_buffer::mem_bitio_buffer() {
+}
+
+mem_bitio_buffer::mem_bitio_buffer(unsigned char *readbuffer, int size) { 
+  pos       = readbuffer;
+  base      = readbuffer;
+  remaining = size;
+  buffer    = 0;
+  btg       = 0;
+}
+
+
+void mem_bitio_buffer::error() {
+  fprintf (stderr, "Unexpected EOF in \"%s\" on line %d\n", __FILE__, __LINE__);
+  exit(1);
+}
+
+void mem_bitio_buffer::encodeStart() {
+  btg = sizeof(buffer) * 8;
+}
+
+
+void mem_bitio_buffer::seek(unsigned long toPos) { 
+  register long _newPos;
+  do {
+    _newPos = toPos;
+    
+    pos    = base + (_newPos >> 3);
+    buffer = *pos++;
+    btg    = 8 - (_newPos & 0x07);
+  } while (0);
+}
+
+long mem_bitio_buffer::addff(long b) {
+  do {
+    if (btg == 0) {
+      if (remaining != 0) {
+	buffer = *pos++;
+	remaining --;
+      } else {
+	buffer = 0xff;
+      }
+      btg = sizeof(buffer) * 8;
+    }
+    b = b + b + ((buffer >> --btg) & 1);
+  } while(0);
+
+  return b;
+}
+
+long mem_bitio_buffer::add00(long b) {
+  do {
+    if (btg == 0) {
+      if (remaining != 0) {
+	buffer = *pos++;
+	remaining --;
+      } else {
+	buffer = 0x00;
+      }
+      btg = sizeof(buffer) * 8;
+    }
+    b = b + b + ((buffer >> --btg) & 1);
+  } while(0);
+  
+  return b;
+}
+
+int mem_bitio_buffer::bit() {
+  if (btg == 0) {
+
+    if (remaining == 0) { 
+      this->error();
+      return 0;
+
+    } else {
+      buffer = *pos++;
+      remaining--;
+      btg = sizeof(buffer) * 8;
+    }
+    
+  }
+  return ((buffer >> --btg) & 0x01);
+}
+
+void mem_bitio_buffer::encodeBit(int b) {
+  btg--;
+  if (b) buffer |= (1 << btg);
+  
+  if (btg == 0) {
+    if (remaining) {
+      *pos++ = buffer;
+      remaining--;
+    }
+    buffer = 0;
+    btg = sizeof(buffer)*8;
+  }
+}
+
+void mem_bitio_buffer::flush() {
+  if (btg != sizeof(buffer)*8) {
+    if (remaining) {
+      *pos++ = buffer;
+      remaining--;
+    }
+  }
+  btg = sizeof(buffer)*8;
+}
+
+void mem_bitio_buffer::done() {
+  // do nothing
+}
+
+int mem_bitio_buffer::encodeLength() {
+  return pos - base;
+}
+
+int mem_bitio_buffer::encodeVerify() {
+  return (remaining != 0);
+}
+
+
+
Index: /trunk/indexers/mgpp/lib/bitio_m_mem.h
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_mem.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_mem.h	(revision 3365)
@@ -0,0 +1,104 @@
+/**************************************************************************
+ *
+ * bitio_m_mem.h -- Macros for bitio to memory
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on an array 
+ *  of chars. These routines are faster than the ones in "mems" files. But 
+ *  with these routines you cannot mix reads and writes on the array of chars,
+ *  or multiple write, at the same time and guarantee them to work, also you 
+ *  cannot seek to a point and do a write. The decode routine can detect when
+ *  you run off the end of the array and will produce an approate error 
+ *  message, and the encode routine will stop when it gets to the end of the 
+ *  character array. 
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_MEM
+#define H_BITIO_M_MEM
+
+#include "bitio_m_abstract.h"
+
+
+class mem_bitio_buffer : public bitio_buffer {
+protected:
+  unsigned char *pos;
+  unsigned char *base;
+  int            remaining;
+  unsigned char  buffer;
+  unsigned char  btg;
+  
+public:
+  mem_bitio_buffer ();
+  mem_bitio_buffer (unsigned char *buffer, int size);
+  void seek (unsigned long topos);
+  void error ();
+  long add00 (long b);
+  long addff (long b);
+  int  bit ();
+  void done ();
+
+  void encodeStart ();
+  void encodeBit (int bit);
+  void encodeDone () { flush(); }
+  int  encodeVerify ();
+  int  encodeLength ();
+  void flush ();
+};
+
+
+/*
+#define ENCODE_START(p,r)						\
+  {									\
+    mem_bitio_buffer buffer(p, r);                                     \
+    buffer.encodeStart();
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    mem_bitio_buffer buffer;                                           \
+    buffer.unpause(b);
+
+#define ENCODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_FLUSH buffer.flush()
+
+#define ENCODE_DONE buffer.flush();}
+
+
+#define DECODE_START(p,r)						\
+  {	                                                                \
+    mem_bitio_buffer buffer(p,r);
+
+#define DECODE_CONTINUE(b)						\
+  {	                                                                \
+    mem_bitio_buffer buffer;                                           \
+    buffer.unpause(b);
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b)
+
+#define DECODE_ADD_00(b) b = buffer.add00(b)
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b) buffer.pause(&b);}
+*/
+
+#endif
Index: /trunk/indexers/mgpp/lib/bitio_m_mems.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_mems.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_mems.cpp	(revision 3365)
@@ -0,0 +1,89 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_abstract.h"
+#include "bitio_m_mems.h"
+
+/**
+ * Create decoding "stream"/"file"
+ */
+mems_bitio_buffer::mems_bitio_buffer()
+{
+}
+
+mems_bitio_buffer::mems_bitio_buffer(unsigned char *readbuffer, long _pos)
+{ 
+  base = readbuffer;
+  pos  = _pos;
+}
+
+void mems_bitio_buffer::encodeStart()
+{
+  // Do nothing
+}
+
+
+void mems_bitio_buffer::error()
+{
+  fprintf(stderr, "Unexpected EOF in \"%s\" on line %d\n", __FILE__, __LINE__);
+  exit(1);
+}
+
+
+void mems_bitio_buffer::seek(unsigned long toPos)
+{ 
+  pos = toPos;
+}
+
+unsigned long mems_bitio_buffer::position()
+{
+  return pos;
+}
+
+long mems_bitio_buffer::addff(long b)
+{
+  do
+    {
+      b = b + b + (base[pos>>3] & (0x80 >> (pos & 0x07)) != 0);
+      pos ++;
+    }
+  while(0);
+
+  return b;
+}
+
+long mems_bitio_buffer::add00(long b)
+{
+  return addff(b);
+}
+
+int mems_bitio_buffer::bit()
+{
+  pos ++;
+  return ((base[(pos-1)>>3] & (0x80 >> ((pos-1) & 0x07))) != 0);
+}
+
+void mems_bitio_buffer::encodeBit(int bit)
+{
+  if (bit)
+    base[pos>>3] |= 0x80 >> (pos & 0x07);
+  else
+    base[pos>>3] &= 0xff7f >> (pos & 0x07);
+  pos++;
+}
+
+void mems_bitio_buffer::flush()
+{
+  // do nothing
+}
+
+void mems_bitio_buffer::done()
+{
+  // do nothing
+}
+
+void mems_bitio_buffer::encodeDone()
+{
+  flush();
+}
+
Index: /trunk/indexers/mgpp/lib/bitio_m_mems.h
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_mems.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_mems.h	(revision 3365)
@@ -0,0 +1,100 @@
+/**************************************************************************
+ *
+ * bitio_m_mems.h -- Macros for bitio to memory (random access)
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on an array 
+ *  of chars. These routines are slower than the ones in "mem" files. but 
+ *  with these routines you can mix reads and writes, or multiple writes,  on
+ *  the array of chars at the same time and guarantee them to work, also you 
+ *  can seek to a point and do a write. The decode and encode macros cannot 
+ *  detect when the end off the character is reached and just continue 
+ *  processing.
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_MEMS
+#define H_BITIO_M_MEMS
+
+#include "bitio_m_abstract.h"
+
+
+class mems_bitio_buffer : public bitio_buffer {
+protected:
+  unsigned char *base;
+  unsigned long  pos;
+  
+public:
+  mems_bitio_buffer();
+  mems_bitio_buffer(unsigned char *readbuffer, long _pos);
+  void seek(unsigned long topos);
+  void error();
+  long add00(long b);
+  long addff(long b);
+  int  bit();
+  //  int  bitOffset();
+  void done();
+
+  unsigned long position();
+
+  void encodeStart();
+  void encodeBit(int b);
+  void encodeDone();
+  void flush();
+};
+
+
+/*#define ENCODE_START(b,p)						\
+  {									\
+    mems_bitio_buffer buffer(b, p);
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    mems_bitio_buffer buffer;                                          \
+    buffer.unpause(b);
+
+#define ENCODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_FLUSH buffer.flush();
+
+
+#define ENCODE_DONE buffer.encodeDone();}
+
+
+#define DECODE_START(b,p)						\
+  {		                                                        \
+    mems_bitio_buffer buffer(b, p);
+
+#define DECODE_CONTINUE(b)						\
+  {									\
+    mems_bitio_buffer buffer;                                          \
+    buffer.unpause(b);
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b);
+
+#define DECODE_ADD_00(b) b = buffer.add00(b);
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b) buffer.pause(&b);}
+*/
+
+#endif
Index: /trunk/indexers/mgpp/lib/bitio_m_random.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_random.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_random.cpp	(revision 3365)
@@ -0,0 +1,218 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "longlong.h"
+#include "sysfuncs.h"
+#include "memlib.h"
+
+#include "bitio_m_abstract.h"
+#include "bitio_m_random.h"
+
+random_bitio_buffer::random_bitio_buffer(FILE *f, unsigned long length)
+{
+  buffer = NULL;
+  attachFile (f, length);
+}
+
+random_bitio_buffer::~random_bitio_buffer() {
+  if (buffer != NULL) delete buffer;
+}
+
+void random_bitio_buffer::attachFile (FILE *f, unsigned long length) {
+  // delete the old buffer
+  if (buffer != NULL) {
+    delete buffer;
+    buffer = NULL;
+  }
+
+  // reset all the values
+  file   = f;
+  pos    = 0;
+  base   = 0;
+  used   = 0;
+  len    = length - 1;
+  shift  = 0;
+
+  // make sure the length is a power of 2
+  while (len != 0) {
+    shift ++;
+    len = len >> 1;
+  }
+  len = 1 << shift;
+
+  // allocate a new buffer
+  buffer = new u_char[len];
+
+  // fill up the buffer
+  if (file != NULL) {
+    fseek(file, 0, 0);
+    fread(buffer, 1, len, file);
+  }
+}
+
+random_bitio_buffer &random_bitio_buffer::operator= (const random_bitio_buffer &_rbb) {
+  // delete the old buffer
+  if (buffer != NULL) {
+    delete buffer;
+    buffer = NULL;
+  }
+
+  file = _rbb.file;
+
+  base = _rbb.base;
+  used = _rbb.used;
+  pos = _rbb.pos;
+  len = _rbb.len;
+  shift = _rbb.shift;
+
+  // copy the buffer
+  buffer = new u_char [len];
+  //  memcpy (buffer, _rbb.buffer, len);
+memcpy (buffer, _rbb.buffer, len);
+  
+  return (*this);
+}
+
+void random_bitio_buffer::error() {
+}
+
+void random_bitio_buffer::writeRead()
+{
+  if (used)
+    {
+      fseek(file, base, 0);
+      fwrite(buffer, 1, len, file);
+    }
+  //  else
+  //    {
+      base += len;
+      fseek(file, base, 0);
+      fread(buffer, 1, len, file);
+      pos = 0;
+      used = 0;
+  //    }
+}
+
+void random_bitio_buffer::encodeBit(int bit)
+{
+  do
+    {
+      if (bit != 0)
+	{
+	  buffer[pos >> 3] |= 0x80 >> (pos & 0x07);
+	}
+      else
+	{
+	  buffer[pos >> 3] &= 0xff7f >> (pos & 0x07);
+	}
+      used = 1;
+      pos ++;
+
+      if ((pos >> 3) >= len)
+	{
+	  writeRead();
+	}
+    }
+  while (0);
+}
+
+void random_bitio_buffer::flush()
+{
+  if (used != 0)
+    {
+      fseek(file, base, 0);
+      fwrite(buffer, 1, len, file);
+      used = 0;
+    }
+}
+
+void random_bitio_buffer::encodeDone()
+{
+  flush();
+  if (buffer != NULL) {
+    delete buffer;
+    buffer = NULL;
+  }
+}
+
+unsigned long random_bitio_buffer::tell()
+{
+  return ((base << 3) + pos);
+}
+
+void random_bitio_buffer::seek(unsigned long toPos)
+{
+  if (((toPos >> 3) >= base) && (((toPos+7) >> 3) < (base + len)))
+    {
+      pos = toPos - (base << 3);
+    }
+  else
+    {
+      flush();
+      base = toPos >> 3;
+      base = (toPos >> (shift + 3)) << shift;
+      fseek(file, base, 0);
+      fread(buffer, 1, len, file);
+      pos = toPos & 0x07;
+      pos = toPos & ((8 << shift) - 1);
+    }
+}
+
+int random_bitio_buffer::bit()
+{
+  //  fprintf(stderr, "Bit in");
+  pos ++;
+  if (((pos - 1) >> 3) >= len)
+    {
+      writeRead();
+      pos = 1;
+    }
+  return (buffer[(pos - 1) >> 3] & (0x80 >> ((pos - 1) & 0x07))) != 0;
+}
+
+long random_bitio_buffer::addff(long b)
+{
+  return b + b + bit();
+}
+
+long random_bitio_buffer::add00(long b)
+{
+  return addff(b);
+}
+
+int random_bitio_buffer::bitOffset()
+{
+  return 0;
+}
+
+void random_bitio_buffer::done()
+{
+  encodeDone();
+}
+
+
+#ifdef USE_LONG_LONG
+void random_bitio_buffer::seek_LL(mg_ullong toPos)
+{
+  if (((toPos >> 3) >= (mg_ullong) base) &&
+      (((toPos + 7) >> 3) < (mg_ullong) (base + len)))
+    {
+      pos = (long) (toPos - (mg_ullong) (base << 3));
+    }
+  else
+    {
+      flush();
+      base = (long) ((toPos >> (shift + 3)) << shift);
+
+      fseek(file, base, 0);
+      fread(buffer, 1, len, file);
+      pos = (long) (toPos & ((8 << shift) - 1));
+    }
+}
+
+mg_ullong random_bitio_buffer::tell_LL()
+{
+  return (((mg_ullong) base) << 3ull) + pos;
+}
+#endif
+
Index: /trunk/indexers/mgpp/lib/bitio_m_random.h
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_random.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_random.h	(revision 3365)
@@ -0,0 +1,127 @@
+/**************************************************************************
+ *
+ * bitio_m_random.h -- Macros for bitio to a file (random access)
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on an array 
+ *  of chars. These routines are slower than the ones in "mem" files. but 
+ *  with these routines you can mix reads and writes, or multiple writes,  on
+ *  the array of chars at the same time and guarantee them to work, also you 
+ *  can seek to a point and do a write. The decode and encode macros cannot 
+ *  detect when the end off the character is reached and just continue 
+ *  processing.
+ *
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_RANDOM
+#define H_BITIO_M_RANDOM
+
+#include "longlong.h"
+#include "bitio_m_abstract.h"
+
+
+class random_bitio_buffer : public bitio_buffer { 
+public:
+  FILE *file;
+  unsigned char *buffer;
+  unsigned long  base;
+  unsigned long  used;
+  unsigned long  pos;
+  unsigned long  len;
+  unsigned long  shift;
+  void writeRead();
+
+  random_bitio_buffer(FILE *f = 0,
+		      unsigned long length = 8*1024);
+  virtual ~random_bitio_buffer();
+  
+  void attachFile (FILE *f, unsigned long length);
+
+  random_bitio_buffer &operator= (const random_bitio_buffer &_rbb);
+  
+  void encodeDone();
+  void encodeBit(int b);
+  void flush();
+  unsigned long tell();
+
+  void seek(unsigned long topos);
+  void error();
+  long add00(long b);
+  long addff(long b);
+  int  bit();
+  int  bitOffset();
+  void done();
+
+#ifdef USE_LONG_LONG
+  // extras for longlong.h
+  void seek_LL(mg_ullong pos);
+  mg_ullong tell_LL();
+#endif
+};
+
+
+/*#define ENCODE_START(f,l)						\
+  {									\
+    random_bitio_buffer buffer(f, l);
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    random_bitio_buffer buffer;                                               \
+    buffer.unpause(b);
+
+#define SEEK fprintf(stderr, "Seek to %d\n",buffer.base)
+#define READ fprintf(stderr, "Read of %d\n",buffer.len)
+#define WRITE fprintf(stderr, "Write of %d\n",buffer.used)
+
+#define WRITE_READ buffer.writeRead()
+
+#define ENCODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_FLUSH buffer.flush();
+
+#define ENCODE_DONE buffer.done();}							
+
+#define DECODE_START(f,l) \
+  {									\
+    random_bitio_buffer buffer(f, l);
+
+#define DECODE_CONTINUE(b) \
+  {									\
+    random_bitio_buffer buffer;                                               \
+    buffer.unpause(b);
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b)
+
+#define DECODE_ADD_00(b) b = buffer.add00(b)
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_TELL buffer.tell()
+
+#define DECODE_TELL buffer.tell()
+*/
+
+
+
+#endif
Index: /trunk/indexers/mgpp/lib/bitio_m_stdio.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_stdio.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_stdio.cpp	(revision 3365)
@@ -0,0 +1,125 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_stdio.h"
+
+stdio_bitio_buffer::stdio_bitio_buffer(FILE *f) {
+  attachFile (f);
+}
+
+void stdio_bitio_buffer::attachFile (FILE *f) {
+  file = f;
+  buffer = 0;
+  btg = 0;
+  bytesWritten = 0;
+}
+
+void stdio_bitio_buffer::encodeStart() {
+  btg = sizeof(buffer) * 8;
+}
+
+void stdio_bitio_buffer::error() {
+  fprintf(stderr, "Unexpected EOF in \"%s\" on line %d", __FILE__, __LINE__);
+  exit(1);
+}
+
+void stdio_bitio_buffer::seek (unsigned long byte, unsigned char bit) {
+  if (fseek (file, byte, SEEK_SET) != 0) return;
+  buffer = getc(file);
+  btg = bit;
+  bytesWritten = 0;
+}
+
+void stdio_bitio_buffer::seek(unsigned long topos) {
+  if (fseek(file, topos >> 3, SEEK_SET) != 0) return;
+  buffer = getc(file);
+  btg = 8 - (topos & 0x07);
+  bytesWritten = 0;
+}
+
+long stdio_bitio_buffer::addff(long b)
+{
+  if (!btg)
+    {
+      register int val;
+      val = getc(file);
+      if (val == EOF)
+	{
+	  buffer = 0x0ff;
+	}
+      else
+	{
+	  buffer = val;
+	}
+      btg = sizeof(buffer) * 8;
+    }
+  b = b + b + ((buffer >> --btg) & 0x01);
+
+  return b;
+}
+
+long stdio_bitio_buffer::add00(long b)
+{
+  if (!btg)
+    {
+      register int val;
+      val = getc(file);
+      if (val == EOF)
+	{
+	  buffer = 0xff;
+	}
+      else
+	{
+	  buffer = val;
+	}
+      btg = sizeof(buffer) * 8;
+    }
+  b = b + b + ((buffer >> --btg) & 0x01);
+  
+  return b;
+}
+
+void stdio_bitio_buffer::encodeBit(int b)
+{
+  btg--;
+  if (b != 0)
+    buffer |= (1 << btg);
+
+  if (btg == 0)
+    {
+      putc(buffer, file);
+      bytesWritten += sizeof (buffer);
+      buffer = 0;
+      btg = sizeof(buffer)*8;
+    }
+}
+
+int stdio_bitio_buffer::bit()
+{
+  if (btg == 0)
+    {
+      buffer = getc(file);
+      if (feof(file))
+	{
+	  error();
+	  return 0;
+	}
+      else
+	{
+	  btg = sizeof(buffer) * 8;
+	}
+    }
+  return ((buffer >> --btg) & 0x01);
+}
+
+void stdio_bitio_buffer::done() {
+  // no tidying required
+}
+
+void stdio_bitio_buffer::encodeDone() {
+  if (btg != sizeof(buffer) * 8) {
+    putc(buffer, file);
+    bytesWritten += sizeof (buffer);
+  }
+}
+
Index: /trunk/indexers/mgpp/lib/bitio_m_stdio.h
===================================================================
--- /trunk/indexers/mgpp/lib/bitio_m_stdio.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/bitio_m_stdio.h	(revision 3365)
@@ -0,0 +1,101 @@
+/**************************************************************************
+ *
+ * bitio_m_stdio.h -- Macros for bitio to a file
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on a FILE*.
+ *  With these routines you cannot mix reads and writes on the FILE,
+ *  or multiple writes, at the same time and guarantee them to work, also you 
+ *  cannot seek to a point and do a write. The decode routine can detect when
+ *  you run off the end of the file and will produce an approate error message.
+ *
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_STDIO
+#define H_BITIO_M_STDIO
+
+#include "bitio_m_abstract.h"
+
+
+class stdio_bitio_buffer : public bitio_buffer {
+protected:
+  FILE *file;
+  unsigned char buffer;
+  unsigned char btg;
+  unsigned long bytesWritten;
+  
+public:
+  stdio_bitio_buffer (FILE *f = NULL);
+
+  void attachFile (FILE *f);
+
+  unsigned char GetBtg () { return btg; }
+  unsigned long GetBytesWritten () { return bytesWritten; }
+  
+  void seek (unsigned long byte, unsigned char bit);
+  void seek (unsigned long topos);
+  void error();
+  long add00(long b);
+  long addff(long b);
+  int  bit();
+  int  bitOffset() { return btg; }
+  void done();
+
+  void encodeStart();
+  void encodeBit(int bit);
+  void encodeDone();
+};
+
+
+/*#define ENCODE_START(f)							\
+  {									\
+    stdio_bitio_buffer buffer(f);                                       \
+    buffer.encodeStart();
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    stdio_bitio_buffer buffer;                                          \
+    buffer.unpause(b);
+
+#define ENCODE_PAUSE(b) buffer.pause(&b);}
+
+#define ENCODE_DONE buffer.encodeDone();}
+
+#define DECODE_START(f)							\
+  {		                                                        \
+    stdio_bitio_buffer buffer(f);
+
+#define DECODE_CONTINUE(b)						\
+  {									\
+    stdio_bitio_buffer buffer;                                          \
+     buffer.unpause(b);
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b)
+
+#define DECODE_ADD_00(b) b = buffer.add00(b)
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b) buffer.pause(&b);}
+*/
+
+#endif
Index: /trunk/indexers/mgpp/lib/ftruncate.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/ftruncate.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/ftruncate.cpp	(revision 3365)
@@ -0,0 +1,99 @@
+/* ftruncate emulations that work on some System V's.
+   This file is in the public domain.  */
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__  /* [RPAP - Feb 97: WIN32 Port] */
+#  include <win32cfg.h>
+# else
+#  include <sysfuncs.h>
+# endif
+#endif
+
+#include <sys/types.h>
+#include <fcntl.h>
+
+#ifdef F_CHSIZE
+
+int
+ftruncate (int fd, off_t length)
+{
+  return fcntl (fd, F_CHSIZE, length);
+}
+
+#else /* not F_CHSIZE */
+#ifdef F_FREESP
+
+/* By William Kucharski <kucharsk@netcom.com>.  */
+
+#include <sys/stat.h>
+#include <errno.h>
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+int
+ftruncate (int fd, off_t length)
+{
+  struct flock fl;
+  struct stat filebuf;
+
+  if (fstat (fd, &filebuf) < 0)
+    return -1;
+
+  if (filebuf.st_size < length)
+    {
+      /* Extend file length. */
+      if (lseek (fd, (length - 1), SEEK_SET) < 0)
+	return -1;
+
+      /* Write a "0" byte. */
+      if (write (fd, "", 1) != 1)
+	return -1;
+    }
+  else
+    {
+
+      /* Truncate length. */
+
+      fl.l_whence = 0;
+      fl.l_len = 0;
+      fl.l_start = length;
+      fl.l_type = F_WRLCK;	/* write lock on file space */
+
+      /* This relies on the *undocumented* F_FREESP argument to fcntl,
+	 which truncates the file so that it ends at the position
+	 indicated by fl.l_start.  Will minor miracles never cease?  */
+
+      if (fcntl (fd, F_FREESP, &fl) < 0)
+	return -1;
+    }
+
+  return 0;
+}
+
+#else /* not F_CHSIZE nor F_FREESP */
+#ifdef HAVE_CHSIZE
+
+int
+ftruncate (int fd, off_t length)
+{
+  return chsize (fd, length);
+}
+
+#else /* not F_CHSIZE nor F_FREESP nor HAVE_CHSIZE */
+
+#include <errno.h>
+#ifndef errno
+extern int errno;
+#endif
+
+int
+ftruncate (int fd, off_t length)
+{
+  errno = EIO;
+  return -1;
+}
+
+#endif /* not HAVE_CHSIZE */
+#endif /* not F_FREESP */
+#endif /* not F_CHSIZE */
Index: /trunk/indexers/mgpp/lib/getopt_old.c
===================================================================
--- /trunk/indexers/mgpp/lib/getopt_old.c	(revision 3365)
+++ /trunk/indexers/mgpp/lib/getopt_old.c	(revision 3365)
@@ -0,0 +1,748 @@
+/* Getopt for GNU.
+   NOTE: getopt is now part of the C library, so if you don't know what
+   "Keep this file name-space clean" means, talk to roland@gnu.ai.mit.edu
+   before changing it!
+
+   Copyright (C) 1987, 88, 89, 90, 91, 92, 1993
+   Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by the
+   Free Software Foundation; either version 2, or (at your option) any
+   later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+
+
+#if !__STDC__ && !defined(const) && IN_GCC
+#define const
+#endif
+
+/* This tells Alpha OSF/1 not to define a getopt prototype in <stdio.h>.  */
+#ifndef _NO_PROTO
+#define _NO_PROTO
+#endif
+
+#include <stdio.h>
+
+/* Comment out all this code if we are using the GNU C Library, and are not
+   actually compiling the library itself.  This code is part of the GNU C
+   Library, but also included in many other GNU distributions.  Compiling
+   and linking in this code is a waste when using the GNU C library
+   (especially if it is a shared library).  Rather than having every GNU
+   program understand `configure --with-gnu-libc' and omit the object files,
+   it is simpler to just do this in the source for each such file.  */
+
+#if defined (_LIBC) || !defined (__GNU_LIBRARY__)
+
+
+/* This needs to come after some library #include
+   to get __GNU_LIBRARY__ defined.  */
+#ifdef	__GNU_LIBRARY__
+/* Don't include stdlib.h for non-GNU C libraries because some of them
+   contain conflicting prototypes for getopt.  */
+#include <stdlib.h>
+#endif /* GNU C library.  */
+
+/* If GETOPT_COMPAT is defined, `+' as well as `--' can introduce a
+   long-named option.  Because this is not POSIX.2 compliant, it is
+   being phased out.  */
+/* #define GETOPT_COMPAT */
+
+/* This version of `getopt' appears to the caller like standard Unix `getopt'
+   but it behaves differently for the user, since it allows the user
+   to intersperse the options with the other arguments.
+
+   As `getopt' works, it permutes the elements of ARGV so that,
+   when it is done, all the options precede everything else.  Thus
+   all application programs are extended to handle flexible argument order.
+
+   Setting the environment variable POSIXLY_CORRECT disables permutation.
+   Then the behavior is completely standard.
+
+   GNU application programs can use a third alternative mode in which
+   they can distinguish the relative order of options and other arguments.  */
+
+/* jrm21 - GSDL: getopt.h -> getopt_old.h */
+#include "getopt_old.h"
+
+/* For communication from `getopt' to the caller.
+   When `getopt' finds an option that takes an argument,
+   the argument value is returned here.
+   Also, when `ordering' is RETURN_IN_ORDER,
+   each non-option ARGV-element is returned here.  */
+
+char *optarg = 0;
+
+/* Index in ARGV of the next element to be scanned.
+   This is used for communication to and from the caller
+   and for communication between successive calls to `getopt'.
+
+   On entry to `getopt', zero means this is the first call; initialize.
+
+   When `getopt' returns EOF, this is the index of the first of the
+   non-option elements that the caller should itself scan.
+
+   Otherwise, `optind' communicates from one call to the next
+   how much of ARGV has been scanned so far.  */
+
+/* XXX 1003.2 says this must be 1 before any call.  */
+int optind = 0;
+
+/* The next char to be scanned in the option-element
+   in which the last option character we returned was found.
+   This allows us to pick up the scan where we left off.
+
+   If this is zero, or a null string, it means resume the scan
+   by advancing to the next ARGV-element.  */
+
+static char *nextchar;
+
+/* Callers store zero here to inhibit the error message
+   for unrecognized options.  */
+
+int opterr = 1;
+
+/* Set to an option character which was unrecognized.
+   This must be initialized on some systems to avoid linking in the
+   system's own getopt implementation.  */
+
+int optopt = '?';
+
+/* Describe how to deal with options that follow non-option ARGV-elements.
+
+   If the caller did not specify anything,
+   the default is REQUIRE_ORDER if the environment variable
+   POSIXLY_CORRECT is defined, PERMUTE otherwise.
+
+   REQUIRE_ORDER means don't recognize them as options;
+   stop option processing when the first non-option is seen.
+   This is what Unix does.
+   This mode of operation is selected by either setting the environment
+   variable POSIXLY_CORRECT, or using `+' as the first character
+   of the list of option characters.
+
+   PERMUTE is the default.  We permute the contents of ARGV as we scan,
+   so that eventually all the non-options are at the end.  This allows options
+   to be given in any order, even with programs that were not written to
+   expect this.
+
+   RETURN_IN_ORDER is an option available to programs that were written
+   to expect options and other ARGV-elements in any order and that care about
+   the ordering of the two.  We describe each non-option ARGV-element
+   as if it were the argument of an option with character code 1.
+   Using `-' as the first character of the list of option characters
+   selects this mode of operation.
+
+   The special argument `--' forces an end of option-scanning regardless
+   of the value of `ordering'.  In the case of RETURN_IN_ORDER, only
+   `--' can cause `getopt' to return EOF with `optind' != ARGC.  */
+
+static enum
+  {
+    REQUIRE_ORDER, PERMUTE, RETURN_IN_ORDER
+  }
+ordering;
+
+
+#ifdef	__GNU_LIBRARY__
+/* We want to avoid inclusion of string.h with non-GNU libraries
+   because there are many ways it can cause trouble.
+   On some systems, it contains special magic macros that don't work
+   in GCC.  */
+#include <string.h>
+#define	my_index	strchr
+#else
+
+/* Avoid depending on library functions or files
+   whose names are inconsistent.  */
+
+char *getenv ();
+
+static char *
+my_index (str, chr)
+     const char *str;
+     int chr;
+{
+  while (*str)
+    {
+      if (*str == chr)
+	return (char *) str;
+      str++;
+    }
+  return 0;
+}
+
+/* If using GCC, we can safely declare strlen this way.
+   If not using GCC, it is ok not to declare it.  */
+#ifdef __GNUC__
+#ifdef IN_GCC
+#include "gstddef.h"
+#else
+#include <stddef.h>
+#endif
+extern size_t strlen (const char *);
+#endif
+
+#endif /* GNU C library.  */
+
+
+/* Handle permutation of arguments.  */
+
+/* Describe the part of ARGV that contains non-options that have
+   been skipped.  `first_nonopt' is the index in ARGV of the first of them;
+   `last_nonopt' is the index after the last of them.  */
+
+static int first_nonopt;
+static int last_nonopt;
+
+/* Exchange two adjacent subsequences of ARGV.
+   One subsequence is elements [first_nonopt,last_nonopt)
+   which contains all the non-options that have been skipped so far.
+   The other is elements [last_nonopt,optind), which contains all
+   the options processed since those non-options were skipped.
+
+   `first_nonopt' and `last_nonopt' are relocated so that they describe
+   the new indices of the non-options in ARGV after they are moved.  */
+
+static void
+exchange (argv)
+     char **argv;
+{
+  int bottom = first_nonopt;
+  int middle = last_nonopt;
+  int top = optind;
+  char *tem;
+
+  /* Exchange the shorter segment with the far end of the longer segment.
+     That puts the shorter segment into the right place.
+     It leaves the longer segment in the right place overall,
+     but it consists of two parts that need to be swapped next.  */
+
+  while (top > middle && middle > bottom)
+    {
+      if (top - middle > middle - bottom)
+	{
+	  /* Bottom segment is the short one.  */
+	  int len = middle - bottom;
+	  register int i;
+
+	  /* Swap it with the top part of the top segment.  */
+	  for (i = 0; i < len; i++)
+	    {
+	      tem = argv[bottom + i];
+	      argv[bottom + i] = argv[top - (middle - bottom) + i];
+	      argv[top - (middle - bottom) + i] = tem;
+	    }
+	  /* Exclude the moved bottom segment from further swapping.  */
+	  top -= len;
+	}
+      else
+	{
+	  /* Top segment is the short one.  */
+	  int len = top - middle;
+	  register int i;
+
+	  /* Swap it with the bottom part of the bottom segment.  */
+	  for (i = 0; i < len; i++)
+	    {
+	      tem = argv[bottom + i];
+	      argv[bottom + i] = argv[middle + i];
+	      argv[middle + i] = tem;
+	    }
+	  /* Exclude the moved top segment from further swapping.  */
+	  bottom += len;
+	}
+    }
+
+  /* Update records for the slots the non-options now occupy.  */
+
+  first_nonopt += (optind - last_nonopt);
+  last_nonopt = optind;
+}
+
+
+/* Scan elements of ARGV (whose length is ARGC) for option characters
+   given in OPTSTRING.
+
+   If an element of ARGV starts with '-', and is not exactly "-" or "--",
+   then it is an option element.  The characters of this element
+   (aside from the initial '-') are option characters.  If `getopt'
+   is called repeatedly, it returns successively each of the option characters
+   from each of the option elements.
+
+   If `getopt' finds another option character, it returns that character,
+   updating `optind' and `nextchar' so that the next call to `getopt' can
+   resume the scan with the following option character or ARGV-element.
+
+   If there are no more option characters, `getopt' returns `EOF'.
+   Then `optind' is the index in ARGV of the first ARGV-element
+   that is not an option.  (The ARGV-elements have been permuted
+   so that those that are not options now come last.)
+
+   OPTSTRING is a string containing the legitimate option characters.
+   If an option character is seen that is not listed in OPTSTRING,
+   return '?' after printing an error message.  If you set `opterr' to
+   zero, the error message is suppressed but we still return '?'.
+
+   If a char in OPTSTRING is followed by a colon, that means it wants an arg,
+   so the following text in the same ARGV-element, or the text of the following
+   ARGV-element, is returned in `optarg'.  Two colons mean an option that
+   wants an optional arg; if there is text in the current ARGV-element,
+   it is returned in `optarg', otherwise `optarg' is set to zero.
+
+   If OPTSTRING starts with `-' or `+', it requests different methods of
+   handling the non-option ARGV-elements.
+   See the comments about RETURN_IN_ORDER and REQUIRE_ORDER, above.
+
+   Long-named options begin with `--' instead of `-'.
+   Their names may be abbreviated as long as the abbreviation is unique
+   or is an exact match for some defined option.  If they have an
+   argument, it follows the option name in the same ARGV-element, separated
+   from the option name by a `=', or else the in next ARGV-element.
+   When `getopt' finds a long-named option, it returns 0 if that option's
+   `flag' field is nonzero, the value of the option's `val' field
+   if the `flag' field is zero.
+
+   The elements of ARGV aren't really const, because we permute them.
+   But we pretend they're const in the prototype to be compatible
+   with other systems.
+
+   LONGOPTS is a vector of `struct option' terminated by an
+   element containing a name which is zero.
+
+   LONGIND returns the index in LONGOPT of the long-named option found.
+   It is only valid when a long-named option has been found by the most
+   recent call.
+
+   If LONG_ONLY is nonzero, '-' as well as '--' can introduce
+   long-named options.  */
+
+int
+_getopt_internal (argc, argv, optstring, longopts, longind, long_only)
+     int argc;
+     char *const *argv;
+     const char *optstring;
+     const struct option *longopts;
+     int *longind;
+     int long_only;
+{
+  int option_index;
+
+  optarg = 0;
+
+  /* Initialize the internal data when the first call is made.
+     Start processing options with ARGV-element 1 (since ARGV-element 0
+     is the program name); the sequence of previously skipped
+     non-option ARGV-elements is empty.  */
+
+  if (optind == 0)
+    {
+      first_nonopt = last_nonopt = optind = 1;
+
+      nextchar = NULL;
+
+      /* Determine how to handle the ordering of options and nonoptions.  */
+
+      if (optstring[0] == '-')
+	{
+	  ordering = RETURN_IN_ORDER;
+	  ++optstring;
+	}
+      else if (optstring[0] == '+')
+	{
+	  ordering = REQUIRE_ORDER;
+	  ++optstring;
+	}
+      else if (getenv ("POSIXLY_CORRECT") != NULL)
+	ordering = REQUIRE_ORDER;
+      else
+	ordering = PERMUTE;
+    }
+
+  if (nextchar == NULL || *nextchar == '\0')
+    {
+      if (ordering == PERMUTE)
+	{
+	  /* If we have just processed some options following some non-options,
+	     exchange them so that the options come first.  */
+
+	  if (first_nonopt != last_nonopt && last_nonopt != optind)
+	    exchange ((char **) argv);
+	  else if (last_nonopt != optind)
+	    first_nonopt = optind;
+
+	  /* Now skip any additional non-options
+	     and extend the range of non-options previously skipped.  */
+
+	  while (optind < argc
+		 && (argv[optind][0] != '-' || argv[optind][1] == '\0')
+#ifdef GETOPT_COMPAT
+		 && (longopts == NULL
+		     || argv[optind][0] != '+' || argv[optind][1] == '\0')
+#endif /* GETOPT_COMPAT */
+	    )
+	    optind++;
+	  last_nonopt = optind;
+	}
+
+      /* Special ARGV-element `--' means premature end of options.
+         Skip it like a null option,
+         then exchange with previous non-options as if it were an option,
+         then skip everything else like a non-option.  */
+
+      if (optind != argc && !strcmp (argv[optind], "--"))
+	{
+	  optind++;
+
+	  if (first_nonopt != last_nonopt && last_nonopt != optind)
+	    exchange ((char **) argv);
+	  else if (first_nonopt == last_nonopt)
+	    first_nonopt = optind;
+	  last_nonopt = argc;
+
+	  optind = argc;
+	}
+
+      /* If we have done all the ARGV-elements, stop the scan
+         and back over any non-options that we skipped and permuted.  */
+
+      if (optind == argc)
+	{
+	  /* Set the next-arg-index to point at the non-options
+	     that we previously skipped, so the caller will digest them.  */
+	  if (first_nonopt != last_nonopt)
+	    optind = first_nonopt;
+	  return EOF;
+	}
+
+      /* If we have come to a non-option and did not permute it,
+         either stop the scan or describe it to the caller and pass it by.  */
+
+      if ((argv[optind][0] != '-' || argv[optind][1] == '\0')
+#ifdef GETOPT_COMPAT
+	  && (longopts == NULL
+	      || argv[optind][0] != '+' || argv[optind][1] == '\0')
+#endif /* GETOPT_COMPAT */
+	)
+	{
+	  if (ordering == REQUIRE_ORDER)
+	    return EOF;
+	  optarg = argv[optind++];
+	  return 1;
+	}
+
+      /* We have found another option-ARGV-element.
+         Start decoding its characters.  */
+
+      nextchar = (argv[optind] + 1
+		  + (longopts != NULL && argv[optind][1] == '-'));
+    }
+
+  if (longopts != NULL
+      && ((argv[optind][0] == '-'
+	   && (argv[optind][1] == '-' || long_only))
+#ifdef GETOPT_COMPAT
+	  || argv[optind][0] == '+'
+#endif /* GETOPT_COMPAT */
+      ))
+    {
+      const struct option *p;
+      char *s = nextchar;
+      int exact = 0;
+      int ambig = 0;
+      const struct option *pfound = NULL;
+      int indfound;
+
+      while (*s && *s != '=')
+	s++;
+
+      /* Test all options for either exact match or abbreviated matches.  */
+      for (p = longopts, option_index = 0; p->name;
+	   p++, option_index++)
+	if (!strncmp (p->name, nextchar, s - nextchar))
+	  {
+	    if (s - nextchar == strlen (p->name))
+	      {
+		/* Exact match found.  */
+		pfound = p;
+		indfound = option_index;
+		exact = 1;
+		break;
+	      }
+	    else if (pfound == NULL)
+	      {
+		/* First nonexact match found.  */
+		pfound = p;
+		indfound = option_index;
+	      }
+	    else
+	      /* Second nonexact match found.  */
+	      ambig = 1;
+	  }
+
+      if (ambig && !exact)
+	{
+	  if (opterr)
+	    fprintf (stderr, "%s: option `%s' is ambiguous\n",
+		     argv[0], argv[optind]);
+	  nextchar += strlen (nextchar);
+	  optind++;
+	  return '?';
+	}
+
+      if (pfound != NULL)
+	{
+	  option_index = indfound;
+	  optind++;
+	  if (*s)
+	    {
+	      /* Don't test has_arg with >, because some C compilers don't
+	         allow it to be used on enums.  */
+	      if (pfound->has_arg)
+		optarg = s + 1;
+	      else
+		{
+		  if (opterr)
+		    {
+		      if (argv[optind - 1][1] == '-')
+			/* --option */
+			fprintf (stderr,
+			    "%s: option `--%s' doesn't allow an argument\n",
+				 argv[0], pfound->name);
+		      else
+			/* +option or -option */
+			fprintf (stderr,
+			    "%s: option `%c%s' doesn't allow an argument\n",
+				 argv[0], argv[optind - 1][0], pfound->name);
+		    }
+		  nextchar += strlen (nextchar);
+		  return '?';
+		}
+	    }
+	  else if (pfound->has_arg == 1)
+	    {
+	      if (optind < argc)
+		optarg = argv[optind++];
+	      else
+		{
+		  if (opterr)
+		    fprintf (stderr, "%s: option `%s' requires an argument\n",
+			     argv[0], argv[optind - 1]);
+		  nextchar += strlen (nextchar);
+		  return optstring[0] == ':' ? ':' : '?';
+		}
+	    }
+	  nextchar += strlen (nextchar);
+	  if (longind != NULL)
+	    *longind = option_index;
+	  if (pfound->flag)
+	    {
+	      *(pfound->flag) = pfound->val;
+	      return 0;
+	    }
+	  return pfound->val;
+	}
+      /* Can't find it as a long option.  If this is not getopt_long_only,
+         or the option starts with '--' or is not a valid short
+         option, then it's an error.
+         Otherwise interpret it as a short option.  */
+      if (!long_only || argv[optind][1] == '-'
+#ifdef GETOPT_COMPAT
+	  || argv[optind][0] == '+'
+#endif /* GETOPT_COMPAT */
+	  || my_index (optstring, *nextchar) == NULL)
+	{
+	  if (opterr)
+	    {
+	      if (argv[optind][1] == '-')
+		/* --option */
+		fprintf (stderr, "%s: unrecognized option `--%s'\n",
+			 argv[0], nextchar);
+	      else
+		/* +option or -option */
+		fprintf (stderr, "%s: unrecognized option `%c%s'\n",
+			 argv[0], argv[optind][0], nextchar);
+	    }
+	  nextchar = (char *) "";
+	  optind++;
+	  return '?';
+	}
+    }
+
+  /* Look at and handle the next option-character.  */
+
+  {
+    char c = *nextchar++;
+    char *temp = my_index (optstring, c);
+
+    /* Increment `optind' when we start to process its last character.  */
+    if (*nextchar == '\0')
+      ++optind;
+
+    if (temp == NULL || c == ':')
+      {
+	if (opterr)
+	  {
+#if 0
+	    if (c < 040 || c >= 0177)
+	      fprintf (stderr, "%s: unrecognized option, character code 0%o\n",
+		       argv[0], c);
+	    else
+	      fprintf (stderr, "%s: unrecognized option `-%c'\n", argv[0], c);
+#else
+	    /* 1003.2 specifies the format of this message.  */
+	    fprintf (stderr, "%s: illegal option -- %c\n", argv[0], c);
+#endif
+	  }
+	optopt = c;
+	return '?';
+      }
+    if (temp[1] == ':')
+      {
+	if (temp[2] == ':')
+	  {
+	    /* This is an option that accepts an argument optionally.  */
+	    if (*nextchar != '\0')
+	      {
+		optarg = nextchar;
+		optind++;
+	      }
+	    else
+	      optarg = 0;
+	    nextchar = NULL;
+	  }
+	else
+	  {
+	    /* This is an option that requires an argument.  */
+	    if (*nextchar != '\0')
+	      {
+		optarg = nextchar;
+		/* If we end this ARGV-element by taking the rest as an arg,
+		   we must advance to the next element now.  */
+		optind++;
+	      }
+	    else if (optind == argc)
+	      {
+		if (opterr)
+		  {
+#if 0
+		    fprintf (stderr, "%s: option `-%c' requires an argument\n",
+			     argv[0], c);
+#else
+		    /* 1003.2 specifies the format of this message.  */
+		    fprintf (stderr, "%s: option requires an argument -- %c\n",
+			     argv[0], c);
+#endif
+		  }
+		optopt = c;
+		if (optstring[0] == ':')
+		  c = ':';
+		else
+		  c = '?';
+	      }
+	    else
+	      /* We already incremented `optind' once;
+	         increment it again when taking next ARGV-elt as argument.  */
+	      optarg = argv[optind++];
+	    nextchar = NULL;
+	  }
+      }
+    return c;
+  }
+}
+
+int
+getopt (argc, argv, optstring)
+     int argc;
+     char *const *argv;
+     const char *optstring;
+{
+  return _getopt_internal (argc, argv, optstring,
+			   (const struct option *) 0,
+			   (int *) 0,
+			   0);
+}
+
+#endif /* _LIBC or not __GNU_LIBRARY__.  */
+
+
+#ifdef TEST
+
+/* Compile with -DTEST to make an executable for use in testing
+   the above definition of `getopt'.  */
+
+int
+main (argc, argv)
+     int argc;
+     char **argv;
+{
+  int c;
+  int digit_optind = 0;
+
+  while (1)
+    {
+      int this_option_optind = optind ? optind : 1;
+
+      c = getopt (argc, argv, "abc:d:0123456789");
+      if (c == EOF)
+	break;
+
+      switch (c)
+	{
+	case '0':
+	case '1':
+	case '2':
+	case '3':
+	case '4':
+	case '5':
+	case '6':
+	case '7':
+	case '8':
+	case '9':
+	  if (digit_optind != 0 && digit_optind != this_option_optind)
+	    printf ("digits occur in two different argv-elements.\n");
+	  digit_optind = this_option_optind;
+	  printf ("option %c\n", c);
+	  break;
+
+	case 'a':
+	  printf ("option a\n");
+	  break;
+
+	case 'b':
+	  printf ("option b\n");
+	  break;
+
+	case 'c':
+	  printf ("option c with value `%s'\n", optarg);
+	  break;
+
+	case '?':
+	  break;
+
+	default:
+	  printf ("?? getopt returned character code 0%o ??\n", c);
+	}
+    }
+
+  if (optind < argc)
+    {
+      printf ("non-option ARGV-elements: ");
+      while (optind < argc)
+	printf ("%s ", argv[optind++]);
+      printf ("\n");
+    }
+
+  exit (0);
+}
+
+#endif /* TEST */
Index: /trunk/indexers/mgpp/lib/getopt_old.h
===================================================================
--- /trunk/indexers/mgpp/lib/getopt_old.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/getopt_old.h	(revision 3365)
@@ -0,0 +1,130 @@
+/* Declarations for getopt.
+   Copyright (C) 1989, 1990, 1991, 1992, 1993 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by the
+   Free Software Foundation; either version 2, or (at your option) any
+   later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifndef _GETOPT_H
+#define _GETOPT_H 1
+
+#ifdef	__cplusplus
+extern "C"
+{
+#endif
+
+/* For communication from `getopt' to the caller.
+   When `getopt' finds an option that takes an argument,
+   the argument value is returned here.
+   Also, when `ordering' is RETURN_IN_ORDER,
+   each non-option ARGV-element is returned here.  */
+
+  extern char *optarg;
+
+/* Index in ARGV of the next element to be scanned.
+   This is used for communication to and from the caller
+   and for communication between successive calls to `getopt'.
+
+   On entry to `getopt', zero means this is the first call; initialize.
+
+   When `getopt' returns EOF, this is the index of the first of the
+   non-option elements that the caller should itself scan.
+
+   Otherwise, `optind' communicates from one call to the next
+   how much of ARGV has been scanned so far.  */
+
+  extern int optind;
+
+/* Callers store zero here to inhibit the error message `getopt' prints
+   for unrecognized options.  */
+
+  extern int opterr;
+
+/* Set to an option character which was unrecognized.  */
+
+  extern int optopt;
+
+/* Describe the long-named options requested by the application.
+   The LONG_OPTIONS argument to getopt_long or getopt_long_only is a vector
+   of `struct option' terminated by an element containing a name which is
+   zero.
+
+   The field `has_arg' is:
+   no_argument          (or 0) if the option does not take an argument,
+   required_argument    (or 1) if the option requires an argument,
+   optional_argument    (or 2) if the option takes an optional argument.
+
+   If the field `flag' is not NULL, it points to a variable that is set
+   to the value given in the field `val' when the option is found, but
+   left unchanged if the option is not found.
+
+   To have a long-named option do something other than set an `int' to
+   a compiled-in constant, such as set a value from `optarg', set the
+   option's `flag' field to zero and its `val' field to a nonzero
+   value (the equivalent single-letter option character, if there is
+   one).  For long options that have a zero `flag' field, `getopt'
+   returns the contents of the `val' field.  */
+
+  struct option
+    {
+#if	__STDC__
+      const char *name;
+#else
+      char *name;
+#endif
+      /* has_arg can't be an enum because some compilers complain about
+         type mismatches in all the code that assumes it is an int.  */
+      int has_arg;
+      int *flag;
+      int val;
+    };
+
+/* Names for the values of the `has_arg' field of `struct option'.  */
+
+#define	no_argument		0
+#define required_argument	1
+#define optional_argument	2
+
+#if __STDC__
+#if defined(__GNU_LIBRARY__)
+/* Many other libraries have conflicting prototypes for getopt, with
+   differences in the consts, in stdlib.h.  To avoid compilation
+   errors, only prototype getopt for the GNU C library.  */
+  extern int getopt (int argc, char *const *argv, const char *shortopts);
+#else				/* not __GNU_LIBRARY__ */
+  extern int getopt ();
+#endif				/* not __GNU_LIBRARY__ */
+  extern int getopt_long (int argc, char *const *argv, const char *shortopts,
+			  const struct option *longopts, int *longind);
+  extern int getopt_long_only (int argc, char *const *argv,
+			       const char *shortopts,
+			       const struct option *longopts, int *longind);
+
+/* Internal only.  Users should not call this directly.  */
+  extern int _getopt_internal (int argc, char *const *argv,
+			       const char *shortopts,
+			       const struct option *longopts, int *longind,
+			       int long_only);
+#else				/* not __STDC__ */
+  extern int getopt ();
+  extern int getopt_long ();
+  extern int getopt_long_only ();
+
+  extern int _getopt_internal ();
+#endif				/* not __STDC__ */
+
+#ifdef	__cplusplus
+}
+#endif
+
+#endif				/* _GETOPT_H */
Index: /trunk/indexers/mgpp/lib/getpagesize.h
===================================================================
--- /trunk/indexers/mgpp/lib/getpagesize.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/getpagesize.h	(revision 3365)
@@ -0,0 +1,38 @@
+/* Emulate getpagesize on systems that lack it.  */
+
+#ifndef HAVE_GETPAGESIZE
+
+#ifdef VMS
+#define getpagesize() 512
+#endif
+
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+#ifdef _SC_PAGESIZE
+#define getpagesize() sysconf(_SC_PAGESIZE)
+#else
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifndef __WIN32__
+#include <sys/param.h>
+#endif
+
+#ifdef EXEC_PAGESIZE
+#define getpagesize() EXEC_PAGESIZE
+#else
+#ifdef NBPG
+#define getpagesize() NBPG * CLSIZE
+#ifndef CLSIZE
+#define CLSIZE 1
+#endif /* no CLSIZE */
+#else /* no NBPG */
+#ifdef NBPC
+#define getpagesize() NBPC
+#endif /* NBPC */
+#endif /* no NBPG */
+#endif /* no EXEC_PAGESIZE */
+#endif /* no _SC_PAGESIZE */
+
+#endif /* not HAVE_GETPAGESIZE */
Index: /trunk/indexers/mgpp/lib/gmalloc.c
===================================================================
--- /trunk/indexers/mgpp/lib/gmalloc.c	(revision 3365)
+++ /trunk/indexers/mgpp/lib/gmalloc.c	(revision 3365)
@@ -0,0 +1,1318 @@
+/* DO NOT EDIT THIS FILE -- it is automagically generated.  -*- C -*- */
+
+#define _MALLOC_INTERNAL
+
+/* The malloc headers and source files from the C library follow here.  */
+
+/* Declarations for `malloc' and friends.
+   Copyright 1990, 1991, 1992, 1993 Free Software Foundation, Inc.
+		  Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef _MALLOC_H
+
+#define _MALLOC_H	1
+
+#ifdef _MALLOC_INTERNAL
+
+#ifdef	HAVE_CONFIG_H
+#include <sysfuncs.h>
+#endif
+
+#if	defined(_LIBC) || defined(STDC_HEADERS) || defined(USG)
+#include <string.h>
+#else
+#ifndef memset
+#define	memset(s, zero, n)	bzero ((s), (n))
+#endif
+#ifndef memcpy
+#define	memcpy(d, s, n)		bcopy ((s), (d), (n))
+#endif
+#endif
+
+#if	defined(__GNU_LIBRARY__) || __STDC__
+#include <limits.h>
+#else
+#define	CHAR_BIT	8
+#endif
+
+#ifdef	HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+#endif	/* _MALLOC_INTERNAL.  */
+
+
+#ifdef	__cplusplus
+extern "C"
+{
+#endif
+
+#if defined (__cplusplus) || __STDC__
+#undef	__P
+#define	__P(args)	args
+#undef	__ptr_t
+#define	__ptr_t		void *
+#else /* Not C++ or ANSI C.  */
+#undef	__P
+#define	__P(args)	()
+#undef	const
+#define	const
+#undef	__ptr_t
+#define	__ptr_t		char *
+#endif /* C++ or ANSI C.  */
+
+#if	__STDC__
+#include <stddef.h>
+#else
+#undef	size_t
+#define	size_t		unsigned int
+#undef	ptrdiff_t
+#define	ptrdiff_t	int
+#endif
+
+#ifndef	NULL
+#define	NULL	0
+#endif
+
+
+/* Allocate SIZE bytes of memory.  */
+extern __ptr_t malloc __P ((size_t __size));
+/* Re-allocate the previously allocated block
+   in __ptr_t, making the new block SIZE bytes long.  */
+extern __ptr_t realloc __P ((__ptr_t __ptr, size_t __size));
+/* Allocate NMEMB elements of SIZE bytes each, all initialized to 0.  */
+extern __ptr_t calloc __P ((size_t __nmemb, size_t __size));
+/* Free a block allocated by `malloc', `realloc' or `calloc'.  */
+extern void free __P ((__ptr_t __ptr));
+
+/* Allocate SIZE bytes allocated to ALIGNMENT bytes.  */
+extern __ptr_t memalign __P ((size_t __alignment, size_t __size));
+
+/* Allocate SIZE bytes on a page boundary.  */
+extern __ptr_t valloc __P ((size_t __size));
+
+
+#ifdef _MALLOC_INTERNAL
+
+/* The allocator divides the heap into blocks of fixed size; large
+   requests receive one or more whole blocks, and small requests
+   receive a fragment of a block.  Fragment sizes are powers of two,
+   and all fragments of a block are the same size.  When all the
+   fragments in a block have been freed, the block itself is freed.  */
+#define INT_BIT		(CHAR_BIT * sizeof(int))
+#define BLOCKLOG	(INT_BIT > 16 ? 12 : 9)
+#define BLOCKSIZE	(1 << BLOCKLOG)
+#define BLOCKIFY(SIZE)	(((SIZE) + BLOCKSIZE - 1) / BLOCKSIZE)
+
+/* Determine the amount of memory spanned by the initial heap table
+   (not an absolute limit).  */
+#define HEAP		(INT_BIT > 16 ? 4194304 : 65536)
+
+/* Number of contiguous free blocks allowed to build up at the end of
+   memory before they will be returned to the system.  */
+#define FINAL_FREE_BLOCKS	8
+
+/* Data structure giving per-block information.  */
+typedef union
+  {
+    /* Heap information for a busy block.  */
+    struct
+      {
+	/* Zero for a large block, or positive giving the
+	   logarithm to the base two of the fragment size.  */
+	int type;
+	union
+	  {
+	    struct
+	      {
+		size_t nfree;	/* Free fragments in a fragmented block.  */
+		size_t first;	/* First free fragment of the block.  */
+	      } frag;
+	    /* Size (in blocks) of a large cluster.  */
+	    size_t size;
+	  } info;
+      } busy;
+    /* Heap information for a free block
+       (that may be the first of a free cluster).  */
+    struct
+      {
+	size_t size;		/* Size (in blocks) of a free cluster.  */
+	size_t next;		/* Index of next free cluster.  */
+	size_t prev;		/* Index of previous free cluster.  */
+      } free;
+  } malloc_info;
+
+/* Pointer to first block of the heap.  */
+extern char *_heapbase;
+
+/* Table indexed by block number giving per-block information.  */
+extern malloc_info *_heapinfo;
+
+/* Address to block number and vice versa.  */
+#define BLOCK(A)	(((char *) (A) - _heapbase) / BLOCKSIZE + 1)
+#define ADDRESS(B)	((__ptr_t) (((B) - 1) * BLOCKSIZE + _heapbase))
+
+/* Current search index for the heap table.  */
+extern size_t _heapindex;
+
+/* Limit of valid info table indices.  */
+extern size_t _heaplimit;
+
+/* Doubly linked lists of free fragments.  */
+struct list
+  {
+    struct list *next;
+    struct list *prev;
+  };
+
+/* Free list headers for each fragment size.  */
+extern struct list _fraghead[];
+
+/* List of blocks allocated with `memalign' (or `valloc').  */
+struct alignlist
+  {
+    struct alignlist *next;
+    __ptr_t aligned;		/* The address that memaligned returned.  */
+    __ptr_t exact;		/* The address that malloc returned.  */
+  };
+extern struct alignlist *_aligned_blocks;
+
+/* Instrumentation.  */
+extern size_t _chunks_used;
+extern size_t _bytes_used;
+extern size_t _chunks_free;
+extern size_t _bytes_free;
+
+/* Internal version of `free' used in `morecore' (malloc.c). */
+extern void _free_internal __P ((__ptr_t __ptr));
+
+#endif /* _MALLOC_INTERNAL.  */
+
+/* Underlying allocation function; successive calls should
+   return contiguous pieces of memory.  */
+extern __ptr_t (*__morecore) __P ((ptrdiff_t __size));
+
+/* Default value of `__morecore'.  */
+extern __ptr_t __default_morecore __P ((ptrdiff_t __size));
+
+/* If not NULL, this function is called after each time
+   `__morecore' is called to increase the data size.  */
+extern void (*__after_morecore_hook) __P ((void));
+
+/* Nonzero if `malloc' has been called and done its initialization.  */
+extern int __malloc_initialized;
+
+/* Hooks for debugging versions.  */
+extern void (*__free_hook) __P ((__ptr_t __ptr));
+extern __ptr_t (*__malloc_hook) __P ((size_t __size));
+extern __ptr_t (*__realloc_hook) __P ((__ptr_t __ptr, size_t __size));
+
+/* Return values for `mprobe': these are the kinds of inconsistencies that
+   `mcheck' enables detection of.  */
+enum mcheck_status
+  {
+    MCHECK_DISABLED = -1,	/* Consistency checking is not turned on.  */
+    MCHECK_OK,			/* Block is fine.  */
+    MCHECK_FREE,		/* Block freed twice.  */
+    MCHECK_HEAD,		/* Memory before the block was clobbered.  */
+    MCHECK_TAIL			/* Memory after the block was clobbered.  */
+  };
+
+/* Activate a standard collection of debugging hooks.  This must be called
+   before `malloc' is ever called.  ABORTFUNC is called with an error code
+   (see enum above) when an inconsistency is detected.  If ABORTFUNC is
+   null, the standard function prints on stderr and then calls `abort'.  */
+extern int mcheck __P ((void (*__abortfunc) __P ((enum mcheck_status))));
+
+/* Check for aberrations in a particular malloc'd block.  You must have
+   called `mcheck' already.  These are the same checks that `mcheck' does
+   when you free or reallocate a block.  */
+extern enum mcheck_status mprobe __P ((__ptr_t __ptr));
+
+/* Activate a standard collection of tracing hooks.  */
+extern void mtrace __P ((void));
+
+/* Statistics available to the user.  */
+struct mstats
+  {
+    size_t bytes_total;		/* Total size of the heap. */
+    size_t chunks_used;		/* Chunks allocated by the user. */
+    size_t bytes_used;		/* Byte total of user-allocated chunks. */
+    size_t chunks_free;		/* Chunks in the free list. */
+    size_t bytes_free;		/* Byte total of chunks in the free list. */
+  };
+
+/* Pick up the current statistics. */
+extern struct mstats mstats __P ((void));
+
+/* Call WARNFUN with a warning message when memory usage is high.  */
+extern void memory_warnings __P ((__ptr_t __start,
+				  void (*__warnfun) __P ((const char *))));
+
+
+/* Relocating allocator.  */
+
+/* Allocate SIZE bytes, and store the address in *HANDLEPTR.  */
+extern __ptr_t r_alloc __P ((__ptr_t *__handleptr, size_t __size));
+
+/* Free the storage allocated in HANDLEPTR.  */
+extern void r_alloc_free __P ((__ptr_t *__handleptr));
+
+/* Adjust the block at HANDLEPTR to be SIZE bytes long.  */
+extern __ptr_t r_re_alloc __P ((__ptr_t *__handleptr, size_t __size));
+
+
+#ifdef	__cplusplus
+}
+#endif
+
+#endif /* malloc.h  */
+/* Allocate memory on a page boundary.
+   Copyright (C) 1991, 1992, 1993 Free Software Foundation, Inc.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#if defined (__GNU_LIBRARY__) || defined (_LIBC)
+#include <stddef.h>
+#include <sys/cdefs.h>
+extern size_t __getpagesize __P ((void));
+#else
+#include "getpagesize.h"
+#define	 __getpagesize()	getpagesize()
+#endif
+
+#ifndef	_MALLOC_INTERNAL
+#define	_MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+static size_t pagesize;
+
+__ptr_t
+valloc (size)
+     size_t size;
+{
+  if (pagesize == 0)
+    pagesize = __getpagesize ();
+
+  return memalign (pagesize, size);
+}
+/* Memory allocator `malloc'.
+   Copyright 1990, 1991, 1992, 1993, 1994 Free Software Foundation
+		  Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+/* How to really get more memory.  */
+__ptr_t (*__morecore) __P ((ptrdiff_t __size)) = __default_morecore;
+
+/* Debugging hook for `malloc'.  */
+__ptr_t (*__malloc_hook) __P ((size_t __size));
+
+/* Pointer to the base of the first block.  */
+char *_heapbase;
+
+/* Block information table.  Allocated with align/__free (not malloc/free).  */
+malloc_info *_heapinfo;
+
+/* Number of info entries.  */
+static size_t heapsize;
+
+/* Search index in the info table.  */
+size_t _heapindex;
+
+/* Limit of valid info table indices.  */
+size_t _heaplimit;
+
+/* Free lists for each fragment size.  */
+struct list _fraghead[BLOCKLOG];
+
+/* Instrumentation.  */
+size_t _chunks_used;
+size_t _bytes_used;
+size_t _chunks_free;
+size_t _bytes_free;
+
+/* Are you experienced?  */
+int __malloc_initialized;
+
+void (*__after_morecore_hook) __P ((void));
+
+/* Aligned allocation.  */
+static __ptr_t align __P ((size_t));
+static __ptr_t
+align (size)
+     size_t size;
+{
+  __ptr_t result;
+  unsigned long int adj;
+
+  result = (*__morecore) (size);
+  adj = (unsigned long int) ((unsigned long int) ((char *) result -
+						  (char *) NULL)) % BLOCKSIZE;
+  if (adj != 0)
+    {
+      adj = BLOCKSIZE - adj;
+      (void) (*__morecore) (adj);
+      result = (char *) result + adj;
+    }
+
+  if (__after_morecore_hook)
+    (*__after_morecore_hook) ();
+
+  return result;
+}
+
+/* Set everything up and remember that we have.  */
+static int initialize __P ((void));
+static int
+initialize ()
+{
+  heapsize = HEAP / BLOCKSIZE;
+  _heapinfo = (malloc_info *) align (heapsize * sizeof (malloc_info));
+  if (_heapinfo == NULL)
+    return 0;
+  memset (_heapinfo, 0, heapsize * sizeof (malloc_info));
+  _heapinfo[0].free.size = 0;
+  _heapinfo[0].free.next = _heapinfo[0].free.prev = 0;
+  _heapindex = 0;
+  _heapbase = (char *) _heapinfo;
+
+  /* Account for the _heapinfo block itself in the statistics.  */
+  _bytes_used = heapsize * sizeof (malloc_info);
+  _chunks_used = 1;
+
+  __malloc_initialized = 1;
+  return 1;
+}
+
+/* Get neatly aligned memory, initializing or
+   growing the heap info table as necessary. */
+static __ptr_t morecore __P ((size_t));
+static __ptr_t
+morecore (size)
+     size_t size;
+{
+  __ptr_t result;
+  malloc_info *newinfo, *oldinfo;
+  size_t newsize;
+
+  result = align (size);
+  if (result == NULL)
+    return NULL;
+
+  /* Check if we need to grow the info table.  */
+  if ((size_t) BLOCK ((char *) result + size) > heapsize)
+    {
+      newsize = heapsize;
+      while ((size_t) BLOCK ((char *) result + size) > newsize)
+	newsize *= 2;
+      newinfo = (malloc_info *) align (newsize * sizeof (malloc_info));
+      if (newinfo == NULL)
+	{
+	  (*__morecore) (-size);
+	  return NULL;
+	}
+      memcpy (newinfo, _heapinfo, heapsize * sizeof (malloc_info));
+      memset (&newinfo[heapsize], 0,
+	      (newsize - heapsize) * sizeof (malloc_info));
+      oldinfo = _heapinfo;
+      newinfo[BLOCK (oldinfo)].busy.type = 0;
+      newinfo[BLOCK (oldinfo)].busy.info.size
+	= BLOCKIFY (heapsize * sizeof (malloc_info));
+      _heapinfo = newinfo;
+      /* Account for the _heapinfo block itself in the statistics.  */
+      _bytes_used += newsize * sizeof (malloc_info);
+      ++_chunks_used;
+      _free_internal (oldinfo);
+      heapsize = newsize;
+    }
+
+  _heaplimit = BLOCK ((char *) result + size);
+  return result;
+}
+
+/* Allocate memory from the heap.  */
+__ptr_t
+malloc (size)
+     size_t size;
+{
+  __ptr_t result;
+  size_t block, blocks, lastblocks, start;
+  register size_t i;
+  struct list *next;
+
+  /* ANSI C allows `malloc (0)' to either return NULL, or to return a
+     valid address you can realloc and free (though not dereference).
+
+     It turns out that some extant code (sunrpc, at least Ultrix's version)
+     expects `malloc (0)' to return non-NULL and breaks otherwise.
+     Be compatible.  */
+
+#if	0
+  if (size == 0)
+    return NULL;
+#endif
+
+  if (__malloc_hook != NULL)
+    return (*__malloc_hook) (size);
+
+  if (!__malloc_initialized)
+    if (!initialize ())
+      return NULL;
+
+  if (size < sizeof (struct list))
+      size = sizeof (struct list);
+
+#ifdef SUNOS_LOCALTIME_BUG
+  if (size < 16)
+    size = 16;
+#endif
+
+  /* Determine the allocation policy based on the request size.  */
+  if (size <= BLOCKSIZE / 2)
+    {
+      /* Small allocation to receive a fragment of a block.
+	 Determine the logarithm to base two of the fragment size. */
+      register size_t log = 1;
+      --size;
+      while ((size /= 2) != 0)
+	++log;
+
+      /* Look in the fragment lists for a
+	 free fragment of the desired size. */
+      next = _fraghead[log].next;
+      if (next != NULL)
+	{
+	  /* There are free fragments of this size.
+	     Pop a fragment out of the fragment list and return it.
+	     Update the block's nfree and first counters. */
+	  result = (__ptr_t) next;
+	  next->prev->next = next->next;
+	  if (next->next != NULL)
+	    next->next->prev = next->prev;
+	  block = BLOCK (result);
+	  if (--_heapinfo[block].busy.info.frag.nfree != 0)
+	    _heapinfo[block].busy.info.frag.first = (unsigned long int)
+	      ((unsigned long int) ((char *) next->next - (char *) NULL)
+	       % BLOCKSIZE) >> log;
+
+	  /* Update the statistics.  */
+	  ++_chunks_used;
+	  _bytes_used += 1 << log;
+	  --_chunks_free;
+	  _bytes_free -= 1 << log;
+	}
+      else
+	{
+	  /* No free fragments of the desired size, so get a new block
+	     and break it into fragments, returning the first.  */
+	  result = malloc (BLOCKSIZE);
+	  if (result == NULL)
+	    return NULL;
+
+	  /* Link all fragments but the first into the free list.  */
+	  for (i = 1; i < (size_t) (BLOCKSIZE >> log); ++i)
+	    {
+	      next = (struct list *) ((char *) result + (i << log));
+	      next->next = _fraghead[log].next;
+	      next->prev = &_fraghead[log];
+	      next->prev->next = next;
+	      if (next->next != NULL)
+		next->next->prev = next;
+	    }
+
+	  /* Initialize the nfree and first counters for this block.  */
+	  block = BLOCK (result);
+	  _heapinfo[block].busy.type = log;
+	  _heapinfo[block].busy.info.frag.nfree = i - 1;
+	  _heapinfo[block].busy.info.frag.first = i - 1;
+
+	  _chunks_free += (BLOCKSIZE >> log) - 1;
+	  _bytes_free += BLOCKSIZE - (1 << log);
+	  _bytes_used -= BLOCKSIZE - (1 << log);
+	}
+    }
+  else
+    {
+      /* Large allocation to receive one or more blocks.
+	 Search the free list in a circle starting at the last place visited.
+	 If we loop completely around without finding a large enough
+	 space we will have to get more memory from the system.  */
+      blocks = BLOCKIFY (size);
+      start = block = _heapindex;
+      while (_heapinfo[block].free.size < blocks)
+	{
+	  block = _heapinfo[block].free.next;
+	  if (block == start)
+	    {
+	      /* Need to get more from the system.  Check to see if
+		 the new core will be contiguous with the final free
+		 block; if so we don't need to get as much.  */
+	      block = _heapinfo[0].free.prev;
+	      lastblocks = _heapinfo[block].free.size;
+	      if (_heaplimit != 0 && block + lastblocks == _heaplimit &&
+		  (*__morecore) (0) == ADDRESS (block + lastblocks) &&
+		  (morecore ((blocks - lastblocks) * BLOCKSIZE)) != NULL)
+		{
+ 		  /* Which block we are extending (the `final free
+ 		     block' referred to above) might have changed, if
+ 		     it got combined with a freed info table.  */
+ 		  block = _heapinfo[0].free.prev;
+  		  _heapinfo[block].free.size += (blocks - lastblocks);
+		  _bytes_free += (blocks - lastblocks) * BLOCKSIZE;
+		  continue;
+		}
+	      result = morecore (blocks * BLOCKSIZE);
+	      if (result == NULL)
+		return NULL;
+	      block = BLOCK (result);
+	      _heapinfo[block].busy.type = 0;
+	      _heapinfo[block].busy.info.size = blocks;
+	      ++_chunks_used;
+	      _bytes_used += blocks * BLOCKSIZE;
+	      return result;
+	    }
+	}
+
+      /* At this point we have found a suitable free list entry.
+	 Figure out how to remove what we need from the list. */
+      result = ADDRESS (block);
+      if (_heapinfo[block].free.size > blocks)
+	{
+	  /* The block we found has a bit left over,
+	     so relink the tail end back into the free list. */
+	  _heapinfo[block + blocks].free.size
+	    = _heapinfo[block].free.size - blocks;
+	  _heapinfo[block + blocks].free.next
+	    = _heapinfo[block].free.next;
+	  _heapinfo[block + blocks].free.prev
+	    = _heapinfo[block].free.prev;
+	  _heapinfo[_heapinfo[block].free.prev].free.next
+	    = _heapinfo[_heapinfo[block].free.next].free.prev
+	    = _heapindex = block + blocks;
+	}
+      else
+	{
+	  /* The block exactly matches our requirements,
+	     so just remove it from the list. */
+	  _heapinfo[_heapinfo[block].free.next].free.prev
+	    = _heapinfo[block].free.prev;
+	  _heapinfo[_heapinfo[block].free.prev].free.next
+	    = _heapindex = _heapinfo[block].free.next;
+	  --_chunks_free;
+	}
+
+      _heapinfo[block].busy.type = 0;
+      _heapinfo[block].busy.info.size = blocks;
+      ++_chunks_used;
+      _bytes_used += blocks * BLOCKSIZE;
+      _bytes_free -= blocks * BLOCKSIZE;
+    }
+
+  return result;
+}
+
+
+#ifndef _LIBC
+
+/* On some ANSI C systems, some libc functions call _malloc, _free
+   and _realloc.  Make them use the GNU functions.  */
+
+__ptr_t
+_malloc (size)
+     size_t size;
+{
+  return malloc (size);
+}
+
+void
+_free (ptr)
+     __ptr_t ptr;
+{
+  free (ptr);
+}
+
+__ptr_t
+_realloc (ptr, size)
+     __ptr_t ptr;
+     size_t size;
+{
+  return realloc (ptr, size);
+}
+
+#endif
+/* Free a block of memory allocated by `malloc'.
+   Copyright 1990, 1991, 1992 Free Software Foundation
+		  Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+/* Debugging hook for free.  */
+void (*__free_hook) __P ((__ptr_t __ptr));
+
+/* List of blocks allocated by memalign.  */
+struct alignlist *_aligned_blocks = NULL;
+
+/* Return memory to the heap.
+   Like `free' but don't call a __free_hook if there is one.  */
+void
+_free_internal (ptr)
+     __ptr_t ptr;
+{
+  int type;
+  size_t block, blocks;
+  register size_t i;
+  struct list *prev, *next;
+
+  block = BLOCK (ptr);
+
+  type = _heapinfo[block].busy.type;
+  switch (type)
+    {
+    case 0:
+      /* Get as many statistics as early as we can.  */
+      --_chunks_used;
+      _bytes_used -= _heapinfo[block].busy.info.size * BLOCKSIZE;
+      _bytes_free += _heapinfo[block].busy.info.size * BLOCKSIZE;
+
+      /* Find the free cluster previous to this one in the free list.
+	 Start searching at the last block referenced; this may benefit
+	 programs with locality of allocation.  */
+      i = _heapindex;
+      if (i > block)
+	while (i > block)
+	  i = _heapinfo[i].free.prev;
+      else
+	{
+	  do
+	    i = _heapinfo[i].free.next;
+	  while (i > 0 && i < block);
+	  i = _heapinfo[i].free.prev;
+	}
+
+      /* Determine how to link this block into the free list.  */
+      if (block == i + _heapinfo[i].free.size)
+	{
+	  /* Coalesce this block with its predecessor.  */
+	  _heapinfo[i].free.size += _heapinfo[block].busy.info.size;
+	  block = i;
+	}
+      else
+	{
+	  /* Really link this block back into the free list.  */
+	  _heapinfo[block].free.size = _heapinfo[block].busy.info.size;
+	  _heapinfo[block].free.next = _heapinfo[i].free.next;
+	  _heapinfo[block].free.prev = i;
+	  _heapinfo[i].free.next = block;
+	  _heapinfo[_heapinfo[block].free.next].free.prev = block;
+	  ++_chunks_free;
+	}
+
+      /* Now that the block is linked in, see if we can coalesce it
+	 with its successor (by deleting its successor from the list
+	 and adding in its size).  */
+      if (block + _heapinfo[block].free.size == _heapinfo[block].free.next)
+	{
+	  _heapinfo[block].free.size
+	    += _heapinfo[_heapinfo[block].free.next].free.size;
+	  _heapinfo[block].free.next
+	    = _heapinfo[_heapinfo[block].free.next].free.next;
+	  _heapinfo[_heapinfo[block].free.next].free.prev = block;
+	  --_chunks_free;
+	}
+
+      /* Now see if we can return stuff to the system.  */
+      blocks = _heapinfo[block].free.size;
+      if (blocks >= FINAL_FREE_BLOCKS && block + blocks == _heaplimit
+	  && (*__morecore) (0) == ADDRESS (block + blocks))
+	{
+	  register size_t bytes = blocks * BLOCKSIZE;
+	  _heaplimit -= blocks;
+	  (*__morecore) (-bytes);
+	  _heapinfo[_heapinfo[block].free.prev].free.next
+	    = _heapinfo[block].free.next;
+	  _heapinfo[_heapinfo[block].free.next].free.prev
+	    = _heapinfo[block].free.prev;
+	  block = _heapinfo[block].free.prev;
+	  --_chunks_free;
+	  _bytes_free -= bytes;
+	}
+
+      /* Set the next search to begin at this block.  */
+      _heapindex = block;
+      break;
+
+    default:
+      /* Do some of the statistics.  */
+      --_chunks_used;
+      _bytes_used -= 1 << type;
+      ++_chunks_free;
+      _bytes_free += 1 << type;
+
+      /* Get the address of the first free fragment in this block.  */
+      prev = (struct list *) ((char *) ADDRESS (block) +
+			   (_heapinfo[block].busy.info.frag.first << type));
+
+      if (_heapinfo[block].busy.info.frag.nfree == (BLOCKSIZE >> type) - 1)
+	{
+	  /* If all fragments of this block are free, remove them
+	     from the fragment list and free the whole block.  */
+	  next = prev;
+	  for (i = 1; i < (size_t) (BLOCKSIZE >> type); ++i)
+	    next = next->next;
+	  prev->prev->next = next;
+	  if (next != NULL)
+	    next->prev = prev->prev;
+	  _heapinfo[block].busy.type = 0;
+	  _heapinfo[block].busy.info.size = 1;
+
+	  /* Keep the statistics accurate.  */
+	  ++_chunks_used;
+	  _bytes_used += BLOCKSIZE;
+	  _chunks_free -= BLOCKSIZE >> type;
+	  _bytes_free -= BLOCKSIZE;
+
+	  free (ADDRESS (block));
+	}
+      else if (_heapinfo[block].busy.info.frag.nfree != 0)
+	{
+	  /* If some fragments of this block are free, link this
+	     fragment into the fragment list after the first free
+	     fragment of this block. */
+	  next = (struct list *) ptr;
+	  next->next = prev->next;
+	  next->prev = prev;
+	  prev->next = next;
+	  if (next->next != NULL)
+	    next->next->prev = next;
+	  ++_heapinfo[block].busy.info.frag.nfree;
+	}
+      else
+	{
+	  /* No fragments of this block are free, so link this
+	     fragment into the fragment list and announce that
+	     it is the first free fragment of this block. */
+	  prev = (struct list *) ptr;
+	  _heapinfo[block].busy.info.frag.nfree = 1;
+	  _heapinfo[block].busy.info.frag.first = (unsigned long int)
+	    ((unsigned long int) ((char *) ptr - (char *) NULL)
+	     % BLOCKSIZE >> type);
+	  prev->next = _fraghead[type].next;
+	  prev->prev = &_fraghead[type];
+	  prev->prev->next = prev;
+	  if (prev->next != NULL)
+	    prev->next->prev = prev;
+	}
+      break;
+    }
+}
+
+/* Return memory to the heap.  */
+void
+free (ptr)
+     __ptr_t ptr;
+{
+  register struct alignlist *l;
+
+  if (ptr == NULL)
+    return;
+
+  for (l = _aligned_blocks; l != NULL; l = l->next)
+    if (l->aligned == ptr)
+      {
+	l->aligned = NULL;	/* Mark the slot in the list as free.  */
+	ptr = l->exact;
+	break;
+      }
+
+  if (__free_hook != NULL)
+    (*__free_hook) (ptr);
+  else
+    _free_internal (ptr);
+}
+/* Copyright (C) 1991, 1993, 1994 Free Software Foundation, Inc.
+This file is part of the GNU C Library.
+
+The GNU C Library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+The GNU C Library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with the GNU C Library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+#ifdef _LIBC
+
+#include <ansidecl.h>
+#include <gnu-stabs.h>
+
+#undef	cfree
+
+function_alias(cfree, free, void, (ptr),
+	       DEFUN(cfree, (ptr), PTR ptr))
+
+#else
+
+void
+cfree (ptr)
+     __ptr_t ptr;
+{
+  free (ptr);
+}
+
+#endif
+/* Change the size of a block allocated by `malloc'.
+   Copyright 1990, 1991, 1992, 1993, 1994 Free Software Foundation, Inc.
+		     Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+#if  (defined (MEMMOVE_MISSING) || \
+      !defined(_LIBC) && !defined(STDC_HEADERS) && !defined(USG))
+
+/* Snarfed directly from Emacs src/dispnew.c:
+   XXX Should use system bcopy if it handles overlap.  */
+#ifndef emacs
+
+/* Like bcopy except never gets confused by overlap.  */
+
+static void
+safe_bcopy (from, to, size)
+     char *from, *to;
+     int size;
+{
+  if (size <= 0 || from == to)
+    return;
+
+  /* If the source and destination don't overlap, then bcopy can
+     handle it.  If they do overlap, but the destination is lower in
+     memory than the source, we'll assume bcopy can handle that.  */
+  if (to < from || from + size <= to)
+    bcopy (from, to, size);
+
+  /* Otherwise, we'll copy from the end.  */
+  else
+    {
+      register char *endf = from + size;
+      register char *endt = to + size;
+
+      /* If TO - FROM is large, then we should break the copy into
+	 nonoverlapping chunks of TO - FROM bytes each.  However, if
+	 TO - FROM is small, then the bcopy function call overhead
+	 makes this not worth it.  The crossover point could be about
+	 anywhere.  Since I don't think the obvious copy loop is too
+	 bad, I'm trying to err in its favor.  */
+      if (to - from < 64)
+	{
+	  do
+	    *--endt = *--endf;
+	  while (endf != from);
+	}
+      else
+	{
+	  for (;;)
+	    {
+	      endt -= (to - from);
+	      endf -= (to - from);
+
+	      if (endt < to)
+		break;
+
+	      bcopy (endf, endt, to - from);
+	    }
+
+	  /* If SIZE wasn't a multiple of TO - FROM, there will be a
+	     little left over.  The amount left over is
+	     (endt + (to - from)) - to, which is endt - from.  */
+	  bcopy (from, to, endt - from);
+	}
+    }
+}     
+#endif	/* Not emacs.  */
+
+#define memmove(to, from, size) safe_bcopy ((from), (to), (size))
+
+#endif
+
+
+#define min(A, B) ((A) < (B) ? (A) : (B))
+
+/* Debugging hook for realloc.  */
+__ptr_t (*__realloc_hook) __P ((__ptr_t __ptr, size_t __size));
+
+/* Resize the given region to the new size, returning a pointer
+   to the (possibly moved) region.  This is optimized for speed;
+   some benchmarks seem to indicate that greater compactness is
+   achieved by unconditionally allocating and copying to a
+   new region.  This module has incestuous knowledge of the
+   internals of both free and malloc. */
+__ptr_t
+realloc (ptr, size)
+     __ptr_t ptr;
+     size_t size;
+{
+  __ptr_t result;
+  int type;
+  size_t block, blocks, oldlimit;
+
+  if (size == 0)
+    {
+      free (ptr);
+      return malloc (0);
+    }
+  else if (ptr == NULL)
+    return malloc (size);
+
+  if (__realloc_hook != NULL)
+    return (*__realloc_hook) (ptr, size);
+
+  block = BLOCK (ptr);
+
+  type = _heapinfo[block].busy.type;
+  switch (type)
+    {
+    case 0:
+      /* Maybe reallocate a large block to a small fragment.  */
+      if (size <= BLOCKSIZE / 2)
+	{
+	  result = malloc (size);
+	  if (result != NULL)
+	    {
+	      memcpy (result, ptr, size);
+	      _free_internal (ptr);
+	      return result;
+	    }
+	}
+
+      /* The new size is a large allocation as well;
+	 see if we can hold it in place. */
+      blocks = BLOCKIFY (size);
+      if (blocks < _heapinfo[block].busy.info.size)
+	{
+	  /* The new size is smaller; return
+	     excess memory to the free list. */
+	  _heapinfo[block + blocks].busy.type = 0;
+	  _heapinfo[block + blocks].busy.info.size
+	    = _heapinfo[block].busy.info.size - blocks;
+	  _heapinfo[block].busy.info.size = blocks;
+	  /* We have just created a new chunk by splitting a chunk in two.
+	     Now we will free this chunk; increment the statistics counter
+	     so it doesn't become wrong when _free_internal decrements it.  */
+	  ++_chunks_used;
+	  _free_internal (ADDRESS (block + blocks));
+	  result = ptr;
+	}
+      else if (blocks == _heapinfo[block].busy.info.size)
+	/* No size change necessary.  */
+	result = ptr;
+      else
+	{
+	  /* Won't fit, so allocate a new region that will.
+	     Free the old region first in case there is sufficient
+	     adjacent free space to grow without moving. */
+	  blocks = _heapinfo[block].busy.info.size;
+	  /* Prevent free from actually returning memory to the system.  */
+	  oldlimit = _heaplimit;
+	  _heaplimit = 0;
+	  _free_internal (ptr);
+	  _heaplimit = oldlimit;
+	  result = malloc (size);
+	  if (result == NULL)
+	    {
+	      /* Now we're really in trouble.  We have to unfree
+		 the thing we just freed.  Unfortunately it might
+		 have been coalesced with its neighbors.  */
+	      if (_heapindex == block)
+	        (void) malloc (blocks * BLOCKSIZE);
+	      else
+		{
+		  __ptr_t previous = malloc ((block - _heapindex) * BLOCKSIZE);
+		  (void) malloc (blocks * BLOCKSIZE);
+		  _free_internal (previous);
+		}
+	      return NULL;
+	    }
+	  if (ptr != result)
+	    memmove (result, ptr, blocks * BLOCKSIZE);
+	}
+      break;
+
+    default:
+      /* Old size is a fragment; type is logarithm
+	 to base two of the fragment size.  */
+      if (size > (size_t) (1 << (type - 1)) && size <= (size_t) (1 << type))
+	/* The new size is the same kind of fragment.  */
+	result = ptr;
+      else
+	{
+	  /* The new size is different; allocate a new space,
+	     and copy the lesser of the new size and the old. */
+	  result = malloc (size);
+	  if (result == NULL)
+	    return NULL;
+	  memcpy (result, ptr, min (size, (size_t) 1 << type));
+	  free (ptr);
+	}
+      break;
+    }
+
+  return result;
+}
+/* Copyright (C) 1991, 1992 Free Software Foundation, Inc.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define	_MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+/* Allocate an array of NMEMB elements each SIZE bytes long.
+   The entire array is initialized to zeros.  */
+__ptr_t
+calloc (nmemb, size)
+     register size_t nmemb;
+     register size_t size;
+{
+  register __ptr_t result = malloc (nmemb * size);
+
+  if (result != NULL)
+    (void) memset (result, 0, nmemb * size);
+
+  return result;
+}
+/* Copyright (C) 1991, 1992, 1993 Free Software Foundation, Inc.
+This file is part of the GNU C Library.
+
+The GNU C Library is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2, or (at your option)
+any later version.
+
+The GNU C Library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with the GNU C Library; see the file COPYING.  If not, write to
+the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define	_MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+#ifndef	__GNU_LIBRARY__
+#define	__sbrk	sbrk
+#endif
+
+#ifdef __GNU_LIBRARY__
+/* It is best not to declare this and cast its result on foreign operating
+   systems with potentially hostile include files.  */
+extern __ptr_t __sbrk __P ((int increment));
+#endif
+
+#ifndef NULL
+#define NULL 0
+#endif
+
+/* Allocate INCREMENT more bytes of data space,
+   and return the start of data space, or NULL on errors.
+   If INCREMENT is negative, shrink data space.  */
+__ptr_t
+__default_morecore (increment)
+     ptrdiff_t increment;
+{
+  __ptr_t result = (__ptr_t) __sbrk ((int) increment);
+  if (result == (__ptr_t) -1)
+    return NULL;
+  return result;
+}
+/* Copyright (C) 1991, 1992, 1993 Free Software Foundation, Inc.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+__ptr_t
+memalign (alignment, size)
+     size_t alignment;
+     size_t size;
+{
+  __ptr_t result;
+  unsigned long int adj;
+
+  size = ((size + alignment - 1) / alignment) * alignment;
+
+  result = malloc (size);
+  if (result == NULL)
+    return NULL;
+  adj = (unsigned long int) ((unsigned long int) ((char *) result -
+						  (char *) NULL)) % alignment;
+  if (adj != 0)
+    {
+      struct alignlist *l;
+      for (l = _aligned_blocks; l != NULL; l = l->next)
+	if (l->aligned == NULL)
+	  /* This slot is free.  Use it.  */
+	  break;
+      if (l == NULL)
+	{
+	  l = (struct alignlist *) malloc (sizeof (struct alignlist));
+	  if (l == NULL)
+	    {
+	      free (result);
+	      return NULL;
+	    }
+	  l->next = _aligned_blocks;
+	  _aligned_blocks = l;
+	}
+      l->exact = result;
+      result = l->aligned = (char *) result + alignment - adj;
+    }
+
+  return result;
+}
Index: /trunk/indexers/mgpp/lib/huffman.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/huffman.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/huffman.cpp	(revision 3365)
@@ -0,0 +1,385 @@
+/**************************************************************************
+ *
+ * huffman.c -- Huffman coding functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "huffman.h"
+#include "messages.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+/* ->data on success,  
+   NULL on error 
+ */
+huff_data *
+Generate_Huffman_Data (int num, long *freqs, huff_data * data,
+		       u_long * mem)
+{
+  int HNum, i, count;
+  unsigned long *heap;
+  huff_data *hd = data;
+  if (!hd)
+    {
+      if (!(hd = new huff_data))
+	goto error0;
+    }
+  if (!(heap = new unsigned long[num * 2]))
+    goto error1;
+  if (!(hd->clens = new char[num]))
+      goto error2;
+  hd->num_codes = num;
+  if (mem)
+    {
+      if (hd != data)
+	*mem += sizeof (huff_data);
+      *mem += num * sizeof (char);
+    }
+  memcpy (&heap[num], freqs, num * sizeof (*heap));
+  memset (heap, '\0', num * sizeof (*heap));
+
+  /* Initialise the pointers to the leaves */
+  for (count = i = 0; i < num; i++)
+    if (heap[num + i])
+      heap[count++] = num + i;
+
+  /* Reorganise the pointers so that it is a heap */
+  HNum = count;
+  for (i = HNum / 2; i > 0; i--)
+    {
+      register int curr, child;
+      curr = i;
+      child = curr * 2;
+      while (child <= HNum)
+	{
+	  if (child < HNum && heap[heap[child]] < heap[heap[child - 1]])
+	    child++;
+	  if (heap[heap[curr - 1]] > heap[heap[child - 1]])
+	    {
+	      register u_long temp;
+	      temp = heap[child - 1];
+	      heap[child - 1] = heap[curr - 1];
+	      heap[curr - 1] = temp;
+	      curr = child;
+	      child = 2 * curr;
+	    }
+	  else
+	    break;
+	}
+    }
+
+
+  /* Form the huffman tree */
+  while (HNum > 1)
+    {
+      int pos[2];
+
+      for (i = 0; i < 2; i++)
+	{
+	  register int curr, child;
+	  pos[i] = heap[0];
+	  heap[0] = heap[--HNum];
+	  curr = 1;
+	  child = 2;
+	  while (child <= HNum)
+	    {
+	      if (child < HNum &&
+		  heap[heap[child]] < heap[heap[child - 1]])
+		child++;
+	      if (heap[heap[curr - 1]] > heap[heap[child - 1]])
+		{
+		  register int temp;
+		  temp = heap[child - 1];
+		  heap[child - 1] = heap[curr - 1];
+		  heap[curr - 1] = temp;
+		  curr = child;
+		  child = 2 * curr;
+		}
+	      else
+		break;
+	    }
+	}
+
+      heap[HNum + 1] = heap[pos[0]] + heap[pos[1]];
+      heap[pos[0]] = heap[pos[1]] = HNum + 1;
+
+      heap[HNum] = HNum + 1;
+
+      {
+	register int parent, curr;
+	HNum++;
+	curr = HNum;
+	parent = curr >> 1;
+	while (parent && heap[heap[parent - 1]] > heap[heap[curr - 1]])
+	  {
+	    register int temp;
+	    temp = heap[parent - 1];
+	    heap[parent - 1] = heap[curr - 1];
+	    heap[curr - 1] = temp;
+	    curr = parent;
+	    parent = curr >> 1;
+	  }
+      }
+    }
+
+
+  /* Calculate the code lens */
+  heap[0] = -1UL;
+  heap[1] = 0;
+  for (i = 2; i < num * 2; i++)
+    heap[i] = heap[heap[i]] + 1;
+
+  /* zero the lencount's array */
+  memset (hd->lencount, '\0', sizeof (hd->lencount));
+  hd->maxcodelen = 0;
+  hd->mincodelen = 100;
+
+  /* Set the code length of each leaf in the huffman tree */
+  for (i = 0; i < num; i++)
+    {
+      register u_long codelen = heap[i + num];
+      hd->clens[i] = (char) codelen;
+      if (!codelen)
+	continue;
+      if (codelen > hd->maxcodelen)
+	hd->maxcodelen = codelen;
+      if (codelen < hd->mincodelen)
+	hd->mincodelen = codelen;
+      hd->lencount[codelen]++;
+    }
+
+
+  if (hd->maxcodelen == 0 && hd->mincodelen == 100)
+    {
+      hd->num_codes = 0;
+    }
+  else
+    {
+      /* Calculate the current codes for each different code length */
+      hd->min_code[hd->maxcodelen] = 0;
+      for (i = hd->maxcodelen - 1; i>=0; i--)
+	hd->min_code[i] = (hd->min_code[i + 1] + hd->lencount[i + 1]) >> 1;
+    }
+  delete heap;
+  return (hd);
+
+error2:
+  delete heap;
+error1:
+  if (!data)
+    delete hd;
+error0:
+  return (NULL);
+
+}
+
+unsigned long *
+Generate_Huffman_Codes (huff_data * data, u_long * mem)
+{
+  int i;
+  unsigned long *codes;
+  unsigned long mc[MAX_HUFFCODE_LEN + 1];
+
+  if (!data)
+    return (NULL);
+  if (!(codes = new unsigned long[data->num_codes]))
+    return (NULL);
+  if (mem)
+    *mem += data->num_codes * sizeof (*codes);
+  memcpy (mc, data->min_code, sizeof (mc));
+  for (i = 0; i < data->num_codes; i++)
+    if (data->clens[i])
+      codes[i] = mc[(int) (data->clens[i])]++;
+
+  return (codes);
+}
+
+
+unsigned long **
+Generate_Huffman_Vals (huff_data * data, u_long * mem)
+{
+  int i;
+  unsigned long *fcode[MAX_HUFFCODE_LEN + 1];
+  unsigned long **values;
+  unsigned long *vals;
+
+  if (!data)
+    return (NULL);
+  if (!(vals = new unsigned long[data->num_codes]))
+    return (NULL);
+  if (!(values = new unsigned long *[MAX_HUFFCODE_LEN + 1]))
+    {
+      delete vals;
+      return (NULL);
+    }
+
+  memset (values, '\0', (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *));
+
+  if (mem)
+    *mem += data->num_codes * sizeof (*vals) +
+      (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *);
+
+  fcode[0] = values[0] = &vals[0];
+  for (i = 1; i <= data->maxcodelen; i++)
+    fcode[i] = values[i] = &vals[(values[i - 1] - vals) + data->lencount[i - 1]];
+
+  for (i = 0; i < data->num_codes; i++)
+    if (data->clens[i])
+      *fcode[(int) (data->clens[i])]++ = i;
+  return (values);
+}
+
+
+
+double 
+Calculate_Huffman_Size (int num, long *freqs, long *counts)
+{
+  double size = 0;
+  huff_data hd;
+  int i;
+  if (!Generate_Huffman_Data (num, freqs, &hd, NULL))
+    return -1;
+  for (i = 0; i < num; i++)
+    size += counts[i] * hd.clens[i];
+  delete hd.clens;
+  return size;
+}
+
+
+
+int 
+Write_Huffman_Data (FILE * f, huff_data * hd)
+{
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONSI(hd->num_codes);
+  HTONSI(hd->mincodelen);
+  HTONSI(hd->maxcodelen);
+
+  if (fwrite (&hd->num_codes, sizeof (hd->num_codes), 1, f) != 1)
+    return -1;
+
+  if (fwrite (&hd->mincodelen, sizeof (hd->mincodelen), 1, f) != 1)
+    return -1;
+
+  if (fwrite (&hd->maxcodelen, sizeof (hd->maxcodelen), 1, f) != 1)
+    return -1;
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(hd->num_codes);
+  NTOHSI(hd->mincodelen);
+  NTOHSI(hd->maxcodelen);
+
+  if (hd->num_codes)
+    {
+      /* [RPAP - Jan 97: Endian Ordering] */
+      int i;
+      for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+	HTONSI(hd->lencount[i]);
+      for (i = 0; i < hd->maxcodelen + 1; i++)
+	HTONUL(hd->min_code[i]);
+
+      if (fwrite (hd->lencount + hd->mincodelen, sizeof (*hd->lencount),
+		  hd->maxcodelen - hd->mincodelen + 1, f) !=
+	  hd->maxcodelen - hd->mincodelen + 1)
+	return -1;
+
+      if (fwrite (hd->min_code, sizeof (*hd->min_code), hd->maxcodelen + 1, f) !=
+	  hd->maxcodelen + 1)
+	return -1;
+
+      if (fwrite (hd->clens, sizeof (*hd->clens), hd->num_codes, f) != hd->num_codes)
+	return -1;
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+	NTOHSI(hd->lencount[i]);
+      for (i = 0; i < hd->maxcodelen + 1; i++)
+	NTOHUL(hd->min_code[i]);
+    }
+  return 0;
+}
+
+
+
+int Read_Huffman_Data (FILE * f, huff_data * hd, u_long * mem, u_long * disk) {
+  if (fread (&hd->num_codes, sizeof (hd->num_codes), 1, f) != 1)
+    return -1;
+
+  if (fread (&hd->mincodelen, sizeof (hd->mincodelen), 1, f) != 1)
+    return -1;
+
+  if (fread (&hd->maxcodelen, sizeof (hd->maxcodelen), 1, f) != 1)
+    return -1;
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(hd->num_codes);
+  NTOHSI(hd->mincodelen);
+  NTOHSI(hd->maxcodelen);
+
+  if (disk)
+    (*disk) += sizeof (hd->num_codes) + sizeof (hd->mincodelen) +
+      sizeof (hd->maxcodelen);
+
+  hd->clens = NULL;
+
+  if (hd->num_codes)
+    {
+      int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+      memset (hd->lencount, '\0', sizeof (hd->lencount));
+      if (fread (hd->lencount + hd->mincodelen, sizeof (*hd->lencount),
+	      hd->maxcodelen - hd->mincodelen + 1, f) !=
+	  hd->maxcodelen - hd->mincodelen + 1)
+	return -1;
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+	NTOHSI(hd->lencount[i]);
+
+      if (disk)
+	(*disk) += sizeof (*hd->lencount) * (hd->maxcodelen - hd->mincodelen + 1);
+
+      memset (hd->min_code, '\0', sizeof (hd->min_code));
+
+      if (fread (hd->min_code, sizeof (*hd->min_code), hd->maxcodelen + 1, f) !=
+	  hd->maxcodelen + 1)
+	return -1;
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (i = 0; i < hd->maxcodelen + 1; i++)
+	NTOHUL(hd->min_code[i]);
+
+      if (disk)
+	(*disk) += sizeof (*hd->min_code) * (hd->maxcodelen + 1);
+
+      if (!(hd->clens = new char[hd->num_codes]))
+	return -1;
+
+
+      if (fread (hd->clens, sizeof (*hd->clens), hd->num_codes, f) != hd->num_codes)
+	return -1;
+
+      if (disk)
+	(*disk) += sizeof (*hd->clens) * hd->num_codes;
+    }
+
+  if (mem)
+    *mem += sizeof (*hd->clens) * hd->num_codes;
+  return 0;
+}
Index: /trunk/indexers/mgpp/lib/huffman.h
===================================================================
--- /trunk/indexers/mgpp/lib/huffman.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/huffman.h	(revision 3365)
@@ -0,0 +1,57 @@
+/**************************************************************************
+ *
+ * huffman.h -- Huffman coding functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_HUFFMAN
+#define H_HUFFMAN
+
+#include "sysfuncs.h"
+#include "bitio_m_abstract.h"
+
+#define MAX_HUFFCODE_LEN 31
+
+struct huff_data {
+  int num_codes;
+  int mincodelen;
+  int maxcodelen;
+  int lencount[MAX_HUFFCODE_LEN + 1];
+  unsigned long min_code[MAX_HUFFCODE_LEN + 1];
+  char *clens;
+};
+
+
+huff_data *Generate_Huffman_Data (int num, long *freqs, huff_data * data,
+				  u_long * mem);
+
+unsigned long *Generate_Huffman_Codes (huff_data * data, u_long * mem);
+
+unsigned long **Generate_Huffman_Vals (huff_data * data, u_long * mem);
+
+int Write_Huffman_Data (FILE * f, huff_data * hd);
+
+int Read_Huffman_Data (FILE * f, huff_data * hd, u_long * mem, u_long * disk);
+
+/* Calculate the number of bits required to code the data with the
+   specified frequencies. Normally freqs and counts should point to
+   the same array. */
+double Calculate_Huffman_Size (int num, long *freqs, long *counts);
+
+
+#endif
Index: /trunk/indexers/mgpp/lib/local_strings.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/local_strings.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/local_strings.cpp	(revision 3365)
@@ -0,0 +1,411 @@
+/**************************************************************************
+ *
+ * local_strings -- provides some extra string routines
+ * Copyright (C) 1994  Authors
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "local_strings.h"
+#include "memlib.h"
+
+/* =========================================================================
+ * Function: arg_atoi
+ * Description: 
+ *      Converts an ascii string to an integer
+ *      but allows a scale string immediately following.
+ *      i.e. "10Mb" or "20Kb" or "40Megabytes"
+ *      There must be no gap between the last digit and the scale.
+ *      Only the fist letter is significant.
+ *      K = Kilo
+ *      M = Mega
+ *      G = Giga
+ * Input: 
+ *      string of form: xxxxxxK or xxxxxxxM or xxxxxxxG
+ *      where x is a digit
+ * Output: 
+ *      a scaled integer
+ * ========================================================================= */
+
+#define KILO 	1024
+#define MEGA	1024*1024
+#define GIGA	1024*1024*1024
+
+int 
+arg_atoi (char *str)
+{
+  char *str_ptr = str;
+  char ch = '\0';
+  long scale = 1;
+
+  /* find any scale letter immediately after digits */
+  ch = *str_ptr;
+  while (!isalpha (ch) && (ch != '\0'))
+    ch = *str_ptr++;
+
+  /* replace scale letter by end-of-string character */
+  *str_ptr = '\0';
+
+  /* set scale appropriately */
+  switch (ch)
+    {
+    case 'K':
+      scale = KILO;
+      break;
+    case 'M':
+      scale = MEGA;
+      break;
+    case 'G':
+      scale = GIGA;
+      break;
+    default:
+      scale = 1;
+      break;
+    }
+
+  return (atoi (str) * scale);
+
+}
+
+/* ===============================================================================
+ * Function: compare
+ * Description: Compare two strings. 
+ * Input: Two binary strings, first byte of string indicates length. 
+ * Output: Like standard strcmp.
+ *      = 0 if s1 = s2
+ *      < 0 if s1 < s2
+ *      > 0 if s1 > s2
+ * =============================================================================== */
+int 
+compare (u_char * s1, u_char * s2)
+{
+  int l1 = *s1, l2 = *s2;
+  int l = (l1 < l2) ? l1 : l2;
+
+  while (l--)
+    if (*++s1 != *++s2)
+      return (*s1 - *s2);
+  return (l1 - l2);
+
+}
+
+/*
+ * This array is designed for mapping upper and lower case letter
+ * together for a case independent comparison.  The mappings are
+ * based upon ascii character sequences.
+ */
+static unsigned char casecharmap[] = {
+	'\000', '\001', '\002', '\003', '\004', '\005', '\006', '\007',
+	'\010', '\011', '\012', '\013', '\014', '\015', '\016', '\017',
+	'\020', '\021', '\022', '\023', '\024', '\025', '\026', '\027',
+	'\030', '\031', '\032', '\033', '\034', '\035', '\036', '\037',
+	'\040', '\041', '\042', '\043', '\044', '\045', '\046', '\047',
+	'\050', '\051', '\052', '\053', '\054', '\055', '\056', '\057',
+	'\060', '\061', '\062', '\063', '\064', '\065', '\066', '\067',
+	'\070', '\071', '\072', '\073', '\074', '\075', '\076', '\077',
+	'\100', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\133', '\134', '\135', '\136', '\137',
+	'\140', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\173', '\174', '\175', '\176', '\177',
+	'\200', '\201', '\202', '\203', '\204', '\205', '\206', '\207',
+	'\210', '\211', '\212', '\213', '\214', '\215', '\216', '\217',
+	'\220', '\221', '\222', '\223', '\224', '\225', '\226', '\227',
+	'\230', '\231', '\232', '\233', '\234', '\235', '\236', '\237',
+	'\240', '\241', '\242', '\243', '\244', '\245', '\246', '\247',
+	'\250', '\251', '\252', '\253', '\254', '\255', '\256', '\257',
+	'\260', '\261', '\262', '\263', '\264', '\265', '\266', '\267',
+	'\270', '\271', '\272', '\273', '\274', '\275', '\276', '\277',
+	'\300', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\333', '\334', '\335', '\336', '\337',
+	'\340', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\373', '\374', '\375', '\376', '\377',
+};
+
+/* ===============================================================================
+ * Function: casecompare   [RPAP - Jan 97: Stem Index Change]
+ * Description: Compare two strings in dictionary order
+ * Input: Two binary strings, first byte of string indicates length. 
+ * Output:
+ *      = 0 if s1 = s2
+ *      < 0 if s1 < s2
+ *      > 0 if s1 > s2
+ * =============================================================================== */
+int 
+casecompare (u_char * s1, u_char * s2)
+{
+  int l1 = *s1, l2 = *s2;
+  int l = (l1 < l2) ? l1 : l2;
+  int pos = 0;
+  register int diff = 0;
+
+  while (l--)
+    if ((diff = casecharmap[*++s1] - casecharmap[*++s2]) != 0)
+      return (diff);
+    else if ((diff = *s1 - *s2) != 0)
+      if (!pos)
+	pos = diff;
+
+  return ((l1 - l2) ? (l1 - l2) : (pos));
+
+}
+
+/* =========================================================================
+ * Function: copy_string
+ * Description: 
+ *      Copies a string with the len in 1st byte
+ * Input: 
+ * Output: 
+ * ========================================================================= */
+
+u_char *
+copy_string (u_char * src_str)
+{
+  u_char *copy = new u_char(src_str[0] + 1);
+
+  if (!copy)
+    return NULL;
+
+  memcpy (copy, src_str, src_str[0] + 1);
+  return copy;
+
+}
+
+
+/* ===============================================================================
+ * Function: prefixlen
+ * Description: Returns length of common prefix string between s1 and s2.
+ * Input: two binary strings
+ * Output: prefix length
+ * =============================================================================== */
+int 
+prefixlen (u_char * s1, u_char * s2)
+{
+  int l = (*s1 < *s2) ? *s1 : *s2;
+  int i = 0;
+
+  while (i < l && *++s1 == *++s2)
+    i++;
+  return (i);
+}
+
+
+/* ===============================================================================
+ * Function: char2str
+ * Description: 
+ *      Ensures that non-printable characters are converted to a string
+ *      which is printable i.e. it indentifies the character.
+ *      e.g. carriage-return = "\n".
+ * Input: character (may be non-printable)
+ * Output: pointer to static string
+ * =============================================================================== */
+
+char *
+char2str (u_char c)
+{
+  static char buf[10];
+  switch (c)
+    {
+    case '\n':
+      sprintf (buf, "\\n");
+      break;
+    case '\b':
+      sprintf (buf, "\\b");
+      break;
+    case '\f':
+      sprintf (buf, "\\f");
+      break;
+    case '\t':
+      sprintf (buf, "\\t");
+      break;
+    default:
+	if (c >= ' ' && c <= 126)
+	switch (c)
+	  {
+	  case '\\':
+	    sprintf (buf, "\\\\");
+	    break;
+	  case '\"':
+	    sprintf (buf, "\\\"");
+	    break;
+	  case '\'':
+	    sprintf (buf, "\\\'");
+	    break;
+	  default:
+	    buf[0] = c;
+	    buf[1] = '\0';
+	    break;
+	  }
+      else
+	sprintf (buf, "\\x%02x", c);
+      break;
+    }
+  return buf;
+}
+
+/* ===============================================================================
+ * Function: word2str
+ * Description: 
+ *      Converts a string with possible non-printable characters into a string
+ *      consisting entirely of printable characters.
+ * Input: string, length stored in first byte
+ * Output: pointer to static string
+ * =============================================================================== */
+
+char *
+word2str (u_char * s)
+{
+  static char buf[1024];
+  char *p = buf;
+  int i, len = (int) *s++;
+
+  memset (buf, '\0', sizeof(buf));
+  for (i = 0; i < len; i++)
+    {
+      strcpy (p, char2str (s[i]));
+      p = strchr (p, '\0');
+    }
+  return buf;
+}
+
+
+/* =========================================================================
+ * Function: de_escape_string
+ * Description: 
+ *      Turn a string containing \ escapes into real characters
+ *      Note this destroys the string s in the process
+ * Input: string to de-escape 
+ * Output: the de-escaped string
+ * ========================================================================= */
+char *
+de_escape_string (char *s)
+{
+  char *org = s;
+  char *d = s;
+  while (*s)
+    {
+      if (*s == '\\')
+	{
+	  s++;
+	  switch (*s++)
+	    {
+	    case '\0':
+	      *d++ = '\\';
+	      s--;
+	      break;
+	    case '\\':
+	      *d++ = '\\';
+	      break;
+	    case 'b':
+	      *d++ = '\b';
+	      break;
+	    case 'f':
+	      *d++ = '\f';
+	      break;
+	    case 'n':
+	      *d++ = '\n';
+	      break;
+	    case 'r':
+	      *d++ = '\r';
+	      break;
+	    case 't':
+	      *d++ = '\r';
+	      break;
+	    case '\"':
+	      *d++ = '\"';
+	      break;
+	    case '\'':
+	      *d++ = '\'';
+	      break;
+	    case 'x':
+	      {
+		int num = 0;
+		if (isxdigit (*s))
+		  {
+		    char ch = toupper (*s++);
+		    num = ch <= '9' ? ch - '0' : ch - 'A' + 10;
+		  }
+		if (isxdigit (*s))
+		  {
+		    char ch = toupper (*s++);
+		    num = (num << 4) + (ch <= '9' ? ch - '0' : ch - 'A' + 10);
+		  }
+		*d++ = num;
+		break;
+	      }
+	    case '0':
+	    case '1':
+	    case '2':
+	    case '3':
+	    case '4':
+	    case '5':
+	    case '6':
+	    case '7':
+	      {
+		int num = *(s - 1) - '0';
+		if (*s >= '0' && *s <= '7')
+		  num = (num << 3) + *s++ - '0';
+		if (*s >= '0' && *s <= '7')
+		  num = (num << 3) + *s++ - '0';
+		*d++ = num;
+		break;
+	      }
+	    }
+	}
+      else
+	*d++ = *s++;
+    }
+  *d = '\0';
+  return (org);
+}
+
+/* =========================================================================
+ * Function: str255_to_string 
+ * Description: 
+ * Input: 
+ * Output: 
+ * ========================================================================= */
+
+char *
+str255_to_string (u_char * str255, char *str)
+{
+  static char buf[256];
+  char *str_base = NULL;
+  int len = *str255++;
+  int i = 0;
+
+  /* if haven't been given a str to write to then use local buffer */
+  if (!str)
+    str = &(buf[0]);
+
+  str_base = str;
+  for (i = 0; i < len; i++)
+    {
+      *str++ = *str255++;
+    }
+  *str = '\0';
+
+  return str_base;
+}
Index: /trunk/indexers/mgpp/lib/local_strings.h
===================================================================
--- /trunk/indexers/mgpp/lib/local_strings.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/local_strings.h	(revision 3365)
@@ -0,0 +1,79 @@
+/**************************************************************************
+ *
+ * local_strings.h -- string routines
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef LOCAL_STRINGS_H
+#define LOCAL_STRINGS_H
+
+#include "sysfuncs.h"
+
+class mgString
+{
+  u_char *data;
+ public:
+  inline int length()
+    {
+      return data[0];
+    }
+  inline u_char *rawdata()
+    {
+      return data;
+    }
+  inline void read(u_char **base)
+    {
+      unsigned int copy, suff;
+      
+      copy  = **base;
+      *base = *base + 1;
+      suff  = **base;
+      *base = *base + 1;
+      memcpy( (data + copy + 1), *base, suff);
+      *data = copy + suff;
+      *base += suff;
+    }
+  inline void read(FILE *dict)
+    {
+      unsigned int copy, suff;
+      copy = fgetc(dict);
+      suff = fgetc(dict);
+      fread(data + copy + 1, sizeof(u_char), suff, dict);
+      *data = copy + suff;
+    }
+  inline static void readString(u_char **base, u_char *prev)
+    {
+      ((mgString *) &prev)->read(base);
+    }
+  inline static void readString(FILE *dict, u_char *prev)
+    { 
+      ((mgString *) &prev)->read(dict);
+    }
+};
+
+int arg_atoi (char *str);
+int compare (u_char * s1, u_char * s2);
+int casecompare (u_char * s1, u_char * s2);   /* [RPAP - Jan 97: Stem Index Change] */
+u_char *copy_string (u_char * src_str);
+char *str255_to_string (u_char * str255, char *str);
+int prefixlen (u_char * s1, u_char * s2);
+char *char2str (u_char c);
+char *word2str (u_char * s);
+char *de_escape_string (char *s);
+
+#endif
Index: /trunk/indexers/mgpp/lib/longlong.h
===================================================================
--- /trunk/indexers/mgpp/lib/longlong.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/longlong.h	(revision 3365)
@@ -0,0 +1,85 @@
+/**************************************************************************
+ *
+ * longlong.h -- Use of GCC's long long integer types
+ * Copyright (C) 1999 Tim A.H. Bell
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_LONGLONG
+#define H_LONGLONG
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__
+#  include <WIN32cfg.h>
+# else
+#  include <config.h>
+# endif
+#endif
+
+/*
+  Use GCC's "long long" integer types for certain variables, to avoid
+  overflowing 32 bit integers.
+
+  Other files affected:
+  
+    lib:
+        bitio_random.c
+        bitio_random.h
+	
+    src/text:
+        ivf.pass1.c
+        ivf.pass2.c
+        build.h
+        mg_passes.c
+        text.h
+    
+*/
+
+/*#define TESTING_OVERFLOW*/
+
+#ifdef TESTING_OVERFLOW
+
+/* Test the overflow detection by using tiny (16-bit) types */
+typedef unsigned short int mg_ullong;
+typedef short int mg_llong;
+
+#define ULL_FS "u"
+#define LL_FS "d"
+
+#elif defined __GNUC__ && ! defined DISABLE_LONG_LONG
+
+/* We're using GCC, so it's okay to use "long long" (64-bit) types */
+#define USE_LONG_LONG
+
+typedef unsigned long long int mg_ullong;
+typedef long long int mg_llong;
+
+#define ULL_FS "llu"
+#define LL_FS "lld"
+
+#else
+
+/* Not using GCC, so fall back on plain "long" (32-bit) types */
+typedef unsigned long int mg_ullong;
+typedef long int mg_llong;
+
+#define ULL_FS "lu"
+#define LL_FS "ld"
+
+#endif /* __GNUC__ */
+
+#endif
Index: /trunk/indexers/mgpp/lib/lovinstem.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/lovinstem.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/lovinstem.cpp	(revision 3365)
@@ -0,0 +1,1172 @@
+/**************************************************************************
+ *
+ * lovinstem.c -- Stemming code
+ * Copyright (C) 1994  Linh Huynh (linh@kbs.citri.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+
+/*****************************************************************************
+ *
+ *   Transformational Rules used in Recoding Stem Terminations
+ *
+ *   Parameter in functions:
+ *       ch : the second last character of the old ending
+ *            (after removing one of double consonants) 
+ *
+ ****************************************************************************/
+
+
+/************ Conditional rules associated with transformations **************/
+
+/*static int aio ();
+static int s ();
+static int pt ();
+static int m ();
+static int n ();*/
+
+static int 
+aio (char ch)			/* Rule number 9     */
+{
+  return ((ch != 'a') && (ch != 'i') && (ch != 'o'));
+}
+
+static int 
+s (char ch)				/* Rule Number 24    */
+{
+  return (ch != 's');
+}
+
+static int 
+pt (char ch)				/* Rule number 28    */
+{
+  return ((ch != 'p') && (ch != 't'));
+}
+
+static int 
+m (char ch)				/* Rule number 30    */
+{
+  return (ch != 'm');
+}
+
+static int 
+n (char ch)				/* Rule number 32    */
+{
+  return (ch != 'n');
+}
+
+
+/**************************************************************************
+ *
+ *  Context-sensitive rules associated with certain endings.
+ *
+ *  The notations of the rules are the same as in Lovins' paper
+ *  except that rule A is replaced by a NULL in the data structure.
+ *
+ *  Function parameters:
+ *      - stem_length: possible stem length
+ *      - end        : pointer points to the end of the possible stem.
+ *
+ *************************************************************************/
+
+
+/********* Context-sensitive rule function declarations ******************/
+
+/*static int B ();
+static int C ();
+static int D ();
+static int E ();
+static int F ();
+static int G ();
+static int H ();
+static int I ();
+static int J ();
+static int K ();
+static int L ();
+static int M ();
+static int N ();
+static int O ();
+static int P ();
+static int Q ();
+static int R ();
+static int S ();
+static int T ();
+static int U ();
+static int V ();
+static int W ();
+static int X ();
+static int Y ();
+static int Z ();
+static int AA ();
+static int BB ();
+static int CC ();
+*/
+static int 
+B (int stem_length, char *end)
+{
+  return (stem_length >= 3);
+}
+
+static int 
+C (int stem_length, char *end)
+{
+  return (stem_length >= 4);
+}
+
+static int 
+D (int stem_length, char *end)
+{
+  return (stem_length >= 5);
+}
+
+static int 
+E (int stem_length, char *end)
+
+     
+{
+  return (*end != 'e');
+}
+
+static int 
+F (int stem_length, char *end)
+     
+     
+{
+  return ((stem_length >= 3) && (*end != 'e'));
+}
+
+static int 
+G (int stem_length, char *end)
+     
+     
+{
+  return ((stem_length >= 3) && (*end == 'f'));
+}
+
+static int 
+H (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+  return ((p1 == 't') || ((p1 == 'l') && (p2 == 'l')));
+}
+
+static int 
+I (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 != 'o') && (p1 != 'e'));
+}
+
+static int 
+J (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 != 'a') && (p1 != 'e'));
+}
+
+static int 
+K (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((stem_length >= 3) &&
+	  ((p1 == 'l') || (p1 == 'i') ||
+	   ((p1 == 'e') && (*(end - 2) == 'u'))));
+}
+
+static int 
+L (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+  if (p1 == 's')
+    return (p2 == 'o');
+  else
+    return ((p1 != 'u') && (p1 != 'x'));
+}
+
+static int 
+M (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 != 'a') && (p1 != 'c') &&
+	  (p1 != 'e') && (p1 != 'm'));
+}
+
+static int 
+N (int stem_length, char *end)
+     
+     
+{
+  if (stem_length >= 3)
+    {
+      if (*(end - 2) == 's')
+	return (stem_length >= 4);
+      else
+	return 1;
+    }
+  else
+    return 0;
+}
+
+static int 
+O (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 == 'l') || (p1 == 'i'));
+}
+
+static int 
+P (int stem_length, char *end)
+     
+     
+{
+  return (*end != 'c');
+}
+
+static int 
+Q (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  return ((stem_length >= 3) && (p1 != 'l') && (p1 != 'n'));
+}
+
+static int 
+R (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 == 'n') || (p1 == 'r'));
+}
+
+static int 
+S (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+
+  if (p1 == 't')
+    return (p2 != 't');
+  else
+    return ((p1 == 'r') && (p2 == 'd'));
+}
+
+static int 
+T (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+
+  if (p1 == 't')
+    return (p2 != 'o');
+  else
+    return (p1 == 's');
+}
+
+static int 
+U (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  return ((p1 == 'l') || (p1 == 'm') ||
+	  (p1 == 'n') || (p1 == 'r'));
+}
+
+static int 
+V (int stem_length, char *end)
+     
+     
+{
+  return (*end == 'c');
+}
+
+static int 
+W (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  return ((p1 != 's') && (p1 != 'u'));
+}
+
+static int 
+X (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  if (p1 == 'e')
+    return ((stem_length >= 3) && (*(end - 2) == 'u'));
+  else
+    return ((p1 == 'l') || (p1 == 'i'));
+}
+
+static int 
+Y (int stem_length, char *end)
+     
+     
+{
+  return ((*end == 'n') && (*(end - 1) == 'i'));
+}
+
+static int 
+Z (int stem_length, char *end)
+     
+     
+{
+  return (*end != 'f');
+}
+
+static int 
+AA (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+
+  if (p1 == 'h')
+    return ((p2 == 'p') || (p2 == 't'));
+  else if (p1 == 'r')
+    return ((p2 == 'e') || (p2 == 'o'));
+  else if (p1 == 's')
+    return (p2 == 'e');
+  else
+    return ((p1 == 'd') || (p1 == 'f') ||
+	    (p1 == 'l') || (p1 == 't'));
+}
+
+static int 
+BB (int stem_length, char *end)
+     
+     
+{
+  if (stem_length >= 3)
+    {
+      if (stem_length >= 4)
+	return (strncmp (end - 3, "ryst", 4) != 0);
+      else
+	return (strncmp (end - 2, "met", 3) != 0);
+    }
+  else
+    return 0;
+}
+
+static int 
+CC (int stem_length, char *end)
+     
+     
+{
+  return (*end == 'l');
+}
+
+/**************************************************************************/
+
+#define MIN_STEM_LENGTH 2
+#define MAX_ENDING_SIZE 11
+#define PREDEFINED_SIZE (MAX_ENDING_SIZE + MIN_STEM_LENGTH)
+#define EOS     '\0'
+/* [RPAP - Jan 97: Stem Index Change] */
+#define MAX_STEM_LENGTH 255
+
+static char *remove_ending (char *word, int w_len);
+static void recode_stem (char *stem_end);
+
+/* =========================================================================
+ * Function: stem
+ * Description: 
+ *      Implemetation of the Lovins' stemming algorithm described in
+ *      J.B. Lovins, "Development of a Stemming Algorithm",
+ *      Mechanical Translation and Computational Linguistics, Vol 11,1968.
+ * Input: a word string with the length in the first byte
+ * Output: the stemmed word
+ * ========================================================================= */
+
+void
+lovinstem (u_char * word)
+{
+  char *word_start;		/* points to first letter of word */
+  char *stem_end;		/* points to last character of stem portion */
+  int old_length;		/* length of word */
+  int new_length;		/* length after stemming */
+
+  /* [RPAP - Jan 97: Stem Index Change] */
+  u_char copy[MAX_STEM_LENGTH + 2];
+  memcpy (copy, word, *word + 1);
+
+  old_length = *copy; /* [RPAP - Jan 97: Stem Index Change] */
+
+  /* The word must be at least MIN_STEM_LENGTH characters long. */
+  if (old_length <= MIN_STEM_LENGTH)
+    return;
+
+  word_start = (char *) copy + 1;	/* jump over length byte */ /* [RPAP - Jan 97: Stem Index Change] */
+  copy[old_length + 1] = '\0';	/* null terminate string */ /* [RPAP - Jan 97: Stem Index Change] */
+
+  stem_end = remove_ending (word_start, old_length);
+  recode_stem (stem_end);
+
+  /* fix up new length */
+  /* have to use strlen because have no other way of telling length */
+  new_length = strlen (word_start);
+  *copy = new_length;  /* [RPAP - Jan 97: Stem Index Change] */
+
+  /* [RPAP - Jan 97: Stem Index Change] */
+  memcpy (word, copy, *copy + 1);
+
+}				/*stem_complex */
+
+/* =========================================================================
+
+ *   Data structure for recoding and the method for recoding the stem. 
+ *
+ * ========================================================================= */
+
+typedef struct
+{
+  char *old_end;		/* old ending */
+  char *new_end;		/* new ending */
+  char old_offset;		/* length of the old ending - 1 */
+  int (*cond) (char);	        /* condition rule */
+  char end_group;		/* signal the end of the group */
+}
+Recode_Rules;
+
+/*  
+   Stem terminations are divided into groups having the same last
+   character
+ */
+
+static Recode_Rules Rules[] =
+{
+  "uad", "uas", 2, NULL, 0,
+  "vad", "vas", 2, NULL, 0,
+  "cid", "cis", 2, NULL, 0,
+  "lid", "lis", 2, NULL, 0,
+  "erid", "eris", 3, NULL, 0,
+  "pand", "pans", 3, NULL, 0,
+  "end", "ens", 2, s, 0,
+  "end", "ens", 2, m, 0,
+  "ond", "ons", 2, NULL, 0,
+  "lud", "lus", 2, NULL, 0,
+  "rud", "rus", 2, NULL, 1,
+
+  "ul", "l", 1, aio, 1,
+
+  "istr", "ister", 3, NULL, 0,
+  "metr", "meter", 3, NULL, 0,
+  "her", "hes", 2, pt, 1,
+
+  "urs", "ur", 2, NULL, 1,
+
+  "uct", "uc", 2, NULL, 0,
+  "umpt", "um", 3, NULL, 0,
+  "rpt", "rb", 2, NULL, 0,
+  "mit", "mis", 2, NULL, 0,
+  "ert", "ers", 2, NULL, 0,
+  "et", "es", 1, n, 0,
+  "yt", "ys", 1, NULL, 1,
+
+  "iev", "ief", 2, NULL, 0,
+  "olv", "olut", 2, NULL, 1,
+
+  "bex", "bic", 2, NULL, 0,
+  "dex", "dic", 2, NULL, 0,
+  "pex", "pic", 2, NULL, 0,
+  "tex", "tic", 2, NULL, 0,
+  "ax", "ac", 1, NULL, 0,
+  "ex", "ec", 1, NULL, 0,
+  "ix", "ic", 1, NULL, 0,
+  "lux", "luc", 2, NULL, 1,
+
+  "yz", "ys", 1, NULL, 1
+};
+
+typedef struct last
+  {
+    char c;			/* The last character */
+    struct last *left;		/* used in balanced   */
+    struct last *right;		/* binary tree        */
+    Recode_Rules *pt;		/* location of approriate group */
+  }
+Last_Char_Node;
+
+/*------------
+       s
+     /   \
+    l     x
+   / \   / \
+  d   r t   z
+         \
+          v
+---------------*/
+static Last_Char_Node pr[] =
+{
+  'd', NULL, NULL, Rules,
+  'l', pr, pr + 2, Rules + 11,
+  'r', NULL, NULL, Rules + 12,
+  's', pr + 1, pr + 6, Rules + 15,
+  't', NULL, pr + 5, Rules + 16,
+  'v', NULL, NULL, Rules + 23,
+  'x', pr + 4, pr + 7, Rules + 25,
+  'z', NULL, NULL, Rules + 33,
+};
+
+
+/*****************************************************************************
+ *
+ *  Recode the stem according to the transformation rules.
+ *
+ *****************************************************************************/
+
+static void
+recode_stem (char *stem_end)
+{
+  static Last_Char_Node *root = pr + 3;
+  Last_Char_Node *p_last = root;
+  Recode_Rules *rule;		/* points to the transformation list */
+  char *h,			/* points to start of possible ending */
+    ch,				/* last character of the old ending  */
+    ch2;
+
+  h = stem_end - 1;
+  ch = *stem_end;
+  if (*h == ch)			/* Check for double consonant        */
+    {
+      if (strchr ("bdglmnprst", ch) != NULL)
+	{
+	  *stem_end = EOS;
+	  stem_end = h;
+	}
+    }
+
+  do				/* Check for the last character       */
+    {
+      ch2 = p_last->c;
+      if (ch == ch2)
+	break;
+      else if (ch > ch2)
+	p_last = p_last->right;
+      else
+	p_last = p_last->left;
+    }
+  while (p_last != NULL);
+
+  if (p_last != NULL)		/* Check for the rest of suffix list */
+    {
+      rule = p_last->pt;
+      for (;;)
+	{
+	  h = stem_end - rule->old_offset;
+	  if (strcmp (h, rule->old_end) == 0)
+	    {
+	      if (!rule->cond || (*rule->cond) (*(h - 1)))
+		{
+		  (void) strcpy (h, rule->new_end);
+		  break;
+		}
+	    }
+	  if (rule->end_group)
+	    break;
+	  rule++;
+	}
+    }
+}
+
+/* ======================================================================
+
+ *    List of endings and the method to remove the ending 
+ *
+ * ======================================================================*/
+
+/************ Data structures for list of endings  ********************/
+
+typedef struct
+{
+  char *ending;			/* old ending                */
+  int (*cond) (int length, char *end);		/* conditional rule          */
+  signed char left_offset;	/* used to find the siblings */
+  signed char right_offset;	/* in balanced binary tree   */
+}
+Ending_List;
+
+/*
+   The original list of endings is re-organised into groups having
+   the same length and the same first character.
+ */
+
+static Ending_List List[] =
+{
+  "a", NULL, 0, 0,
+
+  "ae", NULL, 0, 0,
+  "al", BB, -1, 2,
+  "ar", X, 0, 0,
+  "as", B, -1, 0,
+
+  "acy", NULL, 0, 1,
+  "age", B, 0, 0,
+  "aic", NULL, -2, 1,
+  "als", BB, 0, 0,
+  "ant", B, -2, 2,
+  "ars", O, 0, 0,
+  "ary", F, -1, 2,
+  "ata", NULL, 0, 0,
+  "ate", NULL, -1, 0,
+
+  "able", NULL, 0, 1,
+  "ably", NULL, 0, 0,
+  "ages", B, -2, 2,
+  "ally", B, 0, 0,
+  "ance", B, -1, 1,
+  "ancy", B, 0, 0,
+  "ants", B, -4, 4,
+  "aric", NULL, 0, 0,
+  "arly", K, -1, 1,
+  "ated", I, 0, 0,
+  "ates", NULL, -2, 2,
+  "atic", B, 0, 0,
+  "ator", NULL, -1, 0,
+
+  "acies", NULL, 0, 0,
+  "acity", NULL, -1, 1,
+  "aging", B, 0, 0,
+  "aical", NULL, -2, 2,
+  "alist", NULL, 0, 0,
+  "alism", B, -1, 0,
+  "ality", NULL, -3, 3,
+  "alize", NULL, 0, 1,
+  "allic", BB, 0, 0,
+  "anced", B, -2, 2,
+  "ances", B, 0, 0,
+  "antic", C, -1, 0,
+  "arial", NULL, -6, 6,
+  "aries", NULL, 0, 1,
+  "arily", NULL, 0, 0,
+  "arity", B, -2, 2,
+  "arize", NULL, 0, 0,
+  "aroid", NULL, -1, 0,
+  "ately", NULL, -3, 3,
+  "ating", I, 0, 1,
+  "ation", B, 0, 0,
+  "ative", NULL, -2, 2,
+  "ators", NULL, 0, 0,
+  "atory", NULL, -1, 1,
+  "ature", E, 0, 0,
+
+  "aceous", NULL, 0, 1,
+  "acious", B, 0, 0,
+  "action", G, -2, 2,
+  "alness", NULL, 0, 0,
+  "ancial", NULL, -1, 1,
+  "ancies", NULL, 0, 0,
+  "ancing", B, -4, 4,
+  "ariser", NULL, 0, 0,
+  "arized", NULL, -1, 1,
+  "arizer", NULL, 0, 0,
+  "atable", NULL, -2, 2,
+  "ations", B, 0, 0,
+  "atives", NULL, -1, 0,
+
+  "ability", NULL, 0, 1,
+  "aically", NULL, 0, 0,
+  "alistic", B, -2, 2,
+  "alities", NULL, 0, 0,
+  "ariness", E, -1, 0,
+  "aristic", NULL, -3, 3,
+  "arizing", NULL, 0, 1,
+  "ateness", NULL, 0, 0,
+  "atingly", NULL, -2, 2,
+  "ational", B, 0, 0,
+  "atively", NULL, -1, 1,
+  "ativism", NULL, 0, 0,
+
+  "ableness", NULL, 0, 1,
+  "arizable", NULL, 0, 0,
+
+  "allically", C, 0, 0,
+  "antaneous", NULL, -1, 1,
+  "antiality", NULL, 0, 0,
+  "arisation", NULL, -2, 2,
+  "arization", NULL, 0, 0,
+  "ationally", B, -1, 1,
+  "ativeness", NULL, 0, 0,
+
+  "antialness", NULL, 0, 0,
+  "arisations", NULL, -1, 1,
+  "arizations", NULL, 0, 0,
+
+  "alistically", B, 0, 1,
+  "arizability", NULL, 0, 0,
+
+  "e", NULL, 0, 0,
+
+  "ed", E, 0, 0,
+  "en", F, -1, 1,
+  "es", E, 0, 0,
+
+  "eal", Y, 0, 0,
+  "ear", Y, -1, 1,
+  "ely", E, 0, 0,
+  "ene", E, -2, 2,
+  "ent", C, 0, 0,
+  "ery", E, -1, 1,
+  "ese", NULL, 0, 0,
+
+  "ealy", Y, 0, 1,
+  "edly", E, 0, 0,
+  "eful", NULL, -2, 1,
+  "eity", NULL, 0, 0,
+  "ence", NULL, -2, 2,
+  "ency", NULL, 0, 0,
+  "ened", E, -1, 2,
+  "enly", E, 0, 0,
+  "eous", NULL, -1, 0,
+
+  "early", Y, 0, 1,
+  "ehood", NULL, 0, 0,
+  "eless", NULL, -2, 2,
+  "elity", NULL, 0, 0,
+  "ement", NULL, -1, 0,
+  "enced", NULL, -3, 3,
+  "ences", NULL, 0, 1,
+  "eness", E, 0, 0,
+  "ening", E, -2, 2,
+  "ental", NULL, 0, 0,
+  "ented", C, -1, 1,
+  "ently", NULL, 0, 0,
+
+  "eature", Z, 0, 0,
+  "efully", NULL, -1, 1,
+  "encies", NULL, 0, 0,
+  "encing", NULL, -2, 2,
+  "ential", NULL, 0, 0,
+  "enting", C, -1, 1,
+  "entist", NULL, 0, 1,
+  "eously", NULL, 0, 0,
+
+  "elihood", E, 0, 1,
+  "encible", NULL, 0, 0,
+  "entally", NULL, -2, 2,
+  "entials", NULL, 0, 0,
+  "entiate", NULL, -1, 1,
+  "entness", NULL, 0, 0,
+
+  "entation", NULL, 0, 0,
+  "entially", NULL, -1, 1,
+  "eousness", NULL, 0, 0,
+
+  "eableness", E, 0, 1,
+  "entations", NULL, 0, 0,
+  "entiality", NULL, -2, 2,
+  "entialize", NULL, 0, 0,
+  "entiation", NULL, -1, 0,
+
+  "entialness", NULL, 0, 0,
+
+  "ful", NULL, 0, 0,
+
+  "fully", NULL, 0, 0,
+
+  "fulness", NULL, 0, 0,
+
+  "hood", NULL, 0, 0,
+
+  "i", NULL, 0, 0,
+
+  "ia", NULL, 0, 0,
+  "ic", NULL, -1, 1,
+  "is", NULL, 0, 0,
+
+  "ial", NULL, 0, 0,
+  "ian", NULL, -1, 1,
+  "ics", NULL, 0, 1,
+  "ide", L, 0, 0,
+  "ied", NULL, -3, 3,
+  "ier", NULL, 0, 0,
+  "ies", P, -1, 0,
+  "ily", NULL, -1, 1,
+  "ine", M, 0, 0,
+  "ing", N, -5, 5,
+  "ion", Q, 0, 0,
+  "ish", C, -1, 1,
+  "ism", B, 0, 1,
+  "ist", NULL, 0, 0,
+  "ite", AA, -3, 3,
+  "ity", NULL, 0, 0,
+  "ium", NULL, -1, 0,
+  "ive", NULL, -1, 1,
+  "ize", F, 0, 0,
+
+  "ials", NULL, 0, 0,
+  "ians", NULL, -1, 0,
+  "ible", NULL, -1, 1,
+  "ibly", NULL, 0, 0,
+  "ical", NULL, -2, 2,
+  "ides", L, 0, 0,
+  "iers", NULL, -1, 1,
+  "iful", NULL, 0, 0,
+  "ines", M, -4, 4,
+  "ings", N, 0, 0,
+  "ions", B, -1, 1,
+  "ious", NULL, 0, 0,
+  "isms", B, -2, 2,
+  "ists", NULL, 0, 0,
+  "itic", H, -1, 1,
+  "ized", F, 0, 1,
+  "izer", F, 0, 0,
+
+  "ially", NULL, 0, 0,
+  "icant", NULL, -1, 1,
+  "ician", NULL, 0, 0,
+  "icide", NULL, -2, 2,
+  "icism", NULL, 0, 0,
+  "icist", NULL, -1, 0,
+  "icity", NULL, -3, 3,
+  "idine", I, 0, 1,
+  "iedly", NULL, 0, 0,
+  "ihood", NULL, -2, 2,
+  "inate", NULL, 0, 0,
+  "iness", NULL, -1, 0,
+  "ingly", B, -6, 6,
+  "inism", J, 0, 1,
+  "inity", CC, 0, 0,
+  "ional", NULL, -2, 2,
+  "ioned", NULL, 0, 0,
+  "ished", NULL, -1, 0,
+  "istic", NULL, -3, 3,
+  "ities", NULL, 0, 1,
+  "itous", NULL, 0, 0,
+  "ively", NULL, -2, 2,
+  "ivity", NULL, 0, 0,
+  "izers", F, -1, 1,
+  "izing", F, 0, 0,
+
+  "ialist", NULL, 0, 0,
+  "iality", NULL, -1, 1,
+  "ialize", NULL, 0, 0,
+  "ically", NULL, -2, 2,
+  "icance", NULL, 0, 0,
+  "icians", NULL, -1, 1,
+  "icists", NULL, 0, 0,
+  "ifully", NULL, -4, 4,
+  "ionals", NULL, 0, 0,
+  "ionate", D, -1, 1,
+  "ioning", NULL, 0, 0,
+  "ionist", NULL, -2, 2,
+  "iously", NULL, 0, 0,
+  "istics", NULL, -1, 1,
+  "izable", E, 0, 0,
+
+  "ibility", NULL, 0, 0,
+  "icalism", NULL, -1, 1,
+  "icalist", NULL, 0, 1,
+  "icality", NULL, 0, 0,
+  "icalize", NULL, -3, 3,
+  "ication", G, 0, 0,
+  "icianry", NULL, -1, 0,
+  "ination", NULL, -1, 1,
+  "ingness", NULL, 0, 0,
+  "ionally", NULL, -5, 5,
+  "isation", NULL, 0, 0,
+  "ishness", NULL, -1, 1,
+  "istical", NULL, 0, 1,
+  "iteness", NULL, 0, 0,
+  "iveness", NULL, -3, 3,
+  "ivistic", NULL, 0, 0,
+  "ivities", NULL, -1, 0,
+  "ization", F, -1, 1,
+  "izement", NULL, 0, 0,
+
+  "ibleness", NULL, 0, 0,
+  "icalness", NULL, -1, 1,
+  "ionalism", NULL, 0, 0,
+  "ionality", NULL, -2, 2,
+  "ionalize", NULL, 0, 0,
+  "iousness", NULL, -1, 1,
+  "izations", NULL, 0, 0,
+
+  "ionalness", NULL, 0, 1,
+  "istically", NULL, 0, 0,
+  "itousness", NULL, -2, 2,
+  "izability", NULL, 0, 0,
+  "izational", NULL, -1, 0,
+
+  "izationally", B, 0, 0,
+
+  "ly", B, 0, 0,
+
+  "less", NULL, 0, 1,
+  "lily", NULL, 0, 0,
+
+  "lessly", NULL, 0, 0,
+
+  "lessness", NULL, 0, 0,
+
+  "ness", NULL, 0, 0,
+
+  "nesses", NULL, 0, 0,
+
+  "o", NULL, 0, 0,
+
+  "on", S, 0, 1,
+  "or", T, 0, 0,
+
+  "oid", NULL, 0, 0,
+  "one", R, -1, 1,
+  "ous", NULL, 0, 0,
+
+  "ogen", NULL, 0, 0,
+
+  "oidal", NULL, 0, 0,
+  "oides", NULL, -1, 2,
+  "otide", NULL, 0, 0,
+  "ously", NULL, -1, 0,
+
+  "oidism", NULL, 0, 0,
+
+  "oidally", NULL, 0, 1,
+  "ousness", NULL, 0, 0,
+
+  "s", W, 0, 0,
+
+  "s'", NULL, 0, 0,
+
+  "um", U, 0, 1,
+  "us", V, 0, 0,
+
+  "ward", NULL, 0, 1,
+  "wise", NULL, 0, 0,
+
+  "y", B, 0, 0,
+
+  "yl", R, 0, 0,
+
+  "ying", B, 0, 1,
+  "yish", NULL, 0, 0,
+
+  "'s", NULL, 0, 0,
+};
+
+typedef struct node
+  {
+    char c;			/* First character  */
+    struct node *left;		/* used in balanced */
+    struct node *right;		/* binary tree      */
+    Ending_List *ptr[11];	/* the approriate location */
+  }
+First_Char_Node;
+
+static First_Char_Node First[] =
+{
+  '\'', NULL, NULL,
+  {NULL,
+   List + 293, NULL, NULL, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'a', First, NULL,
+  {List,
+   List + 2, List + 9, List + 20, List + 39, List + 58,
+   List + 70, List + 77, List + 82, List + 87, List + 89},
+
+  'e', First + 1, First + 4,
+  {List + 91,
+   List + 93, List + 98, List + 106, List + 116, List + 126,
+   List + 133, List + 138, List + 142, List + 145, NULL},
+
+  'f', NULL, NULL,
+  {NULL,
+   NULL, List + 146, NULL, List + 147, NULL,
+   List + 148, NULL, NULL, NULL, NULL},
+
+  'h', First + 3, First + 5,
+  {NULL,
+   NULL, NULL, List + 149, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'i', NULL, NULL,
+  {List + 150,
+   List + 152, List + 163, List + 181, List + 202, List + 222,
+   List + 239, List + 252, List + 258, NULL, List + 261},
+
+  'l', First + 2, First + 10,
+  {NULL,
+   List + 262, NULL, List + 263, NULL, List + 265,
+   NULL, List + 266, NULL, NULL, NULL},
+
+  'n', NULL, NULL,
+  {NULL,
+   NULL, NULL, List + 267, NULL, List + 268,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'o', First + 7, First + 9,
+  {List + 269,
+   List + 270, List + 273, List + 275, List + 277, List + 280,
+   List + 281, NULL, NULL, NULL, NULL},
+
+  's', NULL, NULL,
+  {List + 283,
+   List + 284, NULL, NULL, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'u', First + 8, First + 12,
+  {NULL,
+   List + 285, NULL, NULL, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'w', NULL, NULL,
+  {NULL,
+   NULL, NULL, List + 287, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'y', First + 11, NULL,
+  {List + 289,
+   List + 290, NULL, List + 291, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+};
+
+
+/******************************************************************************
+ *
+ *  Look for a match in the suffix list.
+ *  Return the pointer to the end of the new stem if there is a match.
+ *  Otherwise, return the pointer to the end of the original word.
+ *
+ *  Method:
+ *      * Search for the first character in the suffix list.
+ *      * If there is match, search for the rest of the suffix list.
+ *         
+ *****************************************************************************/
+
+static char *
+remove_ending (char *word, int w_len)
+{
+  static First_Char_Node *root = First + 6;	/* the root of binary tree is 'l' */
+  First_Char_Node *p_first;	/* Points to the first character      */
+  /* of the possible suffix             */
+  Ending_List *p_list;		/* Points to the List of Endings      */
+  char *suffix_start,		/* Points to start of possible suffix */
+   *stem_end,			/* Points to the end of possible stem */
+    ch1, ch2;
+  int s_len,			/* Possible stem length               */
+    e_offset,			/* Offset from the end to the start of */
+  /* possible suffix                    */
+    not_done = 1, cmp;
+
+  stem_end = word + w_len - 1;
+  if (w_len > PREDEFINED_SIZE)	/* find the start of suffix   */
+    suffix_start = word + w_len - MAX_ENDING_SIZE;
+  else
+    suffix_start = word + MIN_STEM_LENGTH;
+
+  ch1 = *suffix_start;
+  do				/* Search for the first character     */
+    {
+      p_first = root;
+      do
+	{
+	  ch2 = p_first->c;
+	  if (ch1 == ch2)
+	    break;
+	  else if (ch1 > ch2)
+	    p_first = p_first->right;
+	  else
+	    p_first = p_first->left;
+	}
+      while (p_first != NULL);
+
+      if (p_first != NULL)	/* Search for the rest */
+	{
+	  e_offset = stem_end - suffix_start;
+	  if ((p_list = p_first->ptr[e_offset]) != NULL)
+	    {
+	      for (;;)		/* no need to compare the first char  */
+		{
+		  cmp = strcmp (suffix_start + 1, p_list->ending + 1);
+		  if (cmp > 0)
+		    if (p_list->right_offset)
+		      p_list += p_list->right_offset;
+		    else
+		      break;
+		  else if (cmp < 0)
+		    if (p_list->left_offset)
+		      p_list += p_list->left_offset;
+		    else
+		      break;
+		  else
+		    {
+		      s_len = suffix_start - word;
+		      if (!p_list->cond ||
+			  (*p_list->cond) (s_len, suffix_start - 1))
+			{
+			  *suffix_start = EOS;
+			  stem_end = suffix_start - 1;
+			  not_done = 0;
+			}
+		      break;
+		    }
+		}
+	    }
+	}
+      suffix_start++;
+    }
+  while (not_done && ((ch1 = *suffix_start) != EOS));
+  return (stem_end);
+}
Index: /trunk/indexers/mgpp/lib/lovinstem.h
===================================================================
--- /trunk/indexers/mgpp/lib/lovinstem.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/lovinstem.h	(revision 3365)
@@ -0,0 +1,28 @@
+/**************************************************************************
+ *
+ * lovinstem.h -- Lovins' stemmer header
+ * Copyright (C) 1994  Linh Huynh, 
+ *                     (glued into library by tes@kbs.citri.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef LOVINSTEM_H
+#define LOVINSTEM_H
+
+void lovinstem (unsigned char *word);
+
+#endif
Index: /trunk/indexers/mgpp/lib/memlib.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/memlib.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/memlib.cpp	(revision 3365)
@@ -0,0 +1,41 @@
+/**************************************************************************
+ *
+ * memlib.c -- Malloc wrappers
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "memlib.h"
+
+/* Defined as strdup is not an ANSI function */
+char *
+my_strdup(const char *str)
+{
+  char *ret_str = new char[strlen(str)+1];
+  if (ret_str) return strcpy(ret_str, str);
+  else return (char*) 0;
+}
+
+
+Malloc_func Xmalloc = malloc;
+
+Realloc_func Xrealloc = realloc;
+
+Free_func Xfree = free;
+
+Strdup_func Xstrdup = my_strdup;
Index: /trunk/indexers/mgpp/lib/memlib.h
===================================================================
--- /trunk/indexers/mgpp/lib/memlib.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/memlib.h	(revision 3365)
@@ -0,0 +1,55 @@
+/**************************************************************************
+ *
+ * memlib.h -- Malloc wrappers
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_MEMLIB
+#define H_MEMLIB
+
+/* changed default behaviour - we now assume std memory. - john mcp. */
+/* #ifdef STD_MEM */
+#if !defined(NOT_STD_MEM)
+#include <string.h>
+#define Xmalloc malloc
+#define Xrealloc realloc
+#define Xfree free
+
+#if defined (__WIN32__)
+# define Xstrdup _strdup
+#else
+# define Xstrdup strdup
+#endif
+
+#else
+
+
+#include "sysfuncs.h"
+
+typedef void *(*Malloc_func) (size_t);
+typedef void *(*Realloc_func) (void *, size_t);
+typedef void (*Free_func) (void *);
+typedef char *(*Strdup_func) (const char *);
+
+extern Malloc_func Xmalloc;
+extern Realloc_func Xrealloc;
+extern Free_func Xfree;
+extern Strdup_func Xstrdup;
+#endif
+
+#endif
Index: /trunk/indexers/mgpp/lib/messages.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/messages.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/messages.cpp	(revision 3365)
@@ -0,0 +1,75 @@
+/**************************************************************************
+ *
+ * messages.c -- Message and error functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include <stdarg.h>
+#include "messages.h"
+
+char *msg_prefix = "";
+
+
+void VOLATILE 
+FatalError (int ExitCode, const char *fmt,...)
+{
+  char buf[1024];
+  char *s, *pfx;
+  va_list args;
+  va_start (args, fmt);
+  vsprintf (buf, fmt, args);
+  s = strrchr (buf, '\n');
+  if (!s || *(s + 1) != '\0')
+    strcat (buf, "\n");
+  pfx = strrchr (msg_prefix, '/');
+  pfx = pfx ? pfx + 1 : msg_prefix;
+  fprintf (stderr, "%s%s%s", pfx, *pfx ? " : " : "", buf);
+  exit (ExitCode);
+}
+
+
+
+/*
+ * This function writes messages to stderr. Due to the fact that I can't 
+ * guarantee that the fprintf call is monatomic I have to implement a 
+ * semaphore system.
+ */
+void 
+Message (const char *fmt,...)
+{
+  char buf[1024];
+  char *s, *pfx;
+  va_list args;
+
+#ifdef SILENT
+  return; /* stop those messages!!!! */
+#endif  
+
+  va_start (args, fmt);
+  vsprintf (buf, fmt, args);
+  s = strrchr (buf, '\n');
+  if (!s || *(s + 1) != '\0')
+    strcat (buf, "\n");
+
+  pfx = strrchr (msg_prefix, '/');
+  pfx = pfx ? pfx + 1 : msg_prefix;
+  fprintf (stderr, "%s%s%s", pfx, *pfx ? " : " : "", buf);
+
+}
Index: /trunk/indexers/mgpp/lib/messages.h
===================================================================
--- /trunk/indexers/mgpp/lib/messages.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/messages.h	(revision 3365)
@@ -0,0 +1,49 @@
+/**************************************************************************
+ *
+ * messages.h -- Message and error functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_MESSAGES
+#define H_MESSAGES
+
+#ifdef __GNUC__
+#define VOLATILE volatile
+#else
+#define VOLATILE
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+extern char *msg_prefix;
+
+void VOLATILE FatalError (int ExitCode, const char *fmt,...);
+void Message (const char *fmt,...);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif
+
+
+
+
Index: /trunk/indexers/mgpp/lib/mgheap.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/mgheap.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/mgheap.cpp	(revision 3365)
@@ -0,0 +1,148 @@
+/**************************************************************************
+ *
+ * mgheap.c -- Heap routines
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "mgheap.h"
+
+#define SWAP(s, a, b) 						\
+do {								\
+     register int __i;						\
+     register char *__a = (a), *__b = (b);			\
+     for (__i = (s); __i; __i--)				\
+       {							\
+         register char __t = *__a;				\
+	 *__a++ = *__b; *__b++ = __t;				\
+       }							\
+} while(0)
+
+
+
+#define COPY(s, dst, src) 					\
+do {								\
+     register int __i;						\
+     register char *__dst = (dst), *__src = (src);		\
+     for (__i = (s); __i; __i--)				\
+       *__dst++ = *__src++;					\
+} while(0)
+
+
+
+static void 
+heap_heapify (void *heap, int size, int num, int curr, heap_comp hc)
+{
+  register int child = curr * 2;
+  while (child <= num)
+    {
+      if (child < num && hc ((char *) heap + child * size,
+			     (char *) heap + child * size - size) < 0)
+	child++;
+      if (hc ((char *) heap + (curr - 1) * size, (char *) heap + (child - 1) * size) > 0)
+	{
+	  SWAP (size, (char *) heap + (curr - 1) * size,
+		(char *) heap + (child - 1) * size);
+	  curr = child;
+	  child = child * 2;
+	}
+      else
+	break;
+    }
+}
+
+static void 
+heap_pullup (void *heap, int size, int num, heap_comp hc)
+{
+  register int curr = num;
+  register int parent = curr >> 1;
+  while (parent &&
+	 hc ((char *) heap + (curr - 1) * size, (char *) heap + (parent - 1) * size) < 0)
+    {
+      SWAP (size, (char *) heap + (curr - 1) * size,
+	    (char *) heap + (parent - 1) * size);
+      curr = parent;
+      parent = curr >> 1;
+    }
+}
+
+/************************************************************************
+ *
+ * NOTE: If you choose to change the comparison function the first thing 
+ *       you do after changing the function is call heap_build.
+ *
+ */
+void 
+heap_build (void *heap, int size, int num, heap_comp hc)
+{
+  register int i;
+  for (i = num / 2; i > 0; i--)
+    heap_heapify (heap, size, num, i, hc);
+}
+
+
+
+/****************************************************
+ *
+ * NOTE : The heap must be built before heap_sort is called. 
+ *        This has the effect of reversing the order of the array.
+ *		e.g. if your comparison function is designed to pull the
+ *                   biggest thing off the heap first then the result of
+ *		     sorting with this function will be to put the bigest
+ *                   thing at the end of the array.
+ *
+ */
+void 
+heap_sort (void *heap, int size, int num, heap_comp hc)
+{
+  register int i;
+  for (i = num; i > 1; i--)
+    {
+      SWAP (size, (char *) heap, (char *) heap + (i - 1) * size);
+      heap_heapify (heap, size, i - 1, 1, hc);
+    }
+}
+
+
+void 
+heap_deletehead (void *heap, int size, int *num, heap_comp hc)
+{
+  (*num)--;
+  SWAP (size, (char *) heap, (char *) heap + *num * size);
+  heap_heapify (heap, size, *num, 1, hc);
+}
+
+
+void 
+heap_changedhead (void *heap, int size, int num, heap_comp hc)
+{
+  heap_heapify (heap, size, num, 1, hc);
+}
+
+
+/***********************************************************************
+ *
+ * This assumes that the item has been added to the end of the
+ * array that is the heap. But that num has not been changed. 
+ *
+ */
+void 
+heap_additem (void *heap, int size, int *num, heap_comp hc)
+{
+  (*num)++;
+  heap_pullup (heap, size, *num, hc);
+}
Index: /trunk/indexers/mgpp/lib/mgheap.h
===================================================================
--- /trunk/indexers/mgpp/lib/mgheap.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/mgheap.h	(revision 3365)
@@ -0,0 +1,82 @@
+/**************************************************************************
+ *
+ * mgheap.h -- Heap routines
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+/* Care should be taken when using these routines as NO error checking is
+ * done.
+ */
+
+
+
+/* The heap comparison routine should return 
+ * <0 : If a should be before b
+ *  0 : If a and b are equal
+ * >0 : If a should be after b
+ */
+typedef int (*heap_comp) (void *a, void *b);
+
+
+/************************************************************************
+ *
+ * NOTE: If you choose to change the comparison function the first thing 
+ *       you do after changing the function is call heap_build.
+ *
+ */
+void heap_build (void *heap, int size, int num, heap_comp hc);
+
+
+
+/************************************************************************
+ *
+ * NOTE : The heap must be built before heap_sort is called.
+ *        This has the effect of reversing the order of the array.
+ *		e.g. if your comparison function is designed to pull the
+ *                   biggest thing off the heap first then the result of
+ *		     sorting with this function will be to put the bigest
+ *                   thing at the end of the array.
+ *
+ */
+void heap_sort (void *heap, int size, int num, heap_comp hc);
+
+/***********************************************************************
+ *
+ * NOTE: After deleting the head the heap will be one item shorter.
+ *       The deleted item will be placed at the end of the heap.
+ *
+ */
+void heap_deletehead (void *heap, int size, int *num, heap_comp hc);
+
+
+/***********************************************************************
+ *
+ * If you change the "value" of the root node of the heap then you
+ * should call this to re-pritorize the heap.
+ *
+ */
+void heap_changedhead (void *heap, int size, int num, heap_comp hc);
+
+
+/***********************************************************************
+ *
+ * This assumes that the item has been added to the end of the
+ * array that is the heap. But that num has not been changed. 
+ *
+ */
+void heap_additem (void *heap, int size, int *num, heap_comp hc);
Index: /trunk/indexers/mgpp/lib/netorder.h
===================================================================
--- /trunk/indexers/mgpp/lib/netorder.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/netorder.h	(revision 3365)
@@ -0,0 +1,189 @@
+#ifndef NETORDER_H
+#define NETORDER_H
+
+#include "sysfuncs.h"
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef __WIN32__
+#include "win32in.h"
+#else
+# include <netinet/in.h>
+#endif
+
+#if defined (_BIG_ENDIAN) || (defined (BYTE_ORDER) && BYTE_ORDER == BIG_ENDIAN) || (defined (__BYTE_ORDER) && __BYTE_ORDER == __BIG_ENDIAN)
+# ifndef _BIG_ENDIAN
+#  define _BIG_ENDIAN
+# endif
+#endif
+
+#if defined (_LITTLE_ENDIAN) || (defined (BYTE_ORDER) && BYTE_ORDER == LITTLE_ENDIAN) || (defined (__BYTE_ORDER) && __BYTE_ORDER == __LITTLE_ENDIAN)
+# ifndef _LITTLE_ENDIAN
+#  define _LITTLE_ENDIAN
+# endif
+#endif
+
+#ifdef _LITTLE_ENDIAN
+
+/* double */
+#define HTOND(d)                                                              \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &tmph, (const void *) &d, sizeof(double) >> 1); \
+	     memcpy ((void *) &tmpl, (const void *) ((unsigned int) &d + (sizeof(double) >> 1)), sizeof (double) >> 1);   \
+	     tmph = htonl (tmph);                                             \
+	     tmpl = htonl (tmpl);                                             \
+	     memcpy ( (void *) &d, (const void *) &tmpl, sizeof (double) >> 1);                           \
+	     memcpy ((void *) ((unsigned int) &d + (sizeof(double) >> 1)), (const void *) &tmph, sizeof (double) >> 1);   \
+	}while(0)
+#define NTOHD(d)                                                              \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &tmph, (const void *) &d, sizeof(double) >> 1); \
+	     memcpy ((void *) &tmpl, (const void *) ((int) &d + (sizeof(double) >> 1)), sizeof (double) >> 1);   \
+	     tmph = ntohl (tmph);                                             \
+	     tmpl = ntohl (tmpl);                                             \
+	     memcpy((void *) &d, (const void *) &tmpl, sizeof (double) >> 1); \
+	     memcpy ((void *) ((int) &d + (sizeof(double) >> 1)), (const void *) &tmph, sizeof (double) >> 1);   \
+        }while(0)
+#define HTOND2(hd, nd)                                                        \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &tmph, (const void *) &hd, sizeof(double) >> 1);\
+	     memcpy ((void *) &tmpl, (const void *) ((unsigned int) &hd + (sizeof(double) >> 1)), sizeof (double) >> 1);  \
+	     tmph = htonl (tmph);                                             \
+	     tmpl = htonl (tmpl);                                             \
+	     memcpy((void *) &nd, (const void *) &tmpl, sizeof (double) >> 1);\
+	     memcpy ((void *) ((unsigned int) &nd + (sizeof(double) >> 1)), (void *) &tmph, sizeof (double) >> 1);  \
+	}while(0)
+#define NTOHD2(nd, hd)                                                        \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &nd, (void *) &tmph, sizeof(double) >> 1);      \
+	     memcpy ((void *) &tmpl, (const void *) &nd + (sizeof(double) >> 1), sizeof (double) >> 1);  \
+	     tmph = ntohl (tmph);                                             \
+	     tmpl = ntohl (tmpl);                                             \
+	     memcpy((void *) &hd, (const void *) &tmpl, sizeof (double) >> 1);\
+	     memcpy ((void *) &hd + (sizeof(double) >> 1), (const void *) &tmph, sizeof (double) >> 1);  \
+        }while(0)
+
+/* float */
+#define HTONF(f)                                                              \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(f), sizeof (float));     \
+             HTONUL (tmp);                                                    \
+	     memcpy ((void *) &(f), (const void *) &tmp, sizeof (float));     \
+	}while(0)
+#define NTOHF(f)                                                              \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(f), sizeof (float));     \
+	     NTOHUL (tmp);                                                    \
+	     memcpy ((void *) &(f), (const void *) &tmp, sizeof (float));     \
+	}while(0)
+#define HTONF2(hf, nf)                                                        \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(hf), sizeof (float));    \
+             HTONUL (tmp);                                                    \
+	     memcpy ((void *) &(nf), (const void *) &tmp, sizeof (float));    \
+	}while(0)
+#define NTOHF2(nf, hf)                                                        \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(nf), sizeof (float));    \
+	     NTOHUL (tmp);                                                    \
+	     memcpy ((void *) &(hf), (const void *) &tmp, sizeof (float));    \
+	}while(0)
+
+/* pointers */
+#define HTONP(p)          ((p) = (void *) htonl ((unsigned long) p))
+#define NTOHP(p)          ((p) = (void *) ntohl ((unsigned long) p))
+#define HTONP2(hp, np)    ((np) = (void *) htonl ((unsigned long) hp))
+#define NTOHP2(np, hp)    ((hp) = (void *) ntohl ((unsigned long) np))
+
+/* unsigned long */
+#define HTONUL(l)         ((l) = htonl((l)))
+#define NTOHUL(l)         ((l) = ntohl((l)))
+#define HTONUL2(hl, nl)   ((nl) = htonl ((hl)))
+#define NTOHUL2(nl, hl)   ((hl) = ntohl ((nl)))
+
+/* signed long */
+#define HTONSL(l)         ((l) = (long) htonl ((unsigned long) (l)))
+#define NTOHSL(l)         ((l) = (long) ntohl ((unsigned long) (l)))
+#define HTONSL2(hl, nl)   ((nl) = (long) htonl ((unsigned long) (hl)))
+#define NTOHSL2(nl, hl)   ((hl) = (long) ntohl ((unsigned long) (nl)))
+
+/* unsigned int */
+#define HTONUI(i)         ((i) = (unsigned int) htonl ((unsigned long) (i)))
+#define NTOHUI(i)         ((i) = (unsigned int) ntohl ((unsigned long) (i)))
+#define HTONUI2(hi, ni)   ((ni) = (unsigned int) htonl ((unsigned long) (hi)))
+#define NTOHUI2(ni, hi)   ((hi) = (unsigned int) ntohl ((unsigned long) (ni)))
+
+/* signed int */
+#define HTONSI(i)         ((i) = (int) htonl ((unsigned long) (i)))
+#define NTOHSI(i)         ((i) = (int) ntohl ((unsigned long) (i)))
+#define HTONSI2(hi, ni)   ((ni) = (int) htonl ((unsigned long) (hi)))
+#define NTOHSI2(ni, hi)   ((hi) = (int) ntohl ((unsigned long) (ni)))
+
+/* unsigned short */
+#define HTONUS(s)         ((s) = htons((s)))
+#define NTOHUS(s)         ((s) = ntohs((s)))
+#define HTONUS2(hs, ns)   ((ns) = htons((hs)))
+#define NTOHUS2(ns, hs)   ((hs) = ntohs((ns)))
+
+#else   /* _BIG_ENDIAN */
+
+/* double */
+#define HTOND(d)          (d)
+#define NTOHD(d)          (d)
+#define HTOND2(hd, nd)    ((hd) = (nd))
+#define NTOHD2(nd, hd)    ((nd) = (hd))
+
+/* float */
+#define HTONF(f)          (f)
+#define NTOHF(f)          (f)
+#define HTONF2(hf, nf)    ((nf) = (hf))
+#define NTOHF2(nf, hf)    ((hf) = (nf))
+
+/* pointers */
+#define HTONP(p)          (p)
+#define NTOHP(p)          (p)
+#define HTONP2(hp, np)    ((np) = (hp))
+#define NTOHP2(np, hp)    ((hp) = (np))
+
+/* unsigned long */
+#define HTONUL(l)         (l)
+#define NTOHUL(l)         (l)
+#define HTONUL2(hl, nl)   ((nl) = (hl))
+#define NTOHUL2(nl, hl)   ((hl) = (nl))
+
+/* signed long */
+#define HTONSL(l)         (l)
+#define NTOHSL(l)         (l)
+#define HTONSL2(hl, nl)   ((nl) = (hl))
+#define NTOHSL2(nl, hl)   ((hl) = (nl))
+
+/* unsigned int */
+#define HTONUI(i)         (i)
+#define NTOHUI(i)         (i)
+#define HTONUI2(hi, ni)   ((ni) = (hi))
+#define NTOHUI2(ni, hi)   ((hi) = (ni))
+
+/* signed int */
+#define HTONSI(i)         (i)
+#define NTOHSI(i)         (i)
+#define HTONSI2(hi, ni)   ((ni) = (hi))
+#define NTOHSI2(ni, hi)   ((hi) = (ni))
+
+/* unsigned short */
+#define HTONUS(s)         (s)
+#define NTOHUS(s)         (s)
+#define HTONUS2(hs, ns)   ((ns) = (hs))
+#define NTOHUS2(ns, hs)   ((hs) = (ns))
+
+
+
+#endif
+
+#endif /* netorder.h */
Index: /trunk/indexers/mgpp/lib/non_ansi.h
===================================================================
--- /trunk/indexers/mgpp/lib/non_ansi.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/non_ansi.h	(revision 3365)
@@ -0,0 +1,24 @@
+#if !defined(_NON_ANSI_H)
+#define _NON_ANSI_H 1
+
+#include "config.h" 
+# if !defined(__STDC__)
+
+#include <stdlib.h>  /* for FILE def */
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+int fileno(FILE *);
+int ftruncate(int, long);
+int pclose(FILE *);
+FILE *popen(const char *command, const char *type);
+int getpagesize();
+int strcasecmp(const char *, const char *);
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* !defined STDC, ... */
+#endif
Index: /trunk/indexers/mgpp/lib/pathmax.h
===================================================================
--- /trunk/indexers/mgpp/lib/pathmax.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/pathmax.h	(revision 3365)
@@ -0,0 +1,53 @@
+/* Define PATH_MAX somehow.  Requires sys/types.h.
+   Copyright (C) 1992 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifndef _PATHMAX_H
+#define _PATHMAX_H
+
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+/* Non-POSIX BSD systems might have gcc's limits.h, which doesn't define
+   PATH_MAX but might cause redefinition warnings when sys/param.h is
+   later included (as on MORE/BSD 4.3).  */
+#if defined(_POSIX_VERSION) || (defined(HAVE_LIMITS_H) && !defined(__GNUC__))
+#include <limits.h>
+#endif
+
+#ifndef _POSIX_PATH_MAX
+#define _POSIX_PATH_MAX 255
+#endif
+
+#if !defined(PATH_MAX) && defined(_PC_PATH_MAX)
+#define PATH_MAX (pathconf ("/", _PC_PATH_MAX) < 1 ? 1024 : pathconf ("/", _PC_PATH_MAX))
+#endif
+
+/* Don't include sys/param.h if it already has been.  */
+#if defined(HAVE_SYS_PARAM_H) && !defined(PATH_MAX) && !defined(MAXPATHLEN)
+#include <sys/param.h>
+#endif
+
+#if !defined(PATH_MAX) && defined(MAXPATHLEN)
+#define PATH_MAX MAXPATHLEN
+#endif
+
+#ifndef PATH_MAX
+#define PATH_MAX _POSIX_PATH_MAX
+#endif
+
+#endif /* _PATHMAX_H */
Index: /trunk/indexers/mgpp/lib/perf_hash.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/perf_hash.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/perf_hash.cpp	(revision 3365)
@@ -0,0 +1,1074 @@
+/**************************************************************************
+ *
+ * perf_hash.c -- Perfect hashing functions
+ * Copyright (C) 1992  Bohdan S. Majewski
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ * 1994 Neil Sharman
+ *      Small modifications were made to the interface so that it fitted in
+ *      with mg.
+ *
+ **************************************************************************/
+
+#define STRUCT
+
+#include <time.h>  /* for call to time () in SEED_RANDOM() */
+#include "sysfuncs.h"
+
+#include "local_strings.h"
+#include "memlib.h"
+#include "messages.h"
+#include "perf_hash.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#define FALSE 0
+#define TRUE  1
+
+#define STATIC 0
+
+/* Random Number stuff */
+
+long irandm (long is[2]);  /* from random.c */
+
+static long seed[] = {0, 0};
+#define RANDOM() irandm(seed)
+#define SEED_RANDOM(the_seed) do{ seed[0] = the_seed; }while(0)
+
+/* Max number of keys - must be a power of 2 minus 1 */
+static int MAX_M;
+
+
+static int MAX_CH;
+
+
+/* the width of the mapping tables  */
+static int MAX_L;
+
+
+
+#define c     1.23
+static int MAX_N;
+
+
+
+
+static u_char *translate;
+
+#define MEMBER 0
+#define ERASED 1
+
+
+
+/* direction constants for the edge oriented 3-graph representation */
+#define A              0
+#define B        (MAX_M)
+#define C          (2*B)
+
+/* NODEa, NODEb, NODEc, FIRST and NEXT are used to represent a 3-graph */
+static int *NODEa, *NODEb, *NODEc;
+static int *FIRST;
+static int *NEXT;
+
+
+#define norm(e) ((e) % MAX_M)
+/* in general should be ((e) % MAX_M) but MAX_M = 2^k - 1 */
+
+
+static char *mk;		/* for marking edges/vertices as removed */
+static int *g;			/* array g defines a minimal perfect hash function */
+
+
+
+/* stack of edges, created during the independency */
+/* test, used in the assignment step               */
+static struct
+  {
+    int sp;
+    int *st;
+  }
+S;
+#define push(s, e) s.st[s.sp++] = (e)
+#define pop(s, e)  e = s.st[--s.sp]
+
+#ifndef STRUCT
+static long **tb0, **tb1, **tb2;
+#else
+struct tb_entry **tb;
+#endif
+
+
+static int 
+unique (int v)
+{				/* checks if vertex v belongs to only one edge */
+  return (FIRST[v] != 0 && NEXT[FIRST[v]] == 0) ? FIRST[v] : 0;
+}				/*unique */
+
+
+
+
+
+
+
+
+static void 
+_delete (int v, int e)
+{				/* deletes edge e from list of v */
+
+  int b;
+
+  b = FIRST[v];
+  assert (norm (b) != 0);
+  if (norm (b) == e)
+    FIRST[v] = NEXT[FIRST[v]];
+  else
+    {
+      while (norm (NEXT[b]) != e)
+	{
+	  b = NEXT[b];
+	  assert (norm (b) != 0);
+	}
+      NEXT[b] = NEXT[NEXT[b]];
+    }
+}				/*delete */
+
+
+
+
+
+
+#if 0
+/* recursively removes the edges of a 3-graph which have at least */
+/* one unique vertex. The removed edges are placed on stack S     */
+static void 
+ph_remove (int e, int v)
+{
+  int b;
+  mk[e] = ERASED;
+  push (S, e);
+
+  if (NODEa[e] != v)
+    _delete (NODEa[e], e);
+  if (NODEb[e] != v)
+    _delete (NODEb[e], e);
+  if (NODEc[e] != v)
+    _delete (NODEc[e], e);
+
+  if (NODEa[e] != v && mk[b = norm (unique (NODEa[e]))] == MEMBER)
+    ph_remove (b, NODEa[e]);
+  if (NODEb[e] != v && mk[b = norm (unique (NODEb[e]))] == MEMBER)
+    ph_remove (b, NODEb[e]);
+  if (NODEc[e] != v && mk[b = norm (unique (NODEc[e]))] == MEMBER)
+    ph_remove (b, NODEc[e]);
+}
+#else
+
+
+static int StackPos, StackMax;
+static struct Stack
+  {
+    int e, v, m;
+  }
+ *Stack = NULL;
+
+#define StackAdd(_e, _v, _m)						\
+  do {									\
+    if (StackPos == StackMax)						\
+      {									\
+        StackMax <<= 1;							\
+	if (!(Stack = (struct Stack *)Xrealloc(Stack, sizeof(*Stack) * StackMax)))	\
+	  return(-1);							\
+      }									\
+    Stack[StackPos].e = (_e);						\
+    Stack[StackPos].v = (_v);						\
+    Stack[StackPos++].m = (_m);						\
+  } while(0)
+
+
+static int 
+ph_remove (int e, int v)
+{
+  if (!Stack)
+    {
+      StackMax = 256;
+      if (!(Stack = (struct Stack *) Xmalloc (sizeof (*Stack) * StackMax)))
+	return (-1);
+      StackPos = 0;
+    }
+  Stack[StackPos].e = e;
+  Stack[StackPos].v = v;
+  Stack[StackPos].m = 0;
+  StackPos++;
+  while (StackPos)
+    {
+      int m, b;
+      StackPos--;
+      e = Stack[StackPos].e;
+      v = Stack[StackPos].v;
+      m = Stack[StackPos].m;
+
+
+      switch (m)
+	{
+	case 0:
+	  {
+	    mk[e] = ERASED;
+	    push (S, e);
+
+	    if (NODEa[e] != v)
+	      _delete (NODEa[e], e);
+	    if (NODEb[e] != v)
+	      _delete (NODEb[e], e);
+	    if (NODEc[e] != v)
+	      _delete (NODEc[e], e);
+
+	    if (NODEa[e] != v && mk[b = norm (unique (NODEa[e]))] == MEMBER)
+	      {
+		Stack[StackPos++].m = 1;
+		StackAdd (b, NODEa[e], 0);
+		break;
+	      }
+	  }
+	case 1:
+	  {
+	    if (NODEb[e] != v && mk[b = norm (unique (NODEb[e]))] == MEMBER)
+	      {
+		Stack[StackPos++].m = 2;
+		StackAdd (b, NODEb[e], 0);
+		break;
+	      }
+	  }
+	case 2:
+	  if (NODEc[e] != v && mk[b = norm (unique (NODEc[e]))] == MEMBER)
+	    StackAdd (b, NODEc[e], 0);
+	}
+    }
+  return (0);
+}
+#endif
+
+
+
+
+
+/* returns TRUE if and only if set of edges created in the tree_builder */
+/* procedure form a 3-graph with independent edges. Edges of a 3-graph  */
+/* are independent iff repeated deletion of edges containing vertices   */
+/* of degree 1 results in a graph with no edges in it.                  */
+static int 
+acyclic (int m, int n)
+{
+
+  int v, e;
+  /* build list of edges for each node */
+  for (v = 1; v <= n; FIRST[v++] = 0);
+  for (e = 1; e <= m; e++)
+    {
+      NEXT[A + e] = FIRST[NODEa[e]];
+      FIRST[NODEa[e]] = A + e;
+      NEXT[B + e] = FIRST[NODEb[e]];
+      FIRST[NODEb[e]] = B + e;
+      NEXT[C + e] = FIRST[NODEc[e]];
+      FIRST[NODEc[e]] = C + e;
+
+      /* mark edge e as belonging to the hypergraph */
+      mk[e] = MEMBER;
+    }
+
+
+  S.sp = 0;			/* empty the stack */
+  mk[0] = ERASED;		/* a sentinel for the test below */
+  for (v = 1; v <= n; v++)
+    {
+      if (mk[e = norm (unique (v))] == MEMBER)
+	ph_remove (e, v);
+    }
+
+  /* check if any edges were left in the graph */
+  return S.sp == m;
+}				/*acyclic */
+
+
+
+
+
+
+
+
+/* repeatedly maps words into edges of a 3-graph until the edges */
+/* are independent.                                              */
+static int 
+tree_builder (int num, u_char ** keys, int *n)
+{
+
+  int e, k = 0;
+  int i, j;
+  u_char *w;
+
+  do
+    {
+      *n = (int) (num * c + 1);
+      /* initialize tables of random integers */
+      for (i = 0; i < MAX_CH; i++)
+	for (j = 0; j < MAX_L; j++)
+	  {
+#ifndef STRUCT
+	    tb0[i][j] = RANDOM () % *n;
+	    tb1[i][j] = RANDOM () % *n;
+	    tb2[i][j] = RANDOM () % *n;
+#else
+	    struct tb_entry *tbp = &tb[i][j];
+	    tbp->tb0 = RANDOM () % *n;
+	    tbp->tb1 = RANDOM () % *n;
+	    tbp->tb2 = RANDOM () % *n;
+#endif
+	  }
+
+      /* compute an edge (NODEa[e], NODEb[e], NODEc[e]) for each word */
+      for (e = 1; e <= num; e++)
+	{
+	  /*generate an edge corresponding to the ith word */
+	  int x = 0, y = 0, z = 0, l;
+	  w = keys[e - 1];
+
+	  l = *w++;
+	  j = (l - 1) % MAX_L;
+
+	  do
+	    {
+#ifndef STRUCT
+	      x += tb0[translate[*w]][j];
+	      y += tb1[translate[*w]][j];
+	      z += tb2[translate[*w]][j];
+#else
+	      struct tb_entry *tbp = &tb[translate[*w]][j];
+	      x += tbp->tb0;
+	      y += tbp->tb1;
+	      z += tbp->tb2;
+#endif
+	      if (++j >= MAX_L)
+		j = 0;
+	      w++;
+	    }
+	  while (--l);
+	  x = (x % *n) + 1;
+	  y = (y % *n) + 1;
+	  z = (z % *n) + 1;
+	  if (y == x && ++y > *n)
+	    y = 1;
+	  if (z == x && ++z > *n)
+	    z = 1;
+	  if (z == y)
+	    {
+	      if (++z > *n)
+		z = 1;
+	      if (z == x && ++z > *n)
+		z = 1;
+	    }
+
+	  NODEa[e] = x;
+	  NODEb[e] = y;
+	  NODEc[e] = z;
+	}
+      if (++k > 50)		/* make sure we will not wait for ever */
+	FatalError (1, "Unable to generate the perfect hash function. "
+		    "This is probably because there aren't enough words");
+    }
+  while (!acyclic (num, *n));
+  return k;
+}				/*tree_builder */
+
+
+
+
+
+
+
+
+
+/* calculates values of the g function/array so that i-th word is */
+/* placed at the (i-1)st position of a hash table.                */
+static void 
+assign (int n)
+{
+
+  int v, w = 0, e;
+
+  /* mark all the vertices of the 3-graph as unassigned */
+  for (v = 1; v <= n; mk[v++] = FALSE);
+
+  while (S.sp > 0)
+    {
+      int _xor = 0;
+      pop (S, e);
+      v = NODEa[e];
+      if (mk[v])
+	_xor ^= g[v];
+      else
+	{
+	  g[w = v] = 0;
+	  mk[v] = TRUE;
+	}
+      v = NODEb[e];
+      if (mk[v])
+	_xor ^= g[v];
+      else
+	{
+	  g[w = v] = 0;
+	  mk[v] = TRUE;
+	}
+      v = NODEc[e];
+      if (mk[v])
+	_xor ^= g[v];
+      else
+	{
+	  g[w = v] = 0;
+	  mk[v] = TRUE;
+	}
+      g[w] = (e - 1) ^ _xor;
+    }
+}				/*assign */
+
+
+
+
+
+
+
+
+
+/*check returns 0 if no error in mphf is detected, else returns error code */
+static int 
+check (int m)
+{
+
+  int e, err_code = 0;
+
+  /*for each word compute its place in hash table and check if correct */
+  for (e = 1; e <= m; e++)
+    {
+      if ((e - 1) != (g[NODEa[e]] ^ g[NODEb[e]] ^ g[NODEc[e]]))
+	{
+	  (void) fprintf (stderr, "Error for %dth word detected.\n", e);
+	  err_code = e;
+	}
+    }
+  return err_code;
+}				/*check */
+
+
+
+
+
+
+
+#if 0
+void 
+outputh (int m, int n)
+{
+
+  int i, j, w = (int) ceil (log10 ((double) n + 1.0)) + 1;
+
+#if 0
+  char *PRG =
+  "int h(char *s)\n"
+  "{\n"
+  "  int u, v, w, i;\n"
+  "i = (strlen(s) - 1) % _HXL;\n"
+  "  u = v = w = 0;\n"
+  "  while(*s)\n"
+  "    {\n"
+  "      u += mt1[(*s) - _HFC][i];\n"
+  "      v += mt2[(*s) - _HFC][i];\n"
+  "      w += mt3[(*s++) - _HFC][i];\n"
+  "      if(++i >= _HXL) i = 0;\n"
+  "    }\n"
+  "  u %= _HFN; v %= _HFN; w %= _HFN;\n"
+  "  if(u == v && ++v >= _HFN) v = 0;\n"
+  "  if(w == u && ++w >= _HFN) w = 0;\n"
+  "  if(w == v && ++w >= _HFN) w = 0;\n"
+  "  if(w == u && ++w >= _HFN) w = 0;\n"
+  "  return (g[u] ^ g[v] ^ g[w]) % _HFM;\n"
+  "};\n\n";
+#endif
+
+  char *PRG =
+  "int h(char *s)\n"
+  "{\n"
+  "  int u, v, w, i;\n"
+  "  i = (strlen(s) - 1) % _HXL;\n"
+  "  u = v = w = 0;\n"
+  "  while(*s)\n"
+  "    {\n"
+  "      if((u += mt1[(*s) - _HFC][i]) >= _HFN) u -= _HFN;\n"
+  "      if((v += mt2[(*s) - _HFC][i]) >= _HFN) v -= _HFN;\n"
+  "      if((w += mt3[(*s++) - _HFC][i]) >= _HFN) w -= _HFN;\n"
+  "      if(++i >= _HXL) i = 0;\n"
+  "    }\n"
+  "  if(u == v && ++v >= _HFN) v = 0;\n"
+  "  if(w == v) \n"
+  "    {\n"
+  "      if(++w >= _HFN) w = 0;\n"
+  "      if(w == u && ++w >= _HFN) w = 0;\n"
+  "    }\n"
+  "  else\n"
+  "    if(w == u)\n"
+  "      {\n"
+  "	     if(++w >= _HFN) w = 0;\n"
+  "	     if(w == v && ++w >= _HFN) w = 0;\n"
+  "      }\n"
+  "  return ((g[u] ^ g[v] ^ g[w]) > _HFM)? 0 : (g[u] ^ g[v] ^ g[w]);\n"
+  "}\n\n";
+
+  printf ("#define _HFN %d\n#define _HFC %d\n", n, fc);
+  printf ("#define _HFM %d\n#define _HXL %d\n", m, MAX_L);
+  printf ("#define _HFA %d\n\nstatic int g[_HFN] = {\n", lc - fc + 1);
+  for (i = 1; i < n; i++)
+    {
+      if (i % 8 == 0)
+	putchar ('\n');
+      printf ("%5d, ", g[i]);
+    }
+  printf ("%5d};\n\n", g[n]);
+
+  printf ("static int mt1[_HFA][_HXL] = {\n");
+  for (j = fc; j < lc; j++)
+    {
+      printf ("{");
+      for (i = 0; i < MAX_L - 1; i++)
+	printf ("%*d,", w, tb0[j][i]);
+      printf ("%*d},\n", w, tb0[j][MAX_L - 1]);
+    }
+  printf ("{");
+  for (i = 0; i < MAX_L - 1; i++)
+    printf ("%*d,", w, tb0[lc][i]);
+  printf ("%*d}\n", w, tb0[lc][MAX_L - 1]);
+  printf ("};\n\n");
+
+  printf ("static int mt2[_HFA][_HXL] = {\n");
+  for (j = fc; j < lc; j++)
+    {
+      printf ("{");
+      for (i = 0; i < MAX_L - 1; i++)
+	printf ("%*d,", w, tb1[j][i]);
+      printf ("%*d},\n", w, tb1[j][MAX_L - 1]);
+    }
+  printf ("{");
+  for (i = 0; i < MAX_L - 1; i++)
+    printf ("%*d,", w, tb1[lc][i]);
+  printf ("%*d}\n", w, tb1[lc][MAX_L - 1]);
+  printf ("};\n\n");
+
+  printf ("static int mt3[_HFA][_HXL] = {\n");
+  for (j = fc; j < lc; j++)
+    {
+      printf ("{");
+      for (i = 0; i < MAX_L - 1; i++)
+	printf ("%*d,", w, tb2[j][i]);
+      printf ("%*d},\n", w, tb2[j][MAX_L - 1]);
+    }
+  printf ("{");
+  for (i = 0; i < MAX_L - 1; i++)
+    printf ("%*d,", w, tb2[lc][i]);
+  printf ("%*d}\n", w, tb2[lc][MAX_L - 1]);
+  printf ("};\n\n");
+
+  printf ("%s", PRG);
+}				/*outputh */
+#endif
+
+
+
+
+
+
+
+static void 
+make_trans_func (int num, u_char ** keys)
+{
+  int i, j;
+
+  memset (translate, '\0', sizeof (translate));
+
+  MAX_L = 1;
+  for (i = 0; i < num; i++)
+    {
+      unsigned len = keys[i][0];
+      u_char *s = keys[i] + 1;
+      for (; len; len--, s++)
+	translate[*s] = 1;
+      if (i)
+	{
+	  int l;
+	  l = prefixlen (keys[i - 1], keys[i]);
+	  if (l + 1 > MAX_L)
+	    MAX_L = l + 1;
+	}
+    }
+  j = 0;
+  for (i = 0; i < 256; i++)
+    if (translate[i])
+      translate[i] = j++;
+  MAX_CH = j;
+}
+
+
+static void 
+temp_free (void)
+{
+  if (NODEa)
+    Xfree (NODEa);
+  if (NODEb)
+    Xfree (NODEb);
+  if (NODEc)
+    Xfree (NODEc);
+  if (FIRST)
+    Xfree (FIRST);
+  if (NEXT)
+    Xfree (NEXT);
+  if (S.st)
+    Xfree (S.st);
+  if (mk)
+    Xfree (mk);
+}
+
+static void 
+all_free (void)
+{
+  int i;
+  temp_free ();
+  if (translate)
+    Xfree (translate);
+  if (g)
+    Xfree (g);
+#ifndef STRUCT
+  for (i = 0; i < MAX_CH; i++)
+    {
+      if (tb0 && tb0[i])
+	Xfree (tb0[i]);
+      if (tb1 && tb1[i])
+	Xfree (tb1[i]);
+      if (tb2 && tb2[i])
+	Xfree (tb2[i]);
+    }
+  if (tb0)
+    Xfree (tb0);
+  if (tb1)
+    Xfree (tb1);
+  if (tb2)
+    Xfree (tb2);
+#else
+  for (i = 0; i < MAX_CH; i++)
+    if (tb && tb[i])
+      Xfree (tb[i]);
+  if (tb)
+    Xfree (tb);
+#endif
+}
+
+static int 
+allocate_memory (void)
+{
+  int i, ok = 1;
+  NODEa = (int *) Xmalloc (sizeof (int) * (MAX_M + 1));
+  NODEb = (int *)Xmalloc (sizeof (int) * (MAX_M + 1));
+  NODEc = (int *)Xmalloc (sizeof (int) * (MAX_M + 1));
+  FIRST = (int *)Xmalloc (sizeof (int) * (MAX_N + 1));
+  NEXT = (int *)Xmalloc (sizeof (int) * (MAX_M + 1) * 3);
+  S.st = (int *)Xmalloc (sizeof (int) * MAX_M);
+  mk = (char *)Xmalloc (sizeof (char) * (MAX_N + 1));
+  g = (int *)Xmalloc (sizeof (int) * (MAX_N + 1));
+#ifndef STRUCT
+  tb0 = Xmalloc (sizeof (int *) * MAX_CH);
+  tb1 = Xmalloc (sizeof (int *) * MAX_CH);
+  tb2 = Xmalloc (sizeof (int *) * MAX_CH);
+  for (i = 0; i < MAX_CH; i++)
+    {
+      if (tb0)
+	if (!(tb0[i] = Xmalloc (sizeof (long) * MAX_L)))
+	    ok = 0;
+      if (tb1)
+	if (!(tb1[i] = Xmalloc (sizeof (long) * MAX_L)))
+	    ok = 0;
+      if (tb2)
+	if (!(tb2[i] = Xmalloc (sizeof (long) * MAX_L)))
+	    ok = 0;
+    }
+
+  if (!(NODEa && NODEb && NODEc && FIRST && NEXT && S.st && mk && g &&
+	tb0 && tb1 && tb2 && ok))
+    {
+      all_free ();
+      return (-1);
+    }
+#else
+  tb = (tb_entry **)Xmalloc (sizeof (struct tb_entry *) * MAX_CH);
+  for (i = 0; i < MAX_CH; i++)
+    if (tb)
+      if (!(tb[i] = (tb_entry *)Xmalloc (sizeof (struct tb_entry) * MAX_L)))
+	  ok = 0;
+
+  if (!(NODEa && NODEb && NODEc && FIRST && NEXT && S.st && mk && g &&
+	tb && ok))
+    {
+      all_free ();
+      return (-1);
+    }
+#endif
+  return (0);
+}
+
+
+perf_hash_data *
+gen_hash_func (int num, u_char ** keys, int r)
+{
+  int n;
+  perf_hash_data *phd;
+  translate = (u_char *)Xmalloc (sizeof (u_char) * 256);
+  memset(translate, '\0', sizeof (u_char) * 256);  /* [RPAP - Feb 97: WIN32 Port] */
+  make_trans_func (num, keys);
+
+  if (r <= 0)
+    SEED_RANDOM ((long) time ((time_t *) NULL));
+  else
+    SEED_RANDOM (r);
+
+  MAX_M = num + 1;
+
+  MAX_N = MAX_M + MAX_M / 4;
+
+  if (allocate_memory () == -1)
+    return (NULL);
+
+  /* construct an "acyclic" hypergraph */
+  tree_builder (num, keys, &n);
+  /* compute function g */
+  assign (n);
+
+  if (check (num) != 0)
+    FatalError (1, "An error has been detected in the mphf.");
+  temp_free ();
+
+  if (!(phd = (perf_hash_data *)Xmalloc (sizeof (perf_hash_data))))
+    return (NULL);
+
+  memcpy ((void *) g, (const void *) &g[1], n * sizeof (int));
+
+  phd->MAX_L = MAX_L;
+  phd->MAX_N = n;
+  phd->MAX_M = num;
+  phd->MAX_CH = MAX_CH;
+  phd->translate = translate;
+  phd->g = g;
+#ifndef STRUCT
+  phd->tb0 = tb0;
+  phd->tb1 = tb1;
+  phd->tb2 = tb2;
+#else
+  phd->tb = tb;
+#endif
+  return (phd);
+}
+
+int 
+write_perf_hash_data (FILE * f, perf_hash_data * phd)
+{
+  int tot, i;
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONSI(phd->MAX_L);
+  HTONSI(phd->MAX_N);
+  HTONSI(phd->MAX_M);
+  HTONSI(phd->MAX_CH);
+
+  tot = fwrite ((char *) &phd->MAX_L, sizeof (int), 1, f) * sizeof (int);
+  tot += fwrite ((char *) &phd->MAX_N, sizeof (int), 1, f) * sizeof (int);
+  tot += fwrite ((char *) &phd->MAX_M, sizeof (int), 1, f) * sizeof (int);
+  tot += fwrite ((char *) &phd->MAX_CH, sizeof (int), 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(phd->MAX_L);
+  NTOHSI(phd->MAX_N);
+  NTOHSI(phd->MAX_M);
+  NTOHSI(phd->MAX_CH);
+
+  tot += fwrite ((char *) phd->translate, sizeof (u_char), 256, f);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    HTONSI(phd->g[i]);
+
+  tot += fwrite ((char *) phd->g, sizeof (int), phd->MAX_N + 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    NTOHSI(phd->g[i]);
+
+#ifndef STRUCT
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      /* [RPAP - Jan 97: Endian Ordering] */
+      int j;
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  HTONSI(phd->tb0[i][j]);
+	  HTONSI(phd->tb1[i][j]);
+	  HTONSI(phd->tb2[i][j]);
+	}
+
+      tot += fwrite ((char *) phd->tb0[i], sizeof (int), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fwrite ((char *) phd->tb1[i], sizeof (int), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fwrite ((char *) phd->tb2[i], sizeof (int), phd->MAX_L, f) *
+        sizeof (int);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSI(phd->tb0[i][j]);
+	  NTOHSI(phd->tb1[i][j]);
+	  NTOHSI(phd->tb2[i][j]);
+	}
+    }
+#else
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      /* [RPAP - Jan 97: Endian Ordering] */
+      int j;
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  HTONSL(phd->tb[i][j].tb0);
+	  HTONSL(phd->tb[i][j].tb1);
+	  HTONSL(phd->tb[i][j].tb2);
+	}
+
+      tot += fwrite ((char *) phd->tb[i], sizeof (struct tb_entry), phd->MAX_L, f) *
+	sizeof (struct tb_entry);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSL(phd->tb[i][j].tb0);
+	  NTOHSL(phd->tb[i][j].tb1);
+	  NTOHSL(phd->tb[i][j].tb2);
+	}
+    }
+#endif
+  return (tot);
+}
+
+
+perf_hash_data *
+read_perf_hash_data (FILE * f)
+{
+  perf_hash_data *phd;
+  int i, tot, ok = 1;
+  if (!(phd = (perf_hash_data *)Xmalloc (sizeof (perf_hash_data))))
+    return (NULL);
+  tot = fread ((char *) &phd->MAX_L, sizeof (int), 1, f) * sizeof (int);
+  tot += fread ((char *) &phd->MAX_N, sizeof (int), 1, f) * sizeof (int);
+  tot += fread ((char *) &phd->MAX_M, sizeof (int), 1, f) * sizeof (int);
+  tot += fread ((char *) &phd->MAX_CH, sizeof (int), 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(phd->MAX_L);
+  NTOHSI(phd->MAX_N);
+  NTOHSI(phd->MAX_M);
+  NTOHSI(phd->MAX_CH);
+
+  if (tot != 4 * sizeof (int))
+      return (NULL);
+  phd->translate = (u_char *)Xmalloc (sizeof (u_char) * 256);
+  phd->g = (int *)Xmalloc (sizeof (int) * (phd->MAX_N + 1));
+#ifndef STRUCT
+  phd->tb0 = Xmalloc (sizeof (int *) * phd->MAX_CH);
+  phd->tb1 = Xmalloc (sizeof (int *) * phd->MAX_CH);
+  phd->tb2 = Xmalloc (sizeof (int *) * phd->MAX_CH);
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      if (phd->tb0)
+	if (!(phd->tb0[i] = Xmalloc (sizeof (long) * phd->MAX_L)))
+	    ok = 0;
+      if (phd->tb1)
+	if (!(phd->tb1[i] = Xmalloc (sizeof (long) * phd->MAX_L)))
+	    ok = 0;
+      if (phd->tb2)
+	if (!(phd->tb2[i] = Xmalloc (sizeof (long) * phd->MAX_L)))
+	    ok = 0;
+    }
+  if (!(phd->translate && phd->g && phd->tb0 && phd->tb1 && phd->tb2 && ok))
+    {
+      if (phd->translate)
+	Xfree (phd->translate);
+      if (phd->g)
+	Xfree (phd->g);
+      for (i = 0; i < MAX_CH; i++)
+	{
+	  if (phd->tb0 && phd->tb0[i])
+	    Xfree (phd->tb0[i]);
+	  if (phd->tb1 && phd->tb1[i])
+	    Xfree (phd->tb1[i]);
+	  if (phd->tb2 && phd->tb2[i])
+	    Xfree (phd->tb2[i]);
+	}
+      if (phd->tb0)
+	Xfree (phd->tb0);
+      if (phd->tb1)
+	Xfree (phd->tb1);
+      if (phd->tb2)
+	Xfree (phd->tb2);
+      Xfree (phd);
+      return (NULL);
+    }
+  tot += fread ((char *) phd->translate, sizeof (u_char), 256, f);
+  tot += fread ((char *) phd->g, sizeof (int), phd->MAX_N + 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    NTOHSI(phd->g[i]);
+
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      int j;
+
+      tot += fread ((char *) phd->tb0[i], sizeof (long), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fread ((char *) phd->tb1[i], sizeof (long), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fread ((char *) phd->tb2[i], sizeof (long), phd->MAX_L, f) *
+        sizeof (int);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSI(phd->tb0[i][j]);
+	  NTOHSI(phd->tb1[i][j]);
+	  NTOHSI(phd->tb2[i][j]);
+	}
+    }
+#else
+  phd->tb = (struct tb_entry **)Xmalloc (sizeof (struct tb_entry *) * phd->MAX_CH);
+  for (i = 0; i < phd->MAX_CH; i++)
+    if (phd->tb)
+      if (!(phd->tb[i] = (struct tb_entry *)Xmalloc (sizeof (struct tb_entry) * phd->MAX_L)))
+	  ok = 0;
+  if (!(phd->translate && phd->g && phd->tb && ok))
+    {
+      if (phd->translate)
+	Xfree (phd->translate);
+      if (phd->g)
+	Xfree (phd->g);
+      for (i = 0; i < MAX_CH; i++)
+	if (phd->tb && phd->tb[i])
+	  Xfree (phd->tb[i]);
+      if (phd->tb)
+	Xfree (phd->tb);
+      Xfree (phd);
+      return (NULL);
+    }
+  tot += fread ((char *) phd->translate, sizeof (u_char), 256, f);
+  tot += fread ((char *) phd->g, sizeof (int), phd->MAX_N + 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    NTOHSI(phd->g[i]);
+
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      int j;
+
+      tot += fread ((char *) phd->tb[i], sizeof (struct tb_entry), phd->MAX_L, f) *
+	sizeof (struct tb_entry);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSL(phd->tb[i][j].tb0);
+	  NTOHSL(phd->tb[i][j].tb1);
+	  NTOHSL(phd->tb[i][j].tb2);
+	}
+    }
+#endif
+  return (phd);
+}
+
+void 
+free_perf_hash (perf_hash_data * phd)
+{
+  int i;
+  if (phd->translate)
+    Xfree (phd->translate);
+  if (phd->g)
+    Xfree (phd->g);
+  for (i = 0; i < phd->MAX_CH; i++)
+    if (phd->tb && phd->tb[i])
+      Xfree (phd->tb[i]);
+  if (phd->tb)
+    Xfree (phd->tb);
+  Xfree (phd);
+}
+
+
+int 
+perf_hash (perf_hash_data * phd, u_char * s)
+{
+  int u, v, w, i, l;
+  l = *s++;
+  i = (l - 1) % phd->MAX_L;
+  u = v = w = 0;
+  while (l)
+    {
+#ifndef STRUCT
+      if ((u += phd->tb0[phd->translate[*s]][i]) >= phd->MAX_N)
+	u -= phd->MAX_N;
+      if ((v += phd->tb1[phd->translate[*s]][i]) >= phd->MAX_N)
+	v -= phd->MAX_N;
+      if ((w += phd->tb2[phd->translate[*s]][i]) >= phd->MAX_N)
+	w -= phd->MAX_N;
+#else
+      struct tb_entry *tbp = &phd->tb[phd->translate[*s]][i];
+      if ((u += tbp->tb0) >= phd->MAX_N)
+	u -= phd->MAX_N;
+      if ((v += tbp->tb1) >= phd->MAX_N)
+	v -= phd->MAX_N;
+      if ((w += tbp->tb2) >= phd->MAX_N)
+	w -= phd->MAX_N;
+#endif
+      if (++i >= phd->MAX_L)
+	i = 0;
+      l--;
+      s++;
+    }
+  if (u == v && ++v >= phd->MAX_N)
+    v = 0;
+  if (w == v)
+    {
+      if (++w >= phd->MAX_N)
+	w = 0;
+      if (w == u && ++w >= phd->MAX_N)
+	w = 0;
+    }
+  else if (w == u)
+    {
+      if (++w >= phd->MAX_N)
+	w = 0;
+      if (w == v && ++w >= phd->MAX_N)
+	w = 0;
+    }
+  return ((phd->g[u] ^ phd->g[v] ^ phd->g[w]) > phd->MAX_M) ? 0 :
+    (phd->g[u] ^ phd->g[v] ^ phd->g[w]);
+}
Index: /trunk/indexers/mgpp/lib/perf_hash.h
===================================================================
--- /trunk/indexers/mgpp/lib/perf_hash.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/perf_hash.h	(revision 3365)
@@ -0,0 +1,52 @@
+/**************************************************************************
+ *
+ * perf_hash.h -- Perfect hashing functions
+ * Copyright (C) 1994  Bohdan S. Majewski and Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_PERF_HASH
+#define H_PERF_HASH
+
+struct tb_entry
+  {
+    long tb0, tb1, tb2;
+  };
+
+typedef struct
+  {
+    int MAX_L;
+    int MAX_N;
+    int MAX_M;
+    int MAX_CH;
+    u_char *translate;
+    int *g;
+    struct tb_entry **tb;
+  }
+perf_hash_data;
+
+perf_hash_data *gen_hash_func (int num, u_char ** keys, int r);
+
+int perf_hash (perf_hash_data * phd, u_char * s);
+
+int write_perf_hash_data (FILE * f, perf_hash_data * phd);
+
+void free_perf_hash (perf_hash_data * phd);
+
+perf_hash_data *read_perf_hash_data (FILE * f);
+
+#endif
Index: /trunk/indexers/mgpp/lib/random.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/random.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/random.cpp	(revision 3365)
@@ -0,0 +1,211 @@
+/**************************************************************************
+ *
+ * random.c -- pseudo random number generator
+ * Copyright (C) 1994  Chris Wallace (csw@bruce.cs.monash.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+/*
+ *	A random number generator called as a function by
+ *	random (iseed)	or	irandm (iseed)
+ *	The parameter should be a pointer to a 2-element long vector.
+ *	The first function returns a double uniform in 0 .. 1.
+ *	The second returns a long integer uniform in 0 .. 2**31-1
+ *	Both update iseed[] in exactly the same way.
+ *	iseed[] must be a 2-element integer vector.
+ *	The initial value of the second element may be anything.
+ *
+ *	The period of the random sequence is 2**32 * (2**32-1)
+ *	The table mt[0:127] is defined by mt[i] = 69069 ** (128-i)
+ */
+
+#define MASK ((long) 593970775)
+/*	or in hex, 23674657	*/
+
+#define SCALE ((double) 1.0 / (1024.0 * 1024.0 * 1024.0 * 2.0))
+/*	i.e. 2 to power -31	*/
+
+static long mt [128] =   {
+      902906369,
+     2030498053,
+     -473499623,
+     1640834941,
+      723406961,
+     1993558325,
+     -257162999,
+    -1627724755,
+      913952737,
+      278845029,
+     1327502073,
+    -1261253155,
+      981676113,
+    -1785280363,
+     1700077033,
+      366908557,
+    -1514479167,
+     -682799163,
+      141955545,
+     -830150595,
+      317871153,
+     1542036469,
+     -946413879,
+    -1950779155,
+      985397153,
+      626515237,
+      530871481,
+      783087261,
+    -1512358895,
+     1031357269,
+    -2007710807,
+    -1652747955,
+    -1867214463,
+      928251525,
+     1243003801,
+    -2132510467,
+     1874683889,
+     -717013323,
+      218254473,
+    -1628774995,
+    -2064896159,
+       69678053,
+      281568889,
+    -2104168611,
+     -165128239,
+     1536495125,
+      -39650967,
+      546594317,
+     -725987007,
+     1392966981,
+     1044706649,
+      687331773,
+    -2051306575,
+     1544302965,
+     -758494647,
+    -1243934099,
+      -75073759,
+      293132965,
+    -1935153095,
+      118929437,
+      807830417,
+    -1416222507,
+    -1550074071,
+      -84903219,
+     1355292929,
+     -380482555,
+    -1818444007,
+     -204797315,
+      170442609,
+    -1636797387,
+      868931593,
+     -623503571,
+     1711722209,
+      381210981,
+     -161547783,
+     -272740131,
+    -1450066095,
+     2116588437,
+     1100682473,
+      358442893,
+    -1529216831,
+     2116152005,
+     -776333095,
+     1265240893,
+     -482278607,
+     1067190005,
+      333444553,
+       86502381,
+      753481377,
+       39000101,
+     1779014585,
+      219658653,
+     -920253679,
+     2029538901,
+     1207761577,
+    -1515772851,
+     -236195711,
+      442620293,
+      423166617,
+    -1763648515,
+     -398436623,
+    -1749358155,
+     -538598519,
+     -652439379,
+      430550625,
+    -1481396507,
+     2093206905,
+    -1934691747,
+     -962631983,
+     1454463253,
+    -1877118871,
+     -291917555,
+    -1711673279,
+      201201733,
+     -474645415,
+      -96764739,
+    -1587365199,
+     1945705589,
+     1303896393,
+     1744831853,
+      381957665,
+     2135332261,
+      -55996615,
+    -1190135011,
+     1790562961,
+    -1493191723,
+      475559465,
+          69069
+		};
+
+double 
+random (long is [2])
+{
+	long it, leh, nit;
+
+	it = is [0];
+	leh = is [1];
+	if (it <= 0)	
+		it = (it + it) ^ MASK;
+	else
+		it = it + it;
+	nit = it - 1;
+/*	to ensure all-ones pattern omitted    */
+	leh = leh * mt[nit & 127] + nit;
+	is [0] = it;    is [1] = leh;
+	if (leh < 0) leh = ~leh;
+	return (SCALE * ((long) (leh | 1)));
+}
+
+
+
+long 
+irandm (long is [2])
+{
+	long it, leh, nit;
+
+	it = is [0];
+	leh = is [1];
+	if (it <= 0)	
+		it = (it + it) ^ MASK;
+	else
+		it = it + it;
+	nit = it - 1;
+/*	to ensure all-ones pattern omitted    */
+	leh = leh * mt[nit & 127] + nit;
+	is [0] = it;    is [1] = leh;
+	if (leh < 0) leh = ~leh;
+	return (leh);
+}
Index: /trunk/indexers/mgpp/lib/random.h
===================================================================
--- /trunk/indexers/mgpp/lib/random.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/random.h	(revision 3365)
@@ -0,0 +1,42 @@
+/**************************************************************************
+ *
+ * random.h -- pseudo random number generator
+ * Copyright (C) 1994  Chris Wallace (csw@bruce.cs.monash.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef RANDOM_H
+#define RANDOM_H
+
+/*
+ *	A random number generator called as a function by
+ *	random (iseed)	or	irandm (iseed)
+ *	The parameter should be a pointer to a 2-element long vector.
+ *	The first function returns a double uniform in 0 .. 1.
+ *	The second returns a long integer uniform in 0 .. 2**31-1
+ *	Both update iseed[] in exactly the same way.
+ *	iseed[] must be a 2-element integer vector.
+ *	The initial value of the second element may be anything.
+ *
+ *	The period of the random sequence is 2**32 * (2**32-1)
+ *	The table mt[0:127] is defined by mt[i] = 69069 ** (128-i)
+ */
+
+double random (long is [2]);
+long irandm (long is [2]);
+
+#endif
Index: /trunk/indexers/mgpp/lib/simplefrenchstem.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/simplefrenchstem.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/simplefrenchstem.cpp	(revision 3365)
@@ -0,0 +1,70 @@
+/**************************************************************************
+ *
+ * simplefrenchstem.c -- a simple french stemmer
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "simplefrenchstem.h"
+#include "unitool.h"
+
+
+/* =========================================================================
+ * Function: simplefrenchstem
+ * Description: a simple french stemmer
+ * Input: a word string with the length in the first byte
+ * Output: the stemmed word
+ * ========================================================================= */
+
+void simplefrenchstem (unsigned char *word) {      
+  unsigned short out[256]; /* temp space */
+  unsigned short *wordstart; /* points to first letter of word */
+  int last;                  /* last points to the last character */
+
+  /* decode */
+  utf8_word_to_unicode (word, out, 255);
+  wordstart = out + 1;
+  last = out[0]-1;
+  
+
+  if (last > 4) {
+    if (wordstart[last]=='x') {
+      if (wordstart[last-1]=='u' && wordstart[last-2]=='a') {
+	wordstart[last-1]='l';
+      }
+      last--;
+
+    } else {
+      if (last>=0 && wordstart[last]=='s') last--;
+      if (last>=0 && wordstart[last]=='r') last--;
+      if (last>=0 && wordstart[last]=='e') last--;
+
+      /* letter with accent e + ' -- there are two possible encodings  */
+      if (last>=0 && wordstart[last]==0xe9) { 
+	last--;
+      } else if (last>=1 && wordstart[last-1]=='e' && wordstart[last]==0x301) {
+	last -= 2; 
+      }
+
+      if (last >= 1 && wordstart[last]==wordstart[last-1]) last--;
+    }  /* end else */
+
+    out[0] = (unsigned char)(last+1);
+  } /* end if (len > 4) */
+
+  /* re-code, make sure the result is not longer than the input */
+  unicode_to_utf8_word (out, word, word[0]+1);
+}
Index: /trunk/indexers/mgpp/lib/simplefrenchstem.h
===================================================================
--- /trunk/indexers/mgpp/lib/simplefrenchstem.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/simplefrenchstem.h	(revision 3365)
@@ -0,0 +1,26 @@
+/**************************************************************************
+ *
+ * simplefrenchstem.h -- a simple french stemmer
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef SIMPLEFRENCHSTEM_H
+#define SIMPLEFRENCHSTEM_H
+
+void simplefrenchstem (unsigned char *word);
+
+#endif
Index: /trunk/indexers/mgpp/lib/strcasecmp.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/strcasecmp.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/strcasecmp.cpp	(revision 3365)
@@ -0,0 +1,73 @@
+/*
+ * Copyright (c) 1987 Regents of the University of California.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms are permitted
+ * provided that this notice is preserved and that due credit is given
+ * to the University of California at Berkeley. The name of the University
+ * may not be used to endorse or promote products derived from this
+ * software without specific written prior permission. This software
+ * is provided ``as is'' without express or implied warranty.
+ */
+
+#if defined(LIBC_SCCS) && !defined(lint)
+static char sccsid[] = "@(#)strcasecmp.c	5.5 (Berkeley) 11/24/87";
+#endif /* LIBC_SCCS and not lint */
+
+/*
+ * This array is designed for mapping upper and lower case letter
+ * together for a case independent comparison.  The mappings are
+ * based upon ascii character sequences.
+ */
+static unsigned char charmap[] = {
+	'\000', '\001', '\002', '\003', '\004', '\005', '\006', '\007',
+	'\010', '\011', '\012', '\013', '\014', '\015', '\016', '\017',
+	'\020', '\021', '\022', '\023', '\024', '\025', '\026', '\027',
+	'\030', '\031', '\032', '\033', '\034', '\035', '\036', '\037',
+	'\040', '\041', '\042', '\043', '\044', '\045', '\046', '\047',
+	'\050', '\051', '\052', '\053', '\054', '\055', '\056', '\057',
+	'\060', '\061', '\062', '\063', '\064', '\065', '\066', '\067',
+	'\070', '\071', '\072', '\073', '\074', '\075', '\076', '\077',
+	'\100', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\133', '\134', '\135', '\136', '\137',
+	'\140', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\173', '\174', '\175', '\176', '\177',
+	'\200', '\201', '\202', '\203', '\204', '\205', '\206', '\207',
+	'\210', '\211', '\212', '\213', '\214', '\215', '\216', '\217',
+	'\220', '\221', '\222', '\223', '\224', '\225', '\226', '\227',
+	'\230', '\231', '\232', '\233', '\234', '\235', '\236', '\237',
+	'\240', '\241', '\242', '\243', '\244', '\245', '\246', '\247',
+	'\250', '\251', '\252', '\253', '\254', '\255', '\256', '\257',
+	'\260', '\261', '\262', '\263', '\264', '\265', '\266', '\267',
+	'\270', '\271', '\272', '\273', '\274', '\275', '\276', '\277',
+	'\300', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\333', '\334', '\335', '\336', '\337',
+	'\340', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\373', '\374', '\375', '\376', '\377',
+};
+
+int
+strcasecmp(char *s1, char *s2)
+{
+    register unsigned char u1, u2;
+
+    for (;;) {
+	u1 = (unsigned char) *s1++;
+	u2 = (unsigned char) *s2++;
+	if (charmap[u1] != charmap[u2]) {
+	    return charmap[u1] - charmap[u2];
+	}
+	if (u1 == '\0') {
+	    return 0;
+	}
+    }
+}
+
Index: /trunk/indexers/mgpp/lib/strstr.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/strstr.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/strstr.cpp	(revision 3365)
@@ -0,0 +1,42 @@
+/* strstr.c -- return the offset of one string within another
+   Copyright (C) 1989, 1990 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+/* Written by Mike Rendell <michael@cs.mun.ca>.  */
+
+/* Return the starting address of string S2 in S1;
+   return 0 if it is not found. */
+
+char *
+strstr (char *s1, char *s2)
+{
+  int i;
+  char *p1;
+  char *p2;
+  char *s = s1;
+
+  for (p2 = s2, i = 0; *s; p2 = s2, i++, s++)
+    {
+      for (p1 = s; *p1 && *p2 && *p1 == *p2; p1++, p2++)
+	;
+      if (!*p2)
+	break;
+    }
+  if (!*p2)
+    return s1 + i;
+
+  return 0;
+}
Index: /trunk/indexers/mgpp/lib/sysfuncs.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/sysfuncs.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/sysfuncs.cpp	(revision 3365)
@@ -0,0 +1,6 @@
+#include <stdlib.h>
+
+void *bcopy(void *s, void *d, int n)
+{ return memcpy(d, s, n);
+}
+
Index: /trunk/indexers/mgpp/lib/sysfuncs.h
===================================================================
--- /trunk/indexers/mgpp/lib/sysfuncs.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/sysfuncs.h	(revision 3365)
@@ -0,0 +1,388 @@
+/* System dependent definitions for GNU tar.
+   Copyright (C) 1994 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ */
+
+/*
+ * [TS:Aug/95]
+ * Based on code from system.h file in GNU's tar package 
+ *
+ */
+
+#ifndef SYSFUNCS_H
+#define SYSFUNCS_H
+
+#include <sys/stat.h>
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__
+#  include <WIN32cfg.h>
+# else
+#  include <config.h>
+# endif
+#endif
+
+/* Declare alloca.  AIX requires this to be the first thing in the file.  */
+
+#ifdef __GNUC__
+#ifndef alloca
+#define alloca __builtin_alloca
+#endif
+#else
+#if HAVE_ALLOCA_H
+#include <alloca.h>
+#else
+#ifdef _AIX
+#pragma alloca
+#else
+#ifndef alloca
+char *alloca ();
+#endif
+#endif
+#endif
+#endif
+
+#include <sys/types.h>
+
+/* ------------------------------------------------- */
+/* [TS:Aug/95]
+ * Some extra include files that mg needs
+ */
+#include <stdlib.h>
+#include <math.h>
+
+#ifndef log2
+# ifndef M_LN2
+#  define M_LN2 0.69314718055994530942
+# endif
+# define log2(x) (log(x)/M_LN2)
+#endif
+
+
+#ifndef HAVE_SETBUFFER
+#ifdef __CYGWIN__ 
+/* Cygwin has setbuffer in stdio.h, but can't link against it!! */
+#include <stdio.h>
+#endif
+# define setbuffer(f, b, s)  setvbuf(f, b, _IOFBF, s)
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#if defined(__MSDOS__)  && !defined(__WIN32__)
+# define SHORT_SUFFIX 1
+#endif
+
+
+
+#include <assert.h>
+
+/* ------------------------------------------------- */
+
+/* Declare a generic pointer type.  */
+#if __STDC__ || defined(__TURBOC__)
+#define voidstar void *
+#else
+#define voidstar char *
+#endif
+
+/* Declare ISASCII.  */
+
+#include <ctype.h>
+
+#if STDC_HEADERS
+#define ISASCII(Char) 1
+#else
+#ifdef isascii
+#define ISASCII(Char) isascii (Char)
+#else
+#if HAVE_ISASCII
+#define ISASCII(Char) isascii (Char)
+#else
+#define ISASCII(Char) 1
+#endif
+#endif
+#endif
+
+
+/* ------------------------------------------------- */
+/* [TS:Aug/95] 
+ * The following is copied from GNU Autoconf.Info (1.1)   
+ * It sets up the approriate string and memory functions.
+ */
+
+#if STDC_HEADERS || HAVE_STRING_H
+#include <string.h>
+   /* An ANSI string.h and pre-ANSI memory.h might conflict */
+#if !STDC_HEADERS && HAVE_MEMORY_H
+#include <memory.h>
+#endif /* not STDC_HEADERS and HAVE_MEMORY_H */
+#ifndef index
+/*#define index strchr 
+ *
+ * this is stuffing up our code so its commented out */
+#endif
+#ifndef rindex
+#define rindex strrchr
+#endif
+
+#ifdef __cplusplus /* compiling for C++ */
+#include <stdlib.h>
+#ifndef __bcopy
+inline void *bcopy(void *s, void *d, int n)
+{ return memcpy(d, s, n);
+}
+#define __bcopy 1
+#endif
+#ifndef __bcmp
+inline int bcmp(void *s1, void *s2, int n)
+{ return memcmp(s1, s2, n);
+}
+#define __bcmp 1
+#endif
+#ifndef __bzero
+inline void bzero(void *m, int n)
+{ memset(m, 0, n);
+}
+#define __bzero 1
+#endif
+#else              /* compiling for C */
+#ifndef bcopy
+#define bcopy(s, d, n) memcpy((d), (s), (n))
+#endif
+#ifndef bcmp
+#define bcmp(s1, s2, n) memcmp((s1), (s2), (n))
+#endif
+#ifndef bzero
+#define bzero(s, n) memset((s), 0, (n))
+#endif
+#endif
+
+#else /* not STDC_HEADERS and not HAVE_STRING_H */
+#include <strings.h>
+   /* memory.h and strings.h conflict on some systems */
+#endif
+/* ------------------------------------------------- */
+
+
+/* Declare errno.  */
+
+#include <errno.h>
+#ifndef errno
+extern int errno;
+#endif
+
+/* Declare open parameters.  */
+
+#ifdef HAVE_FCNTL_H
+#include <fcntl.h>
+#else
+#include <sys/file.h>
+#endif
+
+#ifndef	O_BINARY
+#define O_BINARY 0
+#endif
+#ifndef O_CREAT
+#define O_CREAT 0
+#endif
+#ifndef	O_NDELAY
+#define O_NDELAY 0
+#endif
+#ifndef	O_RDONLY
+#define O_RDONLY 0
+#endif
+#ifndef O_RDWR
+#define O_RDWR 2
+#endif
+
+/* Declare file status routines and bits.  */
+
+#ifdef STAT_MACROS_BROKEN
+#undef S_ISBLK
+#undef S_ISCHR
+#undef S_ISDIR
+#undef S_ISFIFO
+#undef S_ISLNK
+#undef S_ISMPB
+#undef S_ISMPC
+#undef S_ISNWK
+#undef S_ISREG
+#undef S_ISSOCK
+#endif
+
+/* On MSDOS, there are missing things from <sys/stat.h>.  */
+#ifdef __MSDOS__
+#define S_ISUID 0
+#define S_ISGID 0
+#define S_ISVTX 0
+#endif
+
+#ifndef S_ISREG			/* POSIX.1 stat stuff missing */
+#define mode_t unsigned short
+#endif
+#if !defined(S_ISBLK) && defined(S_IFBLK)
+#define S_ISBLK(Mode) (((Mode) & S_IFMT) == S_IFBLK)
+#endif
+#if !defined(S_ISCHR) && defined(S_IFCHR)
+#define S_ISCHR(Mode) (((Mode) & S_IFMT) == S_IFCHR)
+#endif
+#if !defined(S_ISDIR) && defined(S_IFDIR)
+#define S_ISDIR(Mode) (((Mode) & S_IFMT) == S_IFDIR)
+#endif
+#if !defined(S_ISREG) && defined(S_IFREG)
+#define S_ISREG(Mode) (((Mode) & S_IFMT) == S_IFREG)
+#endif
+#if !defined(S_ISFIFO) && defined(S_IFIFO)
+#define S_ISFIFO(Mode) (((Mode) & S_IFMT) == S_IFIFO)
+#endif
+#if !defined(S_ISLNK) && defined(S_IFLNK)
+#define S_ISLNK(Mode) (((Mode) & S_IFMT) == S_IFLNK)
+#endif
+#if !defined(S_ISSOCK) && defined(S_IFSOCK)
+#define S_ISSOCK(Mode) (((Mode) & S_IFMT) == S_IFSOCK)
+#endif
+#if !defined(S_ISMPB) && defined(S_IFMPB)	/* V7 */
+#define S_ISMPB(Mode) (((Mode) & S_IFMT) == S_IFMPB)
+#define S_ISMPC(Mode) (((Mode) & S_IFMT) == S_IFMPC)
+#endif
+#if !defined(S_ISNWK) && defined(S_IFNWK)	/* HP/UX */
+#define S_ISNWK(Mode) (((Mode) & S_IFMT) == S_IFNWK)
+#endif
+
+#if !defined(S_ISCTG) && defined(S_IFCTG)	/* contiguous file */
+#define S_ISCTG(Mode) (((Mode) & S_IFMT) == S_IFCTG)
+#endif
+#if !defined(S_ISVTX)
+#define S_ISVTX 0001000
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#if !defined(_POSIX_SOURCE) && !defined(__WIN32__)
+#include <sys/param.h>
+#endif
+
+/* Include <unistd.h> before any preprocessor test of _POSIX_VERSION.  */
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+
+/* Declare standard functions.  */
+
+#ifdef STDC_HEADERS
+#include <stdlib.h>
+#else
+voidstar malloc ();
+voidstar realloc ();
+char *getenv ();
+#endif
+
+#include <stdio.h>
+
+#ifndef _POSIX_VERSION
+#ifdef __MSDOS__
+#include <io.h>
+#else
+off_t lseek ();
+#endif
+#endif
+
+#ifndef __WIN32__
+#include <pathmax.h>
+#endif
+
+/* Until getoptold function is declared in getopt.h, we need this here for
+   struct option.  */
+/*#include <getopt.h>*/
+
+#ifdef WITH_DMALLOC
+#undef HAVE_VALLOC
+#define DMALLOC_FUNC_CHECK
+#include <dmalloc.h>
+#endif
+
+/* Prototypes for external functions.  */
+
+#ifndef __P
+#if PROTOTYPES
+#define __P(Args) Args
+#else
+#define __P(Args) ()
+#endif
+#endif
+
+#if HAVE_LOCALE_H
+#include <locale.h>
+#endif
+/*** setlocale is ansi and posix...(jrm21)
+#if !HAVE_SETLOCALE
+#define setlocale(Category, Locale)
+#endif
+***/
+
+#if ENABLE_NLS
+#include <libintl.h>
+#define _(Text) gettext (Text)
+#else
+#define textdomain(Domain)
+#define _(Text) Text
+#endif
+
+/* Library modules.  */
+
+#ifdef HAVE_VPRINTF
+void error __P ((int, int, const char *,...));
+#else
+void error ();
+#endif
+
+#ifndef HAVE_STRSTR
+char *strstr __P ((const char *, const char *));
+#endif
+
+/* [RB/TS:Oct/95] commented out. Used by lib/gmalloc.c - but it defines it anyway ???
+ *#ifndef HAVE_VALLOC
+ *#define valloc(Size) malloc (Size)
+ *#endif
+ */
+
+voidstar xmalloc __P ((size_t));
+voidstar xrealloc __P ((voidstar, size_t));
+char *xstrdup __P ((const char *));
+
+/* [TS:Aug/95] 
+ * These were required by files which referred 
+ * to these as function pointers. 
+ * e.g. huffman.c - fread,  mgfelics.c - fgetc
+ */
+#ifndef HAVE_FREAD_DECL
+extern size_t   fread __P ((void *, size_t, size_t, FILE *));
+#endif
+#ifndef HAVE_FGETC_DECL
+extern int      fgetc __P ((FILE *));
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifndef u_long
+# define u_long unsigned long
+#endif
+#ifndef u_char
+# define u_char unsigned char
+#endif
+
+
+#endif /* SYSFUNCS_H */
Index: /trunk/indexers/mgpp/lib/unitool.cpp
===================================================================
--- /trunk/indexers/mgpp/lib/unitool.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/lib/unitool.cpp	(revision 3365)
@@ -0,0 +1,1756 @@
+#include "unitool.h"
+#include "sysfuncs.h"
+
+
+/* unicode_cat_data is a combination of the letter, digit, and
+ * space information in a form that is quick to look up.
+ * The information about each byte is stored in two bits
+ *
+ * 00 - neither letter, digit or space
+ * 01 - letter
+ * 10 - space
+ * 11 - digit
+ *
+ * You can test for letter | space by looking at the right
+ * column
+ */
+unsigned long *unicode_cat_data=NULL;
+
+
+typedef struct unidecompose {
+  unsigned short ch;
+  unsigned short str[3];
+} unidecompose;
+
+typedef struct unirelate {
+  unsigned short ch1;
+  unsigned short ch2;
+} unirelate;
+
+typedef struct unirange {
+  unsigned short first;
+  unsigned short last;
+} unirange;
+
+
+
+/* This decomposition data only includes canonical decompositional
+ * characters.  Any unused bytes are set to 0x0. */
+
+#define NUM_DECOMPOSE_INFO 1269
+static const unidecompose decompose_info[NUM_DECOMPOSE_INFO] = {
+  {0xC0,0x41,0x300,0x0}, {0xC1,0x41,0x301,0x0}, {0xC2,0x41,0x302,0x0}, 
+  {0xC3,0x41,0x303,0x0}, {0xC4,0x41,0x308,0x0}, {0xC5,0x41,0x30A,0x0}, 
+  {0xC7,0x43,0x327,0x0}, {0xC8,0x45,0x300,0x0}, {0xC9,0x45,0x301,0x0}, 
+  {0xCA,0x45,0x302,0x0}, {0xCB,0x45,0x308,0x0}, {0xCC,0x49,0x300,0x0}, 
+  {0xCD,0x49,0x301,0x0}, {0xCE,0x49,0x302,0x0}, {0xCF,0x49,0x308,0x0}, 
+  {0xD1,0x4E,0x303,0x0}, {0xD2,0x4F,0x300,0x0}, {0xD3,0x4F,0x301,0x0}, 
+  {0xD4,0x4F,0x302,0x0}, {0xD5,0x4F,0x303,0x0}, {0xD6,0x4F,0x308,0x0}, 
+  {0xD9,0x55,0x300,0x0}, {0xDA,0x55,0x301,0x0}, {0xDB,0x55,0x302,0x0}, 
+  {0xDC,0x55,0x308,0x0}, {0xDD,0x59,0x301,0x0}, {0xE0,0x61,0x300,0x0}, 
+  {0xE1,0x61,0x301,0x0}, {0xE2,0x61,0x302,0x0}, {0xE3,0x61,0x303,0x0}, 
+  {0xE4,0x61,0x308,0x0}, {0xE5,0x61,0x30A,0x0}, {0xE7,0x63,0x327,0x0}, 
+  {0xE8,0x65,0x300,0x0}, {0xE9,0x65,0x301,0x0}, {0xEA,0x65,0x302,0x0}, 
+  {0xEB,0x65,0x308,0x0}, {0xEC,0x69,0x300,0x0}, {0xED,0x69,0x301,0x0}, 
+  {0xEE,0x69,0x302,0x0}, {0xEF,0x69,0x308,0x0}, {0xF1,0x6E,0x303,0x0}, 
+  {0xF2,0x6F,0x300,0x0}, {0xF3,0x6F,0x301,0x0}, {0xF4,0x6F,0x302,0x0}, 
+  {0xF5,0x6F,0x303,0x0}, {0xF6,0x6F,0x308,0x0}, {0xF9,0x75,0x300,0x0}, 
+  {0xFA,0x75,0x301,0x0}, {0xFB,0x75,0x302,0x0}, {0xFC,0x75,0x308,0x0}, 
+  {0xFD,0x79,0x301,0x0}, {0xFF,0x79,0x308,0x0}, {0x100,0x41,0x304,0x0}, 
+  {0x101,0x61,0x304,0x0}, {0x102,0x41,0x306,0x0}, {0x103,0x61,0x306,0x0}, 
+  {0x104,0x41,0x328,0x0}, {0x105,0x61,0x328,0x0}, {0x106,0x43,0x301,0x0}, 
+  {0x107,0x63,0x301,0x0}, {0x108,0x43,0x302,0x0}, {0x109,0x63,0x302,0x0}, 
+  {0x10A,0x43,0x307,0x0}, {0x10B,0x63,0x307,0x0}, {0x10C,0x43,0x30C,0x0}, 
+  {0x10D,0x63,0x30C,0x0}, {0x10E,0x44,0x30C,0x0}, {0x10F,0x64,0x30C,0x0}, 
+  {0x112,0x45,0x304,0x0}, {0x113,0x65,0x304,0x0}, {0x114,0x45,0x306,0x0}, 
+  {0x115,0x65,0x306,0x0}, {0x116,0x45,0x307,0x0}, {0x117,0x65,0x307,0x0}, 
+  {0x118,0x45,0x328,0x0}, {0x119,0x65,0x328,0x0}, {0x11A,0x45,0x30C,0x0}, 
+  {0x11B,0x65,0x30C,0x0}, {0x11C,0x47,0x302,0x0}, {0x11D,0x67,0x302,0x0}, 
+  {0x11E,0x47,0x306,0x0}, {0x11F,0x67,0x306,0x0}, {0x120,0x47,0x307,0x0}, 
+  {0x121,0x67,0x307,0x0}, {0x122,0x47,0x327,0x0}, {0x123,0x67,0x327,0x0}, 
+  {0x124,0x48,0x302,0x0}, {0x125,0x68,0x302,0x0}, {0x128,0x49,0x303,0x0}, 
+  {0x129,0x69,0x303,0x0}, {0x12A,0x49,0x304,0x0}, {0x12B,0x69,0x304,0x0}, 
+  {0x12C,0x49,0x306,0x0}, {0x12D,0x69,0x306,0x0}, {0x12E,0x49,0x328,0x0}, 
+  {0x12F,0x69,0x328,0x0}, {0x130,0x49,0x307,0x0}, {0x134,0x4A,0x302,0x0}, 
+  {0x135,0x6A,0x302,0x0}, {0x136,0x4B,0x327,0x0}, {0x137,0x6B,0x327,0x0}, 
+  {0x139,0x4C,0x301,0x0}, {0x13A,0x6C,0x301,0x0}, {0x13B,0x4C,0x327,0x0}, 
+  {0x13C,0x6C,0x327,0x0}, {0x13D,0x4C,0x30C,0x0}, {0x13E,0x6C,0x30C,0x0}, 
+  {0x143,0x4E,0x301,0x0}, {0x144,0x6E,0x301,0x0}, {0x145,0x4E,0x327,0x0}, 
+  {0x146,0x6E,0x327,0x0}, {0x147,0x4E,0x30C,0x0}, {0x148,0x6E,0x30C,0x0}, 
+  {0x14C,0x4F,0x304,0x0}, {0x14D,0x6F,0x304,0x0}, {0x14E,0x4F,0x306,0x0}, 
+  {0x14F,0x6F,0x306,0x0}, {0x150,0x4F,0x30B,0x0}, {0x151,0x6F,0x30B,0x0}, 
+  {0x154,0x52,0x301,0x0}, {0x155,0x72,0x301,0x0}, {0x156,0x52,0x327,0x0}, 
+  {0x157,0x72,0x327,0x0}, {0x158,0x52,0x30C,0x0}, {0x159,0x72,0x30C,0x0}, 
+  {0x15A,0x53,0x301,0x0}, {0x15B,0x73,0x301,0x0}, {0x15C,0x53,0x302,0x0}, 
+  {0x15D,0x73,0x302,0x0}, {0x15E,0x53,0x327,0x0}, {0x15F,0x73,0x327,0x0}, 
+  {0x160,0x53,0x30C,0x0}, {0x161,0x73,0x30C,0x0}, {0x162,0x54,0x327,0x0}, 
+  {0x163,0x74,0x327,0x0}, {0x164,0x54,0x30C,0x0}, {0x165,0x74,0x30C,0x0}, 
+  {0x168,0x55,0x303,0x0}, {0x169,0x75,0x303,0x0}, {0x16A,0x55,0x304,0x0}, 
+  {0x16B,0x75,0x304,0x0}, {0x16C,0x55,0x306,0x0}, {0x16D,0x75,0x306,0x0}, 
+  {0x16E,0x55,0x30A,0x0}, {0x16F,0x75,0x30A,0x0}, {0x170,0x55,0x30B,0x0}, 
+  {0x171,0x75,0x30B,0x0}, {0x172,0x55,0x328,0x0}, {0x173,0x75,0x328,0x0}, 
+  {0x174,0x57,0x302,0x0}, {0x175,0x77,0x302,0x0}, {0x176,0x59,0x302,0x0}, 
+  {0x177,0x79,0x302,0x0}, {0x178,0x59,0x308,0x0}, {0x179,0x5A,0x301,0x0}, 
+  {0x17A,0x7A,0x301,0x0}, {0x17B,0x5A,0x307,0x0}, {0x17C,0x7A,0x307,0x0}, 
+  {0x17D,0x5A,0x30C,0x0}, {0x17E,0x7A,0x30C,0x0}, {0x1A0,0x4F,0x31B,0x0}, 
+  {0x1A1,0x6F,0x31B,0x0}, {0x1AF,0x55,0x31B,0x0}, {0x1B0,0x75,0x31B,0x0}, 
+  {0x1CD,0x41,0x30C,0x0}, {0x1CE,0x61,0x30C,0x0}, {0x1CF,0x49,0x30C,0x0}, 
+  {0x1D0,0x69,0x30C,0x0}, {0x1D1,0x4F,0x30C,0x0}, {0x1D2,0x6F,0x30C,0x0}, 
+  {0x1D3,0x55,0x30C,0x0}, {0x1D4,0x75,0x30C,0x0}, {0x1D5,0xDC,0x304,0x0}, 
+  {0x1D6,0xFC,0x304,0x0}, {0x1D7,0xDC,0x301,0x0}, {0x1D8,0xFC,0x301,0x0}, 
+  {0x1D9,0xDC,0x30C,0x0}, {0x1DA,0xFC,0x30C,0x0}, {0x1DB,0xDC,0x300,0x0}, 
+  {0x1DC,0xFC,0x300,0x0}, {0x1DE,0xC4,0x304,0x0}, {0x1DF,0xE4,0x304,0x0}, 
+  {0x1E0,0x41,0x307,0x304}, {0x1E1,0x61,0x307,0x304}, {0x1E2,0xC6,0x304,0x0}, 
+  {0x1E3,0xE6,0x304,0x0}, {0x1E6,0x47,0x30C,0x0}, {0x1E7,0x67,0x30C,0x0}, 
+  {0x1E8,0x4B,0x30C,0x0}, {0x1E9,0x6B,0x30C,0x0}, {0x1EA,0x4F,0x328,0x0}, 
+  {0x1EB,0x6F,0x328,0x0}, {0x1EC,0x1EA,0x304,0x0}, {0x1ED,0x1EB,0x304,0x0}, 
+  {0x1EE,0x1B7,0x30C,0x0}, {0x1EF,0x292,0x30C,0x0}, {0x1F0,0x6A,0x30C,0x0}, 
+  {0x1F4,0x47,0x301,0x0}, {0x1F5,0x67,0x301,0x0}, {0x1FA,0xC5,0x301,0x0}, 
+  {0x1FB,0xE5,0x301,0x0}, {0x1FC,0xC6,0x301,0x0}, {0x1FD,0xE6,0x301,0x0}, 
+  {0x1FE,0xD8,0x301,0x0}, {0x1FF,0xF8,0x301,0x0}, {0x200,0x41,0x30F,0x0}, 
+  {0x201,0x61,0x30F,0x0}, {0x202,0x41,0x311,0x0}, {0x203,0x61,0x311,0x0}, 
+  {0x204,0x45,0x30F,0x0}, {0x205,0x65,0x30F,0x0}, {0x206,0x45,0x311,0x0}, 
+  {0x207,0x65,0x311,0x0}, {0x208,0x49,0x30F,0x0}, {0x209,0x69,0x30F,0x0}, 
+  {0x20A,0x49,0x311,0x0}, {0x20B,0x69,0x311,0x0}, {0x20C,0x4F,0x30F,0x0}, 
+  {0x20D,0x6F,0x30F,0x0}, {0x20E,0x4F,0x311,0x0}, {0x20F,0x6F,0x311,0x0}, 
+  {0x210,0x52,0x30F,0x0}, {0x211,0x72,0x30F,0x0}, {0x212,0x52,0x311,0x0}, 
+  {0x213,0x72,0x311,0x0}, {0x214,0x55,0x30F,0x0}, {0x215,0x75,0x30F,0x0}, 
+  {0x216,0x55,0x311,0x0}, {0x217,0x75,0x311,0x0}, {0x310,0x306,0x307,0x0}, 
+  {0x340,0x300,0x0,0x0}, {0x341,0x301,0x0,0x0}, {0x343,0x313,0x0,0x0}, 
+  {0x344,0x308,0x30D,0x0}, {0x374,0x2B9,0x0,0x0}, {0x37E,0x3B,0x0,0x0}, 
+  {0x385,0xA8,0x30D,0x0}, {0x386,0x391,0x30D,0x0}, {0x387,0xB7,0x0,0x0}, 
+  {0x388,0x395,0x30D,0x0}, {0x389,0x397,0x30D,0x0}, {0x38A,0x399,0x30D,0x0}, 
+  {0x38C,0x39F,0x30D,0x0}, {0x38E,0x3A5,0x30D,0x0}, {0x38F,0x3A9,0x30D,0x0}, 
+  {0x390,0x3B9,0x344,0x0}, {0x3AA,0x399,0x308,0x0}, {0x3AB,0x3A5,0x308,0x0}, 
+  {0x3AC,0x3B1,0x30D,0x0}, {0x3AD,0x3B5,0x30D,0x0}, {0x3AE,0x3B7,0x30D,0x0}, 
+  {0x3AF,0x3B9,0x30D,0x0}, {0x3B0,0x3C5,0x344,0x0}, {0x3CA,0x3B9,0x308,0x0}, 
+  {0x3CB,0x3C5,0x308,0x0}, {0x3CC,0x3BF,0x30D,0x0}, {0x3CD,0x3C5,0x30D,0x0}, 
+  {0x3CE,0x3C9,0x30D,0x0}, {0x3D3,0x3D2,0x30D,0x0}, {0x3D4,0x3D2,0x308,0x0}, 
+  {0x401,0x415,0x308,0x0}, {0x403,0x413,0x301,0x0}, {0x407,0x406,0x308,0x0}, 
+  {0x40C,0x41A,0x301,0x0}, {0x40E,0x423,0x306,0x0}, {0x419,0x418,0x306,0x0}, 
+  {0x439,0x438,0x306,0x0}, {0x451,0x435,0x308,0x0}, {0x453,0x433,0x301,0x0}, 
+  {0x457,0x456,0x308,0x0}, {0x45C,0x43A,0x301,0x0}, {0x45E,0x443,0x306,0x0}, 
+  {0x476,0x474,0x30F,0x0}, {0x477,0x475,0x30F,0x0}, {0x4C1,0x416,0x306,0x0}, 
+  {0x4C2,0x436,0x306,0x0}, {0x4D0,0x410,0x306,0x0}, {0x4D1,0x430,0x306,0x0}, 
+  {0x4D2,0x410,0x308,0x0}, {0x4D3,0x430,0x308,0x0}, {0x4D6,0x415,0x306,0x0}, 
+  {0x4D7,0x435,0x306,0x0}, {0x4DA,0x4D8,0x308,0x0}, {0x4DB,0x4D9,0x308,0x0}, 
+  {0x4DC,0x416,0x308,0x0}, {0x4DD,0x436,0x308,0x0}, {0x4DE,0x417,0x308,0x0}, 
+  {0x4DF,0x437,0x308,0x0}, {0x4E2,0x418,0x304,0x0}, {0x4E3,0x438,0x304,0x0}, 
+  {0x4E4,0x418,0x308,0x0}, {0x4E5,0x438,0x308,0x0}, {0x4E6,0x41E,0x308,0x0}, 
+  {0x4E7,0x43E,0x308,0x0}, {0x4EA,0x4E8,0x308,0x0}, {0x4EB,0x4E9,0x308,0x0}, 
+  {0x4EE,0x423,0x304,0x0}, {0x4EF,0x443,0x304,0x0}, {0x4F0,0x423,0x308,0x0}, 
+  {0x4F1,0x443,0x308,0x0}, {0x4F2,0x423,0x30B,0x0}, {0x4F3,0x443,0x30B,0x0}, 
+  {0x4F4,0x427,0x308,0x0}, {0x4F5,0x447,0x308,0x0}, {0x4F8,0x42B,0x308,0x0}, 
+  {0x4F9,0x44B,0x308,0x0}, {0x929,0x928,0x93C,0x0}, {0x931,0x930,0x93C,0x0}, 
+  {0x934,0x933,0x93C,0x0}, {0x958,0x915,0x93C,0x0}, {0x959,0x916,0x93C,0x0}, 
+  {0x95A,0x917,0x93C,0x0}, {0x95B,0x91C,0x93C,0x0}, {0x95C,0x921,0x93C,0x0}, 
+  {0x95D,0x922,0x93C,0x0}, {0x95E,0x92B,0x93C,0x0}, {0x95F,0x92F,0x93C,0x0}, 
+  {0x9B0,0x9AC,0x9BC,0x0}, {0x9CB,0x9C7,0x9BE,0x0}, {0x9CC,0x9C7,0x9D7,0x0}, 
+  {0x9DC,0x9A1,0x9BC,0x0}, {0x9DD,0x9A2,0x9BC,0x0}, {0x9DF,0x9AF,0x9BC,0x0}, 
+  {0xA59,0xA16,0xA3C,0x0}, {0xA5A,0xA17,0xA3C,0x0}, {0xA5B,0xA1C,0xA3C,0x0}, 
+  {0xA5C,0xA21,0xA3C,0x0}, {0xA5E,0xA2B,0xA3C,0x0}, {0xB48,0xB47,0xB56,0x0}, 
+  {0xB4B,0xB47,0xB3E,0x0}, {0xB4C,0xB47,0xB57,0x0}, {0xB5C,0xB21,0xB3C,0x0}, 
+  {0xB5D,0xB22,0xB3C,0x0}, {0xB5F,0xB2F,0xB3C,0x0}, {0xB94,0xB92,0xBD7,0x0}, 
+  {0xBCA,0xBC6,0xBBE,0x0}, {0xBCB,0xBC7,0xBBE,0x0}, {0xBCC,0xBC6,0xBD7,0x0}, 
+  {0xC48,0xC46,0xC56,0x0}, {0xCC0,0xCBF,0xCD5,0x0}, {0xCC7,0xCC6,0xCD5,0x0}, 
+  {0xCC8,0xCC6,0xCD6,0x0}, {0xCCA,0xCC6,0xCC2,0x0}, {0xCCB,0xCC6,0xCC2,0xCD5}, 
+  {0xD4A,0xD46,0xD3E,0x0}, {0xD4B,0xD47,0xD3E,0x0}, {0xD4C,0xD46,0xD57,0x0}, 
+  {0xE33,0xE4D,0xE32,0x0}, {0xEB3,0xECD,0xEB2,0x0}, {0xF43,0xF42,0xFB7,0x0}, 
+  {0xF4D,0xF4C,0xFB7,0x0}, {0xF52,0xF51,0xFB7,0x0}, {0xF57,0xF56,0xFB7,0x0}, 
+  {0xF5C,0xF5B,0xFB7,0x0}, {0xF69,0xF40,0xFB5,0x0}, {0xF73,0xF72,0xF71,0x0}, 
+  {0xF75,0xF71,0xF74,0x0}, {0xF76,0xFB2,0xF80,0x0}, {0xF77,0xF76,0xF71,0x0}, 
+  {0xF78,0xFB3,0xF80,0x0}, {0xF79,0xF78,0xF71,0x0}, {0xF81,0xF80,0xF71,0x0}, 
+  {0xF93,0xF92,0xFB7,0x0}, {0xF9D,0xF9C,0xFB7,0x0}, {0xFA2,0xFA1,0xFB7,0x0}, 
+  {0xFA7,0xFA6,0xFB7,0x0}, {0xFAC,0xFAB,0xFB7,0x0}, {0xFB9,0xF90,0xFB5,0x0}, 
+  {0x1E00,0x41,0x325,0x0}, {0x1E01,0x61,0x325,0x0}, {0x1E02,0x42,0x307,0x0}, 
+  {0x1E03,0x62,0x307,0x0}, {0x1E04,0x42,0x323,0x0}, {0x1E05,0x62,0x323,0x0}, 
+  {0x1E06,0x42,0x331,0x0}, {0x1E07,0x62,0x331,0x0}, {0x1E08,0xC7,0x301,0x0}, 
+  {0x1E09,0xE7,0x301,0x0}, {0x1E0A,0x44,0x307,0x0}, {0x1E0B,0x64,0x307,0x0}, 
+  {0x1E0C,0x44,0x323,0x0}, {0x1E0D,0x64,0x323,0x0}, {0x1E0E,0x44,0x331,0x0}, 
+  {0x1E0F,0x64,0x331,0x0}, {0x1E10,0x44,0x327,0x0}, {0x1E11,0x64,0x327,0x0}, 
+  {0x1E12,0x44,0x32D,0x0}, {0x1E13,0x64,0x32D,0x0}, {0x1E14,0x112,0x300,0x0}, 
+  {0x1E15,0x113,0x300,0x0}, {0x1E16,0x112,0x301,0x0}, {0x1E17,0x113,0x301,0x0}, 
+  {0x1E18,0x45,0x32D,0x0}, {0x1E19,0x65,0x32D,0x0}, {0x1E1A,0x45,0x330,0x0}, 
+  {0x1E1B,0x65,0x330,0x0}, {0x1E1C,0x114,0x327,0x0}, {0x1E1D,0x115,0x327,0x0}, 
+  {0x1E1E,0x46,0x307,0x0}, {0x1E1F,0x66,0x307,0x0}, {0x1E20,0x47,0x304,0x0}, 
+  {0x1E21,0x67,0x304,0x0}, {0x1E22,0x48,0x307,0x0}, {0x1E23,0x68,0x307,0x0}, 
+  {0x1E24,0x48,0x323,0x0}, {0x1E25,0x68,0x323,0x0}, {0x1E26,0x48,0x308,0x0}, 
+  {0x1E27,0x68,0x308,0x0}, {0x1E28,0x48,0x327,0x0}, {0x1E29,0x68,0x327,0x0}, 
+  {0x1E2A,0x48,0x32E,0x0}, {0x1E2B,0x68,0x32E,0x0}, {0x1E2C,0x49,0x330,0x0}, 
+  {0x1E2D,0x69,0x330,0x0}, {0x1E2E,0xCF,0x301,0x0}, {0x1E2F,0xEF,0x301,0x0}, 
+  {0x1E30,0x4B,0x301,0x0}, {0x1E31,0x6B,0x301,0x0}, {0x1E32,0x4B,0x323,0x0}, 
+  {0x1E33,0x6B,0x323,0x0}, {0x1E34,0x4B,0x331,0x0}, {0x1E35,0x6B,0x331,0x0}, 
+  {0x1E36,0x4C,0x323,0x0}, {0x1E37,0x6C,0x323,0x0}, {0x1E38,0x1E36,0x304,0x0}, 
+  {0x1E39,0x1E37,0x304,0x0}, {0x1E3A,0x4C,0x331,0x0}, {0x1E3B,0x6C,0x331,0x0}, 
+  {0x1E3C,0x4C,0x32D,0x0}, {0x1E3D,0x6C,0x32D,0x0}, {0x1E3E,0x4D,0x301,0x0}, 
+  {0x1E3F,0x6D,0x301,0x0}, {0x1E40,0x4D,0x307,0x0}, {0x1E41,0x6D,0x307,0x0}, 
+  {0x1E42,0x4D,0x323,0x0}, {0x1E43,0x6D,0x323,0x0}, {0x1E44,0x4E,0x307,0x0}, 
+  {0x1E45,0x6E,0x307,0x0}, {0x1E46,0x4E,0x323,0x0}, {0x1E47,0x6E,0x323,0x0}, 
+  {0x1E48,0x4E,0x331,0x0}, {0x1E49,0x6E,0x331,0x0}, {0x1E4A,0x4E,0x32D,0x0}, 
+  {0x1E4B,0x6E,0x32D,0x0}, {0x1E4C,0xD5,0x301,0x0}, {0x1E4D,0xF5,0x301,0x0}, 
+  {0x1E4E,0xD5,0x308,0x0}, {0x1E4F,0xF5,0x308,0x0}, {0x1E50,0x14C,0x300,0x0}, 
+  {0x1E51,0x14D,0x300,0x0}, {0x1E52,0x14C,0x301,0x0}, {0x1E53,0x14D,0x301,0x0}, 
+  {0x1E54,0x50,0x301,0x0}, {0x1E55,0x70,0x301,0x0}, {0x1E56,0x50,0x307,0x0}, 
+  {0x1E57,0x70,0x307,0x0}, {0x1E58,0x52,0x307,0x0}, {0x1E59,0x72,0x307,0x0}, 
+  {0x1E5A,0x52,0x323,0x0}, {0x1E5B,0x72,0x323,0x0}, {0x1E5C,0x1E5A,0x304,0x0}, 
+  {0x1E5D,0x1E5B,0x304,0x0}, {0x1E5E,0x52,0x331,0x0}, {0x1E5F,0x72,0x331,0x0}, 
+  {0x1E60,0x53,0x307,0x0}, {0x1E61,0x73,0x307,0x0}, {0x1E62,0x53,0x323,0x0}, 
+  {0x1E63,0x73,0x323,0x0}, {0x1E64,0x15A,0x307,0x0}, {0x1E65,0x15B,0x307,0x0}, 
+  {0x1E66,0x160,0x307,0x0}, {0x1E67,0x161,0x307,0x0}, {0x1E68,0x1E62,0x307,0x0}, 
+  {0x1E69,0x1E63,0x307,0x0}, {0x1E6A,0x54,0x307,0x0}, {0x1E6B,0x74,0x307,0x0}, 
+  {0x1E6C,0x54,0x323,0x0}, {0x1E6D,0x74,0x323,0x0}, {0x1E6E,0x54,0x331,0x0}, 
+  {0x1E6F,0x74,0x331,0x0}, {0x1E70,0x54,0x32D,0x0}, {0x1E71,0x74,0x32D,0x0}, 
+  {0x1E72,0x55,0x324,0x0}, {0x1E73,0x75,0x324,0x0}, {0x1E74,0x55,0x330,0x0}, 
+  {0x1E75,0x75,0x330,0x0}, {0x1E76,0x55,0x32D,0x0}, {0x1E77,0x75,0x32D,0x0}, 
+  {0x1E78,0x168,0x301,0x0}, {0x1E79,0x169,0x301,0x0}, {0x1E7A,0x16A,0x308,0x0}, 
+  {0x1E7B,0x16B,0x308,0x0}, {0x1E7C,0x56,0x303,0x0}, {0x1E7D,0x76,0x303,0x0}, 
+  {0x1E7E,0x56,0x323,0x0}, {0x1E7F,0x76,0x323,0x0}, {0x1E80,0x57,0x300,0x0}, 
+  {0x1E81,0x77,0x300,0x0}, {0x1E82,0x57,0x301,0x0}, {0x1E83,0x77,0x301,0x0}, 
+  {0x1E84,0x57,0x308,0x0}, {0x1E85,0x77,0x308,0x0}, {0x1E86,0x57,0x307,0x0}, 
+  {0x1E87,0x77,0x307,0x0}, {0x1E88,0x57,0x323,0x0}, {0x1E89,0x77,0x323,0x0}, 
+  {0x1E8A,0x58,0x307,0x0}, {0x1E8B,0x78,0x307,0x0}, {0x1E8C,0x58,0x308,0x0}, 
+  {0x1E8D,0x78,0x308,0x0}, {0x1E8E,0x59,0x307,0x0}, {0x1E8F,0x79,0x307,0x0}, 
+  {0x1E90,0x5A,0x302,0x0}, {0x1E91,0x7A,0x302,0x0}, {0x1E92,0x5A,0x323,0x0}, 
+  {0x1E93,0x7A,0x323,0x0}, {0x1E94,0x5A,0x331,0x0}, {0x1E95,0x7A,0x331,0x0}, 
+  {0x1E96,0x68,0x331,0x0}, {0x1E97,0x74,0x308,0x0}, {0x1E98,0x77,0x30A,0x0}, 
+  {0x1E99,0x79,0x30A,0x0}, {0x1E9B,0x17F,0x307,0x0}, {0x1EA0,0x41,0x323,0x0}, 
+  {0x1EA1,0x61,0x323,0x0}, {0x1EA2,0x41,0x309,0x0}, {0x1EA3,0x61,0x309,0x0}, 
+  {0x1EA4,0xC2,0x301,0x0}, {0x1EA5,0xE2,0x301,0x0}, {0x1EA6,0xC2,0x300,0x0}, 
+  {0x1EA7,0xE2,0x300,0x0}, {0x1EA8,0xC2,0x309,0x0}, {0x1EA9,0xE2,0x309,0x0}, 
+  {0x1EAA,0xC2,0x303,0x0}, {0x1EAB,0xE2,0x303,0x0}, {0x1EAC,0xC2,0x323,0x0}, 
+  {0x1EAD,0xE2,0x323,0x0}, {0x1EAE,0x102,0x301,0x0}, {0x1EAF,0x103,0x301,0x0}, 
+  {0x1EB0,0x102,0x300,0x0}, {0x1EB1,0x103,0x300,0x0}, {0x1EB2,0x102,0x309,0x0}, 
+  {0x1EB3,0x103,0x309,0x0}, {0x1EB4,0x102,0x303,0x0}, {0x1EB5,0x103,0x303,0x0}, 
+  {0x1EB6,0x102,0x323,0x0}, {0x1EB7,0x103,0x323,0x0}, {0x1EB8,0x45,0x323,0x0}, 
+  {0x1EB9,0x65,0x323,0x0}, {0x1EBA,0x45,0x309,0x0}, {0x1EBB,0x65,0x309,0x0}, 
+  {0x1EBC,0x45,0x303,0x0}, {0x1EBD,0x65,0x303,0x0}, {0x1EBE,0xCA,0x301,0x0}, 
+  {0x1EBF,0xEA,0x301,0x0}, {0x1EC0,0xCA,0x300,0x0}, {0x1EC1,0xEA,0x300,0x0}, 
+  {0x1EC2,0xCA,0x309,0x0}, {0x1EC3,0xEA,0x309,0x0}, {0x1EC4,0xCA,0x303,0x0}, 
+  {0x1EC5,0xEA,0x303,0x0}, {0x1EC6,0xCA,0x323,0x0}, {0x1EC7,0xEA,0x323,0x0}, 
+  {0x1EC8,0x49,0x309,0x0}, {0x1EC9,0x69,0x309,0x0}, {0x1ECA,0x49,0x323,0x0}, 
+  {0x1ECB,0x69,0x323,0x0}, {0x1ECC,0x4F,0x323,0x0}, {0x1ECD,0x6F,0x323,0x0}, 
+  {0x1ECE,0x4F,0x309,0x0}, {0x1ECF,0x6F,0x309,0x0}, {0x1ED0,0xD4,0x301,0x0}, 
+  {0x1ED1,0xF4,0x301,0x0}, {0x1ED2,0xD4,0x300,0x0}, {0x1ED3,0xF4,0x300,0x0}, 
+  {0x1ED4,0xD4,0x309,0x0}, {0x1ED5,0xF4,0x309,0x0}, {0x1ED6,0xD4,0x303,0x0}, 
+  {0x1ED7,0xF4,0x303,0x0}, {0x1ED8,0xD4,0x323,0x0}, {0x1ED9,0xF4,0x323,0x0}, 
+  {0x1EDA,0x1A0,0x301,0x0}, {0x1EDB,0x1A1,0x301,0x0}, {0x1EDC,0x1A0,0x300,0x0}, 
+  {0x1EDD,0x1A1,0x300,0x0}, {0x1EDE,0x1A0,0x309,0x0}, {0x1EDF,0x1A1,0x309,0x0}, 
+  {0x1EE0,0x1A0,0x303,0x0}, {0x1EE1,0x1A1,0x303,0x0}, {0x1EE2,0x1A0,0x323,0x0}, 
+  {0x1EE3,0x1A1,0x323,0x0}, {0x1EE4,0x55,0x323,0x0}, {0x1EE5,0x75,0x323,0x0}, 
+  {0x1EE6,0x55,0x309,0x0}, {0x1EE7,0x75,0x309,0x0}, {0x1EE8,0x1AF,0x301,0x0}, 
+  {0x1EE9,0x1B0,0x301,0x0}, {0x1EEA,0x1AF,0x300,0x0}, {0x1EEB,0x1B0,0x300,0x0}, 
+  {0x1EEC,0x1AF,0x309,0x0}, {0x1EED,0x1B0,0x309,0x0}, {0x1EEE,0x1AF,0x303,0x0}, 
+  {0x1EEF,0x1B0,0x303,0x0}, {0x1EF0,0x1AF,0x323,0x0}, {0x1EF1,0x1B0,0x323,0x0}, 
+  {0x1EF2,0x59,0x300,0x0}, {0x1EF3,0x79,0x300,0x0}, {0x1EF4,0x59,0x323,0x0}, 
+  {0x1EF5,0x79,0x323,0x0}, {0x1EF6,0x59,0x309,0x0}, {0x1EF7,0x79,0x309,0x0}, 
+  {0x1EF8,0x59,0x303,0x0}, {0x1EF9,0x79,0x303,0x0}, {0x1F00,0x3B1,0x313,0x0}, 
+  {0x1F01,0x3B1,0x314,0x0}, {0x1F02,0x1F00,0x300,0x0}, {0x1F03,0x1F01,0x300,0x0}, 
+  {0x1F04,0x1F00,0x301,0x0}, {0x1F05,0x1F01,0x301,0x0}, {0x1F06,0x1F00,0x342,0x0}, 
+  {0x1F07,0x1F01,0x342,0x0}, {0x1F08,0x391,0x313,0x0}, {0x1F09,0x391,0x314,0x0}, 
+  {0x1F0A,0x1F08,0x300,0x0}, {0x1F0B,0x1F09,0x300,0x0}, {0x1F0C,0x1F08,0x301,0x0}, 
+  {0x1F0D,0x1F09,0x301,0x0}, {0x1F0E,0x1F08,0x342,0x0}, {0x1F0F,0x1F09,0x342,0x0}, 
+  {0x1F10,0x3B5,0x313,0x0}, {0x1F11,0x3B5,0x314,0x0}, {0x1F12,0x1F10,0x300,0x0}, 
+  {0x1F13,0x1F11,0x300,0x0}, {0x1F14,0x1F10,0x301,0x0}, {0x1F15,0x1F11,0x301,0x0}, 
+  {0x1F18,0x395,0x313,0x0}, {0x1F19,0x395,0x314,0x0}, {0x1F1A,0x1F18,0x300,0x0}, 
+  {0x1F1B,0x1F19,0x300,0x0}, {0x1F1C,0x1F18,0x301,0x0}, {0x1F1D,0x1F19,0x301,0x0}, 
+  {0x1F20,0x3B7,0x313,0x0}, {0x1F21,0x3B7,0x314,0x0}, {0x1F22,0x1F20,0x300,0x0}, 
+  {0x1F23,0x1F21,0x300,0x0}, {0x1F24,0x1F20,0x301,0x0}, {0x1F25,0x1F21,0x301,0x0}, 
+  {0x1F26,0x1F20,0x342,0x0}, {0x1F27,0x1F21,0x342,0x0}, {0x1F28,0x397,0x313,0x0}, 
+  {0x1F29,0x397,0x314,0x0}, {0x1F2A,0x1F28,0x300,0x0}, {0x1F2B,0x1F29,0x300,0x0}, 
+  {0x1F2C,0x1F28,0x301,0x0}, {0x1F2D,0x1F29,0x301,0x0}, {0x1F2E,0x1F28,0x342,0x0}, 
+  {0x1F2F,0x1F29,0x342,0x0}, {0x1F30,0x3B9,0x313,0x0}, {0x1F31,0x3B9,0x314,0x0}, 
+  {0x1F32,0x1F30,0x300,0x0}, {0x1F33,0x1F31,0x300,0x0}, {0x1F34,0x1F30,0x301,0x0}, 
+  {0x1F35,0x1F31,0x301,0x0}, {0x1F36,0x1F30,0x342,0x0}, {0x1F37,0x1F31,0x342,0x0}, 
+  {0x1F38,0x399,0x313,0x0}, {0x1F39,0x399,0x314,0x0}, {0x1F3A,0x1F38,0x300,0x0}, 
+  {0x1F3B,0x1F39,0x300,0x0}, {0x1F3C,0x1F38,0x301,0x0}, {0x1F3D,0x1F39,0x301,0x0}, 
+  {0x1F3E,0x1F38,0x342,0x0}, {0x1F3F,0x1F39,0x342,0x0}, {0x1F40,0x3BF,0x313,0x0}, 
+  {0x1F41,0x3BF,0x314,0x0}, {0x1F42,0x1F40,0x300,0x0}, {0x1F43,0x1F41,0x300,0x0}, 
+  {0x1F44,0x1F40,0x301,0x0}, {0x1F45,0x1F41,0x301,0x0}, {0x1F48,0x39F,0x313,0x0}, 
+  {0x1F49,0x39F,0x314,0x0}, {0x1F4A,0x1F48,0x300,0x0}, {0x1F4B,0x1F49,0x300,0x0}, 
+  {0x1F4C,0x1F48,0x301,0x0}, {0x1F4D,0x1F49,0x301,0x0}, {0x1F50,0x3C5,0x313,0x0}, 
+  {0x1F51,0x3C5,0x314,0x0}, {0x1F52,0x1F50,0x300,0x0}, {0x1F53,0x1F51,0x300,0x0}, 
+  {0x1F54,0x1F50,0x301,0x0}, {0x1F55,0x1F51,0x301,0x0}, {0x1F56,0x1F50,0x342,0x0}, 
+  {0x1F57,0x1F51,0x342,0x0}, {0x1F59,0x3A5,0x314,0x0}, {0x1F5B,0x1F59,0x300,0x0}, 
+  {0x1F5D,0x1F59,0x301,0x0}, {0x1F5F,0x1F59,0x342,0x0}, {0x1F60,0x3C9,0x313,0x0}, 
+  {0x1F61,0x3C9,0x314,0x0}, {0x1F62,0x1F60,0x300,0x0}, {0x1F63,0x1F61,0x300,0x0}, 
+  {0x1F64,0x1F60,0x301,0x0}, {0x1F65,0x1F61,0x301,0x0}, {0x1F66,0x1F60,0x342,0x0}, 
+  {0x1F67,0x1F61,0x342,0x0}, {0x1F68,0x3A9,0x313,0x0}, {0x1F69,0x3A9,0x314,0x0}, 
+  {0x1F6A,0x1F68,0x300,0x0}, {0x1F6B,0x1F69,0x300,0x0}, {0x1F6C,0x1F68,0x301,0x0}, 
+  {0x1F6D,0x1F69,0x301,0x0}, {0x1F6E,0x1F68,0x342,0x0}, {0x1F6F,0x1F69,0x342,0x0}, 
+  {0x1F70,0x3B1,0x300,0x0}, {0x1F71,0x3B1,0x301,0x0}, {0x1F72,0x3B5,0x300,0x0}, 
+  {0x1F73,0x3B5,0x301,0x0}, {0x1F74,0x3B7,0x300,0x0}, {0x1F75,0x3B7,0x301,0x0}, 
+  {0x1F76,0x3B9,0x300,0x0}, {0x1F77,0x3B9,0x301,0x0}, {0x1F78,0x3BF,0x300,0x0}, 
+  {0x1F79,0x3BF,0x301,0x0}, {0x1F7A,0x3C5,0x300,0x0}, {0x1F7B,0x3C5,0x301,0x0}, 
+  {0x1F7C,0x3C9,0x300,0x0}, {0x1F7D,0x3C9,0x301,0x0}, {0x1F80,0x1F00,0x345,0x0}, 
+  {0x1F81,0x1F01,0x345,0x0}, {0x1F82,0x1F02,0x345,0x0}, {0x1F83,0x1F03,0x345,0x0}, 
+  {0x1F84,0x1F04,0x345,0x0}, {0x1F85,0x1F05,0x345,0x0}, {0x1F86,0x1F06,0x345,0x0}, 
+  {0x1F87,0x1F07,0x345,0x0}, {0x1F88,0x1F08,0x345,0x0}, {0x1F89,0x1F09,0x345,0x0}, 
+  {0x1F8A,0x1F0A,0x345,0x0}, {0x1F8B,0x1F0B,0x345,0x0}, {0x1F8C,0x1F0C,0x345,0x0}, 
+  {0x1F8D,0x1F0D,0x345,0x0}, {0x1F8E,0x1F0E,0x345,0x0}, {0x1F8F,0x1F0F,0x345,0x0}, 
+  {0x1F90,0x1F20,0x345,0x0}, {0x1F91,0x1F21,0x345,0x0}, {0x1F92,0x1F22,0x345,0x0}, 
+  {0x1F93,0x1F23,0x345,0x0}, {0x1F94,0x1F24,0x345,0x0}, {0x1F95,0x1F25,0x345,0x0}, 
+  {0x1F96,0x1F26,0x345,0x0}, {0x1F97,0x1F27,0x345,0x0}, {0x1F98,0x1F28,0x345,0x0}, 
+  {0x1F99,0x1F29,0x345,0x0}, {0x1F9A,0x1F2A,0x345,0x0}, {0x1F9B,0x1F2B,0x345,0x0}, 
+  {0x1F9C,0x1F2C,0x345,0x0}, {0x1F9D,0x1F2D,0x345,0x0}, {0x1F9E,0x1F2E,0x345,0x0}, 
+  {0x1F9F,0x1F2F,0x345,0x0}, {0x1FA0,0x1F60,0x345,0x0}, {0x1FA1,0x1F61,0x345,0x0}, 
+  {0x1FA2,0x1F62,0x345,0x0}, {0x1FA3,0x1F63,0x345,0x0}, {0x1FA4,0x1F64,0x345,0x0}, 
+  {0x1FA5,0x1F65,0x345,0x0}, {0x1FA6,0x1F66,0x345,0x0}, {0x1FA7,0x1F67,0x345,0x0}, 
+  {0x1FA8,0x1F68,0x345,0x0}, {0x1FA9,0x1F69,0x345,0x0}, {0x1FAA,0x1F6A,0x345,0x0}, 
+  {0x1FAB,0x1F6B,0x345,0x0}, {0x1FAC,0x1F6C,0x345,0x0}, {0x1FAD,0x1F6D,0x345,0x0}, 
+  {0x1FAE,0x1F6E,0x345,0x0}, {0x1FAF,0x1F6F,0x345,0x0}, {0x1FB0,0x3B1,0x306,0x0}, 
+  {0x1FB1,0x3B1,0x304,0x0}, {0x1FB2,0x1F70,0x345,0x0}, {0x1FB3,0x3B1,0x345,0x0}, 
+  {0x1FB4,0x1F71,0x345,0x0}, {0x1FB6,0x3B1,0x342,0x0}, {0x1FB7,0x1FB6,0x345,0x0}, 
+  {0x1FB8,0x391,0x306,0x0}, {0x1FB9,0x391,0x304,0x0}, {0x1FBA,0x391,0x300,0x0}, 
+  {0x1FBB,0x391,0x301,0x0}, {0x1FBC,0x391,0x345,0x0}, {0x1FBE,0x3B9,0x0,0x0}, 
+  {0x1FC1,0xA8,0x342,0x0}, {0x1FC2,0x1F74,0x345,0x0}, {0x1FC3,0x3B7,0x345,0x0}, 
+  {0x1FC4,0x1F75,0x345,0x0}, {0x1FC6,0x3B7,0x342,0x0}, {0x1FC7,0x1FC6,0x345,0x0}, 
+  {0x1FC8,0x395,0x300,0x0}, {0x1FC9,0x395,0x301,0x0}, {0x1FCA,0x397,0x300,0x0}, 
+  {0x1FCB,0x397,0x301,0x0}, {0x1FCC,0x397,0x345,0x0}, {0x1FCD,0x1FBF,0x300,0x0}, 
+  {0x1FCE,0x1FBF,0x301,0x0}, {0x1FCF,0x1FBF,0x342,0x0}, {0x1FD0,0x3B9,0x306,0x0}, 
+  {0x1FD1,0x3B9,0x304,0x0}, {0x1FD2,0x3CA,0x300,0x0}, {0x1FD3,0x3CA,0x301,0x0}, 
+  {0x1FD6,0x3B9,0x342,0x0}, {0x1FD7,0x3CA,0x342,0x0}, {0x1FD8,0x399,0x306,0x0}, 
+  {0x1FD9,0x399,0x304,0x0}, {0x1FDA,0x399,0x300,0x0}, {0x1FDB,0x399,0x301,0x0}, 
+  {0x1FDD,0x1FFE,0x300,0x0}, {0x1FDE,0x1FFE,0x301,0x0}, {0x1FDF,0x1FFE,0x342,0x0}, 
+  {0x1FE0,0x3C5,0x306,0x0}, {0x1FE1,0x3C5,0x304,0x0}, {0x1FE2,0x3CB,0x300,0x0}, 
+  {0x1FE3,0x3CB,0x301,0x0}, {0x1FE4,0x3C1,0x313,0x0}, {0x1FE5,0x3C1,0x314,0x0}, 
+  {0x1FE6,0x3C5,0x342,0x0}, {0x1FE7,0x3CB,0x342,0x0}, {0x1FE8,0x3A5,0x306,0x0}, 
+  {0x1FE9,0x3A5,0x304,0x0}, {0x1FEA,0x3A5,0x300,0x0}, {0x1FEB,0x3A5,0x301,0x0}, 
+  {0x1FEC,0x3A1,0x314,0x0}, {0x1FED,0xA8,0x300,0x0}, {0x1FEE,0xA8,0x301,0x0}, 
+  {0x1FEF,0x60,0x0,0x0}, {0x1FF2,0x1F7C,0x345,0x0}, {0x1FF3,0x3C9,0x345,0x0}, 
+  {0x1FF4,0x1F79,0x345,0x0}, {0x1FF6,0x3C9,0x342,0x0}, {0x1FF7,0x1FF6,0x345,0x0}, 
+  {0x1FF8,0x39F,0x300,0x0}, {0x1FF9,0x39F,0x301,0x0}, {0x1FFA,0x3A9,0x300,0x0}, 
+  {0x1FFB,0x3A9,0x301,0x0}, {0x1FFC,0x3A9,0x345,0x0}, {0x1FFD,0xB4,0x0,0x0}, 
+  {0x2000,0x2002,0x0,0x0}, {0x2001,0x2003,0x0,0x0}, {0x2126,0x3A9,0x0,0x0}, 
+  {0x212A,0x4B,0x0,0x0}, {0x212B,0xC5,0x0,0x0}, {0x2204,0x2203,0x338,0x0}, 
+  {0x2209,0x2208,0x338,0x0}, {0x220C,0x220B,0x338,0x0}, {0x2224,0x2223,0x338,0x0}, 
+  {0x2226,0x2225,0x338,0x0}, {0x2241,0x7E,0x338,0x0}, {0x2244,0x2243,0x338,0x0}, 
+  {0x2247,0x2245,0x338,0x0}, {0x2249,0x2248,0x338,0x0}, {0x2260,0x3D,0x338,0x0}, 
+  {0x2262,0x2261,0x338,0x0}, {0x226D,0x224D,0x338,0x0}, {0x226E,0x3C,0x338,0x0}, 
+  {0x226F,0x3E,0x338,0x0}, {0x2270,0x2264,0x338,0x0}, {0x2271,0x2265,0x338,0x0}, 
+  {0x2274,0x2272,0x338,0x0}, {0x2275,0x2273,0x338,0x0}, {0x2278,0x2276,0x338,0x0}, 
+  {0x2279,0x2277,0x338,0x0}, {0x2280,0x227A,0x338,0x0}, {0x2281,0x227B,0x338,0x0}, 
+  {0x2284,0x2282,0x338,0x0}, {0x2285,0x2283,0x338,0x0}, {0x2288,0x2286,0x338,0x0}, 
+  {0x2289,0x2287,0x338,0x0}, {0x22AC,0x22A2,0x338,0x0}, {0x22AD,0x22A8,0x338,0x0}, 
+  {0x22AE,0x22A9,0x338,0x0}, {0x22AF,0x22AB,0x338,0x0}, {0x22E0,0x227C,0x338,0x0}, 
+  {0x22E1,0x227D,0x338,0x0}, {0x22E2,0x2291,0x338,0x0}, {0x22E3,0x2292,0x338,0x0}, 
+  {0x22EA,0x22B2,0x338,0x0}, {0x22EB,0x22B3,0x338,0x0}, {0x22EC,0x22B4,0x338,0x0}, 
+  {0x22ED,0x22B5,0x338,0x0}, {0x2329,0x3008,0x0,0x0}, {0x232A,0x3009,0x0,0x0}, 
+  {0x304C,0x304B,0x3099,0x0}, {0x304E,0x304D,0x3099,0x0}, {0x3050,0x304F,0x3099,0x0}, 
+  {0x3052,0x3051,0x3099,0x0}, {0x3054,0x3053,0x3099,0x0}, {0x3056,0x3055,0x3099,0x0}, 
+  {0x3058,0x3057,0x3099,0x0}, {0x305A,0x3059,0x3099,0x0}, {0x305C,0x305B,0x3099,0x0}, 
+  {0x305E,0x305D,0x3099,0x0}, {0x3060,0x305F,0x3099,0x0}, {0x3062,0x3061,0x3099,0x0}, 
+  {0x3065,0x3064,0x3099,0x0}, {0x3067,0x3066,0x3099,0x0}, {0x3069,0x3068,0x3099,0x0}, 
+  {0x3070,0x306F,0x3099,0x0}, {0x3071,0x306F,0x309A,0x0}, {0x3073,0x3072,0x3099,0x0}, 
+  {0x3074,0x3072,0x309A,0x0}, {0x3076,0x3075,0x3099,0x0}, {0x3077,0x3075,0x309A,0x0}, 
+  {0x3079,0x3078,0x3099,0x0}, {0x307A,0x3078,0x309A,0x0}, {0x307C,0x307B,0x3099,0x0}, 
+  {0x307D,0x307B,0x309A,0x0}, {0x3094,0x3046,0x3099,0x0}, {0x309E,0x309D,0x3099,0x0}, 
+  {0x30AC,0x30AB,0x3099,0x0}, {0x30AE,0x30AD,0x3099,0x0}, {0x30B0,0x30AF,0x3099,0x0}, 
+  {0x30B2,0x30B1,0x3099,0x0}, {0x30B4,0x30B3,0x3099,0x0}, {0x30B6,0x30B5,0x3099,0x0}, 
+  {0x30B8,0x30B7,0x3099,0x0}, {0x30BA,0x30B9,0x3099,0x0}, {0x30BC,0x30BB,0x3099,0x0}, 
+  {0x30BE,0x30BD,0x3099,0x0}, {0x30C0,0x30BF,0x3099,0x0}, {0x30C2,0x30C1,0x3099,0x0}, 
+  {0x30C5,0x30C4,0x3099,0x0}, {0x30C7,0x30C6,0x3099,0x0}, {0x30C9,0x30C8,0x3099,0x0}, 
+  {0x30D0,0x30CF,0x3099,0x0}, {0x30D1,0x30CF,0x309A,0x0}, {0x30D3,0x30D2,0x3099,0x0}, 
+  {0x30D4,0x30D2,0x309A,0x0}, {0x30D6,0x30D5,0x3099,0x0}, {0x30D7,0x30D5,0x309A,0x0}, 
+  {0x30D9,0x30D8,0x3099,0x0}, {0x30DA,0x30D8,0x309A,0x0}, {0x30DC,0x30DB,0x3099,0x0}, 
+  {0x30DD,0x30DB,0x309A,0x0}, {0x30F4,0x30A6,0x3099,0x0}, {0x30F7,0x30EF,0x3099,0x0}, 
+  {0x30F8,0x30F0,0x3099,0x0}, {0x30F9,0x30F1,0x3099,0x0}, {0x30FA,0x30F2,0x3099,0x0}, 
+  {0x30FE,0x30FD,0x3099,0x0}, {0xF900,0x8C48,0x0,0x0}, {0xF901,0x66F4,0x0,0x0}, 
+  {0xF902,0x8ECA,0x0,0x0}, {0xF903,0x8CC8,0x0,0x0}, {0xF904,0x6ED1,0x0,0x0}, 
+  {0xF905,0x4E32,0x0,0x0}, {0xF906,0x53E5,0x0,0x0}, {0xF907,0x9F9C,0x0,0x0}, 
+  {0xF908,0x9F9C,0x0,0x0}, {0xF909,0x5951,0x0,0x0}, {0xF90A,0x91D1,0x0,0x0}, 
+  {0xF90B,0x5587,0x0,0x0}, {0xF90C,0x5948,0x0,0x0}, {0xF90D,0x61F6,0x0,0x0}, 
+  {0xF90E,0x7669,0x0,0x0}, {0xF90F,0x7F85,0x0,0x0}, {0xF910,0x863F,0x0,0x0}, 
+  {0xF911,0x87BA,0x0,0x0}, {0xF912,0x88F8,0x0,0x0}, {0xF913,0x908F,0x0,0x0}, 
+  {0xF914,0x6A02,0x0,0x0}, {0xF915,0x6D1B,0x0,0x0}, {0xF916,0x70D9,0x0,0x0}, 
+  {0xF917,0x73DE,0x0,0x0}, {0xF918,0x843D,0x0,0x0}, {0xF919,0x916A,0x0,0x0}, 
+  {0xF91A,0x99F1,0x0,0x0}, {0xF91B,0x4E82,0x0,0x0}, {0xF91C,0x5375,0x0,0x0}, 
+  {0xF91D,0x6B04,0x0,0x0}, {0xF91E,0x721B,0x0,0x0}, {0xF91F,0x862D,0x0,0x0}, 
+  {0xF920,0x9E1E,0x0,0x0}, {0xF921,0x5D50,0x0,0x0}, {0xF922,0x6FEB,0x0,0x0}, 
+  {0xF923,0x85CD,0x0,0x0}, {0xF924,0x8964,0x0,0x0}, {0xF925,0x62C9,0x0,0x0}, 
+  {0xF926,0x81D8,0x0,0x0}, {0xF927,0x881F,0x0,0x0}, {0xF928,0x5ECA,0x0,0x0}, 
+  {0xF929,0x6717,0x0,0x0}, {0xF92A,0x6D6A,0x0,0x0}, {0xF92B,0x72FC,0x0,0x0}, 
+  {0xF92C,0x90CE,0x0,0x0}, {0xF92D,0x4F86,0x0,0x0}, {0xF92E,0x51B7,0x0,0x0}, 
+  {0xF92F,0x52DE,0x0,0x0}, {0xF930,0x64C4,0x0,0x0}, {0xF931,0x6AD3,0x0,0x0}, 
+  {0xF932,0x7210,0x0,0x0}, {0xF933,0x76E7,0x0,0x0}, {0xF934,0x8001,0x0,0x0}, 
+  {0xF935,0x8606,0x0,0x0}, {0xF936,0x865C,0x0,0x0}, {0xF937,0x8DEF,0x0,0x0}, 
+  {0xF938,0x9732,0x0,0x0}, {0xF939,0x9B6F,0x0,0x0}, {0xF93A,0x9DFA,0x0,0x0}, 
+  {0xF93B,0x788C,0x0,0x0}, {0xF93C,0x797F,0x0,0x0}, {0xF93D,0x7DA0,0x0,0x0}, 
+  {0xF93E,0x83C9,0x0,0x0}, {0xF93F,0x9304,0x0,0x0}, {0xF940,0x9E7F,0x0,0x0}, 
+  {0xF941,0x8AD6,0x0,0x0}, {0xF942,0x58DF,0x0,0x0}, {0xF943,0x5F04,0x0,0x0}, 
+  {0xF944,0x7C60,0x0,0x0}, {0xF945,0x807E,0x0,0x0}, {0xF946,0x7262,0x0,0x0}, 
+  {0xF947,0x78CA,0x0,0x0}, {0xF948,0x8CC2,0x0,0x0}, {0xF949,0x96F7,0x0,0x0}, 
+  {0xF94A,0x58D8,0x0,0x0}, {0xF94B,0x5C62,0x0,0x0}, {0xF94C,0x6A13,0x0,0x0}, 
+  {0xF94D,0x6DDA,0x0,0x0}, {0xF94E,0x6F0F,0x0,0x0}, {0xF94F,0x7D2F,0x0,0x0}, 
+  {0xF950,0x7E37,0x0,0x0}, {0xF951,0x96FB,0x0,0x0}, {0xF952,0x52D2,0x0,0x0}, 
+  {0xF953,0x808B,0x0,0x0}, {0xF954,0x51DC,0x0,0x0}, {0xF955,0x51CC,0x0,0x0}, 
+  {0xF956,0x7A1C,0x0,0x0}, {0xF957,0x7DBE,0x0,0x0}, {0xF958,0x83F1,0x0,0x0}, 
+  {0xF959,0x9675,0x0,0x0}, {0xF95A,0x8B80,0x0,0x0}, {0xF95B,0x62CF,0x0,0x0}, 
+  {0xF95C,0x6A02,0x0,0x0}, {0xF95D,0x8AFE,0x0,0x0}, {0xF95E,0x4E39,0x0,0x0}, 
+  {0xF95F,0x5BE7,0x0,0x0}, {0xF960,0x6012,0x0,0x0}, {0xF961,0x7387,0x0,0x0}, 
+  {0xF962,0x7570,0x0,0x0}, {0xF963,0x5317,0x0,0x0}, {0xF964,0x78FB,0x0,0x0}, 
+  {0xF965,0x4FBF,0x0,0x0}, {0xF966,0x5FA9,0x0,0x0}, {0xF967,0x4E0D,0x0,0x0}, 
+  {0xF968,0x6CCC,0x0,0x0}, {0xF969,0x6578,0x0,0x0}, {0xF96A,0x7D22,0x0,0x0}, 
+  {0xF96B,0x53C3,0x0,0x0}, {0xF96C,0x585E,0x0,0x0}, {0xF96D,0x7701,0x0,0x0}, 
+  {0xF96E,0x8449,0x0,0x0}, {0xF96F,0x8AAA,0x0,0x0}, {0xF970,0x6BBA,0x0,0x0}, 
+  {0xF971,0x8FB0,0x0,0x0}, {0xF972,0x6C88,0x0,0x0}, {0xF973,0x62FE,0x0,0x0}, 
+  {0xF974,0x82E5,0x0,0x0}, {0xF975,0x63A0,0x0,0x0}, {0xF976,0x7565,0x0,0x0}, 
+  {0xF977,0x4EAE,0x0,0x0}, {0xF978,0x5169,0x0,0x0}, {0xF979,0x51C9,0x0,0x0}, 
+  {0xF97A,0x6881,0x0,0x0}, {0xF97B,0x7CE7,0x0,0x0}, {0xF97C,0x826F,0x0,0x0}, 
+  {0xF97D,0x8AD2,0x0,0x0}, {0xF97E,0x91CF,0x0,0x0}, {0xF97F,0x52F5,0x0,0x0}, 
+  {0xF980,0x5442,0x0,0x0}, {0xF981,0x5973,0x0,0x0}, {0xF982,0x5EEC,0x0,0x0}, 
+  {0xF983,0x65C5,0x0,0x0}, {0xF984,0x6FFE,0x0,0x0}, {0xF985,0x792A,0x0,0x0}, 
+  {0xF986,0x95AD,0x0,0x0}, {0xF987,0x9A6A,0x0,0x0}, {0xF988,0x9E97,0x0,0x0}, 
+  {0xF989,0x9ECE,0x0,0x0}, {0xF98A,0x529B,0x0,0x0}, {0xF98B,0x66C6,0x0,0x0}, 
+  {0xF98C,0x6B77,0x0,0x0}, {0xF98D,0x8F62,0x0,0x0}, {0xF98E,0x5E74,0x0,0x0}, 
+  {0xF98F,0x6190,0x0,0x0}, {0xF990,0x6200,0x0,0x0}, {0xF991,0x649A,0x0,0x0}, 
+  {0xF992,0x6F23,0x0,0x0}, {0xF993,0x7149,0x0,0x0}, {0xF994,0x7489,0x0,0x0}, 
+  {0xF995,0x79CA,0x0,0x0}, {0xF996,0x7DF4,0x0,0x0}, {0xF997,0x806F,0x0,0x0}, 
+  {0xF998,0x8F26,0x0,0x0}, {0xF999,0x84EE,0x0,0x0}, {0xF99A,0x9023,0x0,0x0}, 
+  {0xF99B,0x934A,0x0,0x0}, {0xF99C,0x5217,0x0,0x0}, {0xF99D,0x52A3,0x0,0x0}, 
+  {0xF99E,0x54BD,0x0,0x0}, {0xF99F,0x70C8,0x0,0x0}, {0xF9A0,0x88C2,0x0,0x0}, 
+  {0xF9A1,0x8AAA,0x0,0x0}, {0xF9A2,0x5EC9,0x0,0x0}, {0xF9A3,0x5FF5,0x0,0x0}, 
+  {0xF9A4,0x637B,0x0,0x0}, {0xF9A5,0x6BAE,0x0,0x0}, {0xF9A6,0x7C3E,0x0,0x0}, 
+  {0xF9A7,0x7375,0x0,0x0}, {0xF9A8,0x4EE4,0x0,0x0}, {0xF9A9,0x56F9,0x0,0x0}, 
+  {0xF9AA,0x5BE7,0x0,0x0}, {0xF9AB,0x5DBA,0x0,0x0}, {0xF9AC,0x601C,0x0,0x0}, 
+  {0xF9AD,0x73B2,0x0,0x0}, {0xF9AE,0x7469,0x0,0x0}, {0xF9AF,0x7F9A,0x0,0x0}, 
+  {0xF9B0,0x8046,0x0,0x0}, {0xF9B1,0x9234,0x0,0x0}, {0xF9B2,0x96F6,0x0,0x0}, 
+  {0xF9B3,0x9748,0x0,0x0}, {0xF9B4,0x9818,0x0,0x0}, {0xF9B5,0x4F8B,0x0,0x0}, 
+  {0xF9B6,0x79AE,0x0,0x0}, {0xF9B7,0x91B4,0x0,0x0}, {0xF9B8,0x96B8,0x0,0x0}, 
+  {0xF9B9,0x60E1,0x0,0x0}, {0xF9BA,0x4E86,0x0,0x0}, {0xF9BB,0x50DA,0x0,0x0}, 
+  {0xF9BC,0x5BEE,0x0,0x0}, {0xF9BD,0x5C3F,0x0,0x0}, {0xF9BE,0x6599,0x0,0x0}, 
+  {0xF9BF,0x6A02,0x0,0x0}, {0xF9C0,0x71CE,0x0,0x0}, {0xF9C1,0x7642,0x0,0x0}, 
+  {0xF9C2,0x84FC,0x0,0x0}, {0xF9C3,0x907C,0x0,0x0}, {0xF9C4,0x9F8D,0x0,0x0}, 
+  {0xF9C5,0x6688,0x0,0x0}, {0xF9C6,0x962E,0x0,0x0}, {0xF9C7,0x5289,0x0,0x0}, 
+  {0xF9C8,0x677B,0x0,0x0}, {0xF9C9,0x67F3,0x0,0x0}, {0xF9CA,0x6D41,0x0,0x0}, 
+  {0xF9CB,0x6E9C,0x0,0x0}, {0xF9CC,0x7409,0x0,0x0}, {0xF9CD,0x7559,0x0,0x0}, 
+  {0xF9CE,0x786B,0x0,0x0}, {0xF9CF,0x7D10,0x0,0x0}, {0xF9D0,0x985E,0x0,0x0}, 
+  {0xF9D1,0x516D,0x0,0x0}, {0xF9D2,0x622E,0x0,0x0}, {0xF9D3,0x9678,0x0,0x0}, 
+  {0xF9D4,0x502B,0x0,0x0}, {0xF9D5,0x5D19,0x0,0x0}, {0xF9D6,0x6DEA,0x0,0x0}, 
+  {0xF9D7,0x8F2A,0x0,0x0}, {0xF9D8,0x5F8B,0x0,0x0}, {0xF9D9,0x6144,0x0,0x0}, 
+  {0xF9DA,0x6817,0x0,0x0}, {0xF9DB,0x7387,0x0,0x0}, {0xF9DC,0x9686,0x0,0x0}, 
+  {0xF9DD,0x5229,0x0,0x0}, {0xF9DE,0x540F,0x0,0x0}, {0xF9DF,0x5C65,0x0,0x0}, 
+  {0xF9E0,0x6613,0x0,0x0}, {0xF9E1,0x674E,0x0,0x0}, {0xF9E2,0x68A8,0x0,0x0}, 
+  {0xF9E3,0x6CE5,0x0,0x0}, {0xF9E4,0x7406,0x0,0x0}, {0xF9E5,0x75E2,0x0,0x0}, 
+  {0xF9E6,0x7F79,0x0,0x0}, {0xF9E7,0x88CF,0x0,0x0}, {0xF9E8,0x88E1,0x0,0x0}, 
+  {0xF9E9,0x91CC,0x0,0x0}, {0xF9EA,0x96E2,0x0,0x0}, {0xF9EB,0x533F,0x0,0x0}, 
+  {0xF9EC,0x6EBA,0x0,0x0}, {0xF9ED,0x541D,0x0,0x0}, {0xF9EE,0x71D0,0x0,0x0}, 
+  {0xF9EF,0x7498,0x0,0x0}, {0xF9F0,0x85FA,0x0,0x0}, {0xF9F1,0x96A3,0x0,0x0}, 
+  {0xF9F2,0x9C57,0x0,0x0}, {0xF9F3,0x9E9F,0x0,0x0}, {0xF9F4,0x6797,0x0,0x0}, 
+  {0xF9F5,0x6DCB,0x0,0x0}, {0xF9F6,0x81E8,0x0,0x0}, {0xF9F7,0x7ACB,0x0,0x0}, 
+  {0xF9F8,0x7B20,0x0,0x0}, {0xF9F9,0x7C92,0x0,0x0}, {0xF9FA,0x72C0,0x0,0x0}, 
+  {0xF9FB,0x7099,0x0,0x0}, {0xF9FC,0x8B58,0x0,0x0}, {0xF9FD,0x4EC0,0x0,0x0}, 
+  {0xF9FE,0x8336,0x0,0x0}, {0xF9FF,0x523A,0x0,0x0}, {0xFA00,0x5207,0x0,0x0}, 
+  {0xFA01,0x5EA6,0x0,0x0}, {0xFA02,0x62D3,0x0,0x0}, {0xFA03,0x7CD6,0x0,0x0}, 
+  {0xFA04,0x5B85,0x0,0x0}, {0xFA05,0x6D1E,0x0,0x0}, {0xFA06,0x66B4,0x0,0x0}, 
+  {0xFA07,0x8F3B,0x0,0x0}, {0xFA08,0x884C,0x0,0x0}, {0xFA09,0x964D,0x0,0x0}, 
+  {0xFA0A,0x898B,0x0,0x0}, {0xFA0B,0x5ED3,0x0,0x0}, {0xFA0C,0x5140,0x0,0x0}, 
+  {0xFA0D,0x55C0,0x0,0x0}, {0xFA10,0x585A,0x0,0x0}, {0xFA12,0x6674,0x0,0x0}, 
+  {0xFA15,0x51DE,0x0,0x0}, {0xFA16,0x732A,0x0,0x0}, {0xFA17,0x76CA,0x0,0x0}, 
+  {0xFA18,0x793C,0x0,0x0}, {0xFA19,0x795E,0x0,0x0}, {0xFA1A,0x7965,0x0,0x0}, 
+  {0xFA1B,0x798F,0x0,0x0}, {0xFA1C,0x9756,0x0,0x0}, {0xFA1D,0x7CBE,0x0,0x0}, 
+  {0xFA1E,0x7FBD,0x0,0x0}, {0xFA20,0x8612,0x0,0x0}, {0xFA22,0x8AF8,0x0,0x0}, 
+  {0xFA25,0x9038,0x0,0x0}, {0xFA26,0x90FD,0x0,0x0}, {0xFA2A,0x98EF,0x0,0x0}, 
+  {0xFA2B,0x98FC,0x0,0x0}, {0xFA2C,0x9928,0x0,0x0}, {0xFA2D,0x9DB4,0x0,0x0}, 
+  {0xFB1F,0x5F2,0x5B7,0x0}, {0xFB2A,0x5E9,0x5C1,0x0}, {0xFB2B,0x5E9,0x5C2,0x0}, 
+  {0xFB2C,0x5E9,0x5BC,0x5C1}, {0xFB2D,0x5E9,0x5BC,0x5C2}, {0xFB2E,0x5D0,0x5B7,0x0}, 
+  {0xFB2F,0x5D0,0x5B8,0x0}, {0xFB30,0x5D0,0x5BC,0x0}, {0xFB31,0x5D1,0x5BC,0x0}, 
+  {0xFB32,0x5D2,0x5BC,0x0}, {0xFB33,0x5D3,0x5BC,0x0}, {0xFB34,0x5D4,0x5BC,0x0}, 
+  {0xFB35,0x5D5,0x5BC,0x0}, {0xFB36,0x5D6,0x5BC,0x0}, {0xFB38,0x5D8,0x5BC,0x0}, 
+  {0xFB39,0x5D9,0x5BC,0x0}, {0xFB3A,0x5DA,0x5BC,0x0}, {0xFB3B,0x5DB,0x5BC,0x0}, 
+  {0xFB3C,0x5DC,0x5BC,0x0}, {0xFB3E,0x5DE,0x5BC,0x0}, {0xFB40,0x5E0,0x5BC,0x0}, 
+  {0xFB41,0x5E1,0x5BC,0x0}, {0xFB43,0x5E3,0x5BC,0x0}, {0xFB44,0x5E4,0x5BC,0x0}, 
+  {0xFB46,0x5E6,0x5BC,0x0}, {0xFB47,0x5E7,0x5BC,0x0}, {0xFB48,0x5E8,0x5BC,0x0}, 
+  {0xFB49,0x5E9,0x5BC,0x0}, {0xFB4A,0x5EA,0x5BC,0x0}, {0xFB4B,0x5D5,0x5B9,0x0}, 
+  {0xFB4C,0x5D1,0x5BF,0x0}, {0xFB4D,0x5DB,0x5BF,0x0}, {0xFB4E,0x5E4,0x5BF,0x0} 
+};
+
+
+/* All upper-case translations are included here */
+
+#define NUM_TOUPPER_INFO 677
+static const unirelate toupper_info[NUM_TOUPPER_INFO] = {
+  {0x61,0x41}, {0x62,0x42}, {0x63,0x43}, {0x64,0x44}, {0x65,0x45}, 
+  {0x66,0x46}, {0x67,0x47}, {0x68,0x48}, {0x69,0x49}, {0x6A,0x4A}, 
+  {0x6B,0x4B}, {0x6C,0x4C}, {0x6D,0x4D}, {0x6E,0x4E}, {0x6F,0x4F}, 
+  {0x70,0x50}, {0x71,0x51}, {0x72,0x52}, {0x73,0x53}, {0x74,0x54}, 
+  {0x75,0x55}, {0x76,0x56}, {0x77,0x57}, {0x78,0x58}, {0x79,0x59}, 
+  {0x7A,0x5A}, {0xE0,0xC0}, {0xE1,0xC1}, {0xE2,0xC2}, {0xE3,0xC3}, 
+  {0xE4,0xC4}, {0xE5,0xC5}, {0xE6,0xC6}, {0xE7,0xC7}, {0xE8,0xC8}, 
+  {0xE9,0xC9}, {0xEA,0xCA}, {0xEB,0xCB}, {0xEC,0xCC}, {0xED,0xCD}, 
+  {0xEE,0xCE}, {0xEF,0xCF}, {0xF0,0xD0}, {0xF1,0xD1}, {0xF2,0xD2}, 
+  {0xF3,0xD3}, {0xF4,0xD4}, {0xF5,0xD5}, {0xF6,0xD6}, {0xF8,0xD8}, 
+  {0xF9,0xD9}, {0xFA,0xDA}, {0xFB,0xDB}, {0xFC,0xDC}, {0xFD,0xDD}, 
+  {0xFE,0xDE}, {0xFF,0x178}, {0x101,0x100}, {0x103,0x102}, {0x105,0x104}, 
+  {0x107,0x106}, {0x109,0x108}, {0x10B,0x10A}, {0x10D,0x10C}, {0x10F,0x10E}, 
+  {0x111,0x110}, {0x113,0x112}, {0x115,0x114}, {0x117,0x116}, {0x119,0x118}, 
+  {0x11B,0x11A}, {0x11D,0x11C}, {0x11F,0x11E}, {0x121,0x120}, {0x123,0x122}, 
+  {0x125,0x124}, {0x127,0x126}, {0x129,0x128}, {0x12B,0x12A}, {0x12D,0x12C}, 
+  {0x12F,0x12E}, {0x131,0x49}, {0x133,0x132}, {0x135,0x134}, {0x137,0x136}, 
+  {0x13A,0x139}, {0x13C,0x13B}, {0x13E,0x13D}, {0x140,0x13F}, {0x142,0x141}, 
+  {0x144,0x143}, {0x146,0x145}, {0x148,0x147}, {0x14B,0x14A}, {0x14D,0x14C}, 
+  {0x14F,0x14E}, {0x151,0x150}, {0x153,0x152}, {0x155,0x154}, {0x157,0x156}, 
+  {0x159,0x158}, {0x15B,0x15A}, {0x15D,0x15C}, {0x15F,0x15E}, {0x161,0x160}, 
+  {0x163,0x162}, {0x165,0x164}, {0x167,0x166}, {0x169,0x168}, {0x16B,0x16A}, 
+  {0x16D,0x16C}, {0x16F,0x16E}, {0x171,0x170}, {0x173,0x172}, {0x175,0x174}, 
+  {0x177,0x176}, {0x17A,0x179}, {0x17C,0x17B}, {0x17E,0x17D}, {0x17F,0x53}, 
+  {0x183,0x182}, {0x185,0x184}, {0x188,0x187}, {0x18C,0x18B}, {0x192,0x191}, 
+  {0x199,0x198}, {0x1A1,0x1A0}, {0x1A3,0x1A2}, {0x1A5,0x1A4}, {0x1A8,0x1A7}, 
+  {0x1AD,0x1AC}, {0x1B0,0x1AF}, {0x1B4,0x1B3}, {0x1B6,0x1B5}, {0x1B9,0x1B8}, 
+  {0x1BD,0x1BC}, {0x1C5,0x1C4}, {0x1C6,0x1C4}, {0x1C8,0x1C7}, {0x1C9,0x1C7}, 
+  {0x1CB,0x1CA}, {0x1CC,0x1CA}, {0x1CE,0x1CD}, {0x1D0,0x1CF}, {0x1D2,0x1D1}, 
+  {0x1D4,0x1D3}, {0x1D6,0x1D5}, {0x1D8,0x1D7}, {0x1DA,0x1D9}, {0x1DC,0x1DB}, 
+  {0x1DD,0x18E}, {0x1DF,0x1DE}, {0x1E1,0x1E0}, {0x1E3,0x1E2}, {0x1E5,0x1E4}, 
+  {0x1E7,0x1E6}, {0x1E9,0x1E8}, {0x1EB,0x1EA}, {0x1ED,0x1EC}, {0x1EF,0x1EE}, 
+  {0x1F2,0x1F1}, {0x1F3,0x1F1}, {0x1F5,0x1F4}, {0x1FB,0x1FA}, {0x1FD,0x1FC}, 
+  {0x1FF,0x1FE}, {0x201,0x200}, {0x203,0x202}, {0x205,0x204}, {0x207,0x206}, 
+  {0x209,0x208}, {0x20B,0x20A}, {0x20D,0x20C}, {0x20F,0x20E}, {0x211,0x210}, 
+  {0x213,0x212}, {0x215,0x214}, {0x217,0x216}, {0x253,0x181}, {0x254,0x186}, 
+  {0x256,0x189}, {0x257,0x18A}, {0x259,0x18F}, {0x25B,0x190}, {0x260,0x193}, 
+  {0x263,0x194}, {0x268,0x197}, {0x269,0x196}, {0x26F,0x19C}, {0x272,0x19D}, 
+  {0x275,0x19F}, {0x283,0x1A9}, {0x288,0x1AE}, {0x28A,0x1B1}, {0x28B,0x1B2}, 
+  {0x292,0x1B7}, {0x3AC,0x386}, {0x3AD,0x388}, {0x3AE,0x389}, {0x3AF,0x38A}, 
+  {0x3B1,0x391}, {0x3B2,0x392}, {0x3B3,0x393}, {0x3B4,0x394}, {0x3B5,0x395}, 
+  {0x3B6,0x396}, {0x3B7,0x397}, {0x3B8,0x398}, {0x3B9,0x399}, {0x3BA,0x39A}, 
+  {0x3BB,0x39B}, {0x3BC,0x39C}, {0x3BD,0x39D}, {0x3BE,0x39E}, {0x3BF,0x39F}, 
+  {0x3C0,0x3A0}, {0x3C1,0x3A1}, {0x3C2,0x3A3}, {0x3C3,0x3A3}, {0x3C4,0x3A4}, 
+  {0x3C5,0x3A5}, {0x3C6,0x3A6}, {0x3C7,0x3A7}, {0x3C8,0x3A8}, {0x3C9,0x3A9}, 
+  {0x3CA,0x3AA}, {0x3CB,0x3AB}, {0x3CC,0x38C}, {0x3CD,0x38E}, {0x3CE,0x38F}, 
+  {0x3D0,0x392}, {0x3D1,0x398}, {0x3D5,0x3A6}, {0x3D6,0x3A0}, {0x3E3,0x3E2}, 
+  {0x3E5,0x3E4}, {0x3E7,0x3E6}, {0x3E9,0x3E8}, {0x3EB,0x3EA}, {0x3ED,0x3EC}, 
+  {0x3EF,0x3EE}, {0x3F0,0x39A}, {0x3F1,0x3A1}, {0x3F2,0x3A3}, {0x430,0x410}, 
+  {0x431,0x411}, {0x432,0x412}, {0x433,0x413}, {0x434,0x414}, {0x435,0x415}, 
+  {0x436,0x416}, {0x437,0x417}, {0x438,0x418}, {0x439,0x419}, {0x43A,0x41A}, 
+  {0x43B,0x41B}, {0x43C,0x41C}, {0x43D,0x41D}, {0x43E,0x41E}, {0x43F,0x41F}, 
+  {0x440,0x420}, {0x441,0x421}, {0x442,0x422}, {0x443,0x423}, {0x444,0x424}, 
+  {0x445,0x425}, {0x446,0x426}, {0x447,0x427}, {0x448,0x428}, {0x449,0x429}, 
+  {0x44A,0x42A}, {0x44B,0x42B}, {0x44C,0x42C}, {0x44D,0x42D}, {0x44E,0x42E}, 
+  {0x44F,0x42F}, {0x451,0x401}, {0x452,0x402}, {0x453,0x403}, {0x454,0x404}, 
+  {0x455,0x405}, {0x456,0x406}, {0x457,0x407}, {0x458,0x408}, {0x459,0x409}, 
+  {0x45A,0x40A}, {0x45B,0x40B}, {0x45C,0x40C}, {0x45E,0x40E}, {0x45F,0x40F}, 
+  {0x461,0x460}, {0x463,0x462}, {0x465,0x464}, {0x467,0x466}, {0x469,0x468}, 
+  {0x46B,0x46A}, {0x46D,0x46C}, {0x46F,0x46E}, {0x471,0x470}, {0x473,0x472}, 
+  {0x475,0x474}, {0x477,0x476}, {0x479,0x478}, {0x47B,0x47A}, {0x47D,0x47C}, 
+  {0x47F,0x47E}, {0x481,0x480}, {0x491,0x490}, {0x493,0x492}, {0x495,0x494}, 
+  {0x497,0x496}, {0x499,0x498}, {0x49B,0x49A}, {0x49D,0x49C}, {0x49F,0x49E}, 
+  {0x4A1,0x4A0}, {0x4A3,0x4A2}, {0x4A5,0x4A4}, {0x4A7,0x4A6}, {0x4A9,0x4A8}, 
+  {0x4AB,0x4AA}, {0x4AD,0x4AC}, {0x4AF,0x4AE}, {0x4B1,0x4B0}, {0x4B3,0x4B2}, 
+  {0x4B5,0x4B4}, {0x4B7,0x4B6}, {0x4B9,0x4B8}, {0x4BB,0x4BA}, {0x4BD,0x4BC}, 
+  {0x4BF,0x4BE}, {0x4C2,0x4C1}, {0x4C4,0x4C3}, {0x4C8,0x4C7}, {0x4CC,0x4CB}, 
+  {0x4D1,0x4D0}, {0x4D3,0x4D2}, {0x4D5,0x4D4}, {0x4D7,0x4D6}, {0x4D9,0x4D8}, 
+  {0x4DB,0x4DA}, {0x4DD,0x4DC}, {0x4DF,0x4DE}, {0x4E1,0x4E0}, {0x4E3,0x4E2}, 
+  {0x4E5,0x4E4}, {0x4E7,0x4E6}, {0x4E9,0x4E8}, {0x4EB,0x4EA}, {0x4EF,0x4EE}, 
+  {0x4F1,0x4F0}, {0x4F3,0x4F2}, {0x4F5,0x4F4}, {0x4F9,0x4F8}, {0x561,0x531}, 
+  {0x562,0x532}, {0x563,0x533}, {0x564,0x534}, {0x565,0x535}, {0x566,0x536}, 
+  {0x567,0x537}, {0x568,0x538}, {0x569,0x539}, {0x56A,0x53A}, {0x56B,0x53B}, 
+  {0x56C,0x53C}, {0x56D,0x53D}, {0x56E,0x53E}, {0x56F,0x53F}, {0x570,0x540}, 
+  {0x571,0x541}, {0x572,0x542}, {0x573,0x543}, {0x574,0x544}, {0x575,0x545}, 
+  {0x576,0x546}, {0x577,0x547}, {0x578,0x548}, {0x579,0x549}, {0x57A,0x54A}, 
+  {0x57B,0x54B}, {0x57C,0x54C}, {0x57D,0x54D}, {0x57E,0x54E}, {0x57F,0x54F}, 
+  {0x580,0x550}, {0x581,0x551}, {0x582,0x552}, {0x583,0x553}, {0x584,0x554}, 
+  {0x585,0x555}, {0x586,0x556}, {0x1E01,0x1E00}, {0x1E03,0x1E02}, {0x1E05,0x1E04}, 
+  {0x1E07,0x1E06}, {0x1E09,0x1E08}, {0x1E0B,0x1E0A}, {0x1E0D,0x1E0C}, {0x1E0F,0x1E0E}, 
+  {0x1E11,0x1E10}, {0x1E13,0x1E12}, {0x1E15,0x1E14}, {0x1E17,0x1E16}, {0x1E19,0x1E18}, 
+  {0x1E1B,0x1E1A}, {0x1E1D,0x1E1C}, {0x1E1F,0x1E1E}, {0x1E21,0x1E20}, {0x1E23,0x1E22}, 
+  {0x1E25,0x1E24}, {0x1E27,0x1E26}, {0x1E29,0x1E28}, {0x1E2B,0x1E2A}, {0x1E2D,0x1E2C}, 
+  {0x1E2F,0x1E2E}, {0x1E31,0x1E30}, {0x1E33,0x1E32}, {0x1E35,0x1E34}, {0x1E37,0x1E36}, 
+  {0x1E39,0x1E38}, {0x1E3B,0x1E3A}, {0x1E3D,0x1E3C}, {0x1E3F,0x1E3E}, {0x1E41,0x1E40}, 
+  {0x1E43,0x1E42}, {0x1E45,0x1E44}, {0x1E47,0x1E46}, {0x1E49,0x1E48}, {0x1E4B,0x1E4A}, 
+  {0x1E4D,0x1E4C}, {0x1E4F,0x1E4E}, {0x1E51,0x1E50}, {0x1E53,0x1E52}, {0x1E55,0x1E54}, 
+  {0x1E57,0x1E56}, {0x1E59,0x1E58}, {0x1E5B,0x1E5A}, {0x1E5D,0x1E5C}, {0x1E5F,0x1E5E}, 
+  {0x1E61,0x1E60}, {0x1E63,0x1E62}, {0x1E65,0x1E64}, {0x1E67,0x1E66}, {0x1E69,0x1E68}, 
+  {0x1E6B,0x1E6A}, {0x1E6D,0x1E6C}, {0x1E6F,0x1E6E}, {0x1E71,0x1E70}, {0x1E73,0x1E72}, 
+  {0x1E75,0x1E74}, {0x1E77,0x1E76}, {0x1E79,0x1E78}, {0x1E7B,0x1E7A}, {0x1E7D,0x1E7C}, 
+  {0x1E7F,0x1E7E}, {0x1E81,0x1E80}, {0x1E83,0x1E82}, {0x1E85,0x1E84}, {0x1E87,0x1E86}, 
+  {0x1E89,0x1E88}, {0x1E8B,0x1E8A}, {0x1E8D,0x1E8C}, {0x1E8F,0x1E8E}, {0x1E91,0x1E90}, 
+  {0x1E93,0x1E92}, {0x1E95,0x1E94}, {0x1E9B,0x1E60}, {0x1EA1,0x1EA0}, {0x1EA3,0x1EA2}, 
+  {0x1EA5,0x1EA4}, {0x1EA7,0x1EA6}, {0x1EA9,0x1EA8}, {0x1EAB,0x1EAA}, {0x1EAD,0x1EAC}, 
+  {0x1EAF,0x1EAE}, {0x1EB1,0x1EB0}, {0x1EB3,0x1EB2}, {0x1EB5,0x1EB4}, {0x1EB7,0x1EB6}, 
+  {0x1EB9,0x1EB8}, {0x1EBB,0x1EBA}, {0x1EBD,0x1EBC}, {0x1EBF,0x1EBE}, {0x1EC1,0x1EC0}, 
+  {0x1EC3,0x1EC2}, {0x1EC5,0x1EC4}, {0x1EC7,0x1EC6}, {0x1EC9,0x1EC8}, {0x1ECB,0x1ECA}, 
+  {0x1ECD,0x1ECC}, {0x1ECF,0x1ECE}, {0x1ED1,0x1ED0}, {0x1ED3,0x1ED2}, {0x1ED5,0x1ED4}, 
+  {0x1ED7,0x1ED6}, {0x1ED9,0x1ED8}, {0x1EDB,0x1EDA}, {0x1EDD,0x1EDC}, {0x1EDF,0x1EDE}, 
+  {0x1EE1,0x1EE0}, {0x1EE3,0x1EE2}, {0x1EE5,0x1EE4}, {0x1EE7,0x1EE6}, {0x1EE9,0x1EE8}, 
+  {0x1EEB,0x1EEA}, {0x1EED,0x1EEC}, {0x1EEF,0x1EEE}, {0x1EF1,0x1EF0}, {0x1EF3,0x1EF2}, 
+  {0x1EF5,0x1EF4}, {0x1EF7,0x1EF6}, {0x1EF9,0x1EF8}, {0x1F00,0x1F08}, {0x1F01,0x1F09}, 
+  {0x1F02,0x1F0A}, {0x1F03,0x1F0B}, {0x1F04,0x1F0C}, {0x1F05,0x1F0D}, {0x1F06,0x1F0E}, 
+  {0x1F07,0x1F0F}, {0x1F10,0x1F18}, {0x1F11,0x1F19}, {0x1F12,0x1F1A}, {0x1F13,0x1F1B}, 
+  {0x1F14,0x1F1C}, {0x1F15,0x1F1D}, {0x1F20,0x1F28}, {0x1F21,0x1F29}, {0x1F22,0x1F2A}, 
+  {0x1F23,0x1F2B}, {0x1F24,0x1F2C}, {0x1F25,0x1F2D}, {0x1F26,0x1F2E}, {0x1F27,0x1F2F}, 
+  {0x1F30,0x1F38}, {0x1F31,0x1F39}, {0x1F32,0x1F3A}, {0x1F33,0x1F3B}, {0x1F34,0x1F3C}, 
+  {0x1F35,0x1F3D}, {0x1F36,0x1F3E}, {0x1F37,0x1F3F}, {0x1F40,0x1F48}, {0x1F41,0x1F49}, 
+  {0x1F42,0x1F4A}, {0x1F43,0x1F4B}, {0x1F44,0x1F4C}, {0x1F45,0x1F4D}, {0x1F51,0x1F59}, 
+  {0x1F53,0x1F5B}, {0x1F55,0x1F5D}, {0x1F57,0x1F5F}, {0x1F60,0x1F68}, {0x1F61,0x1F69}, 
+  {0x1F62,0x1F6A}, {0x1F63,0x1F6B}, {0x1F64,0x1F6C}, {0x1F65,0x1F6D}, {0x1F66,0x1F6E}, 
+  {0x1F67,0x1F6F}, {0x1F70,0x1FBA}, {0x1F71,0x1FBB}, {0x1F72,0x1FC8}, {0x1F73,0x1FC9}, 
+  {0x1F74,0x1FCA}, {0x1F75,0x1FCB}, {0x1F76,0x1FDA}, {0x1F77,0x1FDB}, {0x1F78,0x1FF8}, 
+  {0x1F79,0x1FF9}, {0x1F7A,0x1FEA}, {0x1F7B,0x1FEB}, {0x1F7C,0x1FFA}, {0x1F7D,0x1FFB}, 
+  {0x1F80,0x1F88}, {0x1F81,0x1F89}, {0x1F82,0x1F8A}, {0x1F83,0x1F8B}, {0x1F84,0x1F8C}, 
+  {0x1F85,0x1F8D}, {0x1F86,0x1F8E}, {0x1F87,0x1F8F}, {0x1F90,0x1F98}, {0x1F91,0x1F99}, 
+  {0x1F92,0x1F9A}, {0x1F93,0x1F9B}, {0x1F94,0x1F9C}, {0x1F95,0x1F9D}, {0x1F96,0x1F9E}, 
+  {0x1F97,0x1F9F}, {0x1FA0,0x1FA8}, {0x1FA1,0x1FA9}, {0x1FA2,0x1FAA}, {0x1FA3,0x1FAB}, 
+  {0x1FA4,0x1FAC}, {0x1FA5,0x1FAD}, {0x1FA6,0x1FAE}, {0x1FA7,0x1FAF}, {0x1FB0,0x1FB8}, 
+  {0x1FB1,0x1FB9}, {0x1FB3,0x1FBC}, {0x1FC3,0x1FCC}, {0x1FD0,0x1FD8}, {0x1FD1,0x1FD9}, 
+  {0x1FE0,0x1FE8}, {0x1FE1,0x1FE9}, {0x1FE5,0x1FEC}, {0x1FF3,0x1FFC}, {0x2170,0x2160}, 
+  {0x2171,0x2161}, {0x2172,0x2162}, {0x2173,0x2163}, {0x2174,0x2164}, {0x2175,0x2165}, 
+  {0x2176,0x2166}, {0x2177,0x2167}, {0x2178,0x2168}, {0x2179,0x2169}, {0x217A,0x216A}, 
+  {0x217B,0x216B}, {0x217C,0x216C}, {0x217D,0x216D}, {0x217E,0x216E}, {0x217F,0x216F}, 
+  {0x24D0,0x24B6}, {0x24D1,0x24B7}, {0x24D2,0x24B8}, {0x24D3,0x24B9}, {0x24D4,0x24BA}, 
+  {0x24D5,0x24BB}, {0x24D6,0x24BC}, {0x24D7,0x24BD}, {0x24D8,0x24BE}, {0x24D9,0x24BF}, 
+  {0x24DA,0x24C0}, {0x24DB,0x24C1}, {0x24DC,0x24C2}, {0x24DD,0x24C3}, {0x24DE,0x24C4}, 
+  {0x24DF,0x24C5}, {0x24E0,0x24C6}, {0x24E1,0x24C7}, {0x24E2,0x24C8}, {0x24E3,0x24C9}, 
+  {0x24E4,0x24CA}, {0x24E5,0x24CB}, {0x24E6,0x24CC}, {0x24E7,0x24CD}, {0x24E8,0x24CE}, 
+  {0x24E9,0x24CF}, {0xFF41,0xFF21}, {0xFF42,0xFF22}, {0xFF43,0xFF23}, {0xFF44,0xFF24}, 
+  {0xFF45,0xFF25}, {0xFF46,0xFF26}, {0xFF47,0xFF27}, {0xFF48,0xFF28}, {0xFF49,0xFF29}, 
+  {0xFF4A,0xFF2A}, {0xFF4B,0xFF2B}, {0xFF4C,0xFF2C}, {0xFF4D,0xFF2D}, {0xFF4E,0xFF2E}, 
+  {0xFF4F,0xFF2F}, {0xFF50,0xFF30}, {0xFF51,0xFF31}, {0xFF52,0xFF32}, {0xFF53,0xFF33}, 
+  {0xFF54,0xFF34}, {0xFF55,0xFF35}, {0xFF56,0xFF36}, {0xFF57,0xFF37}, {0xFF58,0xFF38}, 
+  {0xFF59,0xFF39}, {0xFF5A,0xFF3A} 
+};
+
+
+
+/* All lower-case translations are included here */
+
+#define NUM_TOLOWER_INFO 705
+static const unirelate tolower_info[NUM_TOLOWER_INFO] = {
+  {0x41,0x61}, {0x42,0x62}, {0x43,0x63}, {0x44,0x64}, {0x45,0x65}, 
+  {0x46,0x66}, {0x47,0x67}, {0x48,0x68}, {0x49,0x69}, {0x4A,0x6A}, 
+  {0x4B,0x6B}, {0x4C,0x6C}, {0x4D,0x6D}, {0x4E,0x6E}, {0x4F,0x6F}, 
+  {0x50,0x70}, {0x51,0x71}, {0x52,0x72}, {0x53,0x73}, {0x54,0x74}, 
+  {0x55,0x75}, {0x56,0x76}, {0x57,0x77}, {0x58,0x78}, {0x59,0x79}, 
+  {0x5A,0x7A}, {0xC0,0xE0}, {0xC1,0xE1}, {0xC2,0xE2}, {0xC3,0xE3}, 
+  {0xC4,0xE4}, {0xC5,0xE5}, {0xC6,0xE6}, {0xC7,0xE7}, {0xC8,0xE8}, 
+  {0xC9,0xE9}, {0xCA,0xEA}, {0xCB,0xEB}, {0xCC,0xEC}, {0xCD,0xED}, 
+  {0xCE,0xEE}, {0xCF,0xEF}, {0xD0,0xF0}, {0xD1,0xF1}, {0xD2,0xF2}, 
+  {0xD3,0xF3}, {0xD4,0xF4}, {0xD5,0xF5}, {0xD6,0xF6}, {0xD8,0xF8}, 
+  {0xD9,0xF9}, {0xDA,0xFA}, {0xDB,0xFB}, {0xDC,0xFC}, {0xDD,0xFD}, 
+  {0xDE,0xFE}, {0x100,0x101}, {0x102,0x103}, {0x104,0x105}, {0x106,0x107}, 
+  {0x108,0x109}, {0x10A,0x10B}, {0x10C,0x10D}, {0x10E,0x10F}, {0x110,0x111}, 
+  {0x112,0x113}, {0x114,0x115}, {0x116,0x117}, {0x118,0x119}, {0x11A,0x11B}, 
+  {0x11C,0x11D}, {0x11E,0x11F}, {0x120,0x121}, {0x122,0x123}, {0x124,0x125}, 
+  {0x126,0x127}, {0x128,0x129}, {0x12A,0x12B}, {0x12C,0x12D}, {0x12E,0x12F}, 
+  {0x130,0x69}, {0x132,0x133}, {0x134,0x135}, {0x136,0x137}, {0x139,0x13A}, 
+  {0x13B,0x13C}, {0x13D,0x13E}, {0x13F,0x140}, {0x141,0x142}, {0x143,0x144}, 
+  {0x145,0x146}, {0x147,0x148}, {0x14A,0x14B}, {0x14C,0x14D}, {0x14E,0x14F}, 
+  {0x150,0x151}, {0x152,0x153}, {0x154,0x155}, {0x156,0x157}, {0x158,0x159}, 
+  {0x15A,0x15B}, {0x15C,0x15D}, {0x15E,0x15F}, {0x160,0x161}, {0x162,0x163}, 
+  {0x164,0x165}, {0x166,0x167}, {0x168,0x169}, {0x16A,0x16B}, {0x16C,0x16D}, 
+  {0x16E,0x16F}, {0x170,0x171}, {0x172,0x173}, {0x174,0x175}, {0x176,0x177}, 
+  {0x178,0xFF}, {0x179,0x17A}, {0x17B,0x17C}, {0x17D,0x17E}, {0x181,0x253}, 
+  {0x182,0x183}, {0x184,0x185}, {0x186,0x254}, {0x187,0x188}, {0x189,0x256}, 
+  {0x18A,0x257}, {0x18B,0x18C}, {0x18E,0x1DD}, {0x18F,0x259}, {0x190,0x25B}, 
+  {0x191,0x192}, {0x193,0x260}, {0x194,0x263}, {0x196,0x269}, {0x197,0x268}, 
+  {0x198,0x199}, {0x19C,0x26F}, {0x19D,0x272}, {0x19F,0x275}, {0x1A0,0x1A1}, 
+  {0x1A2,0x1A3}, {0x1A4,0x1A5}, {0x1A7,0x1A8}, {0x1A9,0x283}, {0x1AC,0x1AD}, 
+  {0x1AE,0x288}, {0x1AF,0x1B0}, {0x1B1,0x28A}, {0x1B2,0x28B}, {0x1B3,0x1B4}, 
+  {0x1B5,0x1B6}, {0x1B7,0x292}, {0x1B8,0x1B9}, {0x1BC,0x1BD}, {0x1C4,0x1C6}, 
+  {0x1C5,0x1C6}, {0x1C7,0x1C9}, {0x1C8,0x1C9}, {0x1CA,0x1CC}, {0x1CB,0x1CC}, 
+  {0x1CD,0x1CE}, {0x1CF,0x1D0}, {0x1D1,0x1D2}, {0x1D3,0x1D4}, {0x1D5,0x1D6}, 
+  {0x1D7,0x1D8}, {0x1D9,0x1DA}, {0x1DB,0x1DC}, {0x1DE,0x1DF}, {0x1E0,0x1E1}, 
+  {0x1E2,0x1E3}, {0x1E4,0x1E5}, {0x1E6,0x1E7}, {0x1E8,0x1E9}, {0x1EA,0x1EB}, 
+  {0x1EC,0x1ED}, {0x1EE,0x1EF}, {0x1F1,0x1F3}, {0x1F2,0x1F3}, {0x1F4,0x1F5}, 
+  {0x1FA,0x1FB}, {0x1FC,0x1FD}, {0x1FE,0x1FF}, {0x200,0x201}, {0x202,0x203}, 
+  {0x204,0x205}, {0x206,0x207}, {0x208,0x209}, {0x20A,0x20B}, {0x20C,0x20D}, 
+  {0x20E,0x20F}, {0x210,0x211}, {0x212,0x213}, {0x214,0x215}, {0x216,0x217}, 
+  {0x386,0x3AC}, {0x388,0x3AD}, {0x389,0x3AE}, {0x38A,0x3AF}, {0x38C,0x3CC}, 
+  {0x38E,0x3CD}, {0x38F,0x3CE}, {0x391,0x3B1}, {0x392,0x3B2}, {0x393,0x3B3}, 
+  {0x394,0x3B4}, {0x395,0x3B5}, {0x396,0x3B6}, {0x397,0x3B7}, {0x398,0x3B8}, 
+  {0x399,0x3B9}, {0x39A,0x3BA}, {0x39B,0x3BB}, {0x39C,0x3BC}, {0x39D,0x3BD}, 
+  {0x39E,0x3BE}, {0x39F,0x3BF}, {0x3A0,0x3C0}, {0x3A1,0x3C1}, {0x3A3,0x3C3}, 
+  {0x3A4,0x3C4}, {0x3A5,0x3C5}, {0x3A6,0x3C6}, {0x3A7,0x3C7}, {0x3A8,0x3C8}, 
+  {0x3A9,0x3C9}, {0x3AA,0x3CA}, {0x3AB,0x3CB}, {0x3E2,0x3E3}, {0x3E4,0x3E5}, 
+  {0x3E6,0x3E7}, {0x3E8,0x3E9}, {0x3EA,0x3EB}, {0x3EC,0x3ED}, {0x3EE,0x3EF}, 
+  {0x401,0x451}, {0x402,0x452}, {0x403,0x453}, {0x404,0x454}, {0x405,0x455}, 
+  {0x406,0x456}, {0x407,0x457}, {0x408,0x458}, {0x409,0x459}, {0x40A,0x45A}, 
+  {0x40B,0x45B}, {0x40C,0x45C}, {0x40E,0x45E}, {0x40F,0x45F}, {0x410,0x430}, 
+  {0x411,0x431}, {0x412,0x432}, {0x413,0x433}, {0x414,0x434}, {0x415,0x435}, 
+  {0x416,0x436}, {0x417,0x437}, {0x418,0x438}, {0x419,0x439}, {0x41A,0x43A}, 
+  {0x41B,0x43B}, {0x41C,0x43C}, {0x41D,0x43D}, {0x41E,0x43E}, {0x41F,0x43F}, 
+  {0x420,0x440}, {0x421,0x441}, {0x422,0x442}, {0x423,0x443}, {0x424,0x444}, 
+  {0x425,0x445}, {0x426,0x446}, {0x427,0x447}, {0x428,0x448}, {0x429,0x449}, 
+  {0x42A,0x44A}, {0x42B,0x44B}, {0x42C,0x44C}, {0x42D,0x44D}, {0x42E,0x44E}, 
+  {0x42F,0x44F}, {0x460,0x461}, {0x462,0x463}, {0x464,0x465}, {0x466,0x467}, 
+  {0x468,0x469}, {0x46A,0x46B}, {0x46C,0x46D}, {0x46E,0x46F}, {0x470,0x471}, 
+  {0x472,0x473}, {0x474,0x475}, {0x476,0x477}, {0x478,0x479}, {0x47A,0x47B}, 
+  {0x47C,0x47D}, {0x47E,0x47F}, {0x480,0x481}, {0x490,0x491}, {0x492,0x493}, 
+  {0x494,0x495}, {0x496,0x497}, {0x498,0x499}, {0x49A,0x49B}, {0x49C,0x49D}, 
+  {0x49E,0x49F}, {0x4A0,0x4A1}, {0x4A2,0x4A3}, {0x4A4,0x4A5}, {0x4A6,0x4A7}, 
+  {0x4A8,0x4A9}, {0x4AA,0x4AB}, {0x4AC,0x4AD}, {0x4AE,0x4AF}, {0x4B0,0x4B1}, 
+  {0x4B2,0x4B3}, {0x4B4,0x4B5}, {0x4B6,0x4B7}, {0x4B8,0x4B9}, {0x4BA,0x4BB}, 
+  {0x4BC,0x4BD}, {0x4BE,0x4BF}, {0x4C1,0x4C2}, {0x4C3,0x4C4}, {0x4C7,0x4C8}, 
+  {0x4CB,0x4CC}, {0x4D0,0x4D1}, {0x4D2,0x4D3}, {0x4D4,0x4D5}, {0x4D6,0x4D7}, 
+  {0x4D8,0x4D9}, {0x4DA,0x4DB}, {0x4DC,0x4DD}, {0x4DE,0x4DF}, {0x4E0,0x4E1}, 
+  {0x4E2,0x4E3}, {0x4E4,0x4E5}, {0x4E6,0x4E7}, {0x4E8,0x4E9}, {0x4EA,0x4EB}, 
+  {0x4EE,0x4EF}, {0x4F0,0x4F1}, {0x4F2,0x4F3}, {0x4F4,0x4F5}, {0x4F8,0x4F9}, 
+  {0x531,0x561}, {0x532,0x562}, {0x533,0x563}, {0x534,0x564}, {0x535,0x565}, 
+  {0x536,0x566}, {0x537,0x567}, {0x538,0x568}, {0x539,0x569}, {0x53A,0x56A}, 
+  {0x53B,0x56B}, {0x53C,0x56C}, {0x53D,0x56D}, {0x53E,0x56E}, {0x53F,0x56F}, 
+  {0x540,0x570}, {0x541,0x571}, {0x542,0x572}, {0x543,0x573}, {0x544,0x574}, 
+  {0x545,0x575}, {0x546,0x576}, {0x547,0x577}, {0x548,0x578}, {0x549,0x579}, 
+  {0x54A,0x57A}, {0x54B,0x57B}, {0x54C,0x57C}, {0x54D,0x57D}, {0x54E,0x57E}, 
+  {0x54F,0x57F}, {0x550,0x580}, {0x551,0x581}, {0x552,0x582}, {0x553,0x583}, 
+  {0x554,0x584}, {0x555,0x585}, {0x556,0x586}, {0x10A0,0x10D0}, {0x10A1,0x10D1}, 
+  {0x10A2,0x10D2}, {0x10A3,0x10D3}, {0x10A4,0x10D4}, {0x10A5,0x10D5}, {0x10A6,0x10D6}, 
+  {0x10A7,0x10D7}, {0x10A8,0x10D8}, {0x10A9,0x10D9}, {0x10AA,0x10DA}, {0x10AB,0x10DB}, 
+  {0x10AC,0x10DC}, {0x10AD,0x10DD}, {0x10AE,0x10DE}, {0x10AF,0x10DF}, {0x10B0,0x10E0}, 
+  {0x10B1,0x10E1}, {0x10B2,0x10E2}, {0x10B3,0x10E3}, {0x10B4,0x10E4}, {0x10B5,0x10E5}, 
+  {0x10B6,0x10E6}, {0x10B7,0x10E7}, {0x10B8,0x10E8}, {0x10B9,0x10E9}, {0x10BA,0x10EA}, 
+  {0x10BB,0x10EB}, {0x10BC,0x10EC}, {0x10BD,0x10ED}, {0x10BE,0x10EE}, {0x10BF,0x10EF}, 
+  {0x10C0,0x10F0}, {0x10C1,0x10F1}, {0x10C2,0x10F2}, {0x10C3,0x10F3}, {0x10C4,0x10F4}, 
+  {0x10C5,0x10F5}, {0x1E00,0x1E01}, {0x1E02,0x1E03}, {0x1E04,0x1E05}, {0x1E06,0x1E07}, 
+  {0x1E08,0x1E09}, {0x1E0A,0x1E0B}, {0x1E0C,0x1E0D}, {0x1E0E,0x1E0F}, {0x1E10,0x1E11}, 
+  {0x1E12,0x1E13}, {0x1E14,0x1E15}, {0x1E16,0x1E17}, {0x1E18,0x1E19}, {0x1E1A,0x1E1B}, 
+  {0x1E1C,0x1E1D}, {0x1E1E,0x1E1F}, {0x1E20,0x1E21}, {0x1E22,0x1E23}, {0x1E24,0x1E25}, 
+  {0x1E26,0x1E27}, {0x1E28,0x1E29}, {0x1E2A,0x1E2B}, {0x1E2C,0x1E2D}, {0x1E2E,0x1E2F}, 
+  {0x1E30,0x1E31}, {0x1E32,0x1E33}, {0x1E34,0x1E35}, {0x1E36,0x1E37}, {0x1E38,0x1E39}, 
+  {0x1E3A,0x1E3B}, {0x1E3C,0x1E3D}, {0x1E3E,0x1E3F}, {0x1E40,0x1E41}, {0x1E42,0x1E43}, 
+  {0x1E44,0x1E45}, {0x1E46,0x1E47}, {0x1E48,0x1E49}, {0x1E4A,0x1E4B}, {0x1E4C,0x1E4D}, 
+  {0x1E4E,0x1E4F}, {0x1E50,0x1E51}, {0x1E52,0x1E53}, {0x1E54,0x1E55}, {0x1E56,0x1E57}, 
+  {0x1E58,0x1E59}, {0x1E5A,0x1E5B}, {0x1E5C,0x1E5D}, {0x1E5E,0x1E5F}, {0x1E60,0x1E61}, 
+  {0x1E62,0x1E63}, {0x1E64,0x1E65}, {0x1E66,0x1E67}, {0x1E68,0x1E69}, {0x1E6A,0x1E6B}, 
+  {0x1E6C,0x1E6D}, {0x1E6E,0x1E6F}, {0x1E70,0x1E71}, {0x1E72,0x1E73}, {0x1E74,0x1E75}, 
+  {0x1E76,0x1E77}, {0x1E78,0x1E79}, {0x1E7A,0x1E7B}, {0x1E7C,0x1E7D}, {0x1E7E,0x1E7F}, 
+  {0x1E80,0x1E81}, {0x1E82,0x1E83}, {0x1E84,0x1E85}, {0x1E86,0x1E87}, {0x1E88,0x1E89}, 
+  {0x1E8A,0x1E8B}, {0x1E8C,0x1E8D}, {0x1E8E,0x1E8F}, {0x1E90,0x1E91}, {0x1E92,0x1E93}, 
+  {0x1E94,0x1E95}, {0x1EA0,0x1EA1}, {0x1EA2,0x1EA3}, {0x1EA4,0x1EA5}, {0x1EA6,0x1EA7}, 
+  {0x1EA8,0x1EA9}, {0x1EAA,0x1EAB}, {0x1EAC,0x1EAD}, {0x1EAE,0x1EAF}, {0x1EB0,0x1EB1}, 
+  {0x1EB2,0x1EB3}, {0x1EB4,0x1EB5}, {0x1EB6,0x1EB7}, {0x1EB8,0x1EB9}, {0x1EBA,0x1EBB}, 
+  {0x1EBC,0x1EBD}, {0x1EBE,0x1EBF}, {0x1EC0,0x1EC1}, {0x1EC2,0x1EC3}, {0x1EC4,0x1EC5}, 
+  {0x1EC6,0x1EC7}, {0x1EC8,0x1EC9}, {0x1ECA,0x1ECB}, {0x1ECC,0x1ECD}, {0x1ECE,0x1ECF}, 
+  {0x1ED0,0x1ED1}, {0x1ED2,0x1ED3}, {0x1ED4,0x1ED5}, {0x1ED6,0x1ED7}, {0x1ED8,0x1ED9}, 
+  {0x1EDA,0x1EDB}, {0x1EDC,0x1EDD}, {0x1EDE,0x1EDF}, {0x1EE0,0x1EE1}, {0x1EE2,0x1EE3}, 
+  {0x1EE4,0x1EE5}, {0x1EE6,0x1EE7}, {0x1EE8,0x1EE9}, {0x1EEA,0x1EEB}, {0x1EEC,0x1EED}, 
+  {0x1EEE,0x1EEF}, {0x1EF0,0x1EF1}, {0x1EF2,0x1EF3}, {0x1EF4,0x1EF5}, {0x1EF6,0x1EF7}, 
+  {0x1EF8,0x1EF9}, {0x1F08,0x1F00}, {0x1F09,0x1F01}, {0x1F0A,0x1F02}, {0x1F0B,0x1F03}, 
+  {0x1F0C,0x1F04}, {0x1F0D,0x1F05}, {0x1F0E,0x1F06}, {0x1F0F,0x1F07}, {0x1F18,0x1F10}, 
+  {0x1F19,0x1F11}, {0x1F1A,0x1F12}, {0x1F1B,0x1F13}, {0x1F1C,0x1F14}, {0x1F1D,0x1F15}, 
+  {0x1F28,0x1F20}, {0x1F29,0x1F21}, {0x1F2A,0x1F22}, {0x1F2B,0x1F23}, {0x1F2C,0x1F24}, 
+  {0x1F2D,0x1F25}, {0x1F2E,0x1F26}, {0x1F2F,0x1F27}, {0x1F38,0x1F30}, {0x1F39,0x1F31}, 
+  {0x1F3A,0x1F32}, {0x1F3B,0x1F33}, {0x1F3C,0x1F34}, {0x1F3D,0x1F35}, {0x1F3E,0x1F36}, 
+  {0x1F3F,0x1F37}, {0x1F48,0x1F40}, {0x1F49,0x1F41}, {0x1F4A,0x1F42}, {0x1F4B,0x1F43}, 
+  {0x1F4C,0x1F44}, {0x1F4D,0x1F45}, {0x1F59,0x1F51}, {0x1F5B,0x1F53}, {0x1F5D,0x1F55}, 
+  {0x1F5F,0x1F57}, {0x1F68,0x1F60}, {0x1F69,0x1F61}, {0x1F6A,0x1F62}, {0x1F6B,0x1F63}, 
+  {0x1F6C,0x1F64}, {0x1F6D,0x1F65}, {0x1F6E,0x1F66}, {0x1F6F,0x1F67}, {0x1F88,0x1F80}, 
+  {0x1F89,0x1F81}, {0x1F8A,0x1F82}, {0x1F8B,0x1F83}, {0x1F8C,0x1F84}, {0x1F8D,0x1F85}, 
+  {0x1F8E,0x1F86}, {0x1F8F,0x1F87}, {0x1F98,0x1F90}, {0x1F99,0x1F91}, {0x1F9A,0x1F92}, 
+  {0x1F9B,0x1F93}, {0x1F9C,0x1F94}, {0x1F9D,0x1F95}, {0x1F9E,0x1F96}, {0x1F9F,0x1F97}, 
+  {0x1FA8,0x1FA0}, {0x1FA9,0x1FA1}, {0x1FAA,0x1FA2}, {0x1FAB,0x1FA3}, {0x1FAC,0x1FA4}, 
+  {0x1FAD,0x1FA5}, {0x1FAE,0x1FA6}, {0x1FAF,0x1FA7}, {0x1FB8,0x1FB0}, {0x1FB9,0x1FB1}, 
+  {0x1FBA,0x1F70}, {0x1FBB,0x1F71}, {0x1FBC,0x1FB3}, {0x1FC8,0x1F72}, {0x1FC9,0x1F73}, 
+  {0x1FCA,0x1F74}, {0x1FCB,0x1F75}, {0x1FCC,0x1FC3}, {0x1FD8,0x1FD0}, {0x1FD9,0x1FD1}, 
+  {0x1FDA,0x1F76}, {0x1FDB,0x1F77}, {0x1FE8,0x1FE0}, {0x1FE9,0x1FE1}, {0x1FEA,0x1F7A}, 
+  {0x1FEB,0x1F7B}, {0x1FEC,0x1FE5}, {0x1FF8,0x1F78}, {0x1FF9,0x1F79}, {0x1FFA,0x1F7C}, 
+  {0x1FFB,0x1F7D}, {0x1FFC,0x1FF3}, {0x2160,0x2170}, {0x2161,0x2171}, {0x2162,0x2172}, 
+  {0x2163,0x2173}, {0x2164,0x2174}, {0x2165,0x2175}, {0x2166,0x2176}, {0x2167,0x2177}, 
+  {0x2168,0x2178}, {0x2169,0x2179}, {0x216A,0x217A}, {0x216B,0x217B}, {0x216C,0x217C}, 
+  {0x216D,0x217D}, {0x216E,0x217E}, {0x216F,0x217F}, {0x24B6,0x24D0}, {0x24B7,0x24D1}, 
+  {0x24B8,0x24D2}, {0x24B9,0x24D3}, {0x24BA,0x24D4}, {0x24BB,0x24D5}, {0x24BC,0x24D6}, 
+  {0x24BD,0x24D7}, {0x24BE,0x24D8}, {0x24BF,0x24D9}, {0x24C0,0x24DA}, {0x24C1,0x24DB}, 
+  {0x24C2,0x24DC}, {0x24C3,0x24DD}, {0x24C4,0x24DE}, {0x24C5,0x24DF}, {0x24C6,0x24E0}, 
+  {0x24C7,0x24E1}, {0x24C8,0x24E2}, {0x24C9,0x24E3}, {0x24CA,0x24E4}, {0x24CB,0x24E5}, 
+  {0x24CC,0x24E6}, {0x24CD,0x24E7}, {0x24CE,0x24E8}, {0x24CF,0x24E9}, {0xFF21,0xFF41}, 
+  {0xFF22,0xFF42}, {0xFF23,0xFF43}, {0xFF24,0xFF44}, {0xFF25,0xFF45}, {0xFF26,0xFF46}, 
+  {0xFF27,0xFF47}, {0xFF28,0xFF48}, {0xFF29,0xFF49}, {0xFF2A,0xFF4A}, {0xFF2B,0xFF4B}, 
+  {0xFF2C,0xFF4C}, {0xFF2D,0xFF4D}, {0xFF2E,0xFF4E}, {0xFF2F,0xFF4F}, {0xFF30,0xFF50}, 
+  {0xFF31,0xFF51}, {0xFF32,0xFF52}, {0xFF33,0xFF53}, {0xFF34,0xFF54}, {0xFF35,0xFF55}, 
+  {0xFF36,0xFF56}, {0xFF37,0xFF57}, {0xFF38,0xFF58}, {0xFF39,0xFF59}, {0xFF3A,0xFF5A}
+};
+
+
+
+#define NUM_TOSIMPLIFIED_INFO 2145
+static const unirelate tosimplified_info[NUM_TOSIMPLIFIED_INFO] = {
+  {0x4E48,0x9EBD}, {0x4E82,0x4E71}, {0x4E9E,0x4E9A}, {0x4F47,0x4F2B}, {0x4F59,0x9980}, 
+  {0x4F86,0x6765}, {0x4F96,0x4ED1}, {0x4FE0,0x4FA0}, {0x5000,0x4F25}, {0x5006,0x4FE9}, 
+  {0x5009,0x4ED3}, {0x500B,0x4E2A}, {0x5011,0x4EEC}, {0x502B,0x4F26}, {0x5049,0x4F1F}, 
+  {0x5074,0x4FA7}, {0x5075,0x4FA6}, {0x5096,0x4F27}, {0x5098,0x4F1E}, {0x5099,0x5907}, 
+  {0x50AD,0x4F63}, {0x50B3,0x4F20}, {0x50B4,0x4F1B}, {0x50B5,0x503A}, {0x50B7,0x4F24}, 
+  {0x50BE,0x503E}, {0x50C2,0x507B}, {0x50C5,0x4EC5}, {0x50C9,0x4F65}, {0x50D1,0x4FA8}, 
+  {0x50D5,0x4EC6}, {0x50DE,0x4F2A}, {0x50E5,0x4FA5}, {0x50E8,0x507E}, {0x50F9,0x4EF7}, 
+  {0x5100,0x4EEA}, {0x5102,0x4FAC}, {0x5104,0x4EBF}, {0x5108,0x4FA9}, {0x5109,0x4FED}, 
+  {0x5110,0x50A7}, {0x5114,0x4FE6}, {0x5115,0x4FAA}, {0x511F,0x507F}, {0x512A,0x4F18}, 
+  {0x5132,0x50A8}, {0x5137,0x4FEA}, {0x513A,0x50A9}, {0x513B,0x50A5}, {0x513C,0x4FE8}, 
+  {0x5152,0x513F}, {0x5169,0x4E24}, {0x51C8,0x51C0}, {0x51CD,0x51BB}, {0x51F1,0x51EF}, 
+  {0x5244,0x522D}, {0x5247,0x5219}, {0x525B,0x521A}, {0x526E,0x5250}, {0x5274,0x5240}, 
+  {0x5275,0x521B}, {0x5283,0x5212}, {0x5287,0x5267}, {0x5289,0x5218}, {0x528A,0x523D}, 
+  {0x528C,0x523F}, {0x528D,0x5251}, {0x5291,0x5242}, {0x52C1,0x52B2}, {0x52D5,0x52A8}, 
+  {0x52D9,0x52A1}, {0x52DB,0x52CB}, {0x52DD,0x80DC}, {0x52DE,0x52B3}, {0x52E2,0x52BF}, 
+  {0x52F1,0x52A2}, {0x52F5,0x52B1}, {0x52F8,0x529D}, {0x532D,0x5326}, {0x532F,0x6C47}, 
+  {0x5331,0x532E}, {0x5340,0x533A}, {0x5354,0x534F}, {0x5399,0x538D}, {0x53A0,0x5395}, 
+  {0x53AD,0x538C}, {0x53B2,0x5389}, {0x53B4,0x53A3}, {0x53C3,0x53C2}, {0x53E2,0x4E1B}, 
+  {0x540E,0x5F8C}, {0x54BC,0x5459}, {0x54E1,0x5458}, {0x5504,0x5457}, {0x554F,0x95EE}, 
+  {0x5553,0x542F}, {0x555E,0x54D1}, {0x559A,0x5524}, {0x55AA,0x4E27}, {0x55AC,0x4E54}, 
+  {0x55AE,0x5355}, {0x55B2,0x54DF}, {0x55C6,0x545B}, {0x55C7,0x556C}, {0x55CE,0x5417}, 
+  {0x55DA,0x545C}, {0x55E9,0x5522}, {0x55F6,0x54D4}, {0x5606,0x53F9}, {0x560D,0x55BD}, 
+  {0x5614,0x5455}, {0x5616,0x5567}, {0x5617,0x5C1D}, {0x561C,0x551B}, {0x5629,0x54D7}, 
+  {0x562E,0x5520}, {0x562F,0x5578}, {0x5630,0x53FD}, {0x5635,0x54D3}, {0x5638,0x5452}, 
+  {0x565D,0x549D}, {0x5660,0x54D2}, {0x5665,0x54DD}, {0x5666,0x54D5}, {0x566F,0x55F3}, 
+  {0x5672,0x54D9}, {0x5674,0x55B7}, {0x5678,0x5428}, {0x5680,0x549B}, {0x5687,0x5413}, 
+  {0x568C,0x54DC}, {0x5695,0x565C}, {0x5699,0x556E}, {0x56A6,0x5456}, {0x56A8,0x5499}, 
+  {0x56B3,0x55BE}, {0x56B4,0x4E25}, {0x56B6,0x5624}, {0x56C0,0x556D}, {0x56C1,0x55EB}, 
+  {0x56C2,0x56A3}, {0x56C5,0x5181}, {0x56C8,0x5453}, {0x56D1,0x5631}, {0x5707,0x56F5}, 
+  {0x570B,0x56FD}, {0x570D,0x56F4}, {0x5712,0x56ED}, {0x5713,0x5706}, {0x5716,0x56FE}, 
+  {0x5718,0x56E2}, {0x57E1,0x57AD}, {0x57F7,0x6267}, {0x5805,0x575A}, {0x580A,0x57A9}, 
+  {0x5816,0x57B4}, {0x581D,0x57DA}, {0x582F,0x5C27}, {0x5831,0x62A5}, {0x5834,0x573A}, 
+  {0x584A,0x5757}, {0x584B,0x8314}, {0x584F,0x57B2}, {0x5852,0x57D8}, {0x5857,0x6D82}, 
+  {0x5862,0x575E}, {0x5864,0x57D9}, {0x5875,0x5C18}, {0x5879,0x5811}, {0x588A,0x57AB}, 
+  {0x589C,0x5760}, {0x58AE,0x5815}, {0x58B3,0x575F}, {0x58BB,0x5899}, {0x58BE,0x57A6}, 
+  {0x58C7,0x575B}, {0x58D3,0x538B}, {0x58D8,0x5792}, {0x58D9,0x5739}, {0x58DA,0x5786}, 
+  {0x58DE,0x574F}, {0x58DF,0x5784}, {0x58E0,0x5785}, {0x58E2,0x575C}, {0x58E9,0x575D}, 
+  {0x58EF,0x58EE}, {0x58FA,0x58F6}, {0x58FD,0x5BFF}, {0x5922,0x68A6}, {0x593E,0x5939}, 
+  {0x5950,0x5942}, {0x5969,0x5941}, {0x596A,0x593A}, {0x596C,0x5956}, {0x596E,0x594B}, 
+  {0x599D,0x5986}, {0x5A41,0x5A04}, {0x5A66,0x5987}, {0x5A6D,0x5A05}, {0x5AA7,0x5A32}, 
+  {0x5ABD,0x5988}, {0x5AD7,0x59AA}, {0x5AF5,0x59A9}, {0x5AFB,0x5A34}, {0x5B00,0x59AB}, 
+  {0x5B08,0x5A06}, {0x5B0B,0x5A75}, {0x5B0C,0x5A07}, {0x5B19,0x5AF1}, {0x5B21,0x5AD2}, 
+  {0x5B2A,0x5AD4}, {0x5B30,0x5A74}, {0x5B38,0x5A76}, {0x5B4C,0x5A08}, {0x5B6B,0x5B59}, 
+  {0x5B78,0x5B66}, {0x5B7F,0x5B6A}, {0x5BE2,0x5BDD}, {0x5BE6,0x5B9E}, {0x5BE7,0x5B81}, 
+  {0x5BE9,0x5BA1}, {0x5BEB,0x5199}, {0x5BEC,0x5BBD}, {0x5BF5,0x5BA0}, {0x5BF6,0x5B9D}, 
+  {0x5C07,0x5C06}, {0x5C08,0x4E13}, {0x5C0B,0x5BFB}, {0x5C0D,0x5BF9}, {0x5C0E,0x5BFC}, 
+  {0x5C37,0x5C34}, {0x5C62,0x5C61}, {0x5C64,0x5C42}, {0x5C68,0x5C66}, {0x5C6C,0x5C5E}, 
+  {0x5CA1,0x5188}, {0x5CF4,0x5C98}, {0x5CF6,0x5C9B}, {0x5CFD,0x5CE1}, {0x5D0D,0x5D03}, 
+  {0x5D17,0x5C97}, {0x5D22,0x5CE5}, {0x5D2C,0x5CBD}, {0x5D50,0x5C9A}, {0x5D81,0x5D5D}, 
+  {0x5D84,0x5D2D}, {0x5D87,0x5C96}, {0x5D97,0x5D02}, {0x5DA0,0x5CE4}, {0x5DA7,0x5CC4}, 
+  {0x5DB8,0x5D58}, {0x5DBA,0x5CAD}, {0x5DBC,0x5C7F}, {0x5DCB,0x5CBF}, {0x5DD2,0x5CE6}, 
+  {0x5DD4,0x5DC5}, {0x5DF0,0x5DEF}, {0x5E25,0x5E05}, {0x5E2B,0x5E08}, {0x5E33,0x5E10}, 
+  {0x5E36,0x5E26}, {0x5E40,0x5E27}, {0x5E43,0x5E0F}, {0x5E57,0x5E3C}, {0x5E58,0x5E3B}, 
+  {0x5E5F,0x5E1C}, {0x5E63,0x5E01}, {0x5E6B,0x5E2E}, {0x5E6C,0x5E31}, {0x5E79,0x5E72}, 
+  {0x5E7E,0x51E0}, {0x5EAB,0x5E93}, {0x5EDF,0x5E99}, {0x5EE0,0x5382}, {0x5EE1,0x5E91}, 
+  {0x5EE2,0x5E9F}, {0x5EE3,0x5E7F}, {0x5EEC,0x5E90}, {0x5EF3,0x5385}, {0x5F33,0x5F2A}, 
+  {0x5F35,0x5F20}, {0x5F48,0x5F39}, {0x5F4C,0x5F25}, {0x5F4E,0x5F2F}, {0x5F8C,0x540E}, 
+  {0x5F91,0x5F84}, {0x5F9E,0x4ECE}, {0x5FA0,0x5F95}, {0x5FA9,0x590D}, {0x5FB9,0x5F7B}, 
+  {0x60B5,0x6005}, {0x60B6,0x95F7}, {0x60E1,0x6076}, {0x60F1,0x607C}, {0x60F2,0x607D}, 
+  {0x60FB,0x607B}, {0x611B,0x7231}, {0x611C,0x60EC}, {0x6134,0x6006}, {0x6137,0x607A}, 
+  {0x613E,0x5FFE}, {0x614B,0x6001}, {0x6158,0x60E8}, {0x615A,0x60ED}, {0x615F,0x6078}, 
+  {0x6163,0x60EF}, {0x6164,0x60AB}, {0x616A,0x6004}, {0x616B,0x6002}, {0x616E,0x8651}, 
+  {0x6173,0x60AD}, {0x6176,0x5E86}, {0x6182,0x5FE7}, {0x618A,0x60EB}, {0x6190,0x601C}, 
+  {0x6191,0x51ED}, {0x6192,0x6126}, {0x619A,0x60EE}, {0x61A4,0x6124}, {0x61AB,0x60AF}, 
+  {0x61AE,0x6003}, {0x61B2,0x5BAA}, {0x61B6,0x5FC6}, {0x61C7,0x6073}, {0x61C9,0x5E94}, 
+  {0x61CC,0x603F}, {0x61DF,0x603C}, {0x61E3,0x61D1}, {0x61E8,0x6079}, {0x61F2,0x60E9}, 
+  {0x61F6,0x61D2}, {0x61F7,0x6000}, {0x61F8,0x60AC}, {0x61FA,0x5FCF}, {0x61FC,0x60E7}, 
+  {0x61FE,0x6151}, {0x6200,0x604B}, {0x6207,0x6206}, {0x6214,0x620B}, {0x6227,0x6217}, 
+  {0x6229,0x622C}, {0x6230,0x6218}, {0x6232,0x620F}, {0x633E,0x631F}, {0x6368,0x820D}, 
+  {0x636B,0x626A}, {0x6383,0x626B}, {0x6384,0x62A1}, {0x6399,0x6323}, {0x63C0,0x62E3}, 
+  {0x63DA,0x626C}, {0x63DB,0x6362}, {0x63EE,0x6325}, {0x640D,0x635F}, {0x6417,0x6363}, 
+  {0x6436,0x62A2}, {0x6451,0x63B4}, {0x645C,0x63BC}, {0x645F,0x6402}, {0x646F,0x631A}, 
+  {0x6473,0x62A0}, {0x6476,0x629F}, {0x647B,0x63BA}, {0x6488,0x635E}, {0x6493,0x6320}, 
+  {0x649F,0x6322}, {0x64A3,0x63B8}, {0x64A5,0x62E8}, {0x64AB,0x629A}, {0x64B2,0x6251}, 
+  {0x64B3,0x63FF}, {0x64BB,0x631E}, {0x64BE,0x631D}, {0x64BF,0x6361}, {0x64C1,0x62E5}, 
+  {0x64C4,0x63B3}, {0x64C7,0x62E9}, {0x64CA,0x51FB}, {0x64CB,0x6321}, {0x64D4,0x62C5}, 
+  {0x64DA,0x636E}, {0x64E0,0x6324}, {0x64EC,0x62DF}, {0x64EF,0x6448}, {0x64F0,0x62E7}, 
+  {0x64F1,0x6401}, {0x64F2,0x63B7}, {0x64F4,0x6269}, {0x64F7,0x64B7}, {0x64FA,0x6446}, 
+  {0x64FB,0x64DE}, {0x64FC,0x64B8}, {0x64FE,0x6270}, {0x6504,0x6445}, {0x6506,0x64B5}, 
+  {0x650F,0x62E2}, {0x6514,0x62E6}, {0x6516,0x6484}, {0x6519,0x6400}, {0x651B,0x64BA}, 
+  {0x651D,0x6444}, {0x6522,0x6512}, {0x6523,0x631B}, {0x6524,0x644A}, {0x652A,0x6405}, 
+  {0x652C,0x63FD}, {0x6557,0x8D25}, {0x6575,0x654C}, {0x6578,0x6570}, {0x6582,0x655B}, 
+  {0x6583,0x6BD9}, {0x6595,0x6593}, {0x65AC,0x65A9}, {0x65B7,0x65AD}, {0x6642,0x65F6}, 
+  {0x6649,0x664B}, {0x665D,0x663C}, {0x6688,0x6655}, {0x6689,0x6656}, {0x66A2,0x7545}, 
+  {0x66AB,0x6682}, {0x66C4,0x6654}, {0x66C7,0x6619}, {0x66C9,0x6653}, {0x66D6,0x66A7}, 
+  {0x66E0,0x65F7}, {0x66EC,0x6652}, {0x66F8,0x4E66}, {0x6703,0x4F1A}, {0x6727,0x80E7}, 
+  {0x6771,0x4E1C}, {0x6898,0x67A7}, {0x689D,0x6761}, {0x689F,0x67AD}, {0x68C4,0x5F03}, 
+  {0x68D6,0x67A8}, {0x68D7,0x67A3}, {0x68DF,0x680B}, {0x68E7,0x6808}, {0x68F2,0x6816}, 
+  {0x690F,0x6860}, {0x694A,0x6768}, {0x6953,0x67AB}, {0x6968,0x6862}, {0x696D,0x4E1A}, 
+  {0x6975,0x6781}, {0x69AA,0x6769}, {0x69AE,0x8363}, {0x69BF,0x6864}, {0x69CB,0x6784}, 
+  {0x69CD,0x67AA}, {0x69E7,0x6920}, {0x69F3,0x6868}, {0x6A01,0x6869}, {0x6A02,0x4E50}, 
+  {0x6A05,0x679E}, {0x6A13,0x697C}, {0x6A19,0x6807}, {0x6A1E,0x67A2}, {0x6A23,0x6837}, 
+  {0x6A38,0x6734}, {0x6A39,0x6811}, {0x6A3A,0x6866}, {0x6A48,0x6861}, {0x6A4B,0x6865}, 
+  {0x6A5F,0x673A}, {0x6A62,0x692D}, {0x6A89,0x67FD}, {0x6A94,0x6863}, {0x6A9C,0x6867}, 
+  {0x6AA2,0x68C0}, {0x6AA3,0x6A2F}, {0x6AB3,0x69DF}, {0x6AB8,0x67E0}, {0x6ABB,0x69DB}, 
+  {0x6AC3,0x67DC}, {0x6AD3,0x6A79}, {0x6ADA,0x6988}, {0x6ADB,0x6809}, {0x6ADD,0x691F}, 
+  {0x6ADE,0x6A7C}, {0x6ADF,0x680E}, {0x6AE7,0x69E0}, {0x6AE8,0x680C}, {0x6AEA,0x67A5}, 
+  {0x6AEC,0x6987}, {0x6AF3,0x680A}, {0x6AF8,0x6989}, {0x6AFB,0x6A31}, {0x6B04,0x680F}, 
+  {0x6B0A,0x6743}, {0x6B0F,0x6924}, {0x6B12,0x683E}, {0x6B16,0x6984}, {0x6B1E,0x68C2}, 
+  {0x6B3D,0x94A6}, {0x6B50,0x6B27}, {0x6B5F,0x6B24}, {0x6B61,0x6B22}, {0x6B72,0x5C81}, 
+  {0x6B77,0x5386}, {0x6B78,0x5F52}, {0x6B98,0x6B8B}, {0x6B9E,0x6B92}, {0x6BA4,0x6B87}, 
+  {0x6BAB,0x6B9A}, {0x6BAE,0x6B93}, {0x6BAF,0x6BA1}, {0x6BB2,0x6B7C}, {0x6BBA,0x6740}, 
+  {0x6BBB,0x58F3}, {0x6BC6,0x6BB4}, {0x6BFF,0x6BF5}, {0x6C08,0x6BE1}, {0x6C0C,0x6C07}, 
+  {0x6C23,0x6C14}, {0x6C2B,0x6C22}, {0x6C2C,0x6C29}, {0x6D79,0x6D43}, {0x6D87,0x6CFE}, 
+  {0x6DEA,0x6CA6}, {0x6DF5,0x6E0A}, {0x6DF6,0x6D9E}, {0x6DFA,0x6D45}, {0x6E19,0x6DA3}, 
+  {0x6E26,0x6DA1}, {0x6E2C,0x6D4B}, {0x6E3E,0x6D51}, {0x6E5E,0x6D48}, {0x6E6F,0x6C64}, 
+  {0x6E96,0x51C6}, {0x6E9D,0x6C9F}, {0x6EC4,0x6CA7}, {0x6EC5,0x706D}, {0x6ECC,0x6DA4}, 
+  {0x6ECE,0x8365}, {0x6EEC,0x6CAA}, {0x6EEF,0x6EDE}, {0x6EF2,0x6E17}, {0x6EF7,0x5364}, 
+  {0x6EF8,0x6D52}, {0x6EFF,0x6EE1}, {0x6F01,0x6E14}, {0x6F1A,0x6CA4}, {0x6F22,0x6C49}, 
+  {0x6F23,0x6D9F}, {0x6F2C,0x6E0D}, {0x6F32,0x6DA8}, {0x6F38,0x6E10}, {0x6F3F,0x6D46}, 
+  {0x6F41,0x988D}, {0x6F51,0x6CFC}, {0x6F54,0x6D01}, {0x6F59,0x6CA9}, {0x6F5B,0x6F5C}, 
+  {0x6F64,0x6DA6}, {0x6F6F,0x6D54}, {0x6F70,0x6E83}, {0x6F77,0x6ED7}, {0x6F7F,0x6DA0}, 
+  {0x6F80,0x6DA9}, {0x6F86,0x6D47}, {0x6F87,0x6D9D}, {0x6F97,0x6DA7}, {0x6FA0,0x6E11}, 
+  {0x6FA4,0x6CFD}, {0x6FA9,0x6CF6}, {0x6FAE,0x6D4D}, {0x6FB1,0x6DC0}, {0x6FC1,0x6D4A}, 
+  {0x6FC3,0x6D53}, {0x6FD5,0x6E7F}, {0x6FD8,0x6CDE}, {0x6FDF,0x6D4E}, {0x6FE4,0x6D9B}, 
+  {0x6FEB,0x6EE5}, {0x6FF0,0x6F4D}, {0x6FF1,0x6EE8}, {0x6FFA,0x6E85}, {0x6FFC,0x6CFA}, 
+  {0x6FFE,0x6EE4}, {0x7005,0x6EE2}, {0x7006,0x6E0E}, {0x7009,0x6CFB}, {0x700B,0x6E16}, 
+  {0x700F,0x6D4F}, {0x7015,0x6FD2}, {0x7018,0x6CF8}, {0x701D,0x6CA5}, {0x701F,0x6F47}, 
+  {0x7020,0x6F46}, {0x7027,0x6CF7}, {0x7028,0x6FD1}, {0x7032,0x6F4B}, {0x703E,0x6F9C}, 
+  {0x7043,0x6CA3}, {0x7044,0x6EE0}, {0x7051,0x6D12}, {0x7055,0x6F13}, {0x7058,0x6EE9}, 
+  {0x705D,0x704F}, {0x7063,0x6E7E}, {0x7064,0x6EE6}, {0x7067,0x6EDF}, {0x70CF,0x4E4C}, 
+  {0x70F4,0x70C3}, {0x7121,0x65E0}, {0x7149,0x70BC}, {0x7152,0x709C}, {0x7162,0x8315}, 
+  {0x7165,0x7115}, {0x7169,0x70E6}, {0x716C,0x7080}, {0x7192,0x8367}, {0x7197,0x709D}, 
+  {0x71B1,0x70ED}, {0x71BE,0x70BD}, {0x71C1,0x70E8}, {0x71C8,0x706F}, {0x71D2,0x70E7}, 
+  {0x71D9,0x70EB}, {0x71DC,0x7116}, {0x71DF,0x8425}, {0x71E6,0x707F}, {0x71ED,0x70DB}, 
+  {0x71F4,0x70E9}, {0x71FC,0x70EC}, {0x71FE,0x7118}, {0x720D,0x70C1}, {0x7210,0x7089}, 
+  {0x721B,0x70C2}, {0x722D,0x4E89}, {0x7232,0x4E3A}, {0x723A,0x7237}, {0x723E,0x5C14}, 
+  {0x7258,0x724D}, {0x727D,0x7275}, {0x7296,0x8366}, {0x72A2,0x728A}, {0x72A7,0x727A}, 
+  {0x72C0,0x72B6}, {0x72F9,0x72ED}, {0x72FD,0x72C8}, {0x7319,0x72F0}, {0x7336,0x72B9}, 
+  {0x733B,0x72F2}, {0x7341,0x72B8}, {0x7344,0x72F1}, {0x7345,0x72EE}, {0x7368,0x72EC}, 
+  {0x736A,0x72EF}, {0x736B,0x7303}, {0x7370,0x72DE}, {0x7372,0x83B7}, {0x7375,0x730E}, 
+  {0x7377,0x72B7}, {0x7378,0x517D}, {0x737A,0x736D}, {0x737B,0x732E}, {0x737C,0x7315}, 
+  {0x7380,0x7321}, {0x73FE,0x73B0}, {0x743F,0x73F2}, {0x744B,0x73AE}, {0x7463,0x7410}, 
+  {0x7469,0x83B9}, {0x746A,0x739B}, {0x7489,0x740F}, {0x74A3,0x7391}, {0x74A6,0x7477}, 
+  {0x74B0,0x73AF}, {0x74BD,0x73BA}, {0x74CA,0x743C}, {0x74CF,0x73D1}, {0x74D4,0x748E}, 
+  {0x74DA,0x74D2}, {0x750C,0x74EF}, {0x7523,0x4EA7}, {0x755D,0x4EA9}, {0x7562,0x6BD5}, 
+  {0x756B,0x753B}, {0x7576,0x5F53}, {0x7587,0x7574}, {0x758A,0x53E0}, {0x75D9,0x75C9}, 
+  {0x7602,0x75D6}, {0x760B,0x75AF}, {0x760D,0x75A1}, {0x7613,0x75EA}, {0x761E,0x7617}, 
+  {0x7621,0x75AE}, {0x7627,0x759F}, {0x763B,0x7618}, {0x7642,0x7597}, {0x7646,0x75E8}, 
+  {0x7647,0x75EB}, {0x7649,0x7605}, {0x7658,0x75A0}, {0x765F,0x762A}, {0x7662,0x75D2}, 
+  {0x7664,0x7596}, {0x7665,0x75C7}, {0x7667,0x75AC}, {0x7669,0x765E}, {0x766C,0x7663}, 
+  {0x766D,0x763F}, {0x766E,0x763E}, {0x7670,0x75C8}, {0x7671,0x762B}, {0x7672,0x766B}, 
+  {0x767C,0x53D1}, {0x769A,0x7691}, {0x76B8,0x76B2}, {0x76BA,0x76B1}, {0x76DE,0x76CF}, 
+  {0x76E1,0x5C3D}, {0x76E3,0x76D1}, {0x76E4,0x76D8}, {0x76E7,0x5362}, {0x775C,0x7741}, 
+  {0x775E,0x7750}, {0x7798,0x770D}, {0x779E,0x7792}, {0x77BC,0x7751}, {0x77DA,0x77A9}, 
+  {0x77EF,0x77EB}, {0x7864,0x7856}, {0x7868,0x7817}, {0x786F,0x781A}, {0x78A9,0x7855}, 
+  {0x78AD,0x7800}, {0x78B8,0x781C}, {0x78BA,0x786E}, {0x78BC,0x7801}, {0x78DA,0x7816}, 
+  {0x78E3,0x789C}, {0x78E7,0x789B}, {0x78EF,0x77F6}, {0x78FD,0x7857}, {0x790E,0x7840}, 
+  {0x7919,0x788D}, {0x7926,0x77FF}, {0x792A,0x783A}, {0x792B,0x783E}, {0x792C,0x77FE}, 
+  {0x7931,0x783B}, {0x798D,0x7978}, {0x798E,0x796F}, {0x79AA,0x7985}, {0x79AE,0x793C}, 
+  {0x79B0,0x7962}, {0x79B1,0x7977}, {0x7A2E,0x79CD}, {0x7A31,0x79F0}, {0x7A4C,0x7A23}, 
+  {0x7A4D,0x79EF}, {0x7A4E,0x9896}, {0x7A61,0x7A51}, {0x7A62,0x79FD}, {0x7A69,0x7A33}, 
+  {0x7AA9,0x7A9D}, {0x7AAA,0x6D3C}, {0x7AAE,0x7A77}, {0x7AB6,0x7AAD}, {0x7ABA,0x7AA5}, 
+  {0x7AC4,0x7A9C}, {0x7AC5,0x7A8D}, {0x7AC7,0x7AA6}, {0x7AC8,0x7076}, {0x7ACA,0x7A83}, 
+  {0x7AEA,0x7AD6}, {0x7AF6,0x7ADE}, {0x7B46,0x7B14}, {0x7B67,0x7B15}, {0x7B8B,0x7B3A}, 
+  {0x7B8F,0x7B5D}, {0x7BC0,0x8282}, {0x7BC4,0x8303}, {0x7BC9,0x7B51}, {0x7BCB,0x7BA7}, 
+  {0x7BE4,0x7B03}, {0x7BE9,0x7B5B}, {0x7BF3,0x7B5A}, {0x7C00,0x7BA6}, {0x7C0D,0x7BD3}, 
+  {0x7C1E,0x7BAA}, {0x7C21,0x7B80}, {0x7C23,0x7BD1}, {0x7C2B,0x7BAB}, {0x7C3D,0x7B7E}, 
+  {0x7C3E,0x5E18}, {0x7C43,0x7BEE}, {0x7C4C,0x7B79}, {0x7C5C,0x7BA8}, {0x7C5F,0x7C41}, 
+  {0x7C60,0x7B3C}, {0x7C69,0x7B3E}, {0x7C6A,0x7C16}, {0x7C6C,0x7BF1}, {0x7C6E,0x7BA9}, 
+  {0x7CDD,0x7CC1}, {0x7CDE,0x7CAA}, {0x7CE7,0x7CAE}, {0x7CF2,0x7C9D}, {0x7CF4,0x7C74}, 
+  {0x7CF6,0x7C9C}, {0x7CF9,0x7E9F}, {0x7CFE,0x7EA0}, {0x7D00,0x7EAA}, {0x7D02,0x7EA3}, 
+  {0x7D04,0x7EA6}, {0x7D05,0x7EA2}, {0x7D06,0x7EA1}, {0x7D07,0x7EA5}, {0x7D08,0x7EA8}, 
+  {0x7D09,0x7EAB}, {0x7D0B,0x7EB9}, {0x7D0D,0x7EB3}, {0x7D10,0x7EBD}, {0x7D13,0x7EBE}, 
+  {0x7D14,0x7EAF}, {0x7D15,0x7EB0}, {0x7D17,0x7EB1}, {0x7D19,0x7EB8}, {0x7D1A,0x7EA7}, 
+  {0x7D1B,0x7EB7}, {0x7D1C,0x7EAD}, {0x7D21,0x7EBA}, {0x7D30,0x7EC6}, {0x7D31,0x7EC2}, 
+  {0x7D32,0x7EC1}, {0x7D33,0x7EC5}, {0x7D39,0x7ECD}, {0x7D3A,0x7EC0}, {0x7D3C,0x7ECB}, 
+  {0x7D3F,0x7ED0}, {0x7D40,0x7ECC}, {0x7D42,0x7EC8}, {0x7D44,0x7EC4}, {0x7D46,0x7ECA}, 
+  {0x7D4E,0x7ED7}, {0x7D50,0x7ED3}, {0x7D5D,0x7ED4}, {0x7D5E,0x7EDE}, {0x7D61,0x7EDC}, 
+  {0x7D62,0x7EDA}, {0x7D66,0x7ED9}, {0x7D68,0x7ED2}, {0x7D71,0x7EDF}, {0x7D72,0x4E1D}, 
+  {0x7D73,0x7EDB}, {0x7D76,0x7EDD}, {0x7D79,0x7EE2}, {0x7D81,0x7ED1}, {0x7D83,0x7EE1}, 
+  {0x7D86,0x7EE0}, {0x7D88,0x7EE8}, {0x7D89,0x7EE3}, {0x7D8F,0x7EE5}, {0x7D93,0x7ECF}, 
+  {0x7D9C,0x7EFC}, {0x7D9E,0x7F0D}, {0x7DA2,0x7EF8}, {0x7DA3,0x7EFB}, {0x7DAB,0x7EBF}, 
+  {0x7DAC,0x7EF6}, {0x7DAD,0x7EF4}, {0x7DB0,0x7EFE}, {0x7DB1,0x7EB2}, {0x7DB2,0x7F51}, 
+  {0x7DB3,0x7EF7}, {0x7DB4,0x7F00}, {0x7DB8,0x7EB6}, {0x7DB9,0x7EFA}, {0x7DBA,0x7EEE}, 
+  {0x7DBB,0x7EFD}, {0x7DBD,0x7EF0}, {0x7DBE,0x7EEB}, {0x7DBF,0x7EF5}, {0x7DC4,0x7EF2}, 
+  {0x7DC7,0x7F01}, {0x7DCA,0x7D27}, {0x7DCB,0x7EEF}, {0x7DD1,0x7EFF}, {0x7DD2,0x7EEA}, 
+  {0x7DD4,0x7EF1}, {0x7DD7,0x7F03}, {0x7DD8,0x7F04}, {0x7DD9,0x7F02}, {0x7DDD,0x7F09}, 
+  {0x7DDE,0x7F0E}, {0x7DE0,0x7F14}, {0x7DE1,0x7F17}, {0x7DE3,0x7F18}, {0x7DE6,0x7F0C}, 
+  {0x7DE8,0x7F16}, {0x7DE9,0x7F13}, {0x7DEC,0x7F05}, {0x7DEF,0x7EAC}, {0x7DF1,0x7F11}, 
+  {0x7DF2,0x7F08}, {0x7DF4,0x7EC3}, {0x7DF6,0x7F0F}, {0x7DF9,0x7F07}, {0x7E08,0x8426}, 
+  {0x7E09,0x7F19}, {0x7E0A,0x7F22}, {0x7E0B,0x7F12}, {0x7E10,0x7EC9}, {0x7E11,0x7F23}, 
+  {0x7E1B,0x7F1A}, {0x7E1D,0x7F1C}, {0x7E1E,0x7F1F}, {0x7E1F,0x7F1B}, {0x7E23,0x53BF}, 
+  {0x7E27,0x7EE6}, {0x7E2B,0x7F1D}, {0x7E2D,0x7F21}, {0x7E2E,0x7F29}, {0x7E31,0x7EB5}, 
+  {0x7E32,0x7F27}, {0x7E35,0x7F26}, {0x7E36,0x7D77}, {0x7E37,0x7F15}, {0x7E39,0x7F25}, 
+  {0x7E3D,0x603B}, {0x7E3E,0x7EE9}, {0x7E45,0x7F2B}, {0x7E46,0x7F2A}, {0x7E52,0x7F2F}, 
+  {0x7E54,0x7EC7}, {0x7E55,0x7F2E}, {0x7E5A,0x7F2D}, {0x7E5E,0x7ED5}, {0x7E62,0x7F0B}, 
+  {0x7E69,0x7EF3}, {0x7E6A,0x7ED8}, {0x7E6D,0x8327}, {0x7E6E,0x7F30}, {0x7E6F,0x7F33}, 
+  {0x7E70,0x7F32}, {0x7E73,0x7F34}, {0x7E79,0x7ECE}, {0x7E7C,0x7EE7}, {0x7E7D,0x7F24}, 
+  {0x7E7E,0x7F31}, {0x7E88,0x7F2C}, {0x7E8A,0x7EA9}, {0x7E8C,0x7EED}, {0x7E8F,0x7F20}, 
+  {0x7E93,0x7F28}, {0x7E96,0x7EA4}, {0x7E98,0x7F35}, {0x7E9C,0x7F06}, {0x7F4C,0x7F42}, 
+  {0x7F70,0x7F5A}, {0x7F77,0x7F62}, {0x7F85,0x7F57}, {0x7F86,0x7F74}, {0x7F88,0x7F81}, 
+  {0x7FA5,0x7F9F}, {0x7FA9,0x4E49}, {0x7FD2,0x4E60}, {0x7FF9,0x7FD8}, {0x802C,0x8027}, 
+  {0x802E,0x8022}, {0x8056,0x5723}, {0x805E,0x95FB}, {0x806F,0x8054}, {0x8070,0x806A}, 
+  {0x8072,0x58F0}, {0x8073,0x8038}, {0x8075,0x8069}, {0x8076,0x8042}, {0x8077,0x804C}, 
+  {0x8079,0x804D}, {0x807D,0x542C}, {0x807E,0x804B}, {0x8085,0x8083}, {0x8105,0x80C1}, 
+  {0x811B,0x80EB}, {0x8139,0x80C0}, {0x814E,0x80BE}, {0x8156,0x80E8}, {0x8161,0x8136}, 
+  {0x8166,0x8111}, {0x816B,0x80BF}, {0x8178,0x80A0}, {0x819A,0x80A4}, {0x81A0,0x80F6}, 
+  {0x81A9,0x817B}, {0x81BD,0x80C6}, {0x81BE,0x810D}, {0x81BF,0x8113}, {0x81C9,0x8138}, 
+  {0x81CD,0x8110}, {0x81CF,0x8191}, {0x81D8,0x814A}, {0x81DA,0x80EA}, {0x81DF,0x810F}, 
+  {0x81E0,0x8114}, {0x81E8,0x4E34}, {0x81FA,0x53F0}, {0x8207,0x4E0E}, {0x8208,0x5174}, 
+  {0x8209,0x4E3E}, {0x820A,0x65E7}, {0x8259,0x8231}, {0x8264,0x8223}, {0x8266,0x8230}, 
+  {0x826B,0x823B}, {0x8271,0x8270}, {0x8277,0x8273}, {0x82BB,0x520D}, {0x82E7,0x82CE}, 
+  {0x838A,0x5E84}, {0x8396,0x830E}, {0x83A2,0x835A}, {0x83A7,0x82CB}, {0x83EF,0x534E}, 
+  {0x8407,0x82CC}, {0x840A,0x83B1}, {0x842C,0x4E07}, {0x8435,0x83B4}, {0x8449,0x53F6}, 
+  {0x8452,0x836D}, {0x8464,0x836E}, {0x8466,0x82C7}, {0x8477,0x8364}, {0x8493,0x83BC}, 
+  {0x8494,0x83B3}, {0x84BC,0x82CD}, {0x84C0,0x836A}, {0x84CB,0x76D6}, {0x84EE,0x83B2}, 
+  {0x84EF,0x82C1}, {0x84FD,0x835C}, {0x851E,0x848C}, {0x8523,0x848B}, {0x8526,0x8311}, 
+  {0x852D,0x836B}, {0x8541,0x8368}, {0x8546,0x8487}, {0x854E,0x835E}, {0x8552,0x836C}, 
+  {0x8553,0x82B8}, {0x8555,0x83B8}, {0x8558,0x835B}, {0x8562,0x8489}, {0x8569,0x8361}, 
+  {0x856A,0x829C}, {0x856D,0x8427}, {0x8577,0x84E3}, {0x8588,0x835F}, {0x858A,0x84DF}, 
+  {0x858C,0x8297}, {0x8594,0x8537}, {0x859F,0x83B6}, {0x85A6,0x8350}, {0x85A9,0x8428}, 
+  {0x85BA,0x8360}, {0x85CD,0x84DD}, {0x85CE,0x8369}, {0x85DD,0x827A}, {0x85E5,0x836F}, 
+  {0x85EA,0x85AE}, {0x85F4,0x8574}, {0x85F6,0x82C8}, {0x85F9,0x853C}, {0x85FA,0x853A}, 
+  {0x8604,0x8572}, {0x8606,0x82A6}, {0x8607,0x82CF}, {0x860B,0x82F9}, {0x861A,0x85D3}, 
+  {0x861E,0x8539}, {0x8622,0x830F}, {0x862D,0x5170}, {0x863A,0x84E0}, {0x863F,0x841D}, 
+  {0x8655,0x5904}, {0x865C,0x864F}, {0x865F,0x53F7}, {0x8667,0x4E8F}, {0x86FA,0x86F1}, 
+  {0x8706,0x86AC}, {0x8755,0x8680}, {0x8766,0x867E}, {0x8778,0x8717}, {0x8784,0x86F3}, 
+  {0x879E,0x8682}, {0x87A2,0x8424}, {0x87BB,0x877C}, {0x87C4,0x86F0}, {0x87C8,0x8748}, 
+  {0x87CE,0x87A8}, {0x87E3,0x866E}, {0x87EC,0x8749}, {0x87EF,0x86F2}, {0x87F2,0x866B}, 
+  {0x87F6,0x86CF}, {0x87FB,0x8681}, {0x8805,0x8747}, {0x8806,0x867F}, {0x8810,0x86F4}, 
+  {0x8811,0x877E}, {0x881F,0x8721}, {0x8823,0x86CE}, {0x8831,0x86CA}, {0x8836,0x8695}, 
+  {0x883B,0x86EE}, {0x8846,0x4F17}, {0x8853,0x672F}, {0x885B,0x536B}, {0x885D,0x51B2}, 
+  {0x8879,0x53EA}, {0x88CA,0x8885}, {0x88CF,0x91CC}, {0x88DC,0x8865}, {0x88DD,0x88C5}, 
+  {0x8932,0x88E4}, {0x8933,0x88E2}, {0x8938,0x891B}, {0x893B,0x4EB5}, {0x8947,0x88E5}, 
+  {0x8956,0x8884}, {0x895D,0x88E3}, {0x8960,0x88C6}, {0x8964,0x8934}, {0x896A,0x889C}, 
+  {0x896F,0x886C}, {0x8972,0x88AD}, {0x898B,0x89C1}, {0x898F,0x89C4}, {0x8993,0x89C5}, 
+  {0x8996,0x89C6}, {0x8998,0x89C7}, {0x89A1,0x89CB}, {0x89A6,0x89CE}, {0x89AA,0x4EB2}, 
+  {0x89AC,0x89CA}, {0x89AF,0x89CF}, {0x89B2,0x89D0}, {0x89B7,0x89D1}, {0x89BA,0x89C9}, 
+  {0x89BD,0x89C8}, {0x89BF,0x89CC}, {0x89C0,0x89C2}, {0x89F4,0x89DE}, {0x89F6,0x89EF}, 
+  {0x89F8,0x89E6}, {0x8A01,0x8BA0}, {0x8A02,0x8BA2}, {0x8A03,0x8BA3}, {0x8A08,0x8BA1}, 
+  {0x8A0A,0x8BAF}, {0x8A0C,0x8BA7}, {0x8A0E,0x8BA8}, {0x8A10,0x8BA6}, {0x8A13,0x8BAD}, 
+  {0x8A15,0x8BAA}, {0x8A16,0x8BAB}, {0x8A18,0x8BB0}, {0x8A1B,0x8BB9}, {0x8A1D,0x8BB6}, 
+  {0x8A1F,0x8BBC}, {0x8A23,0x8BC0}, {0x8A25,0x8BB7}, {0x8A2A,0x8BBF}, {0x8A2D,0x8BBE}, 
+  {0x8A31,0x8BB8}, {0x8A34,0x8BC9}, {0x8A36,0x8BC3}, {0x8A3A,0x8BCA}, {0x8A41,0x8BC2}, 
+  {0x8A46,0x8BCB}, {0x8A4E,0x8BB5}, {0x8A50,0x8BC8}, {0x8A52,0x8BD2}, {0x8A54,0x8BCF}, 
+  {0x8A55,0x8BC4}, {0x8A58,0x8BCE}, {0x8A5B,0x8BC5}, {0x8A5E,0x8BCD}, {0x8A61,0x8BE9}, 
+  {0x8A62,0x8BE2}, {0x8A63,0x8BE3}, {0x8A66,0x8BD5}, {0x8A69,0x8BD7}, {0x8A6B,0x8BE7}, 
+  {0x8A6C,0x8BDF}, {0x8A6D,0x8BE1}, {0x8A6E,0x8BE0}, {0x8A70,0x8BD8}, {0x8A71,0x8BDD}, 
+  {0x8A72,0x8BE5}, {0x8A73,0x8BE6}, {0x8A75,0x8BDC}, {0x8A7C,0x8BD9}, {0x8A7F,0x8BD6}, 
+  {0x8A84,0x8BD4}, {0x8A85,0x8BDB}, {0x8A86,0x8BD3}, {0x8A87,0x5938}, {0x8A8D,0x8BA4}, 
+  {0x8A91,0x8BF3}, {0x8A92,0x8BF6}, {0x8A95,0x8BDE}, {0x8A98,0x8BF1}, {0x8A9A,0x8BEE}, 
+  {0x8A9E,0x8BED}, {0x8AA0,0x8BDA}, {0x8AA1,0x8BEB}, {0x8AA3,0x8BEC}, {0x8AA4,0x8BEF}, 
+  {0x8AA5,0x8BF0}, {0x8AA6,0x8BF5}, {0x8AA8,0x8BF2}, {0x8AAC,0x8BF4}, {0x8AB0,0x8C01}, 
+  {0x8AB2,0x8BFE}, {0x8AB6,0x8C07}, {0x8AB9,0x8BFD}, {0x8ABC,0x8C0A}, {0x8ABF,0x8C03}, 
+  {0x8AC2,0x8C04}, {0x8AC4,0x8C06}, {0x8AC7,0x8C08}, {0x8AC9,0x8BFF}, {0x8ACB,0x8BF7}, 
+  {0x8ACD,0x8BE4}, {0x8ACF,0x8BF9}, {0x8AD1,0x8BFC}, {0x8AD2,0x8C05}, {0x8AD6,0x8BBA}, 
+  {0x8AD7,0x8C02}, {0x8ADB,0x8C00}, {0x8ADC,0x8C0D}, {0x8ADE,0x8C1D}, {0x8AE2,0x8BE8}, 
+  {0x8AE4,0x8C14}, {0x8AE6,0x8C1B}, {0x8AE7,0x8C10}, {0x8AEB,0x8C0F}, {0x8AED,0x8C15}, 
+  {0x8AEE,0x8C18}, {0x8AF1,0x8BB3}, {0x8AF3,0x8C19}, {0x8AF6,0x8C0C}, {0x8AF7,0x8BBD}, 
+  {0x8AF8,0x8BF8}, {0x8AFA,0x8C1A}, {0x8AFC,0x8C16}, {0x8AFE,0x8BFA}, {0x8B00,0x8C0B}, 
+  {0x8B01,0x8C12}, {0x8B02,0x8C13}, {0x8B04,0x8A8A}, {0x8B05,0x8BCC}, {0x8B0A,0x8C0E}, 
+  {0x8B0E,0x8C1C}, {0x8B10,0x8C27}, {0x8B14,0x8C11}, {0x8B16,0x8C21}, {0x8B17,0x8C24}, 
+  {0x8B19,0x8C26}, {0x8B1A,0x8C25}, {0x8B1B,0x8BB2}, {0x8B1D,0x8C22}, {0x8B21,0x8C23}, 
+  {0x8B28,0x8C1F}, {0x8B2B,0x8C2A}, {0x8B2C,0x8C2C}, {0x8B2D,0x8C2B}, {0x8B33,0x8BB4}, 
+  {0x8B39,0x8C28}, {0x8B3E,0x8C29}, {0x8B49,0x8BC1}, {0x8B4E,0x8C32}, {0x8B4F,0x8BA5}, 
+  {0x8B56,0x8C2E}, {0x8B58,0x8BC6}, {0x8B59,0x8C2F}, {0x8B5A,0x8C2D}, {0x8B5C,0x8C31}, 
+  {0x8B6B,0x8C35}, {0x8B6F,0x8BD1}, {0x8B70,0x8BAE}, {0x8B74,0x8C34}, {0x8B77,0x62A4}, 
+  {0x8B7D,0x8A89}, {0x8B80,0x8BFB}, {0x8B8A,0x53D8}, {0x8B8E,0x96E0}, {0x8B92,0x8C17}, 
+  {0x8B93,0x8BA9}, {0x8B95,0x8C30}, {0x8B96,0x8C36}, {0x8B9C,0x8C20}, {0x8B9E,0x8C33}, 
+  {0x8C48,0x5C82}, {0x8C50,0x4E30}, {0x8C9D,0x8D1D}, {0x8C9E,0x8D1E}, {0x8CA0,0x8D1F}, 
+  {0x8CA1,0x8D22}, {0x8CA2,0x8D21}, {0x8CA7,0x8D2B}, {0x8CA8,0x8D27}, {0x8CA9,0x8D29}, 
+  {0x8CAA,0x8D2A}, {0x8CAB,0x8D2F}, {0x8CAC,0x8D23}, {0x8CAF,0x8D2E}, {0x8CB0,0x8D33}, 
+  {0x8CB2,0x8D40}, {0x8CB3,0x8D30}, {0x8CB4,0x8D35}, {0x8CB6,0x8D2C}, {0x8CB7,0x4E70}, 
+  {0x8CB8,0x8D37}, {0x8CBA,0x8D36}, {0x8CBB,0x8D39}, {0x8CBC,0x8D34}, {0x8CBD,0x8D3B}, 
+  {0x8CBF,0x8D38}, {0x8CC0,0x8D3A}, {0x8CC1,0x8D32}, {0x8CC2,0x8D42}, {0x8CC3,0x8D41}, 
+  {0x8CC4,0x8D3F}, {0x8CC5,0x8D45}, {0x8CC7,0x8D44}, {0x8CC8,0x8D3E}, {0x8CCA,0x8D3C}, 
+  {0x8CD1,0x8D48}, {0x8CD2,0x8D4A}, {0x8CD3,0x5BBE}, {0x8CD5,0x8D47}, {0x8CDA,0x8D49}, 
+  {0x8CDC,0x8D50}, {0x8CDE,0x8D4F}, {0x8CE0,0x8D54}, {0x8CE1,0x8D53}, {0x8CE2,0x8D24}, 
+  {0x8CE3,0x5356}, {0x8CE4,0x8D31}, {0x8CE6,0x8D4B}, {0x8CE7,0x8D55}, {0x8CEA,0x8D28}, 
+  {0x8CEB,0x8D4D}, {0x8CEC,0x8D26}, {0x8CED,0x8D4C}, {0x8CF4,0x8D56}, {0x8CFA,0x8D5A}, 
+  {0x8CFB,0x8D59}, {0x8CFC,0x8D2D}, {0x8CFD,0x8D5B}, {0x8CFE,0x8D5C}, {0x8D04,0x8D3D}, 
+  {0x8D05,0x8D58}, {0x8D08,0x8D60}, {0x8D0A,0x8D5E}, {0x8D0B,0x8D5D}, {0x8D0D,0x8D61}, 
+  {0x8D0F,0x8D62}, {0x8D10,0x8D46}, {0x8D16,0x8D4E}, {0x8D1B,0x8D63}, {0x8D1C,0x8D43}, 
+  {0x8D95,0x8D76}, {0x8D99,0x8D75}, {0x8DA8,0x8D8B}, {0x8DB2,0x8DB1}, {0x8E10,0x8DF5}, 
+  {0x8E34,0x8E0A}, {0x8E4C,0x8DC4}, {0x8E55,0x8DF8}, {0x8E63,0x8E52}, {0x8E7A,0x8DF7}, 
+  {0x8E89,0x8DB8}, {0x8E8A,0x8E0C}, {0x8E8B,0x8DFB}, {0x8E8D,0x8DC3}, {0x8E91,0x8E2F}, 
+  {0x8E92,0x8DDE}, {0x8E93,0x8E2C}, {0x8E9A,0x8DF9}, {0x8EA1,0x8E51}, {0x8EA5,0x8E7F}, 
+  {0x8EA6,0x8E9C}, {0x8EAA,0x8E8F}, {0x8EC0,0x8EAF}, {0x8ECA,0x8F66}, {0x8ECB,0x8F67}, 
+  {0x8ECC,0x8F68}, {0x8ECD,0x519B}, {0x8ED2,0x8F69}, {0x8ED4,0x8F6B}, {0x8EDB,0x8F6D}, 
+  {0x8EDF,0x8F6F}, {0x8EE4,0x8F77}, {0x8EEB,0x8F78}, {0x8EF2,0x8F71}, {0x8EF8,0x8F74}, 
+  {0x8EF9,0x8F75}, {0x8EFA,0x8F7A}, {0x8EFB,0x8F72}, {0x8EFC,0x8F76}, {0x8EFE,0x8F7C}, 
+  {0x8F03,0x8F83}, {0x8F05,0x8F82}, {0x8F07,0x8F81}, {0x8F09,0x8F7D}, {0x8F0A,0x8F7E}, 
+  {0x8F12,0x8F84}, {0x8F14,0x8F85}, {0x8F15,0x8F7B}, {0x8F1B,0x8F86}, {0x8F1C,0x8F8E}, 
+  {0x8F1D,0x8F89}, {0x8F1E,0x8F8B}, {0x8F1F,0x8F8D}, {0x8F25,0x8F8A}, {0x8F26,0x8F87}, 
+  {0x8F29,0x8F88}, {0x8F2A,0x8F6E}, {0x8F2F,0x8F91}, {0x8F33,0x8F8F}, {0x8F38,0x8F93}, 
+  {0x8F3B,0x8F90}, {0x8F3E,0x8F97}, {0x8F3F,0x8206}, {0x8F42,0x6BC2}, {0x8F44,0x8F96}, 
+  {0x8F45,0x8F95}, {0x8F46,0x8F98}, {0x8F49,0x8F6C}, {0x8F4D,0x8F99}, {0x8F4E,0x8F7F}, 
+  {0x8F54,0x8F9A}, {0x8F5F,0x8F70}, {0x8F61,0x8F94}, {0x8F62,0x8F79}, {0x8F64,0x8F73}, 
+  {0x8FA6,0x529E}, {0x8FAD,0x8F9E}, {0x8FAE,0x8FAB}, {0x8FAF,0x8FA9}, {0x8FB2,0x519C}, 
+  {0x9015,0x8FF3}, {0x9019,0x8FD9}, {0x9023,0x8FDE}, {0x9032,0x8FDB}, {0x904B,0x8FD0}, 
+  {0x904E,0x8FC7}, {0x9054,0x8FBE}, {0x9055,0x8FDD}, {0x905C,0x900A}, {0x905E,0x9012}, 
+  {0x9060,0x8FDC}, {0x9069,0x9002}, {0x9072,0x8FDF}, {0x9077,0x8FC1}, {0x9078,0x9009}, 
+  {0x907A,0x9057}, {0x907C,0x8FBD}, {0x9081,0x8FC8}, {0x9084,0x8FD8}, {0x9087,0x8FE9}, 
+  {0x908A,0x8FB9}, {0x908F,0x903B}, {0x9090,0x9026}, {0x90DF,0x90CF}, {0x90F5,0x90AE}, 
+  {0x9106,0x90D3}, {0x9109,0x4E61}, {0x9112,0x90B9}, {0x9114,0x90AC}, {0x9116,0x90E7}, 
+  {0x9127,0x9093}, {0x912D,0x90D1}, {0x9130,0x90BB}, {0x9132,0x90F8}, {0x9134,0x90BA}, 
+  {0x9136,0x90D0}, {0x913A,0x909D}, {0x9148,0x90E6}, {0x9196,0x915D}, {0x919C,0x4E11}, 
+  {0x91AB,0x533B}, {0x91AC,0x9171}, {0x91C0,0x917F}, {0x91C1,0x8845}, {0x91C3,0x917E}, 
+  {0x91C5,0x917D}, {0x91CB,0x91CA}, {0x91D2,0x9485}, {0x91D3,0x9486}, {0x91D4,0x9487}, 
+  {0x91D5,0x948C}, {0x91D7,0x948A}, {0x91D8,0x9489}, {0x91D9,0x948B}, {0x91DD,0x9488}, 
+  {0x91E3,0x9493}, {0x91E4,0x9490}, {0x91E7,0x948F}, {0x91E9,0x9492}, {0x91F5,0x9497}, 
+  {0x91F7,0x948D}, {0x91F9,0x9495}, {0x91FA,0x948E}, {0x9200,0x94AF}, {0x9201,0x94AB}, 
+  {0x9204,0x94AD}, {0x9208,0x949A}, {0x9209,0x94A0}, {0x920D,0x949D}, {0x920E,0x94A9}, 
+  {0x9210,0x94A4}, {0x9211,0x94A3}, {0x9214,0x949E}, {0x9215,0x94AE}, {0x921E,0x94A7}, 
+  {0x9223,0x9499}, {0x9225,0x94AC}, {0x9226,0x949B}, {0x9227,0x94AA}, {0x922E,0x94CC}, 
+  {0x9230,0x94C8}, {0x9233,0x94B6}, {0x9234,0x94C3}, {0x9237,0x94B4}, {0x9238,0x94B9}, 
+  {0x9239,0x94CD}, {0x923A,0x94B0}, {0x923D,0x94B8}, {0x923E,0x94C0}, {0x923F,0x94BF}, 
+  {0x9240,0x94BE}, {0x9245,0x949C}, {0x9248,0x94CA}, {0x9249,0x94C9}, {0x924D,0x94CB}, 
+  {0x9251,0x94C2}, {0x9255,0x94B7}, {0x9257,0x94B3}, {0x925A,0x94C6}, {0x925B,0x94C5}, 
+  {0x925E,0x94BA}, {0x9262,0x94B5}, {0x9266,0x94B2}, {0x926C,0x94BC}, {0x926D,0x94BD}, 
+  {0x9278,0x94F0}, {0x927A,0x94D2}, {0x927B,0x94EC}, {0x927F,0x94EA}, {0x9280,0x94F6}, 
+  {0x9283,0x94F3}, {0x9285,0x94DC}, {0x9291,0x94E3}, {0x9293,0x94E8}, {0x9296,0x94E2}, 
+  {0x9298,0x94ED}, {0x929A,0x94EB}, {0x929C,0x8854}, {0x92A0,0x94D1}, {0x92A3,0x94F7}, 
+  {0x92A5,0x94F1}, {0x92A6,0x94DF}, {0x92A8,0x94F5}, {0x92A9,0x94E5}, {0x92AA,0x94D5}, 
+  {0x92AB,0x94EF}, {0x92AC,0x94D0}, {0x92B1,0x94DE}, {0x92B7,0x9500}, {0x92B9,0x9508}, 
+  {0x92BB,0x9511}, {0x92BC,0x9509}, {0x92C1,0x94DD}, {0x92C3,0x9512}, {0x92C5,0x950C}, 
+  {0x92C7,0x94A1}, {0x92CC,0x94E4}, {0x92CF,0x94D7}, {0x92D2,0x950B}, {0x92DD,0x950A}, 
+  {0x92DF,0x9513}, {0x92E3,0x94D8}, {0x92E4,0x9504}, {0x92E5,0x9503}, {0x92E6,0x9514}, 
+  {0x92E8,0x9507}, {0x92EA,0x94FA}, {0x92ED,0x9510}, {0x92EE,0x94D6}, {0x92EF,0x9506}, 
+  {0x92F0,0x9502}, {0x92F1,0x94FD}, {0x92F6,0x950D}, {0x92F8,0x952F}, {0x92FC,0x94A2}, 
+  {0x9301,0x951E}, {0x9306,0x9516}, {0x9307,0x952B}, {0x9308,0x9529}, {0x9310,0x9525}, 
+  {0x9312,0x9515}, {0x9315,0x951F}, {0x9318,0x9524}, {0x9319,0x9531}, {0x931A,0x94EE}, 
+  {0x931B,0x951B}, {0x931F,0x952C}, {0x9320,0x952D}, {0x9322,0x94B1}, {0x9326,0x9526}, 
+  {0x9328,0x951A}, {0x932B,0x9521}, {0x932E,0x9522}, {0x932F,0x9519}, {0x9332,0x5F55}, 
+  {0x9333,0x9530}, {0x9338,0x94FC}, {0x9340,0x951D}, {0x9341,0x9528}, {0x9343,0x952A}, 
+  {0x9346,0x9494}, {0x9347,0x9534}, {0x934B,0x9505}, {0x934D,0x9540}, {0x9354,0x9537}, 
+  {0x9358,0x94E1}, {0x935B,0x953B}, {0x9364,0x9538}, {0x9365,0x9532}, {0x9369,0x9518}, 
+  {0x936C,0x9539}, {0x9370,0x953E}, {0x9375,0x952E}, {0x9376,0x9536}, {0x937A,0x9517}, 
+  {0x937E,0x953A}, {0x9382,0x9541}, {0x9384,0x953F}, {0x9387,0x9545}, {0x938A,0x9551}, 
+  {0x9396,0x9501}, {0x9398,0x9549}, {0x93A2,0x94A8}, {0x93A3,0x84E5}, {0x93A6,0x954F}, 
+  {0x93A7,0x94E0}, {0x93A9,0x94E9}, {0x93AA,0x953C}, {0x93AC,0x9550}, {0x93AE,0x9547}, 
+  {0x93B0,0x9552}, {0x93B3,0x954D}, {0x93B5,0x9553}, {0x93B8,0x954C}, {0x93BF,0x954E}, 
+  {0x93C3,0x955E}, {0x93C7,0x955F}, {0x93C8,0x94FE}, {0x93CC,0x9546}, {0x93CD,0x9559}, 
+  {0x93D1,0x955D}, {0x93D7,0x94FF}, {0x93D8,0x9535}, {0x93DC,0x9557}, {0x93DD,0x9558}, 
+  {0x93DE,0x955B}, {0x93DF,0x94F2}, {0x93E1,0x955C}, {0x93E2,0x9556}, {0x93E4,0x9542}, 
+  {0x93E8,0x933E}, {0x93F5,0x94E7}, {0x93F7,0x9564}, {0x93F9,0x956A}, {0x9403,0x94D9}, 
+  {0x940B,0x94F4}, {0x9410,0x9563}, {0x9412,0x94F9}, {0x9413,0x9566}, {0x9414,0x9561}, 
+  {0x9418,0x949F}, {0x9419,0x956B}, {0x941D,0x9562}, {0x9420,0x9568}, {0x9426,0x950E}, 
+  {0x9427,0x950F}, {0x9428,0x9544}, {0x942E,0x9570}, {0x9432,0x956F}, {0x9433,0x956D}, 
+  {0x9435,0x94C1}, {0x9438,0x94CE}, {0x943A,0x94DB}, {0x943F,0x9571}, {0x9444,0x94F8}, 
+  {0x944A,0x956C}, {0x944C,0x9554}, {0x9452,0x9274}, {0x9454,0x9572}, {0x9460,0x94C4}, 
+  {0x9463,0x9573}, {0x9465,0x9565}, {0x946D,0x9567}, {0x9470,0x94A5}, {0x9472,0x9576}, 
+  {0x9477,0x954A}, {0x9479,0x9569}, {0x947C,0x9523}, {0x947D,0x94BB}, {0x947E,0x92AE}, 
+  {0x947F,0x51FF}, {0x9577,0x957F}, {0x9580,0x95E8}, {0x9582,0x95E9}, {0x9583,0x95EA}, 
+  {0x9586,0x95EB}, {0x9589,0x95ED}, {0x958B,0x5F00}, {0x958C,0x95F6}, {0x958E,0x95F3}, 
+  {0x958F,0x95F0}, {0x9591,0x95F2}, {0x9593,0x95F4}, {0x9594,0x95F5}, {0x9598,0x95F8}, 
+  {0x95A1,0x9602}, {0x95A3,0x9601}, {0x95A5,0x9600}, {0x95A8,0x95FA}, {0x95A9,0x95FD}, 
+  {0x95AB,0x9603}, {0x95AC,0x9606}, {0x95AD,0x95FE}, {0x95B2,0x9605}, {0x95B6,0x960A}, 
+  {0x95B9,0x9609}, {0x95BB,0x960E}, {0x95BC,0x960F}, {0x95BD,0x960D}, {0x95BE,0x9608}, 
+  {0x95BF,0x960C}, {0x95C3,0x9612}, {0x95C8,0x95F1}, {0x95CA,0x9614}, {0x95CB,0x9615}, 
+  {0x95CC,0x9611}, {0x95D0,0x9617}, {0x95D4,0x9616}, {0x95D5,0x9619}, {0x95D6,0x95EF}, 
+  {0x95DC,0x5173}, {0x95DE,0x961A}, {0x95E1,0x9610}, {0x95E5,0x95FC}, {0x9658,0x9649}, 
+  {0x965D,0x9655}, {0x9663,0x9635}, {0x9670,0x9634}, {0x9673,0x9648}, {0x9678,0x9646}, 
+  {0x967D,0x9633}, {0x9689,0x9667}, {0x968A,0x961F}, {0x968E,0x9636}, {0x9695,0x9668}, 
+  {0x969B,0x9645}, {0x96A8,0x968F}, {0x96AA,0x9669}, {0x96B1,0x9690}, {0x96B4,0x9647}, 
+  {0x96B8,0x96B6}, {0x96D6,0x867D}, {0x96D9,0x53CC}, {0x96DB,0x96CF}, {0x96DC,0x6742}, 
+  {0x96E2,0x79BB}, {0x96E3,0x96BE}, {0x96F2,0x4E91}, {0x96FB,0x7535}, {0x9727,0x96FE}, 
+  {0x973D,0x9701}, {0x9742,0x96F3}, {0x9744,0x972D}, {0x9748,0x7075}, {0x975A,0x9753}, 
+  {0x975C,0x9759}, {0x9768,0x9765}, {0x978F,0x5DE9}, {0x97BD,0x9792}, {0x97C3,0x9791}, 
+  {0x97C9,0x97AF}, {0x97CB,0x97E6}, {0x97CC,0x97E7}, {0x97D3,0x97E9}, {0x97D9,0x97EA}, 
+  {0x97DC,0x97EC}, {0x97DE,0x97EB}, {0x97FF,0x54CD}, {0x9801,0x9875}, {0x9802,0x9876}, 
+  {0x9803,0x9877}, {0x9805,0x9879}, {0x9806,0x987A}, {0x9807,0x9878}, {0x9808,0x987B}, 
+  {0x980A,0x987C}, {0x980C,0x9882}, {0x980E,0x9880}, {0x980F,0x9883}, {0x9810,0x9884}, 
+  {0x9811,0x987D}, {0x9812,0x9881}, {0x9813,0x987F}, {0x9817,0x9887}, {0x9818,0x9886}, 
+  {0x981C,0x988C}, {0x9821,0x9889}, {0x9824,0x9890}, {0x9826,0x988F}, {0x982D,0x5934}, 
+  {0x9830,0x988A}, {0x9837,0x9894}, {0x9838,0x9888}, {0x983B,0x9891}, {0x983D,0x9893}, 
+  {0x9846,0x9897}, {0x984C,0x9898}, {0x984D,0x989D}, {0x984E,0x989A}, {0x9853,0x989B}, 
+  {0x9854,0x989C}, {0x9858,0x613F}, {0x9859,0x98A1}, {0x985B,0x98A0}, {0x985E,0x7C7B}, 
+  {0x9862,0x989F}, {0x9865,0x98A2}, {0x9867,0x987E}, {0x986B,0x98A4}, {0x986C,0x98A5}, 
+  {0x986F,0x663E}, {0x9870,0x98A6}, {0x9871,0x9885}, {0x9873,0x989E}, {0x9874,0x98A7}, 
+  {0x98A8,0x98CE}, {0x98AE,0x98D1}, {0x98AF,0x98D2}, {0x98B6,0x98D3}, {0x98BC,0x98D5}, 
+  {0x98C4,0x98D8}, {0x98C6,0x98D9}, {0x98C8,0x98DA}, {0x98DB,0x98DE}, {0x98E0,0x9963}, 
+  {0x98E9,0x9968}, {0x98EA,0x996A}, {0x98EB,0x996B}, {0x98ED,0x996C}, {0x98EF,0x996D}, 
+  {0x98F2,0x996E}, {0x98F4,0x9974}, {0x98FC,0x9972}, {0x98FD,0x9971}, {0x98FE,0x9970}, 
+  {0x9903,0x997A}, {0x9905,0x997C}, {0x9909,0x9977}, {0x990A,0x517B}, {0x990C,0x9975}, 
+  {0x9911,0x997D}, {0x9912,0x9981}, {0x9913,0x997F}, {0x9918,0x4F59}, {0x991B,0x9984}, 
+  {0x991E,0x996F}, {0x9921,0x9985}, {0x9928,0x9986}, {0x9933,0x9967}, {0x9937,0x9987}, 
+  {0x993C,0x9969}, {0x993E,0x998F}, {0x993F,0x998A}, {0x9943,0x998D}, {0x9945,0x9992}, 
+  {0x9948,0x9990}, {0x9949,0x9991}, {0x994A,0x9993}, {0x994B,0x9988}, {0x994C,0x9994}, 
+  {0x9951,0x9965}, {0x9952,0x9976}, {0x9957,0x98E8}, {0x995C,0x990D}, {0x995E,0x998B}, 
+  {0x9962,0x9995}, {0x99AC,0x9A6C}, {0x99AD,0x9A6D}, {0x99AE,0x51AF}, {0x99B1,0x9A6E}, 
+  {0x99B3,0x9A70}, {0x99B4,0x9A6F}, {0x99C1,0x9A73}, {0x99D0,0x9A7B}, {0x99D1,0x9A7D}, 
+  {0x99D2,0x9A79}, {0x99D4,0x9A75}, {0x99D5,0x9A7E}, {0x99D8,0x9A80}, {0x99D9,0x9A78}, 
+  {0x99DB,0x9A76}, {0x99DD,0x9A7C}, {0x99DF,0x9A77}, {0x99E1,0x9A82}, {0x99E2,0x9A88}, 
+  {0x99ED,0x9A87}, {0x99F1,0x9A86}, {0x99FF,0x9A8F}, {0x9A01,0x9A8B}, {0x9A05,0x9A93}, 
+  {0x9A0D,0x9A92}, {0x9A0E,0x9A91}, {0x9A0F,0x9A90}, {0x9A16,0x9A9B}, {0x9A19,0x9A97}, 
+  {0x9A2B,0x9A9E}, {0x9A2D,0x9A98}, {0x9A2E,0x9A9D}, {0x9A30,0x817E}, {0x9A36,0x9A7A}, 
+  {0x9A37,0x9A9A}, {0x9A38,0x9A9F}, {0x9A3E,0x9AA1}, {0x9A40,0x84E6}, {0x9A41,0x9A9C}, 
+  {0x9A42,0x9A96}, {0x9A43,0x9AA0}, {0x9A44,0x9AA2}, {0x9A45,0x9A71}, {0x9A4A,0x9A85}, 
+  {0x9A4D,0x9A81}, {0x9A4F,0x9AA3}, {0x9A55,0x9A84}, {0x9A57,0x9A8C}, {0x9A5A,0x60CA}, 
+  {0x9A5B,0x9A7F}, {0x9A5F,0x9AA4}, {0x9A62,0x9A74}, {0x9A64,0x9AA7}, {0x9A65,0x9AA5}, 
+  {0x9A6A,0x9A8A}, {0x9AAF,0x80AE}, {0x9ACF,0x9AC5}, {0x9AD4,0x4F53}, {0x9AD5,0x9ACC}, 
+  {0x9AD6,0x9ACB}, {0x9B06,0x677E}, {0x9B22,0x9B13}, {0x9B25,0x6597}, {0x9B27,0x95F9}, 
+  {0x9B29,0x960B}, {0x9B2E,0x9604}, {0x9B31,0x90C1}, {0x9B4E,0x9B49}, {0x9B58,0x9B47}, 
+  {0x9B5A,0x9C7C}, {0x9B6F,0x9C81}, {0x9B74,0x9C82}, {0x9B77,0x9C7F}, {0x9B81,0x9C85}, 
+  {0x9B83,0x9C86}, {0x9B8E,0x9C87}, {0x9B90,0x9C90}, {0x9B91,0x9C8D}, {0x9B92,0x9C8B}, 
+  {0x9B9A,0x9C92}, {0x9B9D,0x9C9E}, {0x9B9E,0x9C95}, {0x9BAA,0x9C94}, {0x9BAB,0x9C9B}, 
+  {0x9BAD,0x9C91}, {0x9BAE,0x9C9C}, {0x9BC0,0x9CA7}, {0x9BC1,0x9CA0}, {0x9BC7,0x9CA9}, 
+  {0x9BC9,0x9CA4}, {0x9BCA,0x9CA8}, {0x9BD4,0x9CBB}, {0x9BD6,0x9CAD}, {0x9BDB,0x9CB7}, 
+  {0x9BDD,0x9CB4}, {0x9BE1,0x9CB1}, {0x9BE2,0x9CB5}, {0x9BE4,0x9CB2}, {0x9BE7,0x9CB3}, 
+  {0x9BE8,0x9CB8}, {0x9BEA,0x9CAE}, {0x9BEB,0x9CB0}, {0x9BF0,0x9CB6}, {0x9BF4,0x9CBA}, 
+  {0x9BFD,0x9CAB}, {0x9BFF,0x9CCA}, {0x9C08,0x9CBD}, {0x9C09,0x9CC7}, {0x9C0D,0x9CC5}, 
+  {0x9C10,0x9CC4}, {0x9C12,0x9CC6}, {0x9C13,0x9CC3}, {0x9C20,0x9CCB}, {0x9C23,0x9CA5}, 
+  {0x9C25,0x9CCF}, {0x9C28,0x9CCE}, {0x9C29,0x9CD0}, {0x9C2D,0x9CCD}, {0x9C31,0x9CA2}, 
+  {0x9C32,0x9CCC}, {0x9C33,0x9CD3}, {0x9C35,0x9CD8}, {0x9C37,0x9CA6}, {0x9C39,0x9CA3}, 
+  {0x9C3B,0x9CD7}, {0x9C3E,0x9CD4}, {0x9C45,0x9CD9}, {0x9C48,0x9CD5}, {0x9C49,0x9CD6}, 
+  {0x9C52,0x9CDF}, {0x9C54,0x9CDD}, {0x9C56,0x9CDC}, {0x9C57,0x9CDE}, {0x9C58,0x9C9F}, 
+  {0x9C5D,0x9CBC}, {0x9C5F,0x9C8E}, {0x9C67,0x9CE2}, {0x9C6D,0x9C9A}, {0x9C78,0x9C88}, 
+  {0x9C7A,0x9CA1}, {0x9CE5,0x9E1F}, {0x9CE9,0x9E20}, {0x9CEC,0x51EB}, {0x9CF3,0x51E4}, 
+  {0x9CF4,0x9E23}, {0x9CF6,0x9E22}, {0x9D06,0x9E29}, {0x9D07,0x9E28}, {0x9D09,0x9E26}, 
+  {0x9D15,0x9E35}, {0x9D1B,0x9E33}, {0x9D1D,0x9E32}, {0x9D1F,0x9E31}, {0x9D23,0x9E2A}, 
+  {0x9D26,0x9E2F}, {0x9D28,0x9E2D}, {0x9D2F,0x9E38}, {0x9D30,0x9E39}, {0x9D3B,0x9E3F}, 
+  {0x9D3F,0x9E3D}, {0x9D42,0x9E3A}, {0x9D51,0x9E43}, {0x9D52,0x9E46}, {0x9D53,0x9E41}, 
+  {0x9D5C,0x9E48}, {0x9D5D,0x9E45}, {0x9D60,0x9E44}, {0x9D61,0x9E49}, {0x9D6A,0x9E4C}, 
+  {0x9D6C,0x9E4F}, {0x9D6F,0x9E4E}, {0x9D72,0x9E4A}, {0x9D87,0x9E2B}, {0x9D89,0x9E51}, 
+  {0x9D93,0x9E4B}, {0x9D98,0x9E55}, {0x9D9A,0x9E57}, {0x9DA5,0x9E5B}, {0x9DA9,0x9E5C}, 
+  {0x9DAF,0x83BA}, {0x9DB4,0x9E64}, {0x9DBB,0x9E58}, {0x9DBC,0x9E63}, {0x9DC0,0x9E5A}, 
+  {0x9DC2,0x9E5E}, {0x9DC4,0x9E21}, {0x9DD3,0x9E67}, {0x9DD7,0x9E25}, {0x9DD9,0x9E37}, 
+  {0x9DDA,0x9E68}, {0x9DE5,0x9E36}, {0x9DE6,0x9E6A}, {0x9DEF,0x9E69}, {0x9DF2,0x9E6B}, 
+  {0x9DF3,0x9E47}, {0x9DF8,0x9E6C}, {0x9DF9,0x9E70}, {0x9DFA,0x9E6D}, {0x9E0C,0x9E71}, 
+  {0x9E15,0x9E2C}, {0x9E1A,0x9E66}, {0x9E1B,0x9E73}, {0x9E1D,0x9E42}, {0x9E1E,0x9E3E}, 
+  {0x9E7A,0x9E7E}, {0x9E7C,0x7877}, {0x9E7D,0x76D0}, {0x9E97,0x4E3D}, {0x9EA5,0x9EA6}, 
+  {0x9EA9,0x9EB8}, {0x9EBD,0x4E48}, {0x9ECC,0x9EC9}, {0x9EDE,0x70B9}, {0x9EE8,0x515A}, 
+  {0x9EF2,0x9EEA}, {0x9EF7,0x9EE9}, {0x9EFD,0x9EFE}, {0x9EFF,0x9F0B}, {0x9F09,0x9F0D}, 
+  {0x9F4A,0x9F50}, {0x9F4B,0x658B}, {0x9F4F,0x9F51}, {0x9F52,0x9F7F}, {0x9F54,0x9F80}, 
+  {0x9F59,0x9F85}, {0x9F5C,0x9F87}, {0x9F5F,0x9F83}, {0x9F60,0x9F86}, {0x9F61,0x9F84}, 
+  {0x9F66,0x9F88}, {0x9F6A,0x9F8A}, {0x9F6C,0x9F89}, {0x9F72,0x9F8B}, {0x9F77,0x9F8C}, 
+  {0x9F8D,0x9F99}, {0x9F90,0x5E9E}, {0x9F94,0x9F9A}, {0x9F95,0x9F9B}, {0x9F9C,0x9F9F}
+};
+
+
+
+/* the letters correspond to unicode characters which
+ * have Lu, Ll, Lt, Lm, Lo, Mn, Mc, or Me in their
+ * GENERAL CATEGORY */
+
+#define NUM_LETTER_INFO 274
+static const unirange letter_info[NUM_LETTER_INFO] = {
+  {0x41,0x5a}, {0x61,0x7a}, {0xaa,0xaa}, {0xb5,0xb5}, 
+  {0xba,0xba}, {0xc0,0xd6}, {0xd8,0xf6}, {0xf8,0x1f5}, 
+  {0x1fa,0x217}, {0x250,0x2a8}, {0x2b0,0x2b8}, {0x2bb,0x2c1}, 
+  {0x2d0,0x2d1}, {0x2e0,0x2e4}, {0x300,0x345}, {0x360,0x361}, 
+  {0x37a,0x37a}, {0x386,0x386}, {0x388,0x38a}, {0x38c,0x38c}, 
+  {0x38e,0x3a1}, {0x3a3,0x3ce}, {0x3d0,0x3d6}, {0x3da,0x3da}, 
+  {0x3dc,0x3dc}, {0x3de,0x3de}, {0x3e0,0x3e0}, {0x3e2,0x3f3}, 
+  {0x401,0x40c}, {0x40e,0x44f}, {0x451,0x45c}, {0x45e,0x481}, 
+  {0x483,0x486}, {0x490,0x4c4}, {0x4c7,0x4c8}, {0x4cb,0x4cc}, 
+  {0x4d0,0x4eb}, {0x4ee,0x4f5}, {0x4f8,0x4f9}, {0x531,0x556}, 
+  {0x559,0x559}, {0x561,0x587}, {0x591,0x5a1}, {0x5a3,0x5b9}, 
+  {0x5bb,0x5bd}, {0x5bf,0x5bf}, {0x5c1,0x5c2}, {0x5c4,0x5c4}, 
+  {0x5d0,0x5ea}, {0x5f0,0x5f2}, {0x621,0x63a}, {0x640,0x652}, 
+  {0x670,0x6b7}, {0x6ba,0x6be}, {0x6c0,0x6ce}, {0x6d0,0x6d3}, 
+  {0x6d5,0x6e8}, {0x6ea,0x6ed}, {0x901,0x903}, {0x905,0x939}, 
+  {0x93c,0x94d}, {0x951,0x954}, {0x958,0x963}, {0x981,0x983}, 
+  {0x985,0x98c}, {0x98f,0x990}, {0x993,0x9a8}, {0x9aa,0x9b0}, 
+  {0x9b2,0x9b2}, {0x9b6,0x9b9}, {0x9bc,0x9bc}, {0x9be,0x9c4}, 
+  {0x9c7,0x9c8}, {0x9cb,0x9cd}, {0x9d7,0x9d7}, {0x9dc,0x9dd}, 
+  {0x9df,0x9e3}, {0x9f0,0x9f1}, {0xa02,0xa02}, {0xa05,0xa0a}, 
+  {0xa0f,0xa10}, {0xa13,0xa28}, {0xa2a,0xa30}, {0xa32,0xa33}, 
+  {0xa35,0xa36}, {0xa38,0xa39}, {0xa3c,0xa3c}, {0xa3e,0xa42}, 
+  {0xa47,0xa48}, {0xa4b,0xa4d}, {0xa59,0xa5c}, {0xa5e,0xa5e}, 
+  {0xa70,0xa74}, {0xa81,0xa83}, {0xa85,0xa8b}, {0xa8d,0xa8d}, 
+  {0xa8f,0xa91}, {0xa93,0xaa8}, {0xaaa,0xab0}, {0xab2,0xab3}, 
+  {0xab5,0xab9}, {0xabc,0xac5}, {0xac7,0xac9}, {0xacb,0xacd}, 
+  {0xae0,0xae0}, {0xb01,0xb03}, {0xb05,0xb0c}, {0xb0f,0xb10}, 
+  {0xb13,0xb28}, {0xb2a,0xb30}, {0xb32,0xb33}, {0xb36,0xb39}, 
+  {0xb3c,0xb43}, {0xb47,0xb48}, {0xb4b,0xb4d}, {0xb56,0xb57}, 
+  {0xb5c,0xb5d}, {0xb5f,0xb61}, {0xb82,0xb83}, {0xb85,0xb8a}, 
+  {0xb8e,0xb90}, {0xb92,0xb95}, {0xb99,0xb9a}, {0xb9c,0xb9c}, 
+  {0xb9e,0xb9f}, {0xba3,0xba4}, {0xba8,0xbaa}, {0xbae,0xbb5}, 
+  {0xbb7,0xbb9}, {0xbbe,0xbc2}, {0xbc6,0xbc8}, {0xbca,0xbcd}, 
+  {0xbd7,0xbd7}, {0xc01,0xc03}, {0xc05,0xc0c}, {0xc0e,0xc10}, 
+  {0xc12,0xc28}, {0xc2a,0xc33}, {0xc35,0xc39}, {0xc3e,0xc44}, 
+  {0xc46,0xc48}, {0xc4a,0xc4d}, {0xc55,0xc56}, {0xc60,0xc61}, 
+  {0xc82,0xc83}, {0xc85,0xc8c}, {0xc8e,0xc90}, {0xc92,0xca8}, 
+  {0xcaa,0xcb3}, {0xcb5,0xcb9}, {0xcbe,0xcc4}, {0xcc6,0xcc8}, 
+  {0xcca,0xccd}, {0xcd5,0xcd6}, {0xcde,0xcde}, {0xce0,0xce1}, 
+  {0xd02,0xd03}, {0xd05,0xd0c}, {0xd0e,0xd10}, {0xd12,0xd28}, 
+  {0xd2a,0xd39}, {0xd3e,0xd43}, {0xd46,0xd48}, {0xd4a,0xd4d}, 
+  {0xd57,0xd57}, {0xd60,0xd61}, {0xe01,0xe2e}, {0xe30,0xe3a}, 
+  {0xe40,0xe4e}, {0xe81,0xe82}, {0xe84,0xe84}, {0xe87,0xe88}, 
+  {0xe8a,0xe8a}, {0xe8d,0xe8d}, {0xe94,0xe97}, {0xe99,0xe9f}, 
+  {0xea1,0xea3}, {0xea5,0xea5}, {0xea7,0xea7}, {0xeaa,0xeab}, 
+  {0xead,0xeae}, {0xeb0,0xeb9}, {0xebb,0xebd}, {0xec0,0xec4}, 
+  {0xec6,0xec6}, {0xec8,0xecd}, {0xedc,0xedd}, {0xf18,0xf19}, 
+  {0xf35,0xf35}, {0xf37,0xf37}, {0xf39,0xf39}, {0xf40,0xf47}, 
+  {0xf49,0xf69}, {0xf71,0xf84}, {0xf86,0xf87}, {0xf90,0xf95}, 
+  {0xf97,0xf97}, {0xf99,0xfad}, {0xfb1,0xfb7}, {0xfb9,0xfb9}, 
+  {0x10a0,0x10c5}, {0x10d0,0x10f6}, {0x1100,0x1159}, {0x115f,0x11a2}, 
+  {0x11a8,0x11f9}, {0x1e00,0x1e9b}, {0x1ea0,0x1ef9}, {0x1f00,0x1f15}, 
+  {0x1f18,0x1f1d}, {0x1f20,0x1f45}, {0x1f48,0x1f4d}, {0x1f50,0x1f57}, 
+  {0x1f59,0x1f59}, {0x1f5b,0x1f5b}, {0x1f5d,0x1f5d}, {0x1f5f,0x1f7d}, 
+  {0x1f80,0x1fb4}, {0x1fb6,0x1fbc}, {0x1fbe,0x1fbe}, {0x1fc2,0x1fc4}, 
+  {0x1fc6,0x1fcc}, {0x1fd0,0x1fd3}, {0x1fd6,0x1fdb}, {0x1fe0,0x1fec}, 
+  {0x1ff2,0x1ff4}, {0x1ff6,0x1ffc}, {0x207f,0x207f}, {0x20d0,0x20e1}, 
+  {0x2102,0x2102}, {0x2107,0x2107}, {0x210a,0x2113}, {0x2115,0x2115}, 
+  {0x2118,0x211d}, {0x2124,0x2124}, {0x2126,0x2126}, {0x2128,0x2128}, 
+  {0x212a,0x2131}, {0x2133,0x2138}, {0x3005,0x3005}, {0x302a,0x302f}, 
+  {0x3031,0x3035}, {0x3041,0x3094}, {0x3099,0x309e}, {0x30a1,0x30fa}, 
+  {0x30fc,0x30fe}, {0x3105,0x312c}, {0x3131,0x318e}, {0x4e00,0x9fa5}, 
+  {0xac00,0xd7a3}, {0xf900,0xfa2d}, {0xfb00,0xfb06}, {0xfb13,0xfb17}, 
+  {0xfb1e,0xfb28}, {0xfb2a,0xfb36}, {0xfb38,0xfb3c}, {0xfb3e,0xfb3e}, 
+  {0xfb40,0xfb41}, {0xfb43,0xfb44}, {0xfb46,0xfbb1}, {0xfbd3,0xfd3d}, 
+  {0xfd50,0xfd8f}, {0xfd92,0xfdc7}, {0xfdf0,0xfdfb}, {0xfe20,0xfe23}, 
+  {0xfe70,0xfe72}, {0xfe74,0xfe74}, {0xfe76,0xfefc}, {0xff21,0xff3a}, 
+  {0xff41,0xff5a}, {0xff66,0xffbe}, {0xffc2,0xffc7}, {0xffca,0xffcf}, 
+  {0xffd2,0xffd7}, {0xffda,0xffdc}
+};
+
+
+/* the digits correspond to unicode characters which
+ * have Nd in their GENERAL CATEGORY */
+
+#define NUM_DIGIT_INFO 16
+static const unirange digit_info[NUM_DIGIT_INFO] = {
+  {0x30,0x39}, {0x660,0x669}, {0x6f0,0x6f9}, {0x966,0x96f}, 
+  {0x9e6,0x9ef}, {0xa66,0xa6f}, {0xae6,0xaef}, {0xb66,0xb6f}, 
+  {0xbe7,0xbef}, {0xc66,0xc6f}, {0xce6,0xcef}, {0xd66,0xd6f}, 
+  {0xe50,0xe59}, {0xed0,0xed9}, {0xf20,0xf29}, {0xff10,0xff19}
+};
+
+
+/* the spaces correspond to unicode characters which
+ * have Zs, Zl, or Zp in their GENERAL CATEGORY */
+
+#define NUM_SPACE_INFO 5
+static const unirange space_info[NUM_SPACE_INFO] = {
+  {0x20,0x20}, {0xa0,0xa0}, {0x2000,0x200b}, {0x2028,0x2029}, 
+  {0x3000,0x3000}
+};
+
+
+
+/* parse_utf8_char parses the next utf-8 character, placing its
+ * unicode equivalent in *value. The length of the utf-8 character
+ * is returned */
+int parse_utf8_char(const unsigned char *here,
+		    const unsigned char *end,
+		    unsigned short *value) {
+  unsigned short c1;
+  unsigned short c2;
+  unsigned short c3;
+  
+  *value = 0; /* will be the value for illegal characters */
+  
+  if (here <= end) {
+    c1 = (unsigned short)(*here);
+    
+    if (c1 <= 0x7f) {
+      /* found a one-byte character */
+      *value = c1;
+      return 1;
+      
+    } else if (c1 >= 0xc0 && c1 <= 0xdf && here+1 <= end &&
+	       (c2=(unsigned short)(*(here+1))) >= 0x80) {
+      /* found a two-byte character */
+      *value = ((c1 & 0x1f) << 6) + (c2 & 0x3f);
+      return 2;
+      
+    } else if (c1 >= 0xe0 && c1 <= 0xef && here+2 <= end &&
+	       (c2=(unsigned int)(*(here+1))) >= 0x80 &&
+	       (c3=(unsigned int)(*(here+2))) >= 0x80) {
+      /* found a three-byte character */
+      *value = ((c1 & 0xf) << 12) + ((c2 & 0x3f) << 6) +
+	(c3 & 0x3f);
+      return 3;
+      
+    } else {
+      /* error in encoding, return the length as 1 */
+      return 1;
+    }
+  }
+
+  /* end of string */
+  return 0;
+}
+
+
+/* output_utf8_char encodes a unicode character as a UTF-8 character.
+ * The length of the encoding is returned. If the string was not
+ * long enough to encode the character 0 is returned. */
+int output_utf8_char (unsigned short value,
+		      unsigned char *here,
+		      unsigned char *end) {
+  int len = (int)(end - here + 1); /* end is the last character */
+  if (value < 0x80) {
+    /* will be encoded in one byte */
+    if (len >= 1) {
+      *here = (unsigned char)value;
+      return 1;
+    }
+
+  } else if (value < 0x800) {
+    /* will be encoded in two bytes */
+    if (len >= 2) {
+      *here++ = 0xc0 | (unsigned char)((value >> 6) & 0x1f);
+      *here = 0x80 | (unsigned char)(value & 0x3f);
+
+      return 2;
+    }
+
+  } else {
+    /* will be encoded in three bytes */
+    if (len >= 3) {
+      *here++ = 0xe0 | (unsigned char)((value >> 12) & 0xf);
+      *here++ = 0x80 | (unsigned char)((value >> 6) & 0x3f);
+      *here = 0x80 | (unsigned char)(value & 0x3f);
+      return 3;
+    }
+  }
+
+  return 0;
+}
+
+
+static int lookup_decompose_table (unsigned short value) {
+  int first = 0;
+  int last = NUM_DECOMPOSE_INFO;
+  int here;
+
+  while (first < last) {
+    here = (last-first)/2 + first;
+    if (value < decompose_info[here].ch) {
+      last = here;
+    } else if (value > decompose_info[here].ch) {
+      first = here+1;
+    } else {
+      return here;
+    }
+  }
+
+  return -1;
+}
+
+/* decompose_str will decompose a unicode string into its canonical
+ * equivalents. NULL is returned if the input array was not 
+ * large enough to contain the fully decomposed string (the array
+ * will be in a correct, but partially decomposed state). The input 
+ * must be null-terminated. */
+unsigned short *decompose_str (unsigned short *input,
+			       int max_output_len) {
+  int curpos = 0;
+  int curlen = unicode_strlen (input);
+  int tablepos = 0;
+  int leninc;
+  int i;
+
+  while (input[curpos] != 0) {
+    tablepos = lookup_decompose_table (input[curpos]);
+    if (tablepos >= 0) {
+      /* found a decomposition */
+      leninc = unicode_strnlen (decompose_info[tablepos].str, 3)-1;
+      if (curlen+leninc < max_output_len) {
+	/* enough space for this decomposition */
+
+	if (leninc > 1) {
+	  /* we have to shift the array along */
+	  for (i=curlen-1; i>=curpos; i--) {
+	    input[i+leninc] = input[i];
+	  }
+	  input[curlen+leninc] = 0;
+	  curlen += leninc;
+	}
+
+	/* decompose the character */
+	for (i=0; i<=leninc; i++) {
+	  input[curpos+i] = decompose_info[tablepos].str[i];
+	}
+
+	/* note: curpos is not incremented because the decomposed */
+	/*       characters might need decomposing */
+
+      } else {
+	/* no space left in string */
+	return NULL;
+      }
+
+    } else {
+      /* the character cannot be decomposed */
+      curpos++;
+    }
+  }
+
+  return input;
+}
+
+
+
+/* unicode_cat_data is a combination of the letter, digit, and
+ * space information in a form that is quick to look up.
+ * The information about each byte is stored in two bits
+ *
+ * 00 - neither letter, digit or space
+ * 01 - letter
+ * 10 - space
+ * 11 - digit
+ *
+ * You can test for letter | space by looking at the right
+ * column
+ */
+/*unsigned long *unicode_cat_data;*/
+
+void create_unicat_data () {
+  int i=0;
+  int min_start=0;
+  int min_end=0;
+  register unsigned long min_value=0; /* filling the whole byte */
+  
+  const unirange *letter_here = letter_info;
+  const unirange *letter_end = letter_info+NUM_LETTER_INFO;
+  const unirange *digit_here = digit_info;
+  const unirange *digit_end = digit_info+NUM_DIGIT_INFO;
+  const unirange *space_here = space_info;
+  const unirange *space_end = space_info+NUM_SPACE_INFO;
+
+  if (unicode_cat_data != NULL) return;
+  unicode_cat_data = (unsigned long *) malloc (0x10000/4);
+  if (unicode_cat_data == NULL) return;
+  
+  while (i <= 0xffff) {
+    /* decide what to fill with next */
+    min_start = i;
+    min_end = 0xffff;
+    min_value = 0; /* 00 */
+    
+    while (letter_here < letter_end && letter_here->last < i) letter_here++;
+    if (letter_here < letter_end && letter_here->last <= min_end) {
+      min_start = letter_here->first;
+      min_end = letter_here->last;
+      min_value = 0x55555555; /* 01 */
+    }
+    
+    while (digit_here < digit_end && digit_here->last < i) digit_here++;
+    if (digit_here < digit_end && digit_here->last <= min_end) {
+      min_start = digit_here->first;
+      min_end = digit_here->last;
+      min_value = 0xffffffff; /* 11 */
+    }
+    
+    while (space_here < space_end && space_here->last < i) space_here++;
+    if (space_here < space_end && space_here->last <= min_end) {
+      min_start = space_here->first;
+      min_end = space_here->last;
+      min_value = 0xaaaaaaaa; /* 10 */
+    }
+
+    if (min_start < i) min_start = i;
+    else if (min_start > i) {
+      min_end = min_start-1;
+      min_start = i;
+      min_value = 0; /* 00 */
+    }
+
+    /* fill the entries from min_start to min_end with min_value */
+    while (i <= min_end) {
+      /* special version for common case */
+      if (i % 16 == 0) {
+	unicode_cat_data[i/16] = min_value; /* doesn't matter if too long */
+	i += 16;
+	if (i > min_end) {
+	  i = min_end+1;
+	  break;
+	}
+	
+      } else {
+	register unsigned long mask = 3 << ((i % 16) * 2);
+	unicode_cat_data[i/16] = (unicode_cat_data[i/16] & (~mask)) |
+				  (min_value & mask);
+	i++;
+      }
+    }
+  }
+}
+
+/* does a binary search on a range table */
+static int in_unirange (const unirange *info, int num_info,
+			unsigned short value) {
+  int first = 0;
+  int last = num_info;
+  int here;
+
+  while (first < last) {
+    here = (last-first)/2 + first;
+    if (value < info[here].first) {
+      last = here;
+    } else if (value > info[here].last) {
+      first = here+1;
+    } else {
+      return 1;
+    }
+  }
+
+  return 0;
+}
+
+
+
+/* tests to see whether 'value' is a valid Unicode letter */
+int is_unicode_letter (unsigned short value) {
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return in_unirange(letter_info, NUM_LETTER_INFO, value);
+  }
+
+  return (((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x3) == 0x1);
+}
+
+/* tests to see whether 'value' is a valid Unicode digit */
+int is_unicode_digit (unsigned short value) {
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return in_unirange(digit_info, NUM_DIGIT_INFO, value);
+  }
+
+  return (((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x3) == 0x3);
+}
+
+/* tests to see whether 'value' is a valid Unicode letter or 
+ * digit */
+int is_unicode_letdig (unsigned short value) {
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return (in_unirange(letter_info, NUM_LETTER_INFO, value) ||
+	    in_unirange(digit_info, NUM_DIGIT_INFO, value));
+    
+  }
+
+  return ((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x1);
+}
+
+/* tests to see whether 'value' is a valid space
+ * The test includes both "C" spaces and "Unicode" spaces, i.e. 
+ * form-feed, newline, carriage return, horizontal tab,
+ * vertical tab, and the Zs, Zl, and Zp Unicode categorizations */
+int is_unicode_space (unsigned short value) {
+  if (value < 0x80) return isspace (value);
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return in_unirange(space_info, NUM_SPACE_INFO, value);
+  }
+
+  return (((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x3) == 0x2);
+}
+
+/* returns the length of the unicode string */
+int unicode_strlen (const unsigned short *str) {
+  int len = 0;
+
+  /* keep going till you find the null-terminator */
+  while (*str != 0) {
+    len++;
+    str++;
+  }
+
+  return len;
+}
+
+/* returns the length of the unicode string, up to a maximum */
+int unicode_strnlen (const unsigned short *str, int max_length) {
+  int len = 0;
+
+  /* keep going till you find the null-terminator */
+  while (*str != 0 && len < max_length ) {
+    len++;
+    str++;
+  }
+
+  return len;
+}
+
+
+/* Does a binary search on a relate table, returns -1 if it is not
+ * found. */
+static int search_unirelate (const unirelate *info, int num_info,
+			     unsigned short value) {
+  int first = 0;
+  int last = num_info;
+  int here;
+
+  while (first < last) {
+    here = (last-first)/2 + first;
+    if (value < info[here].ch1) {
+      last = here;
+    } else if (value > info[here].ch1) {
+      first = here+1;
+    } else {
+      return here;
+    }
+  }
+
+  return -1;
+}
+
+
+/* returns the upper-case equivalent of value */
+unsigned short unicode_toupper (unsigned short value) {
+  int related = search_unirelate (toupper_info, NUM_TOUPPER_INFO, value);
+  if (related >= 0) {
+    value = toupper_info[related].ch2;
+  }
+  return value;
+}
+
+/* returns the lower-case equivalent of value */
+unsigned short unicode_tolower (unsigned short value) {
+  int related = search_unirelate (tolower_info, NUM_TOLOWER_INFO, value);
+  if (related >= 0) {
+    value = tolower_info[related].ch2;
+  }
+  return value;
+}
+
+/* returns the simplified Chinese character equivalent of 
+ * another Chinese character */
+unsigned short unicode_tosimplified (unsigned short value) {
+  int related = search_unirelate (tosimplified_info, NUM_TOSIMPLIFIED_INFO, value);
+  if (related >= 0) {
+    value = tosimplified_info[related].ch2;
+  }
+  return value;
+}
+
+
+/* converts a utf-8 word (string with length stored in the first byte
+ * to a Unicode array. To handle all situations the output buffer should
+ * be 256 unsigned shorts long. The output will also have the length as
+ * the first entry. */
+unsigned short *utf8_word_to_unicode (const unsigned char *input,
+				      unsigned short *output,
+				      int max_output_length) {
+  int inlen = *input;
+  const unsigned char *inhere = input+1;
+  const unsigned char *inend = &inhere[inlen-1];
+  unsigned short c;
+  int clen;
+  
+  int outlen = 0;
+  unsigned short *outhere = output+1;
+  
+  while (inhere <= inend && inlen < max_output_length-1) {
+    /* decode the character */
+    clen = parse_utf8_char (inhere, inend, &c);
+    inhere += clen;
+
+    if (clen > 0) {
+      /* store the character */
+      *outhere = c;
+      outhere++;
+      outlen++;
+    }
+  }
+
+  output[0] = outlen;
+  return output;
+}
+
+
+/* converts a unicode word buffer (with the length stored in the
+ * entry) to a utf8 encoded word output (with the length stored in
+ * the first byte. Only 255 bytes (not characters) can be stored 
+ * in the output. */
+unsigned char *unicode_to_utf8_word (const unsigned short *input,
+				     unsigned char *output,
+				     int max_output_length) {
+  int inlen = *input;
+  const unsigned short *word = input + 1;
+  int in_i = 0;
+  int clen;
+
+  int outlen = 0;
+  unsigned char *outhere = output+1;
+  unsigned char *outend = output+max_output_length;
+
+  while (in_i < inlen) {
+    if ((clen=output_utf8_char(word[in_i], outhere, outend)) == 0) {
+      /* ran out of space to encode the character */
+      break;
+    }
+
+    outhere += clen;
+    outlen += clen;
+    in_i++;
+  }
+
+  output[0] = outlen;
+
+  return output;
+}
+
Index: /trunk/indexers/mgpp/lib/unitool.h
===================================================================
--- /trunk/indexers/mgpp/lib/unitool.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/unitool.h	(revision 3365)
@@ -0,0 +1,88 @@
+#ifndef UNITOOL_H
+#define UNITOOL_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+/* This module is based on Unicode 2.1 */
+
+
+/* parse_utf8_char parses the next utf-8 character, placing its
+ * unicode equivalent in *value. The length of the utf-8 character
+ * is returned. end is the address of the last character. */
+int parse_utf8_char (const unsigned char *here,
+		     const unsigned char *end,
+		     unsigned short *value);
+
+/* output_utf8_char encodes a unicode character as a UTF-8 character.
+ * The length of the encoding is returned. If the string was not
+ * long enough to encode the character 0 is returned. end is the
+ * address of the last character */
+int output_utf8_char (unsigned short value,
+		      unsigned char *here,
+		      unsigned char *end);
+
+/* decompose_str will decompose a unicode string into its canonical
+ * equivalents. NULL is returned if the input array was not 
+ * large enough to contain the fully decomposed string (the array
+ * will be in a correct, but partially decomposed state). The input 
+ * must be null-terminated. */
+unsigned short *decompose_str (unsigned short *input,
+			       int max_output_len);
+
+/* tests to see whether 'value' is a valid Unicode letter */
+int is_unicode_letter (unsigned short value);
+
+/* tests to see whether 'value' is a valid Unicode digit */
+int is_unicode_digit (unsigned short value);
+
+/* tests to see whether 'value' is a valid Unicode letter or 
+ * digit */
+int is_unicode_letdig (unsigned short value);
+
+/* tests to see whether 'value' is a valid space
+ * The test includes both "C" spaces and "Unicode" spaces, i.e. 
+ * form-feed, newline, carriage return, horizontal tab,
+ * vertical tab, and the Zs, Zl, and Zp Unicode categorizations */
+int is_unicode_space (unsigned short value);
+
+/* returns the length of the unicode string */
+int unicode_strlen (const unsigned short *str);
+
+/* returns the length of the unicode string, up to a maximum */
+int unicode_strnlen (const unsigned short *str, int max_length);
+
+/* returns the upper-case equivalent of value */
+unsigned short unicode_toupper (unsigned short value);
+
+/* returns the lower-case equivalent of value */
+unsigned short unicode_tolower (unsigned short value);
+
+/* returns the simplified Chinese character equivalent of 
+ * another Chinese character */
+unsigned short unicode_tosimplified (unsigned short value);
+
+
+/* converts a utf-8 word (string with length stored in the first byte
+ * to a Unicode array. To handle all situations the output buffer should
+ * be 256 unsigned shorts long. The output will also have the length as
+ * the first entry. */
+unsigned short *utf8_word_to_unicode (const unsigned char *input,
+				      unsigned short *output,
+				      int max_output_length);
+
+/* converts a unicode word buffer (with the length stored in the
+ * entry) to a utf8 encoded word output (with the length stored in
+ * the first byte. Only 255 bytes (not characters) can be stored 
+ * in the output. */
+unsigned char *unicode_to_utf8_word (const unsigned short *input,
+				     unsigned char *output,
+				     int max_output_length);
+
+#ifdef __cplusplus
+	   }
+#endif
+
+#endif
Index: /trunk/indexers/mgpp/lib/win32in.c
===================================================================
--- /trunk/indexers/mgpp/lib/win32in.c	(revision 3365)
+++ /trunk/indexers/mgpp/lib/win32in.c	(revision 3365)
@@ -0,0 +1,85 @@
+/* This module provides an equivalent of <netinet/in.h> on  */
+/* unix systems. */
+#include "win32in.h"
+
+unsigned long htonl(unsigned long x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 24) & 0x000000FF) | ((x >> 8) & 0x0000FF00) |
+		((x << 8) & 0x00FF0000) | ((x << 24) & 0xFF000000);
+#endif
+
+	return x;
+}
+
+mg_ullong ntohll(mg_ullong x)
+{
+#if defined (LITTLE_ENDIAN) || defined(_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = 
+	  ((x >> 56) & 0x00000000000000FF) | ((x >> 40) & 0x000000000000FF00) |
+	  ((x >> 24) & 0x0000000000FF0000) | ((x >> 8)  & 0x00000000FF000000) |
+	  ((x << 8)  & 0x000000FF00000000) | ((x << 24) & 0x0000FF0000000000) |
+		((x << 40) & 0x00FF000000000000) | ((x << 56) & 0xFF00000000000000);
+#endif
+  return x;
+}
+
+unsigned long ntohl(unsigned long x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 24) & 0x000000FF) | ((x >> 8) & 0x0000FF00) |
+		((x << 8) & 0x00FF0000) | ((x << 24) & 0xFF000000);
+#endif
+	return x;
+}
+
+unsigned short htons(unsigned short x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 8) & 0x00FF) | ((x << 8) & 0xFF00);
+#endif
+	return x;
+}
+
+unsigned short ntohs(unsigned short x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 8) & 0x00FF) | ((x << 8) & 0xFF00);
+#endif
+	return x;
+}
+
+/* just to test stuff
+#include <stdio.h>
+
+main () {
+	long x;
+	short sx;
+
+	x = 0x000000FF;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n",(int)x);
+
+	x = 0x0000FF00;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n",(int)x);
+
+	x = 0x00FF0000;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n",(int)x);
+
+	x = 0xFF000000;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n\n",(int)x);
+
+	sx = 0x00FF;
+	printf ("before %i, ",(int)sx);
+	sx = htons (sx);
+	printf ("after %i\n",(int)sx);
+
+	sx = 0xFF00;
+	printf ("before %i, ",(int)sx);
+	sx = htons (sx);
+	printf ("after %i\n",(int)sx);
+}
+*/
Index: /trunk/indexers/mgpp/lib/win32in.h
===================================================================
--- /trunk/indexers/mgpp/lib/win32in.h	(revision 3365)
+++ /trunk/indexers/mgpp/lib/win32in.h	(revision 3365)
@@ -0,0 +1,20 @@
+/* This module provides an equivalent of <netinet/in.h> on  */
+/* unix systems. */
+#ifdef __WIN32__
+#define _LITTLE_ENDIAN
+#endif
+
+#include "longlong.h"
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+unsigned long htonl(unsigned long x);
+unsigned long ntohl(unsigned long x);
+mg_ullong ntohll(mg_ullong x);
+unsigned short htons(unsigned short x);
+unsigned short ntohs(unsigned short x);
+
+#ifdef __cplusplus
+}
+#endif
Index: /trunk/indexers/mgpp/lib/xmalloc.c
===================================================================
--- /trunk/indexers/mgpp/lib/xmalloc.c	(revision 3365)
+++ /trunk/indexers/mgpp/lib/xmalloc.c	(revision 3365)
@@ -0,0 +1,111 @@
+/* xmalloc.c -- malloc with out of memory checking
+   Copyright (C) 1990, 91, 92, 93, 94 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__  /* [RPAP - Feb 97: WIN32 Port] */
+#  include <win32cfg.h>
+# else
+#  include <sysfuncs.h>
+# endif
+#endif
+
+#if __STDC__
+#define VOID void
+#else
+#define VOID char
+#endif
+
+#include <sys/types.h>
+
+#if STDC_HEADERS
+#include <stdlib.h>
+#else
+VOID *malloc ();
+VOID *realloc ();
+void free ();
+#endif
+
+/* This is for other GNU distributions with internationalized messages.
+   The GNU C Library itself does not yet support such messages.  */
+#if HAVE_LIBINTL_H
+# include <libintl.h>
+#else
+# define gettext(msgid) (msgid)
+#endif
+
+#ifndef EXIT_FAILURE
+#define EXIT_FAILURE 1
+#endif
+
+/* Exit value when the requested amount of memory is not available.
+   The caller may set it to some other value.  */
+int xmalloc_exit_failure = EXIT_FAILURE;
+
+#if defined(__WIN32__)
+void error (int, int, const char *, ...);
+#else
+#if __STDC__ && (HAVE_VPRINTF || HAVE_DOPRNT)
+void error (int, int, const char *, ...);
+#else
+void error (int, int, char *, char *, char *, char *, char *, char *, char *, char *, char *);
+#endif
+#endif
+
+static VOID *
+fixup_null_alloc (n)
+     size_t n;
+{
+  VOID *p;
+
+  p = 0;
+  if (n == 0)
+    p = malloc ((size_t) 1);
+  if (p == 0)
+    error (xmalloc_exit_failure, 0, gettext ("Memory exhausted"));
+  return p;
+}
+
+/* Allocate N bytes of memory dynamically, with error checking.  */
+
+VOID *
+xmalloc (n)
+     size_t n;
+{
+  VOID *p;
+
+  p = malloc (n);
+  if (p == 0)
+    p = fixup_null_alloc (n);
+  return p;
+}
+
+/* Change the size of an allocated block of memory P to N bytes,
+   with error checking.
+   If P is NULL, run xmalloc.  */
+
+VOID *
+xrealloc (p, n)
+     VOID *p;
+     size_t n;
+{
+  if (p == 0)
+    return xmalloc (n);
+  p = realloc (p, n);
+  if (p == 0)
+    p = fixup_null_alloc (n);
+  return p;
+}
Index: /trunk/indexers/mgpp/mkinstalldirs
===================================================================
--- /trunk/indexers/mgpp/mkinstalldirs	(revision 3365)
+++ /trunk/indexers/mgpp/mkinstalldirs	(revision 3365)
@@ -0,0 +1,32 @@
+#! /bin/sh
+# mkinstalldirs --- make directory hierarchy
+# Author: Noah Friedman <friedman@prep.ai.mit.edu>
+# Created: 1993-05-16
+# Last modified: 1994-03-25
+# Public domain
+
+errstatus=0
+
+for file in ${1+"$@"} ; do 
+   set fnord `echo ":$file" | sed -ne 's/^:\//#/;s/^://;s/\// /g;s/^#/\//;p'`
+   shift
+
+   pathcomp=
+   for d in ${1+"$@"} ; do
+     pathcomp="$pathcomp$d"
+     case "$pathcomp" in
+       -* ) pathcomp=./$pathcomp ;;
+     esac
+
+     if test ! -d "$pathcomp"; then
+        echo "mkdir $pathcomp" 1>&2
+        mkdir "$pathcomp" || errstatus=$?
+     fi
+
+     pathcomp="$pathcomp/"
+   done
+done
+
+exit $errstatus
+
+# mkinstalldirs ends here
Index: /trunk/indexers/mgpp/stamp-h.in
===================================================================
--- /trunk/indexers/mgpp/stamp-h.in	(revision 3365)
+++ /trunk/indexers/mgpp/stamp-h.in	(revision 3365)
@@ -0,0 +1,1 @@
+timestamp
Index: /trunk/indexers/mgpp/text/FIvfLevelInfo.cpp
===================================================================
--- /trunk/indexers/mgpp/text/FIvfLevelInfo.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/FIvfLevelInfo.cpp	(revision 3365)
@@ -0,0 +1,129 @@
+/**************************************************************************
+ *
+ * FIvfLevelInfo.cpp -- File functions for document levels
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FIvfLevelInfo.h"
+
+
+IvfLevelInfo::IvfLevelInfo () {
+  Clear();
+}
+
+void IvfLevelInfo::Clear () {
+  levelTag.erase (levelTag.begin(), levelTag.end());
+  numEntries = 0;
+  approxWeightsDiskPtr = 0;
+  exactWeightsDiskPtr = 0;
+  workInLevel = false;
+}
+
+bool IvfLevelInfo::Read (FILE *f) {
+  workInLevel = false;
+  return (ReadUCArray (f, levelTag) &&
+	  ReadUL (f, numEntries) &&
+	  ReadUL (f, approxWeightsDiskPtr) &&
+	  ReadUL (f, exactWeightsDiskPtr));
+}
+
+bool IvfLevelInfo::Write (FILE *f) const {
+  return (WriteUCArray (f, levelTag) &&
+	  WriteUL (f, numEntries) &&
+	  WriteUL (f, approxWeightsDiskPtr) &&
+	  WriteUL (f, exactWeightsDiskPtr));
+}
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const IvfLevelInfo &l) {
+  s << "  Tag: \"" << l.levelTag << "\"\n"
+    << "  numEntries: " << l.numEntries << "\n"
+    << "  approxWeightsDiskPtr: " << l.approxWeightsDiskPtr << "\n"
+    << "  exactWeightsDiskPtr: " << l.exactWeightsDiskPtr << "\n"
+    << "  workInLevel: " << ((l.workInLevel) ? "true" : "false") << "\n\n";
+  
+  return s;
+}
+
+
+
+FIvfLevel::FIvfLevel () {
+  Clear ();
+}
+
+void FIvfLevel::Clear () {
+  docTag.erase (docTag.begin(), docTag.end());
+  indexLevel.erase (indexLevel.begin(), indexLevel.end());
+  levelInfo.erase (levelInfo.begin(), levelInfo.end());
+}
+
+bool FIvfLevel::Read (FILE *f) {
+  if (!ReadUCArray (f, docTag) ||
+      !ReadUCArray (f, indexLevel))
+    return false;
+  
+  // read in the array size
+  unsigned long arrSize = 0;
+  if (!ReadVarLenUL (f, arrSize)) return false;
+  
+  IvfLevelInfo thisLevel;
+  while (arrSize > 0) {
+    // read and insert the next level information
+    if (!thisLevel.Read (f)) return false;
+    levelInfo[thisLevel.levelTag] = thisLevel;
+    
+    arrSize--;
+  }
+
+  return true;
+}
+
+bool FIvfLevel::Write (FILE *f) const {
+  if (!WriteUCArray (f, docTag) ||
+      !WriteUCArray (f, indexLevel))
+    return false;
+  
+  // write out the array size
+  if (!WriteVarLenUL (f, levelInfo.size())) return false;
+
+  // write out each level info
+  IvfLevelInfoMap::const_iterator here = levelInfo.begin();
+  IvfLevelInfoMap::const_iterator end = levelInfo.end();
+  while (here != end) {
+    if (!(*here).second.Write (f)) return false;
+    
+    here++;
+  }
+
+  return true;
+}
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const FIvfLevel &l) {
+  s << "docTag: \"" << l.docTag << "\"\n"
+    << "indexLevel: \"" << l.indexLevel << "\"\n";
+  
+  IvfLevelInfoMap::const_iterator here = l.levelInfo.begin ();
+  IvfLevelInfoMap::const_iterator end = l.levelInfo.end ();
+  while (here != end) {
+    s << (*here).second;
+    here++;
+  }
+  
+  return s;
+}
Index: /trunk/indexers/mgpp/text/FIvfLevelInfo.h
===================================================================
--- /trunk/indexers/mgpp/text/FIvfLevelInfo.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/FIvfLevelInfo.h	(revision 3365)
@@ -0,0 +1,77 @@
+/**************************************************************************
+ *
+ * FIvfLevelInfo.h -- File functions for document levels
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FIVFLEVELINFO_H
+#define FIVFLEVELINFO_H
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+#include <stdio.h>
+#include "UCArray.h"
+
+
+class IvfLevelInfo {
+public:
+  UCArray levelTag;
+  unsigned long numEntries; // N for level
+  unsigned long approxWeightsDiskPtr;
+  unsigned long exactWeightsDiskPtr;
+  bool workInLevel;  // this is just a working variable,
+                     // it is not saved or read from the file
+  
+  IvfLevelInfo ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const IvfLevelInfo &l);
+
+
+typedef map<UCArray, IvfLevelInfo, LTUCArray> IvfLevelInfoMap;
+
+
+class FIvfLevel {
+public:
+  UCArray docTag;
+  UCArray indexLevel;    // empty for word level index
+  IvfLevelInfoMap levelInfo;
+  
+  FIvfLevel ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const FIvfLevel &l);
+
+
+#endif
Index: /trunk/indexers/mgpp/text/FText.cpp
===================================================================
--- /trunk/indexers/mgpp/text/FText.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/FText.cpp	(revision 3365)
@@ -0,0 +1,249 @@
+/**************************************************************************
+ *
+ * FText.cpp -- File structures for text compression
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FText.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+
+BitAddr::BitAddr () {
+  Clear ();  
+}
+
+void BitAddr::Clear () {
+  byte = 0;
+  bit = 0;
+}
+
+bool BitAddr::Read (FILE *f) {
+  return (ReadUL (f, byte) && ReadUC (f, bit));
+}
+
+bool BitAddr::Write (FILE *f) const {
+  return (WriteUL (f, byte) && WriteUC (f, bit));
+}
+
+#define BITADDRLEN (sizeof(unsigned long) + sizeof(unsigned char))
+
+  
+TextLevelInfo::TextLevelInfo () {
+  Clear();
+}
+
+void TextLevelInfo::Clear () {
+  levelTag.erase (levelTag.begin(), levelTag.end());
+  textIdxPtr = 0;
+  numEntries = 0;
+}
+
+bool TextLevelInfo::Read (FILE *f) {
+  return (ReadUCArray (f, levelTag) &&
+	  ReadUL (f, textIdxPtr) &&
+	  ReadUL (f, numEntries));
+}
+
+bool TextLevelInfo::Write (FILE *f) const {
+  return (WriteUCArray (f, levelTag) &&
+	  WriteUL (f, textIdxPtr) &&
+	  WriteUL (f, numEntries));
+}
+
+ostream &operator<<(ostream &s, const TextLevelInfo &l) {
+  s << "  Tag: \"" << l.levelTag << "\"\n"
+    << "  textIdxPtr: " << l.textIdxPtr << "\n"
+    << "  numEntries: " << l.numEntries << "\n\n";
+  
+  return s;
+}
+
+
+FTextLevel::FTextLevel () {
+  Clear();
+}
+
+void FTextLevel::Clear () {
+  levelInfo.erase (levelInfo.begin(), levelInfo.end());
+}
+
+bool FTextLevel::Read (FILE *f) {
+  // read in the array size
+  unsigned long arrSize = 0;
+  if (!ReadVarLenUL (f, arrSize)) return false;
+
+  TextLevelInfo thisLevel;
+  while (arrSize > 0) {
+    // read and insert the next level information
+    if (!thisLevel.Read (f)) return false;
+    levelInfo[thisLevel.levelTag] = thisLevel;
+    
+    arrSize--;
+  }
+
+  return true;
+}
+
+bool FTextLevel::Write (FILE *f) const {
+  // write out the array size
+  if (!WriteVarLenUL (f, levelInfo.size())) return false;
+
+  // write out each level info
+  TextLevelInfoMap::const_iterator here = levelInfo.begin();
+  TextLevelInfoMap::const_iterator end = levelInfo.end();
+  while (here != end) {
+    if (!(*here).second.Write (f)) return false;
+    
+    here++;
+  }
+
+  return true;
+}
+
+ostream &operator<<(ostream &s, const FTextLevel &l) {
+  TextLevelInfoMap::const_iterator here = l.levelInfo.begin ();
+  TextLevelInfoMap::const_iterator end = l.levelInfo.end ();
+  while (here != end) {
+    s << (*here).second;
+    here++;
+  }
+  
+  return s;
+}
+
+
+TextIdx::TextIdx () {
+  Clear ();
+}
+
+void TextIdx::Clear () {
+  start.Clear();
+  end.Clear();
+  which = 0;
+}
+
+#define TEXTIDXLEN (BITADDRLEN*2 + sizeof(char))
+
+bool TextIdx::Read (FILE *f, const TextLevelInfo &levelInfo,
+		    unsigned long docNum) {
+  if (!SeekTextIdx (f, levelInfo, docNum)) return false;
+  return Read (f);
+}
+
+bool TextIdx::Read (FILE *f) {
+  return (start.Read (f) && end.Read (f) && ReadUC (f, which));
+}
+
+bool TextIdx::Write (FILE *f) const {
+  return (start.Write (f) && end.Write (f) && WriteUC (f, which));
+}
+
+ostream &operator<<(ostream &s, const TextIdx &t) {
+  s << "start byte: " << t.start.byte << "\n";
+  s << "start bit:  " << (int)t.start.bit << "\n";
+  s << "end byte:   " << t.end.byte << "\n";
+  s << "end bit:    " << (int)t.end.bit << "\n";
+  s << "which:      " << (int)t.which << "\n\n";
+  
+  return s;
+}
+
+
+bool SeekTextIdx (FILE *f, const TextLevelInfo &levelInfo,
+		  unsigned long docNum) {
+  if (docNum == 0 || docNum > levelInfo.numEntries) return false;
+  
+  unsigned long seekPos = levelInfo.textIdxPtr + (docNum-1) * TEXTIDXLEN;
+  if (fseek (f, seekPos, SEEK_SET) != 0) return false;
+  
+  return true;
+}
+
+
+bool ReadTextIdxArray (FILE *f, TextIdxArray &a, unsigned long arrSize) {
+  // clear the array
+  a.erase (a.begin(), a.end());
+
+  // read in each element
+  TextIdx idx;
+  while (arrSize > 0) {
+    if (!idx.Read(f)) return false;
+    a.push_back (idx);
+    arrSize --;
+  }
+
+  return true;
+}
+
+bool WriteTextIdxArray (FILE *f, const TextIdxArray &a) {
+  TextIdxArray::const_iterator here = a.begin();
+  TextIdxArray::const_iterator end = a.end();
+  while (here != end) {
+    if (!(*here).Write(f)) return false;
+    here++;
+  }
+  
+  return true;
+}
+
+
+CompressTextInfo::CompressTextInfo () {
+  Clear();
+}
+
+void CompressTextInfo::Clear () {
+  ResetStart();
+  docPtrs.erase (docPtrs.begin(), docPtrs.end());
+}
+
+void CompressTextInfo::ResetStart () {
+  inDoc = false;
+  start.Clear();
+  which = 0;
+}
+  
+void CompressTextInfo::SetStart (unsigned long startPos,
+				 unsigned char startBit,
+				 unsigned char startWhich) {
+  // place an imaginary end tag if needed
+  // note: the document tag always needs a closing tag
+  if (inDoc) SetEnd (startPos, startBit);
+
+  // remember start of level
+  inDoc = true;
+  start.byte = startPos;
+  start.bit = startBit;
+  which = startWhich;
+}
+
+void CompressTextInfo::SetEnd (unsigned long endPos,
+			       unsigned char endBit) {
+  if (inDoc) {
+    // add completed entry to list of ptrs
+    TextIdx textIdx;
+    textIdx.start = start;
+    textIdx.end.byte = endPos;
+    textIdx.end.bit = endBit;
+    textIdx.which = which;
+    docPtrs.push_back (textIdx);
+    
+    // reset this entry
+    ResetStart();
+  }
+}
+
Index: /trunk/indexers/mgpp/text/FText.h
===================================================================
--- /trunk/indexers/mgpp/text/FText.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/FText.h	(revision 3365)
@@ -0,0 +1,150 @@
+/**************************************************************************
+ *
+ * FText.h -- File structures for text compression
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FTEXT_H
+#define FTEXT_H
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+#include <stdio.h>
+#include "UCArray.h"
+
+
+// holds a bit address
+class BitAddr {
+public:
+  unsigned long byte;
+  unsigned char bit;
+
+  BitAddr ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+class TextLevelInfo {
+public:
+  UCArray levelTag;
+  unsigned long textIdxPtr;
+  unsigned long numEntries;
+
+  TextLevelInfo ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const TextLevelInfo &l);
+
+
+typedef map<UCArray, TextLevelInfo, LTUCArray> TextLevelInfoMap;
+
+
+class FTextLevel {
+public:
+  TextLevelInfoMap levelInfo;
+  
+  FTextLevel ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const FTextLevel &l);
+
+
+// the text.idx file points into the compressed text
+class TextIdx {
+public:
+  BitAddr start;       // the first bit of the compressed text
+  BitAddr end;         // the first bit past the end of the text
+  unsigned char which; // 0 = non-word, 1 = word
+  
+  TextIdx ();
+  void Clear ();
+
+  // does a seek and reads the appropriate record
+  bool Read (FILE *f, const TextLevelInfo &levelInfo,
+	     unsigned long docNum);
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const TextIdx &t);
+
+
+// note: document numbers start at 1
+
+bool SeekTextIdx (FILE *f, const TextLevelInfo &levelInfo,
+		  unsigned long docNum);
+
+
+// used to store the pointers while compressing the text
+typedef vector<TextIdx> TextIdxArray;
+
+bool ReadTextIdxArray (FILE *f, TextIdxArray &a, unsigned long arrSize);
+bool WriteTextIdxArray (FILE *f, const TextIdxArray &a);
+
+
+// used to store information about a level while compressing the text
+class CompressTextInfo {
+public:
+  bool inDoc;
+  BitAddr start;
+  unsigned char which;
+  TextIdxArray docPtrs;
+
+  CompressTextInfo ();
+  void Clear ();
+
+  // clears inDoc, start, and which (not the docPtrs)
+  void ResetStart ();
+  
+  // set the start of a level (closing off the last opening
+  // tag if needed)
+  void SetStart (unsigned long startPos,
+		 unsigned char startBit,
+		 unsigned char startWhich);
+
+  // if in a document, it will set the end, add the document to the
+  // list of document ptrs, and then call ResetStart
+  void SetEnd (unsigned long endPos,
+	       unsigned char endBit);
+};
+
+typedef map<UCArray, CompressTextInfo, LTUCArray> CompressTextInfoMap;
+
+
+#endif
Index: /trunk/indexers/mgpp/text/FileBuf.cpp
===================================================================
--- /trunk/indexers/mgpp/text/FileBuf.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/FileBuf.cpp	(revision 3365)
@@ -0,0 +1,50 @@
+/**************************************************************************
+ *
+ * FileBuf.cpp -- File buffer
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FileBuf.h"
+#include "sysfuncs.h"
+
+#ifdef __WIN32__
+# include <io.h>
+# define read _read
+#endif
+
+bool FileBuf::NextBuf () {
+  if (fd < 0) return false;
+  int numbytes = read (fd, buf, FILEBUFSIZE);
+  if (numbytes < 0) numbytes = 0;
+  
+  here = buf;
+  end = here + numbytes;
+
+  return (numbytes > 0);
+}
+
+FileBuf::FileBuf () {
+  here = end = buf;
+  fd = -1;
+}
+
+void FileBuf::SetFD (int _fd) {
+  here = end = buf;
+  fd = _fd;
+}
+
Index: /trunk/indexers/mgpp/text/FileBuf.h
===================================================================
--- /trunk/indexers/mgpp/text/FileBuf.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/FileBuf.h	(revision 3365)
@@ -0,0 +1,73 @@
+/**************************************************************************
+ *
+ * FileBuf.h -- File buffer
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FILEBUF_H
+#define FILEBUF_H
+
+#define FILEBUFSIZE 65536
+
+// some elderly windows compilers need to include <vector> (or something
+// similar) here so that they know what a "bool" is
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\vector>
+#elif defined(GSDL_USE_STL_H)
+#  include <vector.h>
+#else
+#  include <vector>
+#endif
+
+
+class FileBuf {
+protected:
+  unsigned char buf[FILEBUFSIZE];
+  unsigned char *here;
+  unsigned char *end;
+
+  int fd;
+
+  // gets a new buffer if need be, returns true if
+  // more could be got. if true is returned the caller
+  // can assume that here != end
+  bool NextBuf ();
+  
+public:
+  FileBuf ();
+
+  void SetFD (int _fd);
+
+  inline bool Get (unsigned char &c) {
+    if (here != end || NextBuf()) { c = *here++; return true; }
+    return false;
+  }
+  
+  inline bool Peek (unsigned char &c) {
+    if (here != end || NextBuf()) { c = *here; return true; }
+    return false;
+  }
+
+  inline bool Next () {
+    if (here != end || NextBuf()) { here++; return true; }
+    return false;
+  }
+};
+
+
+#endif
Index: /trunk/indexers/mgpp/text/FragLevelConvert.cpp
===================================================================
--- /trunk/indexers/mgpp/text/FragLevelConvert.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/FragLevelConvert.cpp	(revision 3365)
@@ -0,0 +1,105 @@
+/**************************************************************************
+ *
+ * FragLevelConvert.cpp -- converting between fragment and document numbers
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FragLevelConvert.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+
+
+FragLevelConvert::FragLevelConvert () {
+  numFrags = 0;
+  numLevelDocs = 0;
+}
+
+bool FragLevelConvert::Read (FILE *invfFile,
+			     unsigned long levelPtr,
+			     unsigned long _numFrags,
+			     unsigned long _numLevelDocs) {
+  levelStarts.erase (levelStarts.begin(), levelStarts.end());
+  numFrags = _numFrags;
+  numLevelDocs = _numLevelDocs;
+
+  // seek to the appropriate place in the inverted file
+  fseek (invfFile, levelPtr, SEEK_SET);
+
+  stdio_bitio_buffer buffer(invfFile);
+  
+  unsigned long pTag = numLevelDocs*2;
+  unsigned long B = BIO_Bblock_Init (numFrags+pTag, pTag);
+  unsigned long fragNum = 0;
+  unsigned long i;
+  for (i=0; i<numLevelDocs; i++) {
+    unsigned long delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+
+    levelStarts.push_back (fragNum);
+
+    // ignore end, assume every fragment is in a level
+    delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+  }
+
+  // fragNum would be the starting fragment of the next document
+  // if there was one
+  levelStarts.push_back (fragNum);
+  
+  if (fragNum != _numFrags) {
+    cerr << "last frag num:"<<fragNum<<" != num frags:"<<_numFrags<<endl;
+  } 
+  buffer.done();
+
+  return true;
+}
+
+bool FragLevelConvert::FragToLevel (unsigned long fragNum,
+				    unsigned long &levelDocNum) const {
+  // do binary search for something containing this fragment number
+  unsigned long low = 1, high = numLevelDocs;
+  unsigned long mid;
+
+  while ((mid = (low+high)/2) >= 1 && low <= high) {
+    if (fragNum > levelStarts[mid]) low = mid+1;
+    else if (fragNum <= levelStarts[mid-1]) high = mid-1;
+    else {
+      levelDocNum = mid;
+      return true;
+    }
+  }
+  return false;
+}
+
+
+bool FragLevelConvert::LevelToLevel (FragLevelConvert sectionLevelConvert,
+		 unsigned long levelNum, unsigned long &DocNum) {
+  
+  if (levelNum==1) {
+    DocNum=1;
+    return true;
+  }
+  unsigned long levelfragnum = sectionLevelConvert.levelStarts[levelNum-1]+1;
+
+  if (FragToLevel(levelfragnum, DocNum)) {
+    return true;
+  } 
+  return false;
+  
+
+}
Index: /trunk/indexers/mgpp/text/FragLevelConvert.h
===================================================================
--- /trunk/indexers/mgpp/text/FragLevelConvert.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/FragLevelConvert.h	(revision 3365)
@@ -0,0 +1,70 @@
+/**************************************************************************
+ *
+ * FragLevelConvert.h -- converting between fragment and document numbers
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FRAGLEVELCONVERT_H
+#define FRAGLEVELCONVERT_H
+
+#include <stdio.h>
+#include "UCArray.h"
+
+
+/*Notes about levelStarts (kjm18) - this is my interpretation of
+whats going on, feel free to correct this if its wrong.
+
+say you have 5 docs, consisting of fragments:
+1:  1-55
+2: 56-83
+3: 84-106
+4: 107-152
+5: 153-211
+
+then levelStarts would have the values
+0  1  2   3   4   5 
+0  55 83 106 152 211
+
+*/
+class FragLevelConvert {
+protected:
+  // note fragment start is one before before real start
+  vector<unsigned long> levelStarts;
+  unsigned long numFrags;
+  unsigned long numLevelDocs;
+  
+public:
+  FragLevelConvert ();
+
+  bool Read (FILE *invfFile, unsigned long levelPtr,
+	     unsigned long _numFrags, unsigned long _numLevelDocs);
+
+  bool FragToLevel (unsigned long fragNum, unsigned long &levelDocNum) const;
+
+  // this function converts from one level number to another
+  // the FragLevelConvert thats calling the method should be loaded with
+  // the level that you want to convert to, while the sectionLevelConvert
+  // should be loaded with the level converting from.
+  // level Num has the old level, DocNum will be returned with the new
+  // coresponding level
+  bool LevelToLevel (FragLevelConvert sectionLevelConvert,
+		   unsigned long levelNum, unsigned long &DocNum);
+};
+
+
+#endif
Index: /trunk/indexers/mgpp/text/GSDLQueryLex.cpp
===================================================================
--- /trunk/indexers/mgpp/text/GSDLQueryLex.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/GSDLQueryLex.cpp	(revision 3365)
@@ -0,0 +1,251 @@
+/**************************************************************************
+ *
+ * GSDLQueryLex.cpp -- Lexical analyser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "GSDLQueryLex.h"
+#include "unitool.h"
+#include "words.h"
+
+inline void AddNChar (UCArray::const_iterator &here,
+		      UCArray &text,
+		      int len) {
+  while (len > 0) {
+    text.push_back (*here++);
+    len--;
+  }
+}
+
+static bool ParseInteger (UCArray::const_iterator &here,
+			  UCArray::const_iterator end,
+			  LexEl &el) {
+  el.Clear();
+
+  // this version of end is used in unitool
+  //  UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (&*here, endMinus1, &c);
+
+  // check for positive or negative
+  bool neg = false;
+  if (c == '+') {
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  } else if (c == '-') {
+    neg = true;
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+
+  // read in number part
+  int numeric=0;
+  el.num = 0;
+  el.lexType = IntegerE;
+  /* stop integers at 4 digits */
+  while (c >= '0' && c <= '9'&& ++numeric<=MAXNUMERIC) {
+    el.num = el.num*10 + c - '0';
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+
+  if (neg) el.num *= -1;
+
+  return (!el.text.empty());
+}
+
+static bool ParsePotentialInteger(UCArray::const_iterator &here,
+				  UCArray::const_iterator end,
+				  LexEl &el) {
+  el.Clear();
+
+  // this version of end is used in unitool
+  //UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+
+  int charLen=0;
+  int length=0;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (&*here, endMinus1, &c);
+
+  // read in number part
+  int numeric=0;
+  el.num = 0;
+  el.lexType = IntegerE;
+  /* stop integers at 4 digits */
+  while (c >= '0' && c <= '9'&& ++numeric<=MAXNUMERIC) {
+    el.num = el.num*10 + c - '0';
+    AddNChar (here, el.text, charLen);
+    length += charLen;
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+  // check the next character -if it is a letter, then have a term, not an integer
+  if (!is_unicode_letter(c)) {
+    // this was just an integer
+    return (!el.text.empty());
+  }
+  // else its a term
+  el.lexType = TermE;
+  el.num = 0;
+  /* this bit taken from ParseIndexWord in words.h*/
+  while (length+charLen<=MAXSTEMLEN && charLen > 0 &&
+	 (is_unicode_letter(c) || (is_unicode_digit(c) &&
+				   ++numeric <= MAXNUMERIC))) {
+    AddNChar (here, el.text, charLen);
+    length += charLen;
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+
+  return (!el.text.empty());
+}
+static bool ParseTerm (UCArray::const_iterator &here,
+		       UCArray::const_iterator end,
+		       UCArray &text) {
+  //UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+  const unsigned char* new_here = ParseIndexWord (&*here, endMinus1, text);
+  here += (new_here - &*here); // advance iterator by number of chars advanced
+  return !text.empty();
+}
+
+
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el) {
+  el.Clear();
+
+  // strange things can happen if here == end == 0
+  if (here == end) return false;
+  
+  // this version of end is used in unitool
+  //UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+
+  // ignore all white space
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (&*here, endMinus1, &c);
+  while (here != end && is_unicode_space (c)) {
+    here += charLen;
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+  if (here == end) return false;
+
+  if (c == '(') {
+    el.lexType = OpenBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ')') {
+    el.lexType = CloseBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+
+  } else if (c =='[') {
+    el.lexType = OpenSquareBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+  
+  } else if (c ==']') {
+    el.lexType = CloseSquareBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+  
+  } else if (c == '\"') {
+    el.lexType = QuoteE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '/') {
+    el.lexType = TermWeightE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '#') {
+    el.lexType = StemMethodE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '^') {
+    el.lexType = RangeE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '@') {
+    el.lexType = AtE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ':') {
+    el.lexType = TagE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c=='&') {
+    el.lexType = AndOpE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '|') {
+    el.lexType = OrOpE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '!') {
+    el.lexType = NotOpE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '+' || c == '-' ) {
+    return  ParseInteger (here, end, el);
+  }
+
+  else if (c >= '0' && c <= '9') {
+    return ParsePotentialInteger (here, end, el);
+  }
+
+  // assume it is a term of some sort
+  if (!ParseTerm (here, end, el.text)) return false;
+
+  UCArray AND; SetCStr (AND, "AND");
+  if (el.text == AND) {
+    el.lexType = AndOpE;
+    return true;
+  }
+  UCArray OR; SetCStr (OR, "OR");
+  if (el.text == OR) {
+    el.lexType = OrOpE;
+    return true;
+  }
+  UCArray NOT; SetCStr (NOT, "NOT");
+  if (el.text == NOT) {
+    el.lexType = NotOpE;
+    return true;
+  }
+  UCArray NEAR; SetCStr (NEAR, "NEAR");
+  if (PrefixLen(el.text, NEAR)==4) {
+    el.lexType = NearOpE;
+    return true;
+  }
+  el.lexType = TermE;
+  return true;
+}
+
Index: /trunk/indexers/mgpp/text/GSDLQueryLex.h
===================================================================
--- /trunk/indexers/mgpp/text/GSDLQueryLex.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/GSDLQueryLex.h	(revision 3365)
@@ -0,0 +1,63 @@
+/**************************************************************************
+ *
+ * GSDLQueryLex.h -- Lexical analyser for a simple query language, for GSDL
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef GSDLQUERYLEX_H
+#define GSDLQUERYLEX_H
+
+#include "UCArray.h"
+
+
+// note: white space is ignored
+enum LexType {NoTypeE, OpenBracketE, CloseBracketE, TermE, AndOpE, OrOpE,
+	      NotOpE, NearOpE, QuoteE, IntegerE, TermWeightE, StemMethodE,
+	      RangeE, AtE, TagE, OpenSquareBracketE, CloseSquareBracketE};
+
+struct LexEl {
+  LexType lexType;
+  UCArray text;
+  int num; // for IntegerE
+
+  void Clear () {
+    lexType = NoTypeE;
+    UCArrayClear (text);
+    num = 0;
+  }
+  LexEl () { Clear (); }
+};
+
+
+// returns true if another lexical element could be parsed.
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el);
+
+
+#endif
+
+
+
+
+
+
+
+
+
+
Index: /trunk/indexers/mgpp/text/GSDLQueryParser.cpp
===================================================================
--- /trunk/indexers/mgpp/text/GSDLQueryParser.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/GSDLQueryParser.cpp	(revision 3365)
@@ -0,0 +1,405 @@
+/**************************************************************************
+ *
+ * QueryParser.cpp -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "GSDLQueryParser.h"
+#include "GSDLQueryLex.h"
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end, 
+				   int defaultBoolCOmbine,
+				   int defaultStemMethod);
+
+static QueryNode *AndAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  AndQueryNode *andNode = new AndQueryNode;
+  andNode->leftNode = t1;
+  andNode->rightNode = t2;
+  return andNode;
+}
+
+static QueryNode *OrAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  OrQueryNode *orNode = new OrQueryNode;
+  orNode->leftNode = t1;
+  orNode->rightNode = t2;
+  return orNode;
+}
+
+static QueryNode *NotAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  NotQueryNode *notNode = new NotQueryNode;
+  notNode->queryNode = t1;
+  notNode->notNode = t2;
+  return notNode;
+}
+
+// expects the opening bracket to have already been parsed
+// and discarded
+static QueryNode *ParseBracketExpression (UCArray::const_iterator &here,
+					  UCArray::const_iterator end, 
+					  int defaultBoolCombine,
+					  int defaultStemMethod) {
+  // get everything in the expression
+  QueryNode *curTree = ParseExpression (here, end, defaultBoolCombine,
+					defaultStemMethod);
+  
+  // gobble up tokens until a closing bracket is found
+  // or the end of the string
+  LexEl el;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == CloseBracketE) break;
+  }
+
+  return curTree;
+}
+
+static int ParseInt (UCArray::const_iterator &here,
+		     UCArray::const_iterator end) {
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  if (ParseLexEl (here, end, el) && el.lexType == IntegerE)
+    return el.num;
+  
+  here = oldHere; // not an integer
+  return 0;
+}
+
+// default is within 20 words
+static void SetRangeValues (TermNode &termNode,
+			    UCArray &nearby) {
+  UCArray NEARBY; SetCStr(NEARBY, "NEAR");
+  if (nearby == NEARBY) { // no modifier
+    termNode.startRange = -21;
+    termNode.endRange = 20;
+  }
+  else { // extract number
+    UCArray::const_iterator here = nearby.begin()+4;
+    UCArray::const_iterator end = nearby.end();
+    int size=0;
+    while (here != end) {
+      size = size*10 + (*here-'0');
+      here++;
+    }
+    termNode.startRange = -1 * (size+1);
+    termNode.endRange = size;
+  }
+}
+
+static unsigned long GetStemMethod(LexEl &el, int defaultStemMethod) {
+  // here expect el to contain some of c,s,i,u
+  // stem method 0 = c,u  00
+  // stem method 1 = i,u  01  - default for DL
+  // stem method 2 = c, s  10
+  // stem method 3 = i,s  11
+
+  unsigned long stem = (unsigned long)defaultStemMethod;
+      
+  UCArray::const_iterator here = el.text.begin();
+  UCArray::const_iterator end = el.text.end();
+
+  unsigned char c1 = *here; 
+  if (!(c1 == 'c'| c1 == 'i' | c1 == 'u' | c1 == 's')) 
+    return 4; // incorrect format
+  
+  here++;
+  unsigned char c2 = 'a';
+  if (here !=end) {
+    c2 = *here;
+    if (!(c2 == 'c'| c2 == 'i' | c2 == 'u' | c2 == 's')) 
+      return 4; // incorrect format
+  }
+  
+  if (c1 == 'i'|| c2=='i') stem |= 1; // set bit 0 to 1
+  if (c1 == 'c' || c2 == 'c') stem &=0xe; //set bit 0 to 0
+  if (c1 == 's'|| c2 == 's') stem |= 2; // set bit 1 to 1
+  if (c1 == 'u' || c2 =='u') stem &=0xd; // set bit 1 to 0
+  return stem;
+}
+  
+
+static void ParseTermModifiers (UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				TermNode &termNode,
+				int defaultStemMethod) {
+ 
+  termNode.stemMethod = defaultStemMethod;
+
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermWeightE) {
+      termNode.termWeight = ParseInt (here, end);
+      
+    } else if (el.lexType == StemMethodE) {
+      oldHere = here;
+      LexEl stem;
+      if (ParseLexEl (here, end, stem) && stem.lexType == TermE) {
+	termNode.stemMethod = GetStemMethod(stem, defaultStemMethod);
+	if (termNode.stemMethod == 4) { // error so backtrack
+	  here = oldHere;
+	  termNode.stemMethod = (unsigned long)defaultStemMethod;
+	} 
+      }else here = oldHere; //ignore - wrong syntax
+
+    } else if (el.lexType == RangeE) {
+      termNode.startRange = ParseInt (here, end);
+      termNode.endRange = ParseInt (here, end);
+      
+    } else if (el.lexType == AtE) {
+      termNode.startRange = termNode.endRange = ParseInt (here, end);
+      
+    } else {
+      // no term modifiers
+      here = oldHere;
+      break;
+    }
+    
+    oldHere = here;
+  }  
+}
+
+static void ParseProxModifiers (UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				ProxMatchQueryNode *proxNode) {
+  // so far only have one - the tag stuff
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TagE) {
+      oldHere = here; // don't backtrack past here
+      if (ParseLexEl (here, end, el) && el.lexType == TermE) {
+	proxNode->tagNodePtr = new TagNode;
+	proxNode->tagNodePtr->tagName = el.text;
+	
+      }
+      else { // error in tag
+	here = oldHere;
+      }
+    } // TagE
+    // add in other cases here
+    else {
+      // no modifiers
+      here = oldHere;
+      break;
+    }
+    oldHere = here;
+  }//while
+
+
+}
+
+// expects starting brackets to have been parsed
+static void ParseSquareBrackets(UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				ProxMatchQueryNode *proxNode,
+				int defaultStemMethod) {
+
+  LexEl el;
+  bool phrase=false;
+  bool first=true;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermE || el.lexType == IntegerE) {
+      TermNode termNode;
+      termNode.term = el.text;
+      ParseTermModifiers (here, end, termNode, defaultStemMethod);
+      if (phrase) {
+        if (first) first=false;
+        else {
+          termNode.startRange = -2;
+          termNode.endRange = -1;
+        }
+      }  
+      proxNode->terms.push_back(termNode);
+    }
+    else if (el.lexType == CloseSquareBracketE) {
+      break;
+    }
+    else if (el.lexType == AndOpE) {
+      // ignore, the words are AND'ed anyway
+      cerr << "and inside []\n";
+    }
+    else if (el.lexType == OrOpE) {
+      cerr << "or inside []\n";
+    }
+    else if (el.lexType == QuoteE) {
+      // phrase inside square brackets
+      if (phrase) phrase=false;
+      else phrase=true;
+    }
+    else {
+      //error
+      // just ignore for now
+      cerr <<"bad syntax inside []\n";
+    }
+  } // while
+
+}
+// expects the starting quote to have been parsed
+// and discarded
+// now phrases use the case and stem preference options
+// ie can search for a phrase ignoring case
+static void ParsePhrase (UCArray::const_iterator &here,
+			 UCArray::const_iterator end,
+			 ProxMatchQueryNode &proxNode, 
+			 int defaultStemMethod) {
+  LexEl el;
+  bool first = true;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermE || el.lexType == IntegerE) {
+      TermNode termNode;
+      termNode.term = el.text;
+      //termNode.stemMethod = defaultStemMethod;
+      ParseTermModifiers (here, end, termNode, defaultStemMethod);
+      if (first) {
+	first = false;
+      }
+      else {
+	termNode.startRange = -2;
+	termNode.endRange = -1;
+      }
+      proxNode.terms.push_back (termNode);
+      
+    } else if (el.lexType == QuoteE) {
+      break;
+      
+    } else {
+      // error
+      break;
+    }
+  }
+}
+
+static QueryNode *ParseTerm (UCArray::const_iterator &here,
+			     UCArray::const_iterator end,
+			     int defaultBoolCombine,
+			     int defaultStemMethod) {
+  LexEl el;
+
+  UCArray::const_iterator oldHere = here;
+  if (!ParseLexEl (here, end, el)) return NULL;
+
+  if (el.lexType == OpenBracketE)
+    return ParseBracketExpression (here, end, defaultBoolCombine,
+				   defaultStemMethod);
+
+  ProxMatchQueryNode *proxNode = new ProxMatchQueryNode;
+
+  if (el.lexType == TermE || el.lexType == IntegerE) {
+    TermNode termNode;
+    termNode.term = el.text;
+    ParseTermModifiers (here, end, termNode, defaultStemMethod);
+    oldHere = here;  // dont backtrack past here
+    if (ParseLexEl(here, end, el) && el.lexType == NearOpE) {
+      delete proxNode;
+      proxNode = (ProxMatchQueryNode *)ParseTerm(here, end, defaultBoolCombine,
+						 defaultStemMethod);
+      SetRangeValues(termNode, el.text);
+      proxNode->terms.push_back (termNode);
+      return proxNode;
+    }
+    else {
+      here = oldHere; // backtrack
+      proxNode->terms.push_back (termNode);
+      ParseProxModifiers(here, end, proxNode);
+      return proxNode;
+    }
+  } else if (el.lexType == QuoteE) {
+    ParsePhrase (here, end, *proxNode, defaultStemMethod);
+    return proxNode;
+  }
+  else if (el.lexType == OpenSquareBracketE) {
+    ParseSquareBrackets (here, end, proxNode, defaultStemMethod);
+    ParseProxModifiers (here, end, proxNode);
+    return proxNode;
+  }
+
+  // not a term
+  here = oldHere;
+  delete proxNode;
+  return NULL;
+}
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end,
+				   int defaultBoolCombine,
+				   int defaultStemMethod) {
+  LexEl el;
+  QueryNode *curTree = NULL;
+
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == OpenSquareBracketE ||
+	el.lexType == OpenBracketE ||
+	el.lexType == TermE ||
+	el.lexType == QuoteE ||
+	el.lexType == IntegerE ) {
+	//	el.lexType == TagE) { //tag at end of term now
+      // some type of term, back track and parse it
+      here = oldHere;
+      // if default==1, AND, else if==0, OR
+      if (defaultBoolCombine) {
+	curTree = AndAdd (curTree, ParseTerm (here, end, defaultBoolCombine, 
+					      defaultStemMethod));
+      }
+      else {
+	curTree = OrAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					     defaultStemMethod));
+      }
+
+    } else if (el.lexType == AndOpE) {
+      curTree = AndAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					    defaultStemMethod));
+      
+    } else if (el.lexType == OrOpE) {
+      curTree = OrAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					   defaultStemMethod));
+      
+    } else if (el.lexType == NotOpE) {
+      curTree = NotAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					    defaultStemMethod));
+      
+    } else if (el.lexType == CloseBracketE) {
+      // parsebracketexpression is waiting for the last bracket, so put it back
+      here = oldHere;
+      break;
+    
+    } else break;
+    
+    oldHere = here;
+  }
+
+  return curTree;
+}
+
+QueryNode *ParseQuery (const UCArray &queryStr, int defaultBoolCombine,
+		       int defaultStemMethod) {
+  UCArray::const_iterator here = queryStr.begin();
+  UCArray::const_iterator end = queryStr.end();
+  return ParseExpression (here, end, defaultBoolCombine, defaultStemMethod);
+}
Index: /trunk/indexers/mgpp/text/GSDLQueryParser.h
===================================================================
--- /trunk/indexers/mgpp/text/GSDLQueryParser.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/GSDLQueryParser.h	(revision 3365)
@@ -0,0 +1,37 @@
+/**************************************************************************
+ *
+ * QueryParser.h -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef GSDLQUERYPARSER_H
+#define GSDLQUERYPARSER_H
+
+#include "UCArray.h"
+#include "MGQuery.h"
+
+// returns NULL if the query could not be parsed
+// defaultBoolCOmbine determines if individual terms are combined using
+// AND or OR in the absence of specified operators.
+// defaultStemMethod used to set stemming and casefolding for terms where
+// its not set explicitly in the query string. THe defaultStemMethod value
+// should be set using the values from the preferences page
+QueryNode *ParseQuery (const UCArray &queryStr, int defaultBoolCombine,
+		       int defaultStemMethod);
+
+#endif
Index: /trunk/indexers/mgpp/text/IndexData.cpp
===================================================================
--- /trunk/indexers/mgpp/text/IndexData.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/IndexData.cpp	(revision 3365)
@@ -0,0 +1,207 @@
+/**************************************************************************
+ *
+ * IndexData.cpp -- Information needed for querying
+ * Copyright (C) 1999 Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "IndexData.h"
+#include "string.h"
+#include "mg_files.h"
+
+IndexData::IndexData () {
+  basePath[0] = '\0';
+  filename[0] = '\0';
+
+  dictFile = NULL;
+
+  stem1File = NULL;
+  stem2File = NULL;
+  stem3File = NULL;
+
+  invfFile = NULL;
+
+  approxWeightsFile = NULL;
+  exactWeightsFile = NULL;
+
+  curLevelNum = 0;
+}
+
+IndexData::~IndexData () {
+  UnloadData ();
+}
+
+bool IndexData::LoadData (const char *_basePath, const char *_filename) {
+  if (_filename[0] == '\0') return false;
+  
+  // make sure this data has not already been loaded
+  if (strcmp (_basePath, basePath) == 0 &&
+      strcmp (_filename, filename) == 0) return true;
+  
+  // make sure there is nothing else loaded
+  UnloadData ();
+
+  strcpy (basePath, _basePath);
+  strcpy (filename, _filename);
+
+  set_basepath (basePath);
+
+  // load in the level information
+  FILE *ivfLevelFile;
+  ivfLevelFile = open_file (filename, INVF_LEVEL_SUFFIX, "rb",
+			    MAGIC_INVF_LEVELS, MG_ABORT);
+  levels.Read (ivfLevelFile);
+  fclose (ivfLevelFile);
+
+  // blocked dictionary
+  dictFile = open_file (filename, INVF_DICT_BLOCKED_SUFFIX, "rb",
+			MAGIC_STEM, MG_ABORT);
+  if (!bdh.Read (dictFile)) { UnloadData (); return false; }
+
+  fseek (dictFile, bdh.wblk_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (dictFile, biWords)) { UnloadData (); return false; }
+
+  fseek (dictFile, bdh.tblk_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (dictFile, biTags)) { UnloadData (); return false; }
+
+  // blocked stem index 1
+  stem1File = open_file (filename, INVF_DICT_BLOCKED_1_SUFFIX,
+			 "rb", MAGIC_STEM_1, MG_ABORT);
+  if (!sih1.Read (stem1File)) { UnloadData (); return false; }
+
+  fseek (stem1File, sih1.block_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (stem1File, sii1)) { UnloadData (); return false; }
+  
+  // blocked stem index 2
+  stem2File = open_file (filename, INVF_DICT_BLOCKED_2_SUFFIX,
+			 "rb", MAGIC_STEM_2, MG_ABORT);
+  if (!sih2.Read (stem2File)) { UnloadData (); return false; }
+
+  fseek (stem2File, sih2.block_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (stem2File, sii2)) { UnloadData (); return false; }
+  
+  // blocked stem index 3
+  stem3File = open_file (filename, INVF_DICT_BLOCKED_3_SUFFIX,
+			 "rb", MAGIC_STEM_3, MG_ABORT);
+  if (!sih3.Read (stem3File)) { UnloadData (); return false; }
+
+  fseek (stem3File, sih3.block_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (stem3File, sii3)) { UnloadData (); return false; }
+
+  // inverted file
+  invfFile = open_file (filename, INVF_SUFFIX, "rb",
+			MAGIC_INVF, MG_ABORT);
+  ifh.Read (invfFile);
+
+  // weights
+  approxWeightsFile = open_file (filename, APPROX_WEIGHTS_SUFFIX, "rb",
+				 MAGIC_WGHT_APPROX, MG_ABORT);
+  exactWeightsFile = open_file (filename, WEIGHTS_SUFFIX, "rb",
+				MAGIC_WGHT, MG_ABORT);
+
+  return true;
+}
+
+bool IndexData::UnloadData () {
+  basePath[0] = '\0';
+  filename[0] = '\0';
+
+  if (dictFile != NULL) {
+    fclose (dictFile); dictFile = NULL;
+  }
+
+  if (stem1File != NULL) {
+    fclose (stem1File); stem1File = NULL;
+  }
+  if (stem2File != NULL) {
+    fclose (stem2File); stem2File = NULL;
+  }
+  if (stem3File != NULL) {
+    fclose (stem3File); stem3File = NULL;
+  }
+
+  if (invfFile != NULL) {
+    fclose (invfFile); invfFile = NULL;
+  }
+
+  if (approxWeightsFile != NULL) {
+    fclose (approxWeightsFile); approxWeightsFile = NULL;
+  }
+  if (exactWeightsFile != NULL) {
+    fclose (exactWeightsFile); exactWeightsFile = NULL;
+  }
+
+  UnloadLevel();
+  
+  return true;
+}
+
+
+bool IndexData::LoadLevel (const UCArray &level) {
+  // make sure this level is not already loaded
+  if (level == curLevel) return true;
+
+  // unload any levels currently loaded
+  UnloadLevel();
+
+  // make sure the required files are open
+  if (dictFile == NULL || invfFile == NULL ||
+      approxWeightsFile == NULL || exactWeightsFile == NULL)
+    return false;
+  
+  // read in the information from the dictionary
+  block_dict_el tagEl;
+  unsigned long tagElNum;
+  if (!SearchBlockDictEl (dictFile, biTags, bdh.entries_per_tblk,
+			  bdh.tag_dict_size, level, tagEl, tagElNum))
+    return false;
+
+  // read in the level conversion information
+  if (!levelConverter.Read (invfFile, tagEl.invf_ptr,
+			    bdh.num_frags, tagEl.frag_occur))
+    return false;
+
+  // read in the approximate weights
+  if (!weightData.Read (approxWeightsFile,
+			levels.levelInfo[level].approxWeightsDiskPtr,
+			levels.levelInfo[level].numEntries))
+    return false;
+  
+  // find the level number
+  curLevelNum = 0;
+  IvfLevelInfoMap::const_iterator levelHere, levelEnd;
+  for (levelHere=levels.levelInfo.begin(), levelEnd=levels.levelInfo.end();
+       levelHere!=levelEnd && (*levelHere).first != level; levelHere++)
+    curLevelNum++;
+
+  // make sure we found the level
+  if (levelHere == levelEnd) return false;
+
+  curLevel = level;
+
+  return true;
+}
+
+bool IndexData::UnloadLevel () {
+  curLevel.erase (curLevel.begin(), curLevel.end());
+  curLevelNum = 0;
+  
+  weightData.Free ();
+    
+  return true;
+}
+
Index: /trunk/indexers/mgpp/text/IndexData.h
===================================================================
--- /trunk/indexers/mgpp/text/IndexData.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/IndexData.h	(revision 3365)
@@ -0,0 +1,82 @@
+/**************************************************************************
+ *
+ * IndexData.h -- Information needed for querying
+ * Copyright (C) 1999 Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef INDEXDATA_H
+#define INDEXDATA_H
+
+#include <stdio.h>
+#include "invf.h"
+#include "FIvfLevelInfo.h"
+#include "FragLevelConvert.h"
+#include "Weights.h"
+
+
+class IndexData {
+public:
+  // general information
+  char basePath[512];
+  char filename[512];
+  FIvfLevel levels;
+  
+  // blocked dictionary
+  FILE *dictFile;
+  block_dict_header bdh;
+  block_idx biWords;
+  block_idx biTags;
+
+  // stem indexes
+  FILE *stem1File;
+  FILE *stem2File;
+  FILE *stem3File;
+  stem_idx_header sih1;
+  stem_idx_header sih2;
+  stem_idx_header sih3;
+  block_idx sii1;
+  block_idx sii2;
+  block_idx sii3;
+
+  // inverted file
+  FILE *invfFile;
+  invf_file_header ifh;
+
+  // weights
+  FILE *approxWeightsFile;
+  FILE *exactWeightsFile;
+
+  // level specific information
+  UCArray curLevel; // last level that was loaded
+  unsigned long curLevelNum;
+  FragLevelConvert levelConverter;
+  ApproxWeightsData weightData;
+  
+  IndexData ();
+  ~IndexData ();
+
+  // LoadData will only load the data if it has not been loaded before
+  bool LoadData (const char *_basePath, const char *_filename);
+  bool UnloadData ();
+
+  bool LoadLevel (const UCArray &level);
+  bool UnloadLevel ();
+};
+
+
+#endif
Index: /trunk/indexers/mgpp/text/MGQuery.cpp
===================================================================
--- /trunk/indexers/mgpp/text/MGQuery.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/MGQuery.cpp	(revision 3365)
@@ -0,0 +1,760 @@
+/**************************************************************************
+ *
+ * MGQuery.cpp -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "MGQuery.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+#include "Terms.h"
+#include "QueryResultsSort.h"
+#include <assert.h>
+
+
+void PrintIndent (ostream &s, int indent) {
+  while (indent-- > 0) s << " ";
+}
+
+
+void PrintIndentText (ostream &s, char *text, int indent) {
+  PrintIndent (s, indent);
+  s << text;
+}
+
+void PrintNode (ostream &s, QueryNode *node, int indent) {
+  if (node == NULL) {
+    PrintIndentText (s, "NULL\n", indent);
+  } else {
+    node->Print (s, indent+2);
+  }
+}
+
+  
+
+QueryNode::QueryNode () {
+}
+
+QueryNode::~QueryNode () {
+}
+
+void QueryNode::Calculate (IndexData &/*indexData*/,
+			   const QueryInfo &/*queryInfo*/,
+			   QueryResult &result) const {
+  result.Clear();
+}
+
+void QueryNode::Free () {
+}
+
+void QueryNode::Print (ostream &/*s*/, int /*indent*/) const {
+}
+
+
+
+AndQueryNode::AndQueryNode () {
+  leftNode = NULL;
+  rightNode = NULL;
+}
+
+AndQueryNode::~AndQueryNode () {
+  Free ();
+}
+
+void AndQueryNode::Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			      QueryResult &result) const {
+  result.Clear();
+
+  // an And between nothing and something is nothing...
+  if (leftNode == NULL || rightNode == NULL) return;
+  
+  // calculate the result from the left tree and the result
+  // from the right tree
+  QueryResult rightResult;
+  leftNode->Calculate (indexData, queryInfo, result);
+  rightNode->Calculate (indexData, queryInfo, rightResult);
+
+  // merge the results, this can be done in place as the results
+  // will always shrink with an And
+
+  bool haveAccum = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  if (haveAccum && (result.ranks.size() != result.docs.size() ||
+		    rightResult.ranks.size() != rightResult.docs.size())) {
+    // shouldn't ever get here
+    haveAccum = false;
+    assert (0);
+  }
+
+  // combine document numbers and corresponding ranks
+  unsigned long leftI = 0;
+  unsigned long rightI = 0;
+  unsigned long outI = 0;
+  while (leftI < result.docs.size() &&
+	 rightI < rightResult.docs.size()) {
+    if (result.docs[leftI] < rightResult.docs[rightI]) {
+      leftI++;
+    } else if (result.docs[leftI] > rightResult.docs[rightI]) {
+      rightI++;
+    } else {
+      // the documents are equal
+      result.docs[outI] = result.docs[leftI];
+      if (haveAccum)
+	result.ranks[outI] = result.ranks[leftI] + rightResult.ranks[rightI];
+      leftI++;
+      rightI++;
+      outI++;
+    }
+  }
+
+  // erase unused document numbers and ranks
+  result.docs.erase(result.docs.begin()+outI, result.docs.end());
+  if (haveAccum)
+    result.ranks.erase(result.ranks.begin()+outI, result.ranks.end());
+    
+  // combine term frequency information
+  if (queryInfo.needTermFreqs)
+    result.termFreqs.insert (result.termFreqs.end(),
+			     rightResult.termFreqs.begin(),
+			     rightResult.termFreqs.end());
+}
+
+void AndQueryNode::Free () {
+  if (leftNode != NULL) {
+    delete leftNode;
+    leftNode = NULL;
+  }
+  if (rightNode != NULL) {
+    delete rightNode;
+    rightNode = NULL;
+  }
+}
+
+void AndQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "leftNode:\n", indent);
+  PrintNode (s, leftNode, indent+2);
+  PrintIndentText (s, "AND\n", indent);
+  PrintIndentText (s, "rightNode:\n", indent);
+  PrintNode (s, rightNode, indent+2);
+}
+
+
+OrQueryNode::OrQueryNode () {
+  leftNode = NULL;
+  rightNode = NULL;
+}
+
+OrQueryNode::~OrQueryNode () {
+  Free ();
+}
+
+void OrQueryNode::Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			      QueryResult &result) const {
+  result.Clear();
+  
+  // calculate the result from the left tree and the result
+  // from the right tree
+  QueryResult leftResult;
+  QueryResult rightResult;
+  if (leftNode != NULL)
+    leftNode->Calculate (indexData, queryInfo, leftResult);
+  if (rightNode != NULL)
+    rightNode->Calculate (indexData, queryInfo, rightResult);
+
+  // merge the results
+  
+  bool haveAccum = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  if (haveAccum && (leftResult.ranks.size() != leftResult.docs.size() ||
+		    rightResult.ranks.size() != rightResult.docs.size())) {
+    // shouldn't ever get here
+    haveAccum = false;
+    assert (0);
+  }
+
+  // combine document numbers and corresponding ranks
+  unsigned long leftSize = leftResult.docs.size();
+  unsigned long rightSize = rightResult.docs.size();
+  unsigned long leftI = 0;
+  unsigned long rightI = 0;
+  unsigned long leftDocNum = 0;
+  unsigned long rightDocNum = 0;
+  while (leftI < leftSize || rightI < rightSize) {
+    // check leftI
+    if (leftI < leftResult.docs.size())
+      leftDocNum = leftResult.docs[leftI];
+    else leftDocNum = ULONG_MAX;
+    
+    // check rightI
+    if (rightI < rightResult.docs.size())
+      rightDocNum = rightResult.docs[rightI];
+    else rightDocNum = ULONG_MAX;
+    
+    // combine
+    if (leftDocNum < rightDocNum) {
+      result.docs.push_back (leftDocNum);
+      if (haveAccum)
+	result.ranks.push_back (leftResult.ranks[leftI]);
+      leftI++;
+      
+    } else if (leftDocNum > rightDocNum) {
+      result.docs.push_back (rightDocNum);
+      if (haveAccum)
+	result.ranks.push_back (rightResult.ranks[rightI]);
+      rightI++;
+      
+    } else { // equal
+      result.docs.push_back (leftDocNum);
+      if (haveAccum)
+	result.ranks.push_back (leftResult.ranks[leftI] +
+				rightResult.ranks[rightI]);
+      leftI++;
+      rightI++;
+    }
+  }
+
+  // combine term frequency information
+  if (queryInfo.needTermFreqs) {
+    result.termFreqs.insert (result.termFreqs.end(),
+			     leftResult.termFreqs.begin(),
+			     leftResult.termFreqs.end());
+    result.termFreqs.insert (result.termFreqs.end(),
+			     rightResult.termFreqs.begin(),
+			     rightResult.termFreqs.end());
+  }
+}
+
+void OrQueryNode::Free () {
+  if (leftNode != NULL) {
+    delete leftNode;
+    leftNode = NULL;
+  }
+  if (rightNode != NULL) {
+    delete rightNode;
+    rightNode = NULL;
+  }
+}
+
+void OrQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "leftNode:\n", indent);
+  PrintNode (s, leftNode, indent+2);
+  PrintIndentText (s, "OR\n", indent);
+  PrintIndentText (s, "rightNode:\n", indent);
+  PrintNode (s, rightNode, indent+2);
+}
+
+
+
+NotQueryNode::NotQueryNode () {
+  queryNode = NULL;
+  notNode = NULL;
+}
+
+NotQueryNode::~NotQueryNode () {
+  Free ();
+}
+
+void NotQueryNode::Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			      QueryResult &result) const {
+  result.Clear();
+
+  // check for nothing
+  if (queryNode == NULL) return;
+  if (notNode == NULL) {
+    queryNode->Calculate (indexData, queryInfo, result);
+    return;
+  }
+  
+  // calculate the result from the query tree and the result
+  // from the not tree
+  QueryResult notResult;
+  queryNode->Calculate (indexData, queryInfo, result);
+  notNode->Calculate (indexData, queryInfo, notResult);
+
+  // merge the results, this can be done in place as the results
+  // will always shrink with a Not
+
+  bool haveAccum = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  if (haveAccum && (result.ranks.size() != result.docs.size() ||
+		    notResult.ranks.size() != notResult.docs.size())) {
+    // shouldn't ever get here
+    haveAccum = false;
+    assert (0);
+  }
+
+  // combine document numbers and corresponding ranks
+  unsigned long queryI = 0;
+  unsigned long notI = 0;
+  unsigned long outI = 0;
+  while (queryI < result.docs.size() &&
+	 notI < notResult.docs.size()) {
+    if (result.docs[queryI] < notResult.docs[notI]) {
+      // found a document not in the notResults
+      result.docs[outI] = result.docs[queryI];
+      if (haveAccum)
+	result.ranks[outI] = result.ranks[queryI];
+      queryI++;
+      outI++;
+    } else if (result.docs[queryI] > notResult.docs[notI]) {
+      notI++;
+    } else {
+      // the documents are equal, ignore both
+      queryI++;
+      notI++;
+    }
+  }
+
+  // erase unused document numbers and ranks
+  result.docs.erase(result.docs.begin()+outI, result.docs.end());
+  if (haveAccum)
+    result.ranks.erase(result.ranks.begin()+outI, result.ranks.end());
+  
+  // combine term frequency information
+  if (queryInfo.needTermFreqs)
+    result.termFreqs.insert (result.termFreqs.end(),
+			     notResult.termFreqs.begin(),
+			     notResult.termFreqs.end());
+}
+
+void NotQueryNode::Free () {
+  if (queryNode != NULL) {
+    delete queryNode;
+    queryNode = NULL;
+  }
+  if (notNode != NULL) {
+    delete notNode;
+    notNode = NULL;
+  }
+}
+
+void NotQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "queryNode:\n", indent);
+  PrintNode (s, queryNode, indent+2);
+  PrintIndentText (s, "NOT\n", indent);
+  PrintIndentText (s, "notNode:\n", indent);
+  PrintNode (s, notNode, indent+2);
+}
+
+
+void TagNode::Clear () {
+  UCArrayClear (tagName);
+}
+
+void TagNode::Calculate (IndexData &indexData,
+			 FragRangeArray &fragRange) const {
+  fragRange.erase (fragRange.begin(), fragRange.end());
+  if (tagName.empty()) return;
+
+  // get information about this tag
+  block_dict_el tagEl;
+  unsigned long tagElNum;
+  if (!SearchBlockDictEl (indexData.dictFile, indexData.biTags,
+			  indexData.bdh.entries_per_tblk,
+			  indexData.bdh.tag_dict_size,
+			  tagName, tagEl, tagElNum))
+    return;
+
+  // seek to the appropriate place in the inverted file
+  fseek (indexData.invfFile, tagEl.invf_ptr, SEEK_SET);
+
+  stdio_bitio_buffer buffer(indexData.invfFile);
+  
+  unsigned long pTag = tagEl.frag_occur*2;
+  unsigned long B = BIO_Bblock_Init (indexData.bdh.num_frags+pTag, pTag);
+  unsigned long fragNum = 0;
+  unsigned long i;
+  FragRange thisFrag;
+  for (i=0; i<tagEl.frag_occur; i++) {
+    // get start
+    unsigned long delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+
+    thisFrag.rangeStart = fragNum;
+
+    // get end
+    delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+
+    thisFrag.rangeEnd = fragNum;
+    fragRange.push_back (thisFrag);
+  }
+
+  buffer.done();
+}
+
+void TagNode::Free () {
+  Clear ();
+}
+
+void TagNode::Print (ostream &s, int indent) const {
+  PrintIndent (s, indent);
+  s << "TAG: \"" << tagName << "\"\n";
+}
+
+
+void TermNode::Clear () {
+  UCArrayClear (term);
+  termWeight = 1;
+  stemMethod = 0;
+  startRange = NO_TERM_RANGE_START;
+  endRange = NO_TERM_RANGE_END;
+}
+
+TermNode::TermNode () {
+  Clear ();
+}
+
+void TermNode::Calculate (IndexData &indexData,
+			  bool needFragFreqs,
+			  FragRangeArray *fragLimits,
+			  FragData &fragData,
+			  UCArrayVector &equivTerms) const {
+  fragData.Clear ();
+  equivTerms.erase(equivTerms.begin(), equivTerms.end());
+
+  // get a list of term numbers
+  vector<unsigned long> equivNums;
+  FindWordNumbers (indexData, term, stemMethod, equivNums);
+
+  // get the information for each word and merge it with
+  // previous results
+  FragData tempFragData1;
+  FragData tempFragData2;
+  UCArray equivWord;
+  vector<unsigned long>::iterator here = equivNums.begin();
+  vector<unsigned long>::iterator end = equivNums.end();
+  while (here != end) {
+    // get the information for this word
+    ReadTermFragData (indexData, needFragFreqs, *here,
+		      tempFragData1, fragLimits, equivWord);
+    equivTerms.push_back(equivWord);
+    // combine with last results
+    tempFragData2 = fragData;
+    CombineFragData (needFragFreqs, tempFragData1, tempFragData2, fragData);
+    
+    here++;
+  }
+}
+
+void TermNode::Free () {
+  Clear ();
+}
+
+void TermNode::Print (ostream &s, int indent) const {
+  PrintIndent (s, indent);
+  s << "TERM: \"" << term << "\"\n";
+  PrintIndent (s, indent+2);
+  s << "termWeight: " << termWeight << "\n";
+  PrintIndent (s, indent+2);
+  s << "stemMethod: " << stemMethod << "\n";
+  PrintIndent (s, indent+2);
+  s << "startRange: " << startRange << "\n";
+  PrintIndent (s, indent+2);
+  s << "endRange: " << endRange << "\n";
+}
+
+
+ProxMatchQueryNode::ProxMatchQueryNode () {
+  tagNodePtr = NULL;
+}
+
+ProxMatchQueryNode::~ProxMatchQueryNode () {
+  Free ();
+}
+
+void ProxMatchQueryNode::Calculate (IndexData &indexData,
+				    const QueryInfo &queryInfo,
+				    QueryResult &result) const {
+  result.Clear ();
+
+  bool needFragFreqs = (queryInfo.sortByRank || queryInfo.needRankInfo);
+
+  // read in the tag if needed
+  FragRangeArray fragLimits;
+  FragRangeArray *fragLimitsPtr = NULL;
+  if (tagNodePtr == NULL && terms.size() > 1) {
+    // multiple terms must be compared relative to some tag
+    // otherwise phrase matches could span documents
+    TagNode tempTagNode;
+    tempTagNode.tagName = indexData.curLevel;
+    tempTagNode.Calculate (indexData, fragLimits);
+    fragLimitsPtr = &fragLimits;
+    
+  } else if (tagNodePtr != NULL) {
+    (*tagNodePtr).Calculate (indexData, fragLimits);
+    fragLimitsPtr = &fragLimits;
+  }
+
+  UCArray tagOrLevel = indexData.curLevel;
+  if (tagNodePtr != NULL) tagOrLevel = (*tagNodePtr).tagName;
+
+  // read in the first term
+  FragData termData;
+  UCArrayVector equivTerms;
+  TermNodeArray::const_iterator termHere=terms.begin(), termEnd = terms.end();
+  if (termHere != termEnd) {
+    (*termHere).Calculate (indexData, needFragFreqs, fragLimitsPtr, 
+			   termData, equivTerms);
+
+    // convert initial fragment information
+    FragsToQueryResult (indexData,
+			queryInfo,
+			termData,
+			tagOrLevel,
+			(*termHere).term,
+			(*termHere).stemMethod,
+			(*termHere).termWeight,
+			equivTerms,
+			result);
+  
+    termHere++;
+
+    if (termHere == termEnd) return; // nothing more to do
+  }
+
+  // read and combine the rest of the terms
+  FragData comTermData;
+  while (termHere != termEnd) {
+    (*termHere).Calculate (indexData, needFragFreqs,
+			   fragLimitsPtr, comTermData, equivTerms);
+
+    AndFragsToQueryResult (indexData,
+			   queryInfo,
+			   comTermData,
+			   tagOrLevel,
+			   (*termHere).term,
+			   (*termHere).stemMethod,
+			   (*termHere).termWeight,
+			   equivTerms,
+			   result);
+    
+    AndCombineFragData (needFragFreqs, termData, comTermData,
+			(*termHere).startRange,
+			(*termHere).endRange,
+			fragLimitsPtr);
+    termHere++;
+  }
+
+  // remove unwanted document numbers
+  RemoveUnwantedResults (indexData,
+			 queryInfo,
+			 termData,
+			 result);
+}
+
+void ProxMatchQueryNode::Free () {
+  if (tagNodePtr != NULL) {
+    delete tagNodePtr;
+    tagNodePtr = NULL;
+  }
+  terms.erase (terms.begin(), terms.end());
+}
+
+void ProxMatchQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "PROXMATCH\n", indent);
+  if (tagNodePtr != NULL) tagNodePtr->Print (s, indent+2);
+  
+  TermNodeArray::const_iterator here = terms.begin();
+  TermNodeArray::const_iterator end = terms.end();
+  while (here != end) {
+    (*here).Print (s, indent+2);
+    here++;
+  }
+}
+
+
+void BrowseQueryNode::Clear () {
+  UCArrayClear(term);
+}
+
+void BrowseQueryNode::Calculate (IndexData &indexData, BrowseQueryResult &result) const {
+
+  unsigned long number=0;
+  FindNearestWordNumber(indexData, term, number);
+  if (number + startPosition > 0 ) {
+    number = number+startPosition;
+  }
+  else {
+    number = 1;
+  }
+
+  GetTermList (indexData, number, numTerms, result.termFreqs);
+
+}
+
+
+
+void BrowseQueryNode::Free () {
+  Clear();
+}
+
+
+void BrowseQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText(s, "BROWSEQUERYNODE\n", indent);
+  PrintIndent (s, indent+2);
+  s << "TERM:"<<term<<"\n";
+  PrintIndent (s, indent+2);
+  s << "Start position: "<< startPosition<<", Num terms: "<< numTerms<<"\n";
+
+
+}
+
+
+
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      QueryResult &result) {
+  result.Clear ();
+
+  // make sure level is current
+  if (!indexData.LoadLevel (queryInfo.docLevel)) {
+    return;
+  }
+  // do query
+  if (queryTree == NULL) return;
+  
+  (*queryTree).Calculate (indexData, queryInfo, result);
+
+  // make weights into ranks if needed
+  unsigned long i;
+  if (queryInfo.sortByRank || queryInfo.needRankInfo) {
+    for (i=0; i<result.ranks.size(); i++) {
+      result.ranks[i] /=
+	indexData.weightData.GetLowerApproxDocWeight (result.docs[i]);
+    }
+  }
+  
+  unsigned long resultsSize = queryInfo.maxDocs;
+  if (resultsSize == 0 || resultsSize > result.docs.size())
+    resultsSize = result.docs.size();
+
+  result.actualNumDocs = result.docs.size(); // the total number of docs 
+                                   //returned, before pruning based on maxDocs
+
+  // sort results by rank if needed
+  GTRank gtRank;
+  if (queryInfo.sortByRank) {
+    // need in ascending order for SelectAddHeap
+    MakeHeap (result.docs.begin(), result.ranks.begin(),
+	      resultsSize, gtRank);
+    SelectAddHeap (result.docs.begin(), result.ranks.begin(), resultsSize,
+		   result.docs.begin()+resultsSize,
+		   result.ranks.begin()+resultsSize,
+		   (unsigned long) result.ranks.size()-resultsSize,
+		   gtRank);
+
+    // sort into descending order
+    SortHeap (result.docs.begin(), result.ranks.begin(), resultsSize, gtRank);
+  }
+  
+  // get exact weights if needed
+  if (queryInfo.exactWeights &&
+      (queryInfo.sortByRank || queryInfo.needRankInfo)) {
+    unsigned long exactDiskPtr =
+      indexData.levels.levelInfo[indexData.curLevel].exactWeightsDiskPtr;
+    
+    for (i=0; i<resultsSize; i++) {
+      result.ranks[i] =  result.ranks[i] *
+	indexData.weightData.GetLowerApproxDocWeight (result.docs[i]) /
+	GetExactDocWeight (indexData.exactWeightsFile, exactDiskPtr,
+			   result.docs[i]);
+    }
+
+    // re-sort top candidates based on exact weights
+    if (queryInfo.sortByRank) {
+      MakeHeap (result.docs.begin(), result.ranks.begin(),
+		resultsSize, gtRank);
+      SortHeap (result.docs.begin(), result.ranks.begin(),
+		resultsSize, gtRank);
+    }
+  }
+
+  // get rid of unwanted ranking results
+  if (result.ranks.empty()) {
+    // do nothing
+  } else if (!queryInfo.needRankInfo) {
+    result.ranks.erase(result.ranks.begin(), result.ranks.end());
+  } else {
+    result.ranks.erase(result.ranks.begin()+resultsSize, result.ranks.end());
+  }
+
+  // remove extra docs that are unwanted
+  result.docs.erase(result.docs.begin()+resultsSize, result.docs.end());
+}
+
+// new MGQuery to retrieve doc and section nums
+// this will return doc nums for the level queried at (set in queryInfo)
+// in QueryResult.docs and if a second level is specified, 
+// it will return corresponding docnums for that level in QueryResult.levels
+// If there is no level specified, or that level is invalid, the query
+// level is used
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      ExtQueryResult &realresult, UCArray &level) {
+  realresult.Clear ();
+  QueryResult result; // temp result
+
+  // do the normal query
+  MGQuery (indexData, queryInfo, queryTree, result);
+
+  // now that have the final result stuff, convert to ExtQueryResult,
+  // add in level nums if needed
+
+  realresult.docs = result.docs;
+  realresult.ranks = result.ranks;
+  realresult.termFreqs = result.termFreqs;
+  realresult.actualNumDocs = result.actualNumDocs;
+
+  if (queryInfo.docLevel == level || level.empty()) {
+    realresult.levels = result.docs;
+    return;
+  }
+
+  // else need to convert from queryInfo.docLevel to level
+
+  // the original level info
+  FragLevelConvert sectionLevelConverter = indexData.levelConverter;
+
+  // the new level info
+  if (!indexData.LoadLevel(level)) {
+    realresult.levels = result.docs;
+    return;
+  }
+  
+  unsigned long DocNum = 0;
+  
+  for (unsigned long i=0; i<realresult.docs.size(); i++) {
+
+    // do an if ! here????
+    indexData.levelConverter.LevelToLevel(sectionLevelConverter, realresult.docs[i], DocNum);
+    realresult.levels.push_back(DocNum);
+  }
+  
+}
+
+
+
+// new function for full text browsing, 
+void MGBrowseQuery (IndexData &indexData, UCArray &level,
+		  const BrowseQueryNode &node,
+		  BrowseQueryResult &result) {
+
+  indexData.LoadLevel(level);
+  node.Calculate(indexData, result); 
+
+}
Index: /trunk/indexers/mgpp/text/MGQuery.h
===================================================================
--- /trunk/indexers/mgpp/text/MGQuery.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/MGQuery.h	(revision 3365)
@@ -0,0 +1,197 @@
+/**************************************************************************
+ *
+ * MGQuery.h -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef MGQUERY_H
+#define MGQUERY_H
+
+#include "IndexData.h"
+#include <limits.h>
+#include "Terms.h"
+
+// forms base for defining and calculating the boolean
+// part of a query
+class QueryNode {
+public:
+  QueryNode ();
+  virtual ~QueryNode ();
+
+  virtual void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			  QueryResult &result) const;
+  virtual void Free ();
+  virtual void Print (ostream &s, int indent=0) const;
+};
+
+
+
+void PrintIndent (ostream &s, int indent);
+void PrintIndentText (ostream &s, char *text, int indent);
+void PrintNode (ostream &s, QueryNode *node, int indent=0);
+
+
+
+// basic boolean nodes
+
+class AndQueryNode : public QueryNode {
+public:
+  QueryNode *leftNode;
+  QueryNode *rightNode;
+  
+  AndQueryNode ();
+  ~AndQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+class OrQueryNode : public QueryNode {
+public:
+  QueryNode *leftNode;
+  QueryNode *rightNode;
+  
+  OrQueryNode ();
+  ~OrQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+class NotQueryNode : public QueryNode {
+public:
+  QueryNode *queryNode;
+  QueryNode *notNode;
+  
+  NotQueryNode ();
+  ~NotQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+
+
+class TagNode {
+public:
+  UCArray tagName;
+
+  void Clear ();
+  TagNode () { Clear (); }
+
+  void Calculate (IndexData &indexData, FragRangeArray &fragRange) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+
+#define NO_TERM_RANGE_START (LONG_MIN/2)
+#define NO_TERM_RANGE_END   (LONG_MAX/2)
+
+/* NOTE: range stuff - the range values are for the previous term relative
+to the current term. So if searching for the phrase "the cat", 'the' doesn't 
+need range limits, but 'cat' has a range of -2 to -1. ie, if have found 'cat'
+then 'the' has to be at position between -2 and -1 relative to 'cat'.
+"the cat" could also be searched for by 'cat' with no range limits, then 'the' with range 0 to 1.
+range values are relative to the gaps between words:
+   x   y   z   X   a   b   c
+    -3   -2  -1  0   1   2   3
+
+ */
+class TermNode {
+public:
+  UCArray term;
+  unsigned long termWeight;
+  unsigned long stemMethod;
+  signed long startRange; // range relative to last term
+  signed long endRange;
+  
+  void Clear ();
+  TermNode ();
+
+  // fragLimits can be NULL if not neede
+  void Calculate (IndexData &indexData,
+		  bool needFragFreqs,
+		  FragRangeArray *fragLimits,
+		  FragData &fragData,
+		  UCArrayVector &equivTerms) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+typedef vector<TermNode> TermNodeArray;
+
+
+class ProxMatchQueryNode : public QueryNode {
+public:
+  TagNode *tagNodePtr; // optional field
+  TermNodeArray terms;
+  
+  ProxMatchQueryNode ();
+  ~ProxMatchQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+class BrowseQueryNode :public QueryNode {
+ public:
+  UCArray term;
+  signed long startPosition;
+  unsigned long numTerms;
+
+  void Clear();
+  BrowseQueryNode () { Clear(); }
+  //  ~BrowseQueryNode ();
+
+  void Calculate (IndexData &indexData, BrowseQueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+
+
+
+
+};
+
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      QueryResult &result);
+
+// this function for retrieving results with both section doc nums
+// and Document docnums
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      ExtQueryResult &result, UCArray &level);
+
+
+// new function for full text browsing, 
+void MGBrowseQuery (IndexData &indexData, UCArray &level,
+		  const BrowseQueryNode &node,
+		  BrowseQueryResult &result);
+
+#endif
Index: /trunk/indexers/mgpp/text/Makefile.in
===================================================================
--- /trunk/indexers/mgpp/text/Makefile.in	(revision 3365)
+++ /trunk/indexers/mgpp/text/Makefile.in	(revision 3365)
@@ -0,0 +1,315 @@
+###########################################################################
+#
+# Makefile.in -- Makefile for the MG system
+# Copyright (C) 1994  Neil Sharman;  1995 Tim Shimmin
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+#       @(#)Makefile.in	1.10 22 Mar 1994
+#
+###########################################################################
+
+PACKAGE = @PACKAGE@
+VERSION = @VERSION@
+
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+bindir = @bindir@
+srcdir = @srcdir@
+subdir = text
+top_srcdir = @top_srcdir@
+transform = @program_transform_name@
+
+AR = ar
+CC = @CC@
+CXX = @CXX@
+CFLAGS = @CFLAGS@ -ansi -DSILENT -DSHORT_SUFFIX
+CXXFLAGS = @CXXFLAGS@ -DSILENT -DSHORT_SUFFIX
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS = ../lib/libmg.a -lstdc++ @LIBS@
+TOUCH = echo timestamp >
+AWK = @AWK@
+
+##########################################
+# George Buchanan - August 1999
+# 
+# compiling with make NZDL=1 to ensure that 
+# the NZDL compile mode is used - this has been changed to the default - kjm18
+#
+##########################################
+
+datadir = $(prefix)/share
+libexecdir = $(exec_prefix)/libexec
+localedir = $(datadir)/locale
+mandir = $(prefix)/man/man1
+
+COMPILE = $(CC) -c $(DEFS) $(TREC_MODE) $(READLINE) $(INCLUDES) $(CFLAGS)
+CXXCOMPILE = $(CXX) -c $(DEFS) $(TREC_MODE) $(READLINE) $(INCLUDES) $(CXXFLAGS)
+LINK = $(CC) $(LDFLAGS) -o $@
+
+.SUFFIXES:
+.SUFFIXES: .c .o
+.c.o:
+	$(COMPILE) $<
+
+.SUFFIXES: .cpp .o
+.cpp.o:
+	$(CXXCOMPILE) $<
+
+
+ANSI2KNR = @ANSI2KNR@
+o = .@U@o
+
+.SUFFIXES: ._c ._o
+.c._c:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $@
+._c._o:
+	@echo $(COMPILE) $<
+	@rm -f _$*.c
+	@ln $< _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+.c._o:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $*._c
+	@echo $(COMPILE) $*._c
+	@rm -f _$*.c
+	@ln $*._c _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+
+INCLUDES = -I.. -I../lib -I.
+
+#############################################################################
+#
+# Uncomment (and possibly edit) the following lines if you wish
+# to use the GNU readline library for the input in mgquery
+# 
+#READLINE = -DGNU_READLINE -I/opt/gnu/include/readline -L/opt/gnu/lib
+#READLINE_LIBS = -lreadline -ltermcap
+#
+#
+#############################################################################
+
+SOURCES = \
+  FIvfLevelInfo.cpp        ivf.pass1.cpp            mgpp_invf_dump.cpp       \
+  FText.cpp                ivf.pass2.cpp            mgpp_passes.cpp          \
+  FileBuf.cpp              locallib.cpp             mgpp_perf_hash_build.cpp \
+  FragLevelConvert.cpp     mg.special.cpp           mgpp_stem_idx.cpp        \
+  IndexData.cpp            mgpp_compression_dict.cpp  mgpp_weights_build.cpp   \
+  TagInfo.cpp              mgpp_decompress_text.cpp   stemmer.cpp            \
+  TextEl.cpp               mg_errors.cpp            text.cpp               \
+  TextGet.cpp              mgpp_fast_comp_dict.cpp    text.pass1.cpp         \
+  UCArray.cpp              mg_files.cpp             text.pass2.cpp         \
+  comp_dict.cpp            mgpp_invf_dict.cpp         words.cpp              \
+  invf.cpp                 mgpp_invf_dict_dump.cpp    Weights.cpp            \
+  MGQuery.cpp              Terms.cpp                QueryTester.cpp        \
+  QueryLex.cpp             QueryParser.cpp         \
+  GSDLQueryLex.cpp	   GSDLQueryParser.cpp      Queryer.cpp
+
+
+HEADERS = \
+  FIvfLevelInfo.h     TextEl.h            hash.h              stemmer.h  \
+  FText.h             TextGet.h           invf.h              text.h     \
+  FileBuf.h           UCArray.h           locallib.h          words.h    \
+  FragLevelConvert.h  WordData.h          mg.h                           \
+  IndexData.h         build.h             mg_errors.h                    \
+  TagInfo.h           comp_dict.h         mg_files.h          Weights.h  \
+  MGQuery.h           Terms.h             QueryLex.h          QueryParser.h \
+  GSDLQueryLex.h	GSDLQueryParser.h
+
+
+MISC          = Makefile.in 
+
+ALLFILES      =	$(SOURCES) $(HEADERS) $(MISC) $(MAN)
+
+DISTFILES     = $(ALLFILES) 
+
+EXEC          = mgpp_passes mgpp_decompress_text mgpp_perf_hash_build \
+                mgpp_invf_dump mgpp_weights_build mgpp_invf_dict mgpp_stem_idx \
+                mgpp_invf_dict_dump mgpp_compression_dict mgpp_fast_comp_dict \
+                QueryTester Queryer
+
+MAN           =  mgpp_weights_build.1  mgpp_invf_dict.1 \
+                 mgpp_passes.1 mgpp_perf_hash_build.1 \
+                 mgpp_compression_dict.1  mgpp_fast_comp_dict.1 \
+                 mgpp_stem_idx.1
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+
+all:	$(EXEC) libtextin.a
+
+depend:
+	makedepend -Y -- $(DEFS) $(INCLUDES) $(CXXFLAGS) $(CFLAGS) -- $(SOURCES)
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+
+PASSES_OBJS = \
+	mgpp_passes$o text.pass1$o comp_dict$o \
+	text.pass2$o locallib$o \
+	ivf.pass1$o ivf.pass2$o mg.special$o mg_files$o \
+	words$o invf$o FileBuf$o TextEl$o UCArray$o TagInfo$o \
+        FText$o text$o FIvfLevelInfo$o
+
+mgpp_passes: $(PASSES_OBJS)
+	$(LINK) $(PASSES_OBJS) $(LIBS)
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+QUERYER_OBJS = Queryer$o mg_files$o mg_errors$o locallib$o invf$o UCArray$o \
+               FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+               IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+               GSDLQueryParser$o GSDLQueryLex$o words$o
+
+Queryer: $(QUERYER_OBJS)
+	$(LINK) $(QUERYER_OBJS) $(LIBS)
+
+QUERY_TESTER_OBJS = QueryTester$o mg_files$o invf$o UCArray$o \
+                    FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+                    IndexData$o stemmer$o Weights$o
+
+QueryTester: $(QUERY_TESTER_OBJS)
+	$(LINK) $(QUERY_TESTER_OBJS) $(LIBS)
+
+WEIGHTS_BUILD_OBJS = mgpp_weights_build$o mg_files$o invf$o UCArray$o \
+                     FIvfLevelInfo$o FragLevelConvert$o
+
+mgpp_weights_build: $(WEIGHTS_BUILD_OBJS)
+	$(LINK) $(WEIGHTS_BUILD_OBJS) $(LIBS)
+
+COMP_DICT_OBJS = mgpp_compression_dict$o mg_files$o locallib$o
+
+mgpp_compression_dict: $(COMP_DICT_OBJS)
+	$(LINK) $(COMP_DICT_OBJS) $(LIBS)
+
+FAST_COMP_OBJS = mgpp_fast_comp_dict$o mg_files$o locallib$o
+
+mgpp_fast_comp_dict: $(FAST_COMP_OBJS)
+	$(LINK) $(FAST_COMP_OBJS) $(LIBS)
+
+INVF_DUMP_OBJS = mgpp_invf_dump$o locallib$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dump: $(INVF_DUMP_OBJS)
+	$(LINK) $(INVF_DUMP_OBJS) $(LIBS)
+
+INVF_DICT_OBJS = mgpp_invf_dict$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict: $(INVF_DICT_OBJS)
+	$(LINK) $(INVF_DICT_OBJS) $(LIBS)
+
+INVF_DICT_DUMP_OBJS = mgpp_invf_dict_dump$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict_dump: $(INVF_DICT_DUMP_OBJS)
+	$(LINK) $(INVF_DICT_DUMP_OBJS) $(LIBS)
+
+PERF_HASH_OBJS = mgpp_perf_hash_build$o mg_files$o invf$o UCArray$o
+
+mgpp_perf_hash_build: $(PERF_HASH_OBJS)
+	$(LINK) $(PERF_HASH_OBJS) $(LIBS)
+
+STEM_IDX_OBJS = mgpp_stem_idx$o mg_files$o stemmer$o \
+                invf$o UCArray$o
+
+mgpp_stem_idx: $(STEM_IDX_OBJS)
+	$(LINK) $(STEM_IDX_OBJS) $(LIBS)
+
+DECOMPRESS_TEXT_OBJS = \
+                mgpp_decompress_text$o mg_files$o mg_errors$o \
+                locallib$o text$o UCArray$o TextGet$o FText$o
+
+mgpp_decompress_text: $(DECOMPRESS_TEXT_OBJS)
+	$(LINK) $(DECOMPRESS_TEXT_OBJS) $(LIBS)
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+ansi2knr: ansi2knr.o
+	$(LINK) ansi2knr.o $(LIBS)
+
+install: execinstall
+uninstall: execuninstall
+
+execinstall: $(EXEC)
+	../mkinstalldirs $(bindir); \
+	for name in $(EXEC); do \
+	  $(INSTALL_PROGRAM) $$name \
+	  $(bindir)/`echo $$name | sed '$(transform)'`; \
+	done
+
+execuninstall:
+	for name in $(EXEC); do \
+	 rm -f $(bindir)/`echo $$name | sed '$(transform)'`; \
+	done
+
+maninstall: $(MAN)
+	$(top_srcdir)/mkinstalldirs $(mandir); \
+	for f in $(MAN); do \
+	  $(INSTALL_DATA) $$f $(mandir)/$$f ;\
+	done
+
+manuninstall: $(MAN)
+	for f in $(MAN); do \
+	  rm -f $(mandir)/$$f ;\
+	done
+
+#mg_errors$o removed from LIB_OBJS to avoid conflict with mg's libtextin.a
+LIB_OBJS = mg_files$o  mg_errors$o locallib$o invf$o UCArray$o \
+           FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+           IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+           GSDLQueryParser$o GSDLQueryLex$o words$o
+
+libtextin.a: $(LIB_OBJS)
+	rm -f libtextin.a
+	$(AR) cru libtextin.a $(LIB_OBJS)
+	$(RANLIB) libtextin.a
+
+
+mostlyclean:
+	rm -f *$o _*.c _*.o *._c *._o core core.*
+
+clean: mostlyclean
+	rm -f $(EXEC) libtextin.a
+
+distclean: clean
+	rm -f ansi2knr
+	rm -f Makefile
+
+maintainer-clean: distclean
+	@echo "This command is intended only for maintainers to use;"
+	@echo "rebuilding the deleted files may require special tools."
+
+dist: $(DISTFILES)
+	for file in $(DISTFILES); do \
+	  ln $(srcdir)/$$file $(distdir) 2> /dev/null \
+	  || cp -p $(srcdir)/$$file $(distdir); \
+	done
+
+movclean:
+	for file in $(SOURCES); do \
+          test = $$file; \
+          echo $("test.c"##*.c); \
+        done 
+
+# jrm21
+#Makefile: Makefile.in ../../../config.status
+#	cd ../../.. && CONFIG_FILES=$(subdir)/$@ CONFIG_HEADERS= ./config.status
+
+# Tell versions [3.59,3.63) of GNU make not to export all variables.
+# Otherwise a system limit (for SysV at least) may be exc
+
+
Index: /trunk/indexers/mgpp/text/QueryLex.cpp
===================================================================
--- /trunk/indexers/mgpp/text/QueryLex.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/QueryLex.cpp	(revision 3365)
@@ -0,0 +1,169 @@
+/**************************************************************************
+ *
+ * QueryLex.cpp -- Lexical analyser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "QueryLex.h"
+#include "unitool.h"
+#include "words.h"
+
+inline void AddNChar (UCArray::const_iterator &here,
+		      UCArray &text,
+		      int len) {
+  while (len > 0) {
+    text.push_back (*here++);
+    len--;
+  }
+}
+
+static bool ParseInteger (UCArray::const_iterator &here,
+			  UCArray::const_iterator end,
+			  LexEl &el) {
+  el.Clear();
+
+  // this version of end is used in unitool
+  UCArray::const_iterator endMinus1 = end-1;
+
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (here, endMinus1, &c);
+
+  // check for positive or negative
+  bool neg = false;
+  if (c == '+') {
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  } else if (c == '-') {
+    neg = true;
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  }
+
+  // read in number part
+  el.num = 0;
+  el.lexType = IntegerE;
+  while (c >= '0' && c <= '9') {
+    el.num = el.num*10 + c - '0';
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  }
+
+  if (neg) el.num *= -1;
+
+  return (!el.text.empty());
+}
+
+static bool ParseTerm (UCArray::const_iterator &here,
+		       UCArray::const_iterator end,
+		       UCArray &text) {
+  UCArray::const_iterator endMinus1 = end-1;
+  here = ParseIndexWord (here, endMinus1, text);
+  return !text.empty();
+}
+
+
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el) {
+  el.Clear();
+
+  // strange things can happen if here == end == 0
+  if (here == end) return false;
+  
+  // this version of end is used in unitool
+  UCArray::const_iterator endMinus1 = end-1;
+  
+  // ignore all white space
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (here, endMinus1, &c);
+  while (here != end && is_unicode_space (c)) {
+    here += charLen;
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  }
+  if (here == end) return false;
+
+  if (c == '(') {
+    el.lexType = OpenBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ')') {
+    el.lexType = CloseBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '\"') {
+    el.lexType = QuoteE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '#') {
+    el.lexType = TermWeightE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '$') {
+    el.lexType = StemMethodE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '^') {
+    el.lexType = RangeE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '@') {
+    el.lexType = AtE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ':') {
+    el.lexType = TagE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '+' || c == '-' ||
+	     (c >= '0' && c <= '9')) {
+    return ParseInteger (here, end, el);
+  }
+
+  // assume it is a term of some sort
+  if (!ParseTerm (here, end, el.text)) return false;
+
+  UCArray AND; SetCStr (AND, "AND");
+  if (el.text == AND) {
+    el.lexType = AndOpE;
+    return true;
+  }
+  UCArray OR; SetCStr (OR, "OR");
+  if (el.text == OR) {
+    el.lexType = OrOpE;
+    return true;
+  }
+  UCArray NOT; SetCStr (NOT, "NOT");
+  if (el.text == NOT) {
+    el.lexType = NotOpE;
+    return true;
+  }
+
+  el.lexType = TermE;
+  return true;
+}
+
Index: /trunk/indexers/mgpp/text/QueryLex.h
===================================================================
--- /trunk/indexers/mgpp/text/QueryLex.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/QueryLex.h	(revision 3365)
@@ -0,0 +1,53 @@
+/**************************************************************************
+ *
+ * QueryLex.h -- Lexical analyser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef QUERYLEX_H
+#define QUERYLEX_H
+
+#include "UCArray.h"
+
+
+// note: white space is ignored
+enum LexType {NoTypeE, OpenBracketE, CloseBracketE, TermE, AndOpE, OrOpE,
+	      NotOpE, QuoteE, IntegerE, TermWeightE, StemMethodE,
+	      RangeE, AtE, TagE};
+
+struct LexEl {
+  LexType lexType;
+  UCArray text;
+  int num; // for IntegerE
+
+  void Clear () {
+    lexType = NoTypeE;
+    UCArrayClear (text);
+    num = 0;
+  }
+  LexEl () { Clear (); }
+};
+
+
+// returns true if another lexical element could be parsed.
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el);
+
+
+#endif
Index: /trunk/indexers/mgpp/text/QueryParser.cpp
===================================================================
--- /trunk/indexers/mgpp/text/QueryParser.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/QueryParser.cpp	(revision 3365)
@@ -0,0 +1,219 @@
+/**************************************************************************
+ *
+ * QueryParser.cpp -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "QueryParser.h"
+#include "QueryLex.h"
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end);
+
+static QueryNode *AndAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  AndQueryNode *andNode = new AndQueryNode;
+  andNode->leftNode = t1;
+  andNode->rightNode = t2;
+  return andNode;
+}
+
+static QueryNode *OrAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  OrQueryNode *orNode = new OrQueryNode;
+  orNode->leftNode = t1;
+  orNode->rightNode = t2;
+  return orNode;
+}
+
+static QueryNode *NotAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  NotQueryNode *notNode = new NotQueryNode;
+  notNode->queryNode = t1;
+  notNode->notNode = t2;
+  return notNode;
+}
+
+// expects the opening bracket to have already been parsed
+// and discarded
+static QueryNode *ParseBracketExpression (UCArray::const_iterator &here,
+					  UCArray::const_iterator end) {
+  // get everything in the expression
+  QueryNode *curTree = ParseExpression (here, end);
+  
+  // gobble up tokens until a closing bracket is found
+  // or the end of the string
+  LexEl el;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == CloseBracketE) break;
+  }
+
+  return curTree;
+}
+
+static int ParseInt (UCArray::const_iterator &here,
+		     UCArray::const_iterator end) {
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  if (ParseLexEl (here, end, el) && el.lexType == IntegerE)
+    return el.num;
+  
+  here = oldHere; // not an integer
+  return 0;
+}
+
+
+static void ParseTermModifiers (UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				TermNode &termNode) {
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermWeightE) {
+      termNode.termWeight = ParseInt (here, end);
+      
+    } else if (el.lexType == StemMethodE) {
+      termNode.stemMethod = ParseInt (here, end);
+      
+    } else if (el.lexType == RangeE) {
+      termNode.startRange = ParseInt (here, end);
+      termNode.endRange = ParseInt (here, end);
+      
+    } else if (el.lexType == AtE) {
+      termNode.startRange = termNode.endRange = ParseInt (here, end);
+      
+    } else {
+      // no term modifiers
+      here = oldHere;
+      break;
+    }
+    
+    oldHere = here;
+  }  
+}
+
+// expects the starting quote to have been parsed
+// and discarded
+static void ParsePhrase (UCArray::const_iterator &here,
+			 UCArray::const_iterator end,
+			 ProxMatchQueryNode &proxNode) {
+  LexEl el;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermE || el.lexType == IntegerE) {
+      TermNode termNode;
+      termNode.term = el.text;
+      ParseTermModifiers (here, end, termNode);
+      proxNode.terms.push_back (termNode);
+      
+    } else if (el.lexType == QuoteE) {
+      break;
+      
+    } else {
+      // error
+      break;
+    }
+  }
+}
+
+static QueryNode *ParseTerm (UCArray::const_iterator &here,
+			     UCArray::const_iterator end) {
+  LexEl el;
+
+  UCArray::const_iterator oldHere = here;
+  if (!ParseLexEl (here, end, el)) return NULL;
+
+  if (el.lexType == OpenBracketE)
+    return ParseBracketExpression (here, end);
+
+  ProxMatchQueryNode *proxNode = new ProxMatchQueryNode;
+
+  // check for a tag
+  if (el.lexType == TagE) {
+    oldHere = here; // don't backtrack past here
+    if (!ParseLexEl (here, end, el)) return NULL;
+    if (el.lexType == TermE) {
+      proxNode->tagNodePtr = new TagNode;
+      proxNode->tagNodePtr->tagName = el.text;
+      if (!ParseLexEl (here, end, el)) return NULL;
+    }
+  }
+  
+  if (el.lexType == TermE || el.lexType == IntegerE) {
+    TermNode termNode;
+    termNode.term = el.text;
+    ParseTermModifiers (here, end, termNode);
+    proxNode->terms.push_back (termNode);
+    return proxNode;
+    
+  } else if (el.lexType == QuoteE) {
+    ParsePhrase (here, end, *proxNode);
+    return proxNode;
+  }
+
+  // not a term
+  here = oldHere;
+  delete proxNode;
+  return NULL;
+}
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end) {
+  LexEl el;
+  QueryNode *curTree = NULL;
+
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == OpenBracketE ||
+	el.lexType == TermE ||
+	el.lexType == QuoteE ||
+	el.lexType == IntegerE ||
+	el.lexType == TagE) {
+      // some type of term, back track and parse it
+      here = oldHere;
+      curTree = OrAdd (curTree, ParseTerm (here, end));
+      
+    } else if (el.lexType == AndOpE) {
+      curTree = AndAdd (curTree, ParseTerm (here, end));
+      
+    } else if (el.lexType == OrOpE) {
+      curTree = OrAdd (curTree, ParseTerm (here, end));
+      
+    } else if (el.lexType == NotOpE) {
+      curTree = NotAdd (curTree, ParseTerm (here, end));
+      
+    } else break;
+    
+    oldHere = here;
+  }
+
+  return curTree;
+}
+
+QueryNode *ParseQuery (const UCArray &queryStr) {
+  UCArray::const_iterator here = queryStr.begin();
+  UCArray::const_iterator end = queryStr.end();
+  return ParseExpression (here, end);
+}
Index: /trunk/indexers/mgpp/text/QueryParser.h
===================================================================
--- /trunk/indexers/mgpp/text/QueryParser.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/QueryParser.h	(revision 3365)
@@ -0,0 +1,31 @@
+/**************************************************************************
+ *
+ * QueryParser.h -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef QUERYPARSER_H
+#define QUERYPARSER_H
+
+#include "UCArray.h"
+#include "MGQuery.h"
+
+// returns NULL if the query could not be parsed
+QueryNode *ParseQuery (const UCArray &queryStr);
+
+#endif
Index: /trunk/indexers/mgpp/text/QueryResultsSort.h
===================================================================
--- /trunk/indexers/mgpp/text/QueryResultsSort.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/QueryResultsSort.h	(revision 3365)
@@ -0,0 +1,140 @@
+/**************************************************************************
+ *
+ * QueryResultsSort.h -- Functions to sort query results and parallel data
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef QUERYRESULTSSORT_H
+#define QUERYRESULTSSORT_H
+
+#include "Terms.h"
+
+
+struct LTRank {
+  bool operator()(const float &f1, const float &f2) const
+    { return (f1 < f2); }
+};
+
+struct GTRank {
+  bool operator()(const float &f1, const float &f2) const
+    { return (f1 > f2); }
+};
+
+
+
+// assumes the rank vector is the same size as the document
+// number vector. assumes the value to be pushed is in
+// docNumHeap + heapSize - 1 and rankHeap + heapSize - 1
+// topIndex can be set
+template <class RandomAccessIterator, class Compare>
+void PushHeap (DocNumArray::iterator docNumHeap,
+	       RandomAccessIterator parallelHeap,
+	       unsigned long heapSize,
+	       const Compare &comp) {
+  unsigned long holeIndex = heapSize-1;
+  unsigned long parent = (holeIndex - 1) / 2;
+  while (holeIndex > 0 && comp (*(parallelHeap+parent),
+				*(parallelHeap+holeIndex))) {
+    swap (*(parallelHeap+holeIndex), *(parallelHeap+parent));
+    swap (*(docNumHeap+holeIndex), *(docNumHeap+parent));
+    holeIndex = parent;
+    parent = (holeIndex - 1) / 2;
+  }
+}
+
+// AdjustHeap is used to add a element to a heap after one
+// element in the heap has been replaced with a new one
+template <class RandomAccessIterator, class Compare>
+void AdjustHeap (DocNumArray::iterator docNumHeap,
+		     RandomAccessIterator parallelHeap,
+		     unsigned long holeIndex,
+		     unsigned long heapSize,
+		     const Compare &comp) {
+  unsigned long secondChild = 2 * holeIndex + 2;
+  while (secondChild < heapSize) {
+    if (comp(*(parallelHeap + secondChild),
+	     *(parallelHeap + (secondChild - 1))))
+      secondChild--;
+    swap (*(parallelHeap+holeIndex), *(parallelHeap+secondChild));
+    swap (*(docNumHeap+holeIndex), *(docNumHeap+secondChild));
+    holeIndex = secondChild;
+    secondChild = 2 * secondChild + 2;
+  }
+  if (secondChild == heapSize) {
+    swap (*(parallelHeap+holeIndex), *(parallelHeap+(secondChild-1)));
+    swap (*(docNumHeap+holeIndex), *(docNumHeap+(secondChild-1)));
+    holeIndex = secondChild - 1;
+  }
+  PushHeap (docNumHeap, parallelHeap, holeIndex+1, comp);
+}
+
+template <class RandomAccessIterator, class Compare>
+void PopHeap (DocNumArray::iterator docNumHeap,
+	      RandomAccessIterator parallelHeap,
+	      unsigned long heapSize,
+	      const Compare &comp) {
+  swap (*(parallelHeap), *(parallelHeap+heapSize-1));
+  swap (*(docNumHeap), *(docNumHeap+heapSize-1));
+  AdjustHeap (docNumHeap, parallelHeap, (unsigned long)0, heapSize-1, comp);
+}
+
+template <class RandomAccessIterator, class Compare>
+void MakeHeap (DocNumArray::iterator docNumHeap,
+	       RandomAccessIterator parallelHeap,
+	       unsigned long size,
+	       const Compare &comp) {
+  if (size < 2) return;
+  
+  unsigned long i = 0;
+  for (i=1; i<=size; i++)
+    PushHeap (docNumHeap, parallelHeap, i, comp);
+}
+
+ 
+template <class RandomAccessIterator, class Compare>
+void SortHeap (DocNumArray::iterator docNumHeap,
+	       RandomAccessIterator parallelHeap,
+	       unsigned long size,
+	       const Compare &comp) {
+  if (size < 2) return;
+
+  unsigned long i;
+  for (i=size; i>1; i--)
+    PopHeap (docNumHeap, parallelHeap, i, comp);
+}
+
+// uses the heap to get the top "size" elements
+template <class RandomAccessIterator, class Compare>
+void SelectAddHeap (DocNumArray::iterator docNumHeap,
+		    RandomAccessIterator parallelHeap,
+		    unsigned long heapSize,
+		    DocNumArray::iterator docNumAdd,
+		    RandomAccessIterator parallelAdd,
+		    unsigned long addSize,
+		    const Compare &comp) {
+  unsigned long i;
+  for (i=0; i<addSize; i++) {
+    if (comp(*(parallelAdd+i), *(parallelHeap))) {
+      swap (*(docNumHeap), *(docNumAdd+i));
+      swap (*(parallelHeap), *(parallelAdd+i));
+      AdjustHeap (docNumHeap, parallelHeap, (unsigned long)0, heapSize, comp);
+    }
+  }
+}
+
+#endif
Index: /trunk/indexers/mgpp/text/QueryTester.cpp
===================================================================
--- /trunk/indexers/mgpp/text/QueryTester.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/QueryTester.cpp	(revision 3365)
@@ -0,0 +1,447 @@
+/**************************************************************************
+ *
+ * QueryTester.cpp -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "MGQuery.h"
+
+
+class SetQueryNode : public QueryNode {
+public:
+  QueryResult queryResult;
+  
+  SetQueryNode () {}
+  ~SetQueryNode () {}
+
+  void Calculate (IndexData &/*indexData*/, const QueryInfo &queryInfo,
+		  QueryResult &result) const {
+    result = queryResult;
+    if (!queryInfo.sortByRank && !queryInfo.needRankInfo)
+      result.ranks.erase (result.ranks.begin(), result.ranks.end());
+    if (!queryInfo.needTermFreqs)
+      result.termFreqs.erase (result.termFreqs.begin(),
+			      result.termFreqs.end());
+  }
+  void Free () { queryResult.Clear(); }
+};
+
+
+bool Test1 () {
+  AndQueryNode andNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  RankArray &rankSet1 = setNode1->queryResult.ranks;
+  docSet1.push_back (1);  rankSet1.push_back (0.1);
+  docSet1.push_back (10); rankSet1.push_back (0.2);
+  docSet1.push_back (15); rankSet1.push_back (0.2);
+  docSet1.push_back (18); rankSet1.push_back (0.4);
+  docSet1.push_back (19); rankSet1.push_back (0.5);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  RankArray &rankSet2 = setNode2->queryResult.ranks;
+  docSet2.push_back (2);  rankSet2.push_back (0.1);
+  docSet2.push_back (11); rankSet2.push_back (0.2);
+  docSet2.push_back (12); rankSet2.push_back (0.3);
+  docSet2.push_back (13); rankSet2.push_back (0.4);
+  docSet2.push_back (14); rankSet2.push_back (0.5);
+  docSet2.push_back (15); rankSet2.push_back (0.6);
+  docSet2.push_back (16); rankSet2.push_back (0.7);
+  docSet2.push_back (17); rankSet2.push_back (0.8);
+  docSet2.push_back (19); rankSet2.push_back (0.9);
+  docSet2.push_back (20); rankSet2.push_back (0.1);
+  docSet2.push_back (21); rankSet2.push_back (0.2);
+
+  cout << "\n" << setNode1->queryResult << "AND\n\n"
+       << setNode2->queryResult;
+  
+
+  andNode.leftNode = setNode1;
+  andNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = true;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  andNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  RankArray &rcRankSet = resultCompare.ranks;
+  rcDocSet.push_back (15);  rcRankSet.push_back (0.2+0.6);
+  rcDocSet.push_back (19);  rcRankSet.push_back (1.4);
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test1b () {
+  AndQueryNode andNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  docSet1.push_back (1); 
+  docSet1.push_back (10);
+  docSet1.push_back (15);
+  docSet1.push_back (18);
+  docSet1.push_back (19);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  docSet2.push_back (2); 
+  docSet2.push_back (11);
+  docSet2.push_back (12);
+  docSet2.push_back (13);
+  docSet2.push_back (14);
+  docSet2.push_back (15);
+  docSet2.push_back (16);
+  docSet2.push_back (17);
+  docSet2.push_back (19);
+  docSet2.push_back (20);
+  docSet2.push_back (21);
+
+  cout << "\n" << setNode1->queryResult << "AND\n\n"
+       << setNode2->queryResult;
+  
+
+  andNode.leftNode = setNode1;
+  andNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = false;
+  queryInfo.needRankInfo = false;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  andNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  rcDocSet.push_back (15); 
+  rcDocSet.push_back (19); 
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test2 () {
+  OrQueryNode orNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  RankArray &rankSet1 = setNode1->queryResult.ranks;
+  docSet1.push_back (1);  rankSet1.push_back (0.1);
+  docSet1.push_back (10); rankSet1.push_back (0.2);
+  docSet1.push_back (15); rankSet1.push_back (0.2);
+  docSet1.push_back (18); rankSet1.push_back (0.4);
+  docSet1.push_back (19); rankSet1.push_back (0.5);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  RankArray &rankSet2 = setNode2->queryResult.ranks;
+  docSet2.push_back (2);  rankSet2.push_back (0.1);
+  docSet2.push_back (11); rankSet2.push_back (0.2);
+  docSet2.push_back (12); rankSet2.push_back (0.3);
+  docSet2.push_back (13); rankSet2.push_back (0.4);
+  docSet2.push_back (14); rankSet2.push_back (0.5);
+  docSet2.push_back (15); rankSet2.push_back (0.6);
+  docSet2.push_back (16); rankSet2.push_back (0.7);
+  docSet2.push_back (17); rankSet2.push_back (0.8);
+  docSet2.push_back (19); rankSet2.push_back (0.9);
+  docSet2.push_back (20); rankSet2.push_back (0.1);
+  docSet2.push_back (21); rankSet2.push_back (0.2);
+
+  cout << "\n" << setNode1->queryResult << "OR\n\n"
+       << setNode2->queryResult;
+  
+
+  orNode.leftNode = setNode1;
+  orNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = true;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  orNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  RankArray &rcRankSet = resultCompare.ranks;
+  rcDocSet.push_back (1);  rcRankSet.push_back (0.1);
+  rcDocSet.push_back (2);  rcRankSet.push_back (0.1);
+  rcDocSet.push_back (10); rcRankSet.push_back (0.2);
+  rcDocSet.push_back (11); rcRankSet.push_back (0.2);
+  rcDocSet.push_back (12); rcRankSet.push_back (0.3);
+  rcDocSet.push_back (13); rcRankSet.push_back (0.4);
+  rcDocSet.push_back (14); rcRankSet.push_back (0.5);
+  rcDocSet.push_back (15); rcRankSet.push_back (0.2+0.6);
+  rcDocSet.push_back (16); rcRankSet.push_back (0.7);
+  rcDocSet.push_back (17); rcRankSet.push_back (0.8);
+  rcDocSet.push_back (18); rcRankSet.push_back (0.4);
+  rcDocSet.push_back (19); rcRankSet.push_back (0.9+0.5);
+  rcDocSet.push_back (20); rcRankSet.push_back (0.1);
+  rcDocSet.push_back (21); rcRankSet.push_back (0.2);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test2b () {
+  OrQueryNode orNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  docSet1.push_back (1); 
+  docSet1.push_back (10);
+  docSet1.push_back (15);
+  docSet1.push_back (18);
+  docSet1.push_back (19);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  docSet2.push_back (2);
+  docSet2.push_back (11);
+  docSet2.push_back (12);
+  docSet2.push_back (13);
+  docSet2.push_back (14);
+  docSet2.push_back (15);
+  docSet2.push_back (16);
+  docSet2.push_back (17);
+  docSet2.push_back (19);
+  docSet2.push_back (20);
+  docSet2.push_back (21);
+
+  cout << "\n" << setNode1->queryResult << "OR\n\n"
+       << setNode2->queryResult;
+  
+
+  orNode.leftNode = setNode1;
+  orNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = false;
+  queryInfo.needRankInfo = false;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  orNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  rcDocSet.push_back (1);
+  rcDocSet.push_back (2); 
+  rcDocSet.push_back (10);
+  rcDocSet.push_back (11);
+  rcDocSet.push_back (12);
+  rcDocSet.push_back (13);
+  rcDocSet.push_back (14);
+  rcDocSet.push_back (15);
+  rcDocSet.push_back (16);
+  rcDocSet.push_back (17);
+  rcDocSet.push_back (18);
+  rcDocSet.push_back (19);
+  rcDocSet.push_back (20);
+  rcDocSet.push_back (21);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test3 () {
+  NotQueryNode notNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  RankArray &rankSet1 = setNode1->queryResult.ranks;
+  docSet1.push_back (1);  rankSet1.push_back (0.1);
+  docSet1.push_back (10); rankSet1.push_back (0.2);
+  docSet1.push_back (15); rankSet1.push_back (0.2);
+  docSet1.push_back (18); rankSet1.push_back (0.4);
+  docSet1.push_back (19); rankSet1.push_back (0.5);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  RankArray &rankSet2 = setNode2->queryResult.ranks;
+  docSet2.push_back (2);  rankSet2.push_back (0.1);
+  docSet2.push_back (11); rankSet2.push_back (0.2);
+  docSet2.push_back (12); rankSet2.push_back (0.3);
+  docSet2.push_back (13); rankSet2.push_back (0.4);
+  docSet2.push_back (14); rankSet2.push_back (0.5);
+  docSet2.push_back (15); rankSet2.push_back (0.6);
+  docSet2.push_back (16); rankSet2.push_back (0.7);
+  docSet2.push_back (17); rankSet2.push_back (0.8);
+  docSet2.push_back (19); rankSet2.push_back (0.9);
+  docSet2.push_back (20); rankSet2.push_back (0.1);
+  docSet2.push_back (21); rankSet2.push_back (0.2);
+
+  cout << "\n" << setNode1->queryResult << "NOT\n\n"
+       << setNode2->queryResult;
+  
+
+  notNode.queryNode = setNode1;
+  notNode.notNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = true;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  notNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  RankArray &rcRankSet = resultCompare.ranks;
+  rcDocSet.push_back (1);  rcRankSet.push_back (0.1);
+  rcDocSet.push_back (10); rcRankSet.push_back (0.2);
+  rcDocSet.push_back (18); rcRankSet.push_back (0.4);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test3b () {
+  NotQueryNode notNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  docSet1.push_back (1); 
+  docSet1.push_back (10);
+  docSet1.push_back (15);
+  docSet1.push_back (18);
+  docSet1.push_back (19);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  docSet2.push_back (2); 
+  docSet2.push_back (11);
+  docSet2.push_back (12);
+  docSet2.push_back (13);
+  docSet2.push_back (14);
+  docSet2.push_back (15);
+  docSet2.push_back (16);
+  docSet2.push_back (17);
+  docSet2.push_back (19);
+  docSet2.push_back (20);
+  docSet2.push_back (21);
+
+  cout << "\n" << setNode1->queryResult << "NOT\n\n"
+       << setNode2->queryResult;
+  
+
+  notNode.queryNode = setNode1;
+  notNode.notNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = false;
+  queryInfo.needRankInfo = false;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  notNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  rcDocSet.push_back (1); 
+  rcDocSet.push_back (10);
+  rcDocSet.push_back (18);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+
+int main () {
+  if (!Test1 () ||
+      !Test2 () ||
+      !Test2b () ||
+      !Test3 () ||
+      !Test3b ()) return 1;
+  
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/Queryer.cpp
===================================================================
--- /trunk/indexers/mgpp/text/Queryer.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/Queryer.cpp	(revision 3365)
@@ -0,0 +1,270 @@
+/**************************************************************************
+ *
+ * Queryer.cpp -- simple interactive query program
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "MGQuery.h"
+#include "TextGet.h"
+
+#include "messages.h"
+#include "mg_files.h"
+
+#include "GSDLQueryParser.h"
+
+void printHelp() {
+
+  cout << "commands available are:\n"
+       << "\t.q\t\tquit\n"
+       << "\t.h\t\tprint the help message\n"
+       << "\t.i\t\tchange the search level (enter the new level at the prompt)\n"
+       << "\t.l\t\tchange the result level ( \"\"   \"\"  )\n"
+       << "\t.b\t\tfull text browse (enter a word or fragment at the prompt)\n"
+       << "\t.r0/.r1\t\tranking off/on\n"
+       << "\t.t0/.t1\t\tquery type some/all\n"
+       << "\t.c0/.c1\t\tcasefolding off/on\n"
+       << "\t.s0/.s1\t\tstemming off/on\n"
+       << "\t.o0/.o1\t\tshort output off/on\n\n"
+       << "\t.p\t\tprint a document (enter the docnum at the prompt)\n"
+       << "otherwise just enter a query\n\n";
+
+}
+
+int main (int argc, char **argv) {
+  int ch;
+  char *textfilename = "";
+  char *indexfilename = "";
+  char *basePath = "";
+  
+  opterr = 0;
+  msg_prefix = argv[0];
+  
+  // process the command line arguments
+  while ((ch = getopt (argc, argv, "f:t:d:h")) != -1) {
+    switch (ch) {
+    case 'f':		/* input file */
+      indexfilename = optarg;
+      break;
+    case 't':
+      textfilename = optarg;
+      break;
+    case 'd':
+      basePath = optarg;
+      set_basepath (basePath);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-d directory] -f indexname -t textname\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  if (textfilename[0] == '\0' || indexfilename[0] == '\0') {
+    FatalError (1, "Index and text file names must be specified with -f and -t \n");
+  }
+  
+  // init the text system
+    TextData textData;
+  if (!textData.LoadData (basePath, textfilename)) {
+    FatalError (1, "Couldn't load text information for \"%s\"", textfilename);
+  }
+  
+  // init the query system
+  IndexData indexData;
+  if (!indexData.LoadData (basePath, indexfilename)) {
+    FatalError (1, "Couldn't load index information for \"%s\"", indexfilename);
+  }
+
+  // debug output
+  cerr << "num docs: "<<indexData.bdh.num_docs       
+       << "\nnum frags: "<<indexData.bdh.num_frags
+       << "\nnum words: "<<indexData.bdh.num_words
+       << "\ntotal bytes: "<<indexData.bdh.total_bytes
+       << "\nindex string bytes: "<<indexData.bdh.index_string_bytes
+       << "\nnum levels: "<<indexData.bdh.num_levels<<endl;
+  
+  // do querying
+  QueryInfo queryInfo;
+  SetCStr (queryInfo.docLevel, "Document");
+  queryInfo.maxDocs = 50;
+  queryInfo.sortByRank = true;
+  queryInfo.exactWeights = false;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  ExtQueryResult queryResult;
+  char query[2048];
+  UCArray queryArray;
+  QueryNode *queryTree = NULL;
+
+
+  UCArray docLevel;
+  SetCStr(docLevel, "Document");
+
+  UCArray level;
+  UCArrayClear(level);
+  //SetCStr(level, "");
+   
+  int defaultStemMethod = 0; // uncasefolded, unstemmed
+  int defaultBoolCombine = 0; // OR
+  bool shortOutput = false;
+  BrowseQueryNode browseNode;
+  browseNode.startPosition = -10;
+  browseNode.numTerms = 40;
+
+  BrowseQueryResult browseResult;
+
+  while (true) {
+    cout << "> ";
+    cin.getline(query, 2048, '\n');
+    SetCStr (queryArray, query);
+
+    // check for commands
+    if (queryArray.size() >= 2 && queryArray[0] == '.') {
+      if (queryArray[1] == 'q') break;  // quit
+
+      if (queryArray[1] == 'h') { // help
+	printHelp();
+      }
+      if (queryArray[1] == 'i') {
+	cout << "current index="<< queryInfo.docLevel << "\nchange to index:";
+	cin >> query;
+	UCArrayClear(queryInfo.docLevel);
+	SetCStr(queryInfo.docLevel, query);
+	cout << "index set to " << queryInfo.docLevel <<"\n";
+	cin.getline(query, 2048, '\n');
+      }
+      if (queryArray[1] == 'l') {
+	cout << "current level="<< level << "\nchange to level:";
+	cin >> query;
+	UCArrayClear(level);
+	SetCStr(level, query);
+	cout << "level set to " << level <<"\n";
+	cin.getline(query, 2048, '\n');
+      }
+
+
+      else if (queryArray[1] == 'p') {
+	// print
+	UCArray docText;
+	unsigned long docNum = 0;
+	cin >> docNum;
+	cin.getline(query, 2048, '\n'); // eat up return
+	
+	if (!GetDocText (textData, queryInfo.docLevel, docNum, docText)) {
+	  FatalError (1, "Error while trying to get document %u", docNum);
+	}
+	
+	cout << docText << "\n";
+      }
+      else if (queryArray[1] == 't') { // query type - all/some
+	if (queryArray[2] == '1') defaultBoolCombine = 1;
+	else if (queryArray[2] == '0') defaultBoolCombine = 0;
+	else {
+	  cout << "Error: please enter .t0 (some) or .t1 (all)\n";
+	}
+      }
+      else if (queryArray[1] == 'r') { // ranking - on/off
+	if (queryArray[2] == '1') queryInfo.sortByRank = true;
+	else if (queryArray[2] == '0') queryInfo.sortByRank = false;
+	else {
+	  cout << "Error: please enter .r0 (non-ranked) or .r1 (ranked)\n";
+	}
+      }
+      else if (queryArray[1] == 'c') { // casefolding - on/off
+	if (queryArray[2] == '1') defaultStemMethod |= 1;
+	else if (queryArray[2] == '0') defaultStemMethod &= 0xe;
+	else {
+	  cout << "Error: please enter .c0 (case sensitive) or .c1 (casefolded)\n";
+	}
+      }
+      else if (queryArray[1] == 's') { // stemming - on/off
+	if (queryArray[2] == '1') defaultStemMethod |=2;
+	else if (queryArray[2] == '0') defaultStemMethod &=0xd;
+	else {
+	  cout << "Error: please enter .s0 (unstemmed) or .s1 (stemmed)\n";
+	}
+      }      
+      else if (queryArray[1] == 'o') { // output - short/long
+	if (queryArray[2] == '1') shortOutput = true;
+	else if (queryArray[2] == '0') shortOutput = false;
+	else {
+	  cout << "Error: please enter .o0 (long output) or .o1 (short output)\n";
+	}
+      }
+      else if (queryArray[1] == 'b') {
+	// full text browse
+	cout<<"enter a few letters to start browsing from:";
+	cin>>query;
+	UCArrayClear(browseNode.term);
+	SetCStr(browseNode.term, query);
+	cin.getline(query, 2048, '\n'); // get rest of line
+
+	// print the query
+	PrintNode (cout, &browseNode);
+
+	MGBrowseQuery(indexData, docLevel, browseNode, browseResult);
+	cout << browseResult;
+	cout << "\n";
+
+      }
+      else { // bad option
+	cout << "bad command\n\n";
+	printHelp();
+      }
+    } // if a .x query
+    else {
+      // regular query
+      queryTree = ParseQuery (queryArray, defaultBoolCombine, defaultStemMethod);
+
+      // print the query
+      PrintNode (cout, queryTree);
+
+      MGQuery (indexData, queryInfo, queryTree, queryResult, level);
+      if (shortOutput) {
+	queryResult.printShort(cout);
+	cout << "\n";
+      }else {
+	cout << queryResult;
+	cout << "\n";
+      }
+      // delete the query
+      if (queryTree != NULL) delete queryTree;
+      queryTree = NULL;
+    }
+  }
+
+
+  // clean up, everybody clean up
+    textData.UnloadData ();
+  indexData.UnloadData ();
+
+  return (0);
+}
+
+
Index: /trunk/indexers/mgpp/text/TagInfo.cpp
===================================================================
--- /trunk/indexers/mgpp/text/TagInfo.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/TagInfo.cpp	(revision 3365)
@@ -0,0 +1,74 @@
+/**************************************************************************
+ *
+ * TagInfo.cpp -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "TagInfo.h"
+
+
+void TagInfo::SetDocTag (const char *cStr) {
+  SetCStr (docTag, cStr);
+}
+
+void TagInfo::SetIndexLevel (const char *cStr) {
+  SetCStr (indexLevel, cStr);
+}
+void TagInfo::AddLevelTag (const char *cStr) {
+  // convert the string
+  UCArray cArr;
+  SetCStr (cArr, cStr);
+
+  // insert the tag
+  levelTags.insert (cArr);
+}
+
+
+TagInfo::TagInfo () {
+  Clear ();
+}
+
+void TagInfo::Clear () {
+  docTag.erase (docTag.begin(), docTag.end());
+  indexLevel.erase (indexLevel.begin(), indexLevel.end());
+  levelTags.erase (levelTags.begin(), levelTags.end());
+}
+
+ostream &operator<<(ostream &s, const TagInfo &t) {
+  // output docTag
+  s << "docTag: \"" << t.docTag << "\"\n";
+
+  // output indexLevel
+  s << "indexLevel: \"" << t.indexLevel << "\"\n";
+
+  // output levelTags
+  s << "levelTags: ";
+
+  UCArraySet::const_iterator here = t.levelTags.begin();
+  UCArraySet::const_iterator end = t.levelTags.end();
+  while (here != end) {
+    s << "\"" << (*here) << "\"";
+    here++;
+    if (here != end) s << ", ";
+  }
+
+  s << "\n";
+  
+  return s;
+}
+
Index: /trunk/indexers/mgpp/text/TagInfo.h
===================================================================
--- /trunk/indexers/mgpp/text/TagInfo.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/TagInfo.h	(revision 3365)
@@ -0,0 +1,46 @@
+/**************************************************************************
+ *
+ * TagInfo.h -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TAGINFO_H
+#define TAGINFO_H
+
+#include "UCArray.h"
+
+
+class TagInfo {
+public:
+  UCArray docTag;
+  UCArray indexLevel;    // empty for word level index
+  UCArraySet levelTags;
+
+  void SetDocTag (const char *cStr);
+  void SetIndexLevel (const char *cStr);
+  void AddLevelTag (const char *cStr);
+  
+  TagInfo ();
+  void Clear ();
+};
+
+// stream operator to print TagInfo
+ostream &operator<<(ostream &s, const TagInfo &t);
+
+
+#endif
Index: /trunk/indexers/mgpp/text/Terms.cpp
===================================================================
--- /trunk/indexers/mgpp/text/Terms.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/Terms.cpp	(revision 3365)
@@ -0,0 +1,797 @@
+/**************************************************************************
+ *
+ * Terms.cpp -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "Terms.h"
+#include "words.h"
+#include "stemmer.h"
+#include "bitio_gen.h"
+#include "bitio_m_stdio.h"
+
+void QueryInfo::Clear () {
+  UCArrayClear (docLevel);
+  maxDocs = 0;
+  sortByRank = true;
+  exactWeights = false;
+  needRankInfo = false;
+  needTermFreqs = false;
+}
+
+
+
+void TermFreqData::Clear () {
+  UCArrayClear (tag);
+  UCArrayClear (term);
+  equivTerms.erase(equivTerms.begin(), equivTerms.end());
+  stemMethod = 0;
+  matchDocs = 0;
+  termFreq = 0;
+}
+
+ostream &operator<< (ostream &s, const TermFreqData &t) {
+  s << "<" << t.tag << ">\"" << t.term << "\"stem("
+    << t.stemMethod << ")equiv terms(";
+  
+  unsigned long i;
+  for (i=0; i<t.equivTerms.size(); i++) {
+    s << t.equivTerms[i] << ", ";
+  }
+  s <<")docs(" << t.matchDocs << ")"
+    << "count("<<t.termFreq<<")";
+  return s;
+}
+
+bool operator== (const TermFreqData &t1, const TermFreqData &t2) {
+  return ((t1.tag == t2.tag) &&
+	  (t1.term == t2.term) &&
+	  (t1.stemMethod == t2.stemMethod) &&
+	  (t1.equivTerms == t2.equivTerms) &&
+	  (t1.matchDocs == t2.matchDocs) &&
+	  (t1.termFreq == t2.termFreq));
+}
+
+
+void QueryResult::Clear () {
+  docs.erase (docs.begin(), docs.end());
+  ranks.erase (ranks.begin(), ranks.end());
+  termFreqs.erase (termFreqs.begin(), termFreqs.end());
+  actualNumDocs = 0;
+}
+
+QueryResult::QueryResult () {
+  Clear ();
+}
+
+void QueryResult::printShort(ostream &s) {
+
+  s << "termFreqs: ";
+  for (unsigned long i=0; i<termFreqs.size(); i++)
+    s << termFreqs[i] << ", ";
+  
+  s << "\nactual number of docs found: " << actualNumDocs;
+  s << "\n\n";
+
+}
+
+
+ostream &operator<< (ostream &s, const QueryResult &r) {
+  s << "docs: ";
+  unsigned long i;
+  for (i=0; i<r.docs.size(); i++)
+    s << r.docs[i] << ", ";
+  
+  s << "\nranks: ";
+  for (i=0; i<r.ranks.size(); i++)
+    s << r.ranks[i] << ", ";
+
+  s << "\ntermFreqs: ";
+  for (i=0; i<r.termFreqs.size(); i++)
+    s << r.termFreqs[i] << ", ";
+
+  s << "\nactual number of docs found: " << r.actualNumDocs;
+  s << "\n\n";
+
+  return s;
+}
+
+
+bool operator== (const QueryResult &r1, const QueryResult &r2) {
+  return ((r1.docs == r2.docs) &&
+	  (r1.ranks == r2.ranks) &&
+	  (r1.termFreqs == r2.termFreqs) &&
+	  (r1.actualNumDocs == r2.actualNumDocs));
+}
+
+//---------------------------------------------------
+// new ExtQueryResult stuff
+void ExtQueryResult::Clear () {
+  docs.erase (docs.begin(), docs.end());
+  levels.erase (levels.begin(), levels.end());
+  ranks.erase (ranks.begin(), ranks.end());
+  termFreqs.erase (termFreqs.begin(), termFreqs.end());
+  actualNumDocs = 0;
+}
+
+ExtQueryResult::ExtQueryResult () {
+  Clear ();
+}
+
+ostream &operator<< (ostream &s, const ExtQueryResult &r) {
+  s << "docs: ";
+  unsigned long i;
+  for (i=0; i<r.docs.size(); i++)
+    s << r.docs[i] << ", ";
+
+  s << "\nlevels: ";
+  for (i=0; i<r.levels.size(); i++)
+    s << r.levels[i] << ", ";
+
+  
+  s << "\nranks: ";
+  for (i=0; i<r.ranks.size(); i++)
+    s << r.ranks[i] << ", ";
+
+  s << "\ntermFreqs: ";
+  for (i=0; i<r.termFreqs.size(); i++)
+    s << r.termFreqs[i] << ", ";
+  s << "\nactual number of docs found: " << r.actualNumDocs;
+  s << "\n\n";
+
+  return s;
+}
+
+
+bool operator== (const ExtQueryResult &r1, const ExtQueryResult &r2) {
+  return ((r1.docs == r2.docs) &&
+	  (r1.levels == r2.levels) &&
+	  (r1.ranks == r2.ranks) &&
+	  (r1.termFreqs == r2.termFreqs) &&
+	  (r1.actualNumDocs == r2.actualNumDocs));
+}
+
+//-------------------------------------------------------
+// new BrowseQueryResult stuff
+void BrowseQueryResult::Clear () {
+  termFreqs.erase (termFreqs.begin(), termFreqs.end());
+}
+
+BrowseQueryResult::BrowseQueryResult () {
+  Clear ();
+}
+
+
+
+ostream &operator<< (ostream &s, const BrowseQueryResult &r) {
+  s << "terms: ";
+  unsigned long i;
+  for (i=0; i<r.termFreqs.size(); i++)
+    s << r.termFreqs[i] << ", ";
+    s << "\n\n";
+  return s;
+}
+
+
+bool operator== (const BrowseQueryResult &r1, const BrowseQueryResult &r2) {
+  return ((r1.termFreqs == r2.termFreqs));
+	 
+}
+
+
+
+
+//--------------------------------------
+void FragData::Clear () {
+  matchDocs = 0;
+  fragNums.erase (fragNums.begin(), fragNums.end());
+  fragFreqs.erase (fragFreqs.begin(), fragFreqs.end());
+}
+
+
+
+
+void FindWordNumbers (IndexData &indexData,
+		      const UCArray &term,
+		      unsigned long stemMethod,
+		      vector<unsigned long> &equivWords) {
+  equivWords.erase (equivWords.begin(), equivWords.end());
+  
+  if (stemMethod == 0) {
+    // don't need to stem the word,
+    // find the word number for this term
+    unsigned long wordElNum = 0;
+    unsigned long numLevels = indexData.bdh.num_levels;
+    word_block_dict_el wordDictEl;
+    wordDictEl.SetNumLevels (numLevels);
+    if (SearchWordBlockDictEl (indexData.dictFile, indexData.biWords,
+			       indexData.bdh.entries_per_wblk,
+			       indexData.bdh.word_dict_size,
+			       numLevels, term, wordDictEl, wordElNum))
+      equivWords.push_back (wordElNum);
+    
+    return;
+    
+  }
+
+
+  // need to stem this word and find it in the blocked stem index
+  
+  unsigned char  mgWord[MAXSTEMLEN + 1];
+  UCArray stemTerm;
+  unsigned long stemmerNum = 0;
+  
+  if (stemMethod == 1) stemmerNum = indexData.sih1.stemmer_num;
+  else if (stemMethod == 2) stemmerNum = indexData.sih2.stemmer_num;
+  else if (stemMethod == 3) stemmerNum = indexData.sih3.stemmer_num;
+  
+  
+  // convert the word to an "mg word"
+  mgWord[0] = term.size();
+  memcpy ((char *)&mgWord[1], &(term[0]), term.size());
+  
+  // stem the word
+  stemmer (stemMethod, stemmerNum, mgWord);
+
+  // convert the result back to a UCArray
+  stemTerm.insert (stemTerm.end(), &mgWord[1], &mgWord[1] + mgWord[0]);
+
+  // need to look up this term in the appropriate dictionary
+  stem_block_dict_el stemDictEl;
+  unsigned long stemElNum;
+  bool result = false;
+  if (stemMethod == 1) {
+    result = SearchStemBlockDictEl (indexData.stem1File,
+			   indexData.sii1,
+			   indexData.sih1.entries_per_block,
+			   indexData.sih1.dict_size,
+			   stemTerm,
+			   stemDictEl,
+			   stemElNum);
+
+  } else if (stemMethod == 2) {
+    result = SearchStemBlockDictEl (indexData.stem2File,
+			   indexData.sii2,
+			   indexData.sih2.entries_per_block,
+			   indexData.sih2.dict_size,
+			   stemTerm,
+			   stemDictEl,
+			   stemElNum);
+
+  } else if (stemMethod == 3) {
+    result = SearchStemBlockDictEl (indexData.stem3File,
+			   indexData.sii3,
+			   indexData.sih3.entries_per_block,
+			   indexData.sih3.dict_size,
+			   stemTerm,
+			   stemDictEl,
+			   stemElNum);
+  }
+
+  if (result) {
+    equivWords = stemDictEl.equivWords;  
+  }
+}
+
+
+
+void ReadTermFragData (IndexData &indexData,
+		       bool needFragFreqs,
+		       unsigned long termNum,
+		       FragData &fragData,
+		       FragRangeArray *fragLimits,
+		       UCArray & termWord) {
+  fragData.Clear();
+
+  // look up the word in the dictionary
+  unsigned long numLevels = indexData.bdh.num_levels;
+  word_block_dict_el wordDictEl;
+  wordDictEl.SetNumLevels (numLevels);
+  if (!SearchWordBlockDictElNum (indexData.dictFile,
+				 indexData.biWords,
+				 indexData.bdh.entries_per_wblk,
+				 indexData.bdh.word_dict_size,
+				 numLevels,
+				 termNum, wordDictEl))
+    return; // nothing more to do
+
+  fragData.matchDocs = wordDictEl.levelFreqs[indexData.curLevelNum];
+  termWord = wordDictEl.el;
+  // seek to the appropriate place in the inverted file
+  fseek (indexData.invfFile, wordDictEl.invf_ptr, SEEK_SET);
+  stdio_bitio_buffer buffer (indexData.invfFile);
+    
+  unsigned long B = BIO_Bblock_Init (indexData.bdh.num_frags,
+				     wordDictEl.frag_occur);
+  unsigned long fragNum = 0;
+  unsigned long termFreq = 0;
+
+  unsigned long fragLimitI = 0;
+  unsigned long i;
+  for (i=0; i<wordDictEl.frag_occur; i++) {
+    fragNum += buffer.bblock_decode (B, NULL);
+    if (!indexData.ifh.word_level_index) termFreq = buffer.gamma_decode (NULL);
+    else termFreq = 1;
+
+    // get the right fragment range
+    if (fragLimits != NULL) {
+      while (fragLimitI+1 < (*fragLimits).size() &&
+	     fragNum > (*fragLimits)[fragLimitI+1].rangeStart) {
+	fragLimitI++;
+      }
+    }
+
+    // add the entry if it is within the limits
+    if ((fragLimits == NULL) ||
+	(fragLimitI < (*fragLimits).size() &&
+	 fragNum > (*fragLimits)[fragLimitI].rangeStart &&
+	 fragNum <= (*fragLimits)[fragLimitI].rangeEnd)) {
+      fragData.fragNums.push_back (fragNum);
+      if (needFragFreqs)
+	fragData.fragFreqs.push_back (termFreq);
+    }
+  }
+
+  buffer.done();
+}
+
+
+void CombineFragData (bool needFragFreqs,
+		      const FragData &f1,
+		      const FragData &f2,
+		      FragData &outFragData) {
+  outFragData.Clear();
+
+  // the new number of matching documents is the maximum
+  // of the two input matching number of documents -- it
+  // is assumed that these are at the same document level
+  outFragData.matchDocs = (f1.matchDocs > f2.matchDocs) ?
+    f1.matchDocs : f2.matchDocs;
+
+  // do or
+  unsigned long f1I = 0, f1Size = f1.fragNums.size();
+  unsigned long f2I = 0, f2Size = f2.fragNums.size();
+  while (f1I < f1Size || f2I < f2Size) {
+    if (f2I < f2Size &&
+	(f1I >= f1Size ||
+	 f1.fragNums[f1I] > f2.fragNums[f2I])) {
+      // output f2I
+      outFragData.fragNums.push_back (f2.fragNums[f2I]);
+      if (needFragFreqs)
+	outFragData.fragFreqs.push_back (f2.fragFreqs[f2I]);
+      f2I++;
+      
+    } else if (f1I < f1Size &&
+	       (f2I >= f2Size ||
+		f1.fragNums[f1I] < f2.fragNums[f2I])) {
+      // output f1I
+      outFragData.fragNums.push_back (f1.fragNums[f1I]);
+      if (needFragFreqs)
+	outFragData.fragFreqs.push_back (f1.fragFreqs[f1I]);
+      f1I++;
+      
+    } else {
+      // must be equal combine f1I and f2I
+      outFragData.fragNums.push_back (f1.fragNums[f1I]);
+      if (needFragFreqs)
+	outFragData.fragFreqs.push_back (f1.fragFreqs[f1I]+f2.fragFreqs[f2I]);
+      f1I++;
+      f2I++;
+    }
+  }
+}
+
+
+void AndCombineFragData (bool needFragFreqs,
+			 FragData &fragData,
+			 const FragData &comFragData,
+			 signed long startRange,
+			 signed long endRange,
+			 const FragRangeArray *fragLimits) {
+  // sanity check on range
+  if (startRange > endRange) {
+    signed long temp = endRange;
+    endRange = startRange;
+    startRange = temp;
+  }
+
+  // get min matchdocs
+  if (comFragData.matchDocs < fragData.matchDocs)
+    fragData.matchDocs = comFragData.matchDocs;
+  
+  unsigned long fragDataI = 0;
+  unsigned long fragDataSize = fragData.fragNums.size();
+  unsigned long comFragDataI = 0;
+  unsigned long comFragDataSize = comFragData.fragNums.size();
+  unsigned long fragLimitI = 0;
+  unsigned long fragLimitSize = (fragLimits==NULL) ? 0 : (*fragLimits).size();
+  unsigned long outI = 0;
+
+  while (fragDataI < fragDataSize &&
+	 comFragDataI < comFragDataSize) {
+    signed long fragNum = (signed long)fragData.fragNums[fragDataI];
+    signed long comFragNum = (signed long)comFragData.fragNums[comFragDataI];
+    
+    // go to the right fragment limit (for the com frag)
+    if (fragLimits != NULL) {
+      while (fragLimitI+1 < fragLimitSize &&
+	     comFragNum > (signed long)(*fragLimits)[fragLimitI+1].rangeStart) {
+	fragLimitI++;
+      }
+    }
+
+    if (fragNum <= comFragNum+startRange ||
+	(fragLimits!=NULL &&
+	 fragNum<=(signed long)(*fragLimits)[fragLimitI].rangeStart)) {
+      fragDataI++;
+      
+    } else if (fragNum > comFragNum+endRange ||
+	       (fragLimits!=NULL &&
+		fragNum>(signed long)(*fragLimits)[fragLimitI].rangeEnd)) {
+      comFragDataI++;
+      
+    } else {
+      // equal and within tag
+      fragData.fragNums[outI] = comFragNum;
+      if (needFragFreqs) {
+	fragData.fragFreqs[outI] =
+	  (fragData.fragFreqs[fragDataI] < comFragData.fragFreqs[comFragDataI]) ?
+	  fragData.fragFreqs[fragDataI] : comFragData.fragFreqs[comFragDataI];
+      }
+      fragDataI++;
+      comFragDataI++;
+      outI++;
+    }
+  }
+
+  // erase unused part of fragData
+  fragData.fragNums.erase (fragData.fragNums.begin()+outI,
+			   fragData.fragNums.end());
+  if (needFragFreqs)
+    fragData.fragFreqs.erase (fragData.fragFreqs.begin()+outI,
+			      fragData.fragFreqs.end());
+  else
+    fragData.fragFreqs.erase (fragData.fragFreqs.begin(),
+			      fragData.fragFreqs.end());
+}
+
+
+void FragsToQueryResult (IndexData &indexData,
+			 const QueryInfo &queryInfo,
+			 const FragData &termData,
+			 const UCArray &tag,
+			 const UCArray &term,
+			 unsigned long stemMethod,
+			 unsigned long termWeight,
+			 UCArrayVector &equivTerms,
+			 QueryResult &result) {
+  bool needRanks = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  
+  result.Clear();
+
+  // log (N / ft)
+  unsigned long N = indexData.levels.levelInfo[indexData.curLevel].numEntries;
+  float wordLog = log((double)N / (double)termData.matchDocs);
+
+  // Wqt = fqt * log (N / ft)
+  // note: terms are allowed to have a weight of zero so
+  // they can be excluded from the ranking
+  float Wqt = termWeight * wordLog;
+
+  // Wdt = fdt * log (N / ft)
+  float Wdt;
+  
+  unsigned long termDataI = 0;
+  unsigned long termDataSize = termData.fragNums.size();
+  unsigned long levelDocNum = 0;
+  
+  unsigned long termDocFreq = 0;
+  unsigned long lastLevelDocNum = 0;
+  unsigned long overallwordfreq = 0;
+  
+  while (termDataI < termDataSize) {
+    if (indexData.levelConverter.FragToLevel (termData.fragNums[termDataI],
+					      levelDocNum)) {
+      if (levelDocNum != lastLevelDocNum) {
+	if (lastLevelDocNum > 0) {
+	  // add this doc information
+	  if (needRanks) {
+	    Wdt = termDocFreq * wordLog;
+	    result.ranks.push_back (Wqt * Wdt);
+	  }
+	  result.docs.push_back (lastLevelDocNum);
+	}
+	
+	lastLevelDocNum = levelDocNum;
+	termDocFreq = 0;
+      }
+
+      if (needRanks){
+	termDocFreq += termData.fragFreqs[termDataI];
+	overallwordfreq += termData.fragFreqs[termDataI];
+      }
+    }
+    termDataI++;
+  }
+
+  if (lastLevelDocNum > 0) {
+    // add the last document information
+    if (needRanks) {
+      Wdt = termDocFreq * wordLog;
+      result.ranks.push_back (Wqt * Wdt);
+    }
+    result.docs.push_back (lastLevelDocNum);
+  }
+
+  // add the term frequency information
+  if (queryInfo.needTermFreqs) {
+    TermFreqData termFreqData;
+    termFreqData.tag = tag;
+    termFreqData.term = term;
+    termFreqData.stemMethod = stemMethod;
+    termFreqData.equivTerms = equivTerms;
+    termFreqData.matchDocs = termData.matchDocs;
+    termFreqData.termFreq = overallwordfreq; // will be zero if needRankInfo 
+                                              //not true
+    result.termFreqs.push_back (termFreqData);
+  }
+}
+
+void AndFragsToQueryResult (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    const UCArray &tag,
+			    const UCArray &term,
+			    unsigned long stemMethod,
+			    unsigned long termWeight,
+			    UCArrayVector &equivTerms,
+			    QueryResult &result) {
+  bool needRanks = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  
+  // log (N / ft)
+  float wordLog =
+    log((double)indexData.levels.levelInfo[indexData.curLevel].numEntries/
+	(double)termData.matchDocs);
+
+  // Wqt = fqt * log (N / ft)
+  // note: terms are allowed to have a weight of zero so
+  // they can be excluded from the ranking
+  float Wqt = termWeight * wordLog;
+
+  // Wdt = fdt * log (N / ft)
+  float Wdt;
+  
+  unsigned long termDataI = 0;
+  unsigned long termDataSize = termData.fragNums.size();
+  unsigned long levelDocNum = 0;
+  
+  unsigned long termDocFreq = 0;
+  unsigned long lastLevelDocNum = 0;
+  unsigned long overallwordfreq = 0;
+  unsigned long resultI = 0;
+  unsigned long resultSize = result.docs.size();
+  unsigned long resultOutI = 0;
+  
+  
+  while (termDataI < termDataSize) {
+    if (indexData.levelConverter.FragToLevel (termData.fragNums[termDataI],
+					      levelDocNum)) {
+      if (levelDocNum != lastLevelDocNum) {
+	if (lastLevelDocNum > 0) {
+	  // add this doc information
+	  Wdt = termDocFreq * wordLog;
+	  
+	  // find this document number
+	  while (resultI < resultSize &&
+		 result.docs[resultI] < lastLevelDocNum)
+	    resultI++;
+	  
+	  // store the result
+	  if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+	    result.docs[resultOutI] = lastLevelDocNum;
+	    if (needRanks)
+	      result.ranks[resultOutI] = result.ranks[resultI] + Wqt * Wdt;
+	    resultI++;
+	    resultOutI++;
+	  }
+	}
+	
+	lastLevelDocNum = levelDocNum;
+	termDocFreq = 0;
+      }
+
+      if (needRanks)
+	termDocFreq += termData.fragFreqs[termDataI];
+     overallwordfreq += termData.fragFreqs[termDataI]; 
+    }
+    
+    termDataI++;
+  } // while
+
+  if (lastLevelDocNum > 0) {
+    // add the last document information
+    Wdt = termDocFreq * wordLog;
+
+    // find this document number
+    while (resultI < resultSize &&
+	   result.docs[resultI] < lastLevelDocNum)
+      resultI++;
+    
+    // store the result
+    if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+      result.docs[resultOutI] = lastLevelDocNum;
+      if (needRanks)
+	result.ranks[resultOutI] = result.ranks[resultI] + Wqt * Wdt;
+      resultI++;
+      resultOutI++;
+    }
+  }
+
+  // remove unneeded entries
+  result.docs.erase (result.docs.begin()+resultOutI, result.docs.end());
+  if (needRanks)
+    result.ranks.erase (result.ranks.begin()+resultOutI, result.ranks.end());
+  else
+    result.ranks.erase (result.ranks.begin(), result.ranks.end());
+  
+  // add the term frequency information
+  if (queryInfo.needTermFreqs) {
+    TermFreqData termFreqData;
+    termFreqData.tag = tag;
+    termFreqData.term = term;
+    termFreqData.stemMethod = stemMethod;
+    termFreqData.equivTerms = equivTerms;
+    termFreqData.matchDocs = termData.matchDocs;
+    termFreqData.termFreq = overallwordfreq;
+    result.termFreqs.push_back (termFreqData);
+  }
+}
+
+
+void RemoveUnwantedResults (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    QueryResult &result) {
+  bool needRanks = (queryInfo.sortByRank || queryInfo.needRankInfo);
+
+  unsigned long termDataI = 0;
+  unsigned long termDataSize = termData.fragNums.size();
+  unsigned long levelDocNum = 0;
+  
+  unsigned long lastLevelDocNum = 0;
+
+  unsigned long resultI = 0;
+  unsigned long resultSize = result.docs.size();
+  unsigned long resultOutI = 0;
+  
+  while (termDataI < termDataSize) {
+    if (indexData.levelConverter.FragToLevel (termData.fragNums[termDataI],
+					      levelDocNum)) {
+      if (levelDocNum != lastLevelDocNum) {
+	if (lastLevelDocNum > 0) {
+	  // find this document number
+	  while (resultI < resultSize &&
+		 result.docs[resultI] < lastLevelDocNum)
+	    resultI++;
+	  
+	  // store the result
+	  if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+	    result.docs[resultOutI] = lastLevelDocNum;
+	    if (needRanks)
+	      result.ranks[resultOutI] = result.ranks[resultI];
+	    resultI++;
+	  resultOutI++;
+	  }
+	}
+	
+	lastLevelDocNum = levelDocNum;
+      }
+    }
+    
+    termDataI++;
+  }
+
+  if (lastLevelDocNum > 0) {
+    // find this document number
+    while (resultI < resultSize &&
+	   result.docs[resultI] < lastLevelDocNum)
+      resultI++;
+    
+    // store the result
+    if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+      result.docs[resultOutI] = lastLevelDocNum;
+      if (needRanks) 
+	result.ranks[resultOutI] = result.ranks[resultI];
+      resultI++;
+      resultOutI++;
+    }
+  }
+
+  // remove unneeded entries
+  result.docs.erase (result.docs.begin()+resultOutI, result.docs.end());
+  if (needRanks)
+    result.ranks.erase (result.ranks.begin()+resultOutI, result.ranks.end());
+  else
+    result.ranks.erase (result.ranks.begin(), result.ranks.end());
+}
+
+
+
+//--------------------------------------------------------------
+// functions to support full text browse
+
+void FindNearestWordNumber (IndexData &indexData,
+			    const UCArray &term,
+			    unsigned long &number) {
+
+    // find the word number for this term
+    unsigned long wordElNum = 0;
+    unsigned long numLevels = indexData.bdh.num_levels;
+    word_block_dict_el wordDictEl;
+    wordDictEl.SetNumLevels (numLevels);
+    if (NearestSearchWordBlockDictEl (indexData.dictFile, indexData.biWords,
+				      indexData.bdh.entries_per_wblk,
+				      indexData.bdh.word_dict_size,
+				      numLevels, term, wordDictEl, wordElNum))
+      number = wordElNum;
+
+}
+
+void GetTermList(IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 TermFreqArray &terms) {
+
+  word_block_dict_el_array wordBlocks; // = new word_block_dict_el_array();
+  TermFreqData termdata;
+
+  terms.erase(terms.begin(), terms.end());
+
+  SearchWordBlockDictElNumRange (indexData.dictFile, indexData.biWords,
+				 indexData.bdh.entries_per_wblk,
+				 indexData.bdh.word_dict_size,
+				 indexData.bdh.num_levels, startTerm,
+				 numTerms, wordBlocks);
+
+  word_block_dict_el_array::iterator here = wordBlocks.begin();
+  word_block_dict_el_array::iterator end = wordBlocks.end();
+
+  while (here != end) {
+    termdata.Clear();
+    termdata.term = (*here).el;
+    termdata.termFreq = (*here).freq;
+    terms.push_back(termdata);
+    here++;
+  }
+
+}
+
+void GetTermList(IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 UCArrayVector &terms) {
+
+  
+  
+  SearchWordBlockDictElNumRange (indexData.dictFile, indexData.biWords,
+				 indexData.bdh.entries_per_wblk,
+				 indexData.bdh.word_dict_size,
+				 indexData.bdh.num_levels, startTerm,
+				 numTerms, terms);
+
+}
Index: /trunk/indexers/mgpp/text/Terms.h
===================================================================
--- /trunk/indexers/mgpp/text/Terms.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/Terms.h	(revision 3365)
@@ -0,0 +1,215 @@
+/**************************************************************************
+ *
+ * Terms.h -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TERMS_H
+#define TERMS_H
+
+#include "IndexData.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\iostream>
+#elif defined(GSDL_USE_IOS_H)
+#  include <iostream.h>
+#else
+#  include <iostream>
+#endif
+
+
+class QueryInfo {
+public:
+  // general query information
+  UCArray docLevel;
+  unsigned long maxDocs; // 0 = all
+  bool sortByRank;
+  bool exactWeights;
+
+  // information need to return
+  bool needRankInfo;
+  bool needTermFreqs;
+
+  void Clear ();
+  QueryInfo () { Clear (); }
+};
+
+
+class TermFreqData {
+public:
+  UCArray tag;  // level tag or query tag
+  UCArray term; // unstemmed term
+  int stemMethod;
+  UCArrayVector equivTerms; // the stemmed and casefolded variants of the term
+  unsigned long matchDocs; // tf for level - num levels
+			   // containing this term
+  unsigned long termFreq;  // overall term freq - num words that
+				// are this term
+  void Clear ();
+  TermFreqData () { Clear (); }
+};
+
+ostream &operator<< (ostream &s, const TermFreqData &t);
+bool operator== (const TermFreqData &t1, const TermFreqData &t2);
+
+typedef vector<TermFreqData> TermFreqArray;
+
+
+typedef vector<float> RankArray;
+typedef vector<unsigned long> DocNumArray;
+
+class QueryResult {
+public:
+  DocNumArray docs;
+  RankArray ranks; // used for accumulators during query
+  
+  TermFreqArray termFreqs;
+
+  unsigned long actualNumDocs;
+  void Clear ();
+  QueryResult ();
+  void printShort(ostream &s);
+};
+
+ostream &operator<< (ostream &s, const QueryResult &r);
+bool operator== (const QueryResult &r1, const QueryResult &r2);
+
+typedef vector<unsigned long> FragNumArray;
+typedef vector<unsigned long> FragFreqArray;
+
+class FragData {
+public:
+  unsigned long matchDocs;    // ft for level
+  FragNumArray fragNums;
+  FragFreqArray fragFreqs;
+
+  void Clear ();
+  FragData () { Clear (); }
+};
+
+
+class FragRange {
+public:
+  unsigned long rangeStart;
+  unsigned long rangeEnd;
+
+  void Clear () { rangeStart = rangeEnd = 0; }
+  FragRange () { Clear (); }
+};
+
+typedef vector<FragRange> FragRangeArray;
+
+
+
+void FindWordNumbers (IndexData &indexData,
+		      const UCArray &term,
+		      unsigned long stemMethod,
+		      vector<unsigned long> &equivWords);
+
+void ReadTermFragData (IndexData &indexData,
+		       bool needFragFreqs,
+		       unsigned long termNum,
+		       FragData &fragData,
+		       FragRangeArray *fragLimits,
+		       UCArray &termWord);
+
+void CombineFragData (bool needFragFreqs,
+		      const FragData &f1,
+		      const FragData &f2,
+		      FragData &outFragData);
+
+// output will be in fragData (as this is an and operation)
+void AndCombineFragData (bool needFragFreqs,
+			 FragData &fragData,
+			 const FragData &comFragData,
+			 signed long startRange,
+			 signed long endRange,
+			 const FragRangeArray *fragLimits);
+
+void FragsToQueryResult (IndexData &indexData,
+			 const QueryInfo &queryInfo,
+			 const FragData &termData,
+			 const UCArray &tag,
+			 const UCArray &term,
+			 unsigned long stemMethod,
+			 unsigned long termWeight,
+			 UCArrayVector &equivTerms,
+			 QueryResult &result);
+
+void AndFragsToQueryResult (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    const UCArray &tag,
+			    const UCArray &term,
+			    unsigned long stemMethod,
+			    unsigned long termWeight,
+			    UCArrayVector &equivTerms,
+			    QueryResult &result);
+
+void RemoveUnwantedResults (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    QueryResult &result);
+
+//-----------------------------------------------------------------
+// new QueryResult class to handle retrieval of doc and level nums.
+// Use this class with extended version of MGQuery
+
+class ExtQueryResult : public QueryResult {
+public:
+  DocNumArray levels; // used for returning a different granularity, eg
+  // search sections but return Document numbers, or search Documents, 
+  // return Section numbers.
+ 
+  void Clear ();
+  ExtQueryResult ();
+};
+
+ostream &operator<< (ostream &s, const ExtQueryResult &r);
+bool operator== (const ExtQueryResult &r1, const ExtQueryResult &r2);
+
+//------------------------------------------------------------
+// new functions to handle full text browse
+
+class BrowseQueryResult {
+ public:
+  TermFreqArray termFreqs;
+  void Clear();
+  BrowseQueryResult ();
+
+};
+
+
+ostream &operator<< (ostream &s, const BrowseQueryResult &r);
+bool operator== (const BrowseQueryResult &r1, const BrowseQueryResult &r2);
+
+void FindNearestWordNumber (IndexData &indexData,
+			    const UCArray &term,
+			    unsigned long &number);
+
+void GetTermList(IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 TermFreqArray &terms);
+
+void GetTermList (IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 UCArrayVector &terms);
+
+#endif
Index: /trunk/indexers/mgpp/text/TextEl.cpp
===================================================================
--- /trunk/indexers/mgpp/text/TextEl.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/TextEl.cpp	(revision 3365)
@@ -0,0 +1,211 @@
+/**************************************************************************
+ *
+ * TextEl.cpp -- Data structures for parsed documents
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "TextEl.h"
+#include "ctype.h"
+
+#define REC_TERM   '\002'
+#define PARA_TERM  '\003'
+
+
+void TextEl::Clear () {
+  elType = TextE;
+  tagName.erase (tagName.begin(), tagName.end());
+  text.erase (text.begin(), text.end());
+}
+
+
+static bool ReadTextTag (FileBuf &buf, TextEl &el) {
+  // we must have some sort of tag
+  unsigned char c;
+  if (!buf.Peek(c)) return false;
+  
+  // get the '<'
+  if (c != '<') return false;
+  el.text.push_back (c);
+  if (!buf.Next() || !buf.Peek (c)) return false;
+  
+  // see if we have a '/'
+  if (c == '/') {
+    // we have a closing tag
+    el.text.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) return false;
+    el.elType = CloseTagE;
+  } else {
+    // we have an opening tag
+    el.elType = OpenTagE;
+  }
+
+  // get the tag name
+  while (isalnum(c)) {
+    el.text.push_back (c);
+    el.tagName.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) return false;
+  }
+  
+  // get everything up to the closing '>'
+  while (c != '>') {
+    el.text.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) return false;
+  }
+
+  // get the closing '>'
+  el.text.push_back (c);
+  buf.Next();
+
+  // if there is no tag name then make this a text element
+  if (el.tagName.empty()) el.elType = TextE;
+  
+  return true;
+}
+
+static void ToggleParaTag (TextEl &el, bool &compatInPara) {
+  SetCStr (el.tagName, "Paragraph");
+  el.text.erase (el.text.begin(), el.text.end());
+  if (compatInPara) {
+    el.elType = CloseTagE;
+    el.text.push_back (PARA_TERM);
+  } else el.elType = OpenTagE;
+  compatInPara = !compatInPara;
+}
+
+static void SetRecTag (TextEl &el, TextElType elType) {
+  el.elType = elType;
+  SetCStr (el.tagName, "Document");
+  el.text.erase (el.text.begin(), el.text.end());
+  if (elType == CloseTagE)
+    el.text.push_back (REC_TERM);
+}
+
+
+bool ReadTextEl (FileBuf &buf, TextEl &el,
+		 bool compatMode, bool &compatInPara) {
+  el.Clear();
+
+  unsigned char c;
+  if (!buf.Peek (c)) return false;
+
+  if (compatMode) {
+    if (c == PARA_TERM) {
+      ToggleParaTag (el, compatInPara);
+      if (!compatInPara) buf.Next();
+      return true;
+    }
+    if (c == REC_TERM) {
+      if (compatInPara) {
+	// need to close this paragraph
+	ToggleParaTag (el, compatInPara);
+	return true;
+      }
+      // can close this document
+      buf.Next();
+      SetRecTag (el, CloseTagE);
+      return true;
+    }
+    if (!compatInPara) {
+      // need to open a paragraph
+      ToggleParaTag (el, compatInPara);
+      return true;
+    }
+  }
+  
+  // see if we have some sort of tag
+  if (c == '<') return ReadTextTag (buf, el);
+
+  // read in a text element
+  el.elType = TextE;
+  while (c != '<' && !(compatMode && (c == PARA_TERM || c == REC_TERM))) {
+    el.text.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) break;
+  }
+
+  return true;
+}
+
+static void AddTextEl (TextElArray &doc, unsigned long &docLen,
+		       const TextEl &el) {
+  doc.push_back (el);
+  docLen += el.text.size();
+
+//    cout << "elType:  " << el.elType << "\n";
+//    cout << "tagName: " << el.tagName << "\n";
+//    cout << "text:    \"" << el.text << "\"\n\n";
+}
+
+
+bool ReadDoc (FileBuf &buf, const UCArray &docTag,
+	      TextElArray &doc, unsigned long &docLen,
+	      bool compatMode) {
+  bool compatInPara = false;
+  bool foundDocEl = false;
+  TextEl el;
+  
+  doc.erase (doc.begin(), doc.end());
+  docLen = 0;
+
+  if (compatMode) {
+    // add <Document><Paragraph>
+    SetRecTag (el, OpenTagE);
+    AddTextEl (doc, docLen, el);
+    ToggleParaTag (el, compatInPara);
+    AddTextEl (doc, docLen, el);
+    
+  } else {
+    // look for an opening docTag
+    do {
+      if (!ReadTextEl (buf, el, compatMode, compatInPara)) return false;
+    } while (el.elType != OpenTagE || el.tagName != docTag);
+    
+    AddTextEl (doc, docLen, el);
+  }
+
+  
+  // get all elements until the closing docTag
+  while (ReadTextEl (buf, el, compatMode, compatInPara)) {
+    foundDocEl = true;
+    AddTextEl (doc, docLen, el);
+    if (el.elType == CloseTagE && el.tagName == docTag)
+      return true;
+  }
+
+  
+  if (compatMode) {
+    if (!foundDocEl) { // end of text
+      doc.erase (doc.begin(), doc.end());
+      return false;
+    }
+
+    // if we got here then the eof was encountered before
+    // the closing document tag
+    if (compatInPara) {
+      // need to close this paragraph
+      ToggleParaTag (el, compatInPara);
+      AddTextEl (doc, docLen, el);
+    }
+    // close this document
+    SetRecTag (el, CloseTagE);
+    AddTextEl (doc, docLen, el);
+  }
+  
+  return true;
+}
+
+
Index: /trunk/indexers/mgpp/text/TextEl.h
===================================================================
--- /trunk/indexers/mgpp/text/TextEl.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/TextEl.h	(revision 3365)
@@ -0,0 +1,62 @@
+/**************************************************************************
+ *
+ * TextEl.h -- Data structures for parsed documents
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TEXTEL_H
+#define TEXTEL_H
+
+#include "UCArray.h"
+#include "FileBuf.h"
+
+
+enum TextElType {OpenTagE, CloseTagE, TextE};
+
+class TextEl {
+protected:
+  bool ReadTextTag (FileBuf &buf, TextEl &el);
+public:
+  TextElType elType;
+  UCArray tagName;
+  UCArray text; // complete text, including tag
+
+  TextEl () { Clear (); }
+  void Clear ();
+};
+
+
+typedef vector<TextEl> TextElArray;
+
+
+// compatMode parses documents similar to "old mg" -- ie with
+// C-b between documents and C-c between paragraphs
+
+// returns true if a text element could be read
+bool ReadTextEl (FileBuf &buf, TextEl &el,
+		 bool compatMode, bool &compatInPara);
+
+// looks for an opening docTag and reads all text elements up
+// to a corresponding closing docTag. note that this does not
+// hierarchically parse the document so you can't nest documents.
+// returns true if a document could be found
+bool ReadDoc (FileBuf &buf, const UCArray &docTag,
+	      TextElArray &doc, unsigned long &docLen,
+	      bool compatMode=false);
+
+#endif
Index: /trunk/indexers/mgpp/text/TextGet.cpp
===================================================================
--- /trunk/indexers/mgpp/text/TextGet.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/TextGet.cpp	(revision 3365)
@@ -0,0 +1,681 @@
+/**************************************************************************
+ *
+ * TextGet.cpp -- Decompressing the text
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "TextGet.h"
+#include "mg_files.h"
+#include "netorder.h"
+#include "mg_errors.h"
+#include "locallib.h"
+#include "words.h"
+#include "local_strings.h"
+#include "bitio_m_stdio.h"
+
+typedef enum huff_type {lengths, chars};
+
+
+static auxiliary_dict *LoadAuxDict (compression_dict &cd, FILE *text_aux_dict) {
+  auxiliary_dict *ad;
+  int i;
+
+  if (!(ad = new auxiliary_dict))
+    {
+      mg_errno = MG_NOMEM;
+      return (NULL);
+    }
+
+  memset (ad, '\0', sizeof (*ad));
+
+  for (i = 0; i <= 1; i++)
+    {
+      int j;
+      u_char *pos;
+
+      fread (&ad->afh[i], sizeof (aux_frags_header), 1, text_aux_dict);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(ad->afh[i].num_frags);
+      NTOHUL(ad->afh[i].mem_for_frags);
+
+      if (!(ad->word_data[i] = new u_char[ad->afh[i].mem_for_frags]))
+	{
+	  mg_errno = MG_NOMEM;
+	  delete ad;
+	  return (NULL);
+	}
+      if (!(ad->words[i] = new u_char* [ad->afh[i].num_frags]))
+	{
+	  mg_errno = MG_NOMEM;
+	  delete ad;
+	  return (NULL);
+	}
+
+      fread (ad->word_data[i], ad->afh[i].mem_for_frags, sizeof (u_char),
+	     text_aux_dict);
+
+      pos = ad->word_data[i];
+      for (j = 0; j < (int)ad->afh[i].num_frags; j++)
+	{
+	  ad->words[i][j] = pos;
+	  pos += *pos + 1;
+	}
+      if (cd.cdh.novel_method == MG_NOVEL_HYBRID)
+	{
+	  int num;
+	  num = 1;
+	  ad->blk_start[i][0] = 0;
+	  ad->blk_end[i][0] = cd.cdh.num_words[i] - 1;
+	  while (num < 33)
+	    {
+	      ad->blk_start[i][num] = ad->blk_end[i][num - 1] + 1;
+	      ad->blk_end[i][num] = ad->blk_start[i][num] +
+		(ad->blk_end[i][num - 1] - ad->blk_start[i][num - 1]) * 2;
+	      num++;
+	    }
+	}
+    }
+  return (ad);
+}
+
+
+static u_char ***ReadInWords (FILE *dict, compression_dict &cd,
+			      comp_frags_header *cfh, u_char **escape) {
+  int i, lookback;
+  int ptrs_reqd = 0;
+  int mem_reqd = 0;
+  int num_set[MAX_HUFFCODE_LEN + 1];
+  u_char *next_word[MAX_HUFFCODE_LEN + 1];
+  u_char **vals;
+  u_char ***values;
+  u_char word[MAXWORDLEN + 1];
+  u_char last_word[MAX_HUFFCODE_LEN + 1][MAXWORDLEN + 1];
+
+  lookback = cd.cdh.lookback;
+
+  for (i = cfh->hd.mincodelen; i <= cfh->hd.maxcodelen; i++) {
+    ptrs_reqd += (cfh->hd.lencount[i] + ((1 << lookback) - 1)) >> lookback;
+    mem_reqd += cfh->huff_words_size[i];
+  }
+
+  if (!(vals = new u_char* [ptrs_reqd]))
+    return (NULL);
+
+  if (!(values = new u_char** [MAX_HUFFCODE_LEN + 1]))
+    return (NULL);
+
+  if (!(next_word[0] = new u_char[mem_reqd])) 
+    return (NULL);
+
+  cd.MemForCompDict += ptrs_reqd * sizeof (*vals) +
+    (MAX_HUFFCODE_LEN + 1) * sizeof (u_char **) +
+    mem_reqd;
+
+  values[0] = vals;
+  values[0][0] = next_word[0];
+  for (i = 1; i <= cfh->hd.maxcodelen; i++)
+    {
+      int next_start = (values[i - 1] - vals) +
+      ((cfh->hd.lencount[i - 1] + ((1 << lookback) - 1)) >> lookback);
+      values[i] = &vals[next_start];
+      next_word[i] = next_word[i - 1] + cfh->huff_words_size[i - 1];
+      values[i][0] = next_word[i];
+    }
+
+  memset (num_set, '\0', sizeof (num_set));
+
+  for (i = 0; i < cfh->hd.num_codes; i++)
+    {
+      register int val, copy;
+      register int len = cfh->hd.clens[i];
+      val = getc (dict);
+      copy = (val >> 4) & 0xf;
+      val &= 0xf;
+
+      fread (word + copy + 1, sizeof (u_char), val, dict);
+      *word = val + copy;
+
+      if ((num_set[len] & ((1 << lookback) - 1)) == 0)
+	{
+	  values[len][num_set[len] >> lookback] = next_word[len];
+	  memcpy (next_word[len], word, *word + 1);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    *escape = next_word[len];
+	  next_word[len] += *word + 1;
+	}
+      else
+	{
+	  copy = prefixlen (last_word[len], word);
+	  memcpy (next_word[len] + 1, word + copy + 1, *word - copy);
+	  *next_word[len] = (copy << 4) + (*word - copy);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    *escape = next_word[len];
+	  next_word[len] += (*word - copy) + 1;
+	}
+      memcpy (last_word[len], word, *word + 1);
+      num_set[len]++;
+    }
+  if (cfh->hd.clens)
+    delete cfh->hd.clens;
+  cfh->hd.clens = NULL;
+  return values;
+}
+
+static int Load_Comp_HuffData(compression_dict &cd, int which, FILE *dict, 
+			      huff_type type) {
+  huff_data * hd;
+  u_long **   vals;
+
+  if (!(hd = new huff_data))
+    return 1;
+  cd.MemForCompDict += sizeof (huff_data);
+  if (Read_Huffman_Data (dict, hd, &cd.MemForCompDict, NULL) == -1)
+    return 2;
+  if (!(vals = Generate_Huffman_Vals (hd, &cd.MemForCompDict)))
+    return 3;
+  if (hd->clens)
+    delete hd->clens;
+  hd->clens = NULL;
+  if (type == chars)
+    {
+      cd.chars_huff[which] = hd;
+      cd.chars_vals[which] = vals;
+    }
+  else
+    {
+      cd.lens_huff[which] = hd;
+      cd.lens_vals[which] = vals;
+    }
+
+  return 0;
+}
+
+static int Load_Comp_FragsHeader(compression_dict &cd, int which, int getEscape,
+				 FILE *dict) {
+  if (!(cd.cfh[which] = new comp_frags_header))
+    return 1;
+  cd.MemForCompDict += sizeof (*cd.cfh[which]);
+  if (Read_cfh (dict, cd.cfh[which], &cd.MemForCompDict, NULL) == -1)
+    return 2;
+
+  if (!(cd.values[which] = ReadInWords (dict, cd, cd.cfh[which],
+					getEscape == 0 ? NULL : &cd.escape[which])))
+    return 3;
+  
+  return 0;
+}
+
+static bool LoadSlowCompDict (FILE *dict, FILE *aux_dict, compression_dict &cd) {
+  if (dict == NULL) return false;
+
+  int which;
+
+  memset (&cd, '\0', sizeof (compression_dict));
+
+  cd.MemForCompDict = sizeof (compression_dict);
+
+  if (Read_cdh (dict, &cd.cdh, &cd.MemForCompDict, NULL) == -1)
+    return false;
+
+  for (which = 0; which < 2; which++)
+    switch (cd.cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  if (Load_Comp_FragsHeader(cd, which, 0, dict) != 0)
+	    return false;
+	  cd.escape[which] = NULL;
+
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (cd.cdh.num_words[which])
+	    {
+	      if (Load_Comp_FragsHeader(cd, which, 1, dict) != 0)
+		return false;
+	    }
+
+	  if (Load_Comp_HuffData(cd, which, dict, chars) != 0)
+	    return false;
+
+	  if (Load_Comp_HuffData(cd, which, dict, lengths) != 0)
+	    return false;
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (cd.cdh.num_words[which])
+	    {
+	      if (Load_Comp_FragsHeader(cd, which, 1, dict) != 0)
+		return false;
+	    }
+	  switch (cd.cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      if (Load_Comp_HuffData(cd, which, dict, chars) != 0)
+		return false;
+
+	      if (Load_Comp_HuffData(cd, which, dict, lengths) != 0)
+		return false;
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    }
+	  break;
+	}
+      }
+
+  if (cd.cdh.novel_method == MG_NOVEL_DELTA ||
+      cd.cdh.novel_method == MG_NOVEL_HYBRID)
+    {
+      if (!aux_dict)
+	{
+	  mg_errno = MG_NOFILE;
+	  cd.Clear();
+	  return false;
+	}
+
+      if (!(cd.ad = LoadAuxDict (cd, aux_dict)))
+	{
+	  cd.Clear();
+	  return false;
+	}
+    }
+
+  mg_errno = MG_NOERROR;
+
+  cd.fast_loaded = 0;
+  
+  return true;
+}
+
+
+
+#define WORDNO(p, base) ((((char*)(p))-((char*)(base)))/sizeof(u_char*))
+#define IS_FIXUP(p) ((fixup[WORDNO(p,cd)/8] & (1<<(WORDNO(p,cd) & 7))) != 0)
+
+// fast loading really needs to be totally re-writen. "Unloading" the
+// text data will currently cause a crash because memory is being
+// deleted multiple times (and probably a zillion other reasons).
+static bool LoadFastCompDict (FILE *text_fast_comp_dict, compression_dict &_cd) {
+  if (text_fast_comp_dict == NULL) return false;
+  
+  u_long *p, *end;
+  u_char *fixup;
+  u_long mem;
+  u_long fixup_mem;
+  int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+  fread (&mem, sizeof (mem), 1, text_fast_comp_dict);
+  NTOHUL(mem);  /* [RPAP - Jan 97: Endian Ordering] */
+  fread (&fixup_mem, sizeof (fixup_mem), 1, text_fast_comp_dict);
+  NTOHUL(fixup_mem);  /* [RPAP - Jan 97: Endian Ordering] */
+
+  compression_dict *cd;
+  if (!(cd = (compression_dict *)malloc (mem))) {
+    mg_errno = MG_NOMEM;
+    return false;
+  }
+
+  end = (u_long *) (((u_char *) cd) + mem);
+  fread (cd, sizeof (u_char), mem, text_fast_comp_dict);
+
+  if (!(fixup = new u_char[fixup_mem]))
+    {
+      mg_errno = MG_NOMEM;
+      return false;
+    }
+
+  fread (fixup, fixup_mem, sizeof (u_char), text_fast_comp_dict);
+
+  for (p = (u_long *) cd; (u_long) p < (u_long) end; p++)
+    if (IS_FIXUP (p))
+      {
+	NTOHUL(*p);  /* [RPAP - Jan 97: Endian Ordering] */
+	*p = *p + (u_long) cd;
+      }
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  /* cdh */
+  NTOHUL(cd->cdh.dict_type);
+  NTOHUL(cd->cdh.novel_method);
+  for (i = 0; i < TEXT_PARAMS; i++)
+    NTOHUL(cd->cdh.params[i]);
+  NTOHUL(cd->cdh.num_words[0]);
+  NTOHUL(cd->cdh.num_words[1]);
+  NTOHUL(cd->cdh.num_word_chars[0]);
+  NTOHUL(cd->cdh.num_word_chars[1]);
+  NTOHUL(cd->cdh.lookback);
+  /* cfh */
+  for (i = 0; i <= 1; i++)
+    {
+      int j;
+
+      NTOHSI(cd->cfh[i]->hd.num_codes);
+      NTOHSI(cd->cfh[i]->hd.mincodelen);
+      NTOHSI(cd->cfh[i]->hd.maxcodelen);
+      for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	{
+	  NTOHSI(cd->cfh[i]->hd.lencount[j]);
+	  NTOHUL(cd->cfh[i]->hd.min_code[j]);
+	}
+      NTOHUL(cd->cfh[i]->uncompressed_size);
+      for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	NTOHUL(cd->cfh[i]->huff_words_size[j]);
+    }
+  NTOHUL(cd->MemForCompDict);
+  /* ad */
+  if (cd->cdh.novel_method == MG_NOVEL_DELTA ||
+      cd->cdh.novel_method == MG_NOVEL_HYBRID)
+    for (i = 0; i <= 1; i++)
+      {
+	int j;
+	
+	NTOHUL(cd->ad->afh[i].num_frags);
+	NTOHUL(cd->ad->afh[i].mem_for_frags);
+	for (j = 0; j < 33; j++)
+	  {
+	    NTOHSI(cd->ad->blk_start[i][j]);
+	    NTOHSI(cd->ad->blk_end[i][j]);
+	  }
+      }
+  NTOHSI(cd->fast_loaded);
+
+  delete fixup;
+
+  // the whole fast comp dict is a bit of a hack so I don't
+  // feel too bad about the next line :-) -- Rodger.
+  _cd = *cd;
+  
+  return true;
+}
+
+
+static bool LoadCompDict (FILE *compDictFile,
+			  FILE *auxDictFile,
+			  FILE *fastCompDictFile,
+			  compression_dict &cd) {
+  // see if we have a fast loading compression dictionary
+  if (fastCompDictFile != NULL)
+    return LoadFastCompDict (fastCompDictFile, cd);
+  
+  // slow compression dictionary
+  return LoadSlowCompDict (compDictFile, auxDictFile, cd);
+}
+
+
+// try to open the dictionary files and load the dictionary
+static bool OpenLoadCompDict (char *textname, compression_dict &cd) {
+  FILE *compDictFile = NULL;
+  FILE *auxDictFile = NULL;
+  FILE *fastCompDictFile = NULL;
+
+  fastCompDictFile = open_file (textname, TEXT_DICT_FAST_SUFFIX,
+				"rb", MAGIC_FAST_DICT, MG_CONTINUE);
+  
+  if (fastCompDictFile == NULL) {
+    compDictFile = open_file (textname, TEXT_DICT_SUFFIX,
+			      "rb", MAGIC_DICT, MG_MESSAGE);
+    auxDictFile =  open_file (textname, TEXT_DICT_AUX_SUFFIX,
+			      "rb", MAGIC_AUX_DICT, MG_CONTINUE);
+  }
+  
+  bool res = LoadCompDict (compDictFile, auxDictFile, fastCompDictFile, cd);
+
+  if (compDictFile != NULL) fclose (compDictFile);
+  if (auxDictFile != NULL) fclose (auxDictFile);
+  if (fastCompDictFile != NULL) fclose (fastCompDictFile);
+  
+  return res;
+}
+
+static bool LoadLevels (char *textname, FTextLevel &levels) {
+  FILE *levelFile = NULL;
+
+  // open the text level file
+  levelFile = open_file (textname, TEXT_LEVEL_SUFFIX,
+			 "rb", MAGIC_TEXT_LEVELS, MG_CONTINUE);
+  if (levelFile == NULL) return false;
+
+  // seek to the appropriate place and read the level information
+  bool res = ((fseek (levelFile, sizeof (u_long), SEEK_SET) == 0) &&
+	      levels.Read (levelFile));
+
+  // close the file
+  fclose (levelFile);
+  
+  return res;
+}
+
+
+TextData::TextData () {
+  // put file pointers in known state first
+  textFile = NULL;
+  textIdxFile = NULL;
+  Clear ();
+}
+
+void TextData::Clear () {
+  cd.Clear(); 
+  textFile = NULL;
+  textIdxFile = NULL;
+  cth.Clear();
+  levels.Clear();
+}
+
+bool TextData::LoadData (char *basepath, char *textname) {
+  
+  if (textname[0] == '\0') return false;
+  
+  // set the basepath
+  set_basepath(basepath);
+  
+  // load the compression dictionary
+  if (!OpenLoadCompDict (textname, cd)) return false;
+
+  // open the compressed text and text index file
+  textFile = open_file (textname, TEXT_SUFFIX, "rb", MAGIC_TEXT, MG_CONTINUE);
+  if (textFile == NULL) return false;
+
+  textIdxFile = open_file (textname, TEXT_IDX_SUFFIX, "rb", MAGIC_TEXI, MG_CONTINUE);
+  if (textIdxFile == NULL) return false;
+
+  // read in the compressed text header
+  if ((fseek (textFile, sizeof (u_long), SEEK_SET) != 0) || !cth.Read (textFile))
+    return false;
+
+  // read in the level information
+  if (!LoadLevels (textname, levels)) return false;
+  
+  return true;
+}
+
+bool TextData::UnloadData () {
+  // close any open files
+  if (textFile != NULL) {
+    fclose (textFile);
+    textFile = NULL;
+  }
+  if (textIdxFile != NULL) {
+    fclose (textIdxFile);
+    textIdxFile = NULL;
+  }
+
+  // do general clear
+  Clear ();
+  
+  return true;  
+}
+
+
+bool GetDocIdx (TextData &td, const UCArray &docLevel,
+		unsigned long docNum, TextIdx &docIdx) {
+  // make sure the text index file was opened successfully
+  if (td.textIdxFile == NULL) return false;
+
+  // read in the index
+  TextLevelInfo &levelInfo = td.levels.levelInfo[docLevel];
+  if (!docIdx.Read (td.textIdxFile, levelInfo, docNum)) return false;
+  
+  return true;
+}
+
+
+
+
+#define MY_HUFF_DECODE(len, code, mcodes)				\
+  do {									\
+    register unsigned long *__min_code = (mcodes);			\
+    register unsigned long *__mclen = __min_code;			\
+    register unsigned long __code = 0;					\
+    do									\
+      {									\
+        __code += __code + buffer.bit();	                  	\
+      }									\
+    while (__code < *++__mclen);					\
+    (len) = __mclen - __min_code;					\
+    (code) = __code - *__mclen;						\
+  } while(0);
+
+
+bool GetDocText (TextData &td, const UCArray &docLevel,
+		 unsigned long docNum, UCArray &docText) {
+  // erase the current text
+  docText.erase (docText.begin(), docText.end());
+
+  // look up the information about this document
+  TextIdx docIdx;
+  if (!GetDocIdx (td, docLevel, docNum, docIdx)) return false;
+
+  // do seek to appropriate position
+  stdio_bitio_buffer buffer (td.textFile);
+  buffer.seek (docIdx.start.byte, docIdx.start.bit);
+  
+  // decompress the document
+  compression_dict &cd = td.cd;
+  auxiliary_dict *ad = cd.ad;
+  int which = docIdx.which;
+  unsigned long num_bits = (docIdx.end.byte*8+(8-docIdx.end.bit)) -
+    (docIdx.start.byte*8+(8-docIdx.start.bit));
+  unsigned long bits = 0;
+
+  // keep decoding bits until enough bits have been decoded
+  while (bits < num_bits) {
+    register unsigned code, len;
+    register int r;
+    register u_char *t, *b = NULL;
+    u_char word[MAXWORDLEN + 1];
+    
+    if (cd.cfh[which]) {
+      MY_HUFF_DECODE (len, code, cd.cfh[which]->hd.min_code);
+      bits += len;
+      
+      r = code & ((1 << cd.cdh.lookback) - 1);
+      t = cd.values[which][len][code >> cd.cdh.lookback];
+      
+      /* step through from base pointer */
+      b = word + 1;
+      while (r--) {
+	register int copy = *t >> 4;
+	memcpy (word + copy + 1, t + 1, *t & 0xf);
+	word[0] = copy + (*t & 0xf);
+	t += ((*t) & 0xf) + 1;
+      }
+    } else t = NULL;
+    
+    if (t == cd.escape[which]) {
+      switch (cd.cdh.novel_method) {
+      case MG_NOVEL_HUFFMAN_CHARS:
+	{
+	  int len, i;
+	  int c;
+	  len = buffer.huff_decode(cd.lens_huff[which]->min_code,
+				   cd.lens_vals[which], &bits);
+	  for (i = 0; i < len; i++) {
+	    c = buffer.huff_decode(cd.chars_huff[which]->min_code,
+				   cd.chars_vals[which], &bits);
+	    docText.push_back (c);
+	  }
+	}
+	break;
+      case MG_NOVEL_DELTA:
+      case MG_NOVEL_HYBRID:
+	{
+	  int idx = 0, len;
+	  u_char *base;
+	  switch (cd.cdh.novel_method)
+	    {
+	    case MG_NOVEL_DELTA:
+	      {
+		idx = buffer.delta_decode (&bits);
+		idx--;
+	      }
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      {
+		int k;
+		k = buffer.gamma_decode (&bits);
+		k--;
+		idx = buffer.binary_decode(ad->blk_end[which][k] -
+					   ad->blk_start[which][k] + 1, 
+					   &bits);
+		idx += ad->blk_start[which][k] - 1;
+	      }
+	      break;
+	    }
+	  base = ad->words[which][idx];
+	  len = *base++;
+	  for (; len; len--)
+	    {
+	      docText.push_back (*base++);
+	    }
+	}
+	break;
+      }
+    }
+    else
+      {
+	/* copy over the matching prefix */
+	r = (*t >> 4);
+	while (r--) {
+	  docText.push_back (*b++);
+	}
+	
+	/* and the stored suffix */
+	r = ((*t) & 0xf);
+	while (r--) {
+	  docText.push_back (*++t);
+	}
+      }
+    which = !which;
+  }
+  
+  buffer.done();
+  
+  return true;
+}
+
Index: /trunk/indexers/mgpp/text/TextGet.h
===================================================================
--- /trunk/indexers/mgpp/text/TextGet.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/TextGet.h	(revision 3365)
@@ -0,0 +1,58 @@
+/**************************************************************************
+ *
+ * TextGet.h -- Decompressing the text
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TEXTGET_H
+#define TEXTGET_H
+
+#include "text.h"
+#include "FText.h"
+
+
+class TextData {
+public:
+  compression_dict cd;
+  FILE *textFile;
+  FILE *textIdxFile;
+  compressed_text_header cth;
+  FTextLevel levels;
+
+  TextData ();
+  void Clear ();
+
+  // loads compression dictionary, the compressed text header,
+  // and all level informaiton
+  bool LoadData (char *basepath, char *textname);
+  bool UnloadData ();
+};
+
+
+// note: document numbers start at 1
+
+// get information about where a document starts and finishes
+bool GetDocIdx (TextData &td, const UCArray &docLevel,
+		unsigned long docNum, TextIdx &docIdx);
+
+
+// decompress a document's text
+bool GetDocText (TextData &td, const UCArray &docLevel,
+		 unsigned long docNum, UCArray &docText);
+
+#endif
Index: /trunk/indexers/mgpp/text/UCArray.cpp
===================================================================
--- /trunk/indexers/mgpp/text/UCArray.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/UCArray.cpp	(revision 3365)
@@ -0,0 +1,278 @@
+/**************************************************************************
+ *
+ * UCArray.cpp -- vector based string class
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "UCArray.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+
+void SetCStr (UCArray &text, const char *cStr) {
+  text.erase(text.begin(), text.end());
+  
+  while (*cStr != '\0') {
+    text.push_back (*cStr);
+    cStr++;
+  }
+}
+
+char * GetCStr(UCArray text) {
+
+  char *cstr = new char[text.size()+1];
+  UCArray::const_iterator here = text.begin();
+  UCArray::const_iterator end = text.end();
+
+  int i = 0;
+  while (here != end) {
+    cstr[i] = text[i];
+    here++; i++;
+  }
+  cstr[i]='\0';
+  return cstr;
+}
+
+ostream &operator<<(ostream &s, const UCArray &a) {
+  UCArray::const_iterator here = a.begin();
+  UCArray::const_iterator end = a.end();
+  while (here != end) {
+    s << *here;
+    here++;
+  }
+
+  return s;
+}
+
+
+bool ReadVarLenUL (FILE *f, unsigned long &n) {
+  register unsigned long temp = 0;
+  register unsigned int bitPos = 0;
+  unsigned char b = 0;
+
+  do {
+    b = fgetc (f);
+    if (feof(f)) return false;
+    temp |= (b & 0x7f) << bitPos;
+    bitPos += 7;
+  } while (b >= 0x80 && bitPos < 32);
+
+  n = temp;
+  
+  return true;
+}
+
+bool WriteVarLenUL (FILE *f, unsigned long n) {
+  register unsigned long temp = n;
+  register unsigned char b = 0;
+  do {
+    b = static_cast<unsigned char> (temp & 0x7f);
+    if (temp >= 0x80) b |= 0x80;
+    fputc (b, f);
+    if (ferror (f) != 0) return false;
+  } while ((temp = temp >> 7) > 0);
+  
+  return true;
+}
+
+
+bool ReadUL (FILE *f, unsigned long &n) {
+  if (fread (&n, sizeof (unsigned long), 1, f) <= 0) return false;
+  NTOHUL (n);
+  return true;
+}
+
+
+bool WriteUL (FILE *f, unsigned long n) {
+  HTONUL (n);
+  return (fwrite (&n, sizeof (unsigned long), 1, f) > 0);
+}
+
+bool ReadF (FILE *f, float &n) {
+  if (fread (&n, sizeof (float), 1, f) <= 0) return false;
+  NTOHF(n);
+  return true;
+}
+
+bool WriteF (FILE *f, float n) {
+  HTONF(n);
+  return (fwrite (&n, sizeof (float), 1, f) > 0);
+}
+
+bool ReadD (FILE *f, double &n) {
+  if (fread (&n, sizeof (double), 1, f) <= 0) return false;
+  NTOHD(n);
+  return true;
+}
+
+bool WriteD (FILE *f, double n) {
+  HTOND(n);
+  return (fwrite (&n, sizeof (double), 1, f) > 0);
+}
+
+bool ReadUCArray (FILE *f, UCArray &a) {
+  // clear the array in preparation
+  a.erase (a.begin(), a.end());
+
+  // read in the array size
+  unsigned long arraySize = 0;
+  if (!ReadVarLenUL (f, arraySize)) return false;
+
+  // read in the array
+  unsigned char b = 0;
+  while (arraySize > 0) {
+    b = fgetc (f);
+    if (feof(f)) return false;
+    a.push_back (b);
+    
+    arraySize--;
+  }
+  
+  return true;
+}
+
+bool WriteUCArray (FILE *f, const UCArray &a) {
+  // write out the array size
+  if (!WriteVarLenUL (f, a.size())) return false;
+
+  UCArray::const_iterator here = a.begin();
+  UCArray::const_iterator end = a.end();
+  while (here != end) {
+    fputc (*here, f);
+    if (ferror (f) != 0) return false;
+    
+    here++;
+  }
+  
+  return true;
+}
+
+/*
+ * This array is designed for mapping upper and lower case letter
+ * together for a case independent comparison.  The mappings are
+ * based upon ascii character sequences.
+ */
+static unsigned char casecharmap[] = {
+	'\000', '\001', '\002', '\003', '\004', '\005', '\006', '\007',
+	'\010', '\011', '\012', '\013', '\014', '\015', '\016', '\017',
+	'\020', '\021', '\022', '\023', '\024', '\025', '\026', '\027',
+	'\030', '\031', '\032', '\033', '\034', '\035', '\036', '\037',
+	'\040', '\041', '\042', '\043', '\044', '\045', '\046', '\047',
+	'\050', '\051', '\052', '\053', '\054', '\055', '\056', '\057',
+	'\060', '\061', '\062', '\063', '\064', '\065', '\066', '\067',
+	'\070', '\071', '\072', '\073', '\074', '\075', '\076', '\077',
+	'\100', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\133', '\134', '\135', '\136', '\137',
+	'\140', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\173', '\174', '\175', '\176', '\177',
+	'\200', '\201', '\202', '\203', '\204', '\205', '\206', '\207',
+	'\210', '\211', '\212', '\213', '\214', '\215', '\216', '\217',
+	'\220', '\221', '\222', '\223', '\224', '\225', '\226', '\227',
+	'\230', '\231', '\232', '\233', '\234', '\235', '\236', '\237',
+	'\240', '\241', '\242', '\243', '\244', '\245', '\246', '\247',
+	'\250', '\251', '\252', '\253', '\254', '\255', '\256', '\257',
+	'\260', '\261', '\262', '\263', '\264', '\265', '\266', '\267',
+	'\270', '\271', '\272', '\273', '\274', '\275', '\276', '\277',
+	'\300', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\333', '\334', '\335', '\336', '\337',
+	'\340', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\373', '\374', '\375', '\376', '\377',
+};
+
+
+int DictCompare (const UCArray &a1, const UCArray &a2) {
+  unsigned int l1 = a1.size();
+  unsigned int l2 = a2.size();
+  unsigned int l = (l1 < l2) ? l1 : l2;
+  int pos = 0;
+  register int diff = 0;
+
+  UCArray::const_iterator a1Here = a1.begin();
+  UCArray::const_iterator a2Here = a2.begin();
+  
+  while (l--) {
+    if ((diff = casecharmap[*a1Here] - casecharmap[*a2Here]) != 0)
+      return diff;
+    if (pos == 0 && (diff = *a1Here - *a2Here) != 0)
+      pos = diff;
+
+    a1Here++;
+    a2Here++;
+  }
+
+  return ((l1 - l2) ? (l1 - l2) : (pos));
+}
+
+unsigned long PrefixLen (const UCArray &a1, const UCArray &a2) {
+  unsigned long l = (a1.size() < a2.size()) ? a1.size() : a2.size();
+  unsigned long i = 0;
+
+  UCArray::const_iterator a1Here = a1.begin();
+  UCArray::const_iterator a2Here = a2.begin();
+
+  while (i < l && *a1Here == *a2Here) {
+    i++; a1Here++; a2Here++;
+  }
+  
+  return i;
+}
+
+bool WritePreSufStr (FILE *f, const UCArray *prev, const UCArray &a) {
+  unsigned char preLen;
+  unsigned char sufLen;
+  
+  if (prev != NULL) preLen = PrefixLen (*prev, a);
+  else preLen = 0;
+  sufLen = a.size() - preLen;
+
+  // output the prefix length, suffix length, and the suffix
+  fputc (preLen, f);
+  if (ferror(f) != 0) return false;
+  fputc (sufLen, f);
+  if (ferror(f) != 0) return false;
+  char* tmp=GetCStr(a);
+  int ret=(fwrite (tmp+preLen, sizeof (char), sufLen, f) == sufLen);
+  delete (tmp);
+  return (ret);
+}
+
+// a also used for prev
+bool ReadPreSufStr (FILE *f, UCArray &a) {
+  unsigned char preLen = 0;
+  unsigned char sufLen = 0;
+
+  preLen = fgetc(f);
+  sufLen = fgetc(f);
+
+  if (a.size () > preLen) a.erase (a.begin()+preLen, a.end());
+  while (sufLen > 0) {
+    unsigned char c = fgetc (f);
+    a.push_back (c);
+    sufLen--;
+  }
+
+  return true;
+}
+
Index: /trunk/indexers/mgpp/text/UCArray.h
===================================================================
--- /trunk/indexers/mgpp/text/UCArray.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/UCArray.h	(revision 3365)
@@ -0,0 +1,118 @@
+/**************************************************************************
+ *
+ * UCArray.h -- vector based string class
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef UCARRAY_H
+#define UCARRAY_H
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if !defined (GSDL_NAMESPACE_BROKEN) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+using namespace std;
+#endif
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\vector>
+#  include <ospace\std\set>
+#elif defined(GSDL_USE_STL_H)
+#  include <vector.h>
+#  include <set.h>
+#else
+#  include <vector>
+#  include <set>
+#endif
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\iostream>
+#elif defined(GSDL_USE_IOS_H)
+#  include <iostream.h>
+#else
+#  include <iostream>
+#endif
+
+#include <stdio.h>
+
+
+typedef vector<unsigned char> UCArray;
+
+// functions to manipulate UCArrays
+void SetCStr (UCArray &text, const char *cStr);
+char * GetCStr(UCArray text);
+inline void UCArrayClear (UCArray &a) {
+  a.erase (a.begin(), a.end());
+}
+
+// stream operator to print UCArray
+ostream &operator<<(ostream &s, const UCArray &a);
+
+// used for reading and writing unsigned characters
+inline bool ReadUC (FILE *f, unsigned char &c) {
+  return (fread (&c, sizeof (unsigned char), 1, f) == 1);
+}
+inline bool WriteUC (FILE *f, unsigned char c) {
+  return (fwrite (&c, sizeof (unsigned char), 1, f) == 1);
+}
+
+// used for reading and writing variable length unsigned longs
+bool ReadVarLenUL (FILE *f, unsigned long &n);
+bool WriteVarLenUL (FILE *f, unsigned long n);
+
+// used for reading and writing unsigned longs
+bool ReadUL (FILE *f, unsigned long &n);
+bool WriteUL (FILE *f, unsigned long n);
+
+// used for reading and writing floats
+bool ReadF (FILE *f, float &n);
+bool WriteF (FILE *f, float n);
+
+// used for reading and writing doubles
+bool ReadD (FILE *f, double &n);
+bool WriteD (FILE *f, double n);
+
+// used for reading and writing arrays to files
+bool ReadUCArray (FILE *f, UCArray &a);
+bool WriteUCArray (FILE *f, const UCArray &a);
+
+
+// compares the two strings in dictionary order
+int DictCompare (const UCArray &a1, const UCArray &a2);
+
+struct LTUCArray {
+  bool operator()(const UCArray &a1, const UCArray &a2) const
+    { return (a1 < a2); }
+};
+
+struct DictLTUCArray {
+  bool operator() (const UCArray &a1, const UCArray &a2) const
+    { return (DictCompare (a1, a2) < 0); }
+};
+
+
+typedef set<UCArray, LTUCArray> UCArraySet;
+typedef vector<UCArray> UCArrayVector;
+
+unsigned long PrefixLen (const UCArray &a1, const UCArray &a2);
+
+// prev == NULL if no previous string
+bool WritePreSufStr (FILE *f, const UCArray *prev, const UCArray &a);
+bool ReadPreSufStr (FILE *f, UCArray &a); // a also used for prev
+
+
+#endif
Index: /trunk/indexers/mgpp/text/WIN32.MAK
===================================================================
--- /trunk/indexers/mgpp/text/WIN32.MAK	(revision 3365)
+++ /trunk/indexers/mgpp/text/WIN32.MAK	(revision 3365)
@@ -0,0 +1,192 @@
+###########################################################################
+#
+# win32 makefile -- mgpp\text
+# A component of the Greenstone digital library software
+# from the New Zealand Digital Library Project at the
+# University of Waikato, New Zealand.
+#
+# Copyright (C) 2000  The New Zealand Digital Library Project
+#
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+###########################################################################
+
+GSDLHOME = ..\..\..
+USE_STLPORT = 0
+
+STLPATH = "$(GSDLHOME)\packages\windows\stlport\stlport"
+
+CC = cl
+CPPFLAGS = -GX
+DEFS = -DSHORT_SUFFIX -DSILENT -DHAVE_CONFIG_H -D__WIN32__ -D__STDC__ -D__GNU_LIBRARY__ -D__MSDOS__
+INCLUDES = -I"$(GSDLHOME)\src\mgpp\lib" -I"$(GSDLHOME)"
+!IF $(USE_STLPORT)
+DEFS = $(DEFS) -DGSDL_NAMESPACE_BROKEN -DGSDL_USE_IOS_H -D__STL_NO_NEW_IOSTREAMS
+INCLUDES = $(INCLUDES) -I$(STLPATH)
+!ENDIF
+
+LDFLAGS =
+
+LIBS = "$(GSDLHOME)\src\mgpp\lib\libmg.lib"
+
+COMPILE = $(CC) -c $(CPPFLAGS) $(DEFS) $(INCLUDES)
+LINK = $(CC) $(LDFLAGS)
+
+.SUFFIXES:
+.SUFFIXES: .c .obj .cpp
+.c.obj:
+	$(COMPILE) $<
+.cpp.obj:
+	$(COMPILE) $<
+
+o = .obj
+e = .exe
+
+SOURCES = \
+  FIvfLevelInfo.cpp        ivf.pass1.cpp            mgpp_invf_dump.cpp       \
+  FText.cpp                ivf.pass2.cpp            mgpp_passes.cpp          \
+  FileBuf.cpp              locallib.cpp             mgpp_perf_hash_build.cpp \
+  FragLevelConvert.cpp     mg.special.cpp           mgpp_stem_idx.cpp        \
+  IndexData.cpp            mgpp_compression_dict.cpp  mgpp_weights_build.cpp   \
+  TagInfo.cpp              mgpp_decompress_text.cpp   stemmer.cpp            \
+  TextEl.cpp               mg_errors.cpp            text.cpp               \
+  TextGet.cpp              mgpp_fast_comp_dict.cpp    text.pass1.cpp         \
+  UCArray.cpp              mg_files.cpp             text.pass2.cpp         \
+  comp_dict.cpp            mgpp_invf_dict.cpp         words.cpp              \
+  invf.cpp                 mgpp_invf_dict_dump.cpp    Weights.cpp            \
+  MGQuery.cpp              Terms.cpp                QueryTester.cpp        \
+  QueryLex.cpp             QueryParser.cpp         \
+  GSDLQueryLex.cpp	   GSDLQueryParser.cpp      Queryer.cpp
+
+
+HEADERS = \
+  FIvfLevelInfo.h     TextEl.h            hash.h              stemmer.h  \
+  FText.h             TextGet.h           invf.h              text.h     \
+  FileBuf.h           UCArray.h           locallib.h          words.h    \
+  FragLevelConvert.h  WordData.h          mg.h                           \
+  IndexData.h         build.h             mg_errors.h                    \
+  TagInfo.h           comp_dict.h         mg_files.h          Weights.h  \
+  MGQuery.h           Terms.h             QueryLex.h          QueryParser.h \
+  GSDLQueryLex.h	GSDLQueryParser.h
+
+EXEC          = mgpp_passes$e mgpp_decompress_text$e mgpp_perf_hash_build$e \
+                mgpp_invf_dump$e mgpp_weights_build$e mgpp_invf_dict$e mgpp_stem_idx$e \
+                mgpp_invf_dict_dump$e mgpp_compression_dict$e mgpp_fast_comp_dict$e \
+                QueryTester$e Queryer$e
+
+all:    $(EXEC) libtextin.lib
+
+PASSES_OBJS = \
+	mgpp_passes$o text.pass1$o comp_dict$o \
+	text.pass2$o locallib$o \
+	ivf.pass1$o ivf.pass2$o mg.special$o mg_files$o \
+	words$o invf$o FileBuf$o TextEl$o UCArray$o TagInfo$o \
+        FText$o text$o FIvfLevelInfo$o
+
+mgpp_passes$e: $(PASSES_OBJS)
+	$(LINK) $(PASSES_OBJS) $(LIBS)
+
+
+QUERYER_OBJS = Queryer$o mg_files$o mg_errors$o locallib$o invf$o UCArray$o \
+               FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+               IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+               GSDLQueryParser$o GSDLQueryLex$o words$o
+
+Queryer$e: $(QUERYER_OBJS)
+	$(LINK) $(QUERYER_OBJS) $(LIBS)
+
+QUERY_TESTER_OBJS = QueryTester$o mg_files$o invf$o UCArray$o \
+                    FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+                    IndexData$o stemmer$o Weights$o
+
+QueryTester$e: $(QUERY_TESTER_OBJS)
+	$(LINK) $(QUERY_TESTER_OBJS) $(LIBS)
+
+WEIGHTS_BUILD_OBJS = mgpp_weights_build$o mg_files$o invf$o UCArray$o \
+                     FIvfLevelInfo$o FragLevelConvert$o
+
+mgpp_weights_build$e: $(WEIGHTS_BUILD_OBJS)
+	$(LINK) $(WEIGHTS_BUILD_OBJS) $(LIBS)
+
+COMP_DICT_OBJS = mgpp_compression_dict$o mg_files$o locallib$o
+
+mgpp_compression_dict$e: $(COMP_DICT_OBJS)
+	$(LINK) $(COMP_DICT_OBJS) $(LIBS)
+
+FAST_COMP_OBJS = mgpp_fast_comp_dict$o mg_files$o locallib$o
+
+mgpp_fast_comp_dict$e: $(FAST_COMP_OBJS)
+	$(LINK) $(FAST_COMP_OBJS) $(LIBS)
+
+INVF_DUMP_OBJS = mgpp_invf_dump$o locallib$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dump$e: $(INVF_DUMP_OBJS)
+	$(LINK) $(INVF_DUMP_OBJS) $(LIBS)
+
+INVF_DICT_OBJS = mgpp_invf_dict$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict$e: $(INVF_DICT_OBJS)
+	$(LINK) $(INVF_DICT_OBJS) $(LIBS)
+
+INVF_DICT_DUMP_OBJS = mgpp_invf_dict_dump$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict_dump$e: $(INVF_DICT_DUMP_OBJS)
+	$(LINK) $(INVF_DICT_DUMP_OBJS) $(LIBS)
+
+PERF_HASH_OBJS = mgpp_perf_hash_build$o mg_files$o invf$o UCArray$o
+
+mgpp_perf_hash_build$e: $(PERF_HASH_OBJS)
+	$(LINK) $(PERF_HASH_OBJS) $(LIBS)
+
+STEM_IDX_OBJS = mgpp_stem_idx$o mg_files$o stemmer$o \
+                invf$o UCArray$o
+
+mgpp_stem_idx$e: $(STEM_IDX_OBJS)
+	$(LINK) $(STEM_IDX_OBJS) $(LIBS)
+
+DECOMPRESS_TEXT_OBJS = \
+                mgpp_decompress_text$o mg_files$o mg_errors$o \
+                locallib$o text$o UCArray$o TextGet$o FText$o
+
+mgpp_decompress_text$e: $(DECOMPRESS_TEXT_OBJS)
+	$(LINK) $(DECOMPRESS_TEXT_OBJS) $(LIBS)
+
+
+PASS_OBJS = \
+	words$o mgpass$o text.pass1$o comp_dict$o stemmer$o \
+	text.pass2$o locallib$o \
+	ivf.pass1$o ivf.pass2$o mg.special$o mg_files$o
+mgpass$e: $(PASS_OBJS)
+	$(LINK) $(PASS_OBJS) $(LIBS)
+
+install: 
+	if NOT EXIST "$(GSDLHOME)\bin\windows" mkdir "$(GSDLHOME)\bin\windows"
+	for %%i in ($(EXEC)) do \
+	  copy %%i "$(GSDLHOME)\bin\windows"
+
+LIB_OBJS = mg_files$o  mg_errors$o locallib$o invf$o UCArray$o \
+           FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+           IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+           GSDLQueryParser$o GSDLQueryLex$o words$o
+
+libtextin.lib: $(LIB_OBJS)
+	if EXIST libtextin.lib del libtextin.lib
+	lib /out:libtextin.lib $(LIB_OBJS)
+
+clean:
+	del *$o 
+	del $(EXEC)
+	del libtextin.lib
Index: /trunk/indexers/mgpp/text/Weights.cpp
===================================================================
--- /trunk/indexers/mgpp/text/Weights.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/Weights.cpp	(revision 3365)
@@ -0,0 +1,126 @@
+/**************************************************************************
+ *
+ * Weights.cpp -- Dealing with document weights
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "UCArray.h"
+#include "Weights.h"
+#include "sysfuncs.h"
+
+#define MAXBITS (sizeof(unsigned long) * 8)
+
+
+ApproxWeightsData::ApproxWeightsData () {
+  weightBuf = NULL;
+  table = NULL;
+  Free();
+}
+
+ApproxWeightsData::~ApproxWeightsData () {
+  Free ();
+}
+
+void ApproxWeightsData::Free () {
+  bits = 0;
+  mask = 0;
+  L = 0.0;
+  B = 0.0;
+  numLevelDocs = 0;
+  if (weightBuf != NULL) {
+    delete [] weightBuf;
+    weightBuf = NULL;
+  }
+  if (table != NULL) {
+    delete [] table;
+    table = NULL;
+  }
+}
+
+bool ApproxWeightsData::Read (FILE *approxWeightsFile, unsigned long diskPtr,
+			      unsigned long _numLevelDocs) {
+  Free ();
+  
+  // go to the appropriate place in the weights file
+  fseek (approxWeightsFile, diskPtr, SEEK_SET);
+
+  // read in approx weights parameters
+  ReadUC (approxWeightsFile, bits);
+  ReadD (approxWeightsFile, L);
+  ReadD (approxWeightsFile, B);
+
+  mask = (bits == 32) ? 0xffffffff : (1 << bits) - 1;
+  numLevelDocs = _numLevelDocs;
+
+  // allocate buffer for the weights
+  unsigned long arrSize = (bits*numLevelDocs+sizeof(unsigned long)-1) / 32;
+  weightBuf = new unsigned long[arrSize];
+
+  // read in the weights
+  unsigned long i;
+  for (i=0; i<arrSize; i++) {
+    ReadUL (approxWeightsFile, weightBuf[i]);
+  }
+
+  // create precomputed table of values if it will be small enough
+  // (if not small enough table == NULL
+  if (bits <= 12) {
+    unsigned long i, tableSize = (1 << bits);
+    table = new float[tableSize];
+    table[0] = L;
+    for (i = 1; i < tableSize; i++)
+      table[i] = table[i - 1] * B;
+  }
+
+  return true;
+}
+
+float ApproxWeightsData::GetLowerApproxDocWeight (unsigned long levelDocNum) {
+  // sanity check
+  if (bits == 0 || weightBuf == NULL) return 1.0;
+
+  register unsigned long c, pos;
+  register unsigned long *dw;
+
+  // get the compressed version of the weight
+  pos = levelDocNum * bits;
+  dw = &weightBuf[pos / MAXBITS];
+  pos &= (MAXBITS - 1);
+  c = *dw >> pos;
+  if (pos + bits > MAXBITS)
+    c |= *(dw + 1) << (MAXBITS - pos);
+  c &= mask;
+
+  // decompress the weight
+  if (table != NULL) return table[c];
+  return (L * pow (B, (double) c));
+}
+
+
+
+float GetExactDocWeight (FILE *exactWeightsFile, unsigned long diskPtr,
+			 unsigned long levelDocNum) {
+  // go to the appropriate place in the weights file
+  fseek (exactWeightsFile, diskPtr+sizeof(float)*levelDocNum, SEEK_SET);
+
+  // read in the exact weight
+  float weight;
+  ReadF (exactWeightsFile, weight);
+  
+  return weight;
+}
Index: /trunk/indexers/mgpp/text/Weights.h
===================================================================
--- /trunk/indexers/mgpp/text/Weights.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/Weights.h	(revision 3365)
@@ -0,0 +1,56 @@
+/**************************************************************************
+ *
+ * Weights.h -- Dealing with document weights
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef WEIGHTS_H
+#define WEIGHTS_H
+
+#include <stdio.h>
+
+
+class ApproxWeightsData {
+public:
+  unsigned char bits;
+  unsigned long mask;
+  double L;
+  double B;
+  unsigned long numLevelDocs;
+  unsigned long *weightBuf;
+  float *table;   // precomputed translation table
+  
+  ApproxWeightsData ();
+  ~ApproxWeightsData ();
+
+  void Free (); // frees any allocated memory
+  
+  bool Read (FILE *approxWeightsFile, unsigned long diskPtr,
+	     unsigned long _numLevelDocs);
+
+  float GetLowerApproxDocWeight (unsigned long levelDocNum);
+};
+
+
+// diskPtr points to the start of the exact document weights
+// for this level
+float GetExactDocWeight (FILE *exactWeightsFile, unsigned long diskPtr,
+			 unsigned long levelDocNum);
+
+
+#endif
Index: /trunk/indexers/mgpp/text/WordData.h
===================================================================
--- /trunk/indexers/mgpp/text/WordData.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/WordData.h	(revision 3365)
@@ -0,0 +1,47 @@
+/**************************************************************************
+ *
+ * WordData.h -- 
+ * Copyright (C) 1999  NZDL
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+class WordData
+{
+
+ public:
+  unsigned long docCount;
+  unsigned long wordCount;
+
+  inline void read(FILE *f)
+    {
+      fread((char *) &docCount, sizeof(docCount), 1, f);
+      fread((char *) &wordCount, sizeof(wordCount), 1, f);
+
+      NTOHUL(docCount);
+      NTOHUL(wordCount);
+    }
+
+  inline unsigned long documents()
+    {
+      return docCount;
+    }
+
+  inline int words()
+    {
+      return wordCount;
+    }
+};
Index: /trunk/indexers/mgpp/text/build.h
===================================================================
--- /trunk/indexers/mgpp/text/build.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/build.h	(revision 3365)
@@ -0,0 +1,86 @@
+/**************************************************************************
+ *
+ * build.h -- Global information for the passes of mg_passes
+ * Copyright (C) 1994  Neil Sharman, Alistair Moffat and Lachlan Andrew
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+
+#ifndef H_BUILD
+#define H_BUILD
+
+#include "longlong.h"
+
+#include "TextEl.h"
+#include "TagInfo.h"
+
+
+int init_special (const TagInfo &tagInfo, char *file_name);
+int init_text_1 (const TagInfo &tagInfo, char *file_name);
+int init_text_2 (const TagInfo &tagInfo, char *file_name);
+int init_ivf_1 (const TagInfo &tagInfo, char *file_name);
+int init_ivf_2 (const TagInfo &tagInfo, char *file_name);
+/*
+ * file_name  IN      The name of the dictionary file
+ * 
+ * returns      COMPALLOK for all ok,
+ *              COMPERROR for any error.  e.g. cannot read file
+ */
+
+
+
+
+
+int process_special (const TagInfo &tagInfo, const TextElArray &doc);
+int process_text_1 (const TagInfo &tagInfo, const TextElArray &doc);
+int process_text_2 (const TagInfo &tagInfo, const TextElArray &doc);
+int process_ivf_1 (const TagInfo &tagInfo, const TextElArray &doc);
+int process_ivf_2 (const TagInfo &tagInfo, const TextElArray &doc);
+/*
+ * returns      COMPALLOK for all ok,
+ *              COMPERROR for any error.  e.g. cannot read file
+ *
+ * The calling routine is responsible for ensuring that s_out is long 
+ * enough.
+ */
+
+
+
+
+
+int done_special (const TagInfo &tagInfo, char *filename);
+int done_text_1 (const TagInfo &tagInfo, char *filename);
+int done_text_2 (const TagInfo &tagInfo, char *filename);
+int done_ivf_1 (const TagInfo &tagInfo, char *filename);
+int done_ivf_2 (const TagInfo &tagInfo, char *filename);
+/*
+ * returns    COMPALLOK for all ok,
+ *              COMPERROR for any error.  e.g. cannot write file
+ */
+
+
+
+
+extern unsigned long invf_buffer_size;
+/*
+ * The amount of memory to allocate to the invertion buffer.
+ */
+
+
+
+#endif
Index: /trunk/indexers/mgpp/text/comp_dict.cpp
===================================================================
--- /trunk/indexers/mgpp/text/comp_dict.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/comp_dict.cpp	(revision 3365)
@@ -0,0 +1,229 @@
+/**************************************************************************
+ *
+ * comp_dict.cpp -- Functions for loading the compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "huffman.h"
+#include "local_strings.h"
+#include "memlib.h"
+#include "messages.h"
+
+#include "mg.h"
+#include "hash.h"
+#include "text.h"
+#include "comp_dict.h"
+#include "locallib.h"
+#include "mg_files.h"
+
+compression_dict_header cdh;
+compressed_text_header cth;
+comp_frags_header cfh[2];
+
+dict_hash_table *ht[2];
+
+huff_data char_huff[2];
+huff_data lens_huff[2];
+u_long *char_codes[2], *lens_codes[2];
+u_long Words_disk = 0;
+u_long Chars_disk = 0;
+
+
+static dict_hash_table *
+ReadInWords (FILE * dict, comp_frags_header * cfh,
+	     int esc)
+{
+  int i;
+  u_char *allwords, *prev = NULL;
+  dict_hash_table *ht;
+  u_char **words;
+  u_long ht_size;
+
+  ht_size = prime (cfh->hd.num_codes * HASH_RATIO);
+  if (!(ht = (dict_hash_table *) Xmalloc (sizeof (dict_hash_table) +
+		                            (ht_size - 1) * sizeof (ht->table[0]))))
+    {
+      Message ("no memory for hash_table\n");
+      return NULL;
+    }
+
+  ht->size = ht_size;
+  ht->hd = &cfh->hd;
+
+  if (!(ht->codes = Generate_Huffman_Codes (&cfh->hd, NULL)))
+    {
+      Message ("no memory for huffman codes\n");
+      return NULL;
+    }
+
+  if (!(ht->words = (u_char **) Xmalloc (sizeof (u_char *) * cfh->hd.num_codes)))
+    {
+      Message ("no memory for word pointers\n");
+      return NULL;
+    }
+  words = ht->words;
+
+  memset (ht->table, '\0', ht_size * sizeof (ht->table[0]));
+
+  if (!(allwords = (u_char *) Xmalloc (sizeof (u_char) * cfh->uncompressed_size)))
+    {
+      Message ("no memory for words\n");
+      return NULL;
+    }
+
+  for (i = 0; i < cfh->hd.num_codes; i++, words++)
+    {
+      register int val, copy;
+      val = fgetc (dict);
+      copy = (val >> 4) & 0xf;
+      val &= 0xf;
+
+      *words = allwords;
+
+      memcpy (allwords + 1, prev + 1, copy);
+      fread (allwords + copy + 1, sizeof (u_char), val, dict);
+      *allwords = val + copy;
+
+      Words_disk += val + 1;
+
+      /* insert into the hash table */
+      if (i < cfh->hd.num_codes - esc)
+	{
+	  register u_char **wptr;
+	  register unsigned long tsize = ht->size;
+	  register unsigned long hashval, step;
+
+	  HASH (hashval, step, allwords, tsize);
+
+	  wptr = ht->table[hashval];
+	  for (; wptr;)
+	    {
+	      hashval += step;
+	      if (hashval >= tsize)
+		hashval -= tsize;
+	      wptr = ht->table[hashval];
+	    }
+	  ht->table[hashval] = words;
+	}
+      prev = allwords;
+      allwords += *allwords + 1;
+    }
+  return ht;
+}
+
+inline int LoadWords(FILE *dict, int which)
+{
+  if (Read_cfh (dict, &cfh[which], NULL, &Words_disk) == -1)
+    return 1;
+  
+  if (!(ht[which] = ReadInWords (dict, &cfh[which], 0)))
+    return 1;
+
+  return 0;
+}
+
+inline int LoadHuffmanCodes(FILE *dict, int which)
+{
+  if (Read_Huffman_Data (dict, &char_huff[which], NULL,
+			 &Chars_disk) == -1)
+    return 1;
+  if (!(char_codes[which] =
+	Generate_Huffman_Codes (&char_huff[which], NULL)))
+    return 2;
+  if (Read_Huffman_Data (dict, &lens_huff[which], NULL,
+			 &Chars_disk) == -1)
+    return 3;
+  if (!(lens_codes[which] =
+	Generate_Huffman_Codes (&lens_huff[which], NULL)))
+    return 4;
+
+  return 0;
+}
+
+int 
+LoadCompressionDictionary (char *dict_file_name)
+{
+  FILE *dict;
+  int which;
+  if (!(dict = open_named_file (dict_file_name, "rb", MAGIC_DICT, MG_MESSAGE)))  /* [RPAP - Feb 97: WIN32 Port] */
+    return COMPERROR;
+
+  Words_disk = sizeof (u_long);
+
+  if (Read_cdh (dict, &cdh, NULL, &Words_disk) == -1)
+    goto error;
+
+  for (which = 0; which < 2; which++)
+    switch (cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  if (LoadWords(dict, which) != 0)
+	    goto error;
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (cdh.num_words[which])
+	    {
+	      if (LoadWords(dict, which) != 0)
+		goto error;
+	    }
+	  else
+	    ht[which] = NULL;
+	  if (LoadHuffmanCodes(dict, which) != 0)
+	    goto error;
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (cdh.num_words[which])
+	    {
+	      if (LoadWords(dict, which) != 0)
+		goto error;
+	    }
+	  else
+	    ht[which] = NULL;
+	  switch (cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      if (LoadHuffmanCodes(dict, which) != 0)
+		goto error;
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    default:
+	      FatalError (1, "Bad novel method");
+	    }
+	}
+	break;
+      default:
+	FatalError (1, "Bad dictionary kind\n");
+      }
+
+  return (COMPALLOK);
+
+
+error:
+  fclose (dict);
+  return (COMPERROR);
+}
Index: /trunk/indexers/mgpp/text/comp_dict.h
===================================================================
--- /trunk/indexers/mgpp/text/comp_dict.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/comp_dict.h	(revision 3365)
@@ -0,0 +1,48 @@
+/**************************************************************************
+ *
+ * comp_dict.h -- Functions for loading the compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define HASH_RATIO 2
+
+typedef struct dict_hash_table
+  {
+    u_long size;
+    huff_data *hd;
+    u_long *codes;
+    u_char **words;
+    u_char **table[1];
+  }
+dict_hash_table;
+
+extern compression_dict_header cdh;
+extern compressed_text_header cth;
+extern comp_frags_header cfh[2];
+
+extern dict_hash_table *ht[2];
+
+extern huff_data char_huff[2];
+extern huff_data lens_huff[2];
+extern u_long *char_codes[2], *lens_codes[2];
+extern u_long Words_disk;
+extern u_long Chars_disk;
+
+
+
+int LoadCompressionDictionary (char *dict_file_name);
Index: /trunk/indexers/mgpp/text/hash.h
===================================================================
--- /trunk/indexers/mgpp/text/hash.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/hash.h	(revision 3365)
@@ -0,0 +1,39 @@
+/**************************************************************************
+ *
+ * hash.h -- Hash function
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_HASH
+#define H_HASH
+
+#define HASH(h, step, wd, tablesize)			\
+	do {						\
+		register u_char *p=wd;			\
+		register long len = *p++;		\
+		register unsigned long hval;		\
+		if (len>25) len = 25;			\
+		for (hval=0; len; len--)		\
+		  hval = (hval << 5) - hval + *p++;	\
+		step = hval % (tablesize - 2) + 1;	\
+		h = hval % tablesize;			\
+	} while (0)
+
+
+
+#endif
Index: /trunk/indexers/mgpp/text/invf.cpp
===================================================================
--- /trunk/indexers/mgpp/text/invf.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/invf.cpp	(revision 3365)
@@ -0,0 +1,840 @@
+/**************************************************************************
+ *
+ * invf.cpp -- Data structures for inverted files
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "invf.h"
+#include "UCArray.h"
+
+
+invf_dict_header::invf_dict_header () {
+  Clear();
+}
+
+invf_dict_header::~invf_dict_header () {
+}
+
+void invf_dict_header::Clear() {
+  lookback = 0;
+  word_dict_start = 0;
+  word_dict_size = 0;
+  tag_dict_start = 0;
+  tag_dict_size = 0;
+  num_docs = 0;
+  num_frags = 0;
+  num_words = 0;
+  total_bytes = 0;
+  index_string_bytes = 0;
+  num_levels = 0;
+}
+
+bool invf_dict_header::Read (FILE *f) {
+  return (ReadUL (f, lookback) &&
+	  ReadUL (f, word_dict_start) &&
+	  ReadUL (f, word_dict_size) &&
+	  ReadUL (f, tag_dict_start) &&
+	  ReadUL (f, tag_dict_size) &&
+	  ReadUL (f, num_docs) &&
+	  ReadUL (f, num_frags) &&
+	  ReadUL (f, num_words) &&
+	  ReadUL (f, total_bytes) &&
+	  ReadUL (f, index_string_bytes) &&
+	  ReadUL (f, num_levels));
+}
+
+bool invf_dict_header::Write (FILE *f) const {
+  return (WriteUL (f, lookback) &&
+	  WriteUL (f, word_dict_start) &&
+	  WriteUL (f, word_dict_size) &&
+	  WriteUL (f, tag_dict_start) &&
+	  WriteUL (f, tag_dict_size) &&
+	  WriteUL (f, num_docs) &&
+	  WriteUL (f, num_frags) &&
+	  WriteUL (f, num_words) &&
+	  WriteUL (f, total_bytes) &&
+	  WriteUL (f, index_string_bytes) &&
+	  WriteUL (f, num_levels));
+}
+
+
+
+void dict_el::Clear () {
+  el.erase (el.begin(), el.end());
+  frag_occur = 0;
+  freq = 0;
+}
+
+bool dict_el::Read (FILE *f) {
+  return (ReadPreSufStr (f, el) &&
+	  ReadUL (f, frag_occur) &&
+	  ReadUL (f, freq));
+}
+
+bool dict_el::Write (FILE *f, const UCArray *lastEl) const {
+  return (WritePreSufStr (f, lastEl, el) &&
+	  WriteUL (f, frag_occur) &&
+	  WriteUL (f, freq));
+}
+
+
+void word_dict_el::Clear () {
+  dict_el::Clear();
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = NULL;
+}
+
+word_dict_el::~word_dict_el () {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+}
+
+void word_dict_el::SetNumLevels (unsigned long numLevels) {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = new unsigned long [numLevels];
+}
+  
+bool word_dict_el::Read (FILE *f, unsigned long numLevels) {
+  if (!dict_el::Read (f)) return false;
+
+  if (levelFreqs == NULL) return false;
+  
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!ReadUL (f, levelFreqs[i])) return false;
+  }
+
+  return true;
+}
+
+bool word_dict_el::Write (FILE *f, const UCArray *lastEl,
+			  unsigned long numLevels) const {
+  if (!dict_el::Write (f, lastEl)) return false;
+
+  if (levelFreqs == NULL) return false;
+
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!WriteUL (f, levelFreqs[i])) return false;
+  }
+
+  return true;
+}
+
+
+
+block_dict_header::block_dict_header () {
+  Clear();
+}
+
+void block_dict_header::Clear () {
+  invf_dict_header::Clear();
+
+  entries_per_wblk = 0;
+  num_wblks = 0;
+  max_wblk_size = 0;
+  wblk_start = 0;
+  wblk_idx_start = 0;
+  
+  entries_per_tblk = 0;
+  num_tblks = 0;
+  max_tblk_size = 0;
+  tblk_start = 0;
+  tblk_idx_start = 0;
+}
+
+bool block_dict_header::Read (FILE *f) {
+  return (invf_dict_header::Read (f) &&
+
+	  ReadUL (f, entries_per_wblk) &&
+	  ReadUL (f, num_wblks) &&
+	  ReadUL (f, max_wblk_size) &&
+	  ReadUL (f, wblk_start) &&
+	  ReadUL (f, wblk_idx_start) &&
+  
+	  ReadUL (f, entries_per_tblk) &&
+	  ReadUL (f, num_tblks) &&
+	  ReadUL (f, max_tblk_size) &&
+	  ReadUL (f, tblk_start) &&
+	  ReadUL (f, tblk_idx_start));
+}
+
+bool block_dict_header::Write (FILE *f) const {
+  return (invf_dict_header::Write (f) &&
+
+	  WriteUL (f, entries_per_wblk) &&
+	  WriteUL (f, num_wblks) &&
+	  WriteUL (f, max_wblk_size) &&
+	  WriteUL (f, wblk_start) &&
+	  WriteUL (f, wblk_idx_start) &&
+  
+	  WriteUL (f, entries_per_tblk) &&
+	  WriteUL (f, num_tblks) &&
+	  WriteUL (f, max_tblk_size) &&
+	  WriteUL (f, tblk_start) &&
+	  WriteUL (f, tblk_idx_start));
+}
+
+
+
+
+
+void block_dict_el::Clear () {
+  el.erase (el.begin(), el.end());
+  frag_occur = 0;
+  freq = 0;
+  invf_ptr = 0;
+}
+
+bool block_dict_el::Read (FILE *f) {
+  return (ReadPreSufStr (f, el) &&
+	  ReadUL (f, frag_occur) &&
+	  ReadUL (f, freq) &&
+	  ReadUL (f, invf_ptr));
+}
+
+bool block_dict_el::Write (FILE *f, const UCArray *lastEl) const {
+  return (WritePreSufStr (f, lastEl, el) &&
+	  WriteUL (f, frag_occur) &&
+	  WriteUL (f, freq) &&
+	  WriteUL (f, invf_ptr));
+}
+
+
+
+
+
+void word_block_dict_el::Clear () {
+  block_dict_el::Clear();
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = NULL;
+}
+
+word_block_dict_el::~word_block_dict_el () {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+}
+
+void word_block_dict_el::SetNumLevels (unsigned long numLevels) {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = new unsigned long [numLevels];
+}
+
+bool word_block_dict_el::Read (FILE *f, unsigned long numLevels) {
+  if (!block_dict_el::Read (f)) return false;
+  
+  if (levelFreqs == NULL) return false;
+  
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!ReadUL (f, levelFreqs[i])) return false;
+  }
+
+  return true;
+}
+
+bool word_block_dict_el::Write (FILE *f, const UCArray *lastEl,
+				unsigned long numLevels) const {
+  if (!block_dict_el::Write (f, lastEl)) return false;
+  
+  if (levelFreqs == NULL) return false;
+  
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!WriteUL (f, levelFreqs[i])) return false;
+  }
+  
+  return true;
+}
+
+
+
+
+block_idx_info::block_idx_info () {
+  Clear ();
+}
+
+void block_idx_info::Clear () {
+  el.erase (el.begin(), el.end());
+  block_ptr = 0;
+}
+
+bool block_idx_info::Read (FILE *f) {
+  return (ReadUCArray (f, el) &&
+	  ReadUL (f, block_ptr));
+}
+
+bool block_idx_info::Write (FILE *f) const {
+  return (WriteUCArray (f, el) &&
+	  WriteUL (f, block_ptr));
+}
+
+
+bool ReadBlockIdx (FILE *f, block_idx &blockIdx) {
+  blockIdx.erase (blockIdx.begin(), blockIdx.end());
+  
+  // read in the array size
+  unsigned long arraySize = 0;
+  if (!ReadVarLenUL (f, arraySize)) return false;
+  
+  // read in the array
+  block_idx_info bi;
+  while (arraySize > 0) {
+    if (!bi.Read (f)) return false;
+    blockIdx.push_back (bi);
+    
+    arraySize--;
+  }
+  
+  return true;
+}
+
+bool WriteBlockIdx (FILE *f, const block_idx &blockIdx) {
+  // write out the array size
+  if (!WriteVarLenUL (f, blockIdx.size())) return false;
+
+  block_idx::const_iterator here = blockIdx.begin();
+  block_idx::const_iterator end = blockIdx.end();
+  while (here != end) {
+    if (!(*here).Write (f)) return false;
+    here++;
+  }
+
+  return true;
+}
+
+
+
+
+stem_idx_header::stem_idx_header () {
+  Clear ();
+}
+
+void stem_idx_header::Clear () {
+  lookback = 0;
+  dict_size = 0;
+
+  entries_per_block = 0;
+  num_blocks = 0;
+  max_block_size = 0;
+  blocks_start = 0;
+  block_idx_start = 0;
+  
+  stemmer_num = 0;
+  stem_method = 0;
+}
+
+bool stem_idx_header::Read (FILE *f) {
+  return (ReadUL (f, lookback) &&
+	  ReadUL (f, dict_size) &&
+	  
+	  ReadUL (f, entries_per_block) &&
+	  ReadUL (f, num_blocks) &&
+	  ReadUL (f, max_block_size) &&
+	  ReadUL (f, blocks_start) &&
+	  ReadUL (f, block_idx_start) &&
+	  
+	  ReadUL (f, stemmer_num) &&
+	  ReadUL (f, stem_method));
+}
+
+bool stem_idx_header::Write (FILE *f) const {
+  return (WriteUL (f, lookback) &&
+	  WriteUL (f, dict_size) &&
+	  
+	  WriteUL (f, entries_per_block) &&
+	  WriteUL (f, num_blocks) &&
+	  WriteUL (f, max_block_size) &&
+	  WriteUL (f, blocks_start) &&
+	  WriteUL (f, block_idx_start) &&
+	  
+	  WriteUL (f, stemmer_num) &&
+	  WriteUL (f, stem_method));
+}
+
+
+
+stem_block_dict_el::stem_block_dict_el () {
+  Clear ();
+}
+
+void stem_block_dict_el::Clear () {
+  el.erase (el.begin(), el.end());
+  equivWords.erase (equivWords.begin(), equivWords.end());
+}
+
+bool stem_block_dict_el::Read (FILE *f) {
+  equivWords.erase (equivWords.begin(), equivWords.end());
+  
+  if (!ReadPreSufStr (f, el)) return false;
+  
+  // read in the array size
+  unsigned long arraySize = 0;
+  if (!ReadVarLenUL (f, arraySize)) return false;
+  
+  // read in the array
+  unsigned long wordNum;
+  while (arraySize > 0) {
+    if (!ReadUL (f, wordNum)) return false;
+    equivWords.push_back (wordNum);
+    
+    arraySize--;
+  }
+  
+  return true;
+}
+
+bool stem_block_dict_el::Write (FILE *f, const UCArray *lastEl) const {
+  if (!WritePreSufStr (f, lastEl, el)) return false;
+
+  // write out the array size
+  if (!WriteVarLenUL (f, equivWords.size())) return false;
+
+  vector<unsigned long>::const_iterator here = equivWords.begin();
+  vector<unsigned long>::const_iterator end = equivWords.end();
+  while (here != end) {
+    if (!WriteUL (f, (*here))) return false;
+    here++;
+  }
+
+  return true;
+}
+
+
+
+
+invf_file_header::invf_file_header () {
+  Clear ();
+}
+
+void invf_file_header::Clear () {
+  no_of_words = 0;
+  no_of_tags = 0;
+  skip_mode = SKIP_MODE_NO_SKIPS;
+  word_level_index = 0;
+  
+  int i;
+  for (i=0; i<16; i++) params[i] = 0;
+}
+  
+bool invf_file_header::Read (FILE *f) {
+  if (!ReadUL (f, no_of_words) ||
+      !ReadUL (f, no_of_tags) ||
+      !ReadUL (f, skip_mode) ||
+      !ReadUL (f, word_level_index)) return false;
+
+  int i;
+  for (i=0; i<16; i++) {
+    if (!ReadUL (f, params[i])) return false;
+  }
+
+  return true;
+}
+
+bool invf_file_header::Write (FILE *f) const {
+  if (!WriteUL (f, no_of_words) ||
+      !WriteUL (f, no_of_tags) ||
+      !WriteUL (f, skip_mode) ||
+      !WriteUL (f, word_level_index)) return false;
+
+  int i;
+  for (i=0; i<16; i++) {
+    if (!WriteUL (f, params[i])) return false;
+  }
+
+  return true;
+}
+
+
+
+
+
+bool SearchElNum (const block_idx &bIdx,
+		  unsigned long entriesPerBlock,
+		  unsigned long elNum,
+		  unsigned long &blockIdxNum,
+		  unsigned long &blockStartElNum) {
+  blockIdxNum = 0;
+  blockStartElNum = 0;
+
+  // make sure the element number isn't out of range
+  if (elNum > bIdx.size()*entriesPerBlock) return false;
+
+  blockIdxNum = elNum / entriesPerBlock;
+  blockStartElNum = blockIdxNum * entriesPerBlock;
+  
+  return true;
+}
+
+bool SearchEl (const block_idx &bIdx,
+	       unsigned long entriesPerBlock,
+	       const UCArray &el,
+	       unsigned long &blockIdxNum,
+	       unsigned long &blockStartElNum) {
+  blockIdxNum = 0;
+  blockStartElNum = 0;
+
+  unsigned long begin = 0;
+  unsigned long bIdxEnd = bIdx.size();
+  unsigned long end = bIdxEnd;
+  unsigned long mid;
+  while (begin < end) {
+    mid = (begin+end)/2;
+    if (DictCompare (el, bIdx[mid].el) < 0) {
+      end = mid;
+    } else if (mid+1 < bIdxEnd &&
+	       DictCompare (el, bIdx[mid+1].el) >= 0) {
+      begin = mid+1;
+    } else {
+      blockIdxNum = mid;
+      blockStartElNum = blockIdxNum * entriesPerBlock;
+      return true;
+    }
+  }
+  
+  return false;
+}
+
+
+
+// use the block dictionary functions for tag entries, and word block dict
+// functions for word entries.
+
+
+bool SearchBlockDictElNum (FILE *dictFile,
+			   const block_idx &bIdx,
+			   unsigned long entriesPerBlock,
+			   unsigned long dictSize,
+			   unsigned long elNum,
+			   block_dict_el &dictEl) {
+  UCArrayClear (dictEl.el);
+  if (elNum >= dictSize) return false;
+
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  do {
+    dictEl.Read (dictFile);
+  } while (curElNum++ < elNum);
+
+  return true;
+}
+
+bool SearchBlockDictEl (FILE *dictFile,
+			const block_idx &bIdx,
+			unsigned long entriesPerBlock,
+			unsigned long dictSize,
+			const UCArray &el,
+			block_dict_el &dictEl,
+			unsigned long &elNum) {
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile);
+    int res = DictCompare (dictEl.el, el);
+    if (res == 0) return true; // found it
+    else if (res > 0) break; // not here
+    
+    elNum++;
+  }
+
+  return false;
+}
+
+
+
+
+bool SearchWordBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long numLevels,
+			       unsigned long elNum,
+			       word_block_dict_el &dictEl) {
+  UCArrayClear (dictEl.el);
+  if (elNum >= dictSize) return false;
+
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  do {
+    dictEl.Read (dictFile, numLevels);
+  } while (curElNum++ < elNum);
+
+  return true;
+}
+
+bool SearchWordBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    unsigned long numLevels,
+			    const UCArray &el,
+			    word_block_dict_el &dictEl,
+			    unsigned long &elNum) {
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile, numLevels);
+    int res = DictCompare (dictEl.el, el);
+    if (res == 0) return true; // found it
+    else if (res > 0) break; // not here
+    
+    elNum++;
+  }
+
+  return false;
+}
+
+
+
+
+bool SearchStemBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long elNum,
+			       stem_block_dict_el &dictEl) {
+  UCArrayClear (dictEl.el);
+  if (elNum >= dictSize) return false;
+
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  do {
+    dictEl.Read (dictFile);
+  } while (curElNum++ < elNum);
+
+  return true;
+}
+
+bool SearchStemBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    const UCArray &el,
+			    stem_block_dict_el &dictEl,
+			    unsigned long &elNum) {
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile);
+    int res = DictCompare (dictEl.el, el);
+    if (res == 0) return true; // found it
+    else if (res > 0) break; // not here
+    
+    elNum++;
+  }
+
+  return false;
+}
+
+//----------------------------------------------------------------
+// functions for full text browse
+
+bool NearestSearchWordBlockDictEl (FILE *dictFile,
+				   const block_idx &bIdx,
+				   unsigned long entriesPerBlock,
+				   unsigned long dictSize,
+				   unsigned long numLevels,
+				   const UCArray &el,
+				   word_block_dict_el &dictEl,
+				   unsigned long &elNum) {
+
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile, numLevels);
+    int res = DictCompare (el, dictEl.el); // look for the first word that is
+                                     // greater or equal to the el
+    if (res <= 0) {
+      return true; // found one
+   } 
+    
+    elNum++;
+  }
+  // it must be the last term
+  return true;
+
+
+}
+
+
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    UCArrayVector &terms) {
+
+  word_block_dict_el dictEl;  
+  dictEl.SetNumLevels (numLevels);
+  UCArrayClear(dictEl.el);
+
+  terms.erase(terms.begin(), terms.end());
+
+  if (elNum >= dictSize) return false;
+  
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  unsigned long lastElNum = elNum + numWords - 1;
+  if (lastElNum > dictSize) lastElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  
+  // get the first term
+  do {
+    dictEl.Read (dictFile, numLevels);
+  } while (curElNum++ < elNum);
+
+  terms.push_back(dictEl.el);
+  while (curElNum <= lastElNum ) {
+    dictEl.Read(dictFile, numLevels);
+    terms.push_back(dictEl.el);
+    curElNum++;
+  }
+  
+
+  return true;
+}
+
+// NOte: before each addition of dictEl to the array, the level freqs array
+// is deleted, as this was causing problems - generating a seg fault, I think if
+// the vector had to be reallocated or something.
+// setNumLevels has to be called each time before a read, now, to set up the level
+//freqs array. this is necessary.
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    word_block_dict_el_array &terms) {
+
+  word_block_dict_el dictEl;
+  dictEl.SetNumLevels (numLevels);
+  UCArrayClear(dictEl.el);
+
+  block_dict_el elem;
+  terms.erase(terms.begin(), terms.end());
+
+  if (elNum >= dictSize) return false;
+  
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  unsigned long lastElNum = elNum + numWords - 1;
+  if (lastElNum > dictSize) lastElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  // get the first term
+  do {
+    dictEl.Read (dictFile, numLevels);
+  } while (curElNum++ < elNum);
+
+  dictEl.levelFreqs = NULL;
+  terms.push_back(dictEl);
+
+  while (curElNum <= lastElNum ) { 
+    dictEl.SetNumLevels(numLevels);
+    dictEl.Read(dictFile, numLevels);
+    dictEl.levelFreqs = NULL;
+    terms.push_back(dictEl);
+    curElNum++;
+  }
+
+  return true;
+}
+
+
+
+
+
+
+
+
+
Index: /trunk/indexers/mgpp/text/invf.h
===================================================================
--- /trunk/indexers/mgpp/text/invf.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/invf.h	(revision 3365)
@@ -0,0 +1,339 @@
+/**************************************************************************
+ *
+ * invf.h -- Data structures for inverted files
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ * $Id$
+ *
+ **************************************************************************/
+
+
+#ifndef H_INVF
+#define H_INVF
+
+#include <stdio.h>
+
+#include "UCArray.h"
+
+// NOTE: This does not include the magic number
+// header info for .invf.dict file
+struct invf_dict_header {
+  unsigned long lookback;
+  unsigned long word_dict_start;
+  unsigned long word_dict_size;
+  unsigned long tag_dict_start;
+  unsigned long tag_dict_size;
+  unsigned long num_docs;
+  unsigned long num_frags;
+  unsigned long num_words;
+  unsigned long total_bytes;
+  unsigned long index_string_bytes;
+  unsigned long num_levels;
+
+  invf_dict_header ();
+  virtual ~invf_dict_header ();
+  virtual void Clear();
+
+  virtual bool Read (FILE *f);
+  virtual bool Write (FILE *f) const;
+};
+
+
+struct dict_el {
+  UCArray el; // word or tag
+  unsigned long frag_occur;
+  unsigned long freq;
+
+  virtual void Clear ();
+  dict_el () { Clear (); }
+  virtual ~dict_el () { }
+
+  // Read assumes that the last word is in el
+  bool Read (FILE *f);
+  bool Write (FILE *f, const UCArray *lastEl) const;
+};
+
+
+struct word_dict_el : public dict_el {
+  unsigned long *levelFreqs;
+
+  void Clear ();
+  word_dict_el () { levelFreqs = NULL; Clear (); }
+  ~word_dict_el ();
+
+  void SetNumLevels (unsigned long numLevels);
+
+  // SetNumLevels should be called before either
+  // reading or writing using Read and Write
+  
+  // Read assumes that the last word is in el
+  bool Read (FILE *f, unsigned long numLevels);
+  bool Write (FILE *f, const UCArray *lastEl,
+	      unsigned long numLevels) const;
+};
+
+
+// wblk = word block
+// tblk = tag block
+// this version of the blocked dictionary uses a fixed number
+// of entries per block, not a fixed block size
+// info for .invf.dict.blocked file
+// blocked dict has a heap of blocks, some for words, some for tags
+// and an index into each set of blocks. The index has pointers to
+// the first entry in each block. Can do a binary search on the index
+// to find out which block an elemnet is in
+struct block_dict_header : public invf_dict_header {
+  // note: word_dict_start and tag_dict_start are undefined
+  // for blocked dictionaries
+
+  unsigned long entries_per_wblk; // word blocks
+  unsigned long num_wblks;
+  unsigned long max_wblk_size;
+  unsigned long wblk_start;
+  unsigned long wblk_idx_start;
+  
+  unsigned long entries_per_tblk; // tag blocks
+  unsigned long num_tblks;
+  unsigned long max_tblk_size;
+  unsigned long tblk_start;
+  unsigned long tblk_idx_start;
+  
+  block_dict_header ();
+  void Clear ();
+  
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+struct block_dict_el {
+  UCArray el; // word or tag
+  unsigned long frag_occur; // # entries in invf file - if have a
+  // word level index, this is the same as freq, otherwise, its the number
+  // of fragments containing this word
+  unsigned long freq; // # of times this word occurs
+  unsigned long invf_ptr; // pointer into inverted file
+
+  virtual void Clear ();
+  block_dict_el () { Clear (); }
+  virtual ~block_dict_el () { }
+
+  // Read assumes that the last word is in el
+  // set lastEl = NULL when no lookback is wanted (eg
+  // for the start of a block
+  bool Read (FILE *f);
+  bool Write (FILE *f, const UCArray *lastEl) const;
+};
+
+struct word_block_dict_el : public block_dict_el {
+  unsigned long *levelFreqs; // freq of the word at each level
+
+  void Clear ();
+  word_block_dict_el () { levelFreqs = NULL; Clear (); }
+  ~word_block_dict_el ();
+
+  void SetNumLevels (unsigned long numLevels);
+
+  // SetNumLevels should be called before either
+  // reading or writing using Read and Write
+  
+  // Read assumes that the last word is in el
+  bool Read (FILE *f, unsigned long numLevels);
+  bool Write (FILE *f, const UCArray *lastEl,
+	      unsigned long numLevels) const;
+};
+
+typedef vector<word_block_dict_el> word_block_dict_el_array;
+
+struct block_idx_info {
+  UCArray el;
+  unsigned long block_ptr;
+
+  block_idx_info ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// used for an index into the word and tag blocks
+typedef vector<block_idx_info> block_idx;
+
+bool ReadBlockIdx (FILE *f, block_idx &blockIdx);
+bool WriteBlockIdx (FILE *f, const block_idx &blockIdx);
+
+
+
+struct stem_idx_header {
+  unsigned long lookback;
+  unsigned long dict_size;
+
+  unsigned long entries_per_block;
+  unsigned long num_blocks;
+  unsigned long max_block_size;
+  unsigned long blocks_start;
+  unsigned long block_idx_start;
+  
+  unsigned long stemmer_num;
+  unsigned long stem_method;
+
+  stem_idx_header ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+struct stem_block_dict_el {
+  UCArray el; // word or tag
+  vector<unsigned long> equivWords;
+  
+  stem_block_dict_el ();
+  void Clear ();
+
+  // Read assumes that the last word is in el
+  // set lastEl = NULL when no lookback is wanted (eg
+  // for the start of a block
+  bool Read (FILE *f);
+  bool Write (FILE *f, const UCArray *lastEl) const;
+};
+
+
+
+#define SKIP_MODE_NO_SKIPS 0
+
+// invf file - has a list of frags for each word, but the word is not 
+//  stored in the invf file - the dictionaries store the words, along
+// with num entries, and a pointer into invf file
+struct invf_file_header {
+  unsigned long no_of_words;
+  unsigned long no_of_tags;
+  unsigned long skip_mode;
+  unsigned long word_level_index; // 1 if word level index
+  unsigned long params[16];
+
+  invf_file_header ();
+  void Clear ();
+  
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+
+
+
+
+// the search functions returns true if a block that could
+// satisfy the request is found. these functions assume that
+// the block index is sorted by DictCompare (or DictLTUCArray)
+bool SearchElNum (const block_idx &bIdx,
+		  unsigned long entriesPerBlock,
+		  unsigned long elNum,
+		  unsigned long &blockIdxNum,
+		  unsigned long &blockStartElNum);
+bool SearchEl (const block_idx &bIdx,
+	       unsigned long entriesPerBlock,
+	       const UCArray &el,
+	       unsigned long &blockIdxNum,
+	       unsigned long &blockStartElNum);
+
+
+// The next six functions use SearchElNum and SearchEl
+// for a particular type of dictionary (Block, WordBlock, or StemBlock)
+// and then look up the entry
+bool SearchBlockDictElNum (FILE *dictFile,
+			   const block_idx &bIdx,
+			   unsigned long entriesPerBlock,
+			   unsigned long dictSize,
+			   unsigned long elNum,
+			   block_dict_el &dictEl);
+bool SearchBlockDictEl (FILE *dictFile,
+			const block_idx &bIdx,
+			unsigned long entriesPerBlock,
+			unsigned long dictSize,
+			const UCArray &el,
+			block_dict_el &dictEl,
+			unsigned long &elNum);
+
+// assumes the numLevels has been set for dictEl
+bool SearchWordBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long numLevels,
+			       unsigned long elNum,
+			       word_block_dict_el &dictEl);
+bool SearchWordBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    unsigned long numLevels,
+			    const UCArray &el,
+			    word_block_dict_el &dictEl,
+			    unsigned long &elNum);
+
+bool SearchStemBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long elNum,
+			       stem_block_dict_el &dictEl);
+bool SearchStemBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    const UCArray &el,
+			    stem_block_dict_el &dictEl,
+			    unsigned long &elNum);
+
+//----------------------------------------------------------
+
+// new functions for full text browse
+
+bool NearestSearchWordBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    unsigned long numLevels,
+			    const UCArray &el,
+			    word_block_dict_el &dictEl,
+			    unsigned long &elNum);
+
+// returns a list of word_block_dict_el, with no levelfreqs
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    word_block_dict_el_array &terms);
+				    
+// just returns a list of terms
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    UCArrayVector &terms);
+				    
+
+
+#endif
Index: /trunk/indexers/mgpp/text/ivf.pass1.cpp
===================================================================
--- /trunk/indexers/mgpp/text/ivf.pass1.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/ivf.pass1.cpp	(revision 3365)
@@ -0,0 +1,775 @@
+/**************************************************************************
+ *
+ * ivf.pass1.cpp -- Memory efficient pass 1 inversion
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "mg_files.h"
+#include "invf.h"
+#include "mg.h"
+#include "build.h"
+#include "locallib.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+#include <stdio.h>
+#include "words.h"
+#include "messages.h"
+#include "netorder.h"
+#include "FIvfLevelInfo.h"
+
+#include "longlong.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+// a fragment corresponds to the index level (word-level is the
+// minimum index level)
+
+
+// structure to determine level information
+struct LevelWorker {
+  unsigned long lastLevelDocNum;
+  unsigned long count;
+
+  LevelWorker () {
+    lastLevelDocNum = count = 0;
+  }
+};
+
+
+// note: the word is stored in the map
+struct IvfWordInfo {
+  unsigned long wordCount;      // word frequency
+  unsigned long fragCount;      // number of fragments that contain the word
+  unsigned long lastFragNum;    // last fragment to contain the word
+  unsigned long chunkWordCount; // word frequency within this chunk
+  unsigned long chunkFragCount; // number of fragments within this chunk that
+                                //   contain the word
+
+  LevelWorker *levels;          // level info for this word
+  
+  IvfWordInfo ();
+  ~IvfWordInfo ();
+  void Clear ();                // will delete levels
+};
+
+typedef map<UCArray, IvfWordInfo, DictLTUCArray> IvfWordInfoMap;
+typedef vector<IvfWordInfoMap::iterator> IvfWordInfoItArray;
+
+// tags don't require as much information
+struct IvfTagInfo {
+  unsigned long tagCount;       // tag frequency
+  unsigned long fragCount;      // number of fragments that contain the tag
+  unsigned long lastFragNum;    // last fragment to contain the tag
+  unsigned long chunkFragCount; // number of fragments within this chunk that
+                                //   contain the tag
+
+  IvfTagInfo ();
+  void Clear ();
+};
+
+typedef map<UCArray, IvfTagInfo, DictLTUCArray> IvfTagInfoMap;
+typedef vector<IvfTagInfoMap::iterator> IvfTagInfoItArray;
+
+
+#define INIT_CHECK_FRAC 0.10
+#define CHECK_FRAC 0.75
+#define CHECK_CLOSE 0.999
+#define CHECK_DIV 1.5
+
+
+static FILE *ic;		// the invf chunk file
+static stdio_bitio_buffer icb;
+
+IvfWordInfoMap ivfWordInfo;
+IvfWordInfoItArray ivfWordInfoOccurOrder;
+IvfTagInfoMap ivfTagInfo;
+IvfTagInfoItArray ivfTagInfoOccurOrder;
+
+static unsigned long chunksWritten;
+static unsigned long maxMemNeeded;
+
+static unsigned long numDocs;
+static unsigned long numChunkDocs;
+
+static unsigned long numFrags;
+static unsigned long numChunkFrags;
+
+static unsigned long numWords;
+
+// the number of document numbers in the inverted file
+static unsigned long numChunkEntries;
+
+// next entry in the inverted file to check memory
+// requirements for the current chunk
+static unsigned long entryCheckPoint;
+
+// information about all the different levels
+static FIvfLevel ivfLevel;
+
+
+
+IvfWordInfo::IvfWordInfo () {
+  levels = NULL;
+  Clear();
+}
+
+IvfWordInfo::~IvfWordInfo () {
+  if (levels != NULL) delete [] levels;
+}
+
+void IvfWordInfo::Clear () {
+  wordCount = 0;
+  chunkWordCount = 0;
+  lastFragNum = 0;
+  fragCount = 0;
+  chunkFragCount = 0;
+
+  if (levels != NULL) {
+    delete [] levels;
+    levels = NULL;
+  }
+}
+
+
+IvfTagInfo::IvfTagInfo () {
+  Clear();
+}
+
+void IvfTagInfo::Clear () {
+  tagCount = 0;
+  lastFragNum = 0;
+  fragCount = 0;
+  chunkFragCount = 0;
+}
+
+
+
+int init_ivf_1 (const TagInfo &tagInfo, char *file_name) {
+  // set up the chunk file
+  if (!(ic = create_file (file_name, INVF_CHUNK_SUFFIX, "wb",
+			  MAGIC_CHUNK, MG_MESSAGE)))
+    return COMPERROR;
+  fwrite ("    ", sizeof (u_long), 1, ic);	// Space for the maxmem
+  icb.attachFile (ic);
+  icb.encodeStart();
+
+  // reset global variables
+  ivfWordInfo.erase (ivfWordInfo.begin(), ivfWordInfo.end());
+  ivfWordInfoOccurOrder.erase (ivfWordInfoOccurOrder.begin(), ivfWordInfoOccurOrder.end());
+  ivfTagInfo.erase (ivfTagInfo.begin(), ivfTagInfo.end());
+  ivfTagInfoOccurOrder.erase (ivfTagInfoOccurOrder.begin(), ivfTagInfoOccurOrder.end());
+  
+  chunksWritten = 0;
+  maxMemNeeded = 0;
+ 
+  numDocs = 0;
+  numChunkDocs = 0;
+
+  numFrags = 0;
+  numChunkFrags = 0;
+
+  numWords = 0;
+
+  numChunkEntries = 0;
+  entryCheckPoint = (unsigned long) ((invf_buffer_size * INIT_CHECK_FRAC) / CHECK_DIV);
+
+  // init the level information
+  ivfLevel.Clear();
+  ivfLevel.docTag = tagInfo.docTag;
+  ivfLevel.indexLevel = tagInfo.indexLevel;
+  IvfLevelInfo blankLevel;
+  UCArraySet::const_iterator levelHere = tagInfo.levelTags.begin();
+  UCArraySet::const_iterator levelEnd = tagInfo.levelTags.end();
+  while (levelHere != levelEnd) {
+    blankLevel.levelTag = *levelHere;
+    ivfLevel.levelInfo[*levelHere] = blankLevel;
+    levelHere++;
+  }
+  
+  return COMPALLOK;
+}
+
+static void ProcessOpenTag (const TagInfo &tagInfo, const TextEl &el,
+			    bool &inFrag) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  // check for start of next fragment
+  if (!wordLevelIndex && el.tagName == tagInfo.indexLevel) {
+    numFrags++;
+    numChunkFrags++;
+    inFrag = true;
+  }
+  
+  // update tag stats
+  IvfTagInfo &i = ivfTagInfo[el.tagName];
+  if (i.tagCount == 0) {
+    // new tag, add to list of iterators
+    IvfTagInfoMap::iterator iIt = ivfTagInfo.find (el.tagName);
+    ivfTagInfoOccurOrder.push_back (iIt);
+  }
+  i.tagCount++;
+
+  // all open tags count as new tags
+  numChunkEntries++;
+  i.fragCount++;
+  i.chunkFragCount++;
+  i.lastFragNum = numFrags;
+  
+  // update level information
+  IvfLevelInfoMap::iterator levelIt = ivfLevel.levelInfo.find (el.tagName);
+  if (levelIt != ivfLevel.levelInfo.end()) {
+    // is a level tag
+    (*levelIt).second.numEntries++;
+    (*levelIt).second.workInLevel = true;
+  }
+}
+
+static void ProcessCloseTag (const TagInfo &tagInfo, const TextEl &el,
+			     bool &inFrag) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  // check for end of fragment
+  if (!wordLevelIndex && el.tagName == tagInfo.indexLevel) {
+    inFrag = false;
+  }
+
+  // update level information
+  IvfLevelInfoMap::iterator levelIt = ivfLevel.levelInfo.find (el.tagName);
+  if (levelIt != ivfLevel.levelInfo.end()) {
+    // is a level tag
+    (*levelIt).second.workInLevel = false;
+  }
+}
+
+static void ProcessText (const TagInfo &tagInfo, const TextEl &el,
+			 bool &inFrag) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  // make sure this text is to be indexed
+  if (!wordLevelIndex && !inFrag) return;
+
+  const unsigned char *textHere = &(el.text[0]);
+  const unsigned char *textEnd = &(el.text[el.text.size() - 1]);
+  UCArray word;
+  
+  if (!inaword (textHere, textEnd))
+    ParseNonindexWord (textHere, textEnd);
+
+  
+  // Alternately parse off words and non-words from the input
+  // Each token is then inserted into the set if it does
+  // not exist or has it's frequency count incremented if it does.
+
+  while (textHere <= textEnd) {
+    textHere = ParseIndexWord (textHere, textEnd, word);
+    textHere = ParseNonindexWord (textHere, textEnd);
+
+    if (!word.empty()) {
+      numWords++;
+
+      if (wordLevelIndex) {
+	numFrags++;
+	numChunkFrags++;
+      }
+      
+      // update word stats
+
+      IvfWordInfo &i = ivfWordInfo[word];
+      if (i.wordCount == 0) {
+	// new word
+	// add to list of iterators
+	IvfWordInfoMap::iterator iIt = ivfWordInfo.find (word);
+	ivfWordInfoOccurOrder.push_back (iIt);
+
+	// add level information array
+	if (ivfLevel.levelInfo.size() > 0) 
+	  i.levels = new LevelWorker [ivfLevel.levelInfo.size()];
+      }
+
+      i.wordCount++;
+      i.chunkWordCount++;
+      if (numFrags > i.lastFragNum) {
+	numChunkEntries++;
+	i.fragCount++;
+	i.chunkFragCount++;
+	i.lastFragNum = numFrags;
+      }
+
+      // update level information for this word
+      if (i.levels != NULL) {
+	IvfLevelInfoMap::iterator levelHere = ivfLevel.levelInfo.begin();
+	IvfLevelInfoMap::iterator levelEnd = ivfLevel.levelInfo.end();
+	LevelWorker *levelWorkerPtr = i.levels;
+	while (levelHere != levelEnd) {
+
+	  // check to make sure the level encompases this fragment
+	  if (!(*levelHere).second.workInLevel) {
+	    cerr << "Level tag <" << (*levelHere).first
+		 << "> does not encompass all fragments\n";
+	    exit (1);
+	  }
+	  
+	  if ((*levelHere).second.numEntries > (*levelWorkerPtr).lastLevelDocNum) {
+	    (*levelWorkerPtr).lastLevelDocNum = (*levelHere).second.numEntries;
+	    (*levelWorkerPtr).count ++;
+	  }
+	  
+	  levelHere++;
+	  levelWorkerPtr++;
+	}
+      }
+    }
+  }
+}
+
+
+static unsigned long MemoryRequired (bool wordLevelIndex) {
+  register unsigned long total = 0;
+
+  // add memory required for word entries
+  IvfWordInfoMap::const_iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::const_iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    register const IvfWordInfo &info = (*wordHere).second;
+    if (info.chunkFragCount > 0) {
+      total += BIO_Bblock_Bound (numChunkFrags, info.chunkFragCount);
+      if (!wordLevelIndex) {
+	total += info.chunkWordCount;
+      }
+    }
+    
+    wordHere++;
+  }
+
+  // add memory required for tag entries
+  IvfTagInfoMap::const_iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::const_iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    register const IvfTagInfo &info = (*tagHere).second;
+    if (info.chunkFragCount > 0) {
+      // two d entries for each frag entry
+      unsigned long pTag = info.chunkFragCount*2;
+      total += BIO_Bblock_Bound (numChunkFrags+pTag, pTag);
+    }
+    
+    tagHere++;
+  }
+  
+  total = (total + 7) >> 3;
+  return total;
+}
+
+
+/*
+static void PrintChunkInfo (unsigned long mem) {
+  cout << "Chunk Number: " << chunksWritten << "\n";
+  cout << "numChunkDocs " << numChunkDocs << "\n";
+  cout << "numChunkFrags " << numChunkFrags << "\n";
+  cout << "mem " << mem << "\n";
+  cout << "numWords " << ivfWordInfo.size() << "\n";
+  cout << "numTags " << ivfTagInfo.size() << "\n\n";
+
+  // output debug tag information in dictionary order
+  IvfTagInfoMap::iterator tagMapHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagMapEnd = ivfTagInfo.end();
+  unsigned long tagNum = 0;
+  while (tagMapHere != tagMapEnd) {
+    cout << (*tagMapHere).first << " " << tagNum
+	 << " " << (*tagMapHere).second.chunkFragCount << "\n";
+    tagNum++;
+    tagMapHere++;
+  }
+}
+*/
+
+static void OutputChunkInfo (unsigned long mem, bool /*wordLevelIndex*/) {
+  chunksWritten++;
+
+  // sanity check
+  if (ivfWordInfo.size() != ivfWordInfoOccurOrder.size()) {
+    Message ("ERROR: Word information size mismatch: %u vs %u\n",
+	     (unsigned int)ivfWordInfo.size(),
+	     (unsigned int)ivfWordInfoOccurOrder.size());
+    exit (1);
+  }
+  if (ivfTagInfo.size() != ivfTagInfoOccurOrder.size()) {
+    Message ("ERROR: Tag information size mismatch: %u vs %u\n",
+	     (unsigned int)ivfTagInfo.size(),
+	     (unsigned int)ivfTagInfoOccurOrder.size());
+    exit (1);
+  }
+
+  icb.gamma_encode (numChunkDocs + 1, NULL);
+  icb.gamma_encode (numChunkFrags + 1, NULL);
+  icb.gamma_encode (mem + 1, NULL);
+  icb.gamma_encode (ivfWordInfo.size() + 1, NULL);
+  icb.gamma_encode (ivfTagInfo.size() + 1, NULL);
+
+  /*  PrintChunkInfo (mem);*/
+
+  // output word information in occurance order
+  IvfWordInfoItArray::iterator wordHere = ivfWordInfoOccurOrder.begin();
+  IvfWordInfoItArray::iterator wordEnd = ivfWordInfoOccurOrder.end();
+  while (wordHere != wordEnd) {
+    register IvfWordInfo &ivfWordInfo = (*(*wordHere)).second;
+    
+    icb.gamma_encode (ivfWordInfo.chunkWordCount + 1, NULL);
+    if (ivfWordInfo.chunkWordCount >= 2) {
+      icb.gamma_encode (ivfWordInfo.chunkFragCount, NULL);
+    }
+
+    ivfWordInfo.lastFragNum = 0;
+    ivfWordInfo.chunkWordCount = 0;
+    ivfWordInfo.chunkFragCount = 0;
+    
+    wordHere++;
+  }
+
+  // output tag information in occurance order
+  IvfTagInfoItArray::iterator tagHere = ivfTagInfoOccurOrder.begin();
+  IvfTagInfoItArray::iterator tagEnd = ivfTagInfoOccurOrder.end();
+  while (tagHere != tagEnd) {
+    register IvfTagInfo &ivfTagInfo = (*(*tagHere)).second;
+
+    icb.gamma_encode (ivfTagInfo.chunkFragCount + 1, NULL);
+
+    ivfTagInfo.lastFragNum = 0;
+    ivfTagInfo.chunkFragCount = 0;
+    
+    tagHere++;
+  }
+
+  numChunkDocs = 0;
+  numChunkFrags = 0;
+  numChunkEntries = 0;
+}
+
+
+int process_ivf_1 (const TagInfo &tagInfo, const TextElArray &doc) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+  bool inFrag = false;
+  if (wordLevelIndex) inFrag = true; // unconditional
+  
+  numDocs++;
+  numChunkDocs++;
+
+  // process each text element in this document
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    if ((*here).elType == OpenTagE)
+      ProcessOpenTag (tagInfo, *here, inFrag);
+    else if ((*here).elType == CloseTagE)
+      ProcessCloseTag (tagInfo, *here, inFrag);
+    else if ((*here).elType == TextE)
+      ProcessText (tagInfo, *here, inFrag);
+    
+    here++;
+  }
+
+  // check the amount of memory needed for this chunk
+  if (numChunkEntries >= entryCheckPoint) {
+    unsigned long mem = MemoryRequired (wordLevelIndex);
+    if (mem >= invf_buffer_size * CHECK_CLOSE) {
+      if (mem > maxMemNeeded) maxMemNeeded = mem;
+      OutputChunkInfo (mem, wordLevelIndex);
+      entryCheckPoint = (unsigned long)
+	((invf_buffer_size * INIT_CHECK_FRAC) / CHECK_DIV);
+      
+    } else {
+      entryCheckPoint = (unsigned long)
+	(entryCheckPoint * ((CHECK_FRAC * (invf_buffer_size - mem)) / mem) +
+	 entryCheckPoint);
+      if (entryCheckPoint <= numChunkEntries)
+	entryCheckPoint = numChunkEntries + 1;
+    }
+  }
+  
+  return COMPALLOK;
+}
+
+
+static void CalcInvfDictSize (unsigned long &totalBytes,
+			      unsigned long &indexStringBytes) {
+  totalBytes = 0;	// The sum of the length of all words, including 
+			// the length byte
+  indexStringBytes = 0; // The amount of space required to store the 
+			// words in the diction, this takes into account 
+			// the prefixes
+  const UCArray *lastWord = NULL;
+  
+  // calculate size of word information
+  IvfWordInfoMap::iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    unsigned long wordSize = (*wordHere).first.size();
+    totalBytes += wordSize + 1;
+    indexStringBytes += wordSize + 2;
+    if (lastWord != NULL)
+      indexStringBytes -= PrefixLen (*lastWord, (*wordHere).first);
+    lastWord = &((*wordHere).first);
+    
+    wordHere++;
+  }
+
+  // calculate size of tag information
+  lastWord = NULL;
+  IvfTagInfoMap::iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    unsigned long tagSize = (*tagHere).first.size();
+    totalBytes += tagSize + 1;
+    indexStringBytes += tagSize + 2;
+    if (lastWord != NULL)
+      indexStringBytes -= PrefixLen (*lastWord, (*tagHere).first);
+    lastWord = &((*tagHere).first);
+    
+    tagHere++;
+  }
+}
+
+
+//      OutputInvfDict ():
+//                writes out the stemmed dictionary file
+//                in the following format 
+//                        lookback value (int)
+//                        totalbytes value (int)
+//                        indexstringbytes (int)
+//                        for each word 
+//                          wordlen (4 bits)
+//                          prefix match (4 bits)
+//                          word (wordlen bytes)
+//                          word frequency (int)
+//                          word count (int)
+//
+//        Accesses outside variables:     
+//
+//        Return value...:                
+static void OutputInvfDict (char *filename) {
+  // create the dictionary header
+  invf_dict_header idh;
+  idh.word_dict_size = ivfWordInfo.size();
+  idh.tag_dict_size = ivfTagInfo.size();
+  idh.num_docs = numDocs;
+  idh.num_frags = numFrags;
+  idh.num_words = numWords;
+  idh.num_levels = ivfLevel.levelInfo.size();
+  CalcInvfDictSize (idh.total_bytes, idh.index_string_bytes);
+
+  // create the inverted dictionary file
+  FILE *sp;
+  if (!(sp = create_file (filename, INVF_DICT_SUFFIX, "wb", MAGIC_STEM_BUILD,
+			  MG_MESSAGE)))
+    return;
+
+  // write out the dictionary header
+  if (!idh.Write (sp)) { fclose (sp); return; }
+
+  // remember where the word dictionary starts
+  idh.word_dict_start = ftell (sp);
+  
+  // output the word dictionary
+  const UCArray *lastWord = NULL;
+  IvfWordInfoMap::iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    // get the prefix and suffix lengths
+    const UCArray &thisWord = (*wordHere).first;
+    WritePreSufStr (sp, lastWord, thisWord);
+
+    // output the number of fragments the word appeared in and the
+    // number of times the word appeared
+    WriteUL (sp, (*wordHere).second.fragCount);
+    WriteUL (sp, (*wordHere).second.wordCount);
+
+    // output frequency information for each level
+    // note that we are expecting every word to have
+    // level information
+    LevelWorker *lwHere = (*wordHere).second.levels;
+    LevelWorker *lwEnd = lwHere + idh.num_levels;
+    while (lwHere != lwEnd) {
+      WriteUL (sp, (*lwHere).count);
+      lwHere++;
+    }
+    
+    lastWord = &thisWord;
+    wordHere++;
+  }
+
+  // remember where the tag dictionary starts
+  idh.tag_dict_start = ftell (sp);
+  
+  // output the tag dictionary
+  const UCArray *lastTag = NULL;
+  IvfTagInfoMap::iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    // get the prefix and suffix lengths
+    const UCArray &thisTag = (*tagHere).first;
+    WritePreSufStr (sp, lastTag, thisTag);
+
+    // output the number of fragments the tag appeared in and the
+    // number of times the tag appeared
+    WriteUL (sp, (*tagHere).second.fragCount);
+    WriteUL (sp, (*tagHere).second.tagCount);
+    
+    lastTag = &thisTag;
+    tagHere++;
+  }
+
+  // write out the updated header
+  fseek (sp, sizeof (u_long), SEEK_SET);
+  if (!idh.Write (sp)) { fclose (sp); return; }
+  
+  fclose (sp);
+}
+
+static void OutputLevelFile (char *filename) {
+  // create the level file
+  FILE *f;
+  if (!(f = create_file (filename, INVF_LEVEL_SUFFIX, "wb", MAGIC_INVF_LEVELS,
+			 MG_MESSAGE)))
+    return;
+
+  // write out the information
+  ivfLevel.Write (f);
+  
+  // close the file
+  fclose (f);
+}
+
+static void OutputTransFile (char *filename) {
+  // save the word number (in lastFragNum :-/)
+  int i = 0;
+  IvfWordInfoMap::iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    (*wordHere).second.lastFragNum = i;
+    i++; wordHere++;
+  }
+
+  // save the tag number (in lastFragNum :-/)
+  i = 0;
+  IvfTagInfoMap::iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    (*tagHere).second.lastFragNum = i;
+    i++; tagHere++;
+  }
+  
+  // create the translation file
+  FILE *f;
+  if (!(f = create_file (filename, INVF_CHUNK_TRANS_SUFFIX, "wb",
+			 MAGIC_CHUNK_TRANS, MG_MESSAGE)))
+    return;
+
+  stdio_bitio_buffer buffer(f);
+  buffer.encodeStart();
+
+  // write out the word translation table
+  unsigned long wordDictSize = ivfWordInfoOccurOrder.size();
+  IvfWordInfoItArray::iterator wordItHere = ivfWordInfoOccurOrder.begin();
+  IvfWordInfoItArray::iterator wordItEnd = ivfWordInfoOccurOrder.end();
+  unsigned long oN = 0;
+  while (wordItHere != wordItEnd) {
+    register IvfWordInfo &ivfWordInfo = (*(*wordItHere)).second;
+    buffer.binary_encode (ivfWordInfo.lastFragNum + 1, wordDictSize + 1, NULL);
+    oN++;
+    wordItHere++;
+  }
+
+  // write out the tag translation table
+  unsigned long tagDictSize = ivfTagInfoOccurOrder.size();
+  IvfTagInfoItArray::iterator tagItHere = ivfTagInfoOccurOrder.begin();
+  IvfTagInfoItArray::iterator tagItEnd = ivfTagInfoOccurOrder.end();
+  while (tagItHere != tagItEnd) {
+    register IvfTagInfo &ivfTagInfo = (*(*tagItHere)).second;
+    buffer.binary_encode (ivfTagInfo.lastFragNum + 1, tagDictSize + 1, NULL);
+    oN++;
+    tagItHere++;
+  }
+
+  // finish encoding and close the file
+  buffer.encodeDone();
+  fclose (f);
+}
+
+#ifndef SILENT
+static void PrintStats () {
+  Message ("Inverted buffer size:          %8u bytes\n", invf_buffer_size);
+  Message ("Max memory needed for 1 chunk: %8u bytes\n", maxMemNeeded);
+
+  Message ("Number of chunks written:      %8u\n", chunksWritten);
+  Message ("Number of documents:           %8u\n", numDocs);
+  Message ("Number of fragments:           %8u\n", numFrags);
+  Message ("Number of words:               %8u\n", numWords);
+
+  Message ("Size of word dictionary:       %8u\n", ivfWordInfo.size());
+  Message ("Size of tag dictionary:        %8u\n", ivfTagInfo.size());
+}
+#endif
+
+int done_ivf_1 (const TagInfo &tagInfo, char * filename) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  char *temp_str = msg_prefix;
+  msg_prefix = "ivf.pass1";
+
+  // output the last chunk
+  if (numChunkDocs > 0) {
+    unsigned long mem = MemoryRequired (wordLevelIndex);
+    OutputChunkInfo (mem, wordLevelIndex);
+    if (mem > maxMemNeeded) maxMemNeeded = mem;
+  }
+  
+  // write out and encoded 1 to say there are no more chunks
+  icb.gamma_encode (1, NULL);
+  icb.encodeDone ();
+
+  // write out the maximum memory required and close the file
+  fseek (ic, sizeof (long), 0);
+  WriteUL (ic, maxMemNeeded);
+  fclose (ic);
+
+  // output the inverted dictionary
+  OutputInvfDict (filename);
+
+  // write out the translation file
+  OutputTransFile (filename);
+
+  // output the level information
+  OutputLevelFile (filename);
+  
+  // output statistics
+#ifndef SILENT
+  PrintStats ();
+#endif
+  
+  msg_prefix = temp_str;
+
+  return COMPALLOK;
+}
+
Index: /trunk/indexers/mgpp/text/ivf.pass2.cpp
===================================================================
--- /trunk/indexers/mgpp/text/ivf.pass2.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/ivf.pass2.cpp	(revision 3365)
@@ -0,0 +1,1111 @@
+/**************************************************************************
+ *
+ * ivf.pass2.cpp -- Memory efficient pass 2 inversion
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+#include <stdio.h>
+
+#if defined __WIN32__
+# include <process.h>
+# include <io.h>
+# define getpid _getpid
+# define unlink _unlink
+#else
+# include <unistd.h>
+# include "non_ansi.h"
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "mg_files.h"
+#include "invf.h"
+#include "mg.h"
+#include "build.h"
+#include "locallib.h"
+#include "bitio_m_random.h"
+#include "bitio_m_stdio.h"
+#include "bitio_m_mems.h"
+#include "bitio_gen.h"
+#include <stdio.h>
+#include "words.h"
+#include "messages.h"
+#include "netorder.h"
+#include "FIvfLevelInfo.h"
+#include "perf_hash.h"
+#include "string.h"
+
+#include "longlong.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+
+#ifdef USE_LONG_LONG
+#define SEEK_X seek_LL
+#define TELL_X tell_LL
+#else
+#define SEEK_X seek
+#define TELL_X tell
+#endif
+
+#ifndef RND_BUF_SIZE
+#define RND_BUF_SIZE 8*1024
+#endif
+
+
+static unsigned long numDocs = 0;
+static unsigned long numChunkDocs = 0;
+static unsigned long numDocsInChunk = 0;
+
+static unsigned long numFrags = 0;
+static unsigned long numFragsInChunk = 0;
+static unsigned long chunkStartFragNum = 0;
+
+
+
+
+struct BitPtr {
+  unsigned long start;
+  unsigned long here;
+  unsigned long lastFragNum;
+  unsigned long lgB;
+  
+  void Clear () { start = here = lastFragNum = lgB = 0; }
+  BitPtr () { Clear(); }
+};
+
+class WordBitPtrs {
+protected:
+  unsigned long numWords;
+  unsigned long numTags;
+  unsigned long size;
+  BitPtr *wordBitPtrs;
+
+  void CheckBufOverrun (unsigned long num) {
+    if (wordBitPtrs[num].here > wordBitPtrs[num+1].start) {
+      cerr << "numDocs: " << numDocs << "\n";
+      cerr << "numChunkDocs: " << numChunkDocs << "\n";
+      cerr << "numDocsInChunk: " << numDocsInChunk << "\n";
+      cerr << "numFrags: " << numFrags << "\n";
+      cerr << "numFragsInChunk: " << numFragsInChunk << "\n";
+      cerr << "chunkStartFragNum: " << chunkStartFragNum << "\n";
+      cerr << "num: " << num << "\n";
+      cerr << "[num].start: " << wordBitPtrs[num].start << "\n";
+      cerr << "[num].here: " << wordBitPtrs[num].here << "\n";
+      cerr << "[num+1].start: " << wordBitPtrs[num+1].start << "\n";
+      FatalError (1, "Bit buffer overrun");
+    }
+  }
+  
+public:
+  void Clear ();
+  WordBitPtrs () { wordBitPtrs = NULL; Clear(); }
+  ~WordBitPtrs ();
+  void SetSize (unsigned long _numWords,
+		unsigned long _numTags);
+
+  void ResetPtrs () {
+    if (wordBitPtrs == NULL) return;
+    unsigned long i;
+    for (i=0; i<size; i++) wordBitPtrs[i].Clear();
+  }
+  
+  BitPtr &GetWordBitPtr (unsigned long wordNum)
+    { return wordBitPtrs[wordNum]; }
+  unsigned long &GetWordStart (unsigned long wordNum)
+    { return wordBitPtrs[wordNum].start; }
+  unsigned long &GetWordHere (unsigned long wordNum)
+    { return wordBitPtrs[wordNum].here; }
+  void CheckWordBufOverrun (unsigned long wordNum)
+    { CheckBufOverrun (wordNum); }
+
+  BitPtr &GetTagBitPtr (unsigned long tagNum)
+    { return wordBitPtrs[tagNum + numWords]; }
+  unsigned long &GetTagStart (unsigned long tagNum)
+    { return wordBitPtrs[tagNum + numWords].start; }
+  unsigned long &GetTagHere (unsigned long tagNum)
+    { return wordBitPtrs[tagNum + numWords].here; }
+  void CheckTagBufOverrun (unsigned long tagNum)
+    { CheckBufOverrun (tagNum + numWords); }
+
+  BitPtr &GetEndBitPtr ()
+    { return wordBitPtrs[size-1]; }
+  unsigned long &GetEndStart ()
+    { return wordBitPtrs[size-1].start; }
+  unsigned long &GetEndHere ()
+    { return wordBitPtrs[size-1].here; }
+};
+
+
+struct IP2TagInfo {
+  bool inTag;
+  unsigned long startFrag;
+  unsigned long tagNum;
+
+  IP2TagInfo () {
+    inTag = false;
+    startFrag = 0;
+    tagNum = 0;
+  }
+};
+
+// maps tags to tag information
+typedef map<UCArray, IP2TagInfo, DictLTUCArray> TagMapDict;
+
+
+// class to handle the translation of occurrence order
+// to dictionary order for words and tags
+class OccurToDictConverter {
+protected:
+  unsigned long pos;
+  unsigned long val;
+  FILE *transFile;
+  random_bitio_buffer rbs;
+
+  unsigned long wordDictSize;
+  unsigned long tagDictSize;
+
+  void SeekStart ();
+  unsigned long TranslateNum (unsigned long num);
+  
+public:
+  OccurToDictConverter ();
+  ~OccurToDictConverter ();
+
+  void Open (char *filename, unsigned long _wordDictSize,
+	     unsigned long _tagDictSize);
+
+  // Close frees all allocated memory
+  void Close ();
+
+  unsigned long TranslateWord (unsigned long occurNum)
+    { return TranslateNum (occurNum); }
+  unsigned long TranslateTag (unsigned long occurNum)
+    { return TranslateNum (occurNum+wordDictSize); }
+};
+
+
+struct InvfStateRec {
+  mg_ullong start;
+  mg_ullong here;
+  unsigned long lastFragNum;
+  unsigned long B;
+
+  void Clear () {
+    start = here = 0;
+    lastFragNum = B = 0;
+  }
+  InvfStateRec () { Clear (); }
+};
+
+
+#define ISR_SIZE 1024
+
+class InvfStateCache {
+protected:
+  InvfStateRec recCache [ISR_SIZE];
+  unsigned long startNum;
+
+  FILE *stateFile;
+
+  void ClearCache () {
+    unsigned int i = 0;
+    for (i=0; i<ISR_SIZE; i++) recCache[i].Clear();
+  }
+  
+public:
+  InvfStateCache ();
+  ~InvfStateCache ();
+  
+  void Open (char *filename);
+  void Close ();
+
+  // previous references to state records may be
+  // invalidated calling GetRec
+  InvfStateRec &GetRec (unsigned long num);
+};
+
+
+static invf_dict_header idh;
+static WordBitPtrs bitPtrs;
+
+static FILE *chunkFile = NULL;
+static stdio_bitio_buffer chunkBuf;
+
+static unsigned long ivfMemBufSize = 0;
+static char *ivfMemBuf = NULL;
+
+// word and tag dictionaries. a map is used for the tag dictionary
+// as it should never be very big (and the perfect hash function
+// sometimes has trouble with small values).
+static perf_hash_data *wordHashDict = NULL;
+static TagMapDict tagMapDict;
+
+// information about all the different levels
+static FIvfLevel ivfLevel;
+
+static OccurToDictConverter occurConvert;
+
+// information about the state of the inverted file
+static InvfStateCache invfState;
+
+static char collectFilename[512];
+
+
+void WordBitPtrs::Clear () {
+  numWords = 0;
+  numTags = 0;
+  size=0;
+  if (wordBitPtrs != NULL) delete [] wordBitPtrs;
+  wordBitPtrs = NULL;
+}
+
+WordBitPtrs::~WordBitPtrs () {
+  if (wordBitPtrs != NULL) delete [] wordBitPtrs;
+}
+
+void WordBitPtrs::SetSize (unsigned long _numWords,
+			   unsigned long _numTags){
+  Clear();
+  numWords = _numWords;
+  numTags = _numTags;
+  size = numWords + numTags + 1;
+  wordBitPtrs = new BitPtr [size];
+}
+
+
+void OccurToDictConverter::SeekStart () {
+  if (transFile == NULL) return;
+  rbs.SEEK_X (sizeof (unsigned long) * 8);
+  pos = 0;
+}
+
+unsigned long OccurToDictConverter::TranslateNum (unsigned long num) {
+  if (num < pos) SeekStart ();
+  while (pos <= num) {
+    if (pos < wordDictSize)
+      val = rbs.binary_decode (wordDictSize + 1, NULL) - 1;
+    else 
+      val = rbs.binary_decode (tagDictSize + 1, NULL) - 1;
+    pos++;
+  }
+  return val;
+}
+
+OccurToDictConverter::OccurToDictConverter () {
+  pos = 0;
+  val = 0;
+  transFile = NULL;
+  wordDictSize = 0;
+  tagDictSize = 0;
+}
+
+OccurToDictConverter::~OccurToDictConverter () {
+  if (transFile != NULL) Close ();
+}
+
+void OccurToDictConverter::Open (char *filename, unsigned long _wordDictSize,
+				 unsigned long _tagDictSize) {
+  if (transFile != NULL) Close ();
+
+  wordDictSize = _wordDictSize;
+  tagDictSize = _tagDictSize;
+
+  transFile = open_file (filename, INVF_CHUNK_TRANS_SUFFIX, "rb",
+			 MAGIC_CHUNK_TRANS, MG_ABORT);
+  rbs.attachFile (transFile, RND_BUF_SIZE);
+  SeekStart ();
+  val = 0;
+}
+
+void OccurToDictConverter::Close () {
+  if (transFile == NULL) return;
+
+  rbs.done ();
+  fclose (transFile);
+  transFile = NULL;
+  pos = 0;
+  val = 0;
+
+  wordDictSize = 0;
+  tagDictSize = 0;
+}
+
+
+
+
+InvfStateCache::InvfStateCache () {
+  startNum = 0;
+  stateFile = NULL;
+}
+
+InvfStateCache::~InvfStateCache () {
+  if (stateFile != NULL) Close ();
+}
+  
+void InvfStateCache::Open (char *filename) {
+  if (stateFile != NULL) Close();
+
+  // open the state file
+  char path[512];
+  sprintf (path, FILE_NAME_FORMAT ".%ld", get_basepath (), filename,
+           ".invf.state", (long) getpid ());
+  if (!(stateFile = fopen (path, "wb+"))) {
+    Message ("Unable to create \"%s\"", path);
+    exit (1);
+  }
+  unlink (path); // file will be deleted after it is closed
+  // reset the buffer
+  startNum = 0;
+  ClearCache();
+}
+
+void InvfStateCache::Close () {
+  if (stateFile == NULL) return;
+  fclose (stateFile);
+  stateFile = NULL;
+  startNum = 0;
+}
+
+InvfStateRec &InvfStateCache::GetRec (unsigned long num) {
+  // see if cached
+  if ((num >= startNum) && (num < startNum + ISR_SIZE))
+    return recCache[num-startNum];
+
+  // not cached, write out this lot of records and read in
+  fseek (stateFile, startNum*sizeof (InvfStateRec), SEEK_SET);
+  fwrite ((char *) recCache, sizeof (InvfStateRec), ISR_SIZE, stateFile);
+
+  // read in the new set of records
+  ClearCache ();
+  startNum = num - (num % ISR_SIZE);
+  fseek (stateFile, startNum*sizeof (InvfStateRec), SEEK_SET);
+  fread ((char *) recCache, sizeof (InvfStateRec), ISR_SIZE, stateFile);
+
+  return recCache[num-startNum];
+}
+
+
+
+static void ClearCharBuf (char *buf, unsigned long size) {
+  char *end = buf + size;
+  while (buf != end) *buf++ = 0;
+}
+
+static void ReadWordDict (char *filename) {
+  // read in the perfect hash function for the word dictionary
+  FILE *wordHashFile = open_file (filename, INVF_DICT_HASH_SUFFIX, "rb",
+				  MAGIC_HASH, MG_ABORT);
+  if (!(wordHashDict = read_perf_hash_data (wordHashFile))) {
+    FatalError (1, "Unable to read in hash data for word dictionary");
+  }
+  fclose (wordHashFile);
+}
+
+static void ReadTagDict (char *filename, invf_dict_header &_idh) {
+  // open the file
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+
+  // seek to the start of the tag dictionary
+  fseek (dictFile, _idh.tag_dict_start, SEEK_SET);
+  
+  unsigned long tagNum;
+  dict_el thisEl;
+  for (tagNum = 0; tagNum < _idh.tag_dict_size; tagNum++) {
+    thisEl.Read (dictFile);
+    tagMapDict[thisEl.el].tagNum = tagNum;
+  }
+  
+  fclose (dictFile);
+}
+
+static void ReadLevelFile (char *filename) {
+  FILE *f;
+  f = open_file (filename, INVF_LEVEL_SUFFIX, "rb",
+		 MAGIC_INVF_LEVELS, MG_ABORT);
+  ivfLevel.Read (f);
+  fclose (f);
+}
+
+void CheckIntOverflow (mg_ullong totalIBits, mg_ullong lastTotalIBits) {
+  if (totalIBits < lastTotalIBits) {
+    fprintf(stderr, "ERROR: The totalIBits counter (%d byte unsigned integer) has overflowed.\n", sizeof (mg_ullong));
+    if (sizeof (mg_ullong) < 8) {
+      fprintf(stderr, "       Try compiling with GCC to enable use of 8 bytes for this counter.\n");
+    }
+    fprintf(stderr, "       Build aborted.\n");
+    exit(1);
+  }
+}
+
+// assumes the inverted file state file has been opened
+static void InitInvfState (char *filename,
+			   invf_dict_header &_idh,
+			   InvfStateCache &_invfState,
+			   mg_ullong &totalIBits,
+			   bool wordLevelIndex) {
+  // read in the dictionary, setting inverted state information
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+
+  // seek to the start of the word dictionary
+  fseek (dictFile, _idh.word_dict_start, SEEK_SET);
+
+  // add the word entries
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (_idh.num_levels);
+  unsigned long dictWordNum, p;
+  mg_ullong lastTotalIBits;
+  unsigned long N = _idh.num_frags;
+  for (dictWordNum=0; dictWordNum<_idh.word_dict_size; dictWordNum++) {
+    // lastTotalIBits is used to detect integer overflow
+    lastTotalIBits = totalIBits;
+
+    // read the next word and associated information
+    wordEl.Read (dictFile, _idh.num_levels);
+
+    // update the state record
+    p = wordEl.frag_occur;
+    InvfStateRec &wisr = _invfState.GetRec (dictWordNum);
+    wisr.start = totalIBits;
+    wisr.here = totalIBits;
+    wisr.B = BIO_Bblock_Init (N, p);
+    
+    // add the length of the fragment numbers
+    totalIBits += BIO_Bblock_Bound_b (N, p, wisr.B);
+
+    // if needed, add the length of the fragment frequency information
+    if (!wordLevelIndex)
+      totalIBits += BIO_Gamma_Bound (wordEl.freq, wordEl.frag_occur);
+
+    // align next byte
+#ifdef USE_LONG_LONG
+    totalIBits = (totalIBits + 7ull) & 0xfffffffffffffff8ull;
+#else
+    totalIBits = (totalIBits + 7ul) & 0xfffffff8ul;
+#endif
+
+    CheckIntOverflow (totalIBits, lastTotalIBits);
+  }
+
+  
+  // seek to the start of the tag dictionary
+  fseek (dictFile, _idh.tag_dict_start, SEEK_SET);
+
+  // add the tag entries
+  dict_el tagEl;
+  unsigned long dictTagNum;
+  N = _idh.num_frags;
+  for (dictTagNum=0; dictTagNum<_idh.tag_dict_size; dictTagNum++) {
+    // lastTotalIBits is used to detect integer overflow
+    lastTotalIBits = totalIBits;
+
+    // read the next tag and associated information
+    tagEl.Read (dictFile);
+
+    // update the state record
+    p = tagEl.frag_occur*2;
+    InvfStateRec &tisr = _invfState.GetRec (dictTagNum + _idh.word_dict_size);
+    tisr.start = totalIBits;
+    tisr.here = totalIBits;
+    tisr.B = BIO_Bblock_Init (N+p, p);
+    
+    // add the length of the fragment numbers (two numbers for each
+    // tag, one for start and one for end)
+    totalIBits += BIO_Bblock_Bound_b (N+p, p, tisr.B);
+
+    // align next byte
+#ifdef USE_LONG_LONG
+    totalIBits = (totalIBits + 7ull) & 0xfffffffffffffff8ull;
+#else
+    totalIBits = (totalIBits + 7ul) & 0xfffffff8ul;
+#endif
+
+    CheckIntOverflow (totalIBits, lastTotalIBits);
+  }
+
+  fclose (dictFile);
+}
+
+/*
+// assumes the chunk tag information has been placed in .first
+static void PrintChunkInfo (unsigned long chunkMem,
+			    unsigned long numChunkWords,
+			    unsigned long numChunkTags) {
+  static unsigned long chunksRead = 0;
+  chunksRead++;
+  cout << "Chunk Number: " << chunksRead << "\n";
+  cout << "numChunkDocs " << numDocsInChunk << "\n";
+  cout << "numChunkFrags " << numFragsInChunk << "\n";
+  cout << "mem " << chunkMem << "\n";
+  cout << "numWords " << numChunkWords << "\n";
+  cout << "numTags " << numChunkTags << "\n\n";
+
+  TagMapDict::iterator tagMapHere = tagMapDict.begin();
+  TagMapDict::iterator tagMapEnd = tagMapDict.end();
+  while (tagMapHere != tagMapEnd) {
+    unsigned long tagMapNum = (*tagMapHere).second.tagNum;
+    cout << (*tagMapHere).first << " " << tagMapNum << " "
+	 << bitPtrs.GetTagBitPtr(tagMapNum).here << "\n";
+    tagMapHere++;
+  }
+}
+*/
+
+void ReadChunk (invf_dict_header &_idh, bool wordLevelIndex) {
+  // reset globals
+  numChunkDocs = 0;
+  chunkStartFragNum = numFrags;
+
+  // read in information about this chunk
+  numDocsInChunk = chunkBuf.gamma_decode (NULL) - 1;
+  if (numDocsInChunk == 0)
+    FatalError (1, "The number of docs in the current chunk is 0");
+
+  numFragsInChunk = chunkBuf.gamma_decode (NULL) - 1;
+  unsigned long chunkMem = chunkBuf.gamma_decode (NULL) - 1;
+
+  if (chunkMem > ivfMemBufSize)
+    FatalError (1, "Chunk memory size is greater than maximum");
+  
+  unsigned long numChunkWords = chunkBuf.gamma_decode (NULL) - 1;
+  unsigned long numChunkTags = chunkBuf.gamma_decode (NULL) - 1;
+
+  
+  // reset stuff
+  ClearCharBuf (ivfMemBuf, ivfMemBufSize);
+  bitPtrs.ResetPtrs();  
+
+  // read in the entries in occurrence order storing the
+  // "chunkWordCount" in "start" and the "chunkFragCount"
+  // in "here"
+  unsigned long numOccur;
+  unsigned long wordNum;
+  for (numOccur=0; numOccur<numChunkWords; numOccur++) {
+    wordNum = occurConvert.TranslateWord (numOccur);
+    BitPtr &wordPtr = bitPtrs.GetWordBitPtr (wordNum);
+    wordPtr.start = chunkBuf.gamma_decode (NULL) - 1;
+    if (wordPtr.start >= 2)
+      wordPtr.here = chunkBuf.gamma_decode (NULL);
+    else wordPtr.here = wordPtr.start;
+  }
+  unsigned long tagNum;
+  for (numOccur=0; numOccur<numChunkTags; numOccur++) {
+    tagNum = occurConvert.TranslateTag (numOccur);
+    BitPtr &tagPtr = bitPtrs.GetTagBitPtr (tagNum);
+    // only chunkFragCount is encoded for tags
+    tagPtr.start = chunkBuf.gamma_decode (NULL) - 1;
+    tagPtr.here = tagPtr.start;
+  }
+
+  /*  PrintChunkInfo (chunkMem, numChunkWords, numChunkTags);*/
+  
+  // create the bit ptrs in dictionary order
+  unsigned long totalIBits = 0; // only dealing with memory
+  unsigned long chunkWordCount, chunkFragCount;
+  for (wordNum=0; wordNum<_idh.word_dict_size; wordNum++) {
+    BitPtr &wordPtr = bitPtrs.GetWordBitPtr (wordNum);
+    chunkWordCount = wordPtr.start;
+    chunkFragCount = wordPtr.here;
+    wordPtr.start = totalIBits;
+    wordPtr.here = totalIBits;
+    wordPtr.lastFragNum = chunkStartFragNum;
+    wordPtr.lgB = 0;
+    if (chunkWordCount > 0) {
+      wordPtr.lgB = floorlog_2 (BIO_Bblock_Init_W (numFragsInChunk,
+						   chunkFragCount));
+      totalIBits += BIO_Bblock_Bound (numFragsInChunk, chunkFragCount);
+      // use unary encoding for memory buffer encoding of fragment freq
+      if (!wordLevelIndex) {
+	totalIBits += chunkWordCount;
+      }
+    }
+  }
+  for (tagNum=0; tagNum<_idh.tag_dict_size; tagNum++) {
+    BitPtr &tagPtr = bitPtrs.GetTagBitPtr (tagNum);
+    chunkFragCount = tagPtr.here;
+    tagPtr.start = totalIBits;
+    tagPtr.here = totalIBits;
+    tagPtr.lastFragNum = chunkStartFragNum;
+    tagPtr.lgB = 0;
+    if (chunkFragCount > 0) {
+      unsigned long pTag = chunkFragCount*2;
+      tagPtr.lgB = floorlog_2 (BIO_Bblock_Init_W (numFragsInChunk+pTag,
+						  pTag));
+      unsigned long bLen = BIO_Bblock_Bound (numFragsInChunk+pTag,
+					     pTag);
+//        cout << tagNum + _idh.word_dict_size << " ";
+//        cout << "numFrags: " << numFragsInChunk
+//    	   << " chunkFragCount: " << chunkFragCount
+//    	   << " B: " << 1 << tagPtr.lgB
+//    	   << " blen: " << blen << "\n";
+      totalIBits += bLen;
+    }
+  }
+  bitPtrs.GetEndStart() = totalIBits;
+  bitPtrs.GetEndHere() = totalIBits;
+
+  if ((totalIBits + 7ul) >> 3ul > chunkMem) {
+    cerr << "totalIBits: " << totalIBits << "\n";
+    cerr << "bytes: " << ((totalIBits + 7ul) >> 3ul) << "\n";
+    cerr << "chunkMem: " << chunkMem << "\n";
+    FatalError (1, "Pointers exceed buffer size");
+  }
+}
+
+
+
+
+int init_ivf_2 (const TagInfo &/*tagInfo*/, char *filename) {
+  // read in compressed dictionary header
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  idh.Read (dictFile);
+  fclose (dictFile);
+
+  // set the size of the bit ptrs
+  bitPtrs.SetSize (idh.word_dict_size, idh.tag_dict_size);
+
+  // open the chunk file and read in the maximum memory needed
+  // for the inverted memory buffer
+  chunkFile = open_file (filename, INVF_CHUNK_SUFFIX, "rb",
+			 MAGIC_CHUNK, MG_ABORT);
+  ReadUL (chunkFile, ivfMemBufSize);
+  chunkBuf.attachFile (chunkFile);
+
+  // allocate memory for the inverted buffer
+  ivfMemBuf = new char [ivfMemBufSize];
+  ClearCharBuf (ivfMemBuf, ivfMemBufSize);
+
+  // read in the word dictionary
+  ReadWordDict (filename);
+
+  // read in the tag dictionary
+  ReadTagDict (filename, idh);
+
+  // read in the level information
+  ReadLevelFile (filename);
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  
+  // set up the translation table file
+  occurConvert.Open (filename, idh.word_dict_size, idh.tag_dict_size);
+
+  // reset some globals
+  numDocs = 0;
+  numChunkDocs = 0;
+  numDocsInChunk = 0;
+  numFrags = 0;
+  numFragsInChunk = 0;
+  chunkStartFragNum = 0;
+
+  strcpy (collectFilename, filename);
+  
+  
+  // create the inverted file
+  mg_ullong totalIBits = 0;
+  FILE *invfFile = create_file (filename, INVF_SUFFIX, "wb",
+				MAGIC_INVF, MG_ABORT);
+  totalIBits += sizeof (unsigned long) * 8; // magic number
+  totalIBits += 8 * 200;                    // 200 byte gap -- why??????
+  fclose (invfFile);
+
+  // init the inverted file state cache
+  invfState.Open (filename);
+  InitInvfState (filename, idh, invfState, totalIBits, wordLevelIndex);
+
+  return COMPALLOK;
+}
+
+static void CloseTextTag (IP2TagInfo &tInfo, const UCArray &/*tagName*/) {
+  if (!tInfo.inTag) return;
+
+    // add this tag to the inverted list
+  BitPtr &tagBitPtr = bitPtrs.GetTagBitPtr (tInfo.tagNum);
+  unsigned long endFrag = numFrags;
+  int b = 1 << tagBitPtr.lgB;
+
+  /*
+  cout << (tInfo.tagNum+idh.word_dict_size) << " \"<" << tagName << ">\" "
+       << tInfo.startFrag << " " << endFrag << "\n";
+  */
+    
+  mems_bitio_buffer buffer ((u_char *) ivfMemBuf, tagBitPtr.here);
+  buffer.bblock_encode (tInfo.startFrag - tagBitPtr.lastFragNum + 1,
+			b, NULL);
+  buffer.bblock_encode (endFrag - tInfo.startFrag + 1, b, NULL);
+  tagBitPtr.lastFragNum = endFrag;
+  tagBitPtr.here = buffer.position();
+  buffer.encodeDone();
+  
+  // check for buffer overrun
+  bitPtrs.CheckTagBufOverrun (tInfo.tagNum);
+  
+  // reset information about this tag
+  tInfo.inTag = false;
+  tInfo.startFrag = 0;
+}
+
+static void ProcessOpenTag (const TextEl &el, bool &inFrag) {
+  // close tag if open
+  IP2TagInfo &tInfo = tagMapDict[el.tagName];
+  if (tInfo.inTag) CloseTextTag (tInfo, el.tagName);
+
+  // open this tag
+  tInfo.inTag = true;
+  tInfo.startFrag = numFrags;
+
+  // check for start of next fragment
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  if (!wordLevelIndex && el.tagName == ivfLevel.indexLevel) {
+    numFrags++;
+    inFrag = true;
+  }
+}
+
+static void ProcessCloseTag (const TextEl &el, bool &inFrag) {
+  // check for end of fragment
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  if (!wordLevelIndex && el.tagName == ivfLevel.indexLevel) {
+    inFrag = false;
+  }
+
+  IP2TagInfo &tInfo = tagMapDict[el.tagName];
+  CloseTextTag (tInfo, el.tagName);
+}
+
+static void ProcessText (const TextEl &el, bool &inFrag) {
+  // make sure this text is to be indexed
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  if (!wordLevelIndex && !inFrag) return;
+
+  const unsigned char *textHere = &(el.text[0]);
+  const unsigned char *textEnd = &(el.text[el.text.size() - 1]);
+  unsigned char mgWord[MAXSTEMLEN + 1];
+  
+  if (!inaword (textHere, textEnd))
+    ParseNonindexWord (textHere, textEnd);
+
+  
+  // Alternately parse off words and non-words from the input
+
+  while (textHere <= textEnd) {
+    textHere = ParseIndexMGWord (textHere, textEnd, mgWord);
+    textHere = ParseNonindexWord (textHere, textEnd);
+
+    if (mgWord[0] > 0) {
+      if (wordLevelIndex) numFrags++;
+
+      unsigned long wordNum = perf_hash (wordHashDict, mgWord);
+
+      /*
+      cout << wordNum << " \"";
+      cout.write (mgWord+1, *mgWord);
+      cout << "\" " << numFrags << "\n";
+      */
+      
+      // add this word to the inverted list
+      BitPtr &wordBitPtr = bitPtrs.GetWordBitPtr (wordNum);
+      unsigned long fragNum = numFrags;
+      int b = 1 << wordBitPtr.lgB;
+  
+      mems_bitio_buffer buffer ((u_char *) ivfMemBuf, wordBitPtr.here);
+
+      // note: this assumes that fragments don't carry over between
+      // chunks (which they don't because all tags are closed at the
+      // end of each document and chunks are based on document
+      // boundaries), i.e. the first fragment number must be greater
+      // than the starting fragment number of the chunk.
+      if (fragNum > wordBitPtr.lastFragNum) {
+	buffer.bblock_encode ((fragNum - wordBitPtr.lastFragNum - 1) + 1,
+			      b, NULL);
+	if (!wordLevelIndex) buffer.encodeBit (1); // freq = 1
+	
+      } else if (!wordLevelIndex) {
+	// add one to the frequency count for this word
+	buffer.seek (buffer.position()-1);
+	buffer.encodeBit (0); // unary encoding -- last = 1
+	buffer.encodeBit (1);
+      }
+      
+      wordBitPtr.lastFragNum = fragNum;
+      wordBitPtr.here = buffer.position();
+      buffer.encodeDone();
+      
+      // check for buffer overrun
+      bitPtrs.CheckWordBufOverrun (wordNum);
+    }
+  }
+}
+
+// combine the in memory inverted buffer with the disk
+// based inverted file
+static void DiskMerge (char *filename) {
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+
+      // make sure we have something to process
+  if (numChunkDocs <= 0) return;
+    
+  // open the inverted file
+  FILE *invfFile = open_file (filename, INVF_SUFFIX, "rb+",
+			      MAGIC_INVF, MG_ABORT);
+  random_bitio_buffer invfOutBuf (invfFile);
+
+  // set up to decode the entries in memory
+  mems_bitio_buffer memInBuf ((u_char *) ivfMemBuf, 0);
+
+  // write out the word information
+  unsigned long wordNum;
+  int b;
+  unsigned long currFragNum;
+  unsigned long delta;
+  unsigned long currFreq;
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+    // go to the end of the last inverted file entry
+    InvfStateRec &wordDiskState = invfState.GetRec (wordNum);
+    invfOutBuf.SEEK_X (wordDiskState.here);
+
+    // go to the start of the inverted chunk info in memory
+    BitPtr &wordBitPtr = bitPtrs.GetWordBitPtr (wordNum);
+    memInBuf.seek(wordBitPtr.start);
+
+    // decode each entry and re-write to disk
+    currFragNum = chunkStartFragNum;
+    while (memInBuf.position() < wordBitPtr.here) {
+      // decode word entry
+      b = 1 << wordBitPtr.lgB;
+      delta = memInBuf.bblock_decode (b, NULL);
+      currFragNum += delta;
+      if (!wordLevelIndex) currFreq = memInBuf.unary_decode (NULL);
+      else currFreq = 1;
+
+      // recode on disk
+      invfOutBuf.bblock_encode (currFragNum-wordDiskState.lastFragNum,
+				wordDiskState.B, NULL);
+      if (!wordLevelIndex) invfOutBuf.gamma_encode (currFreq, NULL);
+      wordDiskState.lastFragNum = currFragNum;
+    }
+
+    wordDiskState.here = invfOutBuf.TELL_X();
+  }
+
+  // write out the tag information
+  unsigned long tagNum;
+  unsigned long currTagStart;
+  unsigned long currTagEnd;
+  for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+    // go to the end of the last inverted file entry
+    InvfStateRec &tagDiskState = invfState.GetRec (tagNum+idh.word_dict_size);
+    invfOutBuf.SEEK_X (tagDiskState.here);
+
+    // go to the start of the inverted chunk info in memory
+    BitPtr &tagBitPtr = bitPtrs.GetTagBitPtr (tagNum);
+    memInBuf.seek(tagBitPtr.start);
+
+    // decode each entry and re-write to disk
+    currTagEnd = chunkStartFragNum;
+    while (memInBuf.position() < tagBitPtr.here) {
+      // decode tag entry
+      b = 1 << tagBitPtr.lgB;
+      delta = memInBuf.bblock_decode (b, NULL) - 1;
+      currTagStart = currTagEnd + delta;
+      delta = memInBuf.bblock_decode (b, NULL) - 1;
+      currTagEnd = currTagStart + delta;
+
+      // recode on disk
+      invfOutBuf.bblock_encode (currTagStart-tagDiskState.lastFragNum+1,
+				tagDiskState.B, NULL);
+      invfOutBuf.bblock_encode (currTagEnd-currTagStart+1,
+				tagDiskState.B, NULL);
+
+      tagDiskState.lastFragNum = currTagEnd;
+    }
+
+    tagDiskState.here = invfOutBuf.TELL_X();
+  }
+
+  memInBuf.done();
+
+  invfOutBuf.encodeDone();
+  fclose (invfFile);
+}
+
+
+int process_ivf_2 (const TagInfo &/*tagInfo*/, const TextElArray &doc) {
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  bool inFrag = false;
+  if (wordLevelIndex) inFrag = true; // unconditional
+  
+  // get next chunk information if need to. the chunk information
+  // is needed before the first document is processed
+  if (numChunkDocs >= numDocsInChunk) ReadChunk (idh, wordLevelIndex);
+
+  // process each text element
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    // process this element
+    if ((*here).elType == OpenTagE) ProcessOpenTag (*here, inFrag);
+    else if ((*here).elType == CloseTagE) ProcessCloseTag (*here, inFrag);
+    else ProcessText (*here, inFrag);
+    
+    here++;
+  }
+
+  // close off any unclosed tags
+  TagMapDict::iterator tdHere = tagMapDict.begin();
+  TagMapDict::iterator tdEnd = tagMapDict.end();
+  while (tdHere != tdEnd) {
+    CloseTextTag ((*tdHere).second, (*tdHere).first);
+    tdHere++;
+  }
+  
+  // we've processed one more document
+  numDocs++;
+  numChunkDocs++;
+
+  // merge the memory based inverted file with the one on
+  // disk if this is the end of this chunk
+  if (numChunkDocs >= numDocsInChunk) DiskMerge (collectFilename);
+  
+  return COMPALLOK;
+}
+
+
+static void CondenseInvfFile (char *filename, unsigned long &bytesOutput) {
+  FILE *inInvfFile = open_file (filename, INVF_SUFFIX, "rb",
+				MAGIC_INVF, MG_ABORT);
+  FILE *outInvfFile = open_file (filename, INVF_SUFFIX, "rb+",
+				 MAGIC_INVF, MG_ABORT);
+
+  // skip the magic number
+  fseek (outInvfFile, sizeof (unsigned long), SEEK_SET);
+  
+  // write the inverted file header -- use defaults for most things
+  invf_file_header ifh;
+  ifh.no_of_words = idh.word_dict_size;
+  ifh.no_of_tags = idh.tag_dict_size;
+  ifh.word_level_index = (ivfLevel.indexLevel.empty()) ? 1 : 0;
+  ifh.Write (outInvfFile);
+
+  bytesOutput = ftell (outInvfFile);
+
+  // process each meaningful byte in the file
+  unsigned long numEntries = ifh.no_of_words + ifh.no_of_tags;
+  unsigned long entryNum;
+  mg_ullong lastStart = 0;
+  for (entryNum = 0; entryNum < numEntries; entryNum++) {
+    InvfStateRec &stateRec = invfState.GetRec (entryNum);
+
+    // overrun check
+    if (stateRec.start < lastStart)
+      FatalError (1, "Inverted file Buffer overrun");
+    lastStart = stateRec.start;
+    
+    unsigned long oldEntryStart = stateRec.start >> 3;
+    unsigned long oldEntryStartOver = stateRec.start & 7; // should be 0
+    unsigned long oldEntryEnd = (stateRec.here + 7) >> 3;  // byte after end
+    unsigned long oldEntryEndOver = stateRec.here & 7;
+
+    fseek (inInvfFile, oldEntryStart, SEEK_SET);
+
+    stateRec.here -= stateRec.start;
+    stateRec.start = (mg_ullong)bytesOutput * 8 + oldEntryStartOver;
+    stateRec.here += stateRec.start;
+    while (oldEntryStart < oldEntryEnd) {
+      unsigned char c = getc (inInvfFile);
+      if (oldEntryStart == oldEntryEnd - 1) {
+	u_char ands[8] =
+	{0xff, 0x80, 0xc0, 0xe0, 0xf0, 0xf8, 0xfc, 0xfe};
+	c &= ands[oldEntryEndOver];
+      }
+      putc (c, outInvfFile);
+      bytesOutput++;
+      oldEntryStart++;
+    }
+  }  
+
+  fclose (inInvfFile);
+
+#ifdef __WIN32__
+  if (_chsize (_fileno (outInvfFile), bytesOutput) != 0)
+    Message ("Could not truncate invf.");
+#else
+  ftruncate (fileno (outInvfFile), bytesOutput);
+#endif
+
+  fclose (outInvfFile);
+}
+
+static void OutputInvfIdx (char *filename, unsigned long invfNumBytes) {
+  FILE *invfIdxFile = create_file (filename, INVF_IDX_SUFFIX, "wb",
+				   MAGIC_INVI, MG_ABORT);
+
+  // process each meaningful byte in the file
+  unsigned long numEntries = idh.word_dict_size + idh.tag_dict_size;
+  unsigned long entryNum;
+  for (entryNum = 0; entryNum < numEntries; entryNum++) {
+    InvfStateRec &stateRec = invfState.GetRec (entryNum);
+
+    // assumes that inverted entries start at beginning of each byte
+    if (!WriteUL (invfIdxFile, (stateRec.start >> 3))) break;
+  }
+
+  WriteUL (invfIdxFile, invfNumBytes);
+
+  fclose (invfIdxFile);
+}
+
+
+int done_ivf_2 (const TagInfo &/*tagInfo*/, char *filename) {
+  // close most open files
+  if (chunkFile != NULL) {
+    chunkBuf.done();
+    fclose (chunkFile);
+    chunkFile = NULL;
+  }
+  occurConvert.Close();
+  
+  // free allocated memory
+  bitPtrs.Clear();
+  if (ivfMemBuf != NULL) { delete [] ivfMemBuf; ivfMemBuf = NULL; }
+  free_perf_hash (wordHashDict);
+  wordHashDict = NULL;
+  tagMapDict.erase (tagMapDict.begin(), tagMapDict.end());
+
+  // condense the inverted file and truncate it
+  // this function also writes out the inverted header
+  unsigned long invfNumBytes = 0;
+  CondenseInvfFile (filename, invfNumBytes);
+
+  OutputInvfIdx (filename, invfNumBytes);
+
+  // close the rest of the open files
+  invfState.Close ();
+  
+  return COMPALLOK;
+}
Index: /trunk/indexers/mgpp/text/locallib.cpp
===================================================================
--- /trunk/indexers/mgpp/text/locallib.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/locallib.cpp	(revision 3365)
@@ -0,0 +1,218 @@
+/**************************************************************************
+ *
+ * locallib.cpp -- Misc functions
+ * Copyright (C) 1994  Gary Eddy, Alistair Moffat and Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "memlib.h"
+#include "bitio_gen.h"
+#include "huffman.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "locallib.h"
+#include "text.h"
+
+int 
+vecentropy (int *A, int n)
+{
+  int total = 0, i;
+  double bits = 0.0;
+
+  for (i = 0; i < n; i++)
+    total += A[i];
+  for (i = 0; i < n; i++)
+    if (A[i])
+      bits += A[i] * log2 (1.0 * total / A[i]);
+  return ((int) (bits + 0.9999));
+
+}
+
+static void 
+siftdown (int *A, int i, int n)
+{
+  int ms, t;
+
+  while (2 * i + 1 <= n - 1)
+    {
+      if (2 * i + 1 == n - 1 || A[2 * i + 1] < A[2 * i + 2])
+	ms = 2 * i + 1;
+      else
+	ms = 2 * i + 2;
+      if (A[i] > A[ms])
+	{
+	  t = A[i];
+	  A[i] = A[ms];
+	  A[ms] = t;
+	}
+      i = ms;
+    }
+}
+
+static void 
+buildheap (int *A, int n)
+{
+  int i;
+  for (i = n / 2 - 1; i >= 0; i--)
+    siftdown (A, i, n);
+}
+
+int 
+huffcodebits (unsigned long *A, int n)
+{
+  int i, j, tot = 0, bits = 0, v1, v2, *B;
+
+  B = (int *) Xmalloc (n * sizeof (int));
+  j = 0;
+  for (i = 0; i < n; i++)
+    {
+      if (A[i])
+	{
+	  tot += (B[j++] = A[i]);
+	}
+    }
+  n = j;
+
+  if (n == 0 || tot == 0)
+    return (0);
+
+  buildheap (B, n);
+  while (n > 1)
+    {
+      v1 = B[0];
+      B[0] = B[n - 1];
+      n--;
+      siftdown (B, 0, n);
+      v2 = B[0];
+      B[0] = v1 + v2;
+      siftdown (B, 0, n);
+      bits += v1 + v2;
+    }
+  Xfree (B);
+  return (bits);
+}
+
+
+int 
+modelbits (unsigned long *A, int n)
+{
+  int i, bits = 0, last, N = 0;
+
+  last = -1;
+  for (i = 0; i < n; i++)
+    {
+      if (A[i])
+	{
+	  bits += BIO_Gamma_Length (i - last) + BIO_Gamma_Length (A[i]);
+	  last = i;
+	  N++;
+	}
+    }
+  if (N == 0)
+    return (0);
+  bits += BIO_Gamma_Length (N);
+  return (bits);
+}
+
+
+
+
+
+/* Find the next prime number larger than p */
+int 
+prime (int p)
+{
+  int limit;
+  p += (p & 1) + 1;
+  limit = (int) sqrt ((double) p) + 1;
+  do
+    {
+      int j;
+      while (limit * limit < p)
+	limit++;
+      for (j = 3; j <= limit && p % j; j += 2)
+	;			/* NULLBODY */
+      if (j > limit)
+	break;
+      p += 2;
+    }
+  while (1);
+  return (p);
+}
+
+
+
+int Read_cdh (FILE * f, compression_dict_header * cdh, u_long * /*mem*/,
+	      u_long * disk) {
+  if (disk)
+    (*disk) += sizeof (*cdh);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  if (fread (cdh, sizeof (*cdh), 1, f) == 1)
+    {
+      int i;
+      NTOHUL(cdh->dict_type);
+      NTOHUL(cdh->novel_method);
+      for (i = 0; i < TEXT_PARAMS; i++)
+	NTOHUL(cdh->params[i]);
+      NTOHUL(cdh->num_words[0]);
+      NTOHUL(cdh->num_words[1]);
+      NTOHUL(cdh->num_word_chars[0]);
+      NTOHUL(cdh->num_word_chars[1]);
+      NTOHUL(cdh->lookback);
+      return 0;
+    }
+  else
+    return -1;
+}
+
+
+int 
+Read_cfh (FILE * f, comp_frags_header * cfh, u_long * mem, u_long * disk)
+{
+  int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+  if (Read_Huffman_Data (f, &cfh->hd, mem, disk) == -1)
+    return -1;
+  if (fread (&cfh->uncompressed_size,
+	     sizeof (cfh->uncompressed_size), 1, f) != 1)
+    return -1;
+  NTOHUL(cfh->uncompressed_size);  /* [RPAP - Jan 97: Endian Ordering] */
+
+  if (disk)
+    (*disk) += sizeof (cfh->uncompressed_size);
+
+  memset (cfh->huff_words_size, '\0', sizeof (cfh->huff_words_size));
+
+  if ((int)fread (cfh->huff_words_size + cfh->hd.mincodelen,
+	     sizeof (*cfh->huff_words_size),
+	     cfh->hd.maxcodelen - cfh->hd.mincodelen + 1, f) !=
+      cfh->hd.maxcodelen - cfh->hd.mincodelen + 1)
+    return -1;
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = cfh->hd.mincodelen; i < cfh->hd.maxcodelen + 1; i++)
+    NTOHUL(cfh->huff_words_size[i]);
+
+  if (disk)
+    (*disk) += sizeof (*cfh->huff_words_size) *
+      (cfh->hd.maxcodelen - cfh->hd.mincodelen + 1);
+  return 0;
+}
+
Index: /trunk/indexers/mgpp/text/locallib.h
===================================================================
--- /trunk/indexers/mgpp/text/locallib.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/locallib.h	(revision 3365)
@@ -0,0 +1,45 @@
+/**************************************************************************
+ *
+ * locallib.h -- Misc functions
+ * Copyright (C) 1994  Gary Eddy, Alistair Moffat and Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef LOCALLIB_H
+#define LOCALLIB_H
+
+#include "sysfuncs.h"
+
+#include "text.h"
+
+#define NUMOF(a) (sizeof(a)/sizeof((a)[0]))
+
+int vecentropy (int *A, int n);
+
+int huffcodebits (unsigned long *A, int n);
+
+int modelbits (unsigned long *A, int n);
+
+int prime (int p);
+
+
+int Read_cdh (FILE * f, compression_dict_header * cdh, u_long * mem, u_long * disk);
+
+int Read_cfh (FILE * f, comp_frags_header * cfh, u_long * mem, u_long * disk);
+
+
+#endif
Index: /trunk/indexers/mgpp/text/mg.h
===================================================================
--- /trunk/indexers/mgpp/text/mg.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/mg.h	(revision 3365)
@@ -0,0 +1,37 @@
+/**************************************************************************
+ *
+ * mg.h -- General header file
+ * Copyright (C) 1994  Neil Sharman, Alistair Moffat and Lachlan Andrew
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+
+#ifndef H_MG
+#define H_MG
+
+
+#define COMPALLOK	(0)
+#define COMPERROR	(-1)
+
+#define MAXSTRING       3000000
+#define BUFFERSIZE      (MAXSTRING+30000)
+
+#define	MAXLN	65536
+#define WORDS           1
+#define NONWORDS        0
+
+#endif
Index: /trunk/indexers/mgpp/text/mg.special.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mg.special.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mg.special.cpp	(revision 3365)
@@ -0,0 +1,58 @@
+/**************************************************************************
+ *
+ * mg.special.cpp -- Special pass for mg_passes
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "sysfuncs.h"
+#include "mg.h"
+#include "build.h"
+
+int init_special (const TagInfo &/*tagInfo*/, char * /*FileName*/) {
+  return COMPALLOK;
+}
+
+int process_special (const TagInfo &/*tagInfo*/, const TextElArray &doc) {
+  // print out the document
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    if ((*here).elType == OpenTagE) {
+      cout << "** OpenTag  \"" << (*here).tagName << "\"\tText \""
+	   << (*here).text << "\"\n"; 
+    } else if ((*here).elType == CloseTagE) {
+      cout << "** CloseTag \"" << (*here).tagName << "\"\tText \""
+	   << (*here).text << "\"\n";
+    } else {
+      cout << "** Text     \"" << (*here).text << "\"\n";
+    }
+    
+    here++;
+  }
+  
+  return COMPALLOK;
+}
+
+int done_special (const TagInfo &/*tagInfo*/, char * /*FileName*/) {
+  return COMPALLOK;
+}
Index: /trunk/indexers/mgpp/text/mg_errors.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mg_errors.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mg_errors.cpp	(revision 3365)
@@ -0,0 +1,72 @@
+/**************************************************************************
+ *
+ * mg_errors.cpp -- Error related stuff for mgquery
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+#include "sysfuncs.h"
+#include "memlib.h"
+
+#include "mg_errors.h"
+
+int mg_errno = MG_NOERROR;
+
+char *mg_errorstrs[] =
+{
+  "No error",
+  "Out of memory",
+  "File \"%s\" not found",
+  "Bad magic number in \"%s\"",
+  "Error reading \"%s\"",
+  "MG_BUFTOOSMALL",
+  "Files required for level 2 and 3 inversion are missing"};
+
+
+/* [RJM 07/98: Memory Leak] -- rest of file */
+
+static char null_data[] = "";
+char *mg_error_data = null_data;
+
+
+void MgErrorData (char *s)
+{
+  /* free the current error string, unless it is the null string */
+  if ((mg_error_data != NULL) && (mg_error_data != null_data)) {
+    delete mg_error_data;
+    mg_error_data = null_data;
+  }
+
+  /* make a copy of the string */
+  mg_error_data = Xstrdup (s);
+
+  /* if the alloc failed set the error to the null string */
+  if (!mg_error_data) mg_error_data = null_data;
+}
+
+
+void MgErrorDeinit (void)
+{
+  /* free the current error string, unless it is the null string */
+  if ((mg_error_data != NULL) && (mg_error_data != null_data)) {
+    delete mg_error_data;
+    mg_error_data = null_data;
+  }
+}
Index: /trunk/indexers/mgpp/text/mg_errors.h
===================================================================
--- /trunk/indexers/mgpp/text/mg_errors.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/mg_errors.h	(revision 3365)
@@ -0,0 +1,37 @@
+/**************************************************************************
+ *
+ * mg_errors.h -- Error related stuff for mgquery
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+
+extern int mg_errno;
+extern char *mg_error_data;
+extern char *mg_errorstrs[];
+
+
+#define MG_NOERROR 0
+#define MG_NOMEM 1
+#define MG_NOFILE 2
+#define MG_BADMAGIC 3
+#define MG_READERR 4
+#define MG_BUFTOOSMALL 5
+#define MG_INVERSION 6
+
+void MgErrorData (char *s);
+void MgErrorDeinit (void);
Index: /trunk/indexers/mgpp/text/mg_files.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mg_files.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mg_files.cpp	(revision 3365)
@@ -0,0 +1,259 @@
+/**************************************************************************
+ *
+ * mg_files.cpp -- Routines for handling files for the auxillary programs
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "mg_files.h"
+
+
+/* This must contain a valid path without a trailing slash */
+static char *basepath = NULL;
+
+/* This sets the base path for all file operations */
+void 
+set_basepath (const char *bp)
+{
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef __WIN32__
+  basepath = "";
+#else
+  char *s;
+  /* Free the memory for the base path if it has already been allocated */
+  if (basepath)
+    {
+      delete basepath;
+      basepath = NULL;
+    }
+
+  if (!bp || *bp == '\0')
+    bp = ".";
+
+  s = strrchr (bp, '/');
+  if (s && *(s + 1) == '\0')
+    {
+      basepath = new char[strlen (bp) + 2];
+      if (!basepath)
+	return;
+      strcpy (basepath, bp);
+      strcat (basepath, ".");
+    }
+  else
+    basepath = Xstrdup (bp);
+#endif
+}
+
+
+
+/* return the currently defined basepath */
+char *
+get_basepath (void)
+{
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  return basepath;
+}
+
+
+
+/* This generates the name of a file. It places the name in the buffer
+   specified or if that is NULL it uses a static buffer. */
+char *
+make_name (const char *name, const char *suffix, char *buffer)
+{
+  static char path[512];
+  if (!buffer)
+    buffer = path;
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  sprintf (buffer, FILE_NAME_FORMAT, basepath, name, suffix);  /* [RPAP - Feb 97: WIN32 Port] */
+  return buffer;
+}
+
+
+
+/* This will open the specified file and check its magic number.
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+open_named_file (const char *name, const char *mode,
+		 u_long magic_num, int err_mode)
+{
+  unsigned long magic;
+  FILE *f = NULL;
+  char *err;
+  f = fopen (name, mode);
+
+  if (!f)
+    {
+      err = "Unable to open \"%s\"";
+      goto error;
+    }
+
+  if (magic_num)
+    {
+      if (fread (&magic, sizeof (magic), 1, f) != 1)
+	{
+	  err = "No magic number \"%s\"";
+	  goto error;
+	}
+
+      NTOHUL(magic);  /* [RPAP - Jan 97: Endian Ordering] */
+
+      if (!IS_MAGIC (magic))
+	{
+	  err = "No MG magic number \"%s\"";
+	  goto error;
+	}
+
+      if (magic != magic_num)
+	{
+	  err = "Wrong MG magic number \"%s\"";
+	  goto error;
+	}
+    }
+  return f;
+
+error:
+  if (f)
+    fclose (f);
+  if (err_mode == MG_ABORT)
+    FatalError (1, err, name);
+  if (err_mode == MG_MESSAGE)
+    Message (err, name);
+  return NULL;
+}
+
+
+
+/* This will open the specified file and check its magic number.
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+open_file (const char *name, const char *suffix, const char *mode,
+	   u_long magic_num, int err_mode)
+{
+  char path[512];
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  sprintf (path, FILE_NAME_FORMAT, basepath, name, suffix);  /* [RPAP - Feb 97: WIN32 Port] */
+  return open_named_file (path, mode, magic_num, err_mode);
+}
+
+
+
+
+
+
+/* This will create the specified file and set its magic number.
+
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+create_named_file (const char *name, const char *mode,
+		   u_long magic_num, int err_mode)
+{
+  FILE *f = NULL;
+  char *err;
+  f = fopen (name, mode);
+
+  if (!f)
+    {
+      err = "Unable to open \"%s\"";
+      goto error;
+    }
+
+  if (magic_num)
+    HTONUL(magic_num);  /* [RPAP - Jan 97: Endian Ordering] */
+    if (fwrite (&magic_num, sizeof (magic_num), 1, f) != 1)
+      {
+	err = "Couldn't write magic number \"%s\"";
+	goto error;
+      }
+
+  return f;
+
+error:
+  if (f)
+    fclose (f);
+  if (err_mode == MG_ABORT)
+    FatalError (1, err, name);
+  if (err_mode == MG_MESSAGE)
+    Message (err, name);
+  return NULL;
+}
+
+
+
+
+
+
+
+/* This will create the specified file and set its magic number.
+
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+create_file (const char *name, const char *suffix, const char *mode,
+	     u_long magic_num, int err_mode)
+{
+  char path[512];
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  sprintf (path, FILE_NAME_FORMAT, basepath, name, suffix);  /* [RPAP - Feb 97: WIN32 Port] */
+  return create_named_file (path, mode, magic_num, err_mode);
+}
Index: /trunk/indexers/mgpp/text/mg_files.h
===================================================================
--- /trunk/indexers/mgpp/text/mg_files.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/mg_files.h	(revision 3365)
@@ -0,0 +1,343 @@
+/**************************************************************************
+ *
+ * mg_files.h -- Routines for handling files for the auxillary programs
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef MG_FILES_H
+#define MG_FILES_H
+
+#include "sysfuncs.h"
+
+/* Magic numbers for the different types of files */
+
+#define GEN_MAGIC(a,b,c,d) ((unsigned long)(((a)<<24) + ((b)<<16) + \
+					    ((c)<<8) + (d)))
+
+#define MAGIC_XXXX		GEN_MAGIC('M','G', 0 , 0)
+#define MAGIC_STATS_DICT	GEN_MAGIC('M','G','S','D')
+#define MAGIC_AUX_DICT		GEN_MAGIC('M','G','A','D')
+#define MAGIC_FAST_DICT		GEN_MAGIC('M','G','F','D')
+#define MAGIC_DICT		GEN_MAGIC('M','G','D', 0 )
+#define MAGIC_STEM_BUILD	GEN_MAGIC('M','G','S', 0 )
+#define MAGIC_HASH		GEN_MAGIC('M','G','H', 0 )
+#define MAGIC_STEM		GEN_MAGIC('M','G','s', 0 )
+#define MAGIC_CHUNK		GEN_MAGIC('M','G','C', 0 )
+#define MAGIC_CHUNK_TRANS	GEN_MAGIC('M','G','c', 0 )
+#define MAGIC_TEXT		GEN_MAGIC('M','G','T', 0 )
+#define MAGIC_TEXI		GEN_MAGIC('M','G','t', 0 )
+#define MAGIC_TEXT_LEVELS	GEN_MAGIC('M','G','t','L')
+#define MAGIC_TEXI_WGT		GEN_MAGIC('M','G','t','W')
+#define MAGIC_INVF		GEN_MAGIC('M','G','I', 0 )
+#define MAGIC_INVF_LEVELS	GEN_MAGIC('M','G','I','L')
+#define MAGIC_INVI		GEN_MAGIC('M','G','i', 0 )
+#define MAGIC_WGHT		GEN_MAGIC('M','G','W', 0 )
+#define MAGIC_WGHT_APPROX	GEN_MAGIC('M','G','w', 0 )
+#define MAGIC_PARAGRAPH		GEN_MAGIC('M','G','P', 0 )
+/* [RPAP - Jan 97: Stem Index Change] */
+#define MAGIC_STEM_1            GEN_MAGIC('M','G','s','1')
+#define MAGIC_STEM_2            GEN_MAGIC('M','G','s','2')
+#define MAGIC_STEM_3            GEN_MAGIC('M','G','s','3')
+
+#define IS_MAGIC(a) ((((u_long)(a)) & 0xffff0000) == MAGIC_XXXX)
+
+
+/* err_mode values for open_file and create_file */
+#define MG_ABORT 0
+#define MG_MESSAGE 1
+#define MG_CONTINUE 2
+
+
+
+
+
+/* File suffixes */
+
+
+/* The compression dictionary built by txt.pass1 */
+#ifdef SHORT_SUFFIX
+# define TEXT_STATS_DICT_SUFFIX	 ".tsd"
+#else
+# define TEXT_STATS_DICT_SUFFIX	 ".text.stats"
+#endif
+
+/* The compression dictionary built by text.pass1 and comp_dict.process */
+#ifdef SHORT_SUFFIX
+# define TEXT_DICT_SUFFIX	 ".td"
+#else
+# define TEXT_DICT_SUFFIX	 ".text.dict"
+#endif
+
+/* The compression dictionary built by mg_make_fast_dict */
+#ifdef SHORT_SUFFIX
+# define TEXT_DICT_FAST_SUFFIX	 ".tdf"
+#else
+# define TEXT_DICT_FAST_SUFFIX	 ".text.dict.fast"
+#endif
+
+/* The auxilary dictionary built by text.pass2 */
+#ifdef SHORT_SUFFIX
+# define TEXT_DICT_AUX_SUFFIX	 ".tda"
+#else
+# define TEXT_DICT_AUX_SUFFIX	 ".text.dict.aux"
+#endif
+
+/* The compressed text build by text.pass2 */
+#ifdef SHORT_SUFFIX
+# define TEXT_SUFFIX		 ".t"
+#else
+# define TEXT_SUFFIX		 ".text"
+#endif
+
+/* The combined compressed text index and document weight file */
+#ifdef SHORT_SUFFIX
+# define TEXT_IDX_WGT_SUFFIX	 ".tiw"
+#else
+# define TEXT_IDX_WGT_SUFFIX	 ".text.idx.wgt"
+#endif
+
+/* The compressed text index file */
+#ifdef SHORT_SUFFIX
+# define TEXT_IDX_SUFFIX	 	".ti"
+#else
+# define TEXT_IDX_SUFFIX	 	".text.idx"
+#endif
+
+/* The document levels in the text */
+#ifdef SHORT_SUFFIX
+# define TEXT_LEVEL_SUFFIX	 	".tl"
+#else
+# define TEXT_LEVEL_SUFFIX	 	".text.level"
+#endif
+
+/* The dictionary of stemmed words build by invf.pass1 and ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_SUFFIX         ".id"
+#else
+# define INVF_DICT_SUFFIX         ".invf.dict"
+#endif
+
+/* The dictionary of stemmed words build by stem.process */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_SUFFIX ".idb"
+#else
+# define INVF_DICT_BLOCKED_SUFFIX ".invf.dict.blocked"
+#endif
+
+/* The exact document weights file build by make.weights, invf.pass2,
+   or ivf.pass2 */
+#ifdef SHORT_SUFFIX
+# define WEIGHTS_SUFFIX           ".w"
+#else
+# define WEIGHTS_SUFFIX           ".weight"
+#endif
+
+/* The approximate weights file built by make.weights */
+#ifdef SHORT_SUFFIX
+# define APPROX_WEIGHTS_SUFFIX    ".wa"
+#else
+# define APPROX_WEIGHTS_SUFFIX    ".weight.approx"
+#endif
+
+/* The inverted file build by invf.pass2 or ivf.pass2 */
+#ifdef SHORT_SUFFIX
+# define INVF_SUFFIX              ".i"
+#else
+# define INVF_SUFFIX              ".invf"
+#endif
+
+/* The level information for the inverted text */
+#ifdef SHORT_SUFFIX
+# define INVF_LEVEL_SUFFIX              ".il"
+#else
+# define INVF_LEVEL_SUFFIX              ".invf.level"
+#endif
+
+/* The inverted file index build by invf.pass2 or ivf.pass2 */
+#ifdef SHORT_SUFFIX
+# define INVF_IDX_SUFFIX          ".ii"
+#else
+# define INVF_IDX_SUFFIX          ".invf.idx"
+#endif
+
+/* The inverted file chunk descriptor built by ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_CHUNK_SUFFIX        ".ic"
+#else
+# define INVF_CHUNK_SUFFIX        ".invf.chunk"
+#endif
+
+/* The word index translation file built by ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_CHUNK_TRANS_SUFFIX  ".ict"
+#else
+# define INVF_CHUNK_TRANS_SUFFIX  ".invf.chunk.trans"
+#endif
+
+/* The hashed stemmed dictionary built by make.perf_hash */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_HASH_SUFFIX    ".idh"
+#else
+# define INVF_DICT_HASH_SUFFIX    ".invf.dict.hash"
+#endif
+
+/* The paragraph descriptior file built by invf.pass1 or ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_PARAGRAPH_SUFFIX    ".ip"
+#else
+# define INVF_PARAGRAPH_SUFFIX    ".invf.paragraph"
+#endif
+
+/* The trace file build by mg.builder. */
+#ifdef SHORT_SUFFIX
+# define TRACE_SUFFIX             ".trc"
+#else
+# define TRACE_SUFFIX             ".trace"
+#endif
+
+/* The compression stats file build by mg.builder. */
+#ifdef SHORT_SUFFIX
+# define COMPRESSION_STATS_SUFFIX ".cs"
+#else
+# define COMPRESSION_STATS_SUFFIX ".compression.stats"
+#endif
+
+/* [RPAP - Jan 97: Stem Index Change] */
+/* The casefolded index into the stemmed dictionary */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_1_SUFFIX ".ib1"
+#else
+# define INVF_DICT_BLOCKED_1_SUFFIX ".invf.dict.blocked.1"
+#endif
+
+/* [RPAP - Jan 97: Stem Index Change] */
+/* The stemmed index into the stemmed dictionary */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_2_SUFFIX ".ib2"
+#else
+# define INVF_DICT_BLOCKED_2_SUFFIX ".invf.dict.blocked.2"
+#endif
+
+/* [RPAP - Jan 97: Stem Index Change] */
+/* The casefolded and stemmed  index into the stemmed dictionary */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_3_SUFFIX ".ib3"
+#else
+# define INVF_DICT_BLOCKED_3_SUFFIX ".invf.dict.blocked.3"
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef __WIN32__
+#define FILE_NAME_FORMAT "%s%s%s"
+#else
+#define FILE_NAME_FORMAT "%s/%s%s"
+#endif
+
+
+/* This sets the base path for all file operations */
+void set_basepath (const char *bp);
+
+
+/* return the currently defined basepath */
+char *get_basepath (void);
+
+
+
+
+/* This generates the name of a file. It places the name in the buffer
+   specified or if that is NULL it uses a static buffer. */
+char *make_name (const char *name, const char *suffix, char *buffer);
+
+
+
+
+
+
+
+/* This will open the specified file and check its magic number.
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *open_named_file (const char *name, const char *mode,
+		       u_long magic_num, int err_mode);
+
+
+
+
+/* This will open the specified file and check its magic number.
+
+   err_mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *open_file (const char *name, const char *suffix, const char *mode,
+		 u_long magic_num, int err_mode);
+
+
+
+
+
+/* This will create the specified file and set its magic number.
+
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *create_named_file (const char *name, const char *mode,
+			 u_long magic_num, int err_mode);
+
+
+
+/* This will create the specified file and set its magic number.
+
+   err_mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *create_file (const char *name, const char *suffix, const char *mode,
+		   u_long magic_num, int err_mode);
+
+
+
+
+#endif
Index: /trunk/indexers/mgpp/text/mgpp_compression_dict.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_compression_dict.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_compression_dict.1	(revision 3365)
@@ -0,0 +1,188 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_compression_dict 1 \*(Dt CITRI
+.SH NAME
+mgpp_compression_dict \- build a compression dictionary.
+.SH SYNOPSIS
+.B mgpp_compression_dict
+[
+.B \-h
+]
+[
+.BR \-C " |"
+.BR \-P " |"
+.B  \-S
+]
+.if n .ti +9n
+[
+.BR \-0 " |"
+.BR \-1 " |"
+.BR \-2 " |"
+.B  \-3
+]
+[
+.BR \-H " |"
+.BR \-B " |"
+.BR \-D " |"
+.BR \-Y " |"
+]
+.if n .ti +9n
+.if t .ti +.5i
+[
+.BI \-l " lookback"
+]
+[
+.BI \-k " mem"
+]
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_compression_dict
+builds a compression dictionary based on the statistics gathered
+during the first pass over the text.  The options to the program are
+mainly concerned with limiting the amount of memory the dictionary
+will use and with how the text compressor will cope with any novel
+words found during the compression phase.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.B \-C
+Build a complete dictionary from the statistics file.  If during the
+text compression phase a novel word is found, then the compressor will
+produce an error message and stop.
+.TP
+.B \-P
+Build a partial dictionary from the statistics file.  This dictionary
+assumes that the statistics file are based on the entire text.  The
+statistics of words not includes in the dictionary are used to
+calculate the escape probability.  If novel words are being coded
+character by character, then there may not be a Huffman code for every
+possible character.  This means that the compressor may fail if a novel
+word contains a novel character.
+.TP
+.B \-S
+Build a seed dictionary from the statistics file.  This dictionary
+assumes that the statistics file is based on only a portion of the
+text to be compressed.  The probability of a novel word is based on the
+number of words that have only occurred once.  If novel words are being
+coded character by character, then the Huffman codes for characters are
+based on the frequency of characters in the dictionary.
+.TP
+.B \-0
+All words from the statistics file are included in the built
+dictionary.
+.TP
+.B \-1
+Words are included in the dictionary until the dictionary reaches the
+desired size.  Words are selected for the dictionary based on the order
+they occurred in the source text.
+.TP
+.B \-2
+Words are included in the dictionary until the dictionary reaches the
+desired size.  The most frequent words are included in the dictionary
+first; where there is a tie for frequency, the shortest word is
+included first.
+.TP
+.B \-3
+Words are included in the dictionary until the dictionary reaches the
+desired size.  The most frequent words are included in the dictionary
+first; where there is a tie for frequency, the shortest word is
+included first.  Words are the shuffled back and forth between the
+`keep' and `discard' lists to find the `optimal' set of words that
+should be in the dictionary.
+.TP
+.B \-H
+This specifies that novel words will be coded character by character
+using Huffman codes.
+.TP
+.B \-B
+This specifies that an auxiliary dictionary will be built by the
+compressor.  Each novel word found will be placed at the end of the
+auxiliary dictionary.  Novel words will be coded in the compressed text
+using binary codes.  The binary code represents their occurrence
+position in the auxiliary dictionary.
+.TP
+.B \-D
+This specifies that an auxiliary dictionary will be built by the
+compressor.  Each novel word found will be placed at the end of the
+auxiliary dictionary.  Novel words will be coded in the compressed text
+using delta codes.  The delta code represents their occurrence position
+in the auxiliary dictionary.
+.TP
+.B \-Y
+This specifies that an auxiliary dictionary will be built by the
+compressor.  Each novel word found will be placed at the end of the
+auxiliary dictionary.  Novel words will be coded in the compressed text
+using a combination of gamma and binary codes.  The code represents
+their occurrence position in the auxiliary dictionary.  This generally
+produces better compression than
+.B \-B
+or
+.BR \-D .
+.TP
+.BI \-l " lookback"
+The generated dictionary is designed to be front coded when it is
+loaded into memory.  Under normal circumstances, a front-coded
+dictionary would require scanning from the beginning in order to find
+any particular word.  However, every
+.I lookback
+words in the dictionary, the whole word is stored and a pointer to that
+word maintained.  E.g., if
+.I lookback
+is 4, then every fourth word is stored in its entirety.
+.TP
+.BI \-k " mem"
+This limits the amount of memory to use for the generated
+dictionary.  Words are selected for the dictionary based of the text
+statistics, and whether
+.BR \-0 , " \-1" , " \-2"
+or
+.B \-3
+is specified.  The memory is calculated assuming a lookback of 0,
+irrespective of what actual lookback is specified.  This means that if
+a non-zero lookback is given, the dictionary will actually occupy
+less space than specified by
+.BR \-k .
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.text.stats
+Statistics about the source text.
+.TP
+.B *.text.dict
+Compression dictionary for the source text.
+.SH "SEE ALSO"
+.na
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/indexers/mgpp/text/mgpp_compression_dict.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_compression_dict.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_compression_dict.cpp	(revision 3365)
@@ -0,0 +1,1021 @@
+/**************************************************************************
+ *
+ * mgpp_compression_dict.cpp -- Routines for creating compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "bitio_gen.h"
+#include "bitio_m_stdio.h"
+#include "mgheap.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "mg_files.h"
+#include "locallib.h"
+#include "invf.h"
+#include "text.h"
+#include "words.h"
+#include "mg.h"
+#include "WordData.h"
+
+#define MAXBITS (sizeof(unsigned long) * 8)
+
+#define is_power_of_two(a) ((a) != 0 && (((a) & ((a)-1)) == 0))
+
+#define MAX_RECALCULATIONS 100
+
+typedef struct DictWordData : public WordData
+  {
+    float saving;
+    float char_bit_cost;
+    u_long num_trans;
+    u_char *word;
+  }
+DictWordData;
+
+static DictWordData *Words[2];
+static u_long Num[2];
+static u_long chars[2];
+
+#define KIND(p) (((p) >= Words[0] && (p) < Words[0] + Num[0]) ? 0 : 1)
+#define IsWord(p) (((p) >= Words[1] && (p) < Words[1] + Num[1]) ? 1 : 0)
+#define IsNonWord(p) (((p) >= Words[0] && (p) < Words[0] + Num[0]) ? 1 : 0)
+
+
+typedef struct DictData
+  {
+    DictWordData **wd;
+    u_long num_wds;
+    u_long chars;
+  }
+DictData;
+
+typedef DictData DictInfo[2];
+
+static DictInfo keep, discard, all;
+static compression_stats_header csh;
+
+static char *file_name = "";
+static u_long novel_method = MG_NOVEL_HUFFMAN_CHARS;
+
+
+static void ReadInWords (char *);
+static void Select_on (int k, heap_comp hc);
+static void Method3 (int k);
+static void Select_all (void);
+static u_long WriteOutWords (char *, u_long, int);
+static int DecFreqIncWL (void *a, void *b);
+static int OccuranceOrder (void *a, void *b);
+
+
+
+int main (int argc, char **argv)
+{
+  int ch;
+  char type = 'C';
+  char mode = '0';
+  int lookback = 2;
+  double k = 0;
+  u_long mem_reqd;
+  opterr = 0;
+  msg_prefix = argv[0];
+  while ((ch = getopt (argc, argv, "0123CPSf:d:l:hk:HDY")) != -1)
+    switch (ch)
+      {
+      case 'H':
+	novel_method = MG_NOVEL_HUFFMAN_CHARS;
+	break;
+      case 'D':
+	novel_method = MG_NOVEL_DELTA;
+	break;
+      case 'Y':
+	novel_method = MG_NOVEL_HYBRID;
+	break;
+      case 'f':		/* input file */
+	file_name = optarg;
+	break;
+      case 'd':
+	set_basepath (optarg);
+	break;
+      case 'C':
+      case 'P':
+      case 'S':
+	type = ch;
+	break;
+      case '0':
+      case '1':
+      case '2':
+      case '3':
+	mode = ch;
+	break;
+      case 'l':
+	lookback = atoi (optarg);
+	if (!is_power_of_two (lookback))
+	  FatalError (1, "The lookback value must be a power of 2");
+	lookback = floorlog_2 (lookback);
+	break;
+      case 'k':
+	k = atof (optarg) * 1024;
+	break;
+      case 'h':
+      case '?':
+	fprintf (stderr, "usage: %s [-l lookback] [-f input_file]"
+	"[-d data directory] [-h] [-0|-1|-2|-3] [-C|-P|-S] [-k mem (Kb)]\n",
+		 argv[0]);
+	exit (1);
+      }
+
+  ReadInWords (file_name);
+
+  if (type == 'C')
+    {
+      Select_all ();
+      mem_reqd = WriteOutWords (file_name, MG_COMPLETE_DICTIONARY, lookback);
+    }
+  else
+    {
+      switch (mode)
+	{
+	case '0':
+	  Select_all ();
+	  break;
+	case '1':
+#ifndef SILENT
+	  Message ("Dictionary limit of %.2f Kb", k / 1024);
+#endif
+	  Select_on ((int) k, OccuranceOrder);
+	  break;
+	case '2':
+#ifndef SILENT
+	  Message ("Dictionary limit of %.2f Kb", k / 1024);
+#endif
+	  Select_on ((int) k, DecFreqIncWL);
+	  break;
+	case '3':
+#ifndef SILENT
+	  Message ("Dictionary limit of %.2f Kb", k / 1024);
+#endif
+	  Method3 ((int) k);
+	  break;
+	}
+      if (type == 'P')
+	{
+	  mem_reqd = WriteOutWords (file_name, MG_PARTIAL_DICTIONARY, lookback);
+	}
+      else
+	{
+	  mem_reqd = WriteOutWords (file_name, MG_SEED_DICTIONARY, lookback);
+	}
+    }
+
+#ifndef SILENT
+  Message ("Num words           : %8u -> %8u\n", Num[1], keep[1].num_wds);
+  Message ("Num non-words       : %8u -> %8u\n", Num[0], keep[0].num_wds);
+  Message ("Chars of words      : %8u -> %8u\n", chars[1], keep[1].chars);
+  Message ("Chars of non-words  : %8u -> %8u\n", chars[0], keep[0].chars);
+  Message ("Mem usage           : %8u -> %8u\n",
+	   (Num[0] + Num[1]) * sizeof (char *) + chars[0] + chars[1],
+	     (keep[0].num_wds + keep[1].num_wds) * sizeof (char *) +
+	   keep[0].chars + keep[1].chars);
+  Message ("Actual mem required : %8u\n", mem_reqd);
+#endif
+  return (0);
+}
+
+
+
+
+static void 
+ReadInWords (char *filename)
+{
+  FILE *f;
+  unsigned long i;
+  f = open_file (filename, TEXT_STATS_DICT_SUFFIX, "rb",
+		 MAGIC_STATS_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  fread (&csh, sizeof (csh), 1, f);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(csh.num_docs);
+  NTOHD(csh.num_bytes);
+
+  for (i = 0; i < 2; i++)
+    {
+      frags_stats_header fsh;
+      char *buf;
+      DictWordData *wd;
+      chars[i] = 0;
+
+      fread (&fsh, sizeof (fsh), 1, f);
+      /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(fsh.num_frags);
+      NTOHUL(fsh.mem_for_frags);
+
+      Num[i] = all[i].num_wds = fsh.num_frags;
+
+      /* The +1 on the following line is to leave room for the esc code. */
+      all[i].wd = (DictWordData **) Xmalloc (sizeof (DictWordData *) * Num[i] + 1);
+
+      buf = (char *) Xmalloc (fsh.mem_for_frags);
+      wd = Words[i] = (DictWordData *) Xmalloc (sizeof (DictWordData) * Num[i]);
+      unsigned int j;
+      for (j = 0; j < Num[i]; j++, wd++)
+	{
+	  int len;
+
+	  // read docCount and wordCount
+	  wd->read(f);
+
+	  // read an mgString
+	  len = fgetc (f);
+	  wd->word = (u_char *) buf;
+	  *buf++ = len;
+	  fread (buf, len, 1, f);
+	  buf += len;
+	  all[i].wd[j] = wd;
+	}
+      chars[i] = fsh.mem_for_frags - fsh.num_frags;
+    }
+  fclose (f);
+}
+
+
+static void 
+Alloc_keep_discard (void)
+{
+  keep[0].num_wds = 0;
+  keep[0].wd = (DictWordData **) Xmalloc ((Num[0] + 1) * sizeof (DictWordData *));
+  keep[1].num_wds = 0;
+  keep[1].wd = (DictWordData **) Xmalloc ((Num[1] + 1) * sizeof (DictWordData *));
+  discard[0].num_wds = 0;
+  discard[0].wd = (DictWordData **) Xmalloc ((Num[0] + 1) * sizeof (DictWordData *));
+  discard[1].num_wds = 0;
+  discard[1].wd = (DictWordData **) Xmalloc ((Num[1] + 1) * sizeof (DictWordData *));
+}
+
+
+static int 
+sort_comp (const void *a, const void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  return casecompare (aa->word, bb->word);  /* [RPAP - Jan 97: Stem Index Change] */
+}
+
+
+
+static void 
+SortAndCount_DictData (DictData * dd)
+{
+  unsigned int i;
+  DictWordData **wd;
+  qsort (dd->wd, dd->num_wds, sizeof (DictWordData *), sort_comp);
+  dd->chars = 0;
+  wd = dd->wd;
+  for (i = 0; i < dd->num_wds; i++, wd++)
+    dd->chars += (*wd)->word[0];
+}
+
+
+static void 
+Select_all (void)
+{
+  unsigned int i;
+  Alloc_keep_discard ();
+  keep[0].num_wds = Num[0];
+  for (i = 0; i < Num[0]; i++)
+    keep[0].wd[i] = Words[0] + i;
+  keep[1].num_wds = Num[1];
+  for (i = 0; i < Num[1]; i++)
+    keep[1].wd[i] = Words[1] + i;
+  SortAndCount_DictData (&keep[0]);
+  SortAndCount_DictData (&keep[1]);
+}
+
+
+
+
+static int 
+DecFreqIncWL (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  if (aa->docCount > bb->docCount)
+    return -1;
+  else if (aa->docCount < bb->docCount)
+    return 1;
+  else
+    return bb->word[0] - aa->word[0];
+}
+
+
+static int 
+OccuranceOrder (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  if (aa->words() > bb->words())
+    return 1;
+  else if (aa->words() < bb->words())
+    return -1;
+  else
+    return 0;
+}
+
+
+static void 
+Select_on (int k, heap_comp hc)
+{
+  int i, num, mem;
+  DictWordData **wd;
+
+  Alloc_keep_discard ();
+
+  num = Num[0] + Num[1];
+  wd = (DictWordData **) Xmalloc (num * sizeof (DictWordData *));
+  for (i = 0; (unsigned int)i < Num[0]; i++)
+    wd[i] = Words[0] + i;
+  for (i = 0; (unsigned int)i < Num[1]; i++)
+    wd[i + Num[0]] = Words[1] + i;
+
+  heap_build (wd, sizeof (*wd), num, hc);
+
+  mem = 0;
+  while (k > mem && num)
+    {
+      int idx;
+      DictWordData *word = wd[0];
+#ifdef DEBUG
+      fprintf (stderr, "%4d:%4d:%8d :%8d :%8d : \"%s\"\n",
+	       keep[0].num_wds, keep[1].num_wds,
+	       mem, word->documents(), word->word, word2str (word->word));
+#endif
+      mem += sizeof (u_char *) + word->word[0];
+      heap_deletehead (wd, sizeof (*wd), &num, hc);
+      idx = KIND (word);
+      keep[idx].wd[keep[idx].num_wds++] = word;
+    }
+
+  for (i = 0; i < num; i++)
+    {
+      DictWordData *word = wd[i];
+      int idx = KIND (word);
+      discard[idx].wd[discard[idx].num_wds++] = word;
+    }
+  SortAndCount_DictData (&keep[0]);
+  SortAndCount_DictData (&keep[1]);
+  SortAndCount_DictData (&discard[0]);
+  SortAndCount_DictData (&discard[1]);
+
+  assert (keep[0].num_wds + discard[0].num_wds == Num[0]);
+  assert (keep[1].num_wds + discard[1].num_wds == Num[1]);
+
+}
+
+
+
+static int 
+BigSaving (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  return (aa->saving > bb->saving) ? -1 : (aa->saving < bb->saving);
+}
+
+static int 
+SmallSaving (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  return (aa->saving < bb->saving) ? -1 : (aa->saving > bb->saving);
+}
+
+
+static void 
+CalcCharCounts (DictWordData ** wd, int num,
+		char *char_lens[2], char *len_lens[2],
+		u_long escape[2])
+{
+  long char_freqs[2][256];
+  long len_freqs[2][16];
+  huff_data hd;
+  int i;
+  escape[0] = 0;
+  escape[1] = 0;
+  memset (char_freqs, '\0', sizeof (char_freqs));
+  memset (len_freqs, '\0', sizeof (len_freqs));
+  for (i = 0; i < num; i++, wd++)
+    {
+      u_long freq = (*wd)->documents();
+      u_char *buf = (*wd)->word;
+      int len = *buf++;
+      int idx = KIND (*wd);
+      len_freqs[idx][len] += freq;
+      escape[idx] += freq;
+      for (; len; len--, buf++)
+	char_freqs[idx][(u_long) (*buf)] += freq;
+    }
+  Generate_Huffman_Data (256, char_freqs[0], &hd, NULL);
+  char_lens[0] = hd.clens;
+  Generate_Huffman_Data (256, char_freqs[1], &hd, NULL);
+  char_lens[1] = hd.clens;
+
+  Generate_Huffman_Data (16, len_freqs[0], &hd, NULL);
+  len_lens[0] = hd.clens;
+  Generate_Huffman_Data (16, len_freqs[1], &hd, NULL);
+  len_lens[1] = hd.clens;
+}
+
+
+
+
+
+inline void CalcBitCostForWordData(DictWordData **word, int num,
+				   double freqs_trans[2], u_long escape[2],
+				   char * char_lens[2], char *len_lens[2], 
+				   double esc[2], int num_trans)
+{ 
+  int j;
+
+  for (j = 0; j < num; j++, word++)
+    {
+      float   cbc, wbc;
+      u_char *buf = (*word)->word;
+      int     len = *buf++;
+      u_long  freq = (*word)->documents();
+      int     idx = KIND (*word);
+
+      cbc = len_lens[idx][len];
+      for (; len; len--, buf++)
+	cbc += char_lens[idx][(u_long) (*buf)];
+
+      (*word)->char_bit_cost = (cbc + esc[idx]) * freq;
+
+      wbc = -log2 (freq / (freqs_trans[idx] + escape[idx])) * freq;
+
+      (*word)->saving = ((*word)->char_bit_cost - wbc) /
+	(sizeof (char *) + (*word)->word[0]);
+
+      (*word)->num_trans = num_trans;
+    }
+}
+
+
+
+void 
+CalcBitCost (DictWordData ** discard_word, int discard_num,
+	     DictWordData ** keep_word, int keep_num, double freqs_trans[2],
+	     u_long escape[2], int num_trans)
+{
+  char *char_lens[2];
+  char *len_lens[2];
+  double esc[2];
+  CalcCharCounts (discard_word, discard_num, char_lens, len_lens, escape);
+  esc[0] = -log2 (escape[0] / (freqs_trans[0] + escape[0]));
+  esc[1] = -log2 (escape[1] / (freqs_trans[1] + escape[1]));
+  CalcBitCostForWordData(discard_word, discard_num, freqs_trans, escape,
+			 char_lens, len_lens, esc, num_trans);
+  CalcBitCostForWordData(keep_word, keep_num, freqs_trans, escape,
+			 char_lens, len_lens, esc, num_trans);
+  Xfree (char_lens[0]);
+  Xfree (char_lens[1]);
+  Xfree (len_lens[0]);
+  Xfree (len_lens[1]);
+}
+
+inline void m3_transferWord(DictWordData **toHeap, int *toNum, heap_comp toSaving,
+			    DictWordData **fromHeap, int *fromNum, heap_comp fromSaving,
+			    int *num_trans, double freqs_trans[], int *mem, double *total,
+			    int *count)
+{
+  /* Transfer the word at the top of the keep heap to the
+     discard heap. */
+  DictWordData *word = fromHeap[0];
+  int idx = KIND (word);
+  heap_deletehead (fromHeap, sizeof (word), fromNum, fromSaving);
+  toHeap[*toNum] = word;
+  heap_additem (toHeap, sizeof (word), toNum, toSaving);
+  freqs_trans[idx] -= word->documents();
+  *mem = (*mem) - sizeof (u_char *) + word->word[0];
+  *num_trans += 1;
+  *total += word->saving;
+  *count += 1;
+#ifdef DEBUG
+  fprintf (stderr,
+	   "KEEP -> DISCARD %8d :%8d :%8.0f :%8.0f : \"%s\"\n",
+	   *mem, word->documents(), word->char_bit_cost,
+	   word->saving, word2str (word->word));
+#endif
+}
+
+inline void m3_storeWord(DictWordData **wordHeap, int wordNum, int num_trans, 
+			 double freqs_trans[], u_long escape[], heap_comp hc)
+{
+  DictWordData *word = wordHeap[0];
+  int idx = KIND (word);
+  float wbc;
+#ifdef DEBUG1
+  fprintf (stderr, "KEEP \"%s\" %.2f ->", word2str (word->word),
+	   word->saving);
+#endif
+  wbc = -log2 (word->documents() / (freqs_trans[idx] + escape[idx])) *
+    word->documents();
+  word->saving = (word->char_bit_cost - wbc) /
+    (sizeof (char *) + word->word[0]);
+#ifdef DEBUG1
+  fprintf (stderr, " %.2f\n", word->saving);
+#endif
+  word->num_trans = num_trans;
+  heap_changedhead (wordHeap, sizeof (word), wordNum, hc);
+}
+
+
+static void 
+Method3 (int k)
+{
+  int i, keep_num, discard_num, mem, num_trans, recalc_reqd;
+  int keep_to_discard = 0;
+  int discard_to_keep = 0;
+  int recalcs = 0;
+  double freqs_trans[2], total;
+  u_long escape[2];
+  DictWordData **keep_heap, **discard_heap;
+
+  freqs_trans[0] = freqs_trans[1] = 0;
+  num_trans = 0;
+
+  discard_num = Num[0] + Num[1];
+  discard_heap = (DictWordData **) Xmalloc (discard_num * sizeof (DictWordData *));
+
+  keep_num = 0;
+  keep_heap = (DictWordData **) Xmalloc (discard_num * sizeof (DictWordData *));
+
+
+  for (i = 0; (unsigned int)i < Num[0]; i++)
+    discard_heap[i] = Words[0] + i;
+  for (i = 0; (unsigned int)i < Num[1]; i++)
+    discard_heap[i + Num[0]] = Words[1] + i;
+
+  heap_build (discard_heap, sizeof (*discard_heap), discard_num, DecFreqIncWL);
+
+  mem = 0;
+  while (k > mem && discard_num)
+    {
+      DictWordData *word = discard_heap[0];
+      mem += sizeof (u_char *) + word->word[0];
+      heap_deletehead (discard_heap, sizeof (word), &discard_num, DecFreqIncWL);
+      keep_heap[keep_num++] = word;
+      freqs_trans[KIND (word)] += word->documents();
+      num_trans++;
+    }
+
+  CalcBitCost (discard_heap, discard_num, keep_heap, keep_num,
+	       freqs_trans, escape, num_trans);
+  heap_build (discard_heap, sizeof (*discard_heap), discard_num, BigSaving);
+  heap_build (keep_heap, sizeof (*keep_heap), keep_num, SmallSaving);
+
+  total = 0;
+  recalc_reqd = 0;
+  while (keep_num && discard_num)
+    {
+      if ((keep_num && keep_heap[0]->num_trans < (unsigned)num_trans) ||
+	  (discard_num && discard_heap[0]->num_trans < (unsigned)num_trans))
+	{
+	  if (keep_num && keep_heap[0]->num_trans < (unsigned)num_trans)
+	    {
+	      m3_storeWord(keep_heap, keep_num, num_trans, freqs_trans, escape, SmallSaving);
+	    }
+
+	  if (discard_num && discard_heap[0]->num_trans < (unsigned)num_trans)
+	    {
+	      m3_storeWord(discard_heap, discard_num, num_trans, freqs_trans, escape, BigSaving);
+	    }
+	}
+      else if (keep_heap[0]->saving < discard_heap[0]->saving)
+	{
+	  assert (keep_num && discard_num);
+	  if (keep_num && mem + sizeof (char *) + discard_heap[0]->word[0] > (unsigned)k)
+	    {
+	      /* Transfer the word at the top of the keep heap to the
+	         discard heap. */
+	      m3_transferWord(discard_heap, &discard_num, BigSaving,
+			      keep_heap, &keep_num, SmallSaving,
+			      &num_trans, freqs_trans, &mem, &total, &keep_to_discard);
+	    }
+	  else
+	    {
+	      /* Transfer the word at the top of the discard heap to the
+	         keep heap. */
+	      m3_transferWord(keep_heap, &keep_num, SmallSaving,
+			      discard_heap, &discard_num, BigSaving,
+			      &num_trans, freqs_trans, &mem, &total, &discard_to_keep);
+	    }
+
+	  recalc_reqd = 1;
+
+	}
+      else
+	{
+	  if (recalc_reqd == 0)
+	    break;
+#ifdef DEBUG1
+	  fprintf (stderr, "--------------\n");
+#endif
+	  if (recalcs == MAX_RECALCULATIONS)
+	    break;
+	  CalcBitCost (discard_heap, discard_num, keep_heap, keep_num,
+		       freqs_trans, escape, num_trans);
+	  heap_build (discard_heap, sizeof (*discard_heap),
+		      discard_num, BigSaving);
+	  heap_build (keep_heap, sizeof (keep_heap), keep_num, SmallSaving);
+	  recalc_reqd = 0;
+	  recalcs++;
+	}
+    }
+
+  Alloc_keep_discard ();
+
+  for (i = 0; i < discard_num; i++)
+    {
+      DictWordData *word = discard_heap[i];
+      int idx = KIND (word);
+      assert (IsWord (word) || IsNonWord (word));
+      discard[idx].wd[discard[idx].num_wds++] = word;
+    }
+  for (i = 0; i < keep_num; i++)
+    {
+      DictWordData *word = keep_heap[i];
+      int idx = KIND (word);
+      assert (IsWord (word) || IsNonWord (word));
+      keep[idx].wd[keep[idx].num_wds++] = word;
+    }
+  SortAndCount_DictData (&keep[0]);
+  SortAndCount_DictData (&keep[1]);
+  SortAndCount_DictData (&discard[0]);
+  SortAndCount_DictData (&discard[1]);
+
+  assert (keep[0].num_wds + discard[0].num_wds == Num[0]);
+  assert (keep[1].num_wds + discard[1].num_wds == Num[1]);
+#ifndef SILENT
+  Message ("Keep -> Discard        : %8d", keep_to_discard);
+  Message ("Discard -> Keep        : %8d", discard_to_keep);
+  Message ("Huffman Recalculations : %8d", recalcs);
+  if (recalcs == MAX_RECALCULATIONS)
+    Message ("WARNING: The number of recalculations == %d", MAX_RECALCULATIONS);
+#endif
+}
+
+
+
+
+
+
+
+
+
+
+/****************************************************************************
+ *****									*****
+ *****			Dictionary saving code				*****
+ *****									*****
+ ****************************************************************************/
+
+
+
+static void 
+Write_cdh (FILE * f, compression_dict_header * cdh)
+{
+  /* [RPAP - Jan 97: Endian Ordering] */
+  int i;
+  compression_dict_header tmp = *cdh;
+  HTONUL(tmp.dict_type);
+  HTONUL(tmp.novel_method);
+  for (i = 0; i < TEXT_PARAMS; i++)
+    HTONUL(tmp.params[i]);
+  HTONUL(tmp.num_words[0]);
+  HTONUL(tmp.num_words[1]);
+  HTONUL(tmp.num_word_chars[0]);
+  HTONUL(tmp.num_word_chars[1]);
+  HTONUL(tmp.lookback);
+
+  fwrite (&tmp, sizeof (tmp), 1, f);
+}
+
+
+static void 
+Write_words (FILE * f, DictData * dd)
+{
+  unsigned int i;
+  u_char *curr, *prev = NULL;
+  for (i = 0; i < dd->num_wds; i++)
+    {
+      int len;
+      curr = dd->wd[i]->word;
+      if (prev)
+	/* look for prefix match with prev string */
+	len = prefixlen (prev, curr);
+      else
+	len = 0;
+      fputc ((len << 4) + (curr[0] - len), f);
+      fwrite (curr + len + 1, sizeof (u_char), curr[0] - len, f);
+      prev = curr;
+    }
+
+}
+
+
+static int 
+Uncompressed_size (DictData * dd)
+{
+  unsigned int i, us;
+  for (us = i = 0; i < dd->num_wds; i++)
+    us += dd->wd[i]->word[0];
+  return us;
+}
+
+
+static u_long 
+Write_data (FILE * f, DictData * dd, int lookback)
+{
+  u_long mem_reqd;
+  huff_data *hd;
+  int i; 
+  u_long us = dd->num_wds;
+  long *freqs;
+  u_long huff_words_size[MAX_HUFFCODE_LEN + 1];
+  u_long lencounts[MAX_HUFFCODE_LEN + 1];
+  u_char *lastword[MAX_HUFFCODE_LEN + 1];
+
+  if (!(freqs = new long [dd->num_wds]))
+    FatalError (1, "Unable to allocate memory for freqs");
+
+  for (i = 0; (unsigned)i < dd->num_wds; i++)
+    {
+      freqs[i] = dd->wd[i]->documents();
+      us += dd->wd[i]->word[0];
+    }
+
+  if (!(hd = Generate_Huffman_Data (dd->num_wds, freqs, NULL, NULL)))
+    FatalError (1, "Unable to allocate memory for huffman data");
+
+  delete (freqs);
+  freqs = NULL;
+
+  if (Write_Huffman_Data (f, hd) == -1)
+    FatalError (1, "Unable to write huffman data");
+
+  HTONUL(us);  /* [RPAP - Jan 97: Endian Ordering] */
+  fwrite (&us, sizeof (us), 1, f);
+  NTOHUL(us);  /* [RPAP - Jan 97: Endian Ordering] */
+
+
+/* Calculate the amount of memory that will be required to store the text for
+   each different huffman code len. Every 1<<lookback words for each different
+   codelen length will not be prefixed by previous strings. */
+
+
+  memset (&huff_words_size, '\0', sizeof (huff_words_size));
+  memset (&lencounts, '\0', sizeof (lencounts));
+
+  mem_reqd = 0;
+
+  for (i = 0; (unsigned)i < dd->num_wds; i++)
+    {
+      int codelen = hd->clens[i];
+      u_char *word = dd->wd[i]->word;
+
+      if (!codelen)
+	FatalError (1, "The length of a code for a word was zero");
+
+      huff_words_size[codelen] += word[0] + 1;
+      mem_reqd += word[0] + (lookback != 0);
+#if 0
+      if ((lencounts[codelen] & ((1 << lookback) - 1)) == 0)
+	lastword[codelen] = word;
+      else
+	huff_words_size[codelen] -= prefixlen (lastword[codelen], word);
+#else
+      if ((lencounts[codelen] & ((1 << lookback) - 1)) != 0)
+	{
+	  int save = prefixlen (lastword[codelen], word);
+	  mem_reqd -= save;
+	  huff_words_size[codelen] -= save;
+	}
+      else
+	{
+	  mem_reqd += sizeof (u_char *);
+	}
+      lastword[codelen] = word;
+#endif
+      lencounts[codelen]++;
+    }
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+    HTONUL(huff_words_size[i]);
+
+  fwrite (huff_words_size + hd->mincodelen, sizeof (*huff_words_size),
+	  hd->maxcodelen - hd->mincodelen + 1, f);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+    NTOHUL(huff_words_size[i]);
+
+  Write_words (f, dd);
+
+  delete hd->clens;
+  delete hd;
+
+  return mem_reqd;
+}
+
+
+
+static void 
+Write_charfreqs (FILE * f, DictData * dd, int words,
+		 int zero_freq_permitted)
+{
+  unsigned int j;
+  long freqs[256];
+  DictWordData **wd = dd->wd;
+  huff_data *hd;
+
+  memset (freqs, '\0', sizeof (freqs));
+
+  for (j = 0; j < dd->num_wds; j++, wd++)
+    {
+      u_char *buf = (*wd)->word;
+      int len = *buf++;
+      for (; len; len--, buf++)
+	freqs[(u_long) (*buf)] += (*wd)->documents();
+    }
+
+  if (!zero_freq_permitted)
+    for (j = 0; j < 256; j++)
+      if (!freqs[j] && PESINAWORD (j) == words)
+	freqs[j] = 1;
+
+  if (!(hd = Generate_Huffman_Data (256, freqs, NULL, NULL)))
+    FatalError (1, "Unable to allocate memory for huffman data");
+
+  if (Write_Huffman_Data (f, hd) == -1)
+    FatalError (1, "Unable to write huffman data");
+
+  delete hd->clens;
+  delete hd;
+}
+
+
+
+
+static void 
+Write_wordlens (FILE * f, DictData * dd, int zero_freq_permitted)
+{
+  unsigned int j;
+  long freqs[16];
+  DictWordData **wd = dd->wd;
+  huff_data *hd;
+
+  memset (freqs, '\0', sizeof (freqs));
+
+  for (j = 0; j < dd->num_wds; j++, wd++)
+    freqs[(*wd)->word[0]] += (*wd)->documents();
+
+  if (!zero_freq_permitted)
+    for (j = 0; j < 16; j++)
+      if (!freqs[j])
+	freqs[j] = 1;
+
+  if (!(hd = Generate_Huffman_Data (16, freqs, NULL, NULL)))
+    FatalError (1, "Unable to allocate memory for huffman data");
+
+  if (Write_Huffman_Data (f, hd) == -1)
+    FatalError (1, "Unable to write huffman data");
+
+
+  delete hd->clens;
+  delete hd;
+}
+
+
+
+static u_long 
+WriteOutWords (char *file_name, u_long type, int lookback)
+{
+  FILE *f;
+  int i;
+  u_long mem_reqd = 0;
+
+  compression_dict_header cdh;
+  f = create_file (file_name, TEXT_DICT_SUFFIX, "w+b",
+		   MAGIC_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  memset (&cdh, '\0', sizeof(cdh));
+
+  cdh.dict_type = type;
+  cdh.novel_method = (type == MG_SEED_DICTIONARY) ? novel_method :
+    MG_NOVEL_HUFFMAN_CHARS;
+
+  cdh.num_words[1] = keep[1].num_wds;
+  cdh.num_words[0] = keep[0].num_wds;
+  cdh.num_word_chars[1] = Uncompressed_size (&keep[1]);
+  cdh.num_word_chars[0] = Uncompressed_size (&keep[0]);
+  cdh.lookback = lookback;
+
+  Write_cdh (f, &cdh);
+
+  for (i = 0; i < 2; i++)
+    switch (type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  mem_reqd += Write_data (f, &keep[i], lookback);
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (keep[i].num_wds)
+	    {
+	      int j;
+	      DictWordData esc;
+	      esc.docCount = 0;
+	      esc.word = (u_char *) "";
+	      keep[i].wd[keep[i].num_wds++] = &esc;
+	      for (j = 0; (unsigned)j < discard[i].num_wds; j++)
+		esc.docCount += discard[i].wd[j]->documents();
+	      if (!esc.docCount)
+		esc.docCount++;
+	      mem_reqd += Write_data (f, &keep[i], lookback);
+	    }
+	  Write_charfreqs (f, &discard[i], i, 1);
+	  Write_wordlens (f, &discard[i], 1);
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (keep[i].num_wds)
+	    {
+	      int j;
+	      DictWordData esc;
+	      esc.docCount = 0;
+	      esc.word = (u_char *) "";
+	      keep[i].wd[keep[i].num_wds++] = &esc;
+	      for (j = 0; (unsigned)j < all[i].num_wds; j++)
+		if (all[i].wd[j]->documents() == 1)
+		  esc.docCount++;
+	      if (!esc.docCount)
+		esc.docCount++;
+	      mem_reqd += Write_data (f, &keep[i], lookback);
+	    }
+	  switch (novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      Write_charfreqs (f, &all[i], i, 0);
+	      Write_wordlens (f, &all[i], 0);
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    default:
+	      FatalError (1, "Bad novel method");
+	    }
+	}
+	break;
+      }
+  fclose (f);
+  return mem_reqd;
+}
Index: /trunk/indexers/mgpp/text/mgpp_decompress_text.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_decompress_text.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_decompress_text.cpp	(revision 3365)
@@ -0,0 +1,100 @@
+/**************************************************************************
+ *
+ * mgpp_decompress_text.cpp -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "TextGet.h"
+#include "messages.h"
+#include "mg_files.h"
+
+int main (int argc, char **argv) {
+  int ch;
+  char *filename = "";
+  char *basePath = "";
+  UCArray level;
+  SetCStr (level, "Document");
+
+  opterr = 0;
+  msg_prefix = argv[0];
+
+  // process the command line arguments
+  while ((ch = getopt (argc, argv, "f:d:K:h")) != -1) {
+    switch (ch) {
+    case 'f':		/* input file */
+      filename = optarg;
+      break;
+    case 'd':
+      basePath = optarg;
+      set_basepath (optarg);
+      break;
+    case 'K':
+      SetCStr (level, optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-K level] [-d directory] -f name\n",
+	       argv[0]);
+      exit (1);
+    }
+  }
+
+  // load up the text information
+  TextData td;
+  if (!td.LoadData (basePath, filename)) {
+    FatalError (1, "Couldn't load text information for \"%s\"", filename);
+  }
+  
+  // output each document in the level
+//    cout << td.levels << "\n";
+  
+  TextLevelInfo levelInfo = td.levels.levelInfo[level];
+  unsigned long docNum = 1;
+  UCArray docText;
+  while (docNum <= levelInfo.numEntries) {
+//      TextIdx docIdx;
+//      if (GetDocIdx (td, level, docNum, docIdx)) {
+//        cout << "doc: " << docNum << "\n";
+//        cout << docIdx;
+//      }
+    
+    if (!GetDocText (td, level, docNum, docText)) {
+      FatalError (1, "Error while trying to get document %u", docNum);
+    }
+
+    cout << docText << "\n";
+    
+    docNum++;
+  }
+  
+  return 0;
+}
+
+
Index: /trunk/indexers/mgpp/text/mgpp_fast_comp_dict.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_fast_comp_dict.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_fast_comp_dict.1	(revision 3365)
@@ -0,0 +1,69 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_fast_comp_dict 1 \*(Dt CITRI
+.SH NAME
+mgpp_fast_comp_dict \- build a fast loading compression dictionary
+.SH SYNOPSIS
+.B mgpp_fast_comp_dict
+[
+.B \-h
+]
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_fast_comp_dict
+builds a fast-loading compression dictionary from the standard
+compression dictionary.  The new dictionary has the property that
+.BR mgppquery (1)
+can load it in very quickly, making the startup time for
+.BR mgppquery (1)
+much shorter.  The new dictionary is larger on disk than the original,
+but they are no different when it comes to memory usage.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.text.dict
+Compression dictionary.
+.TP
+.B *.text.dict.fast
+Fast loading compression dictionary.
+.TP
+.B *.text.dict.aux
+Auxiliary compression dictionary.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/indexers/mgpp/text/mgpp_fast_comp_dict.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_fast_comp_dict.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_fast_comp_dict.cpp	(revision 3365)
@@ -0,0 +1,707 @@
+/**************************************************************************
+ *
+ * mgpp_fast_comp_dict.cpp -- Program to generate a fast compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "huffman.h"
+#include "messages.h"
+#include "memlib.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "local_strings.h"
+#include "mg.h"
+#include "text.h"
+#include "invf.h"
+#include "mg_files.h"
+#include "locallib.h"
+#include "words.h"
+
+
+
+#define ALIGN_SIZE(p,t) (((p) + (sizeof(t)-1)) & (~(sizeof(t)-1)))
+
+#define WORDNO(p, base) ((((char*)(p))-((char*)(base)))/sizeof(u_char*))
+#define FIXUP(p) (fixup[WORDNO(p,buffer)/8] |= (1<<(WORDNO(p, buffer) & 7)))
+
+#define IS_FIXUP(p) ((fixup[WORDNO(p,buffer)/8] & (1<<(WORDNO(p, buffer) & 7))) != 0)
+
+#define FIXUP_VALS(vals) do {						\
+	int i;								\
+	for (i=0; i < MAX_HUFFCODE_LEN+1; i++)				\
+	  FIXUP(&vals[i]);						\
+      } while(0)
+
+
+
+static u_long mem_for_comp_dict (char *filename);
+static void load_comp_dict (char *filename);
+static void save_fast_dict (char *filename);
+static void unfixup_buffer (void);
+
+
+static void *buffer;
+static void *cur;
+static u_char *fixup;
+static u_long mem, fixup_mem;
+
+int main (int argc, char **argv)
+{
+  int ch;
+  char *filename = "";
+  opterr = 0;
+  msg_prefix = argv[0];
+  while ((ch = getopt (argc, argv, "f:d:h")) != -1)
+    switch (ch)
+      {
+      case 'f':		/* input file */
+	filename = optarg;
+	break;
+      case 'd':
+	set_basepath (optarg);
+	break;
+      case 'h':
+      case '?':
+	fprintf (stderr, "usage: %s [-f input_file] [-d data directory] [-h]\n",
+		 argv[0]);
+	exit (1);
+      }
+
+  mem = mem_for_comp_dict (filename);
+
+  fixup_mem = (ALIGN_SIZE (mem, u_char *) / sizeof (u_char *) + 7) / 8;
+
+  cur = buffer = Xmalloc (mem);
+  memset (buffer, 0, mem);
+  fixup = (u_char *) Xmalloc (fixup_mem);
+  memset (fixup, 0, fixup_mem);
+
+#ifndef SILENT
+  Message ("Estimated memory for fast_dict %u", mem);
+  Message ("Estimated memory for fixups %u", fixup_mem);
+#endif
+
+  load_comp_dict (filename);
+
+#ifndef SILENT
+  Message ("Actual memory for fast_dict %u", (char *) cur - (char *) buffer);
+#endif
+
+  if ((u_long) cur > (u_long) buffer + mem)
+    FatalError (1, "The buffer was not big enough for the dictionary");
+
+  {
+    /* [RPAP - Jan 97: Endian Ordering] */
+    compression_dict *cd = (compression_dict *) buffer;
+    int i, which;
+
+    /* cfh */
+    for (which = 0; which <= 1; which++)
+      {
+	int j;
+	
+	HTONSI(cd->cfh[which]->hd.num_codes);
+	HTONSI(cd->cfh[which]->hd.mincodelen);
+	HTONSI(cd->cfh[which]->hd.maxcodelen);
+	for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	  {
+	    HTONSI(cd->cfh[which]->hd.lencount[j]);
+	    HTONUL(cd->cfh[which]->hd.min_code[j]);
+	  }
+	HTONUL(cd->cfh[which]->uncompressed_size);
+	for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	  HTONUL(cd->cfh[which]->huff_words_size[j]);
+      }
+    HTONUL(cd->MemForCompDict);
+    /* ad */
+    if (cd->cdh.novel_method == MG_NOVEL_DELTA ||
+	cd->cdh.novel_method == MG_NOVEL_HYBRID)
+      for (which = 0; which <= 1; which++)
+	{
+	  int j;
+	  
+	  HTONUL(cd->ad->afh[which].num_frags);
+	  HTONUL(cd->ad->afh[which].mem_for_frags);
+	  for (j = 0; j < 33; j++)
+	    {
+	      HTONSI(cd->ad->blk_start[which][j]);
+	      HTONSI(cd->ad->blk_end[which][j]);
+	    }
+	}
+    /* cdh */
+    HTONUL(cd->cdh.dict_type);
+    HTONUL(cd->cdh.novel_method);
+    for (i = 0; i < TEXT_PARAMS; i++)
+      HTONUL(cd->cdh.params[which]);
+    HTONUL(cd->cdh.num_words[0]);
+    HTONUL(cd->cdh.num_words[1]);
+    HTONUL(cd->cdh.num_word_chars[0]);
+    HTONUL(cd->cdh.num_word_chars[1]);
+    HTONUL(cd->cdh.lookback);
+    HTONSI(cd->fast_loaded);
+  }
+
+  unfixup_buffer ();
+
+  save_fast_dict (filename);
+
+  return (0);
+}
+
+
+
+static void 
+unfixup_buffer ()
+{
+  u_long *p;
+  for (p = (u_long *) buffer; (u_long) p < (u_long) cur; p++)
+    {
+      if (IS_FIXUP (p))
+	*p = *p - (u_long) buffer;
+    }
+}
+
+
+
+
+static u_long 
+mem_for_aux_dict (compression_dict_header * /*cdh*/, char *filename)
+{
+  int i;
+  u_long mem = sizeof (auxiliary_dict);
+  FILE *aux;
+
+  aux = open_file (filename, TEXT_DICT_AUX_SUFFIX, "rb",
+		   MAGIC_AUX_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  for (i = 0; i <= 1; i++)
+    {
+      aux_frags_header afh;
+      fread (&afh, sizeof (afh), 1, aux);
+      NTOHUL(afh.num_frags);  /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(afh.mem_for_frags);  /* [RPAP - Jan 97: Endian Ordering] */
+      mem += afh.num_frags * sizeof (u_char *);
+      mem = ALIGN_SIZE (mem + afh.mem_for_frags, u_char *);
+      fseek (aux, afh.mem_for_frags, SEEK_CUR);
+    }
+  fclose (aux);
+
+  return mem;
+}
+
+
+
+static u_long 
+mem_for_words (FILE * dict, compression_dict_header * cdh,
+	       comp_frags_header * cfh)
+{
+  u_long mem = 0;
+  long i, lookback;
+  int ptrs_reqd = 0;
+  int mem_reqd = 0;
+
+  lookback = cdh->lookback;
+
+  for (i = cfh->hd.mincodelen; i <= cfh->hd.maxcodelen; i++)
+    {
+      ptrs_reqd += (cfh->hd.lencount[i] + ((1 << lookback) - 1)) >> lookback;
+      mem_reqd += cfh->huff_words_size[i];
+    }
+
+  mem += ptrs_reqd * sizeof (u_char *);
+  mem += (MAX_HUFFCODE_LEN + 1) * sizeof (u_char **);
+  mem += mem_reqd;
+
+  for (i = 0; i < cfh->hd.num_codes; i++)
+    {
+      register int val;
+      for (val = getc (dict) & 0xf; val; val--)
+	getc (dict);
+    }
+  return ALIGN_SIZE (mem, u_char *);
+}
+
+static u_long mem_skip_hd(FILE *dict, u_long mem)
+{ huff_data hd;
+
+  mem += sizeof (hd);
+  Read_Huffman_Data (dict, &hd, NULL, NULL);
+  if (hd.clens)
+    delete hd.clens;
+  mem += hd.num_codes * sizeof (unsigned long);
+  mem += (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *);
+  return mem;
+}
+
+static u_long mem_read_cfh(FILE *dict, compression_dict_header *cdh, comp_frags_header *cfh, 
+			   u_long mem)
+{
+  mem += sizeof (comp_frags_header);
+  Read_cfh (dict, cfh, NULL, NULL);
+
+  /* Don't need to count the space for the clens of the huffman data */
+
+  mem += mem_for_words (dict, cdh, cfh);
+  if (cfh->hd.clens)
+    delete cfh->hd.clens;
+
+  return mem;
+}
+
+
+static u_long 
+mem_for_comp_dict (char *filename)
+{
+  u_long mem = sizeof (compression_dict);
+  compression_dict_header cdh;
+  comp_frags_header cfh;
+  FILE *dict;
+  int which;
+  int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+  dict = open_file (filename, TEXT_DICT_SUFFIX, "rb",
+		    MAGIC_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  fread (&cdh, sizeof (cdh), 1, dict);
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(cdh.dict_type);
+  NTOHUL(cdh.novel_method);
+  for (i = 0; i < TEXT_PARAMS; i++)
+    NTOHUL(cdh.params[i]);
+  NTOHUL(cdh.num_words[0]);
+  NTOHUL(cdh.num_words[1]);
+  NTOHUL(cdh.num_word_chars[0]);
+  NTOHUL(cdh.num_word_chars[1]);
+  NTOHUL(cdh.lookback);
+
+  for (which = 0; which < 2; which++)
+    switch (cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  mem = mem_read_cfh(dict, &cdh, &cfh, mem);
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  // huff_data hd;
+	  if (cdh.num_words[which])
+	    {
+	      mem = mem_read_cfh(dict, &cdh, &cfh, mem);
+	    }
+
+	  mem = mem_skip_hd(dict, mem);
+	  mem = mem_skip_hd(dict, mem);
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  // huff_data hd;
+	  if (cdh.num_words[which])
+	    {
+	      mem = mem_read_cfh(dict, &cdh, &cfh, mem);
+	    }
+	  switch (cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      mem = mem_skip_hd(dict, mem);
+	      mem = mem_skip_hd(dict, mem);
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    }
+	  break;
+	}
+      }
+  fclose (dict);
+
+  if (cdh.novel_method == MG_NOVEL_DELTA ||
+      cdh.novel_method == MG_NOVEL_HYBRID)
+    mem += mem_for_aux_dict (&cdh, filename);
+
+  return ALIGN_SIZE (mem, u_char *);
+}
+
+
+
+void *
+getmem (u_long size, int align)
+{
+  void *res;
+  cur = (void *) (((u_long) cur + (align - 1)) & (~(align - 1)));
+  res = cur;
+  cur = (char *) cur + size;
+  if ((u_long) cur > (u_long)buffer + mem)
+    FatalError (1, "The buffer was not big enough for the dictionary");
+  return res;
+}
+
+
+
+
+
+
+
+
+
+
+
+static auxiliary_dict *
+LoadAuxDict (compression_dict_header * cdh,
+	     char *filename)
+{
+  auxiliary_dict *ad;
+  int i;
+  FILE *aux;
+
+  aux = open_file (filename, TEXT_DICT_AUX_SUFFIX, "rb",
+		   MAGIC_AUX_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  ad = (auxiliary_dict *) getmem (sizeof (auxiliary_dict), sizeof (u_char *));
+
+  for (i = 0; i <= 1; i++)
+    {
+      unsigned int j;
+      u_char *pos;
+
+      fread (&ad->afh[i], sizeof (aux_frags_header), 1, aux);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(ad->afh[i].num_frags);
+      NTOHUL(ad->afh[i].mem_for_frags);
+
+      ad->word_data[i] = (u_char *) getmem (ad->afh[i].mem_for_frags, 1);
+      FIXUP (&ad->word_data[i]);
+
+      ad->words[i] = (u_char **) getmem (ad->afh[i].num_frags * sizeof (u_char *),
+			     sizeof (u_char *));
+      FIXUP (&ad->words[i]);
+
+      fread (ad->word_data[i], ad->afh[i].mem_for_frags, sizeof (u_char), aux);
+
+      pos = ad->word_data[i];
+      for (j = 0; j < ad->afh[i].num_frags; j++)
+	{
+	  ad->words[i][j] = pos;
+	  FIXUP (&ad->words[i][j]);
+	  pos += *pos + 1;
+	}
+      if (cdh->novel_method == MG_NOVEL_HYBRID)
+	{
+	  int num;
+	  num = 1;
+	  ad->blk_start[i][0] = 0;
+	  ad->blk_end[i][0] = cdh->num_words[i] - 1;
+	  while (num < 33)
+	    {
+	      ad->blk_start[i][num] = ad->blk_end[i][num - 1] + 1;
+	      ad->blk_end[i][num] = ad->blk_start[i][num] +
+		(ad->blk_end[i][num - 1] - ad->blk_start[i][num - 1]) * 2;
+	      num++;
+	    }
+	}
+    }
+  fclose (aux);
+  return (ad);
+}
+
+
+
+
+
+static u_char ***
+ReadInWords (FILE * dict, compression_dict * cd,
+	     comp_frags_header * cfh, u_char ** escape)
+{
+  int i, lookback;
+  int ptrs_reqd = 0;
+  int mem_reqd = 0;
+  int num_set[MAX_HUFFCODE_LEN + 1];
+  u_char *next_word[MAX_HUFFCODE_LEN + 1];
+  u_char **vals;
+  u_char ***values;
+  u_char word[MAXWORDLEN + 1];
+  u_char last_word[MAX_HUFFCODE_LEN + 1][MAXWORDLEN + 1];
+
+  lookback = cd->cdh.lookback;
+
+  for (i = cfh->hd.mincodelen; i <= cfh->hd.maxcodelen; i++)
+    {
+      ptrs_reqd += (cfh->hd.lencount[i] + ((1 << lookback) - 1)) >> lookback;
+      mem_reqd += cfh->huff_words_size[i];
+    }
+
+  vals = (u_char **) getmem (ptrs_reqd * sizeof (*vals), sizeof (u_char *));
+
+  values = (u_char ***) getmem ((MAX_HUFFCODE_LEN + 1) * sizeof (u_char **), sizeof (u_char **));
+
+  next_word[0] = (u_char *) getmem (mem_reqd, sizeof (char));
+
+  cd->MemForCompDict += ptrs_reqd * sizeof (*vals) +
+    (MAX_HUFFCODE_LEN + 1) * sizeof (u_char **) +
+    mem_reqd;
+
+  values[0] = vals;
+  FIXUP (&values[0]);
+  values[0][0] = next_word[0];
+  FIXUP (&values[0][0]);
+  for (i = 1; i <= cfh->hd.maxcodelen; i++)
+    {
+      int next_start = (values[i - 1] - vals) +
+      ((cfh->hd.lencount[i - 1] + ((1 << lookback) - 1)) >> lookback);
+      values[i] = &vals[next_start];
+      FIXUP (&values[i]);
+      next_word[i] = next_word[i - 1] + cfh->huff_words_size[i - 1];
+      values[i][0] = next_word[i];
+      FIXUP (&values[i][0]);
+    }
+
+  memset (num_set, '\0', sizeof (num_set));
+
+  for (i = 0; i < cfh->hd.num_codes; i++)
+    {
+      register int val, copy;
+      register int len = cfh->hd.clens[i];
+      val = getc (dict);
+      copy = (val >> 4) & 0xf;
+      val &= 0xf;
+
+      fread (word + copy + 1, sizeof (u_char), val, dict);
+      *word = val + copy;
+
+      if ((num_set[len] & ((1 << lookback) - 1)) == 0)
+	{
+	  values[len][num_set[len] >> lookback] = next_word[len];
+	  FIXUP (&values[len][num_set[len] >> lookback]);
+	  memcpy (next_word[len], word, *word + 1);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    {
+	      *escape = next_word[len];
+	      FIXUP (escape);
+	    }
+	  next_word[len] += *word + 1;
+	}
+      else
+	{
+	  copy = prefixlen (last_word[len], word);
+	  memcpy (next_word[len] + 1, word + copy + 1, *word - copy);
+	  *next_word[len] = (copy << 4) + (*word - copy);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    {
+	      *escape = next_word[len];
+	      FIXUP (escape);
+	    }
+	  next_word[len] += (*word - copy) + 1;
+	}
+      memcpy (last_word[len], word, *word + 1);
+      num_set[len]++;
+    }
+  if (cfh->hd.clens)
+    delete cfh->hd.clens;
+  cfh->hd.clens = NULL;
+  return values;
+}
+
+
+static unsigned long **
+Generate_Fast_Huffman_Vals (huff_data * data, u_long * mem)
+{
+  int i;
+  unsigned long *fcode[MAX_HUFFCODE_LEN + 1];
+  unsigned long **values;
+  unsigned long *vals;
+
+  if (!data)
+    return (NULL);
+  vals = (unsigned long *) getmem (data->num_codes * sizeof (*vals), sizeof (long *));
+  values = (unsigned long **) getmem ((MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *),
+		   sizeof (long *));
+
+  memset (values, '\0', (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *));
+
+  if (mem)
+    *mem += data->num_codes * sizeof (*vals) +
+      (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *);
+
+  fcode[0] = values[0] = &vals[0];
+  FIXUP (&values[0]);
+  for (i = 1; i <= data->maxcodelen; i++)
+    {
+      fcode[i] = values[i] = &vals[(values[i - 1] - vals) + data->lencount[i - 1]];
+      FIXUP (&values[i]);
+    }
+
+  for (i = 0; i < data->num_codes; i++)
+    if (data->clens[i])
+      *fcode[(int) (data->clens[i])]++ = i;
+  return (values);
+}
+
+
+static void load_read_huffman(FILE *dict, compression_dict *cd, int which)
+{
+  huff_data *hd;
+  u_long   **vals;
+
+  hd = (huff_data *) getmem (sizeof (huff_data), sizeof (char *));
+  cd->MemForCompDict += sizeof (huff_data);
+  Read_Huffman_Data (dict, hd, &cd->MemForCompDict, NULL);
+  vals = Generate_Fast_Huffman_Vals (hd, &cd->MemForCompDict);
+  cd->chars_huff[which] = hd;
+  FIXUP (&cd->chars_huff[which]);
+  cd->chars_vals[which] = vals;
+  FIXUP (&cd->chars_vals[which]);
+  if (hd->clens)
+    delete hd->clens;
+  hd->clens = NULL;
+}
+
+static void load_read_words(FILE *dict, compression_dict *cd, int which, int fixescape)
+{
+  cd->cfh[which] = (comp_frags_header *) getmem (sizeof (*cd->cfh[which]), sizeof (u_char *));
+  cd->MemForCompDict += sizeof (*cd->cfh[which]);
+  Read_cfh (dict, cd->cfh[which], &cd->MemForCompDict, NULL);
+
+  cd->values[which] = ReadInWords (dict, cd, cd->cfh[which], NULL);
+  FIXUP (&cd->cfh[which]);
+  FIXUP (&cd->values[which]);
+  if (fixescape)
+    { FIXUP(&cd->escape[which]);
+    }
+  else
+    {
+      cd->escape[which] = NULL;
+    }
+}
+
+static void 
+load_comp_dict (char *filename)
+{
+  FILE *dict;
+  int which;
+  compression_dict *cd;
+
+  cd = (compression_dict *) getmem (sizeof (compression_dict), sizeof (u_char *));
+  cd->fast_loaded = 1;
+
+  dict = open_file (filename, TEXT_DICT_SUFFIX, "rb",
+		    MAGIC_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  Read_cdh (dict, &cd->cdh, NULL, NULL);
+
+  for (which = 0; which < 2; which++)
+    switch (cd->cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  load_read_words(dict, cd, which, 0);
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (cd->cdh.num_words[which])
+	    {
+	      load_read_words(dict, cd, which, 1);
+	    }
+
+	  load_read_huffman(dict, cd, which);
+	  load_read_huffman(dict, cd, which);
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (cd->cdh.num_words[which])
+	    {
+	      load_read_words(dict, cd, which, 1);
+	    }
+	  switch (cd->cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      load_read_huffman(dict, cd, which);
+	      load_read_huffman(dict, cd, which);
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    }
+	  break;
+	}
+      }
+  fclose (dict);
+
+
+  if (cd->cdh.novel_method == MG_NOVEL_DELTA ||
+      cd->cdh.novel_method == MG_NOVEL_HYBRID)
+    {
+      cd->ad = LoadAuxDict (&cd->cdh, filename);
+      FIXUP (&cd->ad);
+    }
+}
+
+static void 
+save_fast_dict (char *filename)
+{
+  FILE *fdict;
+
+  fdict = create_file (filename, TEXT_DICT_FAST_SUFFIX, "wb",
+		       MAGIC_FAST_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  {
+    u_long *p;
+    for (p = (u_long *) buffer; (u_long) p < (u_long) cur; p++)
+      {
+	if (IS_FIXUP (p))
+	  HTONUL(*p);
+      }
+  }
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONUL(mem);
+  HTONUL(fixup_mem);
+
+  fwrite (&mem, sizeof (mem), 1, fdict);
+  fwrite (&fixup_mem, sizeof (fixup_mem), 1, fdict);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(mem);
+  NTOHUL(fixup_mem);
+
+  fwrite (buffer, sizeof (u_char), mem, fdict);
+  fwrite (fixup, sizeof (u_char), fixup_mem, fdict);
+
+  fclose (fdict);
+}
Index: /trunk/indexers/mgpp/text/mgpp_invf_dict.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_invf_dict.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_invf_dict.1	(revision 3365)
@@ -0,0 +1,79 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_invf_dict 1 \*(Dt CITRI
+.SH NAME
+mgpp_invf_dict \- build an `on-disk' stemmed dictionary
+.SH SYNOPSIS
+.B mgpp_invf_dict
+[
+.B \-h
+]
+[
+.BI \-b " entries-per-block"
+]
+.if n .ti +13n
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_invf_dict
+builds an `on-disk' stemmed dictionary from the
+compressed stemmed dictionary that can be used during querying.
+Each entry in the dictionary contains the disk
+pointer to the associated inverted file entry, and various other pieces
+of information about the stemmed word. The dictionary is blocked with
+each block containing the same number of entries (not the same size).
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectoryyyyyyyy\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-b " entries-per-block"
+The dictionary is stored in blocks on disk; this option is used to 
+specify the number of entries per block. The default is 16.
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.invf
+Inverted file.
+.TP
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.invf.dict.blocked
+The `on-disk' stemmed dictionary.
+.TP
+.B *.invf.idx
+The index into the inverted file.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/indexers/mgpp/text/mgpp_invf_dict.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_invf_dict.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_invf_dict.cpp	(revision 3365)
@@ -0,0 +1,237 @@
+/**************************************************************************
+ *
+ * mgpp_invf_dict.cpp -- Program to build the blocked stemmed dictionary
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "messages.h"
+#include "mg_files.h"
+#include "invf.h"
+
+static void process_files (char *filename, unsigned long entriesPerBlock) {
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // open the inverted index file
+  FILE *invfIdxFile = open_file (filename, INVF_IDX_SUFFIX, "rb",
+				 MAGIC_INVI, MG_ABORT);
+
+  // create the blocked dictionary
+  FILE *blockDictFile = create_file (filename, INVF_DICT_BLOCKED_SUFFIX, "wb",
+				     MAGIC_STEM, MG_ABORT);
+  block_dict_header bdh;
+  bdh.lookback = idh.lookback;
+  bdh.word_dict_start = idh.word_dict_start;
+  bdh.word_dict_size = idh.word_dict_size;
+  bdh.tag_dict_start = idh.tag_dict_start;
+  bdh.tag_dict_size = idh.tag_dict_size;
+  bdh.num_docs = idh.num_docs;
+  bdh.num_frags = idh.num_frags;
+  bdh.num_words = idh.num_words;
+  bdh.total_bytes = idh.total_bytes;
+  bdh.index_string_bytes = idh.index_string_bytes;
+  bdh.num_levels = idh.num_levels;
+  bdh.Write (blockDictFile);
+
+  
+  // write out the word part of the dictionary
+
+  bdh.entries_per_wblk = entriesPerBlock;
+  bdh.max_wblk_size = 0;
+  bdh.wblk_start = ftell (blockDictFile);
+  
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+
+  block_idx wordIdx;
+  word_block_dict_el wordBlockEl;
+  wordBlockEl.SetNumLevels (idh.num_levels);
+  
+  
+  unsigned long wordNum;
+  unsigned long wordInvfPtr;
+  UCArray lastEl;
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+    // read in the next word and inverted file pointer
+    wordEl.Read (dictFile, idh.num_levels);
+    ReadUL (invfIdxFile, wordInvfPtr);
+
+    // remember this word (and position) if this is the start
+    // of a new block
+    if (wordNum % entriesPerBlock == 0) {
+      block_idx_info elIdx;
+      elIdx.el = wordEl.el;
+      elIdx.block_ptr = ftell (blockDictFile);
+
+      // see if this block is the longest so far
+      if (wordIdx.size() > 0) {
+	unsigned long blockLen = elIdx.block_ptr -
+	  (*(wordIdx.end()-1)).block_ptr;
+	if (blockLen > bdh.max_wblk_size) bdh.max_wblk_size = blockLen;
+      }
+      
+      wordIdx.push_back (elIdx);
+      lastEl.erase (lastEl.begin(), lastEl.end()); // output full word
+    }
+
+    // copy the information for this word
+    wordBlockEl.el = wordEl.el;
+    wordBlockEl.frag_occur = wordEl.frag_occur;
+    wordBlockEl.freq = wordEl.freq;
+    wordBlockEl.invf_ptr = wordInvfPtr;
+    unsigned long tempI;
+    for (tempI=0; tempI<idh.num_levels; tempI++)
+      wordBlockEl.levelFreqs[tempI] = wordEl.levelFreqs[tempI];
+
+    // write out the word
+    wordBlockEl.Write (blockDictFile, &lastEl, idh.num_levels);
+
+    lastEl = wordBlockEl.el;
+  }
+
+
+  // write out the tag part of the dictionary
+
+  bdh.entries_per_tblk = entriesPerBlock;
+  bdh.max_tblk_size = 0;
+  bdh.tblk_start = ftell (blockDictFile);
+  
+  fseek (dictFile, idh.tag_dict_start, SEEK_SET);
+
+  block_idx tagIdx;
+  block_dict_el tagBlockEl;
+  
+  unsigned long tagNum;
+  unsigned long tagInvfPtr;
+  dict_el tagEl;
+  lastEl.erase (lastEl.begin(), lastEl.end());
+  for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+    // read in the next tag and inverted file pointer
+    tagEl.Read (dictFile);
+    ReadUL (invfIdxFile, tagInvfPtr);
+
+    // remember this tag (and position) if this is the start
+    // of a new block
+    if (tagNum % entriesPerBlock == 0) {
+      block_idx_info elIdx;
+      elIdx.el = tagEl.el;
+      elIdx.block_ptr = ftell (blockDictFile);
+
+      // see if this block is the longest so far
+      if (tagIdx.size() > 0) {
+	unsigned long blockLen = elIdx.block_ptr -
+	  (*(tagIdx.end()-1)).block_ptr;
+	if (blockLen > bdh.max_tblk_size) bdh.max_tblk_size = blockLen;
+      }
+      
+      tagIdx.push_back (elIdx);
+      lastEl.erase (lastEl.begin(), lastEl.end()); // output full word
+    }
+
+    // copy the information for this tag
+    tagBlockEl.el = tagEl.el;
+    tagBlockEl.frag_occur = tagEl.frag_occur;
+    tagBlockEl.freq = tagEl.freq;
+    tagBlockEl.invf_ptr = tagInvfPtr;
+
+    // write out the tag
+    tagBlockEl.Write (blockDictFile, &lastEl);
+
+    lastEl = tagBlockEl.el;
+  }
+
+
+  // write out the element indexes
+  bdh.num_wblks = wordIdx.size();
+  bdh.wblk_idx_start = ftell (blockDictFile);
+  WriteBlockIdx (blockDictFile, wordIdx);
+  
+  bdh.num_tblks = tagIdx.size();
+  bdh.tblk_idx_start = ftell (blockDictFile);
+  WriteBlockIdx (blockDictFile, tagIdx);
+
+  // write out the blocked dictionary header
+  fseek (blockDictFile, sizeof(unsigned long), SEEK_SET);
+  bdh.Write (blockDictFile);
+
+
+  // close open files
+  fclose (blockDictFile);
+  fclose (invfIdxFile);
+  fclose (dictFile);
+  
+  // print out information
+#ifndef SILENT
+  Message ("Max word block size = %d\n", bdh.max_wblk_size);
+  Message ("Max tag block size = %d\n", bdh.max_tblk_size);
+  Message ("Number of word blocks written = %d\n", bdh.num_wblks);
+  Message ("Number of tag blocks written = %d\n", bdh.num_tblks);
+#endif
+}
+
+
+int main (int argc, char **argv) {
+  unsigned long entriesPerBlock = 16;
+  char *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  while ((ch = getopt (argc, argv, "f:d:b:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'b':
+      entriesPerBlock = atoi (optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-f input_file] "
+	       "[-d data directory] [-b entries-per-block] "
+	       "[-h]\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  process_files (filename, entriesPerBlock);
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/mgpp_invf_dict_dump.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_invf_dict_dump.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_invf_dict_dump.cpp	(revision 3365)
@@ -0,0 +1,141 @@
+/**************************************************************************
+ *
+ * mgpp_invf_dict_dump.cpp -- Program to printthe various inverted dictionaries
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "messages.h"
+#include "mg_files.h"
+#include "invf.h"
+
+static void process_files (char *filename,
+			   bool printHeader,
+			   bool printWords,
+			   bool printTags) {
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // print the information from the inverted dictionary file header
+  if (printHeader) {
+    cout << "lookback: " << idh.lookback << "\n";
+    cout << "word_dict_start: " << idh.word_dict_start << "\n";
+    cout << "word_dict_size: " << idh.word_dict_size << "\n";
+    cout << "tag_dict_start: " << idh.tag_dict_start << "\n";
+    cout << "tag_dict_size: " << idh.tag_dict_size << "\n";
+    cout << "num_docs: " << idh.num_docs << "\n";
+    cout << "num_frags: " << idh.num_frags << "\n";
+    cout << "num_words: " << idh.num_words << "\n";
+    cout << "total_bytes: " << idh.total_bytes << "\n";
+    cout << "index_string_bytes: " << idh.index_string_bytes << "\n";
+    cout << "num_levels: " << idh.num_levels << "\n";
+
+    cout << "\n";
+  }
+
+
+  if (printWords) {
+    fseek (dictFile, idh.word_dict_start, SEEK_SET);
+    
+    unsigned long wordNum;
+    word_dict_el wordEl;
+    wordEl.SetNumLevels (idh.num_levels);
+    for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+      wordEl.Read (dictFile, idh.num_levels);
+      cout << "\"" << wordEl.el << "\"\n";// (" << wordNum << ")\n";
+    }
+    
+    cout << "\n";
+  }
+
+  // write out the tag part of the dictionary
+  if (printTags) {
+    fseek (dictFile, idh.tag_dict_start, SEEK_SET);
+    
+    unsigned long tagNum;
+    dict_el tagEl;
+    for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+      // read in the next tag and inverted file pointer
+      tagEl.Read (dictFile);
+      
+      cout << "\"" << tagEl.el << "\" (" << tagNum << ")\n";
+    }
+  }
+  
+  // close open files
+  fclose (dictFile);
+}
+
+
+int main (int argc, char **argv) {
+  char *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  bool printHeader = false;
+  bool printWords = false;
+  bool printTags = false;
+
+  while ((ch = getopt (argc, argv, "f:d:rwth")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'r':
+      printHeader = true;
+      break;
+    case 'w':
+      printWords = true;
+      break;
+    case 't':
+      printTags = true;
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-r] [-w] [-t] [-f input_file] "
+	       "[-d data directory]\n(-rwt:print header, words, tags)\n", 
+	       argv[0]);
+      exit (1);
+    }
+  }
+
+  process_files (filename, printHeader, printWords, printTags);
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/mgpp_invf_dump.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_invf_dump.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_invf_dump.cpp	(revision 3365)
@@ -0,0 +1,244 @@
+/**************************************************************************
+ *
+ * mgpp_invf_dump.cpp -- Program to dump uot an inverted fil
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "messages.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "mg_files.h"
+#include "locallib.h"
+#include "words.h"
+#include "invf.h"
+#include "WordData.h"
+
+static void PrintInvfWord (FILE *invfFile,
+			   invf_dict_header &idh,
+			   invf_file_header &ifh,
+			   word_dict_el &wordEl,
+			   unsigned long wordStart,
+			   bool printFrags) {
+  cout << wordEl.frag_occur << " \"" << wordEl.el << "\"\n";
+  
+  if (printFrags) {
+    // seek to the appropriate place in the inverted file
+    fseek (invfFile, wordStart, SEEK_SET);
+    
+    stdio_bitio_buffer buffer(invfFile);
+    
+    unsigned long B = BIO_Bblock_Init (idh.num_frags, wordEl.frag_occur);
+    unsigned long fragNum = 0;
+    unsigned long i;
+    for (i=0; i<wordEl.frag_occur; i++) {
+      unsigned long delta = buffer.bblock_decode (B, NULL);
+      fragNum += delta;
+      cout << " " << fragNum;
+      
+      if (!ifh.word_level_index ) {
+	unsigned long count = buffer.gamma_decode (NULL);
+	cout << "(" << count << ")";
+      } else {
+	cout << "(1)";
+      }
+    }
+    
+    cout << "\n";
+  
+    buffer.done();
+  }
+}
+
+static void PrintInvfTag (FILE *invfFile,
+			  invf_dict_header &idh,
+			  invf_file_header &/*ifh*/,
+			  dict_el &tagEl,
+			  unsigned long tagStart,
+			  bool printFrags) {
+  cout << tagEl.frag_occur << " \"<" << tagEl.el << ">\"\n";
+  
+  if (printFrags) {
+    // seek to the appropriate place in the inverted file
+    fseek (invfFile, tagStart, SEEK_SET);
+    
+    stdio_bitio_buffer buffer(invfFile);
+    
+    unsigned long pTag = tagEl.frag_occur*2;
+    unsigned long B = BIO_Bblock_Init (idh.num_frags+pTag, pTag);
+    unsigned long fragNum = 0;
+    unsigned long i;
+    for (i=0; i<tagEl.frag_occur; i++) {
+      unsigned long delta = buffer.bblock_decode (B, NULL)-1;
+      fragNum += delta;
+      cout << " " << fragNum;
+      cout << "-";
+      delta = buffer.bblock_decode (B, NULL)-1;
+      fragNum += delta;
+      cout << fragNum;
+    }
+    
+    cout << "\n";
+    
+    buffer.done();
+  }
+}
+
+static void PrintHeaderInfo (invf_dict_header &idh,
+			     invf_file_header &ifh) {
+  cerr << "Lookback:         " << idh.lookback << "\n";
+  cerr << "Word Dict Size:   " << idh.word_dict_size << "\n";
+  cerr << "Tag Dict Size:    " << idh.tag_dict_size << "\n";
+  cerr << "Num Documents:    " << idh.num_docs << "\n";
+  cerr << "Num Fragments:    " << idh.num_frags << "\n";
+  cerr << "Num Words:        " << idh.num_words << "\n";
+
+  cerr << "Skip Mode:        " << ifh.skip_mode << "\n";
+  cerr << "Word Level Index: " << ifh.word_level_index << "\n";
+
+  cerr << "\n";
+}
+
+
+static void process_files (char *filename,
+			   bool printHeader,
+			   bool printWords,
+			   bool printTags,
+			   bool printFrags) {
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // open the inverted file
+  FILE *invfFile = open_file (filename, INVF_SUFFIX, "rb",
+			MAGIC_INVF, MG_ABORT);
+  
+  invf_file_header ifh;
+  ifh.Read (invfFile);
+
+  if (ifh.skip_mode != SKIP_MODE_NO_SKIPS)
+    FatalError (1, "The invf file contains skips. Unable to dump.");
+
+  // print out header information
+  if (printHeader) {
+    PrintHeaderInfo (idh, ifh);
+  }
+
+  // open the inverted index
+  FILE *invfIdxFile = open_file (filename, INVF_IDX_SUFFIX, "rb",
+				 MAGIC_INVI, MG_ABORT);
+
+  // go to the start of the word dictionary
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+
+  // process all the words
+  if (printWords) {
+    unsigned long wordNum;
+    unsigned long wordStart;
+    word_dict_el wordEl;
+    wordEl.SetNumLevels (idh.num_levels);
+    for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+      wordEl.Read (dictFile, idh.num_levels);
+      ReadUL (invfIdxFile, wordStart);
+      PrintInvfWord (invfFile, idh, ifh, wordEl, wordStart, printFrags);
+    }
+  }
+
+  // process all the tags
+  if (printTags) {
+    unsigned long tagNum;
+    unsigned long tagStart;
+    dict_el tagEl;
+    for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+      tagEl.Read (dictFile);
+      ReadUL (invfIdxFile, tagStart);
+      PrintInvfTag (invfFile, idh, ifh, tagEl, tagStart, printFrags);
+    }
+  }
+  // close the open files
+  fclose (invfIdxFile);
+  fclose (invfFile);
+  fclose (dictFile);
+}
+
+
+int main (int argc, char **argv) {
+  char *dir_name, *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  dir_name = getenv ("MGDATA");
+  opterr = 0;
+
+  bool printHeader = false;
+  bool printWords = false;
+  bool printTags = false;
+  bool printFrags = false;
+  
+  msg_prefix = argv[0];
+  while ((ch = getopt (argc, argv, "hrwtnf:d:")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath(optarg);
+      break;
+    case 'r':
+      printHeader = true;
+      break;
+    case 'w':
+      printWords = true;
+      break;
+    case 'n':
+      printFrags = true;
+      break;
+    case 't':
+      printTags = true;
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-r] [-w] [-t] [-n] [-f input_file]"
+	       "[-d data directory]\n(-rwnt:print header, words, tags, fragnums)\n", 
+	       argv[0]);
+      exit (1);
+    }
+  }
+  
+  process_files (filename, printHeader, printWords, printTags, printFrags);
+
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/mgpp_passes.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_passes.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_passes.1	(revision 3365)
@@ -0,0 +1,323 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_passes 1 \*(Dt CITRI
+.SH NAME
+mgpp_passes \- builds mgpp databases
+.SH SYNOPSIS
+.B mgpp_passes
+[
+.BI \-J " doc-tag"
+]
+[
+.BI \-K " level-tag"
+]
+.if n .ti +10n
+[
+.BI \-L " index-level"
+]
+[
+.BI \-m " invf-mem-buffer"
+]
+.if n .ti +10n
+[
+.B \-T1
+]
+[
+.B \-T2
+]
+[
+.B \-I1
+]
+[
+.B \-I2
+]
+[
+.B \-S
+]
+[
+.B \-C
+]
+.if n .ti +10n
+[
+.B \-h
+]
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+[
+.I filename(s)
+]
+.SH DESCRIPTION
+.B mgpp_passes
+is the program that does most of the work when building mgpp
+database systems.  The input documents can come from either
+.I stdin
+or from a list of files on the command line.  In general,
+.B mgpp_passes
+must be run twice to build a database, first with the
+.B \-T1
+and
+.B \-I1
+options, and second with the
+.B \-T2
+and
+.B \-I2
+options.  Several other programs must be run in order to get an
+mgpp database.  The
+.SB EXAMPLE
+section below gives an example of how to build a complete
+mgpp database.
+.SH OPTIONS
+Options may appear in any order, but the
+.IR filename(s) ,
+if specified, must be last.
+.TP "\w'\fB\-C\fP \fIcompstatpointt\fP'u+2n"
+.BI \-J " doc-tag"
+Specifies the SGML tag that encloses each document.  Text appearing
+outside this tag is ignored.  The document tag defines the highest
+level document that can be queried and printed.  The default document
+tag is 'Document'.
+.TP
+.BI \-K " level-tag"
+Specifies the SGML tag of a sub document level.  A level tag must
+enclose all text enclosed by the document tag.  Levels can be
+queried and printed as if they were separate documents.  Multiple
+document levels can be specified (the document tag is always
+added as a document level).
+.TP
+.BI \-L " index-level"
+Specifies the SGML tag enclosing the smallest indexed element.  The
+index level should be no larger than the smallest document 
+level.  An empty string can be used to specify a word level index 
+(which is the default).
+.TP
+.BI \-m " invf-mem-buffer"
+Maximum amount of memory to use for the pass-2 file inversion in
+megabytes.  This option is only useful when used in conjunction with
+the option
+.BR \-I1 .
+The larger this value, the faster the pass-2 inversion will proceed.
+The default value is 5 MB.
+.TP
+.B \-T1
+Generate the
+.I *.text.stats
+file.
+.TP
+.B \-T2
+Generate the
+.IR *.text ,
+.IR *.text.idx ,
+.IR *.text.level ,
+and possibly the
+.I *.text.dict.aux
+files.  Using this option requires that the
+.I *.text.dict
+file be present.
+.TP
+.B \-I1
+Generate the
+.IR *.invf.dict ,
+.IR *.invf.level ,
+.IR *.invf.chunk ,
+and
+.I *.invf.chunk.trans
+files.
+.TP
+.B \-I2
+Generate the
+.I *.invf
+and
+.I *.invf.idx
+files.  Using this option requires
+that the
+.IR *.invf.dict.hash ,
+.IR *.invf.level ,
+.IR *.invf.chunk ,
+and
+.I *.invf.chunk.trans
+files be present.  The
+.I *.invf.dict.hash
+file is generated by
+.BR mgpp_perf_hash_build (1)
+from the
+.I *.invf.dict
+file.
+.TP
+.B \-S
+This option causes a special pass to be executed.  It is up to a user
+to modify
+.I mg.special.c
+in the source code to do something with the documents it is given.
+.TP 
+.B \-C
+This activates the compatibility parsing mode.  When using this
+mode documents are separated by control-B and paragraphs are separated
+by control-C.  Internally these are converted to documents surrounded
+by 'Document' tags and paragraphs surrounded by 'Paragraph' tags.
+.TP
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection is to be
+written.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection that will be
+created.
+.TP
+.I filename(s)
+This specifies the source text. If this is not specified, then the
+program expects the source text from
+.IR stdin .
+.SH EXAMPLE
+What follows is a UNIX
+.BR csh (1)
+script as an example of how to build an mgpp document collection.
+.LP
+.nf
+.DT
+.ft B
+.I #! /bin/csh
+.I
+# The first argument on the command line specifies the
+.I
+#   source of the text
+set source = ($1)
+.PP
+.I
+# The second argument is the name of the collection
+set text = ($2)
+.PP
+.I
+# Create *.text.stats, *.invf.dict, *.invf.level
+.I
+#   *.invf.chunk and *.invf.chunks.trans
+${source} | mgpp_passes -T1 -I1 -f ${text}
+.PP
+.I
+# Create *.text.dict
+mgpp_compression_dict -f ${text}
+.PP
+.I
+# Create *.invf.dict.hash
+mgpp_perf_hash_build -f ${text}
+.PP
+.I
+# Create *.text, *.text.idx, *.text.level
+.I
+#   *.invf and *.invf.idx
+${source} | mgpp_passes -T2 -I2 -f ${text}
+.PP
+.I
+# Create *.text.weight and *.weight.approx
+mgpp_weights_build -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked
+mgpp_invf_dict -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked.1
+mgpp_stem_idx -s 1 -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked.2
+mgpp_stem_idx -s 2 -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked.3
+mgpp_stem_idx -s 3 -f ${text}
+.PP
+.I
+# Create *.text.dict.fast
+mgpp_fast_comp_dict -f ${text}
+.ft R
+.fi
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 22
+.B *.invf
+Inverted file.
+.TP
+.B *.invf.chunk
+Inverted file chunk descriptor file.  When the inverted file is
+created it is created in chunks that use no more than a set amount of
+memory.  This file describes those chunks.
+.TP
+.B *.invf.chunk.trans
+Word-occurrence-order to lexical-order translation file.  The
+.B *.invf.chunk
+file is written in word-occurrence order but is required by
+.B \-I2
+to be in lexical order.
+.TP
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.invf.dict.blocked
+Compressed stemmed dictionary with index into the dictionary.
+.TP
+.B *.invf.dict.blocked.n
+Transformation dictionary from words stemmed with method 
+.B n
+to unstemmed words.
+.TP
+.B *.invf.dict.hash
+Data for an order-preserving perfect hash function.
+.TP
+.B *.invf.idx
+The index into the inverted file.
+.TP
+.B *.invf.level
+Information about the document levels needed for querying.
+.TP
+.B *.text
+Compressed text.
+.TP
+.B *.text.dict
+Compressed compression dictionary.
+.TP
+.B *.text.dict.fast
+A fast loading version of the compressed compression dictionary.
+.TP
+.B *.text.idx
+Index into the compressed documents.
+.TP
+.B *.text.level
+Information about the document levels needed for text decompression.
+.TP
+.B *.text.stats
+Statistics about the text.
+.TP
+.B *.weight
+The exact weights file.
+.TP
+.B *.weight.approx
+The approximate weights file.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/indexers/mgpp/text/mgpp_passes.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_passes.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_passes.cpp	(revision 3365)
@@ -0,0 +1,264 @@
+/**************************************************************************
+ *
+ * mgpp_passes.cpp -- Driver for the various passes
+ * Copyright (C) 1994  Neil Sharman
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "sysfuncs.h"
+
+#ifdef HAVE_MALLINFO
+# include <malloc.h> 
+#endif
+
+#if defined __WIN32__
+# include <io.h>
+# include "getopt_old.h"
+# define close _close
+# define open _open
+#elif defined __CYGWIN__
+#include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "memlib.h"
+#include "messages.h"
+#include "longlong.h"
+#include "mg_files.h"
+#include "mg.h"
+#include "build.h"
+#include "text.h"
+#include "stemmer.h"
+#include "FileBuf.h"
+#include "TextEl.h"
+#include "TagInfo.h"
+
+#define MAX_PASSES 5
+
+#define SPECIAL 1
+#define TEXT_PASS_1 2
+#define TEXT_PASS_2 4
+#define IVF_PASS_1 8
+#define IVF_PASS_2 16
+
+#define MIN_BUF 8192
+
+
+unsigned long invf_buffer_size = 5 * 1024 * 1024;	/* 5Mb */
+
+static char Passes = 0;
+static char **files = NULL;
+static int num_files = 0;
+
+
+struct pass_data {
+  char *name;
+  int (*init) (const TagInfo &tagInfo, char *);
+  int (*process) (const TagInfo &tagInfo, const TextElArray &doc);
+  int (*done) (const TagInfo &tagInfo, char *);
+};
+
+
+static pass_data PassData[MAX_PASSES] = {
+  {"special", init_special, process_special, done_special},
+  {"text.pass1", init_text_1, process_text_1, done_text_1},
+  {"text.pass2", init_text_2, process_text_2, done_text_2},
+  {"ivf.pass1", init_ivf_1, process_ivf_1, done_ivf_1},
+  {"ivf.pass2", init_ivf_2, process_ivf_2, done_ivf_2},
+};
+
+static char *usage_str = "\nUSAGE:\n"
+"  %s [-J doc-tag] [-K level-tag] [-L index-level]\n"
+"  %*s [-m invf-memory] [-T1] [-T2] [-I1] [-I2] [-S]\n"
+"  %*s [-C] [-h] [-d directory] -f name\n\n";
+
+
+
+static void Usage (char *err) {
+  if (err) Message (err);
+  fprintf (stderr, usage_str, msg_prefix, strlen (msg_prefix), "",
+	   strlen (msg_prefix), "");
+  exit (1);
+}
+
+
+
+
+int OpenNextFile (int in_fd) {
+  if (in_fd > 0) close (in_fd);
+  if (num_files == 0) return (-1);
+  if ((in_fd = open (files[0], O_RDONLY)) == -1)
+    FatalError (1, "Cannot open %s", files[0]);
+  files++;
+  num_files--;
+  return (in_fd);
+}
+
+
+static void Driver (int in_fd, char *file_name,
+		    const TagInfo &tagInfo, bool compatMode) {
+  //  cout << tagInfo;
+
+  int pass;
+
+  unsigned long numBytes = 0;
+  unsigned long numDocs = 0;
+
+  // initialise all the passes
+  for (pass = 0; pass < MAX_PASSES; pass++) {
+    if (Passes & (1 << pass)) {
+      if (PassData[pass].init (tagInfo, file_name) == COMPERROR)
+	FatalError (1, "Error during init of \"%s\"", PassData[pass].name);
+    }
+  }
+
+  
+  // set up various variables
+  FileBuf buf;
+  TextElArray doc;
+  unsigned long docLen = 0;
+
+  // read and process each file (start with an open file)
+  do {
+    
+    // read and process each document in this file
+    buf.SetFD (in_fd);
+    while (ReadDoc (buf, tagInfo.docTag, doc, docLen, compatMode)) {
+
+      // give this document to each pass
+      for (pass = 0; pass < MAX_PASSES; pass++) {
+	if (Passes & (1 << pass)) {
+	  if (PassData[pass].process (tagInfo, doc) == COMPERROR)
+	    FatalError(1,"Error during processing of \"%s\"",PassData[pass].name);
+	}
+      }
+      
+      // another document has been processed
+      numBytes += docLen;
+      numDocs++;
+    }
+    
+  } while ((in_fd = OpenNextFile (in_fd)) > 0);
+
+  
+  // do done for each pass
+  for (pass = 0; pass < MAX_PASSES; pass++) {
+    if (Passes & (1 << pass)) {
+      if (PassData[pass].done (tagInfo, file_name) == COMPERROR)
+	FatalError (1, "Error during done of \"%s\"", PassData[pass].name);
+    }
+  }
+}
+
+int main (int argc, char **argv) {
+  int ch, in_fd;
+  char *filename = NULL;
+  bool compatMode = false;
+  TagInfo tagInfo;
+  tagInfo.SetDocTag ("Document");
+  
+  msg_prefix = argv[0];
+
+  opterr = 0;
+  while ((ch=getopt(argc, argv, "J:K:L:f:d:m:I:T:SCh"))!=-1){
+    switch (ch) {
+    case 'J':
+      tagInfo.SetDocTag (optarg);
+      break;
+    case 'K':
+      tagInfo.AddLevelTag (optarg);
+      break;
+    case 'L':
+      tagInfo.SetIndexLevel (optarg);
+      break;
+    case 'f':
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'm':
+      invf_buffer_size = (int) (atof (optarg) * 1024 * 1024);
+      break;
+    case 'I':
+      if (*optarg == '1')
+	Passes |= IVF_PASS_1;
+      else if (*optarg == '2')
+	Passes |= IVF_PASS_2;
+      else
+	Usage ("Invalid pass number");
+      break;
+    case 'T':
+      if (*optarg == '1')
+	Passes |= TEXT_PASS_1;
+      else if (*optarg == '2')
+	Passes |= TEXT_PASS_2;
+      else
+	Usage ("Invalid pass number");
+      break;
+    case 'S':
+      Passes |= SPECIAL;
+      break;
+    case 'C':
+      compatMode = true;
+      break;
+    case 'h':
+    case '?':
+      Usage (NULL);
+    }
+  }
+
+  if (!filename || *filename == '\0')
+    FatalError (1, "A document collection name must be specified.");
+
+  if ((Passes & (IVF_PASS_1 | IVF_PASS_2)) == (IVF_PASS_1 | IVF_PASS_2))
+    FatalError (1, "I1 and I2 cannot be done simultaneously.");
+
+  if ((Passes & (TEXT_PASS_1 | TEXT_PASS_2)) == (TEXT_PASS_1 | TEXT_PASS_2))
+    FatalError (1, "T1 and T2 cannot be done simultaneously.");
+
+  if (!Passes)
+    FatalError (1, "S, T1, T2, I1 or I2 must be specified.");
+
+  if (optind < argc) {
+    if ((in_fd = open (argv[optind], O_RDONLY)) == -1)
+      FatalError (1, "Cannot open %s", argv[optind]);
+    files = &argv[optind + 1];
+    num_files = argc - (optind + 1);
+    
+  } else in_fd = 0;  // stdin
+
+  
+  if (compatMode) tagInfo.SetDocTag ("Document");
+
+  // a document tag is also a level tag
+  tagInfo.levelTags.insert (tagInfo.docTag); 
+  
+  Driver (in_fd, filename, tagInfo, compatMode);
+
+  return (0);
+}
Index: /trunk/indexers/mgpp/text/mgpp_perf_hash_build.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_perf_hash_build.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_perf_hash_build.1	(revision 3365)
@@ -0,0 +1,72 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_perf_hash_build 1 \*(Dt CITRI
+.SH NAME
+mgpp_perf_hash_build \- generate an order-preserving hash function for the stemmed dictionary
+.SH SYNOPSIS
+.B mgpp_perf_hash_build
+[
+.B \-h
+]
+[
+.BI \-r " rndseed"
+]
+[
+.BI \-d  " directory"
+]
+.if n .ti +9n
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_perf_hash_build
+generates an order-preserving hashing function from the compressed
+stemmed dictionary and writes it out to disk.
+.BR mgpp_passes (1)
+will make use of the hash function when it builds the inverted file.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-r " rndseed"
+This specifies the random seed to be used in the generation of the
+hash function; it must be an integer.  If this is not specified, the
+current time will be used as the random seed.
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mg
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.invf.dict
+Stemmed dictionary.
+.TP
+.B *.invf.dict.hash
+Data for a order-preserving perfect hash function.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/indexers/mgpp/text/mgpp_perf_hash_build.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_perf_hash_build.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_perf_hash_build.cpp	(revision 3365)
@@ -0,0 +1,146 @@
+/**************************************************************************
+ *
+ * mgpp_perf_hash_build.cpp -- Program to build a perfect hash function
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "perf_hash.h"
+#include "netorder.h"
+
+#include "mg_files.h"
+#include "invf.h"
+#include "locallib.h"
+#include "words.h"
+#include "mg.h"
+
+#define POOL_SIZE 1024*1024
+
+static void ProcessFiles (char *filename, int r) {
+  FILE *dictFile, *hashFile;
+  unsigned long i;
+  invf_dict_header idh;
+  perf_hash_data *phd;
+  u_char *pool;
+  unsigned long pool_left;
+  u_char **starts;
+
+  // read in the dictionary
+  dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			MAGIC_STEM_BUILD, MG_ABORT);
+  if (dictFile==NULL) {
+    FatalError(1, "unable to open file");
+  }
+  idh.Read (dictFile);
+
+  //cerr << idh.lookback<<" "<<idh.word_dict_start<<endl;
+  // go to the start of the word dictionary
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+  
+  if (!(pool = (u_char *) Xmalloc (POOL_SIZE)))
+    FatalError (1, "Out of memory");
+  pool_left = POOL_SIZE;
+
+  if (!(starts = (u_char **) Xmalloc (sizeof (u_char *) * idh.word_dict_size)))
+    FatalError (1, "Out of memory");
+  //cerr << "size= "<< idh.word_dict_size<<endl;
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (i = 0; i < idh.word_dict_size; i++) {
+    // read the next word and associated information
+    wordEl.Read (dictFile, idh.num_levels);
+
+    // push string onto pool data
+    register unsigned long l;
+    l = wordEl.el.size() + 1;
+    if (pool_left < l) {
+      pool = (u_char *) Xmalloc (POOL_SIZE);
+      pool_left = POOL_SIZE;
+    }
+    starts[i] = pool;
+    
+    *pool++ = wordEl.el.size();
+    memcpy ((char *) pool, &(wordEl.el[0]), wordEl.el.size());
+    //cerr << pool<<"   " <<starts[i]<<endl;
+    pool += wordEl.el.size();
+    pool_left -= l;
+    
+  }
+  fclose (dictFile);
+  //cerr << pool<<"   " <<starts[i-1]<<endl;
+  //cerr<<"starts  "<<starts[113529]<<endl;
+  //cerr << starts[17][1] << " "<<starts[25][4]<<endl;
+  // create perfect hash file
+  hashFile = create_file (filename, INVF_DICT_HASH_SUFFIX, "wb",
+			  MAGIC_HASH, MG_ABORT);
+  if (!(phd = gen_hash_func (idh.word_dict_size, starts, r)))
+    FatalError (1, "Unable to generate hash function");
+  if (write_perf_hash_data (hashFile, phd) == -1)
+    FatalError (1, "Unable to write hash function");
+  fclose (hashFile);
+}
+
+
+
+int main (int argc, char **argv) {
+  int r = -1;
+  char *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  while ((ch = getopt (argc, argv, "f:d:r:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'r':
+      r = atoi (optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-f input_file]"
+	       "[-d data directory] [-r random seed] [-h]\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  ProcessFiles (filename, r);
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/mgpp_stem_idx.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_stem_idx.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_stem_idx.1	(revision 3365)
@@ -0,0 +1,95 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_stem_idx 1 \*(Dt CITRI
+.SH NAME
+mgpp_stem_idx \- builds a stem index file
+.SH SYNOPSIS
+.B mgpp_stem_idx
+[
+.B \-h
+]
+[
+.BI \-b " entries-per-block"
+]
+.if n .ti +12n
+[
+.BI \-a " stemmer"
+]
+[
+.BI \-d " directory"
+]
+.if n .ti +12n
+.B \-s 1|2|3
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_stem_idx
+generates a stem index file for a collection.
+This program should be called three times: once for each
+.B -s
+parameter.  It uses the stemmed dictionary to create the stem index which
+contains pointers into the stemmed dictionary.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectoryyyyyyyy\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-b " entries-per-block"
+The dictionary is stored in blocks on disk; this option is used to set
+the number of entries per block. The default is 16.
+.TP
+.BI \-a " stemmer"
+The name of the stemmer to use, the default is the Lovin stemmer.
+.TP
+.B -s 1|2|3
+The stem method to apply for the stem index.
+.br
+1 = casefolded and non-stemmed
+.br
+2 = non-casefolded and stemmed
+.br
+3 = casefolded and stemmed
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 22
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.invf.dict.blocked.1
+Stem index with stem index method 1.
+.TP
+.B *.invf.dict.blocked.2
+Stem index with stem index method 2.
+.TP
+.B *.invf.dict.blocked.3
+Stem index with stem index method 3.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_weights_build (1)
Index: /trunk/indexers/mgpp/text/mgpp_stem_idx.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_stem_idx.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_stem_idx.cpp	(revision 3365)
@@ -0,0 +1,237 @@
+/**************************************************************************
+ *
+ * mgpp_stem_idx.cpp -- stem index builder
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "messages.h"
+#include "mg_files.h"
+#include "invf.h"
+#include "words.h"
+#include "stemmer.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#  include <ospace\std\vector>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#  include <vector.h>
+#else
+#  include <map>
+#  include <vector>
+#endif
+
+typedef vector<unsigned long> WordNumList;
+typedef map<UCArray, WordNumList, DictLTUCArray> StemMapDict;
+
+void CreateStemDict (char *filename,
+		     StemMapDict &stemDict,
+		     int stemMethod,
+		     int stemmerNum) {
+  stemDict.erase (stemDict.begin(), stemDict.end());
+
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+
+  unsigned long wordNum;
+  u_char mgWord[MAXSTEMLEN + 1];
+  word_dict_el wordEl;
+  UCArray stemEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+    // read in the next word
+    wordEl.Read (dictFile, idh.num_levels);
+
+    // convert the word to an "mg word"
+    mgWord[0] = wordEl.el.size();
+    memcpy((char *)&mgWord[1], &(wordEl.el[0]), wordEl.el.size());
+
+    // stem the word
+    stemmer (stemMethod, stemmerNum, mgWord);
+
+    // convert the result back to a UCArray
+    stemEl.erase (stemEl.begin(), stemEl.end());
+    stemEl.insert (stemEl.end(), &mgWord[1], &mgWord[1] + mgWord[0]);
+
+//      cout << "\"" << stemEl << "\" -> \"" << wordEl.el << "\"\n";
+    
+    // add this word number to the list of word numbers for this word
+    stemDict[stemEl].push_back (wordNum);
+  }
+
+  fclose (dictFile);
+}
+
+
+void WriteStemDict (char *filename,
+		    StemMapDict &stemDict,
+		    int stemMethod,
+		    int stemmerNum,
+		    unsigned long entriesPerBlock) {
+  // Create appropriate stem index file
+  FILE *stemDictFile = NULL;
+  if (stemMethod == 1) {
+    stemDictFile = create_file (filename, INVF_DICT_BLOCKED_1_SUFFIX,
+				"wb", MAGIC_STEM_1, MG_ABORT);
+  } else if (stemMethod == 2) {
+    stemDictFile = create_file (filename, INVF_DICT_BLOCKED_2_SUFFIX,
+				"wb", MAGIC_STEM_2, MG_ABORT);
+  } else if (stemMethod == 3) {
+    stemDictFile = create_file (filename, INVF_DICT_BLOCKED_3_SUFFIX,
+				"wb", MAGIC_STEM_3, MG_ABORT);
+  } else {
+    FatalError (1, "Unknown stem method %d", stemMethod);
+  }
+
+  stem_idx_header sih;
+  sih.lookback = 0;
+  sih.dict_size = stemDict.size();
+  sih.entries_per_block = entriesPerBlock;
+  sih.max_block_size = 0;
+  
+  sih.stemmer_num = stemmerNum;
+  sih.stem_method = stemMethod;
+
+  // write out a place-holder version of the header
+  sih.Write (stemDictFile);
+
+  sih.blocks_start = ftell (stemDictFile);
+
+  block_idx stemIdx;
+  unsigned long stemNum = 0;
+  stem_block_dict_el stemEl;
+  UCArray lastEl;
+
+  StemMapDict::const_iterator here = stemDict.begin();
+  StemMapDict::const_iterator end = stemDict.end();
+  while (here != end) {
+    // remember this stem (and position) if this is the start
+    // of a new block
+    if (stemNum % entriesPerBlock == 0) {
+      block_idx_info elIdx;
+      elIdx.el = (*here).first;
+      elIdx.block_ptr = ftell (stemDictFile);
+
+      // see if this block is the longest so far
+      if (stemIdx.size() > 0) {
+	unsigned long blockLen = elIdx.block_ptr -
+	  (*(stemIdx.end()-1)).block_ptr;
+	if (blockLen > sih.max_block_size) sih.max_block_size = blockLen;
+      }
+      
+      stemIdx.push_back (elIdx);
+      lastEl.erase (lastEl.begin(), lastEl.end()); // output full word
+    }
+
+    // copy the information for this stem
+    stemEl.el = (*here).first;
+    stemEl.equivWords = (*here).second;
+
+    // write out the stem
+    stemEl.Write (stemDictFile, &lastEl);
+    
+    here++; stemNum++;
+  }
+
+
+  // write out the element indexes
+  sih.num_blocks = stemIdx.size();
+  sih.block_idx_start = ftell (stemDictFile);
+  WriteBlockIdx (stemDictFile, stemIdx);
+  
+  // write out the stem dictionary header
+  fseek (stemDictFile, sizeof(unsigned long), SEEK_SET);
+  sih.Write (stemDictFile);
+
+
+  // close open files
+  fclose (stemDictFile);
+  
+  // print out information
+#ifndef SILENT
+  Message ("Num word stems = %d\n", sih.dict_size);
+  Message ("Max stem block size = %d\n", sih.max_block_size);
+  Message ("Number of stem blocks written = %d\n", sih.num_blocks);
+#endif
+}
+
+
+int main (int argc, char **argv) {
+  unsigned long entriesPerBlock = 16;
+  char *filename = "";
+  int ch;
+  int stemMethod = 0; // illegal value (no translation)
+  int stemmerNum = 0; // English stemmer
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  while ((ch = getopt (argc, argv, "f:d:b:s:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'b':
+      entriesPerBlock = atoi (optarg);
+      break;
+    case 's':
+      stemMethod = atoi (optarg);
+      break;
+    case 'a':
+      stemmerNum = stemmernumber ((unsigned char *) optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-d directory] "
+	       "[-b entries-per-block] [-h] -s 1|2|3 "
+	       "[-a stemmer-method] -f name\n", argv[0]);
+      exit (1);
+    }
+  }
+  
+  if (stemMethod < 1 || stemMethod > 3)
+    FatalError (1, "Stem method must be 1, 2 or 3");
+
+  // read in the dictionary and create the in memory dictionary
+  StemMapDict stemDict;
+  CreateStemDict (filename, stemDict, stemMethod, stemmerNum);
+  
+  // write out the dictionary as a blocked file
+  WriteStemDict (filename, stemDict, stemMethod, stemmerNum, entriesPerBlock);
+  
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/mgpp_weights_build.1
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_weights_build.1	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_weights_build.1	(revision 3365)
@@ -0,0 +1,79 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_weights_build 1 \*(Dt CITRI
+.SH NAME
+mgpp_weights_build \- builds the approximate weights file
+.SH SYNOPSIS
+.B mgpp_weights_build
+[
+.B \-h
+]
+[
+.BI \-b " num-bits"
+]
+.if n .ti +17n
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_weights_build
+is the program that generates the exact and approximate weights 
+files.  To do this, it uses the inverted file
+and dictionary.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-b " num-bits"
+This specifies the number of bits of precision that should be used for
+the approximate weights.  If not specified, it defaults to 8.
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection is.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.invf
+Inverted file.
+.TP
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.text.idx
+Index into the compressed documents.
+.TP
+.B *.weight
+The exact weights file.
+.TP
+.B *.weight.approx
+Approximate document weights.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1)
Index: /trunk/indexers/mgpp/text/mgpp_weights_build.cpp
===================================================================
--- /trunk/indexers/mgpp/text/mgpp_weights_build.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/mgpp_weights_build.cpp	(revision 3365)
@@ -0,0 +1,399 @@
+/**************************************************************************
+ *
+ * mgpp_weights_build.cpp -- Program to build the document weights file
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "bitio_gen.h"
+#include "bitio_m_stdio.h"
+#include "mg_files.h"
+#include "locallib.h"
+#include "invf.h"
+#include "FIvfLevelInfo.h"
+#include "FragLevelConvert.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+#define MAXBITS (sizeof(unsigned long) * 8)
+
+struct WBTagPtr {
+  unsigned long tagNum;
+  unsigned long tagPtr;
+  unsigned long fragOccur;
+  
+  WBTagPtr () {
+    tagNum = 0;
+    tagPtr = 0;
+    fragOccur = 0;
+  }
+};
+
+// maps tags to tag information
+typedef map<UCArray, WBTagPtr, DictLTUCArray> WBTagDict;
+
+typedef vector<float> Weights;
+
+
+static void ReadTagDict (const invf_dict_header &idh,
+			 FILE *dictFile,
+			 FILE *invfIdxFile,
+			 WBTagDict &tagDict) {
+  tagDict.erase (tagDict.begin(), tagDict.end());
+
+  // seek to the start of the tag information
+  fseek (dictFile, idh.tag_dict_start, SEEK_SET);
+  fseek (invfIdxFile, sizeof(unsigned long) +
+	 idh.word_dict_size*sizeof(unsigned long), SEEK_SET);
+  
+  unsigned long tagNum;
+  unsigned long tagPtr;
+  dict_el thisEl;
+  for (tagNum = 0; tagNum < idh.tag_dict_size; tagNum++) {
+    thisEl.Read (dictFile);
+    ReadUL (invfIdxFile, tagPtr);
+    tagDict[thisEl.el].tagNum = tagNum;
+    tagDict[thisEl.el].tagPtr = tagPtr;
+    tagDict[thisEl.el].fragOccur = thisEl.frag_occur;
+  }
+}
+
+static void AddWeight (Weights &w,
+		       unsigned long levelDocNum,
+		       unsigned long termFreq,
+		       float idf) {
+  double weight = termFreq * idf;
+  w[levelDocNum-1] += weight * weight;
+}
+
+static void GenerateLevelWeights (const invf_dict_header &idh,
+				  const invf_file_header &ifh,
+				  unsigned long numLevelDocs,
+				  unsigned long levelNum,
+				  FILE *dictFile,
+				  FILE *invfFile,
+				  FILE *invfIdxFile,
+				  const FragLevelConvert &fragLevelConvert,
+				  Weights &w) {
+  // pre-allocate the right number of weights
+  w.erase (w.begin(), w.end());
+  w.insert (w.end(), (Weights::size_type)numLevelDocs, (float)0.0);
+
+  double logN = log ((double) numLevelDocs);
+
+  // reset the files
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+  fseek (invfIdxFile, sizeof (unsigned long), SEEK_SET);
+  
+  // process each word adding its contributions to the document weights
+  unsigned long wordNum;
+  unsigned long wordStart;
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+#ifndef SILENT
+    // give a little feedback every 4096 words
+    if ((wordNum & 0xfff) == 0) fprintf (stderr, ".");
+#endif
+    
+    wordEl.Read (dictFile, idh.num_levels);
+    ReadUL (invfIdxFile, wordStart);
+
+    float idf = logN - log ((double) wordEl.levelFreqs[levelNum]);
+
+    // seek to the appropriate place in the inverted file
+    fseek (invfFile, wordStart, SEEK_SET);
+    stdio_bitio_buffer buffer (invfFile);
+    
+    unsigned long B = BIO_Bblock_Init (idh.num_frags, wordEl.frag_occur);
+    unsigned long fragNum = 0;
+    unsigned long levelDocNum = 0;
+    unsigned long lastLevelDocNum = 0;
+    unsigned long termFreq = 0;
+    unsigned long checkLevelFreq = 0;
+    
+    unsigned long count, i;
+    for (i=0; i<wordEl.frag_occur; i++) {
+      fragNum += buffer.bblock_decode (B, NULL);
+      if (!ifh.word_level_index) count = buffer.gamma_decode (NULL);
+      else count = 1;
+
+      if (fragNum > idh.num_frags)
+	FatalError (1, "fragNum = %d, "
+		    "number of fragments = %d\n"
+		    "wordNum = %d\n"
+		    "i = %d, frag_occur = %d\n",
+		    fragNum, idh.num_frags, wordNum, i, wordEl.frag_occur);
+
+      if (!fragLevelConvert.FragToLevel (fragNum, levelDocNum))
+	FatalError (1, "could not convert fragment number %d in level %d", fragNum, levelNum);
+
+      if (levelDocNum == 0 || levelDocNum > numLevelDocs)
+	FatalError (1, "bad level document number %d in level %d", levelDocNum, levelNum);
+      
+      if (levelDocNum != lastLevelDocNum) {
+	// new level document
+	if (lastLevelDocNum > 0) {
+	  AddWeight (w, lastLevelDocNum, termFreq, idf);
+	  checkLevelFreq++;
+	}
+	lastLevelDocNum = levelDocNum;
+	termFreq = 0;
+      }
+      termFreq += count;
+    }
+
+    if (lastLevelDocNum > 0) {
+      AddWeight (w, lastLevelDocNum, termFreq, idf);
+      checkLevelFreq++;
+    }
+
+    if (checkLevelFreq != wordEl.levelFreqs[levelNum]) {
+      cerr << "bad level freq at level " <<levelNum<<" "<< checkLevelFreq << " != "
+	   << wordEl.levelFreqs[levelNum] << ", word \""
+	   << wordEl.el << "\" (" << wordNum << ")\n";
+      exit (1);
+    }
+    
+    buffer.done();
+  }
+
+  if (w.size() != numLevelDocs) 
+    FatalError (1, "wrong number of weights created %d != %d",
+		w.size(), numLevelDocs);
+}
+
+static void WriteExactWeights (FILE *weightsFile,
+			       unsigned long &diskPtr,
+			       const Weights &w) {
+  diskPtr = ftell(weightsFile);
+
+  Weights::const_iterator here = w.begin();
+  Weights::const_iterator end = w.end();
+  while (here != end) {
+//      cout << *here << "\n";
+    WriteF (weightsFile, sqrt (*here));
+    here++;
+  }
+}
+
+static void WriteApproxWeights (FILE *approxWeightsFile,
+				unsigned long &diskPtr,
+				const Weights &w,
+				unsigned char bits) {
+  diskPtr = ftell(approxWeightsFile);
+
+  // calculate L, U and B
+  double L = 1e300;
+  double U = 0;
+  float wgt;
+  Weights::const_iterator here = w.begin();
+  Weights::const_iterator end = w.end();
+  while (here != end) {
+    wgt = sqrt (*here);
+    if (wgt > U) U = wgt;
+    if (wgt > 0 && wgt < L) L = wgt;
+    here++;
+  }
+
+  double B = pow (U / L, pow (2.0, -(double) bits));
+
+#ifndef SILENT
+  fprintf (stderr, "L = %f\n", L);
+  fprintf (stderr, "U = %f\n", U);
+  fprintf (stderr, "B = %f\n", B);
+#endif
+  
+  WriteUC (approxWeightsFile, bits);
+  WriteD (approxWeightsFile, L);
+  WriteD (approxWeightsFile, B);
+  
+
+  unsigned long max = (bits == 32) ? 0xffffffff : (1 << bits) - 1;;
+  unsigned long i=0, buf=0, pos=0;
+  here = w.begin();
+  end = w.end();
+  while (here != end) {
+    unsigned long fx;
+    wgt = sqrt (*here);
+    if (wgt == 0) {
+      wgt = L;
+#ifndef SILENT
+      Message ("Warning: Document %d had a weight of 0.", i);
+#endif
+    }
+    fx = (unsigned long) floor (log (wgt / L) / log (B));
+
+    if (fx > max) fx = max;
+
+    buf |= (fx << pos);
+    pos += bits;
+
+    if (pos >= MAXBITS) {
+      WriteUL (approxWeightsFile, buf);
+      buf = fx >> (bits - (pos - MAXBITS));
+      pos = pos - MAXBITS;
+    }
+
+    here++; i++;
+  }
+
+  // write out the last bits
+  if (pos > 0) WriteUL (approxWeightsFile, buf);
+}
+
+int main (int argc, char **argv) {
+  unsigned char bits = 8;
+  char *filename = "";
+  int ch;
+  opterr = 0;
+  msg_prefix = argv[0];
+
+  while ((ch = getopt (argc, argv, "f:d:b:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'b':
+      bits = atoi (optarg);
+      if (bits > 32) {
+	fprintf (stderr, "b may only take values 0-32\n");
+	exit (1);
+      }
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-f input_file]"
+	       "[-d data directory] [-b bits] [-h]\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // open the inverted file
+  FILE *invfFile = open_file (filename, INVF_SUFFIX, "rb",
+			MAGIC_INVF, MG_ABORT);
+  invf_file_header ifh;
+  ifh.Read (invfFile);
+  if (ifh.skip_mode != SKIP_MODE_NO_SKIPS)
+    FatalError (1, "The invf file contains skips. Unable to create weights.");
+
+  // open the inverted index file
+  FILE *invfIdxFile = open_file (filename, INVF_IDX_SUFFIX, "rb",
+				 MAGIC_INVI, MG_ABORT);
+  
+  // read the level information
+  FILE *levelFile = open_file (filename, INVF_LEVEL_SUFFIX, "rb",
+			       MAGIC_INVF_LEVELS, MG_ABORT);
+  FIvfLevel ivfLevel;
+  ivfLevel.Read (levelFile);
+  fclose (levelFile);
+
+  // read in the tag dictionary and inverted file pointers
+  WBTagDict tagDict;
+  ReadTagDict (idh, dictFile, invfIdxFile, tagDict);
+
+  
+  // create the weights file
+  FILE *weightsFile = create_file (filename, WEIGHTS_SUFFIX, "wb",
+				   MAGIC_WGHT, MG_ABORT);
+
+  // create the approx weights file
+  FILE *approxWeightsFile = create_file (filename, APPROX_WEIGHTS_SUFFIX, "wb",
+					 MAGIC_WGHT_APPROX, MG_ABORT);
+
+  
+  // create weights for each document level
+  FragLevelConvert fragLevelConvert;
+  Weights w;
+  IvfLevelInfoMap::iterator levelHere = ivfLevel.levelInfo.begin();
+  IvfLevelInfoMap::iterator levelEnd = ivfLevel.levelInfo.end();
+  unsigned long levelNum = 0;
+  while (levelHere != levelEnd) {
+    const UCArray &levelName = (*levelHere).first;
+    
+    // read the tag information about this level
+    fragLevelConvert.Read(invfFile, tagDict[levelName].tagPtr,
+			  idh.num_frags, tagDict[levelName].fragOccur);
+    
+    // create the weights for this level
+    GenerateLevelWeights (idh, ifh, tagDict[levelName].fragOccur,
+			  levelNum, dictFile, invfFile, invfIdxFile,
+			  fragLevelConvert, w);
+
+    // write out the exact weights
+    WriteExactWeights (weightsFile,
+		       (*levelHere).second.exactWeightsDiskPtr,
+		       w);
+    
+    // write out the approximate weights
+    WriteApproxWeights (approxWeightsFile,
+			(*levelHere).second.approxWeightsDiskPtr,
+			w, bits);
+    
+    levelHere++; levelNum++;
+  }
+  
+
+  // close input files
+  fclose (dictFile);
+  fclose (invfFile);
+  fclose (invfIdxFile);
+
+  // update the level information
+  levelFile = create_file (filename, INVF_LEVEL_SUFFIX, "wb",
+			   MAGIC_INVF_LEVELS, MG_ABORT);
+  ivfLevel.Write (levelFile);
+  fclose (levelFile);
+
+  // close output files
+  fclose (weightsFile);
+  fclose (approxWeightsFile);
+
+
+  return 0;
+}
Index: /trunk/indexers/mgpp/text/stemmer.cpp
===================================================================
--- /trunk/indexers/mgpp/text/stemmer.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/stemmer.cpp	(revision 3365)
@@ -0,0 +1,105 @@
+/**************************************************************************
+ *
+ * stemmer.cpp -- The stemmer/case folder
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "stemmer.h"
+
+#include "lovinstem.h"
+#include "simplefrenchstem.h"
+#include "unitool.h"
+
+#define LOVINSTEMMER        0
+#define SIMPLEFRENCHSTEMMER 1
+
+
+/* decode the utf-8 encoded unicode, casefold and then recode 
+ * making sure the final length doesn't exceed the original
+ * length */
+static void unicode_casefold (u_char *word) {
+  unsigned short out[256]; /* temp space */
+  int i;
+  int len;
+
+  /* decode */
+  utf8_word_to_unicode (word, out, 255);
+  len = out[0];
+
+  /* casefold and simplify-fold */
+  for (i=0; i<len; i++) {
+    out[i+1] = unicode_tosimplified(unicode_tolower(out[i+1]));
+  }
+
+  /* re-code */
+  unicode_to_utf8_word (out, word, word[0]+1);
+}
+
+
+int stemmernumber (u_char *stemmerdescription) {
+  u_char descript[MAX_STEM_DESCRIPTION_LEN];
+  int i;
+
+  /* copy and case-fold the description */
+  for (i=0; (stemmerdescription[i] != '\0') && 
+	 (i < MAX_STEM_DESCRIPTION_LEN-1); i++)
+    descript[i] = tolower (stemmerdescription[i]);
+  descript[i] = '\0';
+
+  /* map the description to its number */
+
+  if ((strcmp ((char *) descript,  "0") == 0) ||
+      (strcmp ((char *) descript, "english") == 0) ||
+      (strcmp ((char *) descript, "lovin") == 0))
+    return LOVINSTEMMER;
+
+  if ((strcmp ((char *) descript, "1") == 1) ||
+      (strcmp ((char *) descript, "french") == 0) ||
+      (strcmp ((char *) descript, "simplefrench") == 0))
+    return SIMPLEFRENCHSTEMMER;
+
+  return -1;
+}
+
+
+
+/*
+ * Method 0 - Do not stem or case fold.
+ * Method 1 - Case fold.
+ * Method 2 - Stem.
+ * Method 3 - Case fold and stem.
+ *
+ * The stemmer number should be obtained using
+ * the stemmernumber function above.
+ */
+void 
+stemmer (int method, int stemmer, u_char *word) {
+  if (method & 1) {
+    unicode_casefold (word);
+  }
+
+  if (method & 2) {
+    switch (stemmer) {
+    case LOVINSTEMMER: lovinstem (word);
+      break;
+    case SIMPLEFRENCHSTEMMER: simplefrenchstem (word);
+      break;
+    }
+  }
+}
Index: /trunk/indexers/mgpp/text/stemmer.h
===================================================================
--- /trunk/indexers/mgpp/text/stemmer.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/stemmer.h	(revision 3365)
@@ -0,0 +1,65 @@
+/**************************************************************************
+ *
+ * stemmer.h -- The stemmer/case folder
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef STEMMER_H
+#define STEMMER_H
+
+#include "sysfuncs.h"
+
+#define STEMMER_MASK 3
+#define MAX_STEM_DESCRIPTION_LEN 16
+
+/* stemmernumber will return the stemmer for
+ * a description of the stemmer. Stemmer descriptions
+ * are not case sensitive. Valid descriptions are:
+ *
+ *   'English'
+ *   'Lovin'
+ *   'French'
+ *   'SimpleFrench'
+ *
+ * More than one description might result in the same
+ * stemmer number (for example, for stemming 'English'
+ * we currently use the 'Lovin' stemmer).
+ *
+ * stemmerdescription is a normal C, null-terminated,
+ * string.
+ *
+ * stemmernumber will return -1 if it doesn't know the
+ * stemmer description.
+ */
+int stemmernumber (u_char *stemmerdescription);
+
+/*
+ * Method 0 - Do not stem or case fold.
+ * Method 1 - Case fold.
+ * Method 2 - Stem.
+ * Method 3 - Case fold and stem.
+ *
+ * The stemmer number should be obtained using function
+ * stemmernumber above.
+ */
+#ifdef __cplusplus
+extern "C" 
+#endif
+void stemmer (int method, int stemmer, u_char * word);
+
+#endif
Index: /trunk/indexers/mgpp/text/text.cpp
===================================================================
--- /trunk/indexers/mgpp/text/text.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/text.cpp	(revision 3365)
@@ -0,0 +1,146 @@
+/**************************************************************************
+ *
+ * text.h -- Header file for compression related stuff
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// UCArray currently contains all the basic reading and writing functions
+#include "UCArray.h"
+#include "text.h"
+
+compressed_text_header::compressed_text_header () {
+  Clear();
+}
+
+void compressed_text_header::Clear () {
+  num_of_docs = 0;
+  num_of_words = 0;
+  num_of_bytes = 0.0;
+}
+
+
+bool compressed_text_header::Read (FILE *f) {
+  return (ReadUL (f, num_of_docs) &&
+	  ReadUL (f, num_of_words) &&
+	  ReadD (f, num_of_bytes));
+}
+
+bool compressed_text_header::Write (FILE *f) const {
+  return (WriteUL (f, num_of_docs) &&
+	  WriteUL (f, num_of_words) &&
+	  WriteD (f, num_of_bytes));
+}
+
+
+auxiliary_dict::auxiliary_dict () {
+  word_data[0] = NULL;
+  word_data[1] = NULL;
+  words[0] = NULL;
+  words[1] = NULL;
+}
+
+auxiliary_dict::~auxiliary_dict () {
+  Clear ();
+}
+
+void auxiliary_dict::Clear () {
+  if (word_data[0] != NULL) {
+    delete word_data[0];
+    word_data[0] = NULL;
+  }
+  if (word_data[1] != NULL) {
+    delete word_data[1];
+    word_data[1] = NULL;
+  }
+  if (words[0] != NULL) {
+    delete words[0];
+    words[0] = NULL;
+  }
+  if (words[1] != NULL) {
+    delete words[1];
+    words[1] = NULL;
+  }
+}
+
+
+compression_dict::compression_dict () {
+  int which;
+  for (which = 0; which < 2; which++) {
+    cfh[which] = NULL;
+    values[which] = NULL;
+    escape[which] = NULL;
+    chars_huff[which] = NULL;
+    chars_vals[which] = NULL;
+    lens_huff[which] = NULL;
+    lens_vals[which] = NULL;
+  }
+
+  MemForCompDict = 0;
+  ad = NULL;
+  fast_loaded = 0;
+}
+
+compression_dict::~compression_dict () {
+  Clear ();
+}
+
+void compression_dict::Clear () {
+  int which;
+  for (which = 0; which < 2; which++) {
+    if (cfh[which] != NULL) {
+      delete cfh[which];
+      cfh[which] = NULL;
+    }
+    if (values[which] != NULL) {
+      delete values[which][0][0];
+      delete values[which][0];
+      delete values[which];
+      values[which] = NULL;
+    }
+    if (escape[which] != NULL) { // don't know if needed
+      //delete escape[which]; // this was causing a seg fault
+      escape[which] = NULL;
+    }
+    if (chars_huff[which] != NULL) {
+      delete chars_huff[which];
+      chars_huff[which] = NULL;
+    }
+    if (chars_vals[which] != NULL) {
+      delete chars_vals[which][0];
+      delete chars_vals[which];
+      chars_vals[which] = NULL;
+    }
+    if (lens_huff[which] != NULL) {
+      delete lens_huff[which];
+      lens_huff[which] = NULL;
+    }
+    if (lens_vals[which] != NULL) {
+      delete lens_vals[which][0];
+      delete lens_vals[which];
+      lens_vals[which] = NULL;
+    }
+  }
+
+  if (ad != NULL) {
+    delete ad;
+    ad = NULL;
+  }
+
+  MemForCompDict = 0;
+  fast_loaded = 0;
+}
Index: /trunk/indexers/mgpp/text/text.h
===================================================================
--- /trunk/indexers/mgpp/text/text.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/text.h	(revision 3365)
@@ -0,0 +1,199 @@
+/**************************************************************************
+ *
+ * text.h -- Header file for compression related stuff
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_TEXT
+#define H_TEXT
+
+// some elderly windows compilers need to include <vector> (or something
+// similar) here so that they know what a "bool" is
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\vector>
+#elif defined(GSDL_USE_STL_H)
+#  include <vector.h>
+#else
+#  include <vector>
+#endif
+
+
+#include "huffman.h"
+#include <stdio.h>
+
+/*****************************************************************************
+ *
+ * There are several different methods of compressing the text of the
+ * database. The following defines list the different methods of text
+ * compression
+ *
+ */
+
+
+/* The dictionary contains all the fragments that occur in the collection
+   i.e. escapes are not possible */
+#define MG_COMPLETE_DICTIONARY 0
+
+
+/* Certain words have been deleted from the dictionary. The words deleted
+   have been used to create the frequency huffman codes of the characters. 
+   This dictionary has an escape code and may be used to compress novel
+   words. This dictionary may fail if there is a novel character. */
+#define MG_PARTIAL_DICTIONARY 1
+
+
+/* This dictionary has an escape so that novel words and non-words can be 
+   coded. The method for coding the novel words and non-words is determined
+   by a dictionary parameter. */
+#define MG_SEED_DICTIONARY 2
+
+
+
+
+
+
+
+
+
+/*****************************************************************************
+ *
+ * With a seed dictionary there are several methods for coding the novel
+ * words and non-words the following defined values specify the different
+ * methods of coding.
+ *
+ */
+
+
+/* Code novel words and non-words character by character using huffman codes.
+   The huffman codes for the word and non-word lengths and characters are 
+   generated from the distribution of lengths and characters in the
+   dictionary. */
+#define MG_NOVEL_HUFFMAN_CHARS 0
+
+
+/* This method codes novel words using delta codes. The novel words are stored
+   in a auxillary dictionary which is built by pass two. */
+#define MG_NOVEL_DELTA 2
+
+
+/* This method codes novel words using hybrid version of delta. The novel
+   words are stored in a auxillary dictionary which is built by pass two. */
+#define MG_NOVEL_HYBRID 3
+
+
+
+
+
+
+/* This specified an amount of extra space allocated in the
+   compression_dict_header for adding new parameters. As new
+   parameters are added this should be decreased. */
+#define TEXT_PARAMS 15
+
+
+
+struct compression_dict_header {
+  u_long dict_type;
+  u_long novel_method;
+  u_long params[TEXT_PARAMS];
+  u_long num_words[2];
+  u_long num_word_chars[2];
+  u_long lookback;
+};
+
+
+struct comp_frags_header {
+  huff_data hd;
+  u_long uncompressed_size;
+  u_long huff_words_size[MAX_HUFFCODE_LEN + 1];
+};
+
+// BOGUSTEXTLEN is used to replace the ratio
+// in the compressed_text_header while preparing
+// for UCArray
+#define BOGUSTEXTLEN 1000000
+
+struct compressed_text_header {
+  u_long num_of_docs;
+  u_long num_of_words; // number of words in collection
+  double num_of_bytes;
+
+  compressed_text_header ();
+  void Clear ();
+
+  // you must seek to the appropriate place before calling
+  // Read or Write
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+struct compression_stats_header {
+  u_long num_docs;
+  u_long dummy;     // alignment
+  double num_bytes; 
+};
+
+
+struct frags_stats_header {
+  u_long num_frags;
+  u_long mem_for_frags;
+};
+
+
+struct aux_frags_header {
+  u_long num_frags;
+  u_long mem_for_frags;
+};
+
+
+struct auxiliary_dict {
+  aux_frags_header afh[2];
+  u_char *word_data[2];
+  u_char **words[2];
+  int blk_start[2][33], blk_end[2][33];	/* blk_start and blk_end are required
+					   for the hybrid methods */
+
+  auxiliary_dict ();
+  ~auxiliary_dict ();
+
+  void Clear ();
+};
+
+
+struct compression_dict {
+  compression_dict_header cdh;
+  comp_frags_header *cfh[2];
+  unsigned long MemForCompDict;
+  u_char ***values[2];
+  u_char *escape[2];
+  huff_data *chars_huff[2];
+  u_long **chars_vals[2];
+  huff_data *lens_huff[2];
+  u_long **lens_vals[2];
+  auxiliary_dict *ad;
+  int fast_loaded;
+
+  compression_dict ();
+  ~compression_dict ();
+
+  void Clear ();
+};
+
+
+#endif
Index: /trunk/indexers/mgpp/text/text.pass1.cpp
===================================================================
--- /trunk/indexers/mgpp/text/text.pass1.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/text.pass1.cpp	(revision 3365)
@@ -0,0 +1,387 @@
+/**************************************************************************
+ *
+ * text.pass1.cpp -- Text compression (Pass 1)
+ * Copyright (C) 1994  Neil Sharman, Gary Eddy and Alistair Moffat
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "huffman.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+#include "mg_files.h"
+#include "mg.h"
+#include "build.h"
+#include "locallib.h"
+#include "words.h"
+#include "text.h"
+#include "hash.h"
+#include "local_strings.h"
+#include "TextEl.h"
+
+/* for more meaningful messages - jrm21 (gsdl) */
+#ifdef HAVE_STRERROR
+#include <string.h>
+#include <errno.h>
+#endif
+
+
+#define POOL_SIZE 1024*1024
+#define INITIAL_HASH_SIZE 7927
+
+
+typedef struct hash_rec
+  {
+    unsigned long wcnt;		/* word frequency */
+    unsigned long occurance_num;
+    u_char *word;
+  }
+hash_rec;
+
+typedef struct dict_data
+  {
+    hash_rec *HashTable;
+    unsigned long HashSize;
+    unsigned long HashUsed;
+    unsigned long wordnum;
+    unsigned long words_read;
+    unsigned long bytes_diff;
+    huff_data hd;
+  }
+dict_data;
+
+
+
+static unsigned long longestDoc = 0;
+static unsigned long occurance_num = 0;
+static dict_data DictData[2];
+
+static u_char *Pool;
+static int PoolLeft;
+static double inputbytes = 0; /* [RJM 07/97: 4G limit] */
+static unsigned long MaxMemInUse = 0;
+static unsigned long MemInUse = 0;
+static compression_stats_header csh = {0, 0, 0.0}; /* [RJM 07/97: 4G limit] */
+
+
+static void ChangeMem (int Change) {
+  MemInUse += Change;
+  if (MemInUse > MaxMemInUse) MaxMemInUse = MemInUse;
+}
+
+
+int init_text_1 (const TagInfo &/*tagInfo*/, char * /*FileName*/) {
+  int which;
+  
+  if (!(Pool = (u_char *) Xmalloc (POOL_SIZE))) {
+    Message ("Unable to allocate memory for pool");
+    return (COMPERROR);
+  }
+  PoolLeft = POOL_SIZE;
+  ChangeMem (POOL_SIZE);
+  
+  for (which = 1; which >= 0; which--) {
+    u_char *word;
+    hash_rec *ent;
+    dict_data *dd = &DictData[which];
+    
+    dd->wordnum = 0;
+    dd->words_read = 0;
+    dd->bytes_diff = 0;
+    dd->HashSize = INITIAL_HASH_SIZE;
+    dd->HashUsed = 0;
+    
+    if (!(dd->HashTable = (hash_rec *) Xmalloc (sizeof (hash_rec) * dd->HashSize))) {
+      Message ("Unable to allocate memory for table");
+      return (COMPERROR);
+    }
+    ChangeMem (sizeof (hash_rec) * dd->HashSize);
+    memset (dd->HashTable,'\0' , sizeof (hash_rec) * dd->HashSize);
+
+    word = Pool;
+    *Pool++ = '\0';
+    PoolLeft--;
+    {
+      register u_char *wptr;
+      register int hsize = dd->HashSize;
+      register unsigned long hashval, step;
+      
+      HASH (hashval, step, word, hsize);
+      wptr = (dd->HashTable + hashval)->word;
+      while (wptr) {
+	hashval += step;
+	if (hashval >= (unsigned long)hsize)
+	  hashval -= hsize;
+	wptr = (dd->HashTable + hashval)->word;
+      }
+      ent = dd->HashTable + hashval;
+    }
+    ent->wcnt = 1;
+    ent->word = word;
+    dd->HashUsed = 1;
+  }
+  return (COMPALLOK);
+}
+
+
+static int process_text_element (const u_char *s_in, int l_in) {
+  const u_char *end = s_in + l_in - 1;
+  
+   /*
+   ** Alternately parse off words and non-words from the input
+   ** stream beginning with a non-word. Each token is then
+   ** inserted into the set if it does not exist or has it's
+   ** frequency count incremented if it does.
+   */
+  
+  bool which = false; // non-word
+  for (; s_in <= end; which = !which) {
+    u_char Word[MAXWORDLEN + 1];
+    dict_data *dd = &DictData[which];
+    
+    /* First parse a word or non-word out of the string */
+    if (which) PARSE_WORD (Word, s_in, end);
+    else PARSE_NON_WORD (Word, s_in, end);
+    
+    dd->wordnum++;
+    inputbytes += *Word;
+    dd->words_read++;
+    
+    /* Search the hash table for Word */
+    {
+      register unsigned long hashval, step;
+      register int hsize = dd->HashSize;
+      HASH (hashval, step, Word, hsize);
+      for (;;) {
+	register u_char *s1;
+	register u_char *s2;
+	register int len;
+	register hash_rec *ent;
+	ent = dd->HashTable + hashval;
+	if (!ent->word) {
+	  int len = *Word + 1;
+	  if (len > PoolLeft) {
+	    if (!(Pool = (u_char *) Xmalloc (POOL_SIZE))) {
+	      Message ("Unable to allocate memory for pool");
+	      return (COMPERROR);
+	    }
+	    PoolLeft = POOL_SIZE;
+	    ChangeMem (POOL_SIZE);
+	  }
+	  ent->occurance_num = occurance_num++;
+	  ent->wcnt = 1;
+	  ent->word = Pool;
+	  memcpy (Pool, Word, len);
+	  Pool += len;
+	  PoolLeft -= len;
+	  dd->HashUsed++;
+	  dd->bytes_diff += Word[0];
+	  break;
+	}
+	
+	/* Compare the words */
+	s1 = Word;
+	s2 = ent->word;
+	len = *s1 + 1;
+	for (; len; len--)
+	  if (*s1++ != *s2++) break;
+	
+	if (len) {
+	  hashval = (hashval + step);
+	  if (hashval >= (unsigned long)hsize) hashval -= hsize;
+	} else {
+	  ent->wcnt++;
+	  break;
+	}
+      }
+    }
+
+    
+    if (dd->HashUsed >= dd->HashSize >> 1) {
+      hash_rec *ht;
+      unsigned long size;
+      unsigned long i;
+      size = prime (dd->HashSize * 2);
+      if (!(ht = (hash_rec *) Xmalloc (sizeof (hash_rec) * size))) {
+	Message ("Unable to allocate memory for table");
+	return (COMPERROR);
+      }
+      ChangeMem (sizeof (hash_rec) * size);
+      memset (ht, '\0', sizeof (hash_rec) * size);
+
+      for (i = 0; i < dd->HashSize; i++)
+	if (dd->HashTable[i].word) {
+	  register u_char *wptr;
+	  register unsigned long hashval, step;
+	  
+	  wptr = dd->HashTable[i].word;
+	  HASH (hashval, step, wptr, size);
+	  wptr = (ht + hashval)->word;
+	  while (wptr) {
+	    hashval += step;
+	    if (hashval >= size) hashval -= size;
+	    wptr = (ht + hashval)->word;
+	  }
+	  ht[hashval] = dd->HashTable[i];
+	}
+      Xfree (dd->HashTable);
+      ChangeMem (-sizeof (hash_rec) * dd->HashSize);
+      dd->HashTable = ht;
+      dd->HashSize = size;
+    }
+  }
+
+  return COMPALLOK;
+}
+
+
+int process_text_1 (const TagInfo &/*tagInfo*/, const TextElArray &doc) {
+  unsigned long textLen = 0;
+  unsigned long docLen = 0;
+  int retValue;
+
+  // process each text element in this document
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    textLen = (*here).text.size();
+    docLen += textLen;
+
+    retValue = process_text_element (&(here->text[0]), textLen);
+    if (retValue != COMPALLOK) return retValue;
+    
+    here++;
+  }
+
+  // get max document length
+  if (docLen > longestDoc) longestDoc = docLen;
+
+  // update header information
+  csh.num_docs++;
+  csh.num_bytes += docLen;
+  
+  return COMPALLOK;
+}
+
+
+static int PackHashTable (dict_data * dd) {
+  int s, d;
+  for (s = d = 0; (unsigned int)s < dd->HashSize; s++)
+    if (dd->HashTable[s].word)
+      dd->HashTable[d++] = dd->HashTable[s];
+  
+  ChangeMem (-sizeof (hash_rec) * dd->HashSize);
+  ChangeMem (sizeof (hash_rec) * dd->HashUsed);
+  
+  if (!(dd->HashTable = (hash_rec *) Xrealloc (dd->HashTable,
+			   sizeof (hash_rec) * dd->HashUsed))) {
+    Message ("Out of memory");
+    return COMPERROR;
+  }
+  dd->HashSize = dd->HashUsed;
+  return COMPALLOK;
+}
+
+
+static int ent_comp (const void *s1, const void *s2) {
+  return casecompare (((hash_rec *) s1)->word, ((hash_rec *) s2)->word);
+}
+
+
+static void WriteHashTable (FILE * fp, dict_data * dd) {
+  frags_stats_header fsh;
+  u_long j = 0;
+  u_char *curr;
+
+  if (PackHashTable (dd) == COMPERROR) return;
+  
+  qsort (dd->HashTable, dd->HashUsed, sizeof (hash_rec), ent_comp);
+  
+  fsh.num_frags = dd->HashSize;
+  fsh.mem_for_frags = dd->HashSize;
+  for (j = 0; j < dd->HashSize; j++)
+    fsh.mem_for_frags += dd->HashTable[j].word[0];
+  
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONUL(fsh.num_frags);
+  HTONUL(fsh.mem_for_frags);
+  
+  fwrite (&fsh, sizeof (fsh), 1, fp);
+  
+  for (j = 0; j < dd->HashSize; j++) {
+    curr = dd->HashTable[j].word;
+    
+    /* [RPAP - Jan 97: Endian Ordering] */
+    HTONUL(dd->HashTable[j].wcnt);
+    HTONUL(dd->HashTable[j].occurance_num);
+    
+    fwrite (&dd->HashTable[j].wcnt, sizeof (dd->HashTable[j].wcnt), 1, fp);
+    fwrite (&dd->HashTable[j].occurance_num,
+	    sizeof (dd->HashTable[j].occurance_num), 1, fp);
+    
+    /* [RPAP - Jan 97: Endian Ordering] */
+    NTOHUL(dd->HashTable[j].wcnt);
+    NTOHUL(dd->HashTable[j].occurance_num);
+    
+    fwrite (curr, sizeof (u_char), curr[0] + 1, fp);
+  }
+}
+
+
+int done_text_1 (const TagInfo &/*tagInfo*/, char *file_name) {
+  char *temp_str;
+  FILE *fp;
+  
+  if (!(fp = create_file (file_name, TEXT_STATS_DICT_SUFFIX, "wb",
+			  MAGIC_STATS_DICT, MG_MESSAGE)))  /* [RPAP - Feb 97: WIN32 Port] */
+    {
+      fprintf(stderr,"Couldn't create file %s%s:%s\n",
+              file_name, TEXT_STATS_DICT_SUFFIX,
+#if defined(HAVE_STRERROR) || defined(__WIN32__)
+              strerror(errno)
+#else
+	      " "
+#endif
+              );
+      return COMPERROR;
+    }
+
+  temp_str = msg_prefix;
+  msg_prefix = "text.pass1";
+  
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONUL(csh.num_docs);
+  HTOND(csh.num_bytes); /* [RJM 07/97: 4G limit] */
+
+  fwrite (&csh, sizeof (csh), 1, fp);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(csh.num_docs);
+  NTOHD(csh.num_bytes); /* [RJM 07/97: 4G limit] */
+
+  WriteHashTable (fp, &DictData[0]);
+  WriteHashTable (fp, &DictData[1]);
+  msg_prefix = temp_str;
+  return COMPALLOK;
+}
Index: /trunk/indexers/mgpp/text/text.pass2.cpp
===================================================================
--- /trunk/indexers/mgpp/text/text.pass2.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/text.pass2.cpp	(revision 3365)
@@ -0,0 +1,581 @@
+/**************************************************************************
+ *
+ * text.pass2.cpp -- Text compression (Pass 2)
+ * Copyright (C) 1994  Neil Sharman, Gary Eddy and Alistair Moffat
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#if defined (__WIN32__)
+# include <stdio.h>
+# define unlink _unlink
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "bitio_m_stdio.h"
+#include "huffman.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+#include "mg.h"
+#include "mg_files.h"
+#include "build.h"
+#include "words.h"
+#include "text.h"
+#include "hash.h"
+#include "locallib.h"
+#include "comp_dict.h"
+#include "FText.h"
+
+#define POOL_SIZE 1024*256
+
+
+struct char_pool {
+  struct char_pool *next;
+  u_long left;
+  u_char *ptr;
+  u_char pool[POOL_SIZE];
+};
+
+struct novel_hash_rec {
+  u_long ordinal_num;
+  u_char *word;
+};
+
+
+#define INITIAL_HASH_SIZE 7927
+
+struct novel_hash_table {
+  novel_hash_rec *HashTable;
+  u_long HashSize, HashUsed;
+  char_pool *first_pool;
+  char_pool *pool;
+  u_long next_num, binary_start;
+  novel_hash_rec **code_to_nhr;
+};
+
+
+static FILE *text;
+static stdio_bitio_buffer textOutBuf;
+static u_long headLen;
+
+static novel_hash_table nht[2];
+
+int blk_start[2][33], blk_end[2][33];
+
+// 0 = non-word, 1 = word
+unsigned char whichWordType = 0;
+
+CompressTextInfoMap textInfoMap;
+
+
+static char_pool *new_pool (char_pool * pool) {
+  char_pool *p = (char_pool *) Xmalloc (sizeof (char_pool));
+  if (!p) FatalError (1, "Unable to allocate memory for pool");
+  
+  if (pool) pool->next = p;
+  p->next = NULL;
+  p->left = POOL_SIZE;
+  p->ptr = p->pool;
+  return p;
+}
+
+
+int init_text_2 (const TagInfo &/*tagInfo*/, char *fileName) {
+  char path[512];
+
+  // load the compression dictionary
+  if (LoadCompressionDictionary (make_name (fileName, TEXT_DICT_SUFFIX,
+					    path)) == COMPERROR)
+    return COMPERROR;
+
+  // create the compressed text file and attach it to the text buffer
+  if (!(text = create_file (fileName, TEXT_SUFFIX, "w+b",
+			    MAGIC_TEXT, MG_MESSAGE)))
+    return COMPERROR;
+  textOutBuf.attachFile (text);
+  textOutBuf.encodeStart ();
+  
+  // write out the compressed text header
+  memset (&cth, '\0', sizeof (cth));
+  if (fwrite (&cth, sizeof (cth), 1, text) != 1)
+    return COMPERROR;
+
+  // the length of the compressed header is the size of the
+  // magic number plus the size of the compressed text header
+  headLen = sizeof (u_long) + sizeof (cth);
+
+  // start off with non-word
+  whichWordType = 0;
+
+  // clear the text pointers
+  textInfoMap.erase (textInfoMap.begin(), textInfoMap.end());
+  
+  // set up the hash table of novel words
+  int i;
+  if (cdh.novel_method != MG_NOVEL_HUFFMAN_CHARS)
+    for (i = 0; i <= 1; i++)
+      {
+	nht[i].HashSize = INITIAL_HASH_SIZE;
+	nht[i].HashTable = (novel_hash_rec *) Xmalloc (sizeof (novel_hash_rec) * nht[i].HashSize);
+	memset (nht[i].HashTable, '\0',
+	       sizeof (novel_hash_rec) * nht[i].HashSize);
+	nht[i].HashUsed = 0;
+	nht[i].HashSize = INITIAL_HASH_SIZE;
+	nht[i].pool = nht[i].first_pool = new_pool (NULL);
+	nht[i].next_num = 1;
+	nht[i].binary_start = 1;
+	nht[i].code_to_nhr = NULL;
+	if (cdh.novel_method == MG_NOVEL_HYBRID)
+	  {
+	    int num;
+	    num = 1;
+	    blk_start[i][0] = 0;
+	    blk_end[i][0] = cdh.num_words[i] - 1;
+	    while (num < 33)
+	      {
+		blk_start[i][num] = blk_end[i][num - 1] + 1;
+		blk_end[i][num] = blk_start[i][num] +
+		  (blk_end[i][num - 1] - blk_start[i][num - 1]) * 2;
+		num++;
+	      }
+	  }
+      }
+
+  return (COMPALLOK);
+}
+
+
+static int compress_text (bitio_buffer &buffer, const u_char *s_in,
+			  int l_in, unsigned char &which,
+			  unsigned long &numWords) {
+  const u_char *end = s_in + l_in - 1;
+
+  for (; s_in <= end; which = !which) {
+    u_char Word[MAXWORDLEN + 1];
+    int res;
+    
+    if (which) numWords++;
+
+    /* First parse a word or non-word out of the string */
+    if (which) PARSE_WORD (Word, s_in, end);
+    else PARSE_NON_WORD (Word, s_in, end);
+
+    /* Search the hash table for Word */
+    if (ht[which]) {
+      register unsigned long hashval, step;
+      register int tsize = ht[which]->size;
+      register u_char **wptr;
+      HASH (hashval, step, Word, tsize);
+      for (;;) {
+	register u_char *s1;
+	register u_char *s2;
+	register int len;
+	wptr = ht[which]->table[hashval];
+	if (wptr == NULL) {
+	  res = COMPERROR;
+	  break;
+	}
+
+	/* Compare the words */
+	s1 = Word;
+	s2 = *wptr;
+	len = *s1 + 1;
+	for (; len; len--)
+	  if (*s1++ != *s2++) break;
+	
+	if (len) {
+	  hashval += step;
+	  if (hashval >= (unsigned long)tsize) hashval -= tsize;
+	  
+	} else {
+	  res = ht[which]->table[hashval] - ht[which]->words;
+	  break;
+	}
+      }
+    }
+    else
+      res = COMPERROR;
+    /* Check that the word was found in the dictionary */
+    if (res == COMPERROR)
+      {
+	if (cdh.dict_type == MG_COMPLETE_DICTIONARY)
+	  {
+	    Message ("Unknown word \"%.*s\"\n", *Word, Word + 1);
+	    return (COMPERROR);
+	  }
+	if (cdh.dict_type == MG_PARTIAL_DICTIONARY)
+	  {
+	    u_long i;
+	    if (ht[which])
+	      {
+		res = ht[which]->hd->num_codes - 1;
+		buffer.huff_encode (res, ht[which]->codes, ht[which]->hd->clens, NULL);
+	      }
+	    buffer.huff_encode (Word[0], lens_codes[which], lens_huff[which].clens, NULL);
+	    for (i = 0; i < Word[0]; i++)
+	      buffer.huff_encode (Word[i + 1], char_codes[which],
+				  char_huff[which].clens, NULL);
+	  }
+	if (cdh.dict_type == MG_SEED_DICTIONARY)
+	  {
+	    if (ht[which])
+	      {
+		res = ht[which]->hd->num_codes - 1;
+		buffer.huff_encode (res, ht[which]->codes, ht[which]->hd->clens, NULL);
+	      }
+	    switch (cdh.novel_method)
+	      {
+	      case MG_NOVEL_HUFFMAN_CHARS:
+		{
+		  u_long i;
+		  buffer.huff_encode (Word[0], lens_codes[which],
+				      lens_huff[which].clens, NULL);
+		  for (i = 0; i < Word[0]; i++)
+		    buffer.huff_encode (Word[i + 1], char_codes[which],
+					char_huff[which].clens, NULL);
+		}
+		break;
+	      case MG_NOVEL_DELTA:
+	      case MG_NOVEL_HYBRID:
+		{
+		  register unsigned long hashval, step;
+		  register novel_hash_table *h = &nht[which];
+		  register int hsize = h->HashSize;
+		  register novel_hash_rec *ent;
+		  HASH (hashval, step, Word, hsize);
+		  for (;;)
+		    {
+		      register u_char *s1, *s2;
+		      register int len;
+		      ent = h->HashTable + hashval;
+		      if (!ent->word)
+			{
+			  int len = *Word + 1;
+			  if ((unsigned)len > h->pool->left)
+			    h->pool = new_pool (h->pool);
+			  ent->word = h->pool->ptr;
+			  ent->ordinal_num = h->next_num++;
+			  memcpy (h->pool->ptr, Word, len);
+			  h->pool->ptr += len;
+			  h->pool->left -= len;
+			  h->HashUsed++;
+			  break;
+			}
+		      /* Compare the words */
+		      s1 = Word;
+		      s2 = ent->word;
+		      len = *s1 + 1;
+		      for (; len; len--)
+			if (*s1++ != *s2++)
+			  break;
+		      
+		      if (!len)
+			break;
+		      
+		      hashval = (hashval + step);
+		      if (hashval >= (unsigned)hsize) hashval -= hsize;
+		    }
+		  
+		  switch (cdh.novel_method) {
+		  case MG_NOVEL_DELTA:
+		    buffer.delta_encode (ent->ordinal_num, NULL);
+		    break;
+		  case MG_NOVEL_HYBRID:
+		    int k = 0;
+		    int j = ent->ordinal_num - 1;
+		    while (j > blk_end[which][k])
+		      k++;
+		    assert (j - blk_start[which][k] + 1 >= 1 &&
+			    j - blk_start[which][k] + 1 <=
+			    blk_end[which][k] - blk_start[which][k] + 1);
+		    
+		    buffer.gamma_encode (k + 1, NULL);
+		    buffer.binary_encode (j - blk_start[which][k] + 1,
+					  blk_end[which][k] - blk_start[which][k] + 1,
+					  NULL);
+		    break;
+		  }
+		  
+		  if (h->HashUsed >= h->HashSize >> 1)
+		    {
+		      novel_hash_rec *ht;
+		      unsigned long size;
+		      unsigned long i;
+		      size = prime (h->HashSize * 2);
+		      if (!(ht = (novel_hash_rec *) Xmalloc (sizeof (novel_hash_rec) * size)))
+			{
+			  Message ("Unable to allocate memory for table");
+			  return (COMPERROR);
+			}
+		      memset (ht, '\0', sizeof (novel_hash_rec) * size);
+		      
+		      for (i = 0; i < h->HashSize; i++)
+			if (h->HashTable[i].word)
+			  {
+			    register u_char *wptr;
+			    register unsigned long hashval, step;
+			    
+			    wptr = h->HashTable[i].word;
+			    HASH (hashval, step, wptr, size);
+			    wptr = (ht + hashval)->word;
+			    while (wptr)
+			      {
+				hashval += step;
+				if (hashval >= size)
+				  hashval -= size;
+				wptr = (ht + hashval)->word;
+			      }
+			    ht[hashval] = h->HashTable[i];
+			  }
+		      Xfree (h->HashTable);
+		      h->HashTable = ht;
+		      h->HashSize = size;
+		    }
+		}
+		break;
+	      }
+	  }
+      }
+    else
+      {
+	buffer.huff_encode (res, ht[which]->codes, ht[which]->hd->clens, NULL);
+      }
+  }
+  
+  return COMPALLOK;
+}
+
+
+int process_text_2 (const TagInfo &tagInfo, const TextElArray &doc) {
+  // we need to remember where the last text element ends
+  unsigned char endBit = textOutBuf.GetBtg (); // 8=high bit, 1=low bit
+  unsigned long endPos = textOutBuf.GetBytesWritten () + headLen;
+
+  // process each text element
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    // remember current place in compressed text
+    unsigned char startWhich = whichWordType;
+    unsigned char startBit = textOutBuf.GetBtg (); // 8=high bit, 1=low bit
+    unsigned long startPos = textOutBuf.GetBytesWritten () + headLen;
+
+    // compress the text
+    if (compress_text (textOutBuf,
+		       &(here->text[0]),
+		       (*here).text.size(),
+		       whichWordType,
+		       cth.num_of_words) != COMPALLOK)
+      return COMPERROR;
+
+    endBit = textOutBuf.GetBtg (); // 8=high bit, 1=low bit
+    endPos = textOutBuf.GetBytesWritten () + headLen;
+    
+    // update the pointers if this is a level tag
+    if (tagInfo.levelTags.find ((*here).tagName) != tagInfo.levelTags.end()) {
+      if ((*here).elType == OpenTagE) {
+	// set the start of the document
+	textInfoMap[(*here).tagName].SetStart (startPos, startBit, startWhich);
+	
+      } else if ((*here).elType == CloseTagE) {
+	// set the end of the document
+	textInfoMap[(*here).tagName].SetEnd (endPos, endBit);
+	
+      } else {
+	// TextE, nothing needs to be done
+      }
+    }
+    
+    cth.num_of_bytes += (*here).text.size();
+    here++;
+  }
+
+  // close off any unclosed tags
+  CompressTextInfoMap::iterator tiHere = textInfoMap.begin ();
+  CompressTextInfoMap::iterator tiEnd = textInfoMap.end ();
+  while (tiHere != tiEnd) {
+    if ((*tiHere).second.inDoc) (*tiHere).second.SetEnd (endPos, endBit);
+    tiHere++;
+  }
+  
+  // we've processed one more document
+  cth.num_of_docs++;
+
+  return COMPALLOK;
+}
+
+
+static int write_aux_dict (char *fileName) {
+  int i;
+  FILE *aux;
+  if (!(aux = create_file (fileName, TEXT_DICT_AUX_SUFFIX, "wb",
+			   MAGIC_AUX_DICT, MG_MESSAGE)))
+    return COMPERROR;
+  
+  for (i = 0; i <= 1; i++)
+    {
+      aux_frags_header afh;
+      char_pool *cp;
+      
+      afh.num_frags = nht[i].HashUsed;
+      afh.mem_for_frags = 0;
+      for (cp = nht[i].first_pool; cp; cp = cp->next)
+	afh.mem_for_frags += POOL_SIZE - cp->left;
+      
+      /* [RPAP - Jan 97: Endian Ordering] */
+      HTONUL(afh.num_frags);
+      HTONUL(afh.mem_for_frags);
+      
+      fwrite (&afh, sizeof (afh), 1, aux);
+      
+      for (cp = nht[i].first_pool; cp; cp = cp->next)
+	fwrite (cp->pool, POOL_SIZE - cp->left, sizeof (u_char), aux);
+    }
+  fclose (aux);
+  return COMPALLOK;
+}
+
+
+static void estimate_compressed_aux_dict (void) {
+  int i;
+  u_long aux_compressed = 0, total_uncomp = 0;
+  for (i = 0; i <= 1; i++)
+    {
+      int j;
+      long chars[256], fchars[256];
+      long lens[16], flens[16];
+      char_pool *cp;
+      memset (chars, '\0', sizeof (chars));
+      memset (lens, '\0', sizeof (lens));
+      for (cp = nht[i].first_pool; cp; cp = cp->next)
+	{
+	  u_char *buf = cp->pool;
+	  while (buf != cp->ptr)
+	    {
+	      int len = *buf++;
+	      lens[len]++;
+	      total_uncomp += len + 4;
+	      for (; len; len--)
+		chars[*buf++]++;
+	    }
+	}
+      for (j = 0; j < 256; j++)
+	if (!chars[j] && PESINAWORD (j) == i)
+	  fchars[j] = 1;
+	else
+	  fchars[j] = chars[j];
+      for (j = 0; j < 16; j++)
+	if (!lens[j])
+	  flens[j] = 1;
+	else
+	  flens[j] = lens[j];
+      
+      aux_compressed += (long unsigned int) ((Calculate_Huffman_Size (16, flens, lens) +
+					      Calculate_Huffman_Size (256, fchars, chars)) / 8);
+      
+    }
+}
+
+
+
+static bool WriteTextIdx (char *fileName, FTextLevel &textLevels) {
+  // create the text index
+  FILE *textIdxFile;
+  if (!(textIdxFile = create_file (fileName, TEXT_IDX_SUFFIX, "w+b",
+				MAGIC_TEXI, MG_MESSAGE)))
+    return false;
+
+  // output each level
+  CompressTextInfoMap::iterator tiHere = textInfoMap.begin ();
+  CompressTextInfoMap::iterator tiEnd = textInfoMap.end ();
+  while (tiHere != tiEnd) {
+    // remember the level information
+    TextLevelInfo levelInfo;
+    levelInfo.levelTag = (*tiHere).first;
+    levelInfo.textIdxPtr = ftell (textIdxFile);
+    levelInfo.numEntries = (*tiHere).second.docPtrs.size();
+    textLevels.levelInfo[levelInfo.levelTag] = levelInfo;
+
+    // write out the index array for this level
+    if (!WriteTextIdxArray (textIdxFile, (*tiHere).second.docPtrs))
+      return false;
+    
+    tiHere++;
+  }
+
+  // close the file
+  fclose (textIdxFile);
+
+  return true;
+}
+
+
+bool WriteTextLevels (char *fileName, const FTextLevel &textLevels) {
+  //  cout << textLevels;
+  
+  // create the text index
+  FILE *textLevelFile;
+  if (!(textLevelFile = create_file (fileName, TEXT_LEVEL_SUFFIX, "w+b",
+				     MAGIC_TEXT_LEVELS, MG_MESSAGE)))
+    return false;
+
+  // output each level
+  if (!textLevels.Write (textLevelFile)) return false;
+  
+  // close the file
+  fclose (textLevelFile);
+
+  return true;
+}
+
+
+int done_text_2 (const TagInfo &/*tagInfo*/, char *fileName) {
+  // write out any remaining bits
+  textOutBuf.encodeDone();
+
+  // write out the compressed text header (with updated statistics)
+  if (fseek (text, sizeof (u_long), SEEK_SET) == -1 || !cth.Write (text))
+    return COMPERROR;
+  fclose (text);
+
+  // write out the text index file and get the level information
+  FTextLevel textLevels;
+  if (!WriteTextIdx (fileName, textLevels)) return COMPERROR;
+  
+  // write out the text level information
+  if (!WriteTextLevels (fileName, textLevels)) return COMPERROR;
+
+  // write out the auxiliary dictionary
+  if (cdh.dict_type != MG_COMPLETE_DICTIONARY &&
+      (cdh.novel_method == MG_NOVEL_DELTA ||
+       cdh.novel_method == MG_NOVEL_HYBRID)) {
+    if (write_aux_dict (fileName) == COMPERROR) return COMPERROR;
+    estimate_compressed_aux_dict ();
+    
+  } else {
+    unlink (make_name (fileName, TEXT_DICT_AUX_SUFFIX, NULL));
+  }
+
+  return (COMPALLOK);
+}
Index: /trunk/indexers/mgpp/text/words.cpp
===================================================================
--- /trunk/indexers/mgpp/text/words.cpp	(revision 3365)
+++ /trunk/indexers/mgpp/text/words.cpp	(revision 3365)
@@ -0,0 +1,100 @@
+/**************************************************************************
+ *
+ * words.cpp -- Functions for parsing out words from the source text
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "words.h"
+
+
+/* Takes the place of the old INAWORD macro. It determines
+   whether a given place in a UTF-8 encoded Unicode string
+   is part of a word. */
+int inaword (const u_char *here, const u_char *end) {
+  unsigned short c;
+  if (parse_utf8_char(here, end, &c) > 0) return is_unicode_letdig (c);
+  return 0;
+}
+
+const unsigned char *ParseIndexWord (const unsigned char *textHere,
+				     const unsigned char *textEnd,
+				     UCArray &word) {
+  word.erase (word.begin(), word.end());
+
+  register int charlength = 0;
+  register int length = 0;
+  register int numeric = 0;
+  unsigned short c;
+  
+  charlength = parse_utf8_char (textHere, textEnd, &c);
+  
+  while (length+charlength <= MAXSTEMLEN && charlength > 0 &&
+	 (is_unicode_letter(c) || (is_unicode_digit(c) &&
+				   ++numeric <= MAXNUMERIC))) {
+    while (charlength-- > 0) {
+      word.push_back (*textHere++); length++;
+    }
+    charlength = parse_utf8_char (textHere, textEnd, &c);
+  }
+  
+  return textHere;
+}
+
+const unsigned char *ParseIndexMGWord (const unsigned char *textHere,
+				       const unsigned char *textEnd,
+				       unsigned char *mgWord) {
+  register int charlength = 0;
+  register int length = 0;
+  register int numeric = 0;
+  unsigned short c;
+  
+  charlength = parse_utf8_char (textHere, textEnd, &c);
+  
+  while (length+charlength <= MAXSTEMLEN && charlength > 0 &&
+	 (is_unicode_letter(c) || (is_unicode_digit(c) &&
+				   ++numeric <= MAXNUMERIC))) {
+    while (charlength-- > 0) {
+      mgWord[++length] = *textHere++;
+    }
+    charlength = parse_utf8_char (textHere, textEnd, &c);
+  }
+
+  mgWord[0] = length;
+  
+  return textHere;
+}
+
+const unsigned char *ParseNonindexWord (const unsigned char *textHere,
+					const unsigned char *textEnd) {
+  register int charlength = 0;
+  unsigned short c;
+  
+  charlength = parse_utf8_char(textHere, textEnd, &c);
+  
+  while (charlength > 0 && !is_unicode_letdig(c)) {
+    textHere += charlength;
+    charlength = parse_utf8_char (textHere, textEnd, &c);
+  }
+  
+  return textHere;
+}
Index: /trunk/indexers/mgpp/text/words.h
===================================================================
--- /trunk/indexers/mgpp/text/words.h	(revision 3365)
+++ /trunk/indexers/mgpp/text/words.h	(revision 3365)
@@ -0,0 +1,155 @@
+/**************************************************************************
+ *
+ * words.h -- Macros for parsing out words from the source text
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "unitool.h"
+#include "UCArray.h"
+
+
+/*
+ * This has been cleaned up by Tim Shimmin. 
+ */
+
+/*
+ * ---NOTE---
+ *
+ * "WORD" refers to a word in the compressed text.
+ * "STEM" or "STEM_WORD" refers to a word for indexing on
+ *
+ */
+
+#define MAXWORDLEN	15
+	/* Maximum length in bytes of any word or non-word. Note that
+	   variations to MAXWORDLEN may have dramatic effects on the rest
+	   of the program, as the length and the prefix match are packed
+	   together into a four bit nibble, and there is not check that
+	   this is possible, i.e., leave MAXWORDLEN alone... */
+
+#define MAXSTEMLEN	255
+	/* Maximum length in bytes of any stem. Note that
+	   variations to MAXSTEMLEN may have dramatic effects on the rest
+	   of the program, , i.e., leave MAXSTEMLEN alone... */
+
+#define MAXNUMERIC	4
+	/* Maximum number of numeric characters permitted in a word.
+	   This avoids long sequences of numbers creating just one
+	   word occurrence for each number. At most 10,000 all numeric
+	   words will be permitted. */
+
+/* [RPAP - Jan 97: Stem Index Change] */
+#define MAXPARAMLEN     20
+        /* Maximum number of bytes to read for a parameter value for a
+	   term in a query. */
+#define WEIGHTPARAM     '/'
+#define STEMPARAM       '#'
+
+/* [RJM 07/97: Ranked Required Terms] */
+#define MUSTMATCHPARAM     '+'
+
+
+#define PESINAWORD(c)      (isalnum(c) || ((c) >= 0x80 && (c) <= 0xff))
+        /* The definition of what characters are permitted in a word.
+	   This macro is pessimistic, you cannot tell from a particular
+	   byte above 0x80 whether it is a character or not. This function
+	   is needed by various functions relating to huffman coding
+	   where frequency counts need to be primed, it should not be
+	   used in parsing the UTF-8 encoded input. */
+
+#ifdef __cplusplus
+extern "C"
+#endif
+int inaword (const u_char *here, const u_char *end);
+        /* Takes the place of the old INAWORD macro. It determines
+	   whether a given place in a UTF-8 encoded Unicode string
+	   is part of a word. */
+
+
+const unsigned char *ParseIndexWord (const unsigned char *textHere,
+				     const unsigned char *textEnd,
+				     UCArray &word);
+const unsigned char *ParseIndexMGWord (const unsigned char *textHere,
+				       const unsigned char *textEnd,
+				       unsigned char *mgWord);
+const unsigned char *ParseNonindexWord (const unsigned char *textHere,
+					const unsigned char *textEnd);
+
+
+
+/* =========================================================================
+ * Macro: PARSE_WORD
+ * Description: 
+ *      Extract a word out for compressing text
+ * Input: 
+ *      s_in = string start in buffer
+ *      end = string end in buffer
+ * Output: 
+ *      Word = extracted word with length in 1st byte
+ *      s_in = ptr to next character in buffer yet to be processed
+ * ========================================================================= */
+#define PARSE_WORD(Word, s_in, end)                                \
+  do {                                                             \
+    register int charlength = 0;                                   \
+    register u_char *wptr = (Word)+1;                              \
+    register int length = 0;                                       \
+    register int numeric = 0;                                      \
+    unsigned short c;                                              \
+                                                                   \
+    charlength = parse_utf8_char((s_in),(end),&c);                 \
+                                                                   \
+    while (length+charlength <= MAXWORDLEN && charlength > 0 &&    \
+	   (is_unicode_letter(c) || (is_unicode_digit(c) &&        \
+				     ++numeric <= MAXNUMERIC))) {  \
+      while (charlength-- > 0) {                                   \
+        *wptr++ = *(s_in)++; length++;                             \
+      }                                                            \
+      charlength = parse_utf8_char((s_in),(end),&c);               \
+    }                                                              \
+    *(Word) = length;                                              \
+  }while(0)
+
+
+
+/* =========================================================================
+ * Macro: PARSE_NON_WORD
+ * Description: 
+ *      Extract a non-word out for storing compressed text
+ * Input: as above
+ * Output: as above
+ * ========================================================================= */
+#define PARSE_NON_WORD(Word, s_in, end)                            \
+  do {                                                             \
+    register int charlength = 0;                                   \
+    register u_char *wptr = (Word)+1;                              \
+    register int length = 0;                                       \
+    unsigned short c;                                              \
+                                                                   \
+    charlength = parse_utf8_char((s_in),(end),&c);                 \
+                                                                   \
+    while (length+charlength <= MAXWORDLEN && charlength > 0 &&    \
+	   !is_unicode_letdig(c)) {                                \
+      while (charlength-- > 0) {                                   \
+        *wptr++ = *(s_in)++; length++;                             \
+      }                                                            \
+      charlength = parse_utf8_char((s_in),(end),&c);               \
+    }                                                              \
+    *(Word) = length;                                              \
+  }while(0)
Index: /trunk/mgpp/COPYING
===================================================================
--- /trunk/mgpp/COPYING	(revision 3365)
+++ /trunk/mgpp/COPYING	(revision 3365)
@@ -0,0 +1,344 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+                          675 Mass Ave, Cambridge, MA 02139, USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) 19yy  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program; if not, write to the Free Software
+    Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) 19yy name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Library General
+Public License instead of this License.
Index: /trunk/mgpp/Makefile.in
===================================================================
--- /trunk/mgpp/Makefile.in	(revision 3365)
+++ /trunk/mgpp/Makefile.in	(revision 3365)
@@ -0,0 +1,111 @@
+# Main Makefile for GNU tar.
+# Copyright (C) 1994, 1995 Free Software Foundation, Inc.
+ 
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+ 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+ 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ 
+PACKAGE = @PACKAGE@
+VERSION = @VERSION@
+ 
+@SET_MAKE@
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+srcdir = @srcdir@
+ 
+
+CC = @CC@
+CPP = @CXX@
+CFLAGS = @CFLAGS@ -ansi
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS = @LIBS@  
+TOUCH = echo timestamp >
+ 
+bindir = $(exec_prefix)/bin
+infodir = $(prefix)/info
+libexecdir = $(exec_prefix)/libexec
+ 
+MDEFINES = 
+#CC='$(CC)' CFLAGS='$(CFLAGS)' LDFLAGS='$(LDFLAGS)' LIBS='$(LIBS)' \
+#prefix='$(prefix)' exec_prefix='$(exec_prefix)' \
+#bindir='$(bindir)' libexecdir='$(libexecdir)' \
+#infodir='$(infodir)' infodir='$(infodir)' libexecdir='$(libexecdir)'
+
+SUBDIRS = lib text java/org/greenstone/mgpp jni
+
+
+.SUFFIXES:
+
+all install uninstall execinstall execuninstall maninstall manuninstall:
+#config.h
+	for subdir in $(SUBDIRS); do \
+	  echo making $@ in $$subdir; \
+	  (cd $$subdir && $(MAKE) $(MDEFINES) $@) || exit 1; \
+	done
+
+# Have not written any texinfo files for the documentation yet
+#info dvi:
+#	cd doc && $(MAKE) $@
+
+mostlyclean: mostlyclean-recursive mostlyclean-local
+
+clean: clean-recursive clean-local
+
+distclean: distclean-recursive distclean-local
+	rm config.status
+
+maintainer-clean: maintainer-clean-recursive maintainer-clean-local
+	rm config.status
+
+mostlyclean-recursive clean-recursive distclean-recursive \
+maintainer-clean-recursive:
+	for subdir in $(SUBDIRS); do \
+	  target=`echo $@ | sed 's/-recursive//'`; \
+	  echo making $$target in $$subdir; \
+	  (cd $$subdir && $(MAKE) $$target) || exit 1; \
+	done
+
+mostlyclean-local:
+
+clean-local: mostlyclean-local
+
+distclean-local: clean-local
+	rm -f Makefile config.cache config.h config.log stamp-h
+
+maintainer-clean-local: distclean-local
+	@echo "This command is intended only for maintainers to use;"
+	@echo "rebuilding the deleted files may require special tools."
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+distdir = $(PACKAGE)-$(VERSION)
+
+distname = $(distdir)
+
+DISTFILES = \
+INSTALL.mg INSTALL README README.port COPYING MODIFICATIONS about_mg.html \
+Makefile.in acconfig.h aclocal.m4 configure.in install-sh mkinstalldirs \
+config.guess config.sub \
+config.h.in configure stamp-h.in
+
+#-----------------------------------------------------------------
+
+# Tell versions [3.59,3.63) of GNU make not to export all variables.
+# Otherwise a system limit (for SysV at least) may be exceeded.
+.NOEXPORT:
+
Index: /trunk/mgpp/acconfig.h
===================================================================
--- /trunk/mgpp/acconfig.h	(revision 3365)
+++ /trunk/mgpp/acconfig.h	(revision 3365)
@@ -0,0 +1,41 @@
+/* Special definitions for mg, processed by autoheader.
+   Created by [TS:Aug/95] 
+   by grepping for AC_DEFINE on configure.in and using the appropriate defs 
+   of acconfig.h used by GNU tar.
+*/
+
+/* Define to the name of the distribution.  */
+#undef PACKAGE
+
+/* Define to the version of the distribution.  */
+#undef VERSION
+
+/* Define if `union wait' is the type of the first arg to wait functions.  */
+#undef HAVE_UNION_WAIT
+
+/* Define to 1 if you have the valloc function.  */
+#undef HAVE_VALLOC
+
+/* Define to 1 if ANSI function prototypes are usable.  */
+#undef PROTOTYPES
+ 
+/* Define to 1 for better use of the debugging malloc library.  See
+   site ftp.antaire.com in antaire/src, file dmalloc/dmalloc.tar.gz.  */
+#undef WITH_DMALLOC
+ 
+/* Define to 1 if GNU regex should be used instead of GNU rx.  */
+#undef WITH_REGEX
+
+/* Define to 1 if GNU readline should be used. */
+#undef WITH_GNU_READLINE
+
+/* Some braindead header files do not have the requ'd function declarations */
+#undef HAVE_FGETC_DECL
+#undef HAVE_FREAD_DECL
+
+/* See if have this field in the prstatus_t structure */
+/* It stores the size of the process heap */
+#undef HAVE_PR_BRKSIZE
+
+/* Define to 1 to override the automatic selection of "long long" */
+#undef DISABLE_LONG_LONG
Index: /trunk/mgpp/aclocal.m4
===================================================================
--- /trunk/mgpp/aclocal.m4	(revision 3365)
+++ /trunk/mgpp/aclocal.m4	(revision 3365)
@@ -0,0 +1,225 @@
+# Local additions to Autoconf macros.
+# Copyright (C) 1992, 1994, 1995 Free Software Foundation, Inc.
+# François Pinard <pinard@iro.umontreal.ca>, 1992.
+
+# [TS:Aug/95] - chopped out the NLS stuff which I don't support
+
+## ------------------------------- ##
+## Check for function prototypes.  ##
+## ------------------------------- ##
+
+AC_DEFUN(fp_C_PROTOTYPES,
+[AC_REQUIRE([fp_PROG_CC_STDC])
+AC_MSG_CHECKING([for function prototypes])
+if test "$ac_cv_prog_cc_stdc" != no; then
+  AC_MSG_RESULT(yes)
+  AC_DEFINE(PROTOTYPES)
+  U= ANSI2KNR=
+else
+  AC_MSG_RESULT(no)
+  U=_ ANSI2KNR=ansi2knr
+fi
+AC_SUBST(U)dnl
+AC_SUBST(ANSI2KNR)dnl
+])
+
+## ----------------------------------------- ##
+## ANSIfy the C compiler whenever possible.  ##
+## ----------------------------------------- ##
+
+# @defmac AC_PROG_CC_STDC
+# @maindex PROG_CC_STDC
+# @ovindex CC
+# If the C compiler in not in ANSI C mode by default, try to add an option
+# to output variable @code{CC} to make it so.  This macro tries various
+# options that select ANSI C on some system or another.  It considers the
+# compiler to be in ANSI C mode if it defines @code{__STDC__} to 1 and
+# handles function prototypes correctly.
+#
+# If you use this macro, you should check after calling it whether the C
+# compiler has been set to accept ANSI C; if not, the shell variable
+# @code{ac_cv_prog_cc_stdc} is set to @samp{no}.  If you wrote your source
+# code in ANSI C, you can make an un-ANSIfied copy of it by using the
+# program @code{ansi2knr}, which comes with Ghostscript.
+# @end defmac
+
+AC_DEFUN(fp_PROG_CC_STDC,
+[AC_MSG_CHECKING(for ${CC-cc} option to accept ANSI C)
+AC_CACHE_VAL(ac_cv_prog_cc_stdc,
+[ac_cv_prog_cc_stdc=no
+ac_save_CFLAGS="$CFLAGS"
+# Don't try gcc -ansi; that turns off useful extensions and
+# breaks some systems' header files.
+# AIX			-qlanglvl=ansi
+# Ultrix and OSF/1	-std1
+# HP-UX			-Aa -D_HPUX_SOURCE
+# SVR4			-Xc
+for ac_arg in "" -qlanglvl=ansi -std1 "-Aa -D_HPUX_SOURCE" -Xc
+do
+  CFLAGS="$ac_save_CFLAGS $ac_arg"
+  AC_TRY_COMPILE(
+[#if !defined(__STDC__) || __STDC__ != 1
+choke me
+#endif	
+], [int test (int i, double x);
+struct s1 {int (*f) (int a);};
+struct s2 {int (*f) (double a);};],
+[ac_cv_prog_cc_stdc="$ac_arg"; break])
+done
+CFLAGS="$ac_save_CFLAGS"
+])
+AC_MSG_RESULT($ac_cv_prog_cc_stdc)
+case "x$ac_cv_prog_cc_stdc" in
+  x|xno) ;;
+  *) CC="$CC $ac_cv_prog_cc_stdc" ;;
+esac
+])
+
+## --------------------------------------------------------- ##
+## Use AC_PROG_INSTALL, supplementing it with INSTALL_SCRIPT ##
+## substitution.					     ##
+## --------------------------------------------------------- ##
+
+AC_DEFUN(fp_PROG_INSTALL,
+[AC_PROG_INSTALL
+test -z "$INSTALL_SCRIPT" && INSTALL_SCRIPT='${INSTALL} -m 755'
+AC_SUBST(INSTALL_SCRIPT)dnl
+])
+
+## ----------------------------------- ##
+## Check if --with-dmalloc was given.  ##
+## ----------------------------------- ##
+
+# I just checked, and GNU rx seems to work fine with a slightly
+# modified GNU m4.  So, I put out the test below in my aclocal.m4,
+# and will try to use it in my things.  The idea is to distribute
+# rx.[hc] and regex.[hc] together, for a while.  The WITH_REGEX symbol
+# (which should also be documented in acconfig.h) is used to decide
+# which of regex.h or rx.h should be included in the application.
+#
+# If `./configure --with-regex' is given, the package will use
+# the older regex.  Else, a check is made to see if rx is already
+# installed, as with newer Linux'es.  If not found, the package will
+# use the rx from the distribution.  If found, the package will use
+# the system's rx which, on Linux at least, will result in a smaller
+# executable file.
+
+AC_DEFUN(fp_WITH_DMALLOC,
+[AC_MSG_CHECKING(if malloc debugging is wanted)
+AC_ARG_WITH(dmalloc,
+[  --with-dmalloc          use dmalloc, as in
+                          ftp://ftp.letters.com/src/dmalloc/dmalloc.tar.gz],
+[if test "$withval" = yes; then
+  AC_MSG_RESULT(yes)
+  AC_DEFINE(WITH_DMALLOC)
+  LIBS="$LIBS -ldmalloc"
+  LDFLAGS="$LDFLAGS -g"
+else
+  AC_MSG_RESULT(no)
+fi], [AC_MSG_RESULT(no)])
+])
+
+## --------------------------------- ##
+## Check if --with-regex was given.  ##
+## --------------------------------- ##
+
+AC_DEFUN(fp_WITH_REGEX,
+[AC_MSG_CHECKING(which of rx or regex is wanted)
+AC_ARG_WITH(regex,
+[  --with-regex            use older regex in lieu of GNU rx for matching],
+[if test "$withval" = yes; then
+  ac_with_regex=1
+  AC_MSG_RESULT(regex)
+  AC_DEFINE(WITH_REGEX)
+  LIBOBJS="$LIBOBJS regex.o"
+fi])
+if test -z "$ac_with_regex"; then
+  AC_MSG_RESULT(rx)
+  AC_CHECK_FUNC(re_rx_search, , 
+   [AC_CHECK_SIZEOF(unsigned char *, unsigned char *)  
+    if test "$ac_cv_sizeof_unsigned_char_p" = 8
+    then
+      AC_MSG_WARN(I'm forcing you to use regex because I can't 
+        find a local rx library and the one included with this 
+        distribution doesn't work on 64-bit machines like yours)
+      [LIBOBJS="$LIBOBJS regex.o"]
+    else
+      [LIBOBJS="$LIBOBJS rx.o"]
+    fi]
+  )
+fi
+AC_SUBST(LIBOBJS)dnl
+])
+
+
+## --------------------------------------- ##
+## Check if --with-gnu-readline was given. ##
+## --------------------------------------- ##
+
+AC_DEFUN(fp_WITH_GNU_READLINE,
+[AC_MSG_CHECKING(whether GNU readline requested)
+  AC_ARG_WITH(gnu_readline,
+    [  --with-gnu-readline     compile with GNU readline support],
+    [if test "$withval" = yes; then
+      AC_MSG_RESULT(yes)
+      ac_with_gnu_readline=1
+    else
+      AC_MSG_RESULT(no)
+    fi],
+    AC_MSG_RESULT(no))
+
+  if test -n "$ac_with_gnu_readline"; then
+    AC_CHECK_HEADER(readline/readline.h, ac_mg_readline_header_found=1,
+	AC_MSG_WARN(Can't find GNU readline headers; configuring without \
+GNU readline support))
+    if test -n "$ac_mg_readline_header_found"; then
+      # first check whether we can find the readline library itself
+      AC_CHECK_LIB(readline, main, 
+        [ac_mg_readline_lib_found=1
+         AC_DEFINE(WITH_GNU_READLINE)
+         LIBS="$LIBS -lreadline"],
+        AC_MSG_WARN(Can't find GNU readline library; configuring without \
+GNU readline support))
+      if test -n "$ac_mg_readline_lib_found"; then
+        # on some systems, readline needs curses.  It is difficult to 
+        #  determine whether this is necessary on the current system,
+        #  since other undefined symbols can be turned up by an
+        #  AC_CHECK_LIB(readline, readline) test that are actually defined
+	#  by mg itself (e.g. xmalloc, xrealloc).  So, if we find libcurses,
+	#  we just bung it on and hope for the best.
+	AC_CHECK_LIB(curses, main,
+	  LIBS="$LIBS -lcurses")
+      fi
+    fi
+  fi
+])	
+
+
+
+#AC_DEFUN(fp_WITH_GNU_READLINE,
+#[AC_MSG_CHECKING(whether GNU readline requested)
+#  AC_ARG_WITH(gnu-readline,
+#    [  --with-gnu-readline   compile with GNU readline support],
+#    [if test "$withval" = yes; then
+#      AC_MSG_RESULT(yes)
+#      ac_with_gnu_readline=1
+#    else
+#      AC_MSG_RESULT(no)
+#    fi],
+#    AC_MSG_RESULT(no))
+#  if test -n "$ac_with_gnu_readline"; then
+#    AC_CHECK_LIB(readline, main, 
+#      AC_CHECK_HEADER(readline/readline.h,
+#	[AC_DEFINE(WITH_GNU_READLINE)
+## curses required on some systems (DEC OSF/1), but not on others (GNU/Linux,
+##  Solaris 2.6).  Forcing its inclusion is unsophisticated; if it causes 
+##  any actual problems on any UNIX platforms, let the maintainer know.
+#	LIBS="$LIBS -lreadline -lcurses"],
+#        AC_MSG_WARN(Can't find GNU readline headers: configuring \
+#without GNU readline support)
+#      ),
+#      AC_MSG_WARN(Can't find GNU readline library: configuring \
+#without GNU readline support)
+#    )
+#  fi 
+#])
Index: /trunk/mgpp/config.guess
===================================================================
--- /trunk/mgpp/config.guess	(revision 3365)
+++ /trunk/mgpp/config.guess	(revision 3365)
@@ -0,0 +1,470 @@
+#!/bin/sh
+# Attempt to guess a canonical system name.
+#   Copyright (C) 1992, 1993, 1994 Free Software Foundation, Inc.
+#
+# This file is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+# As a special exception to the GNU General Public License, if you
+# distribute this file as part of a program that contains a
+# configuration script generated by Autoconf, you may include it under
+# the same distribution terms that you use for the rest of that program.
+
+# Written by Per Bothner <bothner@cygnus.com>.
+# The master version of this file is at the FSF in /home/gd/gnu/lib.
+#
+# This script attempts to guess a canonical system name similar to
+# config.sub.  If it succeeds, it prints the system name on stdout, and
+# exits with 0.  Otherwise, it exits with 1.
+#
+# The plan is that this can be called by configure scripts if you
+# don't specify an explicit system type (host/target name).
+#
+# Only a few systems have been added to this list; please add others
+# (but try to keep the structure clean).
+#
+
+# This is needed to find uname on a Pyramid OSx when run in the BSD universe.
+# (ghazi@noc.rutgers.edu 8/24/94.)
+if (test -f /.attbin/uname) >/dev/null 2>&1 ; then
+	PATH=$PATH:/.attbin ; export PATH
+fi
+
+UNAME_MACHINE=`(uname -m) 2>/dev/null` || UNAME_MACHINE=unknown
+UNAME_RELEASE=`(uname -r) 2>/dev/null` || UNAME_RELEASE=unknown
+UNAME_SYSTEM=`(uname -s) 2>/dev/null` || UNAME_SYSTEM=unknown
+UNAME_VERSION=`(uname -v) 2>/dev/null` || UNAME_VERSION=unknown
+
+trap 'rm -f dummy.c dummy.o dummy; exit 1' 1 2 15
+
+# Note: order is significant - the case branches are not exclusive.
+
+case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in
+    alpha:OSF1:V*:*)
+	# After 1.2, OSF1 uses "V1.3" for uname -r.
+	echo alpha-dec-osf`echo ${UNAME_RELEASE} | sed -e 's/^V//'`
+	exit 0 ;;
+    alpha:OSF1:*:*)
+	# 1.2 uses "1.2" for uname -r.
+	echo alpha-dec-osf${UNAME_RELEASE}
+        exit 0 ;;
+    arm:RISC*:1.[012]*:*|arm:riscix:1.[012]*:*)
+	echo arm-acorn-riscix${UNAME_RELEASE}
+	exit 0;;
+    Pyramid*:OSx*:*:*)
+	if test "`(/bin/universe) 2>/dev/null`" = att ; then
+		echo pyramid-pyramid-sysv3
+	else
+		echo pyramid-pyramid-bsd
+	fi
+	exit 0 ;;
+    sun4*:SunOS:5.*:*)
+	echo sparc-sun-solaris2`echo ${UNAME_RELEASE}|sed -e 's/[^.]*//'`
+	exit 0 ;;
+    sun4*:SunOS:6*:*)
+	# According to config.sub, this is the proper way to canonicalize
+	# SunOS6.  Hard to guess exactly what SunOS6 will be like, but
+	# it's likely to be more like Solaris than SunOS4.
+	echo sparc-sun-solaris3`echo ${UNAME_RELEASE}|sed -e 's/[^.]*//'`
+	exit 0 ;;
+    sun4*:SunOS:*:*)
+	# Japanese Language versions have a version number like `4.1.3-JL'.
+	echo sparc-sun-sunos`echo ${UNAME_RELEASE}|sed -e 's/-/_/'`
+	exit 0 ;;
+    sun3*:SunOS:*:*)
+	echo m68k-sun-sunos${UNAME_RELEASE}
+	exit 0 ;;
+    RISC*:ULTRIX:*:*)
+	echo mips-dec-ultrix${UNAME_RELEASE}
+	exit 0 ;;
+    VAX*:ULTRIX*:*:*)
+	echo vax-dec-ultrix${UNAME_RELEASE}
+	exit 0 ;;
+    mips:*:5*:RISCos)
+	echo mips-mips-riscos${UNAME_RELEASE}
+	exit 0 ;;
+    m88k:CX/UX:7*:*)
+	echo m88k-harris-cxux7
+	exit 0 ;;
+    m88k:*:4*:R4*)
+	echo m88k-motorola-sysv4
+	exit 0 ;;
+    m88k:*:3*:R3*)
+	echo m88k-motorola-sysv3
+	exit 0 ;;
+    AViiON:dgux:*:*)
+	if [ ${TARGET_BINARY_INTERFACE}x = m88kdguxelfx \
+	     -o ${TARGET_BINARY_INTERFACE}x = x ] ; then
+		echo m88k-dg-dgux${UNAME_RELEASE}
+	else
+		echo m88k-dg-dguxbcs${UNAME_RELEASE}
+	fi
+ 	exit 0 ;;
+    M88*:DolphinOS:*:*)	# DolphinOS (SVR3)
+	echo m88k-dolphin-sysv3
+	exit 0 ;;
+    M88*:*:R3*:*)
+	# Delta 88k system running SVR3
+	echo m88k-motorola-sysv3
+	exit 0 ;;
+    XD88*:*:*:*) # Tektronix XD88 system running UTekV (SVR3)
+	echo m88k-tektronix-sysv3
+	exit 0 ;;
+    Tek43[0-9][0-9]:UTek:*:*) # Tektronix 4300 system running UTek (BSD)
+	echo m68k-tektronix-bsd
+	exit 0 ;;
+    *:IRIX:*:*)
+	echo mips-sgi-irix${UNAME_RELEASE}
+	exit 0 ;;
+    i[34]86:AIX:*:*)
+	echo i386-ibm-aix
+	exit 0 ;;
+    *:AIX:2:3)
+	if grep bos325 /usr/include/stdio.h >/dev/null 2>&1; then
+		sed 's/^		//' << EOF >dummy.c
+		#include <sys/systemcfg.h>
+
+		main()
+			{
+			if (!__power_pc())
+				exit(1);
+			puts("powerpc-ibm-aix3.2.5");
+			exit(0);
+			}
+EOF
+		${CC-cc} dummy.c -o dummy && ./dummy && rm dummy.c dummy && exit 0
+		rm -f dummy.c dummy
+		echo rs6000-ibm-aix3.2.5
+	elif grep bos324 /usr/include/stdio.h >/dev/null 2>&1; then
+		echo rs6000-ibm-aix3.2.4
+	else
+		echo rs6000-ibm-aix3.2
+	fi
+	exit 0 ;;
+    *:AIX:*:4)
+	if /usr/sbin/lsattr -EHl proc0 | grep POWER >/dev/null 2>&1; then
+		IBM_ARCH=rs6000
+	else
+		IBM_ARCH=powerpc
+	fi
+	if grep bos410 /usr/include/stdio.h >/dev/null 2>&1; then
+		IBM_REV=4.1
+	elif grep bos411 /usr/include/stdio.h >/dev/null 2>&1; then
+		IBM_REV=4.1.1
+	else
+		IBM_REV=4.${UNAME_RELEASE}
+	fi
+	echo ${IBM_ARCH}-ibm-aix${IBM_REV}
+	exit 0 ;;
+    *:AIX:*:*)
+	echo rs6000-ibm-aix
+	exit 0 ;;
+    *:BOSX:*:*)
+	echo rs6000-bull-bosx
+	exit 0 ;;
+    DPX/2?00:B.O.S.:*:*)
+	echo m68k-bull-sysv3
+	exit 0 ;;
+    9000/[34]??:4.3bsd:1.*:*)
+	echo m68k-hp-bsd
+	exit 0 ;;
+    hp300:4.4BSD:*:* | 9000/[34]??:4.3bsd:2.*:*)
+	echo m68k-hp-bsd4.4
+	exit 0 ;;
+    9000/[3478]??:HP-UX:*:*)
+	case "${UNAME_MACHINE}" in
+	    9000/31? )            HP_ARCH=m68000 ;;
+	    9000/[34]?? )         HP_ARCH=m68k ;;
+	    9000/7?? | 9000/8?7 ) HP_ARCH=hppa1.1 ;;
+	    9000/8?? )            HP_ARCH=hppa1.0 ;;
+	esac
+	HPUX_REV=`echo ${UNAME_RELEASE}|sed -e 's/[^.]*.[0B]*//'`
+	echo ${HP_ARCH}-hp-hpux${HPUX_REV}
+	exit 0 ;;
+    3050*:HI-UX:*:*)
+	sed 's/^	//' << EOF >dummy.c
+	#include <unistd.h>
+	int
+	main ()
+	{
+	  long cpu = sysconf (_SC_CPU_VERSION);
+	  /* The order matters, because CPU_IS_HP_MC68K erroneously returns
+	     true for CPU_PA_RISC1_0.  CPU_IS_PA_RISC returns correct
+	     results, however.  */
+	  if (CPU_IS_PA_RISC (cpu))
+	    {
+	      switch (cpu)
+		{
+		  case CPU_PA_RISC1_0: puts ("hppa1.0-hitachi-hiuxwe2"); break;
+		  case CPU_PA_RISC1_1: puts ("hppa1.1-hitachi-hiuxwe2"); break;
+		  case CPU_PA_RISC2_0: puts ("hppa2.0-hitachi-hiuxwe2"); break;
+		  default: puts ("hppa-hitachi-hiuxwe2"); break;
+		}
+	    }
+	  else if (CPU_IS_HP_MC68K (cpu))
+	    puts ("m68k-hitachi-hiuxwe2");
+	  else puts ("unknown-hitachi-hiuxwe2");
+	  exit (0);
+	}
+EOF
+	${CC-cc} dummy.c -o dummy && ./dummy && rm dummy.c dummy && exit 0
+	rm -f dummy.c dummy
+	echo unknown-hitachi-hiuxwe2
+	exit 0 ;;
+    9000/7??:4.3bsd:*:* | 9000/8?7:4.3bsd:*:* )
+	echo hppa1.1-hp-bsd
+	exit 0 ;;
+    9000/8??:4.3bsd:*:*)
+	echo hppa1.0-hp-bsd
+	exit 0 ;;
+    hp7??:OSF1:*:* | hp8?7:OSF1:*:* )
+	echo hppa1.1-hp-osf
+	exit 0 ;;
+    hp8??:OSF1:*:*)
+	echo hppa1.0-hp-osf
+	exit 0 ;;
+    C1*:ConvexOS:*:* | convex:ConvexOS:C1*:*)
+	echo c1-convex-bsd
+        exit 0 ;;
+    C2*:ConvexOS:*:* | convex:ConvexOS:C2*:*)
+	if getsysinfo -f scalar_acc
+	then echo c32-convex-bsd
+	else echo c2-convex-bsd
+	fi
+        exit 0 ;;
+    C34*:ConvexOS:*:* | convex:ConvexOS:C34*:*)
+	echo c34-convex-bsd
+        exit 0 ;;
+    C38*:ConvexOS:*:* | convex:ConvexOS:C38*:*)
+	echo c38-convex-bsd
+        exit 0 ;;
+    C4*:ConvexOS:*:* | convex:ConvexOS:C4*:*)
+	echo c4-convex-bsd
+        exit 0 ;;
+    CRAY*X-MP:UNICOS:*:*)
+	echo xmp-cray-unicos
+        exit 0 ;;
+    CRAY*Y-MP:UNICOS:*:*)
+	echo ymp-cray-unicos
+        exit 0 ;;
+    CRAY-2:UNICOS:*:*)
+	echo cray2-cray-unicos
+        exit 0 ;;
+    hp3[0-9][05]:NetBSD:*:*)
+	echo m68k-hp-netbsd${UNAME_RELEASE}
+	exit 0 ;;
+    i[34]86:BSD/386:*:*)
+	echo ${UNAME_MACHINE}-unknown-bsdi${UNAME_RELEASE}
+	exit 0 ;;
+    *:FreeBSD:*:*)
+	echo ${UNAME_MACHINE}-unknown-freebsd`echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`
+	exit 0 ;;
+    *:NetBSD:*:*)
+	echo ${UNAME_MACHINE}-unknown-netbsd`echo ${UNAME_RELEASE}|sed -e 's/[-_].*/\./'`
+	exit 0 ;;
+    *:GNU:*:*)
+	echo `echo ${UNAME_MACHINE}|sed -e 's,/.*$,,'`-unknown-gnu`echo ${UNAME_RELEASE}|sed -e 's,/.*$,,'`
+	exit 0 ;;
+    *:Linux:*:*)
+	echo ${UNAME_MACHINE}-unknown-linux
+	exit 0 ;;
+# ptx 4.0 does uname -s correctly, with DYNIX/ptx in there.  earlier versions
+# are messed up and put the nodename in both sysname and nodename.
+    i[34]86:DYNIX/ptx:4*:*)
+	echo i386-sequent-sysv4
+	exit 0 ;;
+    i[34]86:*:4.*:* | i[34]86:SYSTEM_V:4.*:*)
+	if grep Novell /usr/include/link.h >/dev/null 2>/dev/null; then
+		echo ${UNAME_MACHINE}-univel-sysv${UNAME_RELEASE}
+	else
+		echo ${UNAME_MACHINE}-unknown-sysv${UNAME_RELEASE}
+	fi
+	exit 0 ;;
+    i[34]86:*:3.2:*)
+	if /bin/uname -X 2>/dev/null >/dev/null ; then
+		UNAME_REL=`(/bin/uname -X|egrep Release|sed -e 's/.*= //')`
+		(/bin/uname -X|egrep i80486 >/dev/null) && UNAME_MACHINE=i486
+		echo ${UNAME_MACHINE}-unknown-sco$UNAME_REL
+	elif test -f /usr/options/cb.name; then
+		UNAME_REL=`sed -n 's/.*Version //p' </usr/options/cb.name`
+		echo ${UNAME_MACHINE}-unknown-isc$UNAME_REL
+	else
+		echo ${UNAME_MACHINE}-unknown-sysv32
+	fi
+	exit 0 ;;
+    Intel:Mach:3*:*)
+	echo i386-unknown-mach3
+	exit 0 ;;
+    i860:*:4.*:*) # i860-SVR4
+	if grep Stardent /usr/include/sys/uadmin.h >/dev/null 2>&1 ; then
+	  echo i860-stardent-sysv${UNAME_RELEASE} # Stardent Vistra i860-SVR4
+	else # Add other i860-SVR4 vendors below as they are discovered.
+	  echo i860-unknown-sysv${UNAME_RELEASE}  # Unknown i860-SVR4
+	fi
+	exit 0 ;;
+    mini*:CTIX:SYS*5:*)
+	# "miniframe"
+	echo m68010-convergent-sysv
+	exit 0 ;;
+    M680[234]0:*:R3V[567]*:*)
+	test -r /sysV68 && echo 'm68k-motorola-sysv' && exit 0 ;;
+    3[34]??:*:4.0:3.0 | 3[34]??,*:*:4.0:3.0)
+        uname -p 2>/dev/null | grep 86 >/dev/null \
+          && echo i486-ncr-sysv4.3 && exit 0 ;;
+    3[34]??:*:4.0:* | 3[34]??,*:*:4.0:*)
+        uname -p 2>/dev/null | grep 86 >/dev/null \
+          && echo i486-ncr-sysv4 && exit 0 ;;
+    m680[234]0:LynxOS:2.2*:*)
+	echo m68k-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    mc68030:UNIX_System_V:4.*:*)
+	echo m68k-atari-sysv4
+	exit 0 ;;
+    i[34]86:LynxOS:2.2*:*)
+	echo i386-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    TSUNAMI:LynxOS:2.2*:*)
+	echo sparc-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    rs6000:LynxOS:2.2*:*)
+	echo rs6000-lynx-lynxos${UNAME_RELEASE}
+	exit 0 ;;
+    RM*:SINIX-*:*:*)
+	echo mips-sni-sysv4
+	exit 0 ;;
+    *:SINIX-*:*:*)
+	if uname -p 2>/dev/null >/dev/null ; then
+		UNAME_MACHINE=`(uname -p) 2>/dev/null`
+		echo ${UNAME_MACHINE}-sni-sysv4
+	else
+		echo ns32k-sni-sysv
+	fi
+	exit 0 ;;
+esac
+
+#echo '(No uname command or uname output not recognized.)' 1>&2
+#echo "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" 1>&2
+
+cat >dummy.c <<EOF
+main ()
+{
+#if defined (sony)
+#if defined (MIPSEB)
+  /* BFD wants "bsd" instead of "newsos".  Perhaps BFD should be changed,
+     I don't know....  */
+  printf ("mips-sony-bsd\n"); exit (0);
+#else
+  printf ("m68k-sony-newsos\n"); exit (0);
+#endif
+#endif
+
+#if defined (__arm) && defined (__acorn) && defined (__unix)
+  printf ("arm-acorn-riscix"); exit (0);
+#endif
+
+#if defined (hp300) && !defined (hpux)
+  printf ("m68k-hp-bsd\n"); exit (0);
+#endif
+
+#if defined (NeXT)
+#if !defined (__ARCHITECTURE__)
+#define __ARCHITECTURE__ "m68k"
+#endif
+  int version;
+  version=`(hostinfo | sed -n 's/.*NeXT Mach \([0-9]*\).*/\1/p') 2>/dev/null`;
+  printf ("%s-next-nextstep%s\n", __ARCHITECTURE__,  version==2 ? "2" : "3");
+  exit (0);
+#endif
+
+#if defined (MULTIMAX) || defined (n16)
+#if defined (UMAXV)
+  printf ("ns32k-encore-sysv\n"); exit (0);
+#else
+#if defined (CMU)
+  printf ("ns32k-encore-mach\n"); exit (0);
+#else
+  printf ("ns32k-encore-bsd\n"); exit (0);
+#endif
+#endif
+#endif
+
+#if defined (__386BSD__)
+  printf ("i386-unknown-bsd\n"); exit (0);
+#endif
+
+#if defined (sequent)
+#if defined (i386)
+  printf ("i386-sequent-dynix\n"); exit (0);
+#endif
+#if defined (ns32000)
+  printf ("ns32k-sequent-dynix\n"); exit (0);
+#endif
+#endif
+
+#if defined (_SEQUENT_)
+  printf ("i386-sequent-ptx\n"); exit (0);
+#endif
+
+#if defined (vax)
+#if !defined (ultrix)
+  printf ("vax-dec-bsd\n"); exit (0);
+#else
+  printf ("vax-dec-ultrix\n"); exit (0);
+#endif
+#endif
+
+#if defined (alliant) && defined (i860)
+  printf ("i860-alliant-bsd\n"); exit (0);
+#endif
+
+  exit (1);
+}
+EOF
+
+${CC-cc} dummy.c -o dummy 2>/dev/null && ./dummy && rm dummy.c dummy && exit 0
+rm -f dummy.c dummy
+
+# Apollos put the system type in the environment.
+
+test -d /usr/apollo && { echo ${ISP}-apollo-${SYSTYPE}; exit 0; }
+
+# Convex versions that predate uname can use getsysinfo(1)
+
+if [ -x /usr/convex/getsysinfo ]
+then
+    case `getsysinfo -f cpu_type` in
+    c1*)
+	echo c1-convex-bsd
+	exit 0 ;;
+    c2*)
+	if getsysinfo -f scalar_acc
+	then echo c32-convex-bsd
+	else echo c2-convex-bsd
+	fi
+	exit 0 ;;
+    c34*)
+	echo c34-convex-bsd
+	exit 0 ;;
+    c38*)
+	echo c38-convex-bsd
+	exit 0 ;;
+    c4*)
+	echo c4-convex-bsd
+	exit 0 ;;
+    esac
+fi
+
+#echo '(Unable to guess system type)' 1>&2
+
+exit 1
Index: /trunk/mgpp/config.h.in
===================================================================
--- /trunk/mgpp/config.h.in	(revision 3365)
+++ /trunk/mgpp/config.h.in	(revision 3365)
@@ -0,0 +1,183 @@
+/* config.h.in.  Generated automatically from configure.in by autoheader.  */
+
+/* Define if on AIX 3.
+   System headers sometimes define this.
+   We just want to avoid a redefinition error message.  */
+#ifndef _ALL_SOURCE
+#undef _ALL_SOURCE
+#endif
+
+/* Define if using alloca.c.  */
+#undef C_ALLOCA
+
+/* Define to empty if the keyword does not work.  */
+#undef const
+
+/* Define to one of _getb67, GETB67, getb67 for Cray-2 and Cray-YMP systems.
+   This function is required for alloca.c support on those systems.  */
+#undef CRAY_STACKSEG_END
+
+/* Define if you have alloca, as a function or macro.  */
+#undef HAVE_ALLOCA
+
+/* Define if you have <alloca.h> and it should be used (not on Ultrix).  */
+#undef HAVE_ALLOCA_H
+
+/* Define if you don't have vprintf but do have _doprnt.  */
+#undef HAVE_DOPRNT
+
+/* Define if you have the vprintf function.  */
+#undef HAVE_VPRINTF
+
+/* Define if on MINIX.  */
+#undef _MINIX
+
+/* Define to `long' if <sys/types.h> doesn't define.  */
+#undef off_t
+
+/* Define if the system does not provide POSIX.1 features except
+   with this defined.  */
+#undef _POSIX_1_SOURCE
+
+/* Define if you need to in order for stat and other things to work.  */
+#undef _POSIX_SOURCE
+
+/* Define as the return type of signal handlers (int or void).  */
+#undef RETSIGTYPE
+
+/* Define to `unsigned' if <sys/types.h> doesn't define.  */
+#undef size_t
+
+/* If using the C implementation of alloca, define if you know the
+   direction of stack growth for your system; otherwise it will be
+   automatically deduced at run-time.
+ STACK_DIRECTION > 0 => grows toward higher addresses
+ STACK_DIRECTION < 0 => grows toward lower addresses
+ STACK_DIRECTION = 0 => direction of growth unknown
+ */
+#undef STACK_DIRECTION
+
+/* Define if you have the ANSI C header files.  */
+#undef STDC_HEADERS
+
+/* Define if you can safely include both <sys/time.h> and <time.h>.  */
+#undef TIME_WITH_SYS_TIME
+
+/* Define if your <sys/time.h> declares struct tm.  */
+#undef TM_IN_SYS_TIME
+
+/* Define to the name of the distribution.  */
+#undef PACKAGE
+
+/* Define to the version of the distribution.  */
+#undef VERSION
+
+/* Define to 1 if you have the valloc function.  */
+#undef HAVE_VALLOC
+
+/* Define to 1 if ANSI function prototypes are usable.  */
+#undef PROTOTYPES
+
+ 
+/* Define to 1 for better use of the debugging malloc library.  See
+   site ftp.antaire.com in antaire/src, file dmalloc/dmalloc.tar.gz.  */
+#undef WITH_DMALLOC
+
+ 
+/* Define to 1 if GNU regex should be used instead of GNU rx.  */
+#undef WITH_REGEX
+
+/* Define to 1 if GNU readline should be used. */
+#undef WITH_GNU_READLINE
+
+/* Some braindead header files do not have the requ'd function declarations */
+#undef HAVE_FGETC_DECL
+#undef HAVE_FREAD_DECL
+
+/* See if have this field in the prstatus_t structure */
+/* It stores the size of the process heap */
+#undef HAVE_PR_BRKSIZE
+
+/* Define to 1 to override the automatic selection of "long long" */
+#undef DISABLE_LONG_LONG
+
+/* The number of bytes in a unsigned char *.  */
+#undef SIZEOF_UNSIGNED_CHAR_P
+
+/* Define if you have the ftime function.  */
+#undef HAVE_FTIME
+
+/* Define if you have the ftruncate function.  */
+#undef HAVE_FTRUNCATE
+
+/* Define if you have the getpagesize function.  */
+#undef HAVE_GETPAGESIZE
+
+/* Define if you have the getrusage function.  */
+#undef HAVE_GETRUSAGE
+
+/* Define if you have the mallinfo function.  */
+#undef HAVE_MALLINFO
+
+/* Define if you have the select function.  */
+#undef HAVE_SELECT
+
+/* Define if you have the setbuffer function.  */
+#undef HAVE_SETBUFFER
+
+/* Define if you have the strcasecmp function.  */
+#undef HAVE_STRCASECMP
+
+/* Define if you have the strerror function.  */
+#undef HAVE_STRERROR
+
+/* Define if you have the strftime function.  */
+#undef HAVE_STRFTIME
+
+/* Define if you have the strstr function.  */
+#undef HAVE_STRSTR
+
+/* Define if you have the strtol function.  */
+#undef HAVE_STRTOL
+
+/* Define if you have the times function.  */
+#undef HAVE_TIMES
+
+/* Define if you have the valloc function.  */
+#undef HAVE_VALLOC
+
+/* Define if you have the <dirent.h> header file.  */
+#undef HAVE_DIRENT_H
+
+/* Define if you have the <fcntl.h> header file.  */
+#undef HAVE_FCNTL_H
+
+/* Define if you have the <limits.h> header file.  */
+#undef HAVE_LIMITS_H
+
+/* Define if you have the <memory.h> header file.  */
+#undef HAVE_MEMORY_H
+
+/* Define if you have the <ndir.h> header file.  */
+#undef HAVE_NDIR_H
+
+/* Define if you have the <string.h> header file.  */
+#undef HAVE_STRING_H
+
+/* Define if you have the <sys/dir.h> header file.  */
+#undef HAVE_SYS_DIR_H
+
+/* Define if you have the <sys/ndir.h> header file.  */
+#undef HAVE_SYS_NDIR_H
+
+/* Define if you have the <sys/procfs.h> header file.  */
+#undef HAVE_SYS_PROCFS_H
+
+/* Define if you have the <sys/time.h> header file.  */
+#undef HAVE_SYS_TIME_H
+
+/* Define if you have the <unistd.h> header file.  */
+#undef HAVE_UNISTD_H
+
+/* Define if you have the m library (-lm).  */
+#undef HAVE_LIBM
Index: /trunk/mgpp/config.sub
===================================================================
--- /trunk/mgpp/config.sub	(revision 3365)
+++ /trunk/mgpp/config.sub	(revision 3365)
@@ -0,0 +1,793 @@
+#!/bin/sh
+# Configuration validation subroutine script, version 1.1.
+#   Copyright (C) 1991, 1992, 1993, 1994 Free Software Foundation, Inc.
+# This file is (in principle) common to ALL GNU software.
+# The presence of a machine in this file suggests that SOME GNU software
+# can handle that machine.  It does not imply ALL GNU software can. 
+#
+# This file is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+# As a special exception to the GNU General Public License, if you
+# distribute this file as part of a program that contains a
+# configuration script generated by Autoconf, you may include it under
+# the same distribution terms that you use for the rest of that program.
+
+# Configuration subroutine to validate and canonicalize a configuration type.
+# Supply the specified configuration type as an argument.
+# If it is invalid, we print an error message on stderr and exit with code 1.
+# Otherwise, we print the canonical config type on stdout and succeed.
+
+# This file is supposed to be the same for all GNU packages
+# and recognize all the CPU types, system types and aliases
+# that are meaningful with *any* GNU software.
+# Each package is responsible for reporting which valid configurations
+# it does not support.  The user should be able to distinguish
+# a failure to support a valid configuration from a meaningless
+# configuration.
+
+# The goal of this file is to map all the various variations of a given
+# machine specification into a single specification in the form:
+#	CPU_TYPE-MANUFACTURER-OPERATING_SYSTEM
+# It is wrong to echo any other type of specification.
+
+# First pass through any local machine types.
+case $1 in
+	*local*)
+		echo $1
+		exit 0
+		;;
+	*)
+	;;
+esac
+
+# Separate what the user gave into CPU-COMPANY and OS (if any).
+basic_machine=`echo $1 | sed 's/-[^-]*$//'`
+if [ $basic_machine != $1 ]
+then os=`echo $1 | sed 's/.*-/-/'`
+else os=; fi
+
+### Let's recognize common machines as not being operating systems so
+### that things like config.sub decstation-3100 work.  We also
+### recognize some manufacturers as not being operating systems, so we
+### can provide default operating systems below.
+case $os in
+	-sun*os*)
+		# Prevent following clause from handling this invalid input.
+		;;
+	-dec* | -mips* | -sequent* | -encore* | -pc532* | -sgi* | -sony* | \
+	-att* | -7300* | -3300* | -delta* | -motorola* | -sun[234]* | \
+	-unicom* | -ibm* | -next | -hp | -isi* | -apollo | -altos* | \
+	-convergent* | -ncr* | -news | -32* | -3600* | -3100* | -hitachi* |\
+	-c[123]* | -convex* | -sun | -crds | -omron* | -dg | -ultra | -tti* | \
+	-harris | -dolphin | -highlevel | -gould | -cbm | -ns | -masscomp )
+		os=
+		basic_machine=$1
+		;;
+	-hiux*)
+		os=-hiuxwe2
+		;;
+	-sco4)
+		os=-sco3.2v4
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-sco3.2.[4-9]*)
+		os=`echo $os | sed -e 's/sco3.2./sco3.2v/'`
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-sco3.2v[4-9]*)
+		# Don't forget version if it is 3.2v4 or newer.
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-sco*)
+		os=-sco3.2v2
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-isc)
+		os=-isc2.2
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-clix*)
+		basic_machine=clipper-intergraph
+		;;
+	-isc*)
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-unknown/'`
+		;;
+	-lynx)
+		os=-lynxos
+		;;
+	-ptx*)
+		basic_machine=`echo $1 | sed -e 's/86-.*/86-sequent/'`
+		;;
+	-windowsnt*)
+		os=`echo $os | sed -e 's/windowsnt/winnt/'`
+		;;
+esac
+
+# Decode aliases for certain CPU-COMPANY combinations.
+case $basic_machine in
+	# Recognize the basic CPU types without company name.
+	# Some are omitted here because they have special meanings below.
+	tahoe | i[345]86 | i860 | m68k | m68000 | m88k | ns32k | arm | pyramid \
+		| tron | a29k | 580 | i960 | h8300 | hppa1.0 | hppa1.1 \
+		| alpha | we32k | ns16k | clipper | sparclite | i370 | sh \
+		| powerpc | sparc64 | 1750a | dsp16xx | mips64 | mipsel \
+		| pdp11 | mips64el | mips64orion | mips64orionel )
+		basic_machine=$basic_machine-unknown
+		;;
+	# Object if more than one company name word.
+	*-*-*)
+		echo Invalid configuration \`$1\': machine \`$basic_machine\' not recognized 1>&2
+		exit 1
+		;;
+	# Recognize the basic CPU types with company name.
+	vax-* | tahoe-* | i[345]86-* | i860-* | m68k-* | m68000-* | m88k-* \
+	      | sparc-* | ns32k-* | fx80-* | arm-* | c[123]* \
+	      | mips-* | pyramid-* | tron-* | a29k-* | romp-* | rs6000-* \
+	      | none-* | 580-* | cray2-* | h8300-* | i960-* | xmp-* | ymp-* \
+	      | hppa1.0-* | hppa1.1-* | alpha-* | we32k-* | cydra-* | ns16k-* \
+	      | pn-* | np1-* | xps100-* | clipper-* | orion-* | sparclite-* \
+	      | pdp11-* | sh-* | powerpc-* | sparc64-* | mips64-* | mipsel-* \
+	      | mips64el-* | mips64orion-* | mips64orionel-* )
+		;;
+	# Recognize the various machine names and aliases which stand
+	# for a CPU type and a company and sometimes even an OS.
+	3b1 | 7300 | 7300-att | att-7300 | pc7300 | safari | unixpc)
+		basic_machine=m68000-att
+		;;
+	3b*)
+		basic_machine=we32k-att
+		;;
+	alliant | fx80)
+		basic_machine=fx80-alliant
+		;;
+	altos | altos3068)
+		basic_machine=m68k-altos
+		;;
+	am29k)
+		basic_machine=a29k-none
+		os=-bsd
+		;;
+	amdahl)
+		basic_machine=580-amdahl
+		os=-sysv
+		;;
+	amiga | amiga-*)
+		basic_machine=m68k-cbm
+		;;
+	amigados)
+		basic_machine=m68k-cbm
+		os=-amigados
+		;;
+	amigaunix | amix)
+		basic_machine=m68k-cbm
+		os=-sysv4
+		;;
+	apollo68)
+		basic_machine=m68k-apollo
+		os=-sysv
+		;;
+	balance)
+		basic_machine=ns32k-sequent
+		os=-dynix
+		;;
+	convex-c1)
+		basic_machine=c1-convex
+		os=-bsd
+		;;
+	convex-c2)
+		basic_machine=c2-convex
+		os=-bsd
+		;;
+	convex-c32)
+		basic_machine=c32-convex
+		os=-bsd
+		;;
+	convex-c34)
+		basic_machine=c34-convex
+		os=-bsd
+		;;
+	convex-c38)
+		basic_machine=c38-convex
+		os=-bsd
+		;;
+	cray | ymp)
+		basic_machine=ymp-cray
+		os=-unicos
+		;;
+	cray2)
+		basic_machine=cray2-cray
+		os=-unicos
+		;;
+	crds | unos)
+		basic_machine=m68k-crds
+		;;
+	da30 | da30-*)
+		basic_machine=m68k-da30
+		;;
+	decstation | decstation-3100 | pmax | pmax-* | pmin | dec3100 | decstatn)
+		basic_machine=mips-dec
+		;;
+	delta | 3300 | motorola-3300 | motorola-delta \
+	      | 3300-motorola | delta-motorola)
+		basic_machine=m68k-motorola
+		;;
+	delta88)
+		basic_machine=m88k-motorola
+		os=-sysv3
+		;;
+	dpx20 | dpx20-*)
+		basic_machine=rs6000-bull
+		os=-bosx
+		;;
+	dpx2* | dpx2*-bull)
+		basic_machine=m68k-bull
+		os=-sysv3
+		;;
+	ebmon29k)
+		basic_machine=a29k-amd
+		os=-ebmon
+		;;
+	elxsi)
+		basic_machine=elxsi-elxsi
+		os=-bsd
+		;;
+	encore | umax | mmax)
+		basic_machine=ns32k-encore
+		;;
+	fx2800)
+		basic_machine=i860-alliant
+		;;
+	genix)
+		basic_machine=ns32k-ns
+		;;
+	gmicro)
+		basic_machine=tron-gmicro
+		os=-sysv
+		;;
+	h3050r* | hiux*)
+		basic_machine=hppa1.1-hitachi
+		os=-hiuxwe2
+		;;
+	h8300hms)
+		basic_machine=h8300-hitachi
+		os=-hms
+		;;
+	harris)
+		basic_machine=m88k-harris
+		os=-sysv3
+		;;
+	hp300-*)
+		basic_machine=m68k-hp
+		;;
+	hp300bsd)
+		basic_machine=m68k-hp
+		os=-bsd
+		;;
+	hp300hpux)
+		basic_machine=m68k-hp
+		os=-hpux
+		;;
+	hp9k2[0-9][0-9] | hp9k31[0-9])
+		basic_machine=m68000-hp
+		;;
+	hp9k3[2-9][0-9])
+		basic_machine=m68k-hp
+		;;
+	hp9k7[0-9][0-9] | hp7[0-9][0-9] | hp9k8[0-9]7 | hp8[0-9]7)
+		basic_machine=hppa1.1-hp
+		;;
+	hp9k8[0-9][0-9] | hp8[0-9][0-9])
+		basic_machine=hppa1.0-hp
+		;;
+	i370-ibm* | ibm*)
+		basic_machine=i370-ibm
+		os=-mvs
+		;;
+# I'm not sure what "Sysv32" means.  Should this be sysv3.2?
+	i[345]86v32)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-sysv32
+		;;
+	i[345]86v4*)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-sysv4
+		;;
+	i[345]86v)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-sysv
+		;;
+	i[345]86sol2)
+		basic_machine=`echo $1 | sed -e 's/86.*/86-unknown/'`
+		os=-solaris2
+		;;
+	iris | iris4d)
+		basic_machine=mips-sgi
+		case $os in
+		    -irix*)
+			;;
+		    *)
+			os=-irix4
+			;;
+		esac
+		;;
+	isi68 | isi)
+		basic_machine=m68k-isi
+		os=-sysv
+		;;
+	m88k-omron*)
+		basic_machine=m88k-omron
+		;;
+	magnum | m3230)
+		basic_machine=mips-mips
+		os=-sysv
+		;;
+	merlin)
+		basic_machine=ns32k-utek
+		os=-sysv
+		;;
+	miniframe)
+		basic_machine=m68000-convergent
+		;;
+	mips3*-*)
+		basic_machine=`echo $basic_machine | sed -e 's/mips3/mips64/'`
+		;;
+	mips3*)
+		basic_machine=`echo $basic_machine | sed -e 's/mips3/mips64/'`-unknown
+		;;
+	ncr3000)
+		basic_machine=i486-ncr
+		os=-sysv4
+		;;
+	news | news700 | news800 | news900)
+		basic_machine=m68k-sony
+		os=-newsos
+		;;
+	news1000)
+		basic_machine=m68030-sony
+		os=-newsos
+		;;
+	news-3600 | risc-news)
+		basic_machine=mips-sony
+		os=-newsos
+		;;
+	next | m*-next )
+		basic_machine=m68k-next
+		case $os in
+		    -nextstep* )
+			;;
+		    -ns2*)
+		      os=-nextstep2
+			;;
+		    *)
+		      os=-nextstep3
+			;;
+		esac
+		;;
+	nh3000)
+		basic_machine=m68k-harris
+		os=-cxux
+		;;
+	nh[45]000)
+		basic_machine=m88k-harris
+		os=-cxux
+		;;
+	nindy960)
+		basic_machine=i960-intel
+		os=-nindy
+		;;
+	np1)
+		basic_machine=np1-gould
+		;;
+	pa-hitachi)
+		basic_machine=hppa1.1-hitachi
+		os=-hiuxwe2
+		;;
+	paragon)
+		basic_machine=i860-intel
+		os=-osf
+		;;
+	pbd)
+		basic_machine=sparc-tti
+		;;
+	pbb)
+		basic_machine=m68k-tti
+		;;
+        pc532 | pc532-*)
+		basic_machine=ns32k-pc532
+		;;
+	pentium-*)
+		# We will change tis to say i586 once there has been
+		# time for various packages to start to recognize that.
+		basic_machine=i486-`echo $basic_machine | sed 's/^[^-]*-//'`
+		;;
+	pn)
+		basic_machine=pn-gould
+		;;
+	ps2)
+		basic_machine=i386-ibm
+		;;
+	rtpc | rtpc-*)
+		basic_machine=romp-ibm
+		;;
+	sequent)
+		basic_machine=i386-sequent
+		;;
+	sh)
+		basic_machine=sh-hitachi
+		os=-hms
+		;;
+	sps7)
+		basic_machine=m68k-bull
+		os=-sysv2
+		;;
+	spur)
+		basic_machine=spur-unknown
+		;;
+	sun2)
+		basic_machine=m68000-sun
+		;;
+	sun2os3)
+		basic_machine=m68000-sun
+		os=-sunos3
+		;;
+	sun2os4)
+		basic_machine=m68000-sun
+		os=-sunos4
+		;;
+	sun3os3)
+		basic_machine=m68k-sun
+		os=-sunos3
+		;;
+	sun3os4)
+		basic_machine=m68k-sun
+		os=-sunos4
+		;;
+	sun4os3)
+		basic_machine=sparc-sun
+		os=-sunos3
+		;;
+	sun4os4)
+		basic_machine=sparc-sun
+		os=-sunos4
+		;;
+	sun3 | sun3-*)
+		basic_machine=m68k-sun
+		;;
+	sun4)
+		basic_machine=sparc-sun
+		;;
+	sun386 | sun386i | roadrunner)
+		basic_machine=i386-sun
+		;;
+	symmetry)
+		basic_machine=i386-sequent
+		os=-dynix
+		;;
+	tower | tower-32)
+		basic_machine=m68k-ncr
+		;;
+	ultra3)
+		basic_machine=a29k-nyu
+		os=-sym1
+		;;
+	vaxv)
+		basic_machine=vax-dec
+		os=-sysv
+		;;
+	vms)
+		basic_machine=vax-dec
+		os=-vms
+		;;
+	vxworks960)
+		basic_machine=i960-wrs
+		os=-vxworks
+		;;
+	vxworks68)
+		basic_machine=m68k-wrs
+		os=-vxworks
+		;;
+	xmp)
+		basic_machine=xmp-cray
+		os=-unicos
+		;;
+        xps | xps100)
+		basic_machine=xps100-honeywell
+		;;
+	none)
+		basic_machine=none-none
+		os=-none
+		;;
+
+# Here we handle the default manufacturer of certain CPU types.  It is in
+# some cases the only manufacturer, in others, it is the most popular.
+	mips)
+		basic_machine=mips-mips
+		;;
+	romp)
+		basic_machine=romp-ibm
+		;;
+	rs6000)
+		basic_machine=rs6000-ibm
+		;;
+	vax)
+		basic_machine=vax-dec
+		;;
+	pdp11)
+		basic_machine=pdp11-dec
+		;;
+	we32k)
+		basic_machine=we32k-att
+		;;
+	sparc)
+		basic_machine=sparc-sun
+		;;
+        cydra)
+		basic_machine=cydra-cydrome
+		;;
+	orion)
+		basic_machine=orion-highlevel
+		;;
+	orion105)
+		basic_machine=clipper-highlevel
+		;;
+	*)
+		echo Invalid configuration \`$1\': machine \`$basic_machine\' not recognized 1>&2
+		exit 1
+		;;
+esac
+
+# Here we canonicalize certain aliases for manufacturers.
+case $basic_machine in
+	*-digital*)
+		basic_machine=`echo $basic_machine | sed 's/digital.*/dec/'`
+		;;
+	*-commodore*)
+		basic_machine=`echo $basic_machine | sed 's/commodore.*/cbm/'`
+		;;
+	*)
+		;;
+esac
+
+# Decode manufacturer-specific aliases for certain operating systems.
+
+if [ x"$os" != x"" ]
+then
+case $os in
+	# -solaris* is a basic system type, with this one exception.
+	-solaris1 | -solaris1.*)
+		os=`echo $os | sed -e 's|solaris1|sunos4|'`
+		;;
+	-solaris)
+		os=-solaris2
+		;;
+	-gnu/linux*)
+		os=`echo $os | sed -e 's|gnu/linux|linux|'`
+		;;
+	# First accept the basic system types.
+	# The portable systems comes first.
+	# Each alternative must end in a *, to match a version number.
+	# -sysv* is not here because it comes later, after sysvr4.
+	-gnu* | -bsd* | -mach* | -minix* | -genix* | -ultrix* | -irix* \
+	      | -vms* | -sco* | -esix* | -isc* | -aix* | -sunos | -sunos[345]* \
+	      | -hpux* | -unos* | -osf* | -luna* | -dgux* | -solaris* | -sym* \
+	      | -amigados* | -msdos* | -newsos* | -unicos* | -aos* \
+	      | -nindy* | -vxworks* | -ebmon* | -hms* | -mvs* | -clix* \
+	      | -riscos* | -linux* | -uniplus* | -iris* | -rtu* | -xenix* \
+	      | -hiux* | -386bsd* | -netbsd* | -freebsd* | -riscix* \
+	      | -lynxos* | -bosx* | -nextstep* | -cxux* | -aout* | -elf* \
+	      | -ptx* | -coff* | -winnt*)
+		;;
+	-sunos5*)
+		os=`echo $os | sed -e 's|sunos5|solaris2|'`
+		;;
+	-sunos6*)
+		os=`echo $os | sed -e 's|sunos6|solaris3|'`
+		;;
+	-osfrose*)
+		os=-osfrose
+		;;
+	-osf*)
+		os=-osf
+		;;
+	-utek*)
+		os=-bsd
+		;;
+	-dynix*)
+		os=-bsd
+		;;
+	-acis*)
+		os=-aos
+		;;
+	-ctix* | -uts*)
+		os=-sysv
+		;;
+	-triton*)
+		os=-sysv3
+		;;
+	-oss*)
+		os=-sysv3
+		;;
+	-svr4)
+		os=-sysv4
+		;;
+	-svr3)
+		os=-sysv3
+		;;
+	-sysvr4)
+		os=-sysv4
+		;;
+	# This must come after -sysvr4.
+	-sysv*)
+		;;
+	-xenix)
+		os=-xenix
+		;;
+	-none)
+		;;
+	*)
+		# Get rid of the `-' at the beginning of $os.
+		os=`echo $os | sed 's/[^-]*-//'`
+		echo Invalid configuration \`$1\': system \`$os\' not recognized 1>&2
+		exit 1
+		;;
+esac
+else
+
+# Here we handle the default operating systems that come with various machines.
+# The value should be what the vendor currently ships out the door with their
+# machine or put another way, the most popular os provided with the machine.
+
+# Note that if you're going to try to match "-MANUFACTURER" here (say,
+# "-sun"), then you have to tell the case statement up towards the top
+# that MANUFACTURER isn't an operating system.  Otherwise, code above
+# will signal an error saying that MANUFACTURER isn't an operating
+# system, and we'll never get to this point.
+
+case $basic_machine in
+	*-acorn)
+		os=-riscix1.2
+		;;
+        pdp11-*)
+		os=-none
+		;;
+	*-dec | vax-*)
+		os=-ultrix4.2
+		;;
+	i386-sun)
+		os=-sunos4.0.2
+		;;
+	m68000-sun)
+		os=-sunos3
+		# This also exists in the configure program, but was not the
+		# default.
+		# os=-sunos4
+		;;
+	*-tti)	# must be before sparc entry or we get the wrong os.
+		os=-sysv3
+		;;
+	sparc-* | *-sun)
+		os=-sunos4.1.1
+		;;
+	*-ibm)
+		os=-aix
+		;;
+	*-hp)
+		os=-hpux
+		;;
+	*-hitachi)
+		os=-hiux
+		;;
+	i860-* | *-att | *-ncr | *-altos | *-motorola | *-convergent)
+		os=-sysv
+		;;
+	*-cbm)
+		os=-amigados
+		;;
+	*-dg)
+		os=-dgux
+		;;
+	*-dolphin)
+		os=-sysv3
+		;;
+	m68k-ccur)
+		os=-rtu
+		;;
+	m88k-omron*)
+		os=-luna
+		;;
+	*-sequent)
+		os=-ptx
+		;;
+	*-crds)
+		os=-unos
+		;;
+	*-ns)
+		os=-genix
+		;;
+	i370-*)
+		os=-mvs
+		;;
+	*-next)
+		os=-nextstep3
+		;;
+        *-gould)
+		os=-sysv
+		;;
+        *-highlevel)
+		os=-bsd
+		;;
+	*-encore)
+		os=-bsd
+		;;
+        *-sgi)
+		os=-irix
+		;;
+	*-masscomp)
+		os=-rtu
+		;;
+	*)
+		os=-none
+		;;
+esac
+fi
+
+# Here we handle the case where we know the os, and the CPU type, but not the
+# manufacturer.  We pick the logical manufacturer.
+vendor=unknown
+case $basic_machine in
+	*-unknown)
+		case $os in
+			-riscix*)
+				vendor=acorn
+				;;
+			-sunos*)
+				vendor=sun
+				;;
+			-lynxos*)
+				vendor=lynx
+				;;
+			-aix*)
+				vendor=ibm
+				;;
+			-hpux*)
+				vendor=hp
+				;;
+			-hiux*)
+				vendor=hitachi
+				;;
+			-unos*)
+				vendor=crds
+				;;
+			-dgux*)
+				vendor=dg
+				;;
+			-luna*)
+				vendor=omron
+				;;
+			-genix*)
+				vendor=ns
+				;;
+			-mvs*)
+				vendor=ibm
+				;;
+			-ptx*)
+				vendor=sequent
+				;;
+		esac
+		basic_machine=`echo $basic_machine | sed "s/unknown/$vendor/"`
+		;;
+esac
+
+echo $basic_machine$os
Index: /trunk/mgpp/configure
===================================================================
--- /trunk/mgpp/configure	(revision 3365)
+++ /trunk/mgpp/configure	(revision 3365)
@@ -0,0 +1,5792 @@
+#! /bin/sh
+# Guess values for system-dependent variables and create Makefiles.
+# Generated by Autoconf 2.50.
+#
+# Copyright 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001
+# Free Software Foundation, Inc.
+# This configure script is free software; the Free Software Foundation
+# gives unlimited permission to copy, distribute and modify it.
+
+# Be Bourne compatible
+if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then
+  emulate sh
+  NULLCMD=:
+elif test -n "${BASH_VERSION+set}" && (set -o posix) >/dev/null 2>&1; then
+  set -o posix
+fi
+
+# Name of the executable.
+as_me=`echo "$0" |sed 's,.*[\\/],,'`
+
+if expr a : '\(a\)' >/dev/null 2>&1; then
+  as_expr=expr
+else
+  as_expr=false
+fi
+
+rm -f conf$$ conf$$.exe conf$$.file
+echo >conf$$.file
+if ln -s conf$$.file conf$$ 2>/dev/null; then
+  # We could just check for DJGPP; but this test a) works b) is more generic
+  # and c) will remain valid once DJGPP supports symlinks (DJGPP 2.04).
+  if test -f conf$$.exe; then
+    # Don't use ln at all; we don't have any links
+    as_ln_s='cp -p'
+  else
+    as_ln_s='ln -s'
+  fi
+elif ln conf$$.file conf$$ 2>/dev/null; then
+  as_ln_s=ln
+else
+  as_ln_s='cp -p'
+fi
+rm -f conf$$ conf$$.exe conf$$.file
+
+as_executable_p="test -f"
+
+# Support unset when possible.
+if (FOO=FOO; unset FOO) >/dev/null 2>&1; then
+  as_unset=unset
+else
+  as_unset=false
+fi
+
+# NLS nuisances.
+$as_unset LANG || test "${LANG+set}" != set || { LANG=C; export LANG; }
+$as_unset LC_ALL || test "${LC_ALL+set}" != set || { LC_ALL=C; export LC_ALL; }
+$as_unset LC_TIME || test "${LC_TIME+set}" != set || { LC_TIME=C; export LC_TIME; }
+$as_unset LC_CTYPE || test "${LC_CTYPE+set}" != set || { LC_CTYPE=C; export LC_CTYPE; }
+$as_unset LANGUAGE || test "${LANGUAGE+set}" != set || { LANGUAGE=C; export LANGUAGE; }
+$as_unset LC_COLLATE || test "${LC_COLLATE+set}" != set || { LC_COLLATE=C; export LC_COLLATE; }
+$as_unset LC_NUMERIC || test "${LC_NUMERIC+set}" != set || { LC_NUMERIC=C; export LC_NUMERIC; }
+$as_unset LC_MESSAGES || test "${LC_MESSAGES+set}" != set || { LC_MESSAGES=C; export LC_MESSAGES; }
+
+# IFS
+# We need space, tab and new line, in precisely that order.
+as_nl='
+'
+IFS=" 	$as_nl"
+
+# CDPATH.
+$as_unset CDPATH || test "${CDPATH+set}" != set || { CDPATH=:; export CDPATH; }
+
+# Name of the host.
+# hostname on some systems (SVR3.2, Linux) returns a bogus exit status,
+# so uname gets run too.
+ac_hostname=`(hostname || uname -n) 2>/dev/null | sed 1q`
+
+exec 6>&1
+
+#
+# Initializations.
+#
+ac_default_prefix=/usr/local
+cross_compiling=no
+subdirs=
+MFLAGS= MAKEFLAGS=
+SHELL=${CONFIG_SHELL-/bin/sh}
+
+# Maximum number of lines to put in a shell here document.
+# This variable seems obsolete.  It should probably be removed, and
+# only ac_max_sed_lines should be used.
+: ${ac_max_here_lines=38}
+
+# Avoid depending upon Character Ranges.
+ac_cr_az='abcdefghijklmnopqrstuvwxyz'
+ac_cr_AZ='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+ac_cr_09='0123456789'
+ac_cr_alnum=$ac_cr_az$ac_cr_AZ$ac_cr_09
+
+# Sed expression to map a string onto a valid sh and CPP variable names.
+ac_tr_sh="sed y%*+%pp%;s%[^_$ac_cr_alnum]%_%g"
+ac_tr_cpp="sed y%*$ac_cr_az%P$ac_cr_AZ%;s%[^_$ac_cr_alnum]%_%g"
+
+ac_unique_file="text/MGQuery.cpp"
+# Factoring default headers for most tests.
+ac_includes_default="\
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#if STDC_HEADERS
+# include <stdlib.h>
+# include <stddef.h>
+#else
+# if HAVE_STDLIB_H
+#  include <stdlib.h>
+# endif
+#endif
+#if HAVE_STRING_H
+# if !STDC_HEADERS && HAVE_MEMORY_H
+#  include <memory.h>
+# endif
+# include <string.h>
+#else
+# if HAVE_STRINGS_H
+#  include <strings.h>
+# endif
+#endif
+#if HAVE_INTTYPES_H
+# include <inttypes.h>
+#endif
+#if HAVE_UNISTD_H
+# include <unistd.h>
+#endif"
+
+# Initialize some variables set by options.
+ac_init_help=
+ac_init_version=false
+# The variables have the same names as the options, with
+# dashes changed to underlines.
+cache_file=/dev/null
+exec_prefix=NONE
+no_create=
+no_recursion=
+prefix=NONE
+program_prefix=NONE
+program_suffix=NONE
+program_transform_name=s,x,x,
+silent=
+site=
+srcdir=
+verbose=
+x_includes=NONE
+x_libraries=NONE
+
+# Installation directory options.
+# These are left unexpanded so users can "make install exec_prefix=/foo"
+# and all the variables that are supposed to be based on exec_prefix
+# by default will actually change.
+# Use braces instead of parens because sh, perl, etc. also accept them.
+bindir='${exec_prefix}/bin'
+sbindir='${exec_prefix}/sbin'
+libexecdir='${exec_prefix}/libexec'
+datadir='${prefix}/share'
+sysconfdir='${prefix}/etc'
+sharedstatedir='${prefix}/com'
+localstatedir='${prefix}/var'
+libdir='${exec_prefix}/lib'
+includedir='${prefix}/include'
+oldincludedir='/usr/include'
+infodir='${prefix}/info'
+mandir='${prefix}/man'
+
+# Identity of this package.
+PACKAGE_NAME=
+PACKAGE_TARNAME=
+PACKAGE_VERSION=
+PACKAGE_STRING=
+PACKAGE_BUGREPORT=
+
+ac_prev=
+for ac_option
+do
+  # If the previous option needs an argument, assign it.
+  if test -n "$ac_prev"; then
+    eval "$ac_prev=\$ac_option"
+    ac_prev=
+    continue
+  fi
+
+  ac_optarg=`expr "x$ac_option" : 'x[^=]*=\(.*\)'`
+
+  # Accept the important Cygnus configure options, so we can diagnose typos.
+
+  case $ac_option in
+
+  -bindir | --bindir | --bindi | --bind | --bin | --bi)
+    ac_prev=bindir ;;
+  -bindir=* | --bindir=* | --bindi=* | --bind=* | --bin=* | --bi=*)
+    bindir=$ac_optarg ;;
+
+  -build | --build | --buil | --bui | --bu)
+    ac_prev=build_alias ;;
+  -build=* | --build=* | --buil=* | --bui=* | --bu=*)
+    build_alias=$ac_optarg ;;
+
+  -cache-file | --cache-file | --cache-fil | --cache-fi \
+  | --cache-f | --cache- | --cache | --cach | --cac | --ca | --c)
+    ac_prev=cache_file ;;
+  -cache-file=* | --cache-file=* | --cache-fil=* | --cache-fi=* \
+  | --cache-f=* | --cache-=* | --cache=* | --cach=* | --cac=* | --ca=* | --c=*)
+    cache_file=$ac_optarg ;;
+
+  --config-cache | -C)
+    cache_file=config.cache ;;
+
+  -datadir | --datadir | --datadi | --datad | --data | --dat | --da)
+    ac_prev=datadir ;;
+  -datadir=* | --datadir=* | --datadi=* | --datad=* | --data=* | --dat=* \
+  | --da=*)
+    datadir=$ac_optarg ;;
+
+  -disable-* | --disable-*)
+    ac_feature=`expr "x$ac_option" : 'x-*disable-\(.*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_feature" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid feature name: $ac_feature" >&2
+   { (exit 1); exit 1; }; }
+    ac_feature=`echo $ac_feature | sed 's/-/_/g'`
+    eval "enable_$ac_feature=no" ;;
+
+  -enable-* | --enable-*)
+    ac_feature=`expr "x$ac_option" : 'x-*enable-\([^=]*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_feature" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid feature name: $ac_feature" >&2
+   { (exit 1); exit 1; }; }
+    ac_feature=`echo $ac_feature | sed 's/-/_/g'`
+    case $ac_option in
+      *=*) ac_optarg=`echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"`;;
+      *) ac_optarg=yes ;;
+    esac
+    eval "enable_$ac_feature='$ac_optarg'" ;;
+
+  -exec-prefix | --exec_prefix | --exec-prefix | --exec-prefi \
+  | --exec-pref | --exec-pre | --exec-pr | --exec-p | --exec- \
+  | --exec | --exe | --ex)
+    ac_prev=exec_prefix ;;
+  -exec-prefix=* | --exec_prefix=* | --exec-prefix=* | --exec-prefi=* \
+  | --exec-pref=* | --exec-pre=* | --exec-pr=* | --exec-p=* | --exec-=* \
+  | --exec=* | --exe=* | --ex=*)
+    exec_prefix=$ac_optarg ;;
+
+  -gas | --gas | --ga | --g)
+    # Obsolete; use --with-gas.
+    with_gas=yes ;;
+
+  -help | --help | --hel | --he | -h)
+    ac_init_help=long ;;
+  -help=r* | --help=r* | --hel=r* | --he=r* | -hr*)
+    ac_init_help=recursive ;;
+  -help=s* | --help=s* | --hel=s* | --he=s* | -hs*)
+    ac_init_help=short ;;
+
+  -host | --host | --hos | --ho)
+    ac_prev=host_alias ;;
+  -host=* | --host=* | --hos=* | --ho=*)
+    host_alias=$ac_optarg ;;
+
+  -includedir | --includedir | --includedi | --included | --include \
+  | --includ | --inclu | --incl | --inc)
+    ac_prev=includedir ;;
+  -includedir=* | --includedir=* | --includedi=* | --included=* | --include=* \
+  | --includ=* | --inclu=* | --incl=* | --inc=*)
+    includedir=$ac_optarg ;;
+
+  -infodir | --infodir | --infodi | --infod | --info | --inf)
+    ac_prev=infodir ;;
+  -infodir=* | --infodir=* | --infodi=* | --infod=* | --info=* | --inf=*)
+    infodir=$ac_optarg ;;
+
+  -libdir | --libdir | --libdi | --libd)
+    ac_prev=libdir ;;
+  -libdir=* | --libdir=* | --libdi=* | --libd=*)
+    libdir=$ac_optarg ;;
+
+  -libexecdir | --libexecdir | --libexecdi | --libexecd | --libexec \
+  | --libexe | --libex | --libe)
+    ac_prev=libexecdir ;;
+  -libexecdir=* | --libexecdir=* | --libexecdi=* | --libexecd=* | --libexec=* \
+  | --libexe=* | --libex=* | --libe=*)
+    libexecdir=$ac_optarg ;;
+
+  -localstatedir | --localstatedir | --localstatedi | --localstated \
+  | --localstate | --localstat | --localsta | --localst \
+  | --locals | --local | --loca | --loc | --lo)
+    ac_prev=localstatedir ;;
+  -localstatedir=* | --localstatedir=* | --localstatedi=* | --localstated=* \
+  | --localstate=* | --localstat=* | --localsta=* | --localst=* \
+  | --locals=* | --local=* | --loca=* | --loc=* | --lo=*)
+    localstatedir=$ac_optarg ;;
+
+  -mandir | --mandir | --mandi | --mand | --man | --ma | --m)
+    ac_prev=mandir ;;
+  -mandir=* | --mandir=* | --mandi=* | --mand=* | --man=* | --ma=* | --m=*)
+    mandir=$ac_optarg ;;
+
+  -nfp | --nfp | --nf)
+    # Obsolete; use --without-fp.
+    with_fp=no ;;
+
+  -no-create | --no-create | --no-creat | --no-crea | --no-cre \
+  | --no-cr | --no-c)
+    no_create=yes ;;
+
+  -no-recursion | --no-recursion | --no-recursio | --no-recursi \
+  | --no-recurs | --no-recur | --no-recu | --no-rec | --no-re | --no-r)
+    no_recursion=yes ;;
+
+  -oldincludedir | --oldincludedir | --oldincludedi | --oldincluded \
+  | --oldinclude | --oldinclud | --oldinclu | --oldincl | --oldinc \
+  | --oldin | --oldi | --old | --ol | --o)
+    ac_prev=oldincludedir ;;
+  -oldincludedir=* | --oldincludedir=* | --oldincludedi=* | --oldincluded=* \
+  | --oldinclude=* | --oldinclud=* | --oldinclu=* | --oldincl=* | --oldinc=* \
+  | --oldin=* | --oldi=* | --old=* | --ol=* | --o=*)
+    oldincludedir=$ac_optarg ;;
+
+  -prefix | --prefix | --prefi | --pref | --pre | --pr | --p)
+    ac_prev=prefix ;;
+  -prefix=* | --prefix=* | --prefi=* | --pref=* | --pre=* | --pr=* | --p=*)
+    prefix=$ac_optarg ;;
+
+  -program-prefix | --program-prefix | --program-prefi | --program-pref \
+  | --program-pre | --program-pr | --program-p)
+    ac_prev=program_prefix ;;
+  -program-prefix=* | --program-prefix=* | --program-prefi=* \
+  | --program-pref=* | --program-pre=* | --program-pr=* | --program-p=*)
+    program_prefix=$ac_optarg ;;
+
+  -program-suffix | --program-suffix | --program-suffi | --program-suff \
+  | --program-suf | --program-su | --program-s)
+    ac_prev=program_suffix ;;
+  -program-suffix=* | --program-suffix=* | --program-suffi=* \
+  | --program-suff=* | --program-suf=* | --program-su=* | --program-s=*)
+    program_suffix=$ac_optarg ;;
+
+  -program-transform-name | --program-transform-name \
+  | --program-transform-nam | --program-transform-na \
+  | --program-transform-n | --program-transform- \
+  | --program-transform | --program-transfor \
+  | --program-transfo | --program-transf \
+  | --program-trans | --program-tran \
+  | --progr-tra | --program-tr | --program-t)
+    ac_prev=program_transform_name ;;
+  -program-transform-name=* | --program-transform-name=* \
+  | --program-transform-nam=* | --program-transform-na=* \
+  | --program-transform-n=* | --program-transform-=* \
+  | --program-transform=* | --program-transfor=* \
+  | --program-transfo=* | --program-transf=* \
+  | --program-trans=* | --program-tran=* \
+  | --progr-tra=* | --program-tr=* | --program-t=*)
+    program_transform_name=$ac_optarg ;;
+
+  -q | -quiet | --quiet | --quie | --qui | --qu | --q \
+  | -silent | --silent | --silen | --sile | --sil)
+    silent=yes ;;
+
+  -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
+    ac_prev=sbindir ;;
+  -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
+  | --sbi=* | --sb=*)
+    sbindir=$ac_optarg ;;
+
+  -sharedstatedir | --sharedstatedir | --sharedstatedi \
+  | --sharedstated | --sharedstate | --sharedstat | --sharedsta \
+  | --sharedst | --shareds | --shared | --share | --shar \
+  | --sha | --sh)
+    ac_prev=sharedstatedir ;;
+  -sharedstatedir=* | --sharedstatedir=* | --sharedstatedi=* \
+  | --sharedstated=* | --sharedstate=* | --sharedstat=* | --sharedsta=* \
+  | --sharedst=* | --shareds=* | --shared=* | --share=* | --shar=* \
+  | --sha=* | --sh=*)
+    sharedstatedir=$ac_optarg ;;
+
+  -site | --site | --sit)
+    ac_prev=site ;;
+  -site=* | --site=* | --sit=*)
+    site=$ac_optarg ;;
+
+  -srcdir | --srcdir | --srcdi | --srcd | --src | --sr)
+    ac_prev=srcdir ;;
+  -srcdir=* | --srcdir=* | --srcdi=* | --srcd=* | --src=* | --sr=*)
+    srcdir=$ac_optarg ;;
+
+  -sysconfdir | --sysconfdir | --sysconfdi | --sysconfd | --sysconf \
+  | --syscon | --sysco | --sysc | --sys | --sy)
+    ac_prev=sysconfdir ;;
+  -sysconfdir=* | --sysconfdir=* | --sysconfdi=* | --sysconfd=* | --sysconf=* \
+  | --syscon=* | --sysco=* | --sysc=* | --sys=* | --sy=*)
+    sysconfdir=$ac_optarg ;;
+
+  -target | --target | --targe | --targ | --tar | --ta | --t)
+    ac_prev=target_alias ;;
+  -target=* | --target=* | --targe=* | --targ=* | --tar=* | --ta=* | --t=*)
+    target_alias=$ac_optarg ;;
+
+  -v | -verbose | --verbose | --verbos | --verbo | --verb)
+    verbose=yes ;;
+
+  -version | --version | --versio | --versi | --vers | -V)
+    ac_init_version=: ;;
+
+  -with-* | --with-*)
+    ac_package=`expr "x$ac_option" : 'x-*with-\([^=]*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_package" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid package name: $ac_package" >&2
+   { (exit 1); exit 1; }; }
+    ac_package=`echo $ac_package| sed 's/-/_/g'`
+    case $ac_option in
+      *=*) ac_optarg=`echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"`;;
+      *) ac_optarg=yes ;;
+    esac
+    eval "with_$ac_package='$ac_optarg'" ;;
+
+  -without-* | --without-*)
+    ac_package=`expr "x$ac_option" : 'x-*without-\(.*\)'`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_package" : ".*[^-_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid package name: $ac_package" >&2
+   { (exit 1); exit 1; }; }
+    ac_package=`echo $ac_package | sed 's/-/_/g'`
+    eval "with_$ac_package=no" ;;
+
+  --x)
+    # Obsolete; use --with-x.
+    with_x=yes ;;
+
+  -x-includes | --x-includes | --x-include | --x-includ | --x-inclu \
+  | --x-incl | --x-inc | --x-in | --x-i)
+    ac_prev=x_includes ;;
+  -x-includes=* | --x-includes=* | --x-include=* | --x-includ=* | --x-inclu=* \
+  | --x-incl=* | --x-inc=* | --x-in=* | --x-i=*)
+    x_includes=$ac_optarg ;;
+
+  -x-libraries | --x-libraries | --x-librarie | --x-librari \
+  | --x-librar | --x-libra | --x-libr | --x-lib | --x-li | --x-l)
+    ac_prev=x_libraries ;;
+  -x-libraries=* | --x-libraries=* | --x-librarie=* | --x-librari=* \
+  | --x-librar=* | --x-libra=* | --x-libr=* | --x-lib=* | --x-li=* | --x-l=*)
+    x_libraries=$ac_optarg ;;
+
+  -*) { echo "$as_me: error: unrecognized option: $ac_option
+Try \`$0 --help' for more information." >&2
+   { (exit 1); exit 1; }; }
+    ;;
+
+  *=*)
+    ac_envvar=`expr "x$ac_option" : 'x\([^=]*\)='`
+    # Reject names that are not valid shell variable names.
+    expr "x$ac_envvar" : ".*[^_$ac_cr_alnum]" >/dev/null &&
+      { echo "$as_me: error: invalid variable name: $ac_envvar" >&2
+   { (exit 1); exit 1; }; }
+    ac_optarg=`echo "$ac_optarg" | sed "s/'/'\\\\\\\\''/g"`
+    eval "$ac_envvar='$ac_optarg'"
+    export $ac_envvar ;;
+
+  *)
+    # FIXME: should be removed in autoconf 3.0.
+    echo "$as_me: WARNING: you should use --build, --host, --target" >&2
+    expr "x$ac_option" : ".*[^-._$ac_cr_alnum]" >/dev/null &&
+      echo "$as_me: WARNING: invalid host type: $ac_option" >&2
+    : ${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}
+    ;;
+
+  esac
+done
+
+if test -n "$ac_prev"; then
+  ac_option=--`echo $ac_prev | sed 's/_/-/g'`
+  { echo "$as_me: error: missing argument to $ac_option" >&2
+   { (exit 1); exit 1; }; }
+fi
+
+# Be sure to have absolute paths.
+for ac_var in bindir sbindir libexecdir datadir sysconfdir sharedstatedir \
+              localstatedir libdir includedir oldincludedir infodir mandir \
+              exec_prefix prefix
+do
+  eval ac_val=$`echo $ac_var`
+  case $ac_val in
+    [\\/$]* | ?:[\\/]* ) ;;
+    NONE ) ;;
+    *)  { echo "$as_me: error: expected an absolute path for --$ac_var: $ac_val" >&2
+   { (exit 1); exit 1; }; };;
+  esac
+done
+
+# There might be people who depend on the old broken behavior: `$host'
+# used to hold the argument of --host etc.
+build=$build_alias
+host=$host_alias
+target=$target_alias
+
+# FIXME: should be removed in autoconf 3.0.
+if test "x$host_alias" != x; then
+  if test "x$build_alias" = x; then
+    cross_compiling=maybe
+    echo "$as_me: WARNING: If you wanted to set the --build type, don't use --host.
+    If a cross compiler is detected then cross compile mode will be used." >&2
+  elif test "x$build_alias" != "x$host_alias"; then
+    cross_compiling=yes
+  fi
+fi
+
+ac_tool_prefix=
+test -n "$host_alias" && ac_tool_prefix=$host_alias-
+
+test "$silent" = yes && exec 6>/dev/null
+
+# Find the source files, if location was not specified.
+if test -z "$srcdir"; then
+  ac_srcdir_defaulted=yes
+  # Try the directory containing this script, then its parent.
+  ac_prog=$0
+  ac_confdir=`echo "$ac_prog" | sed 's%[\\/][^\\/][^\\/]*$%%'`
+  test "x$ac_confdir" = "x$ac_prog" && ac_confdir=.
+  srcdir=$ac_confdir
+  if test ! -r $srcdir/$ac_unique_file; then
+    srcdir=..
+  fi
+else
+  ac_srcdir_defaulted=no
+fi
+if test ! -r $srcdir/$ac_unique_file; then
+  if test "$ac_srcdir_defaulted" = yes; then
+    { echo "$as_me: error: cannot find sources in $ac_confdir or .." >&2
+   { (exit 1); exit 1; }; }
+  else
+    { echo "$as_me: error: cannot find sources in $srcdir" >&2
+   { (exit 1); exit 1; }; }
+  fi
+fi
+srcdir=`echo "$srcdir" | sed 's%\([^\\/]\)[\\/]*$%\1%'`
+ac_env_build_alias_set=${build_alias+set}
+ac_env_build_alias_value=$build_alias
+ac_cv_env_build_alias_set=${build_alias+set}
+ac_cv_env_build_alias_value=$build_alias
+ac_env_host_alias_set=${host_alias+set}
+ac_env_host_alias_value=$host_alias
+ac_cv_env_host_alias_set=${host_alias+set}
+ac_cv_env_host_alias_value=$host_alias
+ac_env_target_alias_set=${target_alias+set}
+ac_env_target_alias_value=$target_alias
+ac_cv_env_target_alias_set=${target_alias+set}
+ac_cv_env_target_alias_value=$target_alias
+ac_env_CXX_set=${CXX+set}
+ac_env_CXX_value=$CXX
+ac_cv_env_CXX_set=${CXX+set}
+ac_cv_env_CXX_value=$CXX
+ac_env_CXXFLAGS_set=${CXXFLAGS+set}
+ac_env_CXXFLAGS_value=$CXXFLAGS
+ac_cv_env_CXXFLAGS_set=${CXXFLAGS+set}
+ac_cv_env_CXXFLAGS_value=$CXXFLAGS
+ac_env_LDFLAGS_set=${LDFLAGS+set}
+ac_env_LDFLAGS_value=$LDFLAGS
+ac_cv_env_LDFLAGS_set=${LDFLAGS+set}
+ac_cv_env_LDFLAGS_value=$LDFLAGS
+ac_env_CPPFLAGS_set=${CPPFLAGS+set}
+ac_env_CPPFLAGS_value=$CPPFLAGS
+ac_cv_env_CPPFLAGS_set=${CPPFLAGS+set}
+ac_cv_env_CPPFLAGS_value=$CPPFLAGS
+ac_env_CC_set=${CC+set}
+ac_env_CC_value=$CC
+ac_cv_env_CC_set=${CC+set}
+ac_cv_env_CC_value=$CC
+ac_env_CFLAGS_set=${CFLAGS+set}
+ac_env_CFLAGS_value=$CFLAGS
+ac_cv_env_CFLAGS_set=${CFLAGS+set}
+ac_cv_env_CFLAGS_value=$CFLAGS
+ac_env_CPP_set=${CPP+set}
+ac_env_CPP_value=$CPP
+ac_cv_env_CPP_set=${CPP+set}
+ac_cv_env_CPP_value=$CPP
+
+#
+# Report the --help message.
+#
+if test "$ac_init_help" = "long"; then
+  # Omit some internal or obsolete options to make the list less imposing.
+  # This message is too long to be a string in the A/UX 3.1 sh.
+  cat <<EOF
+\`configure' configures this package to adapt to many kinds of systems.
+
+Usage: $0 [OPTION]... [VAR=VALUE]...
+
+To assign environment variables (e.g., CC, CFLAGS...), specify them as
+VAR=VALUE.  See below for descriptions of some of the useful variables.
+
+Defaults for the options are specified in brackets.
+
+Configuration:
+  -h, --help              display this help and exit
+      --help=short        display options specific to this package
+      --help=recursive    display the short help of all the included packages
+  -V, --version           display version information and exit
+  -q, --quiet, --silent   do not print \`checking...' messages
+      --cache-file=FILE   cache test results in FILE [disabled]
+  -C, --config-cache      alias for \`--cache-file=config.cache'
+  -n, --no-create         do not create output files
+      --srcdir=DIR        find the sources in DIR [configure dir or \`..']
+
+EOF
+
+  cat <<EOF
+Installation directories:
+  --prefix=PREFIX         install architecture-independent files in PREFIX
+                          [$ac_default_prefix]
+  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
+                          [PREFIX]
+
+By default, \`make install' will install all the files in
+\`$ac_default_prefix/bin', \`$ac_default_prefix/lib' etc.  You can specify
+an installation prefix other than \`$ac_default_prefix' using \`--prefix',
+for instance \`--prefix=\$HOME'.
+
+For better control, use the options below.
+
+Fine tuning of the installation directories:
+  --bindir=DIR           user executables [EPREFIX/bin]
+  --sbindir=DIR          system admin executables [EPREFIX/sbin]
+  --libexecdir=DIR       program executables [EPREFIX/libexec]
+  --datadir=DIR          read-only architecture-independent data [PREFIX/share]
+  --sysconfdir=DIR       read-only single-machine data [PREFIX/etc]
+  --sharedstatedir=DIR   modifiable architecture-independent data [PREFIX/com]
+  --localstatedir=DIR    modifiable single-machine data [PREFIX/var]
+  --libdir=DIR           object code libraries [EPREFIX/lib]
+  --includedir=DIR       C header files [PREFIX/include]
+  --oldincludedir=DIR    C header files for non-gcc [/usr/include]
+  --infodir=DIR          info documentation [PREFIX/info]
+  --mandir=DIR           man documentation [PREFIX/man]
+EOF
+
+  cat <<\EOF
+
+Program names:
+  --program-prefix=PREFIX            prepend PREFIX to installed program names
+  --program-suffix=SUFFIX            append SUFFIX to installed program names
+  --program-transform-name=PROGRAM   run sed PROGRAM on installed program names
+EOF
+fi
+
+if test -n "$ac_init_help"; then
+
+  cat <<\EOF
+
+Optional Features:
+  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
+  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
+  --enable-override-longlong
+                          overide use of long long with GCC
+
+Optional Packages:
+  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
+  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
+  --with-dmalloc          use dmalloc, as in
+                          ftp://ftp.letters.com/src/dmalloc/dmalloc.tar.gz
+  --with-regex            use older regex in lieu of GNU rx for matching
+  --with-gnu-readline     compile with GNU readline support
+
+Some influential environment variables:
+  CXX         C++ compiler command
+  CXXFLAGS    C++ compiler flags
+  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
+              nonstandard directory <lib dir>
+  CPPFLAGS    C/C++ preprocessor flags, e.g. -I<include dir> if you have
+              headers in a nonstandard directory <include dir>
+  CC          C compiler command
+  CFLAGS      C compiler flags
+  CPP         C preprocessor
+
+Use these variables to override the choices made by `configure' or to help
+it to find libraries and programs with nonstandard names/locations.
+
+EOF
+fi
+
+if test "$ac_init_help" = "recursive"; then
+  # If there are subdirs, report their specific --help.
+  ac_popdir=`pwd`
+  for ac_subdir in : $ac_subdirs_all; do test "x$ac_subdir" = x: && continue
+    cd $ac_subdir
+    # A "../" for each directory in /$ac_subdir.
+    ac_dots=`echo $ac_subdir |
+             sed 's,^\./,,;s,[^/]$,&/,;s,[^/]*/,../,g'`
+
+    case $srcdir in
+    .) # No --srcdir option.  We are building in place.
+      ac_sub_srcdir=$srcdir ;;
+    [\\/]* | ?:[\\/]* ) # Absolute path.
+      ac_sub_srcdir=$srcdir/$ac_subdir ;;
+    *) # Relative path.
+      ac_sub_srcdir=$ac_dots$srcdir/$ac_subdir ;;
+    esac
+
+    # Check for guested configure; otherwise get Cygnus style configure.
+    if test -f $ac_sub_srcdir/configure.gnu; then
+      echo
+      $SHELL $ac_sub_srcdir/configure.gnu  --help=recursive
+    elif test -f $ac_sub_srcdir/configure; then
+      echo
+      $SHELL $ac_sub_srcdir/configure  --help=recursive
+    elif test -f $ac_sub_srcdir/configure.ac ||
+           test -f $ac_sub_srcdir/configure.in; then
+      echo
+      $ac_configure --help
+    else
+      echo "$as_me: WARNING: no configuration information is in $ac_subdir" >&2
+    fi
+    cd $ac_popdir
+  done
+fi
+
+test -n "$ac_init_help" && exit 0
+if $ac_init_version; then
+  cat <<\EOF
+
+Copyright 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001
+Free Software Foundation, Inc.
+This configure script is free software; the Free Software Foundation
+gives unlimited permission to copy, distribute and modify it.
+EOF
+  exit 0
+fi
+exec 5>config.log
+cat >&5 <<EOF
+This file contains any messages produced by compilers while
+running configure, to aid debugging if configure makes a mistake.
+
+It was created by $as_me, which was
+generated by GNU Autoconf 2.50.  Invocation command line was
+
+  $ $0 $@
+
+EOF
+{
+cat <<_ASUNAME
+## ---------- ##
+## Platform.  ##
+## ---------- ##
+
+hostname = `(hostname || uname -n) 2>/dev/null | sed 1q`
+uname -m = `(uname -m) 2>/dev/null || echo unknown`
+uname -r = `(uname -r) 2>/dev/null || echo unknown`
+uname -s = `(uname -s) 2>/dev/null || echo unknown`
+uname -v = `(uname -v) 2>/dev/null || echo unknown`
+
+/usr/bin/uname -p = `(/usr/bin/uname -p) 2>/dev/null || echo unknown`
+/bin/uname -X     = `(/bin/uname -X) 2>/dev/null     || echo unknown`
+
+/bin/arch              = `(/bin/arch) 2>/dev/null              || echo unknown`
+/usr/bin/arch -k       = `(/usr/bin/arch -k) 2>/dev/null       || echo unknown`
+/usr/convex/getsysinfo = `(/usr/convex/getsysinfo) 2>/dev/null || echo unknown`
+hostinfo               = `(hostinfo) 2>/dev/null               || echo unknown`
+/bin/machine           = `(/bin/machine) 2>/dev/null           || echo unknown`
+/usr/bin/oslevel       = `(/usr/bin/oslevel) 2>/dev/null       || echo unknown`
+/bin/universe          = `(/bin/universe) 2>/dev/null          || echo unknown`
+
+PATH = $PATH
+
+_ASUNAME
+} >&5
+
+cat >&5 <<EOF
+## ------------ ##
+## Core tests.  ##
+## ------------ ##
+
+EOF
+
+# Keep a trace of the command line.
+# Strip out --no-create and --no-recursion so they do not pile up.
+# Also quote any args containing shell meta-characters.
+ac_configure_args=
+ac_sep=
+for ac_arg
+do
+  case $ac_arg in
+  -no-create | --no-create | --no-creat | --no-crea | --no-cre \
+  | --no-cr | --no-c) ;;
+  -no-recursion | --no-recursion | --no-recursio | --no-recursi \
+  | --no-recurs | --no-recur | --no-recu | --no-rec | --no-re | --no-r) ;;
+  *" "*|*"	"*|*[\[\]\~\#\$\^\&\*\(\)\{\}\\\|\;\<\>\?\"\']*)
+    ac_arg=`echo "$ac_arg" | sed "s/'/'\\\\\\\\''/g"`
+    ac_configure_args="$ac_configure_args$ac_sep'$ac_arg'"
+    ac_sep=" " ;;
+  *) ac_configure_args="$ac_configure_args$ac_sep$ac_arg"
+     ac_sep=" " ;;
+  esac
+  # Get rid of the leading space.
+done
+
+# When interrupted or exit'd, cleanup temporary files, and complete
+# config.log.  We remove comments because anyway the quotes in there
+# would cause problems or look ugly.
+trap 'exit_status=$?
+  # Save into config.log some information that might help in debugging.
+  echo >&5
+  echo "## ----------------- ##" >&5
+  echo "## Cache variables.  ##" >&5
+  echo "## ----------------- ##" >&5
+  echo >&5
+  # The following way of writing the cache mishandles newlines in values,
+{
+  (set) 2>&1 |
+    case `(ac_space='"'"' '"'"'; set | grep ac_space) 2>&1` in
+    *ac_space=\ *)
+      sed -n \
+        "s/'"'"'/'"'"'\\\\'"'"''"'"'/g;
+    	  s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1='"'"'\\2'"'"'/p"
+      ;;
+    *)
+      sed -n \
+        "s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1=\\2/p"
+      ;;
+    esac;
+} >&5
+  sed "/^$/d" confdefs.h >conftest.log
+  if test -s conftest.log; then
+    echo >&5
+    echo "## ------------ ##" >&5
+    echo "## confdefs.h.  ##" >&5
+    echo "## ------------ ##" >&5
+    echo >&5
+    cat conftest.log >&5
+  fi
+  (echo; echo) >&5
+  test "$ac_signal" != 0 &&
+    echo "$as_me: caught signal $ac_signal" >&5
+  echo "$as_me: exit $exit_status" >&5
+  rm -rf conftest* confdefs* core core.* *.core conf$$* $ac_clean_files &&
+    exit $exit_status
+     ' 0
+for ac_signal in 1 2 13 15; do
+  trap 'ac_status=$?; ac_signal='$ac_signal'; { (exit $ac_status); exit $ac_status; }' $ac_signal
+done
+ac_signal=0
+
+# confdefs.h avoids OS command line length limits that DEFS can exceed.
+rm -rf conftest* confdefs.h
+# AIX cpp loses on an empty file, so make sure it contains at least a newline.
+echo >confdefs.h
+
+# Let the site file select an alternate cache file if it wants to.
+# Prefer explicitly selected file to automatically selected ones.
+if test -z "$CONFIG_SITE"; then
+  if test "x$prefix" != xNONE; then
+    CONFIG_SITE="$prefix/share/config.site $prefix/etc/config.site"
+  else
+    CONFIG_SITE="$ac_default_prefix/share/config.site $ac_default_prefix/etc/config.site"
+  fi
+fi
+for ac_site_file in $CONFIG_SITE; do
+  if test -r "$ac_site_file"; then
+    { echo "$as_me:865: loading site script $ac_site_file" >&5
+echo "$as_me: loading site script $ac_site_file" >&6;}
+    cat "$ac_site_file" >&5
+    . "$ac_site_file"
+  fi
+done
+
+if test -r "$cache_file"; then
+  # Some versions of bash will fail to source /dev/null (special
+  # files actually), so we avoid doing that.
+  if test -f "$cache_file"; then
+    { echo "$as_me:876: loading cache $cache_file" >&5
+echo "$as_me: loading cache $cache_file" >&6;}
+    case $cache_file in
+      [\\/]* | ?:[\\/]* ) . $cache_file;;
+      *)                      . ./$cache_file;;
+    esac
+  fi
+else
+  { echo "$as_me:884: creating cache $cache_file" >&5
+echo "$as_me: creating cache $cache_file" >&6;}
+  >$cache_file
+fi
+
+# Check that the precious variables saved in the cache have kept the same
+# value.
+ac_suggest_removing_cache=false
+for ac_var in `(set) 2>&1 |
+               sed -n 's/^ac_env_\([a-zA-Z_0-9]*\)_set=.*/\1/p'`; do
+  eval ac_old_set=\$ac_cv_env_${ac_var}_set
+  eval ac_new_set=\$ac_env_${ac_var}_set
+  eval ac_old_val="\$ac_cv_env_${ac_var}_value"
+  eval ac_new_val="\$ac_env_${ac_var}_value"
+  case $ac_old_set,$ac_new_set in
+    set,)
+      { echo "$as_me:900: WARNING: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&5
+echo "$as_me: WARNING: \`$ac_var' was set to \`$ac_old_val' in the previous run" >&2;}
+      ac_suggest_removing_cache=: ;;
+    ,set)
+      { echo "$as_me:904: WARNING: \`$ac_var' was not set in the previous run" >&5
+echo "$as_me: WARNING: \`$ac_var' was not set in the previous run" >&2;}
+      ac_suggest_removing_cache=: ;;
+    ,);;
+    *)
+      if test "x$ac_old_val" != "x$ac_new_val"; then
+        { echo "$as_me:910: WARNING: \`$ac_var' has changed since the previous run:" >&5
+echo "$as_me: WARNING: \`$ac_var' has changed since the previous run:" >&2;}
+        { echo "$as_me:912: WARNING:   former value:  $ac_old_val" >&5
+echo "$as_me: WARNING:   former value:  $ac_old_val" >&2;}
+        { echo "$as_me:914: WARNING:   current value: $ac_new_val" >&5
+echo "$as_me: WARNING:   current value: $ac_new_val" >&2;}
+        ac_suggest_removing_cache=:
+      fi;;
+  esac
+done
+if $ac_suggest_removing_cache; then
+  { echo "$as_me:921: WARNING: changes in the environment can compromise the build" >&5
+echo "$as_me: WARNING: changes in the environment can compromise the build" >&2;}
+  { echo "$as_me:923: WARNING: consider removing $cache_file and starting over" >&5
+echo "$as_me: WARNING: consider removing $cache_file and starting over" >&2;}
+fi
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+case `echo "testing\c"; echo 1,2,3`,`echo -n testing; echo 1,2,3` in
+  *c*,-n*) ECHO_N= ECHO_C='
+' ECHO_T='	' ;;
+  *c*,*  ) ECHO_N=-n ECHO_C= ECHO_T= ;;
+  *)      ECHO_N= ECHO_C='\c' ECHO_T= ;;
+esac
+echo "#! $SHELL" >conftest.sh
+echo  "exit 0"   >>conftest.sh
+chmod +x conftest.sh
+if { (echo "$as_me:942: PATH=\".;.\"; conftest.sh") >&5
+  (PATH=".;."; conftest.sh) 2>&5
+  ac_status=$?
+  echo "$as_me:945: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  ac_path_separator=';'
+else
+  ac_path_separator=:
+fi
+PATH_SEPARATOR="$ac_path_separator"
+rm -f conftest.sh
+
+ac_config_headers="$ac_config_headers config.h"
+
+if test "$program_transform_name" = s,x,x,; then
+  program_transform_name=
+else
+  # Double any \ or $.  echo might interpret backslashes.
+  cat <<\EOF >conftest.sed
+s,\\,\\\\,g; s,\$,$$,g
+EOF
+  program_transform_name=`echo $program_transform_name | sed -f conftest.sed`
+  rm -f conftest.sed
+fi
+test "$program_prefix" != NONE &&
+  program_transform_name="s,^,${program_prefix},;$program_transform_name"
+# Use a double $ so make ignores it.
+test "$program_suffix" != NONE &&
+  program_transform_name="s,\$\$,${program_suffix},;$program_transform_name"
+
+# sed with no file args requires a program.
+test -z "$program_transform_name" && program_transform_name="s,x,x,"
+
+PACKAGE=mgpp
+VERSION=1.3
+
+cat >>confdefs.h <<EOF
+#define PACKAGE "$PACKAGE"
+EOF
+
+cat >>confdefs.h <<EOF
+#define VERSION "$VERSION"
+EOF
+
+ac_ext=cc
+ac_cpp='$CXXCPP $CPPFLAGS'
+ac_compile='$CXX -c $CXXFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CXX -o conftest$ac_exeext $CXXFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_cxx_compiler_gnu
+if test -n "$ac_tool_prefix"; then
+  for ac_prog in $CCC g++ c++ gpp aCC CC cxx cc++ cl KCC RCC xlC_r xlC
+  do
+    # Extract the first word of "$ac_tool_prefix$ac_prog", so it can be a program name with args.
+set dummy $ac_tool_prefix$ac_prog; ac_word=$2
+echo "$as_me:996: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CXX+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CXX"; then
+  ac_cv_prog_CXX="$CXX" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CXX="$ac_tool_prefix$ac_prog"
+echo "$as_me:1011: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CXX=$ac_cv_prog_CXX
+if test -n "$CXX"; then
+  echo "$as_me:1019: result: $CXX" >&5
+echo "${ECHO_T}$CXX" >&6
+else
+  echo "$as_me:1022: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+    test -n "$CXX" && break
+  done
+fi
+if test -z "$CXX"; then
+  ac_ct_CXX=$CXX
+  for ac_prog in $CCC g++ c++ gpp aCC CC cxx cc++ cl KCC RCC xlC_r xlC
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1035: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CXX+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CXX"; then
+  ac_cv_prog_ac_ct_CXX="$ac_ct_CXX" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CXX="$ac_prog"
+echo "$as_me:1050: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CXX=$ac_cv_prog_ac_ct_CXX
+if test -n "$ac_ct_CXX"; then
+  echo "$as_me:1058: result: $ac_ct_CXX" >&5
+echo "${ECHO_T}$ac_ct_CXX" >&6
+else
+  echo "$as_me:1061: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$ac_ct_CXX" && break
+done
+test -n "$ac_ct_CXX" || ac_ct_CXX="g++"
+
+  CXX=$ac_ct_CXX
+fi
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 1073 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+ac_clean_files_save=$ac_clean_files
+ac_clean_files="$ac_clean_files a.out a.exe"
+# Try to create an executable without -o first, disregard a.out.
+# It will help us diagnose broken compiler, and finding out an intuition
+# of exeext.
+echo "$as_me:1089: checking for C++ compiler default output" >&5
+echo $ECHO_N "checking for C++ compiler default output... $ECHO_C" >&6
+ac_link_default=`echo "$ac_link" | sed 's/ -o *conftest[^ ]*//'`
+if { (eval echo "$as_me:1092: \"$ac_link_default\"") >&5
+  (eval $ac_link_default) 2>&5
+  ac_status=$?
+  echo "$as_me:1095: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  for ac_file in `ls a.exe conftest.exe a.* conftest conftest.* 2>/dev/null`; do
+  case $ac_file in
+    *.$ac_ext | *.o | *.obj | *.xcoff | *.tds | *.d | *.pdb ) ;;
+    a.out ) # We found the default executable, but exeext='' is most
+            # certainly right.
+            break;;
+    *.* ) ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'`
+          export ac_cv_exeext
+          break;;
+    * ) break;;
+  esac
+done
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+{ { echo "$as_me:1112: error: C++ compiler cannot create executables" >&5
+echo "$as_me: error: C++ compiler cannot create executables" >&2;}
+   { (exit 77); exit 77; }; }
+fi
+
+ac_exeext=$ac_cv_exeext
+echo "$as_me:1118: result: $ac_file" >&5
+echo "${ECHO_T}$ac_file" >&6
+
+# Check the compiler produces executables we can run.  If not, either
+# the compiler is broken, or we cross compile.
+echo "$as_me:1123: checking whether the C++ compiler works" >&5
+echo $ECHO_N "checking whether the C++ compiler works... $ECHO_C" >&6
+# FIXME: These cross compiler hacks should be removed for Autoconf 3.0
+# If not cross compiling, check that we can run a simple program.
+if test "$cross_compiling" != yes; then
+  if { ac_try='./$ac_file'
+  { (eval echo "$as_me:1129: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1132: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+    cross_compiling=no
+  else
+    if test "$cross_compiling" = maybe; then
+	cross_compiling=yes
+    else
+	{ { echo "$as_me:1139: error: cannot run C++ compiled programs.
+If you meant to cross compile, use \`--host'." >&5
+echo "$as_me: error: cannot run C++ compiled programs.
+If you meant to cross compile, use \`--host'." >&2;}
+   { (exit 1); exit 1; }; }
+    fi
+  fi
+fi
+echo "$as_me:1147: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+
+rm -f a.out a.exe conftest$ac_cv_exeext
+ac_clean_files=$ac_clean_files_save
+# Check the compiler produces executables we can run.  If not, either
+# the compiler is broken, or we cross compile.
+echo "$as_me:1154: checking whether we are cross compiling" >&5
+echo $ECHO_N "checking whether we are cross compiling... $ECHO_C" >&6
+echo "$as_me:1156: result: $cross_compiling" >&5
+echo "${ECHO_T}$cross_compiling" >&6
+
+echo "$as_me:1159: checking for executable suffix" >&5
+echo $ECHO_N "checking for executable suffix... $ECHO_C" >&6
+if { (eval echo "$as_me:1161: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:1164: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  # If both `conftest.exe' and `conftest' are `present' (well, observable)
+# catch `conftest.exe'.  For instance with Cygwin, `ls conftest' will
+# work properly (i.e., refer to `conftest.exe'), while it won't with
+# `rm'.
+for ac_file in `(ls conftest.exe; ls conftest; ls conftest.*) 2>/dev/null`; do
+  case $ac_file in
+    *.$ac_ext | *.o | *.obj | *.xcoff | *.tds | *.d | *.pdb ) ;;
+    *.* ) ac_cv_exeext=`expr "$ac_file" : '[^.]*\(\..*\)'`
+          export ac_cv_exeext
+          break;;
+    * ) break;;
+  esac
+done
+else
+  { { echo "$as_me:1180: error: cannot compute EXEEXT: cannot compile and link" >&5
+echo "$as_me: error: cannot compute EXEEXT: cannot compile and link" >&2;}
+   { (exit 1); exit 1; }; }
+fi
+
+rm -f conftest$ac_cv_exeext
+echo "$as_me:1186: result: $ac_cv_exeext" >&5
+echo "${ECHO_T}$ac_cv_exeext" >&6
+
+rm -f conftest.$ac_ext
+EXEEXT=$ac_cv_exeext
+ac_exeext=$EXEEXT
+echo "$as_me:1192: checking for object suffix" >&5
+echo $ECHO_N "checking for object suffix... $ECHO_C" >&6
+if test "${ac_cv_objext+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1198 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.o conftest.obj
+if { (eval echo "$as_me:1210: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1213: \$? = $ac_status" >&5
+  (exit $ac_status); }; then
+  for ac_file in `(ls conftest.o conftest.obj; ls conftest.*) 2>/dev/null`; do
+  case $ac_file in
+    *.$ac_ext | *.xcoff | *.tds | *.d | *.pdb ) ;;
+    *) ac_cv_objext=`expr "$ac_file" : '.*\.\(.*\)'`
+       break;;
+  esac
+done
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+{ { echo "$as_me:1225: error: cannot compute OBJEXT: cannot compile" >&5
+echo "$as_me: error: cannot compute OBJEXT: cannot compile" >&2;}
+   { (exit 1); exit 1; }; }
+fi
+
+rm -f conftest.$ac_cv_objext conftest.$ac_ext
+fi
+echo "$as_me:1232: result: $ac_cv_objext" >&5
+echo "${ECHO_T}$ac_cv_objext" >&6
+OBJEXT=$ac_cv_objext
+ac_objext=$OBJEXT
+echo "$as_me:1236: checking whether we are using the GNU C++ compiler" >&5
+echo $ECHO_N "checking whether we are using the GNU C++ compiler... $ECHO_C" >&6
+if test "${ac_cv_cxx_compiler_gnu+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1242 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+#ifndef __GNUC__
+       choke me
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1257: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1260: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1263: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1266: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_compiler_gnu=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_compiler_gnu=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+ac_cv_cxx_compiler_gnu=$ac_compiler_gnu
+
+fi
+echo "$as_me:1278: result: $ac_cv_cxx_compiler_gnu" >&5
+echo "${ECHO_T}$ac_cv_cxx_compiler_gnu" >&6
+GXX=`test $ac_compiler_gnu = yes && echo yes`
+ac_test_CXXFLAGS=${CXXFLAGS+set}
+ac_save_CXXFLAGS=$CXXFLAGS
+CXXFLAGS="-g"
+echo "$as_me:1284: checking whether $CXX accepts -g" >&5
+echo $ECHO_N "checking whether $CXX accepts -g... $ECHO_C" >&6
+if test "${ac_cv_prog_cxx_g+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1290 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1302: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1305: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1308: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1311: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cxx_g=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_prog_cxx_g=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:1321: result: $ac_cv_prog_cxx_g" >&5
+echo "${ECHO_T}$ac_cv_prog_cxx_g" >&6
+if test "$ac_test_CXXFLAGS" = set; then
+  CXXFLAGS=$ac_save_CXXFLAGS
+elif test $ac_cv_prog_cxx_g = yes; then
+  if test "$GXX" = yes; then
+    CXXFLAGS="-g -O2"
+  else
+    CXXFLAGS="-g"
+  fi
+else
+  if test "$GXX" = yes; then
+    CXXFLAGS="-O2"
+  else
+    CXXFLAGS=
+  fi
+fi
+for ac_declaration in \
+   ''\
+   '#include <stdlib.h>' \
+   'extern "C" void std::exit (int) throw (); using std::exit;' \
+   'extern "C" void std::exit (int); using std::exit;' \
+   'extern "C" void exit (int) throw ();' \
+   'extern "C" void exit (int);' \
+   'void exit (int);'
+do
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1348 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1361: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1364: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1367: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1370: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+continue
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1380 "configure"
+#include "confdefs.h"
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1392: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1395: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1398: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1401: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+echo '#ifdef __cplusplus' >>confdefs.h
+echo $ac_declaration      >>confdefs.h
+echo '#endif'             >>confdefs.h
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+for ac_prog in mawk gawk nawk awk
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1424: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_AWK+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$AWK"; then
+  ac_cv_prog_AWK="$AWK" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_AWK="$ac_prog"
+echo "$as_me:1439: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+AWK=$ac_cv_prog_AWK
+if test -n "$AWK"; then
+  echo "$as_me:1447: result: $AWK" >&5
+echo "${ECHO_T}$AWK" >&6
+else
+  echo "$as_me:1450: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$AWK" && break
+done
+
+for ac_prog in 'bison -y' byacc
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1461: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_YACC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$YACC"; then
+  ac_cv_prog_YACC="$YACC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_YACC="$ac_prog"
+echo "$as_me:1476: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+YACC=$ac_cv_prog_YACC
+if test -n "$YACC"; then
+  echo "$as_me:1484: result: $YACC" >&5
+echo "${ECHO_T}$YACC" >&6
+else
+  echo "$as_me:1487: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$YACC" && break
+done
+test -n "$YACC" || YACC="yacc"
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}gcc", so it can be a program name with args.
+set dummy ${ac_tool_prefix}gcc; ac_word=$2
+echo "$as_me:1503: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CC="${ac_tool_prefix}gcc"
+echo "$as_me:1518: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1526: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1529: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$ac_cv_prog_CC"; then
+  ac_ct_CC=$CC
+  # Extract the first word of "gcc", so it can be a program name with args.
+set dummy gcc; ac_word=$2
+echo "$as_me:1538: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CC"; then
+  ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CC="gcc"
+echo "$as_me:1553: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CC=$ac_cv_prog_ac_ct_CC
+if test -n "$ac_ct_CC"; then
+  echo "$as_me:1561: result: $ac_ct_CC" >&5
+echo "${ECHO_T}$ac_ct_CC" >&6
+else
+  echo "$as_me:1564: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  CC=$ac_ct_CC
+else
+  CC="$ac_cv_prog_CC"
+fi
+
+if test -z "$CC"; then
+  if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}cc", so it can be a program name with args.
+set dummy ${ac_tool_prefix}cc; ac_word=$2
+echo "$as_me:1577: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CC="${ac_tool_prefix}cc"
+echo "$as_me:1592: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1600: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1603: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$ac_cv_prog_CC"; then
+  ac_ct_CC=$CC
+  # Extract the first word of "cc", so it can be a program name with args.
+set dummy cc; ac_word=$2
+echo "$as_me:1612: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CC"; then
+  ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CC="cc"
+echo "$as_me:1627: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CC=$ac_cv_prog_ac_ct_CC
+if test -n "$ac_ct_CC"; then
+  echo "$as_me:1635: result: $ac_ct_CC" >&5
+echo "${ECHO_T}$ac_ct_CC" >&6
+else
+  echo "$as_me:1638: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  CC=$ac_ct_CC
+else
+  CC="$ac_cv_prog_CC"
+fi
+
+fi
+if test -z "$CC"; then
+  # Extract the first word of "cc", so it can be a program name with args.
+set dummy cc; ac_word=$2
+echo "$as_me:1651: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_prog_rejected=no
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+if test "$ac_dir/$ac_word" = "/usr/ucb/cc"; then
+  ac_prog_rejected=yes
+  continue
+fi
+ac_cv_prog_CC="cc"
+echo "$as_me:1671: found $ac_dir/$ac_word" >&5
+break
+done
+
+if test $ac_prog_rejected = yes; then
+  # We found a bogon in the path, so make sure we never use it.
+  set dummy $ac_cv_prog_CC
+  shift
+  if test $# != 0; then
+    # We chose a different compiler from the bogus one.
+    # However, it has the same basename, so the bogon will be chosen
+    # first if we set CC to just the basename; use the full file name.
+    shift
+    set dummy "$ac_dir/$ac_word" ${1+"$@"}
+    shift
+    ac_cv_prog_CC="$@"
+  fi
+fi
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1693: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1696: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$CC"; then
+  if test -n "$ac_tool_prefix"; then
+  for ac_prog in cl
+  do
+    # Extract the first word of "$ac_tool_prefix$ac_prog", so it can be a program name with args.
+set dummy $ac_tool_prefix$ac_prog; ac_word=$2
+echo "$as_me:1707: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$CC"; then
+  ac_cv_prog_CC="$CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_CC="$ac_tool_prefix$ac_prog"
+echo "$as_me:1722: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+CC=$ac_cv_prog_CC
+if test -n "$CC"; then
+  echo "$as_me:1730: result: $CC" >&5
+echo "${ECHO_T}$CC" >&6
+else
+  echo "$as_me:1733: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+    test -n "$CC" && break
+  done
+fi
+if test -z "$CC"; then
+  ac_ct_CC=$CC
+  for ac_prog in cl
+do
+  # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
+echo "$as_me:1746: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_CC"; then
+  ac_cv_prog_ac_ct_CC="$ac_ct_CC" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_CC="$ac_prog"
+echo "$as_me:1761: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+ac_ct_CC=$ac_cv_prog_ac_ct_CC
+if test -n "$ac_ct_CC"; then
+  echo "$as_me:1769: result: $ac_ct_CC" >&5
+echo "${ECHO_T}$ac_ct_CC" >&6
+else
+  echo "$as_me:1772: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  test -n "$ac_ct_CC" && break
+done
+
+  CC=$ac_ct_CC
+fi
+
+fi
+
+test -z "$CC" && { { echo "$as_me:1784: error: no acceptable cc found in \$PATH" >&5
+echo "$as_me: error: no acceptable cc found in \$PATH" >&2;}
+   { (exit 1); exit 1; }; }
+
+echo "$as_me:1788: checking whether we are using the GNU C compiler" >&5
+echo $ECHO_N "checking whether we are using the GNU C compiler... $ECHO_C" >&6
+if test "${ac_cv_c_compiler_gnu+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1794 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+#ifndef __GNUC__
+       choke me
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1809: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1812: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1815: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1818: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_compiler_gnu=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_compiler_gnu=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+ac_cv_c_compiler_gnu=$ac_compiler_gnu
+
+fi
+echo "$as_me:1830: result: $ac_cv_c_compiler_gnu" >&5
+echo "${ECHO_T}$ac_cv_c_compiler_gnu" >&6
+GCC=`test $ac_compiler_gnu = yes && echo yes`
+ac_test_CFLAGS=${CFLAGS+set}
+ac_save_CFLAGS=$CFLAGS
+CFLAGS="-g"
+echo "$as_me:1836: checking whether $CC accepts -g" >&5
+echo $ECHO_N "checking whether $CC accepts -g... $ECHO_C" >&6
+if test "${ac_cv_prog_cc_g+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1842 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1854: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1857: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1860: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1863: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cc_g=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_prog_cc_g=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:1873: result: $ac_cv_prog_cc_g" >&5
+echo "${ECHO_T}$ac_cv_prog_cc_g" >&6
+if test "$ac_test_CFLAGS" = set; then
+  CFLAGS=$ac_save_CFLAGS
+elif test $ac_cv_prog_cc_g = yes; then
+  if test "$GCC" = yes; then
+    CFLAGS="-g -O2"
+  else
+    CFLAGS="-g"
+  fi
+else
+  if test "$GCC" = yes; then
+    CFLAGS="-O2"
+  else
+    CFLAGS=
+  fi
+fi
+# Some people use a C++ compiler to compile C.  Since we use `exit',
+# in C++ we need to declare it.  In case someone uses the same compiler
+# for both compiling C and C++ we need to have the C++ compiler decide
+# the declaration of exit, since it's the most demanding environment.
+cat >conftest.$ac_ext <<_ACEOF
+#ifndef __cplusplus
+  choke me
+#endif
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1900: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1903: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1906: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1909: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  for ac_declaration in \
+   ''\
+   '#include <stdlib.h>' \
+   'extern "C" void std::exit (int) throw (); using std::exit;' \
+   'extern "C" void std::exit (int); using std::exit;' \
+   'extern "C" void exit (int) throw ();' \
+   'extern "C" void exit (int);' \
+   'void exit (int);'
+do
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1921 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1934: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1937: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1940: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1943: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+continue
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  cat >conftest.$ac_ext <<_ACEOF
+#line 1953 "configure"
+#include "confdefs.h"
+$ac_declaration
+int
+main ()
+{
+exit (42);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:1965: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:1968: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:1971: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:1974: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+echo '#ifdef __cplusplus' >>confdefs.h
+echo $ac_declaration      >>confdefs.h
+echo '#endif'             >>confdefs.h
+
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+ac_aux_dir=
+for ac_dir in $srcdir $srcdir/.. $srcdir/../..; do
+  if test -f $ac_dir/install-sh; then
+    ac_aux_dir=$ac_dir
+    ac_install_sh="$ac_aux_dir/install-sh -c"
+    break
+  elif test -f $ac_dir/install.sh; then
+    ac_aux_dir=$ac_dir
+    ac_install_sh="$ac_aux_dir/install.sh -c"
+    break
+  elif test -f $ac_dir/shtool; then
+    ac_aux_dir=$ac_dir
+    ac_install_sh="$ac_aux_dir/shtool install -c"
+    break
+  fi
+done
+if test -z "$ac_aux_dir"; then
+  { { echo "$as_me:2015: error: cannot find install-sh or install.sh in $srcdir $srcdir/.. $srcdir/../.." >&5
+echo "$as_me: error: cannot find install-sh or install.sh in $srcdir $srcdir/.. $srcdir/../.." >&2;}
+   { (exit 1); exit 1; }; }
+fi
+ac_config_guess="$SHELL $ac_aux_dir/config.guess"
+ac_config_sub="$SHELL $ac_aux_dir/config.sub"
+ac_configure="$SHELL $ac_aux_dir/configure" # This should be Cygnus configure.
+
+# Find a good install program.  We prefer a C program (faster),
+# so one script is as good as another.  But avoid the broken or
+# incompatible versions:
+# SysV /etc/install, /usr/sbin/install
+# SunOS /usr/etc/install
+# IRIX /sbin/install
+# AIX /bin/install
+# AmigaOS /C/install, which installs bootblocks on floppy discs
+# AIX 4 /usr/bin/installbsd, which doesn't work without a -g flag
+# AFS /usr/afsws/bin/install, which mishandles nonexistent args
+# SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff"
+# ./install, which can be erroneously created by make from ./install.sh.
+echo "$as_me:2035: checking for a BSD compatible install" >&5
+echo $ECHO_N "checking for a BSD compatible install... $ECHO_C" >&6
+if test -z "$INSTALL"; then
+if test "${ac_cv_path_install+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+    ac_save_IFS=$IFS; IFS=$ac_path_separator
+  for ac_dir in $PATH; do
+    IFS=$ac_save_IFS
+    # Account for people who put trailing slashes in PATH elements.
+    case $ac_dir/ in
+    / | ./ | .// | /cC/* \
+    | /etc/* | /usr/sbin/* | /usr/etc/* | /sbin/* | /usr/afsws/bin/* \
+    | /usr/ucb/* ) ;;
+    *)
+      # OSF1 and SCO ODT 3.0 have their own names for install.
+      # Don't use installbsd from OSF since it installs stuff as root
+      # by default.
+      for ac_prog in ginstall scoinst install; do
+        if $as_executable_p "$ac_dir/$ac_prog"; then
+	  if test $ac_prog = install &&
+            grep dspmsg "$ac_dir/$ac_prog" >/dev/null 2>&1; then
+	    # AIX install.  It has an incompatible calling convention.
+	    :
+	  elif test $ac_prog = install &&
+	    grep pwplus "$ac_dir/$ac_prog" >/dev/null 2>&1; then
+	    # program-specific install script used by HP pwplus--don't use.
+	    :
+	  else
+	    ac_cv_path_install="$ac_dir/$ac_prog -c"
+	    break 2
+	  fi
+	fi
+      done
+      ;;
+    esac
+  done
+
+fi
+  if test "${ac_cv_path_install+set}" = set; then
+    INSTALL=$ac_cv_path_install
+  else
+    # As a last resort, use the slow shell script.  We don't cache a
+    # path for INSTALL within a source directory, because that will
+    # break other packages using the cache if that directory is
+    # removed, or if the path is relative.
+    INSTALL=$ac_install_sh
+  fi
+fi
+echo "$as_me:2084: result: $INSTALL" >&5
+echo "${ECHO_T}$INSTALL" >&6
+
+# Use test -z because SunOS4 sh mishandles braces in ${var-val}.
+# It thinks the first close brace ends the variable substitution.
+test -z "$INSTALL_PROGRAM" && INSTALL_PROGRAM='${INSTALL}'
+
+test -z "$INSTALL_SCRIPT" && INSTALL_SCRIPT='${INSTALL}'
+
+test -z "$INSTALL_DATA" && INSTALL_DATA='${INSTALL} -m 644'
+
+echo "$as_me:2095: checking whether ln -s works" >&5
+echo $ECHO_N "checking whether ln -s works... $ECHO_C" >&6
+LN_S=$as_ln_s
+if test "$LN_S" = "ln -s"; then
+  echo "$as_me:2099: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+else
+  echo "$as_me:2102: result: no, using $LN_S" >&5
+echo "${ECHO_T}no, using $LN_S" >&6
+fi
+
+echo "$as_me:2106: checking whether ${MAKE-make} sets \${MAKE}" >&5
+echo $ECHO_N "checking whether ${MAKE-make} sets \${MAKE}... $ECHO_C" >&6
+set dummy ${MAKE-make}; ac_make=`echo "$2" | sed 'y,./+-,__p_,'`
+if eval "test \"\${ac_cv_prog_make_${ac_make}_set+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.make <<\EOF
+all:
+	@echo 'ac_maketemp="${MAKE}"'
+EOF
+# GNU make sometimes prints "make[1]: Entering...", which would confuse us.
+eval `${MAKE-make} -f conftest.make 2>/dev/null | grep temp=`
+if test -n "$ac_maketemp"; then
+  eval ac_cv_prog_make_${ac_make}_set=yes
+else
+  eval ac_cv_prog_make_${ac_make}_set=no
+fi
+rm -f conftest.make
+fi
+if eval "test \"`echo '$ac_cv_prog_make_'${ac_make}_set`\" = yes"; then
+  echo "$as_me:2126: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  SET_MAKE=
+else
+  echo "$as_me:2130: result: no" >&5
+echo "${ECHO_T}no" >&6
+  SET_MAKE="MAKE=${MAKE-make}"
+fi
+
+if test -n "$ac_tool_prefix"; then
+  # Extract the first word of "${ac_tool_prefix}ranlib", so it can be a program name with args.
+set dummy ${ac_tool_prefix}ranlib; ac_word=$2
+echo "$as_me:2138: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_RANLIB+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$RANLIB"; then
+  ac_cv_prog_RANLIB="$RANLIB" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_RANLIB="${ac_tool_prefix}ranlib"
+echo "$as_me:2153: found $ac_dir/$ac_word" >&5
+break
+done
+
+fi
+fi
+RANLIB=$ac_cv_prog_RANLIB
+if test -n "$RANLIB"; then
+  echo "$as_me:2161: result: $RANLIB" >&5
+echo "${ECHO_T}$RANLIB" >&6
+else
+  echo "$as_me:2164: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+fi
+if test -z "$ac_cv_prog_RANLIB"; then
+  ac_ct_RANLIB=$RANLIB
+  # Extract the first word of "ranlib", so it can be a program name with args.
+set dummy ranlib; ac_word=$2
+echo "$as_me:2173: checking for $ac_word" >&5
+echo $ECHO_N "checking for $ac_word... $ECHO_C" >&6
+if test "${ac_cv_prog_ac_ct_RANLIB+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test -n "$ac_ct_RANLIB"; then
+  ac_cv_prog_ac_ct_RANLIB="$ac_ct_RANLIB" # Let the user override the test.
+else
+  ac_save_IFS=$IFS; IFS=$ac_path_separator
+ac_dummy="$PATH"
+for ac_dir in $ac_dummy; do
+  IFS=$ac_save_IFS
+  test -z "$ac_dir" && ac_dir=.
+  $as_executable_p "$ac_dir/$ac_word" || continue
+ac_cv_prog_ac_ct_RANLIB="ranlib"
+echo "$as_me:2188: found $ac_dir/$ac_word" >&5
+break
+done
+
+  test -z "$ac_cv_prog_ac_ct_RANLIB" && ac_cv_prog_ac_ct_RANLIB=":"
+fi
+fi
+ac_ct_RANLIB=$ac_cv_prog_ac_ct_RANLIB
+if test -n "$ac_ct_RANLIB"; then
+  echo "$as_me:2197: result: $ac_ct_RANLIB" >&5
+echo "${ECHO_T}$ac_ct_RANLIB" >&6
+else
+  echo "$as_me:2200: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+
+  RANLIB=$ac_ct_RANLIB
+else
+  RANLIB="$ac_cv_prog_RANLIB"
+fi
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+echo "$as_me:2214: checking how to run the C preprocessor" >&5
+echo $ECHO_N "checking how to run the C preprocessor... $ECHO_C" >&6
+# On Suns, sometimes $CPP names a directory.
+if test -n "$CPP" && test -d "$CPP"; then
+  CPP=
+fi
+if test -z "$CPP"; then
+  if test "${ac_cv_prog_CPP+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+      # Double quotes because CPP needs to be expanded
+    for CPP in "$CC -E" "$CC -E -traditional-cpp" "/lib/cpp"
+    do
+      # break 2 since there is a loop in there.
+      ac_preproc_ok=false
+for ac_c_preproc_warn_flag in '' yes
+do
+  # Use a header file that comes with gcc, so configuring glibc
+  # with a fresh cross-compiler works.
+  # On the NeXT, cc -E runs the code through the compiler's parser,
+  # not just through cpp. "Syntax error" is here to catch this case.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2236 "configure"
+#include "confdefs.h"
+#include <assert.h>
+                     Syntax error
+_ACEOF
+if { (eval echo "$as_me:2241: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2247: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Broken: fails on valid input.
+continue
+fi
+rm -f conftest.err conftest.$ac_ext
+
+  # OK, works on sane cases.  Now check whether non-existent headers
+  # can be detected and how.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2270 "configure"
+#include "confdefs.h"
+#include <ac_nonexistent.h>
+_ACEOF
+if { (eval echo "$as_me:2274: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2280: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  # Broken: success on invalid input.
+continue
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Passes both tests.
+ac_preproc_ok=:
+break
+fi
+rm -f conftest.err conftest.$ac_ext
+
+done
+# Because of `break', _AC_PREPROC_IFELSE's cleaning code was skipped.
+rm -f conftest.err conftest.$ac_ext
+if $ac_preproc_ok; then
+  break 2
+fi
+
+    done
+    ac_cv_prog_CPP=$CPP
+
+fi
+  CPP=$ac_cv_prog_CPP
+else
+  ac_cv_prog_CPP=$CPP
+fi
+echo "$as_me:2317: result: $CPP" >&5
+echo "${ECHO_T}$CPP" >&6
+ac_preproc_ok=false
+for ac_c_preproc_warn_flag in '' yes
+do
+  # Use a header file that comes with gcc, so configuring glibc
+  # with a fresh cross-compiler works.
+  # On the NeXT, cc -E runs the code through the compiler's parser,
+  # not just through cpp. "Syntax error" is here to catch this case.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2327 "configure"
+#include "confdefs.h"
+#include <assert.h>
+                     Syntax error
+_ACEOF
+if { (eval echo "$as_me:2332: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2338: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Broken: fails on valid input.
+continue
+fi
+rm -f conftest.err conftest.$ac_ext
+
+  # OK, works on sane cases.  Now check whether non-existent headers
+  # can be detected and how.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2361 "configure"
+#include "confdefs.h"
+#include <ac_nonexistent.h>
+_ACEOF
+if { (eval echo "$as_me:2365: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2371: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  # Broken: success on invalid input.
+continue
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  # Passes both tests.
+ac_preproc_ok=:
+break
+fi
+rm -f conftest.err conftest.$ac_ext
+
+done
+# Because of `break', _AC_PREPROC_IFELSE's cleaning code was skipped.
+rm -f conftest.err conftest.$ac_ext
+if $ac_preproc_ok; then
+  :
+else
+  { { echo "$as_me:2399: error: C preprocessor \"$CPP\" fails sanity check" >&5
+echo "$as_me: error: C preprocessor \"$CPP\" fails sanity check" >&2;}
+   { (exit 1); exit 1; }; }
+fi
+
+ac_ext=c
+ac_cpp='$CPP $CPPFLAGS'
+ac_compile='$CC -c $CFLAGS $CPPFLAGS conftest.$ac_ext >&5'
+ac_link='$CC -o conftest$ac_exeext $CFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS >&5'
+ac_compiler_gnu=$ac_cv_c_compiler_gnu
+
+echo "$as_me:2410: checking for AIX" >&5
+echo $ECHO_N "checking for AIX... $ECHO_C" >&6
+cat >conftest.$ac_ext <<_ACEOF
+#line 2413 "configure"
+#include "confdefs.h"
+#ifdef _AIX
+  yes
+#endif
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "yes" >/dev/null 2>&1; then
+  echo "$as_me:2422: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+cat >>confdefs.h <<\EOF
+#define _ALL_SOURCE 1
+EOF
+
+else
+  echo "$as_me:2429: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+rm -f conftest*
+
+echo "$as_me:2434: checking for POSIXized ISC" >&5
+echo $ECHO_N "checking for POSIXized ISC... $ECHO_C" >&6
+if test -d /etc/conf/kconfig.d &&
+   grep _POSIX_VERSION /usr/include/sys/unistd.h >/dev/null 2>&1
+then
+  echo "$as_me:2439: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  ISC=yes # If later tests want to check for ISC.
+
+cat >>confdefs.h <<\EOF
+#define _POSIX_SOURCE 1
+EOF
+
+  if test "$GCC" = yes; then
+    CC="$CC -posix"
+  else
+    CC="$CC -Xp"
+  fi
+else
+  echo "$as_me:2453: result: no" >&5
+echo "${ECHO_T}no" >&6
+  ISC=
+fi
+
+echo "$as_me:2458: checking for minix/config.h" >&5
+echo $ECHO_N "checking for minix/config.h... $ECHO_C" >&6
+if test "${ac_cv_header_minix_config_h+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2464 "configure"
+#include "confdefs.h"
+#include <minix/config.h>
+_ACEOF
+if { (eval echo "$as_me:2468: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2474: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_minix_config_h=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_minix_config_h=no
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:2493: result: $ac_cv_header_minix_config_h" >&5
+echo "${ECHO_T}$ac_cv_header_minix_config_h" >&6
+if test $ac_cv_header_minix_config_h = yes; then
+  MINIX=yes
+else
+  MINIX=
+fi
+
+if test "$MINIX" = yes; then
+
+cat >>confdefs.h <<\EOF
+#define _POSIX_SOURCE 1
+EOF
+
+cat >>confdefs.h <<\EOF
+#define _POSIX_1_SOURCE 2
+EOF
+
+cat >>confdefs.h <<\EOF
+#define _MINIX 1
+EOF
+
+fi
+
+echo "$as_me:2517: checking for ${CC-cc} option to accept ANSI C" >&5
+echo $ECHO_N "checking for ${CC-cc} option to accept ANSI C... $ECHO_C" >&6
+if test "${ac_cv_prog_cc_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_cv_prog_cc_stdc=no
+ac_save_CFLAGS="$CFLAGS"
+# Don't try gcc -ansi; that turns off useful extensions and
+# breaks some systems' header files.
+# AIX			-qlanglvl=ansi
+# Ultrix and OSF/1	-std1
+# HP-UX			-Aa -D_HPUX_SOURCE
+# SVR4			-Xc
+for ac_arg in "" -qlanglvl=ansi -std1 "-Aa -D_HPUX_SOURCE" -Xc
+do
+  CFLAGS="$ac_save_CFLAGS $ac_arg"
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2534 "configure"
+#include "confdefs.h"
+#if !defined(__STDC__) || __STDC__ != 1
+choke me
+#endif
+
+int
+main ()
+{
+int test (int i, double x);
+struct s1 {int (*f) (int a);};
+struct s2 {int (*f) (double a);};
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2551: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2554: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2557: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2560: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cc_stdc="$ac_arg"; break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+CFLAGS="$ac_save_CFLAGS"
+
+fi
+
+echo "$as_me:2573: result: $ac_cv_prog_cc_stdc" >&5
+echo "${ECHO_T}$ac_cv_prog_cc_stdc" >&6
+case "x$ac_cv_prog_cc_stdc" in
+  x|xno) ;;
+  *) CC="$CC $ac_cv_prog_cc_stdc" ;;
+esac
+
+echo "$as_me:2580: checking for function prototypes" >&5
+echo $ECHO_N "checking for function prototypes... $ECHO_C" >&6
+if test "$ac_cv_prog_cc_stdc" != no; then
+  echo "$as_me:2583: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  cat >>confdefs.h <<\EOF
+#define PROTOTYPES 1
+EOF
+
+  U= ANSI2KNR=
+else
+  echo "$as_me:2591: result: no" >&5
+echo "${ECHO_T}no" >&6
+  U=_ ANSI2KNR=ansi2knr
+fi
+
+echo "$as_me:2596: checking for $CC option to accept ANSI C" >&5
+echo $ECHO_N "checking for $CC option to accept ANSI C... $ECHO_C" >&6
+if test "${ac_cv_prog_cc_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_cv_prog_cc_stdc=no
+ac_save_CC=$CC
+cat >conftest.$ac_ext <<_ACEOF
+#line 2604 "configure"
+#include "confdefs.h"
+#include <stdarg.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+/* Most of the following tests are stolen from RCS 5.7's src/conf.sh.  */
+struct buf { int x; };
+FILE * (*rcsopen) (struct buf *, struct stat *, int);
+static char *e (p, i)
+     char **p;
+     int i;
+{
+  return p[i];
+}
+static char *f (char * (*g) (char **, int), char **p, ...)
+{
+  char *s;
+  va_list v;
+  va_start (v,p);
+  s = g (p, va_arg (v,int));
+  va_end (v);
+  return s;
+}
+int test (int i, double x);
+struct s1 {int (*f) (int a);};
+struct s2 {int (*f) (double a);};
+int pairnames (int, char **, FILE *(*)(struct buf *, struct stat *, int), int, int);
+int argc;
+char **argv;
+int
+main ()
+{
+return f (e, argv, 0) != argv[0]  ||  f (e, argv, 1) != argv[1];
+  ;
+  return 0;
+}
+_ACEOF
+# Don't try gcc -ansi; that turns off useful extensions and
+# breaks some systems' header files.
+# AIX			-qlanglvl=ansi
+# Ultrix and OSF/1	-std1
+# HP-UX 10.20 and later	-Ae
+# HP-UX older versions	-Aa -D_HPUX_SOURCE
+# SVR4			-Xc -D__EXTENSIONS__
+for ac_arg in "" -qlanglvl=ansi -std1 -Ae "-Aa -D_HPUX_SOURCE" "-Xc -D__EXTENSIONS__"
+do
+  CC="$ac_save_CC $ac_arg"
+  rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2653: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2656: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2659: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2662: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_prog_cc_stdc=$ac_arg
+break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f conftest.$ac_objext
+done
+rm -f conftest.$ac_ext conftest.$ac_objext
+CC=$ac_save_CC
+
+fi
+
+case "x$ac_cv_prog_cc_stdc" in
+  x|xno)
+    echo "$as_me:2679: result: none needed" >&5
+echo "${ECHO_T}none needed" >&6 ;;
+  *)
+    echo "$as_me:2682: result: $ac_cv_prog_cc_stdc" >&5
+echo "${ECHO_T}$ac_cv_prog_cc_stdc" >&6
+    CC="$CC $ac_cv_prog_cc_stdc" ;;
+esac
+
+echo "$as_me:2687: checking for an ANSI C-conforming const" >&5
+echo $ECHO_N "checking for an ANSI C-conforming const... $ECHO_C" >&6
+if test "${ac_cv_c_const+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2693 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+/* FIXME: Include the comments suggested by Paul. */
+#ifndef __cplusplus
+  /* Ultrix mips cc rejects this.  */
+  typedef int charset[2];
+  const charset x;
+  /* SunOS 4.1.1 cc rejects this.  */
+  char const *const *ccp;
+  char **p;
+  /* NEC SVR4.0.2 mips cc rejects this.  */
+  struct point {int x, y;};
+  static struct point const zero = {0,0};
+  /* AIX XL C 1.02.0.0 rejects this.
+     It does not let you subtract one const X* pointer from another in
+     an arm of an if-expression whose if-part is not a constant
+     expression */
+  const char *g = "string";
+  ccp = &g + (g ? g-g : 0);
+  /* HPUX 7.0 cc rejects these. */
+  ++ccp;
+  p = (char**) ccp;
+  ccp = (char const *const *) p;
+  { /* SCO 3.2v4 cc rejects this.  */
+    char *t;
+    char const *s = 0 ? (char *) 0 : (char const *) 0;
+
+    *t++ = 0;
+  }
+  { /* Someone thinks the Sun supposedly-ANSI compiler will reject this.  */
+    int x[] = {25, 17};
+    const int *foo = &x[0];
+    ++foo;
+  }
+  { /* Sun SC1.0 ANSI compiler rejects this -- but not the above. */
+    typedef const int *iptr;
+    iptr p = 0;
+    ++p;
+  }
+  { /* AIX XL C 1.02.0.0 rejects this saying
+       "k.c", line 2.27: 1506-025 (S) Operand must be a modifiable lvalue. */
+    struct s { int j; const int *ap[3]; };
+    struct s *b; b->j = 5;
+  }
+  { /* ULTRIX-32 V3.1 (Rev 9) vcc rejects this */
+    const int foo = 10;
+  }
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2751: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2754: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2757: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2760: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_c_const=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_c_const=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:2770: result: $ac_cv_c_const" >&5
+echo "${ECHO_T}$ac_cv_c_const" >&6
+if test $ac_cv_c_const = no; then
+
+cat >>confdefs.h <<\EOF
+#define const
+EOF
+
+fi
+
+echo "$as_me:2780: checking for ANSI C header files" >&5
+echo $ECHO_N "checking for ANSI C header files... $ECHO_C" >&6
+if test "${ac_cv_header_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2786 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+#include <stdarg.h>
+#include <string.h>
+#include <float.h>
+
+_ACEOF
+if { (eval echo "$as_me:2794: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2800: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_stdc=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_stdc=no
+fi
+rm -f conftest.err conftest.$ac_ext
+
+if test $ac_cv_header_stdc = yes; then
+  # SunOS 4.x string.h does not declare mem*, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2822 "configure"
+#include "confdefs.h"
+#include <string.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "memchr" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # ISC 2.0.2 stdlib.h does not declare free, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2840 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "free" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # /bin/cc in Irix-4.0.5 gets non-ANSI ctype macros unless using -ansi.
+  if test "$cross_compiling" = yes; then
+  :
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2861 "configure"
+#include "confdefs.h"
+#include <ctype.h>
+#if ((' ' & 0x0FF) == 0x020)
+# define ISLOWER(c) ('a' <= (c) && (c) <= 'z')
+# define TOUPPER(c) (ISLOWER(c) ? 'A' + ((c) - 'a') : (c))
+#else
+# define ISLOWER(c) (('a' <= (c) && (c) <= 'i') \
+                     || ('j' <= (c) && (c) <= 'r') \
+                     || ('s' <= (c) && (c) <= 'z'))
+# define TOUPPER(c) (ISLOWER(c) ? ((c) | 0x40) : (c))
+#endif
+
+#define XOR(e, f) (((e) && !(f)) || (!(e) && (f)))
+int
+main ()
+{
+  int i;
+  for (i = 0; i < 256; i++)
+    if (XOR (islower (i), ISLOWER (i))
+        || toupper (i) != TOUPPER (i))
+      exit(2);
+  exit (0);
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:2887: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:2890: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:2892: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2895: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_header_stdc=no
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+fi
+echo "$as_me:2908: result: $ac_cv_header_stdc" >&5
+echo "${ECHO_T}$ac_cv_header_stdc" >&6
+if test $ac_cv_header_stdc = yes; then
+
+cat >>confdefs.h <<\EOF
+#define STDC_HEADERS 1
+EOF
+
+fi
+
+for ac_header in stdlib.h string.h memory.h strings.h inttypes.h unistd.h
+do
+ac_ac_Header=`echo "ac_cv_header_$ac_header" | $ac_tr_sh`
+echo "$as_me:2921: checking for $ac_header" >&5
+echo $ECHO_N "checking for $ac_header... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_Header+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2927 "configure"
+#include "confdefs.h"
+#include <$ac_header>
+_ACEOF
+if { (eval echo "$as_me:2931: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:2937: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  eval "$ac_ac_Header=yes"
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  eval "$ac_ac_Header=no"
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:2956: result: `eval echo '${'$ac_ac_Header'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_Header'}'`" >&6
+if test `eval echo '${'$ac_ac_Header'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_header" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+echo "$as_me:2966: checking for off_t" >&5
+echo $ECHO_N "checking for off_t... $ECHO_C" >&6
+if test "${ac_cv_type_off_t+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 2972 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+if ((off_t *) 0)
+  return 0;
+if (sizeof (off_t))
+  return 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:2987: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:2990: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:2993: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:2996: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_off_t=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_off_t=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3006: result: $ac_cv_type_off_t" >&5
+echo "${ECHO_T}$ac_cv_type_off_t" >&6
+if test $ac_cv_type_off_t = yes; then
+  :
+else
+
+cat >>confdefs.h <<EOF
+#define off_t long
+EOF
+
+fi
+
+echo "$as_me:3018: checking for size_t" >&5
+echo $ECHO_N "checking for size_t... $ECHO_C" >&6
+if test "${ac_cv_type_size_t+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3024 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+if ((size_t *) 0)
+  return 0;
+if (sizeof (size_t))
+  return 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3039: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3042: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3045: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3048: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_size_t=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_size_t=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3058: result: $ac_cv_type_size_t" >&5
+echo "${ECHO_T}$ac_cv_type_size_t" >&6
+if test $ac_cv_type_size_t = yes; then
+  :
+else
+
+cat >>confdefs.h <<EOF
+#define size_t unsigned
+EOF
+
+fi
+
+echo "$as_me:3070: checking whether time.h and sys/time.h may both be included" >&5
+echo $ECHO_N "checking whether time.h and sys/time.h may both be included... $ECHO_C" >&6
+if test "${ac_cv_header_time+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3076 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <sys/time.h>
+#include <time.h>
+
+int
+main ()
+{
+struct tm *tp;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3091: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3094: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3097: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3100: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_header_time=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_header_time=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3110: result: $ac_cv_header_time" >&5
+echo "${ECHO_T}$ac_cv_header_time" >&6
+if test $ac_cv_header_time = yes; then
+
+cat >>confdefs.h <<\EOF
+#define TIME_WITH_SYS_TIME 1
+EOF
+
+fi
+
+echo "$as_me:3120: checking whether struct tm is in sys/time.h or time.h" >&5
+echo $ECHO_N "checking whether struct tm is in sys/time.h or time.h... $ECHO_C" >&6
+if test "${ac_cv_struct_tm+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3126 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <time.h>
+
+int
+main ()
+{
+struct tm *tp; tp->tm_sec;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3140: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3143: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3146: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3149: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_struct_tm=time.h
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_struct_tm=sys/time.h
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3159: result: $ac_cv_struct_tm" >&5
+echo "${ECHO_T}$ac_cv_struct_tm" >&6
+if test $ac_cv_struct_tm = sys/time.h; then
+
+cat >>confdefs.h <<\EOF
+#define TM_IN_SYS_TIME 1
+EOF
+
+fi
+
+if test "$ac_cv_prog_cc_stdc" = '-Xc'; then
+cat >conftest.$ac_ext <<_ACEOF
+#line 3171 "configure"
+#include "confdefs.h"
+#include <stdio.h>
+
+int
+main ()
+{
+ FILE *rubbish = popen("yes", "r");
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3184: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3187: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3190: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3193: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+CC="`echo $CC | sed 's/-Xc/-Xa/'`"    ac_cv_prog_cc_stdc='-Xa'
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+
+echo "$as_me:3204: checking for main in -lm" >&5
+echo $ECHO_N "checking for main in -lm... $ECHO_C" >&6
+if test "${ac_cv_lib_m_main+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lm  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 3212 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+main ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3224: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3227: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3230: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3233: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_m_main=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_m_main=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:3244: result: $ac_cv_lib_m_main" >&5
+echo "${ECHO_T}$ac_cv_lib_m_main" >&6
+if test $ac_cv_lib_m_main = yes; then
+  cat >>confdefs.h <<EOF
+#define HAVE_LIBM 1
+EOF
+
+  LIBS="-lm $LIBS"
+
+fi
+
+ac_header_dirent=no
+for ac_hdr in dirent.h sys/ndir.h sys/dir.h ndir.h; do
+  ac_ac_Header=`echo "ac_cv_header_dirent_$ac_hdr" | $ac_tr_sh`
+echo "$as_me:3258: checking for $ac_hdr that defines DIR" >&5
+echo $ECHO_N "checking for $ac_hdr that defines DIR... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_Header+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3264 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <$ac_hdr>
+
+int
+main ()
+{
+DIR *dirp = 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:3278: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:3281: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:3284: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3287: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_Header=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_Header=no"
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:3297: result: `eval echo '${'$ac_ac_Header'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_Header'}'`" >&6
+if test `eval echo '${'$ac_ac_Header'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_hdr" | $ac_tr_cpp` 1
+EOF
+
+ac_header_dirent=$ac_hdr; break
+fi
+
+done
+# Two versions of opendir et al. are in -ldir and -lx on SCO Xenix.
+if test $ac_header_dirent = dirent.h; then
+  echo "$as_me:3310: checking for opendir in -ldir" >&5
+echo $ECHO_N "checking for opendir in -ldir... $ECHO_C" >&6
+if test "${ac_cv_lib_dir_opendir+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-ldir  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 3318 "configure"
+#include "confdefs.h"
+
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char opendir ();
+int
+main ()
+{
+opendir ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3337: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3340: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3343: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3346: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_dir_opendir=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_dir_opendir=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:3357: result: $ac_cv_lib_dir_opendir" >&5
+echo "${ECHO_T}$ac_cv_lib_dir_opendir" >&6
+if test $ac_cv_lib_dir_opendir = yes; then
+  LIBS="$LIBS -ldir"
+fi
+
+else
+  echo "$as_me:3364: checking for opendir in -lx" >&5
+echo $ECHO_N "checking for opendir in -lx... $ECHO_C" >&6
+if test "${ac_cv_lib_x_opendir+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lx  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 3372 "configure"
+#include "confdefs.h"
+
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char opendir ();
+int
+main ()
+{
+opendir ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3391: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3394: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3397: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3400: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_x_opendir=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_x_opendir=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:3411: result: $ac_cv_lib_x_opendir" >&5
+echo "${ECHO_T}$ac_cv_lib_x_opendir" >&6
+if test $ac_cv_lib_x_opendir = yes; then
+  LIBS="$LIBS -lx"
+fi
+
+fi
+
+echo "$as_me:3419: checking for ANSI C header files" >&5
+echo $ECHO_N "checking for ANSI C header files... $ECHO_C" >&6
+if test "${ac_cv_header_stdc+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3425 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+#include <stdarg.h>
+#include <string.h>
+#include <float.h>
+
+_ACEOF
+if { (eval echo "$as_me:3433: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:3439: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_stdc=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_stdc=no
+fi
+rm -f conftest.err conftest.$ac_ext
+
+if test $ac_cv_header_stdc = yes; then
+  # SunOS 4.x string.h does not declare mem*, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3461 "configure"
+#include "confdefs.h"
+#include <string.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "memchr" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # ISC 2.0.2 stdlib.h does not declare free, contrary to ANSI.
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3479 "configure"
+#include "confdefs.h"
+#include <stdlib.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "free" >/dev/null 2>&1; then
+  :
+else
+  ac_cv_header_stdc=no
+fi
+rm -f conftest*
+
+fi
+
+if test $ac_cv_header_stdc = yes; then
+  # /bin/cc in Irix-4.0.5 gets non-ANSI ctype macros unless using -ansi.
+  if test "$cross_compiling" = yes; then
+  :
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3500 "configure"
+#include "confdefs.h"
+#include <ctype.h>
+#if ((' ' & 0x0FF) == 0x020)
+# define ISLOWER(c) ('a' <= (c) && (c) <= 'z')
+# define TOUPPER(c) (ISLOWER(c) ? 'A' + ((c) - 'a') : (c))
+#else
+# define ISLOWER(c) (('a' <= (c) && (c) <= 'i') \
+                     || ('j' <= (c) && (c) <= 'r') \
+                     || ('s' <= (c) && (c) <= 'z'))
+# define TOUPPER(c) (ISLOWER(c) ? ((c) | 0x40) : (c))
+#endif
+
+#define XOR(e, f) (((e) && !(f)) || (!(e) && (f)))
+int
+main ()
+{
+  int i;
+  for (i = 0; i < 256; i++)
+    if (XOR (islower (i), ISLOWER (i))
+        || toupper (i) != TOUPPER (i))
+      exit(2);
+  exit (0);
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:3526: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3529: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:3531: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3534: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  :
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_header_stdc=no
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+fi
+echo "$as_me:3547: result: $ac_cv_header_stdc" >&5
+echo "${ECHO_T}$ac_cv_header_stdc" >&6
+if test $ac_cv_header_stdc = yes; then
+
+cat >>confdefs.h <<\EOF
+#define STDC_HEADERS 1
+EOF
+
+fi
+
+for ac_header in fcntl.h limits.h sys/time.h unistd.h string.h memory.h sys/procfs.h
+do
+ac_ac_Header=`echo "ac_cv_header_$ac_header" | $ac_tr_sh`
+echo "$as_me:3560: checking for $ac_header" >&5
+echo $ECHO_N "checking for $ac_header... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_Header+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3566 "configure"
+#include "confdefs.h"
+#include <$ac_header>
+_ACEOF
+if { (eval echo "$as_me:3570: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:3576: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  eval "$ac_ac_Header=yes"
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  eval "$ac_ac_Header=no"
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:3595: result: `eval echo '${'$ac_ac_Header'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_Header'}'`" >&6
+if test `eval echo '${'$ac_ac_Header'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_header" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 3606 "configure"
+#include "confdefs.h"
+#include <stdio.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "fread" >/dev/null 2>&1; then
+  cat >>confdefs.h <<\EOF
+#define HAVE_FREAD_DECL 1
+EOF
+
+fi
+rm -f conftest*
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 3621 "configure"
+#include "confdefs.h"
+#include <stdio.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "fgetc" >/dev/null 2>&1; then
+  cat >>confdefs.h <<\EOF
+#define HAVE_FGETC_DECL 1
+EOF
+
+fi
+rm -f conftest*
+
+cat >conftest.$ac_ext <<_ACEOF
+#line 3636 "configure"
+#include "confdefs.h"
+#include <sys/procfs.h>
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "pr_brksize" >/dev/null 2>&1; then
+  cat >>confdefs.h <<\EOF
+#define HAVE_PR_BRKSIZE 1
+EOF
+
+fi
+rm -f conftest*
+
+# The Ultrix 4.2 mips builtin alloca declared by alloca.h only works
+# for constant arguments.  Useless!
+echo "$as_me:3652: checking for working alloca.h" >&5
+echo $ECHO_N "checking for working alloca.h... $ECHO_C" >&6
+if test "${ac_cv_working_alloca_h+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3658 "configure"
+#include "confdefs.h"
+#include <alloca.h>
+int
+main ()
+{
+char *p = (char *) alloca (2 * sizeof (int));
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3670: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3673: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3676: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3679: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_working_alloca_h=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_working_alloca_h=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:3689: result: $ac_cv_working_alloca_h" >&5
+echo "${ECHO_T}$ac_cv_working_alloca_h" >&6
+if test $ac_cv_working_alloca_h = yes; then
+
+cat >>confdefs.h <<\EOF
+#define HAVE_ALLOCA_H 1
+EOF
+
+fi
+
+echo "$as_me:3699: checking for alloca" >&5
+echo $ECHO_N "checking for alloca... $ECHO_C" >&6
+if test "${ac_cv_func_alloca_works+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3705 "configure"
+#include "confdefs.h"
+#ifdef __GNUC__
+# define alloca __builtin_alloca
+#else
+# ifdef _MSC_VER
+#  include <malloc.h>
+#  define alloca _alloca
+# else
+#  if HAVE_ALLOCA_H
+#   include <alloca.h>
+#  else
+#   ifdef _AIX
+ #pragma alloca
+#   else
+#    ifndef alloca /* predefined by HP cc +Olibcalls */
+char *alloca ();
+#    endif
+#   endif
+#  endif
+# endif
+#endif
+
+int
+main ()
+{
+char *p = (char *) alloca (1);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3737: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3740: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3743: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3746: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_func_alloca_works=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_func_alloca_works=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:3756: result: $ac_cv_func_alloca_works" >&5
+echo "${ECHO_T}$ac_cv_func_alloca_works" >&6
+
+if test $ac_cv_func_alloca_works = yes; then
+
+cat >>confdefs.h <<\EOF
+#define HAVE_ALLOCA 1
+EOF
+
+else
+  # The SVR3 libPW and SVR4 libucb both contain incompatible functions
+# that cause trouble.  Some versions do not even contain alloca or
+# contain a buggy version.  If you still want to use their alloca,
+# use ar to extract alloca.o from them instead of compiling alloca.c.
+
+ALLOCA=alloca.$ac_objext
+
+cat >>confdefs.h <<\EOF
+#define C_ALLOCA 1
+EOF
+
+echo "$as_me:3777: checking whether \`alloca.c' needs Cray hooks" >&5
+echo $ECHO_N "checking whether \`alloca.c' needs Cray hooks... $ECHO_C" >&6
+if test "${ac_cv_os_cray+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3783 "configure"
+#include "confdefs.h"
+#if defined(CRAY) && ! defined(CRAY2)
+webecray
+#else
+wenotbecray
+#endif
+
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "webecray" >/dev/null 2>&1; then
+  ac_cv_os_cray=yes
+else
+  ac_cv_os_cray=no
+fi
+rm -f conftest*
+
+fi
+echo "$as_me:3801: result: $ac_cv_os_cray" >&5
+echo "${ECHO_T}$ac_cv_os_cray" >&6
+if test $ac_cv_os_cray = yes; then
+  for ac_func in _getb67 GETB67 getb67; do
+    ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:3806: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3812 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:3843: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3846: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:3849: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3852: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:3862: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+
+cat >>confdefs.h <<EOF
+#define CRAY_STACKSEG_END $ac_func
+EOF
+
+    break
+fi
+
+  done
+fi
+
+echo "$as_me:3876: checking stack direction for C alloca" >&5
+echo $ECHO_N "checking stack direction for C alloca... $ECHO_C" >&6
+if test "${ac_cv_c_stack_direction+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test "$cross_compiling" = yes; then
+  ac_cv_c_stack_direction=0
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3885 "configure"
+#include "confdefs.h"
+int
+find_stack_direction ()
+{
+  static char *addr = 0;
+  auto char dummy;
+  if (addr == 0)
+    {
+      addr = &dummy;
+      return find_stack_direction ();
+    }
+  else
+    return (&dummy > addr) ? 1 : -1;
+}
+
+int
+main ()
+{
+  exit (find_stack_direction () < 0);
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:3908: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:3911: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:3913: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:3916: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_c_stack_direction=1
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_c_stack_direction=-1
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+echo "$as_me:3928: result: $ac_cv_c_stack_direction" >&5
+echo "${ECHO_T}$ac_cv_c_stack_direction" >&6
+
+cat >>confdefs.h <<EOF
+#define STACK_DIRECTION $ac_cv_c_stack_direction
+EOF
+
+fi
+
+if test $ac_cv_c_compiler_gnu = yes; then
+    echo "$as_me:3938: checking whether $CC needs -traditional" >&5
+echo $ECHO_N "checking whether $CC needs -traditional... $ECHO_C" >&6
+if test "${ac_cv_prog_gcc_traditional+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+    ac_pattern="Autoconf.*'x'"
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3945 "configure"
+#include "confdefs.h"
+#include <sgtty.h>
+Autoconf TIOCGETP
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "$ac_pattern" >/dev/null 2>&1; then
+  ac_cv_prog_gcc_traditional=yes
+else
+  ac_cv_prog_gcc_traditional=no
+fi
+rm -f conftest*
+
+  if test $ac_cv_prog_gcc_traditional = no; then
+    cat >conftest.$ac_ext <<_ACEOF
+#line 3960 "configure"
+#include "confdefs.h"
+#include <termio.h>
+Autoconf TCGETA
+_ACEOF
+if (eval "$ac_cpp conftest.$ac_ext") 2>&5 |
+  egrep "$ac_pattern" >/dev/null 2>&1; then
+  ac_cv_prog_gcc_traditional=yes
+fi
+rm -f conftest*
+
+  fi
+fi
+echo "$as_me:3973: result: $ac_cv_prog_gcc_traditional" >&5
+echo "${ECHO_T}$ac_cv_prog_gcc_traditional" >&6
+  if test $ac_cv_prog_gcc_traditional = yes; then
+    CC="$CC -traditional"
+  fi
+fi
+
+echo "$as_me:3980: checking return type of signal handlers" >&5
+echo $ECHO_N "checking return type of signal handlers... $ECHO_C" >&6
+if test "${ac_cv_type_signal+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 3986 "configure"
+#include "confdefs.h"
+#include <sys/types.h>
+#include <signal.h>
+#ifdef signal
+# undef signal
+#endif
+#ifdef __cplusplus
+extern "C" void (*signal (int, void (*)(int)))(int);
+#else
+void (*signal ()) ();
+#endif
+
+int
+main ()
+{
+int i;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4008: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4011: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4014: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4017: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_signal=void
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_signal=int
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:4027: result: $ac_cv_type_signal" >&5
+echo "${ECHO_T}$ac_cv_type_signal" >&6
+
+cat >>confdefs.h <<EOF
+#define RETSIGTYPE $ac_cv_type_signal
+EOF
+
+for ac_func in vprintf
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4037: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4043 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4074: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4077: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4080: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4083: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4093: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+echo "$as_me:4100: checking for _doprnt" >&5
+echo $ECHO_N "checking for _doprnt... $ECHO_C" >&6
+if test "${ac_cv_func__doprnt+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4106 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char _doprnt (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char _doprnt ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub__doprnt) || defined (__stub____doprnt)
+choke me
+#else
+f = _doprnt;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4137: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4140: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4143: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4146: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_func__doprnt=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_func__doprnt=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4156: result: $ac_cv_func__doprnt" >&5
+echo "${ECHO_T}$ac_cv_func__doprnt" >&6
+if test $ac_cv_func__doprnt = yes; then
+
+cat >>confdefs.h <<\EOF
+#define HAVE_DOPRNT 1
+EOF
+
+fi
+
+fi
+done
+
+for ac_func in ftime select strftime strtol getrusage times mallinfo setbuffer getpagesize strerror
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4172: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4178 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4209: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4212: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4215: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4218: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4228: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+for ac_func in ftruncate strstr strcasecmp
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4241: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4247 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4278: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4281: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4284: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4287: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4297: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+else
+  LIBOBJS="$LIBOBJS $ac_func.$ac_objext"
+fi
+done
+
+# *** Custom checking (based on GNU tar configure.in) ***
+# ---------------------------------------------------------------------------
+echo "$as_me:4311: checking for HP-UX needing gmalloc" >&5
+echo $ECHO_N "checking for HP-UX needing gmalloc... $ECHO_C" >&6
+if test "`(uname -s) 2> /dev/null`" = 'HP-UX'; then
+  echo "$as_me:4314: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  LIBOBJS="$LIBOBJS gmalloc.o"
+  cat >>confdefs.h <<\EOF
+#define HAVE_VALLOC 1
+EOF
+
+else
+  echo "$as_me:4322: result: no" >&5
+echo "${ECHO_T}no" >&6
+
+for ac_func in valloc
+do
+ac_ac_var=`echo "ac_cv_func_$ac_func" | $ac_tr_sh`
+echo "$as_me:4328: checking for $ac_func" >&5
+echo $ECHO_N "checking for $ac_func... $ECHO_C" >&6
+if eval "test \"\${$ac_ac_var+set}\" = set"; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4334 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char $ac_func (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char $ac_func ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_$ac_func) || defined (__stub___$ac_func)
+choke me
+#else
+f = $ac_func;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4365: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4368: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4371: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4374: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  eval "$ac_ac_var=yes"
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+eval "$ac_ac_var=no"
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4384: result: `eval echo '${'$ac_ac_var'}'`" >&5
+echo "${ECHO_T}`eval echo '${'$ac_ac_var'}'`" >&6
+if test `eval echo '${'$ac_ac_var'}'` = yes; then
+  cat >>confdefs.h <<EOF
+#define `echo "HAVE_$ac_func" | $ac_tr_cpp` 1
+EOF
+
+fi
+done
+
+fi
+
+echo "$as_me:4396: checking if malloc debugging is wanted" >&5
+echo $ECHO_N "checking if malloc debugging is wanted... $ECHO_C" >&6
+
+# Check whether --with-dmalloc or --without-dmalloc was given.
+if test "${with_dmalloc+set}" = set; then
+  withval="$with_dmalloc"
+  if test "$withval" = yes; then
+  echo "$as_me:4403: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+  cat >>confdefs.h <<\EOF
+#define WITH_DMALLOC 1
+EOF
+
+  LIBS="$LIBS -ldmalloc"
+  LDFLAGS="$LDFLAGS -g"
+else
+  echo "$as_me:4412: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi
+else
+  echo "$as_me:4416: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi;
+
+echo "$as_me:4420: checking which of rx or regex is wanted" >&5
+echo $ECHO_N "checking which of rx or regex is wanted... $ECHO_C" >&6
+
+# Check whether --with-regex or --without-regex was given.
+if test "${with_regex+set}" = set; then
+  withval="$with_regex"
+  if test "$withval" = yes; then
+  ac_with_regex=1
+  echo "$as_me:4428: result: regex" >&5
+echo "${ECHO_T}regex" >&6
+  cat >>confdefs.h <<\EOF
+#define WITH_REGEX 1
+EOF
+
+  LIBOBJS="$LIBOBJS regex.o"
+fi
+fi;
+if test -z "$ac_with_regex"; then
+  echo "$as_me:4438: result: rx" >&5
+echo "${ECHO_T}rx" >&6
+  echo "$as_me:4440: checking for re_rx_search" >&5
+echo $ECHO_N "checking for re_rx_search... $ECHO_C" >&6
+if test "${ac_cv_func_re_rx_search+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4446 "configure"
+#include "confdefs.h"
+/* System header to define __stub macros and hopefully few prototypes,
+    which can conflict with char re_rx_search (); below.  */
+#include <assert.h>
+/* Override any gcc2 internal prototype to avoid an error.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+/* We use char because int might match the return type of a gcc2
+   builtin and then its argument prototype would still apply.  */
+char re_rx_search ();
+char (*f) ();
+
+int
+main ()
+{
+/* The GNU C library defines this for functions which it implements
+    to always fail with ENOSYS.  Some functions are actually named
+    something starting with __ and the normal name is an alias.  */
+#if defined (__stub_re_rx_search) || defined (__stub___re_rx_search)
+choke me
+#else
+f = re_rx_search;
+#endif
+
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4477: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4480: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4483: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4486: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_func_re_rx_search=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_func_re_rx_search=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+fi
+echo "$as_me:4496: result: $ac_cv_func_re_rx_search" >&5
+echo "${ECHO_T}$ac_cv_func_re_rx_search" >&6
+if test $ac_cv_func_re_rx_search = yes; then
+  :
+else
+  echo "$as_me:4501: checking for unsigned char *" >&5
+echo $ECHO_N "checking for unsigned char *... $ECHO_C" >&6
+if test "${ac_cv_type_unsigned_char_p+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4507 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+if ((unsigned char * *) 0)
+  return 0;
+if (sizeof (unsigned char *))
+  return 0;
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4522: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4525: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4528: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4531: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_type_unsigned_char_p=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_type_unsigned_char_p=no
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+fi
+echo "$as_me:4541: result: $ac_cv_type_unsigned_char_p" >&5
+echo "${ECHO_T}$ac_cv_type_unsigned_char_p" >&6
+
+echo "$as_me:4544: checking size of unsigned char *" >&5
+echo $ECHO_N "checking size of unsigned char *... $ECHO_C" >&6
+if test "${ac_cv_sizeof_unsigned_char_p+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  if test "$ac_cv_type_unsigned_char_p" = yes; then
+  if test "$cross_compiling" = yes; then
+  # Depending upon the size, compute the lo and hi bounds.
+cat >conftest.$ac_ext <<_ACEOF
+#line 4553 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) >= 0)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4565: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4568: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4571: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4574: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_lo=0 ac_mid=0
+  while :; do
+    cat >conftest.$ac_ext <<_ACEOF
+#line 4579 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) <= $ac_mid)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4591: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4594: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4597: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4600: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_hi=$ac_mid; break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_lo=`expr $ac_mid + 1`; ac_mid=`expr 2 '*' $ac_mid + 1`
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  done
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_hi=-1 ac_mid=-1
+  while :; do
+    cat >conftest.$ac_ext <<_ACEOF
+#line 4616 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) >= $ac_mid)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4628: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4631: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4634: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4637: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_lo=$ac_mid; break
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_hi=`expr $ac_mid - 1`; ac_mid=`expr 2 '*' $ac_mid`
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+  done
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+# Binary search between lo and hi bounds.
+while test "x$ac_lo" != "x$ac_hi"; do
+  ac_mid=`expr '(' $ac_hi - $ac_lo ')' / 2 + $ac_lo`
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4653 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+int _array_ [1 - 2 * !((sizeof (unsigned char *)) <= $ac_mid)]
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext
+if { (eval echo "$as_me:4665: \"$ac_compile\"") >&5
+  (eval $ac_compile) 2>&5
+  ac_status=$?
+  echo "$as_me:4668: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest.$ac_objext'
+  { (eval echo "$as_me:4671: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4674: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_hi=$ac_mid
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_lo=`expr $ac_mid + 1`
+fi
+rm -f conftest.$ac_objext conftest.$ac_ext
+done
+ac_cv_sizeof_unsigned_char_p=$ac_lo
+else
+  if test "$cross_compiling" = yes; then
+  { { echo "$as_me:4687: error: cannot run test program while cross compiling" >&5
+echo "$as_me: error: cannot run test program while cross compiling" >&2;}
+   { (exit 1); exit 1; }; }
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4692 "configure"
+#include "confdefs.h"
+$ac_includes_default
+int
+main ()
+{
+FILE *f = fopen ("conftest.val", "w");
+if (!f)
+  exit (1);
+fprintf (f, "%d", (sizeof (unsigned char *)));
+fclose (f);
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest$ac_exeext
+if { (eval echo "$as_me:4708: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4711: \$? = $ac_status" >&5
+  (exit $ac_status); } && { ac_try='./conftest$ac_exeext'
+  { (eval echo "$as_me:4713: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4716: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_sizeof_unsigned_char_p=`cat conftest.val`
+else
+  echo "$as_me: program exited with status $ac_status" >&5
+echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+fi
+rm -f core core.* *.core conftest$ac_exeext conftest.$ac_ext
+fi
+fi
+rm -f conftest.val
+else
+  ac_cv_sizeof_unsigned_char_p=0
+fi
+fi
+echo "$as_me:4732: result: $ac_cv_sizeof_unsigned_char_p" >&5
+echo "${ECHO_T}$ac_cv_sizeof_unsigned_char_p" >&6
+cat >>confdefs.h <<EOF
+#define SIZEOF_UNSIGNED_CHAR_P $ac_cv_sizeof_unsigned_char_p
+EOF
+
+    if test "$ac_cv_sizeof_unsigned_char_p" = 8
+    then
+      { echo "$as_me:4740: WARNING: I'm forcing you to use regex because I can't
+        find a local rx library and the one included with this
+        distribution doesn't work on 64-bit machines like yours" >&5
+echo "$as_me: WARNING: I'm forcing you to use regex because I can't
+        find a local rx library and the one included with this
+        distribution doesn't work on 64-bit machines like yours" >&2;}
+      LIBOBJS="$LIBOBJS regex.o"
+    else
+      LIBOBJS="$LIBOBJS rx.o"
+    fi
+
+fi
+
+fi
+
+echo "$as_me:4755: checking whether GNU readline requested" >&5
+echo $ECHO_N "checking whether GNU readline requested... $ECHO_C" >&6
+
+# Check whether --with-gnu_readline or --without-gnu_readline was given.
+if test "${with_gnu_readline+set}" = set; then
+  withval="$with_gnu_readline"
+  if test "$withval" = yes; then
+      echo "$as_me:4762: result: yes" >&5
+echo "${ECHO_T}yes" >&6
+      ac_with_gnu_readline=1
+    else
+      echo "$as_me:4766: result: no" >&5
+echo "${ECHO_T}no" >&6
+    fi
+else
+  echo "$as_me:4770: result: no" >&5
+echo "${ECHO_T}no" >&6
+fi;
+
+  if test -n "$ac_with_gnu_readline"; then
+    echo "$as_me:4775: checking for readline/readline.h" >&5
+echo $ECHO_N "checking for readline/readline.h... $ECHO_C" >&6
+if test "${ac_cv_header_readline_readline_h+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  cat >conftest.$ac_ext <<_ACEOF
+#line 4781 "configure"
+#include "confdefs.h"
+#include <readline/readline.h>
+_ACEOF
+if { (eval echo "$as_me:4785: \"$ac_cpp conftest.$ac_ext\"") >&5
+  (eval $ac_cpp conftest.$ac_ext) 2>conftest.er1
+  ac_status=$?
+  egrep -v '^ *\+' conftest.er1 >conftest.err
+  rm -f conftest.er1
+  cat conftest.err >&5
+  echo "$as_me:4791: \$? = $ac_status" >&5
+  (exit $ac_status); } >/dev/null; then
+  if test -s conftest.err; then
+    ac_cpp_err=$ac_c_preproc_warn_flag
+  else
+    ac_cpp_err=
+  fi
+else
+  ac_cpp_err=yes
+fi
+if test -z "$ac_cpp_err"; then
+  ac_cv_header_readline_readline_h=yes
+else
+  echo "$as_me: failed program was:" >&5
+  cat conftest.$ac_ext >&5
+  ac_cv_header_readline_readline_h=no
+fi
+rm -f conftest.err conftest.$ac_ext
+fi
+echo "$as_me:4810: result: $ac_cv_header_readline_readline_h" >&5
+echo "${ECHO_T}$ac_cv_header_readline_readline_h" >&6
+if test $ac_cv_header_readline_readline_h = yes; then
+  ac_mg_readline_header_found=1
+else
+  { echo "$as_me:4815: WARNING: Can't find GNU readline headers; configuring without \
+GNU readline support" >&5
+echo "$as_me: WARNING: Can't find GNU readline headers; configuring without \
+GNU readline support" >&2;}
+fi
+
+    if test -n "$ac_mg_readline_header_found"; then
+      # first check whether we can find the readline library itself
+      echo "$as_me:4823: checking for main in -lreadline" >&5
+echo $ECHO_N "checking for main in -lreadline... $ECHO_C" >&6
+if test "${ac_cv_lib_readline_main+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lreadline  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 4831 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+main ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4843: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4846: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4849: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4852: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_readline_main=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_readline_main=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:4863: result: $ac_cv_lib_readline_main" >&5
+echo "${ECHO_T}$ac_cv_lib_readline_main" >&6
+if test $ac_cv_lib_readline_main = yes; then
+  ac_mg_readline_lib_found=1
+         cat >>confdefs.h <<\EOF
+#define WITH_GNU_READLINE 1
+EOF
+
+         LIBS="$LIBS -lreadline"
+else
+  { echo "$as_me:4873: WARNING: Can't find GNU readline library; configuring without \
+GNU readline support" >&5
+echo "$as_me: WARNING: Can't find GNU readline library; configuring without \
+GNU readline support" >&2;}
+fi
+
+      if test -n "$ac_mg_readline_lib_found"; then
+        # on some systems, readline needs curses.  It is difficult to
+        #  determine whether this is necessary on the current system,
+        #  since other undefined symbols can be turned up by an
+        #  AC_CHECK_LIB(readline, readline) test that are actually defined
+	#  by mg itself (e.g. xmalloc, xrealloc).  So, if we find libcurses,
+	#  we just bung it on and hope for the best.
+	echo "$as_me:4886: checking for main in -lcurses" >&5
+echo $ECHO_N "checking for main in -lcurses... $ECHO_C" >&6
+if test "${ac_cv_lib_curses_main+set}" = set; then
+  echo $ECHO_N "(cached) $ECHO_C" >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lcurses  $LIBS"
+cat >conftest.$ac_ext <<_ACEOF
+#line 4894 "configure"
+#include "confdefs.h"
+
+int
+main ()
+{
+main ();
+  ;
+  return 0;
+}
+_ACEOF
+rm -f conftest.$ac_objext conftest$ac_exeext
+if { (eval echo "$as_me:4906: \"$ac_link\"") >&5
+  (eval $ac_link) 2>&5
+  ac_status=$?
+  echo "$as_me:4909: \$? = $ac_status" >&5
+  (exit $ac_status); } &&
+         { ac_try='test -s conftest$ac_exeext'
+  { (eval echo "$as_me:4912: \"$ac_try\"") >&5
+  (eval $ac_try) 2>&5
+  ac_status=$?
+  echo "$as_me:4915: \$? = $ac_status" >&5
+  (exit $ac_status); }; }; then
+  ac_cv_lib_curses_main=yes
+else
+  echo "$as_me: failed program was:" >&5
+cat conftest.$ac_ext >&5
+ac_cv_lib_curses_main=no
+fi
+rm -f conftest.$ac_objext conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+echo "$as_me:4926: result: $ac_cv_lib_curses_main" >&5
+echo "${ECHO_T}$ac_cv_lib_curses_main" >&6
+if test $ac_cv_lib_curses_main = yes; then
+  LIBS="$LIBS -lcurses"
+fi
+
+      fi
+    fi
+  fi
+
+# ---------------------------------------------------------------------------
+if test "$ac_cv_func_alloca" = 'no'; then
+  LIBOBJS="$LIBOBJS xmalloc.o error.o"
+fi
+
+# ---------------------------------------------------------------------------
+# see if the user wants to override use of long long
+# Check whether --enable-override-longlong or --disable-override-longlong was given.
+if test "${enable_override_longlong+set}" = set; then
+  enableval="$enable_override_longlong"
+  cat >>confdefs.h <<\EOF
+#define DISABLE_LONG_LONG 1
+EOF
+
+fi;
+
+# ---------------------------------------------------------------------------
+
+ac_config_files="$ac_config_files Makefile text/Makefile lib/Makefile jni/Makefile java/org/greenstone/mgpp/Makefile"
+ac_config_commands="$ac_config_commands default"
+cat >confcache <<\_ACEOF
+# This file is a shell script that caches the results of configure
+# tests run on this system so they can be shared between configure
+# scripts and configure runs, see configure's option --config-cache.
+# It is not useful on other systems.  If it contains results you don't
+# want to keep, you may remove or edit it.
+#
+# config.status only pays attention to the cache file if you give it
+# the --recheck option to rerun configure.
+#
+# `ac_cv_env_foo' variables (set or unset) will be overriden when
+# loading this file, other *unset* `ac_cv_foo' will be assigned the
+# following values.
+
+_ACEOF
+
+# The following way of writing the cache mishandles newlines in values,
+# but we know of no workaround that is simple, portable, and efficient.
+# So, don't put newlines in cache variables' values.
+# Ultrix sh set writes to stderr and can't be redirected directly,
+# and sets the high bit in the cache file unless we assign to the vars.
+{
+  (set) 2>&1 |
+    case `(ac_space=' '; set | grep ac_space) 2>&1` in
+    *ac_space=\ *)
+      # `set' does not quote correctly, so add quotes (double-quote
+      # substitution turns \\\\ into \\, and sed turns \\ into \).
+      sed -n \
+        "s/'/'\\\\''/g;
+    	  s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1='\\2'/p"
+      ;;
+    *)
+      # `set' quotes correctly as required by POSIX, so do not add quotes.
+      sed -n \
+        "s/^\\([_$ac_cr_alnum]*_cv_[_$ac_cr_alnum]*\\)=\\(.*\\)/\\1=\\2/p"
+      ;;
+    esac;
+} |
+  sed '
+     t clear
+     : clear
+     s/^\([^=]*\)=\(.*[{}].*\)$/test "${\1+set}" = set || &/
+     t end
+     /^ac_cv_env/!s/^\([^=]*\)=\(.*\)$/\1=${\1=\2}/
+     : end' >>confcache
+if cmp -s $cache_file confcache; then :; else
+  if test -w $cache_file; then
+    test "x$cache_file" != "x/dev/null" && echo "updating cache $cache_file"
+    cat confcache >$cache_file
+  else
+    echo "not updating unwritable cache $cache_file"
+  fi
+fi
+rm -f confcache
+
+test "x$prefix" = xNONE && prefix=$ac_default_prefix
+# Let make expand exec_prefix.
+test "x$exec_prefix" = xNONE && exec_prefix='${prefix}'
+
+# VPATH may cause trouble with some makes, so we remove $(srcdir),
+# ${srcdir} and @srcdir@ from VPATH if srcdir is ".", strip leading and
+# trailing colons and then remove the whole line if VPATH becomes empty
+# (actually we leave an empty line to preserve line numbers).
+if test "x$srcdir" = x.; then
+  ac_vpsub='/^[ 	]*VPATH[ 	]*=/{
+s/:*\$(srcdir):*/:/;
+s/:*\${srcdir}:*/:/;
+s/:*@srcdir@:*/:/;
+s/^\([^=]*=[ 	]*\):*/\1/;
+s/:*$//;
+s/^[^=]*=[ 	]*$//;
+}'
+fi
+
+DEFS=-DHAVE_CONFIG_H
+
+: ${CONFIG_STATUS=./config.status}
+ac_clean_files_save=$ac_clean_files
+ac_clean_files="$ac_clean_files $CONFIG_STATUS"
+{ echo "$as_me:5035: creating $CONFIG_STATUS" >&5
+echo "$as_me: creating $CONFIG_STATUS" >&6;}
+cat >$CONFIG_STATUS <<_ACEOF
+#! $SHELL
+# Generated automatically by configure.
+# Run this file to recreate the current configuration.
+# Compiler output produced by configure, useful for debugging
+# configure, is in config.log if it exists.
+
+debug=false
+SHELL=\${CONFIG_SHELL-$SHELL}
+ac_cs_invocation="\$0 \$@"
+
+_ACEOF
+
+cat >>$CONFIG_STATUS <<\_ACEOF
+# Be Bourne compatible
+if test -n "${ZSH_VERSION+set}" && (emulate sh) >/dev/null 2>&1; then
+  emulate sh
+  NULLCMD=:
+elif test -n "${BASH_VERSION+set}" && (set -o posix) >/dev/null 2>&1; then
+  set -o posix
+fi
+
+# Name of the executable.
+as_me=`echo "$0" |sed 's,.*[\\/],,'`
+
+if expr a : '\(a\)' >/dev/null 2>&1; then
+  as_expr=expr
+else
+  as_expr=false
+fi
+
+rm -f conf$$ conf$$.exe conf$$.file
+echo >conf$$.file
+if ln -s conf$$.file conf$$ 2>/dev/null; then
+  # We could just check for DJGPP; but this test a) works b) is more generic
+  # and c) will remain valid once DJGPP supports symlinks (DJGPP 2.04).
+  if test -f conf$$.exe; then
+    # Don't use ln at all; we don't have any links
+    as_ln_s='cp -p'
+  else
+    as_ln_s='ln -s'
+  fi
+elif ln conf$$.file conf$$ 2>/dev/null; then
+  as_ln_s=ln
+else
+  as_ln_s='cp -p'
+fi
+rm -f conf$$ conf$$.exe conf$$.file
+
+as_executable_p="test -f"
+
+# Support unset when possible.
+if (FOO=FOO; unset FOO) >/dev/null 2>&1; then
+  as_unset=unset
+else
+  as_unset=false
+fi
+
+# NLS nuisances.
+$as_unset LANG || test "${LANG+set}" != set || { LANG=C; export LANG; }
+$as_unset LC_ALL || test "${LC_ALL+set}" != set || { LC_ALL=C; export LC_ALL; }
+$as_unset LC_TIME || test "${LC_TIME+set}" != set || { LC_TIME=C; export LC_TIME; }
+$as_unset LC_CTYPE || test "${LC_CTYPE+set}" != set || { LC_CTYPE=C; export LC_CTYPE; }
+$as_unset LANGUAGE || test "${LANGUAGE+set}" != set || { LANGUAGE=C; export LANGUAGE; }
+$as_unset LC_COLLATE || test "${LC_COLLATE+set}" != set || { LC_COLLATE=C; export LC_COLLATE; }
+$as_unset LC_NUMERIC || test "${LC_NUMERIC+set}" != set || { LC_NUMERIC=C; export LC_NUMERIC; }
+$as_unset LC_MESSAGES || test "${LC_MESSAGES+set}" != set || { LC_MESSAGES=C; export LC_MESSAGES; }
+
+# IFS
+# We need space, tab and new line, in precisely that order.
+as_nl='
+'
+IFS=" 	$as_nl"
+
+# CDPATH.
+$as_unset CDPATH || test "${CDPATH+set}" != set || { CDPATH=:; export CDPATH; }
+
+exec 6>&1
+
+_ACEOF
+
+# Files that config.status was made for.
+if test -n "$ac_config_files"; then
+  echo "config_files=\"$ac_config_files\"" >>$CONFIG_STATUS
+fi
+
+if test -n "$ac_config_headers"; then
+  echo "config_headers=\"$ac_config_headers\"" >>$CONFIG_STATUS
+fi
+
+if test -n "$ac_config_links"; then
+  echo "config_links=\"$ac_config_links\"" >>$CONFIG_STATUS
+fi
+
+if test -n "$ac_config_commands"; then
+  echo "config_commands=\"$ac_config_commands\"" >>$CONFIG_STATUS
+fi
+
+cat >>$CONFIG_STATUS <<\EOF
+
+ac_cs_usage="\
+\`$as_me' instantiates files from templates according to the
+current configuration.
+
+Usage: $0 [OPTIONS] [FILE]...
+
+  -h, --help       print this help, then exit
+  -V, --version    print version number, then exit
+  -d, --debug      don't remove temporary files
+      --recheck    update $as_me by reconfiguring in the same conditions
+  --file=FILE[:TEMPLATE]
+                   instantiate the configuration file FILE
+  --header=FILE[:TEMPLATE]
+                   instantiate the configuration header FILE
+
+Configuration files:
+$config_files
+
+Configuration headers:
+$config_headers
+
+Configuration commands:
+$config_commands
+
+Report bugs to <bug-autoconf@gnu.org>."
+EOF
+
+cat >>$CONFIG_STATUS <<EOF
+ac_cs_version="\\
+config.status
+configured by $0, generated by GNU Autoconf 2.50,
+  with options \\"`echo "$ac_configure_args" | sed 's/[\\""\`\$]/\\\\&/g'`\\"
+
+Copyright 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001
+Free Software Foundation, Inc.
+This config.status script is free software; the Free Software Foundation
+gives unlimited permission to copy, distribute and modify it."
+srcdir=$srcdir
+INSTALL="$INSTALL"
+EOF
+
+cat >>$CONFIG_STATUS <<\EOF
+# If no file are specified by the user, then we need to provide default
+# value.  By we need to know if files were specified by the user.
+ac_need_defaults=:
+while test $# != 0
+do
+  case $1 in
+  --*=*)
+    ac_option=`expr "x$1" : 'x\([^=]*\)='`
+    ac_optarg=`expr "x$1" : 'x[^=]*=\(.*\)'`
+    shift
+    set dummy "$ac_option" "$ac_optarg" ${1+"$@"}
+    shift
+    ;;
+  -*);;
+  *) # This is not an option, so the user has probably given explicit
+     # arguments.
+     ac_need_defaults=false;;
+  esac
+
+  case $1 in
+  # Handling of the options.
+EOF
+cat >>$CONFIG_STATUS <<EOF
+  -recheck | --recheck | --rechec | --reche | --rech | --rec | --re | --r)
+    echo "running $SHELL $0 " $ac_configure_args " --no-create --no-recursion"
+    exec $SHELL $0 $ac_configure_args --no-create --no-recursion ;;
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+  --version | --vers* | -V )
+    echo "$ac_cs_version"; exit 0 ;;
+  --he | --h)
+    # Conflict between --help and --header
+    { { echo "$as_me:5211: error: ambiguous option: $1
+Try \`$0 --help' for more information." >&5
+echo "$as_me: error: ambiguous option: $1
+Try \`$0 --help' for more information." >&2;}
+   { (exit 1); exit 1; }; };;
+  --help | --hel | -h )
+    echo "$ac_cs_usage"; exit 0 ;;
+  --debug | --d* | -d )
+    debug=: ;;
+  --file | --fil | --fi | --f )
+    shift
+    CONFIG_FILES="$CONFIG_FILES $1"
+    ac_need_defaults=false;;
+  --header | --heade | --head | --hea )
+    shift
+    CONFIG_HEADERS="$CONFIG_HEADERS $1"
+    ac_need_defaults=false;;
+
+  # Handling of arguments.
+  'Makefile' ) CONFIG_FILES="$CONFIG_FILES Makefile" ;;
+  'text/Makefile' ) CONFIG_FILES="$CONFIG_FILES text/Makefile" ;;
+  'lib/Makefile' ) CONFIG_FILES="$CONFIG_FILES lib/Makefile" ;;
+  'jni/Makefile' ) CONFIG_FILES="$CONFIG_FILES jni/Makefile" ;;
+  'java/org/greenstone/mgpp/Makefile' ) CONFIG_FILES="$CONFIG_FILES java/org/greenstone/mgpp/Makefile" ;;
+  'default' ) CONFIG_COMMANDS="$CONFIG_COMMANDS default" ;;
+  'config.h' ) CONFIG_HEADERS="$CONFIG_HEADERS config.h" ;;
+
+  # This is an error.
+  -*) { { echo "$as_me:5239: error: unrecognized option: $1
+Try \`$0 --help' for more information." >&5
+echo "$as_me: error: unrecognized option: $1
+Try \`$0 --help' for more information." >&2;}
+   { (exit 1); exit 1; }; } ;;
+  *) { { echo "$as_me:5244: error: invalid argument: $1" >&5
+echo "$as_me: error: invalid argument: $1" >&2;}
+   { (exit 1); exit 1; }; };;
+  esac
+  shift
+done
+
+exec 5>>config.log
+cat >&5 << _ACEOF
+
+## ----------------------- ##
+## Running config.status.  ##
+## ----------------------- ##
+
+This file was extended by $as_me 2.50, executed with
+  > $ac_cs_invocation
+on `(hostname || uname -n) 2>/dev/null | sed 1q`
+
+_ACEOF
+EOF
+
+cat >>$CONFIG_STATUS <<\EOF
+# If the user did not use the arguments to specify the items to instantiate,
+# then the envvar interface is used.  Set only those that are not.
+# We use the long form for the default assignment because of an extremely
+# bizarre bug on SunOS 4.1.3.
+if $ac_need_defaults; then
+  test "${CONFIG_FILES+set}" = set || CONFIG_FILES=$config_files
+  test "${CONFIG_HEADERS+set}" = set || CONFIG_HEADERS=$config_headers
+  test "${CONFIG_COMMANDS+set}" = set || CONFIG_COMMANDS=$config_commands
+fi
+
+# Create a temporary directory, and hook for its removal unless debugging.
+$debug ||
+{
+  trap 'exit_status=$?; rm -rf $tmp && exit $exit_status' 0
+  trap '{ (exit $?); exit $?; }' 1 2 13 15
+}
+
+# Create a (secure) tmp directory for tmp files.
+: ${TMPDIR=/tmp}
+{
+  tmp=`(umask 077 && mktemp -d -q "$TMPDIR/csXXXXXX") 2>/dev/null` &&
+  test -n "$tmp" && test -d "$tmp"
+}  ||
+{
+  tmp=$TMPDIR/cs$$-$RANDOM
+  (umask 077 && mkdir $tmp)
+} ||
+{
+   echo "$me: cannot create a temporary directory in $TMPDIR" >&2
+   { (exit 1); exit 1; }
+}
+
+EOF
+
+cat >>$CONFIG_STATUS <<EOF
+
+#
+# CONFIG_FILES section.
+#
+
+# No need to generate the scripts if there are no CONFIG_FILES.
+# This happens for instance when ./config.status config.h
+if test -n "\$CONFIG_FILES"; then
+  # Protect against being on the right side of a sed subst in config.status.
+  sed 's/,@/@@/; s/@,/@@/; s/,;t t\$/@;t t/; /@;t t\$/s/[\\\\&,]/\\\\&/g;
+   s/@@/,@/; s/@@/@,/; s/@;t t\$/,;t t/' >\$tmp/subs.sed <<\\CEOF
+s,@SHELL@,$SHELL,;t t
+s,@exec_prefix@,$exec_prefix,;t t
+s,@prefix@,$prefix,;t t
+s,@program_transform_name@,$program_transform_name,;t t
+s,@bindir@,$bindir,;t t
+s,@sbindir@,$sbindir,;t t
+s,@libexecdir@,$libexecdir,;t t
+s,@datadir@,$datadir,;t t
+s,@sysconfdir@,$sysconfdir,;t t
+s,@sharedstatedir@,$sharedstatedir,;t t
+s,@localstatedir@,$localstatedir,;t t
+s,@libdir@,$libdir,;t t
+s,@includedir@,$includedir,;t t
+s,@oldincludedir@,$oldincludedir,;t t
+s,@infodir@,$infodir,;t t
+s,@mandir@,$mandir,;t t
+s,@PACKAGE_NAME@,$PACKAGE_NAME,;t t
+s,@PACKAGE_TARNAME@,$PACKAGE_TARNAME,;t t
+s,@PACKAGE_VERSION@,$PACKAGE_VERSION,;t t
+s,@PACKAGE_STRING@,$PACKAGE_STRING,;t t
+s,@PACKAGE_BUGREPORT@,$PACKAGE_BUGREPORT,;t t
+s,@ECHO_C@,$ECHO_C,;t t
+s,@ECHO_N@,$ECHO_N,;t t
+s,@ECHO_T@,$ECHO_T,;t t
+s,@PATH_SEPARATOR@,$PATH_SEPARATOR,;t t
+s,@DEFS@,$DEFS,;t t
+s,@LIBS@,$LIBS,;t t
+s,@PACKAGE@,$PACKAGE,;t t
+s,@VERSION@,$VERSION,;t t
+s,@CXX@,$CXX,;t t
+s,@CXXFLAGS@,$CXXFLAGS,;t t
+s,@LDFLAGS@,$LDFLAGS,;t t
+s,@CPPFLAGS@,$CPPFLAGS,;t t
+s,@ac_ct_CXX@,$ac_ct_CXX,;t t
+s,@EXEEXT@,$EXEEXT,;t t
+s,@OBJEXT@,$OBJEXT,;t t
+s,@AWK@,$AWK,;t t
+s,@YACC@,$YACC,;t t
+s,@CC@,$CC,;t t
+s,@CFLAGS@,$CFLAGS,;t t
+s,@ac_ct_CC@,$ac_ct_CC,;t t
+s,@INSTALL_PROGRAM@,$INSTALL_PROGRAM,;t t
+s,@INSTALL_SCRIPT@,$INSTALL_SCRIPT,;t t
+s,@INSTALL_DATA@,$INSTALL_DATA,;t t
+s,@LN_S@,$LN_S,;t t
+s,@SET_MAKE@,$SET_MAKE,;t t
+s,@RANLIB@,$RANLIB,;t t
+s,@ac_ct_RANLIB@,$ac_ct_RANLIB,;t t
+s,@CPP@,$CPP,;t t
+s,@U@,$U,;t t
+s,@ANSI2KNR@,$ANSI2KNR,;t t
+s,@ALLOCA@,$ALLOCA,;t t
+s,@LIBOBJS@,$LIBOBJS,;t t
+CEOF
+
+EOF
+
+  cat >>$CONFIG_STATUS <<\EOF
+  # Split the substitutions into bite-sized pieces for seds with
+  # small command number limits, like on Digital OSF/1 and HP-UX.
+  ac_max_sed_lines=48
+  ac_sed_frag=1 # Number of current file.
+  ac_beg=1 # First line for current file.
+  ac_end=$ac_max_sed_lines # Line after last line for current file.
+  ac_more_lines=:
+  ac_sed_cmds=
+  while $ac_more_lines; do
+    if test $ac_beg -gt 1; then
+      sed "1,${ac_beg}d; ${ac_end}q" $tmp/subs.sed >$tmp/subs.frag
+    else
+      sed "${ac_end}q" $tmp/subs.sed >$tmp/subs.frag
+    fi
+    if test ! -s $tmp/subs.frag; then
+      ac_more_lines=false
+    else
+      # The purpose of the label and of the branching condition is to
+      # speed up the sed processing (if there are no `@' at all, there
+      # is no need to browse any of the substitutions).
+      # These are the two extra sed commands mentioned above.
+      (echo ':t
+  /@[a-zA-Z_][a-zA-Z_0-9]*@/!b' && cat $tmp/subs.frag) >$tmp/subs-$ac_sed_frag.sed
+      if test -z "$ac_sed_cmds"; then
+  	ac_sed_cmds="sed -f $tmp/subs-$ac_sed_frag.sed"
+      else
+  	ac_sed_cmds="$ac_sed_cmds | sed -f $tmp/subs-$ac_sed_frag.sed"
+      fi
+      ac_sed_frag=`expr $ac_sed_frag + 1`
+      ac_beg=$ac_end
+      ac_end=`expr $ac_end + $ac_max_sed_lines`
+    fi
+  done
+  if test -z "$ac_sed_cmds"; then
+    ac_sed_cmds=cat
+  fi
+fi # test -n "$CONFIG_FILES"
+
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+for ac_file in : $CONFIG_FILES; do test "x$ac_file" = x: && continue
+  # Support "outfile[:infile[:infile...]]", defaulting infile="outfile.in".
+  case $ac_file in
+  - | *:- | *:-:* ) # input from stdin
+        cat >$tmp/stdin
+        ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  *:* ) ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  * )   ac_file_in=$ac_file.in ;;
+  esac
+
+  # Compute @srcdir@, @top_srcdir@, and @INSTALL@ for subdirectories.
+  ac_dir=`$as_expr X"$ac_file" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
+         X"$ac_file" : 'X\(//\)[^/]' \| \
+         X"$ac_file" : 'X\(//\)$' \| \
+         X"$ac_file" : 'X\(/\)' \| \
+         .     : '\(.\)' 2>/dev/null ||
+echo X"$ac_file" |
+    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/; q; }
+  	  /^X\(\/\/\)[^/].*/{ s//\1/; q; }
+  	  /^X\(\/\/\)$/{ s//\1/; q; }
+  	  /^X\(\/\).*/{ s//\1/; q; }
+  	  s/.*/./; q'`
+  if test "$ac_dir" != "$ac_file" && test "$ac_dir" != .; then
+    { case "$ac_dir" in
+  [\\/]* | ?:[\\/]* ) as_incr_dir=;;
+  *)                      as_incr_dir=.;;
+esac
+as_dummy="$ac_dir"
+for as_mkdir_dir in `IFS='/\\'; set X $as_dummy; shift; echo "$@"`; do
+  case $as_mkdir_dir in
+    # Skip DOS drivespec
+    ?:) as_incr_dir=$as_mkdir_dir ;;
+    *)
+      as_incr_dir=$as_incr_dir/$as_mkdir_dir
+      test -d "$as_incr_dir" || mkdir "$as_incr_dir"
+    ;;
+  esac
+done; }
+
+    ac_dir_suffix="/`echo $ac_dir|sed 's,^\./,,'`"
+    # A "../" for each directory in $ac_dir_suffix.
+    ac_dots=`echo "$ac_dir_suffix" | sed 's,/[^/]*,../,g'`
+  else
+    ac_dir_suffix= ac_dots=
+  fi
+
+  case $srcdir in
+  .)  ac_srcdir=.
+      if test -z "$ac_dots"; then
+         ac_top_srcdir=.
+      else
+         ac_top_srcdir=`echo $ac_dots | sed 's,/$,,'`
+      fi ;;
+  [\\/]* | ?:[\\/]* )
+      ac_srcdir=$srcdir$ac_dir_suffix;
+      ac_top_srcdir=$srcdir ;;
+  *) # Relative path.
+    ac_srcdir=$ac_dots$srcdir$ac_dir_suffix
+    ac_top_srcdir=$ac_dots$srcdir ;;
+  esac
+
+  case $INSTALL in
+  [\\/$]* | ?:[\\/]* ) ac_INSTALL=$INSTALL ;;
+  *) ac_INSTALL=$ac_dots$INSTALL ;;
+  esac
+
+  if test x"$ac_file" != x-; then
+    { echo "$as_me:5479: creating $ac_file" >&5
+echo "$as_me: creating $ac_file" >&6;}
+    rm -f "$ac_file"
+  fi
+  # Let's still pretend it is `configure' which instantiates (i.e., don't
+  # use $as_me), people would be surprised to read:
+  #    /* config.h.  Generated automatically by config.status.  */
+  configure_input="Generated automatically from `echo $ac_file_in |
+                                                 sed 's,.*/,,'` by configure."
+
+  # First look for the input files in the build tree, otherwise in the
+  # src tree.
+  ac_file_inputs=`IFS=:
+    for f in $ac_file_in; do
+      case $f in
+      -) echo $tmp/stdin ;;
+      [\\/$]*)
+         # Absolute (can't be DOS-style, as IFS=:)
+         test -f "$f" || { { echo "$as_me:5497: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         echo $f;;
+      *) # Relative
+         if test -f "$f"; then
+           # Build tree
+           echo $f
+         elif test -f "$srcdir/$f"; then
+           # Source tree
+           echo $srcdir/$f
+         else
+           # /dev/null tree
+           { { echo "$as_me:5510: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         fi;;
+      esac
+    done` || { (exit 1); exit 1; }
+EOF
+cat >>$CONFIG_STATUS <<EOF
+  sed "$ac_vpsub
+$extrasub
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+:t
+/@[a-zA-Z_][a-zA-Z_0-9]*@/!b
+s,@configure_input@,$configure_input,;t t
+s,@srcdir@,$ac_srcdir,;t t
+s,@top_srcdir@,$ac_top_srcdir,;t t
+s,@INSTALL@,$ac_INSTALL,;t t
+" $ac_file_inputs | (eval "$ac_sed_cmds") >$tmp/out
+  rm -f $tmp/stdin
+  if test x"$ac_file" != x-; then
+    mv $tmp/out $ac_file
+  else
+    cat $tmp/out
+    rm -f $tmp/out
+  fi
+
+done
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+
+#
+# CONFIG_HEADER section.
+#
+
+# These sed commands are passed to sed as "A NAME B NAME C VALUE D", where
+# NAME is the cpp macro being defined and VALUE is the value it is being given.
+#
+# ac_d sets the value in "#define NAME VALUE" lines.
+ac_dA='s,^\([ 	]*\)#\([ 	]*define[ 	][ 	]*\)'
+ac_dB='[ 	].*$,\1#\2'
+ac_dC=' '
+ac_dD=',;t'
+# ac_u turns "#undef NAME" without trailing blanks into "#define NAME VALUE".
+ac_uA='s,^\([ 	]*\)#\([ 	]*\)undef\([ 	][ 	]*\)'
+ac_uB='$,\1#\2define\3'
+ac_uC=' '
+ac_uD=',;t'
+
+for ac_file in : $CONFIG_HEADERS; do test "x$ac_file" = x: && continue
+  # Support "outfile[:infile[:infile...]]", defaulting infile="outfile.in".
+  case $ac_file in
+  - | *:- | *:-:* ) # input from stdin
+        cat >$tmp/stdin
+        ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  *:* ) ac_file_in=`echo "$ac_file" | sed 's,[^:]*:,,'`
+        ac_file=`echo "$ac_file" | sed 's,:.*,,'` ;;
+  * )   ac_file_in=$ac_file.in ;;
+  esac
+
+  test x"$ac_file" != x- && { echo "$as_me:5571: creating $ac_file" >&5
+echo "$as_me: creating $ac_file" >&6;}
+
+  # First look for the input files in the build tree, otherwise in the
+  # src tree.
+  ac_file_inputs=`IFS=:
+    for f in $ac_file_in; do
+      case $f in
+      -) echo $tmp/stdin ;;
+      [\\/$]*)
+         # Absolute (can't be DOS-style, as IFS=:)
+         test -f "$f" || { { echo "$as_me:5582: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         echo $f;;
+      *) # Relative
+         if test -f "$f"; then
+           # Build tree
+           echo $f
+         elif test -f "$srcdir/$f"; then
+           # Source tree
+           echo $srcdir/$f
+         else
+           # /dev/null tree
+           { { echo "$as_me:5595: error: cannot find input file: $f" >&5
+echo "$as_me: error: cannot find input file: $f" >&2;}
+   { (exit 1); exit 1; }; }
+         fi;;
+      esac
+    done` || { (exit 1); exit 1; }
+  # Remove the trailing spaces.
+  sed 's/[ 	]*$//' $ac_file_inputs >$tmp/in
+
+EOF
+
+# Transform confdefs.h into two sed scripts, `conftest.defines' and
+# `conftest.undefs', that substitutes the proper values into
+# config.h.in to produce config.h.  The first handles `#define'
+# templates, and the second `#undef' templates.
+# And first: Protect against being on the right side of a sed subst in
+# config.status.  Protect against being in an unquoted here document
+# in config.status.
+rm -f conftest.defines conftest.undefs
+# Using a here document instead of a string reduces the quoting nightmare.
+# Putting comments in sed scripts is not portable.
+#
+# `end' is used to avoid that the second main sed command (meant for
+# 0-ary CPP macros) applies to n-ary macro definitions.
+# See the Autoconf documentation for `clear'.
+cat >confdef2sed.sed <<\EOF
+s/[\\&,]/\\&/g
+s,[\\$`],\\&,g
+t clear
+: clear
+s,^[ 	]*#[ 	]*define[ 	][ 	]*\(\([^ 	(][^ 	(]*\)([^)]*)\)[ 	]*\(.*\)$,${ac_dA}\2${ac_dB}\1${ac_dC}\3${ac_dD},gp
+t end
+s,^[ 	]*#[ 	]*define[ 	][ 	]*\([^ 	][^ 	]*\)[ 	]*\(.*\)$,${ac_dA}\1${ac_dB}\1${ac_dC}\2${ac_dD},gp
+: end
+EOF
+# If some macros were called several times there might be several times
+# the same #defines, which is useless.  Nevertheless, we may not want to
+# sort them, since we want the *last* AC-DEFINE to be honored.
+uniq confdefs.h | sed -n -f confdef2sed.sed >conftest.defines
+sed 's/ac_d/ac_u/g' conftest.defines >conftest.undefs
+rm -f confdef2sed.sed
+
+# This sed command replaces #undef with comments.  This is necessary, for
+# example, in the case of _POSIX_SOURCE, which is predefined and required
+# on some systems where configure will not decide to define it.
+cat >>conftest.undefs <<\EOF
+s,^[ 	]*#[ 	]*undef[ 	][ 	]*[a-zA-Z_][a-zA-Z_0-9]*,/* & */,
+EOF
+
+# Break up conftest.defines because some shells have a limit on the size
+# of here documents, and old seds have small limits too (100 cmds).
+echo '  # Handle all the #define templates only if necessary.' >>$CONFIG_STATUS
+echo '  if egrep "^[ 	]*#[ 	]*define" $tmp/in >/dev/null; then' >>$CONFIG_STATUS
+echo '  # If there are no defines, we may have an empty if/fi' >>$CONFIG_STATUS
+echo '  :' >>$CONFIG_STATUS
+rm -f conftest.tail
+while grep . conftest.defines >/dev/null
+do
+  # Write a limited-size here document to $tmp/defines.sed.
+  echo '  cat >$tmp/defines.sed <<CEOF' >>$CONFIG_STATUS
+  # Speed up: don't consider the non `#define' lines.
+  echo '/^[ 	]*#[ 	]*define/!b' >>$CONFIG_STATUS
+  # Work around the forget-to-reset-the-flag bug.
+  echo 't clr' >>$CONFIG_STATUS
+  echo ': clr' >>$CONFIG_STATUS
+  sed ${ac_max_here_lines}q conftest.defines >>$CONFIG_STATUS
+  echo 'CEOF
+  sed -f $tmp/defines.sed $tmp/in >$tmp/out
+  rm -f $tmp/in
+  mv $tmp/out $tmp/in
+' >>$CONFIG_STATUS
+  sed 1,${ac_max_here_lines}d conftest.defines >conftest.tail
+  rm -f conftest.defines
+  mv conftest.tail conftest.defines
+done
+rm -f conftest.defines
+echo '  fi # egrep' >>$CONFIG_STATUS
+echo >>$CONFIG_STATUS
+
+# Break up conftest.undefs because some shells have a limit on the size
+# of here documents, and old seds have small limits too (100 cmds).
+echo '  # Handle all the #undef templates' >>$CONFIG_STATUS
+rm -f conftest.tail
+while grep . conftest.undefs >/dev/null
+do
+  # Write a limited-size here document to $tmp/undefs.sed.
+  echo '  cat >$tmp/undefs.sed <<CEOF' >>$CONFIG_STATUS
+  # Speed up: don't consider the non `#undef'
+  echo '/^[ 	]*#[ 	]*undef/!b' >>$CONFIG_STATUS
+  # Work around the forget-to-reset-the-flag bug.
+  echo 't clr' >>$CONFIG_STATUS
+  echo ': clr' >>$CONFIG_STATUS
+  sed ${ac_max_here_lines}q conftest.undefs >>$CONFIG_STATUS
+  echo 'CEOF
+  sed -f $tmp/undefs.sed $tmp/in >$tmp/out
+  rm -f $tmp/in
+  mv $tmp/out $tmp/in
+' >>$CONFIG_STATUS
+  sed 1,${ac_max_here_lines}d conftest.undefs >conftest.tail
+  rm -f conftest.undefs
+  mv conftest.tail conftest.undefs
+done
+rm -f conftest.undefs
+
+cat >>$CONFIG_STATUS <<\EOF
+  # Let's still pretend it is `configure' which instantiates (i.e., don't
+  # use $as_me), people would be surprised to read:
+  #    /* config.h.  Generated automatically by config.status.  */
+  if test x"$ac_file" = x-; then
+    echo "/* Generated automatically by configure.  */" >$tmp/config.h
+  else
+    echo "/* $ac_file.  Generated automatically by configure.  */" >$tmp/config.h
+  fi
+  cat $tmp/in >>$tmp/config.h
+  rm -f $tmp/in
+  if test x"$ac_file" != x-; then
+    if cmp -s $ac_file $tmp/config.h 2>/dev/null; then
+      { echo "$as_me:5712: $ac_file is unchanged" >&5
+echo "$as_me: $ac_file is unchanged" >&6;}
+    else
+      ac_dir=`$as_expr X"$ac_file" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
+         X"$ac_file" : 'X\(//\)[^/]' \| \
+         X"$ac_file" : 'X\(//\)$' \| \
+         X"$ac_file" : 'X\(/\)' \| \
+         .     : '\(.\)' 2>/dev/null ||
+echo X"$ac_file" |
+    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{ s//\1/; q; }
+  	  /^X\(\/\/\)[^/].*/{ s//\1/; q; }
+  	  /^X\(\/\/\)$/{ s//\1/; q; }
+  	  /^X\(\/\).*/{ s//\1/; q; }
+  	  s/.*/./; q'`
+      if test "$ac_dir" != "$ac_file" && test "$ac_dir" != .; then
+        { case "$ac_dir" in
+  [\\/]* | ?:[\\/]* ) as_incr_dir=;;
+  *)                      as_incr_dir=.;;
+esac
+as_dummy="$ac_dir"
+for as_mkdir_dir in `IFS='/\\'; set X $as_dummy; shift; echo "$@"`; do
+  case $as_mkdir_dir in
+    # Skip DOS drivespec
+    ?:) as_incr_dir=$as_mkdir_dir ;;
+    *)
+      as_incr_dir=$as_incr_dir/$as_mkdir_dir
+      test -d "$as_incr_dir" || mkdir "$as_incr_dir"
+    ;;
+  esac
+done; }
+
+      fi
+      rm -f $ac_file
+      mv $tmp/config.h $ac_file
+    fi
+  else
+    cat $tmp/config.h
+    rm -f $tmp/config.h
+  fi
+done
+EOF
+cat >>$CONFIG_STATUS <<\EOF
+
+#
+# CONFIG_COMMANDS section.
+#
+for ac_file in : $CONFIG_COMMANDS; do test "x$ac_file" = x: && continue
+  ac_dest=`echo "$ac_file" | sed 's,:.*,,'`
+  ac_source=`echo "$ac_file" | sed 's,[^:]*:,,'`
+
+  case $ac_dest in
+    default ) test -z "$CONFIG_HEADERS" || echo timestamp > stamp-h ;;
+  esac
+done
+EOF
+
+cat >>$CONFIG_STATUS <<\EOF
+
+{ (exit 0); exit 0; }
+EOF
+chmod +x $CONFIG_STATUS
+ac_clean_files=$ac_clean_files_save
+
+# configure is writing to config.log, and then calls config.status.
+# config.status does its own redirection, appending to config.log.
+# Unfortunately, on DOS this fails, as config.log is still kept open
+# by configure, so config.status won't be able to write to it; its
+# output is simply discarded.  So we exec the FD to /dev/null,
+# effectively closing config.log, so it can be properly (re)opened and
+# appended to by config.status.  When coming back to configure, we
+# need to make the FD available again.
+if test "$no_create" != yes; then
+  ac_cs_success=:
+  exec 5>/dev/null
+  $SHELL $CONFIG_STATUS || ac_cs_success=false
+  exec 5>>config.log
+  # Use ||, not &&, to avoid exiting from the if with $? = 1, which
+  # would make configure fail if this is the last instruction.
+  $ac_cs_success || { (exit 1); exit 1; }
+fi
+
Index: /trunk/mgpp/configure.in
===================================================================
--- /trunk/mgpp/configure.in	(revision 3365)
+++ /trunk/mgpp/configure.in	(revision 3365)
@@ -0,0 +1,106 @@
+dnl Process this file with autoconf to produce a configure script.
+AC_INIT(text/MGQuery.cpp)
+AC_CONFIG_HEADER(config.h)
+AC_ARG_PROGRAM
+
+dnl
+PACKAGE=mgpp
+VERSION=1.3
+
+AC_DEFINE_UNQUOTED(PACKAGE, "$PACKAGE")
+AC_DEFINE_UNQUOTED(VERSION, "$VERSION")
+AC_SUBST(PACKAGE)
+AC_SUBST(VERSION)
+
+dnl Checks for programs.
+AC_PROG_CXX
+AC_PROG_AWK
+AC_PROG_YACC
+AC_PROG_CC
+AC_PROG_INSTALL
+AC_PROG_LN_S
+AC_PROG_MAKE_SET
+AC_PROG_RANLIB
+
+dnl Checks for typedefs, structures, and compiler characteristics.
+AC_AIX
+AC_ISC_POSIX
+AC_MINIX
+fp_C_PROTOTYPES
+AC_C_CONST
+AC_TYPE_OFF_T
+AC_TYPE_SIZE_T
+AC_HEADER_TIME
+AC_STRUCT_TM
+
+dnl --- This should be in the stdc checking -----------------
+if test "$ac_cv_prog_cc_stdc" = '-Xc'; then
+dnl The Pyramids running SMP_DC.OSx on MIPS r3000 chips with
+dnl /usr/opt/cc3.11/bin/cc treat -Xc as full ANSI mode - no POSIX or
+dnl XOPEN extensions (like popen).
+dnl Unfortunately -D_{XOPEN,POSIX}_SOURCE can't provide all that mg
+dnl needs. So drop back to -Xa
+AC_TRY_COMPILE(
+[#include <stdio.h>
+], [ FILE *rubbish = popen("yes", "r"); ],,
+[CC="`echo $CC | sed 's/-Xc/-Xa/'`"    ac_cv_prog_cc_stdc='-Xa'])
+fi
+dnl ----------------------
+
+
+dnl Checks for libraries.
+dnl Replace `main' with a function in -lm:
+AC_CHECK_LIB(m, main)
+
+dnl Checks for header files.
+AC_HEADER_DIRENT
+AC_HEADER_STDC
+AC_CHECK_HEADERS(fcntl.h limits.h sys/time.h unistd.h string.h memory.h sys/procfs.h)
+AC_EGREP_HEADER(fread, stdio.h, AC_DEFINE(HAVE_FREAD_DECL))
+AC_EGREP_HEADER(fgetc, stdio.h, AC_DEFINE(HAVE_FGETC_DECL))
+AC_EGREP_HEADER(pr_brksize, sys/procfs.h, AC_DEFINE(HAVE_PR_BRKSIZE))
+
+dnl Checks for library functions.
+AC_FUNC_ALLOCA
+AC_PROG_GCC_TRADITIONAL
+AC_TYPE_SIGNAL
+AC_FUNC_VPRINTF
+AC_CHECK_FUNCS(ftime select strftime strtol getrusage times mallinfo setbuffer getpagesize strerror)
+AC_REPLACE_FUNCS(ftruncate strstr strcasecmp)
+
+
+# *** Custom checking (based on GNU tar configure.in) ***
+# ---------------------------------------------------------------------------
+AC_MSG_CHECKING(for HP-UX needing gmalloc)
+if test "`(uname -s) 2> /dev/null`" = 'HP-UX'; then
+  AC_MSG_RESULT(yes)
+  LIBOBJS="$LIBOBJS gmalloc.o"
+  AC_DEFINE(HAVE_VALLOC)
+else
+  AC_MSG_RESULT(no)
+  AC_CHECK_FUNCS(valloc)
+fi
+
+fp_WITH_DMALLOC
+fp_WITH_REGEX
+fp_WITH_GNU_READLINE
+
+# ---------------------------------------------------------------------------
+if test "$ac_cv_func_alloca" = 'no'; then
+dnl    support functions for alloca.c
+  LIBOBJS="$LIBOBJS xmalloc.o error.o"
+fi
+
+
+# ---------------------------------------------------------------------------
+# see if the user wants to override use of long long
+AC_ARG_ENABLE(override-longlong,
+[  --enable-override-longlong
+                          overide use of long long with GCC],
+  AC_DEFINE(DISABLE_LONG_LONG))
+
+# ---------------------------------------------------------------------------
+
+AC_OUTPUT([Makefile text/Makefile lib/Makefile 
+   jni/Makefile java/org/greenstone/mgpp/Makefile], 
+[test -z "$CONFIG_HEADERS" || echo timestamp > stamp-h])
Index: /trunk/mgpp/docs/Queryer_help.txt
===================================================================
--- /trunk/mgpp/docs/Queryer_help.txt	(revision 3365)
+++ /trunk/mgpp/docs/Queryer_help.txt	(revision 3365)
@@ -0,0 +1,67 @@
+Queryer is a standalone program enabling search and retrieval of documents
+in an mgpp collection. Even when using greenstone, it is useful for testing
+your collections.
+
+Queryer can be run like
+
+Queryer -f <index files> -t <text files>
+
+index and text files is the path to these files down to the collection name. 
+eg
+if your text files are demo/index/text/demo.t, demo/index/text/demo.ti etc, 
+then the text path is demo/index/text/demo
+similary for the index files.
+
+Commands available in the Queryer are:
+
+.q              quit
+.h              print the help message
+.i              change the search level (enter the new level at the prompt)
+.l              change the result level ( ""   ""  )
+.b              full text browse (enter a word or fragment at the prompt)
+.r0/.r1         ranking off/on
+.t0/.t1         query type some/all
+.c0/.c1         casefolding off/on
+.s0/.s1         stemming off/on
+.o0/.o1         short output off/on
+ 
+.p              print a document (enter the docnum at the prompt)
+otherwise just enter a query
+
+
+The current query syntax involves the following:
+
+boolean operators:  & AND | OR ! NOT, with () for precedence
+
+term modifiers: #icus /x - this is stemming, casefolding and weighting like 
+in gsdl
+
+     #i = case insensitive, #c = case sensitive
+     #u = unstemmed, #s = stemmed
+     /x = term weight (default = 1).
+
+eg computer#is/10  is computer, stemmed and casefolded, with a weight of 10
+compared to other terms in the same query
+
+Proximity searching:  NEARx
+this is used to specify the maximum distance apart two words must be to match
+eg dog NEAR4 cat - cat must be within 4 words either side of dog.
+NEAR by itself defaults to 20(??).
+
+fielded searching: [ terms]:Field
+
+eg [Witten]:Creator
+[Witten]:CR
+
+the field names need to be the names of the metadata elements in your 
+collection. If the collection was built with greenstone, these names are the 
+two letter codes found in the build.cfg file. 
+
+Multiple terms inside the [] are ANDed together.
+
+Different fields can be combined using normal boolean stuff, eg
+
+[Witten]:CR & [Gigabytes]:TI
+
+Term modifiers can be included inside the [].
+
Index: /trunk/mgpp/docs/mgpp_in_greenstone.txt
===================================================================
--- /trunk/mgpp/docs/mgpp_in_greenstone.txt	(revision 3365)
+++ /trunk/mgpp/docs/mgpp_in_greenstone.txt	(revision 3365)
@@ -0,0 +1,116 @@
+BUILDING
+
+Greenstone can build collections using mg or mgpp. The default is mg, but you
+can use mgpp by editing the collection configuration file.
+
+First, add the line 'buildtype	 mgpp'
+
+Second, the way indexes are described is different.
+
+mg uses a line like:
+
+indexes   document:text  section:text,Title
+
+This builds two indexes, one of all the text, at document level, the second one
+of all the text and Title metadata, at section level.
+
+The document and section tags determine the granularity of the results of a 
+search. The first index returns document numbers, while the second index 
+returns section numbers.
+
+mgpp does things differently. By default it builds a word level index. Then 
+you specify levels at which you want results returned. For example, in the 
+one index, you might want to be able to retrieve whole documents, and sections.
+
+The greenstone building code builds a word level index, with Document level
+granularity. To add other levels (Section and Paragraph are permitted), you add
+a line like
+
+levels   Section  Paragraph
+
+Note that Paragraph level indexes can be used for searching, but you cant 
+retrieve Paragraph level documents, only Section and Document.
+
+To specify what goes into the index, we use an indexes line, similar to mg but 
+without the level information (it is specified separately by the levels info).
+eg:
+
+indexes	  text
+
+This will index all the text at word level.
+
+To add metadata fields to the index, you can say
+
+indexes   text,Title,Subject       for example, or
+indexes	  text,metadata
+
+The first one builds one index, with tagged entries for Title and Subject 
+metadata. Unlike levels, metadata names can be anything - obviously they 
+should match the names in your documents though.
+
+The second one builds one index with tagged entries for all the metadata it 
+finds - this is useful if you dont know in advance what metadata are available,
+or want all of it indexed anyway.
+
+After the building has finished, the build.cfg file in the building directory
+has a list of what metadata it has found and indexed, for example
+
+indexfields     Subject TextOnly        Title
+indexfieldmap   TextOnly->TX    Subject->SU     Title->TI
+
+The metadata names are  passed to mgpp during building as two letter codes -
+indexfieldmap specifies what codes were used.
+
+By default, only the text is compressed, not the metadata. To change this, you
+ can add a line to the config file like
+
+textcompress	text,Title
+
+this will add Title metadata to the text that gets passed to the compressor.
+
+QUERYING
+
+A collection built with mgpp can be searched in the usual way through 
+greenstone. Search terms can be combined with & and |, phrases are specified using "". Because it uses a word level index, it has some extended searching capability over mg. If metadata has been specified in the index, fielded search can also be done.
+
+The current query syntax involves the following:
+
+boolean operators:  & AND | OR ! NOT, with () for precedence
+
+term modifiers: #icus /x - this is stemming, casefolding and weighting like 
+in gsdl
+
+     #i = case insensitive, #c = case sensitive
+     #u = unstemmed, #s = stemmed
+     /x = term weight (default = 1).
+
+eg computer#is/10  is computer, stemmed and casefolded, with a weight of 10
+compared to other terms in the same query
+
+Proximity searching:  NEARx
+this is used to specify the maximum distance apart two words must be to match
+eg dog NEAR4 cat - cat must be within 4 words either side of dog.
+NEAR by itself defaults to 20(??).
+
+fielded searching: [ terms]:Field
+
+eg [Witten]:CR
+
+the field names need to be the names of the metadata elements in your 
+collection. If the collection was built with greenstone, these names are the two letter codes found in the build.cfg file. 
+
+Multiple terms inside the [] are ANDed together.
+
+Different fields can be combined using normal boolean stuff, eg
+
+[Witten]:CR & [Gigabytes]:TI
+
+Term modifiers can be included inside the [].
+
+
+This syntax can be entered into the standard greenstone search box. For mgpp 
+collections, however, there are additional query pages using forms. These can 
+be accessed through the preferences page - select form query, then simple/
+advanced.
+hopefully the forms are fairly self explanatory.
+
Index: /trunk/mgpp/docs/standalone_building.txt
===================================================================
--- /trunk/mgpp/docs/standalone_building.txt	(revision 3365)
+++ /trunk/mgpp/docs/standalone_building.txt	(revision 3365)
@@ -0,0 +1,99 @@
+BUILDING COLLECTIONS OUTSIDE OF GREENSTONE
+
+
+The programs for building are like mgpp_passes, mgpp_invf_dict etc. they have 
+man pages with them, like mgpp_passes.1 etc. mgpp_passes is the main
+program used for building and the man page has a simple script that gives an 
+example of how to build a collection.
+
+Here is a simple bash script that can be used to build a collection, 
+  
+#! /bin/bash 
+
+
+# The arguments on the command line specify the
+#   source of the text
+source=$@
+
+# This is the name of the collection
+text=demo
+
+echo $source
+
+# Create *.text.stats, *.invf.dict, *.invf.level
+#   *.invf.chunk and *.invf.chunks.trans
+cat ${source} | mgpp_passes -T1 -I1 -f ${text}
+
+# Create *.text.dict
+mgpp_compression_dict -f ${text}
+
+# Create *.invf.dict.hash
+mgpp_perf_hash_build -f ${text}
+
+# Create *.text, *.text.idx, *.text.level
+#   *.invf and *.invf.idx
+cat ${source} | mgpp_passes -T2 -I2 -f ${text}
+
+# Create *.text.weight and *.weight.approx
+mgpp_weights_build -f ${text}
+
+# Create *.invf.dict.blocked
+mgpp_invf_dict -f ${text}
+
+# Create *.invf.dict.blocked.1
+mgpp_stem_idx -s 1 -f ${text}
+
+# Create *.invf.dict.blocked.2
+mgpp_stem_idx -s 2 -f ${text}
+
+# Create *.invf.dict.blocked.3
+mgpp_stem_idx -s 3 -f ${text}
+
+
+This builds a basic collection, using 'Document' as the document level tag, 
+with a word level index.
+
+Format of documents:
+
+There must be a document level tag thats starts (and optionally ends) each 
+document. The default is 'Document'. Text outside these tags is ignored. To 
+change the document tag add '-J <tagname>' to the mgpp_passes commands. There
+can only be one -J option.
+
+Smaller granularity is added using level tags. These must enclose all the
+ text enclosed by the document tags. They can be specified to mgpp_passes by 
+'-K <level tag>'. There can be many -K options.
+
+The smallest level of granularity (default is word level) can be changed by 
+adding '-L <index level>'. This can be no larger than the smallest level.
+
+Metadata or tagged fields are specified like
+<Title>the text of the title </Title>
+
+these are all automatically indexed and can be searched as fields. 
+
+The output files are placed in the current directory. To change where they go,
+use '-d <directory name> ' for all the commands.
+
+The man pages describe all the different options to the various programs.
+
+
+There are four stages to mgpp_passes, specified by T1 and T2 (for text 
+compression) and I1 and I2 (for indexing). T1 and I1 must be done before T2 
+and I2, respectively. As shown above, T1/I1 and T2/I2 can be run together, 
+reducing the number of passes through the documents to 2. But all output files
+get placed in the same directory, and the same text is passed to the indexer
+and the compressor.
+
+In greenstone, we pass only the text to the compressor, but metadata and text
+to the indexer, and text files and index files are put in separate directories.
+
+So mgpp_passes is run four times in this case, like:
+
+cat <text src> | mgpp_passes -d <text_dir> -T1 ...
+cat <text src> | mgpp_passes -d <text_dir> -T2 ...
+cat <index src> | mgpp_passes -d <index_dir> -I1 ...
+cat <index src> | mgpp_passes -d <index_dir> -I2 ...
+
+
+
Index: /trunk/mgpp/install-sh
===================================================================
--- /trunk/mgpp/install-sh	(revision 3365)
+++ /trunk/mgpp/install-sh	(revision 3365)
@@ -0,0 +1,238 @@
+#! /bin/sh
+#
+# install - install a program, script, or datafile
+# This comes from X11R5.
+#
+# Calling this script install-sh is preferred over install.sh, to prevent
+# `make' implicit rules from creating a file called install from it
+# when there is no Makefile.
+#
+# This script is compatible with the BSD install script, but was written
+# from scratch.
+#
+
+
+# set DOITPROG to echo to test this script
+
+# Don't use :- since 4.3BSD and earlier shells don't like it.
+doit="${DOITPROG-}"
+
+
+# put in absolute paths if you don't have them in your path; or use env. vars.
+
+mvprog="${MVPROG-mv}"
+cpprog="${CPPROG-cp}"
+chmodprog="${CHMODPROG-chmod}"
+chownprog="${CHOWNPROG-chown}"
+chgrpprog="${CHGRPPROG-chgrp}"
+stripprog="${STRIPPROG-strip}"
+rmprog="${RMPROG-rm}"
+mkdirprog="${MKDIRPROG-mkdir}"
+
+tranformbasename=""
+transform_arg=""
+instcmd="$mvprog"
+chmodcmd="$chmodprog 0755"
+chowncmd=""
+chgrpcmd=""
+stripcmd=""
+rmcmd="$rmprog -f"
+mvcmd="$mvprog"
+src=""
+dst=""
+dir_arg=""
+
+while [ x"$1" != x ]; do
+    case $1 in
+	-c) instcmd="$cpprog"
+	    shift
+	    continue;;
+
+	-d) dir_arg=true
+	    shift
+	    continue;;
+
+	-m) chmodcmd="$chmodprog $2"
+	    shift
+	    shift
+	    continue;;
+
+	-o) chowncmd="$chownprog $2"
+	    shift
+	    shift
+	    continue;;
+
+	-g) chgrpcmd="$chgrpprog $2"
+	    shift
+	    shift
+	    continue;;
+
+	-s) stripcmd="$stripprog"
+	    shift
+	    continue;;
+
+	-t=*) transformarg=`echo $1 | sed 's/-t=//'`
+	    shift
+	    continue;;
+
+	-b=*) transformbasename=`echo $1 | sed 's/-b=//'`
+	    shift
+	    continue;;
+
+	*)  if [ x"$src" = x ]
+	    then
+		src=$1
+	    else
+		# this colon is to work around a 386BSD /bin/sh bug
+		:
+		dst=$1
+	    fi
+	    shift
+	    continue;;
+    esac
+done
+
+if [ x"$src" = x ]
+then
+	echo "install:	no input file specified"
+	exit 1
+else
+	true
+fi
+
+if [ x"$dir_arg" != x ]; then
+	dst=$src
+	src=""
+	
+	if [ -d $dst ]; then
+		instcmd=:
+	else
+		instcmd=mkdir
+	fi
+else
+
+# Waiting for this to be detected by the "$instcmd $src $dsttmp" command
+# might cause directories to be created, which would be especially bad 
+# if $src (and thus $dsttmp) contains '*'.
+
+	if [ -f $src -o -d $src ]
+	then
+		true
+	else
+		echo "install:  $src does not exist"
+		exit 1
+	fi
+	
+	if [ x"$dst" = x ]
+	then
+		echo "install:	no destination specified"
+		exit 1
+	else
+		true
+	fi
+
+# If destination is a directory, append the input filename; if your system
+# does not like double slashes in filenames, you may need to add some logic
+
+	if [ -d $dst ]
+	then
+		dst="$dst"/`basename $src`
+	else
+		true
+	fi
+fi
+
+## this sed command emulates the dirname command
+dstdir=`echo $dst | sed -e 's,[^/]*$,,;s,/$,,;s,^$,.,'`
+
+# Make sure that the destination directory exists.
+#  this part is taken from Noah Friedman's mkinstalldirs script
+
+# Skip lots of stat calls in the usual case.
+if [ ! -d "$dstdir" ]; then
+defaultIFS='	
+'
+IFS="${IFS-${defaultIFS}}"
+
+oIFS="${IFS}"
+# Some sh's can't handle IFS=/ for some reason.
+IFS='%'
+set - `echo ${dstdir} | sed -e 's@/@%@g' -e 's@^%@/@'`
+IFS="${oIFS}"
+
+pathcomp=''
+
+while [ $# -ne 0 ] ; do
+	pathcomp="${pathcomp}${1}"
+	shift
+
+	if [ ! -d "${pathcomp}" ] ;
+        then
+		$mkdirprog "${pathcomp}"
+	else
+		true
+	fi
+
+	pathcomp="${pathcomp}/"
+done
+fi
+
+if [ x"$dir_arg" != x ]
+then
+	$doit $instcmd $dst &&
+
+	if [ x"$chowncmd" != x ]; then $doit $chowncmd $dst; else true ; fi &&
+	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd $dst; else true ; fi &&
+	if [ x"$stripcmd" != x ]; then $doit $stripcmd $dst; else true ; fi &&
+	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd $dst; else true ; fi
+else
+
+# If we're going to rename the final executable, determine the name now.
+
+	if [ x"$transformarg" = x ] 
+	then
+		dstfile=`basename $dst`
+	else
+		dstfile=`basename $dst $transformbasename | 
+			sed $transformarg`$transformbasename
+	fi
+
+# don't allow the sed command to completely eliminate the filename
+
+	if [ x"$dstfile" = x ] 
+	then
+		dstfile=`basename $dst`
+	else
+		true
+	fi
+
+# Make a temp file name in the proper directory.
+
+	dsttmp=$dstdir/#inst.$$#
+
+# Move or copy the file name to the temp name
+
+	$doit $instcmd $src $dsttmp &&
+
+	trap "rm -f ${dsttmp}" 0 &&
+
+# and set any options; do chmod last to preserve setuid bits
+
+# If any of these fail, we abort the whole thing.  If we want to
+# ignore errors from any of these, just make sure not to ignore
+# errors from the above "$doit $instcmd $src $dsttmp" command.
+
+	if [ x"$chowncmd" != x ]; then $doit $chowncmd $dsttmp; else true;fi &&
+	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd $dsttmp; else true;fi &&
+	if [ x"$stripcmd" != x ]; then $doit $stripcmd $dsttmp; else true;fi &&
+	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd $dsttmp; else true;fi &&
+
+# Now rename the file to the real destination.
+
+	$doit $rmcmd -f $dstdir/$dstfile &&
+	$doit $mvcmd $dsttmp $dstdir/$dstfile 
+
+fi &&
+
+
+exit 0
Index: /trunk/mgpp/java/org/greenstone/mgpp/MGPPDocInfo.java
===================================================================
--- /trunk/mgpp/java/org/greenstone/mgpp/MGPPDocInfo.java	(revision 3365)
+++ /trunk/mgpp/java/org/greenstone/mgpp/MGPPDocInfo.java	(revision 3365)
@@ -0,0 +1,20 @@
+package org.greenstone.mgpp;
+
+/** DocInfo 'struct' class for a search result from mgpp
+ * used by MGPPQueryResult
+ *
+ * @See MGPPQueryResult
+ */
+public class MGPPDocInfo {
+
+    public long num_=0;
+    public float rank_=0;
+
+    public MGPPDocInfo(long doc, float rank) {
+	num_=doc;
+	rank_=rank;
+    }
+    public String toString(){
+	return ""+num_+"("+rank_+")";
+    }
+}
Index: /trunk/mgpp/java/org/greenstone/mgpp/MGPPQueryResult.java
===================================================================
--- /trunk/mgpp/java/org/greenstone/mgpp/MGPPQueryResult.java	(revision 3365)
+++ /trunk/mgpp/java/org/greenstone/mgpp/MGPPQueryResult.java	(revision 3365)
@@ -0,0 +1,97 @@
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+
+/** a java version of QueryResult for mgpp
+ *
+ * contains get methods for the java side, and simple
+ * set methods to be called from c++ side
+ *
+ */
+public class MGPPQueryResult {
+
+    /** the list of MGPPDocInfo */
+    protected Vector docs_=null;
+    /** the list of MGPPTermInfo */
+    protected Vector terms_=null;
+    /** the total number of docs found - not likely to be the size of docs_*/
+    protected long total_num_docs_=0;
+    
+    MGPPQueryResult() {
+	docs_ = new Vector();
+	terms_ = new Vector();
+    }
+    /** clear the info from the last query - should be called before setting any new docs/terms */
+    public void clear() {
+	total_num_docs_=0;
+	docs_.clear();
+	terms_.clear();
+    }
+    /** returns the result as a String - useful for printing out results */
+    public String toString() {
+	String result = "";
+	result += "docs (ranks): ";
+	for (int i=0; i<docs_.size(); i++) {
+	    result += ((MGPPDocInfo)docs_.elementAt(i)).toString()+", ";
+	}
+	result += "\nterms: ";
+	for (int i=0; i<terms_.size(); i++) {
+	    result += ((MGPPTermInfo)terms_.elementAt(i)).toString()+", ";
+	}
+	result += "\nactual number of docs found = "+total_num_docs_;
+
+	return result;
+    }
+    /** a shorter representation - just terms and total docs - not the 
+	individual docnums and ranks */
+    public String toShortString() {
+	String result = "";
+	result += "\nterms: ";
+	for (int i=0; i<terms_.size(); i++) {
+	    result += ((MGPPTermInfo)terms_.elementAt(i)).toString()+", ";
+	}
+	result += "\nactual number of docs found = "+total_num_docs_;
+	return result;
+    }
+
+    // set methods used by c++ code
+    
+    public void setTotalDocs(long num) {
+	total_num_docs_=num;
+    }
+
+    public void addDoc(long doc, float rank) {
+	MGPPDocInfo doc_info = new MGPPDocInfo(doc, rank);
+	docs_.add(doc_info);
+    }
+
+    public void addTerm(String term, String tag, int stem, long match, long freq, String[] equivs) {
+	MGPPTermInfo ti = new MGPPTermInfo();
+	ti.term_=term;
+	ti.tag_=tag;
+	ti.stem_method_=stem;
+	ti.match_docs_=match;
+	ti.term_freq_=freq;
+	if (equivs!=null) {
+	  for (int i=0; i<equivs.length; i++) {
+	    ti.equiv_terms_.add(equivs[i]);
+	  }
+	}
+	terms_.add(ti);
+    }
+
+    // get methods for the java side
+    public Vector getDocs() {
+	return docs_;
+    }
+
+    public Vector getTerms() {
+	return terms_;
+    }
+
+    public long getTotalDocs() {
+	return total_num_docs_;
+    }
+       
+}
+
Index: /trunk/mgpp/java/org/greenstone/mgpp/MGPPTermInfo.java
===================================================================
--- /trunk/mgpp/java/org/greenstone/mgpp/MGPPTermInfo.java	(revision 3365)
+++ /trunk/mgpp/java/org/greenstone/mgpp/MGPPTermInfo.java	(revision 3365)
@@ -0,0 +1,46 @@
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+
+/** TermInfo 'struct' class for a search result from mgpp
+ * used by MGPPQueryResult
+ * 
+ *
+ *@see MGPPQueryResult
+ */
+public class MGPPTermInfo {
+
+    /** the term itself */
+    public String term_=null;
+    /** the tag - level or metadata - for which the query was done */
+    public String tag_=null;
+    /** what stem and casefold method was used
+	0 = none
+	1 = casefold only
+	2 = stem only
+	3 = casefold and stem */
+    public int stem_method_=0;
+    /** the number of documents containing this term (where document is defined by the tag data */
+    public long match_docs_=0;
+    /** overall term freq  - word level*/
+    public long term_freq_=0;
+    /** list of stemmed and casefolded equivalent terms - if stem_method_ is non-zero 
+      - Vector of Strings */
+    public Vector equiv_terms_=null;
+
+    public MGPPTermInfo() {
+	equiv_terms_ = new Vector();
+    }
+	
+    /** output the class as a string */
+    public String toString() {
+	String result="";
+	result +="<"+tag_+">\""+term_+"\"stem("+stem_method_+")docs("+match_docs_;
+	result +=")freq("+term_freq_+")equiv terms(";
+	for (int i=0; i<equiv_terms_.size(); i++) {
+	    result += (String)equiv_terms_.elementAt(i)+", ";
+	}
+	result += ")";
+	return result;
+    }
+}
Index: /trunk/mgpp/java/org/greenstone/mgpp/MGPPWrapper.java
===================================================================
--- /trunk/mgpp/java/org/greenstone/mgpp/MGPPWrapper.java	(revision 3365)
+++ /trunk/mgpp/java/org/greenstone/mgpp/MGPPWrapper.java	(revision 3365)
@@ -0,0 +1,105 @@
+
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+
+/** java wrapper class for access to mgpp in C++
+ *
+ * the native side implemented in MGPPWrapperImpl.cpp
+ * uses MGPPQueryResult to hold the result of a query. the 
+ * result of a getDocument is a String
+ * uses the jni
+ *
+ *@see MGPPQueryResult
+ */
+public class MGPPWrapper {
+  
+    /** the query result
+     * will be filled in by runQuery
+     */
+    protected MGPPQueryResult mgpp_query_result_=null; 
+    /** pointer to c++ MGPPWrapperData class - cached indexData and queryInfo
+     */
+    protected long mgpp_data_ptr_=0;
+
+    static {
+	System.loadLibrary("mgppjni");
+	initIDs();
+    }
+    
+    public MGPPWrapper() {
+	mgpp_query_result_ = new MGPPQueryResult();
+	initCppSide();
+    }
+
+    /** initialises field and method IDs for java side to enable access on
+	C++ side */
+    private static native void initIDs();
+
+    /** initialises the mgpp_data_ptr_ */
+    private native boolean initCppSide();
+
+    /** returns a document: number docnum at level level 
+     * the base_dir and text_path paths should join together to provide
+     * the absolute location of the mgpp text files eg ..../index/text/demo
+     * returns teh doc in utf-8
+     */
+    public native String getDocument(String base_dir,
+				     String text_path,
+				     String level,
+				     long docnum );
+
+    /** the public method - converts between utf-8 and unicode
+     */
+    // public String getDocument(String base_dir,
+    //		      String text_path,
+    //		      String level,
+    //		      long docnum ) {
+
+    //String utf8_doc = getUtF8Document(base_dir, text_path, level, docnum);
+	
+    /** load up the data structure for index 
+     * - maintains state between requests as can be slow  
+     * base_dir and text_path should join together to provide
+     * the absolute location of the mgpp index files eg ..../index/tt/demo 
+     */
+    public native boolean loadIndexData(String base_dir,
+					String index_path);
+    /** unloads the data */
+    public native boolean unloadIndexData();
+
+    // query param methods
+
+    /** if on=true, sets default stemming on  - its off by default*/
+    public native void setStem(boolean on);
+    /** if on=true, sets default casefolding on - its off by default */
+    public native void setCase(boolean on);
+    /** default is 50 */
+    public native void setMaxDocs(int num);
+    /** if on=true, sorts by rank, otherwise returns in build order - 
+	default is on */
+    public native void setSortByRank(boolean on);
+    /** if on=true, a query returns term freq info - default is on */
+    public native void setReturnTerms(boolean on);
+    /** sets the granularity of the  query  - default 'Section' */
+    public native void setQueryLevel(String level);
+    /** sets the granularity of the result - default 'Section' */
+    public native void setReturnLevel(String level);
+    /** sets the default boolean operator - AND(=1)/OR(=0) */
+    public native void setMatchMode(int mode);
+
+    /** returns a string with all the current query param settings */
+    public native String getQueryParams();
+
+    /** actually carry out the query. Use the set methods to set any query
+	params 
+    * writes the result to query_result
+    */
+    public native void runQuery(String query_string); 
+
+    /** get the result out of the wrapper */
+    public MGPPQueryResult getQueryResult() {
+	return mgpp_query_result_;
+    }
+    
+}
Index: /trunk/mgpp/java/org/greenstone/mgpp/Makefile.in
===================================================================
--- /trunk/mgpp/java/org/greenstone/mgpp/Makefile.in	(revision 3365)
+++ /trunk/mgpp/java/org/greenstone/mgpp/Makefile.in	(revision 3365)
@@ -0,0 +1,64 @@
+
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+bindir = @bindir@
+srcdir = @srcdir@
+libdir = @libdir@
+subdir = java/org/greenstone/mgpp
+top_srcdir = @top_srcdir@
+transform = @program_transform_name@
+
+AR = ar
+CC = @CC@
+CXX = @CXX@
+JAVAC = javac
+JAVAH = javah
+JAR = jar
+CFLAGS = @CFLAGS@ -ansi -DSILENT -DSHORT_SUFFIX
+CXXFLAGS = @CXXFLAGS@ -DSILENT -DSHORT_SUFFIX
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS =  -lstdc++ @LIBS@
+TOUCH = echo timestamp >
+AWK = @AWK@
+
+MGPPHOME=../../../..
+JAVALIBDIR=$(libdir)/java
+JAVACLASSDIR=./classes
+JAVASRCDIR=../../..
+JAVACOPTIONS= -deprecation -g -O
+JAVACLASSPATH=$(JAVACLASSDIR):
+
+datadir = $(prefix)/share
+libexecdir = $(exec_prefix)/libexec
+localedir = $(datadir)/locale
+mandir = $(prefix)/man/man1
+
+
+all: setup compile
+
+setup:
+	mkdir -p $(JAVACLASSDIR)
+
+compile: setup
+	$(JAVAC) -d $(JAVACLASSDIR) -sourcepath $(JAVASRCDIR) $(JAVACOPTIONS) *.java
+	$(JAVAH) -classpath $(CLASSPATH) -d $(MGPPHOME)/jni org.greenstone.mgpp.MGPPWrapper
+	$(JAR) cf mgpp.jar -C $(JAVACLASSDIR) org
+
+clean:
+	rm -rf $(JAVACLASSDIR) mgpp.jar $(MGPPHOME)/jni/org_greenstone_mgpp_MGPPWrapper.h
+
+install:
+	$(MGPPHOME)/mkinstalldirs $(JAVALIBDIR); \
+	cp mgpp.jar $(JAVALIBDIR)/
+
+distclean: clean
+	rm -f Makefile
+
+
Index: /trunk/mgpp/java/org/greenstone/mgpp/Queryer.java
===================================================================
--- /trunk/mgpp/java/org/greenstone/mgpp/Queryer.java	(revision 3365)
+++ /trunk/mgpp/java/org/greenstone/mgpp/Queryer.java	(revision 3365)
@@ -0,0 +1,176 @@
+package org.greenstone.mgpp;
+
+import java.util.Vector;
+import java.io.BufferedReader;
+import java.io.InputStreamReader;
+
+/** Queryer - java port of c++ Queryer 
+ * - uses MGPPWrapper to access mgpp
+ * only has get document and search - full text browse not implemented
+ *
+ * run as:
+ * java org.greenstone.mgpp.Queryer <basedir> <indexdir> <textdir>
+ *@see MGPPWrapper
+ */
+public class Queryer {
+  
+  /** outputs to std out a help message describing the commands used by 
+   * Queryer 
+   */
+  public void printHelp() {
+	
+	System.out.println( "commands available are:\n"+
+			    "\t.q\t\tquit\n"+
+			    "\t.h\t\tprint this help message\n"+
+			    "\t.d\t\tprint the current query parameter settings\n" +
+			    "\t.i<level>\tchange the search level to level\n"+
+			    "\t.l<level>\tchange the result level to level\n"+
+			    "\t.r0/.r1\t\tranking off/on\n"+
+			    "\t.t0/.t1\t\tquery type some/all\n"+
+			    "\t.c0/.c1\t\tcasefolding off/on\n"+
+			    "\t.s0/.s1\t\tstemming off/on\n"+
+			    "\t.o0/.o1\t\tshort output off/on\n"+
+			    "\t.m<num>\t\tset max docs to return to num\n\n"+
+			    "\t.p<docnum>\tprint document docnum\n"+
+			    "\t<query string>\t do a query\n");
+    }
+
+   
+    public static void main(String[] args) {
+	if (args.length != 3) {
+	    System.out.println("Usage: java org.greenstone.mgpp.Queryer <basedir> <indexdir> <textdir>");
+	    return;
+	}
+	
+	Queryer self = new Queryer();
+	
+	String base_dir = args[0];
+	String text_path = args[2];
+	String index_path = args[1];
+
+	// the jni class to access mgpp stuff
+	MGPPWrapper wrapper = new MGPPWrapper();
+	wrapper.loadIndexData(base_dir, index_path);
+	
+	// the return level
+	String level = "Section";
+
+	System.out.println("Welcome to Java Queryer :-)");
+	self.printHelp();
+
+	BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
+	boolean shortOutput=false;
+
+	String command; // user input
+	char x; // the command letter
+	String data=null; // any auxiliary input to a command
+	while (true) {
+	  System.out.print(">"); // the prompt
+            try {
+                command = br.readLine();
+                command=command.trim();
+		if (command.startsWith(".")) {
+		    // a system command
+		    x = command.charAt(1);
+		    if (command.length() > 2) {
+			data = command.substring(2);
+			data = data.trim();
+			
+		    }
+		    switch (x) {
+		    case 'q': // clean up and exit
+			wrapper.unloadIndexData();
+			return;  
+		    case 'h': // print help message
+			self.printHelp();
+			break;
+		    case 'd': // print query param settings
+			String info = wrapper.getQueryParams();
+			System.out.println(info);
+			break;
+		    case 'p': // print doc
+			int docnum = Integer.parseInt(data);
+			String doc = wrapper.getDocument(base_dir, text_path, 
+							 level, docnum);
+			System.out.println(doc);
+			break;
+		    case 'm': //match docs
+			int match = Integer.parseInt(data);
+			wrapper.setMaxDocs(match);
+			break;
+		    case 's': // set stem on/off
+			int stem = Integer.parseInt(data);
+			if (stem==0 ){
+			    wrapper.setStem(false);
+			} else if(stem==1) {
+			    wrapper.setStem(true);
+			} else {
+			    System.err.println("Error: stem should be 0 or 1");
+			}
+			break;
+		    case 'c': // set case on/off
+			int casef = Integer.parseInt(data);
+			if (casef==0) {
+			    wrapper.setCase(false);
+			} else if (casef==1) {
+			    wrapper.setCase(true);
+			} else {
+			    System.err.println("Error: case should be 0 or 1");
+			}
+			break;
+		    case 'i': // set search level
+			wrapper.setQueryLevel(data);
+			break;
+		    case 'l': // set result level
+			wrapper.setReturnLevel(data);
+			break; 
+		    case 'r': // set rank on/off
+			int rank = Integer.parseInt(data);
+			if (rank==0) {
+			    wrapper.setSortByRank(false);
+			} else if (rank==1) {
+			    wrapper.setSortByRank(true);
+			} else {
+			    System.err.println("Error: rank should be 0 or 1");
+			}
+			break;
+		    case 't': // set query type some/all
+			int type = Integer.parseInt(data);
+			if (type==0 || type==1) {
+			    wrapper.setMatchMode(type);
+			} else {
+			    System.err.println("Error: type should be 0 (some) or 1 (all)");
+			}
+			break;
+		    case 'o': // set output short/long
+			int output = Integer.parseInt(data);
+			if (output==0) {
+			    shortOutput = false;
+			} else if (output==1) {
+			    shortOutput = true;
+			}else {
+			    System.err.println("Error: output should be 0 or 1");
+			}
+			break;
+		    }
+		}
+		else {
+		    // a query
+		    wrapper.runQuery(command);
+		    MGPPQueryResult res = wrapper.getQueryResult();
+		    if (shortOutput) {
+		      System.out.println(res.toShortString());
+		    } else {
+		      System.out.println(res.toString());
+		    }
+		}
+		    
+	    } catch (Exception e) {
+		System.out.println("Queryer error: "+e.getClass() + " "+e.getMessage());
+	    }
+
+	}
+    }
+
+
+}
Index: /trunk/mgpp/jni/MGPPWrapperImpl.cpp
===================================================================
--- /trunk/mgpp/jni/MGPPWrapperImpl.cpp	(revision 3365)
+++ /trunk/mgpp/jni/MGPPWrapperImpl.cpp	(revision 3365)
@@ -0,0 +1,486 @@
+#include <jni.h>
+#include "org_greenstone_mgpp_MGPPWrapper.h"
+#include "MGPPWrapperImpl.h"
+#include "TextGet.h"
+#include "GSDLQueryParser.h"
+#include "MGQuery.h"
+#include <iostream.h>
+#include <strstream.h>
+  
+MGPPWrapperData::MGPPWrapperData() {
+  indexData = new IndexData();
+  queryInfo = new QueryInfo();
+
+  if (queryInfo==NULL) {
+    cerr<<"couldn't allocate new query info\n";
+    if (indexData!=NULL) {
+      delete indexData;
+    }
+  }
+
+  // set all the default params
+  SetCStr(queryInfo->docLevel, "Section"); // the level to search at
+  queryInfo->maxDocs = 50;
+  queryInfo->sortByRank = true;
+  queryInfo->exactWeights = false;
+  queryInfo->needRankInfo = true;
+  queryInfo->needTermFreqs = true;
+
+  UCArrayClear(level);
+  SetCStr(level, "Section"); // the level to return docs at
+  defaultStemMethod=0;
+  defaultBoolCombine=0;
+  
+}
+
+MGPPWrapperData::~MGPPWrapperData() { 
+  if (indexData !=NULL) {
+    delete indexData;
+  }
+  if (queryInfo !=NULL) {
+    delete queryInfo;
+  }
+}
+
+// ********************************************
+// initialisation stuff
+// ********************************************
+
+// cached ids for java stuff 
+jfieldID FID_mgpp_data = NULL; // MGPPWrapperData
+jfieldID FID_query_result = NULL; // MGPPQueryResult
+jmethodID MID_addDoc=NULL; // MGPPQueryResult.addDoc()
+jmethodID MID_addTerm=NULL; // MGPPQueryResult.addTerm()
+jmethodID MID_setTotalDocs=NULL; // MGPPQueryResult.setTotalDocs()
+jmethodID MID_clearResult=NULL; //MGPPQueryResult.clear()
+jclass CID_String=NULL; // class ID of String
+
+/* to access objects and methods on java side, need their field/method ids - 
+ this initialises them at the start to avoid recalculating them each time they
+ are needed 
+Note: the descriptors need to be exactly right, otherwise you get an error 
+saying "no such field" but no reference to the fact that it has the right 
+name but the wrong type.
+Note: apparently the jclass is a local ref and should only work 
+in the method that created it. It seems to work ok, but I'll make it
+ global cos the book said I should, and it may avoid future hassles.
+*/
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_initIDs (JNIEnv *j_env, jclass j_cls) {
+  
+  FID_mgpp_data = j_env->GetFieldID(j_cls, "mgpp_data_ptr_", "J"); //a long-"J"
+  if (FID_mgpp_data==NULL) {
+      cerr <<"field mgpp_data_ptr_ not found"<<endl;
+  } 
+
+  FID_query_result = j_env->GetFieldID(j_cls, "mgpp_query_result_", "Lorg/greenstone/mgpp/MGPPQueryResult;"); // an object -"L<class name>;"
+  if (FID_query_result==NULL) {
+      cerr <<"field mgpp_query_result_ not found"<<endl;
+  }
+  // the methods we want to use
+
+  // addDoc(long doc, float rank)
+  jclass JC_MGPPQueryResult = j_env->FindClass("org/greenstone/mgpp/MGPPQueryResult");
+  MID_addDoc = j_env->GetMethodID(JC_MGPPQueryResult, "addDoc", "(JF)V");
+  if (MID_addDoc==NULL) {
+      cerr <<"method not found"<<endl;
+  }
+  // addTerm(String term, String tag, int stem_method, long match_docs, 
+  // long term_freq, String[] equiv_terms)
+  MID_addTerm = j_env->GetMethodID(JC_MGPPQueryResult, "addTerm", "(Ljava/lang/String;Ljava/lang/String;IJJ[Ljava/lang/String;)V");
+  if (MID_addTerm==NULL) {
+      cerr <<"method addTerm not found"<<endl;
+  }
+  
+  // setTotalDocs(long)
+  MID_setTotalDocs = j_env->GetMethodID(JC_MGPPQueryResult, "setTotalDocs", "(J)V");
+  if (MID_setTotalDocs==NULL) {
+      cerr <<"method setTotalDocs not found"<<endl;
+  }
+  
+  MID_clearResult = j_env->GetMethodID(JC_MGPPQueryResult, "clear", "()V");
+  if (MID_clearResult==NULL) {
+      cerr <<"method clear not found"<<endl;
+  }
+
+  // get the class for String to use in NewObjectArray in runQuery()
+  // FindClass returns a local reference - have to convert it to a global one
+  jclass local_CID_String = j_env->FindClass("java/lang/String");
+  if (local_CID_String==NULL) {
+    cerr <<"java String class not found"<<endl;
+  } else { 
+    /* create a global ref */
+    CID_String = (jclass)j_env->NewGlobalRef(local_CID_String);
+    /* The local reference is no longer useful */
+    j_env->DeleteLocalRef(local_CID_String);
+ 
+    /* Is the global reference created successfully? */
+    if (CID_String == NULL) {
+      return; /* out of memory exception thrown */
+    }
+  }
+  
+}
+
+/* the java side MGPPWrapper has a pointer to a C++ object - MGPPWrapperData
+   initialise this and set the pointer 
+*/
+JNIEXPORT jboolean JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_initCppSide (JNIEnv *j_env, jobject j_obj){
+
+  MGPPWrapperData * data = new MGPPWrapperData();
+  j_env->SetIntField(j_obj, FID_mgpp_data, (long)data);
+
+  return true;
+
+}
+
+//****************************************************
+// retrieve a document
+//****************************************************
+
+/* returns a document from mgpp as a string
+Note: TextData isn't cached - just reloaded each time
+*/
+JNIEXPORT jstring JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_getDocument (JNIEnv *j_env, 
+ jobject j_obj, jstring j_base_dir, 
+ jstring j_text_name, jstring j_level, jlong j_docnum) {
+  
+  
+  const char * base_dir = j_env->GetStringUTFChars(j_base_dir, NULL);
+  if (base_dir==NULL) {
+    return NULL; // exception already thrown
+  }
+  const char * text_name = j_env->GetStringUTFChars(j_text_name, NULL);
+  
+  if (text_name==NULL) {
+   j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+   return NULL;
+  }
+
+  const char * level = j_env->GetStringUTFChars( j_level, NULL);
+  if (level==NULL) {
+   j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+   j_env->ReleaseStringUTFChars(j_text_name, text_name);
+   return NULL;
+  }
+    
+  // does this work alright? j_docnum is a long (64 bit)
+  unsigned long docnum = j_docnum;
+  TextData td;
+
+  // cast to char* otherwise complains about const
+  td.LoadData((char *)base_dir, (char *)text_name);
+
+  UCArray mg_level;
+  SetCStr(mg_level, level);
+  UCArray docText;
+  docText.clear();
+  // get the actual text
+  if (!GetDocText(td, mg_level, docnum, docText)) {
+    cerr <<"couldn't retrieve doc text"<<endl;
+  }
+
+  td.UnloadData();
+  
+  char * doc = GetCStr(docText); // do I need to free this char *??
+  jstring result = j_env->NewStringUTF(doc);
+  // release any gets
+  j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+  j_env->ReleaseStringUTFChars(j_text_name, text_name);
+  j_env->ReleaseStringUTFChars(j_level, level);
+
+  // free any C++ stuff
+  delete doc;
+
+  return result;
+ 
+}
+
+//******************************************
+// do a query
+// ****************************************
+
+/* load the IndexData - cached for querying
+ */
+JNIEXPORT jboolean JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_loadIndexData (JNIEnv *j_env, jobject j_obj, jstring j_base_dir, jstring j_index_name) {
+
+  jint data_ptr = j_env->GetIntField(j_obj, FID_mgpp_data);
+  MGPPWrapperData * data = (MGPPWrapperData *)data_ptr;
+
+  const char * base_dir = j_env->GetStringUTFChars( j_base_dir, NULL);
+  if (base_dir==NULL) {
+    return false; // exception already thrown
+  }
+
+  const char * index_name = j_env->GetStringUTFChars( j_index_name, NULL);
+  if (index_name==NULL) {
+    j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+    return false;
+  }
+  
+  jboolean j_result=false;
+  
+  // why doesn't this complain about const??
+  if (data->indexData->LoadData(base_dir, index_name)) {
+    j_result=true;
+  }
+  
+  // release any gets
+  j_env->ReleaseStringUTFChars(j_base_dir, base_dir);
+  j_env->ReleaseStringUTFChars(j_index_name, index_name);
+  
+  return j_result;
+}
+
+/* unload the data 
+ */
+JNIEXPORT jboolean JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_unloadIndexData (JNIEnv *j_env, jobject j_obj) {
+
+  jint data_ptr = j_env->GetIntField(j_obj, FID_mgpp_data);
+  MGPPWrapperData * data = (MGPPWrapperData *)data_ptr;
+
+  data->indexData->UnloadData();
+  return true;
+
+}
+
+/* do the actual query - the results are written to query_result held on the 
+   java side */
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_runQuery (JNIEnv *j_env, jobject j_obj, jstring j_query){
+  
+  jthrowable exc; // an exception - check if something funny has happened
+  const char *query = j_env->GetStringUTFChars(j_query, NULL);
+  if (query==NULL) {
+    return; // exception already thrown
+  }
+  // turn to UCArray for mgpp and then release the string
+  UCArray queryArray;
+  SetCStr(queryArray, query);
+  j_env->ReleaseStringUTFChars(j_query, query);
+  
+  // the query data
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  
+  // the result to write to
+  jobject result_ptr = j_env->GetObjectField(j_obj, FID_query_result);
+  if (result_ptr==NULL) {
+    cerr <<"couldn't access the result to write to"<<endl;
+    return;
+  }
+  
+  // clear the result
+  j_env->CallVoidMethod(result_ptr, MID_clearResult);
+  exc = j_env->ExceptionOccurred(); // this catches teh exception I think - it
+  //wont be thrown any further
+  if (exc) {
+    j_env->ExceptionDescribe();
+    return;
+  }
+  // the mgpp QueryResult that we will use
+  ExtQueryResult queryResult;
+
+  QueryNode * queryTree = NULL;
+  // parse the query string into a tree structure
+  queryTree = ParseQuery(queryArray, data->defaultBoolCombine, 
+			 data->defaultStemMethod);
+  // print the query
+  PrintNode (cout, queryTree);
+  // finally, do the query
+  MGQuery(*(data->indexData), *(data->queryInfo), queryTree, queryResult, data->level);
+
+  delete queryTree;
+
+  // convert queryResult to the java side version
+  // use levels rather than docs of ExtQueryResult
+  // CallVoidMethod(obj, method id, args to method)
+  for (int i=0; i<queryResult.levels.size(); i++) {
+    jlong doc = queryResult.levels[i];
+    jfloat rank = queryResult.ranks[i];
+    j_env->CallVoidMethod(result_ptr, MID_addDoc, doc, rank);
+    exc = j_env->ExceptionOccurred();
+    if (exc) {
+      j_env->ExceptionDescribe();
+      return;
+    }
+    
+  }
+
+  // actual num of docs
+  jlong total = queryResult.actualNumDocs;
+  j_env->CallVoidMethod(result_ptr, MID_setTotalDocs, total);
+  exc = j_env->ExceptionOccurred();
+  if (exc) {
+    j_env->ExceptionDescribe();
+    return;
+  }
+
+  // the terms
+  for (int j=0; j<queryResult.termFreqs.size(); j++) {
+    
+    TermFreqData tf = queryResult.termFreqs[j];
+    jstring term = j_env->NewStringUTF(GetCStr(tf.term));
+    jstring tag = j_env->NewStringUTF(GetCStr(tf.tag));
+    jint stem = tf.stemMethod;
+    jlong match = tf.matchDocs;
+    jlong freq = tf.termFreq;
+    
+    jobjectArray equivs=NULL;
+    jstring empty = j_env->NewStringUTF(""); // the initial object to fill the array
+    jint num_equivs = tf.equivTerms.size();
+    equivs = j_env->NewObjectArray(num_equivs, CID_String, empty);
+    if (equivs==NULL) {
+      cerr<<"couldn't create object array"<<endl;
+      
+    } else {
+      for (int k=0; k<num_equivs;k++) {
+	jstring equiv = j_env->NewStringUTF(GetCStr(tf.equivTerms[k]));
+	j_env->SetObjectArrayElement(equivs, k, equiv);
+      }
+      
+      
+      j_env->CallVoidMethod(result_ptr, MID_addTerm, term, tag, stem, match, freq, equivs);
+      exc = j_env->ExceptionOccurred();
+      if (exc) {
+	j_env->ExceptionDescribe();
+	return;
+      }
+    }
+  }
+
+
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setStem (JNIEnv *j_env, 
+					      jobject j_obj, 
+					      jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  if (j_on) {
+    data->defaultStemMethod |= 2;
+  } else {
+    data->defaultStemMethod &= 0xd;
+  }
+
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setCase (JNIEnv *j_env, 
+					      jobject j_obj, 
+					      jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+
+  if (j_on) {
+    data->defaultStemMethod |= 1;
+  } else {
+    data->defaultStemMethod &= 0xe;
+  }
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setMaxDocs (JNIEnv *j_env, 
+						 jobject j_obj, 
+						 jint j_max) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  data->queryInfo->maxDocs=j_max;
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setSortByRank (JNIEnv *j_env, 
+						    jobject j_obj, 
+						    jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  
+  data->queryInfo->sortByRank=j_on;
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setReturnTerms(JNIEnv *j_env, 
+						    jobject j_obj, 
+						    jboolean j_on) {
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  data->queryInfo->needTermFreqs = j_on;
+ 
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setQueryLevel(JNIEnv *j_env, 
+						   jobject j_obj, 
+						   jstring j_level){
+  
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+
+  const char * level = j_env->GetStringUTFChars(j_level, NULL);
+  if (level==NULL) {
+    return; // exception already thrown
+  }
+  
+  data->queryInfo->docLevel.clear();
+  SetCStr(data->queryInfo->docLevel, level);
+  
+  // release the java stuff
+  j_env->ReleaseStringUTFChars(j_level, level);
+ 
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setReturnLevel(JNIEnv *j_env, 
+						    jobject j_obj, 
+						    jstring j_level){
+
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+
+  const char * level = j_env->GetStringUTFChars(j_level, NULL);
+  if (level==NULL) {
+    return; // exception already thrown
+  }
+  
+  data->level.clear();
+  SetCStr(data->level, level);
+  
+  // release the java stuff
+  j_env->ReleaseStringUTFChars(j_level, level);
+    
+
+}
+
+JNIEXPORT void JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_setMatchMode (JNIEnv *j_env,
+						   jobject j_obj,
+						   jint j_mode){
+
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  data->defaultBoolCombine=j_mode;
+
+}
+
+JNIEXPORT jstring JNICALL 
+Java_org_greenstone_mgpp_MGPPWrapper_getQueryParams (JNIEnv *j_env,
+						     jobject j_obj){
+
+  MGPPWrapperData * data = (MGPPWrapperData *)j_env->GetIntField(j_obj, FID_mgpp_data);
+  
+  // print the data to an output string stream, then convert to char*, then to 
+  //jstring
+
+  ostrstream output;
+  output << "Query params:"<<endl
+    // need to change this to use platform specific separator for niceness
+	 << "index\t\t"<<data->indexData->basePath<<"/"<<data->indexData->filename<<endl
+	 <<"search level\t"<<GetCStr(data->queryInfo->docLevel)<<endl
+	 <<"result level\t"<<GetCStr(data->level)<<endl
+	 <<"casefold\t"<<(data->defaultStemMethod&1)<<endl
+	 <<"stem\t\t"<<(data->defaultStemMethod&2)<<endl
+	 <<"order by rank\t"<<data->queryInfo->sortByRank<<endl
+	 <<"query type\t"<<(data->defaultBoolCombine==1?"all":"some")<<endl
+	 <<"max docs\t"<<data->queryInfo->maxDocs<<endl<<ends;
+
+  char *result = output.str();
+  jstring j_result = j_env->NewStringUTF(result);
+  delete result;
+  return j_result;
+}
Index: /trunk/mgpp/jni/MGPPWrapperImpl.h
===================================================================
--- /trunk/mgpp/jni/MGPPWrapperImpl.h	(revision 3365)
+++ /trunk/mgpp/jni/MGPPWrapperImpl.h	(revision 3365)
@@ -0,0 +1,15 @@
+#include "MGQuery.h"
+
+// the global stuff we need to do a query
+class MGPPWrapperData {
+ public:
+  IndexData * indexData;
+  QueryInfo * queryInfo;
+  int defaultStemMethod;
+  int defaultBoolCombine;
+  UCArray level; // the return level
+
+  MGPPWrapperData();
+  ~MGPPWrapperData();
+
+};
Index: /trunk/mgpp/jni/Makefile.in
===================================================================
--- /trunk/mgpp/jni/Makefile.in	(revision 3365)
+++ /trunk/mgpp/jni/Makefile.in	(revision 3365)
@@ -0,0 +1,59 @@
+
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+bindir = @bindir@
+srcdir = @srcdir@
+libdir = @libdir@
+subdir = jni
+top_srcdir = @top_srcdir@
+transform = @program_transform_name@
+
+AR = ar
+CC = @CC@
+CXX = @CXX@
+CFLAGS = @CFLAGS@ -ansi -DSILENT -DSHORT_SUFFIX
+CXXFLAGS = @CXXFLAGS@ -DSILENT -DSHORT_SUFFIX
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS = ../lib/libmg.a ../text/libtextin.a -lstdc++ @LIBS@
+TOUCH = echo timestamp >
+AWK = @AWK@
+
+datadir = $(prefix)/share
+libexecdir = $(exec_prefix)/libexec
+localedir = $(datadir)/locale
+mandir = $(prefix)/man/man1
+
+CXXCOMPILE = $(CXX) -c $(DEFS) $(INCLUDES) $(CXXFLAGS)
+
+.SUFFIXES:
+
+.SUFFIXES: .cpp .o
+.cpp.o:
+	$(CXXCOMPILE) $<
+
+
+INCLUDES = -I.. -I../lib -I../text -I/usr/local/share/j2sdk1.4.0/include -I/usr/local/share/j2sdk1.4.0/include/linux
+
+all: compile link
+
+compile:
+	$(CXXCOMPILE) MGPPWrapperImpl.cpp
+link:
+	$(CXX) -shared -o libmgppjni.so MGPPWrapperImpl.o $(LIBS)
+
+clean:
+	rm -rf *.o libmgppjni.so 
+
+distclean: clean
+	rm -f Makefile
+
+install:
+	cp libmgppjni.so  $(libdir)/
+
Index: /trunk/mgpp/lib/Makefile.in
===================================================================
--- /trunk/mgpp/lib/Makefile.in	(revision 3365)
+++ /trunk/mgpp/lib/Makefile.in	(revision 3365)
@@ -0,0 +1,154 @@
+###########################################################################
+#
+# Makefile.in -- Makefile for the general purpose library
+# Copyright (C) 1994  Neil Sharman; (C) 1995 Tim Shimmin
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+#       @(#)Makefile.in	1.3 16 Mar 1994
+#
+###########################################################################
+
+
+PACKAGE = @PACKAGE@
+VERSION = @VERSION@
+ 
+SHELL = /bin/sh
+srcdir = @srcdir@
+subdir = lib
+VPATH = @srcdir@
+ 
+AR = ar
+CC = @CC@
+CXX = @CXX@
+ifneq ($(NZDL), 1)
+CFLAGS = @CFLAGS@ -ansi
+else 
+CFLAGS = @CFLAGS@ -ansi -DSILENT
+endif
+CXXFLAGS = @CXXFLAGS@
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+
+prefix = @prefix@
+datadir = $(prefix)/share
+localedir = $(datadir)/locale
+ 
+COMPILE = $(CC) -c $(DEFS) $(INCLUDES)  $(CFLAGS)
+CXXCOMPILE = $(CXX) -c $(DEFS) $(INCLUDES) $(CXXFLAGS)
+ 
+.SUFFIXES:
+.SUFFIXES: .c .o
+.c.o:
+	$(COMPILE) $<
+
+.SUFFIXES: .cpp .o
+.cpp.o:
+	$(COMPILE) -D__cplusplus $<
+
+ANSI2KNR = @ANSI2KNR@
+o = .@U@o
+ 
+.SUFFIXES: ._c ._o
+.c._c:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $@
+._c._o:
+	@echo $(COMPILE) $<
+	@rm -f _$*.c
+	@ln $< _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+.c._o:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $*._c
+	@echo $(COMPILE) $*._c
+	@rm -f _$*.c
+	@ln $*._c _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+
+INCLUDES = -I.. -I$(srcdir)
+ 
+HEADERS = \
+    bitio_m_random.h   local_strings.h    lovinstem.h \
+    bitio_m_stdio.h    memlib.h           netorder.h \
+    bitio_gen.h        mgheap.h           messages.h \
+    bitio_m.h          huffman.h  \
+    bitio_m_mem.h      perf_hash.h \
+    bitio_m_mems.h      \
+    pathmax.h          getpagesize.h      sysfuncs.h         bitio_m_abstract.h  \
+    random.h           simplefrenchstem.h unitool.h	     longlong.h \
+    non_ansi.h
+
+
+SOURCES = \
+    mgheap.cpp         \
+    huffman.cpp       messages.cpp \
+    bitio_gen.cpp     perf_hash.cpp \
+    local_strings.cpp lovinstem.cpp \
+    bitio_m_mem.cpp   bitio_m_stdio.cpp bitio_m_mems.cpp \
+    alloca.c          xmalloc.c         strstr.cpp \
+    ftruncate.cpp     strcasecmp.cpp    random.cpp \
+    simplefrenchstem.cpp                unitool.cpp       bitio_m_random.cpp \
+    bitio_m_abstract.cpp
+# memlib.cpp removed
+
+OBJECTS =  mgheap$o   \
+    huffman$o         messages$o \
+    bitio_gen$o       perf_hash$o \
+     local_strings$o   lovinstem$o \
+    bitio_m_mem$o     bitio_m_stdio$o   bitio_m_mems$o \
+    xmalloc$o         bitio_m_random$o  bitio_m_abstract$o \
+     random$o          simplefrenchstem$o \
+    unitool$o
+
+DISTFILES = Makefile.in $(HEADERS) $(SOURCES)
+
+all : libmg.a
+
+libmg.a : $(OBJECTS)
+	rm -f libmg.a
+	$(AR) cru libmg.a $(OBJECTS)
+	$(RANLIB) libmg.a
+
+install: 
+installdirs:
+uninstall:
+execinstall:
+execuninstall:
+maninstall:
+manuninstall:
+ 
+dist: $(DISTFILES)
+	for file in $(DISTFILES); do \
+	  ln $$file $(distdir) 2> /dev/null \
+	  || cp -p $$file $(distdir); \
+	done
+
+mostlyclean:
+	rm -f *.o *_.o
+ 
+clean: mostlyclean
+	rm -f libmg.a
+ 
+distclean: clean
+	rm -f Makefile
+ 
+maintainer-clean: distclean
+	@echo "This command is intended only for maintainers to use;"
+	@echo "rebuilding the deleted files may require special tools."
+
+# breaks make distclean (jrm21)
+#Makefile: Makefile.in ../../../config.status
+#	cd ../../.. && CONFIG_FILES=$(subdir)/$@ CONFIG_HEADERS= ./config.status
+ 
+# Tell versions [3.59,3.63) of GNU make not to export all variables.
+# Otherwise a system limit (for SysV at least) may be exceeded.
+.NOEXPORT:
Index: /trunk/mgpp/lib/WIN32.MAK
===================================================================
--- /trunk/mgpp/lib/WIN32.MAK	(revision 3365)
+++ /trunk/mgpp/lib/WIN32.MAK	(revision 3365)
@@ -0,0 +1,98 @@
+###########################################################################
+#
+# win32 makefile -- mgpp\text
+# A component of the Greenstone digital library software
+# from the New Zealand Digital Library Project at the
+# University of Waikato, New Zealand.
+#
+# Copyright (C) 2000  The New Zealand Digital Library Project
+#
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+###########################################################################
+
+GSDLHOME = ..\..\..
+USE_STLPORT = 0
+
+STLPATH = "$(GSDLHOME)\packages\windows\stlport\stlport"
+
+CC = cl
+CPPFLAGS =
+DEFS = -DSHORT_SUFFIX -DSILENT -DHAVE_CONFIG_H -D__WIN32__
+INCLUDES = -I"$(GSDLHOME)"
+!IF $(USE_STLPORT)
+DEFS = $(DEFS) -DGSDL_NAMESPACE_BROKEN -DGSDL_USE_IOS_H -D__STL_NO_NEW_IOSTREAMS
+INCLUDES = $(INCLUDES) -I$(STLPATH)
+!ENDIF
+
+LDFLAGS =
+
+LIBS = 
+
+COMPILE = $(CC) -c $(DEFS) $(INCLUDES)
+LINK = $(CC) $(LDFLAGS)
+
+.SUFFIXES:
+.SUFFIXES: .c .obj .cpp
+.c.obj:
+	$(COMPILE) $<
+.cpp.obj:
+	$(COMPILE) $<
+
+o = .obj
+
+HEADERS = \
+    bitio_m_random.h   local_strings.h    lovinstem.h \
+    bitio_m_stdio.h    memlib.h           netorder.h \
+    bitio_gen.h        mgheap.h           messages.h \
+    bitio_m.h          huffman.h  \
+    bitio_m_mem.h      perf_hash.h \
+    bitio_m_mems.h     getopt_old.h \
+    pathmax.h          getpagesize.h      sysfuncs.h         bitio_m_abstract.h  \
+    random.h           simplefrenchstem.h unitool.h	     longlong.h \
+    non_ansi.h
+
+SOURCES = \
+    mgheap.cpp         \
+    huffman.cpp       messages.cpp \
+    bitio_gen.cpp     perf_hash.cpp \
+    local_strings.cpp lovinstem.cpp \
+    bitio_m_mem.cpp   bitio_m_stdio.cpp bitio_m_mems.cpp \
+    alloca.c          xmalloc.c         strstr.cpp \
+    ftruncate.cpp     strcasecmp.cpp    random.cpp \
+    simplefrenchstem.cpp                unitool.cpp       bitio_m_random.cpp \
+    bitio_m_abstract.cpp getopt_old.c
+
+OBJECTS =  mgheap$o   \
+    huffman$o         messages$o \
+    bitio_gen$o       perf_hash$o \
+     local_strings$o   lovinstem$o \
+    bitio_m_mem$o     bitio_m_stdio$o   bitio_m_mems$o \
+    xmalloc$o         bitio_m_random$o  bitio_m_abstract$o \
+     random$o          simplefrenchstem$o \
+    unitool$o  win32in$o getopt_old$o
+
+all : libmg.lib
+
+libmg.lib: $(OBJECTS)
+	if EXIST libmg.lib del libmg.lib
+	lib /out:libmg.lib $(OBJECTS)
+
+clean:
+	del *$o 
+	del libmg.lib
+
+install:
Index: /trunk/mgpp/lib/alloca.c
===================================================================
--- /trunk/mgpp/lib/alloca.c	(revision 3365)
+++ /trunk/mgpp/lib/alloca.c	(revision 3365)
@@ -0,0 +1,496 @@
+/* alloca.c -- allocate automatically reclaimed memory
+   (Mostly) portable public-domain implementation -- D A Gwyn
+
+   This implementation of the PWB library alloca function,
+   which is used to allocate space off the run-time stack so
+   that it is automatically reclaimed upon procedure exit,
+   was inspired by discussions with J. Q. Johnson of Cornell.
+   J.Otto Tennant <jot@cray.com> contributed the Cray support.
+
+   There are some preprocessor constants that can
+   be defined when compiling for your specific system, for
+   improved efficiency; however, the defaults should be okay.
+
+   The general concept of this implementation is to keep
+   track of all alloca-allocated blocks, and reclaim any
+   that are found to be deeper in the stack than the current
+   invocation.  This heuristic does not reclaim storage as
+   soon as it becomes invalid, but it will do so eventually.
+
+   As a special case, alloca(0) reclaims storage without
+   allocating any.  It is a good idea to use alloca(0) in
+   your main control loop, etc. to force garbage collection.  */
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__  /* [RPAP - Feb 97: WIN32 Port] */
+#  include <win32cfg.h>
+# else
+#  include <sysfuncs.h>
+# endif
+#endif
+
+#ifdef emacs
+#include "blockinput.h"
+#endif
+
+/* If compiling with GCC 2, this file's not needed.  */
+#if !defined (__GNUC__) || __GNUC__ < 2
+
+/* If someone has defined alloca as a macro,
+   there must be some other way alloca is supposed to work.  */
+#ifndef alloca
+
+#ifdef emacs
+#ifdef static
+/* actually, only want this if static is defined as ""
+   -- this is for usg, in which emacs must undefine static
+   in order to make unexec workable
+   */
+#ifndef STACK_DIRECTION
+you
+lose
+-- must know STACK_DIRECTION at compile-time
+#endif /* STACK_DIRECTION undefined */
+#endif /* static */
+#endif /* emacs */
+
+/* If your stack is a linked list of frames, you have to
+   provide an "address metric" ADDRESS_FUNCTION macro.  */
+
+#if defined (CRAY) && defined (CRAY_STACKSEG_END)
+long i00afunc ();
+#define ADDRESS_FUNCTION(arg) (char *) i00afunc (&(arg))
+#else
+#define ADDRESS_FUNCTION(arg) &(arg)
+#endif
+
+#if __STDC__
+typedef void *pointer;
+#else
+typedef char *pointer;
+#endif
+
+#define	NULL	0
+
+/* Different portions of Emacs need to call different versions of
+   malloc.  The Emacs executable needs alloca to call xmalloc, because
+   ordinary malloc isn't protected from input signals.  On the other
+   hand, the utilities in lib-src need alloca to call malloc; some of
+   them are very simple, and don't have an xmalloc routine.
+
+   Non-Emacs programs expect this to call use xmalloc.
+
+   Callers below should use malloc.  */
+
+#ifndef emacs
+#define malloc xmalloc
+#endif
+extern pointer malloc ();
+
+/* Define STACK_DIRECTION if you know the direction of stack
+   growth for your system; otherwise it will be automatically
+   deduced at run-time.
+
+   STACK_DIRECTION > 0 => grows toward higher addresses
+   STACK_DIRECTION < 0 => grows toward lower addresses
+   STACK_DIRECTION = 0 => direction of growth unknown  */
+
+#ifndef STACK_DIRECTION
+#define	STACK_DIRECTION	0	/* Direction unknown.  */
+#endif
+
+#if STACK_DIRECTION != 0
+
+#define	STACK_DIR	STACK_DIRECTION	/* Known at compile-time.  */
+
+#else /* STACK_DIRECTION == 0; need run-time code.  */
+
+static int stack_dir;		/* 1 or -1 once known.  */
+#define	STACK_DIR	stack_dir
+
+static void
+find_stack_direction ()
+{
+  static char *addr = NULL;	/* Address of first `dummy', once known.  */
+  auto char dummy;		/* To get stack address.  */
+
+  if (addr == NULL)
+    {				/* Initial entry.  */
+      addr = ADDRESS_FUNCTION (dummy);
+
+      find_stack_direction ();	/* Recurse once.  */
+    }
+  else
+    {
+      /* Second entry.  */
+      if (ADDRESS_FUNCTION (dummy) > addr)
+	stack_dir = 1;		/* Stack grew upward.  */
+      else
+	stack_dir = -1;		/* Stack grew downward.  */
+    }
+}
+
+#endif /* STACK_DIRECTION == 0 */
+
+/* An "alloca header" is used to:
+   (a) chain together all alloca'ed blocks;
+   (b) keep track of stack depth.
+
+   It is very important that sizeof(header) agree with malloc
+   alignment chunk size.  The following default should work okay.  */
+
+#ifndef	ALIGN_SIZE
+#define	ALIGN_SIZE	sizeof(double)
+#endif
+
+typedef union hdr
+{
+  char align[ALIGN_SIZE];	/* To force sizeof(header).  */
+  struct
+    {
+      union hdr *next;		/* For chaining headers.  */
+      char *deep;		/* For stack depth measure.  */
+    } h;
+} header;
+
+static header *last_alloca_header = NULL;	/* -> last alloca header.  */
+
+/* Return a pointer to at least SIZE bytes of storage,
+   which will be automatically reclaimed upon exit from
+   the procedure that called alloca.  Originally, this space
+   was supposed to be taken from the current stack frame of the
+   caller, but that method cannot be made to work for some
+   implementations of C, for example under Gould's UTX/32.  */
+
+pointer
+alloca (size)
+     unsigned size;
+{
+  auto char probe;		/* Probes stack depth: */
+  register char *depth = ADDRESS_FUNCTION (probe);
+
+#if STACK_DIRECTION == 0
+  if (STACK_DIR == 0)		/* Unknown growth direction.  */
+    find_stack_direction ();
+#endif
+
+  /* Reclaim garbage, defined as all alloca'd storage that
+     was allocated from deeper in the stack than currently. */
+
+  {
+    register header *hp;	/* Traverses linked list.  */
+
+#ifdef emacs
+    BLOCK_INPUT;
+#endif
+
+    for (hp = last_alloca_header; hp != NULL;)
+      if ((STACK_DIR > 0 && hp->h.deep > depth)
+	  || (STACK_DIR < 0 && hp->h.deep < depth))
+	{
+	  register header *np = hp->h.next;
+
+	  free ((pointer) hp);	/* Collect garbage.  */
+
+	  hp = np;		/* -> next header.  */
+	}
+      else
+	break;			/* Rest are not deeper.  */
+
+    last_alloca_header = hp;	/* -> last valid storage.  */
+
+#ifdef emacs
+    UNBLOCK_INPUT;
+#endif
+  }
+
+  if (size == 0)
+    return NULL;		/* No allocation required.  */
+
+  /* Allocate combined header + user data storage.  */
+
+  {
+    register pointer new = malloc (sizeof (header) + size);
+    /* Address of header.  */
+
+    ((header *) new)->h.next = last_alloca_header;
+    ((header *) new)->h.deep = depth;
+
+    last_alloca_header = (header *) new;
+
+    /* User storage begins just after header.  */
+
+    return (pointer) ((char *) new + sizeof (header));
+  }
+}
+
+#if defined (CRAY) && defined (CRAY_STACKSEG_END)
+
+#ifdef DEBUG_I00AFUNC
+#include <stdio.h>
+#endif
+
+#ifndef CRAY_STACK
+#define CRAY_STACK
+#ifndef CRAY2
+/* Stack structures for CRAY-1, CRAY X-MP, and CRAY Y-MP */
+struct stack_control_header
+  {
+    long shgrow:32;		/* Number of times stack has grown.  */
+    long shaseg:32;		/* Size of increments to stack.  */
+    long shhwm:32;		/* High water mark of stack.  */
+    long shsize:32;		/* Current size of stack (all segments).  */
+  };
+
+/* The stack segment linkage control information occurs at
+   the high-address end of a stack segment.  (The stack
+   grows from low addresses to high addresses.)  The initial
+   part of the stack segment linkage control information is
+   0200 (octal) words.  This provides for register storage
+   for the routine which overflows the stack.  */
+
+struct stack_segment_linkage
+  {
+    long ss[0200];		/* 0200 overflow words.  */
+    long sssize:32;		/* Number of words in this segment.  */
+    long ssbase:32;		/* Offset to stack base.  */
+    long:32;
+    long sspseg:32;		/* Offset to linkage control of previous
+				   segment of stack.  */
+    long:32;
+    long sstcpt:32;		/* Pointer to task common address block.  */
+    long sscsnm;		/* Private control structure number for
+				   microtasking.  */
+    long ssusr1;		/* Reserved for user.  */
+    long ssusr2;		/* Reserved for user.  */
+    long sstpid;		/* Process ID for pid based multi-tasking.  */
+    long ssgvup;		/* Pointer to multitasking thread giveup.  */
+    long sscray[7];		/* Reserved for Cray Research.  */
+    long ssa0;
+    long ssa1;
+    long ssa2;
+    long ssa3;
+    long ssa4;
+    long ssa5;
+    long ssa6;
+    long ssa7;
+    long sss0;
+    long sss1;
+    long sss2;
+    long sss3;
+    long sss4;
+    long sss5;
+    long sss6;
+    long sss7;
+  };
+
+#else /* CRAY2 */
+/* The following structure defines the vector of words
+   returned by the STKSTAT library routine.  */
+struct stk_stat
+  {
+    long now;			/* Current total stack size.  */
+    long maxc;			/* Amount of contiguous space which would
+				   be required to satisfy the maximum
+				   stack demand to date.  */
+    long high_water;		/* Stack high-water mark.  */
+    long overflows;		/* Number of stack overflow ($STKOFEN) calls.  */
+    long hits;			/* Number of internal buffer hits.  */
+    long extends;		/* Number of block extensions.  */
+    long stko_mallocs;		/* Block allocations by $STKOFEN.  */
+    long underflows;		/* Number of stack underflow calls ($STKRETN).  */
+    long stko_free;		/* Number of deallocations by $STKRETN.  */
+    long stkm_free;		/* Number of deallocations by $STKMRET.  */
+    long segments;		/* Current number of stack segments.  */
+    long maxs;			/* Maximum number of stack segments so far.  */
+    long pad_size;		/* Stack pad size.  */
+    long current_address;	/* Current stack segment address.  */
+    long current_size;		/* Current stack segment size.  This
+				   number is actually corrupted by STKSTAT to
+				   include the fifteen word trailer area.  */
+    long initial_address;	/* Address of initial segment.  */
+    long initial_size;		/* Size of initial segment.  */
+  };
+
+/* The following structure describes the data structure which trails
+   any stack segment.  I think that the description in 'asdef' is
+   out of date.  I only describe the parts that I am sure about.  */
+
+struct stk_trailer
+  {
+    long this_address;		/* Address of this block.  */
+    long this_size;		/* Size of this block (does not include
+				   this trailer).  */
+    long unknown2;
+    long unknown3;
+    long link;			/* Address of trailer block of previous
+				   segment.  */
+    long unknown5;
+    long unknown6;
+    long unknown7;
+    long unknown8;
+    long unknown9;
+    long unknown10;
+    long unknown11;
+    long unknown12;
+    long unknown13;
+    long unknown14;
+  };
+
+#endif /* CRAY2 */
+#endif /* not CRAY_STACK */
+
+#ifdef CRAY2
+/* Determine a "stack measure" for an arbitrary ADDRESS.
+   I doubt that "lint" will like this much. */
+
+static long
+i00afunc (long *address)
+{
+  struct stk_stat status;
+  struct stk_trailer *trailer;
+  long *block, size;
+  long result = 0;
+
+  /* We want to iterate through all of the segments.  The first
+     step is to get the stack status structure.  We could do this
+     more quickly and more directly, perhaps, by referencing the
+     $LM00 common block, but I know that this works.  */
+
+  STKSTAT (&status);
+
+  /* Set up the iteration.  */
+
+  trailer = (struct stk_trailer *) (status.current_address
+				    + status.current_size
+				    - 15);
+
+  /* There must be at least one stack segment.  Therefore it is
+     a fatal error if "trailer" is null.  */
+
+  if (trailer == 0)
+    abort ();
+
+  /* Discard segments that do not contain our argument address.  */
+
+  while (trailer != 0)
+    {
+      block = (long *) trailer->this_address;
+      size = trailer->this_size;
+      if (block == 0 || size == 0)
+	abort ();
+      trailer = (struct stk_trailer *) trailer->link;
+      if ((block <= address) && (address < (block + size)))
+	break;
+    }
+
+  /* Set the result to the offset in this segment and add the sizes
+     of all predecessor segments.  */
+
+  result = address - block;
+
+  if (trailer == 0)
+    {
+      return result;
+    }
+
+  do
+    {
+      if (trailer->this_size <= 0)
+	abort ();
+      result += trailer->this_size;
+      trailer = (struct stk_trailer *) trailer->link;
+    }
+  while (trailer != 0);
+
+  /* We are done.  Note that if you present a bogus address (one
+     not in any segment), you will get a different number back, formed
+     from subtracting the address of the first block.  This is probably
+     not what you want.  */
+
+  return (result);
+}
+
+#else /* not CRAY2 */
+/* Stack address function for a CRAY-1, CRAY X-MP, or CRAY Y-MP.
+   Determine the number of the cell within the stack,
+   given the address of the cell.  The purpose of this
+   routine is to linearize, in some sense, stack addresses
+   for alloca.  */
+
+static long
+i00afunc (long address)
+{
+  long stkl = 0;
+
+  long size, pseg, this_segment, stack;
+  long result = 0;
+
+  struct stack_segment_linkage *ssptr;
+
+  /* Register B67 contains the address of the end of the
+     current stack segment.  If you (as a subprogram) store
+     your registers on the stack and find that you are past
+     the contents of B67, you have overflowed the segment.
+
+     B67 also points to the stack segment linkage control
+     area, which is what we are really interested in.  */
+
+  stkl = CRAY_STACKSEG_END ();
+  ssptr = (struct stack_segment_linkage *) stkl;
+
+  /* If one subtracts 'size' from the end of the segment,
+     one has the address of the first word of the segment.
+
+     If this is not the first segment, 'pseg' will be
+     nonzero.  */
+
+  pseg = ssptr->sspseg;
+  size = ssptr->sssize;
+
+  this_segment = stkl - size;
+
+  /* It is possible that calling this routine itself caused
+     a stack overflow.  Discard stack segments which do not
+     contain the target address.  */
+
+  while (!(this_segment <= address && address <= stkl))
+    {
+#ifdef DEBUG_I00AFUNC
+      fprintf (stderr, "%011o %011o %011o\n", this_segment, address, stkl);
+#endif
+      if (pseg == 0)
+	break;
+      stkl = stkl - pseg;
+      ssptr = (struct stack_segment_linkage *) stkl;
+      size = ssptr->sssize;
+      pseg = ssptr->sspseg;
+      this_segment = stkl - size;
+    }
+
+  result = address - this_segment;
+
+  /* If you subtract pseg from the current end of the stack,
+     you get the address of the previous stack segment's end.
+     This seems a little convoluted to me, but I'll bet you save
+     a cycle somewhere.  */
+
+  while (pseg != 0)
+    {
+#ifdef DEBUG_I00AFUNC
+      fprintf (stderr, "%011o %011o\n", pseg, size);
+#endif
+      stkl = stkl - pseg;
+      ssptr = (struct stack_segment_linkage *) stkl;
+      size = ssptr->sssize;
+      pseg = ssptr->sspseg;
+      result += size;
+    }
+  return (result);
+}
+
+#endif /* not CRAY2 */
+#endif /* CRAY */
+
+#endif /* no alloca */
+#endif /* not GCC version 2 */
Index: /trunk/mgpp/lib/bitio_gen.cpp
===================================================================
--- /trunk/mgpp/lib/bitio_gen.cpp	(revision 3365)
+++ /trunk/mgpp/lib/bitio_gen.cpp	(revision 3365)
@@ -0,0 +1,134 @@
+/**************************************************************************
+ *
+ * bitio_gen.c -- General supoport routines for bitio
+ * Copyright (C) 1994  Neil Sharman and Alistair Moffat
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "bitio_m_abstract.h"
+
+int fprintf (FILE *, const char *,...);
+
+unsigned long 
+BIO_Unary_Length (unsigned long val)
+{
+  register unsigned long num;
+  num = bitio_buffer::unary_length(val);
+  //  UNARY_LENGTH (val, num);
+  return (num);
+}
+
+
+unsigned long 
+BIO_Binary_Length (unsigned long val, unsigned long b)
+{
+  register unsigned long num;
+  num = bitio_buffer::binary_length (val, b);
+  return (num);
+}
+
+
+unsigned long BIO_Gamma_Length (unsigned long val) {
+  register unsigned long num;
+  num = bitio_buffer::gamma_length(val);
+  return (num);
+}
+
+
+unsigned long 
+BIO_Delta_Length (unsigned long val)
+{
+  register unsigned long num;
+  num = bitio_buffer::delta_length(val);
+  return (num);
+}
+
+
+unsigned long 
+BIO_Elias_Length (unsigned long val, unsigned long b, double s)
+{
+  register unsigned long num;
+  num = bitio_buffer::elias_length(val, b, s);
+  return (num);
+}
+
+unsigned long 
+BIO_Bblock_Length (unsigned long val, unsigned long b)
+{
+  register unsigned long num;
+  num = bitio_buffer::bblock_length (val, b);
+  return (num);
+}
+
+
+int 
+BIO_Bblock_Init (int N, int p)
+{
+  int b;
+  b = (int) (0.5 + 0.6931471 * N / p);
+  return (b ? b : 1);
+}
+
+
+int 
+BIO_Bblock_Init_W (int N, int p)
+{
+  int logb;
+  logb = bitio_buffer::floor_log2 ((N - p) / p);
+  return (logb < 0 ? 1 : (1 << logb));
+}
+
+int 
+BIO_Bblock_Bound_b (int N, int p, int b)
+{
+  int clogb;
+  clogb = bitio_buffer::ceil_log2 (b);
+  return (p * (1 + clogb) + (N - p * ((1 << clogb) - b + 1)) / b);
+}
+
+int 
+BIO_Bblock_Bound (int N, int p)
+{
+  int b;
+  b = BIO_Bblock_Init_W (N, p);
+  return (BIO_Bblock_Bound_b (N, p, b));
+}
+
+/* adjustment is a hack to overcome inaccuracies in floating point,
+   where (int)(ln4/ln2) can equal 1, not 2. */
+int 
+BIO_Gamma_Bound (int N, int p)
+{
+  return ((int) (p * (2 * log2 (((double) N * 1.0001) / p) + 1)));
+}
+
+int 
+floorlog_2 (int b)
+{
+  int logb;
+  logb = bitio_buffer::floor_log2 (b);
+  return logb;
+}
+
+int 
+ceillog_2 (int b)
+{
+  int logb;
+  logb = bitio_buffer::ceil_log2 (b);
+  return logb;
+}
Index: /trunk/mgpp/lib/bitio_gen.h
===================================================================
--- /trunk/mgpp/lib/bitio_gen.h	(revision 3365)
+++ /trunk/mgpp/lib/bitio_gen.h	(revision 3365)
@@ -0,0 +1,61 @@
+/**************************************************************************
+ *
+ * bitio_gen.h -- General supoport routines for bitio
+ * Copyright (C) 1994  Neil Sharman and Alistair Moffat
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains function definitions for doing general bitio operations.
+ *
+ *  This file defines functions for calculation the length in bits of a number
+ *  coded unary, binary, delta, gamma, elias, and  bblock coding. It defines 
+ *  routines for calculating the parameter and bounds for bblock coding. 
+ *  Routines for calculating ceil and floor lg are also defined.
+ *
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_GEN
+#define H_BITIO_GEN
+
+unsigned long BIO_Unary_Length (unsigned long val);
+
+unsigned long BIO_Binary_Length (unsigned long val, unsigned long b);
+
+unsigned long BIO_Gamma_Length (unsigned long val);
+
+unsigned long BIO_Delta_Length (unsigned long val);
+
+unsigned long BIO_Elias_Length (unsigned long val, unsigned long b, double s);
+
+unsigned long BIO_Bblock_Length (unsigned long val, unsigned long b);
+
+int BIO_Bblock_Init (int N, int p);
+
+int BIO_Bblock_Init_W (int N, int p);
+
+int BIO_Bblock_Bound_b (int N, int p, int b);
+
+int BIO_Bblock_Bound (int N, int p);
+
+int BIO_Gamma_Bound (int N, int p);
+
+int floorlog_2 (int b);
+
+int ceillog_2 (int b);
+
+#endif
Index: /trunk/mgpp/lib/bitio_m_abstract.cpp
===================================================================
--- /trunk/mgpp/lib/bitio_m_abstract.cpp	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_abstract.cpp	(revision 3365)
@@ -0,0 +1,9 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_abstract.h"
+
+
+
+
+
Index: /trunk/mgpp/lib/bitio_m_abstract.h
===================================================================
--- /trunk/mgpp/lib/bitio_m_abstract.h	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_abstract.h	(revision 3365)
@@ -0,0 +1,340 @@
+#ifndef H_BITIO_ABSTRACT
+#define H_BITIO_ABSTRACT
+
+class bitio_buffer {
+public:
+  virtual int bit() = 0;
+  virtual void encodeBit(int bit) = 0;
+  virtual void seek(unsigned long topos) = 0;
+  
+  static inline void check_positive(char *type, unsigned long x) {
+    if (x <= 0) {
+      fprintf(stderr, "Error: Cannot %s encode %lu\n", type, x);
+      exit(1);
+    }
+  }
+
+  static inline int floor_log2(unsigned long x) {
+    unsigned long  _B_x = x;
+    int reply;
+    
+    reply = -1;
+    while (_B_x != 0) {
+      _B_x >>= 1;
+      reply ++;
+    }
+    return reply;
+  }
+  
+  static inline int ceil_log2(unsigned long x) {
+    register int _B_x;
+    unsigned long reply;
+    
+    _B_x = x - 1;
+    reply = 0;
+    while (_B_x != 0) {
+      _B_x >>= 1;
+      reply ++;
+    }
+    return reply;
+  }
+
+  inline unsigned long unary_decode(unsigned long *count) {
+    unsigned long x;
+    
+    x = 1;
+    while (!this->bit()) x++;
+    if (count != NULL) {
+      *count += x;
+    }
+    return x;
+  }
+
+  inline void unary_encode(unsigned long x, unsigned long *count) {
+    register unsigned long _B_x = x;
+    check_positive("unary", _B_x);
+    if (count != NULL) {
+      *count += _B_x;
+    }
+    while(--_B_x) 
+      encodeBit(0);
+    encodeBit(1);
+  }
+
+
+  inline unsigned long binary_decode(unsigned long b, unsigned long *count) {
+    register unsigned long _B_x = 0;
+    register unsigned long _B_b = b;
+    register int _B_i, _B_logofb, _B_thresh;
+    register unsigned long reply;
+    
+    if (_B_b != 1) {					
+      _B_logofb = ceil_log2(_B_b);
+      _B_thresh = (1<<_B_logofb) - _B_b;
+      _B_logofb--;
+      
+      if (count != NULL) {
+	*count += _B_logofb;
+      }
+      
+      for (_B_i=0; _B_i < _B_logofb; _B_i++) {
+	_B_x += _B_x + this->bit();
+      }
+
+      if ((int)_B_x >= _B_thresh) {
+	_B_x += _B_x + this->bit();
+	_B_x -= _B_thresh;
+	if (count != NULL) {
+	  *count = *count + 1;
+	}
+      }
+      reply = _B_x+1;
+      
+    } else reply = 1;
+    
+    return reply;
+  }
+
+  inline void binary_encode(unsigned long x, unsigned long b, unsigned long *count) {
+    register unsigned long _B_x = (x);
+    register unsigned long _B_b = (b);
+    register int _B_nbits, _B_logofb, _B_thresh;
+    
+    check_positive("binary", _B_x);
+    _B_logofb = ceil_log2(_B_b);
+    
+    _B_thresh = (1<<_B_logofb) - _B_b;
+    if ((int)(--_B_x) < _B_thresh) _B_nbits = _B_logofb-1;
+    else {
+      _B_nbits = _B_logofb;
+      _B_x += _B_thresh;
+    }
+    if (count != NULL) {
+      *count += _B_nbits;
+    }
+    while (--_B_nbits>=0)
+      encodeBit((_B_x>>_B_nbits) & 0x1);
+  }
+
+  inline unsigned long bblock_decode(unsigned long b, unsigned long *count) {
+    register unsigned long _B_x1, _B_xx = 0;
+    register unsigned long _B_bb = (b);
+    register int _B_xdivb;
+    
+    _B_xdivb = this->unary_decode(count);
+    _B_xdivb--;				       
+    while (_B_xdivb--) {
+      _B_xx += _B_bb;
+    }
+    _B_x1 = this->binary_decode(_B_bb, count);
+    return (_B_xx + _B_x1);
+  }
+  
+  inline void bblock_encode(unsigned long x, unsigned long b, unsigned long *count) {
+    register unsigned long _B_xx = (x);
+    register unsigned long _B_bb = (b);
+    register int _B_xdivb = 0;
+    check_positive("bblock", _B_xx);
+    _B_xx--;
+    while (_B_xx >= _B_bb) {
+      _B_xdivb++;
+      _B_xx -= _B_bb;
+    }
+    unary_encode(_B_xdivb+1, count);
+    binary_encode(_B_xx+1, _B_bb, count);
+  }
+
+  inline unsigned long elias_decode(unsigned long b, double s, unsigned long *count) {
+    register unsigned long _B_xx;
+    register unsigned long _B_b = b;
+    register double _B_s = s;
+    register int _B_lower=1, _B_upper=1;
+    register int _B_k;
+    register double _B_pow=1.0;
+    
+    _B_k = this->unary_decode(count);
+    _B_k--;
+    while (_B_k--) {
+      _B_lower = _B_upper+1;
+      _B_pow *= _B_s;
+      _B_upper += (int) (_B_b*_B_pow);
+    }
+    _B_xx = this->binary_decode(_B_upper-_B_lower+1, count);
+    _B_xx--;
+    check_positive("gamma", _B_xx+_B_lower);
+    return _B_xx+_B_lower;
+  }
+  
+  inline void elias_encode(unsigned long x, unsigned long b, double s, unsigned long *count) {
+    register unsigned long _B_xx = x;
+    register unsigned long _B_b = b;
+    register double _B_s = s;
+    register int _B_lower=1, _B_upper=1;
+    register int _B_k=0;
+    register double _B_pow=1.0;
+    
+    check_positive("elias", _B_xx);
+    
+    while ((int)_B_xx>_B_upper) {
+      _B_k++;
+      _B_lower = _B_upper+1;
+      _B_pow *= _B_s;
+      _B_upper += (int) (_B_b*_B_pow);
+    }
+    unary_encode(_B_k+1, count);
+    binary_encode(_B_xx-_B_lower+1, _B_upper-_B_lower+1, count);
+  }
+  
+  inline unsigned long gamma_decode(unsigned long *count) {
+    register unsigned long _B_xx = 1;
+    register int _B_nbits = 0;
+    
+    while(!this->bit()) _B_nbits++;
+    if (count != NULL) {
+      *count += _B_nbits*2+1;
+    }
+    while (_B_nbits-- > 0)
+      _B_xx += _B_xx + this->bit();
+    return _B_xx;
+  }
+
+  inline void gamma_encode(unsigned long x, unsigned long *count) {
+    register unsigned long _B_xx = x;
+    register int _B_logofb;
+    register int _B_nbits;
+    
+    check_positive("gamma", _B_xx);
+    _B_logofb = floor_log2(_B_xx);
+    _B_nbits = _B_logofb+1;
+    
+    if (count != NULL) { 
+      *count += _B_logofb*2+1;
+    }
+    while(_B_logofb--) encodeBit(0);
+    while (--_B_nbits>=0) encodeBit((_B_xx>>_B_nbits) & 0x1);
+  }
+  
+  inline unsigned long delta_decode(unsigned long *count) {
+    register unsigned long _B_xxx;
+    register int _B_logx;
+    _B_logx = gamma_decode(count);
+    _B_logx--;
+    _B_xxx = binary_decode(1<<_B_logx, count); 
+    _B_xxx--;
+    return (_B_xxx + (1<<_B_logx));
+  }
+
+  inline void delta_encode(unsigned long x, unsigned long *count) {
+    register unsigned long _B_xxx = x;
+    register int _B_logx;
+    _B_logx = floor_log2(_B_xxx);
+    gamma_encode(_B_logx+1, count);
+    binary_encode(_B_xxx+1, 1<<_B_logx, count);
+  }
+
+  inline unsigned long huff_decode(unsigned long *mcodes, unsigned long **values,
+				   unsigned long *count = NULL) {
+    register unsigned long *__min_code = mcodes;
+    register unsigned long *__mclen    = mcodes;
+    register unsigned long **__values  = values;
+    register unsigned long __code = 0;
+    do {
+      __code += __code + bit();
+      if (count != NULL) (*count) ++;
+    } while (__code < *++__mclen);
+    
+    return __values[__mclen - __min_code][__code - *__mclen];
+  }
+
+  inline void huff_encode(unsigned long x, unsigned long *codes, char *lens,
+			  unsigned long *count = NULL) {
+    register int __i;
+    register int __clen = lens[x];
+    register unsigned long __code = codes[x];
+    for (__i=__clen-1; __i>=0; __i--) {
+      encodeBit((__code >> __i) & 1);
+    }
+    if (count != NULL) {
+      *count += lens[x];
+    }
+  }
+
+  static inline unsigned long unary_length(unsigned long x) {
+    check_positive("unary", x);
+    return x;
+  }
+
+  static inline int binary_length(unsigned long x, unsigned long b) {
+    register unsigned long _B_x = (x);
+    register unsigned long _B_b = (b);
+    register int _B_nbits, _B_logofb, _B_thresh;
+    
+    check_positive("binary", _B_x);
+    _B_logofb = ceil_log2(_B_b);
+    _B_thresh = (1<<_B_logofb) - _B_b;
+    if ((int)(--_B_x) < _B_thresh) _B_nbits = _B_logofb-1;
+    else _B_nbits = _B_logofb;
+    return _B_nbits;
+  }
+
+  static inline unsigned long bblock_length(unsigned long x, unsigned long b) {
+    register unsigned long _B_bcount, count, _B_xx = (x);
+    register unsigned long _B_bb = (b);
+    register int _B_xdivb = 0;
+    
+    check_positive("bblock", _B_xx);
+    _B_xx--;
+    while (_B_xx >= _B_bb) {
+      _B_xdivb++;
+      _B_xx -= _B_bb;
+    }
+    count = unary_length(_B_xdivb+1);
+    _B_bcount = binary_length(_B_xx+1, _B_bb);
+    count += _B_bcount;
+    return count;
+  }
+  
+  static inline unsigned long elias_length(unsigned long x, unsigned long b, 
+					   double s) {
+    register unsigned long _B_xx = x;
+    register unsigned long _B_b = b;
+    register double _B_s = s;
+    register int _B_lower=1, _B_upper=1;
+    register int _B_k=0, _B_ecount;
+    register double _B_pow=1.0;
+    register unsigned long count;
+    check_positive("gamma", _B_xx);
+    
+    while ((int)(_B_xx)>_B_upper) {
+      _B_k++;
+      _B_lower = _B_upper+1;
+      _B_pow *= _B_s;
+      _B_upper += (int) (_B_b*_B_pow);
+    }
+    count = unary_length(_B_k+1);
+    _B_ecount = binary_length(_B_xx-_B_lower+1, _B_upper-_B_lower+1);
+    count += _B_ecount;
+    return count;
+  }
+  
+  static inline unsigned long gamma_length(unsigned long x) {
+    register unsigned long _B_xx = x;
+    register int _B_logofb;
+    check_positive("gamma", _B_xx);
+    _B_logofb = floor_log2(_B_xx);
+    return _B_logofb*2+1;
+  }
+
+  static inline unsigned long delta_length(unsigned long x) {
+    register unsigned long _B_xxx = x;
+    register int _B_logx, _B_dcount;
+    register unsigned long count;
+    _B_logx = floor_log2 (_B_xxx);
+    count = gamma_length(_B_logx+1);
+    _B_dcount = binary_length (_B_xxx+1, 1<<_B_logx);
+    count += _B_dcount;
+    return count;
+  }
+};
+
+#endif
Index: /trunk/mgpp/lib/bitio_m_mem.cpp
===================================================================
--- /trunk/mgpp/lib/bitio_m_mem.cpp	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_mem.cpp	(revision 3365)
@@ -0,0 +1,128 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_abstract.h"
+#include "bitio_m_mem.h"
+
+mem_bitio_buffer::mem_bitio_buffer() {
+}
+
+mem_bitio_buffer::mem_bitio_buffer(unsigned char *readbuffer, int size) { 
+  pos       = readbuffer;
+  base      = readbuffer;
+  remaining = size;
+  buffer    = 0;
+  btg       = 0;
+}
+
+
+void mem_bitio_buffer::error() {
+  fprintf (stderr, "Unexpected EOF in \"%s\" on line %d\n", __FILE__, __LINE__);
+  exit(1);
+}
+
+void mem_bitio_buffer::encodeStart() {
+  btg = sizeof(buffer) * 8;
+}
+
+
+void mem_bitio_buffer::seek(unsigned long toPos) { 
+  register long _newPos;
+  do {
+    _newPos = toPos;
+    
+    pos    = base + (_newPos >> 3);
+    buffer = *pos++;
+    btg    = 8 - (_newPos & 0x07);
+  } while (0);
+}
+
+long mem_bitio_buffer::addff(long b) {
+  do {
+    if (btg == 0) {
+      if (remaining != 0) {
+	buffer = *pos++;
+	remaining --;
+      } else {
+	buffer = 0xff;
+      }
+      btg = sizeof(buffer) * 8;
+    }
+    b = b + b + ((buffer >> --btg) & 1);
+  } while(0);
+
+  return b;
+}
+
+long mem_bitio_buffer::add00(long b) {
+  do {
+    if (btg == 0) {
+      if (remaining != 0) {
+	buffer = *pos++;
+	remaining --;
+      } else {
+	buffer = 0x00;
+      }
+      btg = sizeof(buffer) * 8;
+    }
+    b = b + b + ((buffer >> --btg) & 1);
+  } while(0);
+  
+  return b;
+}
+
+int mem_bitio_buffer::bit() {
+  if (btg == 0) {
+
+    if (remaining == 0) { 
+      this->error();
+      return 0;
+
+    } else {
+      buffer = *pos++;
+      remaining--;
+      btg = sizeof(buffer) * 8;
+    }
+    
+  }
+  return ((buffer >> --btg) & 0x01);
+}
+
+void mem_bitio_buffer::encodeBit(int b) {
+  btg--;
+  if (b) buffer |= (1 << btg);
+  
+  if (btg == 0) {
+    if (remaining) {
+      *pos++ = buffer;
+      remaining--;
+    }
+    buffer = 0;
+    btg = sizeof(buffer)*8;
+  }
+}
+
+void mem_bitio_buffer::flush() {
+  if (btg != sizeof(buffer)*8) {
+    if (remaining) {
+      *pos++ = buffer;
+      remaining--;
+    }
+  }
+  btg = sizeof(buffer)*8;
+}
+
+void mem_bitio_buffer::done() {
+  // do nothing
+}
+
+int mem_bitio_buffer::encodeLength() {
+  return pos - base;
+}
+
+int mem_bitio_buffer::encodeVerify() {
+  return (remaining != 0);
+}
+
+
+
Index: /trunk/mgpp/lib/bitio_m_mem.h
===================================================================
--- /trunk/mgpp/lib/bitio_m_mem.h	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_mem.h	(revision 3365)
@@ -0,0 +1,104 @@
+/**************************************************************************
+ *
+ * bitio_m_mem.h -- Macros for bitio to memory
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on an array 
+ *  of chars. These routines are faster than the ones in "mems" files. But 
+ *  with these routines you cannot mix reads and writes on the array of chars,
+ *  or multiple write, at the same time and guarantee them to work, also you 
+ *  cannot seek to a point and do a write. The decode routine can detect when
+ *  you run off the end of the array and will produce an approate error 
+ *  message, and the encode routine will stop when it gets to the end of the 
+ *  character array. 
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_MEM
+#define H_BITIO_M_MEM
+
+#include "bitio_m_abstract.h"
+
+
+class mem_bitio_buffer : public bitio_buffer {
+protected:
+  unsigned char *pos;
+  unsigned char *base;
+  int            remaining;
+  unsigned char  buffer;
+  unsigned char  btg;
+  
+public:
+  mem_bitio_buffer ();
+  mem_bitio_buffer (unsigned char *buffer, int size);
+  void seek (unsigned long topos);
+  void error ();
+  long add00 (long b);
+  long addff (long b);
+  int  bit ();
+  void done ();
+
+  void encodeStart ();
+  void encodeBit (int bit);
+  void encodeDone () { flush(); }
+  int  encodeVerify ();
+  int  encodeLength ();
+  void flush ();
+};
+
+
+/*
+#define ENCODE_START(p,r)						\
+  {									\
+    mem_bitio_buffer buffer(p, r);                                     \
+    buffer.encodeStart();
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    mem_bitio_buffer buffer;                                           \
+    buffer.unpause(b);
+
+#define ENCODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_FLUSH buffer.flush()
+
+#define ENCODE_DONE buffer.flush();}
+
+
+#define DECODE_START(p,r)						\
+  {	                                                                \
+    mem_bitio_buffer buffer(p,r);
+
+#define DECODE_CONTINUE(b)						\
+  {	                                                                \
+    mem_bitio_buffer buffer;                                           \
+    buffer.unpause(b);
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b)
+
+#define DECODE_ADD_00(b) b = buffer.add00(b)
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b) buffer.pause(&b);}
+*/
+
+#endif
Index: /trunk/mgpp/lib/bitio_m_mems.cpp
===================================================================
--- /trunk/mgpp/lib/bitio_m_mems.cpp	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_mems.cpp	(revision 3365)
@@ -0,0 +1,89 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_abstract.h"
+#include "bitio_m_mems.h"
+
+/**
+ * Create decoding "stream"/"file"
+ */
+mems_bitio_buffer::mems_bitio_buffer()
+{
+}
+
+mems_bitio_buffer::mems_bitio_buffer(unsigned char *readbuffer, long _pos)
+{ 
+  base = readbuffer;
+  pos  = _pos;
+}
+
+void mems_bitio_buffer::encodeStart()
+{
+  // Do nothing
+}
+
+
+void mems_bitio_buffer::error()
+{
+  fprintf(stderr, "Unexpected EOF in \"%s\" on line %d\n", __FILE__, __LINE__);
+  exit(1);
+}
+
+
+void mems_bitio_buffer::seek(unsigned long toPos)
+{ 
+  pos = toPos;
+}
+
+unsigned long mems_bitio_buffer::position()
+{
+  return pos;
+}
+
+long mems_bitio_buffer::addff(long b)
+{
+  do
+    {
+      b = b + b + (base[pos>>3] & (0x80 >> (pos & 0x07)) != 0);
+      pos ++;
+    }
+  while(0);
+
+  return b;
+}
+
+long mems_bitio_buffer::add00(long b)
+{
+  return addff(b);
+}
+
+int mems_bitio_buffer::bit()
+{
+  pos ++;
+  return ((base[(pos-1)>>3] & (0x80 >> ((pos-1) & 0x07))) != 0);
+}
+
+void mems_bitio_buffer::encodeBit(int bit)
+{
+  if (bit)
+    base[pos>>3] |= 0x80 >> (pos & 0x07);
+  else
+    base[pos>>3] &= 0xff7f >> (pos & 0x07);
+  pos++;
+}
+
+void mems_bitio_buffer::flush()
+{
+  // do nothing
+}
+
+void mems_bitio_buffer::done()
+{
+  // do nothing
+}
+
+void mems_bitio_buffer::encodeDone()
+{
+  flush();
+}
+
Index: /trunk/mgpp/lib/bitio_m_mems.h
===================================================================
--- /trunk/mgpp/lib/bitio_m_mems.h	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_mems.h	(revision 3365)
@@ -0,0 +1,100 @@
+/**************************************************************************
+ *
+ * bitio_m_mems.h -- Macros for bitio to memory (random access)
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on an array 
+ *  of chars. These routines are slower than the ones in "mem" files. but 
+ *  with these routines you can mix reads and writes, or multiple writes,  on
+ *  the array of chars at the same time and guarantee them to work, also you 
+ *  can seek to a point and do a write. The decode and encode macros cannot 
+ *  detect when the end off the character is reached and just continue 
+ *  processing.
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_MEMS
+#define H_BITIO_M_MEMS
+
+#include "bitio_m_abstract.h"
+
+
+class mems_bitio_buffer : public bitio_buffer {
+protected:
+  unsigned char *base;
+  unsigned long  pos;
+  
+public:
+  mems_bitio_buffer();
+  mems_bitio_buffer(unsigned char *readbuffer, long _pos);
+  void seek(unsigned long topos);
+  void error();
+  long add00(long b);
+  long addff(long b);
+  int  bit();
+  //  int  bitOffset();
+  void done();
+
+  unsigned long position();
+
+  void encodeStart();
+  void encodeBit(int b);
+  void encodeDone();
+  void flush();
+};
+
+
+/*#define ENCODE_START(b,p)						\
+  {									\
+    mems_bitio_buffer buffer(b, p);
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    mems_bitio_buffer buffer;                                          \
+    buffer.unpause(b);
+
+#define ENCODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_FLUSH buffer.flush();
+
+
+#define ENCODE_DONE buffer.encodeDone();}
+
+
+#define DECODE_START(b,p)						\
+  {		                                                        \
+    mems_bitio_buffer buffer(b, p);
+
+#define DECODE_CONTINUE(b)						\
+  {									\
+    mems_bitio_buffer buffer;                                          \
+    buffer.unpause(b);
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b);
+
+#define DECODE_ADD_00(b) b = buffer.add00(b);
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b) buffer.pause(&b);}
+*/
+
+#endif
Index: /trunk/mgpp/lib/bitio_m_random.cpp
===================================================================
--- /trunk/mgpp/lib/bitio_m_random.cpp	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_random.cpp	(revision 3365)
@@ -0,0 +1,218 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "longlong.h"
+#include "sysfuncs.h"
+#include "memlib.h"
+
+#include "bitio_m_abstract.h"
+#include "bitio_m_random.h"
+
+random_bitio_buffer::random_bitio_buffer(FILE *f, unsigned long length)
+{
+  buffer = NULL;
+  attachFile (f, length);
+}
+
+random_bitio_buffer::~random_bitio_buffer() {
+  if (buffer != NULL) delete buffer;
+}
+
+void random_bitio_buffer::attachFile (FILE *f, unsigned long length) {
+  // delete the old buffer
+  if (buffer != NULL) {
+    delete buffer;
+    buffer = NULL;
+  }
+
+  // reset all the values
+  file   = f;
+  pos    = 0;
+  base   = 0;
+  used   = 0;
+  len    = length - 1;
+  shift  = 0;
+
+  // make sure the length is a power of 2
+  while (len != 0) {
+    shift ++;
+    len = len >> 1;
+  }
+  len = 1 << shift;
+
+  // allocate a new buffer
+  buffer = new u_char[len];
+
+  // fill up the buffer
+  if (file != NULL) {
+    fseek(file, 0, 0);
+    fread(buffer, 1, len, file);
+  }
+}
+
+random_bitio_buffer &random_bitio_buffer::operator= (const random_bitio_buffer &_rbb) {
+  // delete the old buffer
+  if (buffer != NULL) {
+    delete buffer;
+    buffer = NULL;
+  }
+
+  file = _rbb.file;
+
+  base = _rbb.base;
+  used = _rbb.used;
+  pos = _rbb.pos;
+  len = _rbb.len;
+  shift = _rbb.shift;
+
+  // copy the buffer
+  buffer = new u_char [len];
+  //  memcpy (buffer, _rbb.buffer, len);
+memcpy (buffer, _rbb.buffer, len);
+  
+  return (*this);
+}
+
+void random_bitio_buffer::error() {
+}
+
+void random_bitio_buffer::writeRead()
+{
+  if (used)
+    {
+      fseek(file, base, 0);
+      fwrite(buffer, 1, len, file);
+    }
+  //  else
+  //    {
+      base += len;
+      fseek(file, base, 0);
+      fread(buffer, 1, len, file);
+      pos = 0;
+      used = 0;
+  //    }
+}
+
+void random_bitio_buffer::encodeBit(int bit)
+{
+  do
+    {
+      if (bit != 0)
+	{
+	  buffer[pos >> 3] |= 0x80 >> (pos & 0x07);
+	}
+      else
+	{
+	  buffer[pos >> 3] &= 0xff7f >> (pos & 0x07);
+	}
+      used = 1;
+      pos ++;
+
+      if ((pos >> 3) >= len)
+	{
+	  writeRead();
+	}
+    }
+  while (0);
+}
+
+void random_bitio_buffer::flush()
+{
+  if (used != 0)
+    {
+      fseek(file, base, 0);
+      fwrite(buffer, 1, len, file);
+      used = 0;
+    }
+}
+
+void random_bitio_buffer::encodeDone()
+{
+  flush();
+  if (buffer != NULL) {
+    delete buffer;
+    buffer = NULL;
+  }
+}
+
+unsigned long random_bitio_buffer::tell()
+{
+  return ((base << 3) + pos);
+}
+
+void random_bitio_buffer::seek(unsigned long toPos)
+{
+  if (((toPos >> 3) >= base) && (((toPos+7) >> 3) < (base + len)))
+    {
+      pos = toPos - (base << 3);
+    }
+  else
+    {
+      flush();
+      base = toPos >> 3;
+      base = (toPos >> (shift + 3)) << shift;
+      fseek(file, base, 0);
+      fread(buffer, 1, len, file);
+      pos = toPos & 0x07;
+      pos = toPos & ((8 << shift) - 1);
+    }
+}
+
+int random_bitio_buffer::bit()
+{
+  //  fprintf(stderr, "Bit in");
+  pos ++;
+  if (((pos - 1) >> 3) >= len)
+    {
+      writeRead();
+      pos = 1;
+    }
+  return (buffer[(pos - 1) >> 3] & (0x80 >> ((pos - 1) & 0x07))) != 0;
+}
+
+long random_bitio_buffer::addff(long b)
+{
+  return b + b + bit();
+}
+
+long random_bitio_buffer::add00(long b)
+{
+  return addff(b);
+}
+
+int random_bitio_buffer::bitOffset()
+{
+  return 0;
+}
+
+void random_bitio_buffer::done()
+{
+  encodeDone();
+}
+
+
+#ifdef USE_LONG_LONG
+void random_bitio_buffer::seek_LL(mg_ullong toPos)
+{
+  if (((toPos >> 3) >= (mg_ullong) base) &&
+      (((toPos + 7) >> 3) < (mg_ullong) (base + len)))
+    {
+      pos = (long) (toPos - (mg_ullong) (base << 3));
+    }
+  else
+    {
+      flush();
+      base = (long) ((toPos >> (shift + 3)) << shift);
+
+      fseek(file, base, 0);
+      fread(buffer, 1, len, file);
+      pos = (long) (toPos & ((8 << shift) - 1));
+    }
+}
+
+mg_ullong random_bitio_buffer::tell_LL()
+{
+  return (((mg_ullong) base) << 3ull) + pos;
+}
+#endif
+
Index: /trunk/mgpp/lib/bitio_m_random.h
===================================================================
--- /trunk/mgpp/lib/bitio_m_random.h	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_random.h	(revision 3365)
@@ -0,0 +1,127 @@
+/**************************************************************************
+ *
+ * bitio_m_random.h -- Macros for bitio to a file (random access)
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on an array 
+ *  of chars. These routines are slower than the ones in "mem" files. but 
+ *  with these routines you can mix reads and writes, or multiple writes,  on
+ *  the array of chars at the same time and guarantee them to work, also you 
+ *  can seek to a point and do a write. The decode and encode macros cannot 
+ *  detect when the end off the character is reached and just continue 
+ *  processing.
+ *
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_RANDOM
+#define H_BITIO_M_RANDOM
+
+#include "longlong.h"
+#include "bitio_m_abstract.h"
+
+
+class random_bitio_buffer : public bitio_buffer { 
+public:
+  FILE *file;
+  unsigned char *buffer;
+  unsigned long  base;
+  unsigned long  used;
+  unsigned long  pos;
+  unsigned long  len;
+  unsigned long  shift;
+  void writeRead();
+
+  random_bitio_buffer(FILE *f = 0,
+		      unsigned long length = 8*1024);
+  virtual ~random_bitio_buffer();
+  
+  void attachFile (FILE *f, unsigned long length);
+
+  random_bitio_buffer &operator= (const random_bitio_buffer &_rbb);
+  
+  void encodeDone();
+  void encodeBit(int b);
+  void flush();
+  unsigned long tell();
+
+  void seek(unsigned long topos);
+  void error();
+  long add00(long b);
+  long addff(long b);
+  int  bit();
+  int  bitOffset();
+  void done();
+
+#ifdef USE_LONG_LONG
+  // extras for longlong.h
+  void seek_LL(mg_ullong pos);
+  mg_ullong tell_LL();
+#endif
+};
+
+
+/*#define ENCODE_START(f,l)						\
+  {									\
+    random_bitio_buffer buffer(f, l);
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    random_bitio_buffer buffer;                                               \
+    buffer.unpause(b);
+
+#define SEEK fprintf(stderr, "Seek to %d\n",buffer.base)
+#define READ fprintf(stderr, "Read of %d\n",buffer.len)
+#define WRITE fprintf(stderr, "Write of %d\n",buffer.used)
+
+#define WRITE_READ buffer.writeRead()
+
+#define ENCODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_FLUSH buffer.flush();
+
+#define ENCODE_DONE buffer.done();}							
+
+#define DECODE_START(f,l) \
+  {									\
+    random_bitio_buffer buffer(f, l);
+
+#define DECODE_CONTINUE(b) \
+  {									\
+    random_bitio_buffer buffer;                                               \
+    buffer.unpause(b);
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b)
+
+#define DECODE_ADD_00(b) b = buffer.add00(b)
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b)	buffer.pause(&b);}
+
+#define ENCODE_TELL buffer.tell()
+
+#define DECODE_TELL buffer.tell()
+*/
+
+
+
+#endif
Index: /trunk/mgpp/lib/bitio_m_stdio.cpp
===================================================================
--- /trunk/mgpp/lib/bitio_m_stdio.cpp	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_stdio.cpp	(revision 3365)
@@ -0,0 +1,125 @@
+#include <stdlib.h>
+#include <stdio.h>
+
+#include "bitio_m_stdio.h"
+
+stdio_bitio_buffer::stdio_bitio_buffer(FILE *f) {
+  attachFile (f);
+}
+
+void stdio_bitio_buffer::attachFile (FILE *f) {
+  file = f;
+  buffer = 0;
+  btg = 0;
+  bytesWritten = 0;
+}
+
+void stdio_bitio_buffer::encodeStart() {
+  btg = sizeof(buffer) * 8;
+}
+
+void stdio_bitio_buffer::error() {
+  fprintf(stderr, "Unexpected EOF in \"%s\" on line %d", __FILE__, __LINE__);
+  exit(1);
+}
+
+void stdio_bitio_buffer::seek (unsigned long byte, unsigned char bit) {
+  if (fseek (file, byte, SEEK_SET) != 0) return;
+  buffer = getc(file);
+  btg = bit;
+  bytesWritten = 0;
+}
+
+void stdio_bitio_buffer::seek(unsigned long topos) {
+  if (fseek(file, topos >> 3, SEEK_SET) != 0) return;
+  buffer = getc(file);
+  btg = 8 - (topos & 0x07);
+  bytesWritten = 0;
+}
+
+long stdio_bitio_buffer::addff(long b)
+{
+  if (!btg)
+    {
+      register int val;
+      val = getc(file);
+      if (val == EOF)
+	{
+	  buffer = 0x0ff;
+	}
+      else
+	{
+	  buffer = val;
+	}
+      btg = sizeof(buffer) * 8;
+    }
+  b = b + b + ((buffer >> --btg) & 0x01);
+
+  return b;
+}
+
+long stdio_bitio_buffer::add00(long b)
+{
+  if (!btg)
+    {
+      register int val;
+      val = getc(file);
+      if (val == EOF)
+	{
+	  buffer = 0xff;
+	}
+      else
+	{
+	  buffer = val;
+	}
+      btg = sizeof(buffer) * 8;
+    }
+  b = b + b + ((buffer >> --btg) & 0x01);
+  
+  return b;
+}
+
+void stdio_bitio_buffer::encodeBit(int b)
+{
+  btg--;
+  if (b != 0)
+    buffer |= (1 << btg);
+
+  if (btg == 0)
+    {
+      putc(buffer, file);
+      bytesWritten += sizeof (buffer);
+      buffer = 0;
+      btg = sizeof(buffer)*8;
+    }
+}
+
+int stdio_bitio_buffer::bit()
+{
+  if (btg == 0)
+    {
+      buffer = getc(file);
+      if (feof(file))
+	{
+	  error();
+	  return 0;
+	}
+      else
+	{
+	  btg = sizeof(buffer) * 8;
+	}
+    }
+  return ((buffer >> --btg) & 0x01);
+}
+
+void stdio_bitio_buffer::done() {
+  // no tidying required
+}
+
+void stdio_bitio_buffer::encodeDone() {
+  if (btg != sizeof(buffer) * 8) {
+    putc(buffer, file);
+    bytesWritten += sizeof (buffer);
+  }
+}
+
Index: /trunk/mgpp/lib/bitio_m_stdio.h
===================================================================
--- /trunk/mgpp/lib/bitio_m_stdio.h	(revision 3365)
+++ /trunk/mgpp/lib/bitio_m_stdio.h	(revision 3365)
@@ -0,0 +1,101 @@
+/**************************************************************************
+ *
+ * bitio_m_stdio.h -- Macros for bitio to a file
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ *  This file contains macros for doing bitwise input and output on a FILE*.
+ *  With these routines you cannot mix reads and writes on the FILE,
+ *  or multiple writes, at the same time and guarantee them to work, also you 
+ *  cannot seek to a point and do a write. The decode routine can detect when
+ *  you run off the end of the file and will produce an approate error message.
+ *
+ *
+ **************************************************************************/
+
+#ifndef H_BITIO_M_STDIO
+#define H_BITIO_M_STDIO
+
+#include "bitio_m_abstract.h"
+
+
+class stdio_bitio_buffer : public bitio_buffer {
+protected:
+  FILE *file;
+  unsigned char buffer;
+  unsigned char btg;
+  unsigned long bytesWritten;
+  
+public:
+  stdio_bitio_buffer (FILE *f = NULL);
+
+  void attachFile (FILE *f);
+
+  unsigned char GetBtg () { return btg; }
+  unsigned long GetBytesWritten () { return bytesWritten; }
+  
+  void seek (unsigned long byte, unsigned char bit);
+  void seek (unsigned long topos);
+  void error();
+  long add00(long b);
+  long addff(long b);
+  int  bit();
+  int  bitOffset() { return btg; }
+  void done();
+
+  void encodeStart();
+  void encodeBit(int bit);
+  void encodeDone();
+};
+
+
+/*#define ENCODE_START(f)							\
+  {									\
+    stdio_bitio_buffer buffer(f);                                       \
+    buffer.encodeStart();
+
+#define ENCODE_CONTINUE(b)						\
+  {									\
+    stdio_bitio_buffer buffer;                                          \
+    buffer.unpause(b);
+
+#define ENCODE_PAUSE(b) buffer.pause(&b);}
+
+#define ENCODE_DONE buffer.encodeDone();}
+
+#define DECODE_START(f)							\
+  {		                                                        \
+    stdio_bitio_buffer buffer(f);
+
+#define DECODE_CONTINUE(b)						\
+  {									\
+    stdio_bitio_buffer buffer;                                          \
+     buffer.unpause(b);
+
+#define DECODE_ADD_FF(b) b = buffer.addff(b)
+
+#define DECODE_ADD_00(b) b = buffer.add00(b)
+
+#define DECODE_BIT buffer.bit()
+
+#define DECODE_DONE buffer.done();}
+
+#define DECODE_PAUSE(b) buffer.pause(&b);}
+*/
+
+#endif
Index: /trunk/mgpp/lib/ftruncate.cpp
===================================================================
--- /trunk/mgpp/lib/ftruncate.cpp	(revision 3365)
+++ /trunk/mgpp/lib/ftruncate.cpp	(revision 3365)
@@ -0,0 +1,99 @@
+/* ftruncate emulations that work on some System V's.
+   This file is in the public domain.  */
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__  /* [RPAP - Feb 97: WIN32 Port] */
+#  include <win32cfg.h>
+# else
+#  include <sysfuncs.h>
+# endif
+#endif
+
+#include <sys/types.h>
+#include <fcntl.h>
+
+#ifdef F_CHSIZE
+
+int
+ftruncate (int fd, off_t length)
+{
+  return fcntl (fd, F_CHSIZE, length);
+}
+
+#else /* not F_CHSIZE */
+#ifdef F_FREESP
+
+/* By William Kucharski <kucharsk@netcom.com>.  */
+
+#include <sys/stat.h>
+#include <errno.h>
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+int
+ftruncate (int fd, off_t length)
+{
+  struct flock fl;
+  struct stat filebuf;
+
+  if (fstat (fd, &filebuf) < 0)
+    return -1;
+
+  if (filebuf.st_size < length)
+    {
+      /* Extend file length. */
+      if (lseek (fd, (length - 1), SEEK_SET) < 0)
+	return -1;
+
+      /* Write a "0" byte. */
+      if (write (fd, "", 1) != 1)
+	return -1;
+    }
+  else
+    {
+
+      /* Truncate length. */
+
+      fl.l_whence = 0;
+      fl.l_len = 0;
+      fl.l_start = length;
+      fl.l_type = F_WRLCK;	/* write lock on file space */
+
+      /* This relies on the *undocumented* F_FREESP argument to fcntl,
+	 which truncates the file so that it ends at the position
+	 indicated by fl.l_start.  Will minor miracles never cease?  */
+
+      if (fcntl (fd, F_FREESP, &fl) < 0)
+	return -1;
+    }
+
+  return 0;
+}
+
+#else /* not F_CHSIZE nor F_FREESP */
+#ifdef HAVE_CHSIZE
+
+int
+ftruncate (int fd, off_t length)
+{
+  return chsize (fd, length);
+}
+
+#else /* not F_CHSIZE nor F_FREESP nor HAVE_CHSIZE */
+
+#include <errno.h>
+#ifndef errno
+extern int errno;
+#endif
+
+int
+ftruncate (int fd, off_t length)
+{
+  errno = EIO;
+  return -1;
+}
+
+#endif /* not HAVE_CHSIZE */
+#endif /* not F_FREESP */
+#endif /* not F_CHSIZE */
Index: /trunk/mgpp/lib/getopt_old.c
===================================================================
--- /trunk/mgpp/lib/getopt_old.c	(revision 3365)
+++ /trunk/mgpp/lib/getopt_old.c	(revision 3365)
@@ -0,0 +1,748 @@
+/* Getopt for GNU.
+   NOTE: getopt is now part of the C library, so if you don't know what
+   "Keep this file name-space clean" means, talk to roland@gnu.ai.mit.edu
+   before changing it!
+
+   Copyright (C) 1987, 88, 89, 90, 91, 92, 1993
+   Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by the
+   Free Software Foundation; either version 2, or (at your option) any
+   later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+
+
+#if !__STDC__ && !defined(const) && IN_GCC
+#define const
+#endif
+
+/* This tells Alpha OSF/1 not to define a getopt prototype in <stdio.h>.  */
+#ifndef _NO_PROTO
+#define _NO_PROTO
+#endif
+
+#include <stdio.h>
+
+/* Comment out all this code if we are using the GNU C Library, and are not
+   actually compiling the library itself.  This code is part of the GNU C
+   Library, but also included in many other GNU distributions.  Compiling
+   and linking in this code is a waste when using the GNU C library
+   (especially if it is a shared library).  Rather than having every GNU
+   program understand `configure --with-gnu-libc' and omit the object files,
+   it is simpler to just do this in the source for each such file.  */
+
+#if defined (_LIBC) || !defined (__GNU_LIBRARY__)
+
+
+/* This needs to come after some library #include
+   to get __GNU_LIBRARY__ defined.  */
+#ifdef	__GNU_LIBRARY__
+/* Don't include stdlib.h for non-GNU C libraries because some of them
+   contain conflicting prototypes for getopt.  */
+#include <stdlib.h>
+#endif /* GNU C library.  */
+
+/* If GETOPT_COMPAT is defined, `+' as well as `--' can introduce a
+   long-named option.  Because this is not POSIX.2 compliant, it is
+   being phased out.  */
+/* #define GETOPT_COMPAT */
+
+/* This version of `getopt' appears to the caller like standard Unix `getopt'
+   but it behaves differently for the user, since it allows the user
+   to intersperse the options with the other arguments.
+
+   As `getopt' works, it permutes the elements of ARGV so that,
+   when it is done, all the options precede everything else.  Thus
+   all application programs are extended to handle flexible argument order.
+
+   Setting the environment variable POSIXLY_CORRECT disables permutation.
+   Then the behavior is completely standard.
+
+   GNU application programs can use a third alternative mode in which
+   they can distinguish the relative order of options and other arguments.  */
+
+/* jrm21 - GSDL: getopt.h -> getopt_old.h */
+#include "getopt_old.h"
+
+/* For communication from `getopt' to the caller.
+   When `getopt' finds an option that takes an argument,
+   the argument value is returned here.
+   Also, when `ordering' is RETURN_IN_ORDER,
+   each non-option ARGV-element is returned here.  */
+
+char *optarg = 0;
+
+/* Index in ARGV of the next element to be scanned.
+   This is used for communication to and from the caller
+   and for communication between successive calls to `getopt'.
+
+   On entry to `getopt', zero means this is the first call; initialize.
+
+   When `getopt' returns EOF, this is the index of the first of the
+   non-option elements that the caller should itself scan.
+
+   Otherwise, `optind' communicates from one call to the next
+   how much of ARGV has been scanned so far.  */
+
+/* XXX 1003.2 says this must be 1 before any call.  */
+int optind = 0;
+
+/* The next char to be scanned in the option-element
+   in which the last option character we returned was found.
+   This allows us to pick up the scan where we left off.
+
+   If this is zero, or a null string, it means resume the scan
+   by advancing to the next ARGV-element.  */
+
+static char *nextchar;
+
+/* Callers store zero here to inhibit the error message
+   for unrecognized options.  */
+
+int opterr = 1;
+
+/* Set to an option character which was unrecognized.
+   This must be initialized on some systems to avoid linking in the
+   system's own getopt implementation.  */
+
+int optopt = '?';
+
+/* Describe how to deal with options that follow non-option ARGV-elements.
+
+   If the caller did not specify anything,
+   the default is REQUIRE_ORDER if the environment variable
+   POSIXLY_CORRECT is defined, PERMUTE otherwise.
+
+   REQUIRE_ORDER means don't recognize them as options;
+   stop option processing when the first non-option is seen.
+   This is what Unix does.
+   This mode of operation is selected by either setting the environment
+   variable POSIXLY_CORRECT, or using `+' as the first character
+   of the list of option characters.
+
+   PERMUTE is the default.  We permute the contents of ARGV as we scan,
+   so that eventually all the non-options are at the end.  This allows options
+   to be given in any order, even with programs that were not written to
+   expect this.
+
+   RETURN_IN_ORDER is an option available to programs that were written
+   to expect options and other ARGV-elements in any order and that care about
+   the ordering of the two.  We describe each non-option ARGV-element
+   as if it were the argument of an option with character code 1.
+   Using `-' as the first character of the list of option characters
+   selects this mode of operation.
+
+   The special argument `--' forces an end of option-scanning regardless
+   of the value of `ordering'.  In the case of RETURN_IN_ORDER, only
+   `--' can cause `getopt' to return EOF with `optind' != ARGC.  */
+
+static enum
+  {
+    REQUIRE_ORDER, PERMUTE, RETURN_IN_ORDER
+  }
+ordering;
+
+
+#ifdef	__GNU_LIBRARY__
+/* We want to avoid inclusion of string.h with non-GNU libraries
+   because there are many ways it can cause trouble.
+   On some systems, it contains special magic macros that don't work
+   in GCC.  */
+#include <string.h>
+#define	my_index	strchr
+#else
+
+/* Avoid depending on library functions or files
+   whose names are inconsistent.  */
+
+char *getenv ();
+
+static char *
+my_index (str, chr)
+     const char *str;
+     int chr;
+{
+  while (*str)
+    {
+      if (*str == chr)
+	return (char *) str;
+      str++;
+    }
+  return 0;
+}
+
+/* If using GCC, we can safely declare strlen this way.
+   If not using GCC, it is ok not to declare it.  */
+#ifdef __GNUC__
+#ifdef IN_GCC
+#include "gstddef.h"
+#else
+#include <stddef.h>
+#endif
+extern size_t strlen (const char *);
+#endif
+
+#endif /* GNU C library.  */
+
+
+/* Handle permutation of arguments.  */
+
+/* Describe the part of ARGV that contains non-options that have
+   been skipped.  `first_nonopt' is the index in ARGV of the first of them;
+   `last_nonopt' is the index after the last of them.  */
+
+static int first_nonopt;
+static int last_nonopt;
+
+/* Exchange two adjacent subsequences of ARGV.
+   One subsequence is elements [first_nonopt,last_nonopt)
+   which contains all the non-options that have been skipped so far.
+   The other is elements [last_nonopt,optind), which contains all
+   the options processed since those non-options were skipped.
+
+   `first_nonopt' and `last_nonopt' are relocated so that they describe
+   the new indices of the non-options in ARGV after they are moved.  */
+
+static void
+exchange (argv)
+     char **argv;
+{
+  int bottom = first_nonopt;
+  int middle = last_nonopt;
+  int top = optind;
+  char *tem;
+
+  /* Exchange the shorter segment with the far end of the longer segment.
+     That puts the shorter segment into the right place.
+     It leaves the longer segment in the right place overall,
+     but it consists of two parts that need to be swapped next.  */
+
+  while (top > middle && middle > bottom)
+    {
+      if (top - middle > middle - bottom)
+	{
+	  /* Bottom segment is the short one.  */
+	  int len = middle - bottom;
+	  register int i;
+
+	  /* Swap it with the top part of the top segment.  */
+	  for (i = 0; i < len; i++)
+	    {
+	      tem = argv[bottom + i];
+	      argv[bottom + i] = argv[top - (middle - bottom) + i];
+	      argv[top - (middle - bottom) + i] = tem;
+	    }
+	  /* Exclude the moved bottom segment from further swapping.  */
+	  top -= len;
+	}
+      else
+	{
+	  /* Top segment is the short one.  */
+	  int len = top - middle;
+	  register int i;
+
+	  /* Swap it with the bottom part of the bottom segment.  */
+	  for (i = 0; i < len; i++)
+	    {
+	      tem = argv[bottom + i];
+	      argv[bottom + i] = argv[middle + i];
+	      argv[middle + i] = tem;
+	    }
+	  /* Exclude the moved top segment from further swapping.  */
+	  bottom += len;
+	}
+    }
+
+  /* Update records for the slots the non-options now occupy.  */
+
+  first_nonopt += (optind - last_nonopt);
+  last_nonopt = optind;
+}
+
+
+/* Scan elements of ARGV (whose length is ARGC) for option characters
+   given in OPTSTRING.
+
+   If an element of ARGV starts with '-', and is not exactly "-" or "--",
+   then it is an option element.  The characters of this element
+   (aside from the initial '-') are option characters.  If `getopt'
+   is called repeatedly, it returns successively each of the option characters
+   from each of the option elements.
+
+   If `getopt' finds another option character, it returns that character,
+   updating `optind' and `nextchar' so that the next call to `getopt' can
+   resume the scan with the following option character or ARGV-element.
+
+   If there are no more option characters, `getopt' returns `EOF'.
+   Then `optind' is the index in ARGV of the first ARGV-element
+   that is not an option.  (The ARGV-elements have been permuted
+   so that those that are not options now come last.)
+
+   OPTSTRING is a string containing the legitimate option characters.
+   If an option character is seen that is not listed in OPTSTRING,
+   return '?' after printing an error message.  If you set `opterr' to
+   zero, the error message is suppressed but we still return '?'.
+
+   If a char in OPTSTRING is followed by a colon, that means it wants an arg,
+   so the following text in the same ARGV-element, or the text of the following
+   ARGV-element, is returned in `optarg'.  Two colons mean an option that
+   wants an optional arg; if there is text in the current ARGV-element,
+   it is returned in `optarg', otherwise `optarg' is set to zero.
+
+   If OPTSTRING starts with `-' or `+', it requests different methods of
+   handling the non-option ARGV-elements.
+   See the comments about RETURN_IN_ORDER and REQUIRE_ORDER, above.
+
+   Long-named options begin with `--' instead of `-'.
+   Their names may be abbreviated as long as the abbreviation is unique
+   or is an exact match for some defined option.  If they have an
+   argument, it follows the option name in the same ARGV-element, separated
+   from the option name by a `=', or else the in next ARGV-element.
+   When `getopt' finds a long-named option, it returns 0 if that option's
+   `flag' field is nonzero, the value of the option's `val' field
+   if the `flag' field is zero.
+
+   The elements of ARGV aren't really const, because we permute them.
+   But we pretend they're const in the prototype to be compatible
+   with other systems.
+
+   LONGOPTS is a vector of `struct option' terminated by an
+   element containing a name which is zero.
+
+   LONGIND returns the index in LONGOPT of the long-named option found.
+   It is only valid when a long-named option has been found by the most
+   recent call.
+
+   If LONG_ONLY is nonzero, '-' as well as '--' can introduce
+   long-named options.  */
+
+int
+_getopt_internal (argc, argv, optstring, longopts, longind, long_only)
+     int argc;
+     char *const *argv;
+     const char *optstring;
+     const struct option *longopts;
+     int *longind;
+     int long_only;
+{
+  int option_index;
+
+  optarg = 0;
+
+  /* Initialize the internal data when the first call is made.
+     Start processing options with ARGV-element 1 (since ARGV-element 0
+     is the program name); the sequence of previously skipped
+     non-option ARGV-elements is empty.  */
+
+  if (optind == 0)
+    {
+      first_nonopt = last_nonopt = optind = 1;
+
+      nextchar = NULL;
+
+      /* Determine how to handle the ordering of options and nonoptions.  */
+
+      if (optstring[0] == '-')
+	{
+	  ordering = RETURN_IN_ORDER;
+	  ++optstring;
+	}
+      else if (optstring[0] == '+')
+	{
+	  ordering = REQUIRE_ORDER;
+	  ++optstring;
+	}
+      else if (getenv ("POSIXLY_CORRECT") != NULL)
+	ordering = REQUIRE_ORDER;
+      else
+	ordering = PERMUTE;
+    }
+
+  if (nextchar == NULL || *nextchar == '\0')
+    {
+      if (ordering == PERMUTE)
+	{
+	  /* If we have just processed some options following some non-options,
+	     exchange them so that the options come first.  */
+
+	  if (first_nonopt != last_nonopt && last_nonopt != optind)
+	    exchange ((char **) argv);
+	  else if (last_nonopt != optind)
+	    first_nonopt = optind;
+
+	  /* Now skip any additional non-options
+	     and extend the range of non-options previously skipped.  */
+
+	  while (optind < argc
+		 && (argv[optind][0] != '-' || argv[optind][1] == '\0')
+#ifdef GETOPT_COMPAT
+		 && (longopts == NULL
+		     || argv[optind][0] != '+' || argv[optind][1] == '\0')
+#endif /* GETOPT_COMPAT */
+	    )
+	    optind++;
+	  last_nonopt = optind;
+	}
+
+      /* Special ARGV-element `--' means premature end of options.
+         Skip it like a null option,
+         then exchange with previous non-options as if it were an option,
+         then skip everything else like a non-option.  */
+
+      if (optind != argc && !strcmp (argv[optind], "--"))
+	{
+	  optind++;
+
+	  if (first_nonopt != last_nonopt && last_nonopt != optind)
+	    exchange ((char **) argv);
+	  else if (first_nonopt == last_nonopt)
+	    first_nonopt = optind;
+	  last_nonopt = argc;
+
+	  optind = argc;
+	}
+
+      /* If we have done all the ARGV-elements, stop the scan
+         and back over any non-options that we skipped and permuted.  */
+
+      if (optind == argc)
+	{
+	  /* Set the next-arg-index to point at the non-options
+	     that we previously skipped, so the caller will digest them.  */
+	  if (first_nonopt != last_nonopt)
+	    optind = first_nonopt;
+	  return EOF;
+	}
+
+      /* If we have come to a non-option and did not permute it,
+         either stop the scan or describe it to the caller and pass it by.  */
+
+      if ((argv[optind][0] != '-' || argv[optind][1] == '\0')
+#ifdef GETOPT_COMPAT
+	  && (longopts == NULL
+	      || argv[optind][0] != '+' || argv[optind][1] == '\0')
+#endif /* GETOPT_COMPAT */
+	)
+	{
+	  if (ordering == REQUIRE_ORDER)
+	    return EOF;
+	  optarg = argv[optind++];
+	  return 1;
+	}
+
+      /* We have found another option-ARGV-element.
+         Start decoding its characters.  */
+
+      nextchar = (argv[optind] + 1
+		  + (longopts != NULL && argv[optind][1] == '-'));
+    }
+
+  if (longopts != NULL
+      && ((argv[optind][0] == '-'
+	   && (argv[optind][1] == '-' || long_only))
+#ifdef GETOPT_COMPAT
+	  || argv[optind][0] == '+'
+#endif /* GETOPT_COMPAT */
+      ))
+    {
+      const struct option *p;
+      char *s = nextchar;
+      int exact = 0;
+      int ambig = 0;
+      const struct option *pfound = NULL;
+      int indfound;
+
+      while (*s && *s != '=')
+	s++;
+
+      /* Test all options for either exact match or abbreviated matches.  */
+      for (p = longopts, option_index = 0; p->name;
+	   p++, option_index++)
+	if (!strncmp (p->name, nextchar, s - nextchar))
+	  {
+	    if (s - nextchar == strlen (p->name))
+	      {
+		/* Exact match found.  */
+		pfound = p;
+		indfound = option_index;
+		exact = 1;
+		break;
+	      }
+	    else if (pfound == NULL)
+	      {
+		/* First nonexact match found.  */
+		pfound = p;
+		indfound = option_index;
+	      }
+	    else
+	      /* Second nonexact match found.  */
+	      ambig = 1;
+	  }
+
+      if (ambig && !exact)
+	{
+	  if (opterr)
+	    fprintf (stderr, "%s: option `%s' is ambiguous\n",
+		     argv[0], argv[optind]);
+	  nextchar += strlen (nextchar);
+	  optind++;
+	  return '?';
+	}
+
+      if (pfound != NULL)
+	{
+	  option_index = indfound;
+	  optind++;
+	  if (*s)
+	    {
+	      /* Don't test has_arg with >, because some C compilers don't
+	         allow it to be used on enums.  */
+	      if (pfound->has_arg)
+		optarg = s + 1;
+	      else
+		{
+		  if (opterr)
+		    {
+		      if (argv[optind - 1][1] == '-')
+			/* --option */
+			fprintf (stderr,
+			    "%s: option `--%s' doesn't allow an argument\n",
+				 argv[0], pfound->name);
+		      else
+			/* +option or -option */
+			fprintf (stderr,
+			    "%s: option `%c%s' doesn't allow an argument\n",
+				 argv[0], argv[optind - 1][0], pfound->name);
+		    }
+		  nextchar += strlen (nextchar);
+		  return '?';
+		}
+	    }
+	  else if (pfound->has_arg == 1)
+	    {
+	      if (optind < argc)
+		optarg = argv[optind++];
+	      else
+		{
+		  if (opterr)
+		    fprintf (stderr, "%s: option `%s' requires an argument\n",
+			     argv[0], argv[optind - 1]);
+		  nextchar += strlen (nextchar);
+		  return optstring[0] == ':' ? ':' : '?';
+		}
+	    }
+	  nextchar += strlen (nextchar);
+	  if (longind != NULL)
+	    *longind = option_index;
+	  if (pfound->flag)
+	    {
+	      *(pfound->flag) = pfound->val;
+	      return 0;
+	    }
+	  return pfound->val;
+	}
+      /* Can't find it as a long option.  If this is not getopt_long_only,
+         or the option starts with '--' or is not a valid short
+         option, then it's an error.
+         Otherwise interpret it as a short option.  */
+      if (!long_only || argv[optind][1] == '-'
+#ifdef GETOPT_COMPAT
+	  || argv[optind][0] == '+'
+#endif /* GETOPT_COMPAT */
+	  || my_index (optstring, *nextchar) == NULL)
+	{
+	  if (opterr)
+	    {
+	      if (argv[optind][1] == '-')
+		/* --option */
+		fprintf (stderr, "%s: unrecognized option `--%s'\n",
+			 argv[0], nextchar);
+	      else
+		/* +option or -option */
+		fprintf (stderr, "%s: unrecognized option `%c%s'\n",
+			 argv[0], argv[optind][0], nextchar);
+	    }
+	  nextchar = (char *) "";
+	  optind++;
+	  return '?';
+	}
+    }
+
+  /* Look at and handle the next option-character.  */
+
+  {
+    char c = *nextchar++;
+    char *temp = my_index (optstring, c);
+
+    /* Increment `optind' when we start to process its last character.  */
+    if (*nextchar == '\0')
+      ++optind;
+
+    if (temp == NULL || c == ':')
+      {
+	if (opterr)
+	  {
+#if 0
+	    if (c < 040 || c >= 0177)
+	      fprintf (stderr, "%s: unrecognized option, character code 0%o\n",
+		       argv[0], c);
+	    else
+	      fprintf (stderr, "%s: unrecognized option `-%c'\n", argv[0], c);
+#else
+	    /* 1003.2 specifies the format of this message.  */
+	    fprintf (stderr, "%s: illegal option -- %c\n", argv[0], c);
+#endif
+	  }
+	optopt = c;
+	return '?';
+      }
+    if (temp[1] == ':')
+      {
+	if (temp[2] == ':')
+	  {
+	    /* This is an option that accepts an argument optionally.  */
+	    if (*nextchar != '\0')
+	      {
+		optarg = nextchar;
+		optind++;
+	      }
+	    else
+	      optarg = 0;
+	    nextchar = NULL;
+	  }
+	else
+	  {
+	    /* This is an option that requires an argument.  */
+	    if (*nextchar != '\0')
+	      {
+		optarg = nextchar;
+		/* If we end this ARGV-element by taking the rest as an arg,
+		   we must advance to the next element now.  */
+		optind++;
+	      }
+	    else if (optind == argc)
+	      {
+		if (opterr)
+		  {
+#if 0
+		    fprintf (stderr, "%s: option `-%c' requires an argument\n",
+			     argv[0], c);
+#else
+		    /* 1003.2 specifies the format of this message.  */
+		    fprintf (stderr, "%s: option requires an argument -- %c\n",
+			     argv[0], c);
+#endif
+		  }
+		optopt = c;
+		if (optstring[0] == ':')
+		  c = ':';
+		else
+		  c = '?';
+	      }
+	    else
+	      /* We already incremented `optind' once;
+	         increment it again when taking next ARGV-elt as argument.  */
+	      optarg = argv[optind++];
+	    nextchar = NULL;
+	  }
+      }
+    return c;
+  }
+}
+
+int
+getopt (argc, argv, optstring)
+     int argc;
+     char *const *argv;
+     const char *optstring;
+{
+  return _getopt_internal (argc, argv, optstring,
+			   (const struct option *) 0,
+			   (int *) 0,
+			   0);
+}
+
+#endif /* _LIBC or not __GNU_LIBRARY__.  */
+
+
+#ifdef TEST
+
+/* Compile with -DTEST to make an executable for use in testing
+   the above definition of `getopt'.  */
+
+int
+main (argc, argv)
+     int argc;
+     char **argv;
+{
+  int c;
+  int digit_optind = 0;
+
+  while (1)
+    {
+      int this_option_optind = optind ? optind : 1;
+
+      c = getopt (argc, argv, "abc:d:0123456789");
+      if (c == EOF)
+	break;
+
+      switch (c)
+	{
+	case '0':
+	case '1':
+	case '2':
+	case '3':
+	case '4':
+	case '5':
+	case '6':
+	case '7':
+	case '8':
+	case '9':
+	  if (digit_optind != 0 && digit_optind != this_option_optind)
+	    printf ("digits occur in two different argv-elements.\n");
+	  digit_optind = this_option_optind;
+	  printf ("option %c\n", c);
+	  break;
+
+	case 'a':
+	  printf ("option a\n");
+	  break;
+
+	case 'b':
+	  printf ("option b\n");
+	  break;
+
+	case 'c':
+	  printf ("option c with value `%s'\n", optarg);
+	  break;
+
+	case '?':
+	  break;
+
+	default:
+	  printf ("?? getopt returned character code 0%o ??\n", c);
+	}
+    }
+
+  if (optind < argc)
+    {
+      printf ("non-option ARGV-elements: ");
+      while (optind < argc)
+	printf ("%s ", argv[optind++]);
+      printf ("\n");
+    }
+
+  exit (0);
+}
+
+#endif /* TEST */
Index: /trunk/mgpp/lib/getopt_old.h
===================================================================
--- /trunk/mgpp/lib/getopt_old.h	(revision 3365)
+++ /trunk/mgpp/lib/getopt_old.h	(revision 3365)
@@ -0,0 +1,130 @@
+/* Declarations for getopt.
+   Copyright (C) 1989, 1990, 1991, 1992, 1993 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify it
+   under the terms of the GNU General Public License as published by the
+   Free Software Foundation; either version 2, or (at your option) any
+   later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifndef _GETOPT_H
+#define _GETOPT_H 1
+
+#ifdef	__cplusplus
+extern "C"
+{
+#endif
+
+/* For communication from `getopt' to the caller.
+   When `getopt' finds an option that takes an argument,
+   the argument value is returned here.
+   Also, when `ordering' is RETURN_IN_ORDER,
+   each non-option ARGV-element is returned here.  */
+
+  extern char *optarg;
+
+/* Index in ARGV of the next element to be scanned.
+   This is used for communication to and from the caller
+   and for communication between successive calls to `getopt'.
+
+   On entry to `getopt', zero means this is the first call; initialize.
+
+   When `getopt' returns EOF, this is the index of the first of the
+   non-option elements that the caller should itself scan.
+
+   Otherwise, `optind' communicates from one call to the next
+   how much of ARGV has been scanned so far.  */
+
+  extern int optind;
+
+/* Callers store zero here to inhibit the error message `getopt' prints
+   for unrecognized options.  */
+
+  extern int opterr;
+
+/* Set to an option character which was unrecognized.  */
+
+  extern int optopt;
+
+/* Describe the long-named options requested by the application.
+   The LONG_OPTIONS argument to getopt_long or getopt_long_only is a vector
+   of `struct option' terminated by an element containing a name which is
+   zero.
+
+   The field `has_arg' is:
+   no_argument          (or 0) if the option does not take an argument,
+   required_argument    (or 1) if the option requires an argument,
+   optional_argument    (or 2) if the option takes an optional argument.
+
+   If the field `flag' is not NULL, it points to a variable that is set
+   to the value given in the field `val' when the option is found, but
+   left unchanged if the option is not found.
+
+   To have a long-named option do something other than set an `int' to
+   a compiled-in constant, such as set a value from `optarg', set the
+   option's `flag' field to zero and its `val' field to a nonzero
+   value (the equivalent single-letter option character, if there is
+   one).  For long options that have a zero `flag' field, `getopt'
+   returns the contents of the `val' field.  */
+
+  struct option
+    {
+#if	__STDC__
+      const char *name;
+#else
+      char *name;
+#endif
+      /* has_arg can't be an enum because some compilers complain about
+         type mismatches in all the code that assumes it is an int.  */
+      int has_arg;
+      int *flag;
+      int val;
+    };
+
+/* Names for the values of the `has_arg' field of `struct option'.  */
+
+#define	no_argument		0
+#define required_argument	1
+#define optional_argument	2
+
+#if __STDC__
+#if defined(__GNU_LIBRARY__)
+/* Many other libraries have conflicting prototypes for getopt, with
+   differences in the consts, in stdlib.h.  To avoid compilation
+   errors, only prototype getopt for the GNU C library.  */
+  extern int getopt (int argc, char *const *argv, const char *shortopts);
+#else				/* not __GNU_LIBRARY__ */
+  extern int getopt ();
+#endif				/* not __GNU_LIBRARY__ */
+  extern int getopt_long (int argc, char *const *argv, const char *shortopts,
+			  const struct option *longopts, int *longind);
+  extern int getopt_long_only (int argc, char *const *argv,
+			       const char *shortopts,
+			       const struct option *longopts, int *longind);
+
+/* Internal only.  Users should not call this directly.  */
+  extern int _getopt_internal (int argc, char *const *argv,
+			       const char *shortopts,
+			       const struct option *longopts, int *longind,
+			       int long_only);
+#else				/* not __STDC__ */
+  extern int getopt ();
+  extern int getopt_long ();
+  extern int getopt_long_only ();
+
+  extern int _getopt_internal ();
+#endif				/* not __STDC__ */
+
+#ifdef	__cplusplus
+}
+#endif
+
+#endif				/* _GETOPT_H */
Index: /trunk/mgpp/lib/getpagesize.h
===================================================================
--- /trunk/mgpp/lib/getpagesize.h	(revision 3365)
+++ /trunk/mgpp/lib/getpagesize.h	(revision 3365)
@@ -0,0 +1,38 @@
+/* Emulate getpagesize on systems that lack it.  */
+
+#ifndef HAVE_GETPAGESIZE
+
+#ifdef VMS
+#define getpagesize() 512
+#endif
+
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+#ifdef _SC_PAGESIZE
+#define getpagesize() sysconf(_SC_PAGESIZE)
+#else
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifndef __WIN32__
+#include <sys/param.h>
+#endif
+
+#ifdef EXEC_PAGESIZE
+#define getpagesize() EXEC_PAGESIZE
+#else
+#ifdef NBPG
+#define getpagesize() NBPG * CLSIZE
+#ifndef CLSIZE
+#define CLSIZE 1
+#endif /* no CLSIZE */
+#else /* no NBPG */
+#ifdef NBPC
+#define getpagesize() NBPC
+#endif /* NBPC */
+#endif /* no NBPG */
+#endif /* no EXEC_PAGESIZE */
+#endif /* no _SC_PAGESIZE */
+
+#endif /* not HAVE_GETPAGESIZE */
Index: /trunk/mgpp/lib/gmalloc.c
===================================================================
--- /trunk/mgpp/lib/gmalloc.c	(revision 3365)
+++ /trunk/mgpp/lib/gmalloc.c	(revision 3365)
@@ -0,0 +1,1318 @@
+/* DO NOT EDIT THIS FILE -- it is automagically generated.  -*- C -*- */
+
+#define _MALLOC_INTERNAL
+
+/* The malloc headers and source files from the C library follow here.  */
+
+/* Declarations for `malloc' and friends.
+   Copyright 1990, 1991, 1992, 1993 Free Software Foundation, Inc.
+		  Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef _MALLOC_H
+
+#define _MALLOC_H	1
+
+#ifdef _MALLOC_INTERNAL
+
+#ifdef	HAVE_CONFIG_H
+#include <sysfuncs.h>
+#endif
+
+#if	defined(_LIBC) || defined(STDC_HEADERS) || defined(USG)
+#include <string.h>
+#else
+#ifndef memset
+#define	memset(s, zero, n)	bzero ((s), (n))
+#endif
+#ifndef memcpy
+#define	memcpy(d, s, n)		bcopy ((s), (d), (n))
+#endif
+#endif
+
+#if	defined(__GNU_LIBRARY__) || __STDC__
+#include <limits.h>
+#else
+#define	CHAR_BIT	8
+#endif
+
+#ifdef	HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+#endif	/* _MALLOC_INTERNAL.  */
+
+
+#ifdef	__cplusplus
+extern "C"
+{
+#endif
+
+#if defined (__cplusplus) || __STDC__
+#undef	__P
+#define	__P(args)	args
+#undef	__ptr_t
+#define	__ptr_t		void *
+#else /* Not C++ or ANSI C.  */
+#undef	__P
+#define	__P(args)	()
+#undef	const
+#define	const
+#undef	__ptr_t
+#define	__ptr_t		char *
+#endif /* C++ or ANSI C.  */
+
+#if	__STDC__
+#include <stddef.h>
+#else
+#undef	size_t
+#define	size_t		unsigned int
+#undef	ptrdiff_t
+#define	ptrdiff_t	int
+#endif
+
+#ifndef	NULL
+#define	NULL	0
+#endif
+
+
+/* Allocate SIZE bytes of memory.  */
+extern __ptr_t malloc __P ((size_t __size));
+/* Re-allocate the previously allocated block
+   in __ptr_t, making the new block SIZE bytes long.  */
+extern __ptr_t realloc __P ((__ptr_t __ptr, size_t __size));
+/* Allocate NMEMB elements of SIZE bytes each, all initialized to 0.  */
+extern __ptr_t calloc __P ((size_t __nmemb, size_t __size));
+/* Free a block allocated by `malloc', `realloc' or `calloc'.  */
+extern void free __P ((__ptr_t __ptr));
+
+/* Allocate SIZE bytes allocated to ALIGNMENT bytes.  */
+extern __ptr_t memalign __P ((size_t __alignment, size_t __size));
+
+/* Allocate SIZE bytes on a page boundary.  */
+extern __ptr_t valloc __P ((size_t __size));
+
+
+#ifdef _MALLOC_INTERNAL
+
+/* The allocator divides the heap into blocks of fixed size; large
+   requests receive one or more whole blocks, and small requests
+   receive a fragment of a block.  Fragment sizes are powers of two,
+   and all fragments of a block are the same size.  When all the
+   fragments in a block have been freed, the block itself is freed.  */
+#define INT_BIT		(CHAR_BIT * sizeof(int))
+#define BLOCKLOG	(INT_BIT > 16 ? 12 : 9)
+#define BLOCKSIZE	(1 << BLOCKLOG)
+#define BLOCKIFY(SIZE)	(((SIZE) + BLOCKSIZE - 1) / BLOCKSIZE)
+
+/* Determine the amount of memory spanned by the initial heap table
+   (not an absolute limit).  */
+#define HEAP		(INT_BIT > 16 ? 4194304 : 65536)
+
+/* Number of contiguous free blocks allowed to build up at the end of
+   memory before they will be returned to the system.  */
+#define FINAL_FREE_BLOCKS	8
+
+/* Data structure giving per-block information.  */
+typedef union
+  {
+    /* Heap information for a busy block.  */
+    struct
+      {
+	/* Zero for a large block, or positive giving the
+	   logarithm to the base two of the fragment size.  */
+	int type;
+	union
+	  {
+	    struct
+	      {
+		size_t nfree;	/* Free fragments in a fragmented block.  */
+		size_t first;	/* First free fragment of the block.  */
+	      } frag;
+	    /* Size (in blocks) of a large cluster.  */
+	    size_t size;
+	  } info;
+      } busy;
+    /* Heap information for a free block
+       (that may be the first of a free cluster).  */
+    struct
+      {
+	size_t size;		/* Size (in blocks) of a free cluster.  */
+	size_t next;		/* Index of next free cluster.  */
+	size_t prev;		/* Index of previous free cluster.  */
+      } free;
+  } malloc_info;
+
+/* Pointer to first block of the heap.  */
+extern char *_heapbase;
+
+/* Table indexed by block number giving per-block information.  */
+extern malloc_info *_heapinfo;
+
+/* Address to block number and vice versa.  */
+#define BLOCK(A)	(((char *) (A) - _heapbase) / BLOCKSIZE + 1)
+#define ADDRESS(B)	((__ptr_t) (((B) - 1) * BLOCKSIZE + _heapbase))
+
+/* Current search index for the heap table.  */
+extern size_t _heapindex;
+
+/* Limit of valid info table indices.  */
+extern size_t _heaplimit;
+
+/* Doubly linked lists of free fragments.  */
+struct list
+  {
+    struct list *next;
+    struct list *prev;
+  };
+
+/* Free list headers for each fragment size.  */
+extern struct list _fraghead[];
+
+/* List of blocks allocated with `memalign' (or `valloc').  */
+struct alignlist
+  {
+    struct alignlist *next;
+    __ptr_t aligned;		/* The address that memaligned returned.  */
+    __ptr_t exact;		/* The address that malloc returned.  */
+  };
+extern struct alignlist *_aligned_blocks;
+
+/* Instrumentation.  */
+extern size_t _chunks_used;
+extern size_t _bytes_used;
+extern size_t _chunks_free;
+extern size_t _bytes_free;
+
+/* Internal version of `free' used in `morecore' (malloc.c). */
+extern void _free_internal __P ((__ptr_t __ptr));
+
+#endif /* _MALLOC_INTERNAL.  */
+
+/* Underlying allocation function; successive calls should
+   return contiguous pieces of memory.  */
+extern __ptr_t (*__morecore) __P ((ptrdiff_t __size));
+
+/* Default value of `__morecore'.  */
+extern __ptr_t __default_morecore __P ((ptrdiff_t __size));
+
+/* If not NULL, this function is called after each time
+   `__morecore' is called to increase the data size.  */
+extern void (*__after_morecore_hook) __P ((void));
+
+/* Nonzero if `malloc' has been called and done its initialization.  */
+extern int __malloc_initialized;
+
+/* Hooks for debugging versions.  */
+extern void (*__free_hook) __P ((__ptr_t __ptr));
+extern __ptr_t (*__malloc_hook) __P ((size_t __size));
+extern __ptr_t (*__realloc_hook) __P ((__ptr_t __ptr, size_t __size));
+
+/* Return values for `mprobe': these are the kinds of inconsistencies that
+   `mcheck' enables detection of.  */
+enum mcheck_status
+  {
+    MCHECK_DISABLED = -1,	/* Consistency checking is not turned on.  */
+    MCHECK_OK,			/* Block is fine.  */
+    MCHECK_FREE,		/* Block freed twice.  */
+    MCHECK_HEAD,		/* Memory before the block was clobbered.  */
+    MCHECK_TAIL			/* Memory after the block was clobbered.  */
+  };
+
+/* Activate a standard collection of debugging hooks.  This must be called
+   before `malloc' is ever called.  ABORTFUNC is called with an error code
+   (see enum above) when an inconsistency is detected.  If ABORTFUNC is
+   null, the standard function prints on stderr and then calls `abort'.  */
+extern int mcheck __P ((void (*__abortfunc) __P ((enum mcheck_status))));
+
+/* Check for aberrations in a particular malloc'd block.  You must have
+   called `mcheck' already.  These are the same checks that `mcheck' does
+   when you free or reallocate a block.  */
+extern enum mcheck_status mprobe __P ((__ptr_t __ptr));
+
+/* Activate a standard collection of tracing hooks.  */
+extern void mtrace __P ((void));
+
+/* Statistics available to the user.  */
+struct mstats
+  {
+    size_t bytes_total;		/* Total size of the heap. */
+    size_t chunks_used;		/* Chunks allocated by the user. */
+    size_t bytes_used;		/* Byte total of user-allocated chunks. */
+    size_t chunks_free;		/* Chunks in the free list. */
+    size_t bytes_free;		/* Byte total of chunks in the free list. */
+  };
+
+/* Pick up the current statistics. */
+extern struct mstats mstats __P ((void));
+
+/* Call WARNFUN with a warning message when memory usage is high.  */
+extern void memory_warnings __P ((__ptr_t __start,
+				  void (*__warnfun) __P ((const char *))));
+
+
+/* Relocating allocator.  */
+
+/* Allocate SIZE bytes, and store the address in *HANDLEPTR.  */
+extern __ptr_t r_alloc __P ((__ptr_t *__handleptr, size_t __size));
+
+/* Free the storage allocated in HANDLEPTR.  */
+extern void r_alloc_free __P ((__ptr_t *__handleptr));
+
+/* Adjust the block at HANDLEPTR to be SIZE bytes long.  */
+extern __ptr_t r_re_alloc __P ((__ptr_t *__handleptr, size_t __size));
+
+
+#ifdef	__cplusplus
+}
+#endif
+
+#endif /* malloc.h  */
+/* Allocate memory on a page boundary.
+   Copyright (C) 1991, 1992, 1993 Free Software Foundation, Inc.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#if defined (__GNU_LIBRARY__) || defined (_LIBC)
+#include <stddef.h>
+#include <sys/cdefs.h>
+extern size_t __getpagesize __P ((void));
+#else
+#include "getpagesize.h"
+#define	 __getpagesize()	getpagesize()
+#endif
+
+#ifndef	_MALLOC_INTERNAL
+#define	_MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+static size_t pagesize;
+
+__ptr_t
+valloc (size)
+     size_t size;
+{
+  if (pagesize == 0)
+    pagesize = __getpagesize ();
+
+  return memalign (pagesize, size);
+}
+/* Memory allocator `malloc'.
+   Copyright 1990, 1991, 1992, 1993, 1994 Free Software Foundation
+		  Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+/* How to really get more memory.  */
+__ptr_t (*__morecore) __P ((ptrdiff_t __size)) = __default_morecore;
+
+/* Debugging hook for `malloc'.  */
+__ptr_t (*__malloc_hook) __P ((size_t __size));
+
+/* Pointer to the base of the first block.  */
+char *_heapbase;
+
+/* Block information table.  Allocated with align/__free (not malloc/free).  */
+malloc_info *_heapinfo;
+
+/* Number of info entries.  */
+static size_t heapsize;
+
+/* Search index in the info table.  */
+size_t _heapindex;
+
+/* Limit of valid info table indices.  */
+size_t _heaplimit;
+
+/* Free lists for each fragment size.  */
+struct list _fraghead[BLOCKLOG];
+
+/* Instrumentation.  */
+size_t _chunks_used;
+size_t _bytes_used;
+size_t _chunks_free;
+size_t _bytes_free;
+
+/* Are you experienced?  */
+int __malloc_initialized;
+
+void (*__after_morecore_hook) __P ((void));
+
+/* Aligned allocation.  */
+static __ptr_t align __P ((size_t));
+static __ptr_t
+align (size)
+     size_t size;
+{
+  __ptr_t result;
+  unsigned long int adj;
+
+  result = (*__morecore) (size);
+  adj = (unsigned long int) ((unsigned long int) ((char *) result -
+						  (char *) NULL)) % BLOCKSIZE;
+  if (adj != 0)
+    {
+      adj = BLOCKSIZE - adj;
+      (void) (*__morecore) (adj);
+      result = (char *) result + adj;
+    }
+
+  if (__after_morecore_hook)
+    (*__after_morecore_hook) ();
+
+  return result;
+}
+
+/* Set everything up and remember that we have.  */
+static int initialize __P ((void));
+static int
+initialize ()
+{
+  heapsize = HEAP / BLOCKSIZE;
+  _heapinfo = (malloc_info *) align (heapsize * sizeof (malloc_info));
+  if (_heapinfo == NULL)
+    return 0;
+  memset (_heapinfo, 0, heapsize * sizeof (malloc_info));
+  _heapinfo[0].free.size = 0;
+  _heapinfo[0].free.next = _heapinfo[0].free.prev = 0;
+  _heapindex = 0;
+  _heapbase = (char *) _heapinfo;
+
+  /* Account for the _heapinfo block itself in the statistics.  */
+  _bytes_used = heapsize * sizeof (malloc_info);
+  _chunks_used = 1;
+
+  __malloc_initialized = 1;
+  return 1;
+}
+
+/* Get neatly aligned memory, initializing or
+   growing the heap info table as necessary. */
+static __ptr_t morecore __P ((size_t));
+static __ptr_t
+morecore (size)
+     size_t size;
+{
+  __ptr_t result;
+  malloc_info *newinfo, *oldinfo;
+  size_t newsize;
+
+  result = align (size);
+  if (result == NULL)
+    return NULL;
+
+  /* Check if we need to grow the info table.  */
+  if ((size_t) BLOCK ((char *) result + size) > heapsize)
+    {
+      newsize = heapsize;
+      while ((size_t) BLOCK ((char *) result + size) > newsize)
+	newsize *= 2;
+      newinfo = (malloc_info *) align (newsize * sizeof (malloc_info));
+      if (newinfo == NULL)
+	{
+	  (*__morecore) (-size);
+	  return NULL;
+	}
+      memcpy (newinfo, _heapinfo, heapsize * sizeof (malloc_info));
+      memset (&newinfo[heapsize], 0,
+	      (newsize - heapsize) * sizeof (malloc_info));
+      oldinfo = _heapinfo;
+      newinfo[BLOCK (oldinfo)].busy.type = 0;
+      newinfo[BLOCK (oldinfo)].busy.info.size
+	= BLOCKIFY (heapsize * sizeof (malloc_info));
+      _heapinfo = newinfo;
+      /* Account for the _heapinfo block itself in the statistics.  */
+      _bytes_used += newsize * sizeof (malloc_info);
+      ++_chunks_used;
+      _free_internal (oldinfo);
+      heapsize = newsize;
+    }
+
+  _heaplimit = BLOCK ((char *) result + size);
+  return result;
+}
+
+/* Allocate memory from the heap.  */
+__ptr_t
+malloc (size)
+     size_t size;
+{
+  __ptr_t result;
+  size_t block, blocks, lastblocks, start;
+  register size_t i;
+  struct list *next;
+
+  /* ANSI C allows `malloc (0)' to either return NULL, or to return a
+     valid address you can realloc and free (though not dereference).
+
+     It turns out that some extant code (sunrpc, at least Ultrix's version)
+     expects `malloc (0)' to return non-NULL and breaks otherwise.
+     Be compatible.  */
+
+#if	0
+  if (size == 0)
+    return NULL;
+#endif
+
+  if (__malloc_hook != NULL)
+    return (*__malloc_hook) (size);
+
+  if (!__malloc_initialized)
+    if (!initialize ())
+      return NULL;
+
+  if (size < sizeof (struct list))
+      size = sizeof (struct list);
+
+#ifdef SUNOS_LOCALTIME_BUG
+  if (size < 16)
+    size = 16;
+#endif
+
+  /* Determine the allocation policy based on the request size.  */
+  if (size <= BLOCKSIZE / 2)
+    {
+      /* Small allocation to receive a fragment of a block.
+	 Determine the logarithm to base two of the fragment size. */
+      register size_t log = 1;
+      --size;
+      while ((size /= 2) != 0)
+	++log;
+
+      /* Look in the fragment lists for a
+	 free fragment of the desired size. */
+      next = _fraghead[log].next;
+      if (next != NULL)
+	{
+	  /* There are free fragments of this size.
+	     Pop a fragment out of the fragment list and return it.
+	     Update the block's nfree and first counters. */
+	  result = (__ptr_t) next;
+	  next->prev->next = next->next;
+	  if (next->next != NULL)
+	    next->next->prev = next->prev;
+	  block = BLOCK (result);
+	  if (--_heapinfo[block].busy.info.frag.nfree != 0)
+	    _heapinfo[block].busy.info.frag.first = (unsigned long int)
+	      ((unsigned long int) ((char *) next->next - (char *) NULL)
+	       % BLOCKSIZE) >> log;
+
+	  /* Update the statistics.  */
+	  ++_chunks_used;
+	  _bytes_used += 1 << log;
+	  --_chunks_free;
+	  _bytes_free -= 1 << log;
+	}
+      else
+	{
+	  /* No free fragments of the desired size, so get a new block
+	     and break it into fragments, returning the first.  */
+	  result = malloc (BLOCKSIZE);
+	  if (result == NULL)
+	    return NULL;
+
+	  /* Link all fragments but the first into the free list.  */
+	  for (i = 1; i < (size_t) (BLOCKSIZE >> log); ++i)
+	    {
+	      next = (struct list *) ((char *) result + (i << log));
+	      next->next = _fraghead[log].next;
+	      next->prev = &_fraghead[log];
+	      next->prev->next = next;
+	      if (next->next != NULL)
+		next->next->prev = next;
+	    }
+
+	  /* Initialize the nfree and first counters for this block.  */
+	  block = BLOCK (result);
+	  _heapinfo[block].busy.type = log;
+	  _heapinfo[block].busy.info.frag.nfree = i - 1;
+	  _heapinfo[block].busy.info.frag.first = i - 1;
+
+	  _chunks_free += (BLOCKSIZE >> log) - 1;
+	  _bytes_free += BLOCKSIZE - (1 << log);
+	  _bytes_used -= BLOCKSIZE - (1 << log);
+	}
+    }
+  else
+    {
+      /* Large allocation to receive one or more blocks.
+	 Search the free list in a circle starting at the last place visited.
+	 If we loop completely around without finding a large enough
+	 space we will have to get more memory from the system.  */
+      blocks = BLOCKIFY (size);
+      start = block = _heapindex;
+      while (_heapinfo[block].free.size < blocks)
+	{
+	  block = _heapinfo[block].free.next;
+	  if (block == start)
+	    {
+	      /* Need to get more from the system.  Check to see if
+		 the new core will be contiguous with the final free
+		 block; if so we don't need to get as much.  */
+	      block = _heapinfo[0].free.prev;
+	      lastblocks = _heapinfo[block].free.size;
+	      if (_heaplimit != 0 && block + lastblocks == _heaplimit &&
+		  (*__morecore) (0) == ADDRESS (block + lastblocks) &&
+		  (morecore ((blocks - lastblocks) * BLOCKSIZE)) != NULL)
+		{
+ 		  /* Which block we are extending (the `final free
+ 		     block' referred to above) might have changed, if
+ 		     it got combined with a freed info table.  */
+ 		  block = _heapinfo[0].free.prev;
+  		  _heapinfo[block].free.size += (blocks - lastblocks);
+		  _bytes_free += (blocks - lastblocks) * BLOCKSIZE;
+		  continue;
+		}
+	      result = morecore (blocks * BLOCKSIZE);
+	      if (result == NULL)
+		return NULL;
+	      block = BLOCK (result);
+	      _heapinfo[block].busy.type = 0;
+	      _heapinfo[block].busy.info.size = blocks;
+	      ++_chunks_used;
+	      _bytes_used += blocks * BLOCKSIZE;
+	      return result;
+	    }
+	}
+
+      /* At this point we have found a suitable free list entry.
+	 Figure out how to remove what we need from the list. */
+      result = ADDRESS (block);
+      if (_heapinfo[block].free.size > blocks)
+	{
+	  /* The block we found has a bit left over,
+	     so relink the tail end back into the free list. */
+	  _heapinfo[block + blocks].free.size
+	    = _heapinfo[block].free.size - blocks;
+	  _heapinfo[block + blocks].free.next
+	    = _heapinfo[block].free.next;
+	  _heapinfo[block + blocks].free.prev
+	    = _heapinfo[block].free.prev;
+	  _heapinfo[_heapinfo[block].free.prev].free.next
+	    = _heapinfo[_heapinfo[block].free.next].free.prev
+	    = _heapindex = block + blocks;
+	}
+      else
+	{
+	  /* The block exactly matches our requirements,
+	     so just remove it from the list. */
+	  _heapinfo[_heapinfo[block].free.next].free.prev
+	    = _heapinfo[block].free.prev;
+	  _heapinfo[_heapinfo[block].free.prev].free.next
+	    = _heapindex = _heapinfo[block].free.next;
+	  --_chunks_free;
+	}
+
+      _heapinfo[block].busy.type = 0;
+      _heapinfo[block].busy.info.size = blocks;
+      ++_chunks_used;
+      _bytes_used += blocks * BLOCKSIZE;
+      _bytes_free -= blocks * BLOCKSIZE;
+    }
+
+  return result;
+}
+
+
+#ifndef _LIBC
+
+/* On some ANSI C systems, some libc functions call _malloc, _free
+   and _realloc.  Make them use the GNU functions.  */
+
+__ptr_t
+_malloc (size)
+     size_t size;
+{
+  return malloc (size);
+}
+
+void
+_free (ptr)
+     __ptr_t ptr;
+{
+  free (ptr);
+}
+
+__ptr_t
+_realloc (ptr, size)
+     __ptr_t ptr;
+     size_t size;
+{
+  return realloc (ptr, size);
+}
+
+#endif
+/* Free a block of memory allocated by `malloc'.
+   Copyright 1990, 1991, 1992 Free Software Foundation
+		  Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+/* Debugging hook for free.  */
+void (*__free_hook) __P ((__ptr_t __ptr));
+
+/* List of blocks allocated by memalign.  */
+struct alignlist *_aligned_blocks = NULL;
+
+/* Return memory to the heap.
+   Like `free' but don't call a __free_hook if there is one.  */
+void
+_free_internal (ptr)
+     __ptr_t ptr;
+{
+  int type;
+  size_t block, blocks;
+  register size_t i;
+  struct list *prev, *next;
+
+  block = BLOCK (ptr);
+
+  type = _heapinfo[block].busy.type;
+  switch (type)
+    {
+    case 0:
+      /* Get as many statistics as early as we can.  */
+      --_chunks_used;
+      _bytes_used -= _heapinfo[block].busy.info.size * BLOCKSIZE;
+      _bytes_free += _heapinfo[block].busy.info.size * BLOCKSIZE;
+
+      /* Find the free cluster previous to this one in the free list.
+	 Start searching at the last block referenced; this may benefit
+	 programs with locality of allocation.  */
+      i = _heapindex;
+      if (i > block)
+	while (i > block)
+	  i = _heapinfo[i].free.prev;
+      else
+	{
+	  do
+	    i = _heapinfo[i].free.next;
+	  while (i > 0 && i < block);
+	  i = _heapinfo[i].free.prev;
+	}
+
+      /* Determine how to link this block into the free list.  */
+      if (block == i + _heapinfo[i].free.size)
+	{
+	  /* Coalesce this block with its predecessor.  */
+	  _heapinfo[i].free.size += _heapinfo[block].busy.info.size;
+	  block = i;
+	}
+      else
+	{
+	  /* Really link this block back into the free list.  */
+	  _heapinfo[block].free.size = _heapinfo[block].busy.info.size;
+	  _heapinfo[block].free.next = _heapinfo[i].free.next;
+	  _heapinfo[block].free.prev = i;
+	  _heapinfo[i].free.next = block;
+	  _heapinfo[_heapinfo[block].free.next].free.prev = block;
+	  ++_chunks_free;
+	}
+
+      /* Now that the block is linked in, see if we can coalesce it
+	 with its successor (by deleting its successor from the list
+	 and adding in its size).  */
+      if (block + _heapinfo[block].free.size == _heapinfo[block].free.next)
+	{
+	  _heapinfo[block].free.size
+	    += _heapinfo[_heapinfo[block].free.next].free.size;
+	  _heapinfo[block].free.next
+	    = _heapinfo[_heapinfo[block].free.next].free.next;
+	  _heapinfo[_heapinfo[block].free.next].free.prev = block;
+	  --_chunks_free;
+	}
+
+      /* Now see if we can return stuff to the system.  */
+      blocks = _heapinfo[block].free.size;
+      if (blocks >= FINAL_FREE_BLOCKS && block + blocks == _heaplimit
+	  && (*__morecore) (0) == ADDRESS (block + blocks))
+	{
+	  register size_t bytes = blocks * BLOCKSIZE;
+	  _heaplimit -= blocks;
+	  (*__morecore) (-bytes);
+	  _heapinfo[_heapinfo[block].free.prev].free.next
+	    = _heapinfo[block].free.next;
+	  _heapinfo[_heapinfo[block].free.next].free.prev
+	    = _heapinfo[block].free.prev;
+	  block = _heapinfo[block].free.prev;
+	  --_chunks_free;
+	  _bytes_free -= bytes;
+	}
+
+      /* Set the next search to begin at this block.  */
+      _heapindex = block;
+      break;
+
+    default:
+      /* Do some of the statistics.  */
+      --_chunks_used;
+      _bytes_used -= 1 << type;
+      ++_chunks_free;
+      _bytes_free += 1 << type;
+
+      /* Get the address of the first free fragment in this block.  */
+      prev = (struct list *) ((char *) ADDRESS (block) +
+			   (_heapinfo[block].busy.info.frag.first << type));
+
+      if (_heapinfo[block].busy.info.frag.nfree == (BLOCKSIZE >> type) - 1)
+	{
+	  /* If all fragments of this block are free, remove them
+	     from the fragment list and free the whole block.  */
+	  next = prev;
+	  for (i = 1; i < (size_t) (BLOCKSIZE >> type); ++i)
+	    next = next->next;
+	  prev->prev->next = next;
+	  if (next != NULL)
+	    next->prev = prev->prev;
+	  _heapinfo[block].busy.type = 0;
+	  _heapinfo[block].busy.info.size = 1;
+
+	  /* Keep the statistics accurate.  */
+	  ++_chunks_used;
+	  _bytes_used += BLOCKSIZE;
+	  _chunks_free -= BLOCKSIZE >> type;
+	  _bytes_free -= BLOCKSIZE;
+
+	  free (ADDRESS (block));
+	}
+      else if (_heapinfo[block].busy.info.frag.nfree != 0)
+	{
+	  /* If some fragments of this block are free, link this
+	     fragment into the fragment list after the first free
+	     fragment of this block. */
+	  next = (struct list *) ptr;
+	  next->next = prev->next;
+	  next->prev = prev;
+	  prev->next = next;
+	  if (next->next != NULL)
+	    next->next->prev = next;
+	  ++_heapinfo[block].busy.info.frag.nfree;
+	}
+      else
+	{
+	  /* No fragments of this block are free, so link this
+	     fragment into the fragment list and announce that
+	     it is the first free fragment of this block. */
+	  prev = (struct list *) ptr;
+	  _heapinfo[block].busy.info.frag.nfree = 1;
+	  _heapinfo[block].busy.info.frag.first = (unsigned long int)
+	    ((unsigned long int) ((char *) ptr - (char *) NULL)
+	     % BLOCKSIZE >> type);
+	  prev->next = _fraghead[type].next;
+	  prev->prev = &_fraghead[type];
+	  prev->prev->next = prev;
+	  if (prev->next != NULL)
+	    prev->next->prev = prev;
+	}
+      break;
+    }
+}
+
+/* Return memory to the heap.  */
+void
+free (ptr)
+     __ptr_t ptr;
+{
+  register struct alignlist *l;
+
+  if (ptr == NULL)
+    return;
+
+  for (l = _aligned_blocks; l != NULL; l = l->next)
+    if (l->aligned == ptr)
+      {
+	l->aligned = NULL;	/* Mark the slot in the list as free.  */
+	ptr = l->exact;
+	break;
+      }
+
+  if (__free_hook != NULL)
+    (*__free_hook) (ptr);
+  else
+    _free_internal (ptr);
+}
+/* Copyright (C) 1991, 1993, 1994 Free Software Foundation, Inc.
+This file is part of the GNU C Library.
+
+The GNU C Library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+The GNU C Library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with the GNU C Library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+#ifdef _LIBC
+
+#include <ansidecl.h>
+#include <gnu-stabs.h>
+
+#undef	cfree
+
+function_alias(cfree, free, void, (ptr),
+	       DEFUN(cfree, (ptr), PTR ptr))
+
+#else
+
+void
+cfree (ptr)
+     __ptr_t ptr;
+{
+  free (ptr);
+}
+
+#endif
+/* Change the size of a block allocated by `malloc'.
+   Copyright 1990, 1991, 1992, 1993, 1994 Free Software Foundation, Inc.
+		     Written May 1989 by Mike Haertel.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+#if  (defined (MEMMOVE_MISSING) || \
+      !defined(_LIBC) && !defined(STDC_HEADERS) && !defined(USG))
+
+/* Snarfed directly from Emacs src/dispnew.c:
+   XXX Should use system bcopy if it handles overlap.  */
+#ifndef emacs
+
+/* Like bcopy except never gets confused by overlap.  */
+
+static void
+safe_bcopy (from, to, size)
+     char *from, *to;
+     int size;
+{
+  if (size <= 0 || from == to)
+    return;
+
+  /* If the source and destination don't overlap, then bcopy can
+     handle it.  If they do overlap, but the destination is lower in
+     memory than the source, we'll assume bcopy can handle that.  */
+  if (to < from || from + size <= to)
+    bcopy (from, to, size);
+
+  /* Otherwise, we'll copy from the end.  */
+  else
+    {
+      register char *endf = from + size;
+      register char *endt = to + size;
+
+      /* If TO - FROM is large, then we should break the copy into
+	 nonoverlapping chunks of TO - FROM bytes each.  However, if
+	 TO - FROM is small, then the bcopy function call overhead
+	 makes this not worth it.  The crossover point could be about
+	 anywhere.  Since I don't think the obvious copy loop is too
+	 bad, I'm trying to err in its favor.  */
+      if (to - from < 64)
+	{
+	  do
+	    *--endt = *--endf;
+	  while (endf != from);
+	}
+      else
+	{
+	  for (;;)
+	    {
+	      endt -= (to - from);
+	      endf -= (to - from);
+
+	      if (endt < to)
+		break;
+
+	      bcopy (endf, endt, to - from);
+	    }
+
+	  /* If SIZE wasn't a multiple of TO - FROM, there will be a
+	     little left over.  The amount left over is
+	     (endt + (to - from)) - to, which is endt - from.  */
+	  bcopy (from, to, endt - from);
+	}
+    }
+}     
+#endif	/* Not emacs.  */
+
+#define memmove(to, from, size) safe_bcopy ((from), (to), (size))
+
+#endif
+
+
+#define min(A, B) ((A) < (B) ? (A) : (B))
+
+/* Debugging hook for realloc.  */
+__ptr_t (*__realloc_hook) __P ((__ptr_t __ptr, size_t __size));
+
+/* Resize the given region to the new size, returning a pointer
+   to the (possibly moved) region.  This is optimized for speed;
+   some benchmarks seem to indicate that greater compactness is
+   achieved by unconditionally allocating and copying to a
+   new region.  This module has incestuous knowledge of the
+   internals of both free and malloc. */
+__ptr_t
+realloc (ptr, size)
+     __ptr_t ptr;
+     size_t size;
+{
+  __ptr_t result;
+  int type;
+  size_t block, blocks, oldlimit;
+
+  if (size == 0)
+    {
+      free (ptr);
+      return malloc (0);
+    }
+  else if (ptr == NULL)
+    return malloc (size);
+
+  if (__realloc_hook != NULL)
+    return (*__realloc_hook) (ptr, size);
+
+  block = BLOCK (ptr);
+
+  type = _heapinfo[block].busy.type;
+  switch (type)
+    {
+    case 0:
+      /* Maybe reallocate a large block to a small fragment.  */
+      if (size <= BLOCKSIZE / 2)
+	{
+	  result = malloc (size);
+	  if (result != NULL)
+	    {
+	      memcpy (result, ptr, size);
+	      _free_internal (ptr);
+	      return result;
+	    }
+	}
+
+      /* The new size is a large allocation as well;
+	 see if we can hold it in place. */
+      blocks = BLOCKIFY (size);
+      if (blocks < _heapinfo[block].busy.info.size)
+	{
+	  /* The new size is smaller; return
+	     excess memory to the free list. */
+	  _heapinfo[block + blocks].busy.type = 0;
+	  _heapinfo[block + blocks].busy.info.size
+	    = _heapinfo[block].busy.info.size - blocks;
+	  _heapinfo[block].busy.info.size = blocks;
+	  /* We have just created a new chunk by splitting a chunk in two.
+	     Now we will free this chunk; increment the statistics counter
+	     so it doesn't become wrong when _free_internal decrements it.  */
+	  ++_chunks_used;
+	  _free_internal (ADDRESS (block + blocks));
+	  result = ptr;
+	}
+      else if (blocks == _heapinfo[block].busy.info.size)
+	/* No size change necessary.  */
+	result = ptr;
+      else
+	{
+	  /* Won't fit, so allocate a new region that will.
+	     Free the old region first in case there is sufficient
+	     adjacent free space to grow without moving. */
+	  blocks = _heapinfo[block].busy.info.size;
+	  /* Prevent free from actually returning memory to the system.  */
+	  oldlimit = _heaplimit;
+	  _heaplimit = 0;
+	  _free_internal (ptr);
+	  _heaplimit = oldlimit;
+	  result = malloc (size);
+	  if (result == NULL)
+	    {
+	      /* Now we're really in trouble.  We have to unfree
+		 the thing we just freed.  Unfortunately it might
+		 have been coalesced with its neighbors.  */
+	      if (_heapindex == block)
+	        (void) malloc (blocks * BLOCKSIZE);
+	      else
+		{
+		  __ptr_t previous = malloc ((block - _heapindex) * BLOCKSIZE);
+		  (void) malloc (blocks * BLOCKSIZE);
+		  _free_internal (previous);
+		}
+	      return NULL;
+	    }
+	  if (ptr != result)
+	    memmove (result, ptr, blocks * BLOCKSIZE);
+	}
+      break;
+
+    default:
+      /* Old size is a fragment; type is logarithm
+	 to base two of the fragment size.  */
+      if (size > (size_t) (1 << (type - 1)) && size <= (size_t) (1 << type))
+	/* The new size is the same kind of fragment.  */
+	result = ptr;
+      else
+	{
+	  /* The new size is different; allocate a new space,
+	     and copy the lesser of the new size and the old. */
+	  result = malloc (size);
+	  if (result == NULL)
+	    return NULL;
+	  memcpy (result, ptr, min (size, (size_t) 1 << type));
+	  free (ptr);
+	}
+      break;
+    }
+
+  return result;
+}
+/* Copyright (C) 1991, 1992 Free Software Foundation, Inc.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.
+
+   The author may be reached (Email) at the address mike@ai.mit.edu,
+   or (US mail) as Mike Haertel c/o Free Software Foundation.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define	_MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+/* Allocate an array of NMEMB elements each SIZE bytes long.
+   The entire array is initialized to zeros.  */
+__ptr_t
+calloc (nmemb, size)
+     register size_t nmemb;
+     register size_t size;
+{
+  register __ptr_t result = malloc (nmemb * size);
+
+  if (result != NULL)
+    (void) memset (result, 0, nmemb * size);
+
+  return result;
+}
+/* Copyright (C) 1991, 1992, 1993 Free Software Foundation, Inc.
+This file is part of the GNU C Library.
+
+The GNU C Library is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2, or (at your option)
+any later version.
+
+The GNU C Library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with the GNU C Library; see the file COPYING.  If not, write to
+the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define	_MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+#ifndef	__GNU_LIBRARY__
+#define	__sbrk	sbrk
+#endif
+
+#ifdef __GNU_LIBRARY__
+/* It is best not to declare this and cast its result on foreign operating
+   systems with potentially hostile include files.  */
+extern __ptr_t __sbrk __P ((int increment));
+#endif
+
+#ifndef NULL
+#define NULL 0
+#endif
+
+/* Allocate INCREMENT more bytes of data space,
+   and return the start of data space, or NULL on errors.
+   If INCREMENT is negative, shrink data space.  */
+__ptr_t
+__default_morecore (increment)
+     ptrdiff_t increment;
+{
+  __ptr_t result = (__ptr_t) __sbrk ((int) increment);
+  if (result == (__ptr_t) -1)
+    return NULL;
+  return result;
+}
+/* Copyright (C) 1991, 1992, 1993 Free Software Foundation, Inc.
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Library General Public License as
+published by the Free Software Foundation; either version 2 of the
+License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+Library General Public License for more details.
+
+You should have received a copy of the GNU Library General Public
+License along with this library; see the file COPYING.LIB.  If
+not, write to the Free Software Foundation, Inc., 675 Mass Ave,
+Cambridge, MA 02139, USA.  */
+
+#ifndef	_MALLOC_INTERNAL
+#define _MALLOC_INTERNAL
+#include <malloc.h>
+#endif
+
+__ptr_t
+memalign (alignment, size)
+     size_t alignment;
+     size_t size;
+{
+  __ptr_t result;
+  unsigned long int adj;
+
+  size = ((size + alignment - 1) / alignment) * alignment;
+
+  result = malloc (size);
+  if (result == NULL)
+    return NULL;
+  adj = (unsigned long int) ((unsigned long int) ((char *) result -
+						  (char *) NULL)) % alignment;
+  if (adj != 0)
+    {
+      struct alignlist *l;
+      for (l = _aligned_blocks; l != NULL; l = l->next)
+	if (l->aligned == NULL)
+	  /* This slot is free.  Use it.  */
+	  break;
+      if (l == NULL)
+	{
+	  l = (struct alignlist *) malloc (sizeof (struct alignlist));
+	  if (l == NULL)
+	    {
+	      free (result);
+	      return NULL;
+	    }
+	  l->next = _aligned_blocks;
+	  _aligned_blocks = l;
+	}
+      l->exact = result;
+      result = l->aligned = (char *) result + alignment - adj;
+    }
+
+  return result;
+}
Index: /trunk/mgpp/lib/huffman.cpp
===================================================================
--- /trunk/mgpp/lib/huffman.cpp	(revision 3365)
+++ /trunk/mgpp/lib/huffman.cpp	(revision 3365)
@@ -0,0 +1,385 @@
+/**************************************************************************
+ *
+ * huffman.c -- Huffman coding functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "huffman.h"
+#include "messages.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+/* ->data on success,  
+   NULL on error 
+ */
+huff_data *
+Generate_Huffman_Data (int num, long *freqs, huff_data * data,
+		       u_long * mem)
+{
+  int HNum, i, count;
+  unsigned long *heap;
+  huff_data *hd = data;
+  if (!hd)
+    {
+      if (!(hd = new huff_data))
+	goto error0;
+    }
+  if (!(heap = new unsigned long[num * 2]))
+    goto error1;
+  if (!(hd->clens = new char[num]))
+      goto error2;
+  hd->num_codes = num;
+  if (mem)
+    {
+      if (hd != data)
+	*mem += sizeof (huff_data);
+      *mem += num * sizeof (char);
+    }
+  memcpy (&heap[num], freqs, num * sizeof (*heap));
+  memset (heap, '\0', num * sizeof (*heap));
+
+  /* Initialise the pointers to the leaves */
+  for (count = i = 0; i < num; i++)
+    if (heap[num + i])
+      heap[count++] = num + i;
+
+  /* Reorganise the pointers so that it is a heap */
+  HNum = count;
+  for (i = HNum / 2; i > 0; i--)
+    {
+      register int curr, child;
+      curr = i;
+      child = curr * 2;
+      while (child <= HNum)
+	{
+	  if (child < HNum && heap[heap[child]] < heap[heap[child - 1]])
+	    child++;
+	  if (heap[heap[curr - 1]] > heap[heap[child - 1]])
+	    {
+	      register u_long temp;
+	      temp = heap[child - 1];
+	      heap[child - 1] = heap[curr - 1];
+	      heap[curr - 1] = temp;
+	      curr = child;
+	      child = 2 * curr;
+	    }
+	  else
+	    break;
+	}
+    }
+
+
+  /* Form the huffman tree */
+  while (HNum > 1)
+    {
+      int pos[2];
+
+      for (i = 0; i < 2; i++)
+	{
+	  register int curr, child;
+	  pos[i] = heap[0];
+	  heap[0] = heap[--HNum];
+	  curr = 1;
+	  child = 2;
+	  while (child <= HNum)
+	    {
+	      if (child < HNum &&
+		  heap[heap[child]] < heap[heap[child - 1]])
+		child++;
+	      if (heap[heap[curr - 1]] > heap[heap[child - 1]])
+		{
+		  register int temp;
+		  temp = heap[child - 1];
+		  heap[child - 1] = heap[curr - 1];
+		  heap[curr - 1] = temp;
+		  curr = child;
+		  child = 2 * curr;
+		}
+	      else
+		break;
+	    }
+	}
+
+      heap[HNum + 1] = heap[pos[0]] + heap[pos[1]];
+      heap[pos[0]] = heap[pos[1]] = HNum + 1;
+
+      heap[HNum] = HNum + 1;
+
+      {
+	register int parent, curr;
+	HNum++;
+	curr = HNum;
+	parent = curr >> 1;
+	while (parent && heap[heap[parent - 1]] > heap[heap[curr - 1]])
+	  {
+	    register int temp;
+	    temp = heap[parent - 1];
+	    heap[parent - 1] = heap[curr - 1];
+	    heap[curr - 1] = temp;
+	    curr = parent;
+	    parent = curr >> 1;
+	  }
+      }
+    }
+
+
+  /* Calculate the code lens */
+  heap[0] = -1UL;
+  heap[1] = 0;
+  for (i = 2; i < num * 2; i++)
+    heap[i] = heap[heap[i]] + 1;
+
+  /* zero the lencount's array */
+  memset (hd->lencount, '\0', sizeof (hd->lencount));
+  hd->maxcodelen = 0;
+  hd->mincodelen = 100;
+
+  /* Set the code length of each leaf in the huffman tree */
+  for (i = 0; i < num; i++)
+    {
+      register u_long codelen = heap[i + num];
+      hd->clens[i] = (char) codelen;
+      if (!codelen)
+	continue;
+      if (codelen > hd->maxcodelen)
+	hd->maxcodelen = codelen;
+      if (codelen < hd->mincodelen)
+	hd->mincodelen = codelen;
+      hd->lencount[codelen]++;
+    }
+
+
+  if (hd->maxcodelen == 0 && hd->mincodelen == 100)
+    {
+      hd->num_codes = 0;
+    }
+  else
+    {
+      /* Calculate the current codes for each different code length */
+      hd->min_code[hd->maxcodelen] = 0;
+      for (i = hd->maxcodelen - 1; i>=0; i--)
+	hd->min_code[i] = (hd->min_code[i + 1] + hd->lencount[i + 1]) >> 1;
+    }
+  delete heap;
+  return (hd);
+
+error2:
+  delete heap;
+error1:
+  if (!data)
+    delete hd;
+error0:
+  return (NULL);
+
+}
+
+unsigned long *
+Generate_Huffman_Codes (huff_data * data, u_long * mem)
+{
+  int i;
+  unsigned long *codes;
+  unsigned long mc[MAX_HUFFCODE_LEN + 1];
+
+  if (!data)
+    return (NULL);
+  if (!(codes = new unsigned long[data->num_codes]))
+    return (NULL);
+  if (mem)
+    *mem += data->num_codes * sizeof (*codes);
+  memcpy (mc, data->min_code, sizeof (mc));
+  for (i = 0; i < data->num_codes; i++)
+    if (data->clens[i])
+      codes[i] = mc[(int) (data->clens[i])]++;
+
+  return (codes);
+}
+
+
+unsigned long **
+Generate_Huffman_Vals (huff_data * data, u_long * mem)
+{
+  int i;
+  unsigned long *fcode[MAX_HUFFCODE_LEN + 1];
+  unsigned long **values;
+  unsigned long *vals;
+
+  if (!data)
+    return (NULL);
+  if (!(vals = new unsigned long[data->num_codes]))
+    return (NULL);
+  if (!(values = new unsigned long *[MAX_HUFFCODE_LEN + 1]))
+    {
+      delete vals;
+      return (NULL);
+    }
+
+  memset (values, '\0', (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *));
+
+  if (mem)
+    *mem += data->num_codes * sizeof (*vals) +
+      (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *);
+
+  fcode[0] = values[0] = &vals[0];
+  for (i = 1; i <= data->maxcodelen; i++)
+    fcode[i] = values[i] = &vals[(values[i - 1] - vals) + data->lencount[i - 1]];
+
+  for (i = 0; i < data->num_codes; i++)
+    if (data->clens[i])
+      *fcode[(int) (data->clens[i])]++ = i;
+  return (values);
+}
+
+
+
+double 
+Calculate_Huffman_Size (int num, long *freqs, long *counts)
+{
+  double size = 0;
+  huff_data hd;
+  int i;
+  if (!Generate_Huffman_Data (num, freqs, &hd, NULL))
+    return -1;
+  for (i = 0; i < num; i++)
+    size += counts[i] * hd.clens[i];
+  delete hd.clens;
+  return size;
+}
+
+
+
+int 
+Write_Huffman_Data (FILE * f, huff_data * hd)
+{
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONSI(hd->num_codes);
+  HTONSI(hd->mincodelen);
+  HTONSI(hd->maxcodelen);
+
+  if (fwrite (&hd->num_codes, sizeof (hd->num_codes), 1, f) != 1)
+    return -1;
+
+  if (fwrite (&hd->mincodelen, sizeof (hd->mincodelen), 1, f) != 1)
+    return -1;
+
+  if (fwrite (&hd->maxcodelen, sizeof (hd->maxcodelen), 1, f) != 1)
+    return -1;
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(hd->num_codes);
+  NTOHSI(hd->mincodelen);
+  NTOHSI(hd->maxcodelen);
+
+  if (hd->num_codes)
+    {
+      /* [RPAP - Jan 97: Endian Ordering] */
+      int i;
+      for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+	HTONSI(hd->lencount[i]);
+      for (i = 0; i < hd->maxcodelen + 1; i++)
+	HTONUL(hd->min_code[i]);
+
+      if (fwrite (hd->lencount + hd->mincodelen, sizeof (*hd->lencount),
+		  hd->maxcodelen - hd->mincodelen + 1, f) !=
+	  hd->maxcodelen - hd->mincodelen + 1)
+	return -1;
+
+      if (fwrite (hd->min_code, sizeof (*hd->min_code), hd->maxcodelen + 1, f) !=
+	  hd->maxcodelen + 1)
+	return -1;
+
+      if (fwrite (hd->clens, sizeof (*hd->clens), hd->num_codes, f) != hd->num_codes)
+	return -1;
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+	NTOHSI(hd->lencount[i]);
+      for (i = 0; i < hd->maxcodelen + 1; i++)
+	NTOHUL(hd->min_code[i]);
+    }
+  return 0;
+}
+
+
+
+int Read_Huffman_Data (FILE * f, huff_data * hd, u_long * mem, u_long * disk) {
+  if (fread (&hd->num_codes, sizeof (hd->num_codes), 1, f) != 1)
+    return -1;
+
+  if (fread (&hd->mincodelen, sizeof (hd->mincodelen), 1, f) != 1)
+    return -1;
+
+  if (fread (&hd->maxcodelen, sizeof (hd->maxcodelen), 1, f) != 1)
+    return -1;
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(hd->num_codes);
+  NTOHSI(hd->mincodelen);
+  NTOHSI(hd->maxcodelen);
+
+  if (disk)
+    (*disk) += sizeof (hd->num_codes) + sizeof (hd->mincodelen) +
+      sizeof (hd->maxcodelen);
+
+  hd->clens = NULL;
+
+  if (hd->num_codes)
+    {
+      int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+      memset (hd->lencount, '\0', sizeof (hd->lencount));
+      if (fread (hd->lencount + hd->mincodelen, sizeof (*hd->lencount),
+	      hd->maxcodelen - hd->mincodelen + 1, f) !=
+	  hd->maxcodelen - hd->mincodelen + 1)
+	return -1;
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+	NTOHSI(hd->lencount[i]);
+
+      if (disk)
+	(*disk) += sizeof (*hd->lencount) * (hd->maxcodelen - hd->mincodelen + 1);
+
+      memset (hd->min_code, '\0', sizeof (hd->min_code));
+
+      if (fread (hd->min_code, sizeof (*hd->min_code), hd->maxcodelen + 1, f) !=
+	  hd->maxcodelen + 1)
+	return -1;
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (i = 0; i < hd->maxcodelen + 1; i++)
+	NTOHUL(hd->min_code[i]);
+
+      if (disk)
+	(*disk) += sizeof (*hd->min_code) * (hd->maxcodelen + 1);
+
+      if (!(hd->clens = new char[hd->num_codes]))
+	return -1;
+
+
+      if (fread (hd->clens, sizeof (*hd->clens), hd->num_codes, f) != hd->num_codes)
+	return -1;
+
+      if (disk)
+	(*disk) += sizeof (*hd->clens) * hd->num_codes;
+    }
+
+  if (mem)
+    *mem += sizeof (*hd->clens) * hd->num_codes;
+  return 0;
+}
Index: /trunk/mgpp/lib/huffman.h
===================================================================
--- /trunk/mgpp/lib/huffman.h	(revision 3365)
+++ /trunk/mgpp/lib/huffman.h	(revision 3365)
@@ -0,0 +1,57 @@
+/**************************************************************************
+ *
+ * huffman.h -- Huffman coding functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_HUFFMAN
+#define H_HUFFMAN
+
+#include "sysfuncs.h"
+#include "bitio_m_abstract.h"
+
+#define MAX_HUFFCODE_LEN 31
+
+struct huff_data {
+  int num_codes;
+  int mincodelen;
+  int maxcodelen;
+  int lencount[MAX_HUFFCODE_LEN + 1];
+  unsigned long min_code[MAX_HUFFCODE_LEN + 1];
+  char *clens;
+};
+
+
+huff_data *Generate_Huffman_Data (int num, long *freqs, huff_data * data,
+				  u_long * mem);
+
+unsigned long *Generate_Huffman_Codes (huff_data * data, u_long * mem);
+
+unsigned long **Generate_Huffman_Vals (huff_data * data, u_long * mem);
+
+int Write_Huffman_Data (FILE * f, huff_data * hd);
+
+int Read_Huffman_Data (FILE * f, huff_data * hd, u_long * mem, u_long * disk);
+
+/* Calculate the number of bits required to code the data with the
+   specified frequencies. Normally freqs and counts should point to
+   the same array. */
+double Calculate_Huffman_Size (int num, long *freqs, long *counts);
+
+
+#endif
Index: /trunk/mgpp/lib/local_strings.cpp
===================================================================
--- /trunk/mgpp/lib/local_strings.cpp	(revision 3365)
+++ /trunk/mgpp/lib/local_strings.cpp	(revision 3365)
@@ -0,0 +1,411 @@
+/**************************************************************************
+ *
+ * local_strings -- provides some extra string routines
+ * Copyright (C) 1994  Authors
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "local_strings.h"
+#include "memlib.h"
+
+/* =========================================================================
+ * Function: arg_atoi
+ * Description: 
+ *      Converts an ascii string to an integer
+ *      but allows a scale string immediately following.
+ *      i.e. "10Mb" or "20Kb" or "40Megabytes"
+ *      There must be no gap between the last digit and the scale.
+ *      Only the fist letter is significant.
+ *      K = Kilo
+ *      M = Mega
+ *      G = Giga
+ * Input: 
+ *      string of form: xxxxxxK or xxxxxxxM or xxxxxxxG
+ *      where x is a digit
+ * Output: 
+ *      a scaled integer
+ * ========================================================================= */
+
+#define KILO 	1024
+#define MEGA	1024*1024
+#define GIGA	1024*1024*1024
+
+int 
+arg_atoi (char *str)
+{
+  char *str_ptr = str;
+  char ch = '\0';
+  long scale = 1;
+
+  /* find any scale letter immediately after digits */
+  ch = *str_ptr;
+  while (!isalpha (ch) && (ch != '\0'))
+    ch = *str_ptr++;
+
+  /* replace scale letter by end-of-string character */
+  *str_ptr = '\0';
+
+  /* set scale appropriately */
+  switch (ch)
+    {
+    case 'K':
+      scale = KILO;
+      break;
+    case 'M':
+      scale = MEGA;
+      break;
+    case 'G':
+      scale = GIGA;
+      break;
+    default:
+      scale = 1;
+      break;
+    }
+
+  return (atoi (str) * scale);
+
+}
+
+/* ===============================================================================
+ * Function: compare
+ * Description: Compare two strings. 
+ * Input: Two binary strings, first byte of string indicates length. 
+ * Output: Like standard strcmp.
+ *      = 0 if s1 = s2
+ *      < 0 if s1 < s2
+ *      > 0 if s1 > s2
+ * =============================================================================== */
+int 
+compare (u_char * s1, u_char * s2)
+{
+  int l1 = *s1, l2 = *s2;
+  int l = (l1 < l2) ? l1 : l2;
+
+  while (l--)
+    if (*++s1 != *++s2)
+      return (*s1 - *s2);
+  return (l1 - l2);
+
+}
+
+/*
+ * This array is designed for mapping upper and lower case letter
+ * together for a case independent comparison.  The mappings are
+ * based upon ascii character sequences.
+ */
+static unsigned char casecharmap[] = {
+	'\000', '\001', '\002', '\003', '\004', '\005', '\006', '\007',
+	'\010', '\011', '\012', '\013', '\014', '\015', '\016', '\017',
+	'\020', '\021', '\022', '\023', '\024', '\025', '\026', '\027',
+	'\030', '\031', '\032', '\033', '\034', '\035', '\036', '\037',
+	'\040', '\041', '\042', '\043', '\044', '\045', '\046', '\047',
+	'\050', '\051', '\052', '\053', '\054', '\055', '\056', '\057',
+	'\060', '\061', '\062', '\063', '\064', '\065', '\066', '\067',
+	'\070', '\071', '\072', '\073', '\074', '\075', '\076', '\077',
+	'\100', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\133', '\134', '\135', '\136', '\137',
+	'\140', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\173', '\174', '\175', '\176', '\177',
+	'\200', '\201', '\202', '\203', '\204', '\205', '\206', '\207',
+	'\210', '\211', '\212', '\213', '\214', '\215', '\216', '\217',
+	'\220', '\221', '\222', '\223', '\224', '\225', '\226', '\227',
+	'\230', '\231', '\232', '\233', '\234', '\235', '\236', '\237',
+	'\240', '\241', '\242', '\243', '\244', '\245', '\246', '\247',
+	'\250', '\251', '\252', '\253', '\254', '\255', '\256', '\257',
+	'\260', '\261', '\262', '\263', '\264', '\265', '\266', '\267',
+	'\270', '\271', '\272', '\273', '\274', '\275', '\276', '\277',
+	'\300', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\333', '\334', '\335', '\336', '\337',
+	'\340', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\373', '\374', '\375', '\376', '\377',
+};
+
+/* ===============================================================================
+ * Function: casecompare   [RPAP - Jan 97: Stem Index Change]
+ * Description: Compare two strings in dictionary order
+ * Input: Two binary strings, first byte of string indicates length. 
+ * Output:
+ *      = 0 if s1 = s2
+ *      < 0 if s1 < s2
+ *      > 0 if s1 > s2
+ * =============================================================================== */
+int 
+casecompare (u_char * s1, u_char * s2)
+{
+  int l1 = *s1, l2 = *s2;
+  int l = (l1 < l2) ? l1 : l2;
+  int pos = 0;
+  register int diff = 0;
+
+  while (l--)
+    if ((diff = casecharmap[*++s1] - casecharmap[*++s2]) != 0)
+      return (diff);
+    else if ((diff = *s1 - *s2) != 0)
+      if (!pos)
+	pos = diff;
+
+  return ((l1 - l2) ? (l1 - l2) : (pos));
+
+}
+
+/* =========================================================================
+ * Function: copy_string
+ * Description: 
+ *      Copies a string with the len in 1st byte
+ * Input: 
+ * Output: 
+ * ========================================================================= */
+
+u_char *
+copy_string (u_char * src_str)
+{
+  u_char *copy = new u_char(src_str[0] + 1);
+
+  if (!copy)
+    return NULL;
+
+  memcpy (copy, src_str, src_str[0] + 1);
+  return copy;
+
+}
+
+
+/* ===============================================================================
+ * Function: prefixlen
+ * Description: Returns length of common prefix string between s1 and s2.
+ * Input: two binary strings
+ * Output: prefix length
+ * =============================================================================== */
+int 
+prefixlen (u_char * s1, u_char * s2)
+{
+  int l = (*s1 < *s2) ? *s1 : *s2;
+  int i = 0;
+
+  while (i < l && *++s1 == *++s2)
+    i++;
+  return (i);
+}
+
+
+/* ===============================================================================
+ * Function: char2str
+ * Description: 
+ *      Ensures that non-printable characters are converted to a string
+ *      which is printable i.e. it indentifies the character.
+ *      e.g. carriage-return = "\n".
+ * Input: character (may be non-printable)
+ * Output: pointer to static string
+ * =============================================================================== */
+
+char *
+char2str (u_char c)
+{
+  static char buf[10];
+  switch (c)
+    {
+    case '\n':
+      sprintf (buf, "\\n");
+      break;
+    case '\b':
+      sprintf (buf, "\\b");
+      break;
+    case '\f':
+      sprintf (buf, "\\f");
+      break;
+    case '\t':
+      sprintf (buf, "\\t");
+      break;
+    default:
+	if (c >= ' ' && c <= 126)
+	switch (c)
+	  {
+	  case '\\':
+	    sprintf (buf, "\\\\");
+	    break;
+	  case '\"':
+	    sprintf (buf, "\\\"");
+	    break;
+	  case '\'':
+	    sprintf (buf, "\\\'");
+	    break;
+	  default:
+	    buf[0] = c;
+	    buf[1] = '\0';
+	    break;
+	  }
+      else
+	sprintf (buf, "\\x%02x", c);
+      break;
+    }
+  return buf;
+}
+
+/* ===============================================================================
+ * Function: word2str
+ * Description: 
+ *      Converts a string with possible non-printable characters into a string
+ *      consisting entirely of printable characters.
+ * Input: string, length stored in first byte
+ * Output: pointer to static string
+ * =============================================================================== */
+
+char *
+word2str (u_char * s)
+{
+  static char buf[1024];
+  char *p = buf;
+  int i, len = (int) *s++;
+
+  memset (buf, '\0', sizeof(buf));
+  for (i = 0; i < len; i++)
+    {
+      strcpy (p, char2str (s[i]));
+      p = strchr (p, '\0');
+    }
+  return buf;
+}
+
+
+/* =========================================================================
+ * Function: de_escape_string
+ * Description: 
+ *      Turn a string containing \ escapes into real characters
+ *      Note this destroys the string s in the process
+ * Input: string to de-escape 
+ * Output: the de-escaped string
+ * ========================================================================= */
+char *
+de_escape_string (char *s)
+{
+  char *org = s;
+  char *d = s;
+  while (*s)
+    {
+      if (*s == '\\')
+	{
+	  s++;
+	  switch (*s++)
+	    {
+	    case '\0':
+	      *d++ = '\\';
+	      s--;
+	      break;
+	    case '\\':
+	      *d++ = '\\';
+	      break;
+	    case 'b':
+	      *d++ = '\b';
+	      break;
+	    case 'f':
+	      *d++ = '\f';
+	      break;
+	    case 'n':
+	      *d++ = '\n';
+	      break;
+	    case 'r':
+	      *d++ = '\r';
+	      break;
+	    case 't':
+	      *d++ = '\r';
+	      break;
+	    case '\"':
+	      *d++ = '\"';
+	      break;
+	    case '\'':
+	      *d++ = '\'';
+	      break;
+	    case 'x':
+	      {
+		int num = 0;
+		if (isxdigit (*s))
+		  {
+		    char ch = toupper (*s++);
+		    num = ch <= '9' ? ch - '0' : ch - 'A' + 10;
+		  }
+		if (isxdigit (*s))
+		  {
+		    char ch = toupper (*s++);
+		    num = (num << 4) + (ch <= '9' ? ch - '0' : ch - 'A' + 10);
+		  }
+		*d++ = num;
+		break;
+	      }
+	    case '0':
+	    case '1':
+	    case '2':
+	    case '3':
+	    case '4':
+	    case '5':
+	    case '6':
+	    case '7':
+	      {
+		int num = *(s - 1) - '0';
+		if (*s >= '0' && *s <= '7')
+		  num = (num << 3) + *s++ - '0';
+		if (*s >= '0' && *s <= '7')
+		  num = (num << 3) + *s++ - '0';
+		*d++ = num;
+		break;
+	      }
+	    }
+	}
+      else
+	*d++ = *s++;
+    }
+  *d = '\0';
+  return (org);
+}
+
+/* =========================================================================
+ * Function: str255_to_string 
+ * Description: 
+ * Input: 
+ * Output: 
+ * ========================================================================= */
+
+char *
+str255_to_string (u_char * str255, char *str)
+{
+  static char buf[256];
+  char *str_base = NULL;
+  int len = *str255++;
+  int i = 0;
+
+  /* if haven't been given a str to write to then use local buffer */
+  if (!str)
+    str = &(buf[0]);
+
+  str_base = str;
+  for (i = 0; i < len; i++)
+    {
+      *str++ = *str255++;
+    }
+  *str = '\0';
+
+  return str_base;
+}
Index: /trunk/mgpp/lib/local_strings.h
===================================================================
--- /trunk/mgpp/lib/local_strings.h	(revision 3365)
+++ /trunk/mgpp/lib/local_strings.h	(revision 3365)
@@ -0,0 +1,79 @@
+/**************************************************************************
+ *
+ * local_strings.h -- string routines
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef LOCAL_STRINGS_H
+#define LOCAL_STRINGS_H
+
+#include "sysfuncs.h"
+
+class mgString
+{
+  u_char *data;
+ public:
+  inline int length()
+    {
+      return data[0];
+    }
+  inline u_char *rawdata()
+    {
+      return data;
+    }
+  inline void read(u_char **base)
+    {
+      unsigned int copy, suff;
+      
+      copy  = **base;
+      *base = *base + 1;
+      suff  = **base;
+      *base = *base + 1;
+      memcpy( (data + copy + 1), *base, suff);
+      *data = copy + suff;
+      *base += suff;
+    }
+  inline void read(FILE *dict)
+    {
+      unsigned int copy, suff;
+      copy = fgetc(dict);
+      suff = fgetc(dict);
+      fread(data + copy + 1, sizeof(u_char), suff, dict);
+      *data = copy + suff;
+    }
+  inline static void readString(u_char **base, u_char *prev)
+    {
+      ((mgString *) &prev)->read(base);
+    }
+  inline static void readString(FILE *dict, u_char *prev)
+    { 
+      ((mgString *) &prev)->read(dict);
+    }
+};
+
+int arg_atoi (char *str);
+int compare (u_char * s1, u_char * s2);
+int casecompare (u_char * s1, u_char * s2);   /* [RPAP - Jan 97: Stem Index Change] */
+u_char *copy_string (u_char * src_str);
+char *str255_to_string (u_char * str255, char *str);
+int prefixlen (u_char * s1, u_char * s2);
+char *char2str (u_char c);
+char *word2str (u_char * s);
+char *de_escape_string (char *s);
+
+#endif
Index: /trunk/mgpp/lib/longlong.h
===================================================================
--- /trunk/mgpp/lib/longlong.h	(revision 3365)
+++ /trunk/mgpp/lib/longlong.h	(revision 3365)
@@ -0,0 +1,85 @@
+/**************************************************************************
+ *
+ * longlong.h -- Use of GCC's long long integer types
+ * Copyright (C) 1999 Tim A.H. Bell
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_LONGLONG
+#define H_LONGLONG
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__
+#  include <WIN32cfg.h>
+# else
+#  include <config.h>
+# endif
+#endif
+
+/*
+  Use GCC's "long long" integer types for certain variables, to avoid
+  overflowing 32 bit integers.
+
+  Other files affected:
+  
+    lib:
+        bitio_random.c
+        bitio_random.h
+	
+    src/text:
+        ivf.pass1.c
+        ivf.pass2.c
+        build.h
+        mg_passes.c
+        text.h
+    
+*/
+
+/*#define TESTING_OVERFLOW*/
+
+#ifdef TESTING_OVERFLOW
+
+/* Test the overflow detection by using tiny (16-bit) types */
+typedef unsigned short int mg_ullong;
+typedef short int mg_llong;
+
+#define ULL_FS "u"
+#define LL_FS "d"
+
+#elif defined __GNUC__ && ! defined DISABLE_LONG_LONG
+
+/* We're using GCC, so it's okay to use "long long" (64-bit) types */
+#define USE_LONG_LONG
+
+typedef unsigned long long int mg_ullong;
+typedef long long int mg_llong;
+
+#define ULL_FS "llu"
+#define LL_FS "lld"
+
+#else
+
+/* Not using GCC, so fall back on plain "long" (32-bit) types */
+typedef unsigned long int mg_ullong;
+typedef long int mg_llong;
+
+#define ULL_FS "lu"
+#define LL_FS "ld"
+
+#endif /* __GNUC__ */
+
+#endif
Index: /trunk/mgpp/lib/lovinstem.cpp
===================================================================
--- /trunk/mgpp/lib/lovinstem.cpp	(revision 3365)
+++ /trunk/mgpp/lib/lovinstem.cpp	(revision 3365)
@@ -0,0 +1,1172 @@
+/**************************************************************************
+ *
+ * lovinstem.c -- Stemming code
+ * Copyright (C) 1994  Linh Huynh (linh@kbs.citri.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+
+/*****************************************************************************
+ *
+ *   Transformational Rules used in Recoding Stem Terminations
+ *
+ *   Parameter in functions:
+ *       ch : the second last character of the old ending
+ *            (after removing one of double consonants) 
+ *
+ ****************************************************************************/
+
+
+/************ Conditional rules associated with transformations **************/
+
+/*static int aio ();
+static int s ();
+static int pt ();
+static int m ();
+static int n ();*/
+
+static int 
+aio (char ch)			/* Rule number 9     */
+{
+  return ((ch != 'a') && (ch != 'i') && (ch != 'o'));
+}
+
+static int 
+s (char ch)				/* Rule Number 24    */
+{
+  return (ch != 's');
+}
+
+static int 
+pt (char ch)				/* Rule number 28    */
+{
+  return ((ch != 'p') && (ch != 't'));
+}
+
+static int 
+m (char ch)				/* Rule number 30    */
+{
+  return (ch != 'm');
+}
+
+static int 
+n (char ch)				/* Rule number 32    */
+{
+  return (ch != 'n');
+}
+
+
+/**************************************************************************
+ *
+ *  Context-sensitive rules associated with certain endings.
+ *
+ *  The notations of the rules are the same as in Lovins' paper
+ *  except that rule A is replaced by a NULL in the data structure.
+ *
+ *  Function parameters:
+ *      - stem_length: possible stem length
+ *      - end        : pointer points to the end of the possible stem.
+ *
+ *************************************************************************/
+
+
+/********* Context-sensitive rule function declarations ******************/
+
+/*static int B ();
+static int C ();
+static int D ();
+static int E ();
+static int F ();
+static int G ();
+static int H ();
+static int I ();
+static int J ();
+static int K ();
+static int L ();
+static int M ();
+static int N ();
+static int O ();
+static int P ();
+static int Q ();
+static int R ();
+static int S ();
+static int T ();
+static int U ();
+static int V ();
+static int W ();
+static int X ();
+static int Y ();
+static int Z ();
+static int AA ();
+static int BB ();
+static int CC ();
+*/
+static int 
+B (int stem_length, char *end)
+{
+  return (stem_length >= 3);
+}
+
+static int 
+C (int stem_length, char *end)
+{
+  return (stem_length >= 4);
+}
+
+static int 
+D (int stem_length, char *end)
+{
+  return (stem_length >= 5);
+}
+
+static int 
+E (int stem_length, char *end)
+
+     
+{
+  return (*end != 'e');
+}
+
+static int 
+F (int stem_length, char *end)
+     
+     
+{
+  return ((stem_length >= 3) && (*end != 'e'));
+}
+
+static int 
+G (int stem_length, char *end)
+     
+     
+{
+  return ((stem_length >= 3) && (*end == 'f'));
+}
+
+static int 
+H (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+  return ((p1 == 't') || ((p1 == 'l') && (p2 == 'l')));
+}
+
+static int 
+I (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 != 'o') && (p1 != 'e'));
+}
+
+static int 
+J (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 != 'a') && (p1 != 'e'));
+}
+
+static int 
+K (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((stem_length >= 3) &&
+	  ((p1 == 'l') || (p1 == 'i') ||
+	   ((p1 == 'e') && (*(end - 2) == 'u'))));
+}
+
+static int 
+L (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+  if (p1 == 's')
+    return (p2 == 'o');
+  else
+    return ((p1 != 'u') && (p1 != 'x'));
+}
+
+static int 
+M (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 != 'a') && (p1 != 'c') &&
+	  (p1 != 'e') && (p1 != 'm'));
+}
+
+static int 
+N (int stem_length, char *end)
+     
+     
+{
+  if (stem_length >= 3)
+    {
+      if (*(end - 2) == 's')
+	return (stem_length >= 4);
+      else
+	return 1;
+    }
+  else
+    return 0;
+}
+
+static int 
+O (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 == 'l') || (p1 == 'i'));
+}
+
+static int 
+P (int stem_length, char *end)
+     
+     
+{
+  return (*end != 'c');
+}
+
+static int 
+Q (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  return ((stem_length >= 3) && (p1 != 'l') && (p1 != 'n'));
+}
+
+static int 
+R (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+  return ((p1 == 'n') || (p1 == 'r'));
+}
+
+static int 
+S (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+
+  if (p1 == 't')
+    return (p2 != 't');
+  else
+    return ((p1 == 'r') && (p2 == 'd'));
+}
+
+static int 
+T (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+
+  if (p1 == 't')
+    return (p2 != 'o');
+  else
+    return (p1 == 's');
+}
+
+static int 
+U (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  return ((p1 == 'l') || (p1 == 'm') ||
+	  (p1 == 'n') || (p1 == 'r'));
+}
+
+static int 
+V (int stem_length, char *end)
+     
+     
+{
+  return (*end == 'c');
+}
+
+static int 
+W (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  return ((p1 != 's') && (p1 != 'u'));
+}
+
+static int 
+X (int stem_length, char *end)
+     
+     
+{
+  char p1;
+  p1 = *end;
+
+  if (p1 == 'e')
+    return ((stem_length >= 3) && (*(end - 2) == 'u'));
+  else
+    return ((p1 == 'l') || (p1 == 'i'));
+}
+
+static int 
+Y (int stem_length, char *end)
+     
+     
+{
+  return ((*end == 'n') && (*(end - 1) == 'i'));
+}
+
+static int 
+Z (int stem_length, char *end)
+     
+     
+{
+  return (*end != 'f');
+}
+
+static int 
+AA (int stem_length, char *end)
+     
+     
+{
+  char p1, p2;
+  p1 = *end;
+  p2 = *(end - 1);
+
+  if (p1 == 'h')
+    return ((p2 == 'p') || (p2 == 't'));
+  else if (p1 == 'r')
+    return ((p2 == 'e') || (p2 == 'o'));
+  else if (p1 == 's')
+    return (p2 == 'e');
+  else
+    return ((p1 == 'd') || (p1 == 'f') ||
+	    (p1 == 'l') || (p1 == 't'));
+}
+
+static int 
+BB (int stem_length, char *end)
+     
+     
+{
+  if (stem_length >= 3)
+    {
+      if (stem_length >= 4)
+	return (strncmp (end - 3, "ryst", 4) != 0);
+      else
+	return (strncmp (end - 2, "met", 3) != 0);
+    }
+  else
+    return 0;
+}
+
+static int 
+CC (int stem_length, char *end)
+     
+     
+{
+  return (*end == 'l');
+}
+
+/**************************************************************************/
+
+#define MIN_STEM_LENGTH 2
+#define MAX_ENDING_SIZE 11
+#define PREDEFINED_SIZE (MAX_ENDING_SIZE + MIN_STEM_LENGTH)
+#define EOS     '\0'
+/* [RPAP - Jan 97: Stem Index Change] */
+#define MAX_STEM_LENGTH 255
+
+static char *remove_ending (char *word, int w_len);
+static void recode_stem (char *stem_end);
+
+/* =========================================================================
+ * Function: stem
+ * Description: 
+ *      Implemetation of the Lovins' stemming algorithm described in
+ *      J.B. Lovins, "Development of a Stemming Algorithm",
+ *      Mechanical Translation and Computational Linguistics, Vol 11,1968.
+ * Input: a word string with the length in the first byte
+ * Output: the stemmed word
+ * ========================================================================= */
+
+void
+lovinstem (u_char * word)
+{
+  char *word_start;		/* points to first letter of word */
+  char *stem_end;		/* points to last character of stem portion */
+  int old_length;		/* length of word */
+  int new_length;		/* length after stemming */
+
+  /* [RPAP - Jan 97: Stem Index Change] */
+  u_char copy[MAX_STEM_LENGTH + 2];
+  memcpy (copy, word, *word + 1);
+
+  old_length = *copy; /* [RPAP - Jan 97: Stem Index Change] */
+
+  /* The word must be at least MIN_STEM_LENGTH characters long. */
+  if (old_length <= MIN_STEM_LENGTH)
+    return;
+
+  word_start = (char *) copy + 1;	/* jump over length byte */ /* [RPAP - Jan 97: Stem Index Change] */
+  copy[old_length + 1] = '\0';	/* null terminate string */ /* [RPAP - Jan 97: Stem Index Change] */
+
+  stem_end = remove_ending (word_start, old_length);
+  recode_stem (stem_end);
+
+  /* fix up new length */
+  /* have to use strlen because have no other way of telling length */
+  new_length = strlen (word_start);
+  *copy = new_length;  /* [RPAP - Jan 97: Stem Index Change] */
+
+  /* [RPAP - Jan 97: Stem Index Change] */
+  memcpy (word, copy, *copy + 1);
+
+}				/*stem_complex */
+
+/* =========================================================================
+
+ *   Data structure for recoding and the method for recoding the stem. 
+ *
+ * ========================================================================= */
+
+typedef struct
+{
+  char *old_end;		/* old ending */
+  char *new_end;		/* new ending */
+  char old_offset;		/* length of the old ending - 1 */
+  int (*cond) (char);	        /* condition rule */
+  char end_group;		/* signal the end of the group */
+}
+Recode_Rules;
+
+/*  
+   Stem terminations are divided into groups having the same last
+   character
+ */
+
+static Recode_Rules Rules[] =
+{
+  "uad", "uas", 2, NULL, 0,
+  "vad", "vas", 2, NULL, 0,
+  "cid", "cis", 2, NULL, 0,
+  "lid", "lis", 2, NULL, 0,
+  "erid", "eris", 3, NULL, 0,
+  "pand", "pans", 3, NULL, 0,
+  "end", "ens", 2, s, 0,
+  "end", "ens", 2, m, 0,
+  "ond", "ons", 2, NULL, 0,
+  "lud", "lus", 2, NULL, 0,
+  "rud", "rus", 2, NULL, 1,
+
+  "ul", "l", 1, aio, 1,
+
+  "istr", "ister", 3, NULL, 0,
+  "metr", "meter", 3, NULL, 0,
+  "her", "hes", 2, pt, 1,
+
+  "urs", "ur", 2, NULL, 1,
+
+  "uct", "uc", 2, NULL, 0,
+  "umpt", "um", 3, NULL, 0,
+  "rpt", "rb", 2, NULL, 0,
+  "mit", "mis", 2, NULL, 0,
+  "ert", "ers", 2, NULL, 0,
+  "et", "es", 1, n, 0,
+  "yt", "ys", 1, NULL, 1,
+
+  "iev", "ief", 2, NULL, 0,
+  "olv", "olut", 2, NULL, 1,
+
+  "bex", "bic", 2, NULL, 0,
+  "dex", "dic", 2, NULL, 0,
+  "pex", "pic", 2, NULL, 0,
+  "tex", "tic", 2, NULL, 0,
+  "ax", "ac", 1, NULL, 0,
+  "ex", "ec", 1, NULL, 0,
+  "ix", "ic", 1, NULL, 0,
+  "lux", "luc", 2, NULL, 1,
+
+  "yz", "ys", 1, NULL, 1
+};
+
+typedef struct last
+  {
+    char c;			/* The last character */
+    struct last *left;		/* used in balanced   */
+    struct last *right;		/* binary tree        */
+    Recode_Rules *pt;		/* location of approriate group */
+  }
+Last_Char_Node;
+
+/*------------
+       s
+     /   \
+    l     x
+   / \   / \
+  d   r t   z
+         \
+          v
+---------------*/
+static Last_Char_Node pr[] =
+{
+  'd', NULL, NULL, Rules,
+  'l', pr, pr + 2, Rules + 11,
+  'r', NULL, NULL, Rules + 12,
+  's', pr + 1, pr + 6, Rules + 15,
+  't', NULL, pr + 5, Rules + 16,
+  'v', NULL, NULL, Rules + 23,
+  'x', pr + 4, pr + 7, Rules + 25,
+  'z', NULL, NULL, Rules + 33,
+};
+
+
+/*****************************************************************************
+ *
+ *  Recode the stem according to the transformation rules.
+ *
+ *****************************************************************************/
+
+static void
+recode_stem (char *stem_end)
+{
+  static Last_Char_Node *root = pr + 3;
+  Last_Char_Node *p_last = root;
+  Recode_Rules *rule;		/* points to the transformation list */
+  char *h,			/* points to start of possible ending */
+    ch,				/* last character of the old ending  */
+    ch2;
+
+  h = stem_end - 1;
+  ch = *stem_end;
+  if (*h == ch)			/* Check for double consonant        */
+    {
+      if (strchr ("bdglmnprst", ch) != NULL)
+	{
+	  *stem_end = EOS;
+	  stem_end = h;
+	}
+    }
+
+  do				/* Check for the last character       */
+    {
+      ch2 = p_last->c;
+      if (ch == ch2)
+	break;
+      else if (ch > ch2)
+	p_last = p_last->right;
+      else
+	p_last = p_last->left;
+    }
+  while (p_last != NULL);
+
+  if (p_last != NULL)		/* Check for the rest of suffix list */
+    {
+      rule = p_last->pt;
+      for (;;)
+	{
+	  h = stem_end - rule->old_offset;
+	  if (strcmp (h, rule->old_end) == 0)
+	    {
+	      if (!rule->cond || (*rule->cond) (*(h - 1)))
+		{
+		  (void) strcpy (h, rule->new_end);
+		  break;
+		}
+	    }
+	  if (rule->end_group)
+	    break;
+	  rule++;
+	}
+    }
+}
+
+/* ======================================================================
+
+ *    List of endings and the method to remove the ending 
+ *
+ * ======================================================================*/
+
+/************ Data structures for list of endings  ********************/
+
+typedef struct
+{
+  char *ending;			/* old ending                */
+  int (*cond) (int length, char *end);		/* conditional rule          */
+  signed char left_offset;	/* used to find the siblings */
+  signed char right_offset;	/* in balanced binary tree   */
+}
+Ending_List;
+
+/*
+   The original list of endings is re-organised into groups having
+   the same length and the same first character.
+ */
+
+static Ending_List List[] =
+{
+  "a", NULL, 0, 0,
+
+  "ae", NULL, 0, 0,
+  "al", BB, -1, 2,
+  "ar", X, 0, 0,
+  "as", B, -1, 0,
+
+  "acy", NULL, 0, 1,
+  "age", B, 0, 0,
+  "aic", NULL, -2, 1,
+  "als", BB, 0, 0,
+  "ant", B, -2, 2,
+  "ars", O, 0, 0,
+  "ary", F, -1, 2,
+  "ata", NULL, 0, 0,
+  "ate", NULL, -1, 0,
+
+  "able", NULL, 0, 1,
+  "ably", NULL, 0, 0,
+  "ages", B, -2, 2,
+  "ally", B, 0, 0,
+  "ance", B, -1, 1,
+  "ancy", B, 0, 0,
+  "ants", B, -4, 4,
+  "aric", NULL, 0, 0,
+  "arly", K, -1, 1,
+  "ated", I, 0, 0,
+  "ates", NULL, -2, 2,
+  "atic", B, 0, 0,
+  "ator", NULL, -1, 0,
+
+  "acies", NULL, 0, 0,
+  "acity", NULL, -1, 1,
+  "aging", B, 0, 0,
+  "aical", NULL, -2, 2,
+  "alist", NULL, 0, 0,
+  "alism", B, -1, 0,
+  "ality", NULL, -3, 3,
+  "alize", NULL, 0, 1,
+  "allic", BB, 0, 0,
+  "anced", B, -2, 2,
+  "ances", B, 0, 0,
+  "antic", C, -1, 0,
+  "arial", NULL, -6, 6,
+  "aries", NULL, 0, 1,
+  "arily", NULL, 0, 0,
+  "arity", B, -2, 2,
+  "arize", NULL, 0, 0,
+  "aroid", NULL, -1, 0,
+  "ately", NULL, -3, 3,
+  "ating", I, 0, 1,
+  "ation", B, 0, 0,
+  "ative", NULL, -2, 2,
+  "ators", NULL, 0, 0,
+  "atory", NULL, -1, 1,
+  "ature", E, 0, 0,
+
+  "aceous", NULL, 0, 1,
+  "acious", B, 0, 0,
+  "action", G, -2, 2,
+  "alness", NULL, 0, 0,
+  "ancial", NULL, -1, 1,
+  "ancies", NULL, 0, 0,
+  "ancing", B, -4, 4,
+  "ariser", NULL, 0, 0,
+  "arized", NULL, -1, 1,
+  "arizer", NULL, 0, 0,
+  "atable", NULL, -2, 2,
+  "ations", B, 0, 0,
+  "atives", NULL, -1, 0,
+
+  "ability", NULL, 0, 1,
+  "aically", NULL, 0, 0,
+  "alistic", B, -2, 2,
+  "alities", NULL, 0, 0,
+  "ariness", E, -1, 0,
+  "aristic", NULL, -3, 3,
+  "arizing", NULL, 0, 1,
+  "ateness", NULL, 0, 0,
+  "atingly", NULL, -2, 2,
+  "ational", B, 0, 0,
+  "atively", NULL, -1, 1,
+  "ativism", NULL, 0, 0,
+
+  "ableness", NULL, 0, 1,
+  "arizable", NULL, 0, 0,
+
+  "allically", C, 0, 0,
+  "antaneous", NULL, -1, 1,
+  "antiality", NULL, 0, 0,
+  "arisation", NULL, -2, 2,
+  "arization", NULL, 0, 0,
+  "ationally", B, -1, 1,
+  "ativeness", NULL, 0, 0,
+
+  "antialness", NULL, 0, 0,
+  "arisations", NULL, -1, 1,
+  "arizations", NULL, 0, 0,
+
+  "alistically", B, 0, 1,
+  "arizability", NULL, 0, 0,
+
+  "e", NULL, 0, 0,
+
+  "ed", E, 0, 0,
+  "en", F, -1, 1,
+  "es", E, 0, 0,
+
+  "eal", Y, 0, 0,
+  "ear", Y, -1, 1,
+  "ely", E, 0, 0,
+  "ene", E, -2, 2,
+  "ent", C, 0, 0,
+  "ery", E, -1, 1,
+  "ese", NULL, 0, 0,
+
+  "ealy", Y, 0, 1,
+  "edly", E, 0, 0,
+  "eful", NULL, -2, 1,
+  "eity", NULL, 0, 0,
+  "ence", NULL, -2, 2,
+  "ency", NULL, 0, 0,
+  "ened", E, -1, 2,
+  "enly", E, 0, 0,
+  "eous", NULL, -1, 0,
+
+  "early", Y, 0, 1,
+  "ehood", NULL, 0, 0,
+  "eless", NULL, -2, 2,
+  "elity", NULL, 0, 0,
+  "ement", NULL, -1, 0,
+  "enced", NULL, -3, 3,
+  "ences", NULL, 0, 1,
+  "eness", E, 0, 0,
+  "ening", E, -2, 2,
+  "ental", NULL, 0, 0,
+  "ented", C, -1, 1,
+  "ently", NULL, 0, 0,
+
+  "eature", Z, 0, 0,
+  "efully", NULL, -1, 1,
+  "encies", NULL, 0, 0,
+  "encing", NULL, -2, 2,
+  "ential", NULL, 0, 0,
+  "enting", C, -1, 1,
+  "entist", NULL, 0, 1,
+  "eously", NULL, 0, 0,
+
+  "elihood", E, 0, 1,
+  "encible", NULL, 0, 0,
+  "entally", NULL, -2, 2,
+  "entials", NULL, 0, 0,
+  "entiate", NULL, -1, 1,
+  "entness", NULL, 0, 0,
+
+  "entation", NULL, 0, 0,
+  "entially", NULL, -1, 1,
+  "eousness", NULL, 0, 0,
+
+  "eableness", E, 0, 1,
+  "entations", NULL, 0, 0,
+  "entiality", NULL, -2, 2,
+  "entialize", NULL, 0, 0,
+  "entiation", NULL, -1, 0,
+
+  "entialness", NULL, 0, 0,
+
+  "ful", NULL, 0, 0,
+
+  "fully", NULL, 0, 0,
+
+  "fulness", NULL, 0, 0,
+
+  "hood", NULL, 0, 0,
+
+  "i", NULL, 0, 0,
+
+  "ia", NULL, 0, 0,
+  "ic", NULL, -1, 1,
+  "is", NULL, 0, 0,
+
+  "ial", NULL, 0, 0,
+  "ian", NULL, -1, 1,
+  "ics", NULL, 0, 1,
+  "ide", L, 0, 0,
+  "ied", NULL, -3, 3,
+  "ier", NULL, 0, 0,
+  "ies", P, -1, 0,
+  "ily", NULL, -1, 1,
+  "ine", M, 0, 0,
+  "ing", N, -5, 5,
+  "ion", Q, 0, 0,
+  "ish", C, -1, 1,
+  "ism", B, 0, 1,
+  "ist", NULL, 0, 0,
+  "ite", AA, -3, 3,
+  "ity", NULL, 0, 0,
+  "ium", NULL, -1, 0,
+  "ive", NULL, -1, 1,
+  "ize", F, 0, 0,
+
+  "ials", NULL, 0, 0,
+  "ians", NULL, -1, 0,
+  "ible", NULL, -1, 1,
+  "ibly", NULL, 0, 0,
+  "ical", NULL, -2, 2,
+  "ides", L, 0, 0,
+  "iers", NULL, -1, 1,
+  "iful", NULL, 0, 0,
+  "ines", M, -4, 4,
+  "ings", N, 0, 0,
+  "ions", B, -1, 1,
+  "ious", NULL, 0, 0,
+  "isms", B, -2, 2,
+  "ists", NULL, 0, 0,
+  "itic", H, -1, 1,
+  "ized", F, 0, 1,
+  "izer", F, 0, 0,
+
+  "ially", NULL, 0, 0,
+  "icant", NULL, -1, 1,
+  "ician", NULL, 0, 0,
+  "icide", NULL, -2, 2,
+  "icism", NULL, 0, 0,
+  "icist", NULL, -1, 0,
+  "icity", NULL, -3, 3,
+  "idine", I, 0, 1,
+  "iedly", NULL, 0, 0,
+  "ihood", NULL, -2, 2,
+  "inate", NULL, 0, 0,
+  "iness", NULL, -1, 0,
+  "ingly", B, -6, 6,
+  "inism", J, 0, 1,
+  "inity", CC, 0, 0,
+  "ional", NULL, -2, 2,
+  "ioned", NULL, 0, 0,
+  "ished", NULL, -1, 0,
+  "istic", NULL, -3, 3,
+  "ities", NULL, 0, 1,
+  "itous", NULL, 0, 0,
+  "ively", NULL, -2, 2,
+  "ivity", NULL, 0, 0,
+  "izers", F, -1, 1,
+  "izing", F, 0, 0,
+
+  "ialist", NULL, 0, 0,
+  "iality", NULL, -1, 1,
+  "ialize", NULL, 0, 0,
+  "ically", NULL, -2, 2,
+  "icance", NULL, 0, 0,
+  "icians", NULL, -1, 1,
+  "icists", NULL, 0, 0,
+  "ifully", NULL, -4, 4,
+  "ionals", NULL, 0, 0,
+  "ionate", D, -1, 1,
+  "ioning", NULL, 0, 0,
+  "ionist", NULL, -2, 2,
+  "iously", NULL, 0, 0,
+  "istics", NULL, -1, 1,
+  "izable", E, 0, 0,
+
+  "ibility", NULL, 0, 0,
+  "icalism", NULL, -1, 1,
+  "icalist", NULL, 0, 1,
+  "icality", NULL, 0, 0,
+  "icalize", NULL, -3, 3,
+  "ication", G, 0, 0,
+  "icianry", NULL, -1, 0,
+  "ination", NULL, -1, 1,
+  "ingness", NULL, 0, 0,
+  "ionally", NULL, -5, 5,
+  "isation", NULL, 0, 0,
+  "ishness", NULL, -1, 1,
+  "istical", NULL, 0, 1,
+  "iteness", NULL, 0, 0,
+  "iveness", NULL, -3, 3,
+  "ivistic", NULL, 0, 0,
+  "ivities", NULL, -1, 0,
+  "ization", F, -1, 1,
+  "izement", NULL, 0, 0,
+
+  "ibleness", NULL, 0, 0,
+  "icalness", NULL, -1, 1,
+  "ionalism", NULL, 0, 0,
+  "ionality", NULL, -2, 2,
+  "ionalize", NULL, 0, 0,
+  "iousness", NULL, -1, 1,
+  "izations", NULL, 0, 0,
+
+  "ionalness", NULL, 0, 1,
+  "istically", NULL, 0, 0,
+  "itousness", NULL, -2, 2,
+  "izability", NULL, 0, 0,
+  "izational", NULL, -1, 0,
+
+  "izationally", B, 0, 0,
+
+  "ly", B, 0, 0,
+
+  "less", NULL, 0, 1,
+  "lily", NULL, 0, 0,
+
+  "lessly", NULL, 0, 0,
+
+  "lessness", NULL, 0, 0,
+
+  "ness", NULL, 0, 0,
+
+  "nesses", NULL, 0, 0,
+
+  "o", NULL, 0, 0,
+
+  "on", S, 0, 1,
+  "or", T, 0, 0,
+
+  "oid", NULL, 0, 0,
+  "one", R, -1, 1,
+  "ous", NULL, 0, 0,
+
+  "ogen", NULL, 0, 0,
+
+  "oidal", NULL, 0, 0,
+  "oides", NULL, -1, 2,
+  "otide", NULL, 0, 0,
+  "ously", NULL, -1, 0,
+
+  "oidism", NULL, 0, 0,
+
+  "oidally", NULL, 0, 1,
+  "ousness", NULL, 0, 0,
+
+  "s", W, 0, 0,
+
+  "s'", NULL, 0, 0,
+
+  "um", U, 0, 1,
+  "us", V, 0, 0,
+
+  "ward", NULL, 0, 1,
+  "wise", NULL, 0, 0,
+
+  "y", B, 0, 0,
+
+  "yl", R, 0, 0,
+
+  "ying", B, 0, 1,
+  "yish", NULL, 0, 0,
+
+  "'s", NULL, 0, 0,
+};
+
+typedef struct node
+  {
+    char c;			/* First character  */
+    struct node *left;		/* used in balanced */
+    struct node *right;		/* binary tree      */
+    Ending_List *ptr[11];	/* the approriate location */
+  }
+First_Char_Node;
+
+static First_Char_Node First[] =
+{
+  '\'', NULL, NULL,
+  {NULL,
+   List + 293, NULL, NULL, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'a', First, NULL,
+  {List,
+   List + 2, List + 9, List + 20, List + 39, List + 58,
+   List + 70, List + 77, List + 82, List + 87, List + 89},
+
+  'e', First + 1, First + 4,
+  {List + 91,
+   List + 93, List + 98, List + 106, List + 116, List + 126,
+   List + 133, List + 138, List + 142, List + 145, NULL},
+
+  'f', NULL, NULL,
+  {NULL,
+   NULL, List + 146, NULL, List + 147, NULL,
+   List + 148, NULL, NULL, NULL, NULL},
+
+  'h', First + 3, First + 5,
+  {NULL,
+   NULL, NULL, List + 149, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'i', NULL, NULL,
+  {List + 150,
+   List + 152, List + 163, List + 181, List + 202, List + 222,
+   List + 239, List + 252, List + 258, NULL, List + 261},
+
+  'l', First + 2, First + 10,
+  {NULL,
+   List + 262, NULL, List + 263, NULL, List + 265,
+   NULL, List + 266, NULL, NULL, NULL},
+
+  'n', NULL, NULL,
+  {NULL,
+   NULL, NULL, List + 267, NULL, List + 268,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'o', First + 7, First + 9,
+  {List + 269,
+   List + 270, List + 273, List + 275, List + 277, List + 280,
+   List + 281, NULL, NULL, NULL, NULL},
+
+  's', NULL, NULL,
+  {List + 283,
+   List + 284, NULL, NULL, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'u', First + 8, First + 12,
+  {NULL,
+   List + 285, NULL, NULL, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'w', NULL, NULL,
+  {NULL,
+   NULL, NULL, List + 287, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+
+  'y', First + 11, NULL,
+  {List + 289,
+   List + 290, NULL, List + 291, NULL, NULL,
+   NULL, NULL, NULL, NULL, NULL},
+};
+
+
+/******************************************************************************
+ *
+ *  Look for a match in the suffix list.
+ *  Return the pointer to the end of the new stem if there is a match.
+ *  Otherwise, return the pointer to the end of the original word.
+ *
+ *  Method:
+ *      * Search for the first character in the suffix list.
+ *      * If there is match, search for the rest of the suffix list.
+ *         
+ *****************************************************************************/
+
+static char *
+remove_ending (char *word, int w_len)
+{
+  static First_Char_Node *root = First + 6;	/* the root of binary tree is 'l' */
+  First_Char_Node *p_first;	/* Points to the first character      */
+  /* of the possible suffix             */
+  Ending_List *p_list;		/* Points to the List of Endings      */
+  char *suffix_start,		/* Points to start of possible suffix */
+   *stem_end,			/* Points to the end of possible stem */
+    ch1, ch2;
+  int s_len,			/* Possible stem length               */
+    e_offset,			/* Offset from the end to the start of */
+  /* possible suffix                    */
+    not_done = 1, cmp;
+
+  stem_end = word + w_len - 1;
+  if (w_len > PREDEFINED_SIZE)	/* find the start of suffix   */
+    suffix_start = word + w_len - MAX_ENDING_SIZE;
+  else
+    suffix_start = word + MIN_STEM_LENGTH;
+
+  ch1 = *suffix_start;
+  do				/* Search for the first character     */
+    {
+      p_first = root;
+      do
+	{
+	  ch2 = p_first->c;
+	  if (ch1 == ch2)
+	    break;
+	  else if (ch1 > ch2)
+	    p_first = p_first->right;
+	  else
+	    p_first = p_first->left;
+	}
+      while (p_first != NULL);
+
+      if (p_first != NULL)	/* Search for the rest */
+	{
+	  e_offset = stem_end - suffix_start;
+	  if ((p_list = p_first->ptr[e_offset]) != NULL)
+	    {
+	      for (;;)		/* no need to compare the first char  */
+		{
+		  cmp = strcmp (suffix_start + 1, p_list->ending + 1);
+		  if (cmp > 0)
+		    if (p_list->right_offset)
+		      p_list += p_list->right_offset;
+		    else
+		      break;
+		  else if (cmp < 0)
+		    if (p_list->left_offset)
+		      p_list += p_list->left_offset;
+		    else
+		      break;
+		  else
+		    {
+		      s_len = suffix_start - word;
+		      if (!p_list->cond ||
+			  (*p_list->cond) (s_len, suffix_start - 1))
+			{
+			  *suffix_start = EOS;
+			  stem_end = suffix_start - 1;
+			  not_done = 0;
+			}
+		      break;
+		    }
+		}
+	    }
+	}
+      suffix_start++;
+    }
+  while (not_done && ((ch1 = *suffix_start) != EOS));
+  return (stem_end);
+}
Index: /trunk/mgpp/lib/lovinstem.h
===================================================================
--- /trunk/mgpp/lib/lovinstem.h	(revision 3365)
+++ /trunk/mgpp/lib/lovinstem.h	(revision 3365)
@@ -0,0 +1,28 @@
+/**************************************************************************
+ *
+ * lovinstem.h -- Lovins' stemmer header
+ * Copyright (C) 1994  Linh Huynh, 
+ *                     (glued into library by tes@kbs.citri.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef LOVINSTEM_H
+#define LOVINSTEM_H
+
+void lovinstem (unsigned char *word);
+
+#endif
Index: /trunk/mgpp/lib/memlib.cpp
===================================================================
--- /trunk/mgpp/lib/memlib.cpp	(revision 3365)
+++ /trunk/mgpp/lib/memlib.cpp	(revision 3365)
@@ -0,0 +1,41 @@
+/**************************************************************************
+ *
+ * memlib.c -- Malloc wrappers
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "memlib.h"
+
+/* Defined as strdup is not an ANSI function */
+char *
+my_strdup(const char *str)
+{
+  char *ret_str = new char[strlen(str)+1];
+  if (ret_str) return strcpy(ret_str, str);
+  else return (char*) 0;
+}
+
+
+Malloc_func Xmalloc = malloc;
+
+Realloc_func Xrealloc = realloc;
+
+Free_func Xfree = free;
+
+Strdup_func Xstrdup = my_strdup;
Index: /trunk/mgpp/lib/memlib.h
===================================================================
--- /trunk/mgpp/lib/memlib.h	(revision 3365)
+++ /trunk/mgpp/lib/memlib.h	(revision 3365)
@@ -0,0 +1,55 @@
+/**************************************************************************
+ *
+ * memlib.h -- Malloc wrappers
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_MEMLIB
+#define H_MEMLIB
+
+/* changed default behaviour - we now assume std memory. - john mcp. */
+/* #ifdef STD_MEM */
+#if !defined(NOT_STD_MEM)
+#include <string.h>
+#define Xmalloc malloc
+#define Xrealloc realloc
+#define Xfree free
+
+#if defined (__WIN32__)
+# define Xstrdup _strdup
+#else
+# define Xstrdup strdup
+#endif
+
+#else
+
+
+#include "sysfuncs.h"
+
+typedef void *(*Malloc_func) (size_t);
+typedef void *(*Realloc_func) (void *, size_t);
+typedef void (*Free_func) (void *);
+typedef char *(*Strdup_func) (const char *);
+
+extern Malloc_func Xmalloc;
+extern Realloc_func Xrealloc;
+extern Free_func Xfree;
+extern Strdup_func Xstrdup;
+#endif
+
+#endif
Index: /trunk/mgpp/lib/messages.cpp
===================================================================
--- /trunk/mgpp/lib/messages.cpp	(revision 3365)
+++ /trunk/mgpp/lib/messages.cpp	(revision 3365)
@@ -0,0 +1,75 @@
+/**************************************************************************
+ *
+ * messages.c -- Message and error functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include <stdarg.h>
+#include "messages.h"
+
+char *msg_prefix = "";
+
+
+void VOLATILE 
+FatalError (int ExitCode, const char *fmt,...)
+{
+  char buf[1024];
+  char *s, *pfx;
+  va_list args;
+  va_start (args, fmt);
+  vsprintf (buf, fmt, args);
+  s = strrchr (buf, '\n');
+  if (!s || *(s + 1) != '\0')
+    strcat (buf, "\n");
+  pfx = strrchr (msg_prefix, '/');
+  pfx = pfx ? pfx + 1 : msg_prefix;
+  fprintf (stderr, "%s%s%s", pfx, *pfx ? " : " : "", buf);
+  exit (ExitCode);
+}
+
+
+
+/*
+ * This function writes messages to stderr. Due to the fact that I can't 
+ * guarantee that the fprintf call is monatomic I have to implement a 
+ * semaphore system.
+ */
+void 
+Message (const char *fmt,...)
+{
+  char buf[1024];
+  char *s, *pfx;
+  va_list args;
+
+#ifdef SILENT
+  return; /* stop those messages!!!! */
+#endif  
+
+  va_start (args, fmt);
+  vsprintf (buf, fmt, args);
+  s = strrchr (buf, '\n');
+  if (!s || *(s + 1) != '\0')
+    strcat (buf, "\n");
+
+  pfx = strrchr (msg_prefix, '/');
+  pfx = pfx ? pfx + 1 : msg_prefix;
+  fprintf (stderr, "%s%s%s", pfx, *pfx ? " : " : "", buf);
+
+}
Index: /trunk/mgpp/lib/messages.h
===================================================================
--- /trunk/mgpp/lib/messages.h	(revision 3365)
+++ /trunk/mgpp/lib/messages.h	(revision 3365)
@@ -0,0 +1,49 @@
+/**************************************************************************
+ *
+ * messages.h -- Message and error functions
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_MESSAGES
+#define H_MESSAGES
+
+#ifdef __GNUC__
+#define VOLATILE volatile
+#else
+#define VOLATILE
+#endif
+
+#ifdef __cplusplus
+extern "C"
+{
+#endif
+extern char *msg_prefix;
+
+void VOLATILE FatalError (int ExitCode, const char *fmt,...);
+void Message (const char *fmt,...);
+
+#ifdef __cplusplus
+}
+#endif
+
+
+#endif
+
+
+
+
Index: /trunk/mgpp/lib/mgheap.cpp
===================================================================
--- /trunk/mgpp/lib/mgheap.cpp	(revision 3365)
+++ /trunk/mgpp/lib/mgheap.cpp	(revision 3365)
@@ -0,0 +1,148 @@
+/**************************************************************************
+ *
+ * mgheap.c -- Heap routines
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "mgheap.h"
+
+#define SWAP(s, a, b) 						\
+do {								\
+     register int __i;						\
+     register char *__a = (a), *__b = (b);			\
+     for (__i = (s); __i; __i--)				\
+       {							\
+         register char __t = *__a;				\
+	 *__a++ = *__b; *__b++ = __t;				\
+       }							\
+} while(0)
+
+
+
+#define COPY(s, dst, src) 					\
+do {								\
+     register int __i;						\
+     register char *__dst = (dst), *__src = (src);		\
+     for (__i = (s); __i; __i--)				\
+       *__dst++ = *__src++;					\
+} while(0)
+
+
+
+static void 
+heap_heapify (void *heap, int size, int num, int curr, heap_comp hc)
+{
+  register int child = curr * 2;
+  while (child <= num)
+    {
+      if (child < num && hc ((char *) heap + child * size,
+			     (char *) heap + child * size - size) < 0)
+	child++;
+      if (hc ((char *) heap + (curr - 1) * size, (char *) heap + (child - 1) * size) > 0)
+	{
+	  SWAP (size, (char *) heap + (curr - 1) * size,
+		(char *) heap + (child - 1) * size);
+	  curr = child;
+	  child = child * 2;
+	}
+      else
+	break;
+    }
+}
+
+static void 
+heap_pullup (void *heap, int size, int num, heap_comp hc)
+{
+  register int curr = num;
+  register int parent = curr >> 1;
+  while (parent &&
+	 hc ((char *) heap + (curr - 1) * size, (char *) heap + (parent - 1) * size) < 0)
+    {
+      SWAP (size, (char *) heap + (curr - 1) * size,
+	    (char *) heap + (parent - 1) * size);
+      curr = parent;
+      parent = curr >> 1;
+    }
+}
+
+/************************************************************************
+ *
+ * NOTE: If you choose to change the comparison function the first thing 
+ *       you do after changing the function is call heap_build.
+ *
+ */
+void 
+heap_build (void *heap, int size, int num, heap_comp hc)
+{
+  register int i;
+  for (i = num / 2; i > 0; i--)
+    heap_heapify (heap, size, num, i, hc);
+}
+
+
+
+/****************************************************
+ *
+ * NOTE : The heap must be built before heap_sort is called. 
+ *        This has the effect of reversing the order of the array.
+ *		e.g. if your comparison function is designed to pull the
+ *                   biggest thing off the heap first then the result of
+ *		     sorting with this function will be to put the bigest
+ *                   thing at the end of the array.
+ *
+ */
+void 
+heap_sort (void *heap, int size, int num, heap_comp hc)
+{
+  register int i;
+  for (i = num; i > 1; i--)
+    {
+      SWAP (size, (char *) heap, (char *) heap + (i - 1) * size);
+      heap_heapify (heap, size, i - 1, 1, hc);
+    }
+}
+
+
+void 
+heap_deletehead (void *heap, int size, int *num, heap_comp hc)
+{
+  (*num)--;
+  SWAP (size, (char *) heap, (char *) heap + *num * size);
+  heap_heapify (heap, size, *num, 1, hc);
+}
+
+
+void 
+heap_changedhead (void *heap, int size, int num, heap_comp hc)
+{
+  heap_heapify (heap, size, num, 1, hc);
+}
+
+
+/***********************************************************************
+ *
+ * This assumes that the item has been added to the end of the
+ * array that is the heap. But that num has not been changed. 
+ *
+ */
+void 
+heap_additem (void *heap, int size, int *num, heap_comp hc)
+{
+  (*num)++;
+  heap_pullup (heap, size, *num, hc);
+}
Index: /trunk/mgpp/lib/mgheap.h
===================================================================
--- /trunk/mgpp/lib/mgheap.h	(revision 3365)
+++ /trunk/mgpp/lib/mgheap.h	(revision 3365)
@@ -0,0 +1,82 @@
+/**************************************************************************
+ *
+ * mgheap.h -- Heap routines
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+/* Care should be taken when using these routines as NO error checking is
+ * done.
+ */
+
+
+
+/* The heap comparison routine should return 
+ * <0 : If a should be before b
+ *  0 : If a and b are equal
+ * >0 : If a should be after b
+ */
+typedef int (*heap_comp) (void *a, void *b);
+
+
+/************************************************************************
+ *
+ * NOTE: If you choose to change the comparison function the first thing 
+ *       you do after changing the function is call heap_build.
+ *
+ */
+void heap_build (void *heap, int size, int num, heap_comp hc);
+
+
+
+/************************************************************************
+ *
+ * NOTE : The heap must be built before heap_sort is called.
+ *        This has the effect of reversing the order of the array.
+ *		e.g. if your comparison function is designed to pull the
+ *                   biggest thing off the heap first then the result of
+ *		     sorting with this function will be to put the bigest
+ *                   thing at the end of the array.
+ *
+ */
+void heap_sort (void *heap, int size, int num, heap_comp hc);
+
+/***********************************************************************
+ *
+ * NOTE: After deleting the head the heap will be one item shorter.
+ *       The deleted item will be placed at the end of the heap.
+ *
+ */
+void heap_deletehead (void *heap, int size, int *num, heap_comp hc);
+
+
+/***********************************************************************
+ *
+ * If you change the "value" of the root node of the heap then you
+ * should call this to re-pritorize the heap.
+ *
+ */
+void heap_changedhead (void *heap, int size, int num, heap_comp hc);
+
+
+/***********************************************************************
+ *
+ * This assumes that the item has been added to the end of the
+ * array that is the heap. But that num has not been changed. 
+ *
+ */
+void heap_additem (void *heap, int size, int *num, heap_comp hc);
Index: /trunk/mgpp/lib/netorder.h
===================================================================
--- /trunk/mgpp/lib/netorder.h	(revision 3365)
+++ /trunk/mgpp/lib/netorder.h	(revision 3365)
@@ -0,0 +1,189 @@
+#ifndef NETORDER_H
+#define NETORDER_H
+
+#include "sysfuncs.h"
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef __WIN32__
+#include "win32in.h"
+#else
+# include <netinet/in.h>
+#endif
+
+#if defined (_BIG_ENDIAN) || (defined (BYTE_ORDER) && BYTE_ORDER == BIG_ENDIAN) || (defined (__BYTE_ORDER) && __BYTE_ORDER == __BIG_ENDIAN)
+# ifndef _BIG_ENDIAN
+#  define _BIG_ENDIAN
+# endif
+#endif
+
+#if defined (_LITTLE_ENDIAN) || (defined (BYTE_ORDER) && BYTE_ORDER == LITTLE_ENDIAN) || (defined (__BYTE_ORDER) && __BYTE_ORDER == __LITTLE_ENDIAN)
+# ifndef _LITTLE_ENDIAN
+#  define _LITTLE_ENDIAN
+# endif
+#endif
+
+#ifdef _LITTLE_ENDIAN
+
+/* double */
+#define HTOND(d)                                                              \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &tmph, (const void *) &d, sizeof(double) >> 1); \
+	     memcpy ((void *) &tmpl, (const void *) ((unsigned int) &d + (sizeof(double) >> 1)), sizeof (double) >> 1);   \
+	     tmph = htonl (tmph);                                             \
+	     tmpl = htonl (tmpl);                                             \
+	     memcpy ( (void *) &d, (const void *) &tmpl, sizeof (double) >> 1);                           \
+	     memcpy ((void *) ((unsigned int) &d + (sizeof(double) >> 1)), (const void *) &tmph, sizeof (double) >> 1);   \
+	}while(0)
+#define NTOHD(d)                                                              \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &tmph, (const void *) &d, sizeof(double) >> 1); \
+	     memcpy ((void *) &tmpl, (const void *) ((int) &d + (sizeof(double) >> 1)), sizeof (double) >> 1);   \
+	     tmph = ntohl (tmph);                                             \
+	     tmpl = ntohl (tmpl);                                             \
+	     memcpy((void *) &d, (const void *) &tmpl, sizeof (double) >> 1); \
+	     memcpy ((void *) ((int) &d + (sizeof(double) >> 1)), (const void *) &tmph, sizeof (double) >> 1);   \
+        }while(0)
+#define HTOND2(hd, nd)                                                        \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &tmph, (const void *) &hd, sizeof(double) >> 1);\
+	     memcpy ((void *) &tmpl, (const void *) ((unsigned int) &hd + (sizeof(double) >> 1)), sizeof (double) >> 1);  \
+	     tmph = htonl (tmph);                                             \
+	     tmpl = htonl (tmpl);                                             \
+	     memcpy((void *) &nd, (const void *) &tmpl, sizeof (double) >> 1);\
+	     memcpy ((void *) ((unsigned int) &nd + (sizeof(double) >> 1)), (void *) &tmph, sizeof (double) >> 1);  \
+	}while(0)
+#define NTOHD2(nd, hd)                                                        \
+        do {                                                                  \
+             unsigned long tmph, tmpl;                                        \
+	     memcpy ((void *) &nd, (void *) &tmph, sizeof(double) >> 1);      \
+	     memcpy ((void *) &tmpl, (const void *) &nd + (sizeof(double) >> 1), sizeof (double) >> 1);  \
+	     tmph = ntohl (tmph);                                             \
+	     tmpl = ntohl (tmpl);                                             \
+	     memcpy((void *) &hd, (const void *) &tmpl, sizeof (double) >> 1);\
+	     memcpy ((void *) &hd + (sizeof(double) >> 1), (const void *) &tmph, sizeof (double) >> 1);  \
+        }while(0)
+
+/* float */
+#define HTONF(f)                                                              \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(f), sizeof (float));     \
+             HTONUL (tmp);                                                    \
+	     memcpy ((void *) &(f), (const void *) &tmp, sizeof (float));     \
+	}while(0)
+#define NTOHF(f)                                                              \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(f), sizeof (float));     \
+	     NTOHUL (tmp);                                                    \
+	     memcpy ((void *) &(f), (const void *) &tmp, sizeof (float));     \
+	}while(0)
+#define HTONF2(hf, nf)                                                        \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(hf), sizeof (float));    \
+             HTONUL (tmp);                                                    \
+	     memcpy ((void *) &(nf), (const void *) &tmp, sizeof (float));    \
+	}while(0)
+#define NTOHF2(nf, hf)                                                        \
+        do {                                                                  \
+             unsigned long tmp;                                               \
+             memcpy ((void *) &tmp, (const void *) &(nf), sizeof (float));    \
+	     NTOHUL (tmp);                                                    \
+	     memcpy ((void *) &(hf), (const void *) &tmp, sizeof (float));    \
+	}while(0)
+
+/* pointers */
+#define HTONP(p)          ((p) = (void *) htonl ((unsigned long) p))
+#define NTOHP(p)          ((p) = (void *) ntohl ((unsigned long) p))
+#define HTONP2(hp, np)    ((np) = (void *) htonl ((unsigned long) hp))
+#define NTOHP2(np, hp)    ((hp) = (void *) ntohl ((unsigned long) np))
+
+/* unsigned long */
+#define HTONUL(l)         ((l) = htonl((l)))
+#define NTOHUL(l)         ((l) = ntohl((l)))
+#define HTONUL2(hl, nl)   ((nl) = htonl ((hl)))
+#define NTOHUL2(nl, hl)   ((hl) = ntohl ((nl)))
+
+/* signed long */
+#define HTONSL(l)         ((l) = (long) htonl ((unsigned long) (l)))
+#define NTOHSL(l)         ((l) = (long) ntohl ((unsigned long) (l)))
+#define HTONSL2(hl, nl)   ((nl) = (long) htonl ((unsigned long) (hl)))
+#define NTOHSL2(nl, hl)   ((hl) = (long) ntohl ((unsigned long) (nl)))
+
+/* unsigned int */
+#define HTONUI(i)         ((i) = (unsigned int) htonl ((unsigned long) (i)))
+#define NTOHUI(i)         ((i) = (unsigned int) ntohl ((unsigned long) (i)))
+#define HTONUI2(hi, ni)   ((ni) = (unsigned int) htonl ((unsigned long) (hi)))
+#define NTOHUI2(ni, hi)   ((hi) = (unsigned int) ntohl ((unsigned long) (ni)))
+
+/* signed int */
+#define HTONSI(i)         ((i) = (int) htonl ((unsigned long) (i)))
+#define NTOHSI(i)         ((i) = (int) ntohl ((unsigned long) (i)))
+#define HTONSI2(hi, ni)   ((ni) = (int) htonl ((unsigned long) (hi)))
+#define NTOHSI2(ni, hi)   ((hi) = (int) ntohl ((unsigned long) (ni)))
+
+/* unsigned short */
+#define HTONUS(s)         ((s) = htons((s)))
+#define NTOHUS(s)         ((s) = ntohs((s)))
+#define HTONUS2(hs, ns)   ((ns) = htons((hs)))
+#define NTOHUS2(ns, hs)   ((hs) = ntohs((ns)))
+
+#else   /* _BIG_ENDIAN */
+
+/* double */
+#define HTOND(d)          (d)
+#define NTOHD(d)          (d)
+#define HTOND2(hd, nd)    ((hd) = (nd))
+#define NTOHD2(nd, hd)    ((nd) = (hd))
+
+/* float */
+#define HTONF(f)          (f)
+#define NTOHF(f)          (f)
+#define HTONF2(hf, nf)    ((nf) = (hf))
+#define NTOHF2(nf, hf)    ((hf) = (nf))
+
+/* pointers */
+#define HTONP(p)          (p)
+#define NTOHP(p)          (p)
+#define HTONP2(hp, np)    ((np) = (hp))
+#define NTOHP2(np, hp)    ((hp) = (np))
+
+/* unsigned long */
+#define HTONUL(l)         (l)
+#define NTOHUL(l)         (l)
+#define HTONUL2(hl, nl)   ((nl) = (hl))
+#define NTOHUL2(nl, hl)   ((hl) = (nl))
+
+/* signed long */
+#define HTONSL(l)         (l)
+#define NTOHSL(l)         (l)
+#define HTONSL2(hl, nl)   ((nl) = (hl))
+#define NTOHSL2(nl, hl)   ((hl) = (nl))
+
+/* unsigned int */
+#define HTONUI(i)         (i)
+#define NTOHUI(i)         (i)
+#define HTONUI2(hi, ni)   ((ni) = (hi))
+#define NTOHUI2(ni, hi)   ((hi) = (ni))
+
+/* signed int */
+#define HTONSI(i)         (i)
+#define NTOHSI(i)         (i)
+#define HTONSI2(hi, ni)   ((ni) = (hi))
+#define NTOHSI2(ni, hi)   ((hi) = (ni))
+
+/* unsigned short */
+#define HTONUS(s)         (s)
+#define NTOHUS(s)         (s)
+#define HTONUS2(hs, ns)   ((ns) = (hs))
+#define NTOHUS2(ns, hs)   ((hs) = (ns))
+
+
+
+#endif
+
+#endif /* netorder.h */
Index: /trunk/mgpp/lib/non_ansi.h
===================================================================
--- /trunk/mgpp/lib/non_ansi.h	(revision 3365)
+++ /trunk/mgpp/lib/non_ansi.h	(revision 3365)
@@ -0,0 +1,24 @@
+#if !defined(_NON_ANSI_H)
+#define _NON_ANSI_H 1
+
+#include "config.h" 
+# if !defined(__STDC__)
+
+#include <stdlib.h>  /* for FILE def */
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+int fileno(FILE *);
+int ftruncate(int, long);
+int pclose(FILE *);
+FILE *popen(const char *command, const char *type);
+int getpagesize();
+int strcasecmp(const char *, const char *);
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* !defined STDC, ... */
+#endif
Index: /trunk/mgpp/lib/pathmax.h
===================================================================
--- /trunk/mgpp/lib/pathmax.h	(revision 3365)
+++ /trunk/mgpp/lib/pathmax.h	(revision 3365)
@@ -0,0 +1,53 @@
+/* Define PATH_MAX somehow.  Requires sys/types.h.
+   Copyright (C) 1992 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifndef _PATHMAX_H
+#define _PATHMAX_H
+
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+/* Non-POSIX BSD systems might have gcc's limits.h, which doesn't define
+   PATH_MAX but might cause redefinition warnings when sys/param.h is
+   later included (as on MORE/BSD 4.3).  */
+#if defined(_POSIX_VERSION) || (defined(HAVE_LIMITS_H) && !defined(__GNUC__))
+#include <limits.h>
+#endif
+
+#ifndef _POSIX_PATH_MAX
+#define _POSIX_PATH_MAX 255
+#endif
+
+#if !defined(PATH_MAX) && defined(_PC_PATH_MAX)
+#define PATH_MAX (pathconf ("/", _PC_PATH_MAX) < 1 ? 1024 : pathconf ("/", _PC_PATH_MAX))
+#endif
+
+/* Don't include sys/param.h if it already has been.  */
+#if defined(HAVE_SYS_PARAM_H) && !defined(PATH_MAX) && !defined(MAXPATHLEN)
+#include <sys/param.h>
+#endif
+
+#if !defined(PATH_MAX) && defined(MAXPATHLEN)
+#define PATH_MAX MAXPATHLEN
+#endif
+
+#ifndef PATH_MAX
+#define PATH_MAX _POSIX_PATH_MAX
+#endif
+
+#endif /* _PATHMAX_H */
Index: /trunk/mgpp/lib/perf_hash.cpp
===================================================================
--- /trunk/mgpp/lib/perf_hash.cpp	(revision 3365)
+++ /trunk/mgpp/lib/perf_hash.cpp	(revision 3365)
@@ -0,0 +1,1074 @@
+/**************************************************************************
+ *
+ * perf_hash.c -- Perfect hashing functions
+ * Copyright (C) 1992  Bohdan S. Majewski
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************
+ *
+ * 1994 Neil Sharman
+ *      Small modifications were made to the interface so that it fitted in
+ *      with mg.
+ *
+ **************************************************************************/
+
+#define STRUCT
+
+#include <time.h>  /* for call to time () in SEED_RANDOM() */
+#include "sysfuncs.h"
+
+#include "local_strings.h"
+#include "memlib.h"
+#include "messages.h"
+#include "perf_hash.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#define FALSE 0
+#define TRUE  1
+
+#define STATIC 0
+
+/* Random Number stuff */
+
+long irandm (long is[2]);  /* from random.c */
+
+static long seed[] = {0, 0};
+#define RANDOM() irandm(seed)
+#define SEED_RANDOM(the_seed) do{ seed[0] = the_seed; }while(0)
+
+/* Max number of keys - must be a power of 2 minus 1 */
+static int MAX_M;
+
+
+static int MAX_CH;
+
+
+/* the width of the mapping tables  */
+static int MAX_L;
+
+
+
+#define c     1.23
+static int MAX_N;
+
+
+
+
+static u_char *translate;
+
+#define MEMBER 0
+#define ERASED 1
+
+
+
+/* direction constants for the edge oriented 3-graph representation */
+#define A              0
+#define B        (MAX_M)
+#define C          (2*B)
+
+/* NODEa, NODEb, NODEc, FIRST and NEXT are used to represent a 3-graph */
+static int *NODEa, *NODEb, *NODEc;
+static int *FIRST;
+static int *NEXT;
+
+
+#define norm(e) ((e) % MAX_M)
+/* in general should be ((e) % MAX_M) but MAX_M = 2^k - 1 */
+
+
+static char *mk;		/* for marking edges/vertices as removed */
+static int *g;			/* array g defines a minimal perfect hash function */
+
+
+
+/* stack of edges, created during the independency */
+/* test, used in the assignment step               */
+static struct
+  {
+    int sp;
+    int *st;
+  }
+S;
+#define push(s, e) s.st[s.sp++] = (e)
+#define pop(s, e)  e = s.st[--s.sp]
+
+#ifndef STRUCT
+static long **tb0, **tb1, **tb2;
+#else
+struct tb_entry **tb;
+#endif
+
+
+static int 
+unique (int v)
+{				/* checks if vertex v belongs to only one edge */
+  return (FIRST[v] != 0 && NEXT[FIRST[v]] == 0) ? FIRST[v] : 0;
+}				/*unique */
+
+
+
+
+
+
+
+
+static void 
+_delete (int v, int e)
+{				/* deletes edge e from list of v */
+
+  int b;
+
+  b = FIRST[v];
+  assert (norm (b) != 0);
+  if (norm (b) == e)
+    FIRST[v] = NEXT[FIRST[v]];
+  else
+    {
+      while (norm (NEXT[b]) != e)
+	{
+	  b = NEXT[b];
+	  assert (norm (b) != 0);
+	}
+      NEXT[b] = NEXT[NEXT[b]];
+    }
+}				/*delete */
+
+
+
+
+
+
+#if 0
+/* recursively removes the edges of a 3-graph which have at least */
+/* one unique vertex. The removed edges are placed on stack S     */
+static void 
+ph_remove (int e, int v)
+{
+  int b;
+  mk[e] = ERASED;
+  push (S, e);
+
+  if (NODEa[e] != v)
+    _delete (NODEa[e], e);
+  if (NODEb[e] != v)
+    _delete (NODEb[e], e);
+  if (NODEc[e] != v)
+    _delete (NODEc[e], e);
+
+  if (NODEa[e] != v && mk[b = norm (unique (NODEa[e]))] == MEMBER)
+    ph_remove (b, NODEa[e]);
+  if (NODEb[e] != v && mk[b = norm (unique (NODEb[e]))] == MEMBER)
+    ph_remove (b, NODEb[e]);
+  if (NODEc[e] != v && mk[b = norm (unique (NODEc[e]))] == MEMBER)
+    ph_remove (b, NODEc[e]);
+}
+#else
+
+
+static int StackPos, StackMax;
+static struct Stack
+  {
+    int e, v, m;
+  }
+ *Stack = NULL;
+
+#define StackAdd(_e, _v, _m)						\
+  do {									\
+    if (StackPos == StackMax)						\
+      {									\
+        StackMax <<= 1;							\
+	if (!(Stack = (struct Stack *)Xrealloc(Stack, sizeof(*Stack) * StackMax)))	\
+	  return(-1);							\
+      }									\
+    Stack[StackPos].e = (_e);						\
+    Stack[StackPos].v = (_v);						\
+    Stack[StackPos++].m = (_m);						\
+  } while(0)
+
+
+static int 
+ph_remove (int e, int v)
+{
+  if (!Stack)
+    {
+      StackMax = 256;
+      if (!(Stack = (struct Stack *) Xmalloc (sizeof (*Stack) * StackMax)))
+	return (-1);
+      StackPos = 0;
+    }
+  Stack[StackPos].e = e;
+  Stack[StackPos].v = v;
+  Stack[StackPos].m = 0;
+  StackPos++;
+  while (StackPos)
+    {
+      int m, b;
+      StackPos--;
+      e = Stack[StackPos].e;
+      v = Stack[StackPos].v;
+      m = Stack[StackPos].m;
+
+
+      switch (m)
+	{
+	case 0:
+	  {
+	    mk[e] = ERASED;
+	    push (S, e);
+
+	    if (NODEa[e] != v)
+	      _delete (NODEa[e], e);
+	    if (NODEb[e] != v)
+	      _delete (NODEb[e], e);
+	    if (NODEc[e] != v)
+	      _delete (NODEc[e], e);
+
+	    if (NODEa[e] != v && mk[b = norm (unique (NODEa[e]))] == MEMBER)
+	      {
+		Stack[StackPos++].m = 1;
+		StackAdd (b, NODEa[e], 0);
+		break;
+	      }
+	  }
+	case 1:
+	  {
+	    if (NODEb[e] != v && mk[b = norm (unique (NODEb[e]))] == MEMBER)
+	      {
+		Stack[StackPos++].m = 2;
+		StackAdd (b, NODEb[e], 0);
+		break;
+	      }
+	  }
+	case 2:
+	  if (NODEc[e] != v && mk[b = norm (unique (NODEc[e]))] == MEMBER)
+	    StackAdd (b, NODEc[e], 0);
+	}
+    }
+  return (0);
+}
+#endif
+
+
+
+
+
+/* returns TRUE if and only if set of edges created in the tree_builder */
+/* procedure form a 3-graph with independent edges. Edges of a 3-graph  */
+/* are independent iff repeated deletion of edges containing vertices   */
+/* of degree 1 results in a graph with no edges in it.                  */
+static int 
+acyclic (int m, int n)
+{
+
+  int v, e;
+  /* build list of edges for each node */
+  for (v = 1; v <= n; FIRST[v++] = 0);
+  for (e = 1; e <= m; e++)
+    {
+      NEXT[A + e] = FIRST[NODEa[e]];
+      FIRST[NODEa[e]] = A + e;
+      NEXT[B + e] = FIRST[NODEb[e]];
+      FIRST[NODEb[e]] = B + e;
+      NEXT[C + e] = FIRST[NODEc[e]];
+      FIRST[NODEc[e]] = C + e;
+
+      /* mark edge e as belonging to the hypergraph */
+      mk[e] = MEMBER;
+    }
+
+
+  S.sp = 0;			/* empty the stack */
+  mk[0] = ERASED;		/* a sentinel for the test below */
+  for (v = 1; v <= n; v++)
+    {
+      if (mk[e = norm (unique (v))] == MEMBER)
+	ph_remove (e, v);
+    }
+
+  /* check if any edges were left in the graph */
+  return S.sp == m;
+}				/*acyclic */
+
+
+
+
+
+
+
+
+/* repeatedly maps words into edges of a 3-graph until the edges */
+/* are independent.                                              */
+static int 
+tree_builder (int num, u_char ** keys, int *n)
+{
+
+  int e, k = 0;
+  int i, j;
+  u_char *w;
+
+  do
+    {
+      *n = (int) (num * c + 1);
+      /* initialize tables of random integers */
+      for (i = 0; i < MAX_CH; i++)
+	for (j = 0; j < MAX_L; j++)
+	  {
+#ifndef STRUCT
+	    tb0[i][j] = RANDOM () % *n;
+	    tb1[i][j] = RANDOM () % *n;
+	    tb2[i][j] = RANDOM () % *n;
+#else
+	    struct tb_entry *tbp = &tb[i][j];
+	    tbp->tb0 = RANDOM () % *n;
+	    tbp->tb1 = RANDOM () % *n;
+	    tbp->tb2 = RANDOM () % *n;
+#endif
+	  }
+
+      /* compute an edge (NODEa[e], NODEb[e], NODEc[e]) for each word */
+      for (e = 1; e <= num; e++)
+	{
+	  /*generate an edge corresponding to the ith word */
+	  int x = 0, y = 0, z = 0, l;
+	  w = keys[e - 1];
+
+	  l = *w++;
+	  j = (l - 1) % MAX_L;
+
+	  do
+	    {
+#ifndef STRUCT
+	      x += tb0[translate[*w]][j];
+	      y += tb1[translate[*w]][j];
+	      z += tb2[translate[*w]][j];
+#else
+	      struct tb_entry *tbp = &tb[translate[*w]][j];
+	      x += tbp->tb0;
+	      y += tbp->tb1;
+	      z += tbp->tb2;
+#endif
+	      if (++j >= MAX_L)
+		j = 0;
+	      w++;
+	    }
+	  while (--l);
+	  x = (x % *n) + 1;
+	  y = (y % *n) + 1;
+	  z = (z % *n) + 1;
+	  if (y == x && ++y > *n)
+	    y = 1;
+	  if (z == x && ++z > *n)
+	    z = 1;
+	  if (z == y)
+	    {
+	      if (++z > *n)
+		z = 1;
+	      if (z == x && ++z > *n)
+		z = 1;
+	    }
+
+	  NODEa[e] = x;
+	  NODEb[e] = y;
+	  NODEc[e] = z;
+	}
+      if (++k > 50)		/* make sure we will not wait for ever */
+	FatalError (1, "Unable to generate the perfect hash function. "
+		    "This is probably because there aren't enough words");
+    }
+  while (!acyclic (num, *n));
+  return k;
+}				/*tree_builder */
+
+
+
+
+
+
+
+
+
+/* calculates values of the g function/array so that i-th word is */
+/* placed at the (i-1)st position of a hash table.                */
+static void 
+assign (int n)
+{
+
+  int v, w = 0, e;
+
+  /* mark all the vertices of the 3-graph as unassigned */
+  for (v = 1; v <= n; mk[v++] = FALSE);
+
+  while (S.sp > 0)
+    {
+      int _xor = 0;
+      pop (S, e);
+      v = NODEa[e];
+      if (mk[v])
+	_xor ^= g[v];
+      else
+	{
+	  g[w = v] = 0;
+	  mk[v] = TRUE;
+	}
+      v = NODEb[e];
+      if (mk[v])
+	_xor ^= g[v];
+      else
+	{
+	  g[w = v] = 0;
+	  mk[v] = TRUE;
+	}
+      v = NODEc[e];
+      if (mk[v])
+	_xor ^= g[v];
+      else
+	{
+	  g[w = v] = 0;
+	  mk[v] = TRUE;
+	}
+      g[w] = (e - 1) ^ _xor;
+    }
+}				/*assign */
+
+
+
+
+
+
+
+
+
+/*check returns 0 if no error in mphf is detected, else returns error code */
+static int 
+check (int m)
+{
+
+  int e, err_code = 0;
+
+  /*for each word compute its place in hash table and check if correct */
+  for (e = 1; e <= m; e++)
+    {
+      if ((e - 1) != (g[NODEa[e]] ^ g[NODEb[e]] ^ g[NODEc[e]]))
+	{
+	  (void) fprintf (stderr, "Error for %dth word detected.\n", e);
+	  err_code = e;
+	}
+    }
+  return err_code;
+}				/*check */
+
+
+
+
+
+
+
+#if 0
+void 
+outputh (int m, int n)
+{
+
+  int i, j, w = (int) ceil (log10 ((double) n + 1.0)) + 1;
+
+#if 0
+  char *PRG =
+  "int h(char *s)\n"
+  "{\n"
+  "  int u, v, w, i;\n"
+  "i = (strlen(s) - 1) % _HXL;\n"
+  "  u = v = w = 0;\n"
+  "  while(*s)\n"
+  "    {\n"
+  "      u += mt1[(*s) - _HFC][i];\n"
+  "      v += mt2[(*s) - _HFC][i];\n"
+  "      w += mt3[(*s++) - _HFC][i];\n"
+  "      if(++i >= _HXL) i = 0;\n"
+  "    }\n"
+  "  u %= _HFN; v %= _HFN; w %= _HFN;\n"
+  "  if(u == v && ++v >= _HFN) v = 0;\n"
+  "  if(w == u && ++w >= _HFN) w = 0;\n"
+  "  if(w == v && ++w >= _HFN) w = 0;\n"
+  "  if(w == u && ++w >= _HFN) w = 0;\n"
+  "  return (g[u] ^ g[v] ^ g[w]) % _HFM;\n"
+  "};\n\n";
+#endif
+
+  char *PRG =
+  "int h(char *s)\n"
+  "{\n"
+  "  int u, v, w, i;\n"
+  "  i = (strlen(s) - 1) % _HXL;\n"
+  "  u = v = w = 0;\n"
+  "  while(*s)\n"
+  "    {\n"
+  "      if((u += mt1[(*s) - _HFC][i]) >= _HFN) u -= _HFN;\n"
+  "      if((v += mt2[(*s) - _HFC][i]) >= _HFN) v -= _HFN;\n"
+  "      if((w += mt3[(*s++) - _HFC][i]) >= _HFN) w -= _HFN;\n"
+  "      if(++i >= _HXL) i = 0;\n"
+  "    }\n"
+  "  if(u == v && ++v >= _HFN) v = 0;\n"
+  "  if(w == v) \n"
+  "    {\n"
+  "      if(++w >= _HFN) w = 0;\n"
+  "      if(w == u && ++w >= _HFN) w = 0;\n"
+  "    }\n"
+  "  else\n"
+  "    if(w == u)\n"
+  "      {\n"
+  "	     if(++w >= _HFN) w = 0;\n"
+  "	     if(w == v && ++w >= _HFN) w = 0;\n"
+  "      }\n"
+  "  return ((g[u] ^ g[v] ^ g[w]) > _HFM)? 0 : (g[u] ^ g[v] ^ g[w]);\n"
+  "}\n\n";
+
+  printf ("#define _HFN %d\n#define _HFC %d\n", n, fc);
+  printf ("#define _HFM %d\n#define _HXL %d\n", m, MAX_L);
+  printf ("#define _HFA %d\n\nstatic int g[_HFN] = {\n", lc - fc + 1);
+  for (i = 1; i < n; i++)
+    {
+      if (i % 8 == 0)
+	putchar ('\n');
+      printf ("%5d, ", g[i]);
+    }
+  printf ("%5d};\n\n", g[n]);
+
+  printf ("static int mt1[_HFA][_HXL] = {\n");
+  for (j = fc; j < lc; j++)
+    {
+      printf ("{");
+      for (i = 0; i < MAX_L - 1; i++)
+	printf ("%*d,", w, tb0[j][i]);
+      printf ("%*d},\n", w, tb0[j][MAX_L - 1]);
+    }
+  printf ("{");
+  for (i = 0; i < MAX_L - 1; i++)
+    printf ("%*d,", w, tb0[lc][i]);
+  printf ("%*d}\n", w, tb0[lc][MAX_L - 1]);
+  printf ("};\n\n");
+
+  printf ("static int mt2[_HFA][_HXL] = {\n");
+  for (j = fc; j < lc; j++)
+    {
+      printf ("{");
+      for (i = 0; i < MAX_L - 1; i++)
+	printf ("%*d,", w, tb1[j][i]);
+      printf ("%*d},\n", w, tb1[j][MAX_L - 1]);
+    }
+  printf ("{");
+  for (i = 0; i < MAX_L - 1; i++)
+    printf ("%*d,", w, tb1[lc][i]);
+  printf ("%*d}\n", w, tb1[lc][MAX_L - 1]);
+  printf ("};\n\n");
+
+  printf ("static int mt3[_HFA][_HXL] = {\n");
+  for (j = fc; j < lc; j++)
+    {
+      printf ("{");
+      for (i = 0; i < MAX_L - 1; i++)
+	printf ("%*d,", w, tb2[j][i]);
+      printf ("%*d},\n", w, tb2[j][MAX_L - 1]);
+    }
+  printf ("{");
+  for (i = 0; i < MAX_L - 1; i++)
+    printf ("%*d,", w, tb2[lc][i]);
+  printf ("%*d}\n", w, tb2[lc][MAX_L - 1]);
+  printf ("};\n\n");
+
+  printf ("%s", PRG);
+}				/*outputh */
+#endif
+
+
+
+
+
+
+
+static void 
+make_trans_func (int num, u_char ** keys)
+{
+  int i, j;
+
+  memset (translate, '\0', sizeof (translate));
+
+  MAX_L = 1;
+  for (i = 0; i < num; i++)
+    {
+      unsigned len = keys[i][0];
+      u_char *s = keys[i] + 1;
+      for (; len; len--, s++)
+	translate[*s] = 1;
+      if (i)
+	{
+	  int l;
+	  l = prefixlen (keys[i - 1], keys[i]);
+	  if (l + 1 > MAX_L)
+	    MAX_L = l + 1;
+	}
+    }
+  j = 0;
+  for (i = 0; i < 256; i++)
+    if (translate[i])
+      translate[i] = j++;
+  MAX_CH = j;
+}
+
+
+static void 
+temp_free (void)
+{
+  if (NODEa)
+    Xfree (NODEa);
+  if (NODEb)
+    Xfree (NODEb);
+  if (NODEc)
+    Xfree (NODEc);
+  if (FIRST)
+    Xfree (FIRST);
+  if (NEXT)
+    Xfree (NEXT);
+  if (S.st)
+    Xfree (S.st);
+  if (mk)
+    Xfree (mk);
+}
+
+static void 
+all_free (void)
+{
+  int i;
+  temp_free ();
+  if (translate)
+    Xfree (translate);
+  if (g)
+    Xfree (g);
+#ifndef STRUCT
+  for (i = 0; i < MAX_CH; i++)
+    {
+      if (tb0 && tb0[i])
+	Xfree (tb0[i]);
+      if (tb1 && tb1[i])
+	Xfree (tb1[i]);
+      if (tb2 && tb2[i])
+	Xfree (tb2[i]);
+    }
+  if (tb0)
+    Xfree (tb0);
+  if (tb1)
+    Xfree (tb1);
+  if (tb2)
+    Xfree (tb2);
+#else
+  for (i = 0; i < MAX_CH; i++)
+    if (tb && tb[i])
+      Xfree (tb[i]);
+  if (tb)
+    Xfree (tb);
+#endif
+}
+
+static int 
+allocate_memory (void)
+{
+  int i, ok = 1;
+  NODEa = (int *) Xmalloc (sizeof (int) * (MAX_M + 1));
+  NODEb = (int *)Xmalloc (sizeof (int) * (MAX_M + 1));
+  NODEc = (int *)Xmalloc (sizeof (int) * (MAX_M + 1));
+  FIRST = (int *)Xmalloc (sizeof (int) * (MAX_N + 1));
+  NEXT = (int *)Xmalloc (sizeof (int) * (MAX_M + 1) * 3);
+  S.st = (int *)Xmalloc (sizeof (int) * MAX_M);
+  mk = (char *)Xmalloc (sizeof (char) * (MAX_N + 1));
+  g = (int *)Xmalloc (sizeof (int) * (MAX_N + 1));
+#ifndef STRUCT
+  tb0 = Xmalloc (sizeof (int *) * MAX_CH);
+  tb1 = Xmalloc (sizeof (int *) * MAX_CH);
+  tb2 = Xmalloc (sizeof (int *) * MAX_CH);
+  for (i = 0; i < MAX_CH; i++)
+    {
+      if (tb0)
+	if (!(tb0[i] = Xmalloc (sizeof (long) * MAX_L)))
+	    ok = 0;
+      if (tb1)
+	if (!(tb1[i] = Xmalloc (sizeof (long) * MAX_L)))
+	    ok = 0;
+      if (tb2)
+	if (!(tb2[i] = Xmalloc (sizeof (long) * MAX_L)))
+	    ok = 0;
+    }
+
+  if (!(NODEa && NODEb && NODEc && FIRST && NEXT && S.st && mk && g &&
+	tb0 && tb1 && tb2 && ok))
+    {
+      all_free ();
+      return (-1);
+    }
+#else
+  tb = (tb_entry **)Xmalloc (sizeof (struct tb_entry *) * MAX_CH);
+  for (i = 0; i < MAX_CH; i++)
+    if (tb)
+      if (!(tb[i] = (tb_entry *)Xmalloc (sizeof (struct tb_entry) * MAX_L)))
+	  ok = 0;
+
+  if (!(NODEa && NODEb && NODEc && FIRST && NEXT && S.st && mk && g &&
+	tb && ok))
+    {
+      all_free ();
+      return (-1);
+    }
+#endif
+  return (0);
+}
+
+
+perf_hash_data *
+gen_hash_func (int num, u_char ** keys, int r)
+{
+  int n;
+  perf_hash_data *phd;
+  translate = (u_char *)Xmalloc (sizeof (u_char) * 256);
+  memset(translate, '\0', sizeof (u_char) * 256);  /* [RPAP - Feb 97: WIN32 Port] */
+  make_trans_func (num, keys);
+
+  if (r <= 0)
+    SEED_RANDOM ((long) time ((time_t *) NULL));
+  else
+    SEED_RANDOM (r);
+
+  MAX_M = num + 1;
+
+  MAX_N = MAX_M + MAX_M / 4;
+
+  if (allocate_memory () == -1)
+    return (NULL);
+
+  /* construct an "acyclic" hypergraph */
+  tree_builder (num, keys, &n);
+  /* compute function g */
+  assign (n);
+
+  if (check (num) != 0)
+    FatalError (1, "An error has been detected in the mphf.");
+  temp_free ();
+
+  if (!(phd = (perf_hash_data *)Xmalloc (sizeof (perf_hash_data))))
+    return (NULL);
+
+  memcpy ((void *) g, (const void *) &g[1], n * sizeof (int));
+
+  phd->MAX_L = MAX_L;
+  phd->MAX_N = n;
+  phd->MAX_M = num;
+  phd->MAX_CH = MAX_CH;
+  phd->translate = translate;
+  phd->g = g;
+#ifndef STRUCT
+  phd->tb0 = tb0;
+  phd->tb1 = tb1;
+  phd->tb2 = tb2;
+#else
+  phd->tb = tb;
+#endif
+  return (phd);
+}
+
+int 
+write_perf_hash_data (FILE * f, perf_hash_data * phd)
+{
+  int tot, i;
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONSI(phd->MAX_L);
+  HTONSI(phd->MAX_N);
+  HTONSI(phd->MAX_M);
+  HTONSI(phd->MAX_CH);
+
+  tot = fwrite ((char *) &phd->MAX_L, sizeof (int), 1, f) * sizeof (int);
+  tot += fwrite ((char *) &phd->MAX_N, sizeof (int), 1, f) * sizeof (int);
+  tot += fwrite ((char *) &phd->MAX_M, sizeof (int), 1, f) * sizeof (int);
+  tot += fwrite ((char *) &phd->MAX_CH, sizeof (int), 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(phd->MAX_L);
+  NTOHSI(phd->MAX_N);
+  NTOHSI(phd->MAX_M);
+  NTOHSI(phd->MAX_CH);
+
+  tot += fwrite ((char *) phd->translate, sizeof (u_char), 256, f);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    HTONSI(phd->g[i]);
+
+  tot += fwrite ((char *) phd->g, sizeof (int), phd->MAX_N + 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    NTOHSI(phd->g[i]);
+
+#ifndef STRUCT
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      /* [RPAP - Jan 97: Endian Ordering] */
+      int j;
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  HTONSI(phd->tb0[i][j]);
+	  HTONSI(phd->tb1[i][j]);
+	  HTONSI(phd->tb2[i][j]);
+	}
+
+      tot += fwrite ((char *) phd->tb0[i], sizeof (int), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fwrite ((char *) phd->tb1[i], sizeof (int), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fwrite ((char *) phd->tb2[i], sizeof (int), phd->MAX_L, f) *
+        sizeof (int);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSI(phd->tb0[i][j]);
+	  NTOHSI(phd->tb1[i][j]);
+	  NTOHSI(phd->tb2[i][j]);
+	}
+    }
+#else
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      /* [RPAP - Jan 97: Endian Ordering] */
+      int j;
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  HTONSL(phd->tb[i][j].tb0);
+	  HTONSL(phd->tb[i][j].tb1);
+	  HTONSL(phd->tb[i][j].tb2);
+	}
+
+      tot += fwrite ((char *) phd->tb[i], sizeof (struct tb_entry), phd->MAX_L, f) *
+	sizeof (struct tb_entry);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSL(phd->tb[i][j].tb0);
+	  NTOHSL(phd->tb[i][j].tb1);
+	  NTOHSL(phd->tb[i][j].tb2);
+	}
+    }
+#endif
+  return (tot);
+}
+
+
+perf_hash_data *
+read_perf_hash_data (FILE * f)
+{
+  perf_hash_data *phd;
+  int i, tot, ok = 1;
+  if (!(phd = (perf_hash_data *)Xmalloc (sizeof (perf_hash_data))))
+    return (NULL);
+  tot = fread ((char *) &phd->MAX_L, sizeof (int), 1, f) * sizeof (int);
+  tot += fread ((char *) &phd->MAX_N, sizeof (int), 1, f) * sizeof (int);
+  tot += fread ((char *) &phd->MAX_M, sizeof (int), 1, f) * sizeof (int);
+  tot += fread ((char *) &phd->MAX_CH, sizeof (int), 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHSI(phd->MAX_L);
+  NTOHSI(phd->MAX_N);
+  NTOHSI(phd->MAX_M);
+  NTOHSI(phd->MAX_CH);
+
+  if (tot != 4 * sizeof (int))
+      return (NULL);
+  phd->translate = (u_char *)Xmalloc (sizeof (u_char) * 256);
+  phd->g = (int *)Xmalloc (sizeof (int) * (phd->MAX_N + 1));
+#ifndef STRUCT
+  phd->tb0 = Xmalloc (sizeof (int *) * phd->MAX_CH);
+  phd->tb1 = Xmalloc (sizeof (int *) * phd->MAX_CH);
+  phd->tb2 = Xmalloc (sizeof (int *) * phd->MAX_CH);
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      if (phd->tb0)
+	if (!(phd->tb0[i] = Xmalloc (sizeof (long) * phd->MAX_L)))
+	    ok = 0;
+      if (phd->tb1)
+	if (!(phd->tb1[i] = Xmalloc (sizeof (long) * phd->MAX_L)))
+	    ok = 0;
+      if (phd->tb2)
+	if (!(phd->tb2[i] = Xmalloc (sizeof (long) * phd->MAX_L)))
+	    ok = 0;
+    }
+  if (!(phd->translate && phd->g && phd->tb0 && phd->tb1 && phd->tb2 && ok))
+    {
+      if (phd->translate)
+	Xfree (phd->translate);
+      if (phd->g)
+	Xfree (phd->g);
+      for (i = 0; i < MAX_CH; i++)
+	{
+	  if (phd->tb0 && phd->tb0[i])
+	    Xfree (phd->tb0[i]);
+	  if (phd->tb1 && phd->tb1[i])
+	    Xfree (phd->tb1[i]);
+	  if (phd->tb2 && phd->tb2[i])
+	    Xfree (phd->tb2[i]);
+	}
+      if (phd->tb0)
+	Xfree (phd->tb0);
+      if (phd->tb1)
+	Xfree (phd->tb1);
+      if (phd->tb2)
+	Xfree (phd->tb2);
+      Xfree (phd);
+      return (NULL);
+    }
+  tot += fread ((char *) phd->translate, sizeof (u_char), 256, f);
+  tot += fread ((char *) phd->g, sizeof (int), phd->MAX_N + 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    NTOHSI(phd->g[i]);
+
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      int j;
+
+      tot += fread ((char *) phd->tb0[i], sizeof (long), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fread ((char *) phd->tb1[i], sizeof (long), phd->MAX_L, f) *
+        sizeof (int);
+      tot += fread ((char *) phd->tb2[i], sizeof (long), phd->MAX_L, f) *
+        sizeof (int);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSI(phd->tb0[i][j]);
+	  NTOHSI(phd->tb1[i][j]);
+	  NTOHSI(phd->tb2[i][j]);
+	}
+    }
+#else
+  phd->tb = (struct tb_entry **)Xmalloc (sizeof (struct tb_entry *) * phd->MAX_CH);
+  for (i = 0; i < phd->MAX_CH; i++)
+    if (phd->tb)
+      if (!(phd->tb[i] = (struct tb_entry *)Xmalloc (sizeof (struct tb_entry) * phd->MAX_L)))
+	  ok = 0;
+  if (!(phd->translate && phd->g && phd->tb && ok))
+    {
+      if (phd->translate)
+	Xfree (phd->translate);
+      if (phd->g)
+	Xfree (phd->g);
+      for (i = 0; i < MAX_CH; i++)
+	if (phd->tb && phd->tb[i])
+	  Xfree (phd->tb[i]);
+      if (phd->tb)
+	Xfree (phd->tb);
+      Xfree (phd);
+      return (NULL);
+    }
+  tot += fread ((char *) phd->translate, sizeof (u_char), 256, f);
+  tot += fread ((char *) phd->g, sizeof (int), phd->MAX_N + 1, f) * sizeof (int);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = 0; i < phd->MAX_N + 1; i++)
+    NTOHSI(phd->g[i]);
+
+  for (i = 0; i < phd->MAX_CH; i++)
+    {
+      int j;
+
+      tot += fread ((char *) phd->tb[i], sizeof (struct tb_entry), phd->MAX_L, f) *
+	sizeof (struct tb_entry);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      for (j = 0; j < phd->MAX_L; j++)
+	{
+	  NTOHSL(phd->tb[i][j].tb0);
+	  NTOHSL(phd->tb[i][j].tb1);
+	  NTOHSL(phd->tb[i][j].tb2);
+	}
+    }
+#endif
+  return (phd);
+}
+
+void 
+free_perf_hash (perf_hash_data * phd)
+{
+  int i;
+  if (phd->translate)
+    Xfree (phd->translate);
+  if (phd->g)
+    Xfree (phd->g);
+  for (i = 0; i < phd->MAX_CH; i++)
+    if (phd->tb && phd->tb[i])
+      Xfree (phd->tb[i]);
+  if (phd->tb)
+    Xfree (phd->tb);
+  Xfree (phd);
+}
+
+
+int 
+perf_hash (perf_hash_data * phd, u_char * s)
+{
+  int u, v, w, i, l;
+  l = *s++;
+  i = (l - 1) % phd->MAX_L;
+  u = v = w = 0;
+  while (l)
+    {
+#ifndef STRUCT
+      if ((u += phd->tb0[phd->translate[*s]][i]) >= phd->MAX_N)
+	u -= phd->MAX_N;
+      if ((v += phd->tb1[phd->translate[*s]][i]) >= phd->MAX_N)
+	v -= phd->MAX_N;
+      if ((w += phd->tb2[phd->translate[*s]][i]) >= phd->MAX_N)
+	w -= phd->MAX_N;
+#else
+      struct tb_entry *tbp = &phd->tb[phd->translate[*s]][i];
+      if ((u += tbp->tb0) >= phd->MAX_N)
+	u -= phd->MAX_N;
+      if ((v += tbp->tb1) >= phd->MAX_N)
+	v -= phd->MAX_N;
+      if ((w += tbp->tb2) >= phd->MAX_N)
+	w -= phd->MAX_N;
+#endif
+      if (++i >= phd->MAX_L)
+	i = 0;
+      l--;
+      s++;
+    }
+  if (u == v && ++v >= phd->MAX_N)
+    v = 0;
+  if (w == v)
+    {
+      if (++w >= phd->MAX_N)
+	w = 0;
+      if (w == u && ++w >= phd->MAX_N)
+	w = 0;
+    }
+  else if (w == u)
+    {
+      if (++w >= phd->MAX_N)
+	w = 0;
+      if (w == v && ++w >= phd->MAX_N)
+	w = 0;
+    }
+  return ((phd->g[u] ^ phd->g[v] ^ phd->g[w]) > phd->MAX_M) ? 0 :
+    (phd->g[u] ^ phd->g[v] ^ phd->g[w]);
+}
Index: /trunk/mgpp/lib/perf_hash.h
===================================================================
--- /trunk/mgpp/lib/perf_hash.h	(revision 3365)
+++ /trunk/mgpp/lib/perf_hash.h	(revision 3365)
@@ -0,0 +1,52 @@
+/**************************************************************************
+ *
+ * perf_hash.h -- Perfect hashing functions
+ * Copyright (C) 1994  Bohdan S. Majewski and Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_PERF_HASH
+#define H_PERF_HASH
+
+struct tb_entry
+  {
+    long tb0, tb1, tb2;
+  };
+
+typedef struct
+  {
+    int MAX_L;
+    int MAX_N;
+    int MAX_M;
+    int MAX_CH;
+    u_char *translate;
+    int *g;
+    struct tb_entry **tb;
+  }
+perf_hash_data;
+
+perf_hash_data *gen_hash_func (int num, u_char ** keys, int r);
+
+int perf_hash (perf_hash_data * phd, u_char * s);
+
+int write_perf_hash_data (FILE * f, perf_hash_data * phd);
+
+void free_perf_hash (perf_hash_data * phd);
+
+perf_hash_data *read_perf_hash_data (FILE * f);
+
+#endif
Index: /trunk/mgpp/lib/random.cpp
===================================================================
--- /trunk/mgpp/lib/random.cpp	(revision 3365)
+++ /trunk/mgpp/lib/random.cpp	(revision 3365)
@@ -0,0 +1,211 @@
+/**************************************************************************
+ *
+ * random.c -- pseudo random number generator
+ * Copyright (C) 1994  Chris Wallace (csw@bruce.cs.monash.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+/*
+ *	A random number generator called as a function by
+ *	random (iseed)	or	irandm (iseed)
+ *	The parameter should be a pointer to a 2-element long vector.
+ *	The first function returns a double uniform in 0 .. 1.
+ *	The second returns a long integer uniform in 0 .. 2**31-1
+ *	Both update iseed[] in exactly the same way.
+ *	iseed[] must be a 2-element integer vector.
+ *	The initial value of the second element may be anything.
+ *
+ *	The period of the random sequence is 2**32 * (2**32-1)
+ *	The table mt[0:127] is defined by mt[i] = 69069 ** (128-i)
+ */
+
+#define MASK ((long) 593970775)
+/*	or in hex, 23674657	*/
+
+#define SCALE ((double) 1.0 / (1024.0 * 1024.0 * 1024.0 * 2.0))
+/*	i.e. 2 to power -31	*/
+
+static long mt [128] =   {
+      902906369,
+     2030498053,
+     -473499623,
+     1640834941,
+      723406961,
+     1993558325,
+     -257162999,
+    -1627724755,
+      913952737,
+      278845029,
+     1327502073,
+    -1261253155,
+      981676113,
+    -1785280363,
+     1700077033,
+      366908557,
+    -1514479167,
+     -682799163,
+      141955545,
+     -830150595,
+      317871153,
+     1542036469,
+     -946413879,
+    -1950779155,
+      985397153,
+      626515237,
+      530871481,
+      783087261,
+    -1512358895,
+     1031357269,
+    -2007710807,
+    -1652747955,
+    -1867214463,
+      928251525,
+     1243003801,
+    -2132510467,
+     1874683889,
+     -717013323,
+      218254473,
+    -1628774995,
+    -2064896159,
+       69678053,
+      281568889,
+    -2104168611,
+     -165128239,
+     1536495125,
+      -39650967,
+      546594317,
+     -725987007,
+     1392966981,
+     1044706649,
+      687331773,
+    -2051306575,
+     1544302965,
+     -758494647,
+    -1243934099,
+      -75073759,
+      293132965,
+    -1935153095,
+      118929437,
+      807830417,
+    -1416222507,
+    -1550074071,
+      -84903219,
+     1355292929,
+     -380482555,
+    -1818444007,
+     -204797315,
+      170442609,
+    -1636797387,
+      868931593,
+     -623503571,
+     1711722209,
+      381210981,
+     -161547783,
+     -272740131,
+    -1450066095,
+     2116588437,
+     1100682473,
+      358442893,
+    -1529216831,
+     2116152005,
+     -776333095,
+     1265240893,
+     -482278607,
+     1067190005,
+      333444553,
+       86502381,
+      753481377,
+       39000101,
+     1779014585,
+      219658653,
+     -920253679,
+     2029538901,
+     1207761577,
+    -1515772851,
+     -236195711,
+      442620293,
+      423166617,
+    -1763648515,
+     -398436623,
+    -1749358155,
+     -538598519,
+     -652439379,
+      430550625,
+    -1481396507,
+     2093206905,
+    -1934691747,
+     -962631983,
+     1454463253,
+    -1877118871,
+     -291917555,
+    -1711673279,
+      201201733,
+     -474645415,
+      -96764739,
+    -1587365199,
+     1945705589,
+     1303896393,
+     1744831853,
+      381957665,
+     2135332261,
+      -55996615,
+    -1190135011,
+     1790562961,
+    -1493191723,
+      475559465,
+          69069
+		};
+
+double 
+random (long is [2])
+{
+	long it, leh, nit;
+
+	it = is [0];
+	leh = is [1];
+	if (it <= 0)	
+		it = (it + it) ^ MASK;
+	else
+		it = it + it;
+	nit = it - 1;
+/*	to ensure all-ones pattern omitted    */
+	leh = leh * mt[nit & 127] + nit;
+	is [0] = it;    is [1] = leh;
+	if (leh < 0) leh = ~leh;
+	return (SCALE * ((long) (leh | 1)));
+}
+
+
+
+long 
+irandm (long is [2])
+{
+	long it, leh, nit;
+
+	it = is [0];
+	leh = is [1];
+	if (it <= 0)	
+		it = (it + it) ^ MASK;
+	else
+		it = it + it;
+	nit = it - 1;
+/*	to ensure all-ones pattern omitted    */
+	leh = leh * mt[nit & 127] + nit;
+	is [0] = it;    is [1] = leh;
+	if (leh < 0) leh = ~leh;
+	return (leh);
+}
Index: /trunk/mgpp/lib/random.h
===================================================================
--- /trunk/mgpp/lib/random.h	(revision 3365)
+++ /trunk/mgpp/lib/random.h	(revision 3365)
@@ -0,0 +1,42 @@
+/**************************************************************************
+ *
+ * random.h -- pseudo random number generator
+ * Copyright (C) 1994  Chris Wallace (csw@bruce.cs.monash.edu.au)
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef RANDOM_H
+#define RANDOM_H
+
+/*
+ *	A random number generator called as a function by
+ *	random (iseed)	or	irandm (iseed)
+ *	The parameter should be a pointer to a 2-element long vector.
+ *	The first function returns a double uniform in 0 .. 1.
+ *	The second returns a long integer uniform in 0 .. 2**31-1
+ *	Both update iseed[] in exactly the same way.
+ *	iseed[] must be a 2-element integer vector.
+ *	The initial value of the second element may be anything.
+ *
+ *	The period of the random sequence is 2**32 * (2**32-1)
+ *	The table mt[0:127] is defined by mt[i] = 69069 ** (128-i)
+ */
+
+double random (long is [2]);
+long irandm (long is [2]);
+
+#endif
Index: /trunk/mgpp/lib/simplefrenchstem.cpp
===================================================================
--- /trunk/mgpp/lib/simplefrenchstem.cpp	(revision 3365)
+++ /trunk/mgpp/lib/simplefrenchstem.cpp	(revision 3365)
@@ -0,0 +1,70 @@
+/**************************************************************************
+ *
+ * simplefrenchstem.c -- a simple french stemmer
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "simplefrenchstem.h"
+#include "unitool.h"
+
+
+/* =========================================================================
+ * Function: simplefrenchstem
+ * Description: a simple french stemmer
+ * Input: a word string with the length in the first byte
+ * Output: the stemmed word
+ * ========================================================================= */
+
+void simplefrenchstem (unsigned char *word) {      
+  unsigned short out[256]; /* temp space */
+  unsigned short *wordstart; /* points to first letter of word */
+  int last;                  /* last points to the last character */
+
+  /* decode */
+  utf8_word_to_unicode (word, out, 255);
+  wordstart = out + 1;
+  last = out[0]-1;
+  
+
+  if (last > 4) {
+    if (wordstart[last]=='x') {
+      if (wordstart[last-1]=='u' && wordstart[last-2]=='a') {
+	wordstart[last-1]='l';
+      }
+      last--;
+
+    } else {
+      if (last>=0 && wordstart[last]=='s') last--;
+      if (last>=0 && wordstart[last]=='r') last--;
+      if (last>=0 && wordstart[last]=='e') last--;
+
+      /* letter with accent e + ' -- there are two possible encodings  */
+      if (last>=0 && wordstart[last]==0xe9) { 
+	last--;
+      } else if (last>=1 && wordstart[last-1]=='e' && wordstart[last]==0x301) {
+	last -= 2; 
+      }
+
+      if (last >= 1 && wordstart[last]==wordstart[last-1]) last--;
+    }  /* end else */
+
+    out[0] = (unsigned char)(last+1);
+  } /* end if (len > 4) */
+
+  /* re-code, make sure the result is not longer than the input */
+  unicode_to_utf8_word (out, word, word[0]+1);
+}
Index: /trunk/mgpp/lib/simplefrenchstem.h
===================================================================
--- /trunk/mgpp/lib/simplefrenchstem.h	(revision 3365)
+++ /trunk/mgpp/lib/simplefrenchstem.h	(revision 3365)
@@ -0,0 +1,26 @@
+/**************************************************************************
+ *
+ * simplefrenchstem.h -- a simple french stemmer
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef SIMPLEFRENCHSTEM_H
+#define SIMPLEFRENCHSTEM_H
+
+void simplefrenchstem (unsigned char *word);
+
+#endif
Index: /trunk/mgpp/lib/strcasecmp.cpp
===================================================================
--- /trunk/mgpp/lib/strcasecmp.cpp	(revision 3365)
+++ /trunk/mgpp/lib/strcasecmp.cpp	(revision 3365)
@@ -0,0 +1,73 @@
+/*
+ * Copyright (c) 1987 Regents of the University of California.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms are permitted
+ * provided that this notice is preserved and that due credit is given
+ * to the University of California at Berkeley. The name of the University
+ * may not be used to endorse or promote products derived from this
+ * software without specific written prior permission. This software
+ * is provided ``as is'' without express or implied warranty.
+ */
+
+#if defined(LIBC_SCCS) && !defined(lint)
+static char sccsid[] = "@(#)strcasecmp.c	5.5 (Berkeley) 11/24/87";
+#endif /* LIBC_SCCS and not lint */
+
+/*
+ * This array is designed for mapping upper and lower case letter
+ * together for a case independent comparison.  The mappings are
+ * based upon ascii character sequences.
+ */
+static unsigned char charmap[] = {
+	'\000', '\001', '\002', '\003', '\004', '\005', '\006', '\007',
+	'\010', '\011', '\012', '\013', '\014', '\015', '\016', '\017',
+	'\020', '\021', '\022', '\023', '\024', '\025', '\026', '\027',
+	'\030', '\031', '\032', '\033', '\034', '\035', '\036', '\037',
+	'\040', '\041', '\042', '\043', '\044', '\045', '\046', '\047',
+	'\050', '\051', '\052', '\053', '\054', '\055', '\056', '\057',
+	'\060', '\061', '\062', '\063', '\064', '\065', '\066', '\067',
+	'\070', '\071', '\072', '\073', '\074', '\075', '\076', '\077',
+	'\100', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\133', '\134', '\135', '\136', '\137',
+	'\140', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\173', '\174', '\175', '\176', '\177',
+	'\200', '\201', '\202', '\203', '\204', '\205', '\206', '\207',
+	'\210', '\211', '\212', '\213', '\214', '\215', '\216', '\217',
+	'\220', '\221', '\222', '\223', '\224', '\225', '\226', '\227',
+	'\230', '\231', '\232', '\233', '\234', '\235', '\236', '\237',
+	'\240', '\241', '\242', '\243', '\244', '\245', '\246', '\247',
+	'\250', '\251', '\252', '\253', '\254', '\255', '\256', '\257',
+	'\260', '\261', '\262', '\263', '\264', '\265', '\266', '\267',
+	'\270', '\271', '\272', '\273', '\274', '\275', '\276', '\277',
+	'\300', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\333', '\334', '\335', '\336', '\337',
+	'\340', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\373', '\374', '\375', '\376', '\377',
+};
+
+int
+strcasecmp(char *s1, char *s2)
+{
+    register unsigned char u1, u2;
+
+    for (;;) {
+	u1 = (unsigned char) *s1++;
+	u2 = (unsigned char) *s2++;
+	if (charmap[u1] != charmap[u2]) {
+	    return charmap[u1] - charmap[u2];
+	}
+	if (u1 == '\0') {
+	    return 0;
+	}
+    }
+}
+
Index: /trunk/mgpp/lib/strstr.cpp
===================================================================
--- /trunk/mgpp/lib/strstr.cpp	(revision 3365)
+++ /trunk/mgpp/lib/strstr.cpp	(revision 3365)
@@ -0,0 +1,42 @@
+/* strstr.c -- return the offset of one string within another
+   Copyright (C) 1989, 1990 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+/* Written by Mike Rendell <michael@cs.mun.ca>.  */
+
+/* Return the starting address of string S2 in S1;
+   return 0 if it is not found. */
+
+char *
+strstr (char *s1, char *s2)
+{
+  int i;
+  char *p1;
+  char *p2;
+  char *s = s1;
+
+  for (p2 = s2, i = 0; *s; p2 = s2, i++, s++)
+    {
+      for (p1 = s; *p1 && *p2 && *p1 == *p2; p1++, p2++)
+	;
+      if (!*p2)
+	break;
+    }
+  if (!*p2)
+    return s1 + i;
+
+  return 0;
+}
Index: /trunk/mgpp/lib/sysfuncs.cpp
===================================================================
--- /trunk/mgpp/lib/sysfuncs.cpp	(revision 3365)
+++ /trunk/mgpp/lib/sysfuncs.cpp	(revision 3365)
@@ -0,0 +1,6 @@
+#include <stdlib.h>
+
+void *bcopy(void *s, void *d, int n)
+{ return memcpy(d, s, n);
+}
+
Index: /trunk/mgpp/lib/sysfuncs.h
===================================================================
--- /trunk/mgpp/lib/sysfuncs.h	(revision 3365)
+++ /trunk/mgpp/lib/sysfuncs.h	(revision 3365)
@@ -0,0 +1,388 @@
+/* System dependent definitions for GNU tar.
+   Copyright (C) 1994 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ */
+
+/*
+ * [TS:Aug/95]
+ * Based on code from system.h file in GNU's tar package 
+ *
+ */
+
+#ifndef SYSFUNCS_H
+#define SYSFUNCS_H
+
+#include <sys/stat.h>
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__
+#  include <WIN32cfg.h>
+# else
+#  include <config.h>
+# endif
+#endif
+
+/* Declare alloca.  AIX requires this to be the first thing in the file.  */
+
+#ifdef __GNUC__
+#ifndef alloca
+#define alloca __builtin_alloca
+#endif
+#else
+#if HAVE_ALLOCA_H
+#include <alloca.h>
+#else
+#ifdef _AIX
+#pragma alloca
+#else
+#ifndef alloca
+char *alloca ();
+#endif
+#endif
+#endif
+#endif
+
+#include <sys/types.h>
+
+/* ------------------------------------------------- */
+/* [TS:Aug/95]
+ * Some extra include files that mg needs
+ */
+#include <stdlib.h>
+#include <math.h>
+
+#ifndef log2
+# ifndef M_LN2
+#  define M_LN2 0.69314718055994530942
+# endif
+# define log2(x) (log(x)/M_LN2)
+#endif
+
+
+#ifndef HAVE_SETBUFFER
+#ifdef __CYGWIN__ 
+/* Cygwin has setbuffer in stdio.h, but can't link against it!! */
+#include <stdio.h>
+#endif
+# define setbuffer(f, b, s)  setvbuf(f, b, _IOFBF, s)
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#if defined(__MSDOS__)  && !defined(__WIN32__)
+# define SHORT_SUFFIX 1
+#endif
+
+
+
+#include <assert.h>
+
+/* ------------------------------------------------- */
+
+/* Declare a generic pointer type.  */
+#if __STDC__ || defined(__TURBOC__)
+#define voidstar void *
+#else
+#define voidstar char *
+#endif
+
+/* Declare ISASCII.  */
+
+#include <ctype.h>
+
+#if STDC_HEADERS
+#define ISASCII(Char) 1
+#else
+#ifdef isascii
+#define ISASCII(Char) isascii (Char)
+#else
+#if HAVE_ISASCII
+#define ISASCII(Char) isascii (Char)
+#else
+#define ISASCII(Char) 1
+#endif
+#endif
+#endif
+
+
+/* ------------------------------------------------- */
+/* [TS:Aug/95] 
+ * The following is copied from GNU Autoconf.Info (1.1)   
+ * It sets up the approriate string and memory functions.
+ */
+
+#if STDC_HEADERS || HAVE_STRING_H
+#include <string.h>
+   /* An ANSI string.h and pre-ANSI memory.h might conflict */
+#if !STDC_HEADERS && HAVE_MEMORY_H
+#include <memory.h>
+#endif /* not STDC_HEADERS and HAVE_MEMORY_H */
+#ifndef index
+/*#define index strchr 
+ *
+ * this is stuffing up our code so its commented out */
+#endif
+#ifndef rindex
+#define rindex strrchr
+#endif
+
+#ifdef __cplusplus /* compiling for C++ */
+#include <stdlib.h>
+#ifndef __bcopy
+inline void *bcopy(void *s, void *d, int n)
+{ return memcpy(d, s, n);
+}
+#define __bcopy 1
+#endif
+#ifndef __bcmp
+inline int bcmp(void *s1, void *s2, int n)
+{ return memcmp(s1, s2, n);
+}
+#define __bcmp 1
+#endif
+#ifndef __bzero
+inline void bzero(void *m, int n)
+{ memset(m, 0, n);
+}
+#define __bzero 1
+#endif
+#else              /* compiling for C */
+#ifndef bcopy
+#define bcopy(s, d, n) memcpy((d), (s), (n))
+#endif
+#ifndef bcmp
+#define bcmp(s1, s2, n) memcmp((s1), (s2), (n))
+#endif
+#ifndef bzero
+#define bzero(s, n) memset((s), 0, (n))
+#endif
+#endif
+
+#else /* not STDC_HEADERS and not HAVE_STRING_H */
+#include <strings.h>
+   /* memory.h and strings.h conflict on some systems */
+#endif
+/* ------------------------------------------------- */
+
+
+/* Declare errno.  */
+
+#include <errno.h>
+#ifndef errno
+extern int errno;
+#endif
+
+/* Declare open parameters.  */
+
+#ifdef HAVE_FCNTL_H
+#include <fcntl.h>
+#else
+#include <sys/file.h>
+#endif
+
+#ifndef	O_BINARY
+#define O_BINARY 0
+#endif
+#ifndef O_CREAT
+#define O_CREAT 0
+#endif
+#ifndef	O_NDELAY
+#define O_NDELAY 0
+#endif
+#ifndef	O_RDONLY
+#define O_RDONLY 0
+#endif
+#ifndef O_RDWR
+#define O_RDWR 2
+#endif
+
+/* Declare file status routines and bits.  */
+
+#ifdef STAT_MACROS_BROKEN
+#undef S_ISBLK
+#undef S_ISCHR
+#undef S_ISDIR
+#undef S_ISFIFO
+#undef S_ISLNK
+#undef S_ISMPB
+#undef S_ISMPC
+#undef S_ISNWK
+#undef S_ISREG
+#undef S_ISSOCK
+#endif
+
+/* On MSDOS, there are missing things from <sys/stat.h>.  */
+#ifdef __MSDOS__
+#define S_ISUID 0
+#define S_ISGID 0
+#define S_ISVTX 0
+#endif
+
+#ifndef S_ISREG			/* POSIX.1 stat stuff missing */
+#define mode_t unsigned short
+#endif
+#if !defined(S_ISBLK) && defined(S_IFBLK)
+#define S_ISBLK(Mode) (((Mode) & S_IFMT) == S_IFBLK)
+#endif
+#if !defined(S_ISCHR) && defined(S_IFCHR)
+#define S_ISCHR(Mode) (((Mode) & S_IFMT) == S_IFCHR)
+#endif
+#if !defined(S_ISDIR) && defined(S_IFDIR)
+#define S_ISDIR(Mode) (((Mode) & S_IFMT) == S_IFDIR)
+#endif
+#if !defined(S_ISREG) && defined(S_IFREG)
+#define S_ISREG(Mode) (((Mode) & S_IFMT) == S_IFREG)
+#endif
+#if !defined(S_ISFIFO) && defined(S_IFIFO)
+#define S_ISFIFO(Mode) (((Mode) & S_IFMT) == S_IFIFO)
+#endif
+#if !defined(S_ISLNK) && defined(S_IFLNK)
+#define S_ISLNK(Mode) (((Mode) & S_IFMT) == S_IFLNK)
+#endif
+#if !defined(S_ISSOCK) && defined(S_IFSOCK)
+#define S_ISSOCK(Mode) (((Mode) & S_IFMT) == S_IFSOCK)
+#endif
+#if !defined(S_ISMPB) && defined(S_IFMPB)	/* V7 */
+#define S_ISMPB(Mode) (((Mode) & S_IFMT) == S_IFMPB)
+#define S_ISMPC(Mode) (((Mode) & S_IFMT) == S_IFMPC)
+#endif
+#if !defined(S_ISNWK) && defined(S_IFNWK)	/* HP/UX */
+#define S_ISNWK(Mode) (((Mode) & S_IFMT) == S_IFNWK)
+#endif
+
+#if !defined(S_ISCTG) && defined(S_IFCTG)	/* contiguous file */
+#define S_ISCTG(Mode) (((Mode) & S_IFMT) == S_IFCTG)
+#endif
+#if !defined(S_ISVTX)
+#define S_ISVTX 0001000
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#if !defined(_POSIX_SOURCE) && !defined(__WIN32__)
+#include <sys/param.h>
+#endif
+
+/* Include <unistd.h> before any preprocessor test of _POSIX_VERSION.  */
+#ifdef HAVE_UNISTD_H
+#include <unistd.h>
+#endif
+
+
+/* Declare standard functions.  */
+
+#ifdef STDC_HEADERS
+#include <stdlib.h>
+#else
+voidstar malloc ();
+voidstar realloc ();
+char *getenv ();
+#endif
+
+#include <stdio.h>
+
+#ifndef _POSIX_VERSION
+#ifdef __MSDOS__
+#include <io.h>
+#else
+off_t lseek ();
+#endif
+#endif
+
+#ifndef __WIN32__
+#include <pathmax.h>
+#endif
+
+/* Until getoptold function is declared in getopt.h, we need this here for
+   struct option.  */
+/*#include <getopt.h>*/
+
+#ifdef WITH_DMALLOC
+#undef HAVE_VALLOC
+#define DMALLOC_FUNC_CHECK
+#include <dmalloc.h>
+#endif
+
+/* Prototypes for external functions.  */
+
+#ifndef __P
+#if PROTOTYPES
+#define __P(Args) Args
+#else
+#define __P(Args) ()
+#endif
+#endif
+
+#if HAVE_LOCALE_H
+#include <locale.h>
+#endif
+/*** setlocale is ansi and posix...(jrm21)
+#if !HAVE_SETLOCALE
+#define setlocale(Category, Locale)
+#endif
+***/
+
+#if ENABLE_NLS
+#include <libintl.h>
+#define _(Text) gettext (Text)
+#else
+#define textdomain(Domain)
+#define _(Text) Text
+#endif
+
+/* Library modules.  */
+
+#ifdef HAVE_VPRINTF
+void error __P ((int, int, const char *,...));
+#else
+void error ();
+#endif
+
+#ifndef HAVE_STRSTR
+char *strstr __P ((const char *, const char *));
+#endif
+
+/* [RB/TS:Oct/95] commented out. Used by lib/gmalloc.c - but it defines it anyway ???
+ *#ifndef HAVE_VALLOC
+ *#define valloc(Size) malloc (Size)
+ *#endif
+ */
+
+voidstar xmalloc __P ((size_t));
+voidstar xrealloc __P ((voidstar, size_t));
+char *xstrdup __P ((const char *));
+
+/* [TS:Aug/95] 
+ * These were required by files which referred 
+ * to these as function pointers. 
+ * e.g. huffman.c - fread,  mgfelics.c - fgetc
+ */
+#ifndef HAVE_FREAD_DECL
+extern size_t   fread __P ((void *, size_t, size_t, FILE *));
+#endif
+#ifndef HAVE_FGETC_DECL
+extern int      fgetc __P ((FILE *));
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifndef u_long
+# define u_long unsigned long
+#endif
+#ifndef u_char
+# define u_char unsigned char
+#endif
+
+
+#endif /* SYSFUNCS_H */
Index: /trunk/mgpp/lib/unitool.cpp
===================================================================
--- /trunk/mgpp/lib/unitool.cpp	(revision 3365)
+++ /trunk/mgpp/lib/unitool.cpp	(revision 3365)
@@ -0,0 +1,1756 @@
+#include "unitool.h"
+#include "sysfuncs.h"
+
+
+/* unicode_cat_data is a combination of the letter, digit, and
+ * space information in a form that is quick to look up.
+ * The information about each byte is stored in two bits
+ *
+ * 00 - neither letter, digit or space
+ * 01 - letter
+ * 10 - space
+ * 11 - digit
+ *
+ * You can test for letter | space by looking at the right
+ * column
+ */
+unsigned long *unicode_cat_data=NULL;
+
+
+typedef struct unidecompose {
+  unsigned short ch;
+  unsigned short str[3];
+} unidecompose;
+
+typedef struct unirelate {
+  unsigned short ch1;
+  unsigned short ch2;
+} unirelate;
+
+typedef struct unirange {
+  unsigned short first;
+  unsigned short last;
+} unirange;
+
+
+
+/* This decomposition data only includes canonical decompositional
+ * characters.  Any unused bytes are set to 0x0. */
+
+#define NUM_DECOMPOSE_INFO 1269
+static const unidecompose decompose_info[NUM_DECOMPOSE_INFO] = {
+  {0xC0,0x41,0x300,0x0}, {0xC1,0x41,0x301,0x0}, {0xC2,0x41,0x302,0x0}, 
+  {0xC3,0x41,0x303,0x0}, {0xC4,0x41,0x308,0x0}, {0xC5,0x41,0x30A,0x0}, 
+  {0xC7,0x43,0x327,0x0}, {0xC8,0x45,0x300,0x0}, {0xC9,0x45,0x301,0x0}, 
+  {0xCA,0x45,0x302,0x0}, {0xCB,0x45,0x308,0x0}, {0xCC,0x49,0x300,0x0}, 
+  {0xCD,0x49,0x301,0x0}, {0xCE,0x49,0x302,0x0}, {0xCF,0x49,0x308,0x0}, 
+  {0xD1,0x4E,0x303,0x0}, {0xD2,0x4F,0x300,0x0}, {0xD3,0x4F,0x301,0x0}, 
+  {0xD4,0x4F,0x302,0x0}, {0xD5,0x4F,0x303,0x0}, {0xD6,0x4F,0x308,0x0}, 
+  {0xD9,0x55,0x300,0x0}, {0xDA,0x55,0x301,0x0}, {0xDB,0x55,0x302,0x0}, 
+  {0xDC,0x55,0x308,0x0}, {0xDD,0x59,0x301,0x0}, {0xE0,0x61,0x300,0x0}, 
+  {0xE1,0x61,0x301,0x0}, {0xE2,0x61,0x302,0x0}, {0xE3,0x61,0x303,0x0}, 
+  {0xE4,0x61,0x308,0x0}, {0xE5,0x61,0x30A,0x0}, {0xE7,0x63,0x327,0x0}, 
+  {0xE8,0x65,0x300,0x0}, {0xE9,0x65,0x301,0x0}, {0xEA,0x65,0x302,0x0}, 
+  {0xEB,0x65,0x308,0x0}, {0xEC,0x69,0x300,0x0}, {0xED,0x69,0x301,0x0}, 
+  {0xEE,0x69,0x302,0x0}, {0xEF,0x69,0x308,0x0}, {0xF1,0x6E,0x303,0x0}, 
+  {0xF2,0x6F,0x300,0x0}, {0xF3,0x6F,0x301,0x0}, {0xF4,0x6F,0x302,0x0}, 
+  {0xF5,0x6F,0x303,0x0}, {0xF6,0x6F,0x308,0x0}, {0xF9,0x75,0x300,0x0}, 
+  {0xFA,0x75,0x301,0x0}, {0xFB,0x75,0x302,0x0}, {0xFC,0x75,0x308,0x0}, 
+  {0xFD,0x79,0x301,0x0}, {0xFF,0x79,0x308,0x0}, {0x100,0x41,0x304,0x0}, 
+  {0x101,0x61,0x304,0x0}, {0x102,0x41,0x306,0x0}, {0x103,0x61,0x306,0x0}, 
+  {0x104,0x41,0x328,0x0}, {0x105,0x61,0x328,0x0}, {0x106,0x43,0x301,0x0}, 
+  {0x107,0x63,0x301,0x0}, {0x108,0x43,0x302,0x0}, {0x109,0x63,0x302,0x0}, 
+  {0x10A,0x43,0x307,0x0}, {0x10B,0x63,0x307,0x0}, {0x10C,0x43,0x30C,0x0}, 
+  {0x10D,0x63,0x30C,0x0}, {0x10E,0x44,0x30C,0x0}, {0x10F,0x64,0x30C,0x0}, 
+  {0x112,0x45,0x304,0x0}, {0x113,0x65,0x304,0x0}, {0x114,0x45,0x306,0x0}, 
+  {0x115,0x65,0x306,0x0}, {0x116,0x45,0x307,0x0}, {0x117,0x65,0x307,0x0}, 
+  {0x118,0x45,0x328,0x0}, {0x119,0x65,0x328,0x0}, {0x11A,0x45,0x30C,0x0}, 
+  {0x11B,0x65,0x30C,0x0}, {0x11C,0x47,0x302,0x0}, {0x11D,0x67,0x302,0x0}, 
+  {0x11E,0x47,0x306,0x0}, {0x11F,0x67,0x306,0x0}, {0x120,0x47,0x307,0x0}, 
+  {0x121,0x67,0x307,0x0}, {0x122,0x47,0x327,0x0}, {0x123,0x67,0x327,0x0}, 
+  {0x124,0x48,0x302,0x0}, {0x125,0x68,0x302,0x0}, {0x128,0x49,0x303,0x0}, 
+  {0x129,0x69,0x303,0x0}, {0x12A,0x49,0x304,0x0}, {0x12B,0x69,0x304,0x0}, 
+  {0x12C,0x49,0x306,0x0}, {0x12D,0x69,0x306,0x0}, {0x12E,0x49,0x328,0x0}, 
+  {0x12F,0x69,0x328,0x0}, {0x130,0x49,0x307,0x0}, {0x134,0x4A,0x302,0x0}, 
+  {0x135,0x6A,0x302,0x0}, {0x136,0x4B,0x327,0x0}, {0x137,0x6B,0x327,0x0}, 
+  {0x139,0x4C,0x301,0x0}, {0x13A,0x6C,0x301,0x0}, {0x13B,0x4C,0x327,0x0}, 
+  {0x13C,0x6C,0x327,0x0}, {0x13D,0x4C,0x30C,0x0}, {0x13E,0x6C,0x30C,0x0}, 
+  {0x143,0x4E,0x301,0x0}, {0x144,0x6E,0x301,0x0}, {0x145,0x4E,0x327,0x0}, 
+  {0x146,0x6E,0x327,0x0}, {0x147,0x4E,0x30C,0x0}, {0x148,0x6E,0x30C,0x0}, 
+  {0x14C,0x4F,0x304,0x0}, {0x14D,0x6F,0x304,0x0}, {0x14E,0x4F,0x306,0x0}, 
+  {0x14F,0x6F,0x306,0x0}, {0x150,0x4F,0x30B,0x0}, {0x151,0x6F,0x30B,0x0}, 
+  {0x154,0x52,0x301,0x0}, {0x155,0x72,0x301,0x0}, {0x156,0x52,0x327,0x0}, 
+  {0x157,0x72,0x327,0x0}, {0x158,0x52,0x30C,0x0}, {0x159,0x72,0x30C,0x0}, 
+  {0x15A,0x53,0x301,0x0}, {0x15B,0x73,0x301,0x0}, {0x15C,0x53,0x302,0x0}, 
+  {0x15D,0x73,0x302,0x0}, {0x15E,0x53,0x327,0x0}, {0x15F,0x73,0x327,0x0}, 
+  {0x160,0x53,0x30C,0x0}, {0x161,0x73,0x30C,0x0}, {0x162,0x54,0x327,0x0}, 
+  {0x163,0x74,0x327,0x0}, {0x164,0x54,0x30C,0x0}, {0x165,0x74,0x30C,0x0}, 
+  {0x168,0x55,0x303,0x0}, {0x169,0x75,0x303,0x0}, {0x16A,0x55,0x304,0x0}, 
+  {0x16B,0x75,0x304,0x0}, {0x16C,0x55,0x306,0x0}, {0x16D,0x75,0x306,0x0}, 
+  {0x16E,0x55,0x30A,0x0}, {0x16F,0x75,0x30A,0x0}, {0x170,0x55,0x30B,0x0}, 
+  {0x171,0x75,0x30B,0x0}, {0x172,0x55,0x328,0x0}, {0x173,0x75,0x328,0x0}, 
+  {0x174,0x57,0x302,0x0}, {0x175,0x77,0x302,0x0}, {0x176,0x59,0x302,0x0}, 
+  {0x177,0x79,0x302,0x0}, {0x178,0x59,0x308,0x0}, {0x179,0x5A,0x301,0x0}, 
+  {0x17A,0x7A,0x301,0x0}, {0x17B,0x5A,0x307,0x0}, {0x17C,0x7A,0x307,0x0}, 
+  {0x17D,0x5A,0x30C,0x0}, {0x17E,0x7A,0x30C,0x0}, {0x1A0,0x4F,0x31B,0x0}, 
+  {0x1A1,0x6F,0x31B,0x0}, {0x1AF,0x55,0x31B,0x0}, {0x1B0,0x75,0x31B,0x0}, 
+  {0x1CD,0x41,0x30C,0x0}, {0x1CE,0x61,0x30C,0x0}, {0x1CF,0x49,0x30C,0x0}, 
+  {0x1D0,0x69,0x30C,0x0}, {0x1D1,0x4F,0x30C,0x0}, {0x1D2,0x6F,0x30C,0x0}, 
+  {0x1D3,0x55,0x30C,0x0}, {0x1D4,0x75,0x30C,0x0}, {0x1D5,0xDC,0x304,0x0}, 
+  {0x1D6,0xFC,0x304,0x0}, {0x1D7,0xDC,0x301,0x0}, {0x1D8,0xFC,0x301,0x0}, 
+  {0x1D9,0xDC,0x30C,0x0}, {0x1DA,0xFC,0x30C,0x0}, {0x1DB,0xDC,0x300,0x0}, 
+  {0x1DC,0xFC,0x300,0x0}, {0x1DE,0xC4,0x304,0x0}, {0x1DF,0xE4,0x304,0x0}, 
+  {0x1E0,0x41,0x307,0x304}, {0x1E1,0x61,0x307,0x304}, {0x1E2,0xC6,0x304,0x0}, 
+  {0x1E3,0xE6,0x304,0x0}, {0x1E6,0x47,0x30C,0x0}, {0x1E7,0x67,0x30C,0x0}, 
+  {0x1E8,0x4B,0x30C,0x0}, {0x1E9,0x6B,0x30C,0x0}, {0x1EA,0x4F,0x328,0x0}, 
+  {0x1EB,0x6F,0x328,0x0}, {0x1EC,0x1EA,0x304,0x0}, {0x1ED,0x1EB,0x304,0x0}, 
+  {0x1EE,0x1B7,0x30C,0x0}, {0x1EF,0x292,0x30C,0x0}, {0x1F0,0x6A,0x30C,0x0}, 
+  {0x1F4,0x47,0x301,0x0}, {0x1F5,0x67,0x301,0x0}, {0x1FA,0xC5,0x301,0x0}, 
+  {0x1FB,0xE5,0x301,0x0}, {0x1FC,0xC6,0x301,0x0}, {0x1FD,0xE6,0x301,0x0}, 
+  {0x1FE,0xD8,0x301,0x0}, {0x1FF,0xF8,0x301,0x0}, {0x200,0x41,0x30F,0x0}, 
+  {0x201,0x61,0x30F,0x0}, {0x202,0x41,0x311,0x0}, {0x203,0x61,0x311,0x0}, 
+  {0x204,0x45,0x30F,0x0}, {0x205,0x65,0x30F,0x0}, {0x206,0x45,0x311,0x0}, 
+  {0x207,0x65,0x311,0x0}, {0x208,0x49,0x30F,0x0}, {0x209,0x69,0x30F,0x0}, 
+  {0x20A,0x49,0x311,0x0}, {0x20B,0x69,0x311,0x0}, {0x20C,0x4F,0x30F,0x0}, 
+  {0x20D,0x6F,0x30F,0x0}, {0x20E,0x4F,0x311,0x0}, {0x20F,0x6F,0x311,0x0}, 
+  {0x210,0x52,0x30F,0x0}, {0x211,0x72,0x30F,0x0}, {0x212,0x52,0x311,0x0}, 
+  {0x213,0x72,0x311,0x0}, {0x214,0x55,0x30F,0x0}, {0x215,0x75,0x30F,0x0}, 
+  {0x216,0x55,0x311,0x0}, {0x217,0x75,0x311,0x0}, {0x310,0x306,0x307,0x0}, 
+  {0x340,0x300,0x0,0x0}, {0x341,0x301,0x0,0x0}, {0x343,0x313,0x0,0x0}, 
+  {0x344,0x308,0x30D,0x0}, {0x374,0x2B9,0x0,0x0}, {0x37E,0x3B,0x0,0x0}, 
+  {0x385,0xA8,0x30D,0x0}, {0x386,0x391,0x30D,0x0}, {0x387,0xB7,0x0,0x0}, 
+  {0x388,0x395,0x30D,0x0}, {0x389,0x397,0x30D,0x0}, {0x38A,0x399,0x30D,0x0}, 
+  {0x38C,0x39F,0x30D,0x0}, {0x38E,0x3A5,0x30D,0x0}, {0x38F,0x3A9,0x30D,0x0}, 
+  {0x390,0x3B9,0x344,0x0}, {0x3AA,0x399,0x308,0x0}, {0x3AB,0x3A5,0x308,0x0}, 
+  {0x3AC,0x3B1,0x30D,0x0}, {0x3AD,0x3B5,0x30D,0x0}, {0x3AE,0x3B7,0x30D,0x0}, 
+  {0x3AF,0x3B9,0x30D,0x0}, {0x3B0,0x3C5,0x344,0x0}, {0x3CA,0x3B9,0x308,0x0}, 
+  {0x3CB,0x3C5,0x308,0x0}, {0x3CC,0x3BF,0x30D,0x0}, {0x3CD,0x3C5,0x30D,0x0}, 
+  {0x3CE,0x3C9,0x30D,0x0}, {0x3D3,0x3D2,0x30D,0x0}, {0x3D4,0x3D2,0x308,0x0}, 
+  {0x401,0x415,0x308,0x0}, {0x403,0x413,0x301,0x0}, {0x407,0x406,0x308,0x0}, 
+  {0x40C,0x41A,0x301,0x0}, {0x40E,0x423,0x306,0x0}, {0x419,0x418,0x306,0x0}, 
+  {0x439,0x438,0x306,0x0}, {0x451,0x435,0x308,0x0}, {0x453,0x433,0x301,0x0}, 
+  {0x457,0x456,0x308,0x0}, {0x45C,0x43A,0x301,0x0}, {0x45E,0x443,0x306,0x0}, 
+  {0x476,0x474,0x30F,0x0}, {0x477,0x475,0x30F,0x0}, {0x4C1,0x416,0x306,0x0}, 
+  {0x4C2,0x436,0x306,0x0}, {0x4D0,0x410,0x306,0x0}, {0x4D1,0x430,0x306,0x0}, 
+  {0x4D2,0x410,0x308,0x0}, {0x4D3,0x430,0x308,0x0}, {0x4D6,0x415,0x306,0x0}, 
+  {0x4D7,0x435,0x306,0x0}, {0x4DA,0x4D8,0x308,0x0}, {0x4DB,0x4D9,0x308,0x0}, 
+  {0x4DC,0x416,0x308,0x0}, {0x4DD,0x436,0x308,0x0}, {0x4DE,0x417,0x308,0x0}, 
+  {0x4DF,0x437,0x308,0x0}, {0x4E2,0x418,0x304,0x0}, {0x4E3,0x438,0x304,0x0}, 
+  {0x4E4,0x418,0x308,0x0}, {0x4E5,0x438,0x308,0x0}, {0x4E6,0x41E,0x308,0x0}, 
+  {0x4E7,0x43E,0x308,0x0}, {0x4EA,0x4E8,0x308,0x0}, {0x4EB,0x4E9,0x308,0x0}, 
+  {0x4EE,0x423,0x304,0x0}, {0x4EF,0x443,0x304,0x0}, {0x4F0,0x423,0x308,0x0}, 
+  {0x4F1,0x443,0x308,0x0}, {0x4F2,0x423,0x30B,0x0}, {0x4F3,0x443,0x30B,0x0}, 
+  {0x4F4,0x427,0x308,0x0}, {0x4F5,0x447,0x308,0x0}, {0x4F8,0x42B,0x308,0x0}, 
+  {0x4F9,0x44B,0x308,0x0}, {0x929,0x928,0x93C,0x0}, {0x931,0x930,0x93C,0x0}, 
+  {0x934,0x933,0x93C,0x0}, {0x958,0x915,0x93C,0x0}, {0x959,0x916,0x93C,0x0}, 
+  {0x95A,0x917,0x93C,0x0}, {0x95B,0x91C,0x93C,0x0}, {0x95C,0x921,0x93C,0x0}, 
+  {0x95D,0x922,0x93C,0x0}, {0x95E,0x92B,0x93C,0x0}, {0x95F,0x92F,0x93C,0x0}, 
+  {0x9B0,0x9AC,0x9BC,0x0}, {0x9CB,0x9C7,0x9BE,0x0}, {0x9CC,0x9C7,0x9D7,0x0}, 
+  {0x9DC,0x9A1,0x9BC,0x0}, {0x9DD,0x9A2,0x9BC,0x0}, {0x9DF,0x9AF,0x9BC,0x0}, 
+  {0xA59,0xA16,0xA3C,0x0}, {0xA5A,0xA17,0xA3C,0x0}, {0xA5B,0xA1C,0xA3C,0x0}, 
+  {0xA5C,0xA21,0xA3C,0x0}, {0xA5E,0xA2B,0xA3C,0x0}, {0xB48,0xB47,0xB56,0x0}, 
+  {0xB4B,0xB47,0xB3E,0x0}, {0xB4C,0xB47,0xB57,0x0}, {0xB5C,0xB21,0xB3C,0x0}, 
+  {0xB5D,0xB22,0xB3C,0x0}, {0xB5F,0xB2F,0xB3C,0x0}, {0xB94,0xB92,0xBD7,0x0}, 
+  {0xBCA,0xBC6,0xBBE,0x0}, {0xBCB,0xBC7,0xBBE,0x0}, {0xBCC,0xBC6,0xBD7,0x0}, 
+  {0xC48,0xC46,0xC56,0x0}, {0xCC0,0xCBF,0xCD5,0x0}, {0xCC7,0xCC6,0xCD5,0x0}, 
+  {0xCC8,0xCC6,0xCD6,0x0}, {0xCCA,0xCC6,0xCC2,0x0}, {0xCCB,0xCC6,0xCC2,0xCD5}, 
+  {0xD4A,0xD46,0xD3E,0x0}, {0xD4B,0xD47,0xD3E,0x0}, {0xD4C,0xD46,0xD57,0x0}, 
+  {0xE33,0xE4D,0xE32,0x0}, {0xEB3,0xECD,0xEB2,0x0}, {0xF43,0xF42,0xFB7,0x0}, 
+  {0xF4D,0xF4C,0xFB7,0x0}, {0xF52,0xF51,0xFB7,0x0}, {0xF57,0xF56,0xFB7,0x0}, 
+  {0xF5C,0xF5B,0xFB7,0x0}, {0xF69,0xF40,0xFB5,0x0}, {0xF73,0xF72,0xF71,0x0}, 
+  {0xF75,0xF71,0xF74,0x0}, {0xF76,0xFB2,0xF80,0x0}, {0xF77,0xF76,0xF71,0x0}, 
+  {0xF78,0xFB3,0xF80,0x0}, {0xF79,0xF78,0xF71,0x0}, {0xF81,0xF80,0xF71,0x0}, 
+  {0xF93,0xF92,0xFB7,0x0}, {0xF9D,0xF9C,0xFB7,0x0}, {0xFA2,0xFA1,0xFB7,0x0}, 
+  {0xFA7,0xFA6,0xFB7,0x0}, {0xFAC,0xFAB,0xFB7,0x0}, {0xFB9,0xF90,0xFB5,0x0}, 
+  {0x1E00,0x41,0x325,0x0}, {0x1E01,0x61,0x325,0x0}, {0x1E02,0x42,0x307,0x0}, 
+  {0x1E03,0x62,0x307,0x0}, {0x1E04,0x42,0x323,0x0}, {0x1E05,0x62,0x323,0x0}, 
+  {0x1E06,0x42,0x331,0x0}, {0x1E07,0x62,0x331,0x0}, {0x1E08,0xC7,0x301,0x0}, 
+  {0x1E09,0xE7,0x301,0x0}, {0x1E0A,0x44,0x307,0x0}, {0x1E0B,0x64,0x307,0x0}, 
+  {0x1E0C,0x44,0x323,0x0}, {0x1E0D,0x64,0x323,0x0}, {0x1E0E,0x44,0x331,0x0}, 
+  {0x1E0F,0x64,0x331,0x0}, {0x1E10,0x44,0x327,0x0}, {0x1E11,0x64,0x327,0x0}, 
+  {0x1E12,0x44,0x32D,0x0}, {0x1E13,0x64,0x32D,0x0}, {0x1E14,0x112,0x300,0x0}, 
+  {0x1E15,0x113,0x300,0x0}, {0x1E16,0x112,0x301,0x0}, {0x1E17,0x113,0x301,0x0}, 
+  {0x1E18,0x45,0x32D,0x0}, {0x1E19,0x65,0x32D,0x0}, {0x1E1A,0x45,0x330,0x0}, 
+  {0x1E1B,0x65,0x330,0x0}, {0x1E1C,0x114,0x327,0x0}, {0x1E1D,0x115,0x327,0x0}, 
+  {0x1E1E,0x46,0x307,0x0}, {0x1E1F,0x66,0x307,0x0}, {0x1E20,0x47,0x304,0x0}, 
+  {0x1E21,0x67,0x304,0x0}, {0x1E22,0x48,0x307,0x0}, {0x1E23,0x68,0x307,0x0}, 
+  {0x1E24,0x48,0x323,0x0}, {0x1E25,0x68,0x323,0x0}, {0x1E26,0x48,0x308,0x0}, 
+  {0x1E27,0x68,0x308,0x0}, {0x1E28,0x48,0x327,0x0}, {0x1E29,0x68,0x327,0x0}, 
+  {0x1E2A,0x48,0x32E,0x0}, {0x1E2B,0x68,0x32E,0x0}, {0x1E2C,0x49,0x330,0x0}, 
+  {0x1E2D,0x69,0x330,0x0}, {0x1E2E,0xCF,0x301,0x0}, {0x1E2F,0xEF,0x301,0x0}, 
+  {0x1E30,0x4B,0x301,0x0}, {0x1E31,0x6B,0x301,0x0}, {0x1E32,0x4B,0x323,0x0}, 
+  {0x1E33,0x6B,0x323,0x0}, {0x1E34,0x4B,0x331,0x0}, {0x1E35,0x6B,0x331,0x0}, 
+  {0x1E36,0x4C,0x323,0x0}, {0x1E37,0x6C,0x323,0x0}, {0x1E38,0x1E36,0x304,0x0}, 
+  {0x1E39,0x1E37,0x304,0x0}, {0x1E3A,0x4C,0x331,0x0}, {0x1E3B,0x6C,0x331,0x0}, 
+  {0x1E3C,0x4C,0x32D,0x0}, {0x1E3D,0x6C,0x32D,0x0}, {0x1E3E,0x4D,0x301,0x0}, 
+  {0x1E3F,0x6D,0x301,0x0}, {0x1E40,0x4D,0x307,0x0}, {0x1E41,0x6D,0x307,0x0}, 
+  {0x1E42,0x4D,0x323,0x0}, {0x1E43,0x6D,0x323,0x0}, {0x1E44,0x4E,0x307,0x0}, 
+  {0x1E45,0x6E,0x307,0x0}, {0x1E46,0x4E,0x323,0x0}, {0x1E47,0x6E,0x323,0x0}, 
+  {0x1E48,0x4E,0x331,0x0}, {0x1E49,0x6E,0x331,0x0}, {0x1E4A,0x4E,0x32D,0x0}, 
+  {0x1E4B,0x6E,0x32D,0x0}, {0x1E4C,0xD5,0x301,0x0}, {0x1E4D,0xF5,0x301,0x0}, 
+  {0x1E4E,0xD5,0x308,0x0}, {0x1E4F,0xF5,0x308,0x0}, {0x1E50,0x14C,0x300,0x0}, 
+  {0x1E51,0x14D,0x300,0x0}, {0x1E52,0x14C,0x301,0x0}, {0x1E53,0x14D,0x301,0x0}, 
+  {0x1E54,0x50,0x301,0x0}, {0x1E55,0x70,0x301,0x0}, {0x1E56,0x50,0x307,0x0}, 
+  {0x1E57,0x70,0x307,0x0}, {0x1E58,0x52,0x307,0x0}, {0x1E59,0x72,0x307,0x0}, 
+  {0x1E5A,0x52,0x323,0x0}, {0x1E5B,0x72,0x323,0x0}, {0x1E5C,0x1E5A,0x304,0x0}, 
+  {0x1E5D,0x1E5B,0x304,0x0}, {0x1E5E,0x52,0x331,0x0}, {0x1E5F,0x72,0x331,0x0}, 
+  {0x1E60,0x53,0x307,0x0}, {0x1E61,0x73,0x307,0x0}, {0x1E62,0x53,0x323,0x0}, 
+  {0x1E63,0x73,0x323,0x0}, {0x1E64,0x15A,0x307,0x0}, {0x1E65,0x15B,0x307,0x0}, 
+  {0x1E66,0x160,0x307,0x0}, {0x1E67,0x161,0x307,0x0}, {0x1E68,0x1E62,0x307,0x0}, 
+  {0x1E69,0x1E63,0x307,0x0}, {0x1E6A,0x54,0x307,0x0}, {0x1E6B,0x74,0x307,0x0}, 
+  {0x1E6C,0x54,0x323,0x0}, {0x1E6D,0x74,0x323,0x0}, {0x1E6E,0x54,0x331,0x0}, 
+  {0x1E6F,0x74,0x331,0x0}, {0x1E70,0x54,0x32D,0x0}, {0x1E71,0x74,0x32D,0x0}, 
+  {0x1E72,0x55,0x324,0x0}, {0x1E73,0x75,0x324,0x0}, {0x1E74,0x55,0x330,0x0}, 
+  {0x1E75,0x75,0x330,0x0}, {0x1E76,0x55,0x32D,0x0}, {0x1E77,0x75,0x32D,0x0}, 
+  {0x1E78,0x168,0x301,0x0}, {0x1E79,0x169,0x301,0x0}, {0x1E7A,0x16A,0x308,0x0}, 
+  {0x1E7B,0x16B,0x308,0x0}, {0x1E7C,0x56,0x303,0x0}, {0x1E7D,0x76,0x303,0x0}, 
+  {0x1E7E,0x56,0x323,0x0}, {0x1E7F,0x76,0x323,0x0}, {0x1E80,0x57,0x300,0x0}, 
+  {0x1E81,0x77,0x300,0x0}, {0x1E82,0x57,0x301,0x0}, {0x1E83,0x77,0x301,0x0}, 
+  {0x1E84,0x57,0x308,0x0}, {0x1E85,0x77,0x308,0x0}, {0x1E86,0x57,0x307,0x0}, 
+  {0x1E87,0x77,0x307,0x0}, {0x1E88,0x57,0x323,0x0}, {0x1E89,0x77,0x323,0x0}, 
+  {0x1E8A,0x58,0x307,0x0}, {0x1E8B,0x78,0x307,0x0}, {0x1E8C,0x58,0x308,0x0}, 
+  {0x1E8D,0x78,0x308,0x0}, {0x1E8E,0x59,0x307,0x0}, {0x1E8F,0x79,0x307,0x0}, 
+  {0x1E90,0x5A,0x302,0x0}, {0x1E91,0x7A,0x302,0x0}, {0x1E92,0x5A,0x323,0x0}, 
+  {0x1E93,0x7A,0x323,0x0}, {0x1E94,0x5A,0x331,0x0}, {0x1E95,0x7A,0x331,0x0}, 
+  {0x1E96,0x68,0x331,0x0}, {0x1E97,0x74,0x308,0x0}, {0x1E98,0x77,0x30A,0x0}, 
+  {0x1E99,0x79,0x30A,0x0}, {0x1E9B,0x17F,0x307,0x0}, {0x1EA0,0x41,0x323,0x0}, 
+  {0x1EA1,0x61,0x323,0x0}, {0x1EA2,0x41,0x309,0x0}, {0x1EA3,0x61,0x309,0x0}, 
+  {0x1EA4,0xC2,0x301,0x0}, {0x1EA5,0xE2,0x301,0x0}, {0x1EA6,0xC2,0x300,0x0}, 
+  {0x1EA7,0xE2,0x300,0x0}, {0x1EA8,0xC2,0x309,0x0}, {0x1EA9,0xE2,0x309,0x0}, 
+  {0x1EAA,0xC2,0x303,0x0}, {0x1EAB,0xE2,0x303,0x0}, {0x1EAC,0xC2,0x323,0x0}, 
+  {0x1EAD,0xE2,0x323,0x0}, {0x1EAE,0x102,0x301,0x0}, {0x1EAF,0x103,0x301,0x0}, 
+  {0x1EB0,0x102,0x300,0x0}, {0x1EB1,0x103,0x300,0x0}, {0x1EB2,0x102,0x309,0x0}, 
+  {0x1EB3,0x103,0x309,0x0}, {0x1EB4,0x102,0x303,0x0}, {0x1EB5,0x103,0x303,0x0}, 
+  {0x1EB6,0x102,0x323,0x0}, {0x1EB7,0x103,0x323,0x0}, {0x1EB8,0x45,0x323,0x0}, 
+  {0x1EB9,0x65,0x323,0x0}, {0x1EBA,0x45,0x309,0x0}, {0x1EBB,0x65,0x309,0x0}, 
+  {0x1EBC,0x45,0x303,0x0}, {0x1EBD,0x65,0x303,0x0}, {0x1EBE,0xCA,0x301,0x0}, 
+  {0x1EBF,0xEA,0x301,0x0}, {0x1EC0,0xCA,0x300,0x0}, {0x1EC1,0xEA,0x300,0x0}, 
+  {0x1EC2,0xCA,0x309,0x0}, {0x1EC3,0xEA,0x309,0x0}, {0x1EC4,0xCA,0x303,0x0}, 
+  {0x1EC5,0xEA,0x303,0x0}, {0x1EC6,0xCA,0x323,0x0}, {0x1EC7,0xEA,0x323,0x0}, 
+  {0x1EC8,0x49,0x309,0x0}, {0x1EC9,0x69,0x309,0x0}, {0x1ECA,0x49,0x323,0x0}, 
+  {0x1ECB,0x69,0x323,0x0}, {0x1ECC,0x4F,0x323,0x0}, {0x1ECD,0x6F,0x323,0x0}, 
+  {0x1ECE,0x4F,0x309,0x0}, {0x1ECF,0x6F,0x309,0x0}, {0x1ED0,0xD4,0x301,0x0}, 
+  {0x1ED1,0xF4,0x301,0x0}, {0x1ED2,0xD4,0x300,0x0}, {0x1ED3,0xF4,0x300,0x0}, 
+  {0x1ED4,0xD4,0x309,0x0}, {0x1ED5,0xF4,0x309,0x0}, {0x1ED6,0xD4,0x303,0x0}, 
+  {0x1ED7,0xF4,0x303,0x0}, {0x1ED8,0xD4,0x323,0x0}, {0x1ED9,0xF4,0x323,0x0}, 
+  {0x1EDA,0x1A0,0x301,0x0}, {0x1EDB,0x1A1,0x301,0x0}, {0x1EDC,0x1A0,0x300,0x0}, 
+  {0x1EDD,0x1A1,0x300,0x0}, {0x1EDE,0x1A0,0x309,0x0}, {0x1EDF,0x1A1,0x309,0x0}, 
+  {0x1EE0,0x1A0,0x303,0x0}, {0x1EE1,0x1A1,0x303,0x0}, {0x1EE2,0x1A0,0x323,0x0}, 
+  {0x1EE3,0x1A1,0x323,0x0}, {0x1EE4,0x55,0x323,0x0}, {0x1EE5,0x75,0x323,0x0}, 
+  {0x1EE6,0x55,0x309,0x0}, {0x1EE7,0x75,0x309,0x0}, {0x1EE8,0x1AF,0x301,0x0}, 
+  {0x1EE9,0x1B0,0x301,0x0}, {0x1EEA,0x1AF,0x300,0x0}, {0x1EEB,0x1B0,0x300,0x0}, 
+  {0x1EEC,0x1AF,0x309,0x0}, {0x1EED,0x1B0,0x309,0x0}, {0x1EEE,0x1AF,0x303,0x0}, 
+  {0x1EEF,0x1B0,0x303,0x0}, {0x1EF0,0x1AF,0x323,0x0}, {0x1EF1,0x1B0,0x323,0x0}, 
+  {0x1EF2,0x59,0x300,0x0}, {0x1EF3,0x79,0x300,0x0}, {0x1EF4,0x59,0x323,0x0}, 
+  {0x1EF5,0x79,0x323,0x0}, {0x1EF6,0x59,0x309,0x0}, {0x1EF7,0x79,0x309,0x0}, 
+  {0x1EF8,0x59,0x303,0x0}, {0x1EF9,0x79,0x303,0x0}, {0x1F00,0x3B1,0x313,0x0}, 
+  {0x1F01,0x3B1,0x314,0x0}, {0x1F02,0x1F00,0x300,0x0}, {0x1F03,0x1F01,0x300,0x0}, 
+  {0x1F04,0x1F00,0x301,0x0}, {0x1F05,0x1F01,0x301,0x0}, {0x1F06,0x1F00,0x342,0x0}, 
+  {0x1F07,0x1F01,0x342,0x0}, {0x1F08,0x391,0x313,0x0}, {0x1F09,0x391,0x314,0x0}, 
+  {0x1F0A,0x1F08,0x300,0x0}, {0x1F0B,0x1F09,0x300,0x0}, {0x1F0C,0x1F08,0x301,0x0}, 
+  {0x1F0D,0x1F09,0x301,0x0}, {0x1F0E,0x1F08,0x342,0x0}, {0x1F0F,0x1F09,0x342,0x0}, 
+  {0x1F10,0x3B5,0x313,0x0}, {0x1F11,0x3B5,0x314,0x0}, {0x1F12,0x1F10,0x300,0x0}, 
+  {0x1F13,0x1F11,0x300,0x0}, {0x1F14,0x1F10,0x301,0x0}, {0x1F15,0x1F11,0x301,0x0}, 
+  {0x1F18,0x395,0x313,0x0}, {0x1F19,0x395,0x314,0x0}, {0x1F1A,0x1F18,0x300,0x0}, 
+  {0x1F1B,0x1F19,0x300,0x0}, {0x1F1C,0x1F18,0x301,0x0}, {0x1F1D,0x1F19,0x301,0x0}, 
+  {0x1F20,0x3B7,0x313,0x0}, {0x1F21,0x3B7,0x314,0x0}, {0x1F22,0x1F20,0x300,0x0}, 
+  {0x1F23,0x1F21,0x300,0x0}, {0x1F24,0x1F20,0x301,0x0}, {0x1F25,0x1F21,0x301,0x0}, 
+  {0x1F26,0x1F20,0x342,0x0}, {0x1F27,0x1F21,0x342,0x0}, {0x1F28,0x397,0x313,0x0}, 
+  {0x1F29,0x397,0x314,0x0}, {0x1F2A,0x1F28,0x300,0x0}, {0x1F2B,0x1F29,0x300,0x0}, 
+  {0x1F2C,0x1F28,0x301,0x0}, {0x1F2D,0x1F29,0x301,0x0}, {0x1F2E,0x1F28,0x342,0x0}, 
+  {0x1F2F,0x1F29,0x342,0x0}, {0x1F30,0x3B9,0x313,0x0}, {0x1F31,0x3B9,0x314,0x0}, 
+  {0x1F32,0x1F30,0x300,0x0}, {0x1F33,0x1F31,0x300,0x0}, {0x1F34,0x1F30,0x301,0x0}, 
+  {0x1F35,0x1F31,0x301,0x0}, {0x1F36,0x1F30,0x342,0x0}, {0x1F37,0x1F31,0x342,0x0}, 
+  {0x1F38,0x399,0x313,0x0}, {0x1F39,0x399,0x314,0x0}, {0x1F3A,0x1F38,0x300,0x0}, 
+  {0x1F3B,0x1F39,0x300,0x0}, {0x1F3C,0x1F38,0x301,0x0}, {0x1F3D,0x1F39,0x301,0x0}, 
+  {0x1F3E,0x1F38,0x342,0x0}, {0x1F3F,0x1F39,0x342,0x0}, {0x1F40,0x3BF,0x313,0x0}, 
+  {0x1F41,0x3BF,0x314,0x0}, {0x1F42,0x1F40,0x300,0x0}, {0x1F43,0x1F41,0x300,0x0}, 
+  {0x1F44,0x1F40,0x301,0x0}, {0x1F45,0x1F41,0x301,0x0}, {0x1F48,0x39F,0x313,0x0}, 
+  {0x1F49,0x39F,0x314,0x0}, {0x1F4A,0x1F48,0x300,0x0}, {0x1F4B,0x1F49,0x300,0x0}, 
+  {0x1F4C,0x1F48,0x301,0x0}, {0x1F4D,0x1F49,0x301,0x0}, {0x1F50,0x3C5,0x313,0x0}, 
+  {0x1F51,0x3C5,0x314,0x0}, {0x1F52,0x1F50,0x300,0x0}, {0x1F53,0x1F51,0x300,0x0}, 
+  {0x1F54,0x1F50,0x301,0x0}, {0x1F55,0x1F51,0x301,0x0}, {0x1F56,0x1F50,0x342,0x0}, 
+  {0x1F57,0x1F51,0x342,0x0}, {0x1F59,0x3A5,0x314,0x0}, {0x1F5B,0x1F59,0x300,0x0}, 
+  {0x1F5D,0x1F59,0x301,0x0}, {0x1F5F,0x1F59,0x342,0x0}, {0x1F60,0x3C9,0x313,0x0}, 
+  {0x1F61,0x3C9,0x314,0x0}, {0x1F62,0x1F60,0x300,0x0}, {0x1F63,0x1F61,0x300,0x0}, 
+  {0x1F64,0x1F60,0x301,0x0}, {0x1F65,0x1F61,0x301,0x0}, {0x1F66,0x1F60,0x342,0x0}, 
+  {0x1F67,0x1F61,0x342,0x0}, {0x1F68,0x3A9,0x313,0x0}, {0x1F69,0x3A9,0x314,0x0}, 
+  {0x1F6A,0x1F68,0x300,0x0}, {0x1F6B,0x1F69,0x300,0x0}, {0x1F6C,0x1F68,0x301,0x0}, 
+  {0x1F6D,0x1F69,0x301,0x0}, {0x1F6E,0x1F68,0x342,0x0}, {0x1F6F,0x1F69,0x342,0x0}, 
+  {0x1F70,0x3B1,0x300,0x0}, {0x1F71,0x3B1,0x301,0x0}, {0x1F72,0x3B5,0x300,0x0}, 
+  {0x1F73,0x3B5,0x301,0x0}, {0x1F74,0x3B7,0x300,0x0}, {0x1F75,0x3B7,0x301,0x0}, 
+  {0x1F76,0x3B9,0x300,0x0}, {0x1F77,0x3B9,0x301,0x0}, {0x1F78,0x3BF,0x300,0x0}, 
+  {0x1F79,0x3BF,0x301,0x0}, {0x1F7A,0x3C5,0x300,0x0}, {0x1F7B,0x3C5,0x301,0x0}, 
+  {0x1F7C,0x3C9,0x300,0x0}, {0x1F7D,0x3C9,0x301,0x0}, {0x1F80,0x1F00,0x345,0x0}, 
+  {0x1F81,0x1F01,0x345,0x0}, {0x1F82,0x1F02,0x345,0x0}, {0x1F83,0x1F03,0x345,0x0}, 
+  {0x1F84,0x1F04,0x345,0x0}, {0x1F85,0x1F05,0x345,0x0}, {0x1F86,0x1F06,0x345,0x0}, 
+  {0x1F87,0x1F07,0x345,0x0}, {0x1F88,0x1F08,0x345,0x0}, {0x1F89,0x1F09,0x345,0x0}, 
+  {0x1F8A,0x1F0A,0x345,0x0}, {0x1F8B,0x1F0B,0x345,0x0}, {0x1F8C,0x1F0C,0x345,0x0}, 
+  {0x1F8D,0x1F0D,0x345,0x0}, {0x1F8E,0x1F0E,0x345,0x0}, {0x1F8F,0x1F0F,0x345,0x0}, 
+  {0x1F90,0x1F20,0x345,0x0}, {0x1F91,0x1F21,0x345,0x0}, {0x1F92,0x1F22,0x345,0x0}, 
+  {0x1F93,0x1F23,0x345,0x0}, {0x1F94,0x1F24,0x345,0x0}, {0x1F95,0x1F25,0x345,0x0}, 
+  {0x1F96,0x1F26,0x345,0x0}, {0x1F97,0x1F27,0x345,0x0}, {0x1F98,0x1F28,0x345,0x0}, 
+  {0x1F99,0x1F29,0x345,0x0}, {0x1F9A,0x1F2A,0x345,0x0}, {0x1F9B,0x1F2B,0x345,0x0}, 
+  {0x1F9C,0x1F2C,0x345,0x0}, {0x1F9D,0x1F2D,0x345,0x0}, {0x1F9E,0x1F2E,0x345,0x0}, 
+  {0x1F9F,0x1F2F,0x345,0x0}, {0x1FA0,0x1F60,0x345,0x0}, {0x1FA1,0x1F61,0x345,0x0}, 
+  {0x1FA2,0x1F62,0x345,0x0}, {0x1FA3,0x1F63,0x345,0x0}, {0x1FA4,0x1F64,0x345,0x0}, 
+  {0x1FA5,0x1F65,0x345,0x0}, {0x1FA6,0x1F66,0x345,0x0}, {0x1FA7,0x1F67,0x345,0x0}, 
+  {0x1FA8,0x1F68,0x345,0x0}, {0x1FA9,0x1F69,0x345,0x0}, {0x1FAA,0x1F6A,0x345,0x0}, 
+  {0x1FAB,0x1F6B,0x345,0x0}, {0x1FAC,0x1F6C,0x345,0x0}, {0x1FAD,0x1F6D,0x345,0x0}, 
+  {0x1FAE,0x1F6E,0x345,0x0}, {0x1FAF,0x1F6F,0x345,0x0}, {0x1FB0,0x3B1,0x306,0x0}, 
+  {0x1FB1,0x3B1,0x304,0x0}, {0x1FB2,0x1F70,0x345,0x0}, {0x1FB3,0x3B1,0x345,0x0}, 
+  {0x1FB4,0x1F71,0x345,0x0}, {0x1FB6,0x3B1,0x342,0x0}, {0x1FB7,0x1FB6,0x345,0x0}, 
+  {0x1FB8,0x391,0x306,0x0}, {0x1FB9,0x391,0x304,0x0}, {0x1FBA,0x391,0x300,0x0}, 
+  {0x1FBB,0x391,0x301,0x0}, {0x1FBC,0x391,0x345,0x0}, {0x1FBE,0x3B9,0x0,0x0}, 
+  {0x1FC1,0xA8,0x342,0x0}, {0x1FC2,0x1F74,0x345,0x0}, {0x1FC3,0x3B7,0x345,0x0}, 
+  {0x1FC4,0x1F75,0x345,0x0}, {0x1FC6,0x3B7,0x342,0x0}, {0x1FC7,0x1FC6,0x345,0x0}, 
+  {0x1FC8,0x395,0x300,0x0}, {0x1FC9,0x395,0x301,0x0}, {0x1FCA,0x397,0x300,0x0}, 
+  {0x1FCB,0x397,0x301,0x0}, {0x1FCC,0x397,0x345,0x0}, {0x1FCD,0x1FBF,0x300,0x0}, 
+  {0x1FCE,0x1FBF,0x301,0x0}, {0x1FCF,0x1FBF,0x342,0x0}, {0x1FD0,0x3B9,0x306,0x0}, 
+  {0x1FD1,0x3B9,0x304,0x0}, {0x1FD2,0x3CA,0x300,0x0}, {0x1FD3,0x3CA,0x301,0x0}, 
+  {0x1FD6,0x3B9,0x342,0x0}, {0x1FD7,0x3CA,0x342,0x0}, {0x1FD8,0x399,0x306,0x0}, 
+  {0x1FD9,0x399,0x304,0x0}, {0x1FDA,0x399,0x300,0x0}, {0x1FDB,0x399,0x301,0x0}, 
+  {0x1FDD,0x1FFE,0x300,0x0}, {0x1FDE,0x1FFE,0x301,0x0}, {0x1FDF,0x1FFE,0x342,0x0}, 
+  {0x1FE0,0x3C5,0x306,0x0}, {0x1FE1,0x3C5,0x304,0x0}, {0x1FE2,0x3CB,0x300,0x0}, 
+  {0x1FE3,0x3CB,0x301,0x0}, {0x1FE4,0x3C1,0x313,0x0}, {0x1FE5,0x3C1,0x314,0x0}, 
+  {0x1FE6,0x3C5,0x342,0x0}, {0x1FE7,0x3CB,0x342,0x0}, {0x1FE8,0x3A5,0x306,0x0}, 
+  {0x1FE9,0x3A5,0x304,0x0}, {0x1FEA,0x3A5,0x300,0x0}, {0x1FEB,0x3A5,0x301,0x0}, 
+  {0x1FEC,0x3A1,0x314,0x0}, {0x1FED,0xA8,0x300,0x0}, {0x1FEE,0xA8,0x301,0x0}, 
+  {0x1FEF,0x60,0x0,0x0}, {0x1FF2,0x1F7C,0x345,0x0}, {0x1FF3,0x3C9,0x345,0x0}, 
+  {0x1FF4,0x1F79,0x345,0x0}, {0x1FF6,0x3C9,0x342,0x0}, {0x1FF7,0x1FF6,0x345,0x0}, 
+  {0x1FF8,0x39F,0x300,0x0}, {0x1FF9,0x39F,0x301,0x0}, {0x1FFA,0x3A9,0x300,0x0}, 
+  {0x1FFB,0x3A9,0x301,0x0}, {0x1FFC,0x3A9,0x345,0x0}, {0x1FFD,0xB4,0x0,0x0}, 
+  {0x2000,0x2002,0x0,0x0}, {0x2001,0x2003,0x0,0x0}, {0x2126,0x3A9,0x0,0x0}, 
+  {0x212A,0x4B,0x0,0x0}, {0x212B,0xC5,0x0,0x0}, {0x2204,0x2203,0x338,0x0}, 
+  {0x2209,0x2208,0x338,0x0}, {0x220C,0x220B,0x338,0x0}, {0x2224,0x2223,0x338,0x0}, 
+  {0x2226,0x2225,0x338,0x0}, {0x2241,0x7E,0x338,0x0}, {0x2244,0x2243,0x338,0x0}, 
+  {0x2247,0x2245,0x338,0x0}, {0x2249,0x2248,0x338,0x0}, {0x2260,0x3D,0x338,0x0}, 
+  {0x2262,0x2261,0x338,0x0}, {0x226D,0x224D,0x338,0x0}, {0x226E,0x3C,0x338,0x0}, 
+  {0x226F,0x3E,0x338,0x0}, {0x2270,0x2264,0x338,0x0}, {0x2271,0x2265,0x338,0x0}, 
+  {0x2274,0x2272,0x338,0x0}, {0x2275,0x2273,0x338,0x0}, {0x2278,0x2276,0x338,0x0}, 
+  {0x2279,0x2277,0x338,0x0}, {0x2280,0x227A,0x338,0x0}, {0x2281,0x227B,0x338,0x0}, 
+  {0x2284,0x2282,0x338,0x0}, {0x2285,0x2283,0x338,0x0}, {0x2288,0x2286,0x338,0x0}, 
+  {0x2289,0x2287,0x338,0x0}, {0x22AC,0x22A2,0x338,0x0}, {0x22AD,0x22A8,0x338,0x0}, 
+  {0x22AE,0x22A9,0x338,0x0}, {0x22AF,0x22AB,0x338,0x0}, {0x22E0,0x227C,0x338,0x0}, 
+  {0x22E1,0x227D,0x338,0x0}, {0x22E2,0x2291,0x338,0x0}, {0x22E3,0x2292,0x338,0x0}, 
+  {0x22EA,0x22B2,0x338,0x0}, {0x22EB,0x22B3,0x338,0x0}, {0x22EC,0x22B4,0x338,0x0}, 
+  {0x22ED,0x22B5,0x338,0x0}, {0x2329,0x3008,0x0,0x0}, {0x232A,0x3009,0x0,0x0}, 
+  {0x304C,0x304B,0x3099,0x0}, {0x304E,0x304D,0x3099,0x0}, {0x3050,0x304F,0x3099,0x0}, 
+  {0x3052,0x3051,0x3099,0x0}, {0x3054,0x3053,0x3099,0x0}, {0x3056,0x3055,0x3099,0x0}, 
+  {0x3058,0x3057,0x3099,0x0}, {0x305A,0x3059,0x3099,0x0}, {0x305C,0x305B,0x3099,0x0}, 
+  {0x305E,0x305D,0x3099,0x0}, {0x3060,0x305F,0x3099,0x0}, {0x3062,0x3061,0x3099,0x0}, 
+  {0x3065,0x3064,0x3099,0x0}, {0x3067,0x3066,0x3099,0x0}, {0x3069,0x3068,0x3099,0x0}, 
+  {0x3070,0x306F,0x3099,0x0}, {0x3071,0x306F,0x309A,0x0}, {0x3073,0x3072,0x3099,0x0}, 
+  {0x3074,0x3072,0x309A,0x0}, {0x3076,0x3075,0x3099,0x0}, {0x3077,0x3075,0x309A,0x0}, 
+  {0x3079,0x3078,0x3099,0x0}, {0x307A,0x3078,0x309A,0x0}, {0x307C,0x307B,0x3099,0x0}, 
+  {0x307D,0x307B,0x309A,0x0}, {0x3094,0x3046,0x3099,0x0}, {0x309E,0x309D,0x3099,0x0}, 
+  {0x30AC,0x30AB,0x3099,0x0}, {0x30AE,0x30AD,0x3099,0x0}, {0x30B0,0x30AF,0x3099,0x0}, 
+  {0x30B2,0x30B1,0x3099,0x0}, {0x30B4,0x30B3,0x3099,0x0}, {0x30B6,0x30B5,0x3099,0x0}, 
+  {0x30B8,0x30B7,0x3099,0x0}, {0x30BA,0x30B9,0x3099,0x0}, {0x30BC,0x30BB,0x3099,0x0}, 
+  {0x30BE,0x30BD,0x3099,0x0}, {0x30C0,0x30BF,0x3099,0x0}, {0x30C2,0x30C1,0x3099,0x0}, 
+  {0x30C5,0x30C4,0x3099,0x0}, {0x30C7,0x30C6,0x3099,0x0}, {0x30C9,0x30C8,0x3099,0x0}, 
+  {0x30D0,0x30CF,0x3099,0x0}, {0x30D1,0x30CF,0x309A,0x0}, {0x30D3,0x30D2,0x3099,0x0}, 
+  {0x30D4,0x30D2,0x309A,0x0}, {0x30D6,0x30D5,0x3099,0x0}, {0x30D7,0x30D5,0x309A,0x0}, 
+  {0x30D9,0x30D8,0x3099,0x0}, {0x30DA,0x30D8,0x309A,0x0}, {0x30DC,0x30DB,0x3099,0x0}, 
+  {0x30DD,0x30DB,0x309A,0x0}, {0x30F4,0x30A6,0x3099,0x0}, {0x30F7,0x30EF,0x3099,0x0}, 
+  {0x30F8,0x30F0,0x3099,0x0}, {0x30F9,0x30F1,0x3099,0x0}, {0x30FA,0x30F2,0x3099,0x0}, 
+  {0x30FE,0x30FD,0x3099,0x0}, {0xF900,0x8C48,0x0,0x0}, {0xF901,0x66F4,0x0,0x0}, 
+  {0xF902,0x8ECA,0x0,0x0}, {0xF903,0x8CC8,0x0,0x0}, {0xF904,0x6ED1,0x0,0x0}, 
+  {0xF905,0x4E32,0x0,0x0}, {0xF906,0x53E5,0x0,0x0}, {0xF907,0x9F9C,0x0,0x0}, 
+  {0xF908,0x9F9C,0x0,0x0}, {0xF909,0x5951,0x0,0x0}, {0xF90A,0x91D1,0x0,0x0}, 
+  {0xF90B,0x5587,0x0,0x0}, {0xF90C,0x5948,0x0,0x0}, {0xF90D,0x61F6,0x0,0x0}, 
+  {0xF90E,0x7669,0x0,0x0}, {0xF90F,0x7F85,0x0,0x0}, {0xF910,0x863F,0x0,0x0}, 
+  {0xF911,0x87BA,0x0,0x0}, {0xF912,0x88F8,0x0,0x0}, {0xF913,0x908F,0x0,0x0}, 
+  {0xF914,0x6A02,0x0,0x0}, {0xF915,0x6D1B,0x0,0x0}, {0xF916,0x70D9,0x0,0x0}, 
+  {0xF917,0x73DE,0x0,0x0}, {0xF918,0x843D,0x0,0x0}, {0xF919,0x916A,0x0,0x0}, 
+  {0xF91A,0x99F1,0x0,0x0}, {0xF91B,0x4E82,0x0,0x0}, {0xF91C,0x5375,0x0,0x0}, 
+  {0xF91D,0x6B04,0x0,0x0}, {0xF91E,0x721B,0x0,0x0}, {0xF91F,0x862D,0x0,0x0}, 
+  {0xF920,0x9E1E,0x0,0x0}, {0xF921,0x5D50,0x0,0x0}, {0xF922,0x6FEB,0x0,0x0}, 
+  {0xF923,0x85CD,0x0,0x0}, {0xF924,0x8964,0x0,0x0}, {0xF925,0x62C9,0x0,0x0}, 
+  {0xF926,0x81D8,0x0,0x0}, {0xF927,0x881F,0x0,0x0}, {0xF928,0x5ECA,0x0,0x0}, 
+  {0xF929,0x6717,0x0,0x0}, {0xF92A,0x6D6A,0x0,0x0}, {0xF92B,0x72FC,0x0,0x0}, 
+  {0xF92C,0x90CE,0x0,0x0}, {0xF92D,0x4F86,0x0,0x0}, {0xF92E,0x51B7,0x0,0x0}, 
+  {0xF92F,0x52DE,0x0,0x0}, {0xF930,0x64C4,0x0,0x0}, {0xF931,0x6AD3,0x0,0x0}, 
+  {0xF932,0x7210,0x0,0x0}, {0xF933,0x76E7,0x0,0x0}, {0xF934,0x8001,0x0,0x0}, 
+  {0xF935,0x8606,0x0,0x0}, {0xF936,0x865C,0x0,0x0}, {0xF937,0x8DEF,0x0,0x0}, 
+  {0xF938,0x9732,0x0,0x0}, {0xF939,0x9B6F,0x0,0x0}, {0xF93A,0x9DFA,0x0,0x0}, 
+  {0xF93B,0x788C,0x0,0x0}, {0xF93C,0x797F,0x0,0x0}, {0xF93D,0x7DA0,0x0,0x0}, 
+  {0xF93E,0x83C9,0x0,0x0}, {0xF93F,0x9304,0x0,0x0}, {0xF940,0x9E7F,0x0,0x0}, 
+  {0xF941,0x8AD6,0x0,0x0}, {0xF942,0x58DF,0x0,0x0}, {0xF943,0x5F04,0x0,0x0}, 
+  {0xF944,0x7C60,0x0,0x0}, {0xF945,0x807E,0x0,0x0}, {0xF946,0x7262,0x0,0x0}, 
+  {0xF947,0x78CA,0x0,0x0}, {0xF948,0x8CC2,0x0,0x0}, {0xF949,0x96F7,0x0,0x0}, 
+  {0xF94A,0x58D8,0x0,0x0}, {0xF94B,0x5C62,0x0,0x0}, {0xF94C,0x6A13,0x0,0x0}, 
+  {0xF94D,0x6DDA,0x0,0x0}, {0xF94E,0x6F0F,0x0,0x0}, {0xF94F,0x7D2F,0x0,0x0}, 
+  {0xF950,0x7E37,0x0,0x0}, {0xF951,0x96FB,0x0,0x0}, {0xF952,0x52D2,0x0,0x0}, 
+  {0xF953,0x808B,0x0,0x0}, {0xF954,0x51DC,0x0,0x0}, {0xF955,0x51CC,0x0,0x0}, 
+  {0xF956,0x7A1C,0x0,0x0}, {0xF957,0x7DBE,0x0,0x0}, {0xF958,0x83F1,0x0,0x0}, 
+  {0xF959,0x9675,0x0,0x0}, {0xF95A,0x8B80,0x0,0x0}, {0xF95B,0x62CF,0x0,0x0}, 
+  {0xF95C,0x6A02,0x0,0x0}, {0xF95D,0x8AFE,0x0,0x0}, {0xF95E,0x4E39,0x0,0x0}, 
+  {0xF95F,0x5BE7,0x0,0x0}, {0xF960,0x6012,0x0,0x0}, {0xF961,0x7387,0x0,0x0}, 
+  {0xF962,0x7570,0x0,0x0}, {0xF963,0x5317,0x0,0x0}, {0xF964,0x78FB,0x0,0x0}, 
+  {0xF965,0x4FBF,0x0,0x0}, {0xF966,0x5FA9,0x0,0x0}, {0xF967,0x4E0D,0x0,0x0}, 
+  {0xF968,0x6CCC,0x0,0x0}, {0xF969,0x6578,0x0,0x0}, {0xF96A,0x7D22,0x0,0x0}, 
+  {0xF96B,0x53C3,0x0,0x0}, {0xF96C,0x585E,0x0,0x0}, {0xF96D,0x7701,0x0,0x0}, 
+  {0xF96E,0x8449,0x0,0x0}, {0xF96F,0x8AAA,0x0,0x0}, {0xF970,0x6BBA,0x0,0x0}, 
+  {0xF971,0x8FB0,0x0,0x0}, {0xF972,0x6C88,0x0,0x0}, {0xF973,0x62FE,0x0,0x0}, 
+  {0xF974,0x82E5,0x0,0x0}, {0xF975,0x63A0,0x0,0x0}, {0xF976,0x7565,0x0,0x0}, 
+  {0xF977,0x4EAE,0x0,0x0}, {0xF978,0x5169,0x0,0x0}, {0xF979,0x51C9,0x0,0x0}, 
+  {0xF97A,0x6881,0x0,0x0}, {0xF97B,0x7CE7,0x0,0x0}, {0xF97C,0x826F,0x0,0x0}, 
+  {0xF97D,0x8AD2,0x0,0x0}, {0xF97E,0x91CF,0x0,0x0}, {0xF97F,0x52F5,0x0,0x0}, 
+  {0xF980,0x5442,0x0,0x0}, {0xF981,0x5973,0x0,0x0}, {0xF982,0x5EEC,0x0,0x0}, 
+  {0xF983,0x65C5,0x0,0x0}, {0xF984,0x6FFE,0x0,0x0}, {0xF985,0x792A,0x0,0x0}, 
+  {0xF986,0x95AD,0x0,0x0}, {0xF987,0x9A6A,0x0,0x0}, {0xF988,0x9E97,0x0,0x0}, 
+  {0xF989,0x9ECE,0x0,0x0}, {0xF98A,0x529B,0x0,0x0}, {0xF98B,0x66C6,0x0,0x0}, 
+  {0xF98C,0x6B77,0x0,0x0}, {0xF98D,0x8F62,0x0,0x0}, {0xF98E,0x5E74,0x0,0x0}, 
+  {0xF98F,0x6190,0x0,0x0}, {0xF990,0x6200,0x0,0x0}, {0xF991,0x649A,0x0,0x0}, 
+  {0xF992,0x6F23,0x0,0x0}, {0xF993,0x7149,0x0,0x0}, {0xF994,0x7489,0x0,0x0}, 
+  {0xF995,0x79CA,0x0,0x0}, {0xF996,0x7DF4,0x0,0x0}, {0xF997,0x806F,0x0,0x0}, 
+  {0xF998,0x8F26,0x0,0x0}, {0xF999,0x84EE,0x0,0x0}, {0xF99A,0x9023,0x0,0x0}, 
+  {0xF99B,0x934A,0x0,0x0}, {0xF99C,0x5217,0x0,0x0}, {0xF99D,0x52A3,0x0,0x0}, 
+  {0xF99E,0x54BD,0x0,0x0}, {0xF99F,0x70C8,0x0,0x0}, {0xF9A0,0x88C2,0x0,0x0}, 
+  {0xF9A1,0x8AAA,0x0,0x0}, {0xF9A2,0x5EC9,0x0,0x0}, {0xF9A3,0x5FF5,0x0,0x0}, 
+  {0xF9A4,0x637B,0x0,0x0}, {0xF9A5,0x6BAE,0x0,0x0}, {0xF9A6,0x7C3E,0x0,0x0}, 
+  {0xF9A7,0x7375,0x0,0x0}, {0xF9A8,0x4EE4,0x0,0x0}, {0xF9A9,0x56F9,0x0,0x0}, 
+  {0xF9AA,0x5BE7,0x0,0x0}, {0xF9AB,0x5DBA,0x0,0x0}, {0xF9AC,0x601C,0x0,0x0}, 
+  {0xF9AD,0x73B2,0x0,0x0}, {0xF9AE,0x7469,0x0,0x0}, {0xF9AF,0x7F9A,0x0,0x0}, 
+  {0xF9B0,0x8046,0x0,0x0}, {0xF9B1,0x9234,0x0,0x0}, {0xF9B2,0x96F6,0x0,0x0}, 
+  {0xF9B3,0x9748,0x0,0x0}, {0xF9B4,0x9818,0x0,0x0}, {0xF9B5,0x4F8B,0x0,0x0}, 
+  {0xF9B6,0x79AE,0x0,0x0}, {0xF9B7,0x91B4,0x0,0x0}, {0xF9B8,0x96B8,0x0,0x0}, 
+  {0xF9B9,0x60E1,0x0,0x0}, {0xF9BA,0x4E86,0x0,0x0}, {0xF9BB,0x50DA,0x0,0x0}, 
+  {0xF9BC,0x5BEE,0x0,0x0}, {0xF9BD,0x5C3F,0x0,0x0}, {0xF9BE,0x6599,0x0,0x0}, 
+  {0xF9BF,0x6A02,0x0,0x0}, {0xF9C0,0x71CE,0x0,0x0}, {0xF9C1,0x7642,0x0,0x0}, 
+  {0xF9C2,0x84FC,0x0,0x0}, {0xF9C3,0x907C,0x0,0x0}, {0xF9C4,0x9F8D,0x0,0x0}, 
+  {0xF9C5,0x6688,0x0,0x0}, {0xF9C6,0x962E,0x0,0x0}, {0xF9C7,0x5289,0x0,0x0}, 
+  {0xF9C8,0x677B,0x0,0x0}, {0xF9C9,0x67F3,0x0,0x0}, {0xF9CA,0x6D41,0x0,0x0}, 
+  {0xF9CB,0x6E9C,0x0,0x0}, {0xF9CC,0x7409,0x0,0x0}, {0xF9CD,0x7559,0x0,0x0}, 
+  {0xF9CE,0x786B,0x0,0x0}, {0xF9CF,0x7D10,0x0,0x0}, {0xF9D0,0x985E,0x0,0x0}, 
+  {0xF9D1,0x516D,0x0,0x0}, {0xF9D2,0x622E,0x0,0x0}, {0xF9D3,0x9678,0x0,0x0}, 
+  {0xF9D4,0x502B,0x0,0x0}, {0xF9D5,0x5D19,0x0,0x0}, {0xF9D6,0x6DEA,0x0,0x0}, 
+  {0xF9D7,0x8F2A,0x0,0x0}, {0xF9D8,0x5F8B,0x0,0x0}, {0xF9D9,0x6144,0x0,0x0}, 
+  {0xF9DA,0x6817,0x0,0x0}, {0xF9DB,0x7387,0x0,0x0}, {0xF9DC,0x9686,0x0,0x0}, 
+  {0xF9DD,0x5229,0x0,0x0}, {0xF9DE,0x540F,0x0,0x0}, {0xF9DF,0x5C65,0x0,0x0}, 
+  {0xF9E0,0x6613,0x0,0x0}, {0xF9E1,0x674E,0x0,0x0}, {0xF9E2,0x68A8,0x0,0x0}, 
+  {0xF9E3,0x6CE5,0x0,0x0}, {0xF9E4,0x7406,0x0,0x0}, {0xF9E5,0x75E2,0x0,0x0}, 
+  {0xF9E6,0x7F79,0x0,0x0}, {0xF9E7,0x88CF,0x0,0x0}, {0xF9E8,0x88E1,0x0,0x0}, 
+  {0xF9E9,0x91CC,0x0,0x0}, {0xF9EA,0x96E2,0x0,0x0}, {0xF9EB,0x533F,0x0,0x0}, 
+  {0xF9EC,0x6EBA,0x0,0x0}, {0xF9ED,0x541D,0x0,0x0}, {0xF9EE,0x71D0,0x0,0x0}, 
+  {0xF9EF,0x7498,0x0,0x0}, {0xF9F0,0x85FA,0x0,0x0}, {0xF9F1,0x96A3,0x0,0x0}, 
+  {0xF9F2,0x9C57,0x0,0x0}, {0xF9F3,0x9E9F,0x0,0x0}, {0xF9F4,0x6797,0x0,0x0}, 
+  {0xF9F5,0x6DCB,0x0,0x0}, {0xF9F6,0x81E8,0x0,0x0}, {0xF9F7,0x7ACB,0x0,0x0}, 
+  {0xF9F8,0x7B20,0x0,0x0}, {0xF9F9,0x7C92,0x0,0x0}, {0xF9FA,0x72C0,0x0,0x0}, 
+  {0xF9FB,0x7099,0x0,0x0}, {0xF9FC,0x8B58,0x0,0x0}, {0xF9FD,0x4EC0,0x0,0x0}, 
+  {0xF9FE,0x8336,0x0,0x0}, {0xF9FF,0x523A,0x0,0x0}, {0xFA00,0x5207,0x0,0x0}, 
+  {0xFA01,0x5EA6,0x0,0x0}, {0xFA02,0x62D3,0x0,0x0}, {0xFA03,0x7CD6,0x0,0x0}, 
+  {0xFA04,0x5B85,0x0,0x0}, {0xFA05,0x6D1E,0x0,0x0}, {0xFA06,0x66B4,0x0,0x0}, 
+  {0xFA07,0x8F3B,0x0,0x0}, {0xFA08,0x884C,0x0,0x0}, {0xFA09,0x964D,0x0,0x0}, 
+  {0xFA0A,0x898B,0x0,0x0}, {0xFA0B,0x5ED3,0x0,0x0}, {0xFA0C,0x5140,0x0,0x0}, 
+  {0xFA0D,0x55C0,0x0,0x0}, {0xFA10,0x585A,0x0,0x0}, {0xFA12,0x6674,0x0,0x0}, 
+  {0xFA15,0x51DE,0x0,0x0}, {0xFA16,0x732A,0x0,0x0}, {0xFA17,0x76CA,0x0,0x0}, 
+  {0xFA18,0x793C,0x0,0x0}, {0xFA19,0x795E,0x0,0x0}, {0xFA1A,0x7965,0x0,0x0}, 
+  {0xFA1B,0x798F,0x0,0x0}, {0xFA1C,0x9756,0x0,0x0}, {0xFA1D,0x7CBE,0x0,0x0}, 
+  {0xFA1E,0x7FBD,0x0,0x0}, {0xFA20,0x8612,0x0,0x0}, {0xFA22,0x8AF8,0x0,0x0}, 
+  {0xFA25,0x9038,0x0,0x0}, {0xFA26,0x90FD,0x0,0x0}, {0xFA2A,0x98EF,0x0,0x0}, 
+  {0xFA2B,0x98FC,0x0,0x0}, {0xFA2C,0x9928,0x0,0x0}, {0xFA2D,0x9DB4,0x0,0x0}, 
+  {0xFB1F,0x5F2,0x5B7,0x0}, {0xFB2A,0x5E9,0x5C1,0x0}, {0xFB2B,0x5E9,0x5C2,0x0}, 
+  {0xFB2C,0x5E9,0x5BC,0x5C1}, {0xFB2D,0x5E9,0x5BC,0x5C2}, {0xFB2E,0x5D0,0x5B7,0x0}, 
+  {0xFB2F,0x5D0,0x5B8,0x0}, {0xFB30,0x5D0,0x5BC,0x0}, {0xFB31,0x5D1,0x5BC,0x0}, 
+  {0xFB32,0x5D2,0x5BC,0x0}, {0xFB33,0x5D3,0x5BC,0x0}, {0xFB34,0x5D4,0x5BC,0x0}, 
+  {0xFB35,0x5D5,0x5BC,0x0}, {0xFB36,0x5D6,0x5BC,0x0}, {0xFB38,0x5D8,0x5BC,0x0}, 
+  {0xFB39,0x5D9,0x5BC,0x0}, {0xFB3A,0x5DA,0x5BC,0x0}, {0xFB3B,0x5DB,0x5BC,0x0}, 
+  {0xFB3C,0x5DC,0x5BC,0x0}, {0xFB3E,0x5DE,0x5BC,0x0}, {0xFB40,0x5E0,0x5BC,0x0}, 
+  {0xFB41,0x5E1,0x5BC,0x0}, {0xFB43,0x5E3,0x5BC,0x0}, {0xFB44,0x5E4,0x5BC,0x0}, 
+  {0xFB46,0x5E6,0x5BC,0x0}, {0xFB47,0x5E7,0x5BC,0x0}, {0xFB48,0x5E8,0x5BC,0x0}, 
+  {0xFB49,0x5E9,0x5BC,0x0}, {0xFB4A,0x5EA,0x5BC,0x0}, {0xFB4B,0x5D5,0x5B9,0x0}, 
+  {0xFB4C,0x5D1,0x5BF,0x0}, {0xFB4D,0x5DB,0x5BF,0x0}, {0xFB4E,0x5E4,0x5BF,0x0} 
+};
+
+
+/* All upper-case translations are included here */
+
+#define NUM_TOUPPER_INFO 677
+static const unirelate toupper_info[NUM_TOUPPER_INFO] = {
+  {0x61,0x41}, {0x62,0x42}, {0x63,0x43}, {0x64,0x44}, {0x65,0x45}, 
+  {0x66,0x46}, {0x67,0x47}, {0x68,0x48}, {0x69,0x49}, {0x6A,0x4A}, 
+  {0x6B,0x4B}, {0x6C,0x4C}, {0x6D,0x4D}, {0x6E,0x4E}, {0x6F,0x4F}, 
+  {0x70,0x50}, {0x71,0x51}, {0x72,0x52}, {0x73,0x53}, {0x74,0x54}, 
+  {0x75,0x55}, {0x76,0x56}, {0x77,0x57}, {0x78,0x58}, {0x79,0x59}, 
+  {0x7A,0x5A}, {0xE0,0xC0}, {0xE1,0xC1}, {0xE2,0xC2}, {0xE3,0xC3}, 
+  {0xE4,0xC4}, {0xE5,0xC5}, {0xE6,0xC6}, {0xE7,0xC7}, {0xE8,0xC8}, 
+  {0xE9,0xC9}, {0xEA,0xCA}, {0xEB,0xCB}, {0xEC,0xCC}, {0xED,0xCD}, 
+  {0xEE,0xCE}, {0xEF,0xCF}, {0xF0,0xD0}, {0xF1,0xD1}, {0xF2,0xD2}, 
+  {0xF3,0xD3}, {0xF4,0xD4}, {0xF5,0xD5}, {0xF6,0xD6}, {0xF8,0xD8}, 
+  {0xF9,0xD9}, {0xFA,0xDA}, {0xFB,0xDB}, {0xFC,0xDC}, {0xFD,0xDD}, 
+  {0xFE,0xDE}, {0xFF,0x178}, {0x101,0x100}, {0x103,0x102}, {0x105,0x104}, 
+  {0x107,0x106}, {0x109,0x108}, {0x10B,0x10A}, {0x10D,0x10C}, {0x10F,0x10E}, 
+  {0x111,0x110}, {0x113,0x112}, {0x115,0x114}, {0x117,0x116}, {0x119,0x118}, 
+  {0x11B,0x11A}, {0x11D,0x11C}, {0x11F,0x11E}, {0x121,0x120}, {0x123,0x122}, 
+  {0x125,0x124}, {0x127,0x126}, {0x129,0x128}, {0x12B,0x12A}, {0x12D,0x12C}, 
+  {0x12F,0x12E}, {0x131,0x49}, {0x133,0x132}, {0x135,0x134}, {0x137,0x136}, 
+  {0x13A,0x139}, {0x13C,0x13B}, {0x13E,0x13D}, {0x140,0x13F}, {0x142,0x141}, 
+  {0x144,0x143}, {0x146,0x145}, {0x148,0x147}, {0x14B,0x14A}, {0x14D,0x14C}, 
+  {0x14F,0x14E}, {0x151,0x150}, {0x153,0x152}, {0x155,0x154}, {0x157,0x156}, 
+  {0x159,0x158}, {0x15B,0x15A}, {0x15D,0x15C}, {0x15F,0x15E}, {0x161,0x160}, 
+  {0x163,0x162}, {0x165,0x164}, {0x167,0x166}, {0x169,0x168}, {0x16B,0x16A}, 
+  {0x16D,0x16C}, {0x16F,0x16E}, {0x171,0x170}, {0x173,0x172}, {0x175,0x174}, 
+  {0x177,0x176}, {0x17A,0x179}, {0x17C,0x17B}, {0x17E,0x17D}, {0x17F,0x53}, 
+  {0x183,0x182}, {0x185,0x184}, {0x188,0x187}, {0x18C,0x18B}, {0x192,0x191}, 
+  {0x199,0x198}, {0x1A1,0x1A0}, {0x1A3,0x1A2}, {0x1A5,0x1A4}, {0x1A8,0x1A7}, 
+  {0x1AD,0x1AC}, {0x1B0,0x1AF}, {0x1B4,0x1B3}, {0x1B6,0x1B5}, {0x1B9,0x1B8}, 
+  {0x1BD,0x1BC}, {0x1C5,0x1C4}, {0x1C6,0x1C4}, {0x1C8,0x1C7}, {0x1C9,0x1C7}, 
+  {0x1CB,0x1CA}, {0x1CC,0x1CA}, {0x1CE,0x1CD}, {0x1D0,0x1CF}, {0x1D2,0x1D1}, 
+  {0x1D4,0x1D3}, {0x1D6,0x1D5}, {0x1D8,0x1D7}, {0x1DA,0x1D9}, {0x1DC,0x1DB}, 
+  {0x1DD,0x18E}, {0x1DF,0x1DE}, {0x1E1,0x1E0}, {0x1E3,0x1E2}, {0x1E5,0x1E4}, 
+  {0x1E7,0x1E6}, {0x1E9,0x1E8}, {0x1EB,0x1EA}, {0x1ED,0x1EC}, {0x1EF,0x1EE}, 
+  {0x1F2,0x1F1}, {0x1F3,0x1F1}, {0x1F5,0x1F4}, {0x1FB,0x1FA}, {0x1FD,0x1FC}, 
+  {0x1FF,0x1FE}, {0x201,0x200}, {0x203,0x202}, {0x205,0x204}, {0x207,0x206}, 
+  {0x209,0x208}, {0x20B,0x20A}, {0x20D,0x20C}, {0x20F,0x20E}, {0x211,0x210}, 
+  {0x213,0x212}, {0x215,0x214}, {0x217,0x216}, {0x253,0x181}, {0x254,0x186}, 
+  {0x256,0x189}, {0x257,0x18A}, {0x259,0x18F}, {0x25B,0x190}, {0x260,0x193}, 
+  {0x263,0x194}, {0x268,0x197}, {0x269,0x196}, {0x26F,0x19C}, {0x272,0x19D}, 
+  {0x275,0x19F}, {0x283,0x1A9}, {0x288,0x1AE}, {0x28A,0x1B1}, {0x28B,0x1B2}, 
+  {0x292,0x1B7}, {0x3AC,0x386}, {0x3AD,0x388}, {0x3AE,0x389}, {0x3AF,0x38A}, 
+  {0x3B1,0x391}, {0x3B2,0x392}, {0x3B3,0x393}, {0x3B4,0x394}, {0x3B5,0x395}, 
+  {0x3B6,0x396}, {0x3B7,0x397}, {0x3B8,0x398}, {0x3B9,0x399}, {0x3BA,0x39A}, 
+  {0x3BB,0x39B}, {0x3BC,0x39C}, {0x3BD,0x39D}, {0x3BE,0x39E}, {0x3BF,0x39F}, 
+  {0x3C0,0x3A0}, {0x3C1,0x3A1}, {0x3C2,0x3A3}, {0x3C3,0x3A3}, {0x3C4,0x3A4}, 
+  {0x3C5,0x3A5}, {0x3C6,0x3A6}, {0x3C7,0x3A7}, {0x3C8,0x3A8}, {0x3C9,0x3A9}, 
+  {0x3CA,0x3AA}, {0x3CB,0x3AB}, {0x3CC,0x38C}, {0x3CD,0x38E}, {0x3CE,0x38F}, 
+  {0x3D0,0x392}, {0x3D1,0x398}, {0x3D5,0x3A6}, {0x3D6,0x3A0}, {0x3E3,0x3E2}, 
+  {0x3E5,0x3E4}, {0x3E7,0x3E6}, {0x3E9,0x3E8}, {0x3EB,0x3EA}, {0x3ED,0x3EC}, 
+  {0x3EF,0x3EE}, {0x3F0,0x39A}, {0x3F1,0x3A1}, {0x3F2,0x3A3}, {0x430,0x410}, 
+  {0x431,0x411}, {0x432,0x412}, {0x433,0x413}, {0x434,0x414}, {0x435,0x415}, 
+  {0x436,0x416}, {0x437,0x417}, {0x438,0x418}, {0x439,0x419}, {0x43A,0x41A}, 
+  {0x43B,0x41B}, {0x43C,0x41C}, {0x43D,0x41D}, {0x43E,0x41E}, {0x43F,0x41F}, 
+  {0x440,0x420}, {0x441,0x421}, {0x442,0x422}, {0x443,0x423}, {0x444,0x424}, 
+  {0x445,0x425}, {0x446,0x426}, {0x447,0x427}, {0x448,0x428}, {0x449,0x429}, 
+  {0x44A,0x42A}, {0x44B,0x42B}, {0x44C,0x42C}, {0x44D,0x42D}, {0x44E,0x42E}, 
+  {0x44F,0x42F}, {0x451,0x401}, {0x452,0x402}, {0x453,0x403}, {0x454,0x404}, 
+  {0x455,0x405}, {0x456,0x406}, {0x457,0x407}, {0x458,0x408}, {0x459,0x409}, 
+  {0x45A,0x40A}, {0x45B,0x40B}, {0x45C,0x40C}, {0x45E,0x40E}, {0x45F,0x40F}, 
+  {0x461,0x460}, {0x463,0x462}, {0x465,0x464}, {0x467,0x466}, {0x469,0x468}, 
+  {0x46B,0x46A}, {0x46D,0x46C}, {0x46F,0x46E}, {0x471,0x470}, {0x473,0x472}, 
+  {0x475,0x474}, {0x477,0x476}, {0x479,0x478}, {0x47B,0x47A}, {0x47D,0x47C}, 
+  {0x47F,0x47E}, {0x481,0x480}, {0x491,0x490}, {0x493,0x492}, {0x495,0x494}, 
+  {0x497,0x496}, {0x499,0x498}, {0x49B,0x49A}, {0x49D,0x49C}, {0x49F,0x49E}, 
+  {0x4A1,0x4A0}, {0x4A3,0x4A2}, {0x4A5,0x4A4}, {0x4A7,0x4A6}, {0x4A9,0x4A8}, 
+  {0x4AB,0x4AA}, {0x4AD,0x4AC}, {0x4AF,0x4AE}, {0x4B1,0x4B0}, {0x4B3,0x4B2}, 
+  {0x4B5,0x4B4}, {0x4B7,0x4B6}, {0x4B9,0x4B8}, {0x4BB,0x4BA}, {0x4BD,0x4BC}, 
+  {0x4BF,0x4BE}, {0x4C2,0x4C1}, {0x4C4,0x4C3}, {0x4C8,0x4C7}, {0x4CC,0x4CB}, 
+  {0x4D1,0x4D0}, {0x4D3,0x4D2}, {0x4D5,0x4D4}, {0x4D7,0x4D6}, {0x4D9,0x4D8}, 
+  {0x4DB,0x4DA}, {0x4DD,0x4DC}, {0x4DF,0x4DE}, {0x4E1,0x4E0}, {0x4E3,0x4E2}, 
+  {0x4E5,0x4E4}, {0x4E7,0x4E6}, {0x4E9,0x4E8}, {0x4EB,0x4EA}, {0x4EF,0x4EE}, 
+  {0x4F1,0x4F0}, {0x4F3,0x4F2}, {0x4F5,0x4F4}, {0x4F9,0x4F8}, {0x561,0x531}, 
+  {0x562,0x532}, {0x563,0x533}, {0x564,0x534}, {0x565,0x535}, {0x566,0x536}, 
+  {0x567,0x537}, {0x568,0x538}, {0x569,0x539}, {0x56A,0x53A}, {0x56B,0x53B}, 
+  {0x56C,0x53C}, {0x56D,0x53D}, {0x56E,0x53E}, {0x56F,0x53F}, {0x570,0x540}, 
+  {0x571,0x541}, {0x572,0x542}, {0x573,0x543}, {0x574,0x544}, {0x575,0x545}, 
+  {0x576,0x546}, {0x577,0x547}, {0x578,0x548}, {0x579,0x549}, {0x57A,0x54A}, 
+  {0x57B,0x54B}, {0x57C,0x54C}, {0x57D,0x54D}, {0x57E,0x54E}, {0x57F,0x54F}, 
+  {0x580,0x550}, {0x581,0x551}, {0x582,0x552}, {0x583,0x553}, {0x584,0x554}, 
+  {0x585,0x555}, {0x586,0x556}, {0x1E01,0x1E00}, {0x1E03,0x1E02}, {0x1E05,0x1E04}, 
+  {0x1E07,0x1E06}, {0x1E09,0x1E08}, {0x1E0B,0x1E0A}, {0x1E0D,0x1E0C}, {0x1E0F,0x1E0E}, 
+  {0x1E11,0x1E10}, {0x1E13,0x1E12}, {0x1E15,0x1E14}, {0x1E17,0x1E16}, {0x1E19,0x1E18}, 
+  {0x1E1B,0x1E1A}, {0x1E1D,0x1E1C}, {0x1E1F,0x1E1E}, {0x1E21,0x1E20}, {0x1E23,0x1E22}, 
+  {0x1E25,0x1E24}, {0x1E27,0x1E26}, {0x1E29,0x1E28}, {0x1E2B,0x1E2A}, {0x1E2D,0x1E2C}, 
+  {0x1E2F,0x1E2E}, {0x1E31,0x1E30}, {0x1E33,0x1E32}, {0x1E35,0x1E34}, {0x1E37,0x1E36}, 
+  {0x1E39,0x1E38}, {0x1E3B,0x1E3A}, {0x1E3D,0x1E3C}, {0x1E3F,0x1E3E}, {0x1E41,0x1E40}, 
+  {0x1E43,0x1E42}, {0x1E45,0x1E44}, {0x1E47,0x1E46}, {0x1E49,0x1E48}, {0x1E4B,0x1E4A}, 
+  {0x1E4D,0x1E4C}, {0x1E4F,0x1E4E}, {0x1E51,0x1E50}, {0x1E53,0x1E52}, {0x1E55,0x1E54}, 
+  {0x1E57,0x1E56}, {0x1E59,0x1E58}, {0x1E5B,0x1E5A}, {0x1E5D,0x1E5C}, {0x1E5F,0x1E5E}, 
+  {0x1E61,0x1E60}, {0x1E63,0x1E62}, {0x1E65,0x1E64}, {0x1E67,0x1E66}, {0x1E69,0x1E68}, 
+  {0x1E6B,0x1E6A}, {0x1E6D,0x1E6C}, {0x1E6F,0x1E6E}, {0x1E71,0x1E70}, {0x1E73,0x1E72}, 
+  {0x1E75,0x1E74}, {0x1E77,0x1E76}, {0x1E79,0x1E78}, {0x1E7B,0x1E7A}, {0x1E7D,0x1E7C}, 
+  {0x1E7F,0x1E7E}, {0x1E81,0x1E80}, {0x1E83,0x1E82}, {0x1E85,0x1E84}, {0x1E87,0x1E86}, 
+  {0x1E89,0x1E88}, {0x1E8B,0x1E8A}, {0x1E8D,0x1E8C}, {0x1E8F,0x1E8E}, {0x1E91,0x1E90}, 
+  {0x1E93,0x1E92}, {0x1E95,0x1E94}, {0x1E9B,0x1E60}, {0x1EA1,0x1EA0}, {0x1EA3,0x1EA2}, 
+  {0x1EA5,0x1EA4}, {0x1EA7,0x1EA6}, {0x1EA9,0x1EA8}, {0x1EAB,0x1EAA}, {0x1EAD,0x1EAC}, 
+  {0x1EAF,0x1EAE}, {0x1EB1,0x1EB0}, {0x1EB3,0x1EB2}, {0x1EB5,0x1EB4}, {0x1EB7,0x1EB6}, 
+  {0x1EB9,0x1EB8}, {0x1EBB,0x1EBA}, {0x1EBD,0x1EBC}, {0x1EBF,0x1EBE}, {0x1EC1,0x1EC0}, 
+  {0x1EC3,0x1EC2}, {0x1EC5,0x1EC4}, {0x1EC7,0x1EC6}, {0x1EC9,0x1EC8}, {0x1ECB,0x1ECA}, 
+  {0x1ECD,0x1ECC}, {0x1ECF,0x1ECE}, {0x1ED1,0x1ED0}, {0x1ED3,0x1ED2}, {0x1ED5,0x1ED4}, 
+  {0x1ED7,0x1ED6}, {0x1ED9,0x1ED8}, {0x1EDB,0x1EDA}, {0x1EDD,0x1EDC}, {0x1EDF,0x1EDE}, 
+  {0x1EE1,0x1EE0}, {0x1EE3,0x1EE2}, {0x1EE5,0x1EE4}, {0x1EE7,0x1EE6}, {0x1EE9,0x1EE8}, 
+  {0x1EEB,0x1EEA}, {0x1EED,0x1EEC}, {0x1EEF,0x1EEE}, {0x1EF1,0x1EF0}, {0x1EF3,0x1EF2}, 
+  {0x1EF5,0x1EF4}, {0x1EF7,0x1EF6}, {0x1EF9,0x1EF8}, {0x1F00,0x1F08}, {0x1F01,0x1F09}, 
+  {0x1F02,0x1F0A}, {0x1F03,0x1F0B}, {0x1F04,0x1F0C}, {0x1F05,0x1F0D}, {0x1F06,0x1F0E}, 
+  {0x1F07,0x1F0F}, {0x1F10,0x1F18}, {0x1F11,0x1F19}, {0x1F12,0x1F1A}, {0x1F13,0x1F1B}, 
+  {0x1F14,0x1F1C}, {0x1F15,0x1F1D}, {0x1F20,0x1F28}, {0x1F21,0x1F29}, {0x1F22,0x1F2A}, 
+  {0x1F23,0x1F2B}, {0x1F24,0x1F2C}, {0x1F25,0x1F2D}, {0x1F26,0x1F2E}, {0x1F27,0x1F2F}, 
+  {0x1F30,0x1F38}, {0x1F31,0x1F39}, {0x1F32,0x1F3A}, {0x1F33,0x1F3B}, {0x1F34,0x1F3C}, 
+  {0x1F35,0x1F3D}, {0x1F36,0x1F3E}, {0x1F37,0x1F3F}, {0x1F40,0x1F48}, {0x1F41,0x1F49}, 
+  {0x1F42,0x1F4A}, {0x1F43,0x1F4B}, {0x1F44,0x1F4C}, {0x1F45,0x1F4D}, {0x1F51,0x1F59}, 
+  {0x1F53,0x1F5B}, {0x1F55,0x1F5D}, {0x1F57,0x1F5F}, {0x1F60,0x1F68}, {0x1F61,0x1F69}, 
+  {0x1F62,0x1F6A}, {0x1F63,0x1F6B}, {0x1F64,0x1F6C}, {0x1F65,0x1F6D}, {0x1F66,0x1F6E}, 
+  {0x1F67,0x1F6F}, {0x1F70,0x1FBA}, {0x1F71,0x1FBB}, {0x1F72,0x1FC8}, {0x1F73,0x1FC9}, 
+  {0x1F74,0x1FCA}, {0x1F75,0x1FCB}, {0x1F76,0x1FDA}, {0x1F77,0x1FDB}, {0x1F78,0x1FF8}, 
+  {0x1F79,0x1FF9}, {0x1F7A,0x1FEA}, {0x1F7B,0x1FEB}, {0x1F7C,0x1FFA}, {0x1F7D,0x1FFB}, 
+  {0x1F80,0x1F88}, {0x1F81,0x1F89}, {0x1F82,0x1F8A}, {0x1F83,0x1F8B}, {0x1F84,0x1F8C}, 
+  {0x1F85,0x1F8D}, {0x1F86,0x1F8E}, {0x1F87,0x1F8F}, {0x1F90,0x1F98}, {0x1F91,0x1F99}, 
+  {0x1F92,0x1F9A}, {0x1F93,0x1F9B}, {0x1F94,0x1F9C}, {0x1F95,0x1F9D}, {0x1F96,0x1F9E}, 
+  {0x1F97,0x1F9F}, {0x1FA0,0x1FA8}, {0x1FA1,0x1FA9}, {0x1FA2,0x1FAA}, {0x1FA3,0x1FAB}, 
+  {0x1FA4,0x1FAC}, {0x1FA5,0x1FAD}, {0x1FA6,0x1FAE}, {0x1FA7,0x1FAF}, {0x1FB0,0x1FB8}, 
+  {0x1FB1,0x1FB9}, {0x1FB3,0x1FBC}, {0x1FC3,0x1FCC}, {0x1FD0,0x1FD8}, {0x1FD1,0x1FD9}, 
+  {0x1FE0,0x1FE8}, {0x1FE1,0x1FE9}, {0x1FE5,0x1FEC}, {0x1FF3,0x1FFC}, {0x2170,0x2160}, 
+  {0x2171,0x2161}, {0x2172,0x2162}, {0x2173,0x2163}, {0x2174,0x2164}, {0x2175,0x2165}, 
+  {0x2176,0x2166}, {0x2177,0x2167}, {0x2178,0x2168}, {0x2179,0x2169}, {0x217A,0x216A}, 
+  {0x217B,0x216B}, {0x217C,0x216C}, {0x217D,0x216D}, {0x217E,0x216E}, {0x217F,0x216F}, 
+  {0x24D0,0x24B6}, {0x24D1,0x24B7}, {0x24D2,0x24B8}, {0x24D3,0x24B9}, {0x24D4,0x24BA}, 
+  {0x24D5,0x24BB}, {0x24D6,0x24BC}, {0x24D7,0x24BD}, {0x24D8,0x24BE}, {0x24D9,0x24BF}, 
+  {0x24DA,0x24C0}, {0x24DB,0x24C1}, {0x24DC,0x24C2}, {0x24DD,0x24C3}, {0x24DE,0x24C4}, 
+  {0x24DF,0x24C5}, {0x24E0,0x24C6}, {0x24E1,0x24C7}, {0x24E2,0x24C8}, {0x24E3,0x24C9}, 
+  {0x24E4,0x24CA}, {0x24E5,0x24CB}, {0x24E6,0x24CC}, {0x24E7,0x24CD}, {0x24E8,0x24CE}, 
+  {0x24E9,0x24CF}, {0xFF41,0xFF21}, {0xFF42,0xFF22}, {0xFF43,0xFF23}, {0xFF44,0xFF24}, 
+  {0xFF45,0xFF25}, {0xFF46,0xFF26}, {0xFF47,0xFF27}, {0xFF48,0xFF28}, {0xFF49,0xFF29}, 
+  {0xFF4A,0xFF2A}, {0xFF4B,0xFF2B}, {0xFF4C,0xFF2C}, {0xFF4D,0xFF2D}, {0xFF4E,0xFF2E}, 
+  {0xFF4F,0xFF2F}, {0xFF50,0xFF30}, {0xFF51,0xFF31}, {0xFF52,0xFF32}, {0xFF53,0xFF33}, 
+  {0xFF54,0xFF34}, {0xFF55,0xFF35}, {0xFF56,0xFF36}, {0xFF57,0xFF37}, {0xFF58,0xFF38}, 
+  {0xFF59,0xFF39}, {0xFF5A,0xFF3A} 
+};
+
+
+
+/* All lower-case translations are included here */
+
+#define NUM_TOLOWER_INFO 705
+static const unirelate tolower_info[NUM_TOLOWER_INFO] = {
+  {0x41,0x61}, {0x42,0x62}, {0x43,0x63}, {0x44,0x64}, {0x45,0x65}, 
+  {0x46,0x66}, {0x47,0x67}, {0x48,0x68}, {0x49,0x69}, {0x4A,0x6A}, 
+  {0x4B,0x6B}, {0x4C,0x6C}, {0x4D,0x6D}, {0x4E,0x6E}, {0x4F,0x6F}, 
+  {0x50,0x70}, {0x51,0x71}, {0x52,0x72}, {0x53,0x73}, {0x54,0x74}, 
+  {0x55,0x75}, {0x56,0x76}, {0x57,0x77}, {0x58,0x78}, {0x59,0x79}, 
+  {0x5A,0x7A}, {0xC0,0xE0}, {0xC1,0xE1}, {0xC2,0xE2}, {0xC3,0xE3}, 
+  {0xC4,0xE4}, {0xC5,0xE5}, {0xC6,0xE6}, {0xC7,0xE7}, {0xC8,0xE8}, 
+  {0xC9,0xE9}, {0xCA,0xEA}, {0xCB,0xEB}, {0xCC,0xEC}, {0xCD,0xED}, 
+  {0xCE,0xEE}, {0xCF,0xEF}, {0xD0,0xF0}, {0xD1,0xF1}, {0xD2,0xF2}, 
+  {0xD3,0xF3}, {0xD4,0xF4}, {0xD5,0xF5}, {0xD6,0xF6}, {0xD8,0xF8}, 
+  {0xD9,0xF9}, {0xDA,0xFA}, {0xDB,0xFB}, {0xDC,0xFC}, {0xDD,0xFD}, 
+  {0xDE,0xFE}, {0x100,0x101}, {0x102,0x103}, {0x104,0x105}, {0x106,0x107}, 
+  {0x108,0x109}, {0x10A,0x10B}, {0x10C,0x10D}, {0x10E,0x10F}, {0x110,0x111}, 
+  {0x112,0x113}, {0x114,0x115}, {0x116,0x117}, {0x118,0x119}, {0x11A,0x11B}, 
+  {0x11C,0x11D}, {0x11E,0x11F}, {0x120,0x121}, {0x122,0x123}, {0x124,0x125}, 
+  {0x126,0x127}, {0x128,0x129}, {0x12A,0x12B}, {0x12C,0x12D}, {0x12E,0x12F}, 
+  {0x130,0x69}, {0x132,0x133}, {0x134,0x135}, {0x136,0x137}, {0x139,0x13A}, 
+  {0x13B,0x13C}, {0x13D,0x13E}, {0x13F,0x140}, {0x141,0x142}, {0x143,0x144}, 
+  {0x145,0x146}, {0x147,0x148}, {0x14A,0x14B}, {0x14C,0x14D}, {0x14E,0x14F}, 
+  {0x150,0x151}, {0x152,0x153}, {0x154,0x155}, {0x156,0x157}, {0x158,0x159}, 
+  {0x15A,0x15B}, {0x15C,0x15D}, {0x15E,0x15F}, {0x160,0x161}, {0x162,0x163}, 
+  {0x164,0x165}, {0x166,0x167}, {0x168,0x169}, {0x16A,0x16B}, {0x16C,0x16D}, 
+  {0x16E,0x16F}, {0x170,0x171}, {0x172,0x173}, {0x174,0x175}, {0x176,0x177}, 
+  {0x178,0xFF}, {0x179,0x17A}, {0x17B,0x17C}, {0x17D,0x17E}, {0x181,0x253}, 
+  {0x182,0x183}, {0x184,0x185}, {0x186,0x254}, {0x187,0x188}, {0x189,0x256}, 
+  {0x18A,0x257}, {0x18B,0x18C}, {0x18E,0x1DD}, {0x18F,0x259}, {0x190,0x25B}, 
+  {0x191,0x192}, {0x193,0x260}, {0x194,0x263}, {0x196,0x269}, {0x197,0x268}, 
+  {0x198,0x199}, {0x19C,0x26F}, {0x19D,0x272}, {0x19F,0x275}, {0x1A0,0x1A1}, 
+  {0x1A2,0x1A3}, {0x1A4,0x1A5}, {0x1A7,0x1A8}, {0x1A9,0x283}, {0x1AC,0x1AD}, 
+  {0x1AE,0x288}, {0x1AF,0x1B0}, {0x1B1,0x28A}, {0x1B2,0x28B}, {0x1B3,0x1B4}, 
+  {0x1B5,0x1B6}, {0x1B7,0x292}, {0x1B8,0x1B9}, {0x1BC,0x1BD}, {0x1C4,0x1C6}, 
+  {0x1C5,0x1C6}, {0x1C7,0x1C9}, {0x1C8,0x1C9}, {0x1CA,0x1CC}, {0x1CB,0x1CC}, 
+  {0x1CD,0x1CE}, {0x1CF,0x1D0}, {0x1D1,0x1D2}, {0x1D3,0x1D4}, {0x1D5,0x1D6}, 
+  {0x1D7,0x1D8}, {0x1D9,0x1DA}, {0x1DB,0x1DC}, {0x1DE,0x1DF}, {0x1E0,0x1E1}, 
+  {0x1E2,0x1E3}, {0x1E4,0x1E5}, {0x1E6,0x1E7}, {0x1E8,0x1E9}, {0x1EA,0x1EB}, 
+  {0x1EC,0x1ED}, {0x1EE,0x1EF}, {0x1F1,0x1F3}, {0x1F2,0x1F3}, {0x1F4,0x1F5}, 
+  {0x1FA,0x1FB}, {0x1FC,0x1FD}, {0x1FE,0x1FF}, {0x200,0x201}, {0x202,0x203}, 
+  {0x204,0x205}, {0x206,0x207}, {0x208,0x209}, {0x20A,0x20B}, {0x20C,0x20D}, 
+  {0x20E,0x20F}, {0x210,0x211}, {0x212,0x213}, {0x214,0x215}, {0x216,0x217}, 
+  {0x386,0x3AC}, {0x388,0x3AD}, {0x389,0x3AE}, {0x38A,0x3AF}, {0x38C,0x3CC}, 
+  {0x38E,0x3CD}, {0x38F,0x3CE}, {0x391,0x3B1}, {0x392,0x3B2}, {0x393,0x3B3}, 
+  {0x394,0x3B4}, {0x395,0x3B5}, {0x396,0x3B6}, {0x397,0x3B7}, {0x398,0x3B8}, 
+  {0x399,0x3B9}, {0x39A,0x3BA}, {0x39B,0x3BB}, {0x39C,0x3BC}, {0x39D,0x3BD}, 
+  {0x39E,0x3BE}, {0x39F,0x3BF}, {0x3A0,0x3C0}, {0x3A1,0x3C1}, {0x3A3,0x3C3}, 
+  {0x3A4,0x3C4}, {0x3A5,0x3C5}, {0x3A6,0x3C6}, {0x3A7,0x3C7}, {0x3A8,0x3C8}, 
+  {0x3A9,0x3C9}, {0x3AA,0x3CA}, {0x3AB,0x3CB}, {0x3E2,0x3E3}, {0x3E4,0x3E5}, 
+  {0x3E6,0x3E7}, {0x3E8,0x3E9}, {0x3EA,0x3EB}, {0x3EC,0x3ED}, {0x3EE,0x3EF}, 
+  {0x401,0x451}, {0x402,0x452}, {0x403,0x453}, {0x404,0x454}, {0x405,0x455}, 
+  {0x406,0x456}, {0x407,0x457}, {0x408,0x458}, {0x409,0x459}, {0x40A,0x45A}, 
+  {0x40B,0x45B}, {0x40C,0x45C}, {0x40E,0x45E}, {0x40F,0x45F}, {0x410,0x430}, 
+  {0x411,0x431}, {0x412,0x432}, {0x413,0x433}, {0x414,0x434}, {0x415,0x435}, 
+  {0x416,0x436}, {0x417,0x437}, {0x418,0x438}, {0x419,0x439}, {0x41A,0x43A}, 
+  {0x41B,0x43B}, {0x41C,0x43C}, {0x41D,0x43D}, {0x41E,0x43E}, {0x41F,0x43F}, 
+  {0x420,0x440}, {0x421,0x441}, {0x422,0x442}, {0x423,0x443}, {0x424,0x444}, 
+  {0x425,0x445}, {0x426,0x446}, {0x427,0x447}, {0x428,0x448}, {0x429,0x449}, 
+  {0x42A,0x44A}, {0x42B,0x44B}, {0x42C,0x44C}, {0x42D,0x44D}, {0x42E,0x44E}, 
+  {0x42F,0x44F}, {0x460,0x461}, {0x462,0x463}, {0x464,0x465}, {0x466,0x467}, 
+  {0x468,0x469}, {0x46A,0x46B}, {0x46C,0x46D}, {0x46E,0x46F}, {0x470,0x471}, 
+  {0x472,0x473}, {0x474,0x475}, {0x476,0x477}, {0x478,0x479}, {0x47A,0x47B}, 
+  {0x47C,0x47D}, {0x47E,0x47F}, {0x480,0x481}, {0x490,0x491}, {0x492,0x493}, 
+  {0x494,0x495}, {0x496,0x497}, {0x498,0x499}, {0x49A,0x49B}, {0x49C,0x49D}, 
+  {0x49E,0x49F}, {0x4A0,0x4A1}, {0x4A2,0x4A3}, {0x4A4,0x4A5}, {0x4A6,0x4A7}, 
+  {0x4A8,0x4A9}, {0x4AA,0x4AB}, {0x4AC,0x4AD}, {0x4AE,0x4AF}, {0x4B0,0x4B1}, 
+  {0x4B2,0x4B3}, {0x4B4,0x4B5}, {0x4B6,0x4B7}, {0x4B8,0x4B9}, {0x4BA,0x4BB}, 
+  {0x4BC,0x4BD}, {0x4BE,0x4BF}, {0x4C1,0x4C2}, {0x4C3,0x4C4}, {0x4C7,0x4C8}, 
+  {0x4CB,0x4CC}, {0x4D0,0x4D1}, {0x4D2,0x4D3}, {0x4D4,0x4D5}, {0x4D6,0x4D7}, 
+  {0x4D8,0x4D9}, {0x4DA,0x4DB}, {0x4DC,0x4DD}, {0x4DE,0x4DF}, {0x4E0,0x4E1}, 
+  {0x4E2,0x4E3}, {0x4E4,0x4E5}, {0x4E6,0x4E7}, {0x4E8,0x4E9}, {0x4EA,0x4EB}, 
+  {0x4EE,0x4EF}, {0x4F0,0x4F1}, {0x4F2,0x4F3}, {0x4F4,0x4F5}, {0x4F8,0x4F9}, 
+  {0x531,0x561}, {0x532,0x562}, {0x533,0x563}, {0x534,0x564}, {0x535,0x565}, 
+  {0x536,0x566}, {0x537,0x567}, {0x538,0x568}, {0x539,0x569}, {0x53A,0x56A}, 
+  {0x53B,0x56B}, {0x53C,0x56C}, {0x53D,0x56D}, {0x53E,0x56E}, {0x53F,0x56F}, 
+  {0x540,0x570}, {0x541,0x571}, {0x542,0x572}, {0x543,0x573}, {0x544,0x574}, 
+  {0x545,0x575}, {0x546,0x576}, {0x547,0x577}, {0x548,0x578}, {0x549,0x579}, 
+  {0x54A,0x57A}, {0x54B,0x57B}, {0x54C,0x57C}, {0x54D,0x57D}, {0x54E,0x57E}, 
+  {0x54F,0x57F}, {0x550,0x580}, {0x551,0x581}, {0x552,0x582}, {0x553,0x583}, 
+  {0x554,0x584}, {0x555,0x585}, {0x556,0x586}, {0x10A0,0x10D0}, {0x10A1,0x10D1}, 
+  {0x10A2,0x10D2}, {0x10A3,0x10D3}, {0x10A4,0x10D4}, {0x10A5,0x10D5}, {0x10A6,0x10D6}, 
+  {0x10A7,0x10D7}, {0x10A8,0x10D8}, {0x10A9,0x10D9}, {0x10AA,0x10DA}, {0x10AB,0x10DB}, 
+  {0x10AC,0x10DC}, {0x10AD,0x10DD}, {0x10AE,0x10DE}, {0x10AF,0x10DF}, {0x10B0,0x10E0}, 
+  {0x10B1,0x10E1}, {0x10B2,0x10E2}, {0x10B3,0x10E3}, {0x10B4,0x10E4}, {0x10B5,0x10E5}, 
+  {0x10B6,0x10E6}, {0x10B7,0x10E7}, {0x10B8,0x10E8}, {0x10B9,0x10E9}, {0x10BA,0x10EA}, 
+  {0x10BB,0x10EB}, {0x10BC,0x10EC}, {0x10BD,0x10ED}, {0x10BE,0x10EE}, {0x10BF,0x10EF}, 
+  {0x10C0,0x10F0}, {0x10C1,0x10F1}, {0x10C2,0x10F2}, {0x10C3,0x10F3}, {0x10C4,0x10F4}, 
+  {0x10C5,0x10F5}, {0x1E00,0x1E01}, {0x1E02,0x1E03}, {0x1E04,0x1E05}, {0x1E06,0x1E07}, 
+  {0x1E08,0x1E09}, {0x1E0A,0x1E0B}, {0x1E0C,0x1E0D}, {0x1E0E,0x1E0F}, {0x1E10,0x1E11}, 
+  {0x1E12,0x1E13}, {0x1E14,0x1E15}, {0x1E16,0x1E17}, {0x1E18,0x1E19}, {0x1E1A,0x1E1B}, 
+  {0x1E1C,0x1E1D}, {0x1E1E,0x1E1F}, {0x1E20,0x1E21}, {0x1E22,0x1E23}, {0x1E24,0x1E25}, 
+  {0x1E26,0x1E27}, {0x1E28,0x1E29}, {0x1E2A,0x1E2B}, {0x1E2C,0x1E2D}, {0x1E2E,0x1E2F}, 
+  {0x1E30,0x1E31}, {0x1E32,0x1E33}, {0x1E34,0x1E35}, {0x1E36,0x1E37}, {0x1E38,0x1E39}, 
+  {0x1E3A,0x1E3B}, {0x1E3C,0x1E3D}, {0x1E3E,0x1E3F}, {0x1E40,0x1E41}, {0x1E42,0x1E43}, 
+  {0x1E44,0x1E45}, {0x1E46,0x1E47}, {0x1E48,0x1E49}, {0x1E4A,0x1E4B}, {0x1E4C,0x1E4D}, 
+  {0x1E4E,0x1E4F}, {0x1E50,0x1E51}, {0x1E52,0x1E53}, {0x1E54,0x1E55}, {0x1E56,0x1E57}, 
+  {0x1E58,0x1E59}, {0x1E5A,0x1E5B}, {0x1E5C,0x1E5D}, {0x1E5E,0x1E5F}, {0x1E60,0x1E61}, 
+  {0x1E62,0x1E63}, {0x1E64,0x1E65}, {0x1E66,0x1E67}, {0x1E68,0x1E69}, {0x1E6A,0x1E6B}, 
+  {0x1E6C,0x1E6D}, {0x1E6E,0x1E6F}, {0x1E70,0x1E71}, {0x1E72,0x1E73}, {0x1E74,0x1E75}, 
+  {0x1E76,0x1E77}, {0x1E78,0x1E79}, {0x1E7A,0x1E7B}, {0x1E7C,0x1E7D}, {0x1E7E,0x1E7F}, 
+  {0x1E80,0x1E81}, {0x1E82,0x1E83}, {0x1E84,0x1E85}, {0x1E86,0x1E87}, {0x1E88,0x1E89}, 
+  {0x1E8A,0x1E8B}, {0x1E8C,0x1E8D}, {0x1E8E,0x1E8F}, {0x1E90,0x1E91}, {0x1E92,0x1E93}, 
+  {0x1E94,0x1E95}, {0x1EA0,0x1EA1}, {0x1EA2,0x1EA3}, {0x1EA4,0x1EA5}, {0x1EA6,0x1EA7}, 
+  {0x1EA8,0x1EA9}, {0x1EAA,0x1EAB}, {0x1EAC,0x1EAD}, {0x1EAE,0x1EAF}, {0x1EB0,0x1EB1}, 
+  {0x1EB2,0x1EB3}, {0x1EB4,0x1EB5}, {0x1EB6,0x1EB7}, {0x1EB8,0x1EB9}, {0x1EBA,0x1EBB}, 
+  {0x1EBC,0x1EBD}, {0x1EBE,0x1EBF}, {0x1EC0,0x1EC1}, {0x1EC2,0x1EC3}, {0x1EC4,0x1EC5}, 
+  {0x1EC6,0x1EC7}, {0x1EC8,0x1EC9}, {0x1ECA,0x1ECB}, {0x1ECC,0x1ECD}, {0x1ECE,0x1ECF}, 
+  {0x1ED0,0x1ED1}, {0x1ED2,0x1ED3}, {0x1ED4,0x1ED5}, {0x1ED6,0x1ED7}, {0x1ED8,0x1ED9}, 
+  {0x1EDA,0x1EDB}, {0x1EDC,0x1EDD}, {0x1EDE,0x1EDF}, {0x1EE0,0x1EE1}, {0x1EE2,0x1EE3}, 
+  {0x1EE4,0x1EE5}, {0x1EE6,0x1EE7}, {0x1EE8,0x1EE9}, {0x1EEA,0x1EEB}, {0x1EEC,0x1EED}, 
+  {0x1EEE,0x1EEF}, {0x1EF0,0x1EF1}, {0x1EF2,0x1EF3}, {0x1EF4,0x1EF5}, {0x1EF6,0x1EF7}, 
+  {0x1EF8,0x1EF9}, {0x1F08,0x1F00}, {0x1F09,0x1F01}, {0x1F0A,0x1F02}, {0x1F0B,0x1F03}, 
+  {0x1F0C,0x1F04}, {0x1F0D,0x1F05}, {0x1F0E,0x1F06}, {0x1F0F,0x1F07}, {0x1F18,0x1F10}, 
+  {0x1F19,0x1F11}, {0x1F1A,0x1F12}, {0x1F1B,0x1F13}, {0x1F1C,0x1F14}, {0x1F1D,0x1F15}, 
+  {0x1F28,0x1F20}, {0x1F29,0x1F21}, {0x1F2A,0x1F22}, {0x1F2B,0x1F23}, {0x1F2C,0x1F24}, 
+  {0x1F2D,0x1F25}, {0x1F2E,0x1F26}, {0x1F2F,0x1F27}, {0x1F38,0x1F30}, {0x1F39,0x1F31}, 
+  {0x1F3A,0x1F32}, {0x1F3B,0x1F33}, {0x1F3C,0x1F34}, {0x1F3D,0x1F35}, {0x1F3E,0x1F36}, 
+  {0x1F3F,0x1F37}, {0x1F48,0x1F40}, {0x1F49,0x1F41}, {0x1F4A,0x1F42}, {0x1F4B,0x1F43}, 
+  {0x1F4C,0x1F44}, {0x1F4D,0x1F45}, {0x1F59,0x1F51}, {0x1F5B,0x1F53}, {0x1F5D,0x1F55}, 
+  {0x1F5F,0x1F57}, {0x1F68,0x1F60}, {0x1F69,0x1F61}, {0x1F6A,0x1F62}, {0x1F6B,0x1F63}, 
+  {0x1F6C,0x1F64}, {0x1F6D,0x1F65}, {0x1F6E,0x1F66}, {0x1F6F,0x1F67}, {0x1F88,0x1F80}, 
+  {0x1F89,0x1F81}, {0x1F8A,0x1F82}, {0x1F8B,0x1F83}, {0x1F8C,0x1F84}, {0x1F8D,0x1F85}, 
+  {0x1F8E,0x1F86}, {0x1F8F,0x1F87}, {0x1F98,0x1F90}, {0x1F99,0x1F91}, {0x1F9A,0x1F92}, 
+  {0x1F9B,0x1F93}, {0x1F9C,0x1F94}, {0x1F9D,0x1F95}, {0x1F9E,0x1F96}, {0x1F9F,0x1F97}, 
+  {0x1FA8,0x1FA0}, {0x1FA9,0x1FA1}, {0x1FAA,0x1FA2}, {0x1FAB,0x1FA3}, {0x1FAC,0x1FA4}, 
+  {0x1FAD,0x1FA5}, {0x1FAE,0x1FA6}, {0x1FAF,0x1FA7}, {0x1FB8,0x1FB0}, {0x1FB9,0x1FB1}, 
+  {0x1FBA,0x1F70}, {0x1FBB,0x1F71}, {0x1FBC,0x1FB3}, {0x1FC8,0x1F72}, {0x1FC9,0x1F73}, 
+  {0x1FCA,0x1F74}, {0x1FCB,0x1F75}, {0x1FCC,0x1FC3}, {0x1FD8,0x1FD0}, {0x1FD9,0x1FD1}, 
+  {0x1FDA,0x1F76}, {0x1FDB,0x1F77}, {0x1FE8,0x1FE0}, {0x1FE9,0x1FE1}, {0x1FEA,0x1F7A}, 
+  {0x1FEB,0x1F7B}, {0x1FEC,0x1FE5}, {0x1FF8,0x1F78}, {0x1FF9,0x1F79}, {0x1FFA,0x1F7C}, 
+  {0x1FFB,0x1F7D}, {0x1FFC,0x1FF3}, {0x2160,0x2170}, {0x2161,0x2171}, {0x2162,0x2172}, 
+  {0x2163,0x2173}, {0x2164,0x2174}, {0x2165,0x2175}, {0x2166,0x2176}, {0x2167,0x2177}, 
+  {0x2168,0x2178}, {0x2169,0x2179}, {0x216A,0x217A}, {0x216B,0x217B}, {0x216C,0x217C}, 
+  {0x216D,0x217D}, {0x216E,0x217E}, {0x216F,0x217F}, {0x24B6,0x24D0}, {0x24B7,0x24D1}, 
+  {0x24B8,0x24D2}, {0x24B9,0x24D3}, {0x24BA,0x24D4}, {0x24BB,0x24D5}, {0x24BC,0x24D6}, 
+  {0x24BD,0x24D7}, {0x24BE,0x24D8}, {0x24BF,0x24D9}, {0x24C0,0x24DA}, {0x24C1,0x24DB}, 
+  {0x24C2,0x24DC}, {0x24C3,0x24DD}, {0x24C4,0x24DE}, {0x24C5,0x24DF}, {0x24C6,0x24E0}, 
+  {0x24C7,0x24E1}, {0x24C8,0x24E2}, {0x24C9,0x24E3}, {0x24CA,0x24E4}, {0x24CB,0x24E5}, 
+  {0x24CC,0x24E6}, {0x24CD,0x24E7}, {0x24CE,0x24E8}, {0x24CF,0x24E9}, {0xFF21,0xFF41}, 
+  {0xFF22,0xFF42}, {0xFF23,0xFF43}, {0xFF24,0xFF44}, {0xFF25,0xFF45}, {0xFF26,0xFF46}, 
+  {0xFF27,0xFF47}, {0xFF28,0xFF48}, {0xFF29,0xFF49}, {0xFF2A,0xFF4A}, {0xFF2B,0xFF4B}, 
+  {0xFF2C,0xFF4C}, {0xFF2D,0xFF4D}, {0xFF2E,0xFF4E}, {0xFF2F,0xFF4F}, {0xFF30,0xFF50}, 
+  {0xFF31,0xFF51}, {0xFF32,0xFF52}, {0xFF33,0xFF53}, {0xFF34,0xFF54}, {0xFF35,0xFF55}, 
+  {0xFF36,0xFF56}, {0xFF37,0xFF57}, {0xFF38,0xFF58}, {0xFF39,0xFF59}, {0xFF3A,0xFF5A}
+};
+
+
+
+#define NUM_TOSIMPLIFIED_INFO 2145
+static const unirelate tosimplified_info[NUM_TOSIMPLIFIED_INFO] = {
+  {0x4E48,0x9EBD}, {0x4E82,0x4E71}, {0x4E9E,0x4E9A}, {0x4F47,0x4F2B}, {0x4F59,0x9980}, 
+  {0x4F86,0x6765}, {0x4F96,0x4ED1}, {0x4FE0,0x4FA0}, {0x5000,0x4F25}, {0x5006,0x4FE9}, 
+  {0x5009,0x4ED3}, {0x500B,0x4E2A}, {0x5011,0x4EEC}, {0x502B,0x4F26}, {0x5049,0x4F1F}, 
+  {0x5074,0x4FA7}, {0x5075,0x4FA6}, {0x5096,0x4F27}, {0x5098,0x4F1E}, {0x5099,0x5907}, 
+  {0x50AD,0x4F63}, {0x50B3,0x4F20}, {0x50B4,0x4F1B}, {0x50B5,0x503A}, {0x50B7,0x4F24}, 
+  {0x50BE,0x503E}, {0x50C2,0x507B}, {0x50C5,0x4EC5}, {0x50C9,0x4F65}, {0x50D1,0x4FA8}, 
+  {0x50D5,0x4EC6}, {0x50DE,0x4F2A}, {0x50E5,0x4FA5}, {0x50E8,0x507E}, {0x50F9,0x4EF7}, 
+  {0x5100,0x4EEA}, {0x5102,0x4FAC}, {0x5104,0x4EBF}, {0x5108,0x4FA9}, {0x5109,0x4FED}, 
+  {0x5110,0x50A7}, {0x5114,0x4FE6}, {0x5115,0x4FAA}, {0x511F,0x507F}, {0x512A,0x4F18}, 
+  {0x5132,0x50A8}, {0x5137,0x4FEA}, {0x513A,0x50A9}, {0x513B,0x50A5}, {0x513C,0x4FE8}, 
+  {0x5152,0x513F}, {0x5169,0x4E24}, {0x51C8,0x51C0}, {0x51CD,0x51BB}, {0x51F1,0x51EF}, 
+  {0x5244,0x522D}, {0x5247,0x5219}, {0x525B,0x521A}, {0x526E,0x5250}, {0x5274,0x5240}, 
+  {0x5275,0x521B}, {0x5283,0x5212}, {0x5287,0x5267}, {0x5289,0x5218}, {0x528A,0x523D}, 
+  {0x528C,0x523F}, {0x528D,0x5251}, {0x5291,0x5242}, {0x52C1,0x52B2}, {0x52D5,0x52A8}, 
+  {0x52D9,0x52A1}, {0x52DB,0x52CB}, {0x52DD,0x80DC}, {0x52DE,0x52B3}, {0x52E2,0x52BF}, 
+  {0x52F1,0x52A2}, {0x52F5,0x52B1}, {0x52F8,0x529D}, {0x532D,0x5326}, {0x532F,0x6C47}, 
+  {0x5331,0x532E}, {0x5340,0x533A}, {0x5354,0x534F}, {0x5399,0x538D}, {0x53A0,0x5395}, 
+  {0x53AD,0x538C}, {0x53B2,0x5389}, {0x53B4,0x53A3}, {0x53C3,0x53C2}, {0x53E2,0x4E1B}, 
+  {0x540E,0x5F8C}, {0x54BC,0x5459}, {0x54E1,0x5458}, {0x5504,0x5457}, {0x554F,0x95EE}, 
+  {0x5553,0x542F}, {0x555E,0x54D1}, {0x559A,0x5524}, {0x55AA,0x4E27}, {0x55AC,0x4E54}, 
+  {0x55AE,0x5355}, {0x55B2,0x54DF}, {0x55C6,0x545B}, {0x55C7,0x556C}, {0x55CE,0x5417}, 
+  {0x55DA,0x545C}, {0x55E9,0x5522}, {0x55F6,0x54D4}, {0x5606,0x53F9}, {0x560D,0x55BD}, 
+  {0x5614,0x5455}, {0x5616,0x5567}, {0x5617,0x5C1D}, {0x561C,0x551B}, {0x5629,0x54D7}, 
+  {0x562E,0x5520}, {0x562F,0x5578}, {0x5630,0x53FD}, {0x5635,0x54D3}, {0x5638,0x5452}, 
+  {0x565D,0x549D}, {0x5660,0x54D2}, {0x5665,0x54DD}, {0x5666,0x54D5}, {0x566F,0x55F3}, 
+  {0x5672,0x54D9}, {0x5674,0x55B7}, {0x5678,0x5428}, {0x5680,0x549B}, {0x5687,0x5413}, 
+  {0x568C,0x54DC}, {0x5695,0x565C}, {0x5699,0x556E}, {0x56A6,0x5456}, {0x56A8,0x5499}, 
+  {0x56B3,0x55BE}, {0x56B4,0x4E25}, {0x56B6,0x5624}, {0x56C0,0x556D}, {0x56C1,0x55EB}, 
+  {0x56C2,0x56A3}, {0x56C5,0x5181}, {0x56C8,0x5453}, {0x56D1,0x5631}, {0x5707,0x56F5}, 
+  {0x570B,0x56FD}, {0x570D,0x56F4}, {0x5712,0x56ED}, {0x5713,0x5706}, {0x5716,0x56FE}, 
+  {0x5718,0x56E2}, {0x57E1,0x57AD}, {0x57F7,0x6267}, {0x5805,0x575A}, {0x580A,0x57A9}, 
+  {0x5816,0x57B4}, {0x581D,0x57DA}, {0x582F,0x5C27}, {0x5831,0x62A5}, {0x5834,0x573A}, 
+  {0x584A,0x5757}, {0x584B,0x8314}, {0x584F,0x57B2}, {0x5852,0x57D8}, {0x5857,0x6D82}, 
+  {0x5862,0x575E}, {0x5864,0x57D9}, {0x5875,0x5C18}, {0x5879,0x5811}, {0x588A,0x57AB}, 
+  {0x589C,0x5760}, {0x58AE,0x5815}, {0x58B3,0x575F}, {0x58BB,0x5899}, {0x58BE,0x57A6}, 
+  {0x58C7,0x575B}, {0x58D3,0x538B}, {0x58D8,0x5792}, {0x58D9,0x5739}, {0x58DA,0x5786}, 
+  {0x58DE,0x574F}, {0x58DF,0x5784}, {0x58E0,0x5785}, {0x58E2,0x575C}, {0x58E9,0x575D}, 
+  {0x58EF,0x58EE}, {0x58FA,0x58F6}, {0x58FD,0x5BFF}, {0x5922,0x68A6}, {0x593E,0x5939}, 
+  {0x5950,0x5942}, {0x5969,0x5941}, {0x596A,0x593A}, {0x596C,0x5956}, {0x596E,0x594B}, 
+  {0x599D,0x5986}, {0x5A41,0x5A04}, {0x5A66,0x5987}, {0x5A6D,0x5A05}, {0x5AA7,0x5A32}, 
+  {0x5ABD,0x5988}, {0x5AD7,0x59AA}, {0x5AF5,0x59A9}, {0x5AFB,0x5A34}, {0x5B00,0x59AB}, 
+  {0x5B08,0x5A06}, {0x5B0B,0x5A75}, {0x5B0C,0x5A07}, {0x5B19,0x5AF1}, {0x5B21,0x5AD2}, 
+  {0x5B2A,0x5AD4}, {0x5B30,0x5A74}, {0x5B38,0x5A76}, {0x5B4C,0x5A08}, {0x5B6B,0x5B59}, 
+  {0x5B78,0x5B66}, {0x5B7F,0x5B6A}, {0x5BE2,0x5BDD}, {0x5BE6,0x5B9E}, {0x5BE7,0x5B81}, 
+  {0x5BE9,0x5BA1}, {0x5BEB,0x5199}, {0x5BEC,0x5BBD}, {0x5BF5,0x5BA0}, {0x5BF6,0x5B9D}, 
+  {0x5C07,0x5C06}, {0x5C08,0x4E13}, {0x5C0B,0x5BFB}, {0x5C0D,0x5BF9}, {0x5C0E,0x5BFC}, 
+  {0x5C37,0x5C34}, {0x5C62,0x5C61}, {0x5C64,0x5C42}, {0x5C68,0x5C66}, {0x5C6C,0x5C5E}, 
+  {0x5CA1,0x5188}, {0x5CF4,0x5C98}, {0x5CF6,0x5C9B}, {0x5CFD,0x5CE1}, {0x5D0D,0x5D03}, 
+  {0x5D17,0x5C97}, {0x5D22,0x5CE5}, {0x5D2C,0x5CBD}, {0x5D50,0x5C9A}, {0x5D81,0x5D5D}, 
+  {0x5D84,0x5D2D}, {0x5D87,0x5C96}, {0x5D97,0x5D02}, {0x5DA0,0x5CE4}, {0x5DA7,0x5CC4}, 
+  {0x5DB8,0x5D58}, {0x5DBA,0x5CAD}, {0x5DBC,0x5C7F}, {0x5DCB,0x5CBF}, {0x5DD2,0x5CE6}, 
+  {0x5DD4,0x5DC5}, {0x5DF0,0x5DEF}, {0x5E25,0x5E05}, {0x5E2B,0x5E08}, {0x5E33,0x5E10}, 
+  {0x5E36,0x5E26}, {0x5E40,0x5E27}, {0x5E43,0x5E0F}, {0x5E57,0x5E3C}, {0x5E58,0x5E3B}, 
+  {0x5E5F,0x5E1C}, {0x5E63,0x5E01}, {0x5E6B,0x5E2E}, {0x5E6C,0x5E31}, {0x5E79,0x5E72}, 
+  {0x5E7E,0x51E0}, {0x5EAB,0x5E93}, {0x5EDF,0x5E99}, {0x5EE0,0x5382}, {0x5EE1,0x5E91}, 
+  {0x5EE2,0x5E9F}, {0x5EE3,0x5E7F}, {0x5EEC,0x5E90}, {0x5EF3,0x5385}, {0x5F33,0x5F2A}, 
+  {0x5F35,0x5F20}, {0x5F48,0x5F39}, {0x5F4C,0x5F25}, {0x5F4E,0x5F2F}, {0x5F8C,0x540E}, 
+  {0x5F91,0x5F84}, {0x5F9E,0x4ECE}, {0x5FA0,0x5F95}, {0x5FA9,0x590D}, {0x5FB9,0x5F7B}, 
+  {0x60B5,0x6005}, {0x60B6,0x95F7}, {0x60E1,0x6076}, {0x60F1,0x607C}, {0x60F2,0x607D}, 
+  {0x60FB,0x607B}, {0x611B,0x7231}, {0x611C,0x60EC}, {0x6134,0x6006}, {0x6137,0x607A}, 
+  {0x613E,0x5FFE}, {0x614B,0x6001}, {0x6158,0x60E8}, {0x615A,0x60ED}, {0x615F,0x6078}, 
+  {0x6163,0x60EF}, {0x6164,0x60AB}, {0x616A,0x6004}, {0x616B,0x6002}, {0x616E,0x8651}, 
+  {0x6173,0x60AD}, {0x6176,0x5E86}, {0x6182,0x5FE7}, {0x618A,0x60EB}, {0x6190,0x601C}, 
+  {0x6191,0x51ED}, {0x6192,0x6126}, {0x619A,0x60EE}, {0x61A4,0x6124}, {0x61AB,0x60AF}, 
+  {0x61AE,0x6003}, {0x61B2,0x5BAA}, {0x61B6,0x5FC6}, {0x61C7,0x6073}, {0x61C9,0x5E94}, 
+  {0x61CC,0x603F}, {0x61DF,0x603C}, {0x61E3,0x61D1}, {0x61E8,0x6079}, {0x61F2,0x60E9}, 
+  {0x61F6,0x61D2}, {0x61F7,0x6000}, {0x61F8,0x60AC}, {0x61FA,0x5FCF}, {0x61FC,0x60E7}, 
+  {0x61FE,0x6151}, {0x6200,0x604B}, {0x6207,0x6206}, {0x6214,0x620B}, {0x6227,0x6217}, 
+  {0x6229,0x622C}, {0x6230,0x6218}, {0x6232,0x620F}, {0x633E,0x631F}, {0x6368,0x820D}, 
+  {0x636B,0x626A}, {0x6383,0x626B}, {0x6384,0x62A1}, {0x6399,0x6323}, {0x63C0,0x62E3}, 
+  {0x63DA,0x626C}, {0x63DB,0x6362}, {0x63EE,0x6325}, {0x640D,0x635F}, {0x6417,0x6363}, 
+  {0x6436,0x62A2}, {0x6451,0x63B4}, {0x645C,0x63BC}, {0x645F,0x6402}, {0x646F,0x631A}, 
+  {0x6473,0x62A0}, {0x6476,0x629F}, {0x647B,0x63BA}, {0x6488,0x635E}, {0x6493,0x6320}, 
+  {0x649F,0x6322}, {0x64A3,0x63B8}, {0x64A5,0x62E8}, {0x64AB,0x629A}, {0x64B2,0x6251}, 
+  {0x64B3,0x63FF}, {0x64BB,0x631E}, {0x64BE,0x631D}, {0x64BF,0x6361}, {0x64C1,0x62E5}, 
+  {0x64C4,0x63B3}, {0x64C7,0x62E9}, {0x64CA,0x51FB}, {0x64CB,0x6321}, {0x64D4,0x62C5}, 
+  {0x64DA,0x636E}, {0x64E0,0x6324}, {0x64EC,0x62DF}, {0x64EF,0x6448}, {0x64F0,0x62E7}, 
+  {0x64F1,0x6401}, {0x64F2,0x63B7}, {0x64F4,0x6269}, {0x64F7,0x64B7}, {0x64FA,0x6446}, 
+  {0x64FB,0x64DE}, {0x64FC,0x64B8}, {0x64FE,0x6270}, {0x6504,0x6445}, {0x6506,0x64B5}, 
+  {0x650F,0x62E2}, {0x6514,0x62E6}, {0x6516,0x6484}, {0x6519,0x6400}, {0x651B,0x64BA}, 
+  {0x651D,0x6444}, {0x6522,0x6512}, {0x6523,0x631B}, {0x6524,0x644A}, {0x652A,0x6405}, 
+  {0x652C,0x63FD}, {0x6557,0x8D25}, {0x6575,0x654C}, {0x6578,0x6570}, {0x6582,0x655B}, 
+  {0x6583,0x6BD9}, {0x6595,0x6593}, {0x65AC,0x65A9}, {0x65B7,0x65AD}, {0x6642,0x65F6}, 
+  {0x6649,0x664B}, {0x665D,0x663C}, {0x6688,0x6655}, {0x6689,0x6656}, {0x66A2,0x7545}, 
+  {0x66AB,0x6682}, {0x66C4,0x6654}, {0x66C7,0x6619}, {0x66C9,0x6653}, {0x66D6,0x66A7}, 
+  {0x66E0,0x65F7}, {0x66EC,0x6652}, {0x66F8,0x4E66}, {0x6703,0x4F1A}, {0x6727,0x80E7}, 
+  {0x6771,0x4E1C}, {0x6898,0x67A7}, {0x689D,0x6761}, {0x689F,0x67AD}, {0x68C4,0x5F03}, 
+  {0x68D6,0x67A8}, {0x68D7,0x67A3}, {0x68DF,0x680B}, {0x68E7,0x6808}, {0x68F2,0x6816}, 
+  {0x690F,0x6860}, {0x694A,0x6768}, {0x6953,0x67AB}, {0x6968,0x6862}, {0x696D,0x4E1A}, 
+  {0x6975,0x6781}, {0x69AA,0x6769}, {0x69AE,0x8363}, {0x69BF,0x6864}, {0x69CB,0x6784}, 
+  {0x69CD,0x67AA}, {0x69E7,0x6920}, {0x69F3,0x6868}, {0x6A01,0x6869}, {0x6A02,0x4E50}, 
+  {0x6A05,0x679E}, {0x6A13,0x697C}, {0x6A19,0x6807}, {0x6A1E,0x67A2}, {0x6A23,0x6837}, 
+  {0x6A38,0x6734}, {0x6A39,0x6811}, {0x6A3A,0x6866}, {0x6A48,0x6861}, {0x6A4B,0x6865}, 
+  {0x6A5F,0x673A}, {0x6A62,0x692D}, {0x6A89,0x67FD}, {0x6A94,0x6863}, {0x6A9C,0x6867}, 
+  {0x6AA2,0x68C0}, {0x6AA3,0x6A2F}, {0x6AB3,0x69DF}, {0x6AB8,0x67E0}, {0x6ABB,0x69DB}, 
+  {0x6AC3,0x67DC}, {0x6AD3,0x6A79}, {0x6ADA,0x6988}, {0x6ADB,0x6809}, {0x6ADD,0x691F}, 
+  {0x6ADE,0x6A7C}, {0x6ADF,0x680E}, {0x6AE7,0x69E0}, {0x6AE8,0x680C}, {0x6AEA,0x67A5}, 
+  {0x6AEC,0x6987}, {0x6AF3,0x680A}, {0x6AF8,0x6989}, {0x6AFB,0x6A31}, {0x6B04,0x680F}, 
+  {0x6B0A,0x6743}, {0x6B0F,0x6924}, {0x6B12,0x683E}, {0x6B16,0x6984}, {0x6B1E,0x68C2}, 
+  {0x6B3D,0x94A6}, {0x6B50,0x6B27}, {0x6B5F,0x6B24}, {0x6B61,0x6B22}, {0x6B72,0x5C81}, 
+  {0x6B77,0x5386}, {0x6B78,0x5F52}, {0x6B98,0x6B8B}, {0x6B9E,0x6B92}, {0x6BA4,0x6B87}, 
+  {0x6BAB,0x6B9A}, {0x6BAE,0x6B93}, {0x6BAF,0x6BA1}, {0x6BB2,0x6B7C}, {0x6BBA,0x6740}, 
+  {0x6BBB,0x58F3}, {0x6BC6,0x6BB4}, {0x6BFF,0x6BF5}, {0x6C08,0x6BE1}, {0x6C0C,0x6C07}, 
+  {0x6C23,0x6C14}, {0x6C2B,0x6C22}, {0x6C2C,0x6C29}, {0x6D79,0x6D43}, {0x6D87,0x6CFE}, 
+  {0x6DEA,0x6CA6}, {0x6DF5,0x6E0A}, {0x6DF6,0x6D9E}, {0x6DFA,0x6D45}, {0x6E19,0x6DA3}, 
+  {0x6E26,0x6DA1}, {0x6E2C,0x6D4B}, {0x6E3E,0x6D51}, {0x6E5E,0x6D48}, {0x6E6F,0x6C64}, 
+  {0x6E96,0x51C6}, {0x6E9D,0x6C9F}, {0x6EC4,0x6CA7}, {0x6EC5,0x706D}, {0x6ECC,0x6DA4}, 
+  {0x6ECE,0x8365}, {0x6EEC,0x6CAA}, {0x6EEF,0x6EDE}, {0x6EF2,0x6E17}, {0x6EF7,0x5364}, 
+  {0x6EF8,0x6D52}, {0x6EFF,0x6EE1}, {0x6F01,0x6E14}, {0x6F1A,0x6CA4}, {0x6F22,0x6C49}, 
+  {0x6F23,0x6D9F}, {0x6F2C,0x6E0D}, {0x6F32,0x6DA8}, {0x6F38,0x6E10}, {0x6F3F,0x6D46}, 
+  {0x6F41,0x988D}, {0x6F51,0x6CFC}, {0x6F54,0x6D01}, {0x6F59,0x6CA9}, {0x6F5B,0x6F5C}, 
+  {0x6F64,0x6DA6}, {0x6F6F,0x6D54}, {0x6F70,0x6E83}, {0x6F77,0x6ED7}, {0x6F7F,0x6DA0}, 
+  {0x6F80,0x6DA9}, {0x6F86,0x6D47}, {0x6F87,0x6D9D}, {0x6F97,0x6DA7}, {0x6FA0,0x6E11}, 
+  {0x6FA4,0x6CFD}, {0x6FA9,0x6CF6}, {0x6FAE,0x6D4D}, {0x6FB1,0x6DC0}, {0x6FC1,0x6D4A}, 
+  {0x6FC3,0x6D53}, {0x6FD5,0x6E7F}, {0x6FD8,0x6CDE}, {0x6FDF,0x6D4E}, {0x6FE4,0x6D9B}, 
+  {0x6FEB,0x6EE5}, {0x6FF0,0x6F4D}, {0x6FF1,0x6EE8}, {0x6FFA,0x6E85}, {0x6FFC,0x6CFA}, 
+  {0x6FFE,0x6EE4}, {0x7005,0x6EE2}, {0x7006,0x6E0E}, {0x7009,0x6CFB}, {0x700B,0x6E16}, 
+  {0x700F,0x6D4F}, {0x7015,0x6FD2}, {0x7018,0x6CF8}, {0x701D,0x6CA5}, {0x701F,0x6F47}, 
+  {0x7020,0x6F46}, {0x7027,0x6CF7}, {0x7028,0x6FD1}, {0x7032,0x6F4B}, {0x703E,0x6F9C}, 
+  {0x7043,0x6CA3}, {0x7044,0x6EE0}, {0x7051,0x6D12}, {0x7055,0x6F13}, {0x7058,0x6EE9}, 
+  {0x705D,0x704F}, {0x7063,0x6E7E}, {0x7064,0x6EE6}, {0x7067,0x6EDF}, {0x70CF,0x4E4C}, 
+  {0x70F4,0x70C3}, {0x7121,0x65E0}, {0x7149,0x70BC}, {0x7152,0x709C}, {0x7162,0x8315}, 
+  {0x7165,0x7115}, {0x7169,0x70E6}, {0x716C,0x7080}, {0x7192,0x8367}, {0x7197,0x709D}, 
+  {0x71B1,0x70ED}, {0x71BE,0x70BD}, {0x71C1,0x70E8}, {0x71C8,0x706F}, {0x71D2,0x70E7}, 
+  {0x71D9,0x70EB}, {0x71DC,0x7116}, {0x71DF,0x8425}, {0x71E6,0x707F}, {0x71ED,0x70DB}, 
+  {0x71F4,0x70E9}, {0x71FC,0x70EC}, {0x71FE,0x7118}, {0x720D,0x70C1}, {0x7210,0x7089}, 
+  {0x721B,0x70C2}, {0x722D,0x4E89}, {0x7232,0x4E3A}, {0x723A,0x7237}, {0x723E,0x5C14}, 
+  {0x7258,0x724D}, {0x727D,0x7275}, {0x7296,0x8366}, {0x72A2,0x728A}, {0x72A7,0x727A}, 
+  {0x72C0,0x72B6}, {0x72F9,0x72ED}, {0x72FD,0x72C8}, {0x7319,0x72F0}, {0x7336,0x72B9}, 
+  {0x733B,0x72F2}, {0x7341,0x72B8}, {0x7344,0x72F1}, {0x7345,0x72EE}, {0x7368,0x72EC}, 
+  {0x736A,0x72EF}, {0x736B,0x7303}, {0x7370,0x72DE}, {0x7372,0x83B7}, {0x7375,0x730E}, 
+  {0x7377,0x72B7}, {0x7378,0x517D}, {0x737A,0x736D}, {0x737B,0x732E}, {0x737C,0x7315}, 
+  {0x7380,0x7321}, {0x73FE,0x73B0}, {0x743F,0x73F2}, {0x744B,0x73AE}, {0x7463,0x7410}, 
+  {0x7469,0x83B9}, {0x746A,0x739B}, {0x7489,0x740F}, {0x74A3,0x7391}, {0x74A6,0x7477}, 
+  {0x74B0,0x73AF}, {0x74BD,0x73BA}, {0x74CA,0x743C}, {0x74CF,0x73D1}, {0x74D4,0x748E}, 
+  {0x74DA,0x74D2}, {0x750C,0x74EF}, {0x7523,0x4EA7}, {0x755D,0x4EA9}, {0x7562,0x6BD5}, 
+  {0x756B,0x753B}, {0x7576,0x5F53}, {0x7587,0x7574}, {0x758A,0x53E0}, {0x75D9,0x75C9}, 
+  {0x7602,0x75D6}, {0x760B,0x75AF}, {0x760D,0x75A1}, {0x7613,0x75EA}, {0x761E,0x7617}, 
+  {0x7621,0x75AE}, {0x7627,0x759F}, {0x763B,0x7618}, {0x7642,0x7597}, {0x7646,0x75E8}, 
+  {0x7647,0x75EB}, {0x7649,0x7605}, {0x7658,0x75A0}, {0x765F,0x762A}, {0x7662,0x75D2}, 
+  {0x7664,0x7596}, {0x7665,0x75C7}, {0x7667,0x75AC}, {0x7669,0x765E}, {0x766C,0x7663}, 
+  {0x766D,0x763F}, {0x766E,0x763E}, {0x7670,0x75C8}, {0x7671,0x762B}, {0x7672,0x766B}, 
+  {0x767C,0x53D1}, {0x769A,0x7691}, {0x76B8,0x76B2}, {0x76BA,0x76B1}, {0x76DE,0x76CF}, 
+  {0x76E1,0x5C3D}, {0x76E3,0x76D1}, {0x76E4,0x76D8}, {0x76E7,0x5362}, {0x775C,0x7741}, 
+  {0x775E,0x7750}, {0x7798,0x770D}, {0x779E,0x7792}, {0x77BC,0x7751}, {0x77DA,0x77A9}, 
+  {0x77EF,0x77EB}, {0x7864,0x7856}, {0x7868,0x7817}, {0x786F,0x781A}, {0x78A9,0x7855}, 
+  {0x78AD,0x7800}, {0x78B8,0x781C}, {0x78BA,0x786E}, {0x78BC,0x7801}, {0x78DA,0x7816}, 
+  {0x78E3,0x789C}, {0x78E7,0x789B}, {0x78EF,0x77F6}, {0x78FD,0x7857}, {0x790E,0x7840}, 
+  {0x7919,0x788D}, {0x7926,0x77FF}, {0x792A,0x783A}, {0x792B,0x783E}, {0x792C,0x77FE}, 
+  {0x7931,0x783B}, {0x798D,0x7978}, {0x798E,0x796F}, {0x79AA,0x7985}, {0x79AE,0x793C}, 
+  {0x79B0,0x7962}, {0x79B1,0x7977}, {0x7A2E,0x79CD}, {0x7A31,0x79F0}, {0x7A4C,0x7A23}, 
+  {0x7A4D,0x79EF}, {0x7A4E,0x9896}, {0x7A61,0x7A51}, {0x7A62,0x79FD}, {0x7A69,0x7A33}, 
+  {0x7AA9,0x7A9D}, {0x7AAA,0x6D3C}, {0x7AAE,0x7A77}, {0x7AB6,0x7AAD}, {0x7ABA,0x7AA5}, 
+  {0x7AC4,0x7A9C}, {0x7AC5,0x7A8D}, {0x7AC7,0x7AA6}, {0x7AC8,0x7076}, {0x7ACA,0x7A83}, 
+  {0x7AEA,0x7AD6}, {0x7AF6,0x7ADE}, {0x7B46,0x7B14}, {0x7B67,0x7B15}, {0x7B8B,0x7B3A}, 
+  {0x7B8F,0x7B5D}, {0x7BC0,0x8282}, {0x7BC4,0x8303}, {0x7BC9,0x7B51}, {0x7BCB,0x7BA7}, 
+  {0x7BE4,0x7B03}, {0x7BE9,0x7B5B}, {0x7BF3,0x7B5A}, {0x7C00,0x7BA6}, {0x7C0D,0x7BD3}, 
+  {0x7C1E,0x7BAA}, {0x7C21,0x7B80}, {0x7C23,0x7BD1}, {0x7C2B,0x7BAB}, {0x7C3D,0x7B7E}, 
+  {0x7C3E,0x5E18}, {0x7C43,0x7BEE}, {0x7C4C,0x7B79}, {0x7C5C,0x7BA8}, {0x7C5F,0x7C41}, 
+  {0x7C60,0x7B3C}, {0x7C69,0x7B3E}, {0x7C6A,0x7C16}, {0x7C6C,0x7BF1}, {0x7C6E,0x7BA9}, 
+  {0x7CDD,0x7CC1}, {0x7CDE,0x7CAA}, {0x7CE7,0x7CAE}, {0x7CF2,0x7C9D}, {0x7CF4,0x7C74}, 
+  {0x7CF6,0x7C9C}, {0x7CF9,0x7E9F}, {0x7CFE,0x7EA0}, {0x7D00,0x7EAA}, {0x7D02,0x7EA3}, 
+  {0x7D04,0x7EA6}, {0x7D05,0x7EA2}, {0x7D06,0x7EA1}, {0x7D07,0x7EA5}, {0x7D08,0x7EA8}, 
+  {0x7D09,0x7EAB}, {0x7D0B,0x7EB9}, {0x7D0D,0x7EB3}, {0x7D10,0x7EBD}, {0x7D13,0x7EBE}, 
+  {0x7D14,0x7EAF}, {0x7D15,0x7EB0}, {0x7D17,0x7EB1}, {0x7D19,0x7EB8}, {0x7D1A,0x7EA7}, 
+  {0x7D1B,0x7EB7}, {0x7D1C,0x7EAD}, {0x7D21,0x7EBA}, {0x7D30,0x7EC6}, {0x7D31,0x7EC2}, 
+  {0x7D32,0x7EC1}, {0x7D33,0x7EC5}, {0x7D39,0x7ECD}, {0x7D3A,0x7EC0}, {0x7D3C,0x7ECB}, 
+  {0x7D3F,0x7ED0}, {0x7D40,0x7ECC}, {0x7D42,0x7EC8}, {0x7D44,0x7EC4}, {0x7D46,0x7ECA}, 
+  {0x7D4E,0x7ED7}, {0x7D50,0x7ED3}, {0x7D5D,0x7ED4}, {0x7D5E,0x7EDE}, {0x7D61,0x7EDC}, 
+  {0x7D62,0x7EDA}, {0x7D66,0x7ED9}, {0x7D68,0x7ED2}, {0x7D71,0x7EDF}, {0x7D72,0x4E1D}, 
+  {0x7D73,0x7EDB}, {0x7D76,0x7EDD}, {0x7D79,0x7EE2}, {0x7D81,0x7ED1}, {0x7D83,0x7EE1}, 
+  {0x7D86,0x7EE0}, {0x7D88,0x7EE8}, {0x7D89,0x7EE3}, {0x7D8F,0x7EE5}, {0x7D93,0x7ECF}, 
+  {0x7D9C,0x7EFC}, {0x7D9E,0x7F0D}, {0x7DA2,0x7EF8}, {0x7DA3,0x7EFB}, {0x7DAB,0x7EBF}, 
+  {0x7DAC,0x7EF6}, {0x7DAD,0x7EF4}, {0x7DB0,0x7EFE}, {0x7DB1,0x7EB2}, {0x7DB2,0x7F51}, 
+  {0x7DB3,0x7EF7}, {0x7DB4,0x7F00}, {0x7DB8,0x7EB6}, {0x7DB9,0x7EFA}, {0x7DBA,0x7EEE}, 
+  {0x7DBB,0x7EFD}, {0x7DBD,0x7EF0}, {0x7DBE,0x7EEB}, {0x7DBF,0x7EF5}, {0x7DC4,0x7EF2}, 
+  {0x7DC7,0x7F01}, {0x7DCA,0x7D27}, {0x7DCB,0x7EEF}, {0x7DD1,0x7EFF}, {0x7DD2,0x7EEA}, 
+  {0x7DD4,0x7EF1}, {0x7DD7,0x7F03}, {0x7DD8,0x7F04}, {0x7DD9,0x7F02}, {0x7DDD,0x7F09}, 
+  {0x7DDE,0x7F0E}, {0x7DE0,0x7F14}, {0x7DE1,0x7F17}, {0x7DE3,0x7F18}, {0x7DE6,0x7F0C}, 
+  {0x7DE8,0x7F16}, {0x7DE9,0x7F13}, {0x7DEC,0x7F05}, {0x7DEF,0x7EAC}, {0x7DF1,0x7F11}, 
+  {0x7DF2,0x7F08}, {0x7DF4,0x7EC3}, {0x7DF6,0x7F0F}, {0x7DF9,0x7F07}, {0x7E08,0x8426}, 
+  {0x7E09,0x7F19}, {0x7E0A,0x7F22}, {0x7E0B,0x7F12}, {0x7E10,0x7EC9}, {0x7E11,0x7F23}, 
+  {0x7E1B,0x7F1A}, {0x7E1D,0x7F1C}, {0x7E1E,0x7F1F}, {0x7E1F,0x7F1B}, {0x7E23,0x53BF}, 
+  {0x7E27,0x7EE6}, {0x7E2B,0x7F1D}, {0x7E2D,0x7F21}, {0x7E2E,0x7F29}, {0x7E31,0x7EB5}, 
+  {0x7E32,0x7F27}, {0x7E35,0x7F26}, {0x7E36,0x7D77}, {0x7E37,0x7F15}, {0x7E39,0x7F25}, 
+  {0x7E3D,0x603B}, {0x7E3E,0x7EE9}, {0x7E45,0x7F2B}, {0x7E46,0x7F2A}, {0x7E52,0x7F2F}, 
+  {0x7E54,0x7EC7}, {0x7E55,0x7F2E}, {0x7E5A,0x7F2D}, {0x7E5E,0x7ED5}, {0x7E62,0x7F0B}, 
+  {0x7E69,0x7EF3}, {0x7E6A,0x7ED8}, {0x7E6D,0x8327}, {0x7E6E,0x7F30}, {0x7E6F,0x7F33}, 
+  {0x7E70,0x7F32}, {0x7E73,0x7F34}, {0x7E79,0x7ECE}, {0x7E7C,0x7EE7}, {0x7E7D,0x7F24}, 
+  {0x7E7E,0x7F31}, {0x7E88,0x7F2C}, {0x7E8A,0x7EA9}, {0x7E8C,0x7EED}, {0x7E8F,0x7F20}, 
+  {0x7E93,0x7F28}, {0x7E96,0x7EA4}, {0x7E98,0x7F35}, {0x7E9C,0x7F06}, {0x7F4C,0x7F42}, 
+  {0x7F70,0x7F5A}, {0x7F77,0x7F62}, {0x7F85,0x7F57}, {0x7F86,0x7F74}, {0x7F88,0x7F81}, 
+  {0x7FA5,0x7F9F}, {0x7FA9,0x4E49}, {0x7FD2,0x4E60}, {0x7FF9,0x7FD8}, {0x802C,0x8027}, 
+  {0x802E,0x8022}, {0x8056,0x5723}, {0x805E,0x95FB}, {0x806F,0x8054}, {0x8070,0x806A}, 
+  {0x8072,0x58F0}, {0x8073,0x8038}, {0x8075,0x8069}, {0x8076,0x8042}, {0x8077,0x804C}, 
+  {0x8079,0x804D}, {0x807D,0x542C}, {0x807E,0x804B}, {0x8085,0x8083}, {0x8105,0x80C1}, 
+  {0x811B,0x80EB}, {0x8139,0x80C0}, {0x814E,0x80BE}, {0x8156,0x80E8}, {0x8161,0x8136}, 
+  {0x8166,0x8111}, {0x816B,0x80BF}, {0x8178,0x80A0}, {0x819A,0x80A4}, {0x81A0,0x80F6}, 
+  {0x81A9,0x817B}, {0x81BD,0x80C6}, {0x81BE,0x810D}, {0x81BF,0x8113}, {0x81C9,0x8138}, 
+  {0x81CD,0x8110}, {0x81CF,0x8191}, {0x81D8,0x814A}, {0x81DA,0x80EA}, {0x81DF,0x810F}, 
+  {0x81E0,0x8114}, {0x81E8,0x4E34}, {0x81FA,0x53F0}, {0x8207,0x4E0E}, {0x8208,0x5174}, 
+  {0x8209,0x4E3E}, {0x820A,0x65E7}, {0x8259,0x8231}, {0x8264,0x8223}, {0x8266,0x8230}, 
+  {0x826B,0x823B}, {0x8271,0x8270}, {0x8277,0x8273}, {0x82BB,0x520D}, {0x82E7,0x82CE}, 
+  {0x838A,0x5E84}, {0x8396,0x830E}, {0x83A2,0x835A}, {0x83A7,0x82CB}, {0x83EF,0x534E}, 
+  {0x8407,0x82CC}, {0x840A,0x83B1}, {0x842C,0x4E07}, {0x8435,0x83B4}, {0x8449,0x53F6}, 
+  {0x8452,0x836D}, {0x8464,0x836E}, {0x8466,0x82C7}, {0x8477,0x8364}, {0x8493,0x83BC}, 
+  {0x8494,0x83B3}, {0x84BC,0x82CD}, {0x84C0,0x836A}, {0x84CB,0x76D6}, {0x84EE,0x83B2}, 
+  {0x84EF,0x82C1}, {0x84FD,0x835C}, {0x851E,0x848C}, {0x8523,0x848B}, {0x8526,0x8311}, 
+  {0x852D,0x836B}, {0x8541,0x8368}, {0x8546,0x8487}, {0x854E,0x835E}, {0x8552,0x836C}, 
+  {0x8553,0x82B8}, {0x8555,0x83B8}, {0x8558,0x835B}, {0x8562,0x8489}, {0x8569,0x8361}, 
+  {0x856A,0x829C}, {0x856D,0x8427}, {0x8577,0x84E3}, {0x8588,0x835F}, {0x858A,0x84DF}, 
+  {0x858C,0x8297}, {0x8594,0x8537}, {0x859F,0x83B6}, {0x85A6,0x8350}, {0x85A9,0x8428}, 
+  {0x85BA,0x8360}, {0x85CD,0x84DD}, {0x85CE,0x8369}, {0x85DD,0x827A}, {0x85E5,0x836F}, 
+  {0x85EA,0x85AE}, {0x85F4,0x8574}, {0x85F6,0x82C8}, {0x85F9,0x853C}, {0x85FA,0x853A}, 
+  {0x8604,0x8572}, {0x8606,0x82A6}, {0x8607,0x82CF}, {0x860B,0x82F9}, {0x861A,0x85D3}, 
+  {0x861E,0x8539}, {0x8622,0x830F}, {0x862D,0x5170}, {0x863A,0x84E0}, {0x863F,0x841D}, 
+  {0x8655,0x5904}, {0x865C,0x864F}, {0x865F,0x53F7}, {0x8667,0x4E8F}, {0x86FA,0x86F1}, 
+  {0x8706,0x86AC}, {0x8755,0x8680}, {0x8766,0x867E}, {0x8778,0x8717}, {0x8784,0x86F3}, 
+  {0x879E,0x8682}, {0x87A2,0x8424}, {0x87BB,0x877C}, {0x87C4,0x86F0}, {0x87C8,0x8748}, 
+  {0x87CE,0x87A8}, {0x87E3,0x866E}, {0x87EC,0x8749}, {0x87EF,0x86F2}, {0x87F2,0x866B}, 
+  {0x87F6,0x86CF}, {0x87FB,0x8681}, {0x8805,0x8747}, {0x8806,0x867F}, {0x8810,0x86F4}, 
+  {0x8811,0x877E}, {0x881F,0x8721}, {0x8823,0x86CE}, {0x8831,0x86CA}, {0x8836,0x8695}, 
+  {0x883B,0x86EE}, {0x8846,0x4F17}, {0x8853,0x672F}, {0x885B,0x536B}, {0x885D,0x51B2}, 
+  {0x8879,0x53EA}, {0x88CA,0x8885}, {0x88CF,0x91CC}, {0x88DC,0x8865}, {0x88DD,0x88C5}, 
+  {0x8932,0x88E4}, {0x8933,0x88E2}, {0x8938,0x891B}, {0x893B,0x4EB5}, {0x8947,0x88E5}, 
+  {0x8956,0x8884}, {0x895D,0x88E3}, {0x8960,0x88C6}, {0x8964,0x8934}, {0x896A,0x889C}, 
+  {0x896F,0x886C}, {0x8972,0x88AD}, {0x898B,0x89C1}, {0x898F,0x89C4}, {0x8993,0x89C5}, 
+  {0x8996,0x89C6}, {0x8998,0x89C7}, {0x89A1,0x89CB}, {0x89A6,0x89CE}, {0x89AA,0x4EB2}, 
+  {0x89AC,0x89CA}, {0x89AF,0x89CF}, {0x89B2,0x89D0}, {0x89B7,0x89D1}, {0x89BA,0x89C9}, 
+  {0x89BD,0x89C8}, {0x89BF,0x89CC}, {0x89C0,0x89C2}, {0x89F4,0x89DE}, {0x89F6,0x89EF}, 
+  {0x89F8,0x89E6}, {0x8A01,0x8BA0}, {0x8A02,0x8BA2}, {0x8A03,0x8BA3}, {0x8A08,0x8BA1}, 
+  {0x8A0A,0x8BAF}, {0x8A0C,0x8BA7}, {0x8A0E,0x8BA8}, {0x8A10,0x8BA6}, {0x8A13,0x8BAD}, 
+  {0x8A15,0x8BAA}, {0x8A16,0x8BAB}, {0x8A18,0x8BB0}, {0x8A1B,0x8BB9}, {0x8A1D,0x8BB6}, 
+  {0x8A1F,0x8BBC}, {0x8A23,0x8BC0}, {0x8A25,0x8BB7}, {0x8A2A,0x8BBF}, {0x8A2D,0x8BBE}, 
+  {0x8A31,0x8BB8}, {0x8A34,0x8BC9}, {0x8A36,0x8BC3}, {0x8A3A,0x8BCA}, {0x8A41,0x8BC2}, 
+  {0x8A46,0x8BCB}, {0x8A4E,0x8BB5}, {0x8A50,0x8BC8}, {0x8A52,0x8BD2}, {0x8A54,0x8BCF}, 
+  {0x8A55,0x8BC4}, {0x8A58,0x8BCE}, {0x8A5B,0x8BC5}, {0x8A5E,0x8BCD}, {0x8A61,0x8BE9}, 
+  {0x8A62,0x8BE2}, {0x8A63,0x8BE3}, {0x8A66,0x8BD5}, {0x8A69,0x8BD7}, {0x8A6B,0x8BE7}, 
+  {0x8A6C,0x8BDF}, {0x8A6D,0x8BE1}, {0x8A6E,0x8BE0}, {0x8A70,0x8BD8}, {0x8A71,0x8BDD}, 
+  {0x8A72,0x8BE5}, {0x8A73,0x8BE6}, {0x8A75,0x8BDC}, {0x8A7C,0x8BD9}, {0x8A7F,0x8BD6}, 
+  {0x8A84,0x8BD4}, {0x8A85,0x8BDB}, {0x8A86,0x8BD3}, {0x8A87,0x5938}, {0x8A8D,0x8BA4}, 
+  {0x8A91,0x8BF3}, {0x8A92,0x8BF6}, {0x8A95,0x8BDE}, {0x8A98,0x8BF1}, {0x8A9A,0x8BEE}, 
+  {0x8A9E,0x8BED}, {0x8AA0,0x8BDA}, {0x8AA1,0x8BEB}, {0x8AA3,0x8BEC}, {0x8AA4,0x8BEF}, 
+  {0x8AA5,0x8BF0}, {0x8AA6,0x8BF5}, {0x8AA8,0x8BF2}, {0x8AAC,0x8BF4}, {0x8AB0,0x8C01}, 
+  {0x8AB2,0x8BFE}, {0x8AB6,0x8C07}, {0x8AB9,0x8BFD}, {0x8ABC,0x8C0A}, {0x8ABF,0x8C03}, 
+  {0x8AC2,0x8C04}, {0x8AC4,0x8C06}, {0x8AC7,0x8C08}, {0x8AC9,0x8BFF}, {0x8ACB,0x8BF7}, 
+  {0x8ACD,0x8BE4}, {0x8ACF,0x8BF9}, {0x8AD1,0x8BFC}, {0x8AD2,0x8C05}, {0x8AD6,0x8BBA}, 
+  {0x8AD7,0x8C02}, {0x8ADB,0x8C00}, {0x8ADC,0x8C0D}, {0x8ADE,0x8C1D}, {0x8AE2,0x8BE8}, 
+  {0x8AE4,0x8C14}, {0x8AE6,0x8C1B}, {0x8AE7,0x8C10}, {0x8AEB,0x8C0F}, {0x8AED,0x8C15}, 
+  {0x8AEE,0x8C18}, {0x8AF1,0x8BB3}, {0x8AF3,0x8C19}, {0x8AF6,0x8C0C}, {0x8AF7,0x8BBD}, 
+  {0x8AF8,0x8BF8}, {0x8AFA,0x8C1A}, {0x8AFC,0x8C16}, {0x8AFE,0x8BFA}, {0x8B00,0x8C0B}, 
+  {0x8B01,0x8C12}, {0x8B02,0x8C13}, {0x8B04,0x8A8A}, {0x8B05,0x8BCC}, {0x8B0A,0x8C0E}, 
+  {0x8B0E,0x8C1C}, {0x8B10,0x8C27}, {0x8B14,0x8C11}, {0x8B16,0x8C21}, {0x8B17,0x8C24}, 
+  {0x8B19,0x8C26}, {0x8B1A,0x8C25}, {0x8B1B,0x8BB2}, {0x8B1D,0x8C22}, {0x8B21,0x8C23}, 
+  {0x8B28,0x8C1F}, {0x8B2B,0x8C2A}, {0x8B2C,0x8C2C}, {0x8B2D,0x8C2B}, {0x8B33,0x8BB4}, 
+  {0x8B39,0x8C28}, {0x8B3E,0x8C29}, {0x8B49,0x8BC1}, {0x8B4E,0x8C32}, {0x8B4F,0x8BA5}, 
+  {0x8B56,0x8C2E}, {0x8B58,0x8BC6}, {0x8B59,0x8C2F}, {0x8B5A,0x8C2D}, {0x8B5C,0x8C31}, 
+  {0x8B6B,0x8C35}, {0x8B6F,0x8BD1}, {0x8B70,0x8BAE}, {0x8B74,0x8C34}, {0x8B77,0x62A4}, 
+  {0x8B7D,0x8A89}, {0x8B80,0x8BFB}, {0x8B8A,0x53D8}, {0x8B8E,0x96E0}, {0x8B92,0x8C17}, 
+  {0x8B93,0x8BA9}, {0x8B95,0x8C30}, {0x8B96,0x8C36}, {0x8B9C,0x8C20}, {0x8B9E,0x8C33}, 
+  {0x8C48,0x5C82}, {0x8C50,0x4E30}, {0x8C9D,0x8D1D}, {0x8C9E,0x8D1E}, {0x8CA0,0x8D1F}, 
+  {0x8CA1,0x8D22}, {0x8CA2,0x8D21}, {0x8CA7,0x8D2B}, {0x8CA8,0x8D27}, {0x8CA9,0x8D29}, 
+  {0x8CAA,0x8D2A}, {0x8CAB,0x8D2F}, {0x8CAC,0x8D23}, {0x8CAF,0x8D2E}, {0x8CB0,0x8D33}, 
+  {0x8CB2,0x8D40}, {0x8CB3,0x8D30}, {0x8CB4,0x8D35}, {0x8CB6,0x8D2C}, {0x8CB7,0x4E70}, 
+  {0x8CB8,0x8D37}, {0x8CBA,0x8D36}, {0x8CBB,0x8D39}, {0x8CBC,0x8D34}, {0x8CBD,0x8D3B}, 
+  {0x8CBF,0x8D38}, {0x8CC0,0x8D3A}, {0x8CC1,0x8D32}, {0x8CC2,0x8D42}, {0x8CC3,0x8D41}, 
+  {0x8CC4,0x8D3F}, {0x8CC5,0x8D45}, {0x8CC7,0x8D44}, {0x8CC8,0x8D3E}, {0x8CCA,0x8D3C}, 
+  {0x8CD1,0x8D48}, {0x8CD2,0x8D4A}, {0x8CD3,0x5BBE}, {0x8CD5,0x8D47}, {0x8CDA,0x8D49}, 
+  {0x8CDC,0x8D50}, {0x8CDE,0x8D4F}, {0x8CE0,0x8D54}, {0x8CE1,0x8D53}, {0x8CE2,0x8D24}, 
+  {0x8CE3,0x5356}, {0x8CE4,0x8D31}, {0x8CE6,0x8D4B}, {0x8CE7,0x8D55}, {0x8CEA,0x8D28}, 
+  {0x8CEB,0x8D4D}, {0x8CEC,0x8D26}, {0x8CED,0x8D4C}, {0x8CF4,0x8D56}, {0x8CFA,0x8D5A}, 
+  {0x8CFB,0x8D59}, {0x8CFC,0x8D2D}, {0x8CFD,0x8D5B}, {0x8CFE,0x8D5C}, {0x8D04,0x8D3D}, 
+  {0x8D05,0x8D58}, {0x8D08,0x8D60}, {0x8D0A,0x8D5E}, {0x8D0B,0x8D5D}, {0x8D0D,0x8D61}, 
+  {0x8D0F,0x8D62}, {0x8D10,0x8D46}, {0x8D16,0x8D4E}, {0x8D1B,0x8D63}, {0x8D1C,0x8D43}, 
+  {0x8D95,0x8D76}, {0x8D99,0x8D75}, {0x8DA8,0x8D8B}, {0x8DB2,0x8DB1}, {0x8E10,0x8DF5}, 
+  {0x8E34,0x8E0A}, {0x8E4C,0x8DC4}, {0x8E55,0x8DF8}, {0x8E63,0x8E52}, {0x8E7A,0x8DF7}, 
+  {0x8E89,0x8DB8}, {0x8E8A,0x8E0C}, {0x8E8B,0x8DFB}, {0x8E8D,0x8DC3}, {0x8E91,0x8E2F}, 
+  {0x8E92,0x8DDE}, {0x8E93,0x8E2C}, {0x8E9A,0x8DF9}, {0x8EA1,0x8E51}, {0x8EA5,0x8E7F}, 
+  {0x8EA6,0x8E9C}, {0x8EAA,0x8E8F}, {0x8EC0,0x8EAF}, {0x8ECA,0x8F66}, {0x8ECB,0x8F67}, 
+  {0x8ECC,0x8F68}, {0x8ECD,0x519B}, {0x8ED2,0x8F69}, {0x8ED4,0x8F6B}, {0x8EDB,0x8F6D}, 
+  {0x8EDF,0x8F6F}, {0x8EE4,0x8F77}, {0x8EEB,0x8F78}, {0x8EF2,0x8F71}, {0x8EF8,0x8F74}, 
+  {0x8EF9,0x8F75}, {0x8EFA,0x8F7A}, {0x8EFB,0x8F72}, {0x8EFC,0x8F76}, {0x8EFE,0x8F7C}, 
+  {0x8F03,0x8F83}, {0x8F05,0x8F82}, {0x8F07,0x8F81}, {0x8F09,0x8F7D}, {0x8F0A,0x8F7E}, 
+  {0x8F12,0x8F84}, {0x8F14,0x8F85}, {0x8F15,0x8F7B}, {0x8F1B,0x8F86}, {0x8F1C,0x8F8E}, 
+  {0x8F1D,0x8F89}, {0x8F1E,0x8F8B}, {0x8F1F,0x8F8D}, {0x8F25,0x8F8A}, {0x8F26,0x8F87}, 
+  {0x8F29,0x8F88}, {0x8F2A,0x8F6E}, {0x8F2F,0x8F91}, {0x8F33,0x8F8F}, {0x8F38,0x8F93}, 
+  {0x8F3B,0x8F90}, {0x8F3E,0x8F97}, {0x8F3F,0x8206}, {0x8F42,0x6BC2}, {0x8F44,0x8F96}, 
+  {0x8F45,0x8F95}, {0x8F46,0x8F98}, {0x8F49,0x8F6C}, {0x8F4D,0x8F99}, {0x8F4E,0x8F7F}, 
+  {0x8F54,0x8F9A}, {0x8F5F,0x8F70}, {0x8F61,0x8F94}, {0x8F62,0x8F79}, {0x8F64,0x8F73}, 
+  {0x8FA6,0x529E}, {0x8FAD,0x8F9E}, {0x8FAE,0x8FAB}, {0x8FAF,0x8FA9}, {0x8FB2,0x519C}, 
+  {0x9015,0x8FF3}, {0x9019,0x8FD9}, {0x9023,0x8FDE}, {0x9032,0x8FDB}, {0x904B,0x8FD0}, 
+  {0x904E,0x8FC7}, {0x9054,0x8FBE}, {0x9055,0x8FDD}, {0x905C,0x900A}, {0x905E,0x9012}, 
+  {0x9060,0x8FDC}, {0x9069,0x9002}, {0x9072,0x8FDF}, {0x9077,0x8FC1}, {0x9078,0x9009}, 
+  {0x907A,0x9057}, {0x907C,0x8FBD}, {0x9081,0x8FC8}, {0x9084,0x8FD8}, {0x9087,0x8FE9}, 
+  {0x908A,0x8FB9}, {0x908F,0x903B}, {0x9090,0x9026}, {0x90DF,0x90CF}, {0x90F5,0x90AE}, 
+  {0x9106,0x90D3}, {0x9109,0x4E61}, {0x9112,0x90B9}, {0x9114,0x90AC}, {0x9116,0x90E7}, 
+  {0x9127,0x9093}, {0x912D,0x90D1}, {0x9130,0x90BB}, {0x9132,0x90F8}, {0x9134,0x90BA}, 
+  {0x9136,0x90D0}, {0x913A,0x909D}, {0x9148,0x90E6}, {0x9196,0x915D}, {0x919C,0x4E11}, 
+  {0x91AB,0x533B}, {0x91AC,0x9171}, {0x91C0,0x917F}, {0x91C1,0x8845}, {0x91C3,0x917E}, 
+  {0x91C5,0x917D}, {0x91CB,0x91CA}, {0x91D2,0x9485}, {0x91D3,0x9486}, {0x91D4,0x9487}, 
+  {0x91D5,0x948C}, {0x91D7,0x948A}, {0x91D8,0x9489}, {0x91D9,0x948B}, {0x91DD,0x9488}, 
+  {0x91E3,0x9493}, {0x91E4,0x9490}, {0x91E7,0x948F}, {0x91E9,0x9492}, {0x91F5,0x9497}, 
+  {0x91F7,0x948D}, {0x91F9,0x9495}, {0x91FA,0x948E}, {0x9200,0x94AF}, {0x9201,0x94AB}, 
+  {0x9204,0x94AD}, {0x9208,0x949A}, {0x9209,0x94A0}, {0x920D,0x949D}, {0x920E,0x94A9}, 
+  {0x9210,0x94A4}, {0x9211,0x94A3}, {0x9214,0x949E}, {0x9215,0x94AE}, {0x921E,0x94A7}, 
+  {0x9223,0x9499}, {0x9225,0x94AC}, {0x9226,0x949B}, {0x9227,0x94AA}, {0x922E,0x94CC}, 
+  {0x9230,0x94C8}, {0x9233,0x94B6}, {0x9234,0x94C3}, {0x9237,0x94B4}, {0x9238,0x94B9}, 
+  {0x9239,0x94CD}, {0x923A,0x94B0}, {0x923D,0x94B8}, {0x923E,0x94C0}, {0x923F,0x94BF}, 
+  {0x9240,0x94BE}, {0x9245,0x949C}, {0x9248,0x94CA}, {0x9249,0x94C9}, {0x924D,0x94CB}, 
+  {0x9251,0x94C2}, {0x9255,0x94B7}, {0x9257,0x94B3}, {0x925A,0x94C6}, {0x925B,0x94C5}, 
+  {0x925E,0x94BA}, {0x9262,0x94B5}, {0x9266,0x94B2}, {0x926C,0x94BC}, {0x926D,0x94BD}, 
+  {0x9278,0x94F0}, {0x927A,0x94D2}, {0x927B,0x94EC}, {0x927F,0x94EA}, {0x9280,0x94F6}, 
+  {0x9283,0x94F3}, {0x9285,0x94DC}, {0x9291,0x94E3}, {0x9293,0x94E8}, {0x9296,0x94E2}, 
+  {0x9298,0x94ED}, {0x929A,0x94EB}, {0x929C,0x8854}, {0x92A0,0x94D1}, {0x92A3,0x94F7}, 
+  {0x92A5,0x94F1}, {0x92A6,0x94DF}, {0x92A8,0x94F5}, {0x92A9,0x94E5}, {0x92AA,0x94D5}, 
+  {0x92AB,0x94EF}, {0x92AC,0x94D0}, {0x92B1,0x94DE}, {0x92B7,0x9500}, {0x92B9,0x9508}, 
+  {0x92BB,0x9511}, {0x92BC,0x9509}, {0x92C1,0x94DD}, {0x92C3,0x9512}, {0x92C5,0x950C}, 
+  {0x92C7,0x94A1}, {0x92CC,0x94E4}, {0x92CF,0x94D7}, {0x92D2,0x950B}, {0x92DD,0x950A}, 
+  {0x92DF,0x9513}, {0x92E3,0x94D8}, {0x92E4,0x9504}, {0x92E5,0x9503}, {0x92E6,0x9514}, 
+  {0x92E8,0x9507}, {0x92EA,0x94FA}, {0x92ED,0x9510}, {0x92EE,0x94D6}, {0x92EF,0x9506}, 
+  {0x92F0,0x9502}, {0x92F1,0x94FD}, {0x92F6,0x950D}, {0x92F8,0x952F}, {0x92FC,0x94A2}, 
+  {0x9301,0x951E}, {0x9306,0x9516}, {0x9307,0x952B}, {0x9308,0x9529}, {0x9310,0x9525}, 
+  {0x9312,0x9515}, {0x9315,0x951F}, {0x9318,0x9524}, {0x9319,0x9531}, {0x931A,0x94EE}, 
+  {0x931B,0x951B}, {0x931F,0x952C}, {0x9320,0x952D}, {0x9322,0x94B1}, {0x9326,0x9526}, 
+  {0x9328,0x951A}, {0x932B,0x9521}, {0x932E,0x9522}, {0x932F,0x9519}, {0x9332,0x5F55}, 
+  {0x9333,0x9530}, {0x9338,0x94FC}, {0x9340,0x951D}, {0x9341,0x9528}, {0x9343,0x952A}, 
+  {0x9346,0x9494}, {0x9347,0x9534}, {0x934B,0x9505}, {0x934D,0x9540}, {0x9354,0x9537}, 
+  {0x9358,0x94E1}, {0x935B,0x953B}, {0x9364,0x9538}, {0x9365,0x9532}, {0x9369,0x9518}, 
+  {0x936C,0x9539}, {0x9370,0x953E}, {0x9375,0x952E}, {0x9376,0x9536}, {0x937A,0x9517}, 
+  {0x937E,0x953A}, {0x9382,0x9541}, {0x9384,0x953F}, {0x9387,0x9545}, {0x938A,0x9551}, 
+  {0x9396,0x9501}, {0x9398,0x9549}, {0x93A2,0x94A8}, {0x93A3,0x84E5}, {0x93A6,0x954F}, 
+  {0x93A7,0x94E0}, {0x93A9,0x94E9}, {0x93AA,0x953C}, {0x93AC,0x9550}, {0x93AE,0x9547}, 
+  {0x93B0,0x9552}, {0x93B3,0x954D}, {0x93B5,0x9553}, {0x93B8,0x954C}, {0x93BF,0x954E}, 
+  {0x93C3,0x955E}, {0x93C7,0x955F}, {0x93C8,0x94FE}, {0x93CC,0x9546}, {0x93CD,0x9559}, 
+  {0x93D1,0x955D}, {0x93D7,0x94FF}, {0x93D8,0x9535}, {0x93DC,0x9557}, {0x93DD,0x9558}, 
+  {0x93DE,0x955B}, {0x93DF,0x94F2}, {0x93E1,0x955C}, {0x93E2,0x9556}, {0x93E4,0x9542}, 
+  {0x93E8,0x933E}, {0x93F5,0x94E7}, {0x93F7,0x9564}, {0x93F9,0x956A}, {0x9403,0x94D9}, 
+  {0x940B,0x94F4}, {0x9410,0x9563}, {0x9412,0x94F9}, {0x9413,0x9566}, {0x9414,0x9561}, 
+  {0x9418,0x949F}, {0x9419,0x956B}, {0x941D,0x9562}, {0x9420,0x9568}, {0x9426,0x950E}, 
+  {0x9427,0x950F}, {0x9428,0x9544}, {0x942E,0x9570}, {0x9432,0x956F}, {0x9433,0x956D}, 
+  {0x9435,0x94C1}, {0x9438,0x94CE}, {0x943A,0x94DB}, {0x943F,0x9571}, {0x9444,0x94F8}, 
+  {0x944A,0x956C}, {0x944C,0x9554}, {0x9452,0x9274}, {0x9454,0x9572}, {0x9460,0x94C4}, 
+  {0x9463,0x9573}, {0x9465,0x9565}, {0x946D,0x9567}, {0x9470,0x94A5}, {0x9472,0x9576}, 
+  {0x9477,0x954A}, {0x9479,0x9569}, {0x947C,0x9523}, {0x947D,0x94BB}, {0x947E,0x92AE}, 
+  {0x947F,0x51FF}, {0x9577,0x957F}, {0x9580,0x95E8}, {0x9582,0x95E9}, {0x9583,0x95EA}, 
+  {0x9586,0x95EB}, {0x9589,0x95ED}, {0x958B,0x5F00}, {0x958C,0x95F6}, {0x958E,0x95F3}, 
+  {0x958F,0x95F0}, {0x9591,0x95F2}, {0x9593,0x95F4}, {0x9594,0x95F5}, {0x9598,0x95F8}, 
+  {0x95A1,0x9602}, {0x95A3,0x9601}, {0x95A5,0x9600}, {0x95A8,0x95FA}, {0x95A9,0x95FD}, 
+  {0x95AB,0x9603}, {0x95AC,0x9606}, {0x95AD,0x95FE}, {0x95B2,0x9605}, {0x95B6,0x960A}, 
+  {0x95B9,0x9609}, {0x95BB,0x960E}, {0x95BC,0x960F}, {0x95BD,0x960D}, {0x95BE,0x9608}, 
+  {0x95BF,0x960C}, {0x95C3,0x9612}, {0x95C8,0x95F1}, {0x95CA,0x9614}, {0x95CB,0x9615}, 
+  {0x95CC,0x9611}, {0x95D0,0x9617}, {0x95D4,0x9616}, {0x95D5,0x9619}, {0x95D6,0x95EF}, 
+  {0x95DC,0x5173}, {0x95DE,0x961A}, {0x95E1,0x9610}, {0x95E5,0x95FC}, {0x9658,0x9649}, 
+  {0x965D,0x9655}, {0x9663,0x9635}, {0x9670,0x9634}, {0x9673,0x9648}, {0x9678,0x9646}, 
+  {0x967D,0x9633}, {0x9689,0x9667}, {0x968A,0x961F}, {0x968E,0x9636}, {0x9695,0x9668}, 
+  {0x969B,0x9645}, {0x96A8,0x968F}, {0x96AA,0x9669}, {0x96B1,0x9690}, {0x96B4,0x9647}, 
+  {0x96B8,0x96B6}, {0x96D6,0x867D}, {0x96D9,0x53CC}, {0x96DB,0x96CF}, {0x96DC,0x6742}, 
+  {0x96E2,0x79BB}, {0x96E3,0x96BE}, {0x96F2,0x4E91}, {0x96FB,0x7535}, {0x9727,0x96FE}, 
+  {0x973D,0x9701}, {0x9742,0x96F3}, {0x9744,0x972D}, {0x9748,0x7075}, {0x975A,0x9753}, 
+  {0x975C,0x9759}, {0x9768,0x9765}, {0x978F,0x5DE9}, {0x97BD,0x9792}, {0x97C3,0x9791}, 
+  {0x97C9,0x97AF}, {0x97CB,0x97E6}, {0x97CC,0x97E7}, {0x97D3,0x97E9}, {0x97D9,0x97EA}, 
+  {0x97DC,0x97EC}, {0x97DE,0x97EB}, {0x97FF,0x54CD}, {0x9801,0x9875}, {0x9802,0x9876}, 
+  {0x9803,0x9877}, {0x9805,0x9879}, {0x9806,0x987A}, {0x9807,0x9878}, {0x9808,0x987B}, 
+  {0x980A,0x987C}, {0x980C,0x9882}, {0x980E,0x9880}, {0x980F,0x9883}, {0x9810,0x9884}, 
+  {0x9811,0x987D}, {0x9812,0x9881}, {0x9813,0x987F}, {0x9817,0x9887}, {0x9818,0x9886}, 
+  {0x981C,0x988C}, {0x9821,0x9889}, {0x9824,0x9890}, {0x9826,0x988F}, {0x982D,0x5934}, 
+  {0x9830,0x988A}, {0x9837,0x9894}, {0x9838,0x9888}, {0x983B,0x9891}, {0x983D,0x9893}, 
+  {0x9846,0x9897}, {0x984C,0x9898}, {0x984D,0x989D}, {0x984E,0x989A}, {0x9853,0x989B}, 
+  {0x9854,0x989C}, {0x9858,0x613F}, {0x9859,0x98A1}, {0x985B,0x98A0}, {0x985E,0x7C7B}, 
+  {0x9862,0x989F}, {0x9865,0x98A2}, {0x9867,0x987E}, {0x986B,0x98A4}, {0x986C,0x98A5}, 
+  {0x986F,0x663E}, {0x9870,0x98A6}, {0x9871,0x9885}, {0x9873,0x989E}, {0x9874,0x98A7}, 
+  {0x98A8,0x98CE}, {0x98AE,0x98D1}, {0x98AF,0x98D2}, {0x98B6,0x98D3}, {0x98BC,0x98D5}, 
+  {0x98C4,0x98D8}, {0x98C6,0x98D9}, {0x98C8,0x98DA}, {0x98DB,0x98DE}, {0x98E0,0x9963}, 
+  {0x98E9,0x9968}, {0x98EA,0x996A}, {0x98EB,0x996B}, {0x98ED,0x996C}, {0x98EF,0x996D}, 
+  {0x98F2,0x996E}, {0x98F4,0x9974}, {0x98FC,0x9972}, {0x98FD,0x9971}, {0x98FE,0x9970}, 
+  {0x9903,0x997A}, {0x9905,0x997C}, {0x9909,0x9977}, {0x990A,0x517B}, {0x990C,0x9975}, 
+  {0x9911,0x997D}, {0x9912,0x9981}, {0x9913,0x997F}, {0x9918,0x4F59}, {0x991B,0x9984}, 
+  {0x991E,0x996F}, {0x9921,0x9985}, {0x9928,0x9986}, {0x9933,0x9967}, {0x9937,0x9987}, 
+  {0x993C,0x9969}, {0x993E,0x998F}, {0x993F,0x998A}, {0x9943,0x998D}, {0x9945,0x9992}, 
+  {0x9948,0x9990}, {0x9949,0x9991}, {0x994A,0x9993}, {0x994B,0x9988}, {0x994C,0x9994}, 
+  {0x9951,0x9965}, {0x9952,0x9976}, {0x9957,0x98E8}, {0x995C,0x990D}, {0x995E,0x998B}, 
+  {0x9962,0x9995}, {0x99AC,0x9A6C}, {0x99AD,0x9A6D}, {0x99AE,0x51AF}, {0x99B1,0x9A6E}, 
+  {0x99B3,0x9A70}, {0x99B4,0x9A6F}, {0x99C1,0x9A73}, {0x99D0,0x9A7B}, {0x99D1,0x9A7D}, 
+  {0x99D2,0x9A79}, {0x99D4,0x9A75}, {0x99D5,0x9A7E}, {0x99D8,0x9A80}, {0x99D9,0x9A78}, 
+  {0x99DB,0x9A76}, {0x99DD,0x9A7C}, {0x99DF,0x9A77}, {0x99E1,0x9A82}, {0x99E2,0x9A88}, 
+  {0x99ED,0x9A87}, {0x99F1,0x9A86}, {0x99FF,0x9A8F}, {0x9A01,0x9A8B}, {0x9A05,0x9A93}, 
+  {0x9A0D,0x9A92}, {0x9A0E,0x9A91}, {0x9A0F,0x9A90}, {0x9A16,0x9A9B}, {0x9A19,0x9A97}, 
+  {0x9A2B,0x9A9E}, {0x9A2D,0x9A98}, {0x9A2E,0x9A9D}, {0x9A30,0x817E}, {0x9A36,0x9A7A}, 
+  {0x9A37,0x9A9A}, {0x9A38,0x9A9F}, {0x9A3E,0x9AA1}, {0x9A40,0x84E6}, {0x9A41,0x9A9C}, 
+  {0x9A42,0x9A96}, {0x9A43,0x9AA0}, {0x9A44,0x9AA2}, {0x9A45,0x9A71}, {0x9A4A,0x9A85}, 
+  {0x9A4D,0x9A81}, {0x9A4F,0x9AA3}, {0x9A55,0x9A84}, {0x9A57,0x9A8C}, {0x9A5A,0x60CA}, 
+  {0x9A5B,0x9A7F}, {0x9A5F,0x9AA4}, {0x9A62,0x9A74}, {0x9A64,0x9AA7}, {0x9A65,0x9AA5}, 
+  {0x9A6A,0x9A8A}, {0x9AAF,0x80AE}, {0x9ACF,0x9AC5}, {0x9AD4,0x4F53}, {0x9AD5,0x9ACC}, 
+  {0x9AD6,0x9ACB}, {0x9B06,0x677E}, {0x9B22,0x9B13}, {0x9B25,0x6597}, {0x9B27,0x95F9}, 
+  {0x9B29,0x960B}, {0x9B2E,0x9604}, {0x9B31,0x90C1}, {0x9B4E,0x9B49}, {0x9B58,0x9B47}, 
+  {0x9B5A,0x9C7C}, {0x9B6F,0x9C81}, {0x9B74,0x9C82}, {0x9B77,0x9C7F}, {0x9B81,0x9C85}, 
+  {0x9B83,0x9C86}, {0x9B8E,0x9C87}, {0x9B90,0x9C90}, {0x9B91,0x9C8D}, {0x9B92,0x9C8B}, 
+  {0x9B9A,0x9C92}, {0x9B9D,0x9C9E}, {0x9B9E,0x9C95}, {0x9BAA,0x9C94}, {0x9BAB,0x9C9B}, 
+  {0x9BAD,0x9C91}, {0x9BAE,0x9C9C}, {0x9BC0,0x9CA7}, {0x9BC1,0x9CA0}, {0x9BC7,0x9CA9}, 
+  {0x9BC9,0x9CA4}, {0x9BCA,0x9CA8}, {0x9BD4,0x9CBB}, {0x9BD6,0x9CAD}, {0x9BDB,0x9CB7}, 
+  {0x9BDD,0x9CB4}, {0x9BE1,0x9CB1}, {0x9BE2,0x9CB5}, {0x9BE4,0x9CB2}, {0x9BE7,0x9CB3}, 
+  {0x9BE8,0x9CB8}, {0x9BEA,0x9CAE}, {0x9BEB,0x9CB0}, {0x9BF0,0x9CB6}, {0x9BF4,0x9CBA}, 
+  {0x9BFD,0x9CAB}, {0x9BFF,0x9CCA}, {0x9C08,0x9CBD}, {0x9C09,0x9CC7}, {0x9C0D,0x9CC5}, 
+  {0x9C10,0x9CC4}, {0x9C12,0x9CC6}, {0x9C13,0x9CC3}, {0x9C20,0x9CCB}, {0x9C23,0x9CA5}, 
+  {0x9C25,0x9CCF}, {0x9C28,0x9CCE}, {0x9C29,0x9CD0}, {0x9C2D,0x9CCD}, {0x9C31,0x9CA2}, 
+  {0x9C32,0x9CCC}, {0x9C33,0x9CD3}, {0x9C35,0x9CD8}, {0x9C37,0x9CA6}, {0x9C39,0x9CA3}, 
+  {0x9C3B,0x9CD7}, {0x9C3E,0x9CD4}, {0x9C45,0x9CD9}, {0x9C48,0x9CD5}, {0x9C49,0x9CD6}, 
+  {0x9C52,0x9CDF}, {0x9C54,0x9CDD}, {0x9C56,0x9CDC}, {0x9C57,0x9CDE}, {0x9C58,0x9C9F}, 
+  {0x9C5D,0x9CBC}, {0x9C5F,0x9C8E}, {0x9C67,0x9CE2}, {0x9C6D,0x9C9A}, {0x9C78,0x9C88}, 
+  {0x9C7A,0x9CA1}, {0x9CE5,0x9E1F}, {0x9CE9,0x9E20}, {0x9CEC,0x51EB}, {0x9CF3,0x51E4}, 
+  {0x9CF4,0x9E23}, {0x9CF6,0x9E22}, {0x9D06,0x9E29}, {0x9D07,0x9E28}, {0x9D09,0x9E26}, 
+  {0x9D15,0x9E35}, {0x9D1B,0x9E33}, {0x9D1D,0x9E32}, {0x9D1F,0x9E31}, {0x9D23,0x9E2A}, 
+  {0x9D26,0x9E2F}, {0x9D28,0x9E2D}, {0x9D2F,0x9E38}, {0x9D30,0x9E39}, {0x9D3B,0x9E3F}, 
+  {0x9D3F,0x9E3D}, {0x9D42,0x9E3A}, {0x9D51,0x9E43}, {0x9D52,0x9E46}, {0x9D53,0x9E41}, 
+  {0x9D5C,0x9E48}, {0x9D5D,0x9E45}, {0x9D60,0x9E44}, {0x9D61,0x9E49}, {0x9D6A,0x9E4C}, 
+  {0x9D6C,0x9E4F}, {0x9D6F,0x9E4E}, {0x9D72,0x9E4A}, {0x9D87,0x9E2B}, {0x9D89,0x9E51}, 
+  {0x9D93,0x9E4B}, {0x9D98,0x9E55}, {0x9D9A,0x9E57}, {0x9DA5,0x9E5B}, {0x9DA9,0x9E5C}, 
+  {0x9DAF,0x83BA}, {0x9DB4,0x9E64}, {0x9DBB,0x9E58}, {0x9DBC,0x9E63}, {0x9DC0,0x9E5A}, 
+  {0x9DC2,0x9E5E}, {0x9DC4,0x9E21}, {0x9DD3,0x9E67}, {0x9DD7,0x9E25}, {0x9DD9,0x9E37}, 
+  {0x9DDA,0x9E68}, {0x9DE5,0x9E36}, {0x9DE6,0x9E6A}, {0x9DEF,0x9E69}, {0x9DF2,0x9E6B}, 
+  {0x9DF3,0x9E47}, {0x9DF8,0x9E6C}, {0x9DF9,0x9E70}, {0x9DFA,0x9E6D}, {0x9E0C,0x9E71}, 
+  {0x9E15,0x9E2C}, {0x9E1A,0x9E66}, {0x9E1B,0x9E73}, {0x9E1D,0x9E42}, {0x9E1E,0x9E3E}, 
+  {0x9E7A,0x9E7E}, {0x9E7C,0x7877}, {0x9E7D,0x76D0}, {0x9E97,0x4E3D}, {0x9EA5,0x9EA6}, 
+  {0x9EA9,0x9EB8}, {0x9EBD,0x4E48}, {0x9ECC,0x9EC9}, {0x9EDE,0x70B9}, {0x9EE8,0x515A}, 
+  {0x9EF2,0x9EEA}, {0x9EF7,0x9EE9}, {0x9EFD,0x9EFE}, {0x9EFF,0x9F0B}, {0x9F09,0x9F0D}, 
+  {0x9F4A,0x9F50}, {0x9F4B,0x658B}, {0x9F4F,0x9F51}, {0x9F52,0x9F7F}, {0x9F54,0x9F80}, 
+  {0x9F59,0x9F85}, {0x9F5C,0x9F87}, {0x9F5F,0x9F83}, {0x9F60,0x9F86}, {0x9F61,0x9F84}, 
+  {0x9F66,0x9F88}, {0x9F6A,0x9F8A}, {0x9F6C,0x9F89}, {0x9F72,0x9F8B}, {0x9F77,0x9F8C}, 
+  {0x9F8D,0x9F99}, {0x9F90,0x5E9E}, {0x9F94,0x9F9A}, {0x9F95,0x9F9B}, {0x9F9C,0x9F9F}
+};
+
+
+
+/* the letters correspond to unicode characters which
+ * have Lu, Ll, Lt, Lm, Lo, Mn, Mc, or Me in their
+ * GENERAL CATEGORY */
+
+#define NUM_LETTER_INFO 274
+static const unirange letter_info[NUM_LETTER_INFO] = {
+  {0x41,0x5a}, {0x61,0x7a}, {0xaa,0xaa}, {0xb5,0xb5}, 
+  {0xba,0xba}, {0xc0,0xd6}, {0xd8,0xf6}, {0xf8,0x1f5}, 
+  {0x1fa,0x217}, {0x250,0x2a8}, {0x2b0,0x2b8}, {0x2bb,0x2c1}, 
+  {0x2d0,0x2d1}, {0x2e0,0x2e4}, {0x300,0x345}, {0x360,0x361}, 
+  {0x37a,0x37a}, {0x386,0x386}, {0x388,0x38a}, {0x38c,0x38c}, 
+  {0x38e,0x3a1}, {0x3a3,0x3ce}, {0x3d0,0x3d6}, {0x3da,0x3da}, 
+  {0x3dc,0x3dc}, {0x3de,0x3de}, {0x3e0,0x3e0}, {0x3e2,0x3f3}, 
+  {0x401,0x40c}, {0x40e,0x44f}, {0x451,0x45c}, {0x45e,0x481}, 
+  {0x483,0x486}, {0x490,0x4c4}, {0x4c7,0x4c8}, {0x4cb,0x4cc}, 
+  {0x4d0,0x4eb}, {0x4ee,0x4f5}, {0x4f8,0x4f9}, {0x531,0x556}, 
+  {0x559,0x559}, {0x561,0x587}, {0x591,0x5a1}, {0x5a3,0x5b9}, 
+  {0x5bb,0x5bd}, {0x5bf,0x5bf}, {0x5c1,0x5c2}, {0x5c4,0x5c4}, 
+  {0x5d0,0x5ea}, {0x5f0,0x5f2}, {0x621,0x63a}, {0x640,0x652}, 
+  {0x670,0x6b7}, {0x6ba,0x6be}, {0x6c0,0x6ce}, {0x6d0,0x6d3}, 
+  {0x6d5,0x6e8}, {0x6ea,0x6ed}, {0x901,0x903}, {0x905,0x939}, 
+  {0x93c,0x94d}, {0x951,0x954}, {0x958,0x963}, {0x981,0x983}, 
+  {0x985,0x98c}, {0x98f,0x990}, {0x993,0x9a8}, {0x9aa,0x9b0}, 
+  {0x9b2,0x9b2}, {0x9b6,0x9b9}, {0x9bc,0x9bc}, {0x9be,0x9c4}, 
+  {0x9c7,0x9c8}, {0x9cb,0x9cd}, {0x9d7,0x9d7}, {0x9dc,0x9dd}, 
+  {0x9df,0x9e3}, {0x9f0,0x9f1}, {0xa02,0xa02}, {0xa05,0xa0a}, 
+  {0xa0f,0xa10}, {0xa13,0xa28}, {0xa2a,0xa30}, {0xa32,0xa33}, 
+  {0xa35,0xa36}, {0xa38,0xa39}, {0xa3c,0xa3c}, {0xa3e,0xa42}, 
+  {0xa47,0xa48}, {0xa4b,0xa4d}, {0xa59,0xa5c}, {0xa5e,0xa5e}, 
+  {0xa70,0xa74}, {0xa81,0xa83}, {0xa85,0xa8b}, {0xa8d,0xa8d}, 
+  {0xa8f,0xa91}, {0xa93,0xaa8}, {0xaaa,0xab0}, {0xab2,0xab3}, 
+  {0xab5,0xab9}, {0xabc,0xac5}, {0xac7,0xac9}, {0xacb,0xacd}, 
+  {0xae0,0xae0}, {0xb01,0xb03}, {0xb05,0xb0c}, {0xb0f,0xb10}, 
+  {0xb13,0xb28}, {0xb2a,0xb30}, {0xb32,0xb33}, {0xb36,0xb39}, 
+  {0xb3c,0xb43}, {0xb47,0xb48}, {0xb4b,0xb4d}, {0xb56,0xb57}, 
+  {0xb5c,0xb5d}, {0xb5f,0xb61}, {0xb82,0xb83}, {0xb85,0xb8a}, 
+  {0xb8e,0xb90}, {0xb92,0xb95}, {0xb99,0xb9a}, {0xb9c,0xb9c}, 
+  {0xb9e,0xb9f}, {0xba3,0xba4}, {0xba8,0xbaa}, {0xbae,0xbb5}, 
+  {0xbb7,0xbb9}, {0xbbe,0xbc2}, {0xbc6,0xbc8}, {0xbca,0xbcd}, 
+  {0xbd7,0xbd7}, {0xc01,0xc03}, {0xc05,0xc0c}, {0xc0e,0xc10}, 
+  {0xc12,0xc28}, {0xc2a,0xc33}, {0xc35,0xc39}, {0xc3e,0xc44}, 
+  {0xc46,0xc48}, {0xc4a,0xc4d}, {0xc55,0xc56}, {0xc60,0xc61}, 
+  {0xc82,0xc83}, {0xc85,0xc8c}, {0xc8e,0xc90}, {0xc92,0xca8}, 
+  {0xcaa,0xcb3}, {0xcb5,0xcb9}, {0xcbe,0xcc4}, {0xcc6,0xcc8}, 
+  {0xcca,0xccd}, {0xcd5,0xcd6}, {0xcde,0xcde}, {0xce0,0xce1}, 
+  {0xd02,0xd03}, {0xd05,0xd0c}, {0xd0e,0xd10}, {0xd12,0xd28}, 
+  {0xd2a,0xd39}, {0xd3e,0xd43}, {0xd46,0xd48}, {0xd4a,0xd4d}, 
+  {0xd57,0xd57}, {0xd60,0xd61}, {0xe01,0xe2e}, {0xe30,0xe3a}, 
+  {0xe40,0xe4e}, {0xe81,0xe82}, {0xe84,0xe84}, {0xe87,0xe88}, 
+  {0xe8a,0xe8a}, {0xe8d,0xe8d}, {0xe94,0xe97}, {0xe99,0xe9f}, 
+  {0xea1,0xea3}, {0xea5,0xea5}, {0xea7,0xea7}, {0xeaa,0xeab}, 
+  {0xead,0xeae}, {0xeb0,0xeb9}, {0xebb,0xebd}, {0xec0,0xec4}, 
+  {0xec6,0xec6}, {0xec8,0xecd}, {0xedc,0xedd}, {0xf18,0xf19}, 
+  {0xf35,0xf35}, {0xf37,0xf37}, {0xf39,0xf39}, {0xf40,0xf47}, 
+  {0xf49,0xf69}, {0xf71,0xf84}, {0xf86,0xf87}, {0xf90,0xf95}, 
+  {0xf97,0xf97}, {0xf99,0xfad}, {0xfb1,0xfb7}, {0xfb9,0xfb9}, 
+  {0x10a0,0x10c5}, {0x10d0,0x10f6}, {0x1100,0x1159}, {0x115f,0x11a2}, 
+  {0x11a8,0x11f9}, {0x1e00,0x1e9b}, {0x1ea0,0x1ef9}, {0x1f00,0x1f15}, 
+  {0x1f18,0x1f1d}, {0x1f20,0x1f45}, {0x1f48,0x1f4d}, {0x1f50,0x1f57}, 
+  {0x1f59,0x1f59}, {0x1f5b,0x1f5b}, {0x1f5d,0x1f5d}, {0x1f5f,0x1f7d}, 
+  {0x1f80,0x1fb4}, {0x1fb6,0x1fbc}, {0x1fbe,0x1fbe}, {0x1fc2,0x1fc4}, 
+  {0x1fc6,0x1fcc}, {0x1fd0,0x1fd3}, {0x1fd6,0x1fdb}, {0x1fe0,0x1fec}, 
+  {0x1ff2,0x1ff4}, {0x1ff6,0x1ffc}, {0x207f,0x207f}, {0x20d0,0x20e1}, 
+  {0x2102,0x2102}, {0x2107,0x2107}, {0x210a,0x2113}, {0x2115,0x2115}, 
+  {0x2118,0x211d}, {0x2124,0x2124}, {0x2126,0x2126}, {0x2128,0x2128}, 
+  {0x212a,0x2131}, {0x2133,0x2138}, {0x3005,0x3005}, {0x302a,0x302f}, 
+  {0x3031,0x3035}, {0x3041,0x3094}, {0x3099,0x309e}, {0x30a1,0x30fa}, 
+  {0x30fc,0x30fe}, {0x3105,0x312c}, {0x3131,0x318e}, {0x4e00,0x9fa5}, 
+  {0xac00,0xd7a3}, {0xf900,0xfa2d}, {0xfb00,0xfb06}, {0xfb13,0xfb17}, 
+  {0xfb1e,0xfb28}, {0xfb2a,0xfb36}, {0xfb38,0xfb3c}, {0xfb3e,0xfb3e}, 
+  {0xfb40,0xfb41}, {0xfb43,0xfb44}, {0xfb46,0xfbb1}, {0xfbd3,0xfd3d}, 
+  {0xfd50,0xfd8f}, {0xfd92,0xfdc7}, {0xfdf0,0xfdfb}, {0xfe20,0xfe23}, 
+  {0xfe70,0xfe72}, {0xfe74,0xfe74}, {0xfe76,0xfefc}, {0xff21,0xff3a}, 
+  {0xff41,0xff5a}, {0xff66,0xffbe}, {0xffc2,0xffc7}, {0xffca,0xffcf}, 
+  {0xffd2,0xffd7}, {0xffda,0xffdc}
+};
+
+
+/* the digits correspond to unicode characters which
+ * have Nd in their GENERAL CATEGORY */
+
+#define NUM_DIGIT_INFO 16
+static const unirange digit_info[NUM_DIGIT_INFO] = {
+  {0x30,0x39}, {0x660,0x669}, {0x6f0,0x6f9}, {0x966,0x96f}, 
+  {0x9e6,0x9ef}, {0xa66,0xa6f}, {0xae6,0xaef}, {0xb66,0xb6f}, 
+  {0xbe7,0xbef}, {0xc66,0xc6f}, {0xce6,0xcef}, {0xd66,0xd6f}, 
+  {0xe50,0xe59}, {0xed0,0xed9}, {0xf20,0xf29}, {0xff10,0xff19}
+};
+
+
+/* the spaces correspond to unicode characters which
+ * have Zs, Zl, or Zp in their GENERAL CATEGORY */
+
+#define NUM_SPACE_INFO 5
+static const unirange space_info[NUM_SPACE_INFO] = {
+  {0x20,0x20}, {0xa0,0xa0}, {0x2000,0x200b}, {0x2028,0x2029}, 
+  {0x3000,0x3000}
+};
+
+
+
+/* parse_utf8_char parses the next utf-8 character, placing its
+ * unicode equivalent in *value. The length of the utf-8 character
+ * is returned */
+int parse_utf8_char(const unsigned char *here,
+		    const unsigned char *end,
+		    unsigned short *value) {
+  unsigned short c1;
+  unsigned short c2;
+  unsigned short c3;
+  
+  *value = 0; /* will be the value for illegal characters */
+  
+  if (here <= end) {
+    c1 = (unsigned short)(*here);
+    
+    if (c1 <= 0x7f) {
+      /* found a one-byte character */
+      *value = c1;
+      return 1;
+      
+    } else if (c1 >= 0xc0 && c1 <= 0xdf && here+1 <= end &&
+	       (c2=(unsigned short)(*(here+1))) >= 0x80) {
+      /* found a two-byte character */
+      *value = ((c1 & 0x1f) << 6) + (c2 & 0x3f);
+      return 2;
+      
+    } else if (c1 >= 0xe0 && c1 <= 0xef && here+2 <= end &&
+	       (c2=(unsigned int)(*(here+1))) >= 0x80 &&
+	       (c3=(unsigned int)(*(here+2))) >= 0x80) {
+      /* found a three-byte character */
+      *value = ((c1 & 0xf) << 12) + ((c2 & 0x3f) << 6) +
+	(c3 & 0x3f);
+      return 3;
+      
+    } else {
+      /* error in encoding, return the length as 1 */
+      return 1;
+    }
+  }
+
+  /* end of string */
+  return 0;
+}
+
+
+/* output_utf8_char encodes a unicode character as a UTF-8 character.
+ * The length of the encoding is returned. If the string was not
+ * long enough to encode the character 0 is returned. */
+int output_utf8_char (unsigned short value,
+		      unsigned char *here,
+		      unsigned char *end) {
+  int len = (int)(end - here + 1); /* end is the last character */
+  if (value < 0x80) {
+    /* will be encoded in one byte */
+    if (len >= 1) {
+      *here = (unsigned char)value;
+      return 1;
+    }
+
+  } else if (value < 0x800) {
+    /* will be encoded in two bytes */
+    if (len >= 2) {
+      *here++ = 0xc0 | (unsigned char)((value >> 6) & 0x1f);
+      *here = 0x80 | (unsigned char)(value & 0x3f);
+
+      return 2;
+    }
+
+  } else {
+    /* will be encoded in three bytes */
+    if (len >= 3) {
+      *here++ = 0xe0 | (unsigned char)((value >> 12) & 0xf);
+      *here++ = 0x80 | (unsigned char)((value >> 6) & 0x3f);
+      *here = 0x80 | (unsigned char)(value & 0x3f);
+      return 3;
+    }
+  }
+
+  return 0;
+}
+
+
+static int lookup_decompose_table (unsigned short value) {
+  int first = 0;
+  int last = NUM_DECOMPOSE_INFO;
+  int here;
+
+  while (first < last) {
+    here = (last-first)/2 + first;
+    if (value < decompose_info[here].ch) {
+      last = here;
+    } else if (value > decompose_info[here].ch) {
+      first = here+1;
+    } else {
+      return here;
+    }
+  }
+
+  return -1;
+}
+
+/* decompose_str will decompose a unicode string into its canonical
+ * equivalents. NULL is returned if the input array was not 
+ * large enough to contain the fully decomposed string (the array
+ * will be in a correct, but partially decomposed state). The input 
+ * must be null-terminated. */
+unsigned short *decompose_str (unsigned short *input,
+			       int max_output_len) {
+  int curpos = 0;
+  int curlen = unicode_strlen (input);
+  int tablepos = 0;
+  int leninc;
+  int i;
+
+  while (input[curpos] != 0) {
+    tablepos = lookup_decompose_table (input[curpos]);
+    if (tablepos >= 0) {
+      /* found a decomposition */
+      leninc = unicode_strnlen (decompose_info[tablepos].str, 3)-1;
+      if (curlen+leninc < max_output_len) {
+	/* enough space for this decomposition */
+
+	if (leninc > 1) {
+	  /* we have to shift the array along */
+	  for (i=curlen-1; i>=curpos; i--) {
+	    input[i+leninc] = input[i];
+	  }
+	  input[curlen+leninc] = 0;
+	  curlen += leninc;
+	}
+
+	/* decompose the character */
+	for (i=0; i<=leninc; i++) {
+	  input[curpos+i] = decompose_info[tablepos].str[i];
+	}
+
+	/* note: curpos is not incremented because the decomposed */
+	/*       characters might need decomposing */
+
+      } else {
+	/* no space left in string */
+	return NULL;
+      }
+
+    } else {
+      /* the character cannot be decomposed */
+      curpos++;
+    }
+  }
+
+  return input;
+}
+
+
+
+/* unicode_cat_data is a combination of the letter, digit, and
+ * space information in a form that is quick to look up.
+ * The information about each byte is stored in two bits
+ *
+ * 00 - neither letter, digit or space
+ * 01 - letter
+ * 10 - space
+ * 11 - digit
+ *
+ * You can test for letter | space by looking at the right
+ * column
+ */
+/*unsigned long *unicode_cat_data;*/
+
+void create_unicat_data () {
+  int i=0;
+  int min_start=0;
+  int min_end=0;
+  register unsigned long min_value=0; /* filling the whole byte */
+  
+  const unirange *letter_here = letter_info;
+  const unirange *letter_end = letter_info+NUM_LETTER_INFO;
+  const unirange *digit_here = digit_info;
+  const unirange *digit_end = digit_info+NUM_DIGIT_INFO;
+  const unirange *space_here = space_info;
+  const unirange *space_end = space_info+NUM_SPACE_INFO;
+
+  if (unicode_cat_data != NULL) return;
+  unicode_cat_data = (unsigned long *) malloc (0x10000/4);
+  if (unicode_cat_data == NULL) return;
+  
+  while (i <= 0xffff) {
+    /* decide what to fill with next */
+    min_start = i;
+    min_end = 0xffff;
+    min_value = 0; /* 00 */
+    
+    while (letter_here < letter_end && letter_here->last < i) letter_here++;
+    if (letter_here < letter_end && letter_here->last <= min_end) {
+      min_start = letter_here->first;
+      min_end = letter_here->last;
+      min_value = 0x55555555; /* 01 */
+    }
+    
+    while (digit_here < digit_end && digit_here->last < i) digit_here++;
+    if (digit_here < digit_end && digit_here->last <= min_end) {
+      min_start = digit_here->first;
+      min_end = digit_here->last;
+      min_value = 0xffffffff; /* 11 */
+    }
+    
+    while (space_here < space_end && space_here->last < i) space_here++;
+    if (space_here < space_end && space_here->last <= min_end) {
+      min_start = space_here->first;
+      min_end = space_here->last;
+      min_value = 0xaaaaaaaa; /* 10 */
+    }
+
+    if (min_start < i) min_start = i;
+    else if (min_start > i) {
+      min_end = min_start-1;
+      min_start = i;
+      min_value = 0; /* 00 */
+    }
+
+    /* fill the entries from min_start to min_end with min_value */
+    while (i <= min_end) {
+      /* special version for common case */
+      if (i % 16 == 0) {
+	unicode_cat_data[i/16] = min_value; /* doesn't matter if too long */
+	i += 16;
+	if (i > min_end) {
+	  i = min_end+1;
+	  break;
+	}
+	
+      } else {
+	register unsigned long mask = 3 << ((i % 16) * 2);
+	unicode_cat_data[i/16] = (unicode_cat_data[i/16] & (~mask)) |
+				  (min_value & mask);
+	i++;
+      }
+    }
+  }
+}
+
+/* does a binary search on a range table */
+static int in_unirange (const unirange *info, int num_info,
+			unsigned short value) {
+  int first = 0;
+  int last = num_info;
+  int here;
+
+  while (first < last) {
+    here = (last-first)/2 + first;
+    if (value < info[here].first) {
+      last = here;
+    } else if (value > info[here].last) {
+      first = here+1;
+    } else {
+      return 1;
+    }
+  }
+
+  return 0;
+}
+
+
+
+/* tests to see whether 'value' is a valid Unicode letter */
+int is_unicode_letter (unsigned short value) {
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return in_unirange(letter_info, NUM_LETTER_INFO, value);
+  }
+
+  return (((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x3) == 0x1);
+}
+
+/* tests to see whether 'value' is a valid Unicode digit */
+int is_unicode_digit (unsigned short value) {
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return in_unirange(digit_info, NUM_DIGIT_INFO, value);
+  }
+
+  return (((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x3) == 0x3);
+}
+
+/* tests to see whether 'value' is a valid Unicode letter or 
+ * digit */
+int is_unicode_letdig (unsigned short value) {
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return (in_unirange(letter_info, NUM_LETTER_INFO, value) ||
+	    in_unirange(digit_info, NUM_DIGIT_INFO, value));
+    
+  }
+
+  return ((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x1);
+}
+
+/* tests to see whether 'value' is a valid space
+ * The test includes both "C" spaces and "Unicode" spaces, i.e. 
+ * form-feed, newline, carriage return, horizontal tab,
+ * vertical tab, and the Zs, Zl, and Zp Unicode categorizations */
+int is_unicode_space (unsigned short value) {
+  if (value < 0x80) return isspace (value);
+  if (unicode_cat_data == NULL) create_unicat_data ();
+  if (unicode_cat_data == NULL) {
+    /* use old method */
+    return in_unirange(space_info, NUM_SPACE_INFO, value);
+  }
+
+  return (((unicode_cat_data[value/16] >> ((value%16)*2)) & 0x3) == 0x2);
+}
+
+/* returns the length of the unicode string */
+int unicode_strlen (const unsigned short *str) {
+  int len = 0;
+
+  /* keep going till you find the null-terminator */
+  while (*str != 0) {
+    len++;
+    str++;
+  }
+
+  return len;
+}
+
+/* returns the length of the unicode string, up to a maximum */
+int unicode_strnlen (const unsigned short *str, int max_length) {
+  int len = 0;
+
+  /* keep going till you find the null-terminator */
+  while (*str != 0 && len < max_length ) {
+    len++;
+    str++;
+  }
+
+  return len;
+}
+
+
+/* Does a binary search on a relate table, returns -1 if it is not
+ * found. */
+static int search_unirelate (const unirelate *info, int num_info,
+			     unsigned short value) {
+  int first = 0;
+  int last = num_info;
+  int here;
+
+  while (first < last) {
+    here = (last-first)/2 + first;
+    if (value < info[here].ch1) {
+      last = here;
+    } else if (value > info[here].ch1) {
+      first = here+1;
+    } else {
+      return here;
+    }
+  }
+
+  return -1;
+}
+
+
+/* returns the upper-case equivalent of value */
+unsigned short unicode_toupper (unsigned short value) {
+  int related = search_unirelate (toupper_info, NUM_TOUPPER_INFO, value);
+  if (related >= 0) {
+    value = toupper_info[related].ch2;
+  }
+  return value;
+}
+
+/* returns the lower-case equivalent of value */
+unsigned short unicode_tolower (unsigned short value) {
+  int related = search_unirelate (tolower_info, NUM_TOLOWER_INFO, value);
+  if (related >= 0) {
+    value = tolower_info[related].ch2;
+  }
+  return value;
+}
+
+/* returns the simplified Chinese character equivalent of 
+ * another Chinese character */
+unsigned short unicode_tosimplified (unsigned short value) {
+  int related = search_unirelate (tosimplified_info, NUM_TOSIMPLIFIED_INFO, value);
+  if (related >= 0) {
+    value = tosimplified_info[related].ch2;
+  }
+  return value;
+}
+
+
+/* converts a utf-8 word (string with length stored in the first byte
+ * to a Unicode array. To handle all situations the output buffer should
+ * be 256 unsigned shorts long. The output will also have the length as
+ * the first entry. */
+unsigned short *utf8_word_to_unicode (const unsigned char *input,
+				      unsigned short *output,
+				      int max_output_length) {
+  int inlen = *input;
+  const unsigned char *inhere = input+1;
+  const unsigned char *inend = &inhere[inlen-1];
+  unsigned short c;
+  int clen;
+  
+  int outlen = 0;
+  unsigned short *outhere = output+1;
+  
+  while (inhere <= inend && inlen < max_output_length-1) {
+    /* decode the character */
+    clen = parse_utf8_char (inhere, inend, &c);
+    inhere += clen;
+
+    if (clen > 0) {
+      /* store the character */
+      *outhere = c;
+      outhere++;
+      outlen++;
+    }
+  }
+
+  output[0] = outlen;
+  return output;
+}
+
+
+/* converts a unicode word buffer (with the length stored in the
+ * entry) to a utf8 encoded word output (with the length stored in
+ * the first byte. Only 255 bytes (not characters) can be stored 
+ * in the output. */
+unsigned char *unicode_to_utf8_word (const unsigned short *input,
+				     unsigned char *output,
+				     int max_output_length) {
+  int inlen = *input;
+  const unsigned short *word = input + 1;
+  int in_i = 0;
+  int clen;
+
+  int outlen = 0;
+  unsigned char *outhere = output+1;
+  unsigned char *outend = output+max_output_length;
+
+  while (in_i < inlen) {
+    if ((clen=output_utf8_char(word[in_i], outhere, outend)) == 0) {
+      /* ran out of space to encode the character */
+      break;
+    }
+
+    outhere += clen;
+    outlen += clen;
+    in_i++;
+  }
+
+  output[0] = outlen;
+
+  return output;
+}
+
Index: /trunk/mgpp/lib/unitool.h
===================================================================
--- /trunk/mgpp/lib/unitool.h	(revision 3365)
+++ /trunk/mgpp/lib/unitool.h	(revision 3365)
@@ -0,0 +1,88 @@
+#ifndef UNITOOL_H
+#define UNITOOL_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+
+/* This module is based on Unicode 2.1 */
+
+
+/* parse_utf8_char parses the next utf-8 character, placing its
+ * unicode equivalent in *value. The length of the utf-8 character
+ * is returned. end is the address of the last character. */
+int parse_utf8_char (const unsigned char *here,
+		     const unsigned char *end,
+		     unsigned short *value);
+
+/* output_utf8_char encodes a unicode character as a UTF-8 character.
+ * The length of the encoding is returned. If the string was not
+ * long enough to encode the character 0 is returned. end is the
+ * address of the last character */
+int output_utf8_char (unsigned short value,
+		      unsigned char *here,
+		      unsigned char *end);
+
+/* decompose_str will decompose a unicode string into its canonical
+ * equivalents. NULL is returned if the input array was not 
+ * large enough to contain the fully decomposed string (the array
+ * will be in a correct, but partially decomposed state). The input 
+ * must be null-terminated. */
+unsigned short *decompose_str (unsigned short *input,
+			       int max_output_len);
+
+/* tests to see whether 'value' is a valid Unicode letter */
+int is_unicode_letter (unsigned short value);
+
+/* tests to see whether 'value' is a valid Unicode digit */
+int is_unicode_digit (unsigned short value);
+
+/* tests to see whether 'value' is a valid Unicode letter or 
+ * digit */
+int is_unicode_letdig (unsigned short value);
+
+/* tests to see whether 'value' is a valid space
+ * The test includes both "C" spaces and "Unicode" spaces, i.e. 
+ * form-feed, newline, carriage return, horizontal tab,
+ * vertical tab, and the Zs, Zl, and Zp Unicode categorizations */
+int is_unicode_space (unsigned short value);
+
+/* returns the length of the unicode string */
+int unicode_strlen (const unsigned short *str);
+
+/* returns the length of the unicode string, up to a maximum */
+int unicode_strnlen (const unsigned short *str, int max_length);
+
+/* returns the upper-case equivalent of value */
+unsigned short unicode_toupper (unsigned short value);
+
+/* returns the lower-case equivalent of value */
+unsigned short unicode_tolower (unsigned short value);
+
+/* returns the simplified Chinese character equivalent of 
+ * another Chinese character */
+unsigned short unicode_tosimplified (unsigned short value);
+
+
+/* converts a utf-8 word (string with length stored in the first byte
+ * to a Unicode array. To handle all situations the output buffer should
+ * be 256 unsigned shorts long. The output will also have the length as
+ * the first entry. */
+unsigned short *utf8_word_to_unicode (const unsigned char *input,
+				      unsigned short *output,
+				      int max_output_length);
+
+/* converts a unicode word buffer (with the length stored in the
+ * entry) to a utf8 encoded word output (with the length stored in
+ * the first byte. Only 255 bytes (not characters) can be stored 
+ * in the output. */
+unsigned char *unicode_to_utf8_word (const unsigned short *input,
+				     unsigned char *output,
+				     int max_output_length);
+
+#ifdef __cplusplus
+	   }
+#endif
+
+#endif
Index: /trunk/mgpp/lib/win32in.c
===================================================================
--- /trunk/mgpp/lib/win32in.c	(revision 3365)
+++ /trunk/mgpp/lib/win32in.c	(revision 3365)
@@ -0,0 +1,85 @@
+/* This module provides an equivalent of <netinet/in.h> on  */
+/* unix systems. */
+#include "win32in.h"
+
+unsigned long htonl(unsigned long x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 24) & 0x000000FF) | ((x >> 8) & 0x0000FF00) |
+		((x << 8) & 0x00FF0000) | ((x << 24) & 0xFF000000);
+#endif
+
+	return x;
+}
+
+mg_ullong ntohll(mg_ullong x)
+{
+#if defined (LITTLE_ENDIAN) || defined(_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = 
+	  ((x >> 56) & 0x00000000000000FF) | ((x >> 40) & 0x000000000000FF00) |
+	  ((x >> 24) & 0x0000000000FF0000) | ((x >> 8)  & 0x00000000FF000000) |
+	  ((x << 8)  & 0x000000FF00000000) | ((x << 24) & 0x0000FF0000000000) |
+		((x << 40) & 0x00FF000000000000) | ((x << 56) & 0xFF00000000000000);
+#endif
+  return x;
+}
+
+unsigned long ntohl(unsigned long x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 24) & 0x000000FF) | ((x >> 8) & 0x0000FF00) |
+		((x << 8) & 0x00FF0000) | ((x << 24) & 0xFF000000);
+#endif
+	return x;
+}
+
+unsigned short htons(unsigned short x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 8) & 0x00FF) | ((x << 8) & 0xFF00);
+#endif
+	return x;
+}
+
+unsigned short ntohs(unsigned short x) {
+#if defined (LITTLE_ENDIAN) || defined (_LITTLE_ENDIAN) || defined(__LITTLE_ENDIAN)
+	x = ((x >> 8) & 0x00FF) | ((x << 8) & 0xFF00);
+#endif
+	return x;
+}
+
+/* just to test stuff
+#include <stdio.h>
+
+main () {
+	long x;
+	short sx;
+
+	x = 0x000000FF;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n",(int)x);
+
+	x = 0x0000FF00;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n",(int)x);
+
+	x = 0x00FF0000;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n",(int)x);
+
+	x = 0xFF000000;
+	printf ("before %i, ",(int)x);
+	x = htonl (x);
+	printf ("after %i\n\n",(int)x);
+
+	sx = 0x00FF;
+	printf ("before %i, ",(int)sx);
+	sx = htons (sx);
+	printf ("after %i\n",(int)sx);
+
+	sx = 0xFF00;
+	printf ("before %i, ",(int)sx);
+	sx = htons (sx);
+	printf ("after %i\n",(int)sx);
+}
+*/
Index: /trunk/mgpp/lib/win32in.h
===================================================================
--- /trunk/mgpp/lib/win32in.h	(revision 3365)
+++ /trunk/mgpp/lib/win32in.h	(revision 3365)
@@ -0,0 +1,20 @@
+/* This module provides an equivalent of <netinet/in.h> on  */
+/* unix systems. */
+#ifdef __WIN32__
+#define _LITTLE_ENDIAN
+#endif
+
+#include "longlong.h"
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+unsigned long htonl(unsigned long x);
+unsigned long ntohl(unsigned long x);
+mg_ullong ntohll(mg_ullong x);
+unsigned short htons(unsigned short x);
+unsigned short ntohs(unsigned short x);
+
+#ifdef __cplusplus
+}
+#endif
Index: /trunk/mgpp/lib/xmalloc.c
===================================================================
--- /trunk/mgpp/lib/xmalloc.c	(revision 3365)
+++ /trunk/mgpp/lib/xmalloc.c	(revision 3365)
@@ -0,0 +1,111 @@
+/* xmalloc.c -- malloc with out of memory checking
+   Copyright (C) 1990, 91, 92, 93, 94 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.  */
+
+#ifdef HAVE_CONFIG_H
+# ifdef __WIN32__  /* [RPAP - Feb 97: WIN32 Port] */
+#  include <win32cfg.h>
+# else
+#  include <sysfuncs.h>
+# endif
+#endif
+
+#if __STDC__
+#define VOID void
+#else
+#define VOID char
+#endif
+
+#include <sys/types.h>
+
+#if STDC_HEADERS
+#include <stdlib.h>
+#else
+VOID *malloc ();
+VOID *realloc ();
+void free ();
+#endif
+
+/* This is for other GNU distributions with internationalized messages.
+   The GNU C Library itself does not yet support such messages.  */
+#if HAVE_LIBINTL_H
+# include <libintl.h>
+#else
+# define gettext(msgid) (msgid)
+#endif
+
+#ifndef EXIT_FAILURE
+#define EXIT_FAILURE 1
+#endif
+
+/* Exit value when the requested amount of memory is not available.
+   The caller may set it to some other value.  */
+int xmalloc_exit_failure = EXIT_FAILURE;
+
+#if defined(__WIN32__)
+void error (int, int, const char *, ...);
+#else
+#if __STDC__ && (HAVE_VPRINTF || HAVE_DOPRNT)
+void error (int, int, const char *, ...);
+#else
+void error (int, int, char *, char *, char *, char *, char *, char *, char *, char *, char *);
+#endif
+#endif
+
+static VOID *
+fixup_null_alloc (n)
+     size_t n;
+{
+  VOID *p;
+
+  p = 0;
+  if (n == 0)
+    p = malloc ((size_t) 1);
+  if (p == 0)
+    error (xmalloc_exit_failure, 0, gettext ("Memory exhausted"));
+  return p;
+}
+
+/* Allocate N bytes of memory dynamically, with error checking.  */
+
+VOID *
+xmalloc (n)
+     size_t n;
+{
+  VOID *p;
+
+  p = malloc (n);
+  if (p == 0)
+    p = fixup_null_alloc (n);
+  return p;
+}
+
+/* Change the size of an allocated block of memory P to N bytes,
+   with error checking.
+   If P is NULL, run xmalloc.  */
+
+VOID *
+xrealloc (p, n)
+     VOID *p;
+     size_t n;
+{
+  if (p == 0)
+    return xmalloc (n);
+  p = realloc (p, n);
+  if (p == 0)
+    p = fixup_null_alloc (n);
+  return p;
+}
Index: /trunk/mgpp/mkinstalldirs
===================================================================
--- /trunk/mgpp/mkinstalldirs	(revision 3365)
+++ /trunk/mgpp/mkinstalldirs	(revision 3365)
@@ -0,0 +1,32 @@
+#! /bin/sh
+# mkinstalldirs --- make directory hierarchy
+# Author: Noah Friedman <friedman@prep.ai.mit.edu>
+# Created: 1993-05-16
+# Last modified: 1994-03-25
+# Public domain
+
+errstatus=0
+
+for file in ${1+"$@"} ; do 
+   set fnord `echo ":$file" | sed -ne 's/^:\//#/;s/^://;s/\// /g;s/^#/\//;p'`
+   shift
+
+   pathcomp=
+   for d in ${1+"$@"} ; do
+     pathcomp="$pathcomp$d"
+     case "$pathcomp" in
+       -* ) pathcomp=./$pathcomp ;;
+     esac
+
+     if test ! -d "$pathcomp"; then
+        echo "mkdir $pathcomp" 1>&2
+        mkdir "$pathcomp" || errstatus=$?
+     fi
+
+     pathcomp="$pathcomp/"
+   done
+done
+
+exit $errstatus
+
+# mkinstalldirs ends here
Index: /trunk/mgpp/stamp-h.in
===================================================================
--- /trunk/mgpp/stamp-h.in	(revision 3365)
+++ /trunk/mgpp/stamp-h.in	(revision 3365)
@@ -0,0 +1,1 @@
+timestamp
Index: /trunk/mgpp/text/FIvfLevelInfo.cpp
===================================================================
--- /trunk/mgpp/text/FIvfLevelInfo.cpp	(revision 3365)
+++ /trunk/mgpp/text/FIvfLevelInfo.cpp	(revision 3365)
@@ -0,0 +1,129 @@
+/**************************************************************************
+ *
+ * FIvfLevelInfo.cpp -- File functions for document levels
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FIvfLevelInfo.h"
+
+
+IvfLevelInfo::IvfLevelInfo () {
+  Clear();
+}
+
+void IvfLevelInfo::Clear () {
+  levelTag.erase (levelTag.begin(), levelTag.end());
+  numEntries = 0;
+  approxWeightsDiskPtr = 0;
+  exactWeightsDiskPtr = 0;
+  workInLevel = false;
+}
+
+bool IvfLevelInfo::Read (FILE *f) {
+  workInLevel = false;
+  return (ReadUCArray (f, levelTag) &&
+	  ReadUL (f, numEntries) &&
+	  ReadUL (f, approxWeightsDiskPtr) &&
+	  ReadUL (f, exactWeightsDiskPtr));
+}
+
+bool IvfLevelInfo::Write (FILE *f) const {
+  return (WriteUCArray (f, levelTag) &&
+	  WriteUL (f, numEntries) &&
+	  WriteUL (f, approxWeightsDiskPtr) &&
+	  WriteUL (f, exactWeightsDiskPtr));
+}
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const IvfLevelInfo &l) {
+  s << "  Tag: \"" << l.levelTag << "\"\n"
+    << "  numEntries: " << l.numEntries << "\n"
+    << "  approxWeightsDiskPtr: " << l.approxWeightsDiskPtr << "\n"
+    << "  exactWeightsDiskPtr: " << l.exactWeightsDiskPtr << "\n"
+    << "  workInLevel: " << ((l.workInLevel) ? "true" : "false") << "\n\n";
+  
+  return s;
+}
+
+
+
+FIvfLevel::FIvfLevel () {
+  Clear ();
+}
+
+void FIvfLevel::Clear () {
+  docTag.erase (docTag.begin(), docTag.end());
+  indexLevel.erase (indexLevel.begin(), indexLevel.end());
+  levelInfo.erase (levelInfo.begin(), levelInfo.end());
+}
+
+bool FIvfLevel::Read (FILE *f) {
+  if (!ReadUCArray (f, docTag) ||
+      !ReadUCArray (f, indexLevel))
+    return false;
+  
+  // read in the array size
+  unsigned long arrSize = 0;
+  if (!ReadVarLenUL (f, arrSize)) return false;
+  
+  IvfLevelInfo thisLevel;
+  while (arrSize > 0) {
+    // read and insert the next level information
+    if (!thisLevel.Read (f)) return false;
+    levelInfo[thisLevel.levelTag] = thisLevel;
+    
+    arrSize--;
+  }
+
+  return true;
+}
+
+bool FIvfLevel::Write (FILE *f) const {
+  if (!WriteUCArray (f, docTag) ||
+      !WriteUCArray (f, indexLevel))
+    return false;
+  
+  // write out the array size
+  if (!WriteVarLenUL (f, levelInfo.size())) return false;
+
+  // write out each level info
+  IvfLevelInfoMap::const_iterator here = levelInfo.begin();
+  IvfLevelInfoMap::const_iterator end = levelInfo.end();
+  while (here != end) {
+    if (!(*here).second.Write (f)) return false;
+    
+    here++;
+  }
+
+  return true;
+}
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const FIvfLevel &l) {
+  s << "docTag: \"" << l.docTag << "\"\n"
+    << "indexLevel: \"" << l.indexLevel << "\"\n";
+  
+  IvfLevelInfoMap::const_iterator here = l.levelInfo.begin ();
+  IvfLevelInfoMap::const_iterator end = l.levelInfo.end ();
+  while (here != end) {
+    s << (*here).second;
+    here++;
+  }
+  
+  return s;
+}
Index: /trunk/mgpp/text/FIvfLevelInfo.h
===================================================================
--- /trunk/mgpp/text/FIvfLevelInfo.h	(revision 3365)
+++ /trunk/mgpp/text/FIvfLevelInfo.h	(revision 3365)
@@ -0,0 +1,77 @@
+/**************************************************************************
+ *
+ * FIvfLevelInfo.h -- File functions for document levels
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FIVFLEVELINFO_H
+#define FIVFLEVELINFO_H
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+#include <stdio.h>
+#include "UCArray.h"
+
+
+class IvfLevelInfo {
+public:
+  UCArray levelTag;
+  unsigned long numEntries; // N for level
+  unsigned long approxWeightsDiskPtr;
+  unsigned long exactWeightsDiskPtr;
+  bool workInLevel;  // this is just a working variable,
+                     // it is not saved or read from the file
+  
+  IvfLevelInfo ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const IvfLevelInfo &l);
+
+
+typedef map<UCArray, IvfLevelInfo, LTUCArray> IvfLevelInfoMap;
+
+
+class FIvfLevel {
+public:
+  UCArray docTag;
+  UCArray indexLevel;    // empty for word level index
+  IvfLevelInfoMap levelInfo;
+  
+  FIvfLevel ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const FIvfLevel &l);
+
+
+#endif
Index: /trunk/mgpp/text/FText.cpp
===================================================================
--- /trunk/mgpp/text/FText.cpp	(revision 3365)
+++ /trunk/mgpp/text/FText.cpp	(revision 3365)
@@ -0,0 +1,249 @@
+/**************************************************************************
+ *
+ * FText.cpp -- File structures for text compression
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FText.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+
+BitAddr::BitAddr () {
+  Clear ();  
+}
+
+void BitAddr::Clear () {
+  byte = 0;
+  bit = 0;
+}
+
+bool BitAddr::Read (FILE *f) {
+  return (ReadUL (f, byte) && ReadUC (f, bit));
+}
+
+bool BitAddr::Write (FILE *f) const {
+  return (WriteUL (f, byte) && WriteUC (f, bit));
+}
+
+#define BITADDRLEN (sizeof(unsigned long) + sizeof(unsigned char))
+
+  
+TextLevelInfo::TextLevelInfo () {
+  Clear();
+}
+
+void TextLevelInfo::Clear () {
+  levelTag.erase (levelTag.begin(), levelTag.end());
+  textIdxPtr = 0;
+  numEntries = 0;
+}
+
+bool TextLevelInfo::Read (FILE *f) {
+  return (ReadUCArray (f, levelTag) &&
+	  ReadUL (f, textIdxPtr) &&
+	  ReadUL (f, numEntries));
+}
+
+bool TextLevelInfo::Write (FILE *f) const {
+  return (WriteUCArray (f, levelTag) &&
+	  WriteUL (f, textIdxPtr) &&
+	  WriteUL (f, numEntries));
+}
+
+ostream &operator<<(ostream &s, const TextLevelInfo &l) {
+  s << "  Tag: \"" << l.levelTag << "\"\n"
+    << "  textIdxPtr: " << l.textIdxPtr << "\n"
+    << "  numEntries: " << l.numEntries << "\n\n";
+  
+  return s;
+}
+
+
+FTextLevel::FTextLevel () {
+  Clear();
+}
+
+void FTextLevel::Clear () {
+  levelInfo.erase (levelInfo.begin(), levelInfo.end());
+}
+
+bool FTextLevel::Read (FILE *f) {
+  // read in the array size
+  unsigned long arrSize = 0;
+  if (!ReadVarLenUL (f, arrSize)) return false;
+
+  TextLevelInfo thisLevel;
+  while (arrSize > 0) {
+    // read and insert the next level information
+    if (!thisLevel.Read (f)) return false;
+    levelInfo[thisLevel.levelTag] = thisLevel;
+    
+    arrSize--;
+  }
+
+  return true;
+}
+
+bool FTextLevel::Write (FILE *f) const {
+  // write out the array size
+  if (!WriteVarLenUL (f, levelInfo.size())) return false;
+
+  // write out each level info
+  TextLevelInfoMap::const_iterator here = levelInfo.begin();
+  TextLevelInfoMap::const_iterator end = levelInfo.end();
+  while (here != end) {
+    if (!(*here).second.Write (f)) return false;
+    
+    here++;
+  }
+
+  return true;
+}
+
+ostream &operator<<(ostream &s, const FTextLevel &l) {
+  TextLevelInfoMap::const_iterator here = l.levelInfo.begin ();
+  TextLevelInfoMap::const_iterator end = l.levelInfo.end ();
+  while (here != end) {
+    s << (*here).second;
+    here++;
+  }
+  
+  return s;
+}
+
+
+TextIdx::TextIdx () {
+  Clear ();
+}
+
+void TextIdx::Clear () {
+  start.Clear();
+  end.Clear();
+  which = 0;
+}
+
+#define TEXTIDXLEN (BITADDRLEN*2 + sizeof(char))
+
+bool TextIdx::Read (FILE *f, const TextLevelInfo &levelInfo,
+		    unsigned long docNum) {
+  if (!SeekTextIdx (f, levelInfo, docNum)) return false;
+  return Read (f);
+}
+
+bool TextIdx::Read (FILE *f) {
+  return (start.Read (f) && end.Read (f) && ReadUC (f, which));
+}
+
+bool TextIdx::Write (FILE *f) const {
+  return (start.Write (f) && end.Write (f) && WriteUC (f, which));
+}
+
+ostream &operator<<(ostream &s, const TextIdx &t) {
+  s << "start byte: " << t.start.byte << "\n";
+  s << "start bit:  " << (int)t.start.bit << "\n";
+  s << "end byte:   " << t.end.byte << "\n";
+  s << "end bit:    " << (int)t.end.bit << "\n";
+  s << "which:      " << (int)t.which << "\n\n";
+  
+  return s;
+}
+
+
+bool SeekTextIdx (FILE *f, const TextLevelInfo &levelInfo,
+		  unsigned long docNum) {
+  if (docNum == 0 || docNum > levelInfo.numEntries) return false;
+  
+  unsigned long seekPos = levelInfo.textIdxPtr + (docNum-1) * TEXTIDXLEN;
+  if (fseek (f, seekPos, SEEK_SET) != 0) return false;
+  
+  return true;
+}
+
+
+bool ReadTextIdxArray (FILE *f, TextIdxArray &a, unsigned long arrSize) {
+  // clear the array
+  a.erase (a.begin(), a.end());
+
+  // read in each element
+  TextIdx idx;
+  while (arrSize > 0) {
+    if (!idx.Read(f)) return false;
+    a.push_back (idx);
+    arrSize --;
+  }
+
+  return true;
+}
+
+bool WriteTextIdxArray (FILE *f, const TextIdxArray &a) {
+  TextIdxArray::const_iterator here = a.begin();
+  TextIdxArray::const_iterator end = a.end();
+  while (here != end) {
+    if (!(*here).Write(f)) return false;
+    here++;
+  }
+  
+  return true;
+}
+
+
+CompressTextInfo::CompressTextInfo () {
+  Clear();
+}
+
+void CompressTextInfo::Clear () {
+  ResetStart();
+  docPtrs.erase (docPtrs.begin(), docPtrs.end());
+}
+
+void CompressTextInfo::ResetStart () {
+  inDoc = false;
+  start.Clear();
+  which = 0;
+}
+  
+void CompressTextInfo::SetStart (unsigned long startPos,
+				 unsigned char startBit,
+				 unsigned char startWhich) {
+  // place an imaginary end tag if needed
+  // note: the document tag always needs a closing tag
+  if (inDoc) SetEnd (startPos, startBit);
+
+  // remember start of level
+  inDoc = true;
+  start.byte = startPos;
+  start.bit = startBit;
+  which = startWhich;
+}
+
+void CompressTextInfo::SetEnd (unsigned long endPos,
+			       unsigned char endBit) {
+  if (inDoc) {
+    // add completed entry to list of ptrs
+    TextIdx textIdx;
+    textIdx.start = start;
+    textIdx.end.byte = endPos;
+    textIdx.end.bit = endBit;
+    textIdx.which = which;
+    docPtrs.push_back (textIdx);
+    
+    // reset this entry
+    ResetStart();
+  }
+}
+
Index: /trunk/mgpp/text/FText.h
===================================================================
--- /trunk/mgpp/text/FText.h	(revision 3365)
+++ /trunk/mgpp/text/FText.h	(revision 3365)
@@ -0,0 +1,150 @@
+/**************************************************************************
+ *
+ * FText.h -- File structures for text compression
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FTEXT_H
+#define FTEXT_H
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+#include <stdio.h>
+#include "UCArray.h"
+
+
+// holds a bit address
+class BitAddr {
+public:
+  unsigned long byte;
+  unsigned char bit;
+
+  BitAddr ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+class TextLevelInfo {
+public:
+  UCArray levelTag;
+  unsigned long textIdxPtr;
+  unsigned long numEntries;
+
+  TextLevelInfo ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const TextLevelInfo &l);
+
+
+typedef map<UCArray, TextLevelInfo, LTUCArray> TextLevelInfoMap;
+
+
+class FTextLevel {
+public:
+  TextLevelInfoMap levelInfo;
+  
+  FTextLevel ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const FTextLevel &l);
+
+
+// the text.idx file points into the compressed text
+class TextIdx {
+public:
+  BitAddr start;       // the first bit of the compressed text
+  BitAddr end;         // the first bit past the end of the text
+  unsigned char which; // 0 = non-word, 1 = word
+  
+  TextIdx ();
+  void Clear ();
+
+  // does a seek and reads the appropriate record
+  bool Read (FILE *f, const TextLevelInfo &levelInfo,
+	     unsigned long docNum);
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// stream output for debugging purposes
+ostream &operator<<(ostream &s, const TextIdx &t);
+
+
+// note: document numbers start at 1
+
+bool SeekTextIdx (FILE *f, const TextLevelInfo &levelInfo,
+		  unsigned long docNum);
+
+
+// used to store the pointers while compressing the text
+typedef vector<TextIdx> TextIdxArray;
+
+bool ReadTextIdxArray (FILE *f, TextIdxArray &a, unsigned long arrSize);
+bool WriteTextIdxArray (FILE *f, const TextIdxArray &a);
+
+
+// used to store information about a level while compressing the text
+class CompressTextInfo {
+public:
+  bool inDoc;
+  BitAddr start;
+  unsigned char which;
+  TextIdxArray docPtrs;
+
+  CompressTextInfo ();
+  void Clear ();
+
+  // clears inDoc, start, and which (not the docPtrs)
+  void ResetStart ();
+  
+  // set the start of a level (closing off the last opening
+  // tag if needed)
+  void SetStart (unsigned long startPos,
+		 unsigned char startBit,
+		 unsigned char startWhich);
+
+  // if in a document, it will set the end, add the document to the
+  // list of document ptrs, and then call ResetStart
+  void SetEnd (unsigned long endPos,
+	       unsigned char endBit);
+};
+
+typedef map<UCArray, CompressTextInfo, LTUCArray> CompressTextInfoMap;
+
+
+#endif
Index: /trunk/mgpp/text/FileBuf.cpp
===================================================================
--- /trunk/mgpp/text/FileBuf.cpp	(revision 3365)
+++ /trunk/mgpp/text/FileBuf.cpp	(revision 3365)
@@ -0,0 +1,50 @@
+/**************************************************************************
+ *
+ * FileBuf.cpp -- File buffer
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FileBuf.h"
+#include "sysfuncs.h"
+
+#ifdef __WIN32__
+# include <io.h>
+# define read _read
+#endif
+
+bool FileBuf::NextBuf () {
+  if (fd < 0) return false;
+  int numbytes = read (fd, buf, FILEBUFSIZE);
+  if (numbytes < 0) numbytes = 0;
+  
+  here = buf;
+  end = here + numbytes;
+
+  return (numbytes > 0);
+}
+
+FileBuf::FileBuf () {
+  here = end = buf;
+  fd = -1;
+}
+
+void FileBuf::SetFD (int _fd) {
+  here = end = buf;
+  fd = _fd;
+}
+
Index: /trunk/mgpp/text/FileBuf.h
===================================================================
--- /trunk/mgpp/text/FileBuf.h	(revision 3365)
+++ /trunk/mgpp/text/FileBuf.h	(revision 3365)
@@ -0,0 +1,73 @@
+/**************************************************************************
+ *
+ * FileBuf.h -- File buffer
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FILEBUF_H
+#define FILEBUF_H
+
+#define FILEBUFSIZE 65536
+
+// some elderly windows compilers need to include <vector> (or something
+// similar) here so that they know what a "bool" is
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\vector>
+#elif defined(GSDL_USE_STL_H)
+#  include <vector.h>
+#else
+#  include <vector>
+#endif
+
+
+class FileBuf {
+protected:
+  unsigned char buf[FILEBUFSIZE];
+  unsigned char *here;
+  unsigned char *end;
+
+  int fd;
+
+  // gets a new buffer if need be, returns true if
+  // more could be got. if true is returned the caller
+  // can assume that here != end
+  bool NextBuf ();
+  
+public:
+  FileBuf ();
+
+  void SetFD (int _fd);
+
+  inline bool Get (unsigned char &c) {
+    if (here != end || NextBuf()) { c = *here++; return true; }
+    return false;
+  }
+  
+  inline bool Peek (unsigned char &c) {
+    if (here != end || NextBuf()) { c = *here; return true; }
+    return false;
+  }
+
+  inline bool Next () {
+    if (here != end || NextBuf()) { here++; return true; }
+    return false;
+  }
+};
+
+
+#endif
Index: /trunk/mgpp/text/FragLevelConvert.cpp
===================================================================
--- /trunk/mgpp/text/FragLevelConvert.cpp	(revision 3365)
+++ /trunk/mgpp/text/FragLevelConvert.cpp	(revision 3365)
@@ -0,0 +1,105 @@
+/**************************************************************************
+ *
+ * FragLevelConvert.cpp -- converting between fragment and document numbers
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "FragLevelConvert.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+
+
+FragLevelConvert::FragLevelConvert () {
+  numFrags = 0;
+  numLevelDocs = 0;
+}
+
+bool FragLevelConvert::Read (FILE *invfFile,
+			     unsigned long levelPtr,
+			     unsigned long _numFrags,
+			     unsigned long _numLevelDocs) {
+  levelStarts.erase (levelStarts.begin(), levelStarts.end());
+  numFrags = _numFrags;
+  numLevelDocs = _numLevelDocs;
+
+  // seek to the appropriate place in the inverted file
+  fseek (invfFile, levelPtr, SEEK_SET);
+
+  stdio_bitio_buffer buffer(invfFile);
+  
+  unsigned long pTag = numLevelDocs*2;
+  unsigned long B = BIO_Bblock_Init (numFrags+pTag, pTag);
+  unsigned long fragNum = 0;
+  unsigned long i;
+  for (i=0; i<numLevelDocs; i++) {
+    unsigned long delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+
+    levelStarts.push_back (fragNum);
+
+    // ignore end, assume every fragment is in a level
+    delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+  }
+
+  // fragNum would be the starting fragment of the next document
+  // if there was one
+  levelStarts.push_back (fragNum);
+  
+  if (fragNum != _numFrags) {
+    cerr << "last frag num:"<<fragNum<<" != num frags:"<<_numFrags<<endl;
+  } 
+  buffer.done();
+
+  return true;
+}
+
+bool FragLevelConvert::FragToLevel (unsigned long fragNum,
+				    unsigned long &levelDocNum) const {
+  // do binary search for something containing this fragment number
+  unsigned long low = 1, high = numLevelDocs;
+  unsigned long mid;
+
+  while ((mid = (low+high)/2) >= 1 && low <= high) {
+    if (fragNum > levelStarts[mid]) low = mid+1;
+    else if (fragNum <= levelStarts[mid-1]) high = mid-1;
+    else {
+      levelDocNum = mid;
+      return true;
+    }
+  }
+  return false;
+}
+
+
+bool FragLevelConvert::LevelToLevel (FragLevelConvert sectionLevelConvert,
+		 unsigned long levelNum, unsigned long &DocNum) {
+  
+  if (levelNum==1) {
+    DocNum=1;
+    return true;
+  }
+  unsigned long levelfragnum = sectionLevelConvert.levelStarts[levelNum-1]+1;
+
+  if (FragToLevel(levelfragnum, DocNum)) {
+    return true;
+  } 
+  return false;
+  
+
+}
Index: /trunk/mgpp/text/FragLevelConvert.h
===================================================================
--- /trunk/mgpp/text/FragLevelConvert.h	(revision 3365)
+++ /trunk/mgpp/text/FragLevelConvert.h	(revision 3365)
@@ -0,0 +1,70 @@
+/**************************************************************************
+ *
+ * FragLevelConvert.h -- converting between fragment and document numbers
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef FRAGLEVELCONVERT_H
+#define FRAGLEVELCONVERT_H
+
+#include <stdio.h>
+#include "UCArray.h"
+
+
+/*Notes about levelStarts (kjm18) - this is my interpretation of
+whats going on, feel free to correct this if its wrong.
+
+say you have 5 docs, consisting of fragments:
+1:  1-55
+2: 56-83
+3: 84-106
+4: 107-152
+5: 153-211
+
+then levelStarts would have the values
+0  1  2   3   4   5 
+0  55 83 106 152 211
+
+*/
+class FragLevelConvert {
+protected:
+  // note fragment start is one before before real start
+  vector<unsigned long> levelStarts;
+  unsigned long numFrags;
+  unsigned long numLevelDocs;
+  
+public:
+  FragLevelConvert ();
+
+  bool Read (FILE *invfFile, unsigned long levelPtr,
+	     unsigned long _numFrags, unsigned long _numLevelDocs);
+
+  bool FragToLevel (unsigned long fragNum, unsigned long &levelDocNum) const;
+
+  // this function converts from one level number to another
+  // the FragLevelConvert thats calling the method should be loaded with
+  // the level that you want to convert to, while the sectionLevelConvert
+  // should be loaded with the level converting from.
+  // level Num has the old level, DocNum will be returned with the new
+  // coresponding level
+  bool LevelToLevel (FragLevelConvert sectionLevelConvert,
+		   unsigned long levelNum, unsigned long &DocNum);
+};
+
+
+#endif
Index: /trunk/mgpp/text/GSDLQueryLex.cpp
===================================================================
--- /trunk/mgpp/text/GSDLQueryLex.cpp	(revision 3365)
+++ /trunk/mgpp/text/GSDLQueryLex.cpp	(revision 3365)
@@ -0,0 +1,251 @@
+/**************************************************************************
+ *
+ * GSDLQueryLex.cpp -- Lexical analyser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "GSDLQueryLex.h"
+#include "unitool.h"
+#include "words.h"
+
+inline void AddNChar (UCArray::const_iterator &here,
+		      UCArray &text,
+		      int len) {
+  while (len > 0) {
+    text.push_back (*here++);
+    len--;
+  }
+}
+
+static bool ParseInteger (UCArray::const_iterator &here,
+			  UCArray::const_iterator end,
+			  LexEl &el) {
+  el.Clear();
+
+  // this version of end is used in unitool
+  //  UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (&*here, endMinus1, &c);
+
+  // check for positive or negative
+  bool neg = false;
+  if (c == '+') {
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  } else if (c == '-') {
+    neg = true;
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+
+  // read in number part
+  int numeric=0;
+  el.num = 0;
+  el.lexType = IntegerE;
+  /* stop integers at 4 digits */
+  while (c >= '0' && c <= '9'&& ++numeric<=MAXNUMERIC) {
+    el.num = el.num*10 + c - '0';
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+
+  if (neg) el.num *= -1;
+
+  return (!el.text.empty());
+}
+
+static bool ParsePotentialInteger(UCArray::const_iterator &here,
+				  UCArray::const_iterator end,
+				  LexEl &el) {
+  el.Clear();
+
+  // this version of end is used in unitool
+  //UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+
+  int charLen=0;
+  int length=0;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (&*here, endMinus1, &c);
+
+  // read in number part
+  int numeric=0;
+  el.num = 0;
+  el.lexType = IntegerE;
+  /* stop integers at 4 digits */
+  while (c >= '0' && c <= '9'&& ++numeric<=MAXNUMERIC) {
+    el.num = el.num*10 + c - '0';
+    AddNChar (here, el.text, charLen);
+    length += charLen;
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+  // check the next character -if it is a letter, then have a term, not an integer
+  if (!is_unicode_letter(c)) {
+    // this was just an integer
+    return (!el.text.empty());
+  }
+  // else its a term
+  el.lexType = TermE;
+  el.num = 0;
+  /* this bit taken from ParseIndexWord in words.h*/
+  while (length+charLen<=MAXSTEMLEN && charLen > 0 &&
+	 (is_unicode_letter(c) || (is_unicode_digit(c) &&
+				   ++numeric <= MAXNUMERIC))) {
+    AddNChar (here, el.text, charLen);
+    length += charLen;
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+
+  return (!el.text.empty());
+}
+static bool ParseTerm (UCArray::const_iterator &here,
+		       UCArray::const_iterator end,
+		       UCArray &text) {
+  //UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+  const unsigned char* new_here = ParseIndexWord (&*here, endMinus1, text);
+  here += (new_here - &*here); // advance iterator by number of chars advanced
+  return !text.empty();
+}
+
+
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el) {
+  el.Clear();
+
+  // strange things can happen if here == end == 0
+  if (here == end) return false;
+  
+  // this version of end is used in unitool
+  //UCArray::const_iterator endMinus1 = end-1;
+  const unsigned char* endMinus1 = &(*end)-1;
+
+  // ignore all white space
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (&*here, endMinus1, &c);
+  while (here != end && is_unicode_space (c)) {
+    here += charLen;
+    charLen = parse_utf8_char (&*here, endMinus1, &c);
+  }
+  if (here == end) return false;
+
+  if (c == '(') {
+    el.lexType = OpenBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ')') {
+    el.lexType = CloseBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+
+  } else if (c =='[') {
+    el.lexType = OpenSquareBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+  
+  } else if (c ==']') {
+    el.lexType = CloseSquareBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+  
+  } else if (c == '\"') {
+    el.lexType = QuoteE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '/') {
+    el.lexType = TermWeightE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '#') {
+    el.lexType = StemMethodE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '^') {
+    el.lexType = RangeE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '@') {
+    el.lexType = AtE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ':') {
+    el.lexType = TagE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c=='&') {
+    el.lexType = AndOpE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '|') {
+    el.lexType = OrOpE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '!') {
+    el.lexType = NotOpE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '+' || c == '-' ) {
+    return  ParseInteger (here, end, el);
+  }
+
+  else if (c >= '0' && c <= '9') {
+    return ParsePotentialInteger (here, end, el);
+  }
+
+  // assume it is a term of some sort
+  if (!ParseTerm (here, end, el.text)) return false;
+
+  UCArray AND; SetCStr (AND, "AND");
+  if (el.text == AND) {
+    el.lexType = AndOpE;
+    return true;
+  }
+  UCArray OR; SetCStr (OR, "OR");
+  if (el.text == OR) {
+    el.lexType = OrOpE;
+    return true;
+  }
+  UCArray NOT; SetCStr (NOT, "NOT");
+  if (el.text == NOT) {
+    el.lexType = NotOpE;
+    return true;
+  }
+  UCArray NEAR; SetCStr (NEAR, "NEAR");
+  if (PrefixLen(el.text, NEAR)==4) {
+    el.lexType = NearOpE;
+    return true;
+  }
+  el.lexType = TermE;
+  return true;
+}
+
Index: /trunk/mgpp/text/GSDLQueryLex.h
===================================================================
--- /trunk/mgpp/text/GSDLQueryLex.h	(revision 3365)
+++ /trunk/mgpp/text/GSDLQueryLex.h	(revision 3365)
@@ -0,0 +1,63 @@
+/**************************************************************************
+ *
+ * GSDLQueryLex.h -- Lexical analyser for a simple query language, for GSDL
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef GSDLQUERYLEX_H
+#define GSDLQUERYLEX_H
+
+#include "UCArray.h"
+
+
+// note: white space is ignored
+enum LexType {NoTypeE, OpenBracketE, CloseBracketE, TermE, AndOpE, OrOpE,
+	      NotOpE, NearOpE, QuoteE, IntegerE, TermWeightE, StemMethodE,
+	      RangeE, AtE, TagE, OpenSquareBracketE, CloseSquareBracketE};
+
+struct LexEl {
+  LexType lexType;
+  UCArray text;
+  int num; // for IntegerE
+
+  void Clear () {
+    lexType = NoTypeE;
+    UCArrayClear (text);
+    num = 0;
+  }
+  LexEl () { Clear (); }
+};
+
+
+// returns true if another lexical element could be parsed.
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el);
+
+
+#endif
+
+
+
+
+
+
+
+
+
+
Index: /trunk/mgpp/text/GSDLQueryParser.cpp
===================================================================
--- /trunk/mgpp/text/GSDLQueryParser.cpp	(revision 3365)
+++ /trunk/mgpp/text/GSDLQueryParser.cpp	(revision 3365)
@@ -0,0 +1,405 @@
+/**************************************************************************
+ *
+ * QueryParser.cpp -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "GSDLQueryParser.h"
+#include "GSDLQueryLex.h"
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end, 
+				   int defaultBoolCOmbine,
+				   int defaultStemMethod);
+
+static QueryNode *AndAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  AndQueryNode *andNode = new AndQueryNode;
+  andNode->leftNode = t1;
+  andNode->rightNode = t2;
+  return andNode;
+}
+
+static QueryNode *OrAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  OrQueryNode *orNode = new OrQueryNode;
+  orNode->leftNode = t1;
+  orNode->rightNode = t2;
+  return orNode;
+}
+
+static QueryNode *NotAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  NotQueryNode *notNode = new NotQueryNode;
+  notNode->queryNode = t1;
+  notNode->notNode = t2;
+  return notNode;
+}
+
+// expects the opening bracket to have already been parsed
+// and discarded
+static QueryNode *ParseBracketExpression (UCArray::const_iterator &here,
+					  UCArray::const_iterator end, 
+					  int defaultBoolCombine,
+					  int defaultStemMethod) {
+  // get everything in the expression
+  QueryNode *curTree = ParseExpression (here, end, defaultBoolCombine,
+					defaultStemMethod);
+  
+  // gobble up tokens until a closing bracket is found
+  // or the end of the string
+  LexEl el;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == CloseBracketE) break;
+  }
+
+  return curTree;
+}
+
+static int ParseInt (UCArray::const_iterator &here,
+		     UCArray::const_iterator end) {
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  if (ParseLexEl (here, end, el) && el.lexType == IntegerE)
+    return el.num;
+  
+  here = oldHere; // not an integer
+  return 0;
+}
+
+// default is within 20 words
+static void SetRangeValues (TermNode &termNode,
+			    UCArray &nearby) {
+  UCArray NEARBY; SetCStr(NEARBY, "NEAR");
+  if (nearby == NEARBY) { // no modifier
+    termNode.startRange = -21;
+    termNode.endRange = 20;
+  }
+  else { // extract number
+    UCArray::const_iterator here = nearby.begin()+4;
+    UCArray::const_iterator end = nearby.end();
+    int size=0;
+    while (here != end) {
+      size = size*10 + (*here-'0');
+      here++;
+    }
+    termNode.startRange = -1 * (size+1);
+    termNode.endRange = size;
+  }
+}
+
+static unsigned long GetStemMethod(LexEl &el, int defaultStemMethod) {
+  // here expect el to contain some of c,s,i,u
+  // stem method 0 = c,u  00
+  // stem method 1 = i,u  01  - default for DL
+  // stem method 2 = c, s  10
+  // stem method 3 = i,s  11
+
+  unsigned long stem = (unsigned long)defaultStemMethod;
+      
+  UCArray::const_iterator here = el.text.begin();
+  UCArray::const_iterator end = el.text.end();
+
+  unsigned char c1 = *here; 
+  if (!(c1 == 'c'| c1 == 'i' | c1 == 'u' | c1 == 's')) 
+    return 4; // incorrect format
+  
+  here++;
+  unsigned char c2 = 'a';
+  if (here !=end) {
+    c2 = *here;
+    if (!(c2 == 'c'| c2 == 'i' | c2 == 'u' | c2 == 's')) 
+      return 4; // incorrect format
+  }
+  
+  if (c1 == 'i'|| c2=='i') stem |= 1; // set bit 0 to 1
+  if (c1 == 'c' || c2 == 'c') stem &=0xe; //set bit 0 to 0
+  if (c1 == 's'|| c2 == 's') stem |= 2; // set bit 1 to 1
+  if (c1 == 'u' || c2 =='u') stem &=0xd; // set bit 1 to 0
+  return stem;
+}
+  
+
+static void ParseTermModifiers (UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				TermNode &termNode,
+				int defaultStemMethod) {
+ 
+  termNode.stemMethod = defaultStemMethod;
+
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermWeightE) {
+      termNode.termWeight = ParseInt (here, end);
+      
+    } else if (el.lexType == StemMethodE) {
+      oldHere = here;
+      LexEl stem;
+      if (ParseLexEl (here, end, stem) && stem.lexType == TermE) {
+	termNode.stemMethod = GetStemMethod(stem, defaultStemMethod);
+	if (termNode.stemMethod == 4) { // error so backtrack
+	  here = oldHere;
+	  termNode.stemMethod = (unsigned long)defaultStemMethod;
+	} 
+      }else here = oldHere; //ignore - wrong syntax
+
+    } else if (el.lexType == RangeE) {
+      termNode.startRange = ParseInt (here, end);
+      termNode.endRange = ParseInt (here, end);
+      
+    } else if (el.lexType == AtE) {
+      termNode.startRange = termNode.endRange = ParseInt (here, end);
+      
+    } else {
+      // no term modifiers
+      here = oldHere;
+      break;
+    }
+    
+    oldHere = here;
+  }  
+}
+
+static void ParseProxModifiers (UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				ProxMatchQueryNode *proxNode) {
+  // so far only have one - the tag stuff
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TagE) {
+      oldHere = here; // don't backtrack past here
+      if (ParseLexEl (here, end, el) && el.lexType == TermE) {
+	proxNode->tagNodePtr = new TagNode;
+	proxNode->tagNodePtr->tagName = el.text;
+	
+      }
+      else { // error in tag
+	here = oldHere;
+      }
+    } // TagE
+    // add in other cases here
+    else {
+      // no modifiers
+      here = oldHere;
+      break;
+    }
+    oldHere = here;
+  }//while
+
+
+}
+
+// expects starting brackets to have been parsed
+static void ParseSquareBrackets(UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				ProxMatchQueryNode *proxNode,
+				int defaultStemMethod) {
+
+  LexEl el;
+  bool phrase=false;
+  bool first=true;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermE || el.lexType == IntegerE) {
+      TermNode termNode;
+      termNode.term = el.text;
+      ParseTermModifiers (here, end, termNode, defaultStemMethod);
+      if (phrase) {
+        if (first) first=false;
+        else {
+          termNode.startRange = -2;
+          termNode.endRange = -1;
+        }
+      }  
+      proxNode->terms.push_back(termNode);
+    }
+    else if (el.lexType == CloseSquareBracketE) {
+      break;
+    }
+    else if (el.lexType == AndOpE) {
+      // ignore, the words are AND'ed anyway
+      cerr << "and inside []\n";
+    }
+    else if (el.lexType == OrOpE) {
+      cerr << "or inside []\n";
+    }
+    else if (el.lexType == QuoteE) {
+      // phrase inside square brackets
+      if (phrase) phrase=false;
+      else phrase=true;
+    }
+    else {
+      //error
+      // just ignore for now
+      cerr <<"bad syntax inside []\n";
+    }
+  } // while
+
+}
+// expects the starting quote to have been parsed
+// and discarded
+// now phrases use the case and stem preference options
+// ie can search for a phrase ignoring case
+static void ParsePhrase (UCArray::const_iterator &here,
+			 UCArray::const_iterator end,
+			 ProxMatchQueryNode &proxNode, 
+			 int defaultStemMethod) {
+  LexEl el;
+  bool first = true;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermE || el.lexType == IntegerE) {
+      TermNode termNode;
+      termNode.term = el.text;
+      //termNode.stemMethod = defaultStemMethod;
+      ParseTermModifiers (here, end, termNode, defaultStemMethod);
+      if (first) {
+	first = false;
+      }
+      else {
+	termNode.startRange = -2;
+	termNode.endRange = -1;
+      }
+      proxNode.terms.push_back (termNode);
+      
+    } else if (el.lexType == QuoteE) {
+      break;
+      
+    } else {
+      // error
+      break;
+    }
+  }
+}
+
+static QueryNode *ParseTerm (UCArray::const_iterator &here,
+			     UCArray::const_iterator end,
+			     int defaultBoolCombine,
+			     int defaultStemMethod) {
+  LexEl el;
+
+  UCArray::const_iterator oldHere = here;
+  if (!ParseLexEl (here, end, el)) return NULL;
+
+  if (el.lexType == OpenBracketE)
+    return ParseBracketExpression (here, end, defaultBoolCombine,
+				   defaultStemMethod);
+
+  ProxMatchQueryNode *proxNode = new ProxMatchQueryNode;
+
+  if (el.lexType == TermE || el.lexType == IntegerE) {
+    TermNode termNode;
+    termNode.term = el.text;
+    ParseTermModifiers (here, end, termNode, defaultStemMethod);
+    oldHere = here;  // dont backtrack past here
+    if (ParseLexEl(here, end, el) && el.lexType == NearOpE) {
+      delete proxNode;
+      proxNode = (ProxMatchQueryNode *)ParseTerm(here, end, defaultBoolCombine,
+						 defaultStemMethod);
+      SetRangeValues(termNode, el.text);
+      proxNode->terms.push_back (termNode);
+      return proxNode;
+    }
+    else {
+      here = oldHere; // backtrack
+      proxNode->terms.push_back (termNode);
+      ParseProxModifiers(here, end, proxNode);
+      return proxNode;
+    }
+  } else if (el.lexType == QuoteE) {
+    ParsePhrase (here, end, *proxNode, defaultStemMethod);
+    return proxNode;
+  }
+  else if (el.lexType == OpenSquareBracketE) {
+    ParseSquareBrackets (here, end, proxNode, defaultStemMethod);
+    ParseProxModifiers (here, end, proxNode);
+    return proxNode;
+  }
+
+  // not a term
+  here = oldHere;
+  delete proxNode;
+  return NULL;
+}
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end,
+				   int defaultBoolCombine,
+				   int defaultStemMethod) {
+  LexEl el;
+  QueryNode *curTree = NULL;
+
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == OpenSquareBracketE ||
+	el.lexType == OpenBracketE ||
+	el.lexType == TermE ||
+	el.lexType == QuoteE ||
+	el.lexType == IntegerE ) {
+	//	el.lexType == TagE) { //tag at end of term now
+      // some type of term, back track and parse it
+      here = oldHere;
+      // if default==1, AND, else if==0, OR
+      if (defaultBoolCombine) {
+	curTree = AndAdd (curTree, ParseTerm (here, end, defaultBoolCombine, 
+					      defaultStemMethod));
+      }
+      else {
+	curTree = OrAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					     defaultStemMethod));
+      }
+
+    } else if (el.lexType == AndOpE) {
+      curTree = AndAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					    defaultStemMethod));
+      
+    } else if (el.lexType == OrOpE) {
+      curTree = OrAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					   defaultStemMethod));
+      
+    } else if (el.lexType == NotOpE) {
+      curTree = NotAdd (curTree, ParseTerm (here, end, defaultBoolCombine,
+					    defaultStemMethod));
+      
+    } else if (el.lexType == CloseBracketE) {
+      // parsebracketexpression is waiting for the last bracket, so put it back
+      here = oldHere;
+      break;
+    
+    } else break;
+    
+    oldHere = here;
+  }
+
+  return curTree;
+}
+
+QueryNode *ParseQuery (const UCArray &queryStr, int defaultBoolCombine,
+		       int defaultStemMethod) {
+  UCArray::const_iterator here = queryStr.begin();
+  UCArray::const_iterator end = queryStr.end();
+  return ParseExpression (here, end, defaultBoolCombine, defaultStemMethod);
+}
Index: /trunk/mgpp/text/GSDLQueryParser.h
===================================================================
--- /trunk/mgpp/text/GSDLQueryParser.h	(revision 3365)
+++ /trunk/mgpp/text/GSDLQueryParser.h	(revision 3365)
@@ -0,0 +1,37 @@
+/**************************************************************************
+ *
+ * QueryParser.h -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef GSDLQUERYPARSER_H
+#define GSDLQUERYPARSER_H
+
+#include "UCArray.h"
+#include "MGQuery.h"
+
+// returns NULL if the query could not be parsed
+// defaultBoolCOmbine determines if individual terms are combined using
+// AND or OR in the absence of specified operators.
+// defaultStemMethod used to set stemming and casefolding for terms where
+// its not set explicitly in the query string. THe defaultStemMethod value
+// should be set using the values from the preferences page
+QueryNode *ParseQuery (const UCArray &queryStr, int defaultBoolCombine,
+		       int defaultStemMethod);
+
+#endif
Index: /trunk/mgpp/text/IndexData.cpp
===================================================================
--- /trunk/mgpp/text/IndexData.cpp	(revision 3365)
+++ /trunk/mgpp/text/IndexData.cpp	(revision 3365)
@@ -0,0 +1,207 @@
+/**************************************************************************
+ *
+ * IndexData.cpp -- Information needed for querying
+ * Copyright (C) 1999 Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "IndexData.h"
+#include "string.h"
+#include "mg_files.h"
+
+IndexData::IndexData () {
+  basePath[0] = '\0';
+  filename[0] = '\0';
+
+  dictFile = NULL;
+
+  stem1File = NULL;
+  stem2File = NULL;
+  stem3File = NULL;
+
+  invfFile = NULL;
+
+  approxWeightsFile = NULL;
+  exactWeightsFile = NULL;
+
+  curLevelNum = 0;
+}
+
+IndexData::~IndexData () {
+  UnloadData ();
+}
+
+bool IndexData::LoadData (const char *_basePath, const char *_filename) {
+  if (_filename[0] == '\0') return false;
+  
+  // make sure this data has not already been loaded
+  if (strcmp (_basePath, basePath) == 0 &&
+      strcmp (_filename, filename) == 0) return true;
+  
+  // make sure there is nothing else loaded
+  UnloadData ();
+
+  strcpy (basePath, _basePath);
+  strcpy (filename, _filename);
+
+  set_basepath (basePath);
+
+  // load in the level information
+  FILE *ivfLevelFile;
+  ivfLevelFile = open_file (filename, INVF_LEVEL_SUFFIX, "rb",
+			    MAGIC_INVF_LEVELS, MG_ABORT);
+  levels.Read (ivfLevelFile);
+  fclose (ivfLevelFile);
+
+  // blocked dictionary
+  dictFile = open_file (filename, INVF_DICT_BLOCKED_SUFFIX, "rb",
+			MAGIC_STEM, MG_ABORT);
+  if (!bdh.Read (dictFile)) { UnloadData (); return false; }
+
+  fseek (dictFile, bdh.wblk_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (dictFile, biWords)) { UnloadData (); return false; }
+
+  fseek (dictFile, bdh.tblk_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (dictFile, biTags)) { UnloadData (); return false; }
+
+  // blocked stem index 1
+  stem1File = open_file (filename, INVF_DICT_BLOCKED_1_SUFFIX,
+			 "rb", MAGIC_STEM_1, MG_ABORT);
+  if (!sih1.Read (stem1File)) { UnloadData (); return false; }
+
+  fseek (stem1File, sih1.block_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (stem1File, sii1)) { UnloadData (); return false; }
+  
+  // blocked stem index 2
+  stem2File = open_file (filename, INVF_DICT_BLOCKED_2_SUFFIX,
+			 "rb", MAGIC_STEM_2, MG_ABORT);
+  if (!sih2.Read (stem2File)) { UnloadData (); return false; }
+
+  fseek (stem2File, sih2.block_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (stem2File, sii2)) { UnloadData (); return false; }
+  
+  // blocked stem index 3
+  stem3File = open_file (filename, INVF_DICT_BLOCKED_3_SUFFIX,
+			 "rb", MAGIC_STEM_3, MG_ABORT);
+  if (!sih3.Read (stem3File)) { UnloadData (); return false; }
+
+  fseek (stem3File, sih3.block_idx_start, SEEK_SET);
+  if (!ReadBlockIdx (stem3File, sii3)) { UnloadData (); return false; }
+
+  // inverted file
+  invfFile = open_file (filename, INVF_SUFFIX, "rb",
+			MAGIC_INVF, MG_ABORT);
+  ifh.Read (invfFile);
+
+  // weights
+  approxWeightsFile = open_file (filename, APPROX_WEIGHTS_SUFFIX, "rb",
+				 MAGIC_WGHT_APPROX, MG_ABORT);
+  exactWeightsFile = open_file (filename, WEIGHTS_SUFFIX, "rb",
+				MAGIC_WGHT, MG_ABORT);
+
+  return true;
+}
+
+bool IndexData::UnloadData () {
+  basePath[0] = '\0';
+  filename[0] = '\0';
+
+  if (dictFile != NULL) {
+    fclose (dictFile); dictFile = NULL;
+  }
+
+  if (stem1File != NULL) {
+    fclose (stem1File); stem1File = NULL;
+  }
+  if (stem2File != NULL) {
+    fclose (stem2File); stem2File = NULL;
+  }
+  if (stem3File != NULL) {
+    fclose (stem3File); stem3File = NULL;
+  }
+
+  if (invfFile != NULL) {
+    fclose (invfFile); invfFile = NULL;
+  }
+
+  if (approxWeightsFile != NULL) {
+    fclose (approxWeightsFile); approxWeightsFile = NULL;
+  }
+  if (exactWeightsFile != NULL) {
+    fclose (exactWeightsFile); exactWeightsFile = NULL;
+  }
+
+  UnloadLevel();
+  
+  return true;
+}
+
+
+bool IndexData::LoadLevel (const UCArray &level) {
+  // make sure this level is not already loaded
+  if (level == curLevel) return true;
+
+  // unload any levels currently loaded
+  UnloadLevel();
+
+  // make sure the required files are open
+  if (dictFile == NULL || invfFile == NULL ||
+      approxWeightsFile == NULL || exactWeightsFile == NULL)
+    return false;
+  
+  // read in the information from the dictionary
+  block_dict_el tagEl;
+  unsigned long tagElNum;
+  if (!SearchBlockDictEl (dictFile, biTags, bdh.entries_per_tblk,
+			  bdh.tag_dict_size, level, tagEl, tagElNum))
+    return false;
+
+  // read in the level conversion information
+  if (!levelConverter.Read (invfFile, tagEl.invf_ptr,
+			    bdh.num_frags, tagEl.frag_occur))
+    return false;
+
+  // read in the approximate weights
+  if (!weightData.Read (approxWeightsFile,
+			levels.levelInfo[level].approxWeightsDiskPtr,
+			levels.levelInfo[level].numEntries))
+    return false;
+  
+  // find the level number
+  curLevelNum = 0;
+  IvfLevelInfoMap::const_iterator levelHere, levelEnd;
+  for (levelHere=levels.levelInfo.begin(), levelEnd=levels.levelInfo.end();
+       levelHere!=levelEnd && (*levelHere).first != level; levelHere++)
+    curLevelNum++;
+
+  // make sure we found the level
+  if (levelHere == levelEnd) return false;
+
+  curLevel = level;
+
+  return true;
+}
+
+bool IndexData::UnloadLevel () {
+  curLevel.erase (curLevel.begin(), curLevel.end());
+  curLevelNum = 0;
+  
+  weightData.Free ();
+    
+  return true;
+}
+
Index: /trunk/mgpp/text/IndexData.h
===================================================================
--- /trunk/mgpp/text/IndexData.h	(revision 3365)
+++ /trunk/mgpp/text/IndexData.h	(revision 3365)
@@ -0,0 +1,82 @@
+/**************************************************************************
+ *
+ * IndexData.h -- Information needed for querying
+ * Copyright (C) 1999 Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef INDEXDATA_H
+#define INDEXDATA_H
+
+#include <stdio.h>
+#include "invf.h"
+#include "FIvfLevelInfo.h"
+#include "FragLevelConvert.h"
+#include "Weights.h"
+
+
+class IndexData {
+public:
+  // general information
+  char basePath[512];
+  char filename[512];
+  FIvfLevel levels;
+  
+  // blocked dictionary
+  FILE *dictFile;
+  block_dict_header bdh;
+  block_idx biWords;
+  block_idx biTags;
+
+  // stem indexes
+  FILE *stem1File;
+  FILE *stem2File;
+  FILE *stem3File;
+  stem_idx_header sih1;
+  stem_idx_header sih2;
+  stem_idx_header sih3;
+  block_idx sii1;
+  block_idx sii2;
+  block_idx sii3;
+
+  // inverted file
+  FILE *invfFile;
+  invf_file_header ifh;
+
+  // weights
+  FILE *approxWeightsFile;
+  FILE *exactWeightsFile;
+
+  // level specific information
+  UCArray curLevel; // last level that was loaded
+  unsigned long curLevelNum;
+  FragLevelConvert levelConverter;
+  ApproxWeightsData weightData;
+  
+  IndexData ();
+  ~IndexData ();
+
+  // LoadData will only load the data if it has not been loaded before
+  bool LoadData (const char *_basePath, const char *_filename);
+  bool UnloadData ();
+
+  bool LoadLevel (const UCArray &level);
+  bool UnloadLevel ();
+};
+
+
+#endif
Index: /trunk/mgpp/text/MGQuery.cpp
===================================================================
--- /trunk/mgpp/text/MGQuery.cpp	(revision 3365)
+++ /trunk/mgpp/text/MGQuery.cpp	(revision 3365)
@@ -0,0 +1,760 @@
+/**************************************************************************
+ *
+ * MGQuery.cpp -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "MGQuery.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+#include "Terms.h"
+#include "QueryResultsSort.h"
+#include <assert.h>
+
+
+void PrintIndent (ostream &s, int indent) {
+  while (indent-- > 0) s << " ";
+}
+
+
+void PrintIndentText (ostream &s, char *text, int indent) {
+  PrintIndent (s, indent);
+  s << text;
+}
+
+void PrintNode (ostream &s, QueryNode *node, int indent) {
+  if (node == NULL) {
+    PrintIndentText (s, "NULL\n", indent);
+  } else {
+    node->Print (s, indent+2);
+  }
+}
+
+  
+
+QueryNode::QueryNode () {
+}
+
+QueryNode::~QueryNode () {
+}
+
+void QueryNode::Calculate (IndexData &/*indexData*/,
+			   const QueryInfo &/*queryInfo*/,
+			   QueryResult &result) const {
+  result.Clear();
+}
+
+void QueryNode::Free () {
+}
+
+void QueryNode::Print (ostream &/*s*/, int /*indent*/) const {
+}
+
+
+
+AndQueryNode::AndQueryNode () {
+  leftNode = NULL;
+  rightNode = NULL;
+}
+
+AndQueryNode::~AndQueryNode () {
+  Free ();
+}
+
+void AndQueryNode::Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			      QueryResult &result) const {
+  result.Clear();
+
+  // an And between nothing and something is nothing...
+  if (leftNode == NULL || rightNode == NULL) return;
+  
+  // calculate the result from the left tree and the result
+  // from the right tree
+  QueryResult rightResult;
+  leftNode->Calculate (indexData, queryInfo, result);
+  rightNode->Calculate (indexData, queryInfo, rightResult);
+
+  // merge the results, this can be done in place as the results
+  // will always shrink with an And
+
+  bool haveAccum = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  if (haveAccum && (result.ranks.size() != result.docs.size() ||
+		    rightResult.ranks.size() != rightResult.docs.size())) {
+    // shouldn't ever get here
+    haveAccum = false;
+    assert (0);
+  }
+
+  // combine document numbers and corresponding ranks
+  unsigned long leftI = 0;
+  unsigned long rightI = 0;
+  unsigned long outI = 0;
+  while (leftI < result.docs.size() &&
+	 rightI < rightResult.docs.size()) {
+    if (result.docs[leftI] < rightResult.docs[rightI]) {
+      leftI++;
+    } else if (result.docs[leftI] > rightResult.docs[rightI]) {
+      rightI++;
+    } else {
+      // the documents are equal
+      result.docs[outI] = result.docs[leftI];
+      if (haveAccum)
+	result.ranks[outI] = result.ranks[leftI] + rightResult.ranks[rightI];
+      leftI++;
+      rightI++;
+      outI++;
+    }
+  }
+
+  // erase unused document numbers and ranks
+  result.docs.erase(result.docs.begin()+outI, result.docs.end());
+  if (haveAccum)
+    result.ranks.erase(result.ranks.begin()+outI, result.ranks.end());
+    
+  // combine term frequency information
+  if (queryInfo.needTermFreqs)
+    result.termFreqs.insert (result.termFreqs.end(),
+			     rightResult.termFreqs.begin(),
+			     rightResult.termFreqs.end());
+}
+
+void AndQueryNode::Free () {
+  if (leftNode != NULL) {
+    delete leftNode;
+    leftNode = NULL;
+  }
+  if (rightNode != NULL) {
+    delete rightNode;
+    rightNode = NULL;
+  }
+}
+
+void AndQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "leftNode:\n", indent);
+  PrintNode (s, leftNode, indent+2);
+  PrintIndentText (s, "AND\n", indent);
+  PrintIndentText (s, "rightNode:\n", indent);
+  PrintNode (s, rightNode, indent+2);
+}
+
+
+OrQueryNode::OrQueryNode () {
+  leftNode = NULL;
+  rightNode = NULL;
+}
+
+OrQueryNode::~OrQueryNode () {
+  Free ();
+}
+
+void OrQueryNode::Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			      QueryResult &result) const {
+  result.Clear();
+  
+  // calculate the result from the left tree and the result
+  // from the right tree
+  QueryResult leftResult;
+  QueryResult rightResult;
+  if (leftNode != NULL)
+    leftNode->Calculate (indexData, queryInfo, leftResult);
+  if (rightNode != NULL)
+    rightNode->Calculate (indexData, queryInfo, rightResult);
+
+  // merge the results
+  
+  bool haveAccum = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  if (haveAccum && (leftResult.ranks.size() != leftResult.docs.size() ||
+		    rightResult.ranks.size() != rightResult.docs.size())) {
+    // shouldn't ever get here
+    haveAccum = false;
+    assert (0);
+  }
+
+  // combine document numbers and corresponding ranks
+  unsigned long leftSize = leftResult.docs.size();
+  unsigned long rightSize = rightResult.docs.size();
+  unsigned long leftI = 0;
+  unsigned long rightI = 0;
+  unsigned long leftDocNum = 0;
+  unsigned long rightDocNum = 0;
+  while (leftI < leftSize || rightI < rightSize) {
+    // check leftI
+    if (leftI < leftResult.docs.size())
+      leftDocNum = leftResult.docs[leftI];
+    else leftDocNum = ULONG_MAX;
+    
+    // check rightI
+    if (rightI < rightResult.docs.size())
+      rightDocNum = rightResult.docs[rightI];
+    else rightDocNum = ULONG_MAX;
+    
+    // combine
+    if (leftDocNum < rightDocNum) {
+      result.docs.push_back (leftDocNum);
+      if (haveAccum)
+	result.ranks.push_back (leftResult.ranks[leftI]);
+      leftI++;
+      
+    } else if (leftDocNum > rightDocNum) {
+      result.docs.push_back (rightDocNum);
+      if (haveAccum)
+	result.ranks.push_back (rightResult.ranks[rightI]);
+      rightI++;
+      
+    } else { // equal
+      result.docs.push_back (leftDocNum);
+      if (haveAccum)
+	result.ranks.push_back (leftResult.ranks[leftI] +
+				rightResult.ranks[rightI]);
+      leftI++;
+      rightI++;
+    }
+  }
+
+  // combine term frequency information
+  if (queryInfo.needTermFreqs) {
+    result.termFreqs.insert (result.termFreqs.end(),
+			     leftResult.termFreqs.begin(),
+			     leftResult.termFreqs.end());
+    result.termFreqs.insert (result.termFreqs.end(),
+			     rightResult.termFreqs.begin(),
+			     rightResult.termFreqs.end());
+  }
+}
+
+void OrQueryNode::Free () {
+  if (leftNode != NULL) {
+    delete leftNode;
+    leftNode = NULL;
+  }
+  if (rightNode != NULL) {
+    delete rightNode;
+    rightNode = NULL;
+  }
+}
+
+void OrQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "leftNode:\n", indent);
+  PrintNode (s, leftNode, indent+2);
+  PrintIndentText (s, "OR\n", indent);
+  PrintIndentText (s, "rightNode:\n", indent);
+  PrintNode (s, rightNode, indent+2);
+}
+
+
+
+NotQueryNode::NotQueryNode () {
+  queryNode = NULL;
+  notNode = NULL;
+}
+
+NotQueryNode::~NotQueryNode () {
+  Free ();
+}
+
+void NotQueryNode::Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			      QueryResult &result) const {
+  result.Clear();
+
+  // check for nothing
+  if (queryNode == NULL) return;
+  if (notNode == NULL) {
+    queryNode->Calculate (indexData, queryInfo, result);
+    return;
+  }
+  
+  // calculate the result from the query tree and the result
+  // from the not tree
+  QueryResult notResult;
+  queryNode->Calculate (indexData, queryInfo, result);
+  notNode->Calculate (indexData, queryInfo, notResult);
+
+  // merge the results, this can be done in place as the results
+  // will always shrink with a Not
+
+  bool haveAccum = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  if (haveAccum && (result.ranks.size() != result.docs.size() ||
+		    notResult.ranks.size() != notResult.docs.size())) {
+    // shouldn't ever get here
+    haveAccum = false;
+    assert (0);
+  }
+
+  // combine document numbers and corresponding ranks
+  unsigned long queryI = 0;
+  unsigned long notI = 0;
+  unsigned long outI = 0;
+  while (queryI < result.docs.size() &&
+	 notI < notResult.docs.size()) {
+    if (result.docs[queryI] < notResult.docs[notI]) {
+      // found a document not in the notResults
+      result.docs[outI] = result.docs[queryI];
+      if (haveAccum)
+	result.ranks[outI] = result.ranks[queryI];
+      queryI++;
+      outI++;
+    } else if (result.docs[queryI] > notResult.docs[notI]) {
+      notI++;
+    } else {
+      // the documents are equal, ignore both
+      queryI++;
+      notI++;
+    }
+  }
+
+  // erase unused document numbers and ranks
+  result.docs.erase(result.docs.begin()+outI, result.docs.end());
+  if (haveAccum)
+    result.ranks.erase(result.ranks.begin()+outI, result.ranks.end());
+  
+  // combine term frequency information
+  if (queryInfo.needTermFreqs)
+    result.termFreqs.insert (result.termFreqs.end(),
+			     notResult.termFreqs.begin(),
+			     notResult.termFreqs.end());
+}
+
+void NotQueryNode::Free () {
+  if (queryNode != NULL) {
+    delete queryNode;
+    queryNode = NULL;
+  }
+  if (notNode != NULL) {
+    delete notNode;
+    notNode = NULL;
+  }
+}
+
+void NotQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "queryNode:\n", indent);
+  PrintNode (s, queryNode, indent+2);
+  PrintIndentText (s, "NOT\n", indent);
+  PrintIndentText (s, "notNode:\n", indent);
+  PrintNode (s, notNode, indent+2);
+}
+
+
+void TagNode::Clear () {
+  UCArrayClear (tagName);
+}
+
+void TagNode::Calculate (IndexData &indexData,
+			 FragRangeArray &fragRange) const {
+  fragRange.erase (fragRange.begin(), fragRange.end());
+  if (tagName.empty()) return;
+
+  // get information about this tag
+  block_dict_el tagEl;
+  unsigned long tagElNum;
+  if (!SearchBlockDictEl (indexData.dictFile, indexData.biTags,
+			  indexData.bdh.entries_per_tblk,
+			  indexData.bdh.tag_dict_size,
+			  tagName, tagEl, tagElNum))
+    return;
+
+  // seek to the appropriate place in the inverted file
+  fseek (indexData.invfFile, tagEl.invf_ptr, SEEK_SET);
+
+  stdio_bitio_buffer buffer(indexData.invfFile);
+  
+  unsigned long pTag = tagEl.frag_occur*2;
+  unsigned long B = BIO_Bblock_Init (indexData.bdh.num_frags+pTag, pTag);
+  unsigned long fragNum = 0;
+  unsigned long i;
+  FragRange thisFrag;
+  for (i=0; i<tagEl.frag_occur; i++) {
+    // get start
+    unsigned long delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+
+    thisFrag.rangeStart = fragNum;
+
+    // get end
+    delta = buffer.bblock_decode (B, NULL)-1;
+    fragNum += delta;
+
+    thisFrag.rangeEnd = fragNum;
+    fragRange.push_back (thisFrag);
+  }
+
+  buffer.done();
+}
+
+void TagNode::Free () {
+  Clear ();
+}
+
+void TagNode::Print (ostream &s, int indent) const {
+  PrintIndent (s, indent);
+  s << "TAG: \"" << tagName << "\"\n";
+}
+
+
+void TermNode::Clear () {
+  UCArrayClear (term);
+  termWeight = 1;
+  stemMethod = 0;
+  startRange = NO_TERM_RANGE_START;
+  endRange = NO_TERM_RANGE_END;
+}
+
+TermNode::TermNode () {
+  Clear ();
+}
+
+void TermNode::Calculate (IndexData &indexData,
+			  bool needFragFreqs,
+			  FragRangeArray *fragLimits,
+			  FragData &fragData,
+			  UCArrayVector &equivTerms) const {
+  fragData.Clear ();
+  equivTerms.erase(equivTerms.begin(), equivTerms.end());
+
+  // get a list of term numbers
+  vector<unsigned long> equivNums;
+  FindWordNumbers (indexData, term, stemMethod, equivNums);
+
+  // get the information for each word and merge it with
+  // previous results
+  FragData tempFragData1;
+  FragData tempFragData2;
+  UCArray equivWord;
+  vector<unsigned long>::iterator here = equivNums.begin();
+  vector<unsigned long>::iterator end = equivNums.end();
+  while (here != end) {
+    // get the information for this word
+    ReadTermFragData (indexData, needFragFreqs, *here,
+		      tempFragData1, fragLimits, equivWord);
+    equivTerms.push_back(equivWord);
+    // combine with last results
+    tempFragData2 = fragData;
+    CombineFragData (needFragFreqs, tempFragData1, tempFragData2, fragData);
+    
+    here++;
+  }
+}
+
+void TermNode::Free () {
+  Clear ();
+}
+
+void TermNode::Print (ostream &s, int indent) const {
+  PrintIndent (s, indent);
+  s << "TERM: \"" << term << "\"\n";
+  PrintIndent (s, indent+2);
+  s << "termWeight: " << termWeight << "\n";
+  PrintIndent (s, indent+2);
+  s << "stemMethod: " << stemMethod << "\n";
+  PrintIndent (s, indent+2);
+  s << "startRange: " << startRange << "\n";
+  PrintIndent (s, indent+2);
+  s << "endRange: " << endRange << "\n";
+}
+
+
+ProxMatchQueryNode::ProxMatchQueryNode () {
+  tagNodePtr = NULL;
+}
+
+ProxMatchQueryNode::~ProxMatchQueryNode () {
+  Free ();
+}
+
+void ProxMatchQueryNode::Calculate (IndexData &indexData,
+				    const QueryInfo &queryInfo,
+				    QueryResult &result) const {
+  result.Clear ();
+
+  bool needFragFreqs = (queryInfo.sortByRank || queryInfo.needRankInfo);
+
+  // read in the tag if needed
+  FragRangeArray fragLimits;
+  FragRangeArray *fragLimitsPtr = NULL;
+  if (tagNodePtr == NULL && terms.size() > 1) {
+    // multiple terms must be compared relative to some tag
+    // otherwise phrase matches could span documents
+    TagNode tempTagNode;
+    tempTagNode.tagName = indexData.curLevel;
+    tempTagNode.Calculate (indexData, fragLimits);
+    fragLimitsPtr = &fragLimits;
+    
+  } else if (tagNodePtr != NULL) {
+    (*tagNodePtr).Calculate (indexData, fragLimits);
+    fragLimitsPtr = &fragLimits;
+  }
+
+  UCArray tagOrLevel = indexData.curLevel;
+  if (tagNodePtr != NULL) tagOrLevel = (*tagNodePtr).tagName;
+
+  // read in the first term
+  FragData termData;
+  UCArrayVector equivTerms;
+  TermNodeArray::const_iterator termHere=terms.begin(), termEnd = terms.end();
+  if (termHere != termEnd) {
+    (*termHere).Calculate (indexData, needFragFreqs, fragLimitsPtr, 
+			   termData, equivTerms);
+
+    // convert initial fragment information
+    FragsToQueryResult (indexData,
+			queryInfo,
+			termData,
+			tagOrLevel,
+			(*termHere).term,
+			(*termHere).stemMethod,
+			(*termHere).termWeight,
+			equivTerms,
+			result);
+  
+    termHere++;
+
+    if (termHere == termEnd) return; // nothing more to do
+  }
+
+  // read and combine the rest of the terms
+  FragData comTermData;
+  while (termHere != termEnd) {
+    (*termHere).Calculate (indexData, needFragFreqs,
+			   fragLimitsPtr, comTermData, equivTerms);
+
+    AndFragsToQueryResult (indexData,
+			   queryInfo,
+			   comTermData,
+			   tagOrLevel,
+			   (*termHere).term,
+			   (*termHere).stemMethod,
+			   (*termHere).termWeight,
+			   equivTerms,
+			   result);
+    
+    AndCombineFragData (needFragFreqs, termData, comTermData,
+			(*termHere).startRange,
+			(*termHere).endRange,
+			fragLimitsPtr);
+    termHere++;
+  }
+
+  // remove unwanted document numbers
+  RemoveUnwantedResults (indexData,
+			 queryInfo,
+			 termData,
+			 result);
+}
+
+void ProxMatchQueryNode::Free () {
+  if (tagNodePtr != NULL) {
+    delete tagNodePtr;
+    tagNodePtr = NULL;
+  }
+  terms.erase (terms.begin(), terms.end());
+}
+
+void ProxMatchQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText (s, "PROXMATCH\n", indent);
+  if (tagNodePtr != NULL) tagNodePtr->Print (s, indent+2);
+  
+  TermNodeArray::const_iterator here = terms.begin();
+  TermNodeArray::const_iterator end = terms.end();
+  while (here != end) {
+    (*here).Print (s, indent+2);
+    here++;
+  }
+}
+
+
+void BrowseQueryNode::Clear () {
+  UCArrayClear(term);
+}
+
+void BrowseQueryNode::Calculate (IndexData &indexData, BrowseQueryResult &result) const {
+
+  unsigned long number=0;
+  FindNearestWordNumber(indexData, term, number);
+  if (number + startPosition > 0 ) {
+    number = number+startPosition;
+  }
+  else {
+    number = 1;
+  }
+
+  GetTermList (indexData, number, numTerms, result.termFreqs);
+
+}
+
+
+
+void BrowseQueryNode::Free () {
+  Clear();
+}
+
+
+void BrowseQueryNode::Print (ostream &s, int indent) const {
+  PrintIndentText(s, "BROWSEQUERYNODE\n", indent);
+  PrintIndent (s, indent+2);
+  s << "TERM:"<<term<<"\n";
+  PrintIndent (s, indent+2);
+  s << "Start position: "<< startPosition<<", Num terms: "<< numTerms<<"\n";
+
+
+}
+
+
+
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      QueryResult &result) {
+  result.Clear ();
+
+  // make sure level is current
+  if (!indexData.LoadLevel (queryInfo.docLevel)) {
+    return;
+  }
+  // do query
+  if (queryTree == NULL) return;
+  
+  (*queryTree).Calculate (indexData, queryInfo, result);
+
+  // make weights into ranks if needed
+  unsigned long i;
+  if (queryInfo.sortByRank || queryInfo.needRankInfo) {
+    for (i=0; i<result.ranks.size(); i++) {
+      result.ranks[i] /=
+	indexData.weightData.GetLowerApproxDocWeight (result.docs[i]);
+    }
+  }
+  
+  unsigned long resultsSize = queryInfo.maxDocs;
+  if (resultsSize == 0 || resultsSize > result.docs.size())
+    resultsSize = result.docs.size();
+
+  result.actualNumDocs = result.docs.size(); // the total number of docs 
+                                   //returned, before pruning based on maxDocs
+
+  // sort results by rank if needed
+  GTRank gtRank;
+  if (queryInfo.sortByRank) {
+    // need in ascending order for SelectAddHeap
+    MakeHeap (result.docs.begin(), result.ranks.begin(),
+	      resultsSize, gtRank);
+    SelectAddHeap (result.docs.begin(), result.ranks.begin(), resultsSize,
+		   result.docs.begin()+resultsSize,
+		   result.ranks.begin()+resultsSize,
+		   (unsigned long) result.ranks.size()-resultsSize,
+		   gtRank);
+
+    // sort into descending order
+    SortHeap (result.docs.begin(), result.ranks.begin(), resultsSize, gtRank);
+  }
+  
+  // get exact weights if needed
+  if (queryInfo.exactWeights &&
+      (queryInfo.sortByRank || queryInfo.needRankInfo)) {
+    unsigned long exactDiskPtr =
+      indexData.levels.levelInfo[indexData.curLevel].exactWeightsDiskPtr;
+    
+    for (i=0; i<resultsSize; i++) {
+      result.ranks[i] =  result.ranks[i] *
+	indexData.weightData.GetLowerApproxDocWeight (result.docs[i]) /
+	GetExactDocWeight (indexData.exactWeightsFile, exactDiskPtr,
+			   result.docs[i]);
+    }
+
+    // re-sort top candidates based on exact weights
+    if (queryInfo.sortByRank) {
+      MakeHeap (result.docs.begin(), result.ranks.begin(),
+		resultsSize, gtRank);
+      SortHeap (result.docs.begin(), result.ranks.begin(),
+		resultsSize, gtRank);
+    }
+  }
+
+  // get rid of unwanted ranking results
+  if (result.ranks.empty()) {
+    // do nothing
+  } else if (!queryInfo.needRankInfo) {
+    result.ranks.erase(result.ranks.begin(), result.ranks.end());
+  } else {
+    result.ranks.erase(result.ranks.begin()+resultsSize, result.ranks.end());
+  }
+
+  // remove extra docs that are unwanted
+  result.docs.erase(result.docs.begin()+resultsSize, result.docs.end());
+}
+
+// new MGQuery to retrieve doc and section nums
+// this will return doc nums for the level queried at (set in queryInfo)
+// in QueryResult.docs and if a second level is specified, 
+// it will return corresponding docnums for that level in QueryResult.levels
+// If there is no level specified, or that level is invalid, the query
+// level is used
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      ExtQueryResult &realresult, UCArray &level) {
+  realresult.Clear ();
+  QueryResult result; // temp result
+
+  // do the normal query
+  MGQuery (indexData, queryInfo, queryTree, result);
+
+  // now that have the final result stuff, convert to ExtQueryResult,
+  // add in level nums if needed
+
+  realresult.docs = result.docs;
+  realresult.ranks = result.ranks;
+  realresult.termFreqs = result.termFreqs;
+  realresult.actualNumDocs = result.actualNumDocs;
+
+  if (queryInfo.docLevel == level || level.empty()) {
+    realresult.levels = result.docs;
+    return;
+  }
+
+  // else need to convert from queryInfo.docLevel to level
+
+  // the original level info
+  FragLevelConvert sectionLevelConverter = indexData.levelConverter;
+
+  // the new level info
+  if (!indexData.LoadLevel(level)) {
+    realresult.levels = result.docs;
+    return;
+  }
+  
+  unsigned long DocNum = 0;
+  
+  for (unsigned long i=0; i<realresult.docs.size(); i++) {
+
+    // do an if ! here????
+    indexData.levelConverter.LevelToLevel(sectionLevelConverter, realresult.docs[i], DocNum);
+    realresult.levels.push_back(DocNum);
+  }
+  
+}
+
+
+
+// new function for full text browsing, 
+void MGBrowseQuery (IndexData &indexData, UCArray &level,
+		  const BrowseQueryNode &node,
+		  BrowseQueryResult &result) {
+
+  indexData.LoadLevel(level);
+  node.Calculate(indexData, result); 
+
+}
Index: /trunk/mgpp/text/MGQuery.h
===================================================================
--- /trunk/mgpp/text/MGQuery.h	(revision 3365)
+++ /trunk/mgpp/text/MGQuery.h	(revision 3365)
@@ -0,0 +1,197 @@
+/**************************************************************************
+ *
+ * MGQuery.h -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef MGQUERY_H
+#define MGQUERY_H
+
+#include "IndexData.h"
+#include <limits.h>
+#include "Terms.h"
+
+// forms base for defining and calculating the boolean
+// part of a query
+class QueryNode {
+public:
+  QueryNode ();
+  virtual ~QueryNode ();
+
+  virtual void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+			  QueryResult &result) const;
+  virtual void Free ();
+  virtual void Print (ostream &s, int indent=0) const;
+};
+
+
+
+void PrintIndent (ostream &s, int indent);
+void PrintIndentText (ostream &s, char *text, int indent);
+void PrintNode (ostream &s, QueryNode *node, int indent=0);
+
+
+
+// basic boolean nodes
+
+class AndQueryNode : public QueryNode {
+public:
+  QueryNode *leftNode;
+  QueryNode *rightNode;
+  
+  AndQueryNode ();
+  ~AndQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+class OrQueryNode : public QueryNode {
+public:
+  QueryNode *leftNode;
+  QueryNode *rightNode;
+  
+  OrQueryNode ();
+  ~OrQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+class NotQueryNode : public QueryNode {
+public:
+  QueryNode *queryNode;
+  QueryNode *notNode;
+  
+  NotQueryNode ();
+  ~NotQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+
+
+class TagNode {
+public:
+  UCArray tagName;
+
+  void Clear ();
+  TagNode () { Clear (); }
+
+  void Calculate (IndexData &indexData, FragRangeArray &fragRange) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+
+#define NO_TERM_RANGE_START (LONG_MIN/2)
+#define NO_TERM_RANGE_END   (LONG_MAX/2)
+
+/* NOTE: range stuff - the range values are for the previous term relative
+to the current term. So if searching for the phrase "the cat", 'the' doesn't 
+need range limits, but 'cat' has a range of -2 to -1. ie, if have found 'cat'
+then 'the' has to be at position between -2 and -1 relative to 'cat'.
+"the cat" could also be searched for by 'cat' with no range limits, then 'the' with range 0 to 1.
+range values are relative to the gaps between words:
+   x   y   z   X   a   b   c
+    -3   -2  -1  0   1   2   3
+
+ */
+class TermNode {
+public:
+  UCArray term;
+  unsigned long termWeight;
+  unsigned long stemMethod;
+  signed long startRange; // range relative to last term
+  signed long endRange;
+  
+  void Clear ();
+  TermNode ();
+
+  // fragLimits can be NULL if not neede
+  void Calculate (IndexData &indexData,
+		  bool needFragFreqs,
+		  FragRangeArray *fragLimits,
+		  FragData &fragData,
+		  UCArrayVector &equivTerms) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+typedef vector<TermNode> TermNodeArray;
+
+
+class ProxMatchQueryNode : public QueryNode {
+public:
+  TagNode *tagNodePtr; // optional field
+  TermNodeArray terms;
+  
+  ProxMatchQueryNode ();
+  ~ProxMatchQueryNode ();
+
+  void Calculate (IndexData &indexData, const QueryInfo &queryInfo,
+		  QueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+};
+
+class BrowseQueryNode :public QueryNode {
+ public:
+  UCArray term;
+  signed long startPosition;
+  unsigned long numTerms;
+
+  void Clear();
+  BrowseQueryNode () { Clear(); }
+  //  ~BrowseQueryNode ();
+
+  void Calculate (IndexData &indexData, BrowseQueryResult &result) const;
+  void Free ();
+  void Print (ostream &s, int indent=0) const;
+
+
+
+
+};
+
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      QueryResult &result);
+
+// this function for retrieving results with both section doc nums
+// and Document docnums
+void MGQuery (IndexData &indexData,
+	      const QueryInfo &queryInfo,
+	      const QueryNode *queryTree,
+	      ExtQueryResult &result, UCArray &level);
+
+
+// new function for full text browsing, 
+void MGBrowseQuery (IndexData &indexData, UCArray &level,
+		  const BrowseQueryNode &node,
+		  BrowseQueryResult &result);
+
+#endif
Index: /trunk/mgpp/text/Makefile.in
===================================================================
--- /trunk/mgpp/text/Makefile.in	(revision 3365)
+++ /trunk/mgpp/text/Makefile.in	(revision 3365)
@@ -0,0 +1,315 @@
+###########################################################################
+#
+# Makefile.in -- Makefile for the MG system
+# Copyright (C) 1994  Neil Sharman;  1995 Tim Shimmin
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+#       @(#)Makefile.in	1.10 22 Mar 1994
+#
+###########################################################################
+
+PACKAGE = @PACKAGE@
+VERSION = @VERSION@
+
+SHELL = /bin/sh
+VPATH = @srcdir@
+exec_prefix = @exec_prefix@
+prefix = @prefix@
+bindir = @bindir@
+srcdir = @srcdir@
+subdir = text
+top_srcdir = @top_srcdir@
+transform = @program_transform_name@
+
+AR = ar
+CC = @CC@
+CXX = @CXX@
+CFLAGS = @CFLAGS@ -ansi -DSILENT -DSHORT_SUFFIX
+CXXFLAGS = @CXXFLAGS@ -DSILENT -DSHORT_SUFFIX
+DEFS = @DEFS@
+RANLIB = @RANLIB@
+INSTALL = @INSTALL@
+INSTALL_DATA = @INSTALL_DATA@
+INSTALL_PROGRAM = @INSTALL_PROGRAM@
+LDFLAGS = @LDFLAGS@
+LIBS = ../lib/libmg.a -lstdc++ @LIBS@
+TOUCH = echo timestamp >
+AWK = @AWK@
+
+##########################################
+# George Buchanan - August 1999
+# 
+# compiling with make NZDL=1 to ensure that 
+# the NZDL compile mode is used - this has been changed to the default - kjm18
+#
+##########################################
+
+datadir = $(prefix)/share
+libexecdir = $(exec_prefix)/libexec
+localedir = $(datadir)/locale
+mandir = $(prefix)/man/man1
+
+COMPILE = $(CC) -c $(DEFS) $(TREC_MODE) $(READLINE) $(INCLUDES) $(CFLAGS)
+CXXCOMPILE = $(CXX) -c $(DEFS) $(TREC_MODE) $(READLINE) $(INCLUDES) $(CXXFLAGS)
+LINK = $(CC) $(LDFLAGS) -o $@
+
+.SUFFIXES:
+.SUFFIXES: .c .o
+.c.o:
+	$(COMPILE) $<
+
+.SUFFIXES: .cpp .o
+.cpp.o:
+	$(CXXCOMPILE) $<
+
+
+ANSI2KNR = @ANSI2KNR@
+o = .@U@o
+
+.SUFFIXES: ._c ._o
+.c._c:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $@
+._c._o:
+	@echo $(COMPILE) $<
+	@rm -f _$*.c
+	@ln $< _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+.c._o:
+	./ansi2knr $< > $*.tmp && mv $*.tmp $*._c
+	@echo $(COMPILE) $*._c
+	@rm -f _$*.c
+	@ln $*._c _$*.c && $(COMPILE) _$*.c && mv _$*.o $@ && rm _$*.c
+
+INCLUDES = -I.. -I../lib -I.
+
+#############################################################################
+#
+# Uncomment (and possibly edit) the following lines if you wish
+# to use the GNU readline library for the input in mgquery
+# 
+#READLINE = -DGNU_READLINE -I/opt/gnu/include/readline -L/opt/gnu/lib
+#READLINE_LIBS = -lreadline -ltermcap
+#
+#
+#############################################################################
+
+SOURCES = \
+  FIvfLevelInfo.cpp        ivf.pass1.cpp            mgpp_invf_dump.cpp       \
+  FText.cpp                ivf.pass2.cpp            mgpp_passes.cpp          \
+  FileBuf.cpp              locallib.cpp             mgpp_perf_hash_build.cpp \
+  FragLevelConvert.cpp     mg.special.cpp           mgpp_stem_idx.cpp        \
+  IndexData.cpp            mgpp_compression_dict.cpp  mgpp_weights_build.cpp   \
+  TagInfo.cpp              mgpp_decompress_text.cpp   stemmer.cpp            \
+  TextEl.cpp               mg_errors.cpp            text.cpp               \
+  TextGet.cpp              mgpp_fast_comp_dict.cpp    text.pass1.cpp         \
+  UCArray.cpp              mg_files.cpp             text.pass2.cpp         \
+  comp_dict.cpp            mgpp_invf_dict.cpp         words.cpp              \
+  invf.cpp                 mgpp_invf_dict_dump.cpp    Weights.cpp            \
+  MGQuery.cpp              Terms.cpp                QueryTester.cpp        \
+  QueryLex.cpp             QueryParser.cpp         \
+  GSDLQueryLex.cpp	   GSDLQueryParser.cpp      Queryer.cpp
+
+
+HEADERS = \
+  FIvfLevelInfo.h     TextEl.h            hash.h              stemmer.h  \
+  FText.h             TextGet.h           invf.h              text.h     \
+  FileBuf.h           UCArray.h           locallib.h          words.h    \
+  FragLevelConvert.h  WordData.h          mg.h                           \
+  IndexData.h         build.h             mg_errors.h                    \
+  TagInfo.h           comp_dict.h         mg_files.h          Weights.h  \
+  MGQuery.h           Terms.h             QueryLex.h          QueryParser.h \
+  GSDLQueryLex.h	GSDLQueryParser.h
+
+
+MISC          = Makefile.in 
+
+ALLFILES      =	$(SOURCES) $(HEADERS) $(MISC) $(MAN)
+
+DISTFILES     = $(ALLFILES) 
+
+EXEC          = mgpp_passes mgpp_decompress_text mgpp_perf_hash_build \
+                mgpp_invf_dump mgpp_weights_build mgpp_invf_dict mgpp_stem_idx \
+                mgpp_invf_dict_dump mgpp_compression_dict mgpp_fast_comp_dict \
+                QueryTester Queryer
+
+MAN           =  mgpp_weights_build.1  mgpp_invf_dict.1 \
+                 mgpp_passes.1 mgpp_perf_hash_build.1 \
+                 mgpp_compression_dict.1  mgpp_fast_comp_dict.1 \
+                 mgpp_stem_idx.1
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+
+all:	$(EXEC) libtextin.a
+
+depend:
+	makedepend -Y -- $(DEFS) $(INCLUDES) $(CXXFLAGS) $(CFLAGS) -- $(SOURCES)
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+
+PASSES_OBJS = \
+	mgpp_passes$o text.pass1$o comp_dict$o \
+	text.pass2$o locallib$o \
+	ivf.pass1$o ivf.pass2$o mg.special$o mg_files$o \
+	words$o invf$o FileBuf$o TextEl$o UCArray$o TagInfo$o \
+        FText$o text$o FIvfLevelInfo$o
+
+mgpp_passes: $(PASSES_OBJS)
+	$(LINK) $(PASSES_OBJS) $(LIBS)
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+QUERYER_OBJS = Queryer$o mg_files$o mg_errors$o locallib$o invf$o UCArray$o \
+               FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+               IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+               GSDLQueryParser$o GSDLQueryLex$o words$o
+
+Queryer: $(QUERYER_OBJS)
+	$(LINK) $(QUERYER_OBJS) $(LIBS)
+
+QUERY_TESTER_OBJS = QueryTester$o mg_files$o invf$o UCArray$o \
+                    FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+                    IndexData$o stemmer$o Weights$o
+
+QueryTester: $(QUERY_TESTER_OBJS)
+	$(LINK) $(QUERY_TESTER_OBJS) $(LIBS)
+
+WEIGHTS_BUILD_OBJS = mgpp_weights_build$o mg_files$o invf$o UCArray$o \
+                     FIvfLevelInfo$o FragLevelConvert$o
+
+mgpp_weights_build: $(WEIGHTS_BUILD_OBJS)
+	$(LINK) $(WEIGHTS_BUILD_OBJS) $(LIBS)
+
+COMP_DICT_OBJS = mgpp_compression_dict$o mg_files$o locallib$o
+
+mgpp_compression_dict: $(COMP_DICT_OBJS)
+	$(LINK) $(COMP_DICT_OBJS) $(LIBS)
+
+FAST_COMP_OBJS = mgpp_fast_comp_dict$o mg_files$o locallib$o
+
+mgpp_fast_comp_dict: $(FAST_COMP_OBJS)
+	$(LINK) $(FAST_COMP_OBJS) $(LIBS)
+
+INVF_DUMP_OBJS = mgpp_invf_dump$o locallib$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dump: $(INVF_DUMP_OBJS)
+	$(LINK) $(INVF_DUMP_OBJS) $(LIBS)
+
+INVF_DICT_OBJS = mgpp_invf_dict$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict: $(INVF_DICT_OBJS)
+	$(LINK) $(INVF_DICT_OBJS) $(LIBS)
+
+INVF_DICT_DUMP_OBJS = mgpp_invf_dict_dump$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict_dump: $(INVF_DICT_DUMP_OBJS)
+	$(LINK) $(INVF_DICT_DUMP_OBJS) $(LIBS)
+
+PERF_HASH_OBJS = mgpp_perf_hash_build$o mg_files$o invf$o UCArray$o
+
+mgpp_perf_hash_build: $(PERF_HASH_OBJS)
+	$(LINK) $(PERF_HASH_OBJS) $(LIBS)
+
+STEM_IDX_OBJS = mgpp_stem_idx$o mg_files$o stemmer$o \
+                invf$o UCArray$o
+
+mgpp_stem_idx: $(STEM_IDX_OBJS)
+	$(LINK) $(STEM_IDX_OBJS) $(LIBS)
+
+DECOMPRESS_TEXT_OBJS = \
+                mgpp_decompress_text$o mg_files$o mg_errors$o \
+                locallib$o text$o UCArray$o TextGet$o FText$o
+
+mgpp_decompress_text: $(DECOMPRESS_TEXT_OBJS)
+	$(LINK) $(DECOMPRESS_TEXT_OBJS) $(LIBS)
+
+# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
+
+ansi2knr: ansi2knr.o
+	$(LINK) ansi2knr.o $(LIBS)
+
+install: execinstall
+uninstall: execuninstall
+
+execinstall: $(EXEC)
+	../mkinstalldirs $(bindir); \
+	for name in $(EXEC); do \
+	  $(INSTALL_PROGRAM) $$name \
+	  $(bindir)/`echo $$name | sed '$(transform)'`; \
+	done
+
+execuninstall:
+	for name in $(EXEC); do \
+	 rm -f $(bindir)/`echo $$name | sed '$(transform)'`; \
+	done
+
+maninstall: $(MAN)
+	$(top_srcdir)/mkinstalldirs $(mandir); \
+	for f in $(MAN); do \
+	  $(INSTALL_DATA) $$f $(mandir)/$$f ;\
+	done
+
+manuninstall: $(MAN)
+	for f in $(MAN); do \
+	  rm -f $(mandir)/$$f ;\
+	done
+
+#mg_errors$o removed from LIB_OBJS to avoid conflict with mg's libtextin.a
+LIB_OBJS = mg_files$o  mg_errors$o locallib$o invf$o UCArray$o \
+           FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+           IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+           GSDLQueryParser$o GSDLQueryLex$o words$o
+
+libtextin.a: $(LIB_OBJS)
+	rm -f libtextin.a
+	$(AR) cru libtextin.a $(LIB_OBJS)
+	$(RANLIB) libtextin.a
+
+
+mostlyclean:
+	rm -f *$o _*.c _*.o *._c *._o core core.*
+
+clean: mostlyclean
+	rm -f $(EXEC) libtextin.a
+
+distclean: clean
+	rm -f ansi2knr
+	rm -f Makefile
+
+maintainer-clean: distclean
+	@echo "This command is intended only for maintainers to use;"
+	@echo "rebuilding the deleted files may require special tools."
+
+dist: $(DISTFILES)
+	for file in $(DISTFILES); do \
+	  ln $(srcdir)/$$file $(distdir) 2> /dev/null \
+	  || cp -p $(srcdir)/$$file $(distdir); \
+	done
+
+movclean:
+	for file in $(SOURCES); do \
+          test = $$file; \
+          echo $("test.c"##*.c); \
+        done 
+
+# jrm21
+#Makefile: Makefile.in ../../../config.status
+#	cd ../../.. && CONFIG_FILES=$(subdir)/$@ CONFIG_HEADERS= ./config.status
+
+# Tell versions [3.59,3.63) of GNU make not to export all variables.
+# Otherwise a system limit (for SysV at least) may be exc
+
+
Index: /trunk/mgpp/text/QueryLex.cpp
===================================================================
--- /trunk/mgpp/text/QueryLex.cpp	(revision 3365)
+++ /trunk/mgpp/text/QueryLex.cpp	(revision 3365)
@@ -0,0 +1,169 @@
+/**************************************************************************
+ *
+ * QueryLex.cpp -- Lexical analyser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "QueryLex.h"
+#include "unitool.h"
+#include "words.h"
+
+inline void AddNChar (UCArray::const_iterator &here,
+		      UCArray &text,
+		      int len) {
+  while (len > 0) {
+    text.push_back (*here++);
+    len--;
+  }
+}
+
+static bool ParseInteger (UCArray::const_iterator &here,
+			  UCArray::const_iterator end,
+			  LexEl &el) {
+  el.Clear();
+
+  // this version of end is used in unitool
+  UCArray::const_iterator endMinus1 = end-1;
+
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (here, endMinus1, &c);
+
+  // check for positive or negative
+  bool neg = false;
+  if (c == '+') {
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  } else if (c == '-') {
+    neg = true;
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  }
+
+  // read in number part
+  el.num = 0;
+  el.lexType = IntegerE;
+  while (c >= '0' && c <= '9') {
+    el.num = el.num*10 + c - '0';
+    AddNChar (here, el.text, charLen);
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  }
+
+  if (neg) el.num *= -1;
+
+  return (!el.text.empty());
+}
+
+static bool ParseTerm (UCArray::const_iterator &here,
+		       UCArray::const_iterator end,
+		       UCArray &text) {
+  UCArray::const_iterator endMinus1 = end-1;
+  here = ParseIndexWord (here, endMinus1, text);
+  return !text.empty();
+}
+
+
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el) {
+  el.Clear();
+
+  // strange things can happen if here == end == 0
+  if (here == end) return false;
+  
+  // this version of end is used in unitool
+  UCArray::const_iterator endMinus1 = end-1;
+  
+  // ignore all white space
+  int charLen;
+  unsigned short c; // one character lookahead
+  charLen = parse_utf8_char (here, endMinus1, &c);
+  while (here != end && is_unicode_space (c)) {
+    here += charLen;
+    charLen = parse_utf8_char (here, endMinus1, &c);
+  }
+  if (here == end) return false;
+
+  if (c == '(') {
+    el.lexType = OpenBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ')') {
+    el.lexType = CloseBracketE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '\"') {
+    el.lexType = QuoteE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '#') {
+    el.lexType = TermWeightE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '$') {
+    el.lexType = StemMethodE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '^') {
+    el.lexType = RangeE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '@') {
+    el.lexType = AtE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == ':') {
+    el.lexType = TagE;
+    AddNChar (here, el.text, charLen);
+    return true;
+    
+  } else if (c == '+' || c == '-' ||
+	     (c >= '0' && c <= '9')) {
+    return ParseInteger (here, end, el);
+  }
+
+  // assume it is a term of some sort
+  if (!ParseTerm (here, end, el.text)) return false;
+
+  UCArray AND; SetCStr (AND, "AND");
+  if (el.text == AND) {
+    el.lexType = AndOpE;
+    return true;
+  }
+  UCArray OR; SetCStr (OR, "OR");
+  if (el.text == OR) {
+    el.lexType = OrOpE;
+    return true;
+  }
+  UCArray NOT; SetCStr (NOT, "NOT");
+  if (el.text == NOT) {
+    el.lexType = NotOpE;
+    return true;
+  }
+
+  el.lexType = TermE;
+  return true;
+}
+
Index: /trunk/mgpp/text/QueryLex.h
===================================================================
--- /trunk/mgpp/text/QueryLex.h	(revision 3365)
+++ /trunk/mgpp/text/QueryLex.h	(revision 3365)
@@ -0,0 +1,53 @@
+/**************************************************************************
+ *
+ * QueryLex.h -- Lexical analyser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef QUERYLEX_H
+#define QUERYLEX_H
+
+#include "UCArray.h"
+
+
+// note: white space is ignored
+enum LexType {NoTypeE, OpenBracketE, CloseBracketE, TermE, AndOpE, OrOpE,
+	      NotOpE, QuoteE, IntegerE, TermWeightE, StemMethodE,
+	      RangeE, AtE, TagE};
+
+struct LexEl {
+  LexType lexType;
+  UCArray text;
+  int num; // for IntegerE
+
+  void Clear () {
+    lexType = NoTypeE;
+    UCArrayClear (text);
+    num = 0;
+  }
+  LexEl () { Clear (); }
+};
+
+
+// returns true if another lexical element could be parsed.
+bool ParseLexEl (UCArray::const_iterator &here,
+		 UCArray::const_iterator end,
+		 LexEl &el);
+
+
+#endif
Index: /trunk/mgpp/text/QueryParser.cpp
===================================================================
--- /trunk/mgpp/text/QueryParser.cpp	(revision 3365)
+++ /trunk/mgpp/text/QueryParser.cpp	(revision 3365)
@@ -0,0 +1,219 @@
+/**************************************************************************
+ *
+ * QueryParser.cpp -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "QueryParser.h"
+#include "QueryLex.h"
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end);
+
+static QueryNode *AndAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  AndQueryNode *andNode = new AndQueryNode;
+  andNode->leftNode = t1;
+  andNode->rightNode = t2;
+  return andNode;
+}
+
+static QueryNode *OrAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  OrQueryNode *orNode = new OrQueryNode;
+  orNode->leftNode = t1;
+  orNode->rightNode = t2;
+  return orNode;
+}
+
+static QueryNode *NotAdd (QueryNode *t1, QueryNode *t2) {
+  if (t1 == NULL) return t2;
+  if (t2 == NULL) return t1;
+
+  NotQueryNode *notNode = new NotQueryNode;
+  notNode->queryNode = t1;
+  notNode->notNode = t2;
+  return notNode;
+}
+
+// expects the opening bracket to have already been parsed
+// and discarded
+static QueryNode *ParseBracketExpression (UCArray::const_iterator &here,
+					  UCArray::const_iterator end) {
+  // get everything in the expression
+  QueryNode *curTree = ParseExpression (here, end);
+  
+  // gobble up tokens until a closing bracket is found
+  // or the end of the string
+  LexEl el;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == CloseBracketE) break;
+  }
+
+  return curTree;
+}
+
+static int ParseInt (UCArray::const_iterator &here,
+		     UCArray::const_iterator end) {
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  if (ParseLexEl (here, end, el) && el.lexType == IntegerE)
+    return el.num;
+  
+  here = oldHere; // not an integer
+  return 0;
+}
+
+
+static void ParseTermModifiers (UCArray::const_iterator &here,
+				UCArray::const_iterator end,
+				TermNode &termNode) {
+  LexEl el;
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermWeightE) {
+      termNode.termWeight = ParseInt (here, end);
+      
+    } else if (el.lexType == StemMethodE) {
+      termNode.stemMethod = ParseInt (here, end);
+      
+    } else if (el.lexType == RangeE) {
+      termNode.startRange = ParseInt (here, end);
+      termNode.endRange = ParseInt (here, end);
+      
+    } else if (el.lexType == AtE) {
+      termNode.startRange = termNode.endRange = ParseInt (here, end);
+      
+    } else {
+      // no term modifiers
+      here = oldHere;
+      break;
+    }
+    
+    oldHere = here;
+  }  
+}
+
+// expects the starting quote to have been parsed
+// and discarded
+static void ParsePhrase (UCArray::const_iterator &here,
+			 UCArray::const_iterator end,
+			 ProxMatchQueryNode &proxNode) {
+  LexEl el;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == TermE || el.lexType == IntegerE) {
+      TermNode termNode;
+      termNode.term = el.text;
+      ParseTermModifiers (here, end, termNode);
+      proxNode.terms.push_back (termNode);
+      
+    } else if (el.lexType == QuoteE) {
+      break;
+      
+    } else {
+      // error
+      break;
+    }
+  }
+}
+
+static QueryNode *ParseTerm (UCArray::const_iterator &here,
+			     UCArray::const_iterator end) {
+  LexEl el;
+
+  UCArray::const_iterator oldHere = here;
+  if (!ParseLexEl (here, end, el)) return NULL;
+
+  if (el.lexType == OpenBracketE)
+    return ParseBracketExpression (here, end);
+
+  ProxMatchQueryNode *proxNode = new ProxMatchQueryNode;
+
+  // check for a tag
+  if (el.lexType == TagE) {
+    oldHere = here; // don't backtrack past here
+    if (!ParseLexEl (here, end, el)) return NULL;
+    if (el.lexType == TermE) {
+      proxNode->tagNodePtr = new TagNode;
+      proxNode->tagNodePtr->tagName = el.text;
+      if (!ParseLexEl (here, end, el)) return NULL;
+    }
+  }
+  
+  if (el.lexType == TermE || el.lexType == IntegerE) {
+    TermNode termNode;
+    termNode.term = el.text;
+    ParseTermModifiers (here, end, termNode);
+    proxNode->terms.push_back (termNode);
+    return proxNode;
+    
+  } else if (el.lexType == QuoteE) {
+    ParsePhrase (here, end, *proxNode);
+    return proxNode;
+  }
+
+  // not a term
+  here = oldHere;
+  delete proxNode;
+  return NULL;
+}
+
+
+static QueryNode *ParseExpression (UCArray::const_iterator &here,
+				   UCArray::const_iterator end) {
+  LexEl el;
+  QueryNode *curTree = NULL;
+
+  UCArray::const_iterator oldHere = here;
+  while (ParseLexEl (here, end, el)) {
+    if (el.lexType == OpenBracketE ||
+	el.lexType == TermE ||
+	el.lexType == QuoteE ||
+	el.lexType == IntegerE ||
+	el.lexType == TagE) {
+      // some type of term, back track and parse it
+      here = oldHere;
+      curTree = OrAdd (curTree, ParseTerm (here, end));
+      
+    } else if (el.lexType == AndOpE) {
+      curTree = AndAdd (curTree, ParseTerm (here, end));
+      
+    } else if (el.lexType == OrOpE) {
+      curTree = OrAdd (curTree, ParseTerm (here, end));
+      
+    } else if (el.lexType == NotOpE) {
+      curTree = NotAdd (curTree, ParseTerm (here, end));
+      
+    } else break;
+    
+    oldHere = here;
+  }
+
+  return curTree;
+}
+
+QueryNode *ParseQuery (const UCArray &queryStr) {
+  UCArray::const_iterator here = queryStr.begin();
+  UCArray::const_iterator end = queryStr.end();
+  return ParseExpression (here, end);
+}
Index: /trunk/mgpp/text/QueryParser.h
===================================================================
--- /trunk/mgpp/text/QueryParser.h	(revision 3365)
+++ /trunk/mgpp/text/QueryParser.h	(revision 3365)
@@ -0,0 +1,31 @@
+/**************************************************************************
+ *
+ * QueryParser.h -- Query parser for a simple query language
+ * Copyright (C) 2000  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef QUERYPARSER_H
+#define QUERYPARSER_H
+
+#include "UCArray.h"
+#include "MGQuery.h"
+
+// returns NULL if the query could not be parsed
+QueryNode *ParseQuery (const UCArray &queryStr);
+
+#endif
Index: /trunk/mgpp/text/QueryResultsSort.h
===================================================================
--- /trunk/mgpp/text/QueryResultsSort.h	(revision 3365)
+++ /trunk/mgpp/text/QueryResultsSort.h	(revision 3365)
@@ -0,0 +1,140 @@
+/**************************************************************************
+ *
+ * QueryResultsSort.h -- Functions to sort query results and parallel data
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef QUERYRESULTSSORT_H
+#define QUERYRESULTSSORT_H
+
+#include "Terms.h"
+
+
+struct LTRank {
+  bool operator()(const float &f1, const float &f2) const
+    { return (f1 < f2); }
+};
+
+struct GTRank {
+  bool operator()(const float &f1, const float &f2) const
+    { return (f1 > f2); }
+};
+
+
+
+// assumes the rank vector is the same size as the document
+// number vector. assumes the value to be pushed is in
+// docNumHeap + heapSize - 1 and rankHeap + heapSize - 1
+// topIndex can be set
+template <class RandomAccessIterator, class Compare>
+void PushHeap (DocNumArray::iterator docNumHeap,
+	       RandomAccessIterator parallelHeap,
+	       unsigned long heapSize,
+	       const Compare &comp) {
+  unsigned long holeIndex = heapSize-1;
+  unsigned long parent = (holeIndex - 1) / 2;
+  while (holeIndex > 0 && comp (*(parallelHeap+parent),
+				*(parallelHeap+holeIndex))) {
+    swap (*(parallelHeap+holeIndex), *(parallelHeap+parent));
+    swap (*(docNumHeap+holeIndex), *(docNumHeap+parent));
+    holeIndex = parent;
+    parent = (holeIndex - 1) / 2;
+  }
+}
+
+// AdjustHeap is used to add a element to a heap after one
+// element in the heap has been replaced with a new one
+template <class RandomAccessIterator, class Compare>
+void AdjustHeap (DocNumArray::iterator docNumHeap,
+		     RandomAccessIterator parallelHeap,
+		     unsigned long holeIndex,
+		     unsigned long heapSize,
+		     const Compare &comp) {
+  unsigned long secondChild = 2 * holeIndex + 2;
+  while (secondChild < heapSize) {
+    if (comp(*(parallelHeap + secondChild),
+	     *(parallelHeap + (secondChild - 1))))
+      secondChild--;
+    swap (*(parallelHeap+holeIndex), *(parallelHeap+secondChild));
+    swap (*(docNumHeap+holeIndex), *(docNumHeap+secondChild));
+    holeIndex = secondChild;
+    secondChild = 2 * secondChild + 2;
+  }
+  if (secondChild == heapSize) {
+    swap (*(parallelHeap+holeIndex), *(parallelHeap+(secondChild-1)));
+    swap (*(docNumHeap+holeIndex), *(docNumHeap+(secondChild-1)));
+    holeIndex = secondChild - 1;
+  }
+  PushHeap (docNumHeap, parallelHeap, holeIndex+1, comp);
+}
+
+template <class RandomAccessIterator, class Compare>
+void PopHeap (DocNumArray::iterator docNumHeap,
+	      RandomAccessIterator parallelHeap,
+	      unsigned long heapSize,
+	      const Compare &comp) {
+  swap (*(parallelHeap), *(parallelHeap+heapSize-1));
+  swap (*(docNumHeap), *(docNumHeap+heapSize-1));
+  AdjustHeap (docNumHeap, parallelHeap, (unsigned long)0, heapSize-1, comp);
+}
+
+template <class RandomAccessIterator, class Compare>
+void MakeHeap (DocNumArray::iterator docNumHeap,
+	       RandomAccessIterator parallelHeap,
+	       unsigned long size,
+	       const Compare &comp) {
+  if (size < 2) return;
+  
+  unsigned long i = 0;
+  for (i=1; i<=size; i++)
+    PushHeap (docNumHeap, parallelHeap, i, comp);
+}
+
+ 
+template <class RandomAccessIterator, class Compare>
+void SortHeap (DocNumArray::iterator docNumHeap,
+	       RandomAccessIterator parallelHeap,
+	       unsigned long size,
+	       const Compare &comp) {
+  if (size < 2) return;
+
+  unsigned long i;
+  for (i=size; i>1; i--)
+    PopHeap (docNumHeap, parallelHeap, i, comp);
+}
+
+// uses the heap to get the top "size" elements
+template <class RandomAccessIterator, class Compare>
+void SelectAddHeap (DocNumArray::iterator docNumHeap,
+		    RandomAccessIterator parallelHeap,
+		    unsigned long heapSize,
+		    DocNumArray::iterator docNumAdd,
+		    RandomAccessIterator parallelAdd,
+		    unsigned long addSize,
+		    const Compare &comp) {
+  unsigned long i;
+  for (i=0; i<addSize; i++) {
+    if (comp(*(parallelAdd+i), *(parallelHeap))) {
+      swap (*(docNumHeap), *(docNumAdd+i));
+      swap (*(parallelHeap), *(parallelAdd+i));
+      AdjustHeap (docNumHeap, parallelHeap, (unsigned long)0, heapSize, comp);
+    }
+  }
+}
+
+#endif
Index: /trunk/mgpp/text/QueryTester.cpp
===================================================================
--- /trunk/mgpp/text/QueryTester.cpp	(revision 3365)
+++ /trunk/mgpp/text/QueryTester.cpp	(revision 3365)
@@ -0,0 +1,447 @@
+/**************************************************************************
+ *
+ * QueryTester.cpp -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "MGQuery.h"
+
+
+class SetQueryNode : public QueryNode {
+public:
+  QueryResult queryResult;
+  
+  SetQueryNode () {}
+  ~SetQueryNode () {}
+
+  void Calculate (IndexData &/*indexData*/, const QueryInfo &queryInfo,
+		  QueryResult &result) const {
+    result = queryResult;
+    if (!queryInfo.sortByRank && !queryInfo.needRankInfo)
+      result.ranks.erase (result.ranks.begin(), result.ranks.end());
+    if (!queryInfo.needTermFreqs)
+      result.termFreqs.erase (result.termFreqs.begin(),
+			      result.termFreqs.end());
+  }
+  void Free () { queryResult.Clear(); }
+};
+
+
+bool Test1 () {
+  AndQueryNode andNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  RankArray &rankSet1 = setNode1->queryResult.ranks;
+  docSet1.push_back (1);  rankSet1.push_back (0.1);
+  docSet1.push_back (10); rankSet1.push_back (0.2);
+  docSet1.push_back (15); rankSet1.push_back (0.2);
+  docSet1.push_back (18); rankSet1.push_back (0.4);
+  docSet1.push_back (19); rankSet1.push_back (0.5);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  RankArray &rankSet2 = setNode2->queryResult.ranks;
+  docSet2.push_back (2);  rankSet2.push_back (0.1);
+  docSet2.push_back (11); rankSet2.push_back (0.2);
+  docSet2.push_back (12); rankSet2.push_back (0.3);
+  docSet2.push_back (13); rankSet2.push_back (0.4);
+  docSet2.push_back (14); rankSet2.push_back (0.5);
+  docSet2.push_back (15); rankSet2.push_back (0.6);
+  docSet2.push_back (16); rankSet2.push_back (0.7);
+  docSet2.push_back (17); rankSet2.push_back (0.8);
+  docSet2.push_back (19); rankSet2.push_back (0.9);
+  docSet2.push_back (20); rankSet2.push_back (0.1);
+  docSet2.push_back (21); rankSet2.push_back (0.2);
+
+  cout << "\n" << setNode1->queryResult << "AND\n\n"
+       << setNode2->queryResult;
+  
+
+  andNode.leftNode = setNode1;
+  andNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = true;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  andNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  RankArray &rcRankSet = resultCompare.ranks;
+  rcDocSet.push_back (15);  rcRankSet.push_back (0.2+0.6);
+  rcDocSet.push_back (19);  rcRankSet.push_back (1.4);
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test1b () {
+  AndQueryNode andNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  docSet1.push_back (1); 
+  docSet1.push_back (10);
+  docSet1.push_back (15);
+  docSet1.push_back (18);
+  docSet1.push_back (19);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  docSet2.push_back (2); 
+  docSet2.push_back (11);
+  docSet2.push_back (12);
+  docSet2.push_back (13);
+  docSet2.push_back (14);
+  docSet2.push_back (15);
+  docSet2.push_back (16);
+  docSet2.push_back (17);
+  docSet2.push_back (19);
+  docSet2.push_back (20);
+  docSet2.push_back (21);
+
+  cout << "\n" << setNode1->queryResult << "AND\n\n"
+       << setNode2->queryResult;
+  
+
+  andNode.leftNode = setNode1;
+  andNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = false;
+  queryInfo.needRankInfo = false;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  andNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  rcDocSet.push_back (15); 
+  rcDocSet.push_back (19); 
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test2 () {
+  OrQueryNode orNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  RankArray &rankSet1 = setNode1->queryResult.ranks;
+  docSet1.push_back (1);  rankSet1.push_back (0.1);
+  docSet1.push_back (10); rankSet1.push_back (0.2);
+  docSet1.push_back (15); rankSet1.push_back (0.2);
+  docSet1.push_back (18); rankSet1.push_back (0.4);
+  docSet1.push_back (19); rankSet1.push_back (0.5);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  RankArray &rankSet2 = setNode2->queryResult.ranks;
+  docSet2.push_back (2);  rankSet2.push_back (0.1);
+  docSet2.push_back (11); rankSet2.push_back (0.2);
+  docSet2.push_back (12); rankSet2.push_back (0.3);
+  docSet2.push_back (13); rankSet2.push_back (0.4);
+  docSet2.push_back (14); rankSet2.push_back (0.5);
+  docSet2.push_back (15); rankSet2.push_back (0.6);
+  docSet2.push_back (16); rankSet2.push_back (0.7);
+  docSet2.push_back (17); rankSet2.push_back (0.8);
+  docSet2.push_back (19); rankSet2.push_back (0.9);
+  docSet2.push_back (20); rankSet2.push_back (0.1);
+  docSet2.push_back (21); rankSet2.push_back (0.2);
+
+  cout << "\n" << setNode1->queryResult << "OR\n\n"
+       << setNode2->queryResult;
+  
+
+  orNode.leftNode = setNode1;
+  orNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = true;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  orNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  RankArray &rcRankSet = resultCompare.ranks;
+  rcDocSet.push_back (1);  rcRankSet.push_back (0.1);
+  rcDocSet.push_back (2);  rcRankSet.push_back (0.1);
+  rcDocSet.push_back (10); rcRankSet.push_back (0.2);
+  rcDocSet.push_back (11); rcRankSet.push_back (0.2);
+  rcDocSet.push_back (12); rcRankSet.push_back (0.3);
+  rcDocSet.push_back (13); rcRankSet.push_back (0.4);
+  rcDocSet.push_back (14); rcRankSet.push_back (0.5);
+  rcDocSet.push_back (15); rcRankSet.push_back (0.2+0.6);
+  rcDocSet.push_back (16); rcRankSet.push_back (0.7);
+  rcDocSet.push_back (17); rcRankSet.push_back (0.8);
+  rcDocSet.push_back (18); rcRankSet.push_back (0.4);
+  rcDocSet.push_back (19); rcRankSet.push_back (0.9+0.5);
+  rcDocSet.push_back (20); rcRankSet.push_back (0.1);
+  rcDocSet.push_back (21); rcRankSet.push_back (0.2);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test2b () {
+  OrQueryNode orNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  docSet1.push_back (1); 
+  docSet1.push_back (10);
+  docSet1.push_back (15);
+  docSet1.push_back (18);
+  docSet1.push_back (19);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  docSet2.push_back (2);
+  docSet2.push_back (11);
+  docSet2.push_back (12);
+  docSet2.push_back (13);
+  docSet2.push_back (14);
+  docSet2.push_back (15);
+  docSet2.push_back (16);
+  docSet2.push_back (17);
+  docSet2.push_back (19);
+  docSet2.push_back (20);
+  docSet2.push_back (21);
+
+  cout << "\n" << setNode1->queryResult << "OR\n\n"
+       << setNode2->queryResult;
+  
+
+  orNode.leftNode = setNode1;
+  orNode.rightNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = false;
+  queryInfo.needRankInfo = false;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  orNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  rcDocSet.push_back (1);
+  rcDocSet.push_back (2); 
+  rcDocSet.push_back (10);
+  rcDocSet.push_back (11);
+  rcDocSet.push_back (12);
+  rcDocSet.push_back (13);
+  rcDocSet.push_back (14);
+  rcDocSet.push_back (15);
+  rcDocSet.push_back (16);
+  rcDocSet.push_back (17);
+  rcDocSet.push_back (18);
+  rcDocSet.push_back (19);
+  rcDocSet.push_back (20);
+  rcDocSet.push_back (21);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test3 () {
+  NotQueryNode notNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  RankArray &rankSet1 = setNode1->queryResult.ranks;
+  docSet1.push_back (1);  rankSet1.push_back (0.1);
+  docSet1.push_back (10); rankSet1.push_back (0.2);
+  docSet1.push_back (15); rankSet1.push_back (0.2);
+  docSet1.push_back (18); rankSet1.push_back (0.4);
+  docSet1.push_back (19); rankSet1.push_back (0.5);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  RankArray &rankSet2 = setNode2->queryResult.ranks;
+  docSet2.push_back (2);  rankSet2.push_back (0.1);
+  docSet2.push_back (11); rankSet2.push_back (0.2);
+  docSet2.push_back (12); rankSet2.push_back (0.3);
+  docSet2.push_back (13); rankSet2.push_back (0.4);
+  docSet2.push_back (14); rankSet2.push_back (0.5);
+  docSet2.push_back (15); rankSet2.push_back (0.6);
+  docSet2.push_back (16); rankSet2.push_back (0.7);
+  docSet2.push_back (17); rankSet2.push_back (0.8);
+  docSet2.push_back (19); rankSet2.push_back (0.9);
+  docSet2.push_back (20); rankSet2.push_back (0.1);
+  docSet2.push_back (21); rankSet2.push_back (0.2);
+
+  cout << "\n" << setNode1->queryResult << "NOT\n\n"
+       << setNode2->queryResult;
+  
+
+  notNode.queryNode = setNode1;
+  notNode.notNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = true;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  notNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  RankArray &rcRankSet = resultCompare.ranks;
+  rcDocSet.push_back (1);  rcRankSet.push_back (0.1);
+  rcDocSet.push_back (10); rcRankSet.push_back (0.2);
+  rcDocSet.push_back (18); rcRankSet.push_back (0.4);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+bool Test3b () {
+  NotQueryNode notNode;
+
+  SetQueryNode *setNode1 = new SetQueryNode;
+  DocNumArray &docSet1 = setNode1->queryResult.docs;
+  docSet1.push_back (1); 
+  docSet1.push_back (10);
+  docSet1.push_back (15);
+  docSet1.push_back (18);
+  docSet1.push_back (19);
+
+  SetQueryNode *setNode2 = new SetQueryNode;
+  DocNumArray &docSet2 = setNode2->queryResult.docs;
+  docSet2.push_back (2); 
+  docSet2.push_back (11);
+  docSet2.push_back (12);
+  docSet2.push_back (13);
+  docSet2.push_back (14);
+  docSet2.push_back (15);
+  docSet2.push_back (16);
+  docSet2.push_back (17);
+  docSet2.push_back (19);
+  docSet2.push_back (20);
+  docSet2.push_back (21);
+
+  cout << "\n" << setNode1->queryResult << "NOT\n\n"
+       << setNode2->queryResult;
+  
+
+  notNode.queryNode = setNode1;
+  notNode.notNode = setNode2;
+
+  IndexData indexData; // this will not be used anywhere
+  QueryInfo queryInfo;
+  queryInfo.maxDocs = 0;
+  queryInfo.sortByRank = false;
+  queryInfo.needRankInfo = false;
+  queryInfo.needTermFreqs = true;
+  
+  QueryResult result;
+
+  notNode.Calculate(indexData, queryInfo, result);
+
+  cout << "EQUALS\n\n" << result << "\n";
+
+  QueryResult resultCompare;
+  DocNumArray &rcDocSet = resultCompare.docs;
+  rcDocSet.push_back (1); 
+  rcDocSet.push_back (10);
+  rcDocSet.push_back (18);
+  
+
+  if (!(result == resultCompare)) {
+    cout << "Test failed!!!\n";
+    cout << resultCompare;
+    return false;
+  }
+
+  return true;
+}
+
+
+int main () {
+  if (!Test1 () ||
+      !Test2 () ||
+      !Test2b () ||
+      !Test3 () ||
+      !Test3b ()) return 1;
+  
+  return 0;
+}
Index: /trunk/mgpp/text/Queryer.cpp
===================================================================
--- /trunk/mgpp/text/Queryer.cpp	(revision 3365)
+++ /trunk/mgpp/text/Queryer.cpp	(revision 3365)
@@ -0,0 +1,270 @@
+/**************************************************************************
+ *
+ * Queryer.cpp -- simple interactive query program
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "MGQuery.h"
+#include "TextGet.h"
+
+#include "messages.h"
+#include "mg_files.h"
+
+#include "GSDLQueryParser.h"
+
+void printHelp() {
+
+  cout << "commands available are:\n"
+       << "\t.q\t\tquit\n"
+       << "\t.h\t\tprint the help message\n"
+       << "\t.i\t\tchange the search level (enter the new level at the prompt)\n"
+       << "\t.l\t\tchange the result level ( \"\"   \"\"  )\n"
+       << "\t.b\t\tfull text browse (enter a word or fragment at the prompt)\n"
+       << "\t.r0/.r1\t\tranking off/on\n"
+       << "\t.t0/.t1\t\tquery type some/all\n"
+       << "\t.c0/.c1\t\tcasefolding off/on\n"
+       << "\t.s0/.s1\t\tstemming off/on\n"
+       << "\t.o0/.o1\t\tshort output off/on\n\n"
+       << "\t.p\t\tprint a document (enter the docnum at the prompt)\n"
+       << "otherwise just enter a query\n\n";
+
+}
+
+int main (int argc, char **argv) {
+  int ch;
+  char *textfilename = "";
+  char *indexfilename = "";
+  char *basePath = "";
+  
+  opterr = 0;
+  msg_prefix = argv[0];
+  
+  // process the command line arguments
+  while ((ch = getopt (argc, argv, "f:t:d:h")) != -1) {
+    switch (ch) {
+    case 'f':		/* input file */
+      indexfilename = optarg;
+      break;
+    case 't':
+      textfilename = optarg;
+      break;
+    case 'd':
+      basePath = optarg;
+      set_basepath (basePath);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-d directory] -f indexname -t textname\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  if (textfilename[0] == '\0' || indexfilename[0] == '\0') {
+    FatalError (1, "Index and text file names must be specified with -f and -t \n");
+  }
+  
+  // init the text system
+    TextData textData;
+  if (!textData.LoadData (basePath, textfilename)) {
+    FatalError (1, "Couldn't load text information for \"%s\"", textfilename);
+  }
+  
+  // init the query system
+  IndexData indexData;
+  if (!indexData.LoadData (basePath, indexfilename)) {
+    FatalError (1, "Couldn't load index information for \"%s\"", indexfilename);
+  }
+
+  // debug output
+  cerr << "num docs: "<<indexData.bdh.num_docs       
+       << "\nnum frags: "<<indexData.bdh.num_frags
+       << "\nnum words: "<<indexData.bdh.num_words
+       << "\ntotal bytes: "<<indexData.bdh.total_bytes
+       << "\nindex string bytes: "<<indexData.bdh.index_string_bytes
+       << "\nnum levels: "<<indexData.bdh.num_levels<<endl;
+  
+  // do querying
+  QueryInfo queryInfo;
+  SetCStr (queryInfo.docLevel, "Document");
+  queryInfo.maxDocs = 50;
+  queryInfo.sortByRank = true;
+  queryInfo.exactWeights = false;
+  queryInfo.needRankInfo = true;
+  queryInfo.needTermFreqs = true;
+  
+  ExtQueryResult queryResult;
+  char query[2048];
+  UCArray queryArray;
+  QueryNode *queryTree = NULL;
+
+
+  UCArray docLevel;
+  SetCStr(docLevel, "Document");
+
+  UCArray level;
+  UCArrayClear(level);
+  //SetCStr(level, "");
+   
+  int defaultStemMethod = 0; // uncasefolded, unstemmed
+  int defaultBoolCombine = 0; // OR
+  bool shortOutput = false;
+  BrowseQueryNode browseNode;
+  browseNode.startPosition = -10;
+  browseNode.numTerms = 40;
+
+  BrowseQueryResult browseResult;
+
+  while (true) {
+    cout << "> ";
+    cin.getline(query, 2048, '\n');
+    SetCStr (queryArray, query);
+
+    // check for commands
+    if (queryArray.size() >= 2 && queryArray[0] == '.') {
+      if (queryArray[1] == 'q') break;  // quit
+
+      if (queryArray[1] == 'h') { // help
+	printHelp();
+      }
+      if (queryArray[1] == 'i') {
+	cout << "current index="<< queryInfo.docLevel << "\nchange to index:";
+	cin >> query;
+	UCArrayClear(queryInfo.docLevel);
+	SetCStr(queryInfo.docLevel, query);
+	cout << "index set to " << queryInfo.docLevel <<"\n";
+	cin.getline(query, 2048, '\n');
+      }
+      if (queryArray[1] == 'l') {
+	cout << "current level="<< level << "\nchange to level:";
+	cin >> query;
+	UCArrayClear(level);
+	SetCStr(level, query);
+	cout << "level set to " << level <<"\n";
+	cin.getline(query, 2048, '\n');
+      }
+
+
+      else if (queryArray[1] == 'p') {
+	// print
+	UCArray docText;
+	unsigned long docNum = 0;
+	cin >> docNum;
+	cin.getline(query, 2048, '\n'); // eat up return
+	
+	if (!GetDocText (textData, queryInfo.docLevel, docNum, docText)) {
+	  FatalError (1, "Error while trying to get document %u", docNum);
+	}
+	
+	cout << docText << "\n";
+      }
+      else if (queryArray[1] == 't') { // query type - all/some
+	if (queryArray[2] == '1') defaultBoolCombine = 1;
+	else if (queryArray[2] == '0') defaultBoolCombine = 0;
+	else {
+	  cout << "Error: please enter .t0 (some) or .t1 (all)\n";
+	}
+      }
+      else if (queryArray[1] == 'r') { // ranking - on/off
+	if (queryArray[2] == '1') queryInfo.sortByRank = true;
+	else if (queryArray[2] == '0') queryInfo.sortByRank = false;
+	else {
+	  cout << "Error: please enter .r0 (non-ranked) or .r1 (ranked)\n";
+	}
+      }
+      else if (queryArray[1] == 'c') { // casefolding - on/off
+	if (queryArray[2] == '1') defaultStemMethod |= 1;
+	else if (queryArray[2] == '0') defaultStemMethod &= 0xe;
+	else {
+	  cout << "Error: please enter .c0 (case sensitive) or .c1 (casefolded)\n";
+	}
+      }
+      else if (queryArray[1] == 's') { // stemming - on/off
+	if (queryArray[2] == '1') defaultStemMethod |=2;
+	else if (queryArray[2] == '0') defaultStemMethod &=0xd;
+	else {
+	  cout << "Error: please enter .s0 (unstemmed) or .s1 (stemmed)\n";
+	}
+      }      
+      else if (queryArray[1] == 'o') { // output - short/long
+	if (queryArray[2] == '1') shortOutput = true;
+	else if (queryArray[2] == '0') shortOutput = false;
+	else {
+	  cout << "Error: please enter .o0 (long output) or .o1 (short output)\n";
+	}
+      }
+      else if (queryArray[1] == 'b') {
+	// full text browse
+	cout<<"enter a few letters to start browsing from:";
+	cin>>query;
+	UCArrayClear(browseNode.term);
+	SetCStr(browseNode.term, query);
+	cin.getline(query, 2048, '\n'); // get rest of line
+
+	// print the query
+	PrintNode (cout, &browseNode);
+
+	MGBrowseQuery(indexData, docLevel, browseNode, browseResult);
+	cout << browseResult;
+	cout << "\n";
+
+      }
+      else { // bad option
+	cout << "bad command\n\n";
+	printHelp();
+      }
+    } // if a .x query
+    else {
+      // regular query
+      queryTree = ParseQuery (queryArray, defaultBoolCombine, defaultStemMethod);
+
+      // print the query
+      PrintNode (cout, queryTree);
+
+      MGQuery (indexData, queryInfo, queryTree, queryResult, level);
+      if (shortOutput) {
+	queryResult.printShort(cout);
+	cout << "\n";
+      }else {
+	cout << queryResult;
+	cout << "\n";
+      }
+      // delete the query
+      if (queryTree != NULL) delete queryTree;
+      queryTree = NULL;
+    }
+  }
+
+
+  // clean up, everybody clean up
+    textData.UnloadData ();
+  indexData.UnloadData ();
+
+  return (0);
+}
+
+
Index: /trunk/mgpp/text/TagInfo.cpp
===================================================================
--- /trunk/mgpp/text/TagInfo.cpp	(revision 3365)
+++ /trunk/mgpp/text/TagInfo.cpp	(revision 3365)
@@ -0,0 +1,74 @@
+/**************************************************************************
+ *
+ * TagInfo.cpp -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "TagInfo.h"
+
+
+void TagInfo::SetDocTag (const char *cStr) {
+  SetCStr (docTag, cStr);
+}
+
+void TagInfo::SetIndexLevel (const char *cStr) {
+  SetCStr (indexLevel, cStr);
+}
+void TagInfo::AddLevelTag (const char *cStr) {
+  // convert the string
+  UCArray cArr;
+  SetCStr (cArr, cStr);
+
+  // insert the tag
+  levelTags.insert (cArr);
+}
+
+
+TagInfo::TagInfo () {
+  Clear ();
+}
+
+void TagInfo::Clear () {
+  docTag.erase (docTag.begin(), docTag.end());
+  indexLevel.erase (indexLevel.begin(), indexLevel.end());
+  levelTags.erase (levelTags.begin(), levelTags.end());
+}
+
+ostream &operator<<(ostream &s, const TagInfo &t) {
+  // output docTag
+  s << "docTag: \"" << t.docTag << "\"\n";
+
+  // output indexLevel
+  s << "indexLevel: \"" << t.indexLevel << "\"\n";
+
+  // output levelTags
+  s << "levelTags: ";
+
+  UCArraySet::const_iterator here = t.levelTags.begin();
+  UCArraySet::const_iterator end = t.levelTags.end();
+  while (here != end) {
+    s << "\"" << (*here) << "\"";
+    here++;
+    if (here != end) s << ", ";
+  }
+
+  s << "\n";
+  
+  return s;
+}
+
Index: /trunk/mgpp/text/TagInfo.h
===================================================================
--- /trunk/mgpp/text/TagInfo.h	(revision 3365)
+++ /trunk/mgpp/text/TagInfo.h	(revision 3365)
@@ -0,0 +1,46 @@
+/**************************************************************************
+ *
+ * TagInfo.h -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TAGINFO_H
+#define TAGINFO_H
+
+#include "UCArray.h"
+
+
+class TagInfo {
+public:
+  UCArray docTag;
+  UCArray indexLevel;    // empty for word level index
+  UCArraySet levelTags;
+
+  void SetDocTag (const char *cStr);
+  void SetIndexLevel (const char *cStr);
+  void AddLevelTag (const char *cStr);
+  
+  TagInfo ();
+  void Clear ();
+};
+
+// stream operator to print TagInfo
+ostream &operator<<(ostream &s, const TagInfo &t);
+
+
+#endif
Index: /trunk/mgpp/text/Terms.cpp
===================================================================
--- /trunk/mgpp/text/Terms.cpp	(revision 3365)
+++ /trunk/mgpp/text/Terms.cpp	(revision 3365)
@@ -0,0 +1,797 @@
+/**************************************************************************
+ *
+ * Terms.cpp -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "Terms.h"
+#include "words.h"
+#include "stemmer.h"
+#include "bitio_gen.h"
+#include "bitio_m_stdio.h"
+
+void QueryInfo::Clear () {
+  UCArrayClear (docLevel);
+  maxDocs = 0;
+  sortByRank = true;
+  exactWeights = false;
+  needRankInfo = false;
+  needTermFreqs = false;
+}
+
+
+
+void TermFreqData::Clear () {
+  UCArrayClear (tag);
+  UCArrayClear (term);
+  equivTerms.erase(equivTerms.begin(), equivTerms.end());
+  stemMethod = 0;
+  matchDocs = 0;
+  termFreq = 0;
+}
+
+ostream &operator<< (ostream &s, const TermFreqData &t) {
+  s << "<" << t.tag << ">\"" << t.term << "\"stem("
+    << t.stemMethod << ")equiv terms(";
+  
+  unsigned long i;
+  for (i=0; i<t.equivTerms.size(); i++) {
+    s << t.equivTerms[i] << ", ";
+  }
+  s <<")docs(" << t.matchDocs << ")"
+    << "count("<<t.termFreq<<")";
+  return s;
+}
+
+bool operator== (const TermFreqData &t1, const TermFreqData &t2) {
+  return ((t1.tag == t2.tag) &&
+	  (t1.term == t2.term) &&
+	  (t1.stemMethod == t2.stemMethod) &&
+	  (t1.equivTerms == t2.equivTerms) &&
+	  (t1.matchDocs == t2.matchDocs) &&
+	  (t1.termFreq == t2.termFreq));
+}
+
+
+void QueryResult::Clear () {
+  docs.erase (docs.begin(), docs.end());
+  ranks.erase (ranks.begin(), ranks.end());
+  termFreqs.erase (termFreqs.begin(), termFreqs.end());
+  actualNumDocs = 0;
+}
+
+QueryResult::QueryResult () {
+  Clear ();
+}
+
+void QueryResult::printShort(ostream &s) {
+
+  s << "termFreqs: ";
+  for (unsigned long i=0; i<termFreqs.size(); i++)
+    s << termFreqs[i] << ", ";
+  
+  s << "\nactual number of docs found: " << actualNumDocs;
+  s << "\n\n";
+
+}
+
+
+ostream &operator<< (ostream &s, const QueryResult &r) {
+  s << "docs: ";
+  unsigned long i;
+  for (i=0; i<r.docs.size(); i++)
+    s << r.docs[i] << ", ";
+  
+  s << "\nranks: ";
+  for (i=0; i<r.ranks.size(); i++)
+    s << r.ranks[i] << ", ";
+
+  s << "\ntermFreqs: ";
+  for (i=0; i<r.termFreqs.size(); i++)
+    s << r.termFreqs[i] << ", ";
+
+  s << "\nactual number of docs found: " << r.actualNumDocs;
+  s << "\n\n";
+
+  return s;
+}
+
+
+bool operator== (const QueryResult &r1, const QueryResult &r2) {
+  return ((r1.docs == r2.docs) &&
+	  (r1.ranks == r2.ranks) &&
+	  (r1.termFreqs == r2.termFreqs) &&
+	  (r1.actualNumDocs == r2.actualNumDocs));
+}
+
+//---------------------------------------------------
+// new ExtQueryResult stuff
+void ExtQueryResult::Clear () {
+  docs.erase (docs.begin(), docs.end());
+  levels.erase (levels.begin(), levels.end());
+  ranks.erase (ranks.begin(), ranks.end());
+  termFreqs.erase (termFreqs.begin(), termFreqs.end());
+  actualNumDocs = 0;
+}
+
+ExtQueryResult::ExtQueryResult () {
+  Clear ();
+}
+
+ostream &operator<< (ostream &s, const ExtQueryResult &r) {
+  s << "docs: ";
+  unsigned long i;
+  for (i=0; i<r.docs.size(); i++)
+    s << r.docs[i] << ", ";
+
+  s << "\nlevels: ";
+  for (i=0; i<r.levels.size(); i++)
+    s << r.levels[i] << ", ";
+
+  
+  s << "\nranks: ";
+  for (i=0; i<r.ranks.size(); i++)
+    s << r.ranks[i] << ", ";
+
+  s << "\ntermFreqs: ";
+  for (i=0; i<r.termFreqs.size(); i++)
+    s << r.termFreqs[i] << ", ";
+  s << "\nactual number of docs found: " << r.actualNumDocs;
+  s << "\n\n";
+
+  return s;
+}
+
+
+bool operator== (const ExtQueryResult &r1, const ExtQueryResult &r2) {
+  return ((r1.docs == r2.docs) &&
+	  (r1.levels == r2.levels) &&
+	  (r1.ranks == r2.ranks) &&
+	  (r1.termFreqs == r2.termFreqs) &&
+	  (r1.actualNumDocs == r2.actualNumDocs));
+}
+
+//-------------------------------------------------------
+// new BrowseQueryResult stuff
+void BrowseQueryResult::Clear () {
+  termFreqs.erase (termFreqs.begin(), termFreqs.end());
+}
+
+BrowseQueryResult::BrowseQueryResult () {
+  Clear ();
+}
+
+
+
+ostream &operator<< (ostream &s, const BrowseQueryResult &r) {
+  s << "terms: ";
+  unsigned long i;
+  for (i=0; i<r.termFreqs.size(); i++)
+    s << r.termFreqs[i] << ", ";
+    s << "\n\n";
+  return s;
+}
+
+
+bool operator== (const BrowseQueryResult &r1, const BrowseQueryResult &r2) {
+  return ((r1.termFreqs == r2.termFreqs));
+	 
+}
+
+
+
+
+//--------------------------------------
+void FragData::Clear () {
+  matchDocs = 0;
+  fragNums.erase (fragNums.begin(), fragNums.end());
+  fragFreqs.erase (fragFreqs.begin(), fragFreqs.end());
+}
+
+
+
+
+void FindWordNumbers (IndexData &indexData,
+		      const UCArray &term,
+		      unsigned long stemMethod,
+		      vector<unsigned long> &equivWords) {
+  equivWords.erase (equivWords.begin(), equivWords.end());
+  
+  if (stemMethod == 0) {
+    // don't need to stem the word,
+    // find the word number for this term
+    unsigned long wordElNum = 0;
+    unsigned long numLevels = indexData.bdh.num_levels;
+    word_block_dict_el wordDictEl;
+    wordDictEl.SetNumLevels (numLevels);
+    if (SearchWordBlockDictEl (indexData.dictFile, indexData.biWords,
+			       indexData.bdh.entries_per_wblk,
+			       indexData.bdh.word_dict_size,
+			       numLevels, term, wordDictEl, wordElNum))
+      equivWords.push_back (wordElNum);
+    
+    return;
+    
+  }
+
+
+  // need to stem this word and find it in the blocked stem index
+  
+  unsigned char  mgWord[MAXSTEMLEN + 1];
+  UCArray stemTerm;
+  unsigned long stemmerNum = 0;
+  
+  if (stemMethod == 1) stemmerNum = indexData.sih1.stemmer_num;
+  else if (stemMethod == 2) stemmerNum = indexData.sih2.stemmer_num;
+  else if (stemMethod == 3) stemmerNum = indexData.sih3.stemmer_num;
+  
+  
+  // convert the word to an "mg word"
+  mgWord[0] = term.size();
+  memcpy ((char *)&mgWord[1], &(term[0]), term.size());
+  
+  // stem the word
+  stemmer (stemMethod, stemmerNum, mgWord);
+
+  // convert the result back to a UCArray
+  stemTerm.insert (stemTerm.end(), &mgWord[1], &mgWord[1] + mgWord[0]);
+
+  // need to look up this term in the appropriate dictionary
+  stem_block_dict_el stemDictEl;
+  unsigned long stemElNum;
+  bool result = false;
+  if (stemMethod == 1) {
+    result = SearchStemBlockDictEl (indexData.stem1File,
+			   indexData.sii1,
+			   indexData.sih1.entries_per_block,
+			   indexData.sih1.dict_size,
+			   stemTerm,
+			   stemDictEl,
+			   stemElNum);
+
+  } else if (stemMethod == 2) {
+    result = SearchStemBlockDictEl (indexData.stem2File,
+			   indexData.sii2,
+			   indexData.sih2.entries_per_block,
+			   indexData.sih2.dict_size,
+			   stemTerm,
+			   stemDictEl,
+			   stemElNum);
+
+  } else if (stemMethod == 3) {
+    result = SearchStemBlockDictEl (indexData.stem3File,
+			   indexData.sii3,
+			   indexData.sih3.entries_per_block,
+			   indexData.sih3.dict_size,
+			   stemTerm,
+			   stemDictEl,
+			   stemElNum);
+  }
+
+  if (result) {
+    equivWords = stemDictEl.equivWords;  
+  }
+}
+
+
+
+void ReadTermFragData (IndexData &indexData,
+		       bool needFragFreqs,
+		       unsigned long termNum,
+		       FragData &fragData,
+		       FragRangeArray *fragLimits,
+		       UCArray & termWord) {
+  fragData.Clear();
+
+  // look up the word in the dictionary
+  unsigned long numLevels = indexData.bdh.num_levels;
+  word_block_dict_el wordDictEl;
+  wordDictEl.SetNumLevels (numLevels);
+  if (!SearchWordBlockDictElNum (indexData.dictFile,
+				 indexData.biWords,
+				 indexData.bdh.entries_per_wblk,
+				 indexData.bdh.word_dict_size,
+				 numLevels,
+				 termNum, wordDictEl))
+    return; // nothing more to do
+
+  fragData.matchDocs = wordDictEl.levelFreqs[indexData.curLevelNum];
+  termWord = wordDictEl.el;
+  // seek to the appropriate place in the inverted file
+  fseek (indexData.invfFile, wordDictEl.invf_ptr, SEEK_SET);
+  stdio_bitio_buffer buffer (indexData.invfFile);
+    
+  unsigned long B = BIO_Bblock_Init (indexData.bdh.num_frags,
+				     wordDictEl.frag_occur);
+  unsigned long fragNum = 0;
+  unsigned long termFreq = 0;
+
+  unsigned long fragLimitI = 0;
+  unsigned long i;
+  for (i=0; i<wordDictEl.frag_occur; i++) {
+    fragNum += buffer.bblock_decode (B, NULL);
+    if (!indexData.ifh.word_level_index) termFreq = buffer.gamma_decode (NULL);
+    else termFreq = 1;
+
+    // get the right fragment range
+    if (fragLimits != NULL) {
+      while (fragLimitI+1 < (*fragLimits).size() &&
+	     fragNum > (*fragLimits)[fragLimitI+1].rangeStart) {
+	fragLimitI++;
+      }
+    }
+
+    // add the entry if it is within the limits
+    if ((fragLimits == NULL) ||
+	(fragLimitI < (*fragLimits).size() &&
+	 fragNum > (*fragLimits)[fragLimitI].rangeStart &&
+	 fragNum <= (*fragLimits)[fragLimitI].rangeEnd)) {
+      fragData.fragNums.push_back (fragNum);
+      if (needFragFreqs)
+	fragData.fragFreqs.push_back (termFreq);
+    }
+  }
+
+  buffer.done();
+}
+
+
+void CombineFragData (bool needFragFreqs,
+		      const FragData &f1,
+		      const FragData &f2,
+		      FragData &outFragData) {
+  outFragData.Clear();
+
+  // the new number of matching documents is the maximum
+  // of the two input matching number of documents -- it
+  // is assumed that these are at the same document level
+  outFragData.matchDocs = (f1.matchDocs > f2.matchDocs) ?
+    f1.matchDocs : f2.matchDocs;
+
+  // do or
+  unsigned long f1I = 0, f1Size = f1.fragNums.size();
+  unsigned long f2I = 0, f2Size = f2.fragNums.size();
+  while (f1I < f1Size || f2I < f2Size) {
+    if (f2I < f2Size &&
+	(f1I >= f1Size ||
+	 f1.fragNums[f1I] > f2.fragNums[f2I])) {
+      // output f2I
+      outFragData.fragNums.push_back (f2.fragNums[f2I]);
+      if (needFragFreqs)
+	outFragData.fragFreqs.push_back (f2.fragFreqs[f2I]);
+      f2I++;
+      
+    } else if (f1I < f1Size &&
+	       (f2I >= f2Size ||
+		f1.fragNums[f1I] < f2.fragNums[f2I])) {
+      // output f1I
+      outFragData.fragNums.push_back (f1.fragNums[f1I]);
+      if (needFragFreqs)
+	outFragData.fragFreqs.push_back (f1.fragFreqs[f1I]);
+      f1I++;
+      
+    } else {
+      // must be equal combine f1I and f2I
+      outFragData.fragNums.push_back (f1.fragNums[f1I]);
+      if (needFragFreqs)
+	outFragData.fragFreqs.push_back (f1.fragFreqs[f1I]+f2.fragFreqs[f2I]);
+      f1I++;
+      f2I++;
+    }
+  }
+}
+
+
+void AndCombineFragData (bool needFragFreqs,
+			 FragData &fragData,
+			 const FragData &comFragData,
+			 signed long startRange,
+			 signed long endRange,
+			 const FragRangeArray *fragLimits) {
+  // sanity check on range
+  if (startRange > endRange) {
+    signed long temp = endRange;
+    endRange = startRange;
+    startRange = temp;
+  }
+
+  // get min matchdocs
+  if (comFragData.matchDocs < fragData.matchDocs)
+    fragData.matchDocs = comFragData.matchDocs;
+  
+  unsigned long fragDataI = 0;
+  unsigned long fragDataSize = fragData.fragNums.size();
+  unsigned long comFragDataI = 0;
+  unsigned long comFragDataSize = comFragData.fragNums.size();
+  unsigned long fragLimitI = 0;
+  unsigned long fragLimitSize = (fragLimits==NULL) ? 0 : (*fragLimits).size();
+  unsigned long outI = 0;
+
+  while (fragDataI < fragDataSize &&
+	 comFragDataI < comFragDataSize) {
+    signed long fragNum = (signed long)fragData.fragNums[fragDataI];
+    signed long comFragNum = (signed long)comFragData.fragNums[comFragDataI];
+    
+    // go to the right fragment limit (for the com frag)
+    if (fragLimits != NULL) {
+      while (fragLimitI+1 < fragLimitSize &&
+	     comFragNum > (signed long)(*fragLimits)[fragLimitI+1].rangeStart) {
+	fragLimitI++;
+      }
+    }
+
+    if (fragNum <= comFragNum+startRange ||
+	(fragLimits!=NULL &&
+	 fragNum<=(signed long)(*fragLimits)[fragLimitI].rangeStart)) {
+      fragDataI++;
+      
+    } else if (fragNum > comFragNum+endRange ||
+	       (fragLimits!=NULL &&
+		fragNum>(signed long)(*fragLimits)[fragLimitI].rangeEnd)) {
+      comFragDataI++;
+      
+    } else {
+      // equal and within tag
+      fragData.fragNums[outI] = comFragNum;
+      if (needFragFreqs) {
+	fragData.fragFreqs[outI] =
+	  (fragData.fragFreqs[fragDataI] < comFragData.fragFreqs[comFragDataI]) ?
+	  fragData.fragFreqs[fragDataI] : comFragData.fragFreqs[comFragDataI];
+      }
+      fragDataI++;
+      comFragDataI++;
+      outI++;
+    }
+  }
+
+  // erase unused part of fragData
+  fragData.fragNums.erase (fragData.fragNums.begin()+outI,
+			   fragData.fragNums.end());
+  if (needFragFreqs)
+    fragData.fragFreqs.erase (fragData.fragFreqs.begin()+outI,
+			      fragData.fragFreqs.end());
+  else
+    fragData.fragFreqs.erase (fragData.fragFreqs.begin(),
+			      fragData.fragFreqs.end());
+}
+
+
+void FragsToQueryResult (IndexData &indexData,
+			 const QueryInfo &queryInfo,
+			 const FragData &termData,
+			 const UCArray &tag,
+			 const UCArray &term,
+			 unsigned long stemMethod,
+			 unsigned long termWeight,
+			 UCArrayVector &equivTerms,
+			 QueryResult &result) {
+  bool needRanks = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  
+  result.Clear();
+
+  // log (N / ft)
+  unsigned long N = indexData.levels.levelInfo[indexData.curLevel].numEntries;
+  float wordLog = log((double)N / (double)termData.matchDocs);
+
+  // Wqt = fqt * log (N / ft)
+  // note: terms are allowed to have a weight of zero so
+  // they can be excluded from the ranking
+  float Wqt = termWeight * wordLog;
+
+  // Wdt = fdt * log (N / ft)
+  float Wdt;
+  
+  unsigned long termDataI = 0;
+  unsigned long termDataSize = termData.fragNums.size();
+  unsigned long levelDocNum = 0;
+  
+  unsigned long termDocFreq = 0;
+  unsigned long lastLevelDocNum = 0;
+  unsigned long overallwordfreq = 0;
+  
+  while (termDataI < termDataSize) {
+    if (indexData.levelConverter.FragToLevel (termData.fragNums[termDataI],
+					      levelDocNum)) {
+      if (levelDocNum != lastLevelDocNum) {
+	if (lastLevelDocNum > 0) {
+	  // add this doc information
+	  if (needRanks) {
+	    Wdt = termDocFreq * wordLog;
+	    result.ranks.push_back (Wqt * Wdt);
+	  }
+	  result.docs.push_back (lastLevelDocNum);
+	}
+	
+	lastLevelDocNum = levelDocNum;
+	termDocFreq = 0;
+      }
+
+      if (needRanks){
+	termDocFreq += termData.fragFreqs[termDataI];
+	overallwordfreq += termData.fragFreqs[termDataI];
+      }
+    }
+    termDataI++;
+  }
+
+  if (lastLevelDocNum > 0) {
+    // add the last document information
+    if (needRanks) {
+      Wdt = termDocFreq * wordLog;
+      result.ranks.push_back (Wqt * Wdt);
+    }
+    result.docs.push_back (lastLevelDocNum);
+  }
+
+  // add the term frequency information
+  if (queryInfo.needTermFreqs) {
+    TermFreqData termFreqData;
+    termFreqData.tag = tag;
+    termFreqData.term = term;
+    termFreqData.stemMethod = stemMethod;
+    termFreqData.equivTerms = equivTerms;
+    termFreqData.matchDocs = termData.matchDocs;
+    termFreqData.termFreq = overallwordfreq; // will be zero if needRankInfo 
+                                              //not true
+    result.termFreqs.push_back (termFreqData);
+  }
+}
+
+void AndFragsToQueryResult (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    const UCArray &tag,
+			    const UCArray &term,
+			    unsigned long stemMethod,
+			    unsigned long termWeight,
+			    UCArrayVector &equivTerms,
+			    QueryResult &result) {
+  bool needRanks = (queryInfo.sortByRank || queryInfo.needRankInfo);
+  
+  // log (N / ft)
+  float wordLog =
+    log((double)indexData.levels.levelInfo[indexData.curLevel].numEntries/
+	(double)termData.matchDocs);
+
+  // Wqt = fqt * log (N / ft)
+  // note: terms are allowed to have a weight of zero so
+  // they can be excluded from the ranking
+  float Wqt = termWeight * wordLog;
+
+  // Wdt = fdt * log (N / ft)
+  float Wdt;
+  
+  unsigned long termDataI = 0;
+  unsigned long termDataSize = termData.fragNums.size();
+  unsigned long levelDocNum = 0;
+  
+  unsigned long termDocFreq = 0;
+  unsigned long lastLevelDocNum = 0;
+  unsigned long overallwordfreq = 0;
+  unsigned long resultI = 0;
+  unsigned long resultSize = result.docs.size();
+  unsigned long resultOutI = 0;
+  
+  
+  while (termDataI < termDataSize) {
+    if (indexData.levelConverter.FragToLevel (termData.fragNums[termDataI],
+					      levelDocNum)) {
+      if (levelDocNum != lastLevelDocNum) {
+	if (lastLevelDocNum > 0) {
+	  // add this doc information
+	  Wdt = termDocFreq * wordLog;
+	  
+	  // find this document number
+	  while (resultI < resultSize &&
+		 result.docs[resultI] < lastLevelDocNum)
+	    resultI++;
+	  
+	  // store the result
+	  if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+	    result.docs[resultOutI] = lastLevelDocNum;
+	    if (needRanks)
+	      result.ranks[resultOutI] = result.ranks[resultI] + Wqt * Wdt;
+	    resultI++;
+	    resultOutI++;
+	  }
+	}
+	
+	lastLevelDocNum = levelDocNum;
+	termDocFreq = 0;
+      }
+
+      if (needRanks)
+	termDocFreq += termData.fragFreqs[termDataI];
+     overallwordfreq += termData.fragFreqs[termDataI]; 
+    }
+    
+    termDataI++;
+  } // while
+
+  if (lastLevelDocNum > 0) {
+    // add the last document information
+    Wdt = termDocFreq * wordLog;
+
+    // find this document number
+    while (resultI < resultSize &&
+	   result.docs[resultI] < lastLevelDocNum)
+      resultI++;
+    
+    // store the result
+    if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+      result.docs[resultOutI] = lastLevelDocNum;
+      if (needRanks)
+	result.ranks[resultOutI] = result.ranks[resultI] + Wqt * Wdt;
+      resultI++;
+      resultOutI++;
+    }
+  }
+
+  // remove unneeded entries
+  result.docs.erase (result.docs.begin()+resultOutI, result.docs.end());
+  if (needRanks)
+    result.ranks.erase (result.ranks.begin()+resultOutI, result.ranks.end());
+  else
+    result.ranks.erase (result.ranks.begin(), result.ranks.end());
+  
+  // add the term frequency information
+  if (queryInfo.needTermFreqs) {
+    TermFreqData termFreqData;
+    termFreqData.tag = tag;
+    termFreqData.term = term;
+    termFreqData.stemMethod = stemMethod;
+    termFreqData.equivTerms = equivTerms;
+    termFreqData.matchDocs = termData.matchDocs;
+    termFreqData.termFreq = overallwordfreq;
+    result.termFreqs.push_back (termFreqData);
+  }
+}
+
+
+void RemoveUnwantedResults (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    QueryResult &result) {
+  bool needRanks = (queryInfo.sortByRank || queryInfo.needRankInfo);
+
+  unsigned long termDataI = 0;
+  unsigned long termDataSize = termData.fragNums.size();
+  unsigned long levelDocNum = 0;
+  
+  unsigned long lastLevelDocNum = 0;
+
+  unsigned long resultI = 0;
+  unsigned long resultSize = result.docs.size();
+  unsigned long resultOutI = 0;
+  
+  while (termDataI < termDataSize) {
+    if (indexData.levelConverter.FragToLevel (termData.fragNums[termDataI],
+					      levelDocNum)) {
+      if (levelDocNum != lastLevelDocNum) {
+	if (lastLevelDocNum > 0) {
+	  // find this document number
+	  while (resultI < resultSize &&
+		 result.docs[resultI] < lastLevelDocNum)
+	    resultI++;
+	  
+	  // store the result
+	  if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+	    result.docs[resultOutI] = lastLevelDocNum;
+	    if (needRanks)
+	      result.ranks[resultOutI] = result.ranks[resultI];
+	    resultI++;
+	  resultOutI++;
+	  }
+	}
+	
+	lastLevelDocNum = levelDocNum;
+      }
+    }
+    
+    termDataI++;
+  }
+
+  if (lastLevelDocNum > 0) {
+    // find this document number
+    while (resultI < resultSize &&
+	   result.docs[resultI] < lastLevelDocNum)
+      resultI++;
+    
+    // store the result
+    if (resultI < resultSize && result.docs[resultI] == lastLevelDocNum) {
+      result.docs[resultOutI] = lastLevelDocNum;
+      if (needRanks) 
+	result.ranks[resultOutI] = result.ranks[resultI];
+      resultI++;
+      resultOutI++;
+    }
+  }
+
+  // remove unneeded entries
+  result.docs.erase (result.docs.begin()+resultOutI, result.docs.end());
+  if (needRanks)
+    result.ranks.erase (result.ranks.begin()+resultOutI, result.ranks.end());
+  else
+    result.ranks.erase (result.ranks.begin(), result.ranks.end());
+}
+
+
+
+//--------------------------------------------------------------
+// functions to support full text browse
+
+void FindNearestWordNumber (IndexData &indexData,
+			    const UCArray &term,
+			    unsigned long &number) {
+
+    // find the word number for this term
+    unsigned long wordElNum = 0;
+    unsigned long numLevels = indexData.bdh.num_levels;
+    word_block_dict_el wordDictEl;
+    wordDictEl.SetNumLevels (numLevels);
+    if (NearestSearchWordBlockDictEl (indexData.dictFile, indexData.biWords,
+				      indexData.bdh.entries_per_wblk,
+				      indexData.bdh.word_dict_size,
+				      numLevels, term, wordDictEl, wordElNum))
+      number = wordElNum;
+
+}
+
+void GetTermList(IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 TermFreqArray &terms) {
+
+  word_block_dict_el_array wordBlocks; // = new word_block_dict_el_array();
+  TermFreqData termdata;
+
+  terms.erase(terms.begin(), terms.end());
+
+  SearchWordBlockDictElNumRange (indexData.dictFile, indexData.biWords,
+				 indexData.bdh.entries_per_wblk,
+				 indexData.bdh.word_dict_size,
+				 indexData.bdh.num_levels, startTerm,
+				 numTerms, wordBlocks);
+
+  word_block_dict_el_array::iterator here = wordBlocks.begin();
+  word_block_dict_el_array::iterator end = wordBlocks.end();
+
+  while (here != end) {
+    termdata.Clear();
+    termdata.term = (*here).el;
+    termdata.termFreq = (*here).freq;
+    terms.push_back(termdata);
+    here++;
+  }
+
+}
+
+void GetTermList(IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 UCArrayVector &terms) {
+
+  
+  
+  SearchWordBlockDictElNumRange (indexData.dictFile, indexData.biWords,
+				 indexData.bdh.entries_per_wblk,
+				 indexData.bdh.word_dict_size,
+				 indexData.bdh.num_levels, startTerm,
+				 numTerms, terms);
+
+}
Index: /trunk/mgpp/text/Terms.h
===================================================================
--- /trunk/mgpp/text/Terms.h	(revision 3365)
+++ /trunk/mgpp/text/Terms.h	(revision 3365)
@@ -0,0 +1,215 @@
+/**************************************************************************
+ *
+ * Terms.h -- Query related functions
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TERMS_H
+#define TERMS_H
+
+#include "IndexData.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\iostream>
+#elif defined(GSDL_USE_IOS_H)
+#  include <iostream.h>
+#else
+#  include <iostream>
+#endif
+
+
+class QueryInfo {
+public:
+  // general query information
+  UCArray docLevel;
+  unsigned long maxDocs; // 0 = all
+  bool sortByRank;
+  bool exactWeights;
+
+  // information need to return
+  bool needRankInfo;
+  bool needTermFreqs;
+
+  void Clear ();
+  QueryInfo () { Clear (); }
+};
+
+
+class TermFreqData {
+public:
+  UCArray tag;  // level tag or query tag
+  UCArray term; // unstemmed term
+  int stemMethod;
+  UCArrayVector equivTerms; // the stemmed and casefolded variants of the term
+  unsigned long matchDocs; // tf for level - num levels
+			   // containing this term
+  unsigned long termFreq;  // overall term freq - num words that
+				// are this term
+  void Clear ();
+  TermFreqData () { Clear (); }
+};
+
+ostream &operator<< (ostream &s, const TermFreqData &t);
+bool operator== (const TermFreqData &t1, const TermFreqData &t2);
+
+typedef vector<TermFreqData> TermFreqArray;
+
+
+typedef vector<float> RankArray;
+typedef vector<unsigned long> DocNumArray;
+
+class QueryResult {
+public:
+  DocNumArray docs;
+  RankArray ranks; // used for accumulators during query
+  
+  TermFreqArray termFreqs;
+
+  unsigned long actualNumDocs;
+  void Clear ();
+  QueryResult ();
+  void printShort(ostream &s);
+};
+
+ostream &operator<< (ostream &s, const QueryResult &r);
+bool operator== (const QueryResult &r1, const QueryResult &r2);
+
+typedef vector<unsigned long> FragNumArray;
+typedef vector<unsigned long> FragFreqArray;
+
+class FragData {
+public:
+  unsigned long matchDocs;    // ft for level
+  FragNumArray fragNums;
+  FragFreqArray fragFreqs;
+
+  void Clear ();
+  FragData () { Clear (); }
+};
+
+
+class FragRange {
+public:
+  unsigned long rangeStart;
+  unsigned long rangeEnd;
+
+  void Clear () { rangeStart = rangeEnd = 0; }
+  FragRange () { Clear (); }
+};
+
+typedef vector<FragRange> FragRangeArray;
+
+
+
+void FindWordNumbers (IndexData &indexData,
+		      const UCArray &term,
+		      unsigned long stemMethod,
+		      vector<unsigned long> &equivWords);
+
+void ReadTermFragData (IndexData &indexData,
+		       bool needFragFreqs,
+		       unsigned long termNum,
+		       FragData &fragData,
+		       FragRangeArray *fragLimits,
+		       UCArray &termWord);
+
+void CombineFragData (bool needFragFreqs,
+		      const FragData &f1,
+		      const FragData &f2,
+		      FragData &outFragData);
+
+// output will be in fragData (as this is an and operation)
+void AndCombineFragData (bool needFragFreqs,
+			 FragData &fragData,
+			 const FragData &comFragData,
+			 signed long startRange,
+			 signed long endRange,
+			 const FragRangeArray *fragLimits);
+
+void FragsToQueryResult (IndexData &indexData,
+			 const QueryInfo &queryInfo,
+			 const FragData &termData,
+			 const UCArray &tag,
+			 const UCArray &term,
+			 unsigned long stemMethod,
+			 unsigned long termWeight,
+			 UCArrayVector &equivTerms,
+			 QueryResult &result);
+
+void AndFragsToQueryResult (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    const UCArray &tag,
+			    const UCArray &term,
+			    unsigned long stemMethod,
+			    unsigned long termWeight,
+			    UCArrayVector &equivTerms,
+			    QueryResult &result);
+
+void RemoveUnwantedResults (IndexData &indexData,
+			    const QueryInfo &queryInfo,
+			    const FragData &termData,
+			    QueryResult &result);
+
+//-----------------------------------------------------------------
+// new QueryResult class to handle retrieval of doc and level nums.
+// Use this class with extended version of MGQuery
+
+class ExtQueryResult : public QueryResult {
+public:
+  DocNumArray levels; // used for returning a different granularity, eg
+  // search sections but return Document numbers, or search Documents, 
+  // return Section numbers.
+ 
+  void Clear ();
+  ExtQueryResult ();
+};
+
+ostream &operator<< (ostream &s, const ExtQueryResult &r);
+bool operator== (const ExtQueryResult &r1, const ExtQueryResult &r2);
+
+//------------------------------------------------------------
+// new functions to handle full text browse
+
+class BrowseQueryResult {
+ public:
+  TermFreqArray termFreqs;
+  void Clear();
+  BrowseQueryResult ();
+
+};
+
+
+ostream &operator<< (ostream &s, const BrowseQueryResult &r);
+bool operator== (const BrowseQueryResult &r1, const BrowseQueryResult &r2);
+
+void FindNearestWordNumber (IndexData &indexData,
+			    const UCArray &term,
+			    unsigned long &number);
+
+void GetTermList(IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 TermFreqArray &terms);
+
+void GetTermList (IndexData &indexData,
+		 unsigned long startTerm,
+		 unsigned long numTerms,
+		 UCArrayVector &terms);
+
+#endif
Index: /trunk/mgpp/text/TextEl.cpp
===================================================================
--- /trunk/mgpp/text/TextEl.cpp	(revision 3365)
+++ /trunk/mgpp/text/TextEl.cpp	(revision 3365)
@@ -0,0 +1,211 @@
+/**************************************************************************
+ *
+ * TextEl.cpp -- Data structures for parsed documents
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "TextEl.h"
+#include "ctype.h"
+
+#define REC_TERM   '\002'
+#define PARA_TERM  '\003'
+
+
+void TextEl::Clear () {
+  elType = TextE;
+  tagName.erase (tagName.begin(), tagName.end());
+  text.erase (text.begin(), text.end());
+}
+
+
+static bool ReadTextTag (FileBuf &buf, TextEl &el) {
+  // we must have some sort of tag
+  unsigned char c;
+  if (!buf.Peek(c)) return false;
+  
+  // get the '<'
+  if (c != '<') return false;
+  el.text.push_back (c);
+  if (!buf.Next() || !buf.Peek (c)) return false;
+  
+  // see if we have a '/'
+  if (c == '/') {
+    // we have a closing tag
+    el.text.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) return false;
+    el.elType = CloseTagE;
+  } else {
+    // we have an opening tag
+    el.elType = OpenTagE;
+  }
+
+  // get the tag name
+  while (isalnum(c)) {
+    el.text.push_back (c);
+    el.tagName.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) return false;
+  }
+  
+  // get everything up to the closing '>'
+  while (c != '>') {
+    el.text.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) return false;
+  }
+
+  // get the closing '>'
+  el.text.push_back (c);
+  buf.Next();
+
+  // if there is no tag name then make this a text element
+  if (el.tagName.empty()) el.elType = TextE;
+  
+  return true;
+}
+
+static void ToggleParaTag (TextEl &el, bool &compatInPara) {
+  SetCStr (el.tagName, "Paragraph");
+  el.text.erase (el.text.begin(), el.text.end());
+  if (compatInPara) {
+    el.elType = CloseTagE;
+    el.text.push_back (PARA_TERM);
+  } else el.elType = OpenTagE;
+  compatInPara = !compatInPara;
+}
+
+static void SetRecTag (TextEl &el, TextElType elType) {
+  el.elType = elType;
+  SetCStr (el.tagName, "Document");
+  el.text.erase (el.text.begin(), el.text.end());
+  if (elType == CloseTagE)
+    el.text.push_back (REC_TERM);
+}
+
+
+bool ReadTextEl (FileBuf &buf, TextEl &el,
+		 bool compatMode, bool &compatInPara) {
+  el.Clear();
+
+  unsigned char c;
+  if (!buf.Peek (c)) return false;
+
+  if (compatMode) {
+    if (c == PARA_TERM) {
+      ToggleParaTag (el, compatInPara);
+      if (!compatInPara) buf.Next();
+      return true;
+    }
+    if (c == REC_TERM) {
+      if (compatInPara) {
+	// need to close this paragraph
+	ToggleParaTag (el, compatInPara);
+	return true;
+      }
+      // can close this document
+      buf.Next();
+      SetRecTag (el, CloseTagE);
+      return true;
+    }
+    if (!compatInPara) {
+      // need to open a paragraph
+      ToggleParaTag (el, compatInPara);
+      return true;
+    }
+  }
+  
+  // see if we have some sort of tag
+  if (c == '<') return ReadTextTag (buf, el);
+
+  // read in a text element
+  el.elType = TextE;
+  while (c != '<' && !(compatMode && (c == PARA_TERM || c == REC_TERM))) {
+    el.text.push_back (c);
+    if (!buf.Next() || !buf.Peek (c)) break;
+  }
+
+  return true;
+}
+
+static void AddTextEl (TextElArray &doc, unsigned long &docLen,
+		       const TextEl &el) {
+  doc.push_back (el);
+  docLen += el.text.size();
+
+//    cout << "elType:  " << el.elType << "\n";
+//    cout << "tagName: " << el.tagName << "\n";
+//    cout << "text:    \"" << el.text << "\"\n\n";
+}
+
+
+bool ReadDoc (FileBuf &buf, const UCArray &docTag,
+	      TextElArray &doc, unsigned long &docLen,
+	      bool compatMode) {
+  bool compatInPara = false;
+  bool foundDocEl = false;
+  TextEl el;
+  
+  doc.erase (doc.begin(), doc.end());
+  docLen = 0;
+
+  if (compatMode) {
+    // add <Document><Paragraph>
+    SetRecTag (el, OpenTagE);
+    AddTextEl (doc, docLen, el);
+    ToggleParaTag (el, compatInPara);
+    AddTextEl (doc, docLen, el);
+    
+  } else {
+    // look for an opening docTag
+    do {
+      if (!ReadTextEl (buf, el, compatMode, compatInPara)) return false;
+    } while (el.elType != OpenTagE || el.tagName != docTag);
+    
+    AddTextEl (doc, docLen, el);
+  }
+
+  
+  // get all elements until the closing docTag
+  while (ReadTextEl (buf, el, compatMode, compatInPara)) {
+    foundDocEl = true;
+    AddTextEl (doc, docLen, el);
+    if (el.elType == CloseTagE && el.tagName == docTag)
+      return true;
+  }
+
+  
+  if (compatMode) {
+    if (!foundDocEl) { // end of text
+      doc.erase (doc.begin(), doc.end());
+      return false;
+    }
+
+    // if we got here then the eof was encountered before
+    // the closing document tag
+    if (compatInPara) {
+      // need to close this paragraph
+      ToggleParaTag (el, compatInPara);
+      AddTextEl (doc, docLen, el);
+    }
+    // close this document
+    SetRecTag (el, CloseTagE);
+    AddTextEl (doc, docLen, el);
+  }
+  
+  return true;
+}
+
+
Index: /trunk/mgpp/text/TextEl.h
===================================================================
--- /trunk/mgpp/text/TextEl.h	(revision 3365)
+++ /trunk/mgpp/text/TextEl.h	(revision 3365)
@@ -0,0 +1,62 @@
+/**************************************************************************
+ *
+ * TextEl.h -- Data structures for parsed documents
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TEXTEL_H
+#define TEXTEL_H
+
+#include "UCArray.h"
+#include "FileBuf.h"
+
+
+enum TextElType {OpenTagE, CloseTagE, TextE};
+
+class TextEl {
+protected:
+  bool ReadTextTag (FileBuf &buf, TextEl &el);
+public:
+  TextElType elType;
+  UCArray tagName;
+  UCArray text; // complete text, including tag
+
+  TextEl () { Clear (); }
+  void Clear ();
+};
+
+
+typedef vector<TextEl> TextElArray;
+
+
+// compatMode parses documents similar to "old mg" -- ie with
+// C-b between documents and C-c between paragraphs
+
+// returns true if a text element could be read
+bool ReadTextEl (FileBuf &buf, TextEl &el,
+		 bool compatMode, bool &compatInPara);
+
+// looks for an opening docTag and reads all text elements up
+// to a corresponding closing docTag. note that this does not
+// hierarchically parse the document so you can't nest documents.
+// returns true if a document could be found
+bool ReadDoc (FileBuf &buf, const UCArray &docTag,
+	      TextElArray &doc, unsigned long &docLen,
+	      bool compatMode=false);
+
+#endif
Index: /trunk/mgpp/text/TextGet.cpp
===================================================================
--- /trunk/mgpp/text/TextGet.cpp	(revision 3365)
+++ /trunk/mgpp/text/TextGet.cpp	(revision 3365)
@@ -0,0 +1,681 @@
+/**************************************************************************
+ *
+ * TextGet.cpp -- Decompressing the text
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "TextGet.h"
+#include "mg_files.h"
+#include "netorder.h"
+#include "mg_errors.h"
+#include "locallib.h"
+#include "words.h"
+#include "local_strings.h"
+#include "bitio_m_stdio.h"
+
+typedef enum huff_type {lengths, chars};
+
+
+static auxiliary_dict *LoadAuxDict (compression_dict &cd, FILE *text_aux_dict) {
+  auxiliary_dict *ad;
+  int i;
+
+  if (!(ad = new auxiliary_dict))
+    {
+      mg_errno = MG_NOMEM;
+      return (NULL);
+    }
+
+  memset (ad, '\0', sizeof (*ad));
+
+  for (i = 0; i <= 1; i++)
+    {
+      int j;
+      u_char *pos;
+
+      fread (&ad->afh[i], sizeof (aux_frags_header), 1, text_aux_dict);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(ad->afh[i].num_frags);
+      NTOHUL(ad->afh[i].mem_for_frags);
+
+      if (!(ad->word_data[i] = new u_char[ad->afh[i].mem_for_frags]))
+	{
+	  mg_errno = MG_NOMEM;
+	  delete ad;
+	  return (NULL);
+	}
+      if (!(ad->words[i] = new u_char* [ad->afh[i].num_frags]))
+	{
+	  mg_errno = MG_NOMEM;
+	  delete ad;
+	  return (NULL);
+	}
+
+      fread (ad->word_data[i], ad->afh[i].mem_for_frags, sizeof (u_char),
+	     text_aux_dict);
+
+      pos = ad->word_data[i];
+      for (j = 0; j < (int)ad->afh[i].num_frags; j++)
+	{
+	  ad->words[i][j] = pos;
+	  pos += *pos + 1;
+	}
+      if (cd.cdh.novel_method == MG_NOVEL_HYBRID)
+	{
+	  int num;
+	  num = 1;
+	  ad->blk_start[i][0] = 0;
+	  ad->blk_end[i][0] = cd.cdh.num_words[i] - 1;
+	  while (num < 33)
+	    {
+	      ad->blk_start[i][num] = ad->blk_end[i][num - 1] + 1;
+	      ad->blk_end[i][num] = ad->blk_start[i][num] +
+		(ad->blk_end[i][num - 1] - ad->blk_start[i][num - 1]) * 2;
+	      num++;
+	    }
+	}
+    }
+  return (ad);
+}
+
+
+static u_char ***ReadInWords (FILE *dict, compression_dict &cd,
+			      comp_frags_header *cfh, u_char **escape) {
+  int i, lookback;
+  int ptrs_reqd = 0;
+  int mem_reqd = 0;
+  int num_set[MAX_HUFFCODE_LEN + 1];
+  u_char *next_word[MAX_HUFFCODE_LEN + 1];
+  u_char **vals;
+  u_char ***values;
+  u_char word[MAXWORDLEN + 1];
+  u_char last_word[MAX_HUFFCODE_LEN + 1][MAXWORDLEN + 1];
+
+  lookback = cd.cdh.lookback;
+
+  for (i = cfh->hd.mincodelen; i <= cfh->hd.maxcodelen; i++) {
+    ptrs_reqd += (cfh->hd.lencount[i] + ((1 << lookback) - 1)) >> lookback;
+    mem_reqd += cfh->huff_words_size[i];
+  }
+
+  if (!(vals = new u_char* [ptrs_reqd]))
+    return (NULL);
+
+  if (!(values = new u_char** [MAX_HUFFCODE_LEN + 1]))
+    return (NULL);
+
+  if (!(next_word[0] = new u_char[mem_reqd])) 
+    return (NULL);
+
+  cd.MemForCompDict += ptrs_reqd * sizeof (*vals) +
+    (MAX_HUFFCODE_LEN + 1) * sizeof (u_char **) +
+    mem_reqd;
+
+  values[0] = vals;
+  values[0][0] = next_word[0];
+  for (i = 1; i <= cfh->hd.maxcodelen; i++)
+    {
+      int next_start = (values[i - 1] - vals) +
+      ((cfh->hd.lencount[i - 1] + ((1 << lookback) - 1)) >> lookback);
+      values[i] = &vals[next_start];
+      next_word[i] = next_word[i - 1] + cfh->huff_words_size[i - 1];
+      values[i][0] = next_word[i];
+    }
+
+  memset (num_set, '\0', sizeof (num_set));
+
+  for (i = 0; i < cfh->hd.num_codes; i++)
+    {
+      register int val, copy;
+      register int len = cfh->hd.clens[i];
+      val = getc (dict);
+      copy = (val >> 4) & 0xf;
+      val &= 0xf;
+
+      fread (word + copy + 1, sizeof (u_char), val, dict);
+      *word = val + copy;
+
+      if ((num_set[len] & ((1 << lookback) - 1)) == 0)
+	{
+	  values[len][num_set[len] >> lookback] = next_word[len];
+	  memcpy (next_word[len], word, *word + 1);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    *escape = next_word[len];
+	  next_word[len] += *word + 1;
+	}
+      else
+	{
+	  copy = prefixlen (last_word[len], word);
+	  memcpy (next_word[len] + 1, word + copy + 1, *word - copy);
+	  *next_word[len] = (copy << 4) + (*word - copy);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    *escape = next_word[len];
+	  next_word[len] += (*word - copy) + 1;
+	}
+      memcpy (last_word[len], word, *word + 1);
+      num_set[len]++;
+    }
+  if (cfh->hd.clens)
+    delete cfh->hd.clens;
+  cfh->hd.clens = NULL;
+  return values;
+}
+
+static int Load_Comp_HuffData(compression_dict &cd, int which, FILE *dict, 
+			      huff_type type) {
+  huff_data * hd;
+  u_long **   vals;
+
+  if (!(hd = new huff_data))
+    return 1;
+  cd.MemForCompDict += sizeof (huff_data);
+  if (Read_Huffman_Data (dict, hd, &cd.MemForCompDict, NULL) == -1)
+    return 2;
+  if (!(vals = Generate_Huffman_Vals (hd, &cd.MemForCompDict)))
+    return 3;
+  if (hd->clens)
+    delete hd->clens;
+  hd->clens = NULL;
+  if (type == chars)
+    {
+      cd.chars_huff[which] = hd;
+      cd.chars_vals[which] = vals;
+    }
+  else
+    {
+      cd.lens_huff[which] = hd;
+      cd.lens_vals[which] = vals;
+    }
+
+  return 0;
+}
+
+static int Load_Comp_FragsHeader(compression_dict &cd, int which, int getEscape,
+				 FILE *dict) {
+  if (!(cd.cfh[which] = new comp_frags_header))
+    return 1;
+  cd.MemForCompDict += sizeof (*cd.cfh[which]);
+  if (Read_cfh (dict, cd.cfh[which], &cd.MemForCompDict, NULL) == -1)
+    return 2;
+
+  if (!(cd.values[which] = ReadInWords (dict, cd, cd.cfh[which],
+					getEscape == 0 ? NULL : &cd.escape[which])))
+    return 3;
+  
+  return 0;
+}
+
+static bool LoadSlowCompDict (FILE *dict, FILE *aux_dict, compression_dict &cd) {
+  if (dict == NULL) return false;
+
+  int which;
+
+  memset (&cd, '\0', sizeof (compression_dict));
+
+  cd.MemForCompDict = sizeof (compression_dict);
+
+  if (Read_cdh (dict, &cd.cdh, &cd.MemForCompDict, NULL) == -1)
+    return false;
+
+  for (which = 0; which < 2; which++)
+    switch (cd.cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  if (Load_Comp_FragsHeader(cd, which, 0, dict) != 0)
+	    return false;
+	  cd.escape[which] = NULL;
+
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (cd.cdh.num_words[which])
+	    {
+	      if (Load_Comp_FragsHeader(cd, which, 1, dict) != 0)
+		return false;
+	    }
+
+	  if (Load_Comp_HuffData(cd, which, dict, chars) != 0)
+	    return false;
+
+	  if (Load_Comp_HuffData(cd, which, dict, lengths) != 0)
+	    return false;
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (cd.cdh.num_words[which])
+	    {
+	      if (Load_Comp_FragsHeader(cd, which, 1, dict) != 0)
+		return false;
+	    }
+	  switch (cd.cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      if (Load_Comp_HuffData(cd, which, dict, chars) != 0)
+		return false;
+
+	      if (Load_Comp_HuffData(cd, which, dict, lengths) != 0)
+		return false;
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    }
+	  break;
+	}
+      }
+
+  if (cd.cdh.novel_method == MG_NOVEL_DELTA ||
+      cd.cdh.novel_method == MG_NOVEL_HYBRID)
+    {
+      if (!aux_dict)
+	{
+	  mg_errno = MG_NOFILE;
+	  cd.Clear();
+	  return false;
+	}
+
+      if (!(cd.ad = LoadAuxDict (cd, aux_dict)))
+	{
+	  cd.Clear();
+	  return false;
+	}
+    }
+
+  mg_errno = MG_NOERROR;
+
+  cd.fast_loaded = 0;
+  
+  return true;
+}
+
+
+
+#define WORDNO(p, base) ((((char*)(p))-((char*)(base)))/sizeof(u_char*))
+#define IS_FIXUP(p) ((fixup[WORDNO(p,cd)/8] & (1<<(WORDNO(p,cd) & 7))) != 0)
+
+// fast loading really needs to be totally re-writen. "Unloading" the
+// text data will currently cause a crash because memory is being
+// deleted multiple times (and probably a zillion other reasons).
+static bool LoadFastCompDict (FILE *text_fast_comp_dict, compression_dict &_cd) {
+  if (text_fast_comp_dict == NULL) return false;
+  
+  u_long *p, *end;
+  u_char *fixup;
+  u_long mem;
+  u_long fixup_mem;
+  int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+  fread (&mem, sizeof (mem), 1, text_fast_comp_dict);
+  NTOHUL(mem);  /* [RPAP - Jan 97: Endian Ordering] */
+  fread (&fixup_mem, sizeof (fixup_mem), 1, text_fast_comp_dict);
+  NTOHUL(fixup_mem);  /* [RPAP - Jan 97: Endian Ordering] */
+
+  compression_dict *cd;
+  if (!(cd = (compression_dict *)malloc (mem))) {
+    mg_errno = MG_NOMEM;
+    return false;
+  }
+
+  end = (u_long *) (((u_char *) cd) + mem);
+  fread (cd, sizeof (u_char), mem, text_fast_comp_dict);
+
+  if (!(fixup = new u_char[fixup_mem]))
+    {
+      mg_errno = MG_NOMEM;
+      return false;
+    }
+
+  fread (fixup, fixup_mem, sizeof (u_char), text_fast_comp_dict);
+
+  for (p = (u_long *) cd; (u_long) p < (u_long) end; p++)
+    if (IS_FIXUP (p))
+      {
+	NTOHUL(*p);  /* [RPAP - Jan 97: Endian Ordering] */
+	*p = *p + (u_long) cd;
+      }
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  /* cdh */
+  NTOHUL(cd->cdh.dict_type);
+  NTOHUL(cd->cdh.novel_method);
+  for (i = 0; i < TEXT_PARAMS; i++)
+    NTOHUL(cd->cdh.params[i]);
+  NTOHUL(cd->cdh.num_words[0]);
+  NTOHUL(cd->cdh.num_words[1]);
+  NTOHUL(cd->cdh.num_word_chars[0]);
+  NTOHUL(cd->cdh.num_word_chars[1]);
+  NTOHUL(cd->cdh.lookback);
+  /* cfh */
+  for (i = 0; i <= 1; i++)
+    {
+      int j;
+
+      NTOHSI(cd->cfh[i]->hd.num_codes);
+      NTOHSI(cd->cfh[i]->hd.mincodelen);
+      NTOHSI(cd->cfh[i]->hd.maxcodelen);
+      for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	{
+	  NTOHSI(cd->cfh[i]->hd.lencount[j]);
+	  NTOHUL(cd->cfh[i]->hd.min_code[j]);
+	}
+      NTOHUL(cd->cfh[i]->uncompressed_size);
+      for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	NTOHUL(cd->cfh[i]->huff_words_size[j]);
+    }
+  NTOHUL(cd->MemForCompDict);
+  /* ad */
+  if (cd->cdh.novel_method == MG_NOVEL_DELTA ||
+      cd->cdh.novel_method == MG_NOVEL_HYBRID)
+    for (i = 0; i <= 1; i++)
+      {
+	int j;
+	
+	NTOHUL(cd->ad->afh[i].num_frags);
+	NTOHUL(cd->ad->afh[i].mem_for_frags);
+	for (j = 0; j < 33; j++)
+	  {
+	    NTOHSI(cd->ad->blk_start[i][j]);
+	    NTOHSI(cd->ad->blk_end[i][j]);
+	  }
+      }
+  NTOHSI(cd->fast_loaded);
+
+  delete fixup;
+
+  // the whole fast comp dict is a bit of a hack so I don't
+  // feel too bad about the next line :-) -- Rodger.
+  _cd = *cd;
+  
+  return true;
+}
+
+
+static bool LoadCompDict (FILE *compDictFile,
+			  FILE *auxDictFile,
+			  FILE *fastCompDictFile,
+			  compression_dict &cd) {
+  // see if we have a fast loading compression dictionary
+  if (fastCompDictFile != NULL)
+    return LoadFastCompDict (fastCompDictFile, cd);
+  
+  // slow compression dictionary
+  return LoadSlowCompDict (compDictFile, auxDictFile, cd);
+}
+
+
+// try to open the dictionary files and load the dictionary
+static bool OpenLoadCompDict (char *textname, compression_dict &cd) {
+  FILE *compDictFile = NULL;
+  FILE *auxDictFile = NULL;
+  FILE *fastCompDictFile = NULL;
+
+  fastCompDictFile = open_file (textname, TEXT_DICT_FAST_SUFFIX,
+				"rb", MAGIC_FAST_DICT, MG_CONTINUE);
+  
+  if (fastCompDictFile == NULL) {
+    compDictFile = open_file (textname, TEXT_DICT_SUFFIX,
+			      "rb", MAGIC_DICT, MG_MESSAGE);
+    auxDictFile =  open_file (textname, TEXT_DICT_AUX_SUFFIX,
+			      "rb", MAGIC_AUX_DICT, MG_CONTINUE);
+  }
+  
+  bool res = LoadCompDict (compDictFile, auxDictFile, fastCompDictFile, cd);
+
+  if (compDictFile != NULL) fclose (compDictFile);
+  if (auxDictFile != NULL) fclose (auxDictFile);
+  if (fastCompDictFile != NULL) fclose (fastCompDictFile);
+  
+  return res;
+}
+
+static bool LoadLevels (char *textname, FTextLevel &levels) {
+  FILE *levelFile = NULL;
+
+  // open the text level file
+  levelFile = open_file (textname, TEXT_LEVEL_SUFFIX,
+			 "rb", MAGIC_TEXT_LEVELS, MG_CONTINUE);
+  if (levelFile == NULL) return false;
+
+  // seek to the appropriate place and read the level information
+  bool res = ((fseek (levelFile, sizeof (u_long), SEEK_SET) == 0) &&
+	      levels.Read (levelFile));
+
+  // close the file
+  fclose (levelFile);
+  
+  return res;
+}
+
+
+TextData::TextData () {
+  // put file pointers in known state first
+  textFile = NULL;
+  textIdxFile = NULL;
+  Clear ();
+}
+
+void TextData::Clear () {
+  cd.Clear(); 
+  textFile = NULL;
+  textIdxFile = NULL;
+  cth.Clear();
+  levels.Clear();
+}
+
+bool TextData::LoadData (char *basepath, char *textname) {
+  
+  if (textname[0] == '\0') return false;
+  
+  // set the basepath
+  set_basepath(basepath);
+  
+  // load the compression dictionary
+  if (!OpenLoadCompDict (textname, cd)) return false;
+
+  // open the compressed text and text index file
+  textFile = open_file (textname, TEXT_SUFFIX, "rb", MAGIC_TEXT, MG_CONTINUE);
+  if (textFile == NULL) return false;
+
+  textIdxFile = open_file (textname, TEXT_IDX_SUFFIX, "rb", MAGIC_TEXI, MG_CONTINUE);
+  if (textIdxFile == NULL) return false;
+
+  // read in the compressed text header
+  if ((fseek (textFile, sizeof (u_long), SEEK_SET) != 0) || !cth.Read (textFile))
+    return false;
+
+  // read in the level information
+  if (!LoadLevels (textname, levels)) return false;
+  
+  return true;
+}
+
+bool TextData::UnloadData () {
+  // close any open files
+  if (textFile != NULL) {
+    fclose (textFile);
+    textFile = NULL;
+  }
+  if (textIdxFile != NULL) {
+    fclose (textIdxFile);
+    textIdxFile = NULL;
+  }
+
+  // do general clear
+  Clear ();
+  
+  return true;  
+}
+
+
+bool GetDocIdx (TextData &td, const UCArray &docLevel,
+		unsigned long docNum, TextIdx &docIdx) {
+  // make sure the text index file was opened successfully
+  if (td.textIdxFile == NULL) return false;
+
+  // read in the index
+  TextLevelInfo &levelInfo = td.levels.levelInfo[docLevel];
+  if (!docIdx.Read (td.textIdxFile, levelInfo, docNum)) return false;
+  
+  return true;
+}
+
+
+
+
+#define MY_HUFF_DECODE(len, code, mcodes)				\
+  do {									\
+    register unsigned long *__min_code = (mcodes);			\
+    register unsigned long *__mclen = __min_code;			\
+    register unsigned long __code = 0;					\
+    do									\
+      {									\
+        __code += __code + buffer.bit();	                  	\
+      }									\
+    while (__code < *++__mclen);					\
+    (len) = __mclen - __min_code;					\
+    (code) = __code - *__mclen;						\
+  } while(0);
+
+
+bool GetDocText (TextData &td, const UCArray &docLevel,
+		 unsigned long docNum, UCArray &docText) {
+  // erase the current text
+  docText.erase (docText.begin(), docText.end());
+
+  // look up the information about this document
+  TextIdx docIdx;
+  if (!GetDocIdx (td, docLevel, docNum, docIdx)) return false;
+
+  // do seek to appropriate position
+  stdio_bitio_buffer buffer (td.textFile);
+  buffer.seek (docIdx.start.byte, docIdx.start.bit);
+  
+  // decompress the document
+  compression_dict &cd = td.cd;
+  auxiliary_dict *ad = cd.ad;
+  int which = docIdx.which;
+  unsigned long num_bits = (docIdx.end.byte*8+(8-docIdx.end.bit)) -
+    (docIdx.start.byte*8+(8-docIdx.start.bit));
+  unsigned long bits = 0;
+
+  // keep decoding bits until enough bits have been decoded
+  while (bits < num_bits) {
+    register unsigned code, len;
+    register int r;
+    register u_char *t, *b = NULL;
+    u_char word[MAXWORDLEN + 1];
+    
+    if (cd.cfh[which]) {
+      MY_HUFF_DECODE (len, code, cd.cfh[which]->hd.min_code);
+      bits += len;
+      
+      r = code & ((1 << cd.cdh.lookback) - 1);
+      t = cd.values[which][len][code >> cd.cdh.lookback];
+      
+      /* step through from base pointer */
+      b = word + 1;
+      while (r--) {
+	register int copy = *t >> 4;
+	memcpy (word + copy + 1, t + 1, *t & 0xf);
+	word[0] = copy + (*t & 0xf);
+	t += ((*t) & 0xf) + 1;
+      }
+    } else t = NULL;
+    
+    if (t == cd.escape[which]) {
+      switch (cd.cdh.novel_method) {
+      case MG_NOVEL_HUFFMAN_CHARS:
+	{
+	  int len, i;
+	  int c;
+	  len = buffer.huff_decode(cd.lens_huff[which]->min_code,
+				   cd.lens_vals[which], &bits);
+	  for (i = 0; i < len; i++) {
+	    c = buffer.huff_decode(cd.chars_huff[which]->min_code,
+				   cd.chars_vals[which], &bits);
+	    docText.push_back (c);
+	  }
+	}
+	break;
+      case MG_NOVEL_DELTA:
+      case MG_NOVEL_HYBRID:
+	{
+	  int idx = 0, len;
+	  u_char *base;
+	  switch (cd.cdh.novel_method)
+	    {
+	    case MG_NOVEL_DELTA:
+	      {
+		idx = buffer.delta_decode (&bits);
+		idx--;
+	      }
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      {
+		int k;
+		k = buffer.gamma_decode (&bits);
+		k--;
+		idx = buffer.binary_decode(ad->blk_end[which][k] -
+					   ad->blk_start[which][k] + 1, 
+					   &bits);
+		idx += ad->blk_start[which][k] - 1;
+	      }
+	      break;
+	    }
+	  base = ad->words[which][idx];
+	  len = *base++;
+	  for (; len; len--)
+	    {
+	      docText.push_back (*base++);
+	    }
+	}
+	break;
+      }
+    }
+    else
+      {
+	/* copy over the matching prefix */
+	r = (*t >> 4);
+	while (r--) {
+	  docText.push_back (*b++);
+	}
+	
+	/* and the stored suffix */
+	r = ((*t) & 0xf);
+	while (r--) {
+	  docText.push_back (*++t);
+	}
+      }
+    which = !which;
+  }
+  
+  buffer.done();
+  
+  return true;
+}
+
Index: /trunk/mgpp/text/TextGet.h
===================================================================
--- /trunk/mgpp/text/TextGet.h	(revision 3365)
+++ /trunk/mgpp/text/TextGet.h	(revision 3365)
@@ -0,0 +1,58 @@
+/**************************************************************************
+ *
+ * TextGet.h -- Decompressing the text
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef TEXTGET_H
+#define TEXTGET_H
+
+#include "text.h"
+#include "FText.h"
+
+
+class TextData {
+public:
+  compression_dict cd;
+  FILE *textFile;
+  FILE *textIdxFile;
+  compressed_text_header cth;
+  FTextLevel levels;
+
+  TextData ();
+  void Clear ();
+
+  // loads compression dictionary, the compressed text header,
+  // and all level informaiton
+  bool LoadData (char *basepath, char *textname);
+  bool UnloadData ();
+};
+
+
+// note: document numbers start at 1
+
+// get information about where a document starts and finishes
+bool GetDocIdx (TextData &td, const UCArray &docLevel,
+		unsigned long docNum, TextIdx &docIdx);
+
+
+// decompress a document's text
+bool GetDocText (TextData &td, const UCArray &docLevel,
+		 unsigned long docNum, UCArray &docText);
+
+#endif
Index: /trunk/mgpp/text/UCArray.cpp
===================================================================
--- /trunk/mgpp/text/UCArray.cpp	(revision 3365)
+++ /trunk/mgpp/text/UCArray.cpp	(revision 3365)
@@ -0,0 +1,278 @@
+/**************************************************************************
+ *
+ * UCArray.cpp -- vector based string class
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "UCArray.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+
+void SetCStr (UCArray &text, const char *cStr) {
+  text.erase(text.begin(), text.end());
+  
+  while (*cStr != '\0') {
+    text.push_back (*cStr);
+    cStr++;
+  }
+}
+
+char * GetCStr(UCArray text) {
+
+  char *cstr = new char[text.size()+1];
+  UCArray::const_iterator here = text.begin();
+  UCArray::const_iterator end = text.end();
+
+  int i = 0;
+  while (here != end) {
+    cstr[i] = text[i];
+    here++; i++;
+  }
+  cstr[i]='\0';
+  return cstr;
+}
+
+ostream &operator<<(ostream &s, const UCArray &a) {
+  UCArray::const_iterator here = a.begin();
+  UCArray::const_iterator end = a.end();
+  while (here != end) {
+    s << *here;
+    here++;
+  }
+
+  return s;
+}
+
+
+bool ReadVarLenUL (FILE *f, unsigned long &n) {
+  register unsigned long temp = 0;
+  register unsigned int bitPos = 0;
+  unsigned char b = 0;
+
+  do {
+    b = fgetc (f);
+    if (feof(f)) return false;
+    temp |= (b & 0x7f) << bitPos;
+    bitPos += 7;
+  } while (b >= 0x80 && bitPos < 32);
+
+  n = temp;
+  
+  return true;
+}
+
+bool WriteVarLenUL (FILE *f, unsigned long n) {
+  register unsigned long temp = n;
+  register unsigned char b = 0;
+  do {
+    b = static_cast<unsigned char> (temp & 0x7f);
+    if (temp >= 0x80) b |= 0x80;
+    fputc (b, f);
+    if (ferror (f) != 0) return false;
+  } while ((temp = temp >> 7) > 0);
+  
+  return true;
+}
+
+
+bool ReadUL (FILE *f, unsigned long &n) {
+  if (fread (&n, sizeof (unsigned long), 1, f) <= 0) return false;
+  NTOHUL (n);
+  return true;
+}
+
+
+bool WriteUL (FILE *f, unsigned long n) {
+  HTONUL (n);
+  return (fwrite (&n, sizeof (unsigned long), 1, f) > 0);
+}
+
+bool ReadF (FILE *f, float &n) {
+  if (fread (&n, sizeof (float), 1, f) <= 0) return false;
+  NTOHF(n);
+  return true;
+}
+
+bool WriteF (FILE *f, float n) {
+  HTONF(n);
+  return (fwrite (&n, sizeof (float), 1, f) > 0);
+}
+
+bool ReadD (FILE *f, double &n) {
+  if (fread (&n, sizeof (double), 1, f) <= 0) return false;
+  NTOHD(n);
+  return true;
+}
+
+bool WriteD (FILE *f, double n) {
+  HTOND(n);
+  return (fwrite (&n, sizeof (double), 1, f) > 0);
+}
+
+bool ReadUCArray (FILE *f, UCArray &a) {
+  // clear the array in preparation
+  a.erase (a.begin(), a.end());
+
+  // read in the array size
+  unsigned long arraySize = 0;
+  if (!ReadVarLenUL (f, arraySize)) return false;
+
+  // read in the array
+  unsigned char b = 0;
+  while (arraySize > 0) {
+    b = fgetc (f);
+    if (feof(f)) return false;
+    a.push_back (b);
+    
+    arraySize--;
+  }
+  
+  return true;
+}
+
+bool WriteUCArray (FILE *f, const UCArray &a) {
+  // write out the array size
+  if (!WriteVarLenUL (f, a.size())) return false;
+
+  UCArray::const_iterator here = a.begin();
+  UCArray::const_iterator end = a.end();
+  while (here != end) {
+    fputc (*here, f);
+    if (ferror (f) != 0) return false;
+    
+    here++;
+  }
+  
+  return true;
+}
+
+/*
+ * This array is designed for mapping upper and lower case letter
+ * together for a case independent comparison.  The mappings are
+ * based upon ascii character sequences.
+ */
+static unsigned char casecharmap[] = {
+	'\000', '\001', '\002', '\003', '\004', '\005', '\006', '\007',
+	'\010', '\011', '\012', '\013', '\014', '\015', '\016', '\017',
+	'\020', '\021', '\022', '\023', '\024', '\025', '\026', '\027',
+	'\030', '\031', '\032', '\033', '\034', '\035', '\036', '\037',
+	'\040', '\041', '\042', '\043', '\044', '\045', '\046', '\047',
+	'\050', '\051', '\052', '\053', '\054', '\055', '\056', '\057',
+	'\060', '\061', '\062', '\063', '\064', '\065', '\066', '\067',
+	'\070', '\071', '\072', '\073', '\074', '\075', '\076', '\077',
+	'\100', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\133', '\134', '\135', '\136', '\137',
+	'\140', '\141', '\142', '\143', '\144', '\145', '\146', '\147',
+	'\150', '\151', '\152', '\153', '\154', '\155', '\156', '\157',
+	'\160', '\161', '\162', '\163', '\164', '\165', '\166', '\167',
+	'\170', '\171', '\172', '\173', '\174', '\175', '\176', '\177',
+	'\200', '\201', '\202', '\203', '\204', '\205', '\206', '\207',
+	'\210', '\211', '\212', '\213', '\214', '\215', '\216', '\217',
+	'\220', '\221', '\222', '\223', '\224', '\225', '\226', '\227',
+	'\230', '\231', '\232', '\233', '\234', '\235', '\236', '\237',
+	'\240', '\241', '\242', '\243', '\244', '\245', '\246', '\247',
+	'\250', '\251', '\252', '\253', '\254', '\255', '\256', '\257',
+	'\260', '\261', '\262', '\263', '\264', '\265', '\266', '\267',
+	'\270', '\271', '\272', '\273', '\274', '\275', '\276', '\277',
+	'\300', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\333', '\334', '\335', '\336', '\337',
+	'\340', '\341', '\342', '\343', '\344', '\345', '\346', '\347',
+	'\350', '\351', '\352', '\353', '\354', '\355', '\356', '\357',
+	'\360', '\361', '\362', '\363', '\364', '\365', '\366', '\367',
+	'\370', '\371', '\372', '\373', '\374', '\375', '\376', '\377',
+};
+
+
+int DictCompare (const UCArray &a1, const UCArray &a2) {
+  unsigned int l1 = a1.size();
+  unsigned int l2 = a2.size();
+  unsigned int l = (l1 < l2) ? l1 : l2;
+  int pos = 0;
+  register int diff = 0;
+
+  UCArray::const_iterator a1Here = a1.begin();
+  UCArray::const_iterator a2Here = a2.begin();
+  
+  while (l--) {
+    if ((diff = casecharmap[*a1Here] - casecharmap[*a2Here]) != 0)
+      return diff;
+    if (pos == 0 && (diff = *a1Here - *a2Here) != 0)
+      pos = diff;
+
+    a1Here++;
+    a2Here++;
+  }
+
+  return ((l1 - l2) ? (l1 - l2) : (pos));
+}
+
+unsigned long PrefixLen (const UCArray &a1, const UCArray &a2) {
+  unsigned long l = (a1.size() < a2.size()) ? a1.size() : a2.size();
+  unsigned long i = 0;
+
+  UCArray::const_iterator a1Here = a1.begin();
+  UCArray::const_iterator a2Here = a2.begin();
+
+  while (i < l && *a1Here == *a2Here) {
+    i++; a1Here++; a2Here++;
+  }
+  
+  return i;
+}
+
+bool WritePreSufStr (FILE *f, const UCArray *prev, const UCArray &a) {
+  unsigned char preLen;
+  unsigned char sufLen;
+  
+  if (prev != NULL) preLen = PrefixLen (*prev, a);
+  else preLen = 0;
+  sufLen = a.size() - preLen;
+
+  // output the prefix length, suffix length, and the suffix
+  fputc (preLen, f);
+  if (ferror(f) != 0) return false;
+  fputc (sufLen, f);
+  if (ferror(f) != 0) return false;
+  char* tmp=GetCStr(a);
+  int ret=(fwrite (tmp+preLen, sizeof (char), sufLen, f) == sufLen);
+  delete (tmp);
+  return (ret);
+}
+
+// a also used for prev
+bool ReadPreSufStr (FILE *f, UCArray &a) {
+  unsigned char preLen = 0;
+  unsigned char sufLen = 0;
+
+  preLen = fgetc(f);
+  sufLen = fgetc(f);
+
+  if (a.size () > preLen) a.erase (a.begin()+preLen, a.end());
+  while (sufLen > 0) {
+    unsigned char c = fgetc (f);
+    a.push_back (c);
+    sufLen--;
+  }
+
+  return true;
+}
+
Index: /trunk/mgpp/text/UCArray.h
===================================================================
--- /trunk/mgpp/text/UCArray.h	(revision 3365)
+++ /trunk/mgpp/text/UCArray.h	(revision 3365)
@@ -0,0 +1,118 @@
+/**************************************************************************
+ *
+ * UCArray.h -- vector based string class
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef UCARRAY_H
+#define UCARRAY_H
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if !defined (GSDL_NAMESPACE_BROKEN) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+using namespace std;
+#endif
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\vector>
+#  include <ospace\std\set>
+#elif defined(GSDL_USE_STL_H)
+#  include <vector.h>
+#  include <set.h>
+#else
+#  include <vector>
+#  include <set>
+#endif
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\iostream>
+#elif defined(GSDL_USE_IOS_H)
+#  include <iostream.h>
+#else
+#  include <iostream>
+#endif
+
+#include <stdio.h>
+
+
+typedef vector<unsigned char> UCArray;
+
+// functions to manipulate UCArrays
+void SetCStr (UCArray &text, const char *cStr);
+char * GetCStr(UCArray text);
+inline void UCArrayClear (UCArray &a) {
+  a.erase (a.begin(), a.end());
+}
+
+// stream operator to print UCArray
+ostream &operator<<(ostream &s, const UCArray &a);
+
+// used for reading and writing unsigned characters
+inline bool ReadUC (FILE *f, unsigned char &c) {
+  return (fread (&c, sizeof (unsigned char), 1, f) == 1);
+}
+inline bool WriteUC (FILE *f, unsigned char c) {
+  return (fwrite (&c, sizeof (unsigned char), 1, f) == 1);
+}
+
+// used for reading and writing variable length unsigned longs
+bool ReadVarLenUL (FILE *f, unsigned long &n);
+bool WriteVarLenUL (FILE *f, unsigned long n);
+
+// used for reading and writing unsigned longs
+bool ReadUL (FILE *f, unsigned long &n);
+bool WriteUL (FILE *f, unsigned long n);
+
+// used for reading and writing floats
+bool ReadF (FILE *f, float &n);
+bool WriteF (FILE *f, float n);
+
+// used for reading and writing doubles
+bool ReadD (FILE *f, double &n);
+bool WriteD (FILE *f, double n);
+
+// used for reading and writing arrays to files
+bool ReadUCArray (FILE *f, UCArray &a);
+bool WriteUCArray (FILE *f, const UCArray &a);
+
+
+// compares the two strings in dictionary order
+int DictCompare (const UCArray &a1, const UCArray &a2);
+
+struct LTUCArray {
+  bool operator()(const UCArray &a1, const UCArray &a2) const
+    { return (a1 < a2); }
+};
+
+struct DictLTUCArray {
+  bool operator() (const UCArray &a1, const UCArray &a2) const
+    { return (DictCompare (a1, a2) < 0); }
+};
+
+
+typedef set<UCArray, LTUCArray> UCArraySet;
+typedef vector<UCArray> UCArrayVector;
+
+unsigned long PrefixLen (const UCArray &a1, const UCArray &a2);
+
+// prev == NULL if no previous string
+bool WritePreSufStr (FILE *f, const UCArray *prev, const UCArray &a);
+bool ReadPreSufStr (FILE *f, UCArray &a); // a also used for prev
+
+
+#endif
Index: /trunk/mgpp/text/WIN32.MAK
===================================================================
--- /trunk/mgpp/text/WIN32.MAK	(revision 3365)
+++ /trunk/mgpp/text/WIN32.MAK	(revision 3365)
@@ -0,0 +1,192 @@
+###########################################################################
+#
+# win32 makefile -- mgpp\text
+# A component of the Greenstone digital library software
+# from the New Zealand Digital Library Project at the
+# University of Waikato, New Zealand.
+#
+# Copyright (C) 2000  The New Zealand Digital Library Project
+#
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+###########################################################################
+
+GSDLHOME = ..\..\..
+USE_STLPORT = 0
+
+STLPATH = "$(GSDLHOME)\packages\windows\stlport\stlport"
+
+CC = cl
+CPPFLAGS = -GX
+DEFS = -DSHORT_SUFFIX -DSILENT -DHAVE_CONFIG_H -D__WIN32__ -D__STDC__ -D__GNU_LIBRARY__ -D__MSDOS__
+INCLUDES = -I"$(GSDLHOME)\src\mgpp\lib" -I"$(GSDLHOME)"
+!IF $(USE_STLPORT)
+DEFS = $(DEFS) -DGSDL_NAMESPACE_BROKEN -DGSDL_USE_IOS_H -D__STL_NO_NEW_IOSTREAMS
+INCLUDES = $(INCLUDES) -I$(STLPATH)
+!ENDIF
+
+LDFLAGS =
+
+LIBS = "$(GSDLHOME)\src\mgpp\lib\libmg.lib"
+
+COMPILE = $(CC) -c $(CPPFLAGS) $(DEFS) $(INCLUDES)
+LINK = $(CC) $(LDFLAGS)
+
+.SUFFIXES:
+.SUFFIXES: .c .obj .cpp
+.c.obj:
+	$(COMPILE) $<
+.cpp.obj:
+	$(COMPILE) $<
+
+o = .obj
+e = .exe
+
+SOURCES = \
+  FIvfLevelInfo.cpp        ivf.pass1.cpp            mgpp_invf_dump.cpp       \
+  FText.cpp                ivf.pass2.cpp            mgpp_passes.cpp          \
+  FileBuf.cpp              locallib.cpp             mgpp_perf_hash_build.cpp \
+  FragLevelConvert.cpp     mg.special.cpp           mgpp_stem_idx.cpp        \
+  IndexData.cpp            mgpp_compression_dict.cpp  mgpp_weights_build.cpp   \
+  TagInfo.cpp              mgpp_decompress_text.cpp   stemmer.cpp            \
+  TextEl.cpp               mg_errors.cpp            text.cpp               \
+  TextGet.cpp              mgpp_fast_comp_dict.cpp    text.pass1.cpp         \
+  UCArray.cpp              mg_files.cpp             text.pass2.cpp         \
+  comp_dict.cpp            mgpp_invf_dict.cpp         words.cpp              \
+  invf.cpp                 mgpp_invf_dict_dump.cpp    Weights.cpp            \
+  MGQuery.cpp              Terms.cpp                QueryTester.cpp        \
+  QueryLex.cpp             QueryParser.cpp         \
+  GSDLQueryLex.cpp	   GSDLQueryParser.cpp      Queryer.cpp
+
+
+HEADERS = \
+  FIvfLevelInfo.h     TextEl.h            hash.h              stemmer.h  \
+  FText.h             TextGet.h           invf.h              text.h     \
+  FileBuf.h           UCArray.h           locallib.h          words.h    \
+  FragLevelConvert.h  WordData.h          mg.h                           \
+  IndexData.h         build.h             mg_errors.h                    \
+  TagInfo.h           comp_dict.h         mg_files.h          Weights.h  \
+  MGQuery.h           Terms.h             QueryLex.h          QueryParser.h \
+  GSDLQueryLex.h	GSDLQueryParser.h
+
+EXEC          = mgpp_passes$e mgpp_decompress_text$e mgpp_perf_hash_build$e \
+                mgpp_invf_dump$e mgpp_weights_build$e mgpp_invf_dict$e mgpp_stem_idx$e \
+                mgpp_invf_dict_dump$e mgpp_compression_dict$e mgpp_fast_comp_dict$e \
+                QueryTester$e Queryer$e
+
+all:    $(EXEC) libtextin.lib
+
+PASSES_OBJS = \
+	mgpp_passes$o text.pass1$o comp_dict$o \
+	text.pass2$o locallib$o \
+	ivf.pass1$o ivf.pass2$o mg.special$o mg_files$o \
+	words$o invf$o FileBuf$o TextEl$o UCArray$o TagInfo$o \
+        FText$o text$o FIvfLevelInfo$o
+
+mgpp_passes$e: $(PASSES_OBJS)
+	$(LINK) $(PASSES_OBJS) $(LIBS)
+
+
+QUERYER_OBJS = Queryer$o mg_files$o mg_errors$o locallib$o invf$o UCArray$o \
+               FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+               IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+               GSDLQueryParser$o GSDLQueryLex$o words$o
+
+Queryer$e: $(QUERYER_OBJS)
+	$(LINK) $(QUERYER_OBJS) $(LIBS)
+
+QUERY_TESTER_OBJS = QueryTester$o mg_files$o invf$o UCArray$o \
+                    FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+                    IndexData$o stemmer$o Weights$o
+
+QueryTester$e: $(QUERY_TESTER_OBJS)
+	$(LINK) $(QUERY_TESTER_OBJS) $(LIBS)
+
+WEIGHTS_BUILD_OBJS = mgpp_weights_build$o mg_files$o invf$o UCArray$o \
+                     FIvfLevelInfo$o FragLevelConvert$o
+
+mgpp_weights_build$e: $(WEIGHTS_BUILD_OBJS)
+	$(LINK) $(WEIGHTS_BUILD_OBJS) $(LIBS)
+
+COMP_DICT_OBJS = mgpp_compression_dict$o mg_files$o locallib$o
+
+mgpp_compression_dict$e: $(COMP_DICT_OBJS)
+	$(LINK) $(COMP_DICT_OBJS) $(LIBS)
+
+FAST_COMP_OBJS = mgpp_fast_comp_dict$o mg_files$o locallib$o
+
+mgpp_fast_comp_dict$e: $(FAST_COMP_OBJS)
+	$(LINK) $(FAST_COMP_OBJS) $(LIBS)
+
+INVF_DUMP_OBJS = mgpp_invf_dump$o locallib$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dump$e: $(INVF_DUMP_OBJS)
+	$(LINK) $(INVF_DUMP_OBJS) $(LIBS)
+
+INVF_DICT_OBJS = mgpp_invf_dict$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict$e: $(INVF_DICT_OBJS)
+	$(LINK) $(INVF_DICT_OBJS) $(LIBS)
+
+INVF_DICT_DUMP_OBJS = mgpp_invf_dict_dump$o mg_files$o invf$o UCArray$o
+
+mgpp_invf_dict_dump$e: $(INVF_DICT_DUMP_OBJS)
+	$(LINK) $(INVF_DICT_DUMP_OBJS) $(LIBS)
+
+PERF_HASH_OBJS = mgpp_perf_hash_build$o mg_files$o invf$o UCArray$o
+
+mgpp_perf_hash_build$e: $(PERF_HASH_OBJS)
+	$(LINK) $(PERF_HASH_OBJS) $(LIBS)
+
+STEM_IDX_OBJS = mgpp_stem_idx$o mg_files$o stemmer$o \
+                invf$o UCArray$o
+
+mgpp_stem_idx$e: $(STEM_IDX_OBJS)
+	$(LINK) $(STEM_IDX_OBJS) $(LIBS)
+
+DECOMPRESS_TEXT_OBJS = \
+                mgpp_decompress_text$o mg_files$o mg_errors$o \
+                locallib$o text$o UCArray$o TextGet$o FText$o
+
+mgpp_decompress_text$e: $(DECOMPRESS_TEXT_OBJS)
+	$(LINK) $(DECOMPRESS_TEXT_OBJS) $(LIBS)
+
+
+PASS_OBJS = \
+	words$o mgpass$o text.pass1$o comp_dict$o stemmer$o \
+	text.pass2$o locallib$o \
+	ivf.pass1$o ivf.pass2$o mg.special$o mg_files$o
+mgpass$e: $(PASS_OBJS)
+	$(LINK) $(PASS_OBJS) $(LIBS)
+
+install: 
+	if NOT EXIST "$(GSDLHOME)\bin\windows" mkdir "$(GSDLHOME)\bin\windows"
+	for %%i in ($(EXEC)) do \
+	  copy %%i "$(GSDLHOME)\bin\windows"
+
+LIB_OBJS = mg_files$o  mg_errors$o locallib$o invf$o UCArray$o \
+           FIvfLevelInfo$o FragLevelConvert$o Terms$o MGQuery$o \
+           IndexData$o stemmer$o Weights$o TextGet$o text$o FText$o \
+           GSDLQueryParser$o GSDLQueryLex$o words$o
+
+libtextin.lib: $(LIB_OBJS)
+	if EXIST libtextin.lib del libtextin.lib
+	lib /out:libtextin.lib $(LIB_OBJS)
+
+clean:
+	del *$o 
+	del $(EXEC)
+	del libtextin.lib
Index: /trunk/mgpp/text/Weights.cpp
===================================================================
--- /trunk/mgpp/text/Weights.cpp	(revision 3365)
+++ /trunk/mgpp/text/Weights.cpp	(revision 3365)
@@ -0,0 +1,126 @@
+/**************************************************************************
+ *
+ * Weights.cpp -- Dealing with document weights
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "UCArray.h"
+#include "Weights.h"
+#include "sysfuncs.h"
+
+#define MAXBITS (sizeof(unsigned long) * 8)
+
+
+ApproxWeightsData::ApproxWeightsData () {
+  weightBuf = NULL;
+  table = NULL;
+  Free();
+}
+
+ApproxWeightsData::~ApproxWeightsData () {
+  Free ();
+}
+
+void ApproxWeightsData::Free () {
+  bits = 0;
+  mask = 0;
+  L = 0.0;
+  B = 0.0;
+  numLevelDocs = 0;
+  if (weightBuf != NULL) {
+    delete [] weightBuf;
+    weightBuf = NULL;
+  }
+  if (table != NULL) {
+    delete [] table;
+    table = NULL;
+  }
+}
+
+bool ApproxWeightsData::Read (FILE *approxWeightsFile, unsigned long diskPtr,
+			      unsigned long _numLevelDocs) {
+  Free ();
+  
+  // go to the appropriate place in the weights file
+  fseek (approxWeightsFile, diskPtr, SEEK_SET);
+
+  // read in approx weights parameters
+  ReadUC (approxWeightsFile, bits);
+  ReadD (approxWeightsFile, L);
+  ReadD (approxWeightsFile, B);
+
+  mask = (bits == 32) ? 0xffffffff : (1 << bits) - 1;
+  numLevelDocs = _numLevelDocs;
+
+  // allocate buffer for the weights
+  unsigned long arrSize = (bits*numLevelDocs+sizeof(unsigned long)-1) / 32;
+  weightBuf = new unsigned long[arrSize];
+
+  // read in the weights
+  unsigned long i;
+  for (i=0; i<arrSize; i++) {
+    ReadUL (approxWeightsFile, weightBuf[i]);
+  }
+
+  // create precomputed table of values if it will be small enough
+  // (if not small enough table == NULL
+  if (bits <= 12) {
+    unsigned long i, tableSize = (1 << bits);
+    table = new float[tableSize];
+    table[0] = L;
+    for (i = 1; i < tableSize; i++)
+      table[i] = table[i - 1] * B;
+  }
+
+  return true;
+}
+
+float ApproxWeightsData::GetLowerApproxDocWeight (unsigned long levelDocNum) {
+  // sanity check
+  if (bits == 0 || weightBuf == NULL) return 1.0;
+
+  register unsigned long c, pos;
+  register unsigned long *dw;
+
+  // get the compressed version of the weight
+  pos = levelDocNum * bits;
+  dw = &weightBuf[pos / MAXBITS];
+  pos &= (MAXBITS - 1);
+  c = *dw >> pos;
+  if (pos + bits > MAXBITS)
+    c |= *(dw + 1) << (MAXBITS - pos);
+  c &= mask;
+
+  // decompress the weight
+  if (table != NULL) return table[c];
+  return (L * pow (B, (double) c));
+}
+
+
+
+float GetExactDocWeight (FILE *exactWeightsFile, unsigned long diskPtr,
+			 unsigned long levelDocNum) {
+  // go to the appropriate place in the weights file
+  fseek (exactWeightsFile, diskPtr+sizeof(float)*levelDocNum, SEEK_SET);
+
+  // read in the exact weight
+  float weight;
+  ReadF (exactWeightsFile, weight);
+  
+  return weight;
+}
Index: /trunk/mgpp/text/Weights.h
===================================================================
--- /trunk/mgpp/text/Weights.h	(revision 3365)
+++ /trunk/mgpp/text/Weights.h	(revision 3365)
@@ -0,0 +1,56 @@
+/**************************************************************************
+ *
+ * Weights.h -- Dealing with document weights
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef WEIGHTS_H
+#define WEIGHTS_H
+
+#include <stdio.h>
+
+
+class ApproxWeightsData {
+public:
+  unsigned char bits;
+  unsigned long mask;
+  double L;
+  double B;
+  unsigned long numLevelDocs;
+  unsigned long *weightBuf;
+  float *table;   // precomputed translation table
+  
+  ApproxWeightsData ();
+  ~ApproxWeightsData ();
+
+  void Free (); // frees any allocated memory
+  
+  bool Read (FILE *approxWeightsFile, unsigned long diskPtr,
+	     unsigned long _numLevelDocs);
+
+  float GetLowerApproxDocWeight (unsigned long levelDocNum);
+};
+
+
+// diskPtr points to the start of the exact document weights
+// for this level
+float GetExactDocWeight (FILE *exactWeightsFile, unsigned long diskPtr,
+			 unsigned long levelDocNum);
+
+
+#endif
Index: /trunk/mgpp/text/WordData.h
===================================================================
--- /trunk/mgpp/text/WordData.h	(revision 3365)
+++ /trunk/mgpp/text/WordData.h	(revision 3365)
@@ -0,0 +1,47 @@
+/**************************************************************************
+ *
+ * WordData.h -- 
+ * Copyright (C) 1999  NZDL
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+class WordData
+{
+
+ public:
+  unsigned long docCount;
+  unsigned long wordCount;
+
+  inline void read(FILE *f)
+    {
+      fread((char *) &docCount, sizeof(docCount), 1, f);
+      fread((char *) &wordCount, sizeof(wordCount), 1, f);
+
+      NTOHUL(docCount);
+      NTOHUL(wordCount);
+    }
+
+  inline unsigned long documents()
+    {
+      return docCount;
+    }
+
+  inline int words()
+    {
+      return wordCount;
+    }
+};
Index: /trunk/mgpp/text/build.h
===================================================================
--- /trunk/mgpp/text/build.h	(revision 3365)
+++ /trunk/mgpp/text/build.h	(revision 3365)
@@ -0,0 +1,86 @@
+/**************************************************************************
+ *
+ * build.h -- Global information for the passes of mg_passes
+ * Copyright (C) 1994  Neil Sharman, Alistair Moffat and Lachlan Andrew
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+
+#ifndef H_BUILD
+#define H_BUILD
+
+#include "longlong.h"
+
+#include "TextEl.h"
+#include "TagInfo.h"
+
+
+int init_special (const TagInfo &tagInfo, char *file_name);
+int init_text_1 (const TagInfo &tagInfo, char *file_name);
+int init_text_2 (const TagInfo &tagInfo, char *file_name);
+int init_ivf_1 (const TagInfo &tagInfo, char *file_name);
+int init_ivf_2 (const TagInfo &tagInfo, char *file_name);
+/*
+ * file_name  IN      The name of the dictionary file
+ * 
+ * returns      COMPALLOK for all ok,
+ *              COMPERROR for any error.  e.g. cannot read file
+ */
+
+
+
+
+
+int process_special (const TagInfo &tagInfo, const TextElArray &doc);
+int process_text_1 (const TagInfo &tagInfo, const TextElArray &doc);
+int process_text_2 (const TagInfo &tagInfo, const TextElArray &doc);
+int process_ivf_1 (const TagInfo &tagInfo, const TextElArray &doc);
+int process_ivf_2 (const TagInfo &tagInfo, const TextElArray &doc);
+/*
+ * returns      COMPALLOK for all ok,
+ *              COMPERROR for any error.  e.g. cannot read file
+ *
+ * The calling routine is responsible for ensuring that s_out is long 
+ * enough.
+ */
+
+
+
+
+
+int done_special (const TagInfo &tagInfo, char *filename);
+int done_text_1 (const TagInfo &tagInfo, char *filename);
+int done_text_2 (const TagInfo &tagInfo, char *filename);
+int done_ivf_1 (const TagInfo &tagInfo, char *filename);
+int done_ivf_2 (const TagInfo &tagInfo, char *filename);
+/*
+ * returns    COMPALLOK for all ok,
+ *              COMPERROR for any error.  e.g. cannot write file
+ */
+
+
+
+
+extern unsigned long invf_buffer_size;
+/*
+ * The amount of memory to allocate to the invertion buffer.
+ */
+
+
+
+#endif
Index: /trunk/mgpp/text/comp_dict.cpp
===================================================================
--- /trunk/mgpp/text/comp_dict.cpp	(revision 3365)
+++ /trunk/mgpp/text/comp_dict.cpp	(revision 3365)
@@ -0,0 +1,229 @@
+/**************************************************************************
+ *
+ * comp_dict.cpp -- Functions for loading the compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "huffman.h"
+#include "local_strings.h"
+#include "memlib.h"
+#include "messages.h"
+
+#include "mg.h"
+#include "hash.h"
+#include "text.h"
+#include "comp_dict.h"
+#include "locallib.h"
+#include "mg_files.h"
+
+compression_dict_header cdh;
+compressed_text_header cth;
+comp_frags_header cfh[2];
+
+dict_hash_table *ht[2];
+
+huff_data char_huff[2];
+huff_data lens_huff[2];
+u_long *char_codes[2], *lens_codes[2];
+u_long Words_disk = 0;
+u_long Chars_disk = 0;
+
+
+static dict_hash_table *
+ReadInWords (FILE * dict, comp_frags_header * cfh,
+	     int esc)
+{
+  int i;
+  u_char *allwords, *prev = NULL;
+  dict_hash_table *ht;
+  u_char **words;
+  u_long ht_size;
+
+  ht_size = prime (cfh->hd.num_codes * HASH_RATIO);
+  if (!(ht = (dict_hash_table *) Xmalloc (sizeof (dict_hash_table) +
+		                            (ht_size - 1) * sizeof (ht->table[0]))))
+    {
+      Message ("no memory for hash_table\n");
+      return NULL;
+    }
+
+  ht->size = ht_size;
+  ht->hd = &cfh->hd;
+
+  if (!(ht->codes = Generate_Huffman_Codes (&cfh->hd, NULL)))
+    {
+      Message ("no memory for huffman codes\n");
+      return NULL;
+    }
+
+  if (!(ht->words = (u_char **) Xmalloc (sizeof (u_char *) * cfh->hd.num_codes)))
+    {
+      Message ("no memory for word pointers\n");
+      return NULL;
+    }
+  words = ht->words;
+
+  memset (ht->table, '\0', ht_size * sizeof (ht->table[0]));
+
+  if (!(allwords = (u_char *) Xmalloc (sizeof (u_char) * cfh->uncompressed_size)))
+    {
+      Message ("no memory for words\n");
+      return NULL;
+    }
+
+  for (i = 0; i < cfh->hd.num_codes; i++, words++)
+    {
+      register int val, copy;
+      val = fgetc (dict);
+      copy = (val >> 4) & 0xf;
+      val &= 0xf;
+
+      *words = allwords;
+
+      memcpy (allwords + 1, prev + 1, copy);
+      fread (allwords + copy + 1, sizeof (u_char), val, dict);
+      *allwords = val + copy;
+
+      Words_disk += val + 1;
+
+      /* insert into the hash table */
+      if (i < cfh->hd.num_codes - esc)
+	{
+	  register u_char **wptr;
+	  register unsigned long tsize = ht->size;
+	  register unsigned long hashval, step;
+
+	  HASH (hashval, step, allwords, tsize);
+
+	  wptr = ht->table[hashval];
+	  for (; wptr;)
+	    {
+	      hashval += step;
+	      if (hashval >= tsize)
+		hashval -= tsize;
+	      wptr = ht->table[hashval];
+	    }
+	  ht->table[hashval] = words;
+	}
+      prev = allwords;
+      allwords += *allwords + 1;
+    }
+  return ht;
+}
+
+inline int LoadWords(FILE *dict, int which)
+{
+  if (Read_cfh (dict, &cfh[which], NULL, &Words_disk) == -1)
+    return 1;
+  
+  if (!(ht[which] = ReadInWords (dict, &cfh[which], 0)))
+    return 1;
+
+  return 0;
+}
+
+inline int LoadHuffmanCodes(FILE *dict, int which)
+{
+  if (Read_Huffman_Data (dict, &char_huff[which], NULL,
+			 &Chars_disk) == -1)
+    return 1;
+  if (!(char_codes[which] =
+	Generate_Huffman_Codes (&char_huff[which], NULL)))
+    return 2;
+  if (Read_Huffman_Data (dict, &lens_huff[which], NULL,
+			 &Chars_disk) == -1)
+    return 3;
+  if (!(lens_codes[which] =
+	Generate_Huffman_Codes (&lens_huff[which], NULL)))
+    return 4;
+
+  return 0;
+}
+
+int 
+LoadCompressionDictionary (char *dict_file_name)
+{
+  FILE *dict;
+  int which;
+  if (!(dict = open_named_file (dict_file_name, "rb", MAGIC_DICT, MG_MESSAGE)))  /* [RPAP - Feb 97: WIN32 Port] */
+    return COMPERROR;
+
+  Words_disk = sizeof (u_long);
+
+  if (Read_cdh (dict, &cdh, NULL, &Words_disk) == -1)
+    goto error;
+
+  for (which = 0; which < 2; which++)
+    switch (cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  if (LoadWords(dict, which) != 0)
+	    goto error;
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (cdh.num_words[which])
+	    {
+	      if (LoadWords(dict, which) != 0)
+		goto error;
+	    }
+	  else
+	    ht[which] = NULL;
+	  if (LoadHuffmanCodes(dict, which) != 0)
+	    goto error;
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (cdh.num_words[which])
+	    {
+	      if (LoadWords(dict, which) != 0)
+		goto error;
+	    }
+	  else
+	    ht[which] = NULL;
+	  switch (cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      if (LoadHuffmanCodes(dict, which) != 0)
+		goto error;
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    default:
+	      FatalError (1, "Bad novel method");
+	    }
+	}
+	break;
+      default:
+	FatalError (1, "Bad dictionary kind\n");
+      }
+
+  return (COMPALLOK);
+
+
+error:
+  fclose (dict);
+  return (COMPERROR);
+}
Index: /trunk/mgpp/text/comp_dict.h
===================================================================
--- /trunk/mgpp/text/comp_dict.h	(revision 3365)
+++ /trunk/mgpp/text/comp_dict.h	(revision 3365)
@@ -0,0 +1,48 @@
+/**************************************************************************
+ *
+ * comp_dict.h -- Functions for loading the compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define HASH_RATIO 2
+
+typedef struct dict_hash_table
+  {
+    u_long size;
+    huff_data *hd;
+    u_long *codes;
+    u_char **words;
+    u_char **table[1];
+  }
+dict_hash_table;
+
+extern compression_dict_header cdh;
+extern compressed_text_header cth;
+extern comp_frags_header cfh[2];
+
+extern dict_hash_table *ht[2];
+
+extern huff_data char_huff[2];
+extern huff_data lens_huff[2];
+extern u_long *char_codes[2], *lens_codes[2];
+extern u_long Words_disk;
+extern u_long Chars_disk;
+
+
+
+int LoadCompressionDictionary (char *dict_file_name);
Index: /trunk/mgpp/text/hash.h
===================================================================
--- /trunk/mgpp/text/hash.h	(revision 3365)
+++ /trunk/mgpp/text/hash.h	(revision 3365)
@@ -0,0 +1,39 @@
+/**************************************************************************
+ *
+ * hash.h -- Hash function
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_HASH
+#define H_HASH
+
+#define HASH(h, step, wd, tablesize)			\
+	do {						\
+		register u_char *p=wd;			\
+		register long len = *p++;		\
+		register unsigned long hval;		\
+		if (len>25) len = 25;			\
+		for (hval=0; len; len--)		\
+		  hval = (hval << 5) - hval + *p++;	\
+		step = hval % (tablesize - 2) + 1;	\
+		h = hval % tablesize;			\
+	} while (0)
+
+
+
+#endif
Index: /trunk/mgpp/text/invf.cpp
===================================================================
--- /trunk/mgpp/text/invf.cpp	(revision 3365)
+++ /trunk/mgpp/text/invf.cpp	(revision 3365)
@@ -0,0 +1,840 @@
+/**************************************************************************
+ *
+ * invf.cpp -- Data structures for inverted files
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "invf.h"
+#include "UCArray.h"
+
+
+invf_dict_header::invf_dict_header () {
+  Clear();
+}
+
+invf_dict_header::~invf_dict_header () {
+}
+
+void invf_dict_header::Clear() {
+  lookback = 0;
+  word_dict_start = 0;
+  word_dict_size = 0;
+  tag_dict_start = 0;
+  tag_dict_size = 0;
+  num_docs = 0;
+  num_frags = 0;
+  num_words = 0;
+  total_bytes = 0;
+  index_string_bytes = 0;
+  num_levels = 0;
+}
+
+bool invf_dict_header::Read (FILE *f) {
+  return (ReadUL (f, lookback) &&
+	  ReadUL (f, word_dict_start) &&
+	  ReadUL (f, word_dict_size) &&
+	  ReadUL (f, tag_dict_start) &&
+	  ReadUL (f, tag_dict_size) &&
+	  ReadUL (f, num_docs) &&
+	  ReadUL (f, num_frags) &&
+	  ReadUL (f, num_words) &&
+	  ReadUL (f, total_bytes) &&
+	  ReadUL (f, index_string_bytes) &&
+	  ReadUL (f, num_levels));
+}
+
+bool invf_dict_header::Write (FILE *f) const {
+  return (WriteUL (f, lookback) &&
+	  WriteUL (f, word_dict_start) &&
+	  WriteUL (f, word_dict_size) &&
+	  WriteUL (f, tag_dict_start) &&
+	  WriteUL (f, tag_dict_size) &&
+	  WriteUL (f, num_docs) &&
+	  WriteUL (f, num_frags) &&
+	  WriteUL (f, num_words) &&
+	  WriteUL (f, total_bytes) &&
+	  WriteUL (f, index_string_bytes) &&
+	  WriteUL (f, num_levels));
+}
+
+
+
+void dict_el::Clear () {
+  el.erase (el.begin(), el.end());
+  frag_occur = 0;
+  freq = 0;
+}
+
+bool dict_el::Read (FILE *f) {
+  return (ReadPreSufStr (f, el) &&
+	  ReadUL (f, frag_occur) &&
+	  ReadUL (f, freq));
+}
+
+bool dict_el::Write (FILE *f, const UCArray *lastEl) const {
+  return (WritePreSufStr (f, lastEl, el) &&
+	  WriteUL (f, frag_occur) &&
+	  WriteUL (f, freq));
+}
+
+
+void word_dict_el::Clear () {
+  dict_el::Clear();
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = NULL;
+}
+
+word_dict_el::~word_dict_el () {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+}
+
+void word_dict_el::SetNumLevels (unsigned long numLevels) {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = new unsigned long [numLevels];
+}
+  
+bool word_dict_el::Read (FILE *f, unsigned long numLevels) {
+  if (!dict_el::Read (f)) return false;
+
+  if (levelFreqs == NULL) return false;
+  
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!ReadUL (f, levelFreqs[i])) return false;
+  }
+
+  return true;
+}
+
+bool word_dict_el::Write (FILE *f, const UCArray *lastEl,
+			  unsigned long numLevels) const {
+  if (!dict_el::Write (f, lastEl)) return false;
+
+  if (levelFreqs == NULL) return false;
+
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!WriteUL (f, levelFreqs[i])) return false;
+  }
+
+  return true;
+}
+
+
+
+block_dict_header::block_dict_header () {
+  Clear();
+}
+
+void block_dict_header::Clear () {
+  invf_dict_header::Clear();
+
+  entries_per_wblk = 0;
+  num_wblks = 0;
+  max_wblk_size = 0;
+  wblk_start = 0;
+  wblk_idx_start = 0;
+  
+  entries_per_tblk = 0;
+  num_tblks = 0;
+  max_tblk_size = 0;
+  tblk_start = 0;
+  tblk_idx_start = 0;
+}
+
+bool block_dict_header::Read (FILE *f) {
+  return (invf_dict_header::Read (f) &&
+
+	  ReadUL (f, entries_per_wblk) &&
+	  ReadUL (f, num_wblks) &&
+	  ReadUL (f, max_wblk_size) &&
+	  ReadUL (f, wblk_start) &&
+	  ReadUL (f, wblk_idx_start) &&
+  
+	  ReadUL (f, entries_per_tblk) &&
+	  ReadUL (f, num_tblks) &&
+	  ReadUL (f, max_tblk_size) &&
+	  ReadUL (f, tblk_start) &&
+	  ReadUL (f, tblk_idx_start));
+}
+
+bool block_dict_header::Write (FILE *f) const {
+  return (invf_dict_header::Write (f) &&
+
+	  WriteUL (f, entries_per_wblk) &&
+	  WriteUL (f, num_wblks) &&
+	  WriteUL (f, max_wblk_size) &&
+	  WriteUL (f, wblk_start) &&
+	  WriteUL (f, wblk_idx_start) &&
+  
+	  WriteUL (f, entries_per_tblk) &&
+	  WriteUL (f, num_tblks) &&
+	  WriteUL (f, max_tblk_size) &&
+	  WriteUL (f, tblk_start) &&
+	  WriteUL (f, tblk_idx_start));
+}
+
+
+
+
+
+void block_dict_el::Clear () {
+  el.erase (el.begin(), el.end());
+  frag_occur = 0;
+  freq = 0;
+  invf_ptr = 0;
+}
+
+bool block_dict_el::Read (FILE *f) {
+  return (ReadPreSufStr (f, el) &&
+	  ReadUL (f, frag_occur) &&
+	  ReadUL (f, freq) &&
+	  ReadUL (f, invf_ptr));
+}
+
+bool block_dict_el::Write (FILE *f, const UCArray *lastEl) const {
+  return (WritePreSufStr (f, lastEl, el) &&
+	  WriteUL (f, frag_occur) &&
+	  WriteUL (f, freq) &&
+	  WriteUL (f, invf_ptr));
+}
+
+
+
+
+
+void word_block_dict_el::Clear () {
+  block_dict_el::Clear();
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = NULL;
+}
+
+word_block_dict_el::~word_block_dict_el () {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+}
+
+void word_block_dict_el::SetNumLevels (unsigned long numLevels) {
+  if (levelFreqs != NULL) delete [] levelFreqs;
+  levelFreqs = new unsigned long [numLevels];
+}
+
+bool word_block_dict_el::Read (FILE *f, unsigned long numLevels) {
+  if (!block_dict_el::Read (f)) return false;
+  
+  if (levelFreqs == NULL) return false;
+  
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!ReadUL (f, levelFreqs[i])) return false;
+  }
+
+  return true;
+}
+
+bool word_block_dict_el::Write (FILE *f, const UCArray *lastEl,
+				unsigned long numLevels) const {
+  if (!block_dict_el::Write (f, lastEl)) return false;
+  
+  if (levelFreqs == NULL) return false;
+  
+  unsigned long i;
+  for (i=0; i<numLevels; i++) {
+    if (!WriteUL (f, levelFreqs[i])) return false;
+  }
+  
+  return true;
+}
+
+
+
+
+block_idx_info::block_idx_info () {
+  Clear ();
+}
+
+void block_idx_info::Clear () {
+  el.erase (el.begin(), el.end());
+  block_ptr = 0;
+}
+
+bool block_idx_info::Read (FILE *f) {
+  return (ReadUCArray (f, el) &&
+	  ReadUL (f, block_ptr));
+}
+
+bool block_idx_info::Write (FILE *f) const {
+  return (WriteUCArray (f, el) &&
+	  WriteUL (f, block_ptr));
+}
+
+
+bool ReadBlockIdx (FILE *f, block_idx &blockIdx) {
+  blockIdx.erase (blockIdx.begin(), blockIdx.end());
+  
+  // read in the array size
+  unsigned long arraySize = 0;
+  if (!ReadVarLenUL (f, arraySize)) return false;
+  
+  // read in the array
+  block_idx_info bi;
+  while (arraySize > 0) {
+    if (!bi.Read (f)) return false;
+    blockIdx.push_back (bi);
+    
+    arraySize--;
+  }
+  
+  return true;
+}
+
+bool WriteBlockIdx (FILE *f, const block_idx &blockIdx) {
+  // write out the array size
+  if (!WriteVarLenUL (f, blockIdx.size())) return false;
+
+  block_idx::const_iterator here = blockIdx.begin();
+  block_idx::const_iterator end = blockIdx.end();
+  while (here != end) {
+    if (!(*here).Write (f)) return false;
+    here++;
+  }
+
+  return true;
+}
+
+
+
+
+stem_idx_header::stem_idx_header () {
+  Clear ();
+}
+
+void stem_idx_header::Clear () {
+  lookback = 0;
+  dict_size = 0;
+
+  entries_per_block = 0;
+  num_blocks = 0;
+  max_block_size = 0;
+  blocks_start = 0;
+  block_idx_start = 0;
+  
+  stemmer_num = 0;
+  stem_method = 0;
+}
+
+bool stem_idx_header::Read (FILE *f) {
+  return (ReadUL (f, lookback) &&
+	  ReadUL (f, dict_size) &&
+	  
+	  ReadUL (f, entries_per_block) &&
+	  ReadUL (f, num_blocks) &&
+	  ReadUL (f, max_block_size) &&
+	  ReadUL (f, blocks_start) &&
+	  ReadUL (f, block_idx_start) &&
+	  
+	  ReadUL (f, stemmer_num) &&
+	  ReadUL (f, stem_method));
+}
+
+bool stem_idx_header::Write (FILE *f) const {
+  return (WriteUL (f, lookback) &&
+	  WriteUL (f, dict_size) &&
+	  
+	  WriteUL (f, entries_per_block) &&
+	  WriteUL (f, num_blocks) &&
+	  WriteUL (f, max_block_size) &&
+	  WriteUL (f, blocks_start) &&
+	  WriteUL (f, block_idx_start) &&
+	  
+	  WriteUL (f, stemmer_num) &&
+	  WriteUL (f, stem_method));
+}
+
+
+
+stem_block_dict_el::stem_block_dict_el () {
+  Clear ();
+}
+
+void stem_block_dict_el::Clear () {
+  el.erase (el.begin(), el.end());
+  equivWords.erase (equivWords.begin(), equivWords.end());
+}
+
+bool stem_block_dict_el::Read (FILE *f) {
+  equivWords.erase (equivWords.begin(), equivWords.end());
+  
+  if (!ReadPreSufStr (f, el)) return false;
+  
+  // read in the array size
+  unsigned long arraySize = 0;
+  if (!ReadVarLenUL (f, arraySize)) return false;
+  
+  // read in the array
+  unsigned long wordNum;
+  while (arraySize > 0) {
+    if (!ReadUL (f, wordNum)) return false;
+    equivWords.push_back (wordNum);
+    
+    arraySize--;
+  }
+  
+  return true;
+}
+
+bool stem_block_dict_el::Write (FILE *f, const UCArray *lastEl) const {
+  if (!WritePreSufStr (f, lastEl, el)) return false;
+
+  // write out the array size
+  if (!WriteVarLenUL (f, equivWords.size())) return false;
+
+  vector<unsigned long>::const_iterator here = equivWords.begin();
+  vector<unsigned long>::const_iterator end = equivWords.end();
+  while (here != end) {
+    if (!WriteUL (f, (*here))) return false;
+    here++;
+  }
+
+  return true;
+}
+
+
+
+
+invf_file_header::invf_file_header () {
+  Clear ();
+}
+
+void invf_file_header::Clear () {
+  no_of_words = 0;
+  no_of_tags = 0;
+  skip_mode = SKIP_MODE_NO_SKIPS;
+  word_level_index = 0;
+  
+  int i;
+  for (i=0; i<16; i++) params[i] = 0;
+}
+  
+bool invf_file_header::Read (FILE *f) {
+  if (!ReadUL (f, no_of_words) ||
+      !ReadUL (f, no_of_tags) ||
+      !ReadUL (f, skip_mode) ||
+      !ReadUL (f, word_level_index)) return false;
+
+  int i;
+  for (i=0; i<16; i++) {
+    if (!ReadUL (f, params[i])) return false;
+  }
+
+  return true;
+}
+
+bool invf_file_header::Write (FILE *f) const {
+  if (!WriteUL (f, no_of_words) ||
+      !WriteUL (f, no_of_tags) ||
+      !WriteUL (f, skip_mode) ||
+      !WriteUL (f, word_level_index)) return false;
+
+  int i;
+  for (i=0; i<16; i++) {
+    if (!WriteUL (f, params[i])) return false;
+  }
+
+  return true;
+}
+
+
+
+
+
+bool SearchElNum (const block_idx &bIdx,
+		  unsigned long entriesPerBlock,
+		  unsigned long elNum,
+		  unsigned long &blockIdxNum,
+		  unsigned long &blockStartElNum) {
+  blockIdxNum = 0;
+  blockStartElNum = 0;
+
+  // make sure the element number isn't out of range
+  if (elNum > bIdx.size()*entriesPerBlock) return false;
+
+  blockIdxNum = elNum / entriesPerBlock;
+  blockStartElNum = blockIdxNum * entriesPerBlock;
+  
+  return true;
+}
+
+bool SearchEl (const block_idx &bIdx,
+	       unsigned long entriesPerBlock,
+	       const UCArray &el,
+	       unsigned long &blockIdxNum,
+	       unsigned long &blockStartElNum) {
+  blockIdxNum = 0;
+  blockStartElNum = 0;
+
+  unsigned long begin = 0;
+  unsigned long bIdxEnd = bIdx.size();
+  unsigned long end = bIdxEnd;
+  unsigned long mid;
+  while (begin < end) {
+    mid = (begin+end)/2;
+    if (DictCompare (el, bIdx[mid].el) < 0) {
+      end = mid;
+    } else if (mid+1 < bIdxEnd &&
+	       DictCompare (el, bIdx[mid+1].el) >= 0) {
+      begin = mid+1;
+    } else {
+      blockIdxNum = mid;
+      blockStartElNum = blockIdxNum * entriesPerBlock;
+      return true;
+    }
+  }
+  
+  return false;
+}
+
+
+
+// use the block dictionary functions for tag entries, and word block dict
+// functions for word entries.
+
+
+bool SearchBlockDictElNum (FILE *dictFile,
+			   const block_idx &bIdx,
+			   unsigned long entriesPerBlock,
+			   unsigned long dictSize,
+			   unsigned long elNum,
+			   block_dict_el &dictEl) {
+  UCArrayClear (dictEl.el);
+  if (elNum >= dictSize) return false;
+
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  do {
+    dictEl.Read (dictFile);
+  } while (curElNum++ < elNum);
+
+  return true;
+}
+
+bool SearchBlockDictEl (FILE *dictFile,
+			const block_idx &bIdx,
+			unsigned long entriesPerBlock,
+			unsigned long dictSize,
+			const UCArray &el,
+			block_dict_el &dictEl,
+			unsigned long &elNum) {
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile);
+    int res = DictCompare (dictEl.el, el);
+    if (res == 0) return true; // found it
+    else if (res > 0) break; // not here
+    
+    elNum++;
+  }
+
+  return false;
+}
+
+
+
+
+bool SearchWordBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long numLevels,
+			       unsigned long elNum,
+			       word_block_dict_el &dictEl) {
+  UCArrayClear (dictEl.el);
+  if (elNum >= dictSize) return false;
+
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  do {
+    dictEl.Read (dictFile, numLevels);
+  } while (curElNum++ < elNum);
+
+  return true;
+}
+
+bool SearchWordBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    unsigned long numLevels,
+			    const UCArray &el,
+			    word_block_dict_el &dictEl,
+			    unsigned long &elNum) {
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile, numLevels);
+    int res = DictCompare (dictEl.el, el);
+    if (res == 0) return true; // found it
+    else if (res > 0) break; // not here
+    
+    elNum++;
+  }
+
+  return false;
+}
+
+
+
+
+bool SearchStemBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long elNum,
+			       stem_block_dict_el &dictEl) {
+  UCArrayClear (dictEl.el);
+  if (elNum >= dictSize) return false;
+
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  do {
+    dictEl.Read (dictFile);
+  } while (curElNum++ < elNum);
+
+  return true;
+}
+
+bool SearchStemBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    const UCArray &el,
+			    stem_block_dict_el &dictEl,
+			    unsigned long &elNum) {
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile);
+    int res = DictCompare (dictEl.el, el);
+    if (res == 0) return true; // found it
+    else if (res > 0) break; // not here
+    
+    elNum++;
+  }
+
+  return false;
+}
+
+//----------------------------------------------------------------
+// functions for full text browse
+
+bool NearestSearchWordBlockDictEl (FILE *dictFile,
+				   const block_idx &bIdx,
+				   unsigned long entriesPerBlock,
+				   unsigned long dictSize,
+				   unsigned long numLevels,
+				   const UCArray &el,
+				   word_block_dict_el &dictEl,
+				   unsigned long &elNum) {
+
+  UCArrayClear (dictEl.el);
+
+  // find the block that contains the element
+  unsigned long blockIdxNum;
+  if (!SearchEl (bIdx, entriesPerBlock, el,
+		 blockIdxNum, elNum))
+    return false;
+
+  unsigned long blockEndElNum = elNum + entriesPerBlock;
+  if (blockEndElNum > dictSize) blockEndElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  while (elNum < blockEndElNum) {
+    dictEl.Read (dictFile, numLevels);
+    int res = DictCompare (el, dictEl.el); // look for the first word that is
+                                     // greater or equal to the el
+    if (res <= 0) {
+      return true; // found one
+   } 
+    
+    elNum++;
+  }
+  // it must be the last term
+  return true;
+
+
+}
+
+
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    UCArrayVector &terms) {
+
+  word_block_dict_el dictEl;  
+  dictEl.SetNumLevels (numLevels);
+  UCArrayClear(dictEl.el);
+
+  terms.erase(terms.begin(), terms.end());
+
+  if (elNum >= dictSize) return false;
+  
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  unsigned long lastElNum = elNum + numWords - 1;
+  if (lastElNum > dictSize) lastElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  
+  // get the first term
+  do {
+    dictEl.Read (dictFile, numLevels);
+  } while (curElNum++ < elNum);
+
+  terms.push_back(dictEl.el);
+  while (curElNum <= lastElNum ) {
+    dictEl.Read(dictFile, numLevels);
+    terms.push_back(dictEl.el);
+    curElNum++;
+  }
+  
+
+  return true;
+}
+
+// NOte: before each addition of dictEl to the array, the level freqs array
+// is deleted, as this was causing problems - generating a seg fault, I think if
+// the vector had to be reallocated or something.
+// setNumLevels has to be called each time before a read, now, to set up the level
+//freqs array. this is necessary.
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    word_block_dict_el_array &terms) {
+
+  word_block_dict_el dictEl;
+  dictEl.SetNumLevels (numLevels);
+  UCArrayClear(dictEl.el);
+
+  block_dict_el elem;
+  terms.erase(terms.begin(), terms.end());
+
+  if (elNum >= dictSize) return false;
+  
+  // find the block that contains the element
+  unsigned long blockIdxNum, curElNum;
+  if (!SearchElNum (bIdx, entriesPerBlock, elNum,
+		    blockIdxNum, curElNum))
+    return false;
+
+  unsigned long lastElNum = elNum + numWords - 1;
+  if (lastElNum > dictSize) lastElNum = dictSize;
+
+  // look for the block
+  fseek (dictFile, bIdx[blockIdxNum].block_ptr, SEEK_SET);
+  // get the first term
+  do {
+    dictEl.Read (dictFile, numLevels);
+  } while (curElNum++ < elNum);
+
+  dictEl.levelFreqs = NULL;
+  terms.push_back(dictEl);
+
+  while (curElNum <= lastElNum ) { 
+    dictEl.SetNumLevels(numLevels);
+    dictEl.Read(dictFile, numLevels);
+    dictEl.levelFreqs = NULL;
+    terms.push_back(dictEl);
+    curElNum++;
+  }
+
+  return true;
+}
+
+
+
+
+
+
+
+
+
Index: /trunk/mgpp/text/invf.h
===================================================================
--- /trunk/mgpp/text/invf.h	(revision 3365)
+++ /trunk/mgpp/text/invf.h	(revision 3365)
@@ -0,0 +1,339 @@
+/**************************************************************************
+ *
+ * invf.h -- Data structures for inverted files
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ * $Id$
+ *
+ **************************************************************************/
+
+
+#ifndef H_INVF
+#define H_INVF
+
+#include <stdio.h>
+
+#include "UCArray.h"
+
+// NOTE: This does not include the magic number
+// header info for .invf.dict file
+struct invf_dict_header {
+  unsigned long lookback;
+  unsigned long word_dict_start;
+  unsigned long word_dict_size;
+  unsigned long tag_dict_start;
+  unsigned long tag_dict_size;
+  unsigned long num_docs;
+  unsigned long num_frags;
+  unsigned long num_words;
+  unsigned long total_bytes;
+  unsigned long index_string_bytes;
+  unsigned long num_levels;
+
+  invf_dict_header ();
+  virtual ~invf_dict_header ();
+  virtual void Clear();
+
+  virtual bool Read (FILE *f);
+  virtual bool Write (FILE *f) const;
+};
+
+
+struct dict_el {
+  UCArray el; // word or tag
+  unsigned long frag_occur;
+  unsigned long freq;
+
+  virtual void Clear ();
+  dict_el () { Clear (); }
+  virtual ~dict_el () { }
+
+  // Read assumes that the last word is in el
+  bool Read (FILE *f);
+  bool Write (FILE *f, const UCArray *lastEl) const;
+};
+
+
+struct word_dict_el : public dict_el {
+  unsigned long *levelFreqs;
+
+  void Clear ();
+  word_dict_el () { levelFreqs = NULL; Clear (); }
+  ~word_dict_el ();
+
+  void SetNumLevels (unsigned long numLevels);
+
+  // SetNumLevels should be called before either
+  // reading or writing using Read and Write
+  
+  // Read assumes that the last word is in el
+  bool Read (FILE *f, unsigned long numLevels);
+  bool Write (FILE *f, const UCArray *lastEl,
+	      unsigned long numLevels) const;
+};
+
+
+// wblk = word block
+// tblk = tag block
+// this version of the blocked dictionary uses a fixed number
+// of entries per block, not a fixed block size
+// info for .invf.dict.blocked file
+// blocked dict has a heap of blocks, some for words, some for tags
+// and an index into each set of blocks. The index has pointers to
+// the first entry in each block. Can do a binary search on the index
+// to find out which block an elemnet is in
+struct block_dict_header : public invf_dict_header {
+  // note: word_dict_start and tag_dict_start are undefined
+  // for blocked dictionaries
+
+  unsigned long entries_per_wblk; // word blocks
+  unsigned long num_wblks;
+  unsigned long max_wblk_size;
+  unsigned long wblk_start;
+  unsigned long wblk_idx_start;
+  
+  unsigned long entries_per_tblk; // tag blocks
+  unsigned long num_tblks;
+  unsigned long max_tblk_size;
+  unsigned long tblk_start;
+  unsigned long tblk_idx_start;
+  
+  block_dict_header ();
+  void Clear ();
+  
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+struct block_dict_el {
+  UCArray el; // word or tag
+  unsigned long frag_occur; // # entries in invf file - if have a
+  // word level index, this is the same as freq, otherwise, its the number
+  // of fragments containing this word
+  unsigned long freq; // # of times this word occurs
+  unsigned long invf_ptr; // pointer into inverted file
+
+  virtual void Clear ();
+  block_dict_el () { Clear (); }
+  virtual ~block_dict_el () { }
+
+  // Read assumes that the last word is in el
+  // set lastEl = NULL when no lookback is wanted (eg
+  // for the start of a block
+  bool Read (FILE *f);
+  bool Write (FILE *f, const UCArray *lastEl) const;
+};
+
+struct word_block_dict_el : public block_dict_el {
+  unsigned long *levelFreqs; // freq of the word at each level
+
+  void Clear ();
+  word_block_dict_el () { levelFreqs = NULL; Clear (); }
+  ~word_block_dict_el ();
+
+  void SetNumLevels (unsigned long numLevels);
+
+  // SetNumLevels should be called before either
+  // reading or writing using Read and Write
+  
+  // Read assumes that the last word is in el
+  bool Read (FILE *f, unsigned long numLevels);
+  bool Write (FILE *f, const UCArray *lastEl,
+	      unsigned long numLevels) const;
+};
+
+typedef vector<word_block_dict_el> word_block_dict_el_array;
+
+struct block_idx_info {
+  UCArray el;
+  unsigned long block_ptr;
+
+  block_idx_info ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+// used for an index into the word and tag blocks
+typedef vector<block_idx_info> block_idx;
+
+bool ReadBlockIdx (FILE *f, block_idx &blockIdx);
+bool WriteBlockIdx (FILE *f, const block_idx &blockIdx);
+
+
+
+struct stem_idx_header {
+  unsigned long lookback;
+  unsigned long dict_size;
+
+  unsigned long entries_per_block;
+  unsigned long num_blocks;
+  unsigned long max_block_size;
+  unsigned long blocks_start;
+  unsigned long block_idx_start;
+  
+  unsigned long stemmer_num;
+  unsigned long stem_method;
+
+  stem_idx_header ();
+  void Clear ();
+
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+struct stem_block_dict_el {
+  UCArray el; // word or tag
+  vector<unsigned long> equivWords;
+  
+  stem_block_dict_el ();
+  void Clear ();
+
+  // Read assumes that the last word is in el
+  // set lastEl = NULL when no lookback is wanted (eg
+  // for the start of a block
+  bool Read (FILE *f);
+  bool Write (FILE *f, const UCArray *lastEl) const;
+};
+
+
+
+#define SKIP_MODE_NO_SKIPS 0
+
+// invf file - has a list of frags for each word, but the word is not 
+//  stored in the invf file - the dictionaries store the words, along
+// with num entries, and a pointer into invf file
+struct invf_file_header {
+  unsigned long no_of_words;
+  unsigned long no_of_tags;
+  unsigned long skip_mode;
+  unsigned long word_level_index; // 1 if word level index
+  unsigned long params[16];
+
+  invf_file_header ();
+  void Clear ();
+  
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+
+
+
+
+// the search functions returns true if a block that could
+// satisfy the request is found. these functions assume that
+// the block index is sorted by DictCompare (or DictLTUCArray)
+bool SearchElNum (const block_idx &bIdx,
+		  unsigned long entriesPerBlock,
+		  unsigned long elNum,
+		  unsigned long &blockIdxNum,
+		  unsigned long &blockStartElNum);
+bool SearchEl (const block_idx &bIdx,
+	       unsigned long entriesPerBlock,
+	       const UCArray &el,
+	       unsigned long &blockIdxNum,
+	       unsigned long &blockStartElNum);
+
+
+// The next six functions use SearchElNum and SearchEl
+// for a particular type of dictionary (Block, WordBlock, or StemBlock)
+// and then look up the entry
+bool SearchBlockDictElNum (FILE *dictFile,
+			   const block_idx &bIdx,
+			   unsigned long entriesPerBlock,
+			   unsigned long dictSize,
+			   unsigned long elNum,
+			   block_dict_el &dictEl);
+bool SearchBlockDictEl (FILE *dictFile,
+			const block_idx &bIdx,
+			unsigned long entriesPerBlock,
+			unsigned long dictSize,
+			const UCArray &el,
+			block_dict_el &dictEl,
+			unsigned long &elNum);
+
+// assumes the numLevels has been set for dictEl
+bool SearchWordBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long numLevels,
+			       unsigned long elNum,
+			       word_block_dict_el &dictEl);
+bool SearchWordBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    unsigned long numLevels,
+			    const UCArray &el,
+			    word_block_dict_el &dictEl,
+			    unsigned long &elNum);
+
+bool SearchStemBlockDictElNum (FILE *dictFile,
+			       const block_idx &bIdx,
+			       unsigned long entriesPerBlock,
+			       unsigned long dictSize,
+			       unsigned long elNum,
+			       stem_block_dict_el &dictEl);
+bool SearchStemBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    const UCArray &el,
+			    stem_block_dict_el &dictEl,
+			    unsigned long &elNum);
+
+//----------------------------------------------------------
+
+// new functions for full text browse
+
+bool NearestSearchWordBlockDictEl (FILE *dictFile,
+			    const block_idx &bIdx,
+			    unsigned long entriesPerBlock,
+			    unsigned long dictSize,
+			    unsigned long numLevels,
+			    const UCArray &el,
+			    word_block_dict_el &dictEl,
+			    unsigned long &elNum);
+
+// returns a list of word_block_dict_el, with no levelfreqs
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    word_block_dict_el_array &terms);
+				    
+// just returns a list of terms
+bool SearchWordBlockDictElNumRange (FILE *dictFile,
+				    const block_idx &bIdx,
+				    unsigned long entriesPerBlock,
+				    unsigned long dictSize,
+				    unsigned long numLevels,
+				    unsigned long elNum,
+				    unsigned long numWords,
+				    UCArrayVector &terms);
+				    
+
+
+#endif
Index: /trunk/mgpp/text/ivf.pass1.cpp
===================================================================
--- /trunk/mgpp/text/ivf.pass1.cpp	(revision 3365)
+++ /trunk/mgpp/text/ivf.pass1.cpp	(revision 3365)
@@ -0,0 +1,775 @@
+/**************************************************************************
+ *
+ * ivf.pass1.cpp -- Memory efficient pass 1 inversion
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "mg_files.h"
+#include "invf.h"
+#include "mg.h"
+#include "build.h"
+#include "locallib.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+#include <stdio.h>
+#include "words.h"
+#include "messages.h"
+#include "netorder.h"
+#include "FIvfLevelInfo.h"
+
+#include "longlong.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+// a fragment corresponds to the index level (word-level is the
+// minimum index level)
+
+
+// structure to determine level information
+struct LevelWorker {
+  unsigned long lastLevelDocNum;
+  unsigned long count;
+
+  LevelWorker () {
+    lastLevelDocNum = count = 0;
+  }
+};
+
+
+// note: the word is stored in the map
+struct IvfWordInfo {
+  unsigned long wordCount;      // word frequency
+  unsigned long fragCount;      // number of fragments that contain the word
+  unsigned long lastFragNum;    // last fragment to contain the word
+  unsigned long chunkWordCount; // word frequency within this chunk
+  unsigned long chunkFragCount; // number of fragments within this chunk that
+                                //   contain the word
+
+  LevelWorker *levels;          // level info for this word
+  
+  IvfWordInfo ();
+  ~IvfWordInfo ();
+  void Clear ();                // will delete levels
+};
+
+typedef map<UCArray, IvfWordInfo, DictLTUCArray> IvfWordInfoMap;
+typedef vector<IvfWordInfoMap::iterator> IvfWordInfoItArray;
+
+// tags don't require as much information
+struct IvfTagInfo {
+  unsigned long tagCount;       // tag frequency
+  unsigned long fragCount;      // number of fragments that contain the tag
+  unsigned long lastFragNum;    // last fragment to contain the tag
+  unsigned long chunkFragCount; // number of fragments within this chunk that
+                                //   contain the tag
+
+  IvfTagInfo ();
+  void Clear ();
+};
+
+typedef map<UCArray, IvfTagInfo, DictLTUCArray> IvfTagInfoMap;
+typedef vector<IvfTagInfoMap::iterator> IvfTagInfoItArray;
+
+
+#define INIT_CHECK_FRAC 0.10
+#define CHECK_FRAC 0.75
+#define CHECK_CLOSE 0.999
+#define CHECK_DIV 1.5
+
+
+static FILE *ic;		// the invf chunk file
+static stdio_bitio_buffer icb;
+
+IvfWordInfoMap ivfWordInfo;
+IvfWordInfoItArray ivfWordInfoOccurOrder;
+IvfTagInfoMap ivfTagInfo;
+IvfTagInfoItArray ivfTagInfoOccurOrder;
+
+static unsigned long chunksWritten;
+static unsigned long maxMemNeeded;
+
+static unsigned long numDocs;
+static unsigned long numChunkDocs;
+
+static unsigned long numFrags;
+static unsigned long numChunkFrags;
+
+static unsigned long numWords;
+
+// the number of document numbers in the inverted file
+static unsigned long numChunkEntries;
+
+// next entry in the inverted file to check memory
+// requirements for the current chunk
+static unsigned long entryCheckPoint;
+
+// information about all the different levels
+static FIvfLevel ivfLevel;
+
+
+
+IvfWordInfo::IvfWordInfo () {
+  levels = NULL;
+  Clear();
+}
+
+IvfWordInfo::~IvfWordInfo () {
+  if (levels != NULL) delete [] levels;
+}
+
+void IvfWordInfo::Clear () {
+  wordCount = 0;
+  chunkWordCount = 0;
+  lastFragNum = 0;
+  fragCount = 0;
+  chunkFragCount = 0;
+
+  if (levels != NULL) {
+    delete [] levels;
+    levels = NULL;
+  }
+}
+
+
+IvfTagInfo::IvfTagInfo () {
+  Clear();
+}
+
+void IvfTagInfo::Clear () {
+  tagCount = 0;
+  lastFragNum = 0;
+  fragCount = 0;
+  chunkFragCount = 0;
+}
+
+
+
+int init_ivf_1 (const TagInfo &tagInfo, char *file_name) {
+  // set up the chunk file
+  if (!(ic = create_file (file_name, INVF_CHUNK_SUFFIX, "wb",
+			  MAGIC_CHUNK, MG_MESSAGE)))
+    return COMPERROR;
+  fwrite ("    ", sizeof (u_long), 1, ic);	// Space for the maxmem
+  icb.attachFile (ic);
+  icb.encodeStart();
+
+  // reset global variables
+  ivfWordInfo.erase (ivfWordInfo.begin(), ivfWordInfo.end());
+  ivfWordInfoOccurOrder.erase (ivfWordInfoOccurOrder.begin(), ivfWordInfoOccurOrder.end());
+  ivfTagInfo.erase (ivfTagInfo.begin(), ivfTagInfo.end());
+  ivfTagInfoOccurOrder.erase (ivfTagInfoOccurOrder.begin(), ivfTagInfoOccurOrder.end());
+  
+  chunksWritten = 0;
+  maxMemNeeded = 0;
+ 
+  numDocs = 0;
+  numChunkDocs = 0;
+
+  numFrags = 0;
+  numChunkFrags = 0;
+
+  numWords = 0;
+
+  numChunkEntries = 0;
+  entryCheckPoint = (unsigned long) ((invf_buffer_size * INIT_CHECK_FRAC) / CHECK_DIV);
+
+  // init the level information
+  ivfLevel.Clear();
+  ivfLevel.docTag = tagInfo.docTag;
+  ivfLevel.indexLevel = tagInfo.indexLevel;
+  IvfLevelInfo blankLevel;
+  UCArraySet::const_iterator levelHere = tagInfo.levelTags.begin();
+  UCArraySet::const_iterator levelEnd = tagInfo.levelTags.end();
+  while (levelHere != levelEnd) {
+    blankLevel.levelTag = *levelHere;
+    ivfLevel.levelInfo[*levelHere] = blankLevel;
+    levelHere++;
+  }
+  
+  return COMPALLOK;
+}
+
+static void ProcessOpenTag (const TagInfo &tagInfo, const TextEl &el,
+			    bool &inFrag) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  // check for start of next fragment
+  if (!wordLevelIndex && el.tagName == tagInfo.indexLevel) {
+    numFrags++;
+    numChunkFrags++;
+    inFrag = true;
+  }
+  
+  // update tag stats
+  IvfTagInfo &i = ivfTagInfo[el.tagName];
+  if (i.tagCount == 0) {
+    // new tag, add to list of iterators
+    IvfTagInfoMap::iterator iIt = ivfTagInfo.find (el.tagName);
+    ivfTagInfoOccurOrder.push_back (iIt);
+  }
+  i.tagCount++;
+
+  // all open tags count as new tags
+  numChunkEntries++;
+  i.fragCount++;
+  i.chunkFragCount++;
+  i.lastFragNum = numFrags;
+  
+  // update level information
+  IvfLevelInfoMap::iterator levelIt = ivfLevel.levelInfo.find (el.tagName);
+  if (levelIt != ivfLevel.levelInfo.end()) {
+    // is a level tag
+    (*levelIt).second.numEntries++;
+    (*levelIt).second.workInLevel = true;
+  }
+}
+
+static void ProcessCloseTag (const TagInfo &tagInfo, const TextEl &el,
+			     bool &inFrag) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  // check for end of fragment
+  if (!wordLevelIndex && el.tagName == tagInfo.indexLevel) {
+    inFrag = false;
+  }
+
+  // update level information
+  IvfLevelInfoMap::iterator levelIt = ivfLevel.levelInfo.find (el.tagName);
+  if (levelIt != ivfLevel.levelInfo.end()) {
+    // is a level tag
+    (*levelIt).second.workInLevel = false;
+  }
+}
+
+static void ProcessText (const TagInfo &tagInfo, const TextEl &el,
+			 bool &inFrag) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  // make sure this text is to be indexed
+  if (!wordLevelIndex && !inFrag) return;
+
+  const unsigned char *textHere = &(el.text[0]);
+  const unsigned char *textEnd = &(el.text[el.text.size() - 1]);
+  UCArray word;
+  
+  if (!inaword (textHere, textEnd))
+    ParseNonindexWord (textHere, textEnd);
+
+  
+  // Alternately parse off words and non-words from the input
+  // Each token is then inserted into the set if it does
+  // not exist or has it's frequency count incremented if it does.
+
+  while (textHere <= textEnd) {
+    textHere = ParseIndexWord (textHere, textEnd, word);
+    textHere = ParseNonindexWord (textHere, textEnd);
+
+    if (!word.empty()) {
+      numWords++;
+
+      if (wordLevelIndex) {
+	numFrags++;
+	numChunkFrags++;
+      }
+      
+      // update word stats
+
+      IvfWordInfo &i = ivfWordInfo[word];
+      if (i.wordCount == 0) {
+	// new word
+	// add to list of iterators
+	IvfWordInfoMap::iterator iIt = ivfWordInfo.find (word);
+	ivfWordInfoOccurOrder.push_back (iIt);
+
+	// add level information array
+	if (ivfLevel.levelInfo.size() > 0) 
+	  i.levels = new LevelWorker [ivfLevel.levelInfo.size()];
+      }
+
+      i.wordCount++;
+      i.chunkWordCount++;
+      if (numFrags > i.lastFragNum) {
+	numChunkEntries++;
+	i.fragCount++;
+	i.chunkFragCount++;
+	i.lastFragNum = numFrags;
+      }
+
+      // update level information for this word
+      if (i.levels != NULL) {
+	IvfLevelInfoMap::iterator levelHere = ivfLevel.levelInfo.begin();
+	IvfLevelInfoMap::iterator levelEnd = ivfLevel.levelInfo.end();
+	LevelWorker *levelWorkerPtr = i.levels;
+	while (levelHere != levelEnd) {
+
+	  // check to make sure the level encompases this fragment
+	  if (!(*levelHere).second.workInLevel) {
+	    cerr << "Level tag <" << (*levelHere).first
+		 << "> does not encompass all fragments\n";
+	    exit (1);
+	  }
+	  
+	  if ((*levelHere).second.numEntries > (*levelWorkerPtr).lastLevelDocNum) {
+	    (*levelWorkerPtr).lastLevelDocNum = (*levelHere).second.numEntries;
+	    (*levelWorkerPtr).count ++;
+	  }
+	  
+	  levelHere++;
+	  levelWorkerPtr++;
+	}
+      }
+    }
+  }
+}
+
+
+static unsigned long MemoryRequired (bool wordLevelIndex) {
+  register unsigned long total = 0;
+
+  // add memory required for word entries
+  IvfWordInfoMap::const_iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::const_iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    register const IvfWordInfo &info = (*wordHere).second;
+    if (info.chunkFragCount > 0) {
+      total += BIO_Bblock_Bound (numChunkFrags, info.chunkFragCount);
+      if (!wordLevelIndex) {
+	total += info.chunkWordCount;
+      }
+    }
+    
+    wordHere++;
+  }
+
+  // add memory required for tag entries
+  IvfTagInfoMap::const_iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::const_iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    register const IvfTagInfo &info = (*tagHere).second;
+    if (info.chunkFragCount > 0) {
+      // two d entries for each frag entry
+      unsigned long pTag = info.chunkFragCount*2;
+      total += BIO_Bblock_Bound (numChunkFrags+pTag, pTag);
+    }
+    
+    tagHere++;
+  }
+  
+  total = (total + 7) >> 3;
+  return total;
+}
+
+
+/*
+static void PrintChunkInfo (unsigned long mem) {
+  cout << "Chunk Number: " << chunksWritten << "\n";
+  cout << "numChunkDocs " << numChunkDocs << "\n";
+  cout << "numChunkFrags " << numChunkFrags << "\n";
+  cout << "mem " << mem << "\n";
+  cout << "numWords " << ivfWordInfo.size() << "\n";
+  cout << "numTags " << ivfTagInfo.size() << "\n\n";
+
+  // output debug tag information in dictionary order
+  IvfTagInfoMap::iterator tagMapHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagMapEnd = ivfTagInfo.end();
+  unsigned long tagNum = 0;
+  while (tagMapHere != tagMapEnd) {
+    cout << (*tagMapHere).first << " " << tagNum
+	 << " " << (*tagMapHere).second.chunkFragCount << "\n";
+    tagNum++;
+    tagMapHere++;
+  }
+}
+*/
+
+static void OutputChunkInfo (unsigned long mem, bool /*wordLevelIndex*/) {
+  chunksWritten++;
+
+  // sanity check
+  if (ivfWordInfo.size() != ivfWordInfoOccurOrder.size()) {
+    Message ("ERROR: Word information size mismatch: %u vs %u\n",
+	     (unsigned int)ivfWordInfo.size(),
+	     (unsigned int)ivfWordInfoOccurOrder.size());
+    exit (1);
+  }
+  if (ivfTagInfo.size() != ivfTagInfoOccurOrder.size()) {
+    Message ("ERROR: Tag information size mismatch: %u vs %u\n",
+	     (unsigned int)ivfTagInfo.size(),
+	     (unsigned int)ivfTagInfoOccurOrder.size());
+    exit (1);
+  }
+
+  icb.gamma_encode (numChunkDocs + 1, NULL);
+  icb.gamma_encode (numChunkFrags + 1, NULL);
+  icb.gamma_encode (mem + 1, NULL);
+  icb.gamma_encode (ivfWordInfo.size() + 1, NULL);
+  icb.gamma_encode (ivfTagInfo.size() + 1, NULL);
+
+  /*  PrintChunkInfo (mem);*/
+
+  // output word information in occurance order
+  IvfWordInfoItArray::iterator wordHere = ivfWordInfoOccurOrder.begin();
+  IvfWordInfoItArray::iterator wordEnd = ivfWordInfoOccurOrder.end();
+  while (wordHere != wordEnd) {
+    register IvfWordInfo &ivfWordInfo = (*(*wordHere)).second;
+    
+    icb.gamma_encode (ivfWordInfo.chunkWordCount + 1, NULL);
+    if (ivfWordInfo.chunkWordCount >= 2) {
+      icb.gamma_encode (ivfWordInfo.chunkFragCount, NULL);
+    }
+
+    ivfWordInfo.lastFragNum = 0;
+    ivfWordInfo.chunkWordCount = 0;
+    ivfWordInfo.chunkFragCount = 0;
+    
+    wordHere++;
+  }
+
+  // output tag information in occurance order
+  IvfTagInfoItArray::iterator tagHere = ivfTagInfoOccurOrder.begin();
+  IvfTagInfoItArray::iterator tagEnd = ivfTagInfoOccurOrder.end();
+  while (tagHere != tagEnd) {
+    register IvfTagInfo &ivfTagInfo = (*(*tagHere)).second;
+
+    icb.gamma_encode (ivfTagInfo.chunkFragCount + 1, NULL);
+
+    ivfTagInfo.lastFragNum = 0;
+    ivfTagInfo.chunkFragCount = 0;
+    
+    tagHere++;
+  }
+
+  numChunkDocs = 0;
+  numChunkFrags = 0;
+  numChunkEntries = 0;
+}
+
+
+int process_ivf_1 (const TagInfo &tagInfo, const TextElArray &doc) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+  bool inFrag = false;
+  if (wordLevelIndex) inFrag = true; // unconditional
+  
+  numDocs++;
+  numChunkDocs++;
+
+  // process each text element in this document
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    if ((*here).elType == OpenTagE)
+      ProcessOpenTag (tagInfo, *here, inFrag);
+    else if ((*here).elType == CloseTagE)
+      ProcessCloseTag (tagInfo, *here, inFrag);
+    else if ((*here).elType == TextE)
+      ProcessText (tagInfo, *here, inFrag);
+    
+    here++;
+  }
+
+  // check the amount of memory needed for this chunk
+  if (numChunkEntries >= entryCheckPoint) {
+    unsigned long mem = MemoryRequired (wordLevelIndex);
+    if (mem >= invf_buffer_size * CHECK_CLOSE) {
+      if (mem > maxMemNeeded) maxMemNeeded = mem;
+      OutputChunkInfo (mem, wordLevelIndex);
+      entryCheckPoint = (unsigned long)
+	((invf_buffer_size * INIT_CHECK_FRAC) / CHECK_DIV);
+      
+    } else {
+      entryCheckPoint = (unsigned long)
+	(entryCheckPoint * ((CHECK_FRAC * (invf_buffer_size - mem)) / mem) +
+	 entryCheckPoint);
+      if (entryCheckPoint <= numChunkEntries)
+	entryCheckPoint = numChunkEntries + 1;
+    }
+  }
+  
+  return COMPALLOK;
+}
+
+
+static void CalcInvfDictSize (unsigned long &totalBytes,
+			      unsigned long &indexStringBytes) {
+  totalBytes = 0;	// The sum of the length of all words, including 
+			// the length byte
+  indexStringBytes = 0; // The amount of space required to store the 
+			// words in the diction, this takes into account 
+			// the prefixes
+  const UCArray *lastWord = NULL;
+  
+  // calculate size of word information
+  IvfWordInfoMap::iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    unsigned long wordSize = (*wordHere).first.size();
+    totalBytes += wordSize + 1;
+    indexStringBytes += wordSize + 2;
+    if (lastWord != NULL)
+      indexStringBytes -= PrefixLen (*lastWord, (*wordHere).first);
+    lastWord = &((*wordHere).first);
+    
+    wordHere++;
+  }
+
+  // calculate size of tag information
+  lastWord = NULL;
+  IvfTagInfoMap::iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    unsigned long tagSize = (*tagHere).first.size();
+    totalBytes += tagSize + 1;
+    indexStringBytes += tagSize + 2;
+    if (lastWord != NULL)
+      indexStringBytes -= PrefixLen (*lastWord, (*tagHere).first);
+    lastWord = &((*tagHere).first);
+    
+    tagHere++;
+  }
+}
+
+
+//      OutputInvfDict ():
+//                writes out the stemmed dictionary file
+//                in the following format 
+//                        lookback value (int)
+//                        totalbytes value (int)
+//                        indexstringbytes (int)
+//                        for each word 
+//                          wordlen (4 bits)
+//                          prefix match (4 bits)
+//                          word (wordlen bytes)
+//                          word frequency (int)
+//                          word count (int)
+//
+//        Accesses outside variables:     
+//
+//        Return value...:                
+static void OutputInvfDict (char *filename) {
+  // create the dictionary header
+  invf_dict_header idh;
+  idh.word_dict_size = ivfWordInfo.size();
+  idh.tag_dict_size = ivfTagInfo.size();
+  idh.num_docs = numDocs;
+  idh.num_frags = numFrags;
+  idh.num_words = numWords;
+  idh.num_levels = ivfLevel.levelInfo.size();
+  CalcInvfDictSize (idh.total_bytes, idh.index_string_bytes);
+
+  // create the inverted dictionary file
+  FILE *sp;
+  if (!(sp = create_file (filename, INVF_DICT_SUFFIX, "wb", MAGIC_STEM_BUILD,
+			  MG_MESSAGE)))
+    return;
+
+  // write out the dictionary header
+  if (!idh.Write (sp)) { fclose (sp); return; }
+
+  // remember where the word dictionary starts
+  idh.word_dict_start = ftell (sp);
+  
+  // output the word dictionary
+  const UCArray *lastWord = NULL;
+  IvfWordInfoMap::iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    // get the prefix and suffix lengths
+    const UCArray &thisWord = (*wordHere).first;
+    WritePreSufStr (sp, lastWord, thisWord);
+
+    // output the number of fragments the word appeared in and the
+    // number of times the word appeared
+    WriteUL (sp, (*wordHere).second.fragCount);
+    WriteUL (sp, (*wordHere).second.wordCount);
+
+    // output frequency information for each level
+    // note that we are expecting every word to have
+    // level information
+    LevelWorker *lwHere = (*wordHere).second.levels;
+    LevelWorker *lwEnd = lwHere + idh.num_levels;
+    while (lwHere != lwEnd) {
+      WriteUL (sp, (*lwHere).count);
+      lwHere++;
+    }
+    
+    lastWord = &thisWord;
+    wordHere++;
+  }
+
+  // remember where the tag dictionary starts
+  idh.tag_dict_start = ftell (sp);
+  
+  // output the tag dictionary
+  const UCArray *lastTag = NULL;
+  IvfTagInfoMap::iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    // get the prefix and suffix lengths
+    const UCArray &thisTag = (*tagHere).first;
+    WritePreSufStr (sp, lastTag, thisTag);
+
+    // output the number of fragments the tag appeared in and the
+    // number of times the tag appeared
+    WriteUL (sp, (*tagHere).second.fragCount);
+    WriteUL (sp, (*tagHere).second.tagCount);
+    
+    lastTag = &thisTag;
+    tagHere++;
+  }
+
+  // write out the updated header
+  fseek (sp, sizeof (u_long), SEEK_SET);
+  if (!idh.Write (sp)) { fclose (sp); return; }
+  
+  fclose (sp);
+}
+
+static void OutputLevelFile (char *filename) {
+  // create the level file
+  FILE *f;
+  if (!(f = create_file (filename, INVF_LEVEL_SUFFIX, "wb", MAGIC_INVF_LEVELS,
+			 MG_MESSAGE)))
+    return;
+
+  // write out the information
+  ivfLevel.Write (f);
+  
+  // close the file
+  fclose (f);
+}
+
+static void OutputTransFile (char *filename) {
+  // save the word number (in lastFragNum :-/)
+  int i = 0;
+  IvfWordInfoMap::iterator wordHere = ivfWordInfo.begin();
+  IvfWordInfoMap::iterator wordEnd = ivfWordInfo.end();
+  while (wordHere != wordEnd) {
+    (*wordHere).second.lastFragNum = i;
+    i++; wordHere++;
+  }
+
+  // save the tag number (in lastFragNum :-/)
+  i = 0;
+  IvfTagInfoMap::iterator tagHere = ivfTagInfo.begin();
+  IvfTagInfoMap::iterator tagEnd = ivfTagInfo.end();
+  while (tagHere != tagEnd) {
+    (*tagHere).second.lastFragNum = i;
+    i++; tagHere++;
+  }
+  
+  // create the translation file
+  FILE *f;
+  if (!(f = create_file (filename, INVF_CHUNK_TRANS_SUFFIX, "wb",
+			 MAGIC_CHUNK_TRANS, MG_MESSAGE)))
+    return;
+
+  stdio_bitio_buffer buffer(f);
+  buffer.encodeStart();
+
+  // write out the word translation table
+  unsigned long wordDictSize = ivfWordInfoOccurOrder.size();
+  IvfWordInfoItArray::iterator wordItHere = ivfWordInfoOccurOrder.begin();
+  IvfWordInfoItArray::iterator wordItEnd = ivfWordInfoOccurOrder.end();
+  unsigned long oN = 0;
+  while (wordItHere != wordItEnd) {
+    register IvfWordInfo &ivfWordInfo = (*(*wordItHere)).second;
+    buffer.binary_encode (ivfWordInfo.lastFragNum + 1, wordDictSize + 1, NULL);
+    oN++;
+    wordItHere++;
+  }
+
+  // write out the tag translation table
+  unsigned long tagDictSize = ivfTagInfoOccurOrder.size();
+  IvfTagInfoItArray::iterator tagItHere = ivfTagInfoOccurOrder.begin();
+  IvfTagInfoItArray::iterator tagItEnd = ivfTagInfoOccurOrder.end();
+  while (tagItHere != tagItEnd) {
+    register IvfTagInfo &ivfTagInfo = (*(*tagItHere)).second;
+    buffer.binary_encode (ivfTagInfo.lastFragNum + 1, tagDictSize + 1, NULL);
+    oN++;
+    tagItHere++;
+  }
+
+  // finish encoding and close the file
+  buffer.encodeDone();
+  fclose (f);
+}
+
+#ifndef SILENT
+static void PrintStats () {
+  Message ("Inverted buffer size:          %8u bytes\n", invf_buffer_size);
+  Message ("Max memory needed for 1 chunk: %8u bytes\n", maxMemNeeded);
+
+  Message ("Number of chunks written:      %8u\n", chunksWritten);
+  Message ("Number of documents:           %8u\n", numDocs);
+  Message ("Number of fragments:           %8u\n", numFrags);
+  Message ("Number of words:               %8u\n", numWords);
+
+  Message ("Size of word dictionary:       %8u\n", ivfWordInfo.size());
+  Message ("Size of tag dictionary:        %8u\n", ivfTagInfo.size());
+}
+#endif
+
+int done_ivf_1 (const TagInfo &tagInfo, char * filename) {
+  bool wordLevelIndex = tagInfo.indexLevel.empty();
+
+  char *temp_str = msg_prefix;
+  msg_prefix = "ivf.pass1";
+
+  // output the last chunk
+  if (numChunkDocs > 0) {
+    unsigned long mem = MemoryRequired (wordLevelIndex);
+    OutputChunkInfo (mem, wordLevelIndex);
+    if (mem > maxMemNeeded) maxMemNeeded = mem;
+  }
+  
+  // write out and encoded 1 to say there are no more chunks
+  icb.gamma_encode (1, NULL);
+  icb.encodeDone ();
+
+  // write out the maximum memory required and close the file
+  fseek (ic, sizeof (long), 0);
+  WriteUL (ic, maxMemNeeded);
+  fclose (ic);
+
+  // output the inverted dictionary
+  OutputInvfDict (filename);
+
+  // write out the translation file
+  OutputTransFile (filename);
+
+  // output the level information
+  OutputLevelFile (filename);
+  
+  // output statistics
+#ifndef SILENT
+  PrintStats ();
+#endif
+  
+  msg_prefix = temp_str;
+
+  return COMPALLOK;
+}
+
Index: /trunk/mgpp/text/ivf.pass2.cpp
===================================================================
--- /trunk/mgpp/text/ivf.pass2.cpp	(revision 3365)
+++ /trunk/mgpp/text/ivf.pass2.cpp	(revision 3365)
@@ -0,0 +1,1111 @@
+/**************************************************************************
+ *
+ * ivf.pass2.cpp -- Memory efficient pass 2 inversion
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+#include <stdio.h>
+
+#if defined __WIN32__
+# include <process.h>
+# include <io.h>
+# define getpid _getpid
+# define unlink _unlink
+#else
+# include <unistd.h>
+# include "non_ansi.h"
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "mg_files.h"
+#include "invf.h"
+#include "mg.h"
+#include "build.h"
+#include "locallib.h"
+#include "bitio_m_random.h"
+#include "bitio_m_stdio.h"
+#include "bitio_m_mems.h"
+#include "bitio_gen.h"
+#include <stdio.h>
+#include "words.h"
+#include "messages.h"
+#include "netorder.h"
+#include "FIvfLevelInfo.h"
+#include "perf_hash.h"
+#include "string.h"
+
+#include "longlong.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+
+#ifdef USE_LONG_LONG
+#define SEEK_X seek_LL
+#define TELL_X tell_LL
+#else
+#define SEEK_X seek
+#define TELL_X tell
+#endif
+
+#ifndef RND_BUF_SIZE
+#define RND_BUF_SIZE 8*1024
+#endif
+
+
+static unsigned long numDocs = 0;
+static unsigned long numChunkDocs = 0;
+static unsigned long numDocsInChunk = 0;
+
+static unsigned long numFrags = 0;
+static unsigned long numFragsInChunk = 0;
+static unsigned long chunkStartFragNum = 0;
+
+
+
+
+struct BitPtr {
+  unsigned long start;
+  unsigned long here;
+  unsigned long lastFragNum;
+  unsigned long lgB;
+  
+  void Clear () { start = here = lastFragNum = lgB = 0; }
+  BitPtr () { Clear(); }
+};
+
+class WordBitPtrs {
+protected:
+  unsigned long numWords;
+  unsigned long numTags;
+  unsigned long size;
+  BitPtr *wordBitPtrs;
+
+  void CheckBufOverrun (unsigned long num) {
+    if (wordBitPtrs[num].here > wordBitPtrs[num+1].start) {
+      cerr << "numDocs: " << numDocs << "\n";
+      cerr << "numChunkDocs: " << numChunkDocs << "\n";
+      cerr << "numDocsInChunk: " << numDocsInChunk << "\n";
+      cerr << "numFrags: " << numFrags << "\n";
+      cerr << "numFragsInChunk: " << numFragsInChunk << "\n";
+      cerr << "chunkStartFragNum: " << chunkStartFragNum << "\n";
+      cerr << "num: " << num << "\n";
+      cerr << "[num].start: " << wordBitPtrs[num].start << "\n";
+      cerr << "[num].here: " << wordBitPtrs[num].here << "\n";
+      cerr << "[num+1].start: " << wordBitPtrs[num+1].start << "\n";
+      FatalError (1, "Bit buffer overrun");
+    }
+  }
+  
+public:
+  void Clear ();
+  WordBitPtrs () { wordBitPtrs = NULL; Clear(); }
+  ~WordBitPtrs ();
+  void SetSize (unsigned long _numWords,
+		unsigned long _numTags);
+
+  void ResetPtrs () {
+    if (wordBitPtrs == NULL) return;
+    unsigned long i;
+    for (i=0; i<size; i++) wordBitPtrs[i].Clear();
+  }
+  
+  BitPtr &GetWordBitPtr (unsigned long wordNum)
+    { return wordBitPtrs[wordNum]; }
+  unsigned long &GetWordStart (unsigned long wordNum)
+    { return wordBitPtrs[wordNum].start; }
+  unsigned long &GetWordHere (unsigned long wordNum)
+    { return wordBitPtrs[wordNum].here; }
+  void CheckWordBufOverrun (unsigned long wordNum)
+    { CheckBufOverrun (wordNum); }
+
+  BitPtr &GetTagBitPtr (unsigned long tagNum)
+    { return wordBitPtrs[tagNum + numWords]; }
+  unsigned long &GetTagStart (unsigned long tagNum)
+    { return wordBitPtrs[tagNum + numWords].start; }
+  unsigned long &GetTagHere (unsigned long tagNum)
+    { return wordBitPtrs[tagNum + numWords].here; }
+  void CheckTagBufOverrun (unsigned long tagNum)
+    { CheckBufOverrun (tagNum + numWords); }
+
+  BitPtr &GetEndBitPtr ()
+    { return wordBitPtrs[size-1]; }
+  unsigned long &GetEndStart ()
+    { return wordBitPtrs[size-1].start; }
+  unsigned long &GetEndHere ()
+    { return wordBitPtrs[size-1].here; }
+};
+
+
+struct IP2TagInfo {
+  bool inTag;
+  unsigned long startFrag;
+  unsigned long tagNum;
+
+  IP2TagInfo () {
+    inTag = false;
+    startFrag = 0;
+    tagNum = 0;
+  }
+};
+
+// maps tags to tag information
+typedef map<UCArray, IP2TagInfo, DictLTUCArray> TagMapDict;
+
+
+// class to handle the translation of occurrence order
+// to dictionary order for words and tags
+class OccurToDictConverter {
+protected:
+  unsigned long pos;
+  unsigned long val;
+  FILE *transFile;
+  random_bitio_buffer rbs;
+
+  unsigned long wordDictSize;
+  unsigned long tagDictSize;
+
+  void SeekStart ();
+  unsigned long TranslateNum (unsigned long num);
+  
+public:
+  OccurToDictConverter ();
+  ~OccurToDictConverter ();
+
+  void Open (char *filename, unsigned long _wordDictSize,
+	     unsigned long _tagDictSize);
+
+  // Close frees all allocated memory
+  void Close ();
+
+  unsigned long TranslateWord (unsigned long occurNum)
+    { return TranslateNum (occurNum); }
+  unsigned long TranslateTag (unsigned long occurNum)
+    { return TranslateNum (occurNum+wordDictSize); }
+};
+
+
+struct InvfStateRec {
+  mg_ullong start;
+  mg_ullong here;
+  unsigned long lastFragNum;
+  unsigned long B;
+
+  void Clear () {
+    start = here = 0;
+    lastFragNum = B = 0;
+  }
+  InvfStateRec () { Clear (); }
+};
+
+
+#define ISR_SIZE 1024
+
+class InvfStateCache {
+protected:
+  InvfStateRec recCache [ISR_SIZE];
+  unsigned long startNum;
+
+  FILE *stateFile;
+
+  void ClearCache () {
+    unsigned int i = 0;
+    for (i=0; i<ISR_SIZE; i++) recCache[i].Clear();
+  }
+  
+public:
+  InvfStateCache ();
+  ~InvfStateCache ();
+  
+  void Open (char *filename);
+  void Close ();
+
+  // previous references to state records may be
+  // invalidated calling GetRec
+  InvfStateRec &GetRec (unsigned long num);
+};
+
+
+static invf_dict_header idh;
+static WordBitPtrs bitPtrs;
+
+static FILE *chunkFile = NULL;
+static stdio_bitio_buffer chunkBuf;
+
+static unsigned long ivfMemBufSize = 0;
+static char *ivfMemBuf = NULL;
+
+// word and tag dictionaries. a map is used for the tag dictionary
+// as it should never be very big (and the perfect hash function
+// sometimes has trouble with small values).
+static perf_hash_data *wordHashDict = NULL;
+static TagMapDict tagMapDict;
+
+// information about all the different levels
+static FIvfLevel ivfLevel;
+
+static OccurToDictConverter occurConvert;
+
+// information about the state of the inverted file
+static InvfStateCache invfState;
+
+static char collectFilename[512];
+
+
+void WordBitPtrs::Clear () {
+  numWords = 0;
+  numTags = 0;
+  size=0;
+  if (wordBitPtrs != NULL) delete [] wordBitPtrs;
+  wordBitPtrs = NULL;
+}
+
+WordBitPtrs::~WordBitPtrs () {
+  if (wordBitPtrs != NULL) delete [] wordBitPtrs;
+}
+
+void WordBitPtrs::SetSize (unsigned long _numWords,
+			   unsigned long _numTags){
+  Clear();
+  numWords = _numWords;
+  numTags = _numTags;
+  size = numWords + numTags + 1;
+  wordBitPtrs = new BitPtr [size];
+}
+
+
+void OccurToDictConverter::SeekStart () {
+  if (transFile == NULL) return;
+  rbs.SEEK_X (sizeof (unsigned long) * 8);
+  pos = 0;
+}
+
+unsigned long OccurToDictConverter::TranslateNum (unsigned long num) {
+  if (num < pos) SeekStart ();
+  while (pos <= num) {
+    if (pos < wordDictSize)
+      val = rbs.binary_decode (wordDictSize + 1, NULL) - 1;
+    else 
+      val = rbs.binary_decode (tagDictSize + 1, NULL) - 1;
+    pos++;
+  }
+  return val;
+}
+
+OccurToDictConverter::OccurToDictConverter () {
+  pos = 0;
+  val = 0;
+  transFile = NULL;
+  wordDictSize = 0;
+  tagDictSize = 0;
+}
+
+OccurToDictConverter::~OccurToDictConverter () {
+  if (transFile != NULL) Close ();
+}
+
+void OccurToDictConverter::Open (char *filename, unsigned long _wordDictSize,
+				 unsigned long _tagDictSize) {
+  if (transFile != NULL) Close ();
+
+  wordDictSize = _wordDictSize;
+  tagDictSize = _tagDictSize;
+
+  transFile = open_file (filename, INVF_CHUNK_TRANS_SUFFIX, "rb",
+			 MAGIC_CHUNK_TRANS, MG_ABORT);
+  rbs.attachFile (transFile, RND_BUF_SIZE);
+  SeekStart ();
+  val = 0;
+}
+
+void OccurToDictConverter::Close () {
+  if (transFile == NULL) return;
+
+  rbs.done ();
+  fclose (transFile);
+  transFile = NULL;
+  pos = 0;
+  val = 0;
+
+  wordDictSize = 0;
+  tagDictSize = 0;
+}
+
+
+
+
+InvfStateCache::InvfStateCache () {
+  startNum = 0;
+  stateFile = NULL;
+}
+
+InvfStateCache::~InvfStateCache () {
+  if (stateFile != NULL) Close ();
+}
+  
+void InvfStateCache::Open (char *filename) {
+  if (stateFile != NULL) Close();
+
+  // open the state file
+  char path[512];
+  sprintf (path, FILE_NAME_FORMAT ".%ld", get_basepath (), filename,
+           ".invf.state", (long) getpid ());
+  if (!(stateFile = fopen (path, "wb+"))) {
+    Message ("Unable to create \"%s\"", path);
+    exit (1);
+  }
+  unlink (path); // file will be deleted after it is closed
+  // reset the buffer
+  startNum = 0;
+  ClearCache();
+}
+
+void InvfStateCache::Close () {
+  if (stateFile == NULL) return;
+  fclose (stateFile);
+  stateFile = NULL;
+  startNum = 0;
+}
+
+InvfStateRec &InvfStateCache::GetRec (unsigned long num) {
+  // see if cached
+  if ((num >= startNum) && (num < startNum + ISR_SIZE))
+    return recCache[num-startNum];
+
+  // not cached, write out this lot of records and read in
+  fseek (stateFile, startNum*sizeof (InvfStateRec), SEEK_SET);
+  fwrite ((char *) recCache, sizeof (InvfStateRec), ISR_SIZE, stateFile);
+
+  // read in the new set of records
+  ClearCache ();
+  startNum = num - (num % ISR_SIZE);
+  fseek (stateFile, startNum*sizeof (InvfStateRec), SEEK_SET);
+  fread ((char *) recCache, sizeof (InvfStateRec), ISR_SIZE, stateFile);
+
+  return recCache[num-startNum];
+}
+
+
+
+static void ClearCharBuf (char *buf, unsigned long size) {
+  char *end = buf + size;
+  while (buf != end) *buf++ = 0;
+}
+
+static void ReadWordDict (char *filename) {
+  // read in the perfect hash function for the word dictionary
+  FILE *wordHashFile = open_file (filename, INVF_DICT_HASH_SUFFIX, "rb",
+				  MAGIC_HASH, MG_ABORT);
+  if (!(wordHashDict = read_perf_hash_data (wordHashFile))) {
+    FatalError (1, "Unable to read in hash data for word dictionary");
+  }
+  fclose (wordHashFile);
+}
+
+static void ReadTagDict (char *filename, invf_dict_header &_idh) {
+  // open the file
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+
+  // seek to the start of the tag dictionary
+  fseek (dictFile, _idh.tag_dict_start, SEEK_SET);
+  
+  unsigned long tagNum;
+  dict_el thisEl;
+  for (tagNum = 0; tagNum < _idh.tag_dict_size; tagNum++) {
+    thisEl.Read (dictFile);
+    tagMapDict[thisEl.el].tagNum = tagNum;
+  }
+  
+  fclose (dictFile);
+}
+
+static void ReadLevelFile (char *filename) {
+  FILE *f;
+  f = open_file (filename, INVF_LEVEL_SUFFIX, "rb",
+		 MAGIC_INVF_LEVELS, MG_ABORT);
+  ivfLevel.Read (f);
+  fclose (f);
+}
+
+void CheckIntOverflow (mg_ullong totalIBits, mg_ullong lastTotalIBits) {
+  if (totalIBits < lastTotalIBits) {
+    fprintf(stderr, "ERROR: The totalIBits counter (%d byte unsigned integer) has overflowed.\n", sizeof (mg_ullong));
+    if (sizeof (mg_ullong) < 8) {
+      fprintf(stderr, "       Try compiling with GCC to enable use of 8 bytes for this counter.\n");
+    }
+    fprintf(stderr, "       Build aborted.\n");
+    exit(1);
+  }
+}
+
+// assumes the inverted file state file has been opened
+static void InitInvfState (char *filename,
+			   invf_dict_header &_idh,
+			   InvfStateCache &_invfState,
+			   mg_ullong &totalIBits,
+			   bool wordLevelIndex) {
+  // read in the dictionary, setting inverted state information
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+
+  // seek to the start of the word dictionary
+  fseek (dictFile, _idh.word_dict_start, SEEK_SET);
+
+  // add the word entries
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (_idh.num_levels);
+  unsigned long dictWordNum, p;
+  mg_ullong lastTotalIBits;
+  unsigned long N = _idh.num_frags;
+  for (dictWordNum=0; dictWordNum<_idh.word_dict_size; dictWordNum++) {
+    // lastTotalIBits is used to detect integer overflow
+    lastTotalIBits = totalIBits;
+
+    // read the next word and associated information
+    wordEl.Read (dictFile, _idh.num_levels);
+
+    // update the state record
+    p = wordEl.frag_occur;
+    InvfStateRec &wisr = _invfState.GetRec (dictWordNum);
+    wisr.start = totalIBits;
+    wisr.here = totalIBits;
+    wisr.B = BIO_Bblock_Init (N, p);
+    
+    // add the length of the fragment numbers
+    totalIBits += BIO_Bblock_Bound_b (N, p, wisr.B);
+
+    // if needed, add the length of the fragment frequency information
+    if (!wordLevelIndex)
+      totalIBits += BIO_Gamma_Bound (wordEl.freq, wordEl.frag_occur);
+
+    // align next byte
+#ifdef USE_LONG_LONG
+    totalIBits = (totalIBits + 7ull) & 0xfffffffffffffff8ull;
+#else
+    totalIBits = (totalIBits + 7ul) & 0xfffffff8ul;
+#endif
+
+    CheckIntOverflow (totalIBits, lastTotalIBits);
+  }
+
+  
+  // seek to the start of the tag dictionary
+  fseek (dictFile, _idh.tag_dict_start, SEEK_SET);
+
+  // add the tag entries
+  dict_el tagEl;
+  unsigned long dictTagNum;
+  N = _idh.num_frags;
+  for (dictTagNum=0; dictTagNum<_idh.tag_dict_size; dictTagNum++) {
+    // lastTotalIBits is used to detect integer overflow
+    lastTotalIBits = totalIBits;
+
+    // read the next tag and associated information
+    tagEl.Read (dictFile);
+
+    // update the state record
+    p = tagEl.frag_occur*2;
+    InvfStateRec &tisr = _invfState.GetRec (dictTagNum + _idh.word_dict_size);
+    tisr.start = totalIBits;
+    tisr.here = totalIBits;
+    tisr.B = BIO_Bblock_Init (N+p, p);
+    
+    // add the length of the fragment numbers (two numbers for each
+    // tag, one for start and one for end)
+    totalIBits += BIO_Bblock_Bound_b (N+p, p, tisr.B);
+
+    // align next byte
+#ifdef USE_LONG_LONG
+    totalIBits = (totalIBits + 7ull) & 0xfffffffffffffff8ull;
+#else
+    totalIBits = (totalIBits + 7ul) & 0xfffffff8ul;
+#endif
+
+    CheckIntOverflow (totalIBits, lastTotalIBits);
+  }
+
+  fclose (dictFile);
+}
+
+/*
+// assumes the chunk tag information has been placed in .first
+static void PrintChunkInfo (unsigned long chunkMem,
+			    unsigned long numChunkWords,
+			    unsigned long numChunkTags) {
+  static unsigned long chunksRead = 0;
+  chunksRead++;
+  cout << "Chunk Number: " << chunksRead << "\n";
+  cout << "numChunkDocs " << numDocsInChunk << "\n";
+  cout << "numChunkFrags " << numFragsInChunk << "\n";
+  cout << "mem " << chunkMem << "\n";
+  cout << "numWords " << numChunkWords << "\n";
+  cout << "numTags " << numChunkTags << "\n\n";
+
+  TagMapDict::iterator tagMapHere = tagMapDict.begin();
+  TagMapDict::iterator tagMapEnd = tagMapDict.end();
+  while (tagMapHere != tagMapEnd) {
+    unsigned long tagMapNum = (*tagMapHere).second.tagNum;
+    cout << (*tagMapHere).first << " " << tagMapNum << " "
+	 << bitPtrs.GetTagBitPtr(tagMapNum).here << "\n";
+    tagMapHere++;
+  }
+}
+*/
+
+void ReadChunk (invf_dict_header &_idh, bool wordLevelIndex) {
+  // reset globals
+  numChunkDocs = 0;
+  chunkStartFragNum = numFrags;
+
+  // read in information about this chunk
+  numDocsInChunk = chunkBuf.gamma_decode (NULL) - 1;
+  if (numDocsInChunk == 0)
+    FatalError (1, "The number of docs in the current chunk is 0");
+
+  numFragsInChunk = chunkBuf.gamma_decode (NULL) - 1;
+  unsigned long chunkMem = chunkBuf.gamma_decode (NULL) - 1;
+
+  if (chunkMem > ivfMemBufSize)
+    FatalError (1, "Chunk memory size is greater than maximum");
+  
+  unsigned long numChunkWords = chunkBuf.gamma_decode (NULL) - 1;
+  unsigned long numChunkTags = chunkBuf.gamma_decode (NULL) - 1;
+
+  
+  // reset stuff
+  ClearCharBuf (ivfMemBuf, ivfMemBufSize);
+  bitPtrs.ResetPtrs();  
+
+  // read in the entries in occurrence order storing the
+  // "chunkWordCount" in "start" and the "chunkFragCount"
+  // in "here"
+  unsigned long numOccur;
+  unsigned long wordNum;
+  for (numOccur=0; numOccur<numChunkWords; numOccur++) {
+    wordNum = occurConvert.TranslateWord (numOccur);
+    BitPtr &wordPtr = bitPtrs.GetWordBitPtr (wordNum);
+    wordPtr.start = chunkBuf.gamma_decode (NULL) - 1;
+    if (wordPtr.start >= 2)
+      wordPtr.here = chunkBuf.gamma_decode (NULL);
+    else wordPtr.here = wordPtr.start;
+  }
+  unsigned long tagNum;
+  for (numOccur=0; numOccur<numChunkTags; numOccur++) {
+    tagNum = occurConvert.TranslateTag (numOccur);
+    BitPtr &tagPtr = bitPtrs.GetTagBitPtr (tagNum);
+    // only chunkFragCount is encoded for tags
+    tagPtr.start = chunkBuf.gamma_decode (NULL) - 1;
+    tagPtr.here = tagPtr.start;
+  }
+
+  /*  PrintChunkInfo (chunkMem, numChunkWords, numChunkTags);*/
+  
+  // create the bit ptrs in dictionary order
+  unsigned long totalIBits = 0; // only dealing with memory
+  unsigned long chunkWordCount, chunkFragCount;
+  for (wordNum=0; wordNum<_idh.word_dict_size; wordNum++) {
+    BitPtr &wordPtr = bitPtrs.GetWordBitPtr (wordNum);
+    chunkWordCount = wordPtr.start;
+    chunkFragCount = wordPtr.here;
+    wordPtr.start = totalIBits;
+    wordPtr.here = totalIBits;
+    wordPtr.lastFragNum = chunkStartFragNum;
+    wordPtr.lgB = 0;
+    if (chunkWordCount > 0) {
+      wordPtr.lgB = floorlog_2 (BIO_Bblock_Init_W (numFragsInChunk,
+						   chunkFragCount));
+      totalIBits += BIO_Bblock_Bound (numFragsInChunk, chunkFragCount);
+      // use unary encoding for memory buffer encoding of fragment freq
+      if (!wordLevelIndex) {
+	totalIBits += chunkWordCount;
+      }
+    }
+  }
+  for (tagNum=0; tagNum<_idh.tag_dict_size; tagNum++) {
+    BitPtr &tagPtr = bitPtrs.GetTagBitPtr (tagNum);
+    chunkFragCount = tagPtr.here;
+    tagPtr.start = totalIBits;
+    tagPtr.here = totalIBits;
+    tagPtr.lastFragNum = chunkStartFragNum;
+    tagPtr.lgB = 0;
+    if (chunkFragCount > 0) {
+      unsigned long pTag = chunkFragCount*2;
+      tagPtr.lgB = floorlog_2 (BIO_Bblock_Init_W (numFragsInChunk+pTag,
+						  pTag));
+      unsigned long bLen = BIO_Bblock_Bound (numFragsInChunk+pTag,
+					     pTag);
+//        cout << tagNum + _idh.word_dict_size << " ";
+//        cout << "numFrags: " << numFragsInChunk
+//    	   << " chunkFragCount: " << chunkFragCount
+//    	   << " B: " << 1 << tagPtr.lgB
+//    	   << " blen: " << blen << "\n";
+      totalIBits += bLen;
+    }
+  }
+  bitPtrs.GetEndStart() = totalIBits;
+  bitPtrs.GetEndHere() = totalIBits;
+
+  if ((totalIBits + 7ul) >> 3ul > chunkMem) {
+    cerr << "totalIBits: " << totalIBits << "\n";
+    cerr << "bytes: " << ((totalIBits + 7ul) >> 3ul) << "\n";
+    cerr << "chunkMem: " << chunkMem << "\n";
+    FatalError (1, "Pointers exceed buffer size");
+  }
+}
+
+
+
+
+int init_ivf_2 (const TagInfo &/*tagInfo*/, char *filename) {
+  // read in compressed dictionary header
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  idh.Read (dictFile);
+  fclose (dictFile);
+
+  // set the size of the bit ptrs
+  bitPtrs.SetSize (idh.word_dict_size, idh.tag_dict_size);
+
+  // open the chunk file and read in the maximum memory needed
+  // for the inverted memory buffer
+  chunkFile = open_file (filename, INVF_CHUNK_SUFFIX, "rb",
+			 MAGIC_CHUNK, MG_ABORT);
+  ReadUL (chunkFile, ivfMemBufSize);
+  chunkBuf.attachFile (chunkFile);
+
+  // allocate memory for the inverted buffer
+  ivfMemBuf = new char [ivfMemBufSize];
+  ClearCharBuf (ivfMemBuf, ivfMemBufSize);
+
+  // read in the word dictionary
+  ReadWordDict (filename);
+
+  // read in the tag dictionary
+  ReadTagDict (filename, idh);
+
+  // read in the level information
+  ReadLevelFile (filename);
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  
+  // set up the translation table file
+  occurConvert.Open (filename, idh.word_dict_size, idh.tag_dict_size);
+
+  // reset some globals
+  numDocs = 0;
+  numChunkDocs = 0;
+  numDocsInChunk = 0;
+  numFrags = 0;
+  numFragsInChunk = 0;
+  chunkStartFragNum = 0;
+
+  strcpy (collectFilename, filename);
+  
+  
+  // create the inverted file
+  mg_ullong totalIBits = 0;
+  FILE *invfFile = create_file (filename, INVF_SUFFIX, "wb",
+				MAGIC_INVF, MG_ABORT);
+  totalIBits += sizeof (unsigned long) * 8; // magic number
+  totalIBits += 8 * 200;                    // 200 byte gap -- why??????
+  fclose (invfFile);
+
+  // init the inverted file state cache
+  invfState.Open (filename);
+  InitInvfState (filename, idh, invfState, totalIBits, wordLevelIndex);
+
+  return COMPALLOK;
+}
+
+static void CloseTextTag (IP2TagInfo &tInfo, const UCArray &/*tagName*/) {
+  if (!tInfo.inTag) return;
+
+    // add this tag to the inverted list
+  BitPtr &tagBitPtr = bitPtrs.GetTagBitPtr (tInfo.tagNum);
+  unsigned long endFrag = numFrags;
+  int b = 1 << tagBitPtr.lgB;
+
+  /*
+  cout << (tInfo.tagNum+idh.word_dict_size) << " \"<" << tagName << ">\" "
+       << tInfo.startFrag << " " << endFrag << "\n";
+  */
+    
+  mems_bitio_buffer buffer ((u_char *) ivfMemBuf, tagBitPtr.here);
+  buffer.bblock_encode (tInfo.startFrag - tagBitPtr.lastFragNum + 1,
+			b, NULL);
+  buffer.bblock_encode (endFrag - tInfo.startFrag + 1, b, NULL);
+  tagBitPtr.lastFragNum = endFrag;
+  tagBitPtr.here = buffer.position();
+  buffer.encodeDone();
+  
+  // check for buffer overrun
+  bitPtrs.CheckTagBufOverrun (tInfo.tagNum);
+  
+  // reset information about this tag
+  tInfo.inTag = false;
+  tInfo.startFrag = 0;
+}
+
+static void ProcessOpenTag (const TextEl &el, bool &inFrag) {
+  // close tag if open
+  IP2TagInfo &tInfo = tagMapDict[el.tagName];
+  if (tInfo.inTag) CloseTextTag (tInfo, el.tagName);
+
+  // open this tag
+  tInfo.inTag = true;
+  tInfo.startFrag = numFrags;
+
+  // check for start of next fragment
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  if (!wordLevelIndex && el.tagName == ivfLevel.indexLevel) {
+    numFrags++;
+    inFrag = true;
+  }
+}
+
+static void ProcessCloseTag (const TextEl &el, bool &inFrag) {
+  // check for end of fragment
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  if (!wordLevelIndex && el.tagName == ivfLevel.indexLevel) {
+    inFrag = false;
+  }
+
+  IP2TagInfo &tInfo = tagMapDict[el.tagName];
+  CloseTextTag (tInfo, el.tagName);
+}
+
+static void ProcessText (const TextEl &el, bool &inFrag) {
+  // make sure this text is to be indexed
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  if (!wordLevelIndex && !inFrag) return;
+
+  const unsigned char *textHere = &(el.text[0]);
+  const unsigned char *textEnd = &(el.text[el.text.size() - 1]);
+  unsigned char mgWord[MAXSTEMLEN + 1];
+  
+  if (!inaword (textHere, textEnd))
+    ParseNonindexWord (textHere, textEnd);
+
+  
+  // Alternately parse off words and non-words from the input
+
+  while (textHere <= textEnd) {
+    textHere = ParseIndexMGWord (textHere, textEnd, mgWord);
+    textHere = ParseNonindexWord (textHere, textEnd);
+
+    if (mgWord[0] > 0) {
+      if (wordLevelIndex) numFrags++;
+
+      unsigned long wordNum = perf_hash (wordHashDict, mgWord);
+
+      /*
+      cout << wordNum << " \"";
+      cout.write (mgWord+1, *mgWord);
+      cout << "\" " << numFrags << "\n";
+      */
+      
+      // add this word to the inverted list
+      BitPtr &wordBitPtr = bitPtrs.GetWordBitPtr (wordNum);
+      unsigned long fragNum = numFrags;
+      int b = 1 << wordBitPtr.lgB;
+  
+      mems_bitio_buffer buffer ((u_char *) ivfMemBuf, wordBitPtr.here);
+
+      // note: this assumes that fragments don't carry over between
+      // chunks (which they don't because all tags are closed at the
+      // end of each document and chunks are based on document
+      // boundaries), i.e. the first fragment number must be greater
+      // than the starting fragment number of the chunk.
+      if (fragNum > wordBitPtr.lastFragNum) {
+	buffer.bblock_encode ((fragNum - wordBitPtr.lastFragNum - 1) + 1,
+			      b, NULL);
+	if (!wordLevelIndex) buffer.encodeBit (1); // freq = 1
+	
+      } else if (!wordLevelIndex) {
+	// add one to the frequency count for this word
+	buffer.seek (buffer.position()-1);
+	buffer.encodeBit (0); // unary encoding -- last = 1
+	buffer.encodeBit (1);
+      }
+      
+      wordBitPtr.lastFragNum = fragNum;
+      wordBitPtr.here = buffer.position();
+      buffer.encodeDone();
+      
+      // check for buffer overrun
+      bitPtrs.CheckWordBufOverrun (wordNum);
+    }
+  }
+}
+
+// combine the in memory inverted buffer with the disk
+// based inverted file
+static void DiskMerge (char *filename) {
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+
+      // make sure we have something to process
+  if (numChunkDocs <= 0) return;
+    
+  // open the inverted file
+  FILE *invfFile = open_file (filename, INVF_SUFFIX, "rb+",
+			      MAGIC_INVF, MG_ABORT);
+  random_bitio_buffer invfOutBuf (invfFile);
+
+  // set up to decode the entries in memory
+  mems_bitio_buffer memInBuf ((u_char *) ivfMemBuf, 0);
+
+  // write out the word information
+  unsigned long wordNum;
+  int b;
+  unsigned long currFragNum;
+  unsigned long delta;
+  unsigned long currFreq;
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+    // go to the end of the last inverted file entry
+    InvfStateRec &wordDiskState = invfState.GetRec (wordNum);
+    invfOutBuf.SEEK_X (wordDiskState.here);
+
+    // go to the start of the inverted chunk info in memory
+    BitPtr &wordBitPtr = bitPtrs.GetWordBitPtr (wordNum);
+    memInBuf.seek(wordBitPtr.start);
+
+    // decode each entry and re-write to disk
+    currFragNum = chunkStartFragNum;
+    while (memInBuf.position() < wordBitPtr.here) {
+      // decode word entry
+      b = 1 << wordBitPtr.lgB;
+      delta = memInBuf.bblock_decode (b, NULL);
+      currFragNum += delta;
+      if (!wordLevelIndex) currFreq = memInBuf.unary_decode (NULL);
+      else currFreq = 1;
+
+      // recode on disk
+      invfOutBuf.bblock_encode (currFragNum-wordDiskState.lastFragNum,
+				wordDiskState.B, NULL);
+      if (!wordLevelIndex) invfOutBuf.gamma_encode (currFreq, NULL);
+      wordDiskState.lastFragNum = currFragNum;
+    }
+
+    wordDiskState.here = invfOutBuf.TELL_X();
+  }
+
+  // write out the tag information
+  unsigned long tagNum;
+  unsigned long currTagStart;
+  unsigned long currTagEnd;
+  for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+    // go to the end of the last inverted file entry
+    InvfStateRec &tagDiskState = invfState.GetRec (tagNum+idh.word_dict_size);
+    invfOutBuf.SEEK_X (tagDiskState.here);
+
+    // go to the start of the inverted chunk info in memory
+    BitPtr &tagBitPtr = bitPtrs.GetTagBitPtr (tagNum);
+    memInBuf.seek(tagBitPtr.start);
+
+    // decode each entry and re-write to disk
+    currTagEnd = chunkStartFragNum;
+    while (memInBuf.position() < tagBitPtr.here) {
+      // decode tag entry
+      b = 1 << tagBitPtr.lgB;
+      delta = memInBuf.bblock_decode (b, NULL) - 1;
+      currTagStart = currTagEnd + delta;
+      delta = memInBuf.bblock_decode (b, NULL) - 1;
+      currTagEnd = currTagStart + delta;
+
+      // recode on disk
+      invfOutBuf.bblock_encode (currTagStart-tagDiskState.lastFragNum+1,
+				tagDiskState.B, NULL);
+      invfOutBuf.bblock_encode (currTagEnd-currTagStart+1,
+				tagDiskState.B, NULL);
+
+      tagDiskState.lastFragNum = currTagEnd;
+    }
+
+    tagDiskState.here = invfOutBuf.TELL_X();
+  }
+
+  memInBuf.done();
+
+  invfOutBuf.encodeDone();
+  fclose (invfFile);
+}
+
+
+int process_ivf_2 (const TagInfo &/*tagInfo*/, const TextElArray &doc) {
+  bool wordLevelIndex = ivfLevel.indexLevel.empty();
+  bool inFrag = false;
+  if (wordLevelIndex) inFrag = true; // unconditional
+  
+  // get next chunk information if need to. the chunk information
+  // is needed before the first document is processed
+  if (numChunkDocs >= numDocsInChunk) ReadChunk (idh, wordLevelIndex);
+
+  // process each text element
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    // process this element
+    if ((*here).elType == OpenTagE) ProcessOpenTag (*here, inFrag);
+    else if ((*here).elType == CloseTagE) ProcessCloseTag (*here, inFrag);
+    else ProcessText (*here, inFrag);
+    
+    here++;
+  }
+
+  // close off any unclosed tags
+  TagMapDict::iterator tdHere = tagMapDict.begin();
+  TagMapDict::iterator tdEnd = tagMapDict.end();
+  while (tdHere != tdEnd) {
+    CloseTextTag ((*tdHere).second, (*tdHere).first);
+    tdHere++;
+  }
+  
+  // we've processed one more document
+  numDocs++;
+  numChunkDocs++;
+
+  // merge the memory based inverted file with the one on
+  // disk if this is the end of this chunk
+  if (numChunkDocs >= numDocsInChunk) DiskMerge (collectFilename);
+  
+  return COMPALLOK;
+}
+
+
+static void CondenseInvfFile (char *filename, unsigned long &bytesOutput) {
+  FILE *inInvfFile = open_file (filename, INVF_SUFFIX, "rb",
+				MAGIC_INVF, MG_ABORT);
+  FILE *outInvfFile = open_file (filename, INVF_SUFFIX, "rb+",
+				 MAGIC_INVF, MG_ABORT);
+
+  // skip the magic number
+  fseek (outInvfFile, sizeof (unsigned long), SEEK_SET);
+  
+  // write the inverted file header -- use defaults for most things
+  invf_file_header ifh;
+  ifh.no_of_words = idh.word_dict_size;
+  ifh.no_of_tags = idh.tag_dict_size;
+  ifh.word_level_index = (ivfLevel.indexLevel.empty()) ? 1 : 0;
+  ifh.Write (outInvfFile);
+
+  bytesOutput = ftell (outInvfFile);
+
+  // process each meaningful byte in the file
+  unsigned long numEntries = ifh.no_of_words + ifh.no_of_tags;
+  unsigned long entryNum;
+  mg_ullong lastStart = 0;
+  for (entryNum = 0; entryNum < numEntries; entryNum++) {
+    InvfStateRec &stateRec = invfState.GetRec (entryNum);
+
+    // overrun check
+    if (stateRec.start < lastStart)
+      FatalError (1, "Inverted file Buffer overrun");
+    lastStart = stateRec.start;
+    
+    unsigned long oldEntryStart = stateRec.start >> 3;
+    unsigned long oldEntryStartOver = stateRec.start & 7; // should be 0
+    unsigned long oldEntryEnd = (stateRec.here + 7) >> 3;  // byte after end
+    unsigned long oldEntryEndOver = stateRec.here & 7;
+
+    fseek (inInvfFile, oldEntryStart, SEEK_SET);
+
+    stateRec.here -= stateRec.start;
+    stateRec.start = (mg_ullong)bytesOutput * 8 + oldEntryStartOver;
+    stateRec.here += stateRec.start;
+    while (oldEntryStart < oldEntryEnd) {
+      unsigned char c = getc (inInvfFile);
+      if (oldEntryStart == oldEntryEnd - 1) {
+	u_char ands[8] =
+	{0xff, 0x80, 0xc0, 0xe0, 0xf0, 0xf8, 0xfc, 0xfe};
+	c &= ands[oldEntryEndOver];
+      }
+      putc (c, outInvfFile);
+      bytesOutput++;
+      oldEntryStart++;
+    }
+  }  
+
+  fclose (inInvfFile);
+
+#ifdef __WIN32__
+  if (_chsize (_fileno (outInvfFile), bytesOutput) != 0)
+    Message ("Could not truncate invf.");
+#else
+  ftruncate (fileno (outInvfFile), bytesOutput);
+#endif
+
+  fclose (outInvfFile);
+}
+
+static void OutputInvfIdx (char *filename, unsigned long invfNumBytes) {
+  FILE *invfIdxFile = create_file (filename, INVF_IDX_SUFFIX, "wb",
+				   MAGIC_INVI, MG_ABORT);
+
+  // process each meaningful byte in the file
+  unsigned long numEntries = idh.word_dict_size + idh.tag_dict_size;
+  unsigned long entryNum;
+  for (entryNum = 0; entryNum < numEntries; entryNum++) {
+    InvfStateRec &stateRec = invfState.GetRec (entryNum);
+
+    // assumes that inverted entries start at beginning of each byte
+    if (!WriteUL (invfIdxFile, (stateRec.start >> 3))) break;
+  }
+
+  WriteUL (invfIdxFile, invfNumBytes);
+
+  fclose (invfIdxFile);
+}
+
+
+int done_ivf_2 (const TagInfo &/*tagInfo*/, char *filename) {
+  // close most open files
+  if (chunkFile != NULL) {
+    chunkBuf.done();
+    fclose (chunkFile);
+    chunkFile = NULL;
+  }
+  occurConvert.Close();
+  
+  // free allocated memory
+  bitPtrs.Clear();
+  if (ivfMemBuf != NULL) { delete [] ivfMemBuf; ivfMemBuf = NULL; }
+  free_perf_hash (wordHashDict);
+  wordHashDict = NULL;
+  tagMapDict.erase (tagMapDict.begin(), tagMapDict.end());
+
+  // condense the inverted file and truncate it
+  // this function also writes out the inverted header
+  unsigned long invfNumBytes = 0;
+  CondenseInvfFile (filename, invfNumBytes);
+
+  OutputInvfIdx (filename, invfNumBytes);
+
+  // close the rest of the open files
+  invfState.Close ();
+  
+  return COMPALLOK;
+}
Index: /trunk/mgpp/text/locallib.cpp
===================================================================
--- /trunk/mgpp/text/locallib.cpp	(revision 3365)
+++ /trunk/mgpp/text/locallib.cpp	(revision 3365)
@@ -0,0 +1,218 @@
+/**************************************************************************
+ *
+ * locallib.cpp -- Misc functions
+ * Copyright (C) 1994  Gary Eddy, Alistair Moffat and Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "memlib.h"
+#include "bitio_gen.h"
+#include "huffman.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "locallib.h"
+#include "text.h"
+
+int 
+vecentropy (int *A, int n)
+{
+  int total = 0, i;
+  double bits = 0.0;
+
+  for (i = 0; i < n; i++)
+    total += A[i];
+  for (i = 0; i < n; i++)
+    if (A[i])
+      bits += A[i] * log2 (1.0 * total / A[i]);
+  return ((int) (bits + 0.9999));
+
+}
+
+static void 
+siftdown (int *A, int i, int n)
+{
+  int ms, t;
+
+  while (2 * i + 1 <= n - 1)
+    {
+      if (2 * i + 1 == n - 1 || A[2 * i + 1] < A[2 * i + 2])
+	ms = 2 * i + 1;
+      else
+	ms = 2 * i + 2;
+      if (A[i] > A[ms])
+	{
+	  t = A[i];
+	  A[i] = A[ms];
+	  A[ms] = t;
+	}
+      i = ms;
+    }
+}
+
+static void 
+buildheap (int *A, int n)
+{
+  int i;
+  for (i = n / 2 - 1; i >= 0; i--)
+    siftdown (A, i, n);
+}
+
+int 
+huffcodebits (unsigned long *A, int n)
+{
+  int i, j, tot = 0, bits = 0, v1, v2, *B;
+
+  B = (int *) Xmalloc (n * sizeof (int));
+  j = 0;
+  for (i = 0; i < n; i++)
+    {
+      if (A[i])
+	{
+	  tot += (B[j++] = A[i]);
+	}
+    }
+  n = j;
+
+  if (n == 0 || tot == 0)
+    return (0);
+
+  buildheap (B, n);
+  while (n > 1)
+    {
+      v1 = B[0];
+      B[0] = B[n - 1];
+      n--;
+      siftdown (B, 0, n);
+      v2 = B[0];
+      B[0] = v1 + v2;
+      siftdown (B, 0, n);
+      bits += v1 + v2;
+    }
+  Xfree (B);
+  return (bits);
+}
+
+
+int 
+modelbits (unsigned long *A, int n)
+{
+  int i, bits = 0, last, N = 0;
+
+  last = -1;
+  for (i = 0; i < n; i++)
+    {
+      if (A[i])
+	{
+	  bits += BIO_Gamma_Length (i - last) + BIO_Gamma_Length (A[i]);
+	  last = i;
+	  N++;
+	}
+    }
+  if (N == 0)
+    return (0);
+  bits += BIO_Gamma_Length (N);
+  return (bits);
+}
+
+
+
+
+
+/* Find the next prime number larger than p */
+int 
+prime (int p)
+{
+  int limit;
+  p += (p & 1) + 1;
+  limit = (int) sqrt ((double) p) + 1;
+  do
+    {
+      int j;
+      while (limit * limit < p)
+	limit++;
+      for (j = 3; j <= limit && p % j; j += 2)
+	;			/* NULLBODY */
+      if (j > limit)
+	break;
+      p += 2;
+    }
+  while (1);
+  return (p);
+}
+
+
+
+int Read_cdh (FILE * f, compression_dict_header * cdh, u_long * /*mem*/,
+	      u_long * disk) {
+  if (disk)
+    (*disk) += sizeof (*cdh);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  if (fread (cdh, sizeof (*cdh), 1, f) == 1)
+    {
+      int i;
+      NTOHUL(cdh->dict_type);
+      NTOHUL(cdh->novel_method);
+      for (i = 0; i < TEXT_PARAMS; i++)
+	NTOHUL(cdh->params[i]);
+      NTOHUL(cdh->num_words[0]);
+      NTOHUL(cdh->num_words[1]);
+      NTOHUL(cdh->num_word_chars[0]);
+      NTOHUL(cdh->num_word_chars[1]);
+      NTOHUL(cdh->lookback);
+      return 0;
+    }
+  else
+    return -1;
+}
+
+
+int 
+Read_cfh (FILE * f, comp_frags_header * cfh, u_long * mem, u_long * disk)
+{
+  int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+  if (Read_Huffman_Data (f, &cfh->hd, mem, disk) == -1)
+    return -1;
+  if (fread (&cfh->uncompressed_size,
+	     sizeof (cfh->uncompressed_size), 1, f) != 1)
+    return -1;
+  NTOHUL(cfh->uncompressed_size);  /* [RPAP - Jan 97: Endian Ordering] */
+
+  if (disk)
+    (*disk) += sizeof (cfh->uncompressed_size);
+
+  memset (cfh->huff_words_size, '\0', sizeof (cfh->huff_words_size));
+
+  if ((int)fread (cfh->huff_words_size + cfh->hd.mincodelen,
+	     sizeof (*cfh->huff_words_size),
+	     cfh->hd.maxcodelen - cfh->hd.mincodelen + 1, f) !=
+      cfh->hd.maxcodelen - cfh->hd.mincodelen + 1)
+    return -1;
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = cfh->hd.mincodelen; i < cfh->hd.maxcodelen + 1; i++)
+    NTOHUL(cfh->huff_words_size[i]);
+
+  if (disk)
+    (*disk) += sizeof (*cfh->huff_words_size) *
+      (cfh->hd.maxcodelen - cfh->hd.mincodelen + 1);
+  return 0;
+}
+
Index: /trunk/mgpp/text/locallib.h
===================================================================
--- /trunk/mgpp/text/locallib.h	(revision 3365)
+++ /trunk/mgpp/text/locallib.h	(revision 3365)
@@ -0,0 +1,45 @@
+/**************************************************************************
+ *
+ * locallib.h -- Misc functions
+ * Copyright (C) 1994  Gary Eddy, Alistair Moffat and Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef LOCALLIB_H
+#define LOCALLIB_H
+
+#include "sysfuncs.h"
+
+#include "text.h"
+
+#define NUMOF(a) (sizeof(a)/sizeof((a)[0]))
+
+int vecentropy (int *A, int n);
+
+int huffcodebits (unsigned long *A, int n);
+
+int modelbits (unsigned long *A, int n);
+
+int prime (int p);
+
+
+int Read_cdh (FILE * f, compression_dict_header * cdh, u_long * mem, u_long * disk);
+
+int Read_cfh (FILE * f, comp_frags_header * cfh, u_long * mem, u_long * disk);
+
+
+#endif
Index: /trunk/mgpp/text/mg.h
===================================================================
--- /trunk/mgpp/text/mg.h	(revision 3365)
+++ /trunk/mgpp/text/mg.h	(revision 3365)
@@ -0,0 +1,37 @@
+/**************************************************************************
+ *
+ * mg.h -- General header file
+ * Copyright (C) 1994  Neil Sharman, Alistair Moffat and Lachlan Andrew
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+
+#ifndef H_MG
+#define H_MG
+
+
+#define COMPALLOK	(0)
+#define COMPERROR	(-1)
+
+#define MAXSTRING       3000000
+#define BUFFERSIZE      (MAXSTRING+30000)
+
+#define	MAXLN	65536
+#define WORDS           1
+#define NONWORDS        0
+
+#endif
Index: /trunk/mgpp/text/mg.special.cpp
===================================================================
--- /trunk/mgpp/text/mg.special.cpp	(revision 3365)
+++ /trunk/mgpp/text/mg.special.cpp	(revision 3365)
@@ -0,0 +1,58 @@
+/**************************************************************************
+ *
+ * mg.special.cpp -- Special pass for mg_passes
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "sysfuncs.h"
+#include "mg.h"
+#include "build.h"
+
+int init_special (const TagInfo &/*tagInfo*/, char * /*FileName*/) {
+  return COMPALLOK;
+}
+
+int process_special (const TagInfo &/*tagInfo*/, const TextElArray &doc) {
+  // print out the document
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    if ((*here).elType == OpenTagE) {
+      cout << "** OpenTag  \"" << (*here).tagName << "\"\tText \""
+	   << (*here).text << "\"\n"; 
+    } else if ((*here).elType == CloseTagE) {
+      cout << "** CloseTag \"" << (*here).tagName << "\"\tText \""
+	   << (*here).text << "\"\n";
+    } else {
+      cout << "** Text     \"" << (*here).text << "\"\n";
+    }
+    
+    here++;
+  }
+  
+  return COMPALLOK;
+}
+
+int done_special (const TagInfo &/*tagInfo*/, char * /*FileName*/) {
+  return COMPALLOK;
+}
Index: /trunk/mgpp/text/mg_errors.cpp
===================================================================
--- /trunk/mgpp/text/mg_errors.cpp	(revision 3365)
+++ /trunk/mgpp/text/mg_errors.cpp	(revision 3365)
@@ -0,0 +1,72 @@
+/**************************************************************************
+ *
+ * mg_errors.cpp -- Error related stuff for mgquery
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+#include "sysfuncs.h"
+#include "memlib.h"
+
+#include "mg_errors.h"
+
+int mg_errno = MG_NOERROR;
+
+char *mg_errorstrs[] =
+{
+  "No error",
+  "Out of memory",
+  "File \"%s\" not found",
+  "Bad magic number in \"%s\"",
+  "Error reading \"%s\"",
+  "MG_BUFTOOSMALL",
+  "Files required for level 2 and 3 inversion are missing"};
+
+
+/* [RJM 07/98: Memory Leak] -- rest of file */
+
+static char null_data[] = "";
+char *mg_error_data = null_data;
+
+
+void MgErrorData (char *s)
+{
+  /* free the current error string, unless it is the null string */
+  if ((mg_error_data != NULL) && (mg_error_data != null_data)) {
+    delete mg_error_data;
+    mg_error_data = null_data;
+  }
+
+  /* make a copy of the string */
+  mg_error_data = Xstrdup (s);
+
+  /* if the alloc failed set the error to the null string */
+  if (!mg_error_data) mg_error_data = null_data;
+}
+
+
+void MgErrorDeinit (void)
+{
+  /* free the current error string, unless it is the null string */
+  if ((mg_error_data != NULL) && (mg_error_data != null_data)) {
+    delete mg_error_data;
+    mg_error_data = null_data;
+  }
+}
Index: /trunk/mgpp/text/mg_errors.h
===================================================================
--- /trunk/mgpp/text/mg_errors.h	(revision 3365)
+++ /trunk/mgpp/text/mg_errors.h	(revision 3365)
@@ -0,0 +1,37 @@
+/**************************************************************************
+ *
+ * mg_errors.h -- Error related stuff for mgquery
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+
+extern int mg_errno;
+extern char *mg_error_data;
+extern char *mg_errorstrs[];
+
+
+#define MG_NOERROR 0
+#define MG_NOMEM 1
+#define MG_NOFILE 2
+#define MG_BADMAGIC 3
+#define MG_READERR 4
+#define MG_BUFTOOSMALL 5
+#define MG_INVERSION 6
+
+void MgErrorData (char *s);
+void MgErrorDeinit (void);
Index: /trunk/mgpp/text/mg_files.cpp
===================================================================
--- /trunk/mgpp/text/mg_files.cpp	(revision 3365)
+++ /trunk/mgpp/text/mg_files.cpp	(revision 3365)
@@ -0,0 +1,259 @@
+/**************************************************************************
+ *
+ * mg_files.cpp -- Routines for handling files for the auxillary programs
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "mg_files.h"
+
+
+/* This must contain a valid path without a trailing slash */
+static char *basepath = NULL;
+
+/* This sets the base path for all file operations */
+void 
+set_basepath (const char *bp)
+{
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef __WIN32__
+  basepath = "";
+#else
+  char *s;
+  /* Free the memory for the base path if it has already been allocated */
+  if (basepath)
+    {
+      delete basepath;
+      basepath = NULL;
+    }
+
+  if (!bp || *bp == '\0')
+    bp = ".";
+
+  s = strrchr (bp, '/');
+  if (s && *(s + 1) == '\0')
+    {
+      basepath = new char[strlen (bp) + 2];
+      if (!basepath)
+	return;
+      strcpy (basepath, bp);
+      strcat (basepath, ".");
+    }
+  else
+    basepath = Xstrdup (bp);
+#endif
+}
+
+
+
+/* return the currently defined basepath */
+char *
+get_basepath (void)
+{
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  return basepath;
+}
+
+
+
+/* This generates the name of a file. It places the name in the buffer
+   specified or if that is NULL it uses a static buffer. */
+char *
+make_name (const char *name, const char *suffix, char *buffer)
+{
+  static char path[512];
+  if (!buffer)
+    buffer = path;
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  sprintf (buffer, FILE_NAME_FORMAT, basepath, name, suffix);  /* [RPAP - Feb 97: WIN32 Port] */
+  return buffer;
+}
+
+
+
+/* This will open the specified file and check its magic number.
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+open_named_file (const char *name, const char *mode,
+		 u_long magic_num, int err_mode)
+{
+  unsigned long magic;
+  FILE *f = NULL;
+  char *err;
+  f = fopen (name, mode);
+
+  if (!f)
+    {
+      err = "Unable to open \"%s\"";
+      goto error;
+    }
+
+  if (magic_num)
+    {
+      if (fread (&magic, sizeof (magic), 1, f) != 1)
+	{
+	  err = "No magic number \"%s\"";
+	  goto error;
+	}
+
+      NTOHUL(magic);  /* [RPAP - Jan 97: Endian Ordering] */
+
+      if (!IS_MAGIC (magic))
+	{
+	  err = "No MG magic number \"%s\"";
+	  goto error;
+	}
+
+      if (magic != magic_num)
+	{
+	  err = "Wrong MG magic number \"%s\"";
+	  goto error;
+	}
+    }
+  return f;
+
+error:
+  if (f)
+    fclose (f);
+  if (err_mode == MG_ABORT)
+    FatalError (1, err, name);
+  if (err_mode == MG_MESSAGE)
+    Message (err, name);
+  return NULL;
+}
+
+
+
+/* This will open the specified file and check its magic number.
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+open_file (const char *name, const char *suffix, const char *mode,
+	   u_long magic_num, int err_mode)
+{
+  char path[512];
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  sprintf (path, FILE_NAME_FORMAT, basepath, name, suffix);  /* [RPAP - Feb 97: WIN32 Port] */
+  return open_named_file (path, mode, magic_num, err_mode);
+}
+
+
+
+
+
+
+/* This will create the specified file and set its magic number.
+
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+create_named_file (const char *name, const char *mode,
+		   u_long magic_num, int err_mode)
+{
+  FILE *f = NULL;
+  char *err;
+  f = fopen (name, mode);
+
+  if (!f)
+    {
+      err = "Unable to open \"%s\"";
+      goto error;
+    }
+
+  if (magic_num)
+    HTONUL(magic_num);  /* [RPAP - Jan 97: Endian Ordering] */
+    if (fwrite (&magic_num, sizeof (magic_num), 1, f) != 1)
+      {
+	err = "Couldn't write magic number \"%s\"";
+	goto error;
+      }
+
+  return f;
+
+error:
+  if (f)
+    fclose (f);
+  if (err_mode == MG_ABORT)
+    FatalError (1, err, name);
+  if (err_mode == MG_MESSAGE)
+    Message (err, name);
+  return NULL;
+}
+
+
+
+
+
+
+
+/* This will create the specified file and set its magic number.
+
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *
+create_file (const char *name, const char *suffix, const char *mode,
+	     u_long magic_num, int err_mode)
+{
+  char path[512];
+  if (!basepath)
+    set_basepath (getenv ("MGDATA"));
+  sprintf (path, FILE_NAME_FORMAT, basepath, name, suffix);  /* [RPAP - Feb 97: WIN32 Port] */
+  return create_named_file (path, mode, magic_num, err_mode);
+}
Index: /trunk/mgpp/text/mg_files.h
===================================================================
--- /trunk/mgpp/text/mg_files.h	(revision 3365)
+++ /trunk/mgpp/text/mg_files.h	(revision 3365)
@@ -0,0 +1,343 @@
+/**************************************************************************
+ *
+ * mg_files.h -- Routines for handling files for the auxillary programs
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef MG_FILES_H
+#define MG_FILES_H
+
+#include "sysfuncs.h"
+
+/* Magic numbers for the different types of files */
+
+#define GEN_MAGIC(a,b,c,d) ((unsigned long)(((a)<<24) + ((b)<<16) + \
+					    ((c)<<8) + (d)))
+
+#define MAGIC_XXXX		GEN_MAGIC('M','G', 0 , 0)
+#define MAGIC_STATS_DICT	GEN_MAGIC('M','G','S','D')
+#define MAGIC_AUX_DICT		GEN_MAGIC('M','G','A','D')
+#define MAGIC_FAST_DICT		GEN_MAGIC('M','G','F','D')
+#define MAGIC_DICT		GEN_MAGIC('M','G','D', 0 )
+#define MAGIC_STEM_BUILD	GEN_MAGIC('M','G','S', 0 )
+#define MAGIC_HASH		GEN_MAGIC('M','G','H', 0 )
+#define MAGIC_STEM		GEN_MAGIC('M','G','s', 0 )
+#define MAGIC_CHUNK		GEN_MAGIC('M','G','C', 0 )
+#define MAGIC_CHUNK_TRANS	GEN_MAGIC('M','G','c', 0 )
+#define MAGIC_TEXT		GEN_MAGIC('M','G','T', 0 )
+#define MAGIC_TEXI		GEN_MAGIC('M','G','t', 0 )
+#define MAGIC_TEXT_LEVELS	GEN_MAGIC('M','G','t','L')
+#define MAGIC_TEXI_WGT		GEN_MAGIC('M','G','t','W')
+#define MAGIC_INVF		GEN_MAGIC('M','G','I', 0 )
+#define MAGIC_INVF_LEVELS	GEN_MAGIC('M','G','I','L')
+#define MAGIC_INVI		GEN_MAGIC('M','G','i', 0 )
+#define MAGIC_WGHT		GEN_MAGIC('M','G','W', 0 )
+#define MAGIC_WGHT_APPROX	GEN_MAGIC('M','G','w', 0 )
+#define MAGIC_PARAGRAPH		GEN_MAGIC('M','G','P', 0 )
+/* [RPAP - Jan 97: Stem Index Change] */
+#define MAGIC_STEM_1            GEN_MAGIC('M','G','s','1')
+#define MAGIC_STEM_2            GEN_MAGIC('M','G','s','2')
+#define MAGIC_STEM_3            GEN_MAGIC('M','G','s','3')
+
+#define IS_MAGIC(a) ((((u_long)(a)) & 0xffff0000) == MAGIC_XXXX)
+
+
+/* err_mode values for open_file and create_file */
+#define MG_ABORT 0
+#define MG_MESSAGE 1
+#define MG_CONTINUE 2
+
+
+
+
+
+/* File suffixes */
+
+
+/* The compression dictionary built by txt.pass1 */
+#ifdef SHORT_SUFFIX
+# define TEXT_STATS_DICT_SUFFIX	 ".tsd"
+#else
+# define TEXT_STATS_DICT_SUFFIX	 ".text.stats"
+#endif
+
+/* The compression dictionary built by text.pass1 and comp_dict.process */
+#ifdef SHORT_SUFFIX
+# define TEXT_DICT_SUFFIX	 ".td"
+#else
+# define TEXT_DICT_SUFFIX	 ".text.dict"
+#endif
+
+/* The compression dictionary built by mg_make_fast_dict */
+#ifdef SHORT_SUFFIX
+# define TEXT_DICT_FAST_SUFFIX	 ".tdf"
+#else
+# define TEXT_DICT_FAST_SUFFIX	 ".text.dict.fast"
+#endif
+
+/* The auxilary dictionary built by text.pass2 */
+#ifdef SHORT_SUFFIX
+# define TEXT_DICT_AUX_SUFFIX	 ".tda"
+#else
+# define TEXT_DICT_AUX_SUFFIX	 ".text.dict.aux"
+#endif
+
+/* The compressed text build by text.pass2 */
+#ifdef SHORT_SUFFIX
+# define TEXT_SUFFIX		 ".t"
+#else
+# define TEXT_SUFFIX		 ".text"
+#endif
+
+/* The combined compressed text index and document weight file */
+#ifdef SHORT_SUFFIX
+# define TEXT_IDX_WGT_SUFFIX	 ".tiw"
+#else
+# define TEXT_IDX_WGT_SUFFIX	 ".text.idx.wgt"
+#endif
+
+/* The compressed text index file */
+#ifdef SHORT_SUFFIX
+# define TEXT_IDX_SUFFIX	 	".ti"
+#else
+# define TEXT_IDX_SUFFIX	 	".text.idx"
+#endif
+
+/* The document levels in the text */
+#ifdef SHORT_SUFFIX
+# define TEXT_LEVEL_SUFFIX	 	".tl"
+#else
+# define TEXT_LEVEL_SUFFIX	 	".text.level"
+#endif
+
+/* The dictionary of stemmed words build by invf.pass1 and ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_SUFFIX         ".id"
+#else
+# define INVF_DICT_SUFFIX         ".invf.dict"
+#endif
+
+/* The dictionary of stemmed words build by stem.process */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_SUFFIX ".idb"
+#else
+# define INVF_DICT_BLOCKED_SUFFIX ".invf.dict.blocked"
+#endif
+
+/* The exact document weights file build by make.weights, invf.pass2,
+   or ivf.pass2 */
+#ifdef SHORT_SUFFIX
+# define WEIGHTS_SUFFIX           ".w"
+#else
+# define WEIGHTS_SUFFIX           ".weight"
+#endif
+
+/* The approximate weights file built by make.weights */
+#ifdef SHORT_SUFFIX
+# define APPROX_WEIGHTS_SUFFIX    ".wa"
+#else
+# define APPROX_WEIGHTS_SUFFIX    ".weight.approx"
+#endif
+
+/* The inverted file build by invf.pass2 or ivf.pass2 */
+#ifdef SHORT_SUFFIX
+# define INVF_SUFFIX              ".i"
+#else
+# define INVF_SUFFIX              ".invf"
+#endif
+
+/* The level information for the inverted text */
+#ifdef SHORT_SUFFIX
+# define INVF_LEVEL_SUFFIX              ".il"
+#else
+# define INVF_LEVEL_SUFFIX              ".invf.level"
+#endif
+
+/* The inverted file index build by invf.pass2 or ivf.pass2 */
+#ifdef SHORT_SUFFIX
+# define INVF_IDX_SUFFIX          ".ii"
+#else
+# define INVF_IDX_SUFFIX          ".invf.idx"
+#endif
+
+/* The inverted file chunk descriptor built by ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_CHUNK_SUFFIX        ".ic"
+#else
+# define INVF_CHUNK_SUFFIX        ".invf.chunk"
+#endif
+
+/* The word index translation file built by ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_CHUNK_TRANS_SUFFIX  ".ict"
+#else
+# define INVF_CHUNK_TRANS_SUFFIX  ".invf.chunk.trans"
+#endif
+
+/* The hashed stemmed dictionary built by make.perf_hash */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_HASH_SUFFIX    ".idh"
+#else
+# define INVF_DICT_HASH_SUFFIX    ".invf.dict.hash"
+#endif
+
+/* The paragraph descriptior file built by invf.pass1 or ivf.pass1 */
+#ifdef SHORT_SUFFIX
+# define INVF_PARAGRAPH_SUFFIX    ".ip"
+#else
+# define INVF_PARAGRAPH_SUFFIX    ".invf.paragraph"
+#endif
+
+/* The trace file build by mg.builder. */
+#ifdef SHORT_SUFFIX
+# define TRACE_SUFFIX             ".trc"
+#else
+# define TRACE_SUFFIX             ".trace"
+#endif
+
+/* The compression stats file build by mg.builder. */
+#ifdef SHORT_SUFFIX
+# define COMPRESSION_STATS_SUFFIX ".cs"
+#else
+# define COMPRESSION_STATS_SUFFIX ".compression.stats"
+#endif
+
+/* [RPAP - Jan 97: Stem Index Change] */
+/* The casefolded index into the stemmed dictionary */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_1_SUFFIX ".ib1"
+#else
+# define INVF_DICT_BLOCKED_1_SUFFIX ".invf.dict.blocked.1"
+#endif
+
+/* [RPAP - Jan 97: Stem Index Change] */
+/* The stemmed index into the stemmed dictionary */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_2_SUFFIX ".ib2"
+#else
+# define INVF_DICT_BLOCKED_2_SUFFIX ".invf.dict.blocked.2"
+#endif
+
+/* [RPAP - Jan 97: Stem Index Change] */
+/* The casefolded and stemmed  index into the stemmed dictionary */
+#ifdef SHORT_SUFFIX
+# define INVF_DICT_BLOCKED_3_SUFFIX ".ib3"
+#else
+# define INVF_DICT_BLOCKED_3_SUFFIX ".invf.dict.blocked.3"
+#endif
+
+/* [RPAP - Feb 97: WIN32 Port] */
+#ifdef __WIN32__
+#define FILE_NAME_FORMAT "%s%s%s"
+#else
+#define FILE_NAME_FORMAT "%s/%s%s"
+#endif
+
+
+/* This sets the base path for all file operations */
+void set_basepath (const char *bp);
+
+
+/* return the currently defined basepath */
+char *get_basepath (void);
+
+
+
+
+/* This generates the name of a file. It places the name in the buffer
+   specified or if that is NULL it uses a static buffer. */
+char *make_name (const char *name, const char *suffix, char *buffer);
+
+
+
+
+
+
+
+/* This will open the specified file and check its magic number.
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *open_named_file (const char *name, const char *mode,
+		       u_long magic_num, int err_mode);
+
+
+
+
+/* This will open the specified file and check its magic number.
+
+   err_mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *open_file (const char *name, const char *suffix, const char *mode,
+		 u_long magic_num, int err_mode);
+
+
+
+
+
+/* This will create the specified file and set its magic number.
+
+   Mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *create_named_file (const char *name, const char *mode,
+			 u_long magic_num, int err_mode);
+
+
+
+/* This will create the specified file and set its magic number.
+
+   err_mode may take on the following values
+   MG_ABORT    : causes an error message to be generated and the
+   program aborted if there is an error.
+   MG_MESSAGE  : causes a message to be generated and a NULL value to
+   be returned if there is an error.
+   MG_CONTINUE : causes a NULL value to be returned if there is an error.
+
+   On success if returns the FILE *. On failure it will return a NULL value
+   and possibly generate an error message, or it will exit the program with
+   an error message.   */
+FILE *create_file (const char *name, const char *suffix, const char *mode,
+		   u_long magic_num, int err_mode);
+
+
+
+
+#endif
Index: /trunk/mgpp/text/mgpp_compression_dict.1
===================================================================
--- /trunk/mgpp/text/mgpp_compression_dict.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_compression_dict.1	(revision 3365)
@@ -0,0 +1,188 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_compression_dict 1 \*(Dt CITRI
+.SH NAME
+mgpp_compression_dict \- build a compression dictionary.
+.SH SYNOPSIS
+.B mgpp_compression_dict
+[
+.B \-h
+]
+[
+.BR \-C " |"
+.BR \-P " |"
+.B  \-S
+]
+.if n .ti +9n
+[
+.BR \-0 " |"
+.BR \-1 " |"
+.BR \-2 " |"
+.B  \-3
+]
+[
+.BR \-H " |"
+.BR \-B " |"
+.BR \-D " |"
+.BR \-Y " |"
+]
+.if n .ti +9n
+.if t .ti +.5i
+[
+.BI \-l " lookback"
+]
+[
+.BI \-k " mem"
+]
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_compression_dict
+builds a compression dictionary based on the statistics gathered
+during the first pass over the text.  The options to the program are
+mainly concerned with limiting the amount of memory the dictionary
+will use and with how the text compressor will cope with any novel
+words found during the compression phase.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.B \-C
+Build a complete dictionary from the statistics file.  If during the
+text compression phase a novel word is found, then the compressor will
+produce an error message and stop.
+.TP
+.B \-P
+Build a partial dictionary from the statistics file.  This dictionary
+assumes that the statistics file are based on the entire text.  The
+statistics of words not includes in the dictionary are used to
+calculate the escape probability.  If novel words are being coded
+character by character, then there may not be a Huffman code for every
+possible character.  This means that the compressor may fail if a novel
+word contains a novel character.
+.TP
+.B \-S
+Build a seed dictionary from the statistics file.  This dictionary
+assumes that the statistics file is based on only a portion of the
+text to be compressed.  The probability of a novel word is based on the
+number of words that have only occurred once.  If novel words are being
+coded character by character, then the Huffman codes for characters are
+based on the frequency of characters in the dictionary.
+.TP
+.B \-0
+All words from the statistics file are included in the built
+dictionary.
+.TP
+.B \-1
+Words are included in the dictionary until the dictionary reaches the
+desired size.  Words are selected for the dictionary based on the order
+they occurred in the source text.
+.TP
+.B \-2
+Words are included in the dictionary until the dictionary reaches the
+desired size.  The most frequent words are included in the dictionary
+first; where there is a tie for frequency, the shortest word is
+included first.
+.TP
+.B \-3
+Words are included in the dictionary until the dictionary reaches the
+desired size.  The most frequent words are included in the dictionary
+first; where there is a tie for frequency, the shortest word is
+included first.  Words are the shuffled back and forth between the
+`keep' and `discard' lists to find the `optimal' set of words that
+should be in the dictionary.
+.TP
+.B \-H
+This specifies that novel words will be coded character by character
+using Huffman codes.
+.TP
+.B \-B
+This specifies that an auxiliary dictionary will be built by the
+compressor.  Each novel word found will be placed at the end of the
+auxiliary dictionary.  Novel words will be coded in the compressed text
+using binary codes.  The binary code represents their occurrence
+position in the auxiliary dictionary.
+.TP
+.B \-D
+This specifies that an auxiliary dictionary will be built by the
+compressor.  Each novel word found will be placed at the end of the
+auxiliary dictionary.  Novel words will be coded in the compressed text
+using delta codes.  The delta code represents their occurrence position
+in the auxiliary dictionary.
+.TP
+.B \-Y
+This specifies that an auxiliary dictionary will be built by the
+compressor.  Each novel word found will be placed at the end of the
+auxiliary dictionary.  Novel words will be coded in the compressed text
+using a combination of gamma and binary codes.  The code represents
+their occurrence position in the auxiliary dictionary.  This generally
+produces better compression than
+.B \-B
+or
+.BR \-D .
+.TP
+.BI \-l " lookback"
+The generated dictionary is designed to be front coded when it is
+loaded into memory.  Under normal circumstances, a front-coded
+dictionary would require scanning from the beginning in order to find
+any particular word.  However, every
+.I lookback
+words in the dictionary, the whole word is stored and a pointer to that
+word maintained.  E.g., if
+.I lookback
+is 4, then every fourth word is stored in its entirety.
+.TP
+.BI \-k " mem"
+This limits the amount of memory to use for the generated
+dictionary.  Words are selected for the dictionary based of the text
+statistics, and whether
+.BR \-0 , " \-1" , " \-2"
+or
+.B \-3
+is specified.  The memory is calculated assuming a lookback of 0,
+irrespective of what actual lookback is specified.  This means that if
+a non-zero lookback is given, the dictionary will actually occupy
+less space than specified by
+.BR \-k .
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.text.stats
+Statistics about the source text.
+.TP
+.B *.text.dict
+Compression dictionary for the source text.
+.SH "SEE ALSO"
+.na
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/mgpp/text/mgpp_compression_dict.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_compression_dict.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_compression_dict.cpp	(revision 3365)
@@ -0,0 +1,1021 @@
+/**************************************************************************
+ *
+ * mgpp_compression_dict.cpp -- Routines for creating compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "bitio_gen.h"
+#include "bitio_m_stdio.h"
+#include "mgheap.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "mg_files.h"
+#include "locallib.h"
+#include "invf.h"
+#include "text.h"
+#include "words.h"
+#include "mg.h"
+#include "WordData.h"
+
+#define MAXBITS (sizeof(unsigned long) * 8)
+
+#define is_power_of_two(a) ((a) != 0 && (((a) & ((a)-1)) == 0))
+
+#define MAX_RECALCULATIONS 100
+
+typedef struct DictWordData : public WordData
+  {
+    float saving;
+    float char_bit_cost;
+    u_long num_trans;
+    u_char *word;
+  }
+DictWordData;
+
+static DictWordData *Words[2];
+static u_long Num[2];
+static u_long chars[2];
+
+#define KIND(p) (((p) >= Words[0] && (p) < Words[0] + Num[0]) ? 0 : 1)
+#define IsWord(p) (((p) >= Words[1] && (p) < Words[1] + Num[1]) ? 1 : 0)
+#define IsNonWord(p) (((p) >= Words[0] && (p) < Words[0] + Num[0]) ? 1 : 0)
+
+
+typedef struct DictData
+  {
+    DictWordData **wd;
+    u_long num_wds;
+    u_long chars;
+  }
+DictData;
+
+typedef DictData DictInfo[2];
+
+static DictInfo keep, discard, all;
+static compression_stats_header csh;
+
+static char *file_name = "";
+static u_long novel_method = MG_NOVEL_HUFFMAN_CHARS;
+
+
+static void ReadInWords (char *);
+static void Select_on (int k, heap_comp hc);
+static void Method3 (int k);
+static void Select_all (void);
+static u_long WriteOutWords (char *, u_long, int);
+static int DecFreqIncWL (void *a, void *b);
+static int OccuranceOrder (void *a, void *b);
+
+
+
+int main (int argc, char **argv)
+{
+  int ch;
+  char type = 'C';
+  char mode = '0';
+  int lookback = 2;
+  double k = 0;
+  u_long mem_reqd;
+  opterr = 0;
+  msg_prefix = argv[0];
+  while ((ch = getopt (argc, argv, "0123CPSf:d:l:hk:HDY")) != -1)
+    switch (ch)
+      {
+      case 'H':
+	novel_method = MG_NOVEL_HUFFMAN_CHARS;
+	break;
+      case 'D':
+	novel_method = MG_NOVEL_DELTA;
+	break;
+      case 'Y':
+	novel_method = MG_NOVEL_HYBRID;
+	break;
+      case 'f':		/* input file */
+	file_name = optarg;
+	break;
+      case 'd':
+	set_basepath (optarg);
+	break;
+      case 'C':
+      case 'P':
+      case 'S':
+	type = ch;
+	break;
+      case '0':
+      case '1':
+      case '2':
+      case '3':
+	mode = ch;
+	break;
+      case 'l':
+	lookback = atoi (optarg);
+	if (!is_power_of_two (lookback))
+	  FatalError (1, "The lookback value must be a power of 2");
+	lookback = floorlog_2 (lookback);
+	break;
+      case 'k':
+	k = atof (optarg) * 1024;
+	break;
+      case 'h':
+      case '?':
+	fprintf (stderr, "usage: %s [-l lookback] [-f input_file]"
+	"[-d data directory] [-h] [-0|-1|-2|-3] [-C|-P|-S] [-k mem (Kb)]\n",
+		 argv[0]);
+	exit (1);
+      }
+
+  ReadInWords (file_name);
+
+  if (type == 'C')
+    {
+      Select_all ();
+      mem_reqd = WriteOutWords (file_name, MG_COMPLETE_DICTIONARY, lookback);
+    }
+  else
+    {
+      switch (mode)
+	{
+	case '0':
+	  Select_all ();
+	  break;
+	case '1':
+#ifndef SILENT
+	  Message ("Dictionary limit of %.2f Kb", k / 1024);
+#endif
+	  Select_on ((int) k, OccuranceOrder);
+	  break;
+	case '2':
+#ifndef SILENT
+	  Message ("Dictionary limit of %.2f Kb", k / 1024);
+#endif
+	  Select_on ((int) k, DecFreqIncWL);
+	  break;
+	case '3':
+#ifndef SILENT
+	  Message ("Dictionary limit of %.2f Kb", k / 1024);
+#endif
+	  Method3 ((int) k);
+	  break;
+	}
+      if (type == 'P')
+	{
+	  mem_reqd = WriteOutWords (file_name, MG_PARTIAL_DICTIONARY, lookback);
+	}
+      else
+	{
+	  mem_reqd = WriteOutWords (file_name, MG_SEED_DICTIONARY, lookback);
+	}
+    }
+
+#ifndef SILENT
+  Message ("Num words           : %8u -> %8u\n", Num[1], keep[1].num_wds);
+  Message ("Num non-words       : %8u -> %8u\n", Num[0], keep[0].num_wds);
+  Message ("Chars of words      : %8u -> %8u\n", chars[1], keep[1].chars);
+  Message ("Chars of non-words  : %8u -> %8u\n", chars[0], keep[0].chars);
+  Message ("Mem usage           : %8u -> %8u\n",
+	   (Num[0] + Num[1]) * sizeof (char *) + chars[0] + chars[1],
+	     (keep[0].num_wds + keep[1].num_wds) * sizeof (char *) +
+	   keep[0].chars + keep[1].chars);
+  Message ("Actual mem required : %8u\n", mem_reqd);
+#endif
+  return (0);
+}
+
+
+
+
+static void 
+ReadInWords (char *filename)
+{
+  FILE *f;
+  unsigned long i;
+  f = open_file (filename, TEXT_STATS_DICT_SUFFIX, "rb",
+		 MAGIC_STATS_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  fread (&csh, sizeof (csh), 1, f);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(csh.num_docs);
+  NTOHD(csh.num_bytes);
+
+  for (i = 0; i < 2; i++)
+    {
+      frags_stats_header fsh;
+      char *buf;
+      DictWordData *wd;
+      chars[i] = 0;
+
+      fread (&fsh, sizeof (fsh), 1, f);
+      /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(fsh.num_frags);
+      NTOHUL(fsh.mem_for_frags);
+
+      Num[i] = all[i].num_wds = fsh.num_frags;
+
+      /* The +1 on the following line is to leave room for the esc code. */
+      all[i].wd = (DictWordData **) Xmalloc (sizeof (DictWordData *) * Num[i] + 1);
+
+      buf = (char *) Xmalloc (fsh.mem_for_frags);
+      wd = Words[i] = (DictWordData *) Xmalloc (sizeof (DictWordData) * Num[i]);
+      unsigned int j;
+      for (j = 0; j < Num[i]; j++, wd++)
+	{
+	  int len;
+
+	  // read docCount and wordCount
+	  wd->read(f);
+
+	  // read an mgString
+	  len = fgetc (f);
+	  wd->word = (u_char *) buf;
+	  *buf++ = len;
+	  fread (buf, len, 1, f);
+	  buf += len;
+	  all[i].wd[j] = wd;
+	}
+      chars[i] = fsh.mem_for_frags - fsh.num_frags;
+    }
+  fclose (f);
+}
+
+
+static void 
+Alloc_keep_discard (void)
+{
+  keep[0].num_wds = 0;
+  keep[0].wd = (DictWordData **) Xmalloc ((Num[0] + 1) * sizeof (DictWordData *));
+  keep[1].num_wds = 0;
+  keep[1].wd = (DictWordData **) Xmalloc ((Num[1] + 1) * sizeof (DictWordData *));
+  discard[0].num_wds = 0;
+  discard[0].wd = (DictWordData **) Xmalloc ((Num[0] + 1) * sizeof (DictWordData *));
+  discard[1].num_wds = 0;
+  discard[1].wd = (DictWordData **) Xmalloc ((Num[1] + 1) * sizeof (DictWordData *));
+}
+
+
+static int 
+sort_comp (const void *a, const void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  return casecompare (aa->word, bb->word);  /* [RPAP - Jan 97: Stem Index Change] */
+}
+
+
+
+static void 
+SortAndCount_DictData (DictData * dd)
+{
+  unsigned int i;
+  DictWordData **wd;
+  qsort (dd->wd, dd->num_wds, sizeof (DictWordData *), sort_comp);
+  dd->chars = 0;
+  wd = dd->wd;
+  for (i = 0; i < dd->num_wds; i++, wd++)
+    dd->chars += (*wd)->word[0];
+}
+
+
+static void 
+Select_all (void)
+{
+  unsigned int i;
+  Alloc_keep_discard ();
+  keep[0].num_wds = Num[0];
+  for (i = 0; i < Num[0]; i++)
+    keep[0].wd[i] = Words[0] + i;
+  keep[1].num_wds = Num[1];
+  for (i = 0; i < Num[1]; i++)
+    keep[1].wd[i] = Words[1] + i;
+  SortAndCount_DictData (&keep[0]);
+  SortAndCount_DictData (&keep[1]);
+}
+
+
+
+
+static int 
+DecFreqIncWL (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  if (aa->docCount > bb->docCount)
+    return -1;
+  else if (aa->docCount < bb->docCount)
+    return 1;
+  else
+    return bb->word[0] - aa->word[0];
+}
+
+
+static int 
+OccuranceOrder (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  if (aa->words() > bb->words())
+    return 1;
+  else if (aa->words() < bb->words())
+    return -1;
+  else
+    return 0;
+}
+
+
+static void 
+Select_on (int k, heap_comp hc)
+{
+  int i, num, mem;
+  DictWordData **wd;
+
+  Alloc_keep_discard ();
+
+  num = Num[0] + Num[1];
+  wd = (DictWordData **) Xmalloc (num * sizeof (DictWordData *));
+  for (i = 0; (unsigned int)i < Num[0]; i++)
+    wd[i] = Words[0] + i;
+  for (i = 0; (unsigned int)i < Num[1]; i++)
+    wd[i + Num[0]] = Words[1] + i;
+
+  heap_build (wd, sizeof (*wd), num, hc);
+
+  mem = 0;
+  while (k > mem && num)
+    {
+      int idx;
+      DictWordData *word = wd[0];
+#ifdef DEBUG
+      fprintf (stderr, "%4d:%4d:%8d :%8d :%8d : \"%s\"\n",
+	       keep[0].num_wds, keep[1].num_wds,
+	       mem, word->documents(), word->word, word2str (word->word));
+#endif
+      mem += sizeof (u_char *) + word->word[0];
+      heap_deletehead (wd, sizeof (*wd), &num, hc);
+      idx = KIND (word);
+      keep[idx].wd[keep[idx].num_wds++] = word;
+    }
+
+  for (i = 0; i < num; i++)
+    {
+      DictWordData *word = wd[i];
+      int idx = KIND (word);
+      discard[idx].wd[discard[idx].num_wds++] = word;
+    }
+  SortAndCount_DictData (&keep[0]);
+  SortAndCount_DictData (&keep[1]);
+  SortAndCount_DictData (&discard[0]);
+  SortAndCount_DictData (&discard[1]);
+
+  assert (keep[0].num_wds + discard[0].num_wds == Num[0]);
+  assert (keep[1].num_wds + discard[1].num_wds == Num[1]);
+
+}
+
+
+
+static int 
+BigSaving (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  return (aa->saving > bb->saving) ? -1 : (aa->saving < bb->saving);
+}
+
+static int 
+SmallSaving (void *a, void *b)
+{
+  DictWordData *aa = *(DictWordData **) a;
+  DictWordData *bb = *(DictWordData **) b;
+  return (aa->saving < bb->saving) ? -1 : (aa->saving > bb->saving);
+}
+
+
+static void 
+CalcCharCounts (DictWordData ** wd, int num,
+		char *char_lens[2], char *len_lens[2],
+		u_long escape[2])
+{
+  long char_freqs[2][256];
+  long len_freqs[2][16];
+  huff_data hd;
+  int i;
+  escape[0] = 0;
+  escape[1] = 0;
+  memset (char_freqs, '\0', sizeof (char_freqs));
+  memset (len_freqs, '\0', sizeof (len_freqs));
+  for (i = 0; i < num; i++, wd++)
+    {
+      u_long freq = (*wd)->documents();
+      u_char *buf = (*wd)->word;
+      int len = *buf++;
+      int idx = KIND (*wd);
+      len_freqs[idx][len] += freq;
+      escape[idx] += freq;
+      for (; len; len--, buf++)
+	char_freqs[idx][(u_long) (*buf)] += freq;
+    }
+  Generate_Huffman_Data (256, char_freqs[0], &hd, NULL);
+  char_lens[0] = hd.clens;
+  Generate_Huffman_Data (256, char_freqs[1], &hd, NULL);
+  char_lens[1] = hd.clens;
+
+  Generate_Huffman_Data (16, len_freqs[0], &hd, NULL);
+  len_lens[0] = hd.clens;
+  Generate_Huffman_Data (16, len_freqs[1], &hd, NULL);
+  len_lens[1] = hd.clens;
+}
+
+
+
+
+
+inline void CalcBitCostForWordData(DictWordData **word, int num,
+				   double freqs_trans[2], u_long escape[2],
+				   char * char_lens[2], char *len_lens[2], 
+				   double esc[2], int num_trans)
+{ 
+  int j;
+
+  for (j = 0; j < num; j++, word++)
+    {
+      float   cbc, wbc;
+      u_char *buf = (*word)->word;
+      int     len = *buf++;
+      u_long  freq = (*word)->documents();
+      int     idx = KIND (*word);
+
+      cbc = len_lens[idx][len];
+      for (; len; len--, buf++)
+	cbc += char_lens[idx][(u_long) (*buf)];
+
+      (*word)->char_bit_cost = (cbc + esc[idx]) * freq;
+
+      wbc = -log2 (freq / (freqs_trans[idx] + escape[idx])) * freq;
+
+      (*word)->saving = ((*word)->char_bit_cost - wbc) /
+	(sizeof (char *) + (*word)->word[0]);
+
+      (*word)->num_trans = num_trans;
+    }
+}
+
+
+
+void 
+CalcBitCost (DictWordData ** discard_word, int discard_num,
+	     DictWordData ** keep_word, int keep_num, double freqs_trans[2],
+	     u_long escape[2], int num_trans)
+{
+  char *char_lens[2];
+  char *len_lens[2];
+  double esc[2];
+  CalcCharCounts (discard_word, discard_num, char_lens, len_lens, escape);
+  esc[0] = -log2 (escape[0] / (freqs_trans[0] + escape[0]));
+  esc[1] = -log2 (escape[1] / (freqs_trans[1] + escape[1]));
+  CalcBitCostForWordData(discard_word, discard_num, freqs_trans, escape,
+			 char_lens, len_lens, esc, num_trans);
+  CalcBitCostForWordData(keep_word, keep_num, freqs_trans, escape,
+			 char_lens, len_lens, esc, num_trans);
+  Xfree (char_lens[0]);
+  Xfree (char_lens[1]);
+  Xfree (len_lens[0]);
+  Xfree (len_lens[1]);
+}
+
+inline void m3_transferWord(DictWordData **toHeap, int *toNum, heap_comp toSaving,
+			    DictWordData **fromHeap, int *fromNum, heap_comp fromSaving,
+			    int *num_trans, double freqs_trans[], int *mem, double *total,
+			    int *count)
+{
+  /* Transfer the word at the top of the keep heap to the
+     discard heap. */
+  DictWordData *word = fromHeap[0];
+  int idx = KIND (word);
+  heap_deletehead (fromHeap, sizeof (word), fromNum, fromSaving);
+  toHeap[*toNum] = word;
+  heap_additem (toHeap, sizeof (word), toNum, toSaving);
+  freqs_trans[idx] -= word->documents();
+  *mem = (*mem) - sizeof (u_char *) + word->word[0];
+  *num_trans += 1;
+  *total += word->saving;
+  *count += 1;
+#ifdef DEBUG
+  fprintf (stderr,
+	   "KEEP -> DISCARD %8d :%8d :%8.0f :%8.0f : \"%s\"\n",
+	   *mem, word->documents(), word->char_bit_cost,
+	   word->saving, word2str (word->word));
+#endif
+}
+
+inline void m3_storeWord(DictWordData **wordHeap, int wordNum, int num_trans, 
+			 double freqs_trans[], u_long escape[], heap_comp hc)
+{
+  DictWordData *word = wordHeap[0];
+  int idx = KIND (word);
+  float wbc;
+#ifdef DEBUG1
+  fprintf (stderr, "KEEP \"%s\" %.2f ->", word2str (word->word),
+	   word->saving);
+#endif
+  wbc = -log2 (word->documents() / (freqs_trans[idx] + escape[idx])) *
+    word->documents();
+  word->saving = (word->char_bit_cost - wbc) /
+    (sizeof (char *) + word->word[0]);
+#ifdef DEBUG1
+  fprintf (stderr, " %.2f\n", word->saving);
+#endif
+  word->num_trans = num_trans;
+  heap_changedhead (wordHeap, sizeof (word), wordNum, hc);
+}
+
+
+static void 
+Method3 (int k)
+{
+  int i, keep_num, discard_num, mem, num_trans, recalc_reqd;
+  int keep_to_discard = 0;
+  int discard_to_keep = 0;
+  int recalcs = 0;
+  double freqs_trans[2], total;
+  u_long escape[2];
+  DictWordData **keep_heap, **discard_heap;
+
+  freqs_trans[0] = freqs_trans[1] = 0;
+  num_trans = 0;
+
+  discard_num = Num[0] + Num[1];
+  discard_heap = (DictWordData **) Xmalloc (discard_num * sizeof (DictWordData *));
+
+  keep_num = 0;
+  keep_heap = (DictWordData **) Xmalloc (discard_num * sizeof (DictWordData *));
+
+
+  for (i = 0; (unsigned int)i < Num[0]; i++)
+    discard_heap[i] = Words[0] + i;
+  for (i = 0; (unsigned int)i < Num[1]; i++)
+    discard_heap[i + Num[0]] = Words[1] + i;
+
+  heap_build (discard_heap, sizeof (*discard_heap), discard_num, DecFreqIncWL);
+
+  mem = 0;
+  while (k > mem && discard_num)
+    {
+      DictWordData *word = discard_heap[0];
+      mem += sizeof (u_char *) + word->word[0];
+      heap_deletehead (discard_heap, sizeof (word), &discard_num, DecFreqIncWL);
+      keep_heap[keep_num++] = word;
+      freqs_trans[KIND (word)] += word->documents();
+      num_trans++;
+    }
+
+  CalcBitCost (discard_heap, discard_num, keep_heap, keep_num,
+	       freqs_trans, escape, num_trans);
+  heap_build (discard_heap, sizeof (*discard_heap), discard_num, BigSaving);
+  heap_build (keep_heap, sizeof (*keep_heap), keep_num, SmallSaving);
+
+  total = 0;
+  recalc_reqd = 0;
+  while (keep_num && discard_num)
+    {
+      if ((keep_num && keep_heap[0]->num_trans < (unsigned)num_trans) ||
+	  (discard_num && discard_heap[0]->num_trans < (unsigned)num_trans))
+	{
+	  if (keep_num && keep_heap[0]->num_trans < (unsigned)num_trans)
+	    {
+	      m3_storeWord(keep_heap, keep_num, num_trans, freqs_trans, escape, SmallSaving);
+	    }
+
+	  if (discard_num && discard_heap[0]->num_trans < (unsigned)num_trans)
+	    {
+	      m3_storeWord(discard_heap, discard_num, num_trans, freqs_trans, escape, BigSaving);
+	    }
+	}
+      else if (keep_heap[0]->saving < discard_heap[0]->saving)
+	{
+	  assert (keep_num && discard_num);
+	  if (keep_num && mem + sizeof (char *) + discard_heap[0]->word[0] > (unsigned)k)
+	    {
+	      /* Transfer the word at the top of the keep heap to the
+	         discard heap. */
+	      m3_transferWord(discard_heap, &discard_num, BigSaving,
+			      keep_heap, &keep_num, SmallSaving,
+			      &num_trans, freqs_trans, &mem, &total, &keep_to_discard);
+	    }
+	  else
+	    {
+	      /* Transfer the word at the top of the discard heap to the
+	         keep heap. */
+	      m3_transferWord(keep_heap, &keep_num, SmallSaving,
+			      discard_heap, &discard_num, BigSaving,
+			      &num_trans, freqs_trans, &mem, &total, &discard_to_keep);
+	    }
+
+	  recalc_reqd = 1;
+
+	}
+      else
+	{
+	  if (recalc_reqd == 0)
+	    break;
+#ifdef DEBUG1
+	  fprintf (stderr, "--------------\n");
+#endif
+	  if (recalcs == MAX_RECALCULATIONS)
+	    break;
+	  CalcBitCost (discard_heap, discard_num, keep_heap, keep_num,
+		       freqs_trans, escape, num_trans);
+	  heap_build (discard_heap, sizeof (*discard_heap),
+		      discard_num, BigSaving);
+	  heap_build (keep_heap, sizeof (keep_heap), keep_num, SmallSaving);
+	  recalc_reqd = 0;
+	  recalcs++;
+	}
+    }
+
+  Alloc_keep_discard ();
+
+  for (i = 0; i < discard_num; i++)
+    {
+      DictWordData *word = discard_heap[i];
+      int idx = KIND (word);
+      assert (IsWord (word) || IsNonWord (word));
+      discard[idx].wd[discard[idx].num_wds++] = word;
+    }
+  for (i = 0; i < keep_num; i++)
+    {
+      DictWordData *word = keep_heap[i];
+      int idx = KIND (word);
+      assert (IsWord (word) || IsNonWord (word));
+      keep[idx].wd[keep[idx].num_wds++] = word;
+    }
+  SortAndCount_DictData (&keep[0]);
+  SortAndCount_DictData (&keep[1]);
+  SortAndCount_DictData (&discard[0]);
+  SortAndCount_DictData (&discard[1]);
+
+  assert (keep[0].num_wds + discard[0].num_wds == Num[0]);
+  assert (keep[1].num_wds + discard[1].num_wds == Num[1]);
+#ifndef SILENT
+  Message ("Keep -> Discard        : %8d", keep_to_discard);
+  Message ("Discard -> Keep        : %8d", discard_to_keep);
+  Message ("Huffman Recalculations : %8d", recalcs);
+  if (recalcs == MAX_RECALCULATIONS)
+    Message ("WARNING: The number of recalculations == %d", MAX_RECALCULATIONS);
+#endif
+}
+
+
+
+
+
+
+
+
+
+
+/****************************************************************************
+ *****									*****
+ *****			Dictionary saving code				*****
+ *****									*****
+ ****************************************************************************/
+
+
+
+static void 
+Write_cdh (FILE * f, compression_dict_header * cdh)
+{
+  /* [RPAP - Jan 97: Endian Ordering] */
+  int i;
+  compression_dict_header tmp = *cdh;
+  HTONUL(tmp.dict_type);
+  HTONUL(tmp.novel_method);
+  for (i = 0; i < TEXT_PARAMS; i++)
+    HTONUL(tmp.params[i]);
+  HTONUL(tmp.num_words[0]);
+  HTONUL(tmp.num_words[1]);
+  HTONUL(tmp.num_word_chars[0]);
+  HTONUL(tmp.num_word_chars[1]);
+  HTONUL(tmp.lookback);
+
+  fwrite (&tmp, sizeof (tmp), 1, f);
+}
+
+
+static void 
+Write_words (FILE * f, DictData * dd)
+{
+  unsigned int i;
+  u_char *curr, *prev = NULL;
+  for (i = 0; i < dd->num_wds; i++)
+    {
+      int len;
+      curr = dd->wd[i]->word;
+      if (prev)
+	/* look for prefix match with prev string */
+	len = prefixlen (prev, curr);
+      else
+	len = 0;
+      fputc ((len << 4) + (curr[0] - len), f);
+      fwrite (curr + len + 1, sizeof (u_char), curr[0] - len, f);
+      prev = curr;
+    }
+
+}
+
+
+static int 
+Uncompressed_size (DictData * dd)
+{
+  unsigned int i, us;
+  for (us = i = 0; i < dd->num_wds; i++)
+    us += dd->wd[i]->word[0];
+  return us;
+}
+
+
+static u_long 
+Write_data (FILE * f, DictData * dd, int lookback)
+{
+  u_long mem_reqd;
+  huff_data *hd;
+  int i; 
+  u_long us = dd->num_wds;
+  long *freqs;
+  u_long huff_words_size[MAX_HUFFCODE_LEN + 1];
+  u_long lencounts[MAX_HUFFCODE_LEN + 1];
+  u_char *lastword[MAX_HUFFCODE_LEN + 1];
+
+  if (!(freqs = new long [dd->num_wds]))
+    FatalError (1, "Unable to allocate memory for freqs");
+
+  for (i = 0; (unsigned)i < dd->num_wds; i++)
+    {
+      freqs[i] = dd->wd[i]->documents();
+      us += dd->wd[i]->word[0];
+    }
+
+  if (!(hd = Generate_Huffman_Data (dd->num_wds, freqs, NULL, NULL)))
+    FatalError (1, "Unable to allocate memory for huffman data");
+
+  delete (freqs);
+  freqs = NULL;
+
+  if (Write_Huffman_Data (f, hd) == -1)
+    FatalError (1, "Unable to write huffman data");
+
+  HTONUL(us);  /* [RPAP - Jan 97: Endian Ordering] */
+  fwrite (&us, sizeof (us), 1, f);
+  NTOHUL(us);  /* [RPAP - Jan 97: Endian Ordering] */
+
+
+/* Calculate the amount of memory that will be required to store the text for
+   each different huffman code len. Every 1<<lookback words for each different
+   codelen length will not be prefixed by previous strings. */
+
+
+  memset (&huff_words_size, '\0', sizeof (huff_words_size));
+  memset (&lencounts, '\0', sizeof (lencounts));
+
+  mem_reqd = 0;
+
+  for (i = 0; (unsigned)i < dd->num_wds; i++)
+    {
+      int codelen = hd->clens[i];
+      u_char *word = dd->wd[i]->word;
+
+      if (!codelen)
+	FatalError (1, "The length of a code for a word was zero");
+
+      huff_words_size[codelen] += word[0] + 1;
+      mem_reqd += word[0] + (lookback != 0);
+#if 0
+      if ((lencounts[codelen] & ((1 << lookback) - 1)) == 0)
+	lastword[codelen] = word;
+      else
+	huff_words_size[codelen] -= prefixlen (lastword[codelen], word);
+#else
+      if ((lencounts[codelen] & ((1 << lookback) - 1)) != 0)
+	{
+	  int save = prefixlen (lastword[codelen], word);
+	  mem_reqd -= save;
+	  huff_words_size[codelen] -= save;
+	}
+      else
+	{
+	  mem_reqd += sizeof (u_char *);
+	}
+      lastword[codelen] = word;
+#endif
+      lencounts[codelen]++;
+    }
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+    HTONUL(huff_words_size[i]);
+
+  fwrite (huff_words_size + hd->mincodelen, sizeof (*huff_words_size),
+	  hd->maxcodelen - hd->mincodelen + 1, f);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  for (i = hd->mincodelen; i < hd->maxcodelen + 1; i++)
+    NTOHUL(huff_words_size[i]);
+
+  Write_words (f, dd);
+
+  delete hd->clens;
+  delete hd;
+
+  return mem_reqd;
+}
+
+
+
+static void 
+Write_charfreqs (FILE * f, DictData * dd, int words,
+		 int zero_freq_permitted)
+{
+  unsigned int j;
+  long freqs[256];
+  DictWordData **wd = dd->wd;
+  huff_data *hd;
+
+  memset (freqs, '\0', sizeof (freqs));
+
+  for (j = 0; j < dd->num_wds; j++, wd++)
+    {
+      u_char *buf = (*wd)->word;
+      int len = *buf++;
+      for (; len; len--, buf++)
+	freqs[(u_long) (*buf)] += (*wd)->documents();
+    }
+
+  if (!zero_freq_permitted)
+    for (j = 0; j < 256; j++)
+      if (!freqs[j] && PESINAWORD (j) == words)
+	freqs[j] = 1;
+
+  if (!(hd = Generate_Huffman_Data (256, freqs, NULL, NULL)))
+    FatalError (1, "Unable to allocate memory for huffman data");
+
+  if (Write_Huffman_Data (f, hd) == -1)
+    FatalError (1, "Unable to write huffman data");
+
+  delete hd->clens;
+  delete hd;
+}
+
+
+
+
+static void 
+Write_wordlens (FILE * f, DictData * dd, int zero_freq_permitted)
+{
+  unsigned int j;
+  long freqs[16];
+  DictWordData **wd = dd->wd;
+  huff_data *hd;
+
+  memset (freqs, '\0', sizeof (freqs));
+
+  for (j = 0; j < dd->num_wds; j++, wd++)
+    freqs[(*wd)->word[0]] += (*wd)->documents();
+
+  if (!zero_freq_permitted)
+    for (j = 0; j < 16; j++)
+      if (!freqs[j])
+	freqs[j] = 1;
+
+  if (!(hd = Generate_Huffman_Data (16, freqs, NULL, NULL)))
+    FatalError (1, "Unable to allocate memory for huffman data");
+
+  if (Write_Huffman_Data (f, hd) == -1)
+    FatalError (1, "Unable to write huffman data");
+
+
+  delete hd->clens;
+  delete hd;
+}
+
+
+
+static u_long 
+WriteOutWords (char *file_name, u_long type, int lookback)
+{
+  FILE *f;
+  int i;
+  u_long mem_reqd = 0;
+
+  compression_dict_header cdh;
+  f = create_file (file_name, TEXT_DICT_SUFFIX, "w+b",
+		   MAGIC_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  memset (&cdh, '\0', sizeof(cdh));
+
+  cdh.dict_type = type;
+  cdh.novel_method = (type == MG_SEED_DICTIONARY) ? novel_method :
+    MG_NOVEL_HUFFMAN_CHARS;
+
+  cdh.num_words[1] = keep[1].num_wds;
+  cdh.num_words[0] = keep[0].num_wds;
+  cdh.num_word_chars[1] = Uncompressed_size (&keep[1]);
+  cdh.num_word_chars[0] = Uncompressed_size (&keep[0]);
+  cdh.lookback = lookback;
+
+  Write_cdh (f, &cdh);
+
+  for (i = 0; i < 2; i++)
+    switch (type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  mem_reqd += Write_data (f, &keep[i], lookback);
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (keep[i].num_wds)
+	    {
+	      int j;
+	      DictWordData esc;
+	      esc.docCount = 0;
+	      esc.word = (u_char *) "";
+	      keep[i].wd[keep[i].num_wds++] = &esc;
+	      for (j = 0; (unsigned)j < discard[i].num_wds; j++)
+		esc.docCount += discard[i].wd[j]->documents();
+	      if (!esc.docCount)
+		esc.docCount++;
+	      mem_reqd += Write_data (f, &keep[i], lookback);
+	    }
+	  Write_charfreqs (f, &discard[i], i, 1);
+	  Write_wordlens (f, &discard[i], 1);
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (keep[i].num_wds)
+	    {
+	      int j;
+	      DictWordData esc;
+	      esc.docCount = 0;
+	      esc.word = (u_char *) "";
+	      keep[i].wd[keep[i].num_wds++] = &esc;
+	      for (j = 0; (unsigned)j < all[i].num_wds; j++)
+		if (all[i].wd[j]->documents() == 1)
+		  esc.docCount++;
+	      if (!esc.docCount)
+		esc.docCount++;
+	      mem_reqd += Write_data (f, &keep[i], lookback);
+	    }
+	  switch (novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      Write_charfreqs (f, &all[i], i, 0);
+	      Write_wordlens (f, &all[i], 0);
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    default:
+	      FatalError (1, "Bad novel method");
+	    }
+	}
+	break;
+      }
+  fclose (f);
+  return mem_reqd;
+}
Index: /trunk/mgpp/text/mgpp_decompress_text.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_decompress_text.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_decompress_text.cpp	(revision 3365)
@@ -0,0 +1,100 @@
+/**************************************************************************
+ *
+ * mgpp_decompress_text.cpp -- 
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "TextGet.h"
+#include "messages.h"
+#include "mg_files.h"
+
+int main (int argc, char **argv) {
+  int ch;
+  char *filename = "";
+  char *basePath = "";
+  UCArray level;
+  SetCStr (level, "Document");
+
+  opterr = 0;
+  msg_prefix = argv[0];
+
+  // process the command line arguments
+  while ((ch = getopt (argc, argv, "f:d:K:h")) != -1) {
+    switch (ch) {
+    case 'f':		/* input file */
+      filename = optarg;
+      break;
+    case 'd':
+      basePath = optarg;
+      set_basepath (optarg);
+      break;
+    case 'K':
+      SetCStr (level, optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-K level] [-d directory] -f name\n",
+	       argv[0]);
+      exit (1);
+    }
+  }
+
+  // load up the text information
+  TextData td;
+  if (!td.LoadData (basePath, filename)) {
+    FatalError (1, "Couldn't load text information for \"%s\"", filename);
+  }
+  
+  // output each document in the level
+//    cout << td.levels << "\n";
+  
+  TextLevelInfo levelInfo = td.levels.levelInfo[level];
+  unsigned long docNum = 1;
+  UCArray docText;
+  while (docNum <= levelInfo.numEntries) {
+//      TextIdx docIdx;
+//      if (GetDocIdx (td, level, docNum, docIdx)) {
+//        cout << "doc: " << docNum << "\n";
+//        cout << docIdx;
+//      }
+    
+    if (!GetDocText (td, level, docNum, docText)) {
+      FatalError (1, "Error while trying to get document %u", docNum);
+    }
+
+    cout << docText << "\n";
+    
+    docNum++;
+  }
+  
+  return 0;
+}
+
+
Index: /trunk/mgpp/text/mgpp_fast_comp_dict.1
===================================================================
--- /trunk/mgpp/text/mgpp_fast_comp_dict.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_fast_comp_dict.1	(revision 3365)
@@ -0,0 +1,69 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_fast_comp_dict 1 \*(Dt CITRI
+.SH NAME
+mgpp_fast_comp_dict \- build a fast loading compression dictionary
+.SH SYNOPSIS
+.B mgpp_fast_comp_dict
+[
+.B \-h
+]
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_fast_comp_dict
+builds a fast-loading compression dictionary from the standard
+compression dictionary.  The new dictionary has the property that
+.BR mgppquery (1)
+can load it in very quickly, making the startup time for
+.BR mgppquery (1)
+much shorter.  The new dictionary is larger on disk than the original,
+but they are no different when it comes to memory usage.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.text.dict
+Compression dictionary.
+.TP
+.B *.text.dict.fast
+Fast loading compression dictionary.
+.TP
+.B *.text.dict.aux
+Auxiliary compression dictionary.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/mgpp/text/mgpp_fast_comp_dict.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_fast_comp_dict.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_fast_comp_dict.cpp	(revision 3365)
@@ -0,0 +1,707 @@
+/**************************************************************************
+ *
+ * mgpp_fast_comp_dict.cpp -- Program to generate a fast compression dictionary
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "huffman.h"
+#include "messages.h"
+#include "memlib.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "local_strings.h"
+#include "mg.h"
+#include "text.h"
+#include "invf.h"
+#include "mg_files.h"
+#include "locallib.h"
+#include "words.h"
+
+
+
+#define ALIGN_SIZE(p,t) (((p) + (sizeof(t)-1)) & (~(sizeof(t)-1)))
+
+#define WORDNO(p, base) ((((char*)(p))-((char*)(base)))/sizeof(u_char*))
+#define FIXUP(p) (fixup[WORDNO(p,buffer)/8] |= (1<<(WORDNO(p, buffer) & 7)))
+
+#define IS_FIXUP(p) ((fixup[WORDNO(p,buffer)/8] & (1<<(WORDNO(p, buffer) & 7))) != 0)
+
+#define FIXUP_VALS(vals) do {						\
+	int i;								\
+	for (i=0; i < MAX_HUFFCODE_LEN+1; i++)				\
+	  FIXUP(&vals[i]);						\
+      } while(0)
+
+
+
+static u_long mem_for_comp_dict (char *filename);
+static void load_comp_dict (char *filename);
+static void save_fast_dict (char *filename);
+static void unfixup_buffer (void);
+
+
+static void *buffer;
+static void *cur;
+static u_char *fixup;
+static u_long mem, fixup_mem;
+
+int main (int argc, char **argv)
+{
+  int ch;
+  char *filename = "";
+  opterr = 0;
+  msg_prefix = argv[0];
+  while ((ch = getopt (argc, argv, "f:d:h")) != -1)
+    switch (ch)
+      {
+      case 'f':		/* input file */
+	filename = optarg;
+	break;
+      case 'd':
+	set_basepath (optarg);
+	break;
+      case 'h':
+      case '?':
+	fprintf (stderr, "usage: %s [-f input_file] [-d data directory] [-h]\n",
+		 argv[0]);
+	exit (1);
+      }
+
+  mem = mem_for_comp_dict (filename);
+
+  fixup_mem = (ALIGN_SIZE (mem, u_char *) / sizeof (u_char *) + 7) / 8;
+
+  cur = buffer = Xmalloc (mem);
+  memset (buffer, 0, mem);
+  fixup = (u_char *) Xmalloc (fixup_mem);
+  memset (fixup, 0, fixup_mem);
+
+#ifndef SILENT
+  Message ("Estimated memory for fast_dict %u", mem);
+  Message ("Estimated memory for fixups %u", fixup_mem);
+#endif
+
+  load_comp_dict (filename);
+
+#ifndef SILENT
+  Message ("Actual memory for fast_dict %u", (char *) cur - (char *) buffer);
+#endif
+
+  if ((u_long) cur > (u_long) buffer + mem)
+    FatalError (1, "The buffer was not big enough for the dictionary");
+
+  {
+    /* [RPAP - Jan 97: Endian Ordering] */
+    compression_dict *cd = (compression_dict *) buffer;
+    int i, which;
+
+    /* cfh */
+    for (which = 0; which <= 1; which++)
+      {
+	int j;
+	
+	HTONSI(cd->cfh[which]->hd.num_codes);
+	HTONSI(cd->cfh[which]->hd.mincodelen);
+	HTONSI(cd->cfh[which]->hd.maxcodelen);
+	for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	  {
+	    HTONSI(cd->cfh[which]->hd.lencount[j]);
+	    HTONUL(cd->cfh[which]->hd.min_code[j]);
+	  }
+	HTONUL(cd->cfh[which]->uncompressed_size);
+	for (j = 0; j < MAX_HUFFCODE_LEN + 1; j++)
+	  HTONUL(cd->cfh[which]->huff_words_size[j]);
+      }
+    HTONUL(cd->MemForCompDict);
+    /* ad */
+    if (cd->cdh.novel_method == MG_NOVEL_DELTA ||
+	cd->cdh.novel_method == MG_NOVEL_HYBRID)
+      for (which = 0; which <= 1; which++)
+	{
+	  int j;
+	  
+	  HTONUL(cd->ad->afh[which].num_frags);
+	  HTONUL(cd->ad->afh[which].mem_for_frags);
+	  for (j = 0; j < 33; j++)
+	    {
+	      HTONSI(cd->ad->blk_start[which][j]);
+	      HTONSI(cd->ad->blk_end[which][j]);
+	    }
+	}
+    /* cdh */
+    HTONUL(cd->cdh.dict_type);
+    HTONUL(cd->cdh.novel_method);
+    for (i = 0; i < TEXT_PARAMS; i++)
+      HTONUL(cd->cdh.params[which]);
+    HTONUL(cd->cdh.num_words[0]);
+    HTONUL(cd->cdh.num_words[1]);
+    HTONUL(cd->cdh.num_word_chars[0]);
+    HTONUL(cd->cdh.num_word_chars[1]);
+    HTONUL(cd->cdh.lookback);
+    HTONSI(cd->fast_loaded);
+  }
+
+  unfixup_buffer ();
+
+  save_fast_dict (filename);
+
+  return (0);
+}
+
+
+
+static void 
+unfixup_buffer ()
+{
+  u_long *p;
+  for (p = (u_long *) buffer; (u_long) p < (u_long) cur; p++)
+    {
+      if (IS_FIXUP (p))
+	*p = *p - (u_long) buffer;
+    }
+}
+
+
+
+
+static u_long 
+mem_for_aux_dict (compression_dict_header * /*cdh*/, char *filename)
+{
+  int i;
+  u_long mem = sizeof (auxiliary_dict);
+  FILE *aux;
+
+  aux = open_file (filename, TEXT_DICT_AUX_SUFFIX, "rb",
+		   MAGIC_AUX_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  for (i = 0; i <= 1; i++)
+    {
+      aux_frags_header afh;
+      fread (&afh, sizeof (afh), 1, aux);
+      NTOHUL(afh.num_frags);  /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(afh.mem_for_frags);  /* [RPAP - Jan 97: Endian Ordering] */
+      mem += afh.num_frags * sizeof (u_char *);
+      mem = ALIGN_SIZE (mem + afh.mem_for_frags, u_char *);
+      fseek (aux, afh.mem_for_frags, SEEK_CUR);
+    }
+  fclose (aux);
+
+  return mem;
+}
+
+
+
+static u_long 
+mem_for_words (FILE * dict, compression_dict_header * cdh,
+	       comp_frags_header * cfh)
+{
+  u_long mem = 0;
+  long i, lookback;
+  int ptrs_reqd = 0;
+  int mem_reqd = 0;
+
+  lookback = cdh->lookback;
+
+  for (i = cfh->hd.mincodelen; i <= cfh->hd.maxcodelen; i++)
+    {
+      ptrs_reqd += (cfh->hd.lencount[i] + ((1 << lookback) - 1)) >> lookback;
+      mem_reqd += cfh->huff_words_size[i];
+    }
+
+  mem += ptrs_reqd * sizeof (u_char *);
+  mem += (MAX_HUFFCODE_LEN + 1) * sizeof (u_char **);
+  mem += mem_reqd;
+
+  for (i = 0; i < cfh->hd.num_codes; i++)
+    {
+      register int val;
+      for (val = getc (dict) & 0xf; val; val--)
+	getc (dict);
+    }
+  return ALIGN_SIZE (mem, u_char *);
+}
+
+static u_long mem_skip_hd(FILE *dict, u_long mem)
+{ huff_data hd;
+
+  mem += sizeof (hd);
+  Read_Huffman_Data (dict, &hd, NULL, NULL);
+  if (hd.clens)
+    delete hd.clens;
+  mem += hd.num_codes * sizeof (unsigned long);
+  mem += (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *);
+  return mem;
+}
+
+static u_long mem_read_cfh(FILE *dict, compression_dict_header *cdh, comp_frags_header *cfh, 
+			   u_long mem)
+{
+  mem += sizeof (comp_frags_header);
+  Read_cfh (dict, cfh, NULL, NULL);
+
+  /* Don't need to count the space for the clens of the huffman data */
+
+  mem += mem_for_words (dict, cdh, cfh);
+  if (cfh->hd.clens)
+    delete cfh->hd.clens;
+
+  return mem;
+}
+
+
+static u_long 
+mem_for_comp_dict (char *filename)
+{
+  u_long mem = sizeof (compression_dict);
+  compression_dict_header cdh;
+  comp_frags_header cfh;
+  FILE *dict;
+  int which;
+  int i;  /* [RPAP - Jan 97: Endian Ordering] */
+
+  dict = open_file (filename, TEXT_DICT_SUFFIX, "rb",
+		    MAGIC_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  fread (&cdh, sizeof (cdh), 1, dict);
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(cdh.dict_type);
+  NTOHUL(cdh.novel_method);
+  for (i = 0; i < TEXT_PARAMS; i++)
+    NTOHUL(cdh.params[i]);
+  NTOHUL(cdh.num_words[0]);
+  NTOHUL(cdh.num_words[1]);
+  NTOHUL(cdh.num_word_chars[0]);
+  NTOHUL(cdh.num_word_chars[1]);
+  NTOHUL(cdh.lookback);
+
+  for (which = 0; which < 2; which++)
+    switch (cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  mem = mem_read_cfh(dict, &cdh, &cfh, mem);
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  // huff_data hd;
+	  if (cdh.num_words[which])
+	    {
+	      mem = mem_read_cfh(dict, &cdh, &cfh, mem);
+	    }
+
+	  mem = mem_skip_hd(dict, mem);
+	  mem = mem_skip_hd(dict, mem);
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  // huff_data hd;
+	  if (cdh.num_words[which])
+	    {
+	      mem = mem_read_cfh(dict, &cdh, &cfh, mem);
+	    }
+	  switch (cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      mem = mem_skip_hd(dict, mem);
+	      mem = mem_skip_hd(dict, mem);
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    }
+	  break;
+	}
+      }
+  fclose (dict);
+
+  if (cdh.novel_method == MG_NOVEL_DELTA ||
+      cdh.novel_method == MG_NOVEL_HYBRID)
+    mem += mem_for_aux_dict (&cdh, filename);
+
+  return ALIGN_SIZE (mem, u_char *);
+}
+
+
+
+void *
+getmem (u_long size, int align)
+{
+  void *res;
+  cur = (void *) (((u_long) cur + (align - 1)) & (~(align - 1)));
+  res = cur;
+  cur = (char *) cur + size;
+  if ((u_long) cur > (u_long)buffer + mem)
+    FatalError (1, "The buffer was not big enough for the dictionary");
+  return res;
+}
+
+
+
+
+
+
+
+
+
+
+
+static auxiliary_dict *
+LoadAuxDict (compression_dict_header * cdh,
+	     char *filename)
+{
+  auxiliary_dict *ad;
+  int i;
+  FILE *aux;
+
+  aux = open_file (filename, TEXT_DICT_AUX_SUFFIX, "rb",
+		   MAGIC_AUX_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  ad = (auxiliary_dict *) getmem (sizeof (auxiliary_dict), sizeof (u_char *));
+
+  for (i = 0; i <= 1; i++)
+    {
+      unsigned int j;
+      u_char *pos;
+
+      fread (&ad->afh[i], sizeof (aux_frags_header), 1, aux);
+
+      /* [RPAP - Jan 97: Endian Ordering] */
+      NTOHUL(ad->afh[i].num_frags);
+      NTOHUL(ad->afh[i].mem_for_frags);
+
+      ad->word_data[i] = (u_char *) getmem (ad->afh[i].mem_for_frags, 1);
+      FIXUP (&ad->word_data[i]);
+
+      ad->words[i] = (u_char **) getmem (ad->afh[i].num_frags * sizeof (u_char *),
+			     sizeof (u_char *));
+      FIXUP (&ad->words[i]);
+
+      fread (ad->word_data[i], ad->afh[i].mem_for_frags, sizeof (u_char), aux);
+
+      pos = ad->word_data[i];
+      for (j = 0; j < ad->afh[i].num_frags; j++)
+	{
+	  ad->words[i][j] = pos;
+	  FIXUP (&ad->words[i][j]);
+	  pos += *pos + 1;
+	}
+      if (cdh->novel_method == MG_NOVEL_HYBRID)
+	{
+	  int num;
+	  num = 1;
+	  ad->blk_start[i][0] = 0;
+	  ad->blk_end[i][0] = cdh->num_words[i] - 1;
+	  while (num < 33)
+	    {
+	      ad->blk_start[i][num] = ad->blk_end[i][num - 1] + 1;
+	      ad->blk_end[i][num] = ad->blk_start[i][num] +
+		(ad->blk_end[i][num - 1] - ad->blk_start[i][num - 1]) * 2;
+	      num++;
+	    }
+	}
+    }
+  fclose (aux);
+  return (ad);
+}
+
+
+
+
+
+static u_char ***
+ReadInWords (FILE * dict, compression_dict * cd,
+	     comp_frags_header * cfh, u_char ** escape)
+{
+  int i, lookback;
+  int ptrs_reqd = 0;
+  int mem_reqd = 0;
+  int num_set[MAX_HUFFCODE_LEN + 1];
+  u_char *next_word[MAX_HUFFCODE_LEN + 1];
+  u_char **vals;
+  u_char ***values;
+  u_char word[MAXWORDLEN + 1];
+  u_char last_word[MAX_HUFFCODE_LEN + 1][MAXWORDLEN + 1];
+
+  lookback = cd->cdh.lookback;
+
+  for (i = cfh->hd.mincodelen; i <= cfh->hd.maxcodelen; i++)
+    {
+      ptrs_reqd += (cfh->hd.lencount[i] + ((1 << lookback) - 1)) >> lookback;
+      mem_reqd += cfh->huff_words_size[i];
+    }
+
+  vals = (u_char **) getmem (ptrs_reqd * sizeof (*vals), sizeof (u_char *));
+
+  values = (u_char ***) getmem ((MAX_HUFFCODE_LEN + 1) * sizeof (u_char **), sizeof (u_char **));
+
+  next_word[0] = (u_char *) getmem (mem_reqd, sizeof (char));
+
+  cd->MemForCompDict += ptrs_reqd * sizeof (*vals) +
+    (MAX_HUFFCODE_LEN + 1) * sizeof (u_char **) +
+    mem_reqd;
+
+  values[0] = vals;
+  FIXUP (&values[0]);
+  values[0][0] = next_word[0];
+  FIXUP (&values[0][0]);
+  for (i = 1; i <= cfh->hd.maxcodelen; i++)
+    {
+      int next_start = (values[i - 1] - vals) +
+      ((cfh->hd.lencount[i - 1] + ((1 << lookback) - 1)) >> lookback);
+      values[i] = &vals[next_start];
+      FIXUP (&values[i]);
+      next_word[i] = next_word[i - 1] + cfh->huff_words_size[i - 1];
+      values[i][0] = next_word[i];
+      FIXUP (&values[i][0]);
+    }
+
+  memset (num_set, '\0', sizeof (num_set));
+
+  for (i = 0; i < cfh->hd.num_codes; i++)
+    {
+      register int val, copy;
+      register int len = cfh->hd.clens[i];
+      val = getc (dict);
+      copy = (val >> 4) & 0xf;
+      val &= 0xf;
+
+      fread (word + copy + 1, sizeof (u_char), val, dict);
+      *word = val + copy;
+
+      if ((num_set[len] & ((1 << lookback) - 1)) == 0)
+	{
+	  values[len][num_set[len] >> lookback] = next_word[len];
+	  FIXUP (&values[len][num_set[len] >> lookback]);
+	  memcpy (next_word[len], word, *word + 1);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    {
+	      *escape = next_word[len];
+	      FIXUP (escape);
+	    }
+	  next_word[len] += *word + 1;
+	}
+      else
+	{
+	  copy = prefixlen (last_word[len], word);
+	  memcpy (next_word[len] + 1, word + copy + 1, *word - copy);
+	  *next_word[len] = (copy << 4) + (*word - copy);
+	  if (escape && i == cfh->hd.num_codes - 1)
+	    {
+	      *escape = next_word[len];
+	      FIXUP (escape);
+	    }
+	  next_word[len] += (*word - copy) + 1;
+	}
+      memcpy (last_word[len], word, *word + 1);
+      num_set[len]++;
+    }
+  if (cfh->hd.clens)
+    delete cfh->hd.clens;
+  cfh->hd.clens = NULL;
+  return values;
+}
+
+
+static unsigned long **
+Generate_Fast_Huffman_Vals (huff_data * data, u_long * mem)
+{
+  int i;
+  unsigned long *fcode[MAX_HUFFCODE_LEN + 1];
+  unsigned long **values;
+  unsigned long *vals;
+
+  if (!data)
+    return (NULL);
+  vals = (unsigned long *) getmem (data->num_codes * sizeof (*vals), sizeof (long *));
+  values = (unsigned long **) getmem ((MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *),
+		   sizeof (long *));
+
+  memset (values, '\0', (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *));
+
+  if (mem)
+    *mem += data->num_codes * sizeof (*vals) +
+      (MAX_HUFFCODE_LEN + 1) * sizeof (unsigned long *);
+
+  fcode[0] = values[0] = &vals[0];
+  FIXUP (&values[0]);
+  for (i = 1; i <= data->maxcodelen; i++)
+    {
+      fcode[i] = values[i] = &vals[(values[i - 1] - vals) + data->lencount[i - 1]];
+      FIXUP (&values[i]);
+    }
+
+  for (i = 0; i < data->num_codes; i++)
+    if (data->clens[i])
+      *fcode[(int) (data->clens[i])]++ = i;
+  return (values);
+}
+
+
+static void load_read_huffman(FILE *dict, compression_dict *cd, int which)
+{
+  huff_data *hd;
+  u_long   **vals;
+
+  hd = (huff_data *) getmem (sizeof (huff_data), sizeof (char *));
+  cd->MemForCompDict += sizeof (huff_data);
+  Read_Huffman_Data (dict, hd, &cd->MemForCompDict, NULL);
+  vals = Generate_Fast_Huffman_Vals (hd, &cd->MemForCompDict);
+  cd->chars_huff[which] = hd;
+  FIXUP (&cd->chars_huff[which]);
+  cd->chars_vals[which] = vals;
+  FIXUP (&cd->chars_vals[which]);
+  if (hd->clens)
+    delete hd->clens;
+  hd->clens = NULL;
+}
+
+static void load_read_words(FILE *dict, compression_dict *cd, int which, int fixescape)
+{
+  cd->cfh[which] = (comp_frags_header *) getmem (sizeof (*cd->cfh[which]), sizeof (u_char *));
+  cd->MemForCompDict += sizeof (*cd->cfh[which]);
+  Read_cfh (dict, cd->cfh[which], &cd->MemForCompDict, NULL);
+
+  cd->values[which] = ReadInWords (dict, cd, cd->cfh[which], NULL);
+  FIXUP (&cd->cfh[which]);
+  FIXUP (&cd->values[which]);
+  if (fixescape)
+    { FIXUP(&cd->escape[which]);
+    }
+  else
+    {
+      cd->escape[which] = NULL;
+    }
+}
+
+static void 
+load_comp_dict (char *filename)
+{
+  FILE *dict;
+  int which;
+  compression_dict *cd;
+
+  cd = (compression_dict *) getmem (sizeof (compression_dict), sizeof (u_char *));
+  cd->fast_loaded = 1;
+
+  dict = open_file (filename, TEXT_DICT_SUFFIX, "rb",
+		    MAGIC_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  Read_cdh (dict, &cd->cdh, NULL, NULL);
+
+  for (which = 0; which < 2; which++)
+    switch (cd->cdh.dict_type)
+      {
+      case MG_COMPLETE_DICTIONARY:
+	{
+	  load_read_words(dict, cd, which, 0);
+	}
+	break;
+      case MG_PARTIAL_DICTIONARY:
+	{
+	  if (cd->cdh.num_words[which])
+	    {
+	      load_read_words(dict, cd, which, 1);
+	    }
+
+	  load_read_huffman(dict, cd, which);
+	  load_read_huffman(dict, cd, which);
+	}
+	break;
+      case MG_SEED_DICTIONARY:
+	{
+	  if (cd->cdh.num_words[which])
+	    {
+	      load_read_words(dict, cd, which, 1);
+	    }
+	  switch (cd->cdh.novel_method)
+	    {
+	    case MG_NOVEL_HUFFMAN_CHARS:
+	      load_read_huffman(dict, cd, which);
+	      load_read_huffman(dict, cd, which);
+	      break;
+	    case MG_NOVEL_DELTA:
+	      break;
+	    case MG_NOVEL_HYBRID:
+	      break;
+	    }
+	  break;
+	}
+      }
+  fclose (dict);
+
+
+  if (cd->cdh.novel_method == MG_NOVEL_DELTA ||
+      cd->cdh.novel_method == MG_NOVEL_HYBRID)
+    {
+      cd->ad = LoadAuxDict (&cd->cdh, filename);
+      FIXUP (&cd->ad);
+    }
+}
+
+static void 
+save_fast_dict (char *filename)
+{
+  FILE *fdict;
+
+  fdict = create_file (filename, TEXT_DICT_FAST_SUFFIX, "wb",
+		       MAGIC_FAST_DICT, MG_ABORT);  /* [RPAP - Feb 97: WIN32 Port] */
+
+  {
+    u_long *p;
+    for (p = (u_long *) buffer; (u_long) p < (u_long) cur; p++)
+      {
+	if (IS_FIXUP (p))
+	  HTONUL(*p);
+      }
+  }
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONUL(mem);
+  HTONUL(fixup_mem);
+
+  fwrite (&mem, sizeof (mem), 1, fdict);
+  fwrite (&fixup_mem, sizeof (fixup_mem), 1, fdict);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(mem);
+  NTOHUL(fixup_mem);
+
+  fwrite (buffer, sizeof (u_char), mem, fdict);
+  fwrite (fixup, sizeof (u_char), fixup_mem, fdict);
+
+  fclose (fdict);
+}
Index: /trunk/mgpp/text/mgpp_invf_dict.1
===================================================================
--- /trunk/mgpp/text/mgpp_invf_dict.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_invf_dict.1	(revision 3365)
@@ -0,0 +1,79 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_invf_dict 1 \*(Dt CITRI
+.SH NAME
+mgpp_invf_dict \- build an `on-disk' stemmed dictionary
+.SH SYNOPSIS
+.B mgpp_invf_dict
+[
+.B \-h
+]
+[
+.BI \-b " entries-per-block"
+]
+.if n .ti +13n
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_invf_dict
+builds an `on-disk' stemmed dictionary from the
+compressed stemmed dictionary that can be used during querying.
+Each entry in the dictionary contains the disk
+pointer to the associated inverted file entry, and various other pieces
+of information about the stemmed word. The dictionary is blocked with
+each block containing the same number of entries (not the same size).
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectoryyyyyyyy\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-b " entries-per-block"
+The dictionary is stored in blocks on disk; this option is used to 
+specify the number of entries per block. The default is 16.
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.invf
+Inverted file.
+.TP
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.invf.dict.blocked
+The `on-disk' stemmed dictionary.
+.TP
+.B *.invf.idx
+The index into the inverted file.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/mgpp/text/mgpp_invf_dict.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_invf_dict.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_invf_dict.cpp	(revision 3365)
@@ -0,0 +1,237 @@
+/**************************************************************************
+ *
+ * mgpp_invf_dict.cpp -- Program to build the blocked stemmed dictionary
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "messages.h"
+#include "mg_files.h"
+#include "invf.h"
+
+static void process_files (char *filename, unsigned long entriesPerBlock) {
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // open the inverted index file
+  FILE *invfIdxFile = open_file (filename, INVF_IDX_SUFFIX, "rb",
+				 MAGIC_INVI, MG_ABORT);
+
+  // create the blocked dictionary
+  FILE *blockDictFile = create_file (filename, INVF_DICT_BLOCKED_SUFFIX, "wb",
+				     MAGIC_STEM, MG_ABORT);
+  block_dict_header bdh;
+  bdh.lookback = idh.lookback;
+  bdh.word_dict_start = idh.word_dict_start;
+  bdh.word_dict_size = idh.word_dict_size;
+  bdh.tag_dict_start = idh.tag_dict_start;
+  bdh.tag_dict_size = idh.tag_dict_size;
+  bdh.num_docs = idh.num_docs;
+  bdh.num_frags = idh.num_frags;
+  bdh.num_words = idh.num_words;
+  bdh.total_bytes = idh.total_bytes;
+  bdh.index_string_bytes = idh.index_string_bytes;
+  bdh.num_levels = idh.num_levels;
+  bdh.Write (blockDictFile);
+
+  
+  // write out the word part of the dictionary
+
+  bdh.entries_per_wblk = entriesPerBlock;
+  bdh.max_wblk_size = 0;
+  bdh.wblk_start = ftell (blockDictFile);
+  
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+
+  block_idx wordIdx;
+  word_block_dict_el wordBlockEl;
+  wordBlockEl.SetNumLevels (idh.num_levels);
+  
+  
+  unsigned long wordNum;
+  unsigned long wordInvfPtr;
+  UCArray lastEl;
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+    // read in the next word and inverted file pointer
+    wordEl.Read (dictFile, idh.num_levels);
+    ReadUL (invfIdxFile, wordInvfPtr);
+
+    // remember this word (and position) if this is the start
+    // of a new block
+    if (wordNum % entriesPerBlock == 0) {
+      block_idx_info elIdx;
+      elIdx.el = wordEl.el;
+      elIdx.block_ptr = ftell (blockDictFile);
+
+      // see if this block is the longest so far
+      if (wordIdx.size() > 0) {
+	unsigned long blockLen = elIdx.block_ptr -
+	  (*(wordIdx.end()-1)).block_ptr;
+	if (blockLen > bdh.max_wblk_size) bdh.max_wblk_size = blockLen;
+      }
+      
+      wordIdx.push_back (elIdx);
+      lastEl.erase (lastEl.begin(), lastEl.end()); // output full word
+    }
+
+    // copy the information for this word
+    wordBlockEl.el = wordEl.el;
+    wordBlockEl.frag_occur = wordEl.frag_occur;
+    wordBlockEl.freq = wordEl.freq;
+    wordBlockEl.invf_ptr = wordInvfPtr;
+    unsigned long tempI;
+    for (tempI=0; tempI<idh.num_levels; tempI++)
+      wordBlockEl.levelFreqs[tempI] = wordEl.levelFreqs[tempI];
+
+    // write out the word
+    wordBlockEl.Write (blockDictFile, &lastEl, idh.num_levels);
+
+    lastEl = wordBlockEl.el;
+  }
+
+
+  // write out the tag part of the dictionary
+
+  bdh.entries_per_tblk = entriesPerBlock;
+  bdh.max_tblk_size = 0;
+  bdh.tblk_start = ftell (blockDictFile);
+  
+  fseek (dictFile, idh.tag_dict_start, SEEK_SET);
+
+  block_idx tagIdx;
+  block_dict_el tagBlockEl;
+  
+  unsigned long tagNum;
+  unsigned long tagInvfPtr;
+  dict_el tagEl;
+  lastEl.erase (lastEl.begin(), lastEl.end());
+  for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+    // read in the next tag and inverted file pointer
+    tagEl.Read (dictFile);
+    ReadUL (invfIdxFile, tagInvfPtr);
+
+    // remember this tag (and position) if this is the start
+    // of a new block
+    if (tagNum % entriesPerBlock == 0) {
+      block_idx_info elIdx;
+      elIdx.el = tagEl.el;
+      elIdx.block_ptr = ftell (blockDictFile);
+
+      // see if this block is the longest so far
+      if (tagIdx.size() > 0) {
+	unsigned long blockLen = elIdx.block_ptr -
+	  (*(tagIdx.end()-1)).block_ptr;
+	if (blockLen > bdh.max_tblk_size) bdh.max_tblk_size = blockLen;
+      }
+      
+      tagIdx.push_back (elIdx);
+      lastEl.erase (lastEl.begin(), lastEl.end()); // output full word
+    }
+
+    // copy the information for this tag
+    tagBlockEl.el = tagEl.el;
+    tagBlockEl.frag_occur = tagEl.frag_occur;
+    tagBlockEl.freq = tagEl.freq;
+    tagBlockEl.invf_ptr = tagInvfPtr;
+
+    // write out the tag
+    tagBlockEl.Write (blockDictFile, &lastEl);
+
+    lastEl = tagBlockEl.el;
+  }
+
+
+  // write out the element indexes
+  bdh.num_wblks = wordIdx.size();
+  bdh.wblk_idx_start = ftell (blockDictFile);
+  WriteBlockIdx (blockDictFile, wordIdx);
+  
+  bdh.num_tblks = tagIdx.size();
+  bdh.tblk_idx_start = ftell (blockDictFile);
+  WriteBlockIdx (blockDictFile, tagIdx);
+
+  // write out the blocked dictionary header
+  fseek (blockDictFile, sizeof(unsigned long), SEEK_SET);
+  bdh.Write (blockDictFile);
+
+
+  // close open files
+  fclose (blockDictFile);
+  fclose (invfIdxFile);
+  fclose (dictFile);
+  
+  // print out information
+#ifndef SILENT
+  Message ("Max word block size = %d\n", bdh.max_wblk_size);
+  Message ("Max tag block size = %d\n", bdh.max_tblk_size);
+  Message ("Number of word blocks written = %d\n", bdh.num_wblks);
+  Message ("Number of tag blocks written = %d\n", bdh.num_tblks);
+#endif
+}
+
+
+int main (int argc, char **argv) {
+  unsigned long entriesPerBlock = 16;
+  char *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  while ((ch = getopt (argc, argv, "f:d:b:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'b':
+      entriesPerBlock = atoi (optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-f input_file] "
+	       "[-d data directory] [-b entries-per-block] "
+	       "[-h]\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  process_files (filename, entriesPerBlock);
+  return 0;
+}
Index: /trunk/mgpp/text/mgpp_invf_dict_dump.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_invf_dict_dump.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_invf_dict_dump.cpp	(revision 3365)
@@ -0,0 +1,141 @@
+/**************************************************************************
+ *
+ * mgpp_invf_dict_dump.cpp -- Program to printthe various inverted dictionaries
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "messages.h"
+#include "mg_files.h"
+#include "invf.h"
+
+static void process_files (char *filename,
+			   bool printHeader,
+			   bool printWords,
+			   bool printTags) {
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // print the information from the inverted dictionary file header
+  if (printHeader) {
+    cout << "lookback: " << idh.lookback << "\n";
+    cout << "word_dict_start: " << idh.word_dict_start << "\n";
+    cout << "word_dict_size: " << idh.word_dict_size << "\n";
+    cout << "tag_dict_start: " << idh.tag_dict_start << "\n";
+    cout << "tag_dict_size: " << idh.tag_dict_size << "\n";
+    cout << "num_docs: " << idh.num_docs << "\n";
+    cout << "num_frags: " << idh.num_frags << "\n";
+    cout << "num_words: " << idh.num_words << "\n";
+    cout << "total_bytes: " << idh.total_bytes << "\n";
+    cout << "index_string_bytes: " << idh.index_string_bytes << "\n";
+    cout << "num_levels: " << idh.num_levels << "\n";
+
+    cout << "\n";
+  }
+
+
+  if (printWords) {
+    fseek (dictFile, idh.word_dict_start, SEEK_SET);
+    
+    unsigned long wordNum;
+    word_dict_el wordEl;
+    wordEl.SetNumLevels (idh.num_levels);
+    for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+      wordEl.Read (dictFile, idh.num_levels);
+      cout << "\"" << wordEl.el << "\"\n";// (" << wordNum << ")\n";
+    }
+    
+    cout << "\n";
+  }
+
+  // write out the tag part of the dictionary
+  if (printTags) {
+    fseek (dictFile, idh.tag_dict_start, SEEK_SET);
+    
+    unsigned long tagNum;
+    dict_el tagEl;
+    for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+      // read in the next tag and inverted file pointer
+      tagEl.Read (dictFile);
+      
+      cout << "\"" << tagEl.el << "\" (" << tagNum << ")\n";
+    }
+  }
+  
+  // close open files
+  fclose (dictFile);
+}
+
+
+int main (int argc, char **argv) {
+  char *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  bool printHeader = false;
+  bool printWords = false;
+  bool printTags = false;
+
+  while ((ch = getopt (argc, argv, "f:d:rwth")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'r':
+      printHeader = true;
+      break;
+    case 'w':
+      printWords = true;
+      break;
+    case 't':
+      printTags = true;
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-r] [-w] [-t] [-f input_file] "
+	       "[-d data directory]\n(-rwt:print header, words, tags)\n", 
+	       argv[0]);
+      exit (1);
+    }
+  }
+
+  process_files (filename, printHeader, printWords, printTags);
+  return 0;
+}
Index: /trunk/mgpp/text/mgpp_invf_dump.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_invf_dump.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_invf_dump.cpp	(revision 3365)
@@ -0,0 +1,244 @@
+/**************************************************************************
+ *
+ * mgpp_invf_dump.cpp -- Program to dump uot an inverted fil
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "messages.h"
+#include "bitio_m_stdio.h"
+#include "bitio_gen.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+
+#include "mg_files.h"
+#include "locallib.h"
+#include "words.h"
+#include "invf.h"
+#include "WordData.h"
+
+static void PrintInvfWord (FILE *invfFile,
+			   invf_dict_header &idh,
+			   invf_file_header &ifh,
+			   word_dict_el &wordEl,
+			   unsigned long wordStart,
+			   bool printFrags) {
+  cout << wordEl.frag_occur << " \"" << wordEl.el << "\"\n";
+  
+  if (printFrags) {
+    // seek to the appropriate place in the inverted file
+    fseek (invfFile, wordStart, SEEK_SET);
+    
+    stdio_bitio_buffer buffer(invfFile);
+    
+    unsigned long B = BIO_Bblock_Init (idh.num_frags, wordEl.frag_occur);
+    unsigned long fragNum = 0;
+    unsigned long i;
+    for (i=0; i<wordEl.frag_occur; i++) {
+      unsigned long delta = buffer.bblock_decode (B, NULL);
+      fragNum += delta;
+      cout << " " << fragNum;
+      
+      if (!ifh.word_level_index ) {
+	unsigned long count = buffer.gamma_decode (NULL);
+	cout << "(" << count << ")";
+      } else {
+	cout << "(1)";
+      }
+    }
+    
+    cout << "\n";
+  
+    buffer.done();
+  }
+}
+
+static void PrintInvfTag (FILE *invfFile,
+			  invf_dict_header &idh,
+			  invf_file_header &/*ifh*/,
+			  dict_el &tagEl,
+			  unsigned long tagStart,
+			  bool printFrags) {
+  cout << tagEl.frag_occur << " \"<" << tagEl.el << ">\"\n";
+  
+  if (printFrags) {
+    // seek to the appropriate place in the inverted file
+    fseek (invfFile, tagStart, SEEK_SET);
+    
+    stdio_bitio_buffer buffer(invfFile);
+    
+    unsigned long pTag = tagEl.frag_occur*2;
+    unsigned long B = BIO_Bblock_Init (idh.num_frags+pTag, pTag);
+    unsigned long fragNum = 0;
+    unsigned long i;
+    for (i=0; i<tagEl.frag_occur; i++) {
+      unsigned long delta = buffer.bblock_decode (B, NULL)-1;
+      fragNum += delta;
+      cout << " " << fragNum;
+      cout << "-";
+      delta = buffer.bblock_decode (B, NULL)-1;
+      fragNum += delta;
+      cout << fragNum;
+    }
+    
+    cout << "\n";
+    
+    buffer.done();
+  }
+}
+
+static void PrintHeaderInfo (invf_dict_header &idh,
+			     invf_file_header &ifh) {
+  cerr << "Lookback:         " << idh.lookback << "\n";
+  cerr << "Word Dict Size:   " << idh.word_dict_size << "\n";
+  cerr << "Tag Dict Size:    " << idh.tag_dict_size << "\n";
+  cerr << "Num Documents:    " << idh.num_docs << "\n";
+  cerr << "Num Fragments:    " << idh.num_frags << "\n";
+  cerr << "Num Words:        " << idh.num_words << "\n";
+
+  cerr << "Skip Mode:        " << ifh.skip_mode << "\n";
+  cerr << "Word Level Index: " << ifh.word_level_index << "\n";
+
+  cerr << "\n";
+}
+
+
+static void process_files (char *filename,
+			   bool printHeader,
+			   bool printWords,
+			   bool printTags,
+			   bool printFrags) {
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // open the inverted file
+  FILE *invfFile = open_file (filename, INVF_SUFFIX, "rb",
+			MAGIC_INVF, MG_ABORT);
+  
+  invf_file_header ifh;
+  ifh.Read (invfFile);
+
+  if (ifh.skip_mode != SKIP_MODE_NO_SKIPS)
+    FatalError (1, "The invf file contains skips. Unable to dump.");
+
+  // print out header information
+  if (printHeader) {
+    PrintHeaderInfo (idh, ifh);
+  }
+
+  // open the inverted index
+  FILE *invfIdxFile = open_file (filename, INVF_IDX_SUFFIX, "rb",
+				 MAGIC_INVI, MG_ABORT);
+
+  // go to the start of the word dictionary
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+
+  // process all the words
+  if (printWords) {
+    unsigned long wordNum;
+    unsigned long wordStart;
+    word_dict_el wordEl;
+    wordEl.SetNumLevels (idh.num_levels);
+    for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+      wordEl.Read (dictFile, idh.num_levels);
+      ReadUL (invfIdxFile, wordStart);
+      PrintInvfWord (invfFile, idh, ifh, wordEl, wordStart, printFrags);
+    }
+  }
+
+  // process all the tags
+  if (printTags) {
+    unsigned long tagNum;
+    unsigned long tagStart;
+    dict_el tagEl;
+    for (tagNum=0; tagNum<idh.tag_dict_size; tagNum++) {
+      tagEl.Read (dictFile);
+      ReadUL (invfIdxFile, tagStart);
+      PrintInvfTag (invfFile, idh, ifh, tagEl, tagStart, printFrags);
+    }
+  }
+  // close the open files
+  fclose (invfIdxFile);
+  fclose (invfFile);
+  fclose (dictFile);
+}
+
+
+int main (int argc, char **argv) {
+  char *dir_name, *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  dir_name = getenv ("MGDATA");
+  opterr = 0;
+
+  bool printHeader = false;
+  bool printWords = false;
+  bool printTags = false;
+  bool printFrags = false;
+  
+  msg_prefix = argv[0];
+  while ((ch = getopt (argc, argv, "hrwtnf:d:")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath(optarg);
+      break;
+    case 'r':
+      printHeader = true;
+      break;
+    case 'w':
+      printWords = true;
+      break;
+    case 'n':
+      printFrags = true;
+      break;
+    case 't':
+      printTags = true;
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-h] [-r] [-w] [-t] [-n] [-f input_file]"
+	       "[-d data directory]\n(-rwnt:print header, words, tags, fragnums)\n", 
+	       argv[0]);
+      exit (1);
+    }
+  }
+  
+  process_files (filename, printHeader, printWords, printTags, printFrags);
+
+  return 0;
+}
Index: /trunk/mgpp/text/mgpp_passes.1
===================================================================
--- /trunk/mgpp/text/mgpp_passes.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_passes.1	(revision 3365)
@@ -0,0 +1,323 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_passes 1 \*(Dt CITRI
+.SH NAME
+mgpp_passes \- builds mgpp databases
+.SH SYNOPSIS
+.B mgpp_passes
+[
+.BI \-J " doc-tag"
+]
+[
+.BI \-K " level-tag"
+]
+.if n .ti +10n
+[
+.BI \-L " index-level"
+]
+[
+.BI \-m " invf-mem-buffer"
+]
+.if n .ti +10n
+[
+.B \-T1
+]
+[
+.B \-T2
+]
+[
+.B \-I1
+]
+[
+.B \-I2
+]
+[
+.B \-S
+]
+[
+.B \-C
+]
+.if n .ti +10n
+[
+.B \-h
+]
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+[
+.I filename(s)
+]
+.SH DESCRIPTION
+.B mgpp_passes
+is the program that does most of the work when building mgpp
+database systems.  The input documents can come from either
+.I stdin
+or from a list of files on the command line.  In general,
+.B mgpp_passes
+must be run twice to build a database, first with the
+.B \-T1
+and
+.B \-I1
+options, and second with the
+.B \-T2
+and
+.B \-I2
+options.  Several other programs must be run in order to get an
+mgpp database.  The
+.SB EXAMPLE
+section below gives an example of how to build a complete
+mgpp database.
+.SH OPTIONS
+Options may appear in any order, but the
+.IR filename(s) ,
+if specified, must be last.
+.TP "\w'\fB\-C\fP \fIcompstatpointt\fP'u+2n"
+.BI \-J " doc-tag"
+Specifies the SGML tag that encloses each document.  Text appearing
+outside this tag is ignored.  The document tag defines the highest
+level document that can be queried and printed.  The default document
+tag is 'Document'.
+.TP
+.BI \-K " level-tag"
+Specifies the SGML tag of a sub document level.  A level tag must
+enclose all text enclosed by the document tag.  Levels can be
+queried and printed as if they were separate documents.  Multiple
+document levels can be specified (the document tag is always
+added as a document level).
+.TP
+.BI \-L " index-level"
+Specifies the SGML tag enclosing the smallest indexed element.  The
+index level should be no larger than the smallest document 
+level.  An empty string can be used to specify a word level index 
+(which is the default).
+.TP
+.BI \-m " invf-mem-buffer"
+Maximum amount of memory to use for the pass-2 file inversion in
+megabytes.  This option is only useful when used in conjunction with
+the option
+.BR \-I1 .
+The larger this value, the faster the pass-2 inversion will proceed.
+The default value is 5 MB.
+.TP
+.B \-T1
+Generate the
+.I *.text.stats
+file.
+.TP
+.B \-T2
+Generate the
+.IR *.text ,
+.IR *.text.idx ,
+.IR *.text.level ,
+and possibly the
+.I *.text.dict.aux
+files.  Using this option requires that the
+.I *.text.dict
+file be present.
+.TP
+.B \-I1
+Generate the
+.IR *.invf.dict ,
+.IR *.invf.level ,
+.IR *.invf.chunk ,
+and
+.I *.invf.chunk.trans
+files.
+.TP
+.B \-I2
+Generate the
+.I *.invf
+and
+.I *.invf.idx
+files.  Using this option requires
+that the
+.IR *.invf.dict.hash ,
+.IR *.invf.level ,
+.IR *.invf.chunk ,
+and
+.I *.invf.chunk.trans
+files be present.  The
+.I *.invf.dict.hash
+file is generated by
+.BR mgpp_perf_hash_build (1)
+from the
+.I *.invf.dict
+file.
+.TP
+.B \-S
+This option causes a special pass to be executed.  It is up to a user
+to modify
+.I mg.special.c
+in the source code to do something with the documents it is given.
+.TP 
+.B \-C
+This activates the compatibility parsing mode.  When using this
+mode documents are separated by control-B and paragraphs are separated
+by control-C.  Internally these are converted to documents surrounded
+by 'Document' tags and paragraphs surrounded by 'Paragraph' tags.
+.TP
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection is to be
+written.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection that will be
+created.
+.TP
+.I filename(s)
+This specifies the source text. If this is not specified, then the
+program expects the source text from
+.IR stdin .
+.SH EXAMPLE
+What follows is a UNIX
+.BR csh (1)
+script as an example of how to build an mgpp document collection.
+.LP
+.nf
+.DT
+.ft B
+.I #! /bin/csh
+.I
+# The first argument on the command line specifies the
+.I
+#   source of the text
+set source = ($1)
+.PP
+.I
+# The second argument is the name of the collection
+set text = ($2)
+.PP
+.I
+# Create *.text.stats, *.invf.dict, *.invf.level
+.I
+#   *.invf.chunk and *.invf.chunks.trans
+${source} | mgpp_passes -T1 -I1 -f ${text}
+.PP
+.I
+# Create *.text.dict
+mgpp_compression_dict -f ${text}
+.PP
+.I
+# Create *.invf.dict.hash
+mgpp_perf_hash_build -f ${text}
+.PP
+.I
+# Create *.text, *.text.idx, *.text.level
+.I
+#   *.invf and *.invf.idx
+${source} | mgpp_passes -T2 -I2 -f ${text}
+.PP
+.I
+# Create *.text.weight and *.weight.approx
+mgpp_weights_build -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked
+mgpp_invf_dict -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked.1
+mgpp_stem_idx -s 1 -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked.2
+mgpp_stem_idx -s 2 -f ${text}
+.PP
+.I
+# Create *.invf.dict.blocked.3
+mgpp_stem_idx -s 3 -f ${text}
+.PP
+.I
+# Create *.text.dict.fast
+mgpp_fast_comp_dict -f ${text}
+.ft R
+.fi
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 22
+.B *.invf
+Inverted file.
+.TP
+.B *.invf.chunk
+Inverted file chunk descriptor file.  When the inverted file is
+created it is created in chunks that use no more than a set amount of
+memory.  This file describes those chunks.
+.TP
+.B *.invf.chunk.trans
+Word-occurrence-order to lexical-order translation file.  The
+.B *.invf.chunk
+file is written in word-occurrence order but is required by
+.B \-I2
+to be in lexical order.
+.TP
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.invf.dict.blocked
+Compressed stemmed dictionary with index into the dictionary.
+.TP
+.B *.invf.dict.blocked.n
+Transformation dictionary from words stemmed with method 
+.B n
+to unstemmed words.
+.TP
+.B *.invf.dict.hash
+Data for an order-preserving perfect hash function.
+.TP
+.B *.invf.idx
+The index into the inverted file.
+.TP
+.B *.invf.level
+Information about the document levels needed for querying.
+.TP
+.B *.text
+Compressed text.
+.TP
+.B *.text.dict
+Compressed compression dictionary.
+.TP
+.B *.text.dict.fast
+A fast loading version of the compressed compression dictionary.
+.TP
+.B *.text.idx
+Index into the compressed documents.
+.TP
+.B *.text.level
+Information about the document levels needed for text decompression.
+.TP
+.B *.text.stats
+Statistics about the text.
+.TP
+.B *.weight
+The exact weights file.
+.TP
+.B *.weight.approx
+The approximate weights file.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/mgpp/text/mgpp_passes.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_passes.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_passes.cpp	(revision 3365)
@@ -0,0 +1,264 @@
+/**************************************************************************
+ *
+ * mgpp_passes.cpp -- Driver for the various passes
+ * Copyright (C) 1994  Neil Sharman
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "sysfuncs.h"
+
+#ifdef HAVE_MALLINFO
+# include <malloc.h> 
+#endif
+
+#if defined __WIN32__
+# include <io.h>
+# include "getopt_old.h"
+# define close _close
+# define open _open
+#elif defined __CYGWIN__
+#include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "memlib.h"
+#include "messages.h"
+#include "longlong.h"
+#include "mg_files.h"
+#include "mg.h"
+#include "build.h"
+#include "text.h"
+#include "stemmer.h"
+#include "FileBuf.h"
+#include "TextEl.h"
+#include "TagInfo.h"
+
+#define MAX_PASSES 5
+
+#define SPECIAL 1
+#define TEXT_PASS_1 2
+#define TEXT_PASS_2 4
+#define IVF_PASS_1 8
+#define IVF_PASS_2 16
+
+#define MIN_BUF 8192
+
+
+unsigned long invf_buffer_size = 5 * 1024 * 1024;	/* 5Mb */
+
+static char Passes = 0;
+static char **files = NULL;
+static int num_files = 0;
+
+
+struct pass_data {
+  char *name;
+  int (*init) (const TagInfo &tagInfo, char *);
+  int (*process) (const TagInfo &tagInfo, const TextElArray &doc);
+  int (*done) (const TagInfo &tagInfo, char *);
+};
+
+
+static pass_data PassData[MAX_PASSES] = {
+  {"special", init_special, process_special, done_special},
+  {"text.pass1", init_text_1, process_text_1, done_text_1},
+  {"text.pass2", init_text_2, process_text_2, done_text_2},
+  {"ivf.pass1", init_ivf_1, process_ivf_1, done_ivf_1},
+  {"ivf.pass2", init_ivf_2, process_ivf_2, done_ivf_2},
+};
+
+static char *usage_str = "\nUSAGE:\n"
+"  %s [-J doc-tag] [-K level-tag] [-L index-level]\n"
+"  %*s [-m invf-memory] [-T1] [-T2] [-I1] [-I2] [-S]\n"
+"  %*s [-C] [-h] [-d directory] -f name\n\n";
+
+
+
+static void Usage (char *err) {
+  if (err) Message (err);
+  fprintf (stderr, usage_str, msg_prefix, strlen (msg_prefix), "",
+	   strlen (msg_prefix), "");
+  exit (1);
+}
+
+
+
+
+int OpenNextFile (int in_fd) {
+  if (in_fd > 0) close (in_fd);
+  if (num_files == 0) return (-1);
+  if ((in_fd = open (files[0], O_RDONLY)) == -1)
+    FatalError (1, "Cannot open %s", files[0]);
+  files++;
+  num_files--;
+  return (in_fd);
+}
+
+
+static void Driver (int in_fd, char *file_name,
+		    const TagInfo &tagInfo, bool compatMode) {
+  //  cout << tagInfo;
+
+  int pass;
+
+  unsigned long numBytes = 0;
+  unsigned long numDocs = 0;
+
+  // initialise all the passes
+  for (pass = 0; pass < MAX_PASSES; pass++) {
+    if (Passes & (1 << pass)) {
+      if (PassData[pass].init (tagInfo, file_name) == COMPERROR)
+	FatalError (1, "Error during init of \"%s\"", PassData[pass].name);
+    }
+  }
+
+  
+  // set up various variables
+  FileBuf buf;
+  TextElArray doc;
+  unsigned long docLen = 0;
+
+  // read and process each file (start with an open file)
+  do {
+    
+    // read and process each document in this file
+    buf.SetFD (in_fd);
+    while (ReadDoc (buf, tagInfo.docTag, doc, docLen, compatMode)) {
+
+      // give this document to each pass
+      for (pass = 0; pass < MAX_PASSES; pass++) {
+	if (Passes & (1 << pass)) {
+	  if (PassData[pass].process (tagInfo, doc) == COMPERROR)
+	    FatalError(1,"Error during processing of \"%s\"",PassData[pass].name);
+	}
+      }
+      
+      // another document has been processed
+      numBytes += docLen;
+      numDocs++;
+    }
+    
+  } while ((in_fd = OpenNextFile (in_fd)) > 0);
+
+  
+  // do done for each pass
+  for (pass = 0; pass < MAX_PASSES; pass++) {
+    if (Passes & (1 << pass)) {
+      if (PassData[pass].done (tagInfo, file_name) == COMPERROR)
+	FatalError (1, "Error during done of \"%s\"", PassData[pass].name);
+    }
+  }
+}
+
+int main (int argc, char **argv) {
+  int ch, in_fd;
+  char *filename = NULL;
+  bool compatMode = false;
+  TagInfo tagInfo;
+  tagInfo.SetDocTag ("Document");
+  
+  msg_prefix = argv[0];
+
+  opterr = 0;
+  while ((ch=getopt(argc, argv, "J:K:L:f:d:m:I:T:SCh"))!=-1){
+    switch (ch) {
+    case 'J':
+      tagInfo.SetDocTag (optarg);
+      break;
+    case 'K':
+      tagInfo.AddLevelTag (optarg);
+      break;
+    case 'L':
+      tagInfo.SetIndexLevel (optarg);
+      break;
+    case 'f':
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'm':
+      invf_buffer_size = (int) (atof (optarg) * 1024 * 1024);
+      break;
+    case 'I':
+      if (*optarg == '1')
+	Passes |= IVF_PASS_1;
+      else if (*optarg == '2')
+	Passes |= IVF_PASS_2;
+      else
+	Usage ("Invalid pass number");
+      break;
+    case 'T':
+      if (*optarg == '1')
+	Passes |= TEXT_PASS_1;
+      else if (*optarg == '2')
+	Passes |= TEXT_PASS_2;
+      else
+	Usage ("Invalid pass number");
+      break;
+    case 'S':
+      Passes |= SPECIAL;
+      break;
+    case 'C':
+      compatMode = true;
+      break;
+    case 'h':
+    case '?':
+      Usage (NULL);
+    }
+  }
+
+  if (!filename || *filename == '\0')
+    FatalError (1, "A document collection name must be specified.");
+
+  if ((Passes & (IVF_PASS_1 | IVF_PASS_2)) == (IVF_PASS_1 | IVF_PASS_2))
+    FatalError (1, "I1 and I2 cannot be done simultaneously.");
+
+  if ((Passes & (TEXT_PASS_1 | TEXT_PASS_2)) == (TEXT_PASS_1 | TEXT_PASS_2))
+    FatalError (1, "T1 and T2 cannot be done simultaneously.");
+
+  if (!Passes)
+    FatalError (1, "S, T1, T2, I1 or I2 must be specified.");
+
+  if (optind < argc) {
+    if ((in_fd = open (argv[optind], O_RDONLY)) == -1)
+      FatalError (1, "Cannot open %s", argv[optind]);
+    files = &argv[optind + 1];
+    num_files = argc - (optind + 1);
+    
+  } else in_fd = 0;  // stdin
+
+  
+  if (compatMode) tagInfo.SetDocTag ("Document");
+
+  // a document tag is also a level tag
+  tagInfo.levelTags.insert (tagInfo.docTag); 
+  
+  Driver (in_fd, filename, tagInfo, compatMode);
+
+  return (0);
+}
Index: /trunk/mgpp/text/mgpp_perf_hash_build.1
===================================================================
--- /trunk/mgpp/text/mgpp_perf_hash_build.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_perf_hash_build.1	(revision 3365)
@@ -0,0 +1,72 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_perf_hash_build 1 \*(Dt CITRI
+.SH NAME
+mgpp_perf_hash_build \- generate an order-preserving hash function for the stemmed dictionary
+.SH SYNOPSIS
+.B mgpp_perf_hash_build
+[
+.B \-h
+]
+[
+.BI \-r " rndseed"
+]
+[
+.BI \-d  " directory"
+]
+.if n .ti +9n
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_perf_hash_build
+generates an order-preserving hashing function from the compressed
+stemmed dictionary and writes it out to disk.
+.BR mgpp_passes (1)
+will make use of the hash function when it builds the inverted file.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-r " rndseed"
+This specifies the random seed to be used in the generation of the
+hash function; it must be an integer.  If this is not specified, the
+current time will be used as the random seed.
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mg
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.invf.dict
+Stemmed dictionary.
+.TP
+.B *.invf.dict.hash
+Data for a order-preserving perfect hash function.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_stem_idx (1),
+.BR mgpp_weights_build (1)
Index: /trunk/mgpp/text/mgpp_perf_hash_build.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_perf_hash_build.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_perf_hash_build.cpp	(revision 3365)
@@ -0,0 +1,146 @@
+/**************************************************************************
+ *
+ * mgpp_perf_hash_build.cpp -- Program to build a perfect hash function
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "perf_hash.h"
+#include "netorder.h"
+
+#include "mg_files.h"
+#include "invf.h"
+#include "locallib.h"
+#include "words.h"
+#include "mg.h"
+
+#define POOL_SIZE 1024*1024
+
+static void ProcessFiles (char *filename, int r) {
+  FILE *dictFile, *hashFile;
+  unsigned long i;
+  invf_dict_header idh;
+  perf_hash_data *phd;
+  u_char *pool;
+  unsigned long pool_left;
+  u_char **starts;
+
+  // read in the dictionary
+  dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			MAGIC_STEM_BUILD, MG_ABORT);
+  if (dictFile==NULL) {
+    FatalError(1, "unable to open file");
+  }
+  idh.Read (dictFile);
+
+  //cerr << idh.lookback<<" "<<idh.word_dict_start<<endl;
+  // go to the start of the word dictionary
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+  
+  if (!(pool = (u_char *) Xmalloc (POOL_SIZE)))
+    FatalError (1, "Out of memory");
+  pool_left = POOL_SIZE;
+
+  if (!(starts = (u_char **) Xmalloc (sizeof (u_char *) * idh.word_dict_size)))
+    FatalError (1, "Out of memory");
+  //cerr << "size= "<< idh.word_dict_size<<endl;
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (i = 0; i < idh.word_dict_size; i++) {
+    // read the next word and associated information
+    wordEl.Read (dictFile, idh.num_levels);
+
+    // push string onto pool data
+    register unsigned long l;
+    l = wordEl.el.size() + 1;
+    if (pool_left < l) {
+      pool = (u_char *) Xmalloc (POOL_SIZE);
+      pool_left = POOL_SIZE;
+    }
+    starts[i] = pool;
+    
+    *pool++ = wordEl.el.size();
+    memcpy ((char *) pool, &(wordEl.el[0]), wordEl.el.size());
+    //cerr << pool<<"   " <<starts[i]<<endl;
+    pool += wordEl.el.size();
+    pool_left -= l;
+    
+  }
+  fclose (dictFile);
+  //cerr << pool<<"   " <<starts[i-1]<<endl;
+  //cerr<<"starts  "<<starts[113529]<<endl;
+  //cerr << starts[17][1] << " "<<starts[25][4]<<endl;
+  // create perfect hash file
+  hashFile = create_file (filename, INVF_DICT_HASH_SUFFIX, "wb",
+			  MAGIC_HASH, MG_ABORT);
+  if (!(phd = gen_hash_func (idh.word_dict_size, starts, r)))
+    FatalError (1, "Unable to generate hash function");
+  if (write_perf_hash_data (hashFile, phd) == -1)
+    FatalError (1, "Unable to write hash function");
+  fclose (hashFile);
+}
+
+
+
+int main (int argc, char **argv) {
+  int r = -1;
+  char *filename = "";
+  int ch;
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  while ((ch = getopt (argc, argv, "f:d:r:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'r':
+      r = atoi (optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-f input_file]"
+	       "[-d data directory] [-r random seed] [-h]\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  ProcessFiles (filename, r);
+  return 0;
+}
Index: /trunk/mgpp/text/mgpp_stem_idx.1
===================================================================
--- /trunk/mgpp/text/mgpp_stem_idx.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_stem_idx.1	(revision 3365)
@@ -0,0 +1,95 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_stem_idx 1 \*(Dt CITRI
+.SH NAME
+mgpp_stem_idx \- builds a stem index file
+.SH SYNOPSIS
+.B mgpp_stem_idx
+[
+.B \-h
+]
+[
+.BI \-b " entries-per-block"
+]
+.if n .ti +12n
+[
+.BI \-a " stemmer"
+]
+[
+.BI \-d " directory"
+]
+.if n .ti +12n
+.B \-s 1|2|3
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_stem_idx
+generates a stem index file for a collection.
+This program should be called three times: once for each
+.B -s
+parameter.  It uses the stemmed dictionary to create the stem index which
+contains pointers into the stemmed dictionary.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIdirectoryyyyyyyy\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-b " entries-per-block"
+The dictionary is stored in blocks on disk; this option is used to set
+the number of entries per block. The default is 16.
+.TP
+.BI \-a " stemmer"
+The name of the stemmer to use, the default is the Lovin stemmer.
+.TP
+.B -s 1|2|3
+The stem method to apply for the stem index.
+.br
+1 = casefolded and non-stemmed
+.br
+2 = non-casefolded and stemmed
+.br
+3 = casefolded and stemmed
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection can be found.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 22
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.invf.dict.blocked.1
+Stem index with stem index method 1.
+.TP
+.B *.invf.dict.blocked.2
+Stem index with stem index method 2.
+.TP
+.B *.invf.dict.blocked.3
+Stem index with stem index method 3.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_weights_build (1)
Index: /trunk/mgpp/text/mgpp_stem_idx.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_stem_idx.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_stem_idx.cpp	(revision 3365)
@@ -0,0 +1,237 @@
+/**************************************************************************
+ *
+ * mgpp_stem_idx.cpp -- stem index builder
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "messages.h"
+#include "mg_files.h"
+#include "invf.h"
+#include "words.h"
+#include "stemmer.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#  include <ospace\std\vector>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#  include <vector.h>
+#else
+#  include <map>
+#  include <vector>
+#endif
+
+typedef vector<unsigned long> WordNumList;
+typedef map<UCArray, WordNumList, DictLTUCArray> StemMapDict;
+
+void CreateStemDict (char *filename,
+		     StemMapDict &stemDict,
+		     int stemMethod,
+		     int stemmerNum) {
+  stemDict.erase (stemDict.begin(), stemDict.end());
+
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+
+  unsigned long wordNum;
+  u_char mgWord[MAXSTEMLEN + 1];
+  word_dict_el wordEl;
+  UCArray stemEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+    // read in the next word
+    wordEl.Read (dictFile, idh.num_levels);
+
+    // convert the word to an "mg word"
+    mgWord[0] = wordEl.el.size();
+    memcpy((char *)&mgWord[1], &(wordEl.el[0]), wordEl.el.size());
+
+    // stem the word
+    stemmer (stemMethod, stemmerNum, mgWord);
+
+    // convert the result back to a UCArray
+    stemEl.erase (stemEl.begin(), stemEl.end());
+    stemEl.insert (stemEl.end(), &mgWord[1], &mgWord[1] + mgWord[0]);
+
+//      cout << "\"" << stemEl << "\" -> \"" << wordEl.el << "\"\n";
+    
+    // add this word number to the list of word numbers for this word
+    stemDict[stemEl].push_back (wordNum);
+  }
+
+  fclose (dictFile);
+}
+
+
+void WriteStemDict (char *filename,
+		    StemMapDict &stemDict,
+		    int stemMethod,
+		    int stemmerNum,
+		    unsigned long entriesPerBlock) {
+  // Create appropriate stem index file
+  FILE *stemDictFile = NULL;
+  if (stemMethod == 1) {
+    stemDictFile = create_file (filename, INVF_DICT_BLOCKED_1_SUFFIX,
+				"wb", MAGIC_STEM_1, MG_ABORT);
+  } else if (stemMethod == 2) {
+    stemDictFile = create_file (filename, INVF_DICT_BLOCKED_2_SUFFIX,
+				"wb", MAGIC_STEM_2, MG_ABORT);
+  } else if (stemMethod == 3) {
+    stemDictFile = create_file (filename, INVF_DICT_BLOCKED_3_SUFFIX,
+				"wb", MAGIC_STEM_3, MG_ABORT);
+  } else {
+    FatalError (1, "Unknown stem method %d", stemMethod);
+  }
+
+  stem_idx_header sih;
+  sih.lookback = 0;
+  sih.dict_size = stemDict.size();
+  sih.entries_per_block = entriesPerBlock;
+  sih.max_block_size = 0;
+  
+  sih.stemmer_num = stemmerNum;
+  sih.stem_method = stemMethod;
+
+  // write out a place-holder version of the header
+  sih.Write (stemDictFile);
+
+  sih.blocks_start = ftell (stemDictFile);
+
+  block_idx stemIdx;
+  unsigned long stemNum = 0;
+  stem_block_dict_el stemEl;
+  UCArray lastEl;
+
+  StemMapDict::const_iterator here = stemDict.begin();
+  StemMapDict::const_iterator end = stemDict.end();
+  while (here != end) {
+    // remember this stem (and position) if this is the start
+    // of a new block
+    if (stemNum % entriesPerBlock == 0) {
+      block_idx_info elIdx;
+      elIdx.el = (*here).first;
+      elIdx.block_ptr = ftell (stemDictFile);
+
+      // see if this block is the longest so far
+      if (stemIdx.size() > 0) {
+	unsigned long blockLen = elIdx.block_ptr -
+	  (*(stemIdx.end()-1)).block_ptr;
+	if (blockLen > sih.max_block_size) sih.max_block_size = blockLen;
+      }
+      
+      stemIdx.push_back (elIdx);
+      lastEl.erase (lastEl.begin(), lastEl.end()); // output full word
+    }
+
+    // copy the information for this stem
+    stemEl.el = (*here).first;
+    stemEl.equivWords = (*here).second;
+
+    // write out the stem
+    stemEl.Write (stemDictFile, &lastEl);
+    
+    here++; stemNum++;
+  }
+
+
+  // write out the element indexes
+  sih.num_blocks = stemIdx.size();
+  sih.block_idx_start = ftell (stemDictFile);
+  WriteBlockIdx (stemDictFile, stemIdx);
+  
+  // write out the stem dictionary header
+  fseek (stemDictFile, sizeof(unsigned long), SEEK_SET);
+  sih.Write (stemDictFile);
+
+
+  // close open files
+  fclose (stemDictFile);
+  
+  // print out information
+#ifndef SILENT
+  Message ("Num word stems = %d\n", sih.dict_size);
+  Message ("Max stem block size = %d\n", sih.max_block_size);
+  Message ("Number of stem blocks written = %d\n", sih.num_blocks);
+#endif
+}
+
+
+int main (int argc, char **argv) {
+  unsigned long entriesPerBlock = 16;
+  char *filename = "";
+  int ch;
+  int stemMethod = 0; // illegal value (no translation)
+  int stemmerNum = 0; // English stemmer
+  msg_prefix = argv[0];
+  opterr = 0;
+
+  while ((ch = getopt (argc, argv, "f:d:b:s:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'b':
+      entriesPerBlock = atoi (optarg);
+      break;
+    case 's':
+      stemMethod = atoi (optarg);
+      break;
+    case 'a':
+      stemmerNum = stemmernumber ((unsigned char *) optarg);
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-d directory] "
+	       "[-b entries-per-block] [-h] -s 1|2|3 "
+	       "[-a stemmer-method] -f name\n", argv[0]);
+      exit (1);
+    }
+  }
+  
+  if (stemMethod < 1 || stemMethod > 3)
+    FatalError (1, "Stem method must be 1, 2 or 3");
+
+  // read in the dictionary and create the in memory dictionary
+  StemMapDict stemDict;
+  CreateStemDict (filename, stemDict, stemMethod, stemmerNum);
+  
+  // write out the dictionary as a blocked file
+  WriteStemDict (filename, stemDict, stemMethod, stemmerNum, entriesPerBlock);
+  
+  return 0;
+}
Index: /trunk/mgpp/text/mgpp_weights_build.1
===================================================================
--- /trunk/mgpp/text/mgpp_weights_build.1	(revision 3365)
+++ /trunk/mgpp/text/mgpp_weights_build.1	(revision 3365)
@@ -0,0 +1,79 @@
+.\"------------------------------------------------------------
+.\" Id - set Rv,revision, and Dt, Date using rcs-Id tag.
+.de Id
+.ds Rv \\$3
+.ds Dt \\$4
+..
+.\"------------------------------------------------------------
+.TH mgpp_weights_build 1 \*(Dt CITRI
+.SH NAME
+mgpp_weights_build \- builds the approximate weights file
+.SH SYNOPSIS
+.B mgpp_weights_build
+[
+.B \-h
+]
+[
+.BI \-b " num-bits"
+]
+.if n .ti +17n
+[
+.BI \-d " directory"
+]
+.BI \-f " name"
+.SH DESCRIPTION
+.B mgpp_weights_build
+is the program that generates the exact and approximate weights 
+files.  To do this, it uses the inverted file
+and dictionary.
+.SH OPTIONS
+Options may appear in any order.
+.TP "\w'\fB\-d\fP \fIirectory\fP'u+2n"
+.B \-h
+This displays a usage line on
+.IR stderr .
+.TP
+.BI \-b " num-bits"
+This specifies the number of bits of precision that should be used for
+the approximate weights.  If not specified, it defaults to 8.
+.TP
+.BI \-d " directory"
+This specifies the directory where the document collection is.
+.TP
+.BI \-f " name"
+This specifies the base name of the document collection.
+.SH ENVIRONMENT
+.TP "\w'\fBMGDATA\fP'u+2n"
+.SB MGDATA
+If this environment variable exists, then its value is used as the
+default directory where the mgpp
+collection files are.  If this variable does not exist, then the
+directory \*(lq\fB.\fP\*(rq is used by default.  The command line
+option
+.BI \-d " directory"
+overrides the directory in
+.BR MGDATA .
+.SH FILES
+.TP 20
+.B *.invf
+Inverted file.
+.TP
+.B *.invf.dict
+Compressed stemmed dictionary.
+.TP
+.B *.text.idx
+Index into the compressed documents.
+.TP
+.B *.weight
+The exact weights file.
+.TP
+.B *.weight.approx
+Approximate document weights.
+.SH "SEE ALSO"
+.na
+.BR mgpp_compression_dict (1),
+.BR mgpp_fast_comp_dict (1),
+.BR mgpp_invf_dict (1),
+.BR mgpp_passes (1),
+.BR mgpp_perf_hash_build (1),
+.BR mgpp_stem_idx (1)
Index: /trunk/mgpp/text/mgpp_weights_build.cpp
===================================================================
--- /trunk/mgpp/text/mgpp_weights_build.cpp	(revision 3365)
+++ /trunk/mgpp/text/mgpp_weights_build.cpp	(revision 3365)
@@ -0,0 +1,399 @@
+/**************************************************************************
+ *
+ * mgpp_weights_build.cpp -- Program to build the document weights file
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#define _XOPEN_SOURCE 1
+#define _XOPEN_SOURCE_EXTENDED 1
+
+/* getopt is in posix.2, so cygwin should have it in unistd, but doesn't */
+#if defined (__WIN32__) || defined (__CYGWIN__)
+# include "getopt_old.h"
+#else
+# include <unistd.h>
+#endif
+
+#include "UCArray.h"
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "bitio_gen.h"
+#include "bitio_m_stdio.h"
+#include "mg_files.h"
+#include "locallib.h"
+#include "invf.h"
+#include "FIvfLevelInfo.h"
+#include "FragLevelConvert.h"
+
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\map>
+#elif defined(GSDL_USE_STL_H)
+#  include <map.h>
+#else
+#  include <map>
+#endif
+
+#define MAXBITS (sizeof(unsigned long) * 8)
+
+struct WBTagPtr {
+  unsigned long tagNum;
+  unsigned long tagPtr;
+  unsigned long fragOccur;
+  
+  WBTagPtr () {
+    tagNum = 0;
+    tagPtr = 0;
+    fragOccur = 0;
+  }
+};
+
+// maps tags to tag information
+typedef map<UCArray, WBTagPtr, DictLTUCArray> WBTagDict;
+
+typedef vector<float> Weights;
+
+
+static void ReadTagDict (const invf_dict_header &idh,
+			 FILE *dictFile,
+			 FILE *invfIdxFile,
+			 WBTagDict &tagDict) {
+  tagDict.erase (tagDict.begin(), tagDict.end());
+
+  // seek to the start of the tag information
+  fseek (dictFile, idh.tag_dict_start, SEEK_SET);
+  fseek (invfIdxFile, sizeof(unsigned long) +
+	 idh.word_dict_size*sizeof(unsigned long), SEEK_SET);
+  
+  unsigned long tagNum;
+  unsigned long tagPtr;
+  dict_el thisEl;
+  for (tagNum = 0; tagNum < idh.tag_dict_size; tagNum++) {
+    thisEl.Read (dictFile);
+    ReadUL (invfIdxFile, tagPtr);
+    tagDict[thisEl.el].tagNum = tagNum;
+    tagDict[thisEl.el].tagPtr = tagPtr;
+    tagDict[thisEl.el].fragOccur = thisEl.frag_occur;
+  }
+}
+
+static void AddWeight (Weights &w,
+		       unsigned long levelDocNum,
+		       unsigned long termFreq,
+		       float idf) {
+  double weight = termFreq * idf;
+  w[levelDocNum-1] += weight * weight;
+}
+
+static void GenerateLevelWeights (const invf_dict_header &idh,
+				  const invf_file_header &ifh,
+				  unsigned long numLevelDocs,
+				  unsigned long levelNum,
+				  FILE *dictFile,
+				  FILE *invfFile,
+				  FILE *invfIdxFile,
+				  const FragLevelConvert &fragLevelConvert,
+				  Weights &w) {
+  // pre-allocate the right number of weights
+  w.erase (w.begin(), w.end());
+  w.insert (w.end(), (Weights::size_type)numLevelDocs, (float)0.0);
+
+  double logN = log ((double) numLevelDocs);
+
+  // reset the files
+  fseek (dictFile, idh.word_dict_start, SEEK_SET);
+  fseek (invfIdxFile, sizeof (unsigned long), SEEK_SET);
+  
+  // process each word adding its contributions to the document weights
+  unsigned long wordNum;
+  unsigned long wordStart;
+  word_dict_el wordEl;
+  wordEl.SetNumLevels (idh.num_levels);
+  for (wordNum=0; wordNum<idh.word_dict_size; wordNum++) {
+#ifndef SILENT
+    // give a little feedback every 4096 words
+    if ((wordNum & 0xfff) == 0) fprintf (stderr, ".");
+#endif
+    
+    wordEl.Read (dictFile, idh.num_levels);
+    ReadUL (invfIdxFile, wordStart);
+
+    float idf = logN - log ((double) wordEl.levelFreqs[levelNum]);
+
+    // seek to the appropriate place in the inverted file
+    fseek (invfFile, wordStart, SEEK_SET);
+    stdio_bitio_buffer buffer (invfFile);
+    
+    unsigned long B = BIO_Bblock_Init (idh.num_frags, wordEl.frag_occur);
+    unsigned long fragNum = 0;
+    unsigned long levelDocNum = 0;
+    unsigned long lastLevelDocNum = 0;
+    unsigned long termFreq = 0;
+    unsigned long checkLevelFreq = 0;
+    
+    unsigned long count, i;
+    for (i=0; i<wordEl.frag_occur; i++) {
+      fragNum += buffer.bblock_decode (B, NULL);
+      if (!ifh.word_level_index) count = buffer.gamma_decode (NULL);
+      else count = 1;
+
+      if (fragNum > idh.num_frags)
+	FatalError (1, "fragNum = %d, "
+		    "number of fragments = %d\n"
+		    "wordNum = %d\n"
+		    "i = %d, frag_occur = %d\n",
+		    fragNum, idh.num_frags, wordNum, i, wordEl.frag_occur);
+
+      if (!fragLevelConvert.FragToLevel (fragNum, levelDocNum))
+	FatalError (1, "could not convert fragment number %d in level %d", fragNum, levelNum);
+
+      if (levelDocNum == 0 || levelDocNum > numLevelDocs)
+	FatalError (1, "bad level document number %d in level %d", levelDocNum, levelNum);
+      
+      if (levelDocNum != lastLevelDocNum) {
+	// new level document
+	if (lastLevelDocNum > 0) {
+	  AddWeight (w, lastLevelDocNum, termFreq, idf);
+	  checkLevelFreq++;
+	}
+	lastLevelDocNum = levelDocNum;
+	termFreq = 0;
+      }
+      termFreq += count;
+    }
+
+    if (lastLevelDocNum > 0) {
+      AddWeight (w, lastLevelDocNum, termFreq, idf);
+      checkLevelFreq++;
+    }
+
+    if (checkLevelFreq != wordEl.levelFreqs[levelNum]) {
+      cerr << "bad level freq at level " <<levelNum<<" "<< checkLevelFreq << " != "
+	   << wordEl.levelFreqs[levelNum] << ", word \""
+	   << wordEl.el << "\" (" << wordNum << ")\n";
+      exit (1);
+    }
+    
+    buffer.done();
+  }
+
+  if (w.size() != numLevelDocs) 
+    FatalError (1, "wrong number of weights created %d != %d",
+		w.size(), numLevelDocs);
+}
+
+static void WriteExactWeights (FILE *weightsFile,
+			       unsigned long &diskPtr,
+			       const Weights &w) {
+  diskPtr = ftell(weightsFile);
+
+  Weights::const_iterator here = w.begin();
+  Weights::const_iterator end = w.end();
+  while (here != end) {
+//      cout << *here << "\n";
+    WriteF (weightsFile, sqrt (*here));
+    here++;
+  }
+}
+
+static void WriteApproxWeights (FILE *approxWeightsFile,
+				unsigned long &diskPtr,
+				const Weights &w,
+				unsigned char bits) {
+  diskPtr = ftell(approxWeightsFile);
+
+  // calculate L, U and B
+  double L = 1e300;
+  double U = 0;
+  float wgt;
+  Weights::const_iterator here = w.begin();
+  Weights::const_iterator end = w.end();
+  while (here != end) {
+    wgt = sqrt (*here);
+    if (wgt > U) U = wgt;
+    if (wgt > 0 && wgt < L) L = wgt;
+    here++;
+  }
+
+  double B = pow (U / L, pow (2.0, -(double) bits));
+
+#ifndef SILENT
+  fprintf (stderr, "L = %f\n", L);
+  fprintf (stderr, "U = %f\n", U);
+  fprintf (stderr, "B = %f\n", B);
+#endif
+  
+  WriteUC (approxWeightsFile, bits);
+  WriteD (approxWeightsFile, L);
+  WriteD (approxWeightsFile, B);
+  
+
+  unsigned long max = (bits == 32) ? 0xffffffff : (1 << bits) - 1;;
+  unsigned long i=0, buf=0, pos=0;
+  here = w.begin();
+  end = w.end();
+  while (here != end) {
+    unsigned long fx;
+    wgt = sqrt (*here);
+    if (wgt == 0) {
+      wgt = L;
+#ifndef SILENT
+      Message ("Warning: Document %d had a weight of 0.", i);
+#endif
+    }
+    fx = (unsigned long) floor (log (wgt / L) / log (B));
+
+    if (fx > max) fx = max;
+
+    buf |= (fx << pos);
+    pos += bits;
+
+    if (pos >= MAXBITS) {
+      WriteUL (approxWeightsFile, buf);
+      buf = fx >> (bits - (pos - MAXBITS));
+      pos = pos - MAXBITS;
+    }
+
+    here++; i++;
+  }
+
+  // write out the last bits
+  if (pos > 0) WriteUL (approxWeightsFile, buf);
+}
+
+int main (int argc, char **argv) {
+  unsigned char bits = 8;
+  char *filename = "";
+  int ch;
+  opterr = 0;
+  msg_prefix = argv[0];
+
+  while ((ch = getopt (argc, argv, "f:d:b:h")) != -1) {
+    switch (ch) {
+    case 'f':		// input file
+      filename = optarg;
+      break;
+    case 'd':
+      set_basepath (optarg);
+      break;
+    case 'b':
+      bits = atoi (optarg);
+      if (bits > 32) {
+	fprintf (stderr, "b may only take values 0-32\n");
+	exit (1);
+      }
+      break;
+    case 'h':
+    case '?':
+      fprintf (stderr, "usage: %s [-f input_file]"
+	       "[-d data directory] [-b bits] [-h]\n", argv[0]);
+      exit (1);
+    }
+  }
+
+  
+  // open the dictionary
+  FILE *dictFile = open_file (filename, INVF_DICT_SUFFIX, "rb",
+			      MAGIC_STEM_BUILD, MG_ABORT);
+  invf_dict_header idh;
+  idh.Read (dictFile);
+
+  // open the inverted file
+  FILE *invfFile = open_file (filename, INVF_SUFFIX, "rb",
+			MAGIC_INVF, MG_ABORT);
+  invf_file_header ifh;
+  ifh.Read (invfFile);
+  if (ifh.skip_mode != SKIP_MODE_NO_SKIPS)
+    FatalError (1, "The invf file contains skips. Unable to create weights.");
+
+  // open the inverted index file
+  FILE *invfIdxFile = open_file (filename, INVF_IDX_SUFFIX, "rb",
+				 MAGIC_INVI, MG_ABORT);
+  
+  // read the level information
+  FILE *levelFile = open_file (filename, INVF_LEVEL_SUFFIX, "rb",
+			       MAGIC_INVF_LEVELS, MG_ABORT);
+  FIvfLevel ivfLevel;
+  ivfLevel.Read (levelFile);
+  fclose (levelFile);
+
+  // read in the tag dictionary and inverted file pointers
+  WBTagDict tagDict;
+  ReadTagDict (idh, dictFile, invfIdxFile, tagDict);
+
+  
+  // create the weights file
+  FILE *weightsFile = create_file (filename, WEIGHTS_SUFFIX, "wb",
+				   MAGIC_WGHT, MG_ABORT);
+
+  // create the approx weights file
+  FILE *approxWeightsFile = create_file (filename, APPROX_WEIGHTS_SUFFIX, "wb",
+					 MAGIC_WGHT_APPROX, MG_ABORT);
+
+  
+  // create weights for each document level
+  FragLevelConvert fragLevelConvert;
+  Weights w;
+  IvfLevelInfoMap::iterator levelHere = ivfLevel.levelInfo.begin();
+  IvfLevelInfoMap::iterator levelEnd = ivfLevel.levelInfo.end();
+  unsigned long levelNum = 0;
+  while (levelHere != levelEnd) {
+    const UCArray &levelName = (*levelHere).first;
+    
+    // read the tag information about this level
+    fragLevelConvert.Read(invfFile, tagDict[levelName].tagPtr,
+			  idh.num_frags, tagDict[levelName].fragOccur);
+    
+    // create the weights for this level
+    GenerateLevelWeights (idh, ifh, tagDict[levelName].fragOccur,
+			  levelNum, dictFile, invfFile, invfIdxFile,
+			  fragLevelConvert, w);
+
+    // write out the exact weights
+    WriteExactWeights (weightsFile,
+		       (*levelHere).second.exactWeightsDiskPtr,
+		       w);
+    
+    // write out the approximate weights
+    WriteApproxWeights (approxWeightsFile,
+			(*levelHere).second.approxWeightsDiskPtr,
+			w, bits);
+    
+    levelHere++; levelNum++;
+  }
+  
+
+  // close input files
+  fclose (dictFile);
+  fclose (invfFile);
+  fclose (invfIdxFile);
+
+  // update the level information
+  levelFile = create_file (filename, INVF_LEVEL_SUFFIX, "wb",
+			   MAGIC_INVF_LEVELS, MG_ABORT);
+  ivfLevel.Write (levelFile);
+  fclose (levelFile);
+
+  // close output files
+  fclose (weightsFile);
+  fclose (approxWeightsFile);
+
+
+  return 0;
+}
Index: /trunk/mgpp/text/stemmer.cpp
===================================================================
--- /trunk/mgpp/text/stemmer.cpp	(revision 3365)
+++ /trunk/mgpp/text/stemmer.cpp	(revision 3365)
@@ -0,0 +1,105 @@
+/**************************************************************************
+ *
+ * stemmer.cpp -- The stemmer/case folder
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+#include "stemmer.h"
+
+#include "lovinstem.h"
+#include "simplefrenchstem.h"
+#include "unitool.h"
+
+#define LOVINSTEMMER        0
+#define SIMPLEFRENCHSTEMMER 1
+
+
+/* decode the utf-8 encoded unicode, casefold and then recode 
+ * making sure the final length doesn't exceed the original
+ * length */
+static void unicode_casefold (u_char *word) {
+  unsigned short out[256]; /* temp space */
+  int i;
+  int len;
+
+  /* decode */
+  utf8_word_to_unicode (word, out, 255);
+  len = out[0];
+
+  /* casefold and simplify-fold */
+  for (i=0; i<len; i++) {
+    out[i+1] = unicode_tosimplified(unicode_tolower(out[i+1]));
+  }
+
+  /* re-code */
+  unicode_to_utf8_word (out, word, word[0]+1);
+}
+
+
+int stemmernumber (u_char *stemmerdescription) {
+  u_char descript[MAX_STEM_DESCRIPTION_LEN];
+  int i;
+
+  /* copy and case-fold the description */
+  for (i=0; (stemmerdescription[i] != '\0') && 
+	 (i < MAX_STEM_DESCRIPTION_LEN-1); i++)
+    descript[i] = tolower (stemmerdescription[i]);
+  descript[i] = '\0';
+
+  /* map the description to its number */
+
+  if ((strcmp ((char *) descript,  "0") == 0) ||
+      (strcmp ((char *) descript, "english") == 0) ||
+      (strcmp ((char *) descript, "lovin") == 0))
+    return LOVINSTEMMER;
+
+  if ((strcmp ((char *) descript, "1") == 1) ||
+      (strcmp ((char *) descript, "french") == 0) ||
+      (strcmp ((char *) descript, "simplefrench") == 0))
+    return SIMPLEFRENCHSTEMMER;
+
+  return -1;
+}
+
+
+
+/*
+ * Method 0 - Do not stem or case fold.
+ * Method 1 - Case fold.
+ * Method 2 - Stem.
+ * Method 3 - Case fold and stem.
+ *
+ * The stemmer number should be obtained using
+ * the stemmernumber function above.
+ */
+void 
+stemmer (int method, int stemmer, u_char *word) {
+  if (method & 1) {
+    unicode_casefold (word);
+  }
+
+  if (method & 2) {
+    switch (stemmer) {
+    case LOVINSTEMMER: lovinstem (word);
+      break;
+    case SIMPLEFRENCHSTEMMER: simplefrenchstem (word);
+      break;
+    }
+  }
+}
Index: /trunk/mgpp/text/stemmer.h
===================================================================
--- /trunk/mgpp/text/stemmer.h	(revision 3365)
+++ /trunk/mgpp/text/stemmer.h	(revision 3365)
@@ -0,0 +1,65 @@
+/**************************************************************************
+ *
+ * stemmer.h -- The stemmer/case folder
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef STEMMER_H
+#define STEMMER_H
+
+#include "sysfuncs.h"
+
+#define STEMMER_MASK 3
+#define MAX_STEM_DESCRIPTION_LEN 16
+
+/* stemmernumber will return the stemmer for
+ * a description of the stemmer. Stemmer descriptions
+ * are not case sensitive. Valid descriptions are:
+ *
+ *   'English'
+ *   'Lovin'
+ *   'French'
+ *   'SimpleFrench'
+ *
+ * More than one description might result in the same
+ * stemmer number (for example, for stemming 'English'
+ * we currently use the 'Lovin' stemmer).
+ *
+ * stemmerdescription is a normal C, null-terminated,
+ * string.
+ *
+ * stemmernumber will return -1 if it doesn't know the
+ * stemmer description.
+ */
+int stemmernumber (u_char *stemmerdescription);
+
+/*
+ * Method 0 - Do not stem or case fold.
+ * Method 1 - Case fold.
+ * Method 2 - Stem.
+ * Method 3 - Case fold and stem.
+ *
+ * The stemmer number should be obtained using function
+ * stemmernumber above.
+ */
+#ifdef __cplusplus
+extern "C" 
+#endif
+void stemmer (int method, int stemmer, u_char * word);
+
+#endif
Index: /trunk/mgpp/text/text.cpp
===================================================================
--- /trunk/mgpp/text/text.cpp	(revision 3365)
+++ /trunk/mgpp/text/text.cpp	(revision 3365)
@@ -0,0 +1,146 @@
+/**************************************************************************
+ *
+ * text.h -- Header file for compression related stuff
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// UCArray currently contains all the basic reading and writing functions
+#include "UCArray.h"
+#include "text.h"
+
+compressed_text_header::compressed_text_header () {
+  Clear();
+}
+
+void compressed_text_header::Clear () {
+  num_of_docs = 0;
+  num_of_words = 0;
+  num_of_bytes = 0.0;
+}
+
+
+bool compressed_text_header::Read (FILE *f) {
+  return (ReadUL (f, num_of_docs) &&
+	  ReadUL (f, num_of_words) &&
+	  ReadD (f, num_of_bytes));
+}
+
+bool compressed_text_header::Write (FILE *f) const {
+  return (WriteUL (f, num_of_docs) &&
+	  WriteUL (f, num_of_words) &&
+	  WriteD (f, num_of_bytes));
+}
+
+
+auxiliary_dict::auxiliary_dict () {
+  word_data[0] = NULL;
+  word_data[1] = NULL;
+  words[0] = NULL;
+  words[1] = NULL;
+}
+
+auxiliary_dict::~auxiliary_dict () {
+  Clear ();
+}
+
+void auxiliary_dict::Clear () {
+  if (word_data[0] != NULL) {
+    delete word_data[0];
+    word_data[0] = NULL;
+  }
+  if (word_data[1] != NULL) {
+    delete word_data[1];
+    word_data[1] = NULL;
+  }
+  if (words[0] != NULL) {
+    delete words[0];
+    words[0] = NULL;
+  }
+  if (words[1] != NULL) {
+    delete words[1];
+    words[1] = NULL;
+  }
+}
+
+
+compression_dict::compression_dict () {
+  int which;
+  for (which = 0; which < 2; which++) {
+    cfh[which] = NULL;
+    values[which] = NULL;
+    escape[which] = NULL;
+    chars_huff[which] = NULL;
+    chars_vals[which] = NULL;
+    lens_huff[which] = NULL;
+    lens_vals[which] = NULL;
+  }
+
+  MemForCompDict = 0;
+  ad = NULL;
+  fast_loaded = 0;
+}
+
+compression_dict::~compression_dict () {
+  Clear ();
+}
+
+void compression_dict::Clear () {
+  int which;
+  for (which = 0; which < 2; which++) {
+    if (cfh[which] != NULL) {
+      delete cfh[which];
+      cfh[which] = NULL;
+    }
+    if (values[which] != NULL) {
+      delete values[which][0][0];
+      delete values[which][0];
+      delete values[which];
+      values[which] = NULL;
+    }
+    if (escape[which] != NULL) { // don't know if needed
+      //delete escape[which]; // this was causing a seg fault
+      escape[which] = NULL;
+    }
+    if (chars_huff[which] != NULL) {
+      delete chars_huff[which];
+      chars_huff[which] = NULL;
+    }
+    if (chars_vals[which] != NULL) {
+      delete chars_vals[which][0];
+      delete chars_vals[which];
+      chars_vals[which] = NULL;
+    }
+    if (lens_huff[which] != NULL) {
+      delete lens_huff[which];
+      lens_huff[which] = NULL;
+    }
+    if (lens_vals[which] != NULL) {
+      delete lens_vals[which][0];
+      delete lens_vals[which];
+      lens_vals[which] = NULL;
+    }
+  }
+
+  if (ad != NULL) {
+    delete ad;
+    ad = NULL;
+  }
+
+  MemForCompDict = 0;
+  fast_loaded = 0;
+}
Index: /trunk/mgpp/text/text.h
===================================================================
--- /trunk/mgpp/text/text.h	(revision 3365)
+++ /trunk/mgpp/text/text.h	(revision 3365)
@@ -0,0 +1,199 @@
+/**************************************************************************
+ *
+ * text.h -- Header file for compression related stuff
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#ifndef H_TEXT
+#define H_TEXT
+
+// some elderly windows compilers need to include <vector> (or something
+// similar) here so that they know what a "bool" is
+#if defined(GSDL_USE_OBJECTSPACE)
+#  include <ospace\std\vector>
+#elif defined(GSDL_USE_STL_H)
+#  include <vector.h>
+#else
+#  include <vector>
+#endif
+
+
+#include "huffman.h"
+#include <stdio.h>
+
+/*****************************************************************************
+ *
+ * There are several different methods of compressing the text of the
+ * database. The following defines list the different methods of text
+ * compression
+ *
+ */
+
+
+/* The dictionary contains all the fragments that occur in the collection
+   i.e. escapes are not possible */
+#define MG_COMPLETE_DICTIONARY 0
+
+
+/* Certain words have been deleted from the dictionary. The words deleted
+   have been used to create the frequency huffman codes of the characters. 
+   This dictionary has an escape code and may be used to compress novel
+   words. This dictionary may fail if there is a novel character. */
+#define MG_PARTIAL_DICTIONARY 1
+
+
+/* This dictionary has an escape so that novel words and non-words can be 
+   coded. The method for coding the novel words and non-words is determined
+   by a dictionary parameter. */
+#define MG_SEED_DICTIONARY 2
+
+
+
+
+
+
+
+
+
+/*****************************************************************************
+ *
+ * With a seed dictionary there are several methods for coding the novel
+ * words and non-words the following defined values specify the different
+ * methods of coding.
+ *
+ */
+
+
+/* Code novel words and non-words character by character using huffman codes.
+   The huffman codes for the word and non-word lengths and characters are 
+   generated from the distribution of lengths and characters in the
+   dictionary. */
+#define MG_NOVEL_HUFFMAN_CHARS 0
+
+
+/* This method codes novel words using delta codes. The novel words are stored
+   in a auxillary dictionary which is built by pass two. */
+#define MG_NOVEL_DELTA 2
+
+
+/* This method codes novel words using hybrid version of delta. The novel
+   words are stored in a auxillary dictionary which is built by pass two. */
+#define MG_NOVEL_HYBRID 3
+
+
+
+
+
+
+/* This specified an amount of extra space allocated in the
+   compression_dict_header for adding new parameters. As new
+   parameters are added this should be decreased. */
+#define TEXT_PARAMS 15
+
+
+
+struct compression_dict_header {
+  u_long dict_type;
+  u_long novel_method;
+  u_long params[TEXT_PARAMS];
+  u_long num_words[2];
+  u_long num_word_chars[2];
+  u_long lookback;
+};
+
+
+struct comp_frags_header {
+  huff_data hd;
+  u_long uncompressed_size;
+  u_long huff_words_size[MAX_HUFFCODE_LEN + 1];
+};
+
+// BOGUSTEXTLEN is used to replace the ratio
+// in the compressed_text_header while preparing
+// for UCArray
+#define BOGUSTEXTLEN 1000000
+
+struct compressed_text_header {
+  u_long num_of_docs;
+  u_long num_of_words; // number of words in collection
+  double num_of_bytes;
+
+  compressed_text_header ();
+  void Clear ();
+
+  // you must seek to the appropriate place before calling
+  // Read or Write
+  bool Read (FILE *f);
+  bool Write (FILE *f) const;
+};
+
+
+struct compression_stats_header {
+  u_long num_docs;
+  u_long dummy;     // alignment
+  double num_bytes; 
+};
+
+
+struct frags_stats_header {
+  u_long num_frags;
+  u_long mem_for_frags;
+};
+
+
+struct aux_frags_header {
+  u_long num_frags;
+  u_long mem_for_frags;
+};
+
+
+struct auxiliary_dict {
+  aux_frags_header afh[2];
+  u_char *word_data[2];
+  u_char **words[2];
+  int blk_start[2][33], blk_end[2][33];	/* blk_start and blk_end are required
+					   for the hybrid methods */
+
+  auxiliary_dict ();
+  ~auxiliary_dict ();
+
+  void Clear ();
+};
+
+
+struct compression_dict {
+  compression_dict_header cdh;
+  comp_frags_header *cfh[2];
+  unsigned long MemForCompDict;
+  u_char ***values[2];
+  u_char *escape[2];
+  huff_data *chars_huff[2];
+  u_long **chars_vals[2];
+  huff_data *lens_huff[2];
+  u_long **lens_vals[2];
+  auxiliary_dict *ad;
+  int fast_loaded;
+
+  compression_dict ();
+  ~compression_dict ();
+
+  void Clear ();
+};
+
+
+#endif
Index: /trunk/mgpp/text/text.pass1.cpp
===================================================================
--- /trunk/mgpp/text/text.pass1.cpp	(revision 3365)
+++ /trunk/mgpp/text/text.pass1.cpp	(revision 3365)
@@ -0,0 +1,387 @@
+/**************************************************************************
+ *
+ * text.pass1.cpp -- Text compression (Pass 1)
+ * Copyright (C) 1994  Neil Sharman, Gary Eddy and Alistair Moffat
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "huffman.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+#include "mg_files.h"
+#include "mg.h"
+#include "build.h"
+#include "locallib.h"
+#include "words.h"
+#include "text.h"
+#include "hash.h"
+#include "local_strings.h"
+#include "TextEl.h"
+
+/* for more meaningful messages - jrm21 (gsdl) */
+#ifdef HAVE_STRERROR
+#include <string.h>
+#include <errno.h>
+#endif
+
+
+#define POOL_SIZE 1024*1024
+#define INITIAL_HASH_SIZE 7927
+
+
+typedef struct hash_rec
+  {
+    unsigned long wcnt;		/* word frequency */
+    unsigned long occurance_num;
+    u_char *word;
+  }
+hash_rec;
+
+typedef struct dict_data
+  {
+    hash_rec *HashTable;
+    unsigned long HashSize;
+    unsigned long HashUsed;
+    unsigned long wordnum;
+    unsigned long words_read;
+    unsigned long bytes_diff;
+    huff_data hd;
+  }
+dict_data;
+
+
+
+static unsigned long longestDoc = 0;
+static unsigned long occurance_num = 0;
+static dict_data DictData[2];
+
+static u_char *Pool;
+static int PoolLeft;
+static double inputbytes = 0; /* [RJM 07/97: 4G limit] */
+static unsigned long MaxMemInUse = 0;
+static unsigned long MemInUse = 0;
+static compression_stats_header csh = {0, 0, 0.0}; /* [RJM 07/97: 4G limit] */
+
+
+static void ChangeMem (int Change) {
+  MemInUse += Change;
+  if (MemInUse > MaxMemInUse) MaxMemInUse = MemInUse;
+}
+
+
+int init_text_1 (const TagInfo &/*tagInfo*/, char * /*FileName*/) {
+  int which;
+  
+  if (!(Pool = (u_char *) Xmalloc (POOL_SIZE))) {
+    Message ("Unable to allocate memory for pool");
+    return (COMPERROR);
+  }
+  PoolLeft = POOL_SIZE;
+  ChangeMem (POOL_SIZE);
+  
+  for (which = 1; which >= 0; which--) {
+    u_char *word;
+    hash_rec *ent;
+    dict_data *dd = &DictData[which];
+    
+    dd->wordnum = 0;
+    dd->words_read = 0;
+    dd->bytes_diff = 0;
+    dd->HashSize = INITIAL_HASH_SIZE;
+    dd->HashUsed = 0;
+    
+    if (!(dd->HashTable = (hash_rec *) Xmalloc (sizeof (hash_rec) * dd->HashSize))) {
+      Message ("Unable to allocate memory for table");
+      return (COMPERROR);
+    }
+    ChangeMem (sizeof (hash_rec) * dd->HashSize);
+    memset (dd->HashTable,'\0' , sizeof (hash_rec) * dd->HashSize);
+
+    word = Pool;
+    *Pool++ = '\0';
+    PoolLeft--;
+    {
+      register u_char *wptr;
+      register int hsize = dd->HashSize;
+      register unsigned long hashval, step;
+      
+      HASH (hashval, step, word, hsize);
+      wptr = (dd->HashTable + hashval)->word;
+      while (wptr) {
+	hashval += step;
+	if (hashval >= (unsigned long)hsize)
+	  hashval -= hsize;
+	wptr = (dd->HashTable + hashval)->word;
+      }
+      ent = dd->HashTable + hashval;
+    }
+    ent->wcnt = 1;
+    ent->word = word;
+    dd->HashUsed = 1;
+  }
+  return (COMPALLOK);
+}
+
+
+static int process_text_element (const u_char *s_in, int l_in) {
+  const u_char *end = s_in + l_in - 1;
+  
+   /*
+   ** Alternately parse off words and non-words from the input
+   ** stream beginning with a non-word. Each token is then
+   ** inserted into the set if it does not exist or has it's
+   ** frequency count incremented if it does.
+   */
+  
+  bool which = false; // non-word
+  for (; s_in <= end; which = !which) {
+    u_char Word[MAXWORDLEN + 1];
+    dict_data *dd = &DictData[which];
+    
+    /* First parse a word or non-word out of the string */
+    if (which) PARSE_WORD (Word, s_in, end);
+    else PARSE_NON_WORD (Word, s_in, end);
+    
+    dd->wordnum++;
+    inputbytes += *Word;
+    dd->words_read++;
+    
+    /* Search the hash table for Word */
+    {
+      register unsigned long hashval, step;
+      register int hsize = dd->HashSize;
+      HASH (hashval, step, Word, hsize);
+      for (;;) {
+	register u_char *s1;
+	register u_char *s2;
+	register int len;
+	register hash_rec *ent;
+	ent = dd->HashTable + hashval;
+	if (!ent->word) {
+	  int len = *Word + 1;
+	  if (len > PoolLeft) {
+	    if (!(Pool = (u_char *) Xmalloc (POOL_SIZE))) {
+	      Message ("Unable to allocate memory for pool");
+	      return (COMPERROR);
+	    }
+	    PoolLeft = POOL_SIZE;
+	    ChangeMem (POOL_SIZE);
+	  }
+	  ent->occurance_num = occurance_num++;
+	  ent->wcnt = 1;
+	  ent->word = Pool;
+	  memcpy (Pool, Word, len);
+	  Pool += len;
+	  PoolLeft -= len;
+	  dd->HashUsed++;
+	  dd->bytes_diff += Word[0];
+	  break;
+	}
+	
+	/* Compare the words */
+	s1 = Word;
+	s2 = ent->word;
+	len = *s1 + 1;
+	for (; len; len--)
+	  if (*s1++ != *s2++) break;
+	
+	if (len) {
+	  hashval = (hashval + step);
+	  if (hashval >= (unsigned long)hsize) hashval -= hsize;
+	} else {
+	  ent->wcnt++;
+	  break;
+	}
+      }
+    }
+
+    
+    if (dd->HashUsed >= dd->HashSize >> 1) {
+      hash_rec *ht;
+      unsigned long size;
+      unsigned long i;
+      size = prime (dd->HashSize * 2);
+      if (!(ht = (hash_rec *) Xmalloc (sizeof (hash_rec) * size))) {
+	Message ("Unable to allocate memory for table");
+	return (COMPERROR);
+      }
+      ChangeMem (sizeof (hash_rec) * size);
+      memset (ht, '\0', sizeof (hash_rec) * size);
+
+      for (i = 0; i < dd->HashSize; i++)
+	if (dd->HashTable[i].word) {
+	  register u_char *wptr;
+	  register unsigned long hashval, step;
+	  
+	  wptr = dd->HashTable[i].word;
+	  HASH (hashval, step, wptr, size);
+	  wptr = (ht + hashval)->word;
+	  while (wptr) {
+	    hashval += step;
+	    if (hashval >= size) hashval -= size;
+	    wptr = (ht + hashval)->word;
+	  }
+	  ht[hashval] = dd->HashTable[i];
+	}
+      Xfree (dd->HashTable);
+      ChangeMem (-sizeof (hash_rec) * dd->HashSize);
+      dd->HashTable = ht;
+      dd->HashSize = size;
+    }
+  }
+
+  return COMPALLOK;
+}
+
+
+int process_text_1 (const TagInfo &/*tagInfo*/, const TextElArray &doc) {
+  unsigned long textLen = 0;
+  unsigned long docLen = 0;
+  int retValue;
+
+  // process each text element in this document
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    textLen = (*here).text.size();
+    docLen += textLen;
+
+    retValue = process_text_element (&(here->text[0]), textLen);
+    if (retValue != COMPALLOK) return retValue;
+    
+    here++;
+  }
+
+  // get max document length
+  if (docLen > longestDoc) longestDoc = docLen;
+
+  // update header information
+  csh.num_docs++;
+  csh.num_bytes += docLen;
+  
+  return COMPALLOK;
+}
+
+
+static int PackHashTable (dict_data * dd) {
+  int s, d;
+  for (s = d = 0; (unsigned int)s < dd->HashSize; s++)
+    if (dd->HashTable[s].word)
+      dd->HashTable[d++] = dd->HashTable[s];
+  
+  ChangeMem (-sizeof (hash_rec) * dd->HashSize);
+  ChangeMem (sizeof (hash_rec) * dd->HashUsed);
+  
+  if (!(dd->HashTable = (hash_rec *) Xrealloc (dd->HashTable,
+			   sizeof (hash_rec) * dd->HashUsed))) {
+    Message ("Out of memory");
+    return COMPERROR;
+  }
+  dd->HashSize = dd->HashUsed;
+  return COMPALLOK;
+}
+
+
+static int ent_comp (const void *s1, const void *s2) {
+  return casecompare (((hash_rec *) s1)->word, ((hash_rec *) s2)->word);
+}
+
+
+static void WriteHashTable (FILE * fp, dict_data * dd) {
+  frags_stats_header fsh;
+  u_long j = 0;
+  u_char *curr;
+
+  if (PackHashTable (dd) == COMPERROR) return;
+  
+  qsort (dd->HashTable, dd->HashUsed, sizeof (hash_rec), ent_comp);
+  
+  fsh.num_frags = dd->HashSize;
+  fsh.mem_for_frags = dd->HashSize;
+  for (j = 0; j < dd->HashSize; j++)
+    fsh.mem_for_frags += dd->HashTable[j].word[0];
+  
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONUL(fsh.num_frags);
+  HTONUL(fsh.mem_for_frags);
+  
+  fwrite (&fsh, sizeof (fsh), 1, fp);
+  
+  for (j = 0; j < dd->HashSize; j++) {
+    curr = dd->HashTable[j].word;
+    
+    /* [RPAP - Jan 97: Endian Ordering] */
+    HTONUL(dd->HashTable[j].wcnt);
+    HTONUL(dd->HashTable[j].occurance_num);
+    
+    fwrite (&dd->HashTable[j].wcnt, sizeof (dd->HashTable[j].wcnt), 1, fp);
+    fwrite (&dd->HashTable[j].occurance_num,
+	    sizeof (dd->HashTable[j].occurance_num), 1, fp);
+    
+    /* [RPAP - Jan 97: Endian Ordering] */
+    NTOHUL(dd->HashTable[j].wcnt);
+    NTOHUL(dd->HashTable[j].occurance_num);
+    
+    fwrite (curr, sizeof (u_char), curr[0] + 1, fp);
+  }
+}
+
+
+int done_text_1 (const TagInfo &/*tagInfo*/, char *file_name) {
+  char *temp_str;
+  FILE *fp;
+  
+  if (!(fp = create_file (file_name, TEXT_STATS_DICT_SUFFIX, "wb",
+			  MAGIC_STATS_DICT, MG_MESSAGE)))  /* [RPAP - Feb 97: WIN32 Port] */
+    {
+      fprintf(stderr,"Couldn't create file %s%s:%s\n",
+              file_name, TEXT_STATS_DICT_SUFFIX,
+#if defined(HAVE_STRERROR) || defined(__WIN32__)
+              strerror(errno)
+#else
+	      " "
+#endif
+              );
+      return COMPERROR;
+    }
+
+  temp_str = msg_prefix;
+  msg_prefix = "text.pass1";
+  
+  /* [RPAP - Jan 97: Endian Ordering] */
+  HTONUL(csh.num_docs);
+  HTOND(csh.num_bytes); /* [RJM 07/97: 4G limit] */
+
+  fwrite (&csh, sizeof (csh), 1, fp);
+
+  /* [RPAP - Jan 97: Endian Ordering] */
+  NTOHUL(csh.num_docs);
+  NTOHD(csh.num_bytes); /* [RJM 07/97: 4G limit] */
+
+  WriteHashTable (fp, &DictData[0]);
+  WriteHashTable (fp, &DictData[1]);
+  msg_prefix = temp_str;
+  return COMPALLOK;
+}
Index: /trunk/mgpp/text/text.pass2.cpp
===================================================================
--- /trunk/mgpp/text/text.pass2.cpp	(revision 3365)
+++ /trunk/mgpp/text/text.pass2.cpp	(revision 3365)
@@ -0,0 +1,581 @@
+/**************************************************************************
+ *
+ * text.pass2.cpp -- Text compression (Pass 2)
+ * Copyright (C) 1994  Neil Sharman, Gary Eddy and Alistair Moffat
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#if defined (__WIN32__)
+# include <stdio.h>
+# define unlink _unlink
+#endif
+
+#include "sysfuncs.h"
+#include "memlib.h"
+#include "messages.h"
+#include "local_strings.h"
+#include "bitio_m_stdio.h"
+#include "huffman.h"
+#include "netorder.h"  /* [RPAP - Jan 97: Endian Ordering] */
+#include "mg.h"
+#include "mg_files.h"
+#include "build.h"
+#include "words.h"
+#include "text.h"
+#include "hash.h"
+#include "locallib.h"
+#include "comp_dict.h"
+#include "FText.h"
+
+#define POOL_SIZE 1024*256
+
+
+struct char_pool {
+  struct char_pool *next;
+  u_long left;
+  u_char *ptr;
+  u_char pool[POOL_SIZE];
+};
+
+struct novel_hash_rec {
+  u_long ordinal_num;
+  u_char *word;
+};
+
+
+#define INITIAL_HASH_SIZE 7927
+
+struct novel_hash_table {
+  novel_hash_rec *HashTable;
+  u_long HashSize, HashUsed;
+  char_pool *first_pool;
+  char_pool *pool;
+  u_long next_num, binary_start;
+  novel_hash_rec **code_to_nhr;
+};
+
+
+static FILE *text;
+static stdio_bitio_buffer textOutBuf;
+static u_long headLen;
+
+static novel_hash_table nht[2];
+
+int blk_start[2][33], blk_end[2][33];
+
+// 0 = non-word, 1 = word
+unsigned char whichWordType = 0;
+
+CompressTextInfoMap textInfoMap;
+
+
+static char_pool *new_pool (char_pool * pool) {
+  char_pool *p = (char_pool *) Xmalloc (sizeof (char_pool));
+  if (!p) FatalError (1, "Unable to allocate memory for pool");
+  
+  if (pool) pool->next = p;
+  p->next = NULL;
+  p->left = POOL_SIZE;
+  p->ptr = p->pool;
+  return p;
+}
+
+
+int init_text_2 (const TagInfo &/*tagInfo*/, char *fileName) {
+  char path[512];
+
+  // load the compression dictionary
+  if (LoadCompressionDictionary (make_name (fileName, TEXT_DICT_SUFFIX,
+					    path)) == COMPERROR)
+    return COMPERROR;
+
+  // create the compressed text file and attach it to the text buffer
+  if (!(text = create_file (fileName, TEXT_SUFFIX, "w+b",
+			    MAGIC_TEXT, MG_MESSAGE)))
+    return COMPERROR;
+  textOutBuf.attachFile (text);
+  textOutBuf.encodeStart ();
+  
+  // write out the compressed text header
+  memset (&cth, '\0', sizeof (cth));
+  if (fwrite (&cth, sizeof (cth), 1, text) != 1)
+    return COMPERROR;
+
+  // the length of the compressed header is the size of the
+  // magic number plus the size of the compressed text header
+  headLen = sizeof (u_long) + sizeof (cth);
+
+  // start off with non-word
+  whichWordType = 0;
+
+  // clear the text pointers
+  textInfoMap.erase (textInfoMap.begin(), textInfoMap.end());
+  
+  // set up the hash table of novel words
+  int i;
+  if (cdh.novel_method != MG_NOVEL_HUFFMAN_CHARS)
+    for (i = 0; i <= 1; i++)
+      {
+	nht[i].HashSize = INITIAL_HASH_SIZE;
+	nht[i].HashTable = (novel_hash_rec *) Xmalloc (sizeof (novel_hash_rec) * nht[i].HashSize);
+	memset (nht[i].HashTable, '\0',
+	       sizeof (novel_hash_rec) * nht[i].HashSize);
+	nht[i].HashUsed = 0;
+	nht[i].HashSize = INITIAL_HASH_SIZE;
+	nht[i].pool = nht[i].first_pool = new_pool (NULL);
+	nht[i].next_num = 1;
+	nht[i].binary_start = 1;
+	nht[i].code_to_nhr = NULL;
+	if (cdh.novel_method == MG_NOVEL_HYBRID)
+	  {
+	    int num;
+	    num = 1;
+	    blk_start[i][0] = 0;
+	    blk_end[i][0] = cdh.num_words[i] - 1;
+	    while (num < 33)
+	      {
+		blk_start[i][num] = blk_end[i][num - 1] + 1;
+		blk_end[i][num] = blk_start[i][num] +
+		  (blk_end[i][num - 1] - blk_start[i][num - 1]) * 2;
+		num++;
+	      }
+	  }
+      }
+
+  return (COMPALLOK);
+}
+
+
+static int compress_text (bitio_buffer &buffer, const u_char *s_in,
+			  int l_in, unsigned char &which,
+			  unsigned long &numWords) {
+  const u_char *end = s_in + l_in - 1;
+
+  for (; s_in <= end; which = !which) {
+    u_char Word[MAXWORDLEN + 1];
+    int res;
+    
+    if (which) numWords++;
+
+    /* First parse a word or non-word out of the string */
+    if (which) PARSE_WORD (Word, s_in, end);
+    else PARSE_NON_WORD (Word, s_in, end);
+
+    /* Search the hash table for Word */
+    if (ht[which]) {
+      register unsigned long hashval, step;
+      register int tsize = ht[which]->size;
+      register u_char **wptr;
+      HASH (hashval, step, Word, tsize);
+      for (;;) {
+	register u_char *s1;
+	register u_char *s2;
+	register int len;
+	wptr = ht[which]->table[hashval];
+	if (wptr == NULL) {
+	  res = COMPERROR;
+	  break;
+	}
+
+	/* Compare the words */
+	s1 = Word;
+	s2 = *wptr;
+	len = *s1 + 1;
+	for (; len; len--)
+	  if (*s1++ != *s2++) break;
+	
+	if (len) {
+	  hashval += step;
+	  if (hashval >= (unsigned long)tsize) hashval -= tsize;
+	  
+	} else {
+	  res = ht[which]->table[hashval] - ht[which]->words;
+	  break;
+	}
+      }
+    }
+    else
+      res = COMPERROR;
+    /* Check that the word was found in the dictionary */
+    if (res == COMPERROR)
+      {
+	if (cdh.dict_type == MG_COMPLETE_DICTIONARY)
+	  {
+	    Message ("Unknown word \"%.*s\"\n", *Word, Word + 1);
+	    return (COMPERROR);
+	  }
+	if (cdh.dict_type == MG_PARTIAL_DICTIONARY)
+	  {
+	    u_long i;
+	    if (ht[which])
+	      {
+		res = ht[which]->hd->num_codes - 1;
+		buffer.huff_encode (res, ht[which]->codes, ht[which]->hd->clens, NULL);
+	      }
+	    buffer.huff_encode (Word[0], lens_codes[which], lens_huff[which].clens, NULL);
+	    for (i = 0; i < Word[0]; i++)
+	      buffer.huff_encode (Word[i + 1], char_codes[which],
+				  char_huff[which].clens, NULL);
+	  }
+	if (cdh.dict_type == MG_SEED_DICTIONARY)
+	  {
+	    if (ht[which])
+	      {
+		res = ht[which]->hd->num_codes - 1;
+		buffer.huff_encode (res, ht[which]->codes, ht[which]->hd->clens, NULL);
+	      }
+	    switch (cdh.novel_method)
+	      {
+	      case MG_NOVEL_HUFFMAN_CHARS:
+		{
+		  u_long i;
+		  buffer.huff_encode (Word[0], lens_codes[which],
+				      lens_huff[which].clens, NULL);
+		  for (i = 0; i < Word[0]; i++)
+		    buffer.huff_encode (Word[i + 1], char_codes[which],
+					char_huff[which].clens, NULL);
+		}
+		break;
+	      case MG_NOVEL_DELTA:
+	      case MG_NOVEL_HYBRID:
+		{
+		  register unsigned long hashval, step;
+		  register novel_hash_table *h = &nht[which];
+		  register int hsize = h->HashSize;
+		  register novel_hash_rec *ent;
+		  HASH (hashval, step, Word, hsize);
+		  for (;;)
+		    {
+		      register u_char *s1, *s2;
+		      register int len;
+		      ent = h->HashTable + hashval;
+		      if (!ent->word)
+			{
+			  int len = *Word + 1;
+			  if ((unsigned)len > h->pool->left)
+			    h->pool = new_pool (h->pool);
+			  ent->word = h->pool->ptr;
+			  ent->ordinal_num = h->next_num++;
+			  memcpy (h->pool->ptr, Word, len);
+			  h->pool->ptr += len;
+			  h->pool->left -= len;
+			  h->HashUsed++;
+			  break;
+			}
+		      /* Compare the words */
+		      s1 = Word;
+		      s2 = ent->word;
+		      len = *s1 + 1;
+		      for (; len; len--)
+			if (*s1++ != *s2++)
+			  break;
+		      
+		      if (!len)
+			break;
+		      
+		      hashval = (hashval + step);
+		      if (hashval >= (unsigned)hsize) hashval -= hsize;
+		    }
+		  
+		  switch (cdh.novel_method) {
+		  case MG_NOVEL_DELTA:
+		    buffer.delta_encode (ent->ordinal_num, NULL);
+		    break;
+		  case MG_NOVEL_HYBRID:
+		    int k = 0;
+		    int j = ent->ordinal_num - 1;
+		    while (j > blk_end[which][k])
+		      k++;
+		    assert (j - blk_start[which][k] + 1 >= 1 &&
+			    j - blk_start[which][k] + 1 <=
+			    blk_end[which][k] - blk_start[which][k] + 1);
+		    
+		    buffer.gamma_encode (k + 1, NULL);
+		    buffer.binary_encode (j - blk_start[which][k] + 1,
+					  blk_end[which][k] - blk_start[which][k] + 1,
+					  NULL);
+		    break;
+		  }
+		  
+		  if (h->HashUsed >= h->HashSize >> 1)
+		    {
+		      novel_hash_rec *ht;
+		      unsigned long size;
+		      unsigned long i;
+		      size = prime (h->HashSize * 2);
+		      if (!(ht = (novel_hash_rec *) Xmalloc (sizeof (novel_hash_rec) * size)))
+			{
+			  Message ("Unable to allocate memory for table");
+			  return (COMPERROR);
+			}
+		      memset (ht, '\0', sizeof (novel_hash_rec) * size);
+		      
+		      for (i = 0; i < h->HashSize; i++)
+			if (h->HashTable[i].word)
+			  {
+			    register u_char *wptr;
+			    register unsigned long hashval, step;
+			    
+			    wptr = h->HashTable[i].word;
+			    HASH (hashval, step, wptr, size);
+			    wptr = (ht + hashval)->word;
+			    while (wptr)
+			      {
+				hashval += step;
+				if (hashval >= size)
+				  hashval -= size;
+				wptr = (ht + hashval)->word;
+			      }
+			    ht[hashval] = h->HashTable[i];
+			  }
+		      Xfree (h->HashTable);
+		      h->HashTable = ht;
+		      h->HashSize = size;
+		    }
+		}
+		break;
+	      }
+	  }
+      }
+    else
+      {
+	buffer.huff_encode (res, ht[which]->codes, ht[which]->hd->clens, NULL);
+      }
+  }
+  
+  return COMPALLOK;
+}
+
+
+int process_text_2 (const TagInfo &tagInfo, const TextElArray &doc) {
+  // we need to remember where the last text element ends
+  unsigned char endBit = textOutBuf.GetBtg (); // 8=high bit, 1=low bit
+  unsigned long endPos = textOutBuf.GetBytesWritten () + headLen;
+
+  // process each text element
+  TextElArray::const_iterator here = doc.begin();
+  TextElArray::const_iterator end = doc.end();
+  while (here != end) {
+    // remember current place in compressed text
+    unsigned char startWhich = whichWordType;
+    unsigned char startBit = textOutBuf.GetBtg (); // 8=high bit, 1=low bit
+    unsigned long startPos = textOutBuf.GetBytesWritten () + headLen;
+
+    // compress the text
+    if (compress_text (textOutBuf,
+		       &(here->text[0]),
+		       (*here).text.size(),
+		       whichWordType,
+		       cth.num_of_words) != COMPALLOK)
+      return COMPERROR;
+
+    endBit = textOutBuf.GetBtg (); // 8=high bit, 1=low bit
+    endPos = textOutBuf.GetBytesWritten () + headLen;
+    
+    // update the pointers if this is a level tag
+    if (tagInfo.levelTags.find ((*here).tagName) != tagInfo.levelTags.end()) {
+      if ((*here).elType == OpenTagE) {
+	// set the start of the document
+	textInfoMap[(*here).tagName].SetStart (startPos, startBit, startWhich);
+	
+      } else if ((*here).elType == CloseTagE) {
+	// set the end of the document
+	textInfoMap[(*here).tagName].SetEnd (endPos, endBit);
+	
+      } else {
+	// TextE, nothing needs to be done
+      }
+    }
+    
+    cth.num_of_bytes += (*here).text.size();
+    here++;
+  }
+
+  // close off any unclosed tags
+  CompressTextInfoMap::iterator tiHere = textInfoMap.begin ();
+  CompressTextInfoMap::iterator tiEnd = textInfoMap.end ();
+  while (tiHere != tiEnd) {
+    if ((*tiHere).second.inDoc) (*tiHere).second.SetEnd (endPos, endBit);
+    tiHere++;
+  }
+  
+  // we've processed one more document
+  cth.num_of_docs++;
+
+  return COMPALLOK;
+}
+
+
+static int write_aux_dict (char *fileName) {
+  int i;
+  FILE *aux;
+  if (!(aux = create_file (fileName, TEXT_DICT_AUX_SUFFIX, "wb",
+			   MAGIC_AUX_DICT, MG_MESSAGE)))
+    return COMPERROR;
+  
+  for (i = 0; i <= 1; i++)
+    {
+      aux_frags_header afh;
+      char_pool *cp;
+      
+      afh.num_frags = nht[i].HashUsed;
+      afh.mem_for_frags = 0;
+      for (cp = nht[i].first_pool; cp; cp = cp->next)
+	afh.mem_for_frags += POOL_SIZE - cp->left;
+      
+      /* [RPAP - Jan 97: Endian Ordering] */
+      HTONUL(afh.num_frags);
+      HTONUL(afh.mem_for_frags);
+      
+      fwrite (&afh, sizeof (afh), 1, aux);
+      
+      for (cp = nht[i].first_pool; cp; cp = cp->next)
+	fwrite (cp->pool, POOL_SIZE - cp->left, sizeof (u_char), aux);
+    }
+  fclose (aux);
+  return COMPALLOK;
+}
+
+
+static void estimate_compressed_aux_dict (void) {
+  int i;
+  u_long aux_compressed = 0, total_uncomp = 0;
+  for (i = 0; i <= 1; i++)
+    {
+      int j;
+      long chars[256], fchars[256];
+      long lens[16], flens[16];
+      char_pool *cp;
+      memset (chars, '\0', sizeof (chars));
+      memset (lens, '\0', sizeof (lens));
+      for (cp = nht[i].first_pool; cp; cp = cp->next)
+	{
+	  u_char *buf = cp->pool;
+	  while (buf != cp->ptr)
+	    {
+	      int len = *buf++;
+	      lens[len]++;
+	      total_uncomp += len + 4;
+	      for (; len; len--)
+		chars[*buf++]++;
+	    }
+	}
+      for (j = 0; j < 256; j++)
+	if (!chars[j] && PESINAWORD (j) == i)
+	  fchars[j] = 1;
+	else
+	  fchars[j] = chars[j];
+      for (j = 0; j < 16; j++)
+	if (!lens[j])
+	  flens[j] = 1;
+	else
+	  flens[j] = lens[j];
+      
+      aux_compressed += (long unsigned int) ((Calculate_Huffman_Size (16, flens, lens) +
+					      Calculate_Huffman_Size (256, fchars, chars)) / 8);
+      
+    }
+}
+
+
+
+static bool WriteTextIdx (char *fileName, FTextLevel &textLevels) {
+  // create the text index
+  FILE *textIdxFile;
+  if (!(textIdxFile = create_file (fileName, TEXT_IDX_SUFFIX, "w+b",
+				MAGIC_TEXI, MG_MESSAGE)))
+    return false;
+
+  // output each level
+  CompressTextInfoMap::iterator tiHere = textInfoMap.begin ();
+  CompressTextInfoMap::iterator tiEnd = textInfoMap.end ();
+  while (tiHere != tiEnd) {
+    // remember the level information
+    TextLevelInfo levelInfo;
+    levelInfo.levelTag = (*tiHere).first;
+    levelInfo.textIdxPtr = ftell (textIdxFile);
+    levelInfo.numEntries = (*tiHere).second.docPtrs.size();
+    textLevels.levelInfo[levelInfo.levelTag] = levelInfo;
+
+    // write out the index array for this level
+    if (!WriteTextIdxArray (textIdxFile, (*tiHere).second.docPtrs))
+      return false;
+    
+    tiHere++;
+  }
+
+  // close the file
+  fclose (textIdxFile);
+
+  return true;
+}
+
+
+bool WriteTextLevels (char *fileName, const FTextLevel &textLevels) {
+  //  cout << textLevels;
+  
+  // create the text index
+  FILE *textLevelFile;
+  if (!(textLevelFile = create_file (fileName, TEXT_LEVEL_SUFFIX, "w+b",
+				     MAGIC_TEXT_LEVELS, MG_MESSAGE)))
+    return false;
+
+  // output each level
+  if (!textLevels.Write (textLevelFile)) return false;
+  
+  // close the file
+  fclose (textLevelFile);
+
+  return true;
+}
+
+
+int done_text_2 (const TagInfo &/*tagInfo*/, char *fileName) {
+  // write out any remaining bits
+  textOutBuf.encodeDone();
+
+  // write out the compressed text header (with updated statistics)
+  if (fseek (text, sizeof (u_long), SEEK_SET) == -1 || !cth.Write (text))
+    return COMPERROR;
+  fclose (text);
+
+  // write out the text index file and get the level information
+  FTextLevel textLevels;
+  if (!WriteTextIdx (fileName, textLevels)) return COMPERROR;
+  
+  // write out the text level information
+  if (!WriteTextLevels (fileName, textLevels)) return COMPERROR;
+
+  // write out the auxiliary dictionary
+  if (cdh.dict_type != MG_COMPLETE_DICTIONARY &&
+      (cdh.novel_method == MG_NOVEL_DELTA ||
+       cdh.novel_method == MG_NOVEL_HYBRID)) {
+    if (write_aux_dict (fileName) == COMPERROR) return COMPERROR;
+    estimate_compressed_aux_dict ();
+    
+  } else {
+    unlink (make_name (fileName, TEXT_DICT_AUX_SUFFIX, NULL));
+  }
+
+  return (COMPALLOK);
+}
Index: /trunk/mgpp/text/words.cpp
===================================================================
--- /trunk/mgpp/text/words.cpp	(revision 3365)
+++ /trunk/mgpp/text/words.cpp	(revision 3365)
@@ -0,0 +1,100 @@
+/**************************************************************************
+ *
+ * words.cpp -- Functions for parsing out words from the source text
+ * Copyright (C) 1999  Rodger McNab
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+// need this to avoid bizarre compiler problems under VC++ 6.0
+#if defined (__WIN32__) && !defined (GSDL_USE_IOS_H)
+# include <iostream>
+#endif
+
+#include "words.h"
+
+
+/* Takes the place of the old INAWORD macro. It determines
+   whether a given place in a UTF-8 encoded Unicode string
+   is part of a word. */
+int inaword (const u_char *here, const u_char *end) {
+  unsigned short c;
+  if (parse_utf8_char(here, end, &c) > 0) return is_unicode_letdig (c);
+  return 0;
+}
+
+const unsigned char *ParseIndexWord (const unsigned char *textHere,
+				     const unsigned char *textEnd,
+				     UCArray &word) {
+  word.erase (word.begin(), word.end());
+
+  register int charlength = 0;
+  register int length = 0;
+  register int numeric = 0;
+  unsigned short c;
+  
+  charlength = parse_utf8_char (textHere, textEnd, &c);
+  
+  while (length+charlength <= MAXSTEMLEN && charlength > 0 &&
+	 (is_unicode_letter(c) || (is_unicode_digit(c) &&
+				   ++numeric <= MAXNUMERIC))) {
+    while (charlength-- > 0) {
+      word.push_back (*textHere++); length++;
+    }
+    charlength = parse_utf8_char (textHere, textEnd, &c);
+  }
+  
+  return textHere;
+}
+
+const unsigned char *ParseIndexMGWord (const unsigned char *textHere,
+				       const unsigned char *textEnd,
+				       unsigned char *mgWord) {
+  register int charlength = 0;
+  register int length = 0;
+  register int numeric = 0;
+  unsigned short c;
+  
+  charlength = parse_utf8_char (textHere, textEnd, &c);
+  
+  while (length+charlength <= MAXSTEMLEN && charlength > 0 &&
+	 (is_unicode_letter(c) || (is_unicode_digit(c) &&
+				   ++numeric <= MAXNUMERIC))) {
+    while (charlength-- > 0) {
+      mgWord[++length] = *textHere++;
+    }
+    charlength = parse_utf8_char (textHere, textEnd, &c);
+  }
+
+  mgWord[0] = length;
+  
+  return textHere;
+}
+
+const unsigned char *ParseNonindexWord (const unsigned char *textHere,
+					const unsigned char *textEnd) {
+  register int charlength = 0;
+  unsigned short c;
+  
+  charlength = parse_utf8_char(textHere, textEnd, &c);
+  
+  while (charlength > 0 && !is_unicode_letdig(c)) {
+    textHere += charlength;
+    charlength = parse_utf8_char (textHere, textEnd, &c);
+  }
+  
+  return textHere;
+}
Index: /trunk/mgpp/text/words.h
===================================================================
--- /trunk/mgpp/text/words.h	(revision 3365)
+++ /trunk/mgpp/text/words.h	(revision 3365)
@@ -0,0 +1,155 @@
+/**************************************************************************
+ *
+ * words.h -- Macros for parsing out words from the source text
+ * Copyright (C) 1994  Neil Sharman
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ **************************************************************************/
+
+#include "sysfuncs.h"
+
+#include "unitool.h"
+#include "UCArray.h"
+
+
+/*
+ * This has been cleaned up by Tim Shimmin. 
+ */
+
+/*
+ * ---NOTE---
+ *
+ * "WORD" refers to a word in the compressed text.
+ * "STEM" or "STEM_WORD" refers to a word for indexing on
+ *
+ */
+
+#define MAXWORDLEN	15
+	/* Maximum length in bytes of any word or non-word. Note that
+	   variations to MAXWORDLEN may have dramatic effects on the rest
+	   of the program, as the length and the prefix match are packed
+	   together into a four bit nibble, and there is not check that
+	   this is possible, i.e., leave MAXWORDLEN alone... */
+
+#define MAXSTEMLEN	255
+	/* Maximum length in bytes of any stem. Note that
+	   variations to MAXSTEMLEN may have dramatic effects on the rest
+	   of the program, , i.e., leave MAXSTEMLEN alone... */
+
+#define MAXNUMERIC	4
+	/* Maximum number of numeric characters permitted in a word.
+	   This avoids long sequences of numbers creating just one
+	   word occurrence for each number. At most 10,000 all numeric
+	   words will be permitted. */
+
+/* [RPAP - Jan 97: Stem Index Change] */
+#define MAXPARAMLEN     20
+        /* Maximum number of bytes to read for a parameter value for a
+	   term in a query. */
+#define WEIGHTPARAM     '/'
+#define STEMPARAM       '#'
+
+/* [RJM 07/97: Ranked Required Terms] */
+#define MUSTMATCHPARAM     '+'
+
+
+#define PESINAWORD(c)      (isalnum(c) || ((c) >= 0x80 && (c) <= 0xff))
+        /* The definition of what characters are permitted in a word.
+	   This macro is pessimistic, you cannot tell from a particular
+	   byte above 0x80 whether it is a character or not. This function
+	   is needed by various functions relating to huffman coding
+	   where frequency counts need to be primed, it should not be
+	   used in parsing the UTF-8 encoded input. */
+
+#ifdef __cplusplus
+extern "C"
+#endif
+int inaword (const u_char *here, const u_char *end);
+        /* Takes the place of the old INAWORD macro. It determines
+	   whether a given place in a UTF-8 encoded Unicode string
+	   is part of a word. */
+
+
+const unsigned char *ParseIndexWord (const unsigned char *textHere,
+				     const unsigned char *textEnd,
+				     UCArray &word);
+const unsigned char *ParseIndexMGWord (const unsigned char *textHere,
+				       const unsigned char *textEnd,
+				       unsigned char *mgWord);
+const unsigned char *ParseNonindexWord (const unsigned char *textHere,
+					const unsigned char *textEnd);
+
+
+
+/* =========================================================================
+ * Macro: PARSE_WORD
+ * Description: 
+ *      Extract a word out for compressing text
+ * Input: 
+ *      s_in = string start in buffer
+ *      end = string end in buffer
+ * Output: 
+ *      Word = extracted word with length in 1st byte
+ *      s_in = ptr to next character in buffer yet to be processed
+ * ========================================================================= */
+#define PARSE_WORD(Word, s_in, end)                                \
+  do {                                                             \
+    register int charlength = 0;                                   \
+    register u_char *wptr = (Word)+1;                              \
+    register int length = 0;                                       \
+    register int numeric = 0;                                      \
+    unsigned short c;                                              \
+                                                                   \
+    charlength = parse_utf8_char((s_in),(end),&c);                 \
+                                                                   \
+    while (length+charlength <= MAXWORDLEN && charlength > 0 &&    \
+	   (is_unicode_letter(c) || (is_unicode_digit(c) &&        \
+				     ++numeric <= MAXNUMERIC))) {  \
+      while (charlength-- > 0) {                                   \
+        *wptr++ = *(s_in)++; length++;                             \
+      }                                                            \
+      charlength = parse_utf8_char((s_in),(end),&c);               \
+    }                                                              \
+    *(Word) = length;                                              \
+  }while(0)
+
+
+
+/* =========================================================================
+ * Macro: PARSE_NON_WORD
+ * Description: 
+ *      Extract a non-word out for storing compressed text
+ * Input: as above
+ * Output: as above
+ * ========================================================================= */
+#define PARSE_NON_WORD(Word, s_in, end)                            \
+  do {                                                             \
+    register int charlength = 0;                                   \
+    register u_char *wptr = (Word)+1;                              \
+    register int length = 0;                                       \
+    unsigned short c;                                              \
+                                                                   \
+    charlength = parse_utf8_char((s_in),(end),&c);                 \
+                                                                   \
+    while (length+charlength <= MAXWORDLEN && charlength > 0 &&    \
+	   !is_unicode_letdig(c)) {                                \
+      while (charlength-- > 0) {                                   \
+        *wptr++ = *(s_in)++; length++;                             \
+      }                                                            \
+      charlength = parse_utf8_char((s_in),(end),&c);               \
+    }                                                              \
+    *(Word) = length;                                              \
+  }while(0)
