source: gs2-extensions/parallel-building

Revision Log Mode:


Legend:

Added
Modified
Copied or renamed
Diff Rev Age Author Log Message
(edit) @27424   11 years jmt12 Adding canRead() and isAbsolute() functions, and some more debugging …
(edit) @27423   11 years jmt12 Adding canRead() and isAbsolute() functions
(edit) @27422   11 years jmt12 Adding canRead() and isAbsolute() functions
(edit) @27421   11 years jmt12 Fixing bug in filePutContents (wasn't shifting parameters off front of …
(edit) @27420   11 years jmt12 Updated to reflect changes in the perllib customization (most of it …
(edit) @27419   11 years jmt12 Adding more reasonable default values for HDFS host and port (so the …
(edit) @27418   11 years jmt12 Also recompile the HadoopGreenstoneIngest.java file during cascade make
(edit) @27417   11 years jmt12 A 'pseudo'-CASCASE-MAKE file for building java code. It doesn't really …
(edit) @27416   11 years jmt12 Forgot to ensure the manifests generated here are also version 2
(edit) @27415   11 years jmt12 Ensuring the manifests generated are marked as version 2 - i.e. should …
(edit) @27414   11 years jmt12 Allowing more processing arguments to be configured at the call, and …
(edit) @27413   11 years jmt12 Making the Hadoop processing aware that there are two different …
(edit) @27412   11 years jmt12 I obviously hadn't run this script on Karearea before - assumed all …
(edit) @27409   11 years jmt12 Unit test like testing for the FileUtils class and LocalFS, HDFSShell, …
(edit) @27408   11 years jmt12 A symbolic link to the actual script in the packages directory
(edit) @27407   11 years jmt12 updating ignore
(edit) @27406   11 years jmt12 Updating ignore
(edit) @27405   11 years jmt12 Now that I can get the Perl version, make the cascade make script …
(edit) @27404   11 years jmt12 Thrift access to HDFS and required Bit-Vector Perl module
(edit) @27386   11 years jmt12 Forgot these were just symbolic links to my Dropbox folder - adding in …
(edit) @27385   11 years jmt12 removing soft links to my dropbox folder - sorry
(edit) @27384   11 years jmt12 An overriding version of FileUtils that provides several drivers …
(edit) @27383   11 years jmt12 Adding a perl module for the slightly more efficient Base91 encoding …
(edit) @27382   11 years jmt12 Adding Thrift and its prerequisite BitVector packages
(edit) @27381   11 years jmt12 An image used during FileUtils testing to check binary transfer in …
(edit) @27378   11 years jmt12 Parallel processing support now added (via buildcolutil subclass) to …
(edit) @27377   11 years jmt12 Updating calls to intermediate util functions to the new FileUtils …
(edit) @27373   11 years jmt12 No longer needs a custom util.pm - file utility functions now moved …
(edit) @27372   11 years jmt12 Fixing where I'd used uppercase letters in a package name that didn't …
(edit) @27371   11 years jmt12 Replacing with a proper class inheriting from gdbm and just overriding …
(edit) @27370   11 years jmt12 fixing minor bug in inheritence tree
(edit) @27280   11 years jmt12 A subclass of buildcolutils.pm with added functionality for parallel …
(edit) @27279   11 years jmt12 A subclass of inexport.pm with added functionality for parallel …
(edit) @27278   11 years jmt12 Renamed so it can be found in an automated fashion (prefixed by the …
(edit) @27237   11 years jmt12 Moved several generally applicable changes (manifest version, …
(edit) @27236   11 years jmt12 extended loader (to support extensions) and conditional additions to …
(edit) @27235   11 years jmt12 Moving the conditional additions to INC into the main trunk's version
(edit) @27234   11 years jmt12 renamed prepare_build_recipe() to the more meaningful …
(edit) @27233   11 years jmt12 Recipe generation code moved into subclass of mgbuilder called …
(edit) @27232   11 years jmt12 Recipe generation code moved into subclass of mgppbuilder called …
(edit) @27231   11 years jmt12 Recipe generation code moved into subclass of mgppbuilder called …
(edit) @27230   11 years jmt12 Recipe generation code moved into subclass of mgbuilder called …
(edit) @27229   11 years jmt12 A subclass of basebuilder that includes an 'abstract' function for the …
(edit) @27228   11 years jmt12 A subclass of lucenebuilder that includes a function to generate a …
(edit) @27227   11 years jmt12 The indexlevel processing code has been moved into the main trunk's …
(edit) @27226   11 years jmt12 Moved the version property into the main trunk's version
(edit) @27216   11 years jmt12 renaming (away from Java/PHP conventions) to make my life easier when …
(edit) @27215   11 years jmt12 Moved the conditional INC and PATH addition code into the main trunk
(edit) @27214   11 years jmt12 Copied the two new strings (buildcol.indexname and …
(edit) @27213   11 years jmt12 Unchanged from version in main trunk
(edit) @27212   11 years jmt12 No difference from version in main trunk (apart from some debugging …
(edit) @27211   11 years jmt12 The parallel processing changes (pretty much just the placeholder …
(edit) @27210   11 years jmt12 DavidB already ported the few minor changes (complexmeta, indexname, …
(edit) @27209   11 years jmt12 No difference from version in trunk
(edit) @27208   11 years jmt12 The only change was to ensure that Greenstone's 'perllib' and …
(edit) @27207   11 years jmt12 Classify had been edited to do a better job of searching for …
(edit) @27206   11 years jmt12 The only real change in doc.pm was the replacement of a -e with a call …
(edit) @27205   11 years jmt12 Unchanged from base perllib version
(edit) @27204   11 years jmt12 Unchanged from base perllib version
(edit) @27203   11 years jmt12 SQLite support for Write Ahead Logging (offering basic parallel …
(edit) @27202   11 years jmt12 Unchanged from base perllib version
(edit) @27201   11 years jmt12 Should never have been checked in as I made no changes here - caused …
(edit) @27200   11 years jmt12 A subclass of basebuilder that provides a placeholder function for the …
(edit) @27195   11 years davidb Not changed for this extension
(edit) @27194   11 years davidb Not changed for this extension
(edit) @27193   11 years davidb Not changed for this extension
(edit) @27184   11 years davidb Removing 'downloaders' from parallel importing extension, as there are …
(edit) @27179   11 years davidb Mods to allow code to run with Greenstone3
(edit) @27177   11 years davidb Test for and create (if needed) the 'tmp' directory
(edit) @27126   11 years jmt12 Extra clean up commands (like removing cached versions of video …
(edit) @27125   11 years jmt12 A script to try and flush all caches - I'm certain it's flushing disk …
(edit) @27124   11 years jmt12 Use the new perl version script to extract the version number - so as …
(edit) @27119   11 years jmt12 Merging version finder from Medusa with the one lurking on Karearea
(edit) @27102   11 years jmt12 Adding in a progress reporter, as the set timeout to 10 hours trick …
(edit) @27058   11 years jmt12 Adding data locality report generation to Hadoop greenstone imports
(edit) @27052   11 years jmt12 Turns out the Perl on Medusa doesn't support $V, so I've had to …
(edit) @27051   11 years jmt12 Trying to force Sort::Key to install to a sensible location on Medusa …
(edit) @27048   11 years jmt12 Ignoring untarred Hadoop and Sort::Key directories
(edit) @27047   11 years jmt12 Script to allow the Perl module Sort::Key to be compiled/installed as …
(edit) @27046   11 years jmt12 Adding in Sort::Key cascade makefile
(edit) @27045   11 years jmt12 Ignoring untarred Hadoop and Sort::Key directories
(edit) @27044   11 years jmt12 The Hadoop reporting stuff needs natural sorting
(edit) @27043   11 years jmt12 Sort appears to be platform specific, so I'll need to include it as a …
(edit) @27042   11 years jmt12 Sort appears to be platform specific, so I'll need to include it as a …
(edit) @27041   11 years jmt12 INC path now includes the installed extensions perl path (including …
(edit) @27040   11 years jmt12 A simple script that returns just the version number of Perl
(edit) @27039   11 years jmt12 Adding natural sort for Perl. Shared object files appear to be ignored …
(edit) @27038   11 years jmt12 Allow natural sorting in Perl
(edit) @27037   11 years jmt12 Allow natural sorting in Perl
(edit) @27036   11 years jmt12 A script to extract data locality and other task information from the …
(edit) @27006   11 years jmt12 A companion script to stop-hadoop-processes that just reports running …
(edit) @27005   11 years jmt12 Similar to stop-impt.pl, this script uses kill to stop runaway Hadoop …
(edit) @27004   11 years jmt12 A script to stop (using kill) a runaway import process and any related …
(edit) @27003   11 years jmt12 Adding some Hadoopy stuff to setup.bash - will need to be configured …
(edit) @27002   11 years jmt12 Fixing bug in maxdocs processing where I compared to 'file_count' …
(edit) @27001   11 years jmt12 Passing more environment variables (HADOOPPREFIX, HDFSHOST, HDFSPORT) …
(edit) @26999   11 years jmt12 Ensuring MPI binds to correct interface, and passing through …
(edit) @26998   11 years jmt12 Adding maxdocs variable, lots of debug comments, added some tests for …
(edit) @26987   11 years jmt12 Removing some debug comments I was using to track down why TDB support …
(edit) @26986   11 years jmt12 Commenting out some debug comments
Note: See TracRevisionLog for help on using the revision log.