root/gs2-extensions/parallel-building

Revision Log Mode:


Legend:

Added
Modified
Copied or renamed
Rev Chgset Date Author Log Message
(edit) @27437 [27437] 7 years jmt12 Why doesn't this exist on Medusa... I thought it was a built-in
(edit) @27436 [27436] 7 years jmt12 Adding the actual script - rather than a symlink to my dropbox. doh
(edit) @27435 [27435] 7 years jmt12 Gah - only a symbolic link
(edit) @27434 [27434] 7 years jmt12 Updated the starting script to look in the correct place (in this package, …
(edit) @27433 [27433] 7 years jmt12 Adding a few more details to the Hadoop log to assist with later work
(edit) @27424 [27424] 7 years jmt12 Adding canRead() and isAbsolute() functions, and some more debugging while …
(edit) @27423 [27423] 7 years jmt12 Adding canRead() and isAbsolute() functions
(edit) @27422 [27422] 7 years jmt12 Adding canRead() and isAbsolute() functions
(edit) @27421 [27421] 7 years jmt12 Fixing bug in filePutContents (wasn't shifting parameters off front of @_, …
(edit) @27420 [27420] 7 years jmt12 Updated to reflect changes in the perllib customization (most of it being …
(edit) @27419 [27419] 7 years jmt12 Adding more reasonable default values for HDFS host and port (so the …
(edit) @27418 [27418] 7 years jmt12 Also recompile the HadoopGreenstoneIngest?.java file during cascade make
(edit) @27417 [27417] 7 years jmt12 A 'pseudo'-CASCASE-MAKE file for building java code. It doesn't really use …
(edit) @27416 [27416] 7 years jmt12 Forgot to ensure the manifests generated here are also version 2
(edit) @27415 [27415] 7 years jmt12 Ensuring the manifests generated are marked as version 2 - i.e. should be …
(edit) @27414 [27414] 7 years jmt12 Allowing more processing arguments to be configured at the call, and …
(edit) @27413 [27413] 7 years jmt12 Making the Hadoop processing aware that there are two different possible …
(edit) @27412 [27412] 7 years jmt12 I obviously hadn't run this script on Karearea before - assumed all …
(edit) @27409 [27409] 7 years jmt12 Unit test like testing for the FileUtils? class and LocalFS, HDFSShell, and …
(edit) @27408 [27408] 7 years jmt12 A symbolic link to the actual script in the packages directory
(edit) @27407 [27407] 7 years jmt12 updating ignore
(edit) @27406 [27406] 7 years jmt12 Updating ignore
(edit) @27405 [27405] 7 years jmt12 Now that I can get the Perl version, make the cascade make script smart …
(edit) @27404 [27404] 7 years jmt12 Thrift access to HDFS and required Bit-Vector Perl module
(edit) @27386 [27386] 7 years jmt12 Forgot these were just symbolic links to my Dropbox folder - adding in the …
(edit) @27385 [27385] 7 years jmt12 removing soft links to my dropbox folder - sorry
(edit) @27384 [27384] 7 years jmt12 An overriding version of FileUtils? that provides several drivers required …
(edit) @27383 [27383] 7 years jmt12 Adding a perl module for the slightly more efficient Base91 encoding (as …
(edit) @27382 [27382] 7 years jmt12 Adding Thrift and its prerequisite BitVector? packages
(edit) @27381 [27381] 7 years jmt12 An image used during FileUtils? testing to check binary transfer in …
(edit) @27378 [27378] 7 years jmt12 Parallel processing support now added (via buildcolutil subclass) to …
(edit) @27377 [27377] 7 years jmt12 Updating calls to intermediate util functions to the new FileUtils?
(edit) @27373 [27373] 7 years jmt12 No longer needs a custom util.pm - file utility functions now moved into …
(edit) @27372 [27372] 7 years jmt12 Fixing where I'd used uppercase letters in a package name that didn't have …
(edit) @27371 [27371] 7 years jmt12 Replacing with a proper class inheriting from gdbm and just overriding the …
(edit) @27370 [27370] 7 years jmt12 fixing minor bug in inheritence tree
(edit) @27280 [27280] 7 years jmt12 A subclass of buildcolutils.pm with added functionality for parallel …
(edit) @27279 [27279] 7 years jmt12 A subclass of inexport.pm with added functionality for parallel processing …
(edit) @27278 [27278] 7 years jmt12 Renamed so it can be found in an automated fashion (prefixed by the …
(edit) @27237 [27237] 7 years jmt12 Moved several generally applicable changes (manifest version, conditional …
(edit) @27236 [27236] 7 years jmt12 extended loader (to support extensions) and conditional additions to INC …
(edit) @27235 [27235] 7 years jmt12 Moving the conditional additions to INC into the main trunk's version
(edit) @27234 [27234] 7 years jmt12 renamed prepare_build_recipe() to the more meaningful prepareIndexRecipe() …
(edit) @27233 [27233] 7 years jmt12 Recipe generation code moved into subclass of mgbuilder called …
(edit) @27232 [27232] 7 years jmt12 Recipe generation code moved into subclass of mgppbuilder called …
(edit) @27231 [27231] 7 years jmt12 Recipe generation code moved into subclass of mgppbuilder called …
(edit) @27230 [27230] 7 years jmt12 Recipe generation code moved into subclass of mgbuilder called …
(edit) @27229 [27229] 7 years jmt12 A subclass of basebuilder that includes an 'abstract' function for the …
(edit) @27228 [27228] 7 years jmt12 A subclass of lucenebuilder that includes a function to generate a …
(edit) @27227 [27227] 7 years jmt12 The indexlevel processing code has been moved into the main trunk's …
(edit) @27226 [27226] 7 years jmt12 Moved the version property into the main trunk's version
(edit) @27216 [27216] 7 years jmt12 renaming (away from Java/PHP conventions) to make my life easier when …
(edit) @27215 [27215] 7 years jmt12 Moved the conditional INC and PATH addition code into the main trunk
(edit) @27214 [27214] 7 years jmt12 Copied the two new strings (buildcol.indexname and buildcol.indexlevel) …
(edit) @27213 [27213] 7 years jmt12 Unchanged from version in main trunk
(edit) @27212 [27212] 7 years jmt12 No difference from version in main trunk (apart from some debugging code …
(edit) @27211 [27211] 7 years jmt12 The parallel processing changes (pretty much just the placeholder function …
(edit) @27210 [27210] 7 years jmt12 DavidB already ported the few minor changes (complexmeta, indexname, and …
(edit) @27209 [27209] 7 years jmt12 No difference from version in trunk
(edit) @27208 [27208] 7 years jmt12 The only change was to ensure that Greenstone's 'perllib' and …
(edit) @27207 [27207] 7 years jmt12 Classify had been edited to do a better job of searching for …
(edit) @27206 [27206] 7 years jmt12 The only real change in doc.pm was the replacement of a -e with a call to …
(edit) @27205 [27205] 7 years jmt12 Unchanged from base perllib version
(edit) @27204 [27204] 7 years jmt12 Unchanged from base perllib version
(edit) @27203 [27203] 7 years jmt12 SQLite support for Write Ahead Logging (offering basic parallel processing …
(edit) @27202 [27202] 7 years jmt12 Unchanged from base perllib version
(edit) @27201 [27201] 7 years jmt12 Should never have been checked in as I made no changes here - caused by …
(edit) @27200 [27200] 7 years jmt12 A subclass of basebuilder that provides a placeholder function for the …
(edit) @27195 [27195] 7 years davidb Not changed for this extension
(edit) @27194 [27194] 7 years davidb Not changed for this extension
(edit) @27193 [27193] 7 years davidb Not changed for this extension
(edit) @27184 [27184] 7 years davidb Removing 'downloaders' from parallel importing extension, as there are no …
(edit) @27179 [27179] 7 years davidb Mods to allow code to run with Greenstone3
(edit) @27177 [27177] 7 years davidb Test for and create (if needed) the 'tmp' directory
(edit) @27126 [27126] 7 years jmt12 Extra clean up commands (like removing cached versions of video …
(edit) @27125 [27125] 7 years jmt12 A script to try and flush all caches - I'm certain it's flushing disk …
(edit) @27124 [27124] 7 years jmt12 Use the new perl version script to extract the version number - so as to …
(edit) @27119 [27119] 7 years jmt12 Merging version finder from Medusa with the one lurking on Karearea
(edit) @27102 [27102] 7 years jmt12 Adding in a progress reporter, as the set timeout to 10 hours trick …
(edit) @27058 [27058] 7 years jmt12 Adding data locality report generation to Hadoop greenstone imports
(edit) @27052 [27052] 7 years jmt12 Turns out the Perl on Medusa doesn't support $V, so I've had to change …
(edit) @27051 [27051] 7 years jmt12 Trying to force Sort::Key to install to a sensible location on Medusa by …
(edit) @27048 [27048] 7 years jmt12 Ignoring untarred Hadoop and Sort::Key directories
(edit) @27047 [27047] 7 years jmt12 Script to allow the Perl module Sort::Key to be compiled/installed as part …
(edit) @27046 [27046] 7 years jmt12 Adding in Sort::Key cascade makefile
(edit) @27045 [27045] 7 years jmt12 Ignoring untarred Hadoop and Sort::Key directories
(edit) @27044 [27044] 7 years jmt12 The Hadoop reporting stuff needs natural sorting
(edit) @27043 [27043] 7 years jmt12 Sort appears to be platform specific, so I'll need to include it as a …
(edit) @27042 [27042] 7 years jmt12 Sort appears to be platform specific, so I'll need to include it as a …
(edit) @27041 [27041] 7 years jmt12 INC path now includes the installed extensions perl path (including …
(edit) @27040 [27040] 7 years jmt12 A simple script that returns just the version number of Perl
(edit) @27039 [27039] 7 years jmt12 Adding natural sort for Perl. Shared object files appear to be ignored by …
(edit) @27038 [27038] 7 years jmt12 Allow natural sorting in Perl
(edit) @27037 [27037] 7 years jmt12 Allow natural sorting in Perl
(edit) @27036 [27036] 7 years jmt12 A script to extract data locality and other task information from the …
(edit) @27006 [27006] 7 years jmt12 A companion script to stop-hadoop-processes that just reports running …
(edit) @27005 [27005] 7 years jmt12 Similar to stop-impt.pl, this script uses kill to stop runaway Hadoop …
(edit) @27004 [27004] 7 years jmt12 A script to stop (using kill) a runaway import process and any related TDB …
(edit) @27003 [27003] 7 years jmt12 Adding some Hadoopy stuff to setup.bash - will need to be configured each …
(edit) @27002 [27002] 7 years jmt12 Fixing bug in maxdocs processing where I compared to 'file_count' rather …
Note: See TracRevisionLog for help on using the revision log.