# # ChangeLog for other-projects/hathitrust # # Generated by Trac 1.4.2 # 2024-03-29T02:14:52+13:00 Sun, 11 Dec 2016 21:02:37 GMT davidb [31197] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/DictionaryWhitelist.java (deleted) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/WhitelistBloomFilter.java (deleted) File renaming to make way for newer version of classes needed in the ... Sun, 11 Dec 2016 21:01:30 GMT davidb [31196] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/TestWhitelistBloomFilter.java (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/TestWhitelistDictionaryMain.java (added) File renaming to make way for newer version of classes needed in the ... Sun, 11 Dec 2016 21:00:08 GMT davidb [31195] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/TESTWhitelistHashmap.java (moved) File renaming to make way for newer version of classes needed in the ... Sun, 11 Dec 2016 20:51:07 GMT davidb [31194] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/WhitelistBloomFilter.java (modified) Serialize in and out methods added Sun, 11 Dec 2016 20:32:50 GMT davidb [31193] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/wcsa-whitelist1.csv.gz (added) Peter's white-list file Wed, 07 Dec 2016 20:21:25 GMT davidb [31184] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK-GEN-WHITELIST.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/_RUN.sh (modified) New provision to run different main classes in _RUN.sh; New top-level ... Wed, 07 Dec 2016 20:19:36 GMT davidb [31183] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/.classpath (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/.settings/org.eclipse.jdt.core.prefs (modified) Bump up to project using Java 1.8 Sat, 03 Dec 2016 08:23:51 GMT davidb [31177] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/pom.xml (modified) Adding in Google jar that supports Bloom filters Sat, 03 Dec 2016 08:16:38 GMT davidb [31176] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/SolrDocJSON.java (modified) Support added for producing whitelist word count Sat, 03 Dec 2016 08:15:52 GMT davidb [31175] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/DictionaryWhitelist.java (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerVolumeWordStreamFlatmap.java (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForWhitelist.java (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/WhitelistBloomFilter.java (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/WhitelistHashmap.java (added) Trial to find memory difference betwen Hashmap and Bloom filters Sat, 03 Dec 2016 01:16:01 GMT davidb [31174] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-EF-HDFS-extra10-njp-missing.sh (added) One of the last scripts developed for getting ef dataset into HDFS Sat, 03 Dec 2016 01:14:20 GMT davidb [31173] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-aeu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-bc.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-caia.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-chi.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-coo.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-coo1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-dul1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-emu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-gri.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-hvd.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-iau.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ien.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-inu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-keio.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ku01.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-loc.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mcg.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mdp.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-miua.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-miun.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mmet.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mou.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nc01.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ncs1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-njp.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nnc1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nnc2.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nyp.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-osu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-psia.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-pst.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-pur1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-txa.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uc1-filename.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uc1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uc2.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ucm.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ucw.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-udel.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ufl1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ufl2.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uiuc.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uiug.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uiuo.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uma.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-umn.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-usu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uva.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-wau.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-wu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-yale.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-yul.txt (added) individual file sizes per top-level folder Fri, 02 Dec 2016 20:40:17 GMT davidb [31172] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-FILE-SIZE-COUNT.sh (added) to help track down missing files in HDFS copy Thu, 01 Dec 2016 21:15:59 GMT davidb [31171] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK-SUBFOLDERS.pl (added) Util to help find where missing files are Thu, 01 Dec 2016 21:15:25 GMT davidb [31170] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-TL-TARGET-DEPTH2-FOREACH-DEPTH3-HDFS-PUT.sh (added) Targetted sub-dir copy Thu, 01 Dec 2016 21:14:47 GMT davidb [31169] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-EF-HDFS.sh (modified) Improved logic Fri, 25 Nov 2016 10:00:10 GMT davidb [31161] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK-HDFS.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK.sh (added) Comparison of local disk version with HDFS version Wed, 23 Nov 2016 10:11:16 GMT davidb [31152] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-FOREACH-HDFS-PUT.sh (modified) Development of script Wed, 23 Nov 2016 10:10:20 GMT davidb [31151] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-TL-TARGET-DEPTH1-FOREACH-DEPTH2-HDFS-PUT.sh (added) More nuanced version to help finish off the 'big put' Sun, 20 Nov 2016 09:14:09 GMT davidb [31128] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/CHECK-EF-HDFS.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-EF-HDFS.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-FOREACH-HDFS-PUT.sh (added) Some scripts to help with pushing and monitoring the progress of the ... Tue, 15 Nov 2016 20:24:05 GMT davidb [31112] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-rsync2nema-local-shard-all.sh (added) To move out shards saved in /tmp on gsliscluter1 nodes to nema Fri, 11 Nov 2016 04:27:54 GMT davidb [31106] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-init-full-ef-collection.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-start-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-stop-all.sh (added) Scripts to help run an rsync'd copy of gslistcluster1 /tmp/gcX-solr- ... Fri, 11 Nov 2016 03:01:24 GMT davidb [31105] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-check-local-disk-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-check-local-shardsize-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-delete-full-ef-collection.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-setup-local-disk-all.sh (added) Additional scripts to help with running solr locally out of /tmp area Fri, 11 Nov 2016 03:00:37 GMT davidb [31104] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) now configurable to be run from local disk (/tmp) Fri, 11 Nov 2016 00:22:02 GMT davidb [31103] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-init-full-ef-collection.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-stop-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-spark.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-zookeeper.bash (modified) Changes made after testing with 20 solr nodes Thu, 10 Nov 2016 10:15:32 GMT davidb [31102] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-QUERY.sh (added) Command line way of running a Solr test query Thu, 10 Nov 2016 10:08:48 GMT davidb [31101] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-init-full-ef-collection.sh (modified) Correction to collection name Thu, 10 Nov 2016 09:58:19 GMT davidb [31100] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Change to using solr-cloud-nodes that include port number Thu, 10 Nov 2016 09:43:57 GMT davidb [31099] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-init-full-ef-collection.sh (added) Changes resulting from test runs to get Zookeeper and Solr running on ... Thu, 10 Nov 2016 09:42:31 GMT davidb [31098] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-stop-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (modified) Changes resulting from test runs to get Zookeeper and Solr running on ... Thu, 10 Nov 2016 09:41:33 GMT davidb [31097] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF/zoo.cfg.in (moved) Changed to .in style namne Thu, 10 Nov 2016 06:25:14 GMT davidb [31096] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONFlatmap.java (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (modified) Only need to create a volume's pages output directory is _output_dir ... Thu, 10 Nov 2016 05:58:06 GMT davidb [31095] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Introduced num-partitions property Thu, 10 Nov 2016 03:21:20 GMT davidb [31094] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) Changes triggered by running on gsliscluster1 Thu, 10 Nov 2016 03:20:02 GMT davidb [31093] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/_RUN.sh (modified) Changes triggered by running on gsliscluster1 Thu, 10 Nov 2016 03:18:45 GMT davidb [31092] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/packages/GET-PACKAGES.sh (modified) Minor tweak to spark/hadoop combination downloaded Thu, 10 Nov 2016 03:15:30 GMT davidb [31091] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Change of number of core for 'gsliscluster1' machine; commmented out ... Thu, 10 Nov 2016 03:14:21 GMT davidb [31090] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONFlatmap.java (modified) Memory monitor debugging code, commented out Thu, 10 Nov 2016 03:13:12 GMT davidb [31089] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/JSONClusterFileIO.java (modified) Change in way the JSON file is read in. Motivation was a out-of- ... Thu, 10 Nov 2016 03:09:55 GMT davidb [31088] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ClusterFileIO.java (modified) Shift to newIstance for FileSystem due to StackOverflow page ... Mon, 07 Nov 2016 21:38:35 GMT davidb [31082] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-stop-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) Changes in response to testing on gchead Mon, 07 Nov 2016 10:51:29 GMT davidb [31081] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) Going live with generation of spark slaves file Mon, 07 Nov 2016 10:49:42 GMT davidb [31080] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-spark.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-zookeeper.bash (modified) echo formatting tidy up. Fixed some typos Mon, 07 Nov 2016 10:31:48 GMT davidb [31079] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/GET-PACKAGES-SOLR.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/GET-PACKAGES-SPARK.sh (added) Useful get started scripts Mon, 07 Nov 2016 10:27:52 GMT davidb [31078] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF/htrc_configs.tar.gz (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF/zoo.cfg (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-stop-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-start-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-stop-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-zookeeper-start.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-zookeeper-stop.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-spark.bash (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-zookeeper.bash (added) Some setup files and scripts to make running Spark and Solr easier on ... Mon, 07 Nov 2016 09:34:31 GMT davidb [31077] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/Vagrantfile (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/manifests/base-hadoop.pp (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/modules/hadoop/manifests/init.pp (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/modules/hadoop/templates/hadoop-env.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/modules/hadoop/templates/hdfs-site.xml (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/modules/hadoop/templates/masters (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/modules/hadoop/templates/setup-hadoop.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster/modules/spark/manifests/init.pp (modified) Move up to JDK1.8. Tidy up of Vagrant machine names. Support for ... Sun, 06 Nov 2016 20:09:03 GMT davidb [31065] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh (modified) Additional echo output Sat, 05 Nov 2016 02:04:01 GMT davidb [31062] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh (modified) Added in -W option so check-sum calculation is skipped Thu, 03 Nov 2016 22:01:29 GMT davidb [31058] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh (modified) echo for additional information added Thu, 03 Nov 2016 21:59:03 GMT davidb [31057] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/_RUN.sh (modified) Tweak to jps output formatting Thu, 03 Nov 2016 01:26:13 GMT davidb [31053] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh (modified) Addition of second argument, optional, for where to save the files Thu, 03 Nov 2016 00:46:49 GMT davidb [31051] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/packages/GET-PACKAGES.sh (modified) Added in JDK to list of possible packages needed Wed, 02 Nov 2016 09:52:43 GMT davidb [31046] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) Added property to control how severe a JSON IO problem is Wed, 02 Nov 2016 08:34:47 GMT davidb [31045] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/JSONClusterFileIO.java (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONFlatmap.java (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONMap.java (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) More careful treatment of what to do when a JSON file isn't there Wed, 02 Nov 2016 08:30:49 GMT davidb [31044] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/_RUN.sh (modified) Fixed up error when output_dir is empty Wed, 02 Nov 2016 08:24:32 GMT davidb [31043] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK.sh (added) Version for processing full EF set Wed, 02 Nov 2016 07:18:22 GMT davidb [31042] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/PD-RUN-MASTER-LOCAL.sh (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/PD-RUN-MASTER-SPARK.sh (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/_RUN.sh (moved) Name changes, preparing the way for FULL-RUN versions Wed, 02 Nov 2016 07:07:40 GMT davidb [31041] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Test needs to be more careful if -read-only specified Wed, 02 Nov 2016 04:20:52 GMT davidb [31036] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/RUN-PD-MASTER-LOCAL.bash (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/RUN-PD-MASTER-SPARK.bash (moved) Renaming to prepare way for YARN version of script Wed, 02 Nov 2016 04:16:04 GMT davidb [31035] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-SELECT-EVERY-N.sh (modified) Changes after testing scripts Wed, 02 Nov 2016 04:10:29 GMT davidb [31034] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-DOWNLOAD-EVERY-N.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-GET-FILE-LIST.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-SELECT-EVERY-N.sh (added) Development of scripts for working with Full EF dataset Wed, 02 Nov 2016 04:10:17 GMT davidb [31033] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/PD-GET-FILE-LIST.sh (moved) Development of scripts for working with Full EF dataset Wed, 02 Nov 2016 01:28:39 GMT davidb [31030] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONFlatmap.java (modified) Tweak to some verbosity level 2 printing Wed, 02 Nov 2016 01:19:23 GMT davidb [31029] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) Newline at end of file added Wed, 02 Nov 2016 01:17:45 GMT davidb [31028] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/_RUN.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONMap.java (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Support for randonly choosing Solr endpoints added in Wed, 02 Nov 2016 00:06:15 GMT davidb [31027] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Mixed typo in property name used Wed, 02 Nov 2016 00:01:16 GMT davidb [31026] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Corrected flag setting Tue, 01 Nov 2016 22:59:37 GMT davidb [31025] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Use property process-json-mode to determine which sort of Spark ... Tue, 01 Nov 2016 22:37:07 GMT davidb [31024] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Support for Java properties file Tue, 01 Nov 2016 01:14:51 GMT davidb [31022] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/web-portal-trunk (deleted) No longer used Tue, 01 Nov 2016 01:14:21 GMT davidb [31021] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/web-portal (copied) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/web-portal-trunk (copied) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/web-portal/trunk (deleted) Folder restructure to remove 'trunk' part Tue, 01 Nov 2016 01:13:11 GMT davidb [31020] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-solr-cluster-trunk (deleted) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster-trunk (deleted) No longer used Tue, 01 Nov 2016 01:12:15 GMT davidb [31019] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-solr-cluster (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster (moved) Part 2 or two-step folder restructure Tue, 01 Nov 2016 01:10:29 GMT davidb [31018] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-solr-cluster-trunk (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster-trunk (moved) Part 1 or two-step folder restructure Tue, 01 Nov 2016 01:08:24 GMT davidb [31017] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/web-portal (moved) Moved to correct position Tue, 01 Nov 2016 01:07:24 GMT davidb [31016] * other-projects/hathitrust/solr-extracted-features (deleted) No longer used Tue, 01 Nov 2016 01:06:05 GMT davidb [31015] * other-projects/hathitrust/wcsa (added) * other-projects/hathitrust/wcsa/extracted-features-solr (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-solr-cluster (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/vagrant-spark-hdfs-cluster (moved) * other-projects/hathitrust/wcsa/extracted-features-solr/web-portal (moved) Restructuring of projects into one Mon, 31 Oct 2016 07:51:39 GMT davidb [31013] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONMap.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Accumulator for PerPageMap Mon, 31 Oct 2016 02:40:36 GMT davidb [31011] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONForeach.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONMap.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/__PerPageJSONForeach.java (added) Further RDD flatMap/map restructuring and refactoring, for per-page Mon, 31 Oct 2016 02:14:07 GMT davidb [31010] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Tidy up on generating Spark App name Mon, 31 Oct 2016 02:10:48 GMT davidb [31009] * other-projects/hathitrust/vagrant-solr-cluster/trunk/README.txt (modified) * other-projects/hathitrust/vagrant-solr-cluster/trunk/modules/solr/manifests/init.pp (modified) * other-projects/hathitrust/vagrant-solr-cluster/trunk/modules/zookeeper/templates/solr-start-all.sh (modified) Adjustments after latest fresh 'vagrant up' trial Sun, 30 Oct 2016 20:40:20 GMT davidb [31008] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Additional detail added into Spark app name Sun, 30 Oct 2016 20:35:06 GMT davidb [31007] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/BasePerJSON.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/JSONSolrTransform.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSONForeach.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONFlatmap.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerPageJSONForeach.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/SolrDocJSON.java (added) Class name refactoring Sun, 30 Oct 2016 20:27:42 GMT davidb [31006] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSONForeach.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Further reversal of Base class. Switch to PerPage Sun, 30 Oct 2016 20:16:16 GMT davidb [31005] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (modified) Reversal of Base class in PerVolumeJSON Sun, 30 Oct 2016 11:25:57 GMT davidb [31004] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) added debug Sun, 30 Oct 2016 11:13:28 GMT davidb [31003] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/BasePerJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSONForeach.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (modified) Explicity default constructors added Sun, 30 Oct 2016 11:07:39 GMT davidb [31002] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/BasePerJSON.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSONForeach.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Need to separate flatMap and foreach calls in PagedJSON Sun, 30 Oct 2016 10:51:07 GMT davidb [31001] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/JSONSolrTransform.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PerVolumeJSON.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) Code to work per-volume and per-page Sun, 30 Oct 2016 09:51:02 GMT davidb [31000] * other-projects/hathitrust/solr-extracted-features/trunk/_RUN.bash (modified) Class name refactoring Sun, 30 Oct 2016 09:50:10 GMT davidb [30999] * other-projects/hathitrust/solr-extracted-features/trunk/pom.xml (modified) Class name refactoring Sun, 30 Oct 2016 09:49:56 GMT davidb [30998] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/PrepareForIngest.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (added) Class name refactoring Sun, 30 Oct 2016 09:49:39 GMT davidb [30997] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (modified) Verbosity control over printing Sun, 30 Oct 2016 09:25:42 GMT davidb [30996] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/ClusterFileIO.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/PagedJSON.java (deleted) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/PrepareForIngest.java (modified) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ClusterFileIO.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/JSONClusterFileIO.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/JSONSolrTransform.java (added) * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/PagedJSON.java (added) Code refactoring Sun, 30 Oct 2016 08:43:02 GMT davidb [30995] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/PrepareForIngest.java (modified) Adjustment of NUM_PARTITIONS to be based on Spark recommended calculation Sun, 30 Oct 2016 04:22:41 GMT davidb [30994] * other-projects/hathitrust/worksets-from-extracted-features/trunk/index.html (modified) Additional useful links. Links open in new tab Sun, 30 Oct 2016 04:10:57 GMT davidb [30993] * other-projects/hathitrust/worksets-from-extracted-features (added) * other-projects/hathitrust/worksets-from-extracted-features/trunk (added) * other-projects/hathitrust/worksets-from-extracted-features/trunk/index.html (added) Placeholder page to provide useful links to hadoop and solr cluster ... Sun, 30 Oct 2016 02:49:35 GMT davidb [30992] * other-projects/hathitrust/vagrant-solr-cluster/trunk/README.txt (modified) Additional adjustments after test run on cluster Sun, 30 Oct 2016 02:41:11 GMT davidb [30991] * other-projects/hathitrust/vagrant-solr-cluster/trunk/NOTES-AND-SOURCES.txt (added) * other-projects/hathitrust/vagrant-solr-cluster/trunk/README.txt (modified) Inital cut at README notes, and supporting links Sat, 29 Oct 2016 22:39:31 GMT davidb [30990] * other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/PrepareForIngest.java (modified) opt name change