# # ChangeLog for other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS # # Generated by Trac 1.4.2 # 2024-04-28T08:55:10+12:00 Thu, 19 Jan 2017 04:25:05 GMT davidb [31302] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-mongodb-setup-local-disk-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-mongodb-start-configservers.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-mongodb-stop-configservers.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-mongodb.bash (modified) Initial commit of scripts, after some testing, and subsequent changes ... Tue, 13 Dec 2016 09:41:54 GMT davidb [31231] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-check-local-shardsize-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-rsync2nema-local-shard-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-setup-local-disk-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (modified) Changes to allow SOLR to run on nodes in /hdfsd05/dbbridge/solr-ef Sat, 03 Dec 2016 01:16:01 GMT davidb [31174] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-EF-HDFS-extra10-njp-missing.sh (added) One of the last scripts developed for getting ef dataset into HDFS Sat, 03 Dec 2016 01:14:20 GMT davidb [31173] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-aeu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-bc.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-caia.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-chi.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-coo.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-coo1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-dul1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-emu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-gri.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-hvd.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-iau.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ien.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-inu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-keio.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ku01.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-loc.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mcg.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mdp.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-miua.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-miun.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mmet.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-mou.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nc01.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ncs1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-njp.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nnc1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nnc2.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-nyp.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-osu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-psia.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-pst.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-pur1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-txa.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uc1-filename.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uc1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uc2.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ucm.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ucw.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-udel.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ufl1.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-ufl2.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uiuc.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uiug.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uiuo.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uma.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-umn.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-usu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-uva.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-wau.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-wu.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-yale.txt (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/file-size-local/ef-full-yul.txt (added) individual file sizes per top-level folder Fri, 02 Dec 2016 20:40:17 GMT davidb [31172] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-FILE-SIZE-COUNT.sh (added) to help track down missing files in HDFS copy Thu, 01 Dec 2016 21:15:59 GMT davidb [31171] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK-SUBFOLDERS.pl (added) Util to help find where missing files are Thu, 01 Dec 2016 21:15:25 GMT davidb [31170] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-TL-TARGET-DEPTH2-FOREACH-DEPTH3-HDFS-PUT.sh (added) Targetted sub-dir copy Thu, 01 Dec 2016 21:14:47 GMT davidb [31169] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-EF-HDFS.sh (modified) Improved logic Fri, 25 Nov 2016 10:00:10 GMT davidb [31161] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK-HDFS.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FILE-SIZE-CHECK.sh (added) Comparison of local disk version with HDFS version Wed, 23 Nov 2016 10:11:16 GMT davidb [31152] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-FOREACH-HDFS-PUT.sh (modified) Development of script Wed, 23 Nov 2016 10:10:20 GMT davidb [31151] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-TL-TARGET-DEPTH1-FOREACH-DEPTH2-HDFS-PUT.sh (added) More nuanced version to help finish off the 'big put' Sun, 20 Nov 2016 09:14:09 GMT davidb [31128] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/CHECK-EF-HDFS.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/FULL-EF-HDFS.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/PAIRTREE-FOREACH-HDFS-PUT.sh (added) Some scripts to help with pushing and monitoring the progress of the ... Tue, 15 Nov 2016 20:24:05 GMT davidb [31112] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-rsync2nema-local-shard-all.sh (added) To move out shards saved in /tmp on gsliscluter1 nodes to nema Fri, 11 Nov 2016 04:27:54 GMT davidb [31106] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-init-full-ef-collection.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-start-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/nema-solr-stop-all.sh (added) Scripts to help run an rsync'd copy of gslistcluster1 /tmp/gcX-solr- ... Fri, 11 Nov 2016 03:01:24 GMT davidb [31105] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-check-local-disk-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-check-local-shardsize-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-delete-full-ef-collection.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-setup-local-disk-all.sh (added) Additional scripts to help with running solr locally out of /tmp area Fri, 11 Nov 2016 03:00:37 GMT davidb [31104] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) now configurable to be run from local disk (/tmp) Fri, 11 Nov 2016 00:22:02 GMT davidb [31103] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-init-full-ef-collection.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-stop-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-spark.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-zookeeper.bash (modified) Changes made after testing with 20 solr nodes Thu, 10 Nov 2016 10:08:48 GMT davidb [31101] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-init-full-ef-collection.sh (modified) Correction to collection name Thu, 10 Nov 2016 09:43:57 GMT davidb [31099] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-init-full-ef-collection.sh (added) Changes resulting from test runs to get Zookeeper and Solr running on ... Thu, 10 Nov 2016 09:42:31 GMT davidb [31098] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-stop-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (modified) Changes resulting from test runs to get Zookeeper and Solr running on ... Mon, 07 Nov 2016 21:38:35 GMT davidb [31082] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-start-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-stop-all.sh (modified) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (modified) Changes in response to testing on gchead Mon, 07 Nov 2016 10:27:52 GMT davidb [31078] * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF/htrc_configs.tar.gz (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/CONF/zoo.cfg (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-start-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-solr-stop-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-start-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-spark-stop-all.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-zookeeper-start.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SCRIPTS/remote-zookeeper-stop.sh (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP.bash (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-solr.bash (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-spark.bash (added) * other-projects/hathitrust/wcsa/extracted-features-solr/trunk/gslis-cluster/SETUP/setup-zookeeper.bash (added) Some setup files and scripts to make running Spark and Solr easier on ...