root/other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK-CATALOG-LANG-COUNT.sh @ 31452

Revision 31452, 0.6 KB (checked in by davidb, 3 years ago)

Additional Spark progs to run

  • Property svn:executable set to *
Line 
1#!/bin/bash
2
3json_filelist=${1:-full-listing-step100000.txt}
4shift
5
6input_dir="hdfs://$SPARK_MASTER_HOST:9000/user/dbbridge/full-ef-json-files"
7
8solr_url="http://gc0:8983/solr/htrc-full-ef/update"
9
10
11##master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077"
12
13#--conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError
14master_opt="--driver-memory 30g --executor-memory 50g --master spark://$SPARK_MASTER_HOST:7077"
15
16classmain="org.hathitrust.extractedfeatures.ProcessForCatalogLangCount"
17
18. scripts/_RUN.sh
19
Note: See TracBrowser for help on using the browser.