Changeset 31268 for other-projects/hathitrust/wcsa
- Timestamp:
- 2016-12-27T18:54:09+13:00 (7 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK.sh
r31093 r31268 4 4 shift 5 5 6 input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files" 6 input_dir="hdfs://$SPARK_MASTER_HOST:9000/user/dbbridge/full-ef-json-files" 7 #input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files" 7 8 #input_dir="hdfs://master:9000/user/htrc/full-ef-json-files" 8 9 #input_dir="hdfs://10.10.0.52:9000/user/htrc/full-ef-json-files" … … 13 14 # 14 15 15 master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077" 16 #master_opt="--driver-memory 50g --executor-memory 12g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077" 17 master_opt="--driver-memory 50g --executor-memory 70g --master spark://$SPARK_MASTER_HOST:7077" 16 18 17 19 . scripts/_RUN.sh
Note:
See TracChangeset
for help on using the changeset viewer.