Ignore:
Timestamp:
2016-12-27T18:54:09+13:00 (7 years ago)
Author:
davidb
Message:

Adjustments to memory allocation in response to test runs on 10% of dataset

File:
1 edited

Legend:

Unmodified
Added
Removed

  • other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK.sh

    r31093 r31268  
    44shift
    55
    6 input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files"
     6input_dir="hdfs://$SPARK_MASTER_HOST:9000/user/dbbridge/full-ef-json-files"
     7#input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files"
    78#input_dir="hdfs://master:9000/user/htrc/full-ef-json-files"
    89#input_dir="hdfs://10.10.0.52:9000/user/htrc/full-ef-json-files"
    … …  
    1314#
    1415
    15 master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077"
     16#master_opt="--driver-memory 50g --executor-memory 12g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077"
     17master_opt="--driver-memory 50g --executor-memory 70g --master spark://$SPARK_MASTER_HOST:7077"
    1618
    1719. scripts/_RUN.sh
Note: See TracChangeset for help on using the changeset viewer.