Changeset 31268 for other-projects

Show
Ignore:
Timestamp:
27.12.2016 18:54:09 (3 years ago)
Author:
davidb
Message:

Adjustments to memory allocation in response to test runs on 10% of dataset

Files:
1 modified

Legend:

Unmodified
Added
Removed
  • other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK.sh

    r31093 r31268  
    44shift 
    55 
    6 input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files" 
     6input_dir="hdfs://$SPARK_MASTER_HOST:9000/user/dbbridge/full-ef-json-files" 
     7#input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files" 
    78#input_dir="hdfs://master:9000/user/htrc/full-ef-json-files" 
    89#input_dir="hdfs://10.10.0.52:9000/user/htrc/full-ef-json-files" 
     
    1314#  
    1415 
    15 master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077" 
     16#master_opt="--driver-memory 50g --executor-memory 12g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077" 
     17master_opt="--driver-memory 50g --executor-memory 70g --master spark://$SPARK_MASTER_HOST:7077" 
    1618 
    1719. scripts/_RUN.sh