source: other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-YARN-SPARK-CATALOG-LANG-COUNT.sh

Last change on this file was 31452, checked in by davidb, 7 years ago

Additional Spark progs to run

  • Property svn:executable set to *
File size: 650 bytes
Line 
1#!/bin/bash
2
3json_filelist=${1:-full-listing-step100000.txt}
4shift
5
6input_dir="hdfs://$SPARK_MASTER_HOST:9000/user/dbbridge/full-ef-json-files"
7
8solr_url="http://gc0:8983/solr/htrc-full-ef/update"
9
10
11##master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077"
12
13#--conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError
14master_opt="--num-executors 10 --executor-cores 4 --driver-memory 20g --executor-memory 8g --master yarn --deploy-mode cluster"
15
16# 33
17
18
19classmain="org.hathitrust.extractedfeatures.ProcessForCatalogLangCount"
20
21. scripts/_RUN.sh
22
Note: See TracBrowser for help on using the repository browser.