|
Last change
on this file since 31452 was 31452, checked in by davidb, 7 years ago |
|
Additional Spark progs to run
|
-
Property svn:executable
set to
*
|
|
File size:
650 bytes
|
| Line | |
|---|
| 1 | #!/bin/bash
|
|---|
| 2 |
|
|---|
| 3 | json_filelist=${1:-full-listing-step100000.txt}
|
|---|
| 4 | shift
|
|---|
| 5 |
|
|---|
| 6 | input_dir="hdfs://$SPARK_MASTER_HOST:9000/user/dbbridge/full-ef-json-files"
|
|---|
| 7 |
|
|---|
| 8 | solr_url="http://gc0:8983/solr/htrc-full-ef/update"
|
|---|
| 9 |
|
|---|
| 10 |
|
|---|
| 11 | ##master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077"
|
|---|
| 12 |
|
|---|
| 13 | #--conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError
|
|---|
| 14 | master_opt="--num-executors 10 --executor-cores 4 --driver-memory 20g --executor-memory 8g --master yarn --deploy-mode cluster"
|
|---|
| 15 |
|
|---|
| 16 | # 33
|
|---|
| 17 |
|
|---|
| 18 |
|
|---|
| 19 | classmain="org.hathitrust.extractedfeatures.ProcessForCatalogLangCount"
|
|---|
| 20 |
|
|---|
| 21 | . scripts/_RUN.sh
|
|---|
| 22 |
|
|---|
Note:
See
TracBrowser
for help on using the repository browser.
