source: other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/scripts/FULL-RUN-MASTER-SPARK-LANG-COUNT.sh

Last change on this file was 31260, checked in by davidb, 7 years ago

Language counting

  • Property svn:executable set to *
File size: 622 bytes
Line 
1#!/bin/bash
2
3json_filelist=${1:-full-listing-step100000.txt}
4shift
5
6input_dir="hdfs://$SPARK_MASTER_HOST:9000/tmp/dbbridge/full-ef-json-files"
7#input_dir="hdfs://master:9000/user/htrc/full-ef-json-files"
8#input_dir="hdfs://10.10.0.52:9000/user/htrc/full-ef-json-files"
9
10#output_dir=hdfs://master:9000/user/htrc/full-solr-json-files
11solr_url="http://gc0:8983/solr/htrc-full-ef/update"
12
13#
14
15master_opt="--executor-memory 10g --conf spark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError --master spark://$SPARK_MASTER_HOST:7077"
16
17classmain="org.hathitrust.extractedfeatures.ProcessForLangCount"
18
19. scripts/_RUN.sh
20
Note: See TracBrowser for help on using the repository browser.