root/other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/ef-solr.properties @ 31095

Revision 31095, 0.7 KB (checked in by davidb, 4 years ago)

Introduced num-partitions property

Line 
1
2#wcsa-ef-ingest.process-ef-json-mode = per-volume
3wcsa-ef-ingest.process-ef-json-mode = per-page
4
5#wcsa-ef-ingest.solr-endpoint-ips = 10.11.0.53,10.11.0.54,10.11.0.55
6wcsa-ef-ingest.solr-endpoint-ips = gc0,gc1,gc2,gc3,gc4,gc5,gc6,gc7,gc8,gc9
7wcsa-ef-ingest.strict-file-io = false
8
9# For guide on number of partitions to use, see "Parallelized collections" section of:
10#   https://spark.apache.org/docs/2.0.1/programming-guide.html
11# which suggests 2-4 * num_cores
12#
13# For a more detailed discussion see:
14#   http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
15   
16# wcsa-ef-ingest.num-partitions = 12
17wcsa-ef-ingest.num-partitions = 120
18
19
Note: See TracBrowser for help on using the browser.