Changeset 31091 for other-projects/hathitrust/wcsa/extracted-features-solr
- Timestamp:
- 2016-11-10T16:15:30+13:00 (7 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java
r31045 r31091 28 28 // http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ 29 29 30 public static final int NUM_CORES = 6;30 public static final int NUM_CORES = 10*12; 31 31 public static final int NUM_PARTITIONS = 2*NUM_CORES; // default would appear to be 2 32 32 … … 103 103 double per_vol = 100.0/(double)num_volumes; 104 104 105 //JavaRDD<String> json_list_data_rp = json_list_data.repartition((int)(num_volumes/100)); 106 105 107 DoubleAccumulator progress_accum = jsc.sc().doubleAccumulator("Progress Percent"); 106 108 … … 147 149 double per_vol = 100.0/(double)num_volumes; 148 150 151 //JavaRDD<String> json_list_data_rp = json_list_data.repartition((int)(num_volumes/100)); 152 149 153 DoubleAccumulator per_vol_progress_accum = jsc.sc().doubleAccumulator("Per Volume Progress Percent"); 150 154
Note:
See TracChangeset
for help on using the changeset viewer.