Changeset 31091

Show
Ignore:
Timestamp:
10.11.2016 16:15:30 (3 years ago)
Author:
davidb
Message:

Change of number of core for 'gsliscluster1' machine; commmented out experimental repartition added.

Files:
1 modified

Legend:

Unmodified
Added
Removed
  • other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java

    r31045 r31091  
    2828    //   http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ 
    2929     
    30     public static final int NUM_CORES = 6; 
     30    public static final int NUM_CORES = 10*12; 
    3131    public static final int NUM_PARTITIONS = 2*NUM_CORES; // default would appear to be 2 
    3232     
     
    103103        double per_vol = 100.0/(double)num_volumes; 
    104104         
     105        //JavaRDD<String> json_list_data_rp = json_list_data.repartition((int)(num_volumes/100)); 
     106 
    105107        DoubleAccumulator progress_accum = jsc.sc().doubleAccumulator("Progress Percent"); 
    106108         
     
    147149        double per_vol = 100.0/(double)num_volumes; 
    148150         
     151        //JavaRDD<String> json_list_data_rp = json_list_data.repartition((int)(num_volumes/100)); 
     152 
    149153        DoubleAccumulator per_vol_progress_accum = jsc.sc().doubleAccumulator("Per Volume Progress Percent"); 
    150154