Changeset 31502

Show
Ignore:
Timestamp:
13.03.2017 14:16:15 (3 years ago)
Author:
davidb
Message:

Comment out section, useful for controlling a smaller run

Files:
1 modified

Legend:

Unmodified
Added
Removed
  • other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java

    r31451 r31502  
    107107         
    108108        JavaPairRDD<Text, Text> input_pair_rdd = jsc.sequenceFile(packed_sequence_path, Text.class, Text.class); 
    109      
     109        //JavaPairRDD<String, String> input_pair_rdd = jsc.wholeTextFiles(packed_sequence_path); 
     110 
     111        //JavaPairRDD<Text, Text> input_pair_sampled_rdd = input_pair_rdd.sample(false,0.5,42); 
     112 
     113        //JavaRDD<Text> json_text_rdd = input_pair_sampled_rdd.map(item -> item._2); 
     114        //JavaRDD<Text> json_text_rdd = input_pair_rdd.map(item -> new Text(item._2)); 
    110115        JavaRDD<Text> json_text_rdd = input_pair_rdd.map(item -> item._2); 
    111116