Changeset 31007 for other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java
- Timestamp:
- 2016-10-31T09:35:06+13:00 (7 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
other-projects/hathitrust/solr-extracted-features/trunk/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java
r31006 r31007 6 6 import org.apache.spark.api.java.*; 7 7 import org.apache.spark.util.DoubleAccumulator; 8 import org.hathitrust.extractedfeatures.P agedJSON;8 import org.hathitrust.extractedfeatures.PerPageJSONFlatmap; 9 9 import org.json.JSONObject; 10 10 import org.apache.spark.SparkConf; … … 105 105 DoubleAccumulator progress_accum = jsc.sc().doubleAccumulator("Progress Percent"); 106 106 107 P agedJSON paged_json = new PagedJSON(_input_dir,_solr_url,_output_dir,_verbosity, progress_accum,per_vol);107 PerPageJSONFlatmap paged_json = new PerPageJSONFlatmap(_input_dir,_solr_url,_output_dir,_verbosity, progress_accum,per_vol); 108 108 JavaRDD<JSONObject> json_per_page_ids = json_list_data.flatMap(paged_json).cache(); 109 109 110 P agedJSONForeach paged_json_foreach = new PagedJSONForeach(_input_dir,_solr_url,_output_dir,_verbosity, progress_accum,per_vol);110 PerPageJSONForeach paged_json_foreach = new PerPageJSONForeach(_input_dir,_solr_url,_output_dir,_verbosity, progress_accum,per_vol); 111 111 json_per_page_ids.foreach(paged_json_foreach); 112 112
Note:
See TracChangeset
for help on using the changeset viewer.