Changeset 31272 for other-projects/hathitrust/wcsa/extracted-features-solr
- Timestamp:
- 2017-01-04T09:37:56+13:00 (7 years ago)
- Location:
- other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures
- Files:
-
- 2 edited
Legend:
- Unmodified
- Added
- Removed
-
other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/JSONClusterFileIO.java
r31089 r31272 14 14 StringBuilder sb = new StringBuilder(); 15 15 16 String str;17 16 BufferedReader br = ClusterFileIO.getBufferedReaderForCompressedFile(filename); 18 17 … … 23 22 24 23 /* 24 String str; 25 25 while ((str = br.readLine()) != null) { 26 26 sb.append(str); -
other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForLangCount.java
r31271 r31272 11 11 import org.apache.spark.api.java.function.Function2; 12 12 import org.apache.spark.api.java.function.PairFunction; 13 import org.apache.spark.storage.StorageLevel; 13 14 import org.apache.spark.util.DoubleAccumulator; 14 15 import scala.Tuple2; … … 91 92 strict_file_io); 92 93 JavaRDD<String> lang_list = json_list_data_rp.flatMap(paged_solr_langfreq_flatmap); 94 lang_list.persist(StorageLevel.MEMORY_AND_DISK()); 93 95 lang_list.setName("lang-stream"); 94 96
Note:
See TracChangeset
for help on using the changeset viewer.