Changeset 31597 for other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java

Timestamp:

2017-04-11T23:41:07+12:00 (7 years ago)

Author:

davidb

Message:

Additional _s and _ss fields to help with faceting. Temporarily commented out the full-text page part.

File:

• other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java (modified) (1 diff)

other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/ProcessForSolrIngest.java

@@ -124 +124 @@
                                                        solr_endpoints,_output_dir,_verbosity,
                                                        icu_tokenize,strict_file_io);
-
         
         JavaRDD<Integer> per_volume_page_count = json_text_rdd.map(per_vol_json);
         
-        Integer num_page_ids = per_volume_page_count.reduce((a, b) -> a + b);
-        
-        System.out.println("");
-        System.out.println("############");
-        System.out.println("# Number of page ids: " + num_page_ids);
+        //Integer num_page_ids = per_volume_page_count.reduce((a, b) -> a + b);
+        long num_vol_ids = per_volume_page_count.count();
+        
+        System.out.println("");
+        System.out.println("############");
+        //System.out.println("# Number of page ids: " + num_page_ids);
+        System.out.println("# Number of volume ids: " + num_vol_ids);
         System.out.println("############");
         System.out.println("");