Changeset 31089 for other-projects/hathitrust/wcsa/extracted-features-solr/trunk

Timestamp:

2016-11-10T16:13:12+13:00 (7 years ago)

Author:

davidb

Message:

Change in way the JSON file is read in. Motivation was a out-of-memory error, but now looks like it might not have been needed. That said, the JSON files are all done as single lines, so reading in the file constantly checking for a new-line char that will never appear is probably about the same as the new explicity character by character one

File:

• other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/JSONClusterFileIO.java (modified) (1 diff)

other-projects/hathitrust/wcsa/extracted-features-solr/trunk/solr-ingest/src/main/java/org/hathitrust/extractedfeatures/JSONClusterFileIO.java

@@ -16 +16 @@
             String str;
             BufferedReader br = ClusterFileIO.getBufferedReaderForCompressedFile(filename);
+
+            int cp;
+            while ((cp = br.read()) != -1) {
+                sb.append((char) cp);
+            }
+
+            /*
             while ((str = br.readLine()) != null) {
                 sb.append(str);
             }
+            */
     
             br.close();