Changeset 33466 for gs3-extensions/maori-lang-detection/src/org/greenstone/atea/NZTLDProcessor.java

Timestamp:

2019-09-12T21:37:39+12:00 (5 years ago)

Author:

ak19

Message:

1. WETProcessor.main() now processes a folder of *.warc.wet(.gz) files. Each file's WET records is written out into an individual file and put into either the keep folder or discard folder, based on amount of content (number lines and/or content-length). 2. Moved unzipFile() from NZTLDProcessor.java into new Utility.java class as a static method.

File:

• gs3-extensions/maori-lang-detection/src/org/greenstone/atea/NZTLDProcessor.java (modified) (3 diffs)

gs3-extensions/maori-lang-detection/src/org/greenstone/atea/NZTLDProcessor.java

@@ -169 +169 @@
         // don't have the WET file yet. Get it from the zip file, which we know we should have by now
         
-        boolean success = unzipFile(inZipFile, WETfile);
+        boolean success = Utility.unzipFile(inZipFile, WETfile);
+        log("Unzipped " + inZipFile + " to " + WETfile);
+        
         // whether we succeeded or not, get rid of the zipped file:
         if(!inZipFile.delete()) {
@@ -185 +187 @@
     }
 
+    /*
     // Run gunzip
     // To avoid making this linux specific, use Java to unzip, instead of running gunzip as process
@@ -216 +219 @@
     return true;
     }
+    */
     
     // wget will be launched from the specified directory, SITES_DIR