Changeset 33737 for main/trunk/gli/src/org/greenstone/gatherer/metadata/FilenameEncoding.java

Timestamp:

2019-12-02T20:03:57+13:00 (4 years ago)

Author:

ak19

Message:

A larger fix but not complete fix to the problem of attaching and retaining file level assigned meta to filenames containing non-ASCII characters. 1. Committing intermediate version of bugfix containing the idea suggested by Kathy to reuse the steps in fileToURLEncoding(File) for a String parameter as she felt that since the String represents a filename, a URI object should be instantiable on a String. Worked with some massaging. Can't yet get the new fileToURLEncoding(String) to work by calling fileToURLEncoding(File). So am committing the version of fileToURLEncoding(String) that is largely a copy of fileToURLEncoding(File), until I can get the simpler variant working. 2. The new method is called after each successful parseXML call from MetadataXMLFile, so that the decoded entities resulting from parseXML() are reencoded in the DOM. This allows us to retain the correct filenames originally mentioned in metadata.xml files, do proper comparisons against them to attach/modify further metdata and so that the correct values get written out again into metadata.xml files. 3. Still want to get simpler version of fileToURLEncoding(String) to work that reuses fileToURLEncoding(File). 4. Want to get ampersand and plus signs in filenames to work (+ signs in filenames are lost when filenames are converted to URL). 5. Still need to investigate the missing ex. metadata for filenames containing non-ASCII.

File:

• main/trunk/gli/src/org/greenstone/gatherer/metadata/FilenameEncoding.java (modified) (2 diffs)

main/trunk/gli/src/org/greenstone/gatherer/metadata/FilenameEncoding.java

@@ -364 +364 @@
             // ASCII table: https://cdn.sparkfun.com/assets/home_page_posts/2/1/2/1/ascii_table_black.png
             // If the unicode character code pt is less than the ASCII code for space and greater than for tilda, let's display the char in hex (x0000 format)
-            if((charCode >= 20 && charCode <= 126) || charCode == 9 || charCode == 10 || charCode == 13) { // space, tilda, TAB, LF, CR are printable, leave them in for XML element printing
+            if((charCode >= 20 && charCode <= 126) || charCode == 9 || charCode == 10 || charCode == 13 || charCode == 36 || charCode == 43) { // space, tilda, TAB, LF, CR are printable, leave them in for XML element printing. And spaces and plus signs (ASCII codes 36 and 43) need to be converted to hex too 
                 hexFilename += filename.charAt(i);
             } else {
@@ -374 +374 @@
      }
 
+    
+    // follows Dr Bainbridge's method below, but with a String parameter instead of a file parameter
+    public static String filenameToURLEncoding(String filename) {
+        if(!MULTIPLE_FILENAME_ENCODINGS_SUPPORTED) {
+            return filename;
+        }
+
+         // Can't create a URI out of a filename containing spaces. Spaces must be encoded as %20
+        String filename_url_encoded = filename.replace(" ", "%20");
+        //filename_url_encoded = filename_url_encoded.replace("&", "%26"); // &'s ASCII code is 36 in decimal, and 26 in hex
+        //filename_url_encoded = filename_url_encoded.replace("+", "%2B"); // +'s ASCII code is 43 decimal, 2b in hex, 2B when uppercased
+
+        try {
+            URI filename_uri = new URI(filename_url_encoded);
+            // The trick:
+            //  1. toASCIIString() will %xx encode values > 127
+            //  2. Decode the result to "ISO-8859-1"
+            //  3. URL encode the bytes to string
+            
+            // Step 2 forces the string to be 8-bit values.  It
+            // doesn't matter if the starting raw filename was *not*
+            // in the ISO-8859-1 encoding, the effect is to ensure
+            // we have an 8-bit byte string that (numerically)
+            // captures the right value.  These numerical values are
+            // then used to determine how to URL encode it
+            
+            String filename_ascii = filename_uri.toASCIIString();
+            //filename_ascii = filename_ascii.replace("&", "%26"); // &'s ASCII code is 36 in decimal, and 26 in hex
+            //filename_ascii = filename_ascii.replace("+", "%2B"); // +'s ASCII code is 43 decimal, 2b in hex, 2B when uppercased
+            String filename_raw_bytes = URLDecoder.decode(filename_ascii,"ISO-8859-1");
+            filename_url_encoded = iso_8859_1_filename_to_url_encoded(filename_raw_bytes);
+            
+            // DEALING WITH & and + in filenames: NOT WORKING YET
+            //if(filename_url_encoded.contains("&amp;")) {
+            //  filename_url_encoded = filename_url_encoded.replace("&amp;", "%36amp;");
+            //} else if(filename_url_encoded.contains("&")) {
+            //  filename_url_encoded = filename_url_encoded.replace("&", "%36");
+            //}
+            
+        }
+        catch (Exception e) {
+            e.printStackTrace();
+            // Give up trying to convert
+            filename_url_encoded = filename; 
+        }
+        return filename_url_encoded;
+    }
+    
+    
+    // follows Dr Bainbridge's method below, but with a String parameter instead of a file parameter    
+    public static String _filenameToURLEncoding(String filename) {
+        if(!MULTIPLE_FILENAME_ENCODINGS_SUPPORTED) { // on a UTF-8 file system, DO NOT do the stuff below, just return input param
+            return filename;
+        }
+        
+        File file = new File (filename);
+        return fileToURLEncoding(file);
+    }       
+
+    
     // Dr Bainbridge's methods
     /* On Linux machines that are set to using an ISO-8859 (Latin) type encoding,