Changeset 33727 for main/trunk

Timestamp:

2019-11-28T22:17:15+13:00 (4 years ago)

Author:

ak19

Message:

Experimental encoding related bugfix to GLI. In GLI, meta assigned at file level to filenames with non-ascii chars were not sticking to the file, because repeated entries were written out to metadata.xml under 2 variants of the filename but never loaded back into GLI again. This problem was not apparent with the old FilenameEncodings test set of docs or Kathy's complex test case of Russian filenames gathered into a folder structure. In the latter case, meta was assigned at folder level and so the regex to match was .* which is just ASCII. Neither test document sets were tested with meta assigned at file level. I can't now remember whether we tested today whether assigning file level meta to docs in the FilenameEncodings test set worked or not, but if it did, maybe that was because the special characters were not too complex and just Latin-1 or Win codepage 850 (like 1252) for the docs where meta was assigned. In any case, with test docs where filenames had A-macrons in them, the problem showed up and also in the Russian test set if meta got assigned at doc level. GLI was correctly saving filenames that had meta into metadata.xml as hex-encoded filenames the first time around. It just wasn't comparing them to hex values on subsequent times, and thus not finding a match. Method FilenameEncoding.fileNameToHex() introduced to fix this (experimental, need to run some questions by Dr Bainbridge). For all current tests, this appears to have fixed it. However, there must be somewhere else that ex.meta is being loaded in, as that is still not appearing for specially named files.

Location:

main/trunk/gli/src/org/greenstone/gatherer/metadata

Files:

• FilenameEncoding.java (modified) (1 diff)
• MetadataXMLFile.java (modified) (4 diffs)
• MetadataXMLFileManager.java (modified) (2 diffs)

main/trunk/gli/src/org/greenstone/gatherer/metadata/FilenameEncoding.java

@@ -359 +359 @@
     }
 
+    /** Attempting to produce the equivalent method fileToURLEncoding() above, but taking a String as input parameter */
+     public static String fileNameToHex(String filename) {
+        /*String filename_url_encoded = "";
+        try {
+            URI filename_uri = new URI(filename);
+            String filename_ascii = filename_uri.toASCIIString();
+            String filename_raw_bytes = URLDecoder.decode(filename_ascii,"ISO-8859-1");
+            filename_url_encoded = iso_8859_1_filename_to_url_encoded(filename_raw_bytes); 
+            return filename_url_encoded;
+        } catch (Exception e) {
+            e.printStackTrace();
+            // Give up trying to convert
+            filename_url_encoded = filename; 
+        }
+        return filename_url_encoded;
+        */
+        
+        String hexFilename = "";
+        for(int i = 0; i < filename.length(); i++) {
+            int charCode = filename.codePointAt(i); // unicode codepoint / ASCII code
+            
+            // ASCII table: https://cdn.sparkfun.com/assets/home_page_posts/2/1/2/1/ascii_table_black.png
+            // If the unicode character code pt is less than the ASCII code for space and greater than for tilda, let's display the char in hex (x0000 format)
+            if((charCode >= 20 && charCode <= 126) || charCode == 9 || charCode == 10 || charCode == 13) { // space, tilda, TAB, LF, CR are printable, leave them in for XML element printing
+                hexFilename += filename.charAt(i);
+            } else {
+                hexFilename += "&#x" + String.format("%x", charCode).toUpperCase() + ";"; // looks like: "&#x[up-to-4-hexdigits-in-UPPERCASE];"
+            }
+        }
+          
+        return hexFilename;
+     }
+    
     
     // For unicode codepoints see:

main/trunk/gli/src/org/greenstone/gatherer/metadata/MetadataXMLFile.java

@@ -100 +100 @@
     }
 
+    //System.err.println("MetadataXMLFile.addMetadata() Adding meta for file regexp: "
+    //  + file_path_regexp + " - " + org.greenstone.gatherer.util.Utility.debugUnicodeString(file_path_regexp));
+    
     // Find the appropriate FileSet element for this file
     Element appropriate_fileset_element = null;
@@ -259 +262 @@
         String current_filename_element_value = XMLTools.getElementTextValue(current_filename_element);
 
+        String regexed_file_relative_path = MetadataTools.getRegularExpressionThatMatchesFilePath(file_relative_path);
+        //System.err.println("Looking in meta.xml for regexed version of filename: " + regexed_file_relative_path);
+    
         // Does this fileset specify metadata for one file only?
         is_one_file_only_metadata = true;
@@ -266 +272 @@
         }
 
+        String current_filename_element_value_hex = FilenameEncoding.fileNameToHex(current_filename_element_value);
+        
         // This fileset specifies metadata for the file
-        if (file_relative_path.matches(current_filename_element_value)) {
+        // MetadataXMLFile.addMetadata(CollectionTreeNode, ArrayList) stored filename in uppercase hex, so need to compare with the same
+        if (file_relative_path.matches(current_filename_element_value_hex)) { //if (file_relative_path.matches(current_filename_element_value)) {
+            //System.err.println("Found a match in meta.xml for file name: " + regexed_file_relative_path);
             current_fileset_matches = true;
             if (!file_relative_path.equals("") && current_filename_element_value.equals(DIRECTORY_FILENAME)) {
@@ -273 +283 @@
             }
             break;
-        }
-
+        }       
+        
         // This fileset specifies metadata for the folder the file is in
-        if (file_relative_path.startsWith(current_filename_element_value + FilenameEncoding.URL_FILE_SEPARATOR)) {
+        if (regexed_file_relative_path.startsWith(current_filename_element_value + FilenameEncoding.URL_FILE_SEPARATOR)) {
             current_fileset_matches = true;
             folder_metadata_inherited_from = new File(metadata_xml_file_directory, current_filename_element_value);

main/trunk/gli/src/org/greenstone/gatherer/metadata/MetadataXMLFileManager.java

@@ -37 +37 @@
 import org.greenstone.gatherer.util.XMLTools;
 
+import org.greenstone.gatherer.util.Utility;
+
 
 /** This class is a static class that manages the metadata.xml files */
@@ -78 +80 @@
     for (int i = 0; i < file_nodes.length; i++) {
         File current_file = file_nodes[i].getFile();
-        DebugStream.println("Adding metadata to " + current_file.getAbsolutePath());
+        DebugStream.println("Adding metadata to " + current_file.getAbsolutePath() + " - hex: " + Utility.debugUnicodeString(current_file.getAbsolutePath()));      
 
         // Find which metadata.xml file needs editing