Changeset 23387 for main/trunk/greenstone2/perllib/plugins/ReadTextFile.pm

Timestamp:

2010-12-06T13:15:10+13:00 (13 years ago)

Author:

davidb

Message:

Further changes to deal with documents that use different filename encodings on the file-system. Now sets UTF8URL metadata to perform the cross-document look up. Files stored in doc.pm as associated files are now always raw filenames (rather than potentially UTF8 encoded). Storing of filenames seen by HTMLPlug when scanning for files to block on is now done in Unicode aware strings rather than utf8 but unware strings.

File:

• main/trunk/greenstone2/perllib/plugins/ReadTextFile.pm (modified) (1 diff)

main/trunk/greenstone2/perllib/plugins/ReadTextFile.pm

@@ -307 +307 @@
     $reader->set_encoding($encoding);
     $reader->decode_text($raw_text,$textref);
+
+    # At this point $$textref is a binary byte string
+    # => turn it into a Unicode aware string, so full
+    # Unicode aware pattern matching can be used.
+    # For instance: 's/\x{0101}//g' or '[[:upper:]]'    
+    
+    $$textref = decode("utf8",$$textref);
     }
 }