Changeset 28355 for main/trunk/greenstone2/perllib/plugins/PagedImagePlugin.pm

Timestamp:

2013-10-04T19:52:18+13:00 (11 years ago)

Author:

ak19

Message:

1. Now gsConvert.pl calls the new pptextract.vbs VBScript (which creates .item files and ppt slide.txt files in utf-8) instead of the older VB pptextract.exe executable which created .item and slide.txt files in windows default utf-16 LE. 2. PagedImagePlugin.pm::tidy_item_file now reads in the .item files in utf-8 mode, so that its strings are unicode aware. Substitutions are of unicode code points instead of byte sequences, since the strings in the file are now unicode aware.

File:

• main/trunk/greenstone2/perllib/plugins/PagedImagePlugin.pm (modified) (2 diffs)

main/trunk/greenstone2/perllib/plugins/PagedImagePlugin.pm

@@ -407 +407 @@
     my ($filename) = @_;
 
-    open (ITEMFILE, $filename) || die "couldn't open $filename\n";
+    open (ITEMFILE, "<:encoding(UTF-8)", $filename) || die "couldn't open $filename\n";
     my $backup_filename = "backup.item";
     open (BACKUP,">$backup_filename")|| die "couldn't write to $backup_filename\n";
+    binmode(BACKUP, ":utf8");
     my $line = "";
     $line = <ITEMFILE>;
-    $line =~ s/^\xEF\xBB\xBF//; # strip BOM
-    $line =~ s/\x0B+//ig;
+    #$line =~ s/^\xEF\xBB\xBF//; # strip BOM in text file read in as a sequence of bytes (not unicode aware strings)
+    $line =~ s/^\x{FEFF}//; # strip BOM in file opened *as UTF-8*. Strings in the file just read in are now unicode-aware,
+                            # this means the BOM is now a unicode codepoint instead of a byte sequence
+                            # See http://en.wikipedia.org/wiki/Byte_order_mark and http://perldoc.perl.org/5.14.0/perlunicode.html 
+    $line =~ s/\x{0B}+//ig; # removing \vt-vertical tabs using the unicode codepoint for \vt
     $line =~ s/&/&amp;/g;
     print BACKUP ($line);
     #Tidy up the item file some metadata title contains \vt-vertical tab
     while ($line = <ITEMFILE>) {
-    $line =~ s/\x0B+//ig;
+    $line =~ s/\x{0B}+//ig; # removing \vt-vertical tabs using the unicode codepoint for \vt
     $line =~ s/&/&amp;/g;
     print BACKUP ($line);
@@ -657 +661 @@
     my $self = shift (@_);
     my ($filename_full_path, $dir, $block_hash) = @_;
+
 
     open (ITEMFILE, $filename_full_path) || die "couldn't open $filename_full_path to work out which files to block\n";