Changeset 2799 for trunk/gsdl/perllib/plugins/ConvertToPlug.pm

Timestamp:

2001-10-12T12:45:17+13:00 (23 years ago)

Author:

sjboddie

Message:

Fixed a bug where Word documents containing non-ascii characters weren't
being handled correctly. The problem occurred because Greenstone doesn't
currently detect utf-8 encoded text automatically and the html produced by
wvWare is utf-8. The Greenstone build process was assuming it was
iso-8859-1 and so not importing the documents correctly. I've fixed it by
forcing the build process to assume html imported with WordPlug is
utf-8. This works but we should also include support for detecting utf-8
encoded documents sometime.

File:

• trunk/gsdl/perllib/plugins/ConvertToPlug.pm (modified) (2 diffs)

trunk/gsdl/perllib/plugins/ConvertToPlug.pm

@@ -258 +258 @@
     # Do encoding stuff
     my ($language, $encoding);
+
+    # WordPlug's wvWare will always produce html files encoded as utf-8
+    if ($plugin_name eq "WordPlug") {
+    $self->{'input_encoding'} = "utf8";
+    $self->{'extract_language'} = 1;
+    }
+
     if ($self->{'input_encoding'} eq "auto") {
         # use textcat to automatically work out the input encoding and language
@@ -264 +271 @@
         # use textcat to get language metadata
 
-        my ($language, $extracted_encoding) = $self->get_language_encoding ($conv_filename);
+    my ($extracted_encoding);
+        ($language, $extracted_encoding) = $self->get_language_encoding ($conv_filename);
         $encoding = $self->{'input_encoding'};
         if ($extracted_encoding ne $encoding && $self->{'verbosity'}) {