Changeset 32089 for main/trunk/greenstone2/perllib/FileUtils.pm

Timestamp:

2017-12-08T19:16:12+13:00 (6 years ago)

Author:

ak19

Message:

1. Attempted fix by Kathy and me for Diego's problem of PDFBox's handling of a PDF. When it was set to convert_to_html, it built fine, but convert_to_text produced something that was invalid XML in doc.XML and build failed. Diego reasoned correctly that building ought to succeed in both cases if it succeeded in one case. Kathy found the correct fix for escaping the ampersand character (it wasn't & to & that I'd attempted, nor did using HTML::Entities' encode work either). 2. The fix needed to read and write files, so introducing readUTF8File() and writeUTF8File() into FileUtils.pm for reusability. Need to still contact John Thompson to ask him if and how these functions need to be modified to support parallel processing, for which FileUtils was written.

File:

• main/trunk/greenstone2/perllib/FileUtils.pm (modified) (2 diffs)

main/trunk/greenstone2/perllib/FileUtils.pm

@@ -60 +60 @@
 # util::soft_link()            => FileUtils::softLink()
 
+# Functions that have been added, but not by John Thompson, 
+# So the implementations don't support parallel processing yet, but they print a warning and the
+# correct implementation can be put into here. So that if all calls for reading and writing UTF8
+# file content go through here, then they will do the right thing when the functions are updated.
+#
+#  => FileUtils::readUTF8File()
+#  => FileUtils::writeUTF8File()
+#
+
 # Other functions in this file (perhaps some of these may have counterparts in util.pm too):
 
@@ -962 +971 @@
 ## readDirectory()
 
+## @function readUTF8File()
+#
+# read contents from a file containing UTF8.
+#
+# Parameter filename, the filepath to read from
+#
+sub readUTF8File
+{
+    my $filename = shift(@_);
+
+    print STDERR "@@@ Warning FileUtils::readFile() not yet implemented for parallel processing. Using regular version...\n";
+    
+    open(FIN,"<$filename") or die "FileUtils::readFile: Unable to open $filename for reading...ERROR: $!\n";
+
+    # decode the bytes in the file with UTF8 enc,
+    # to get unicode aware strings that represent utf8 chars
+    binmode(FIN,":utf8");
+    
+    my $contents;
+    # Read in the entire contents of the file in one hit
+    sysread(FIN, $contents, -s FIN);
+    close(FIN);
+    return $contents;    
+}
+## readUTF8File()
+
+## @function writeUTF8File()
+#
+# write UTF8 contents to a file.
+#
+# Parameter filename, the filepath to write to
+# Parameter contentRef, a *reference* to the contents to write out
+#
+sub writeUTF8File
+{
+    my ($filename, $contentRef) = @_;
+
+    print STDERR "@@@ Warning FileUtils::writeFile() not yet implemented for parallel processing. Using regular version...\n";
+
+    open(FOUT, ">$filename") or die "FileUtils::writeFile: Unable to open $filename for writing out contents...ERROR: $!\n";
+    # encode the unicode aware characters in the string as utf8
+    # before writing out the resulting bytes
+    binmode(FOUT,":utf8");
+    
+    print FOUT $$contentRef;
+    close(FOUT);
+}
+## writeUTF8File()
+
 ## @function removeFiles()
 #