Changeset 31757

Timestamp:

2017-06-28T20:39:16+12:00 (7 years ago)

Author:

ak19

Message:

Fixed the earlier problems, which, it turned out, had to do with the order in which the superclass plugin instances were merged to create the subclass plugin. I can run the PDFBox command via UnknownConverterPlugin now at last, but while the text does go into doc.xml, previewing doesn't give me access to the HTML file. Not sure if this is requires fixing up a Formatting statement, or I'm not doing enough in the plugin.

Location:

main/trunk/greenstone2/perllib

Files:

• plugins/UnknownConverterPlugin.pm.bak (modified) (9 diffs)
• strings.properties (modified) (1 diff)

main/trunk/greenstone2/perllib/plugins/UnknownConverterPlugin.pm.bak

@@ -45 +45 @@
 # At present, a file or folder of files is assumed.
 # Need to look in there for files with extension process_ext.
+# Support html_multi as output? Then a folder of html files is generated per document? OR Flag that indicates whether an html file + associated folder (such as of images) gets generated. And name of assoc folder. Such output gets generated for instance when a doc file is replaced by its html version.
 
 sub BEGIN {
@@ -78 +79 @@
     'desc' => "{UnknownConverterPlugin.output_file_or_dir_name}",
     'type' => "string",
-    'reqd' => "yes",
+    'reqd' => "no",
     'deft' => "" } ];
 
@@ -96 +97 @@
     push(@{$hashArgOptLists->{"OptList"}},$options);
 
+    my $unknown_converter_self = new UnknownPlugin($pluginlist, $inputargs, $hashArgOptLists);
     my $cbf_self = new ConvertBinaryFile($pluginlist, $inputargs, $hashArgOptLists);
-    my $unknown_converter_self = new UnknownPlugin($pluginlist, $inputargs, $hashArgOptLists);
-    my $self = BaseImporter::merge_inheritance($cbf_self, $unknown_converter_self);
+    
+    # Need to feed the superclass plugins to merge_inheritance() below in the order that the
+    # superclass plugins were declared in the ISA listing earlier in this file:
+    my $self = BaseImporter::merge_inheritance($unknown_converter_self, $cbf_self);
 
     $self = bless $self, $class;
 
-$self->{'convert_to'} = "text"; # why do I have to set a value for convert_to here, when a default's already set at start of this file???????
+my $outhandle = $self->{'outhandle'};
+    print STDERR "\n\n**** convert_to is |" . $self->{'convert_to'} . "|\n\n";
+    if(!defined $self->{'convert_to'}) {
+    $self->{'convert_to'} = "text"; # why do I have to set a value for convert_to here, when a default's already set at the start of this file???????
+    }
 
     # Convert_To set up, including secondary_plugins for processing the text or html generated
@@ -173 +181 @@
     my $plugin_name = $self->{'plugin_type'}; # inherited from BaseImporter
 
+    #### COPIED FROM ConvertBinaryFile::tmp_area_convert_file()
+    my $outhandle = $self->{'outhandle'};
+    my $convert_to = $self->{'convert_to'};
+    my $failhandle = $self->{'failhandle'};
+    my $convert_to_ext = $self->{'convert_to_ext'}; #set by ConvertBinaryFile::set_standard_convert_settings()
+    
+
+    my $upgraded_input_filename = &util::upgrade_if_dos_filename($input_filename);
+
+    # derive tmp filename from input filename
+    my ($tailname, $dirname, $suffix)
+    = &File::Basename::fileparse($upgraded_input_filename, "\\.[^\\.]+\$");
+
+    # softlink to collection tmp dir
+    my $tmp_dirname = &util::get_timestamped_tmp_folder();
+    if (defined $tmp_dirname) {
+    $self->{'tmp_dir'} = $tmp_dirname;
+    } else {
+    $tmp_dirname = $dirname;
+    }
+    
+#    # convert to utf-8 otherwise we have problems with the doc.xml file later on
+#    my $utf8_tailname = (&unicode::check_is_utf8($tailname)) ? $tailname : $self->filepath_to_utf8($tailname);
+
+    # make sure filename to be used can be stored OK in a UTF-8 compliant doc.xml file
+     my $utf8_tailname = &unicode::raw_filename_to_utf8_url_encoded($tailname);
+
+
+    # URLEncode this since htmls with images where the html filename is utf8 don't seem
+    # to work on Windows (IE or Firefox), as browsers are looking for filesystem-encoded
+    # files on the filesystem.
+    $utf8_tailname = &util::rename_file($utf8_tailname, $self->{'file_rename_method'}, "without_suffix");
+
+    my $lc_suffix = lc($suffix);
+    my $tmp_filename = &FileUtils::filenameConcatenate($tmp_dirname, "$utf8_tailname$lc_suffix");
+    
+    # If gsdl is remote, we're given relative path to input file, of the form import/utf8_tailname.suffix
+    # But we can't softlink to relative paths. Therefore, we need to ensure that
+    # the input_filename is the absolute path, see http://perldoc.perl.org/File/Spec.html
+    my $ensure_path_absolute = 1; # true
+    &FileUtils::softLink($input_filename, $tmp_filename, $ensure_path_absolute);
+    my $verbosity = $self->{'verbosity'};
+    if ($verbosity > 0) {
+    print $outhandle "Converting $tailname$suffix to $convert_to format with extension $convert_to_ext\n";
+    }
+
+    my $errlog = &FileUtils::filenameConcatenate($tmp_dirname, "err.log");
+    
+ 
+    my $output_type=$self->{'convert_to'};
+
+    # store the *actual* output type and return the output filename
+    # it's possible we requested conversion to html, but only to text succeeded
+    #$self->{'convert_to_ext'} = $output_type;
+    if ($output_type =~ /html/i) {
+    $self->{'converted_to'} = "HTML";
+    } elsif ($output_type =~ /te?xt/i) {
+    $self->{'converted_to'} = "Text";
+    } elsif ($output_type =~ /item/i || $output_type =~ /^pagedimg/){
+    $self->{'converted_to'} = "PagedImage";
+    }
+    
+    my $output_filename = $tmp_filename;
+    my $output_dirname;
+    if ($output_type =~ /item/i || $output_type =~ /^pagedimg/) {
+    # running under windows
+    if ($ENV{'GSDLOS'} =~ /^windows$/i) {
+        $output_dirname = $tmp_dirname . "\\$utf8_tailname\\" . $utf8_tailname;
+    } else {
+        $output_dirname = $tmp_dirname . "\/$utf8_tailname\/" . $utf8_tailname;
+    }
+    $output_filename .= ".item";
+    } else {
+    $output_filename =~ s/$lc_suffix$/.$output_type/;
+    }
+
+    #### END COPIED FROM ConvertBinaryFile::tmp_area_convert_file()
+
+    # Execute the conversion command and get the type of the result,
+    # making sure the converter gives us the appropriate output type
+
     # On Linux: if the program isn't installed, $? tends to come back with 127, in any case neither 0 nor 1.
     # On Windows: echo %ERRORLEVEL% ends up as 9009 if the program is not installed.
@@ -178 +267 @@
     # should produce either a text file or output to stdout.
 
-    my $outhandle=$self->{'outhandle'};
-
     my $cmd = $self->{'exec_cmd'};
     if(!$cmd) { # empty string for instance
-    print $outhandle "$plugin_name Conversion error: invalid cmd $cmd\n";
+    print $outhandle "$plugin_name Conversion error: a command to execute is required, cmd provided is |$cmd|\n";
     return "";
     }
 
-    # replace occurrences of '*' placeholder in cmd string with input filename
-    my ($tailname, $dir, $suffix) = &File::Basename::fileparse($input_filename, "\\.[^\\.]+\$");
-    $cmd =~ s/\*/$tailname/g;
-    print STDERR "@@@@ $plugin_name: executing conversion cmd $cmd\n";
+    # HARDCODING CMD FOR NOW
+    #$cmd ="/Scratch/ak19/gs3-svn-15Nov2016/packages/jre/bin/java -cp \"/Scratch/ak19/gs3-svn-15Nov2016/gs2build/ext/pdf-box/lib/java/pdfbox-app.jar\" -Dline.separator=\"<br />\" org.apache.pdfbox.ExtractText -html \"/Scratch/ak19/tutorial_sample_files/pdfbox/A9-access-best-practices.pdf\" \"/Scratch/ak19/gs3-svn-15Nov2016/pdf-tmp/1.html\"";
+
+    #$cmd ="/Scratch/ak19/gs3-svn-15Nov2016/packages/jre/bin/java -cp \"/Scratch/ak19/gs3-svn-15Nov2016/gs2build/ext/pdf-box/lib/java/pdfbox-app.jar\" -Dline.separator=\"<br />\" org.apache.pdfbox.ExtractText -html INPUT_FILE OUTPUT";
+
+    # replace occurrences of placeholders in cmd string
+    #$cmd =~ s@\"@\\"@g;
+    $cmd =~ s@INPUT_FILE@\"$input_filename\"@g;
+    if(defined $output_dirname) {
+    $cmd =~ s@OUTPUT@\"$output_dirname\"@g;
+    } else {
+    $cmd =~ s@OUTPUT@\"$output_filename\"@g;
+    }   
+
+    print STDERR "@@@@ $plugin_name: executing conversion cmd \n|$cmd|\n";
+    print STDERR "   on infile |$input_filename|\n";
+    print STDERR "   to produce expected $output_filename\n";
     my $status = system($cmd);
 
@@ -202 +302 @@
     }
 
-    my $output_file_or_dir = $self->{'output_file_or_dir_name'};
-    if (!-e $output_file_or_dir) {
-    print $outhandle "$plugin_name Conversion error: Output file/dir $output_file_or_dir doesn't exist\n";
+    # remove symbolic link to original file
+    &FileUtils::removeFiles($tmp_filename);
+
+
+    if(defined $output_dirname && -d $output_dirname) {
+    print $outhandle "$plugin_name Conversion error: Output directory $output_dirname doesn't exist\n";
     return "";
     }
+    elsif (!-e $output_filename) {
+    print $outhandle "$plugin_name Conversion error: Output file $output_filename doesn't exist\n";
+    return "";
+    }
 
     # else, conversion success
     
     # if multiple images were generated by running the conversion
-    if ($self->{'convert_to'} eq "pagedimg") {
-    my $item_filename = $self->generate_item_file($output_file_or_dir);
-    return $item_filename;
-    }
-
-    return $output_file_or_dir;
+    if ($self->{'convert_to'} =~ /^pagedimg/) {
+    my $item_filename = $self->generate_item_file($output_filename); #my $item_filename = $self->generate_item_file($output_file_or_dir);
+
+    if (!-e $item_filename) {
+        print $outhandle "$plugin_name Conversion error: Item file $item_filename was not generated\n";
+        return "";
+    }   
+    $output_filename = $item_filename;
+    }
+
+    $self->{'output_dirname'} = $output_dirname;
+    $self->{'output_filename'} = $output_filename;
+    
+    return $output_filename; #$output_file_or_dir;
 }
 
@@ -230 +345 @@
     return undef unless $self->can_process_this_file($filename_full_path);
     
-    my $output_file_or_dir = $self->{'output_file_or_dir_name'};
-    my $is_output_dir = (-d $output_file_or_dir) ? 1 : 0;
+    my $is_output_dir = (defined $self->{'output_dirname'}) ? 1 : 0;
 
     # we are only doing something special if we have a directory of html files
-    if (!$is_output_dir || $self->{'convert_to'} ne "html") {
+    #if ($is_output_dir || $self->{'convert_to'} ne "html") {
+    if ($self->{'convert_to'} ne "html_multi") {
     return $self->BaseImporter::read(@_); # no read in ConvertBinaryFile.pm
     }
@@ -320 +435 @@
     # deleted some commented out code here that exists in PowerPointPlugin
 
-    # for UnknownConverterPlugin, don't delete any temp files that the conversion may have created
-    # as we don't know where it was created
-    #$self->clean_up_after_doc_obj_processing();
+    # for UnknownConverterPlugin, don't delete any temp files that the conversion may have created?
+    # as we don't know where it was created. No. Now creating in tmp.
+    $self->clean_up_after_doc_obj_processing();
 
 
@@ -333 +448 @@
 sub read_into_doc_obj {
     my $self = shift (@_);
-    $self->ConvertBinaryFile::deinit(@_);
+    $self->ConvertBinaryFile::read_into_doc_obj(@_);
 }
 

main/trunk/greenstone2/perllib/strings.properties

@@ -1271 +1271 @@
 TextPlugin.title_sub:Substitution expression to modify string stored as Title. Used by, for example, PostScriptPlugin to remove "Page 1" etc from text used as the title.
 
-UnknownConverterPlugin.desc:If you have a custom conversion tool installed that you're able to run from the command line to convert from an unsupported document format to either text or HTML, provide that command to this Plugin and it will run the command for you, capturing the output for indexing by Greenstone, making your document searchable. Use * as placeholder for input file name, but specify suffix of file to be converted (and also of any output file generated, if a file and not dir of files is generated).
-
-UnknownConverterPlugin.exec_cmd:Command line command string to execute that will do the conversion.
+UnknownConverterPlugin.desc:If you have a custom conversion tool installed that you're able to run from the command line to convert from an unsupported document format to text, HTML or a series of images in jpg, png or gif form, then provide that command to this Plugin. It will then run the command for you, capturing the output for indexing by Greenstone, making any documents that aren't converted to images searchable. Set the process_extension to the suffix of files to be converted. Set convert_to to be the output format that the conversion command will generate, which will determine the output file's suffix. Use INPUT_FILE and OUTPUT as place holders in the command, which Greenstone will replace. It will pass in the full path to each file that matches the process_extension suffix in turn as INPUT_FILE. OUTPUT will be replaced with a path in the temporary folder of the output file with suffix determined by the value of convert_to. If convert_to is a pagedimg type, Greenstone sets OUTPUT to be a directory to contain the expected files and will create an item file collating the parts of the document.
+
+UnknownConverterPlugin.exec_cmd:Command line command string to execute that will do the conversion. Quoted elements need to have the quotes escaped with a backslash to preserve them.
 
 UnknownConverterPlugin.output_file_or_dir_name: Full pathname of the output file or of the directory (of output files) that get generated by the conversion