Changeset 32591 for main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm

Timestamp:

2018-11-09T19:01:04+13:00 (5 years ago)

Author:

ak19

Message:

1. gssql destructor DESTROY doesn't really do anything now, as DBI's destructor will ensure disconnection if not already disconnected. So we don't disconnect in gssql DESTROY, which we used to do but which would only happen on Cancel specifically as other gssql code would already disconnect on natural termination. 2. gssql no longer sets connection parameter fallbacks for those that have fallbacks, since the defaults for these parameters are set by the GS SQL Plugs' configuration options. 3. No need to (html) escape meta/full text when storing in DB or unescape when reading it out of DB, since DB is like a regular text file: can just put text in there. It's only html/xml files where text needs to be stored escaped. 4. Having committed doc.pm::add_utf8_textREF() method in previous commit, GreenstoneSQLPlugin can now pass the fulltxt by reference using this method when constructing the doc_obj from what's read in from the DB.

File:

• main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm (modified) (8 diffs)

main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm

@@ -43 +43 @@
 # back in from the sql db while the remainder is to be read back in from the docsql .xml files.
 
-# TODO: Add public instructions on using this plugin and its plugout: start with installing mysql binary, changing pwd, running the server (and the client against it for checking: basic cmds like create and drop). Then discuss db name, table names (per coll), db cols and col types, and how the plugout and plugin work.
+# + TODO: Add public instructions on using this plugin and its plugout: start with installing mysql binary, changing pwd, running the server (and the client against it for checking: basic cmds like create and drop). Then discuss db name, table names (per coll), db cols and col types, and how the plugout and plugin work.
 # Discuss the plugin/plugout parameters.
 
@@ -145 +145 @@
         'desc' => "{GreenstoneSQLPlug.rollbacl_on_cancel}" } ];
 
+# TODO: If subclassing gssql for other supporting databases and if they have different required
+# connection parameters, we can check how WordPlugin, upon detecting Word is installed,
+# dynamically loads Word specific configuration options.
 my $arguments =
     [ { 'name' => "process_exp",
@@ -161 +164 @@
     'type' => "enum",
     'list' => $rollback_on_cancel_list,
-    'deft' => "false", # TODO Q: what's the better default? If "true", any memory concerns?
+    'deft' => "false", # better default than true
     'reqd' => "no",
     'hiddengli' => "no"},
@@ -178 +181 @@
     'type' => "string",
     'deft' => "",
-    'reqd' => "no"}, # pwd required? NO.
+    'reqd' => "no"}, # pwd not required: can create mysql accounts without pwd
       { 'name' => "db_host", 
     'desc' => "{GreenstoneSQLPlug.db_host}",
     'type' => "string",
-    'deft' => "127.0.0.1",
+    'deft' => "127.0.0.1", # NOTE: make this int? No default for port, since it's not a required connection param
     'reqd' => "yes"},
       { 'name' => "db_port", 
@@ -226 +229 @@
 # as removeold, which should drop the collection tables, happens during the import phase,
 # calling GreenstoneSQLPlugin::and therefore also requires a db connection.
-# TODO: Eventually can try moving get_gssql_instance into gssql.pm? That way both GS SQL Plugin
+# + TODO: Eventually can try moving get_gssql_instance into gssql.pm? That way both GS SQL Plugin
 # and Plugout would be using one connection during import.pl phase when both plugs exist.
 
@@ -421 +424 @@
 }
 
-# TODO Q: Why are there 4 passes when we're only indexing at doc and section level (2 passes)? What's the dummy pass, why is there a pass for infodb?
+# There are multiple passes processing the document (see buildcol's mode parameter description):
+# - compressing the text which may be a dummy pass for lucene/solr, wherein they still want the
+# docobj for different purposes,
+# - the pass(es) for indexing, e.g. doc/didx and section/sidx level passes
+# - and an infodb pass for processing the classifiers. This pass too needs the docobj
+# Since all passes need the doc_obj, all are read in from docsql + SQL db into the docobj in memory
 
 # We should only ever get here during the buildcol.pl phase
@@ -472 +480 @@
         if $self->{'verbosity'} > 2;
         
-        # TODO:  we accessed the db in utf8 mode, so, we can call doc_obj->add_utf8_meta directly:
-        $doc_obj->add_utf8_metadata($sid, $metaname, &docprint::unescape_text($metaval));
+        # + TODO:  we accessed the db in utf8 mode, so, we can call doc_obj->add_utf8_meta directly:
+        #$doc_obj->add_utf8_metadata($sid, $metaname, &docprint::unescape_text($metaval));
+        
+        # data stored unescaped in db: escaping only for html/xml files, not for txt files or db
+        $doc_obj->add_utf8_metadata($sid, $metaname, $metaval);
     }
     print $outhandle "----------FIN READING DOC's META FROM SQL DB------------\n"
@@ -501 +512 @@
         
         # TODO - pass by ref?
-        # TODO: we accessed the db in utf8 mode, so, we can call doc_obj->add_utf8_text directly:
-        my $textref = &docprint::unescape_textref(\$text);
-        $doc_obj->add_utf8_text($sid, $$textref);
+        # + TODO: we accessed the db in utf8 mode, so, we can call doc_obj->add_utf8_text directly:
+        # data stored unescaped in db: escaping is only for html/xml files, not for txt files or db
+        #my $textref = &docprint::unescape_textref(\$text);
+        $doc_obj->add_utf8_textref($sid, \$text);
     }   
     print $outhandle "----------FIN READING DOC's TXT FROM SQL DB------------\n"