- Timestamp:
- 2018-11-09T22:33:51+13:00 (5 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
main/trunk/greenstone2/perllib/plugins/GreenstoneSQLPlugin.pm
r32592 r32595 38 38 39 39 40 # TODO:41 # - Run TODOs here, in Plugout and in gsmysql.pm by Dr Bainbridge.42 # - Have not yet tested writing out just meta or just fulltxt to sql db and reading just that43 # back in from the sql db while the remainder is to be read back in from the docsql .xml files.44 45 # + TODO: Add public instructions on using this plugin and its plugout: start with installing mysql binary, changing pwd, running the server (and the client against it for checking: basic cmds like create and drop). Then discuss db name, table names (per coll), db cols and col types, and how the plugout and plugin work.46 # Discuss the plugin/plugout parameters.47 48 # TODO, test on windows and mac.49 # Note: if parsing fails (e.g. using wrong plugout like GS XML plugout, which chokes on args intended for SQL plugout) then SQL plugin init would have already been called and done connection, but disconnect would not have been done because SQL plugin disconnect would not have been called upon parse failure.50 51 # DONE:52 # + TODO: For on cancel, add a SIGTERM handler or so to call end()53 # or to explicitly call gs_sql->close_connection if $gs_sql def54 #55 # + TODO: Incremental delete can't work until GSSQLPlugout has implemented build_mode = incremental56 # (instead of tossing away db on every build)57 # + Ask about docsql naming convention adopted to identify OID. Better way?58 # collection names -> table names: it seems hyphens not allowed. Changed to underscores.59 # + Startup parameters (except removeold/build_mode)60 # + how do we detect we're to do removeold during plugout in import.pl phase61 # + incremental building: where do we need to add code to delete rows from our sql table after62 # incrementally importing a coll with fewer docs (for instance)? What about deleted/modified meta?63 # + Ask if I can assume that all SQL dbs (not just MySQL) will preserve the order of inserted nodes64 # (sections) which in this case had made it easy to reconstruct the doc_obj in memory in the correct order.65 # YES: Otherwise for later db types (drivers), can set order by primary key column and then order by did column66 # + NOTTODO: when db is not running GLI is paralyzed -> can we set timeout on DBI connection attempt?67 # NOT A PROBLEM: Tested to find DBI connection attempt fails immediately when MySQL server not68 # running. The GLI "paralyzing" incident last time was not because of the gs sql connection code,69 # but because my computer was freezing on-and-off.70 # + "Courier" demo documents in lucene-sql collection: character (degree symbol) not preserved in title. Is this because we encode in utf8 when putting into db and reading back in?71 # Test doc with meta and text like macron in Maori text.72 # + TODO Q: During import, the GS SQL Plugin is called before the GS SQL Plugout with undesirable side73 # effect that if the db doesn't exist, gsmysql::use_db() fails, as it won't create db.74 # This got fixed when GSSQLPlugin stopped connecting on init().75 #76 #77 #+ TODO: deal with incremental vs removeold. If docs removed from import folder, then import step78 # won't delete it from archives but buildcol step will. Need to implement this with this database plugin or wherever the actual flow is.79 #80 # + TODO Q: is "reindex" = del from db + add to db?81 # - is this okay for reindexing, or will it need to modify existing values (update table)82 # - if it's okay, what does reindex need to accomplish (and how) if the OID changes because hash id produced is different?83 # - delete is accomplished in GS SQL Plugin, during buildcol.pl. When should reindexing take place?84 # during SQL plugout/import.pl or during plugin? If adding is done by GSSQLPlugout, does it need to85 # be reimplemented in GSSQLPlugin to support the adding portion of reindexing.86 #87 # INCREMENTAL REBUILDING IMPLEMENTED CORRECTLY AND WORKS:88 # Overriding plugins' remove_all() method covered removeold.89 # Overriding plugins' remove_one() method is all I needed to do for reindex and deletion90 # (incremental and non-incremental) to work.91 # but doing all this needed an overhaul of gsmysql.pm and its use by the GS SQL plugin and plugout.92 # - needed to correct plugin.pm::remove_some() to process all files93 # - and needed to correct GreenstoneSQLPlugin::close_document() to setOID() after all94 # All incremental import and buildcol worked after that:95 # - deleting files and running incr-import and incr-buildcol (= "incr delete"),96 # - deleting files and running incr-import and buildcol (="non-incr delete")97 # - modifying meta and doing an incr rebuild98 # - modifying fulltext and doing an incr rebuild99 # - renaming a file forces a reindex: doc is removed from db and added back in, due to remove_one()100 # - tested CSV file: adding some records, changing some records101 # + CSVPlugin test (collection csvsql)102 # + MetadataCSVPlugin test (modified collection sqltest to have metadata.csv refer to the103 # filenames of sqltest's documents)104 # + shared image test (collection shareimg): if 2 html files reference the same image, the docs105 # are indeed both reindexed if the image is modified (e.g. I replaced the image with another106 # of the same name) which in the GS SQL plugin/plugout case is that the 2 docs are deleted107 # and added in again.108 40 109 41 ######################################################################################## 110 42 111 # GreenstoneSQLPlugin inherits from GreenstoneXMLPlugin so that i t if meta or fulltext43 # GreenstoneSQLPlugin inherits from GreenstoneXMLPlugin so that if meta or fulltext 112 44 # is still written out to doc.xml (docsql .xml), that will be processed as usual, 113 45 # whereas GreenstoneSQLPlugin will process all the rest (full text and/or meta, whichever … … 145 77 'desc' => "{GreenstoneSQLPlug.rollbacl_on_cancel}" } ]; 146 78 147 # TODO: If subclassing gsmysql for other supporting databases and if they have different required79 # NOTE: If subclassing gsmysql for other supporting databases and if they have different required 148 80 # connection parameters, we can check how WordPlugin, upon detecting Word is installed, 149 81 # dynamically loads Word specific configuration options. … … 225 157 } 226 158 227 # GS SQL Plugin::init() (and deinit()) is called by import.pl and also by buildcol.pl228 # This means it connects and deconnects during import.pl as well. This is okay229 # as removeold, which should drop the collection tables, happens during the import phase,230 # calling GreenstoneSQLPlugin::and therefore also requires a db connection.231 # + TODO: Eventually can try moving get_gssql_instance into gsmysql.pm? That way both GS SQL Plugin232 # and Plugout would be using one connection during import.pl phase when both plugs exist.233 234 159 # Call init() not begin() because there can be multiple plugin passes and begin() called for 235 160 # each pass (one for doc level and another for section level indexing), whereas init() should 236 161 # be called before any and all passes. 237 162 # This way, we can connect to the SQL database once per buildcol run. 163 # Although now it doesn't matter, since gsmysql.pm uses the get_instance pattern to return a 164 # singleton db connection, regardless of the number of gsmysql objects instantiated and 165 # the number of connect() calls made on them. 238 166 sub init { 239 167 my ($self) = shift (@_); 240 ##print STDERR "@@@@@@@@@@ INIT CALLED\n";241 168 242 169 $self->SUPER::init(@_); # super (GreenstoneXMLPlugin) will not yet be trying to read from doc.xml (docsql .xml) files in init(). 243 170 244 ####################245 # print "@@@ SITE NAME: ". $self->{'site'} . "\n" if defined $self->{'site'};246 # print "@@@ COLL NAME: ". $ENV{'GSDLCOLLECTION'} . "\n";247 248 # print STDERR "@@@@ db_pwd: " . $self->{'db_client_pwd'} . "\n";249 # print STDERR "@@@@ user: " . $self->{'db_client_user'} . "\n";250 # print STDERR "@@@@ db_host: " . $self->{'db_host'} . "\n";251 # print STDERR "@@@@ db_driver: " . $self->{'db_driver'} . "\n";252 ####################253 171 254 172 # create gsmysql object. … … 287 205 # PrintError would already have displayed the warning message on load fail 288 206 # And on die() perl will call gsmysql destroy which will ensure a disconnect() from db 289 #$gs_sql->force_disconnect_from_db();290 207 die("Could not use db $db_name. Can't proceed.\n"); 291 208 } … … 297 214 298 215 299 # This method also runs on import.pl if gs_sql has a value. But we just want to run it on buildcol216 # This method also runs on import.pl if gs_sql has a value. 300 217 # Call deinit() not end() because there can be multiple plugin passes: 301 218 # one for doc level and another for section level indexing 302 219 # and deinit() should be called before all passes 303 220 # This way, we can close the SQL database once per buildcol run. 221 # Again, this doesn't matter because we gsmysql the ensures the connection 222 # is a singleton connection instance, which connects once and disconnects once per perl process. 304 223 sub deinit { 305 224 my ($self) = shift (@_); 306 307 ##print STDERR "@@@@@@@@@@ GreenstoneSQLPlugin::DEINIT CALLED\n";308 225 309 226 if($self->{'gs_sql'}) { … … 360 277 # but if a plugin does any stuff outside of creating doc obj, then it may need 361 278 # to clear something. 362 # remove the doc(s) denoted by oids from GS SQL db279 # In the case of GreenstoneSQL plugs: Remove the doc(s) denoted by oids from GS SQL db. 363 280 # This takes care of incremental deletes (docs marked D by ArchivesInfPlugin when building 364 # incrementally) as well as cases of "Non-icremental Delete", see ArchivesInfPlugin.pm 281 # incrementally) as well as cases of "Non-icremental Delete", see ArchivesInfPlugin.pm. 282 # As well as cases involving reindexing, which are implemented here as delete followed by add. 365 283 sub remove_one { 366 284 my $self = shift (@_); … … 379 297 380 298 my $gs_sql = $self->{'gs_sql'} || return 0; # couldn't make the connection or no db etc 381 382 print STDERR "*****************************\nAsked to remove_one oid\n***********************\n";383 print STDERR "Num oids: " . scalar (@$oids) . "\n";384 299 385 300 my $proc_mode = $self->{'process_mode'}; … … 431 346 # Since all passes need the doc_obj, all are read in from docsql + SQL db into the docobj in memory 432 347 433 # We should only ever get here during the buildcol.pl phase348 # We only ever get here or do any parsing of the docsql.xml file during the buildcol.pl phase. 434 349 # At the end of superclass GreenstoneXMLPlugin.pm's close_document() method, 435 350 # the doc_obj in memory is processed (indexed) and then made undef. … … 438 353 my $self = shift(@_); 439 354 440 ##print STDERR "XXXXXXXXX in SQLPlugin::close_doc()\n";441 442 355 my $gs_sql = $self->{'gs_sql'}; 443 356 … … 468 381 469 382 print $outhandle "----------SQL DB contains meta-----------\n" if $self->{'verbosity'} > 2; 470 # https://www.effectiveperlprogramming.com/2010/07/set-custom-dbi-error-handlers/471 383 472 384 foreach my $row (@$records) { 473 #print $outhandle "row: @$row\n";474 385 my ($primary_key, $did, $sid, $metaname, $metaval) = @$row; 475 386 … … 480 391 if $self->{'verbosity'} > 2; 481 392 482 # + TODO: we accessed the db in utf8 mode, so, we can call doc_obj->add_utf8_meta directly: 483 #$doc_obj->add_utf8_metadata($sid, $metaname, &docprint::unescape_text($metaval)); 484 485 # data stored unescaped in db: escaping only for html/xml files, not for txt files or db 393 # We're only dealing with utf8 data where docobj is concerned 394 # Data stored unescaped in db: escaping only for html/xml files, not for txt files or db 486 395 $doc_obj->add_utf8_metadata($sid, $metaname, $metaval); 487 396 } … … 510 419 print $outhandle "### did: $did, sid: |$sid|, fulltext: <TXT>\n" 511 420 if $self->{'verbosity'} > 2; 512 513 # TODO - pass by ref? 514 # + TODO: we accessed the db in utf8 mode, so, we can call doc_obj->add_utf8_text directly: 515 # data stored unescaped in db: escaping is only for html/xml files, not for txt files or db 516 #my $textref = &docprint::unescape_textref(\$text); 421 422 # We're only dealing with utf8 data where docobj is concerned 423 # Data stored unescaped in db: escaping is only for html/xml files, not for txt files or db 517 424 $doc_obj->add_utf8_textref($sid, \$text); 518 425 }
Note:
See TracChangeset
for help on using the changeset viewer.
