Changeset 16240

Timestamp:

2008-06-27T16:36:27+12:00 (15 years ago)

Author:

mdewsnip

Message:

Added "CREATE INDEX" on document_metadata(docOID) to hugely increase the speed at which large amounts of data get imported -- the nshore-newspapers collection reduced from 879 minutes to less than 5 minutes!

File:

• gsdl/trunk/perllib/dbutil.pm (modified) (1 diff)

gsdl/trunk/perllib/dbutil.pm

@@ -251 +251 @@
     print $infodb_handle "CREATE TABLE IF NOT EXISTS data (key TEXT PRIMARY KEY, value TEXT);\n";
     print $infodb_handle "CREATE TABLE IF NOT EXISTS document_metadata (id INTEGER PRIMARY KEY, docOID TEXT, element TEXT, value TEXT);\n";
+
+    # This is crucial for efficiency when importing large amounts of data
+    print $infodb_handle "CREATE INDEX IF NOT EXISTS dmd ON document_metadata(docOID);\n";
 
     # This is very important for efficiency, otherwise each command will be actioned one at a time