greenstone.org greenstone wiki greenstone trac planet greenstone

Changeset 16240

Show
Ignore:
Timestamp:
2008-06-27 16:36:27 (5 months ago)
Author:
mdewsnip
Message:

Added "CREATE INDEX" on document_metadata(docOID) to hugely increase the speed at which large amounts of data get imported -- the nshore-newspapers collection reduced from 879 minutes to less than 5 minutes!

Files:

Legend:

Unmodified
Added
Removed
Modified
Copied
Moved

  • gsdl/trunk/perllib/dbutil.pm

    r16226 r16240  
    251251    print $infodb_handle "CREATE TABLE IF NOT EXISTS data (key TEXT PRIMARY KEY, value TEXT);\n"; 
    252252    print $infodb_handle "CREATE TABLE IF NOT EXISTS document_metadata (id INTEGER PRIMARY KEY, docOID TEXT, element TEXT, value TEXT);\n"; 
     253 
     254    # This is crucial for efficiency when importing large amounts of data 
     255    print $infodb_handle "CREATE INDEX IF NOT EXISTS dmd ON document_metadata(docOID);\n"; 
    253256 
    254257    # This is very important for efficiency, otherwise each command will be actioned one at a time