Changeset 27604 for other-projects/nightly-tasks/diffcol/trunk/diffcol/gdbdiff.pm

Timestamp:

2013-06-12T18:43:04+12:00 (11 years ago)

Author:

ak19

Message:

Fixing up diffcol process so it works better. Current state finds no errors in Small-HTML model-collection. 1. Better handling of gdb database (and ignores .idh) by filtering out fields that are expected to differ such as date before doing the diff. Handles archiveinf-doc.gdb and -src.gdb files and with the sort flag Dr Bainbridge added to db2text and the sorting of keys in perllib/dbutil/gdbmtxtgz, the ordering of keys in the database is no longer affecting the outcome. 2. Better handling of doc.xml files. Once more date fields that will differ are filtered out before performing the diff. EarliestDatestamp file is ignored. 3. The task script now ensures that model-collect is up to date with the svn version when about to perform the diff col testing.

File:

• other-projects/nightly-tasks/diffcol/trunk/diffcol/gdbdiff.pm (modified) (1 diff)

other-projects/nightly-tasks/diffcol/trunk/diffcol/gdbdiff.pm

@@ -32 +32 @@
 sub test_gdb
 {
-    my ($full_modeldb, $full_testdb) = @_;
+    my ($full_modeldb, $full_testdb,$strColName) = @_;
 
 
    # print "Now is testing database\n";
     
-    my $model_cmd = "db2txt $full_modeldb 2>&1";
-    my $test_cmd  = "db2txt $full_testdb 2>&1";
+    # need to sort text output of both test and model col database files, to normalise them for the comparison
+    # the -sort option to db2txt was added specifically to support diffcol
+    my $model_cmd = "db2txt -sort $full_modeldb 2>&1";
+    my $test_cmd  = "db2txt -sort $full_testdb 2>&1";
 
     my $model_text = readin_gdb($model_cmd);
     my $test_text = readin_gdb($test_cmd);
+
+
+    # filter out the fields that can be ignored in the two database files
+    my $ignore_line_re = "\n<(lastmodified|lastmodifieddate|oailastmodified|oailastmodifieddate)>([^\n])*";
+    $model_text =~ s/$ignore_line_re//g;
+    $test_text =~ s/$ignore_line_re//g;
+
+
+    # ignore absolute path prefixes in modelcol and testcol (necessary for archiveinf-doc and -src.gdb files)
+
+    # Remember the original model col on SVN could have been built anywhere, 
+    # and in the gdb files, absolute paths are stored to the collection location. 
+    # Crop these paths to the collect/<colname> point.
+    
+    # Entries are of the form [Entry] or <Entry>. In order to do a sensible diff,
+    # need to remove the prefix to the collect/colname folder in any (absolute) path that occurs in Entry
+    # E.g. [/full/path/collect/colname/import/file.ext] should become [collect/colname/import/file.ext]
+    # Better regex is of the form /BEGIN((?:(?!BEGIN).)*)END/, see http://docstore.mik.ua/orelly/perl/cookbook/ch06_16.htm
+
+    $model_text =~ s@^([^\\//]*).*(\\|/)(collect(\\|/)$strColName)(.*)$@$1$3$5@mg;
+    $test_text =~ s@^([^\\//]*).*(\\|/)(collect(\\|/)$strColName)(.*)$@$1$3$5@mg;
+
+
     my $report_type = "OldStyle"; # Can not change this type.
     my $diff_gdb = diff \$model_text, \$test_text, { STYLE => $report_type };
     
+    # leaving the ignore regex as it used to be in the following, in case it helps with single line comparisons
     $diff_gdb = &diffutil::GenerateOutput($diff_gdb,"^<(lastmodified|lastmodifieddate|oailastmodified|oailastmodifieddate)>.*");