Changeset 36807 for other-projects/nightly-tasks/diffcol/trunk/diffcol/gdbdiff.pm

Timestamp:

2022-10-16T18:51:49+13:00 (19 months ago)

Author:

anupama

Message:

We used to run diffcol as a nightly task only for GS2. Commit 36655 was the first stage of getting diffcol to work for GS3, but skipped a lot of important code branches (like comparing the index\text\j/gdb files) in order to fix up the easier parts of the code. Now that I think the remainder of the diffcol scripts have been got to work with diffcol for GS3, where the index\text\flatdb files are compared and diffcol works for them, I can commit the important changes as well as commented out debugging statements made to the diffcol scripts that get the full diffcol code to work for GS3 diffcol. I will recommit again after removing the debugging statements. And I still need to do a full local diffcol run again, as well as testing if diffcol still works after locally undoing my sort field changes to some GS3 model cols (the recent commits to Tudor, Word-PDF, Images-GPS and Multimedia collections) to see if Dr Bainbridge's PERL_HASH_SEED env var addition fixes all of those collections diffcol failures, making the extra sorting redundant. In that case, I will recommit those model collections after updating their col configurations to not do the extra sorting.

File:

• other-projects/nightly-tasks/diffcol/trunk/diffcol/gdbdiff.pm (modified) (3 diffs)

other-projects/nightly-tasks/diffcol/trunk/diffcol/gdbdiff.pm

@@ -106 +106 @@
     $test_text =~ s@(tmp[\\\/])(\d*[\\\/])@$1@g;
 
+
+#print STDERR "@@@@ DEBUGGING: $debugging\n";
+#print STDERR "******** full_modeldb: $full_modeldb\n$model_text\n\n";
+#print STDERR "******** full_testdb: $full_testdb\n$test_text\n\n";
+    
     # if the OS doesn't match and one of them is windows, extra work needs to be done to bring the db files
     # in test and model collection to an even base for comparison
@@ -231 +236 @@
     } # end of equalising differences between a windows collection's db file and linux coll's db file
     
+    
+    # Windows or linux: if index is a flat db file, then ensure the docIDs listed in <contains> field of
+    # both test and model flat db file are alphabetically sorted. So to the numbers in <mdoffset> field.
+    # Despite PERL_PERTURB_KEYS envvar being set to 0 on both machine when generating model collections
+    # and when test collections were generated on test machine, still collections like Images-GPS and some
+    # other colls list items in <contains> and <mdoffset> in different orders. So reordering alphabetically.    
+    #if($dbname =~ m/$strColName/) {    
+        # regex modifiers mge: multi-line, global (replace as many as match), e allows function call in substitution
+        ##$model_text =~ s@^<contains>(.*)@sort_contains_field($1, "MODEL", $debugging)@mge;
+        ##$test_text =~ s@^<contains>(.*)@sort_contains_field($1, "TEST", $debugging)@mge;
+    #   $model_text =~ s@^<(contains|mdoffset)>(.*)@sort_field_value($1, $2, "MODEL", $debugging)@mge;
+    #   $test_text =~ s@^<(contains|mdoffset)>(.*)@sort_field_value($1, $2, "TEST", $debugging)@mge;        
+    #}
+    
     # The following block of code is necessary to deal with tmp (html) source files generated when using PDFBox
     # These tmpdirs are located inside the toplevel *greenstone* directory
@@ -302 +321 @@
     # Call diff?
 }
+
+# Unused, but may come in handy when debugging again: regex substitution helper function
+sub sort_field_value {
+    my($fieldname, $fieldvalue, $displayStr, $debugging) = @_;
+    
+    print STDERR "\n$displayStr BEFORE sort: <$fieldname>$fieldvalue\n" if($debugging);
+    
+    $fieldvalue =~ s@(\r|\n|\\n)*$@@; # get rid of trailing newlines/carriage returns
+    my @values_list = split(';', $fieldvalue);
+    @values_list = sort @values_list;
+    $fieldvalue = "<$fieldname>".join(';', @values_list). "\n";
+    
+    print STDERR "$displayStr AFTER  sort: $fieldvalue\n" if($debugging);
+    
+    return $fieldvalue;
+}
+
 
 # returns true if the contents are windows AND it matters for the diffing on the db that it's windows