Changeset 33327 for gs3-extensions/solr/trunk/src/perllib/solrbuildproc.pm

Timestamp:

2019-07-18T22:45:22+12:00 (5 years ago)

Author:

ak19

Message:

In order to get map coordinate metadata stored correctly in solr, changes were required. These changes revealed that the way in which some index fields were stored in solr but also lucene were not exactly correct and required changing too. 1. Coordinate/CD, CoordShort/CS and GPSMapOverlayLabel/ML meta are now being stored. The schema elements created for these indexed fields notably need to say they're multivalued (multiple values per docOID) and are of type=string rather than type=text_en_splitting as the other meta have been so far. No term related information being stored for them as that doesn't appear important for these indexed fields. 2. Changes to solrbuildproc required and these changes were also repeated into lucenebuildproc: in their code before this commit, <field name=... /> elements were stored once for all meta elements in that field. It sort of worked out so far since the type=text_en_splitting for these fields. This however created the problem that for example all Coordinate meta for a docOID went into a single <field name=CD .../> element separate by spaces rather than a <field name=CD .../> element for each Coordinate meta. We wanted the latter behaviour for CD, CS and ML meta but also for all other indexed meta fields such as TI for titles. But also for indexed fields that include multiple meta in one index such as a hypothetical TT where TT would include dc.Title,ex.Title,text. In that case too we want a <field name=TT /> element for each title meta and for the text meta. 3. The num_processed_bytes calculation is left untouched and still includes the encapsulating <field name=.../> element and has not been changed to be calculated over just the meta data value of each field. This is because not only is it calculated to include the field in super -buildproc.pm classes, but also because the definition of num_processed_bytes in basebuilder.pm is defined as the number of bytes actually passed to (mg) for the current index, where lucene and mgpp buildprocs both include the enclosing element in the calculation which seems deliberate. Further, num_processed_bytes contrasts against num_bytes, declared and defined in basebuildproc.pm too as The actual number of bytes in the collection, normally the same as what's processed during text compression. num_bytes seems to be what Dr Bainbridge had in mind today when he said that actually the enclosing <field/> element shouldn't be included in the calculation of num_processed_bytes. Since the definition of num_processed_bytes seems ambiguous to me now, I leave it alone until discussed with Dr Bainbridge again, as there are many places where it needs changing otherwise.

File:

• gs3-extensions/solr/trunk/src/perllib/solrbuildproc.pm (modified) (3 diffs)

gs3-extensions/solr/trunk/src/perllib/solrbuildproc.pm

@@ -202 +202 @@
     
 }
-sub create_shortname {
+
+# UNUSED now by default.
+# Georgy overrode the mgppbuildproc::create_shortname() method in commit 32441 to create the method below, to override the inherited
+# behaviour so that create_shortname() worked appropriately for his use cases involving multiple analyzers.
+# As a result, create_shortname() for solr no longer did a lookup into the %mgppbuildproc::static_indexfield_map for registered shortnames.
+# For the rest, this method is a copy mgppbuildproc::create_shortname().
+# But we want the original mgppbuildproc::create_shortname() behaviour restored, as it does the lookups into %static_indexfield_map that's necessary for us.
+# So we've renamed this function to create_shortname_multi_solr_analyzer below so it won't get called as default beahviour any more.
+# Rename to create_shortname() when requiring Georgy's behaviour.
+sub create_shortname_multi_solr_analyzer {
     my $self = shift(@_);
 
@@ -500 +509 @@
         
         if ($section_text ne "") {
-            $new_text .= "$section_text ";
+            
+            if ($allfields_index) {
+                $allfields_text .= "$section_text ";
+            }
+            
+            # Remove any leading or trailing white space
+            $section_text =~ s/\s+$//;
+            $section_text =~ s/^\s+//;
+            
+            if ($self->{'indexing_text'}) {
+                # add the tag
+                $new_text .= "<field name=\"$shortname\" >$section_text</field>\n";
+            } else {
+                $new_text .= "$section_text ";
+            }
         }
         
         foreach my $item (@metadata_list) {
             &ghtml::htmlsafe($item);
-            $new_text .= "$item ";
-        }
-
-        if ($allfields_index) {
-            $allfields_text .= $new_text;
-        }
-
-        # Remove any leading or trailing white space
-        $new_text =~ s/\s+$//;
-        $new_text =~ s/^\s+//;
-    
+
+            if ($allfields_index) {
+                $allfields_text .= "$item ";
+            }
+
+            # Remove any leading or trailing white space
+            $item =~ s/\s+$//;
+            $item =~ s/^\s+//;
+            
+            if ($self->{'indexing_text'}) {
+                # add the tag
+                $new_text .= "<field name=\"$shortname\" >$item</field>\n";
+            } else {
+                $new_text .= "$item ";
+            }
+        } # end for loop processing @metadata_list
         
-        if ($self->{'indexing_text'}) {
-            # add the tag
-            $new_text = "<field name=\"$shortname\" >$new_text</field>\n";
-        }
         # filter the text
         $new_text = $self->filter_text ($field, $new_text);
@@ -669 +693 @@
 
         foreach my $item (@metadata_list) {
-        &ghtml::htmlsafe($item);
-        
-        $item = "<field name=\"$sf_shortname\">$item</field>\n";
-        # filter the text???
-        $text .= "$item"; # add it to the main text block
-        #print "#### new_text: $item\n";
+            &ghtml::htmlsafe($item);
+            if ($item =~ /\S/) {
+                $item = "<field name=\"$sf_shortname\">$item</field>\n";
+                # filter the text???
+                $text .= "$item"; # add it to the main text block
+                #print "#### new_text: $item\n";
+            }
         }
         if(scalar @metadata_list > 0) {