Changeset 14923

Timestamp:

2007-12-17T13:47:08+13:00 (16 years ago)

Author:

mdewsnip

Message:

Undid a change I made back in August 2006 regarding removing entities from the text. Turns out this is necessary because HTML entities like   are not valid XML, so Lucene will barf on them. Improved the code so only named entities are removed, and &#nnnn; and &#xhhhh; entities are kept (as these are valid XML).

File:

• gsdl/trunk/perllib/lucenebuildproc.pm (modified) (1 diff)

gsdl/trunk/perllib/lucenebuildproc.pm

@@ -446 +446 @@
     }
 
+    # It's important that we remove name entities because otherwise the text passed to Lucene for indexing
+    #   may not be valid XML (eg. if HTML-only entities like   are used)
+    $new_text =~ s/&\w{1,10};//g;
+    # Remove stray '&' characters, except in &#nnnn; or &#xhhhh; entities (which are valid XML)
+    $new_text =~ s/&([^\#])/ $1/g;
+
     return $new_text;
 }