Changeset 21876

Timestamp:

2010-04-13T15:29:42+12:00 (14 years ago)

Author:

kjdon

Message:

only process into english clauses if english is the only language, not for eg with ar|en. Don't remove all non \w characters - this removes all non alphanumeric chars. I have made up a punctuation match, some replaced with new lines, some with space.

File:

• main/trunk/greenstone2/perllib/classify/Phind.pm (modified) (5 diffs)

main/trunk/greenstone2/perllib/classify/Phind.pm

@@ -29 +29 @@
 # The Phind clasifier plugin. 
 # Type "classinfo.pl Phind" at the command line for a summary.
-
-# 12/05/02 Added usage datastructure - John Thompson
 
 package Phind;
@@ -472 +470 @@
     }
 
-    if ($language_exp =~ /en/) {
+    if ($language_exp =~ /^en$/) {
     return &convert_gml_to_tokens_EN($text);
     }
@@ -504 +502 @@
 
     
-
-
     # 2. Split the remaining text into space-delimited tokens
 
@@ -513 +509 @@
     # Split text at word boundaries
     s/\b/ /go;
-
+    
     # 3. Convert the remaining text to "clause format"
 
@@ -521 +517 @@
 
     # remove unnecessary punctuation and replace with clause break symbol (\n)
-    s/[^\w ]/\n/go;
+    # the following very nicely removes all non alphanumeric characters. too bad if you are not using english...
+    #s/[^\w ]/\n/go;
+    # replace punct with new lines - is this what we want??
+    s/\s*[\?\;\:\!\,\.\"\[\]\{\}\(\)]\s*/\n/go; #"
+    # then remove other punct with space
+    s/[\'\`\\\_]/ /go;
 
     # remove extraneous whitespace