Changeset 16980 for gsdl/trunk/runtime-src/src

Timestamp:

2008-08-25T09:58:13+12:00 (16 years ago)

Author:

kjdon

Message:

cjk character segmentation. text_t chars not big enough to handle numbers > 0xffff. have commented these ranges out in c++ and perl until we implement a better solution. these high ranges are only for extension sets anyway, so most common words will be segmented

File:

• gsdl/trunk/runtime-src/src/recpt/querytools.cpp (modified) (1 diff)

gsdl/trunk/runtime-src/src/recpt/querytools.cpp

@@ -305 +305 @@
       formattedstring.push_back(' ');
     } else if (segment) {
-      if ((*here >= 0x2e80 && *here <= 0xfa6a) ||
-          (*here >= 0x20000 && *here <= 0x2a6d6) ||
-      (*here >= 0x2f800 && *here <= 0x2fa1d)) {
+      if ((*here >= 0x2e80 && *here <= 0xd7a3) ||
+      ( *here >= 0xf900 && *here <= 0xfa6a)) {
+    /* text_t not big enough to handle these. */
+    /*    (*here >= 0x20000 && *here <= 0x2a6d6) ||
+      (*here >= 0x2f800 && *here <= 0x2fa1d)) { */
     
     // CJK character