Opened 12 years ago
Closed 12 years ago
#848 closed defect (fixed)
mgpp crash: bit buffer overrun
| Reported by: | kjdon | Owned by: | nobody |
|---|---|---|---|
| Priority: | high | Milestone: | 3.05 Release |
| Component: | Collection Building | Severity: | major |
| Keywords: | Cc: |
Description
Diego had some very messy PDFs containing strings like
fffffffff (x500ish) - which triggered word splitting at max_stem_len which is 255.
Building an mgpp index would crash the build, with a bit buffer overrun.
Note:
See TracTickets
for help on using tickets.
When generating the list of gaps for each word, there was not enough space to store all the gaps - a buffer overrun.
Problem stemmed from bug in utf8_word_to_unicode - comapring inlen instead of outlen with max_output_length. For a 255 long word, inlen was greater than max_output_len, and the conversion never happened, so in another part of the code where it was adding up how many occurrences of each word there were, all 255 long words (which were different) got added into the same score. later on when they were separate words, the right counts weren't available.
Fixing up this also showed up an off-by-one error in max_output_length, between what the code was expecting and what the calling code was giving. Position 0 of the string/array holds the length. Should that char be included in the length or not? Made it all be the max length of the string, NOT including the length char.