1 |
|
---|
2 | # This document contains text in Perl "POD" format.
|
---|
3 | # Use a POD viewer like perldoc or perlman to render it.
|
---|
4 |
|
---|
5 | # This corrects some typoes in the previous release.
|
---|
6 |
|
---|
7 | =head1 NAME
|
---|
8 |
|
---|
9 | Locale::Maketext::TPJ13 -- article about software localization
|
---|
10 |
|
---|
11 | =head1 SYNOPSIS
|
---|
12 |
|
---|
13 | # This an article, not a module.
|
---|
14 |
|
---|
15 | =head1 DESCRIPTION
|
---|
16 |
|
---|
17 | The following article by Sean M. Burke and Jordan Lachler
|
---|
18 | first appeared in I<The Perl
|
---|
19 | Journal> #13 and is copyright 1999 The Perl Journal. It appears
|
---|
20 | courtesy of Jon Orwant and The Perl Journal. This document may be
|
---|
21 | distributed under the same terms as Perl itself.
|
---|
22 |
|
---|
23 | =head1 Localization and Perl: gettext breaks, Maketext fixes
|
---|
24 |
|
---|
25 | by Sean M. Burke and Jordan Lachler
|
---|
26 |
|
---|
27 | This article points out cases where gettext (a common system for
|
---|
28 | localizing software interfaces -- i.e., making them work in the user's
|
---|
29 | language of choice) fails because of basic differences between human
|
---|
30 | languages. This article then describes Maketext, a new system capable
|
---|
31 | of correctly treating these differences.
|
---|
32 |
|
---|
33 | =head2 A Localization Horror Story: It Could Happen To You
|
---|
34 |
|
---|
35 | =over
|
---|
36 |
|
---|
37 | "There are a number of languages spoken by human beings in this
|
---|
38 | world."
|
---|
39 |
|
---|
40 | -- Harald Tveit Alvestrand, in RFC 1766, "Tags for the
|
---|
41 | Identification of Languages"
|
---|
42 |
|
---|
43 | =back
|
---|
44 |
|
---|
45 | Imagine that your task for the day is to localize a piece of software
|
---|
46 | -- and luckily for you, the only output the program emits is two
|
---|
47 | messages, like this:
|
---|
48 |
|
---|
49 | I scanned 12 directories.
|
---|
50 |
|
---|
51 | Your query matched 10 files in 4 directories.
|
---|
52 |
|
---|
53 | So how hard could that be? You look at the code that
|
---|
54 | produces the first item, and it reads:
|
---|
55 |
|
---|
56 | printf("I scanned %g directories.",
|
---|
57 | $directory_count);
|
---|
58 |
|
---|
59 | You think about that, and realize that it doesn't even work right for
|
---|
60 | English, as it can produce this output:
|
---|
61 |
|
---|
62 | I scanned 1 directories.
|
---|
63 |
|
---|
64 | So you rewrite it to read:
|
---|
65 |
|
---|
66 | printf("I scanned %g %s.",
|
---|
67 | $directory_count,
|
---|
68 | $directory_count == 1 ?
|
---|
69 | "directory" : "directories",
|
---|
70 | );
|
---|
71 |
|
---|
72 | ...which does the Right Thing. (In case you don't recall, "%g" is for
|
---|
73 | locale-specific number interpolation, and "%s" is for string
|
---|
74 | interpolation.)
|
---|
75 |
|
---|
76 | But you still have to localize it for all the languages you're
|
---|
77 | producing this software for, so you pull Locale::gettext off of CPAN
|
---|
78 | so you can access the C<gettext> C functions you've heard are standard
|
---|
79 | for localization tasks.
|
---|
80 |
|
---|
81 | And you write:
|
---|
82 |
|
---|
83 | printf(gettext("I scanned %g %s."),
|
---|
84 | $dir_scan_count,
|
---|
85 | $dir_scan_count == 1 ?
|
---|
86 | gettext("directory") : gettext("directories"),
|
---|
87 | );
|
---|
88 |
|
---|
89 | But you then read in the gettext manual (Drepper, Miller, and Pinard 1995)
|
---|
90 | that this is not a good idea, since how a single word like "directory"
|
---|
91 | or "directories" is translated may depend on context -- and this is
|
---|
92 | true, since in a case language like German or Russian, you'd may need
|
---|
93 | these words with a different case ending in the first instance (where the
|
---|
94 | word is the object of a verb) than in the second instance, which you haven't even
|
---|
95 | gotten to yet (where the word is the object of a preposition, "in %g
|
---|
96 | directories") -- assuming these keep the same syntax when translated
|
---|
97 | into those languages.
|
---|
98 |
|
---|
99 | So, on the advice of the gettext manual, you rewrite:
|
---|
100 |
|
---|
101 | printf( $dir_scan_count == 1 ?
|
---|
102 | gettext("I scanned %g directory.") :
|
---|
103 | gettext("I scanned %g directories."),
|
---|
104 | $dir_scan_count );
|
---|
105 |
|
---|
106 | So, you email your various translators (the boss decides that the
|
---|
107 | languages du jour are Chinese, Arabic, Russian, and Italian, so you
|
---|
108 | have one translator for each), asking for translations for "I scanned
|
---|
109 | %g directory." and "I scanned %g directories.". When they reply,
|
---|
110 | you'll put that in the lexicons for gettext to use when it localizes
|
---|
111 | your software, so that when the user is running under the "zh"
|
---|
112 | (Chinese) locale, gettext("I scanned %g directory.") will return the
|
---|
113 | appropriate Chinese text, with a "%g" in there where printf can then
|
---|
114 | interpolate $dir_scan.
|
---|
115 |
|
---|
116 | Your Chinese translator emails right back -- he says both of these
|
---|
117 | phrases translate to the same thing in Chinese, because, in linguistic
|
---|
118 | jargon, Chinese "doesn't have number as a grammatical category" --
|
---|
119 | whereas English does. That is, English has grammatical rules that
|
---|
120 | refer to "number", i.e., whether something is grammatically singular
|
---|
121 | or plural; and one of these rules is the one that forces nouns to take
|
---|
122 | a plural suffix (generally "s") when in a plural context, as they are when
|
---|
123 | they follow a number other than "one" (including, oddly enough, "zero").
|
---|
124 | Chinese has no such rules, and so has just the one phrase where English
|
---|
125 | has two. But, no problem, you can have this one Chinese phrase appear
|
---|
126 | as the translation for the two English phrases in the "zh" gettext
|
---|
127 | lexicon for your program.
|
---|
128 |
|
---|
129 | Emboldened by this, you dive into the second phrase that your software
|
---|
130 | needs to output: "Your query matched 10 files in 4 directories.". You notice
|
---|
131 | that if you want to treat phrases as indivisible, as the gettext
|
---|
132 | manual wisely advises, you need four cases now, instead of two, to
|
---|
133 | cover the permutations of singular and plural on the two items,
|
---|
134 | $dir_count and $file_count. So you try this:
|
---|
135 |
|
---|
136 | printf( $file_count == 1 ?
|
---|
137 | ( $directory_count == 1 ?
|
---|
138 | gettext("Your query matched %g file in %g directory.") :
|
---|
139 | gettext("Your query matched %g file in %g directories.") ) :
|
---|
140 | ( $directory_count == 1 ?
|
---|
141 | gettext("Your query matched %g files in %g directory.") :
|
---|
142 | gettext("Your query matched %g files in %g directories.") ),
|
---|
143 | $file_count, $directory_count,
|
---|
144 | );
|
---|
145 |
|
---|
146 | (The case of "1 file in 2 [or more] directories" could, I suppose,
|
---|
147 | occur in the case of symlinking or something of the sort.)
|
---|
148 |
|
---|
149 | It occurs to you that this is not the prettiest code you've ever
|
---|
150 | written, but this seems the way to go. You mail off to the
|
---|
151 | translators asking for translations for these four cases. The
|
---|
152 | Chinese guy replies with the one phrase that these all translate to in
|
---|
153 | Chinese, and that phrase has two "%g"s in it, as it should -- but
|
---|
154 | there's a problem. He translates it word-for-word back: "In %g
|
---|
155 | directories contains %g files match your query." The %g
|
---|
156 | slots are in an order reverse to what they are in English. You wonder
|
---|
157 | how you'll get gettext to handle that.
|
---|
158 |
|
---|
159 | But you put it aside for the moment, and optimistically hope that the
|
---|
160 | other translators won't have this problem, and that their languages
|
---|
161 | will be better behaved -- i.e., that they will be just like English.
|
---|
162 |
|
---|
163 | But the Arabic translator is the next to write back. First off, your
|
---|
164 | code for "I scanned %g directory." or "I scanned %g directories."
|
---|
165 | assumes there's only singular or plural. But, to use linguistic
|
---|
166 | jargon again, Arabic has grammatical number, like English (but unlike
|
---|
167 | Chinese), but it's a three-term category: singular, dual, and plural.
|
---|
168 | In other words, the way you say "directory" depends on whether there's
|
---|
169 | one directory, or I<two> of them, or I<more than two> of them. Your
|
---|
170 | test of C<($directory == 1)> no longer does the job. And it means
|
---|
171 | that where English's grammatical category of number necessitates
|
---|
172 | only the two permutations of the first sentence based on "directory
|
---|
173 | [singular]" and "directories [plural]", Arabic has three -- and,
|
---|
174 | worse, in the second sentence ("Your query matched %g file in %g
|
---|
175 | directory."), where English has four, Arabic has nine. You sense
|
---|
176 | an unwelcome, exponential trend taking shape.
|
---|
177 |
|
---|
178 | Your Italian translator emails you back and says that "I searched 0
|
---|
179 | directories" (a possible English output of your program) is stilted,
|
---|
180 | and if you think that's fine English, that's your problem, but that
|
---|
181 | I<just will not do> in the language of Dante. He insists that where
|
---|
182 | $directory_count is 0, your program should produce the Italian text
|
---|
183 | for "I I<didn't> scan I<any> directories.". And ditto for "I didn't
|
---|
184 | match any files in any directories", although he says the last part
|
---|
185 | about "in any directories" should probably just be left off.
|
---|
186 |
|
---|
187 | You wonder how you'll get gettext to handle this; to accomodate the
|
---|
188 | ways Arabic, Chinese, and Italian deal with numbers in just these few
|
---|
189 | very simple phrases, you need to write code that will ask gettext for
|
---|
190 | different queries depending on whether the numerical values in
|
---|
191 | question are 1, 2, more than 2, or in some cases 0, and you still haven't
|
---|
192 | figured out the problem with the different word order in Chinese.
|
---|
193 |
|
---|
194 | Then your Russian translator calls on the phone, to I<personally> tell
|
---|
195 | you the bad news about how really unpleasant your life is about to
|
---|
196 | become:
|
---|
197 |
|
---|
198 | Russian, like German or Latin, is an inflectional language; that is, nouns
|
---|
199 | and adjectives have to take endings that depend on their case
|
---|
200 | (i.e., nominative, accusative, genitive, etc...) -- which is roughly a matter of
|
---|
201 | what role they have in syntax of the sentence --
|
---|
202 | as well as on the grammatical gender (i.e., masculine, feminine, neuter)
|
---|
203 | and number (i.e., singular or plural) of the noun, as well as on the
|
---|
204 | declension class of the noun. But unlike with most other inflected languages,
|
---|
205 | putting a number-phrase (like "ten" or "forty-three", or their Arabic
|
---|
206 | numeral equivalents) in front of noun in Russian can change the case and
|
---|
207 | number that noun is, and therefore the endings you have to put on it.
|
---|
208 |
|
---|
209 | He elaborates: In "I scanned %g directories", you'd I<expect>
|
---|
210 | "directories" to be in the accusative case (since it is the direct
|
---|
211 | object in the sentnce) and the plural number,
|
---|
212 | except where $directory_count is 1, then you'd expect the singular, of
|
---|
213 | course. Just like Latin or German. I<But!> Where $directory_count %
|
---|
214 | 10 is 1 ("%" for modulo, remember), assuming $directory count is an
|
---|
215 | integer, and except where $directory_count % 100 is 11, "directories"
|
---|
216 | is forced to become grammatically singular, which means it gets the
|
---|
217 | ending for the accusative singular... You begin to visualize the code
|
---|
218 | it'd take to test for the problem so far, I<and still work for Chinese
|
---|
219 | and Arabic and Italian>, and how many gettext items that'd take, but
|
---|
220 | he keeps going... But where $directory_count % 10 is 2, 3, or 4
|
---|
221 | (except where $directory_count % 100 is 12, 13, or 14), the word for
|
---|
222 | "directories" is forced to be genitive singular -- which means another
|
---|
223 | ending... The room begins to spin around you, slowly at first... But
|
---|
224 | with I<all other> integer values, since "directory" is an inanimate
|
---|
225 | noun, when preceded by a number and in the nominative or accusative
|
---|
226 | cases (as it is here, just your luck!), it does stay plural, but it is
|
---|
227 | forced into the genitive case -- yet another ending... And
|
---|
228 | you never hear him get to the part about how you're going to run into
|
---|
229 | similar (but maybe subtly different) problems with other Slavic
|
---|
230 | languages like Polish, because the floor comes up to meet you, and you
|
---|
231 | fade into unconsciousness.
|
---|
232 |
|
---|
233 |
|
---|
234 | The above cautionary tale relates how an attempt at localization can
|
---|
235 | lead from programmer consternation, to program obfuscation, to a need
|
---|
236 | for sedation. But careful evaluation shows that your choice of tools
|
---|
237 | merely needed further consideration.
|
---|
238 |
|
---|
239 | =head2 The Linguistic View
|
---|
240 |
|
---|
241 | =over
|
---|
242 |
|
---|
243 | "It is more complicated than you think."
|
---|
244 |
|
---|
245 | -- The Eighth Networking Truth, from RFC 1925
|
---|
246 |
|
---|
247 | =back
|
---|
248 |
|
---|
249 | The field of Linguistics has expended a great deal of effort over the
|
---|
250 | past century trying to find grammatical patterns which hold across
|
---|
251 | languages; it's been a constant process
|
---|
252 | of people making generalizations that should apply to all languages,
|
---|
253 | only to find out that, all too often, these generalizations fail --
|
---|
254 | sometimes failing for just a few languages, sometimes whole classes of
|
---|
255 | languages, and sometimes nearly every language in the world except
|
---|
256 | English. Broad statistical trends are evident in what the "average
|
---|
257 | language" is like as far as what its rules can look like, must look
|
---|
258 | like, and cannot look like. But the "average language" is just as
|
---|
259 | unreal a concept as the "average person" -- it runs up against the
|
---|
260 | fact no language (or person) is, in fact, average. The wisdom of past
|
---|
261 | experience leads us to believe that any given language can do whatever
|
---|
262 | it wants, in any order, with appeal to any kind of grammatical
|
---|
263 | categories wants -- case, number, tense, real or metaphoric
|
---|
264 | characteristics of the things that words refer to, arbitrary or
|
---|
265 | predictable classifications of words based on what endings or prefixes
|
---|
266 | they can take, degree or means of certainty about the truth of
|
---|
267 | statements expressed, and so on, ad infinitum.
|
---|
268 |
|
---|
269 | Mercifully, most localization tasks are a matter of finding ways to
|
---|
270 | translate whole phrases, generally sentences, where the context is
|
---|
271 | relatively set, and where the only variation in content is I<usually>
|
---|
272 | in a number being expressed -- as in the example sentences above.
|
---|
273 | Translating specific, fully-formed sentences is, in practice, fairly
|
---|
274 | foolproof -- which is good, because that's what's in the phrasebooks
|
---|
275 | that so many tourists rely on. Now, a given phrase (whether in a
|
---|
276 | phrasebook or in a gettext lexicon) in one language I<might> have a
|
---|
277 | greater or lesser applicability than that phrase's translation into
|
---|
278 | another language -- for example, strictly speaking, in Arabic, the
|
---|
279 | "your" in "Your query matched..." would take a different form
|
---|
280 | depending on whether the user is male or female; so the Arabic
|
---|
281 | translation "your[feminine] query" is applicable in fewer cases than
|
---|
282 | the corresponding English phrase, which doesn't distinguish the user's
|
---|
283 | gender. (In practice, it's not feasable to have a program know the
|
---|
284 | user's gender, so the masculine "you" in Arabic is usually used, by
|
---|
285 | default.)
|
---|
286 |
|
---|
287 | But in general, such surprises are rare when entire sentences are
|
---|
288 | being translated, especially when the functional context is restricted
|
---|
289 | to that of a computer interacting with a user either to convey a fact
|
---|
290 | or to prompt for a piece of information. So, for purposes of
|
---|
291 | localization, translation by phrase (generally by sentence) is both the
|
---|
292 | simplest and the least problematic.
|
---|
293 |
|
---|
294 | =head2 Breaking gettext
|
---|
295 |
|
---|
296 | =over
|
---|
297 |
|
---|
298 | "It Has To Work."
|
---|
299 |
|
---|
300 | -- First Networking Truth, RFC 1925
|
---|
301 |
|
---|
302 | =back
|
---|
303 |
|
---|
304 | Consider that sentences in a tourist phrasebook are of two types: ones
|
---|
305 | like "How do I get to the marketplace?" that don't have any blanks to
|
---|
306 | fill in, and ones like "How much do these ___ cost?", where there's
|
---|
307 | one or more blanks to fill in (and these are usually linked to a
|
---|
308 | list of words that you can put in that blank: "fish", "potatoes",
|
---|
309 | "tomatoes", etc.) The ones with no blanks are no problem, but the
|
---|
310 | fill-in-the-blank ones may not be really straightforward. If it's a
|
---|
311 | Swahili phrasebook, for example, the authors probably didn't bother to
|
---|
312 | tell you the complicated ways that the verb "cost" changes its
|
---|
313 | inflectional prefix depending on the noun you're putting in the blank.
|
---|
314 | The trader in the marketplace will still understand what you're saying if
|
---|
315 | you say "how much do these potatoes cost?" with the wrong
|
---|
316 | inflectional prefix on "cost". After all, I<you> can't speak proper Swahili,
|
---|
317 | I<you're> just a tourist. But while tourists can be stupid, computers
|
---|
318 | are supposed to be smart; the computer should be able to fill in the
|
---|
319 | blank, and still have the results be grammatical.
|
---|
320 |
|
---|
321 | In other words, a phrasebook entry takes some values as parameters
|
---|
322 | (the things that you fill in the blank or blanks), and provides a value
|
---|
323 | based on these parameters, where the way you get that final value from
|
---|
324 | the given values can, properly speaking, involve an arbitrarily
|
---|
325 | complex series of operations. (In the case of Chinese, it'd be not at
|
---|
326 | all complex, at least in cases like the examples at the beginning of
|
---|
327 | this article; whereas in the case of Russian it'd be a rather complex
|
---|
328 | series of operations. And in some languages, the
|
---|
329 | complexity could be spread around differently: while the act of
|
---|
330 | putting a number-expression in front of a noun phrase might not be
|
---|
331 | complex by itself, it may change how you have to, for example, inflect
|
---|
332 | a verb elsewhere in the sentence. This is what in syntax is called
|
---|
333 | "long-distance dependencies".)
|
---|
334 |
|
---|
335 | This talk of parameters and arbitrary complexity is just another way
|
---|
336 | to say that an entry in a phrasebook is what in a programming language
|
---|
337 | would be called a "function". Just so you don't miss it, this is the
|
---|
338 | crux of this article: I<A phrase is a function; a phrasebook is a
|
---|
339 | bunch of functions.>
|
---|
340 |
|
---|
341 | The reason that using gettext runs into walls (as in the above
|
---|
342 | second-person horror story) is that you're trying to use a string (or
|
---|
343 | worse, a choice among a bunch of strings) to do what you really need a
|
---|
344 | function for -- which is futile. Preforming (s)printf interpolation
|
---|
345 | on the strings which you get back from gettext does allow you to do I<some>
|
---|
346 | common things passably well... sometimes... sort of; but, to paraphrase
|
---|
347 | what some people say about C<csh> script programming, "it fools you
|
---|
348 | into thinking you can use it for real things, but you can't, and you
|
---|
349 | don't discover this until you've already spent too much time trying,
|
---|
350 | and by then it's too late."
|
---|
351 |
|
---|
352 | =head2 Replacing gettext
|
---|
353 |
|
---|
354 | So, what needs to replace gettext is a system that supports lexicons
|
---|
355 | of functions instead of lexicons of strings. An entry in a lexicon
|
---|
356 | from such a system should I<not> look like this:
|
---|
357 |
|
---|
358 | "J'ai trouv\xE9 %g fichiers dans %g r\xE9pertoires"
|
---|
359 |
|
---|
360 | [\xE9 is e-acute in Latin-1. Some pod renderers would
|
---|
361 | scream if I used the actual character here. -- SB]
|
---|
362 |
|
---|
363 | but instead like this, bearing in mind that this is just a first stab:
|
---|
364 |
|
---|
365 | sub I_found_X1_files_in_X2_directories {
|
---|
366 | my( $files, $dirs ) = @_[0,1];
|
---|
367 | $files = sprintf("%g %s", $files,
|
---|
368 | $files == 1 ? 'fichier' : 'fichiers');
|
---|
369 | $dirs = sprintf("%g %s", $dirs,
|
---|
370 | $dirs == 1 ? "r\xE9pertoire" : "r\xE9pertoires");
|
---|
371 | return "J'ai trouv\xE9 $files dans $dirs.";
|
---|
372 | }
|
---|
373 |
|
---|
374 | Now, there's no particularly obvious way to store anything but strings
|
---|
375 | in a gettext lexicon; so it looks like we just have to start over and
|
---|
376 | make something better, from scratch. I call my shot at a
|
---|
377 | gettext-replacement system "Maketext", or, in CPAN terms,
|
---|
378 | Locale::Maketext.
|
---|
379 |
|
---|
380 | When designing Maketext, I chose to plan its main features in terms of
|
---|
381 | "buzzword compliance". And here are the buzzwords:
|
---|
382 |
|
---|
383 | =head2 Buzzwords: Abstraction and Encapsulation
|
---|
384 |
|
---|
385 | The complexity of the language you're trying to output a phrase in is
|
---|
386 | entirely abstracted inside (and encapsulated within) the Maketext module
|
---|
387 | for that interface. When you call:
|
---|
388 |
|
---|
389 | print $lang->maketext("You have [quant,_1,piece] of new mail.",
|
---|
390 | scalar(@messages));
|
---|
391 |
|
---|
392 | you don't know (and in fact can't easily find out) whether this will
|
---|
393 | involve lots of figuring, as in Russian (if $lang is a handle to the
|
---|
394 | Russian module), or relatively little, as in Chinese. That kind of
|
---|
395 | abstraction and encapsulation may encourage other pleasant buzzwords
|
---|
396 | like modularization and stratification, depending on what design
|
---|
397 | decisions you make.
|
---|
398 |
|
---|
399 | =head2 Buzzword: Isomorphism
|
---|
400 |
|
---|
401 | "Isomorphism" means "having the same structure or form"; in discussions
|
---|
402 | of program design, the word takes on the special, specific meaning that
|
---|
403 | your implementation of a solution to a problem I<has the same
|
---|
404 | structure> as, say, an informal verbal description of the solution, or
|
---|
405 | maybe of the problem itself. Isomorphism is, all things considered,
|
---|
406 | a good thing -- it's what problem-solving (and solution-implementing)
|
---|
407 | should look like.
|
---|
408 |
|
---|
409 | What's wrong the with gettext-using code like this...
|
---|
410 |
|
---|
411 | printf( $file_count == 1 ?
|
---|
412 | ( $directory_count == 1 ?
|
---|
413 | "Your query matched %g file in %g directory." :
|
---|
414 | "Your query matched %g file in %g directories." ) :
|
---|
415 | ( $directory_count == 1 ?
|
---|
416 | "Your query matched %g files in %g directory." :
|
---|
417 | "Your query matched %g files in %g directories." ),
|
---|
418 | $file_count, $directory_count,
|
---|
419 | );
|
---|
420 |
|
---|
421 | is first off that it's not well abstracted -- these ways of testing
|
---|
422 | for grammatical number (as in the expressions like C<foo == 1 ?
|
---|
423 | singular_form : plural_form>) should be abstracted to each language
|
---|
424 | module, since how you get grammatical number is language-specific.
|
---|
425 |
|
---|
426 | But second off, it's not isomorphic -- the "solution" (i.e., the
|
---|
427 | phrasebook entries) for Chinese maps from these four English phrases to
|
---|
428 | the one Chinese phrase that fits for all of them. In other words, the
|
---|
429 | informal solution would be "The way to say what you want in Chinese is
|
---|
430 | with the one phrase 'For your question, in Y directories you would
|
---|
431 | find X files'" -- and so the implemented solution should be,
|
---|
432 | isomorphically, just a straightforward way to spit out that one
|
---|
433 | phrase, with numerals properly interpolated. It shouldn't have to map
|
---|
434 | from the complexity of other languages to the simplicity of this one.
|
---|
435 |
|
---|
436 | =head2 Buzzword: Inheritance
|
---|
437 |
|
---|
438 | There's a great deal of reuse possible for sharing of phrases between
|
---|
439 | modules for related dialects, or for sharing of auxiliary functions
|
---|
440 | between related languages. (By "auxiliary functions", I mean
|
---|
441 | functions that don't produce phrase-text, but which, say, return an
|
---|
442 | answer to "does this number require a plural noun after it?". Such
|
---|
443 | auxiliary functions would be used in the internal logic of functions
|
---|
444 | that actually do produce phrase-text.)
|
---|
445 |
|
---|
446 | In the case of sharing phrases, consider that you have an interface
|
---|
447 | already localized for American English (probably by having been
|
---|
448 | written with that as the native locale, but that's incidental).
|
---|
449 | Localizing it for UK English should, in practical terms, be just a
|
---|
450 | matter of running it past a British person with the instructions to
|
---|
451 | indicate what few phrases would benefit from a change in spelling or
|
---|
452 | possibly minor rewording. In that case, you should be able to put in
|
---|
453 | the UK English localization module I<only> those phrases that are
|
---|
454 | UK-specific, and for all the rest, I<inherit> from the American
|
---|
455 | English module. (And I expect this same situation would apply with
|
---|
456 | Brazilian and Continental Portugese, possbily with some I<very>
|
---|
457 | closely related languages like Czech and Slovak, and possibly with the
|
---|
458 | slightly different "versions" of written Mandarin Chinese, as I hear exist in
|
---|
459 | Taiwan and mainland China.)
|
---|
460 |
|
---|
461 | As to sharing of auxiliary functions, consider the problem of Russian
|
---|
462 | numbers from the beginning of this article; obviously, you'd want to
|
---|
463 | write only once the hairy code that, given a numeric value, would
|
---|
464 | return some specification of which case and number a given quanitified
|
---|
465 | noun should use. But suppose that you discover, while localizing an
|
---|
466 | interface for, say, Ukranian (a Slavic language related to Russian,
|
---|
467 | spoken by several million people, many of whom would be relieved to
|
---|
468 | find that your Web site's or software's interface is available in
|
---|
469 | their language), that the rules in Ukranian are the same as in Russian
|
---|
470 | for quantification, and probably for many other grammatical functions.
|
---|
471 | While there may well be no phrases in common between Russian and
|
---|
472 | Ukranian, you could still choose to have the Ukranian module inherit
|
---|
473 | from the Russian module, just for the sake of inheriting all the
|
---|
474 | various grammatical methods. Or, probably better organizationally,
|
---|
475 | you could move those functions to a module called C<_E_Slavic> or
|
---|
476 | something, which Russian and Ukranian could inherit useful functions
|
---|
477 | from, but which would (presumably) provide no lexicon.
|
---|
478 |
|
---|
479 | =head2 Buzzword: Concision
|
---|
480 |
|
---|
481 | Okay, concision isn't a buzzword. But it should be, so I decree that
|
---|
482 | as a new buzzword, "concision" means that simple common things should
|
---|
483 | be expressible in very few lines (or maybe even just a few characters)
|
---|
484 | of code -- call it a special case of "making simple things easy and
|
---|
485 | hard things possible", and see also the role it played in the
|
---|
486 | MIDI::Simple language, discussed elsewhere in this issue [TPJ#13].
|
---|
487 |
|
---|
488 | Consider our first stab at an entry in our "phrasebook of functions":
|
---|
489 |
|
---|
490 | sub I_found_X1_files_in_X2_directories {
|
---|
491 | my( $files, $dirs ) = @_[0,1];
|
---|
492 | $files = sprintf("%g %s", $files,
|
---|
493 | $files == 1 ? 'fichier' : 'fichiers');
|
---|
494 | $dirs = sprintf("%g %s", $dirs,
|
---|
495 | $dirs == 1 ? "r\xE9pertoire" : "r\xE9pertoires");
|
---|
496 | return "J'ai trouv\xE9 $files dans $dirs.";
|
---|
497 | }
|
---|
498 |
|
---|
499 | You may sense that a lexicon (to use a non-committal catch-all term for a
|
---|
500 | collection of things you know how to say, regardless of whether they're
|
---|
501 | phrases or words) consisting of functions I<expressed> as above would
|
---|
502 | make for rather long-winded and repetitive code -- even if you wisely
|
---|
503 | rewrote this to have quantification (as we call adding a number
|
---|
504 | expression to a noun phrase) be a function called like:
|
---|
505 |
|
---|
506 | sub I_found_X1_files_in_X2_directories {
|
---|
507 | my( $files, $dirs ) = @_[0,1];
|
---|
508 | $files = quant($files, "fichier");
|
---|
509 | $dirs = quant($dirs, "r\xE9pertoire");
|
---|
510 | return "J'ai trouv\xE9 $files dans $dirs.";
|
---|
511 | }
|
---|
512 |
|
---|
513 | And you may also sense that you do not want to bother your translators
|
---|
514 | with having to write Perl code -- you'd much rather that they spend
|
---|
515 | their I<very costly time> on just translation. And this is to say
|
---|
516 | nothing of the near impossibility of finding a commercial translator
|
---|
517 | who would know even simple Perl.
|
---|
518 |
|
---|
519 | In a first-hack implementation of Maketext, each language-module's
|
---|
520 | lexicon looked like this:
|
---|
521 |
|
---|
522 | %Lexicon = (
|
---|
523 | "I found %g files in %g directories"
|
---|
524 | => sub {
|
---|
525 | my( $files, $dirs ) = @_[0,1];
|
---|
526 | $files = quant($files, "fichier");
|
---|
527 | $dirs = quant($dirs, "r\xE9pertoire");
|
---|
528 | return "J'ai trouv\xE9 $files dans $dirs.";
|
---|
529 | },
|
---|
530 | ... and so on with other phrase => sub mappings ...
|
---|
531 | );
|
---|
532 |
|
---|
533 | but I immediately went looking for some more concise way to basically
|
---|
534 | denote the same phrase-function -- a way that would also serve to
|
---|
535 | concisely denote I<most> phrase-functions in the lexicon for I<most>
|
---|
536 | languages. After much time and even some actual thought, I decided on
|
---|
537 | this system:
|
---|
538 |
|
---|
539 | * Where a value in a %Lexicon hash is a contentful string instead of
|
---|
540 | an anonymous sub (or, conceivably, a coderef), it would be interpreted
|
---|
541 | as a sort of shorthand expression of what the sub does. When accessed
|
---|
542 | for the first time in a session, it is parsed, turned into Perl code,
|
---|
543 | and then eval'd into an anonymous sub; then that sub replaces the
|
---|
544 | original string in that lexicon. (That way, the work of parsing and
|
---|
545 | evaling the shorthand form for a given phrase is done no more than
|
---|
546 | once per session.)
|
---|
547 |
|
---|
548 | * Calls to C<maketext> (as Maketext's main function is called) happen
|
---|
549 | thru a "language session handle", notionally very much like an IO
|
---|
550 | handle, in that you open one at the start of the session, and use it
|
---|
551 | for "sending signals" to an object in order to have it return the text
|
---|
552 | you want.
|
---|
553 |
|
---|
554 | So, this:
|
---|
555 |
|
---|
556 | $lang->maketext("You have [quant,_1,piece] of new mail.",
|
---|
557 | scalar(@messages));
|
---|
558 |
|
---|
559 | basically means this: look in the lexicon for $lang (which may inherit
|
---|
560 | from any number of other lexicons), and find the function that we
|
---|
561 | happen to associate with the string "You have [quant,_1,piece] of new
|
---|
562 | mail" (which is, and should be, a functioning "shorthand" for this
|
---|
563 | function in the native locale -- English in this case). If you find
|
---|
564 | such a function, call it with $lang as its first parameter (as if it
|
---|
565 | were a method), and then a copy of scalar(@messages) as its second,
|
---|
566 | and then return that value. If that function was found, but was in
|
---|
567 | string shorthand instead of being a fully specified function, parse it
|
---|
568 | and make it into a function before calling it the first time.
|
---|
569 |
|
---|
570 | * The shorthand uses code in brackets to indicate method calls that
|
---|
571 | should be performed. A full explanation is not in order here, but a
|
---|
572 | few examples will suffice:
|
---|
573 |
|
---|
574 | "You have [quant,_1,piece] of new mail."
|
---|
575 |
|
---|
576 | The above code is shorthand for, and will be interpreted as,
|
---|
577 | this:
|
---|
578 |
|
---|
579 | sub {
|
---|
580 | my $handle = $_[0];
|
---|
581 | my(@params) = @_;
|
---|
582 | return join '',
|
---|
583 | "You have ",
|
---|
584 | $handle->quant($params[1], 'piece'),
|
---|
585 | "of new mail.";
|
---|
586 | }
|
---|
587 |
|
---|
588 | where "quant" is the name of a method you're using to quantify the
|
---|
589 | noun "piece" with the number $params[0].
|
---|
590 |
|
---|
591 | A string with no brackety calls, like this:
|
---|
592 |
|
---|
593 | "Your search expression was malformed."
|
---|
594 |
|
---|
595 | is somewhat of a degerate case, and just gets turned into:
|
---|
596 |
|
---|
597 | sub { return "Your search expression was malformed." }
|
---|
598 |
|
---|
599 | However, not everything you can write in Perl code can be written in
|
---|
600 | the above shorthand system -- not by a long shot. For example, consider
|
---|
601 | the Italian translator from the beginning of this article, who wanted
|
---|
602 | the Italian for "I didn't find any files" as a special case, instead
|
---|
603 | of "I found 0 files". That couldn't be specified (at least not easily
|
---|
604 | or simply) in our shorthand system, and it would have to be written
|
---|
605 | out in full, like this:
|
---|
606 |
|
---|
607 | sub { # pretend the English strings are in Italian
|
---|
608 | my($handle, $files, $dirs) = @_[0,1,2];
|
---|
609 | return "I didn't find any files" unless $files;
|
---|
610 | return join '',
|
---|
611 | "I found ",
|
---|
612 | $handle->quant($files, 'file'),
|
---|
613 | " in ",
|
---|
614 | $handle->quant($dirs, 'directory'),
|
---|
615 | ".";
|
---|
616 | }
|
---|
617 |
|
---|
618 | Next to a lexicon full of shorthand code, that sort of sticks out like a
|
---|
619 | sore thumb -- but this I<is> a special case, after all; and at least
|
---|
620 | it's possible, if not as concise as usual.
|
---|
621 |
|
---|
622 | As to how you'd implement the Russian example from the beginning of
|
---|
623 | the article, well, There's More Than One Way To Do It, but it could be
|
---|
624 | something like this (using English words for Russian, just so you know
|
---|
625 | what's going on):
|
---|
626 |
|
---|
627 | "I [quant,_1,directory,accusative] scanned."
|
---|
628 |
|
---|
629 | This shifts the burden of complexity off to the quant method. That
|
---|
630 | method's parameters are: the numeric value it's going to use to
|
---|
631 | quantify something; the Russian word it's going to quantify; and the
|
---|
632 | parameter "accusative", which you're using to mean that this
|
---|
633 | sentence's syntax wants a noun in the accusative case there, although
|
---|
634 | that quantification method may have to overrule, for grammatical
|
---|
635 | reasons you may recall from the beginning of this article.
|
---|
636 |
|
---|
637 | Now, the Russian quant method here is responsible not only for
|
---|
638 | implementing the strange logic necessary for figuring out how Russian
|
---|
639 | number-phrases impose case and number on their noun-phrases, but also
|
---|
640 | for inflecting the Russian word for "directory". How that inflection
|
---|
641 | is to be carried out is no small issue, and among the solutions I've
|
---|
642 | seen, some (like variations on a simple lookup in a hash where all
|
---|
643 | possible forms are provided for all necessary words) are
|
---|
644 | straightforward but I<can> become cumbersome when you need to inflect
|
---|
645 | more than a few dozen words; and other solutions (like using
|
---|
646 | algorithms to model the inflections, storing only root forms and
|
---|
647 | irregularities) I<can> involve more overhead than is justifiable for
|
---|
648 | all but the largest lexicons.
|
---|
649 |
|
---|
650 | Mercifully, this design decision becomes crucial only in the hairiest
|
---|
651 | of inflected languages, of which Russian is by no means the I<worst> case
|
---|
652 | scenario, but is worse than most. Most languages have simpler
|
---|
653 | inflection systems; for example, in English or Swahili, there are
|
---|
654 | generally no more than two possible inflected forms for a given noun
|
---|
655 | ("error/errors"; "kosa/makosa"), and the
|
---|
656 | rules for producing these forms are fairly simple -- or at least,
|
---|
657 | simple rules can be formulated that work for most words, and you can
|
---|
658 | then treat the exceptions as just "irregular", at least relative to
|
---|
659 | your ad hoc rules. A simpler inflection system (simpler rules, fewer
|
---|
660 | forms) means that design decisions are less crucial to maintaining
|
---|
661 | sanity, whereas the same decisions could incur
|
---|
662 | overhead-versus-scalability problems in languages like Russian. It
|
---|
663 | may I<also> be likely that code (possibly in Perl, as with
|
---|
664 | Lingua::EN::Inflect, for English nouns) has already
|
---|
665 | been written for the language in question, whether simple or complex.
|
---|
666 |
|
---|
667 | Moreover, a third possibility may even be simpler than anything
|
---|
668 | discussed above: "Just require that all possible (or at least
|
---|
669 | applicable) forms be provided in the call to the given language's quant
|
---|
670 | method, as in:"
|
---|
671 |
|
---|
672 | "I found [quant,_1,file,files]."
|
---|
673 |
|
---|
674 | That way, quant just has to chose which form it needs, without having
|
---|
675 | to look up or generate anything. While possibly not optimal for
|
---|
676 | Russian, this should work well for most other languages, where
|
---|
677 | quantification is not as complicated an operation.
|
---|
678 |
|
---|
679 | =head2 The Devil in the Details
|
---|
680 |
|
---|
681 | There's plenty more to Maketext than described above -- for example,
|
---|
682 | there's the details of how language tags ("en-US", "i-pwn", "fi",
|
---|
683 | etc.) or locale IDs ("en_US") interact with actual module naming
|
---|
684 | ("BogoQuery/Locale/en_us.pm"), and what magic can ensue; there's the
|
---|
685 | details of how to record (and possibly negotiate) what character
|
---|
686 | encoding Maketext will return text in (UTF8? Latin-1? KOI8?). There's
|
---|
687 | the interesting fact that Maketext is for localization, but nowhere
|
---|
688 | actually has a "C<use locale;>" anywhere in it. For the curious,
|
---|
689 | there's the somewhat frightening details of how I actually
|
---|
690 | implement something like data inheritance so that searches across
|
---|
691 | modules' %Lexicon hashes can parallel how Perl implements method
|
---|
692 | inheritance.
|
---|
693 |
|
---|
694 | And, most importantly, there's all the practical details of how to
|
---|
695 | actually go about deriving from Maketext so you can use it for your
|
---|
696 | interfaces, and the various tools and conventions for starting out and
|
---|
697 | maintaining individual language modules.
|
---|
698 |
|
---|
699 | That is all covered in the documentation for Locale::Maketext and the
|
---|
700 | modules that come with it, available in CPAN. After having read this
|
---|
701 | article, which covers the why's of Maketext, the documentation,
|
---|
702 | which covers the how's of it, should be quite straightfoward.
|
---|
703 |
|
---|
704 | =head2 The Proof in the Pudding: Localizing Web Sites
|
---|
705 |
|
---|
706 | Maketext and gettext have a notable difference: gettext is in C,
|
---|
707 | accessible thru C library calls, whereas Maketext is in Perl, and
|
---|
708 | really can't work without a Perl interpreter (although I suppose
|
---|
709 | something like it could be written for C). Accidents of history (and
|
---|
710 | not necessarily lucky ones) have made C++ the most common language for
|
---|
711 | the implementation of applications like word processors, Web browsers,
|
---|
712 | and even many in-house applications like custom query systems. Current
|
---|
713 | conditions make it somewhat unlikely that the next one of any of these
|
---|
714 | kinds of applications will be written in Perl, albeit clearly more for
|
---|
715 | reasons of custom and inertia than out of consideration of what is the
|
---|
716 | right tool for the job.
|
---|
717 |
|
---|
718 | However, other accidents of history have made Perl a well-accepted
|
---|
719 | language for design of server-side programs (generally in CGI form)
|
---|
720 | for Web site interfaces. Localization of static pages in Web sites is
|
---|
721 | trivial, feasable either with simple language-negotiation features in
|
---|
722 | servers like Apache, or with some kind of server-side inclusions of
|
---|
723 | language-appropriate text into layout templates. However, I think
|
---|
724 | that the localization of Perl-based search systems (or other kinds of
|
---|
725 | dynamic content) in Web sites, be they public or access-restricted,
|
---|
726 | is where Maketext will see the greatest use.
|
---|
727 |
|
---|
728 | I presume that it would be only the exceptional Web site that gets
|
---|
729 | localized for English I<and> Chinese I<and> Italian I<and> Arabic
|
---|
730 | I<and> Russian, to recall the languages from the beginning of this
|
---|
731 | article -- to say nothing of German, Spanish, French, Japanese,
|
---|
732 | Finnish, and Hindi, to name a few languages that benefit from large
|
---|
733 | numbers of programmers or Web viewers or both.
|
---|
734 |
|
---|
735 | However, the ever-increasing internationalization of the Web (whether
|
---|
736 | measured in terms of amount of content, of numbers of content writers
|
---|
737 | or programmers, or of size of content audiences) makes it increasingly
|
---|
738 | likely that the interface to the average Web-based dynamic content
|
---|
739 | service will be localized for two or maybe three languages. It is my
|
---|
740 | hope that Maketext will make that task as simple as possible, and will
|
---|
741 | remove previous barriers to localization for languages dissimilar to
|
---|
742 | English.
|
---|
743 |
|
---|
744 | __END__
|
---|
745 |
|
---|
746 | Sean M. Burke (sburkeE<64>cpan.org) has a Master's in linguistics
|
---|
747 | from Northwestern University; he specializes in language technology.
|
---|
748 | Jordan Lachler (lachlerE<64>unm.edu) is a PhD student in the Department of
|
---|
749 | Linguistics at the University of New Mexico; he specializes in
|
---|
750 | morphology and pedagogy of North American native languages.
|
---|
751 |
|
---|
752 | =head2 References
|
---|
753 |
|
---|
754 | Alvestrand, Harald Tveit. 1995. I<RFC 1766: Tags for the
|
---|
755 | Identification of Languages.>
|
---|
756 | C<ftp://ftp.isi.edu/in-notes/rfc1766.txt>
|
---|
757 | [Now see RFC 3066.]
|
---|
758 |
|
---|
759 | Callon, Ross, editor. 1996. I<RFC 1925: The Twelve
|
---|
760 | Networking Truths.>
|
---|
761 | C<ftp://ftp.isi.edu/in-notes/rfc1925.txt>
|
---|
762 |
|
---|
763 | Drepper, Ulrich, Peter Miller,
|
---|
764 | and FranE<ccedil>ois Pinard. 1995-2001. GNU
|
---|
765 | C<gettext>. Available in C<ftp://prep.ai.mit.edu/pub/gnu/>, with
|
---|
766 | extensive docs in the distribution tarball. [Since
|
---|
767 | I wrote this article in 1998, I now see that the
|
---|
768 | gettext docs are now trying more to come to terms with
|
---|
769 | plurality. Whether useful conclusions have come from it
|
---|
770 | is another question altogether. -- SMB, May 2001]
|
---|
771 |
|
---|
772 | Forbes, Nevill. 1964. I<Russian Grammar.> Third Edition, revised
|
---|
773 | by J. C. Dumbreck. Oxford University Press.
|
---|
774 |
|
---|
775 | =cut
|
---|
776 |
|
---|
777 | #End
|
---|
778 |
|
---|