[31809] | 1 | Mapping files for Japanese encodings
|
---|
| 2 |
|
---|
| 3 | 1998 12/25
|
---|
| 4 |
|
---|
| 5 | Fuji Xerox Information Systems
|
---|
| 6 | MURATA Makoto
|
---|
| 7 |
|
---|
| 8 | 1. Overview
|
---|
| 9 |
|
---|
| 10 | This version of XML::Parser and XML::Encoding does not come with map files for
|
---|
| 11 | the charset "Shift_JIS" and the charset "euc-jp". Unfortunately, each of these
|
---|
| 12 | charsets has more than one mapping. None of these mappings are
|
---|
| 13 | considered as authoritative.
|
---|
| 14 |
|
---|
| 15 | Therefore, we have come to believe that it is dangerous to provide map files
|
---|
| 16 | for these charsets. Rather, we introduce several private charsets and map
|
---|
| 17 | files for these private charsets. If IANA, Unicode Consoritum, and JIS
|
---|
| 18 | eventually reach a consensus, we will be able to provide map files for
|
---|
| 19 | "Shift_JIS" and "euc-jp".
|
---|
| 20 |
|
---|
| 21 | 2. Different mappings from existing charsets to Unicode
|
---|
| 22 |
|
---|
| 23 | 1) Different mappings in JIS X0221 and Unicode
|
---|
| 24 |
|
---|
| 25 | The mapping between JIS X0208:1990 and Unicode 1.1 and the mapping
|
---|
| 26 | between JIS X0212:1990 and Unicode 1.1 are published from Unicode
|
---|
| 27 | consortium. They are available at
|
---|
| 28 | ftp://ftp.unicode.org/Public/MAPPINGS/EASTASIA/JIS/JIS0208.TXT and
|
---|
| 29 | ftp://ftp.unicode.org/Public/MAPPINGS/EASTASIA/JIS/JIS0212.TXT,
|
---|
| 30 | respectively.) These mapping files have a note as below:
|
---|
| 31 |
|
---|
| 32 | # The kanji mappings are a normative part of ISO/IEC 10646. The
|
---|
| 33 | # non-kanji mappings are provisional, pending definition of
|
---|
| 34 | # official mappings by Japanese standards bodies.
|
---|
| 35 |
|
---|
| 36 | Unfortunately, the non-kanji mappings in the Japanese standard for ISO 10646/1,
|
---|
| 37 | namely JIS X 0221:1995, is different from the Unicode Consortium mapping since
|
---|
| 38 | 0x213D of JIS X 0208 is mapped to U+2014 (em dash) rather than U+2015
|
---|
| 39 | (horizontal bar). Furthermore, JIS X 0221 clearly says that the mapping is
|
---|
| 40 | informational and non-normative. As a result, some companies (e.g., Microsoft and
|
---|
| 41 | Apple) have introduced slightly different mappings. Therefore, neither the
|
---|
| 42 | Unicode consortium mapping nor the JIS X 0221 mapping are considered as
|
---|
| 43 | authoritative.
|
---|
| 44 |
|
---|
| 45 | 2) Shift-JIS
|
---|
| 46 |
|
---|
| 47 | This charset is especially problematic, since its definition has been unclear
|
---|
| 48 | since its inception.
|
---|
| 49 |
|
---|
| 50 | The current registration of the charset "Shift_JIS" is as below:
|
---|
| 51 |
|
---|
| 52 | >Name: Shift_JIS (preferred MIME name)
|
---|
| 53 | >MIBenum: 17
|
---|
| 54 | >Source: A Microsoft code that extends csHalfWidthKatakana to include
|
---|
| 55 | > kanji by adding a second byte when the value of the first
|
---|
| 56 | > byte is in the ranges 81-9F or E0-EF.
|
---|
| 57 | >Alias: MS_Kanji
|
---|
| 58 | >Alias: csShiftJIS
|
---|
| 59 |
|
---|
| 60 | First, this does not reference to the mapping "Shift-JIS to Unicode"
|
---|
| 61 | published by the Unicode consortium (available at
|
---|
| 62 | ftp://ftp.unicode.org/Public/MAPPINGS/EASTASIA/JIS/SHIFTJIS.TXT).
|
---|
| 63 |
|
---|
| 64 | Second, "kanji" in this registration can be interepreted in different ways.
|
---|
| 65 | Does this "kanji" reference to JIS X0208:1978, JIS X0208:1983, or JIS
|
---|
| 66 | X0208:1990(== JIS X0208:1997)? These three standards are *incompatible* with
|
---|
| 67 | each other. Moreover, we can even argue that "kanji" refers to JIS X0212 or
|
---|
| 68 | ideographic characters in other countries.
|
---|
| 69 |
|
---|
| 70 | Third, each company has extended Shift JIS. For example, Microsoft introduced
|
---|
| 71 | OEM extensions (NEC extensionsand IBM extensions).
|
---|
| 72 |
|
---|
| 73 | Forth, Shift JIS uses JIS X0201, which is almost upper-compatible with US-ASCII
|
---|
| 74 | but is not quite. 5C and 7E of JIS X 0201 are different from backslash and
|
---|
| 75 | tilde, respectively. However, many programming languages (e.g., Java)
|
---|
| 76 | ignore this difference and assumes that 5C and 7E of Shift JIS are backslash
|
---|
| 77 | and tilde.
|
---|
| 78 |
|
---|
| 79 |
|
---|
| 80 | 3. Proposed charsets and mappings
|
---|
| 81 |
|
---|
| 82 | As a tentative solution, we introduce two private charsets for EUC-JP and four
|
---|
| 83 | priviate charsets for Shift JIS.
|
---|
| 84 |
|
---|
| 85 | 1) EUC-JP
|
---|
| 86 |
|
---|
| 87 | We have two charsets, namely "x-eucjp-unicode" and "x-eucjp-jisx0221". Their
|
---|
| 88 | difference is only one code point. The mapping for the former is based
|
---|
| 89 | on the Unicode Consortium mapping, while the latter is based on the JIS X0221
|
---|
| 90 | mapping.
|
---|
| 91 |
|
---|
| 92 | 2) Shift JIS
|
---|
| 93 |
|
---|
| 94 | We have four charsets, namely x-sjis-unicode, x-sjis-jisx0221,
|
---|
| 95 | x-sjis-jdk117, and x-sjis-cp932.
|
---|
| 96 |
|
---|
| 97 | The mapping for the charset x-sjis-unicode is the one published by the Unicode
|
---|
| 98 | consortium. The mapping for x-sjis-jisx0221 is almost equivalent to
|
---|
| 99 | x-sjis-unicode, but 0x213D of JIS X 0208 is mapped to U+2014 (em dash) rather
|
---|
| 100 | than U+2015. The charset x-sjis-jdk117 is again almost equivalent to
|
---|
| 101 | x-sjis-unicode, but 0x5C and 0x7E of JIS X0201 are mapped to backslash and
|
---|
| 102 | tilde.
|
---|
| 103 |
|
---|
| 104 | The charset x-sjis-cp932 is used by Microsoft Windows, and its mapping is
|
---|
| 105 | published from the Unicode Consortium (available at:
|
---|
| 106 | ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.txt). The
|
---|
| 107 | coded character set for this charset includes NEC-extensions and
|
---|
| 108 | IBM-extensions. 0x5C and 0x7E of JIS X0201 are mapped to backslash and tilde;
|
---|
| 109 | 0x213D is mapped to U+2015; and 0x2140, 0x2141, 0x2142, and 0x215E of JIS X
|
---|
| 110 | 0208 are mapped to compatibility characters.
|
---|
| 111 |
|
---|
| 112 | Makoto
|
---|
| 113 |
|
---|
| 114 | Fuji Xerox Information Systems
|
---|
| 115 |
|
---|
| 116 | Tel: +81-44-812-7230 Fax: +81-44-812-7231
|
---|
| 117 | E-mail: [email protected]
|
---|