1 | package utf8;
|
---|
2 |
|
---|
3 | $utf8::hint_bits = 0x00800000;
|
---|
4 |
|
---|
5 | our $VERSION = '1.06';
|
---|
6 |
|
---|
7 | sub import {
|
---|
8 | $^H |= $utf8::hint_bits;
|
---|
9 | $enc{caller()} = $_[1] if $_[1];
|
---|
10 | }
|
---|
11 |
|
---|
12 | sub unimport {
|
---|
13 | $^H &= ~$utf8::hint_bits;
|
---|
14 | }
|
---|
15 |
|
---|
16 | sub AUTOLOAD {
|
---|
17 | require "utf8_heavy.pl";
|
---|
18 | goto &$AUTOLOAD if defined &$AUTOLOAD;
|
---|
19 | require Carp;
|
---|
20 | Carp::croak("Undefined subroutine $AUTOLOAD called");
|
---|
21 | }
|
---|
22 |
|
---|
23 | 1;
|
---|
24 | __END__
|
---|
25 |
|
---|
26 | =head1 NAME
|
---|
27 |
|
---|
28 | utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code
|
---|
29 |
|
---|
30 | =head1 SYNOPSIS
|
---|
31 |
|
---|
32 | use utf8;
|
---|
33 | no utf8;
|
---|
34 |
|
---|
35 | # Convert a Perl scalar to/from UTF-8.
|
---|
36 | $num_octets = utf8::upgrade($string);
|
---|
37 | $success = utf8::downgrade($string[, FAIL_OK]);
|
---|
38 |
|
---|
39 | # Change the native bytes of a Perl scalar to/from UTF-8 bytes.
|
---|
40 | utf8::encode($string);
|
---|
41 | utf8::decode($string);
|
---|
42 |
|
---|
43 | $flag = utf8::is_utf8(STRING); # since Perl 5.8.1
|
---|
44 | $flag = utf8::valid(STRING);
|
---|
45 |
|
---|
46 | =head1 DESCRIPTION
|
---|
47 |
|
---|
48 | The C<use utf8> pragma tells the Perl parser to allow UTF-8 in the
|
---|
49 | program text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based
|
---|
50 | platforms). The C<no utf8> pragma tells Perl to switch back to treating
|
---|
51 | the source text as literal bytes in the current lexical scope.
|
---|
52 |
|
---|
53 | This pragma is primarily a compatibility device. Perl versions
|
---|
54 | earlier than 5.6 allowed arbitrary bytes in source code, whereas
|
---|
55 | in future we would like to standardize on the UTF-8 encoding for
|
---|
56 | source text.
|
---|
57 |
|
---|
58 | B<Do not use this pragma for anything else than telling Perl that your
|
---|
59 | script is written in UTF-8.> The utility functions described below are
|
---|
60 | useful for their own purposes, but they are not really part of the
|
---|
61 | "pragmatic" effect.
|
---|
62 |
|
---|
63 | Until UTF-8 becomes the default format for source text, either this
|
---|
64 | pragma or the L<encoding> pragma should be used to recognize UTF-8
|
---|
65 | in the source. When UTF-8 becomes the standard source format, this
|
---|
66 | pragma will effectively become a no-op. For convenience in what
|
---|
67 | follows the term I<UTF-X> is used to refer to UTF-8 on ASCII and ISO
|
---|
68 | Latin based platforms and UTF-EBCDIC on EBCDIC based platforms.
|
---|
69 |
|
---|
70 | See also the effects of the C<-C> switch and its cousin, the
|
---|
71 | C<$ENV{PERL_UNICODE}>, in L<perlrun>.
|
---|
72 |
|
---|
73 | Enabling the C<utf8> pragma has the following effect:
|
---|
74 |
|
---|
75 | =over 4
|
---|
76 |
|
---|
77 | =item *
|
---|
78 |
|
---|
79 | Bytes in the source text that have their high-bit set will be treated
|
---|
80 | as being part of a literal UTF-8 character. This includes most
|
---|
81 | literals such as identifier names, string constants, and constant
|
---|
82 | regular expression patterns.
|
---|
83 |
|
---|
84 | On EBCDIC platforms characters in the Latin 1 character set are
|
---|
85 | treated as being part of a literal UTF-EBCDIC character.
|
---|
86 |
|
---|
87 | =back
|
---|
88 |
|
---|
89 | Note that if you have bytes with the eighth bit on in your script
|
---|
90 | (for example embedded Latin-1 in your string literals), C<use utf8>
|
---|
91 | will be unhappy since the bytes are most probably not well-formed
|
---|
92 | UTF-8. If you want to have such bytes and use utf8, you can disable
|
---|
93 | utf8 until the end the block (or file, if at top level) by C<no utf8;>.
|
---|
94 |
|
---|
95 | If you want to automatically upgrade your 8-bit legacy bytes to UTF-8,
|
---|
96 | use the L<encoding> pragma instead of this pragma. For example, if
|
---|
97 | you want to implicitly upgrade your ISO 8859-1 (Latin-1) bytes to UTF-8
|
---|
98 | as used in e.g. C<chr()> and C<\x{...}>, try this:
|
---|
99 |
|
---|
100 | use encoding "latin-1";
|
---|
101 | my $c = chr(0xc4);
|
---|
102 | my $x = "\x{c5}";
|
---|
103 |
|
---|
104 | In case you are wondering: yes, C<use encoding 'utf8';> works much
|
---|
105 | the same as C<use utf8;>.
|
---|
106 |
|
---|
107 | =head2 Utility functions
|
---|
108 |
|
---|
109 | The following functions are defined in the C<utf8::> package by the
|
---|
110 | Perl core. You do not need to say C<use utf8> to use these and in fact
|
---|
111 | you should not say that unless you really want to have UTF-8 source code.
|
---|
112 |
|
---|
113 | =over 4
|
---|
114 |
|
---|
115 | =item * $num_octets = utf8::upgrade($string)
|
---|
116 |
|
---|
117 | Converts in-place the octet sequence in the native encoding
|
---|
118 | (Latin-1 or EBCDIC) to the equivalent character sequence in I<UTF-X>.
|
---|
119 | I<$string> already encoded as characters does no harm.
|
---|
120 | Returns the number of octets necessary to represent the string as I<UTF-X>.
|
---|
121 | Can be used to make sure that the UTF-8 flag is on,
|
---|
122 | so that C<\w> or C<lc()> work as Unicode on strings
|
---|
123 | containing characters in the range 0x80-0xFF (on ASCII and
|
---|
124 | derivatives).
|
---|
125 |
|
---|
126 | B<Note that this function does not handle arbitrary encodings.>
|
---|
127 | Therefore I<Encode.pm> is recommended for the general purposes.
|
---|
128 |
|
---|
129 | Affected by the encoding pragma.
|
---|
130 |
|
---|
131 | =item * $success = utf8::downgrade($string[, FAIL_OK])
|
---|
132 |
|
---|
133 | Converts in-place the character sequence in I<UTF-X>
|
---|
134 | to the equivalent octet sequence in the native encoding (Latin-1 or EBCDIC).
|
---|
135 | I<$string> already encoded as octets does no harm.
|
---|
136 | Returns true on success. On failure dies or, if the value of
|
---|
137 | C<FAIL_OK> is true, returns false.
|
---|
138 | Can be used to make sure that the UTF-8 flag is off,
|
---|
139 | e.g. when you want to make sure that the substr() or length() function
|
---|
140 | works with the usually faster byte algorithm.
|
---|
141 |
|
---|
142 | B<Note that this function does not handle arbitrary encodings.>
|
---|
143 | Therefore I<Encode.pm> is recommended for the general purposes.
|
---|
144 |
|
---|
145 | B<Not> affected by the encoding pragma.
|
---|
146 |
|
---|
147 | B<NOTE:> this function is experimental and may change
|
---|
148 | or be removed without notice.
|
---|
149 |
|
---|
150 | =item * utf8::encode($string)
|
---|
151 |
|
---|
152 | Converts in-place the character sequence to the corresponding octet sequence
|
---|
153 | in I<UTF-X>. The UTF-8 flag is turned off. Returns nothing.
|
---|
154 |
|
---|
155 | B<Note that this function does not handle arbitrary encodings.>
|
---|
156 | Therefore I<Encode.pm> is recommended for the general purposes.
|
---|
157 |
|
---|
158 | =item * utf8::decode($string)
|
---|
159 |
|
---|
160 | Attempts to convert in-place the octet sequence in I<UTF-X>
|
---|
161 | to the corresponding character sequence. The UTF-8 flag is turned on
|
---|
162 | only if the source string contains multiple-byte I<UTF-X> characters.
|
---|
163 | If I<$string> is invalid as I<UTF-X>, returns false; otherwise returns true.
|
---|
164 |
|
---|
165 | B<Note that this function does not handle arbitrary encodings.>
|
---|
166 | Therefore I<Encode.pm> is recommended for the general purposes.
|
---|
167 |
|
---|
168 | B<NOTE:> this function is experimental and may change
|
---|
169 | or be removed without notice.
|
---|
170 |
|
---|
171 | =item * $flag = utf8::is_utf8(STRING)
|
---|
172 |
|
---|
173 | (Since Perl 5.8.1) Test whether STRING is in UTF-8. Functionally
|
---|
174 | the same as Encode::is_utf8().
|
---|
175 |
|
---|
176 | =item * $flag = utf8::valid(STRING)
|
---|
177 |
|
---|
178 | [INTERNAL] Test whether STRING is in a consistent state regarding
|
---|
179 | UTF-8. Will return true is well-formed UTF-8 and has the UTF-8 flag
|
---|
180 | on B<or> if string is held as bytes (both these states are 'consistent').
|
---|
181 | Main reason for this routine is to allow Perl's testsuite to check
|
---|
182 | that operations have left strings in a consistent state. You most
|
---|
183 | probably want to use utf8::is_utf8() instead.
|
---|
184 |
|
---|
185 | =back
|
---|
186 |
|
---|
187 | C<utf8::encode> is like C<utf8::upgrade>, but the UTF8 flag is
|
---|
188 | cleared. See L<perlunicode> for more on the UTF8 flag and the C API
|
---|
189 | functions C<sv_utf8_upgrade>, C<sv_utf8_downgrade>, C<sv_utf8_encode>,
|
---|
190 | and C<sv_utf8_decode>, which are wrapped by the Perl functions
|
---|
191 | C<utf8::upgrade>, C<utf8::downgrade>, C<utf8::encode> and
|
---|
192 | C<utf8::decode>. Note that in the Perl 5.8.0 and 5.8.1 implementation
|
---|
193 | the functions utf8::is_utf8, utf8::valid, utf8::encode, utf8::decode,
|
---|
194 | utf8::upgrade, and utf8::downgrade are always available, without a
|
---|
195 | C<require utf8> statement-- this may change in future releases.
|
---|
196 |
|
---|
197 | =head1 BUGS
|
---|
198 |
|
---|
199 | One can have Unicode in identifier names, but not in package/class or
|
---|
200 | subroutine names. While some limited functionality towards this does
|
---|
201 | exist as of Perl 5.8.0, that is more accidental than designed; use of
|
---|
202 | Unicode for the said purposes is unsupported.
|
---|
203 |
|
---|
204 | One reason of this unfinishedness is its (currently) inherent
|
---|
205 | unportability: since both package names and subroutine names may need
|
---|
206 | to be mapped to file and directory names, the Unicode capability of
|
---|
207 | the filesystem becomes important-- and there unfortunately aren't
|
---|
208 | portable answers.
|
---|
209 |
|
---|
210 | =head1 SEE ALSO
|
---|
211 |
|
---|
212 | L<perluniintro>, L<encoding>, L<perlrun>, L<bytes>, L<perlunicode>
|
---|
213 |
|
---|
214 | =cut
|
---|