FM / pod

/usr/share/perl5/pod

Name	Size	Mode	Actions
perl.pod	16270	0644	edit dl rm
perl56delta.pod	107201	0644	edit dl rm
perl58delta.pod	115165	0644	edit dl rm
perl561delta.pod	124713	0644	edit dl rm
perl581delta.pod	38061	0644	edit dl rm
perl582delta.pod	4470	0644	edit dl rm
perl583delta.pod	6335	0644	edit dl rm
perl584delta.pod	7363	0644	edit dl rm
perl585delta.pod	5889	0644	edit dl rm
perl586delta.pod	4651	0644	edit dl rm
perl587delta.pod	8357	0644	edit dl rm
perl588delta.pod	25272	0644	edit dl rm
perl589delta.pod	53900	0644	edit dl rm
perl5004delta.pod	56240	0644	edit dl rm
perl5005delta.pod	34283	0644	edit dl rm
perl5100delta.pod	55535	0644	edit dl rm
perl5101delta.pod	43888	0644	edit dl rm
perl5120delta.pod	89272	0644	edit dl rm
perl5121delta.pod	10141	0644	edit dl rm
perl5122delta.pod	9603	0644	edit dl rm
perl5123delta.pod	4100	0644	edit dl rm
perl5124delta.pod	3672	0644	edit dl rm
perl5125delta.pod	7683	0644	edit dl rm
perl5140delta.pod	144324	0644	edit dl rm
perl5141delta.pod	7966	0644	edit dl rm
perl5142delta.pod	6892	0644	edit dl rm
perl5143delta.pod	7760	0644	edit dl rm
perl5144delta.pod	6327	0644	edit dl rm
perl5160delta.pod	133651	0644	edit dl rm
perl5161delta.pod	6142	0644	edit dl rm
perl5162delta.pod	3594	0644	edit dl rm
perl5163delta.pod	4085	0644	edit dl rm
perl5180delta.pod	119431	0644	edit dl rm
perl5181delta.pod	6595	0644	edit dl rm
perl5182delta.pod	5335	0644	edit dl rm
perl5184delta.pod	4642	0644	edit dl rm
perl5200delta.pod	115699	0644	edit dl rm
perl5201delta.pod	10899	0644	edit dl rm
perl5202delta.pod	12509	0644	edit dl rm
perl5203delta.pod	9392	0644	edit dl rm
perl5220delta.pod	130963	0644	edit dl rm
perl5221delta.pod	10767	0644	edit dl rm
perl5222delta.pod	12629	0644	edit dl rm
perl5223delta.pod	8456	0644	edit dl rm
perl5224delta.pod	4460	0644	edit dl rm
perl5240delta.pod	64927	0644	edit dl rm
perl5241delta.pod	8215	0644	edit dl rm
perl5242delta.pod	4113	0644	edit dl rm
perl5243delta.pod	11428	0644	edit dl rm
perl5244delta.pod	4510	0644	edit dl rm
perl5260delta.pod	101836	0644	edit dl rm
perl5261delta.pod	7927	0644	edit dl rm
perl5262delta.pod	7880	0644	edit dl rm
perl5263delta.pod	7063	0644	edit dl rm
perl5280delta.pod	72113	0644	edit dl rm
perlaix.pod	20437	0644	edit dl rm
perlamiga.pod	5749	0644	edit dl rm
perlandroid.pod	7871	0644	edit dl rm
perlapi.pod	443535	0644	edit dl rm
perlapio.pod	19285	0644	edit dl rm
perlartistic.pod	7010	0644	edit dl rm
perlbook.pod	8338	0644	edit dl rm
perlboot.pod	294	0644	edit dl rm
perlbot.pod	304	0644	edit dl rm
perlbs2000.pod	8058	0644	edit dl rm
perlcall.pod	56706	0644	edit dl rm
perlce.pod	14602	0644	edit dl rm
perlcheat.pod	4481	0644	edit dl rm
perlclib.pod	9619	0644	edit dl rm
perlcn.pod	4691	0644	edit dl rm
perlcommunity.pod	7217	0644	edit dl rm
perlcygwin.pod	27199	0644	edit dl rm
perldata.pod	46743	0644	edit dl rm
perldbmfilter.pod	4981	0644	edit dl rm
perldebguts.pod	38535	0644	edit dl rm
perldebtut.pod	22152	0644	edit dl rm
perldebug.pod	39258	0644	edit dl rm
perldelta.pod	7063	0644	edit dl rm
perldeprecation.pod	18169	0644	edit dl rm
perldiag.pod	284572	0644	edit dl rm
perldos.pod	10522	0644	edit dl rm
perldsc.pod	25614	0644	edit dl rm
perldtrace.pod	7958	0644	edit dl rm
perlebcdic.pod	84233	0644	edit dl rm
perlembed.pod	37196	0644	edit dl rm
perlexperiment.pod	7195	0644	edit dl rm
perlfork.pod	13355	0644	edit dl rm
perlform.pod	16608	0644	edit dl rm
perlfreebsd.pod	1610	0644	edit dl rm
perlfunc.pod	392957	0644	edit dl rm
perlgit.pod	33509	0644	edit dl rm
perlgpl.pod	13815	0644	edit dl rm
perlguts.pod	139329	0644	edit dl rm
perlhack.pod	40445	0644	edit dl rm
perlhacktips.pod	55509	0644	edit dl rm
perlhacktut.pod	6153	0644	edit dl rm
perlhaiku.pod	1508	0644	edit dl rm
perlhist.pod	53546	0644	edit dl rm
perlhpux.pod	30509	0644	edit dl rm
perlhurd.pod	1993	0644	edit dl rm
perlintern.pod	54572	0644	edit dl rm
perlinterp.pod	33687	0644	edit dl rm
perlintro.pod	22119	0644	edit dl rm
perliol.pod	34185	0644	edit dl rm
perlipc.pod	70829	0644	edit dl rm
perlirix.pod	4395	0644	edit dl rm
perljp.pod	7521	0644	edit dl rm
perlko.pod	12259	0644	edit dl rm
perllexwarn.pod	355	0644	edit dl rm
perllinux.pod	1488	0644	edit dl rm
perllocale.pod	68678	0644	edit dl rm
perllol.pod	9580	0644	edit dl rm
perlmacos.pod	1001	0644	edit dl rm
perlmacosx.pod	12060	0644	edit dl rm
perlmod.pod	26250	0644	edit dl rm
perlmodinstall.pod	12792	0644	edit dl rm
perlmodlib.pod	76482	0644	edit dl rm
perlmodstyle.pod	22575	0644	edit dl rm
perlmroapi.pod	3212	0644	edit dl rm
perlnetware.pod	6648	0644	edit dl rm
perlnewmod.pod	11036	0644	edit dl rm
perlnumber.pod	8353	0644	edit dl rm
perlobj.pod	35537	0644	edit dl rm
perlootut.pod	26783	0644	edit dl rm
perlop.pod	136252	0644	edit dl rm
perlopenbsd.pod	1204	0644	edit dl rm
perlopentut.pod	9455	0644	edit dl rm
perlos2.pod	93351	0644	edit dl rm
perlos390.pod	15674	0644	edit dl rm
perlos400.pod	4768	0644	edit dl rm
perlpacktut.pod	51282	0644	edit dl rm
perlperf.pod	49881	0644	edit dl rm
perlplan9.pod	5125	0644	edit dl rm
perlpod.pod	22196	0644	edit dl rm
perlpodspec.pod	68476	0644	edit dl rm
perlpolicy.pod	25629	0644	edit dl rm
perlport.pod	87602	0644	edit dl rm
perlpragma.pod	5176	0644	edit dl rm
perlqnx.pod	6673	0644	edit dl rm
perlre.pod	120901	0644	edit dl rm
perlreapi.pod	30334	0644	edit dl rm
perlrebackslash.pod	31817	0644	edit dl rm
perlrecharclass.pod	49029	0644	edit dl rm
perlref.pod	35304	0644	edit dl rm
perlreftut.pod	18790	0644	edit dl rm
perlreguts.pod	38328	0644	edit dl rm
perlrepository.pod	509	0644	edit dl rm
perlrequick.pod	18497	0644	edit dl rm
perlreref.pod	14744	0644	edit dl rm
perlretut.pod	121257	0644	edit dl rm
perlriscos.pod	1529	0644	edit dl rm
perlrun.pod	53550	0644	edit dl rm
perlsec.pod	26184	0644	edit dl rm
perlsolaris.pod	29822	0644	edit dl rm
perlsource.pod	6876	0644	edit dl rm
perlstyle.pod	8630	0644	edit dl rm
perlsub.pod	72967	0644	edit dl rm
perlsymbian.pod	15359	0644	edit dl rm
perlsyn.pod	44512	0644	edit dl rm
perlsynology.pod	7778	0644	edit dl rm
perlthrtut.pod	46459	0644	edit dl rm
perltie.pod	38607	0644	edit dl rm
perltoc.pod	694171	0644	edit dl rm
perltodo.pod	376	0644	edit dl rm
perltooc.pod	294	0644	edit dl rm
perltoot.pod	294	0644	edit dl rm
perltrap.pod	10620	0644	edit dl rm
perltru64.pod	8492	0644	edit dl rm
perltw.pod	4477	0644	edit dl rm
perlunicode.pod	82491	0644	edit dl rm
perlunicook.pod	25488	0644	edit dl rm
perlunifaq.pod	13647	0644	edit dl rm
perluniintro.pod	38340	0644	edit dl rm
perluniprops.pod	285306	0644	edit dl rm
perlunitut.pod	7951	0644	edit dl rm
perlutil.pod	7640	0644	edit dl rm
perlvar.pod	78364	0644	edit dl rm
perlvms.pod	50823	0644	edit dl rm
perlvos.pod	3843	0644	edit dl rm
perlwin32.pod	39298	0644	edit dl rm
perlxs.pod	78920	0644	edit dl rm
perlxstut.pod	50095	0644	edit dl rm
perlxstypemap.pod	24001	0644	edit dl rm

Edit: /usr/share/perl5/pod/perlunitut.pod (7951B)

=head1 NAME

perlunitut - Perl Unicode Tutorial

=head1 DESCRIPTION

The days of just flinging strings around are over. It's well established that
modern programs need to be capable of communicating funny accented letters, and
things like euro symbols. This means that programmers need new habits. It's
easy to program Unicode capable software, but it does require discipline to do
it right.

There's a lot to know about character sets, and text encodings. It's probably
best to spend a full day learning all this, but the basics can be learned in
minutes.

These are not the very basics, though. It is assumed that you already
know the difference between bytes and characters, and realise (and accept!)
that there are many different character sets and encodings, and that your
program has to be explicit about them. Recommended reading is "The Absolute
Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
and Character Sets (No Excuses!)" by Joel Spolsky, at
L<http://joelonsoftware.com/articles/Unicode.html>.

This tutorial speaks in rather absolute terms, and provides only a limited view
of the wealth of character string related features that Perl has to offer. For
most projects, this information will probably suffice.

=head2 Definitions

It's important to set a few things straight first. This is the most important
part of this tutorial. This view may conflict with other information that you
may have found on the web, but that's mostly because many sources are wrong.

You may have to re-read this entire section a few times...

=head3 Unicode

B<Unicode> is a character set with room for lots of characters. The ordinal
value of a character is called a B<code point>.   (But in practice, the
distinction between code point and character is blurred, so the terms often
are used interchangeably.)

There are many, many code points, but computers work with bytes, and a byte has
room for only 256 values.  Unicode has many more characters than that,
so you need a method to make these accessible.

Unicode is encoded using several competing encodings, of which UTF-8 is the
most used. In a Unicode encoding, multiple subsequent bytes can be used to
store a single code point, or simply: character.

=head3 UTF-8

B<UTF-8> is a Unicode encoding. Many people think that Unicode and UTF-8 are
the same thing, but they're not. There are more Unicode encodings, but much of
the world has standardized on UTF-8.

UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
only one byte per character. All other characters are encoded as two to
four bytes using a complex scheme. Fortunately, Perl handles this for
us, so we don't have to worry about this.

=head3 Text strings (character strings)

B<Text strings>, or B<character strings> are made of characters. Bytes are
irrelevant here, and so are encodings. Each character is just that: the
character.

On a text string, you would do things like:

$text =~ s/foo/bar/;
    if ($string =~ /^\d+$/) { ... }
    $text = ucfirst $text;
    my $character_count = length $text;

The value of a character (C<ord>, C<chr>) is the corresponding Unicode code
point.

=head3 Binary strings (byte strings)

B<Binary strings>, or B<byte strings> are made of bytes. Here, you don't have
characters, just bytes. All communication with the outside world (anything
outside of your current Perl process) is done in binary.

On a binary string, you would do things like:

my (@length_content) = unpack "(V/a)*", $binary;
    $binary =~ s/\x00\x0F/\xFF\xF0/;  # for the brave :)
    print {$fh} $binary;
    my $byte_count = length $binary;

=head3 Encoding

B<Encoding> (as a verb) is the conversion from I<text> to I<binary>. To encode,
you have to supply the target encoding, for example C<iso-8859-1> or C<UTF-8>.
Some encodings, like the C<iso-8859> ("latin") range, do not support the full
Unicode standard; characters that can't be represented are lost in the
conversion.

=head3 Decoding

B<Decoding> is the conversion from I<binary> to I<text>. To decode, you have to
know what encoding was used during the encoding phase. And most of all, it must
be something decodable. It doesn't make much sense to decode a PNG image into a
text string.

=head3 Internal format

Perl has an B<internal format>, an encoding that it uses to encode text strings
so it can store them in memory. All text strings are in this internal format.
In fact, text strings are never in any other format!

You shouldn't worry about what this format is, because conversion is
automatically done when you decode or encode.

=head2 Your new toolkit

Add to your standard heading the following line:

use Encode qw(encode decode);

Or, if you're lazy, just:

use Encode;

=head2 I/O flow (the actual 5 minute tutorial)

The typical input/output flow of a program is:

1. Receive and decode
    2. Process
    3. Encode and output

If your input is binary, and is supposed to remain binary, you shouldn't decode
it to a text string, of course. But in all other cases, you should decode it.

Decoding can't happen reliably if you don't know how the data was encoded. If
you get to choose, it's a good idea to standardize on UTF-8.

my $foo   = decode('UTF-8', get 'http://example.com/');
    my $bar   = decode('ISO-8859-1', readline STDIN);
    my $xyzzy = decode('Windows-1251', $cgi->param('foo'));

Processing happens as you knew before. The only difference is that you're now
using characters instead of bytes. That's very useful if you use things like
C<substr>, or C<length>.

It's important to realize that there are no bytes in a text string. Of course,
Perl has its internal encoding to store the string in memory, but ignore that.
If you have to do anything with the number of bytes, it's probably best to move
that part to step 3, just after you've encoded the string. Then you know
exactly how many bytes it will be in the destination string.

The syntax for encoding text strings to binary strings is as simple as decoding:

$body = encode('UTF-8', $body);

If you needed to know the length of the string in bytes, now's the perfect time
for that. Because C<$body> is now a byte string, C<length> will report the
number of bytes, instead of the number of characters. The number of
characters is no longer known, because characters only exist in text strings.

my $byte_count = length $body;

And if the protocol you're using supports a way of letting the recipient know
which character encoding you used, please help the receiving end by using that
feature! For example, E-mail and HTTP support MIME headers, so you can use the
C<Content-Type> header. They can also have C<Content-Length> to indicate the
number of I<bytes>, which is always a good idea to supply if the number is
known.

"Content-Type: text/plain; charset=UTF-8",
    "Content-Length: $byte_count"

=head1 SUMMARY

Decode everything you receive, encode everything you send out. (If it's text
data.)

=head1 Q and A (or FAQ)

After reading this document, you ought to read L<perlunifaq> too, then
L<perluniintro>.

=head1 ACKNOWLEDGEMENTS

Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the
Amsterdam Perl Mongers meetings got me interested and determined to find out
how to use character encodings in Perl in ways that don't break easily.

Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch
Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.

Thanks to the people who asked about this kind of stuff in several Perl IRC
channels, and have constantly reminded me that a simpler explanation was
needed.

Thanks to the people who reviewed this document for me, before it went public.
They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
Gray.

=head1 AUTHOR

Juerd Waalboer <#####@juerd.nl>

=head1 SEE ALSO

L<perlunifaq>, L<perlunicode>, L<perluniintro>, L<Encode>

cmd: