FM / pod

/usr/share/perl5/pod

Name	Size	Mode	Actions
perl.pod	16270	0644	edit dl rm
perl56delta.pod	107201	0644	edit dl rm
perl58delta.pod	115165	0644	edit dl rm
perl561delta.pod	124713	0644	edit dl rm
perl581delta.pod	38061	0644	edit dl rm
perl582delta.pod	4470	0644	edit dl rm
perl583delta.pod	6335	0644	edit dl rm
perl584delta.pod	7363	0644	edit dl rm
perl585delta.pod	5889	0644	edit dl rm
perl586delta.pod	4651	0644	edit dl rm
perl587delta.pod	8357	0644	edit dl rm
perl588delta.pod	25272	0644	edit dl rm
perl589delta.pod	53900	0644	edit dl rm
perl5004delta.pod	56240	0644	edit dl rm
perl5005delta.pod	34283	0644	edit dl rm
perl5100delta.pod	55535	0644	edit dl rm
perl5101delta.pod	43888	0644	edit dl rm
perl5120delta.pod	89272	0644	edit dl rm
perl5121delta.pod	10141	0644	edit dl rm
perl5122delta.pod	9603	0644	edit dl rm
perl5123delta.pod	4100	0644	edit dl rm
perl5124delta.pod	3672	0644	edit dl rm
perl5125delta.pod	7683	0644	edit dl rm
perl5140delta.pod	144324	0644	edit dl rm
perl5141delta.pod	7966	0644	edit dl rm
perl5142delta.pod	6892	0644	edit dl rm
perl5143delta.pod	7760	0644	edit dl rm
perl5144delta.pod	6327	0644	edit dl rm
perl5160delta.pod	133651	0644	edit dl rm
perl5161delta.pod	6142	0644	edit dl rm
perl5162delta.pod	3594	0644	edit dl rm
perl5163delta.pod	4085	0644	edit dl rm
perl5180delta.pod	119431	0644	edit dl rm
perl5181delta.pod	6595	0644	edit dl rm
perl5182delta.pod	5335	0644	edit dl rm
perl5184delta.pod	4642	0644	edit dl rm
perl5200delta.pod	115699	0644	edit dl rm
perl5201delta.pod	10899	0644	edit dl rm
perl5202delta.pod	12509	0644	edit dl rm
perl5203delta.pod	9392	0644	edit dl rm
perl5220delta.pod	130963	0644	edit dl rm
perl5221delta.pod	10767	0644	edit dl rm
perl5222delta.pod	12629	0644	edit dl rm
perl5223delta.pod	8456	0644	edit dl rm
perl5224delta.pod	4460	0644	edit dl rm
perl5240delta.pod	64927	0644	edit dl rm
perl5241delta.pod	8215	0644	edit dl rm
perl5242delta.pod	4113	0644	edit dl rm
perl5243delta.pod	11428	0644	edit dl rm
perl5244delta.pod	4510	0644	edit dl rm
perl5260delta.pod	101836	0644	edit dl rm
perl5261delta.pod	7927	0644	edit dl rm
perl5262delta.pod	7880	0644	edit dl rm
perl5263delta.pod	7063	0644	edit dl rm
perl5280delta.pod	72113	0644	edit dl rm
perlaix.pod	20437	0644	edit dl rm
perlamiga.pod	5749	0644	edit dl rm
perlandroid.pod	7871	0644	edit dl rm
perlapi.pod	443535	0644	edit dl rm
perlapio.pod	19285	0644	edit dl rm
perlartistic.pod	7010	0644	edit dl rm
perlbook.pod	8338	0644	edit dl rm
perlboot.pod	294	0644	edit dl rm
perlbot.pod	304	0644	edit dl rm
perlbs2000.pod	8058	0644	edit dl rm
perlcall.pod	56706	0644	edit dl rm
perlce.pod	14602	0644	edit dl rm
perlcheat.pod	4481	0644	edit dl rm
perlclib.pod	9619	0644	edit dl rm
perlcn.pod	4691	0644	edit dl rm
perlcommunity.pod	7217	0644	edit dl rm
perlcygwin.pod	27199	0644	edit dl rm
perldata.pod	46743	0644	edit dl rm
perldbmfilter.pod	4981	0644	edit dl rm
perldebguts.pod	38535	0644	edit dl rm
perldebtut.pod	22152	0644	edit dl rm
perldebug.pod	39258	0644	edit dl rm
perldelta.pod	7063	0644	edit dl rm
perldeprecation.pod	18169	0644	edit dl rm
perldiag.pod	284572	0644	edit dl rm
perldos.pod	10522	0644	edit dl rm
perldsc.pod	25614	0644	edit dl rm
perldtrace.pod	7958	0644	edit dl rm
perlebcdic.pod	84233	0644	edit dl rm
perlembed.pod	37196	0644	edit dl rm
perlexperiment.pod	7195	0644	edit dl rm
perlfork.pod	13355	0644	edit dl rm
perlform.pod	16608	0644	edit dl rm
perlfreebsd.pod	1610	0644	edit dl rm
perlfunc.pod	392957	0644	edit dl rm
perlgit.pod	33509	0644	edit dl rm
perlgpl.pod	13815	0644	edit dl rm
perlguts.pod	139329	0644	edit dl rm
perlhack.pod	40445	0644	edit dl rm
perlhacktips.pod	55509	0644	edit dl rm
perlhacktut.pod	6153	0644	edit dl rm
perlhaiku.pod	1508	0644	edit dl rm
perlhist.pod	53546	0644	edit dl rm
perlhpux.pod	30509	0644	edit dl rm
perlhurd.pod	1993	0644	edit dl rm
perlintern.pod	54572	0644	edit dl rm
perlinterp.pod	33687	0644	edit dl rm
perlintro.pod	22119	0644	edit dl rm
perliol.pod	34185	0644	edit dl rm
perlipc.pod	70829	0644	edit dl rm
perlirix.pod	4395	0644	edit dl rm
perljp.pod	7521	0644	edit dl rm
perlko.pod	12259	0644	edit dl rm
perllexwarn.pod	355	0644	edit dl rm
perllinux.pod	1488	0644	edit dl rm
perllocale.pod	68678	0644	edit dl rm
perllol.pod	9580	0644	edit dl rm
perlmacos.pod	1001	0644	edit dl rm
perlmacosx.pod	12060	0644	edit dl rm
perlmod.pod	26250	0644	edit dl rm
perlmodinstall.pod	12792	0644	edit dl rm
perlmodlib.pod	76482	0644	edit dl rm
perlmodstyle.pod	22575	0644	edit dl rm
perlmroapi.pod	3212	0644	edit dl rm
perlnetware.pod	6648	0644	edit dl rm
perlnewmod.pod	11036	0644	edit dl rm
perlnumber.pod	8353	0644	edit dl rm
perlobj.pod	35537	0644	edit dl rm
perlootut.pod	26783	0644	edit dl rm
perlop.pod	136252	0644	edit dl rm
perlopenbsd.pod	1204	0644	edit dl rm
perlopentut.pod	9455	0644	edit dl rm
perlos2.pod	93351	0644	edit dl rm
perlos390.pod	15674	0644	edit dl rm
perlos400.pod	4768	0644	edit dl rm
perlpacktut.pod	51282	0644	edit dl rm
perlperf.pod	49881	0644	edit dl rm
perlplan9.pod	5125	0644	edit dl rm
perlpod.pod	22196	0644	edit dl rm
perlpodspec.pod	68476	0644	edit dl rm
perlpolicy.pod	25629	0644	edit dl rm
perlport.pod	87602	0644	edit dl rm
perlpragma.pod	5176	0644	edit dl rm
perlqnx.pod	6673	0644	edit dl rm
perlre.pod	120901	0644	edit dl rm
perlreapi.pod	30334	0644	edit dl rm
perlrebackslash.pod	31817	0644	edit dl rm
perlrecharclass.pod	49029	0644	edit dl rm
perlref.pod	35304	0644	edit dl rm
perlreftut.pod	18790	0644	edit dl rm
perlreguts.pod	38328	0644	edit dl rm
perlrepository.pod	509	0644	edit dl rm
perlrequick.pod	18497	0644	edit dl rm
perlreref.pod	14744	0644	edit dl rm
perlretut.pod	121257	0644	edit dl rm
perlriscos.pod	1529	0644	edit dl rm
perlrun.pod	53550	0644	edit dl rm
perlsec.pod	26184	0644	edit dl rm
perlsolaris.pod	29822	0644	edit dl rm
perlsource.pod	6876	0644	edit dl rm
perlstyle.pod	8630	0644	edit dl rm
perlsub.pod	72967	0644	edit dl rm
perlsymbian.pod	15359	0644	edit dl rm
perlsyn.pod	44512	0644	edit dl rm
perlsynology.pod	7778	0644	edit dl rm
perlthrtut.pod	46459	0644	edit dl rm
perltie.pod	38607	0644	edit dl rm
perltoc.pod	694171	0644	edit dl rm
perltodo.pod	376	0644	edit dl rm
perltooc.pod	294	0644	edit dl rm
perltoot.pod	294	0644	edit dl rm
perltrap.pod	10620	0644	edit dl rm
perltru64.pod	8492	0644	edit dl rm
perltw.pod	4477	0644	edit dl rm
perlunicode.pod	82491	0644	edit dl rm
perlunicook.pod	25488	0644	edit dl rm
perlunifaq.pod	13647	0644	edit dl rm
perluniintro.pod	38340	0644	edit dl rm
perluniprops.pod	285306	0644	edit dl rm
perlunitut.pod	7951	0644	edit dl rm
perlutil.pod	7640	0644	edit dl rm
perlvar.pod	78364	0644	edit dl rm
perlvms.pod	50823	0644	edit dl rm
perlvos.pod	3843	0644	edit dl rm
perlwin32.pod	39298	0644	edit dl rm
perlxs.pod	78920	0644	edit dl rm
perlxstut.pod	50095	0644	edit dl rm
perlxstypemap.pod	24001	0644	edit dl rm

Edit: /usr/share/perl5/pod/perlunicook.pod (25488B)

=encoding utf8

=head1 NAME

perlunicook - cookbookish examples of handling Unicode in Perl

=head1 DESCRIPTION

This manpage contains short recipes demonstrating how to handle common Unicode
operations in Perl, plus one complete program at the end. Any undeclared
variables in individual recipes are assumed to have a previous appropriate
value in them.

=head1 EXAMPLES

=head2 ℞ 0: Standard preamble

Unless otherwise notes, all examples below require this standard preamble
to work correctly, with the C<#!> adjusted to work on your system:

#!/usr/bin/env perl

use utf8;      # so literals and identifiers can be in UTF-8
 use v5.12;     # or later to get "unicode_strings" feature
 use strict;    # quote strings, declare variables
 use warnings;  # on by default
 use warnings  qw(FATAL utf8);    # fatalize encoding glitches
 use open      qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8
 use charnames qw(:full :short);  # unneeded in v5.16

This I<does> make even Unix programmers C<binmode> your binary streams,
or open them with C<:raw>, but that's the only way to get at them
portably anyway.

B<WARNING>: C<use autodie> (pre 2.26) and C<use open> do not get along with each
other.

=head2 ℞ 1: Generic Unicode-savvy filter

Always decompose on the way in, then recompose on the way out.

use Unicode::Normalize;

while (<>) {
     $_ = NFD($_);   # decompose + reorder canonically
     ...
 } continue {
     print NFC($_);  # recompose (where possible) + reorder canonically
 }

=head2 ℞ 2: Fine-tuning Unicode warnings

As of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings.

use v5.14;                  # subwarnings unavailable any earlier
 no warnings "nonchar";      # the 66 forbidden non-characters
 no warnings "surrogate";    # UTF-16/CESU-8 nonsense
 no warnings "non_unicode";  # for codepoints over 0x10_FFFF

=head2 ℞ 3: Declare source in utf8 for identifiers and literals

Without the all-critical C<use utf8> declaration, putting UTF‑8 in your
literals and identifiers won’t work right.  If you used the standard
preamble just given above, this already happened.  If you did, you can
do things like this:

use utf8;

my $measure   = "Ångström";
 my @μsoft     = qw( cp852 cp1251 cp1252 );
 my @ὑπέρμεγας = qw( ὑπέρ  μεγας );
 my @鯉        = qw( koi8-f koi8-u koi8-r );
 my $motto     = "👪 💗 🐪"; # FAMILY, GROWING HEART, DROMEDARY CAMEL

If you forget C<use utf8>, high bytes will be misunderstood as
separate characters, and nothing will work right.

=head2 ℞ 4: Characters and their numbers

The C<ord> and C<chr> functions work transparently on all codepoints,
not just on ASCII alone — nor in fact, not even just on Unicode alone.

# ASCII characters
 ord("A")
 chr(65)

# characters from the Basic Multilingual Plane
 ord("Σ")
 chr(0x3A3)

# beyond the BMP
 ord("𝑛")               # MATHEMATICAL ITALIC SMALL N
 chr(0x1D45B)

# beyond Unicode! (up to MAXINT)
 ord("\x{20_0000}")
 chr(0x20_0000)

=head2 ℞ 5: Unicode literals by character number

In an interpolated literal, whether a double-quoted string or a
regex, you may specify a character by its number using the
C<\x{I<HHHHHH>}> escape.

String: "\x{3a3}"
 Regex:  /\x{3a3}/

String: "\x{1d45b}"
 Regex:  /\x{1d45b}/

# even non-BMP ranges in regex work fine
 /[\x{1D434}-\x{1D467}]/

=head2 ℞ 6: Get character name by number

use charnames ();
 my $name = charnames::viacode(0x03A3);

=head2 ℞ 7: Get character number by name

use charnames ();
 my $number = charnames::vianame("GREEK CAPITAL LETTER SIGMA");

=head2 ℞ 8: Unicode named characters

Use the C<< \N{I<charname>} >> notation to get the character
by that name for use in interpolated literals (double-quoted
strings and regexes).  In v5.16, there is an implicit

use charnames qw(:full :short);

But prior to v5.16, you must be explicit about which set of charnames you
want.  The C<:full> names are the official Unicode character name, alias, or
sequence, which all share a namespace.

use charnames qw(:full :short latin greek);

"\N{MATHEMATICAL ITALIC SMALL N}"      # :full
 "\N{GREEK CAPITAL LETTER SIGMA}"       # :full

Anything else is a Perl-specific convenience abbreviation.  Specify one or
more scripts by names if you want short names that are script-specific.

"\N{Greek:Sigma}"                      # :short
 "\N{ae}"                               #  latin
 "\N{epsilon}"                          #  greek

The v5.16 release also supports a C<:loose> import for loose matching of
character names, which works just like loose matching of property names:
that is, it disregards case, whitespace, and underscores:

"\N{euro sign}"                        # :loose (from v5.16)

=head2 ℞ 9: Unicode named sequences

These look just like character names but return multiple codepoints.
Notice the C<%vx> vector-print functionality in C<printf>.

use charnames qw(:full);
 my $seq = "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}";
 printf "U+%v04X\n", $seq;
 U+0100.0300

=head2 ℞ 10: Custom named characters

Use C<:alias> to give your own lexically scoped nicknames to existing
characters, or even to give unnamed private-use characters useful names.

use charnames ":full", ":alias" => {
     ecute => "LATIN SMALL LETTER E WITH ACUTE",
     "APPLE LOGO" => 0xF8FF, # private use character
 };

"\N{ecute}"
 "\N{APPLE LOGO}"

=head2 ℞ 11: Names of CJK codepoints

Sinograms like “東京” come back with character names of
C<CJK UNIFIED IDEOGRAPH-6771> and C<CJK UNIFIED IDEOGRAPH-4EAC>,
because their “names” vary.  The CPAN C<Unicode::Unihan> module
has a large database for decoding these (and a whole lot more), provided you
know how to understand its output.

# cpan -i Unicode::Unihan
 use Unicode::Unihan;
 my $str = "東京";
 my $unhan = Unicode::Unihan->new;
 for my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) {
     printf "CJK $str in %-12s is ", $lang;
     say $unhan->$lang($str);
 }

prints:

CJK 東京 in Mandarin     is DONG1JING1
 CJK 東京 in Cantonese    is dung1ging1
 CJK 東京 in Korean       is TONGKYENG
 CJK 東京 in JapaneseOn   is TOUKYOU KEI KIN
 CJK 東京 in JapaneseKun  is HIGASHI AZUMAMIYAKO

If you have a specific romanization scheme in mind,
use the specific module:

# cpan -i Lingua::JA::Romanize::Japanese
 use Lingua::JA::Romanize::Japanese;
 my $k2r = Lingua::JA::Romanize::Japanese->new;
 my $str = "東京";
 say "Japanese for $str is ", $k2r->chars($str);

prints

Japanese for 東京 is toukyou

=head2 ℞ 12: Explicit encode/decode

On rare occasion, such as a database read, you may be
given encoded text you need to decode.

use Encode qw(encode decode);

my $chars = decode("shiftjis", $bytes, 1);
 # OR
  my $bytes = encode("MIME-Header-ISO_2022_JP", $chars, 1);

For streams all in the same encoding, don't use encode/decode; instead
set the file encoding when you open the file or immediately after with
C<binmode> as described later below.

=head2 ℞ 13: Decode program arguments as utf8

$ perl -CA ...
 or
     $ export PERL_UNICODE=A
 or
    use Encode qw(decode);
    @ARGV = map { decode('UTF-8', $_, 1) } @ARGV;

=head2 ℞ 14: Decode program arguments as locale encoding

# cpan -i Encode::Locale
    use Encode qw(locale);
    use Encode::Locale;

# use "locale" as an arg to encode/decode
    @ARGV = map { decode(locale => $_, 1) } @ARGV;

=head2 ℞ 15: Declare STD{IN,OUT,ERR} to be utf8

Use a command-line option, an environment variable, or else
call C<binmode> explicitly:

$ perl -CS ...
 or
     $ export PERL_UNICODE=S
 or
     use open qw(:std :encoding(UTF-8));
 or
     binmode(STDIN,  ":encoding(UTF-8)");
     binmode(STDOUT, ":utf8");
     binmode(STDERR, ":utf8");

=head2 ℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding

# cpan -i Encode::Locale
    use Encode;
    use Encode::Locale;

# or as a stream for binmode or open
    binmode STDIN,  ":encoding(console_in)"  if -t STDIN;
    binmode STDOUT, ":encoding(console_out)" if -t STDOUT;
    binmode STDERR, ":encoding(console_out)" if -t STDERR;

=head2 ℞ 17: Make file I/O default to utf8

Files opened without an encoding argument will be in UTF-8:

$ perl -CD ...
 or
     $ export PERL_UNICODE=D
 or
     use open qw(:encoding(UTF-8));

=head2 ℞ 18: Make all I/O and args default to utf8

$ perl -CSDA ...
 or
     $ export PERL_UNICODE=SDA
 or
     use open qw(:std :encoding(UTF-8));
     use Encode qw(decode);
     @ARGV = map { decode('UTF-8', $_, 1) } @ARGV;

=head2 ℞ 19: Open file with specific encoding

Specify stream encoding.  This is the normal way
to deal with encoded text, not by calling low-level
functions.

# input file
     open(my $in_file, "< :encoding(UTF-16)", "wintext");
 OR
     open(my $in_file, "<", "wintext");
     binmode($in_file, ":encoding(UTF-16)");
 THEN
     my $line = <$in_file>;

# output file
     open($out_file, "> :encoding(cp1252)", "wintext");
 OR
     open(my $out_file, ">", "wintext");
     binmode($out_file, ":encoding(cp1252)");
 THEN
     print $out_file "some text\n";

More layers than just the encoding can be specified here. For example,
the incantation C<":raw :encoding(UTF-16LE) :crlf"> includes implicit
CRLF handling.

=head2 ℞ 20: Unicode casing

Unicode casing is very different from ASCII casing.

uc("henry ⅷ")  # "HENRY Ⅷ"
 uc("tschüß")   # "TSCHÜSS"  notice ß => SS

# both are true:
 "tschüß"  =~ /TSCHÜSS/i   # notice ß => SS
 "Σίσυφος" =~ /ΣΊΣΥΦΟΣ/i   # notice Σ,σ,ς sameness

=head2 ℞ 21: Unicode case-insensitive comparisons

Also available in the CPAN L<Unicode::CaseFold> module,
the new C<fc> “foldcase” function from v5.16 grants
access to the same Unicode casefolding as the C</i>
pattern modifier has always used:

use feature "fc"; # fc() function is from v5.16

# sort case-insensitively
 my @sorted = sort { fc($a) cmp fc($b) } @list;

# both are true:
 fc("tschüß")  eq fc("TSCHÜSS")
 fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ")

=head2 ℞ 22: Match Unicode linebreak sequence in regex

A Unicode linebreak matches the two-character CRLF
grapheme or any of seven vertical whitespace characters.
Good for dealing with textfiles coming from different
operating systems.

s/\R/\n/g;  # normalize all linebreaks to \n

=head2 ℞ 23: Get character category

Find the general category of a numeric codepoint.

use Unicode::UCD qw(charinfo);
 my $cat = charinfo(0x3A3)->{category};  # "Lu"

=head2 ℞ 24: Disabling Unicode-awareness in builtin charclasses

Disable C<\w>, C<\b>, C<\s>, C<\d>, and the POSIX
classes from working correctly on Unicode either in this
scope, or in just one regex.

use v5.14;
 use re "/a";

# OR

my($num) = $str =~ /(\d+)/a;

Or use specific un-Unicode properties, like C<\p{ahex}>
and C<\p{POSIX_Digit>}.  Properties still work normally
no matter what charset modifiers (C</d /u /l /a /aa>)
should be effect.

=head2 ℞ 25: Match Unicode properties in regex with \p, \P

These all match a single codepoint with the given
property.  Use C<\P> in place of C<\p> to match
one codepoint lacking that property.

\pL, \pN, \pS, \pP, \pM, \pZ, \pC
 \p{Sk}, \p{Ps}, \p{Lt}
 \p{alpha}, \p{upper}, \p{lower}
 \p{Latin}, \p{Greek}
 \p{script_extensions=Latin}, \p{scx=Greek}
 \p{East_Asian_Width=Wide}, \p{EA=W}
 \p{Line_Break=Hyphen}, \p{LB=HY}
 \p{Numeric_Value=4}, \p{NV=4}

=head2 ℞ 26: Custom character properties

Define at compile-time your own custom character
properties for use in regexes.

# using private-use characters
 sub In_Tengwar { "E000\tE07F\n" }

if (/\p{In_Tengwar}/) { ... }

# blending existing properties
 sub Is_GraecoRoman_Title {<<'END_OF_SET'}
 +utf8::IsLatin
 +utf8::IsGreek
 &utf8::IsTitle
 END_OF_SET

if (/\p{Is_GraecoRoman_Title}/ { ... }

=head2 ℞ 27: Unicode normalization

Typically render into NFD on input and NFC on output. Using NFKC or NFKD
functions improves recall on searches, assuming you've already done to the
same text to be searched. Note that this is about much more than just pre-
combined compatibility glyphs; it also reorders marks according to their
canonical combining classes and weeds out singletons.

use Unicode::Normalize;
 my $nfd  = NFD($orig);
 my $nfc  = NFC($orig);
 my $nfkd = NFKD($orig);
 my $nfkc = NFKC($orig);

=head2 ℞ 28: Convert non-ASCII Unicode numerics

Unless you’ve used C</a> or C</aa>, C<\d> matches more than
ASCII digits only, but Perl’s implicit string-to-number
conversion does not current recognize these.  Here’s how to
convert such strings manually.

use v5.14;  # needed for num() function
 use Unicode::UCD qw(num);
 my $str = "got Ⅻ and ४५६७ and ⅞ and here";
 my @nums = ();
 while ($str =~ /(\d+|\N)/g) {  # not just ASCII!
    push @nums, num($1);
 }
 say "@nums";   #     12      4567      0.875

use charnames qw(:full);
 my $nv = num("\N{RUMI DIGIT ONE}\N{RUMI DIGIT TWO}");

=head2 ℞ 29: Match Unicode grapheme cluster in regex

Programmer-visible “characters” are codepoints matched by C</./s>,
but user-visible “characters” are graphemes matched by C</\X/>.

# Find vowel *plus* any combining diacritics,underlining,etc.
 my $nfd = NFD($orig);
 $nfd =~ / (?=[aeiou]) \X /xi

=head2 ℞ 30: Extract by grapheme instead of by codepoint (regex)

# match and grab five first graphemes
 my($first_five) = $str =~ /^ ( \X{5} ) /x;

=head2 ℞ 31: Extract by grapheme instead of by codepoint (substr)

# cpan -i Unicode::GCString
 use Unicode::GCString;
 my $gcs = Unicode::GCString->new($str);
 my $first_five = $gcs->substr(0, 5);

=head2 ℞ 32: Reverse string by grapheme

Reversing by codepoint messes up diacritics, mistakenly converting
C<crème brûlée> into C<éel̂urb em̀erc> instead of into C<eélûrb emèrc>;
so reverse by grapheme instead.  Both these approaches work
right no matter what normalization the string is in:

$str = join("", reverse $str =~ /\X/g);

# OR: cpan -i Unicode::GCString
 use Unicode::GCString;
 $str = reverse Unicode::GCString->new($str);

=head2 ℞ 33: String length in graphemes

The string C<brûlée> has six graphemes but up to eight codepoints.
This counts by grapheme, not by codepoint:

my $str = "brûlée";
 my $count = 0;
 while ($str =~ /\X/g) { $count++ }

# OR: cpan -i Unicode::GCString
 use Unicode::GCString;
 my $gcs = Unicode::GCString->new($str);
 my $count = $gcs->length;

=head2 ℞ 34: Unicode column-width for printing

Perl’s C<printf>, C<sprintf>, and C<format> think all
codepoints take up 1 print column, but many take 0 or 2.
Here to show that normalization makes no difference,
we print out both forms:

use Unicode::GCString;
 use Unicode::Normalize;

my @words = qw/crème brûlée/;
 @words = map { NFC($_), NFD($_) } @words;

for my $str (@words) {
     my $gcs = Unicode::GCString->new($str);
     my $cols = $gcs->columns;
     my $pad = " " x (10 - $cols);
     say str, $pad, " |";
 }

generates this to show that it pads correctly no matter
the normalization:

crème      |
 crème      |
 brûlée     |
 brûlée     |

=head2 ℞ 35: Unicode collation

Text sorted by numeric codepoint follows no reasonable alphabetic order;
use the UCA for sorting text.

use Unicode::Collate;
 my $col = Unicode::Collate->new();
 my @list = $col->sort(@old_list);

See the I<ucsort> program from the L<Unicode::Tussle> CPAN module
for a convenient command-line interface to this module.

=head2 ℞ 36: Case- I<and> accent-insensitive Unicode sort

Specify a collation strength of level 1 to ignore case and
diacritics, only looking at the basic character.

use Unicode::Collate;
 my $col = Unicode::Collate->new(level => 1);
 my @list = $col->sort(@old_list);

=head2 ℞ 37: Unicode locale collation

Some locales have special sorting rules.

# either use v5.12, OR: cpan -i Unicode::Collate::Locale
 use Unicode::Collate::Locale;
 my $col = Unicode::Collate::Locale->new(locale => "de__phonebook");
 my @list = $col->sort(@old_list);

The I<ucsort> program mentioned above accepts a C<--locale> parameter.

=head2 ℞ 38: Making C<cmp> work on text instead of codepoints

Instead of this:

@srecs = sort {
     $b->{AGE}   <=>  $a->{AGE}
                 ||
     $a->{NAME}  cmp  $b->{NAME}
 } @recs;

Use this:

my $coll = Unicode::Collate->new();
 for my $rec (@recs) {
     $rec->{NAME_key} = $coll->getSortKey( $rec->{NAME} );
 }
 @srecs = sort {
     $b->{AGE}       <=>  $a->{AGE}
                     ||
     $a->{NAME_key}  cmp  $b->{NAME_key}
 } @recs;

=head2 ℞ 39: Case- I<and> accent-insensitive comparisons

Use a collator object to compare Unicode text by character
instead of by codepoint.

use Unicode::Collate;
 my $es = Unicode::Collate->new(
     level => 1,
     normalization => undef
 );

# now both are true:
 $es->eq("García",  "GARCIA" );
 $es->eq("Márquez", "MARQUEZ");

=head2 ℞ 40: Case- I<and> accent-insensitive locale comparisons

Same, but in a specific locale.

my $de = Unicode::Collate::Locale->new(
            locale => "de__phonebook",
          );

# now this is true:
 $de->eq("tschüß", "TSCHUESS");  # notice ü => UE, ß => SS

=head2 ℞ 41: Unicode linebreaking

Break up text into lines according to Unicode rules.

# cpan -i Unicode::LineBreak
 use Unicode::LineBreak;
 use charnames qw(:full);

my $para = "This is a super\N{HYPHEN}long string. " x 20;
 my $fmt = Unicode::LineBreak->new;
 print $fmt->break($para), "\n";

=head2 ℞ 42: Unicode text in DBM hashes, the tedious way

Using a regular Perl string as a key or value for a DBM
hash will trigger a wide character exception if any codepoints
won’t fit into a byte.  Here’s how to manually manage the translation:

use DB_File;
    use Encode qw(encode decode);
    tie %dbhash, "DB_File", "pathname";

# STORE

# assume $uni_key and $uni_value are abstract Unicode strings
    my $enc_key   = encode("UTF-8", $uni_key, 1);
    my $enc_value = encode("UTF-8", $uni_value, 1);
    $dbhash{$enc_key} = $enc_value;

# FETCH

# assume $uni_key holds a normal Perl string (abstract Unicode)
    my $enc_key   = encode("UTF-8", $uni_key, 1);
    my $enc_value = $dbhash{$enc_key};
    my $uni_value = decode("UTF-8", $enc_value, 1);

=head2 ℞ 43: Unicode text in DBM hashes, the easy way

Here’s how to implicitly manage the translation; all encoding
and decoding is done automatically, just as with streams that
have a particular encoding attached to them:

use DB_File;
    use DBM_Filter;

my $dbobj = tie %dbhash, "DB_File", "pathname";
    $dbobj->Filter_Value("utf8");  # this is the magic bit

# STORE

# assume $uni_key and $uni_value are abstract Unicode strings
    $dbhash{$uni_key} = $uni_value;

# FETCH

# $uni_key holds a normal Perl string (abstract Unicode)
    my $uni_value = $dbhash{$uni_key};

=head2 ℞ 44: PROGRAM: Demo of Unicode collation and printing

Here’s a full program showing how to make use of locale-sensitive
sorting, Unicode casing, and managing print widths when some of the
characters take up zero or two columns, not just one column each time.
When run, the following program produces this nicely aligned output:

Crème Brûlée....... €2.00
    Éclair............. €1.60
    Fideuà............. €4.20
    Hamburger.......... €6.00
    Jamón Serrano...... €4.45
    Linguiça........... €7.00
    Pâté............... €4.15
    Pears.............. €2.00
    Pêches............. €2.25
    Smørbrød........... €5.75
    Spätzle............ €5.50
    Xoriço............. €3.00
    Γύρος.............. €6.50
    막걸리............. €4.00
    おもち............. €2.65
    お好み焼き......... €8.00
    シュークリーム..... €1.85
    寿司............... €9.99
    包子............... €7.50

Here's that program; tested on v5.14.

#!/usr/bin/env perl
 # umenu - demo sorting and printing of Unicode food
 #
 # (obligatory and increasingly long preamble)
 #
 use utf8;
 use v5.14;                       # for locale sorting
 use strict;
 use warnings;
 use warnings  qw(FATAL utf8);    # fatalize encoding faults
 use open      qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8
 use charnames qw(:full :short);  # unneeded in v5.16

# std modules
 use Unicode::Normalize;          # std perl distro as of v5.8
 use List::Util qw(max);          # std perl distro as of v5.10
 use Unicode::Collate::Locale;    # std perl distro as of v5.14

# cpan modules
 use Unicode::GCString;           # from CPAN

# forward defs
 sub pad($$$);
 sub colwidth(_);
 sub entitle(_);

my %price = (
     "γύρος"             => 6.50, # gyros
     "pears"             => 2.00, # like um, pears
     "linguiça"          => 7.00, # spicy sausage, Portuguese
     "xoriço"            => 3.00, # chorizo sausage, Catalan
     "hamburger"         => 6.00, # burgermeister meisterburger
     "éclair"            => 1.60, # dessert, French
     "smørbrød"          => 5.75, # sandwiches, Norwegian
     "spätzle"           => 5.50, # Bayerisch noodles, little sparrows
     "包子"              => 7.50, # bao1 zi5, steamed pork buns, Mandarin
     "jamón serrano"     => 4.45, # country ham, Spanish
     "pêches"            => 2.25, # peaches, French
     "シュークリーム"    => 1.85, # cream-filled pastry like eclair
     "막걸리"            => 4.00, # makgeolli, Korean rice wine
     "寿司"              => 9.99, # sushi, Japanese
     "おもち"            => 2.65, # omochi, rice cakes, Japanese
     "crème brûlée"      => 2.00, # crema catalana
     "fideuà"            => 4.20, # more noodles, Valencian
                                  # (Catalan=fideuada)
     "pâté"              => 4.15, # gooseliver paste, French
     "お好み焼き"        => 8.00, # okonomiyaki, Japanese
 );

my $width = 5 + max map { colwidth } keys %price;

# So the Asian stuff comes out in an order that someone
 # who reads those scripts won't freak out over; the
 # CJK stuff will be in JIS X 0208 order that way.
 my $coll  = Unicode::Collate::Locale->new(locale => "ja");

for my $item ($coll->sort(keys %price)) {
     print pad(entitle($item), $width, ".");
     printf " €%.2f\n", $price{$item};
 }

sub pad($$$) {
     my($str, $width, $padchar) = @_;
     return $str . ($padchar x ($width - colwidth($str)));
 }

sub colwidth(_) {
     my($str) = @_;
     return Unicode::GCString->new($str)->columns;
 }

sub entitle(_) {
     my($str) = @_;
     $str =~ s{ (?=\pL)(\S)     (\S*) }
              { ucfirst($1) . lc($2)  }xge;
     return $str;
 }

=head1 SEE ALSO

See these manpages, some of which are CPAN modules:
L<perlunicode>, L<perluniprops>,
L<perlre>, L<perlrecharclass>,
L<perluniintro>, L<perlunitut>, L<perlunifaq>,
L<PerlIO>, L<DB_File>, L<DBM_Filter>, L<DBM_Filter::utf8>,
L<Encode>, L<Encode::Locale>,
L<Unicode::UCD>,
L<Unicode::Normalize>,
L<Unicode::GCString>, L<Unicode::LineBreak>,
L<Unicode::Collate>, L<Unicode::Collate::Locale>,
L<Unicode::Unihan>,
L<Unicode::CaseFold>,
L<Unicode::Tussle>,
L<Lingua::JA::Romanize::Japanese>,
L<Lingua::ZH::Romanize::Pinyin>,
L<Lingua::KO::Romanize::Hangul>.

The L<Unicode::Tussle> CPAN module includes many programs
to help with working with Unicode, including
these programs to fully or partly replace standard utilities:
I<tcgrep> instead of I<egrep>,
I<uniquote> instead of I<cat -v> or I<hexdump>,
I<uniwc> instead of I<wc>,
I<unilook> instead of I<look>,
I<unifmt> instead of I<fmt>,
and
I<ucsort> instead of I<sort>.
For exploring Unicode character names and character properties,
see its I<uniprops>, I<unichars>, and I<uninames> programs.
It also supplies these programs, all of which are general filters that do Unicode-y things:
I<unititle> and I<unicaps>;
I<uniwide> and I<uninarrow>;
I<unisupers> and I<unisubs>;
I<nfd>, I<nfc>, I<nfkd>, and I<nfkc>;
and I<uc>, I<lc>, and I<tc>.

Finally, see the published Unicode Standard (page numbers are from version
6.0.0), including these specific annexes and technical reports:

=over

=item §3.13 Default Case Algorithms, page 113;
§4.2  Case, pages 120–122;
Case Mappings, page 166–172, especially Caseless Matching starting on page 170.

=item UAX #44: Unicode Character Database

=item UTS #18: Unicode Regular Expressions

=item UAX #15: Unicode Normalization Forms

=item UTS #10: Unicode Collation Algorithm

=item UAX #29: Unicode Text Segmentation

=item UAX #14: Unicode Line Breaking Algorithm

=item UAX #11: East Asian Width

=back

=head1 AUTHOR

Tom Christiansen E<lt>tchrist@perl.comE<gt> wrote this, with occasional
kibbitzing from Larry Wall and Jeffrey Friedl in the background.

=head1 COPYRIGHT AND LICENCE

This program is free software; you may redistribute it and/or modify it
under the same terms as Perl itself.

Most of these examples taken from the current edition of the “Camel Book”;
that is, from the 4ᵗʰ Edition of I<Programming Perl>, Copyright © 2012 Tom
Christiansen <et al.>, 2012-02-13 by O’Reilly Media.  The code itself is
freely redistributable, and you are encouraged to transplant, fold,
spindle, and mutilate any of the examples in this manpage however you please
for inclusion into your own programs without any encumbrance whatsoever.
Acknowledgement via code comment is polite but not required.

=head1 REVISION HISTORY

v1.0.0 – first public release, 2012-02-27

=encoding utf8 =head1 NAME perlunicook - cookbookish examples of handling Unicode in Perl =head1 DESCRIPTION This manpage contains short recipes demonstrating how to handle common Unicode operations in Perl, plus one complete program at the end. Any undeclared variables in individual recipes are assumed to have a previous appropriate value in them. =head1 EXAMPLES =head2 ℞ 0: Standard preamble Unless otherwise notes, all examples below require this standard preamble to work correctly, with the C<#!> adjusted to work on your system: #!/usr/bin/env perl use utf8; # so literals and identifiers can be in UTF-8 use v5.12; # or later to get "unicode_strings" feature use strict; # quote strings, declare variables use warnings; # on by default use warnings qw(FATAL utf8); # fatalize encoding glitches use open qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8 use charnames qw(:full :short); # unneeded in v5.16 This I make even Unix programmers C your binary streams, or open them with C<:raw>, but that's the only way to get at them portably anyway. B: C (pre 2.26) and C do not get along with each other. =head2 ℞ 1: Generic Unicode-savvy filter Always decompose on the way in, then recompose on the way out. use Unicode::Normalize; while (<>) { $_ = NFD($_); # decompose + reorder canonically ... } continue { print NFC($_); # recompose (where possible) + reorder canonically } =head2 ℞ 2: Fine-tuning Unicode warnings As of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings. use v5.14; # subwarnings unavailable any earlier no warnings "nonchar"; # the 66 forbidden non-characters no warnings "surrogate"; # UTF-16/CESU-8 nonsense no warnings "non_unicode"; # for codepoints over 0x10_FFFF =head2 ℞ 3: Declare source in utf8 for identifiers and literals Without the all-critical C declaration, putting UTF‑8 in your literals and identifiers won’t work right. If you used the standard preamble just given above, this already happened. If you did, you can do things like this: use utf8; my $measure = "Ångström"; my @μsoft = qw( cp852 cp1251 cp1252 ); my @ὑπέρμεγας = qw( ὑπέρ μεγας ); my @鯉 = qw( koi8-f koi8-u koi8-r ); my $motto = "👪 💗 🐪"; # FAMILY, GROWING HEART, DROMEDARY CAMEL If you forget C, high bytes will be misunderstood as separate characters, and nothing will work right. =head2 ℞ 4: Characters and their numbers The C and C functions work transparently on all codepoints, not just on ASCII alone — nor in fact, not even just on Unicode alone. # ASCII characters ord("A") chr(65) # characters from the Basic Multilingual Plane ord("Σ") chr(0x3A3) # beyond the BMP ord("𝑛") # MATHEMATICAL ITALIC SMALL N chr(0x1D45B) # beyond Unicode! (up to MAXINT) ord("\x{20_0000}") chr(0x20_0000) =head2 ℞ 5: Unicode literals by character number In an interpolated literal, whether a double-quoted string or a regex, you may specify a character by its number using the C<\x{I}> escape. String: "\x{3a3}" Regex: /\x{3a3}/ String: "\x{1d45b}" Regex: /\x{1d45b}/ # even non-BMP ranges in regex work fine /[\x{1D434}-\x{1D467}]/ =head2 ℞ 6: Get character name by number use charnames (); my $name = charnames::viacode(0x03A3); =head2 ℞ 7: Get character number by name use charnames (); my $number = charnames::vianame("GREEK CAPITAL LETTER SIGMA"); =head2 ℞ 8: Unicode named characters Use the C<< \N{I} >> notation to get the character by that name for use in interpolated literals (double-quoted strings and regexes). In v5.16, there is an implicit use charnames qw(:full :short); But prior to v5.16, you must be explicit about which set of charnames you want. The C<:full> names are the official Unicode character name, alias, or sequence, which all share a namespace. use charnames qw(:full :short latin greek); "\N{MATHEMATICAL ITALIC SMALL N}" # :full "\N{GREEK CAPITAL LETTER SIGMA}" # :full Anything else is a Perl-specific convenience abbreviation. Specify one or more scripts by names if you want short names that are script-specific. "\N{Greek:Sigma}" # :short "\N{ae}" # latin "\N{epsilon}" # greek The v5.16 release also supports a C<:loose> import for loose matching of character names, which works just like loose matching of property names: that is, it disregards case, whitespace, and underscores: "\N{euro sign}" # :loose (from v5.16) =head2 ℞ 9: Unicode named sequences These look just like character names but return multiple codepoints. Notice the C<%vx> vector-print functionality in C. use charnames qw(:full); my $seq = "\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}"; printf "U+%v04X\n", $seq; U+0100.0300 =head2 ℞ 10: Custom named characters Use C<:alias> to give your own lexically scoped nicknames to existing characters, or even to give unnamed private-use characters useful names. use charnames ":full", ":alias" => { ecute => "LATIN SMALL LETTER E WITH ACUTE", "APPLE LOGO" => 0xF8FF, # private use character }; "\N{ecute}" "\N{APPLE LOGO}" =head2 ℞ 11: Names of CJK codepoints Sinograms like “東京” come back with character names of C and C, because their “names” vary. The CPAN C module has a large database for decoding these (and a whole lot more), provided you know how to understand its output. # cpan -i Unicode::Unihan use Unicode::Unihan; my $str = "東京"; my $unhan = Unicode::Unihan->new; for my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) { printf "CJK $str in %-12s is ", $lang; say $unhan->$lang($str); } prints: CJK 東京 in Mandarin is DONG1JING1 CJK 東京 in Cantonese is dung1ging1 CJK 東京 in Korean is TONGKYENG CJK 東京 in JapaneseOn is TOUKYOU KEI KIN CJK 東京 in JapaneseKun is HIGASHI AZUMAMIYAKO If you have a specific romanization scheme in mind, use the specific module: # cpan -i Lingua::JA::Romanize::Japanese use Lingua::JA::Romanize::Japanese; my $k2r = Lingua::JA::Romanize::Japanese->new; my $str = "東京"; say "Japanese for $str is ", $k2r->chars($str); prints Japanese for 東京 is toukyou =head2 ℞ 12: Explicit encode/decode On rare occasion, such as a database read, you may be given encoded text you need to decode. use Encode qw(encode decode); my $chars = decode("shiftjis", $bytes, 1); # OR my $bytes = encode("MIME-Header-ISO_2022_JP", $chars, 1); For streams all in the same encoding, don't use encode/decode; instead set the file encoding when you open the file or immediately after with C as described later below. =head2 ℞ 13: Decode program arguments as utf8 $ perl -CA ... or $ export PERL_UNICODE=A or use Encode qw(decode); @ARGV = map { decode('UTF-8', $_, 1) } @ARGV; =head2 ℞ 14: Decode program arguments as locale encoding # cpan -i Encode::Locale use Encode qw(locale); use Encode::Locale; # use "locale" as an arg to encode/decode @ARGV = map { decode(locale => $_, 1) } @ARGV; =head2 ℞ 15: Declare STD{IN,OUT,ERR} to be utf8 Use a command-line option, an environment variable, or else call C explicitly: $ perl -CS ... or $ export PERL_UNICODE=S or use open qw(:std :encoding(UTF-8)); or binmode(STDIN, ":encoding(UTF-8)"); binmode(STDOUT, ":utf8"); binmode(STDERR, ":utf8"); =head2 ℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding # cpan -i Encode::Locale use Encode; use Encode::Locale; # or as a stream for binmode or open binmode STDIN, ":encoding(console_in)" if -t STDIN; binmode STDOUT, ":encoding(console_out)" if -t STDOUT; binmode STDERR, ":encoding(console_out)" if -t STDERR; =head2 ℞ 17: Make file I/O default to utf8 Files opened without an encoding argument will be in UTF-8: $ perl -CD ... or $ export PERL_UNICODE=D or use open qw(:encoding(UTF-8)); =head2 ℞ 18: Make all I/O and args default to utf8 $ perl -CSDA ... or $ export PERL_UNICODE=SDA or use open qw(:std :encoding(UTF-8)); use Encode qw(decode); @ARGV = map { decode('UTF-8', $_, 1) } @ARGV; =head2 ℞ 19: Open file with specific encoding Specify stream encoding. This is the normal way to deal with encoded text, not by calling low-level functions. # input file open(my $in_file, "< :encoding(UTF-16)", "wintext"); OR open(my $in_file, "<", "wintext"); binmode($in_file, ":encoding(UTF-16)"); THEN my $line = <$in_file>; # output file open($out_file, "> :encoding(cp1252)", "wintext"); OR open(my $out_file, ">", "wintext"); binmode($out_file, ":encoding(cp1252)"); THEN print $out_file "some text\n"; More layers than just the encoding can be specified here. For example, the incantation C<":raw :encoding(UTF-16LE) :crlf"> includes implicit CRLF handling. =head2 ℞ 20: Unicode casing Unicode casing is very different from ASCII casing. uc("henry ⅷ") # "HENRY Ⅷ" uc("tschüß") # "TSCHÜSS" notice ß => SS # both are true: "tschüß" =~ /TSCHÜSS/i # notice ß => SS "Σίσυφος" =~ /ΣΊΣΥΦΟΣ/i # notice Σ,σ,ς sameness =head2 ℞ 21: Unicode case-insensitive comparisons Also available in the CPAN L module, the new C “foldcase” function from v5.16 grants access to the same Unicode casefolding as the C pattern modifier has always used: use feature "fc"; # fc() function is from v5.16 # sort case-insensitively my @sorted = sort { fc($a) cmp fc($b) } @list; # both are true: fc("tschüß") eq fc("TSCHÜSS") fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ") =head2 ℞ 22: Match Unicode linebreak sequence in regex A Unicode linebreak matches the two-character CRLF grapheme or any of seven vertical whitespace characters. Good for dealing with textfiles coming from different operating systems. \R s/\R/\n/g; # normalize all linebreaks to \n =head2 ℞ 23: Get character category Find the general category of a numeric codepoint. use Unicode::UCD qw(charinfo); my $cat = charinfo(0x3A3)->{category}; # "Lu" =head2 ℞ 24: Disabling Unicode-awareness in builtin charclasses Disable C<\w>, C<\b>, C<\s>, C<\d>, and the POSIX classes from working correctly on Unicode either in this scope, or in just one regex. use v5.14; use re "/a"; # OR my($num) = $str =~ /(\d+)/a; Or use specific un-Unicode properties, like C<\p{ahex}> and C<\p{POSIX_Digit>}. Properties still work normally no matter what charset modifiers (C) should be effect. =head2 ℞ 25: Match Unicode properties in regex with \p, \P These all match a single codepoint with the given property. Use C<\P> in place of C<\p> to match one codepoint lacking that property. \pL, \pN, \pS, \pP, \pM, \pZ, \pC \p{Sk}, \p{Ps}, \p{Lt} \p{alpha}, \p{upper}, \p{lower} \p{Latin}, \p{Greek} \p{script_extensions=Latin}, \p{scx=Greek} \p{East_Asian_Width=Wide}, \p{EA=W} \p{Line_Break=Hyphen}, \p{LB=HY} \p{Numeric_Value=4}, \p{NV=4} =head2 ℞ 26: Custom character properties Define at compile-time your own custom character properties for use in regexes. # using private-use characters sub In_Tengwar { "E000\tE07F\n" } if (/\p{In_Tengwar}/) { ... } # blending existing properties sub Is_GraecoRoman_Title {<<'END_OF_SET'} +utf8::IsLatin +utf8::IsGreek &utf8::IsTitle END_OF_SET if (/\p{Is_GraecoRoman_Title}/ { ... } =head2 ℞ 27: Unicode normalization Typically render into NFD on input and NFC on output. Using NFKC or NFKD functions improves recall on searches, assuming you've already done to the same text to be searched. Note that this is about much more than just pre- combined compatibility glyphs; it also reorders marks according to their canonical combining classes and weeds out singletons. use Unicode::Normalize; my $nfd = NFD($orig); my $nfc = NFC($orig); my $nfkd = NFKD($orig); my $nfkc = NFKC($orig); =head2 ℞ 28: Convert non-ASCII Unicode numerics Unless you’ve used C or C, C<\d> matches more than ASCII digits only, but Perl’s implicit string-to-number conversion does not current recognize these. Here’s how to convert such strings manually. use v5.14; # needed for num() function use Unicode::UCD qw(num); my $str = "got Ⅻ and ४५६७ and ⅞ and here"; my @nums = (); while ($str =~ /(\d+|\N)/g) { # not just ASCII! push @nums, num($1); } say "@nums"; # 12 4567 0.875 use charnames qw(:full); my $nv = num("\N{RUMI DIGIT ONE}\N{RUMI DIGIT TWO}"); =head2 ℞ 29: Match Unicode grapheme cluster in regex Programmer-visible “characters” are codepoints matched by C, but user-visible “characters” are graphemes matched by C. # Find vowel *plus* any combining diacritics,underlining,etc. my $nfd = NFD($orig); $nfd =~ / (?=[aeiou]) \X /xi =head2 ℞ 30: Extract by grapheme instead of by codepoint (regex) # match and grab five first graphemes my($first_five) = $str =~ /^ ( \X{5} ) /x; =head2 ℞ 31: Extract by grapheme instead of by codepoint (substr) # cpan -i Unicode::GCString use Unicode::GCString; my $gcs = Unicode::GCString->new($str); my $first_five = $gcs->substr(0, 5); =head2 ℞ 32: Reverse string by grapheme Reversing by codepoint messes up diacritics, mistakenly converting C into C<éel̂urb em̀erc> instead of into C; so reverse by grapheme instead. Both these approaches work right no matter what normalization the string is in: $str = join("", reverse $str =~ /\X/g); # OR: cpan -i Unicode::GCString use Unicode::GCString; $str = reverse Unicode::GCString->new($str); =head2 ℞ 33: String length in graphemes The string C has six graphemes but up to eight codepoints. This counts by grapheme, not by codepoint: my $str = "brûlée"; my $count = 0; while ($str =~ /\X/g) { $count++ } # OR: cpan -i Unicode::GCString use Unicode::GCString; my $gcs = Unicode::GCString->new($str); my $count = $gcs->length; =head2 ℞ 34: Unicode column-width for printing Perl’s C, C, and C think all codepoints take up 1 print column, but many take 0 or 2. Here to show that normalization makes no difference, we print out both forms: use Unicode::GCString; use Unicode::Normalize; my @words = qw/crème brûlée/; @words = map { NFC($_), NFD($_) } @words; for my $str (@words) { my $gcs = Unicode::GCString->new($str); my $cols = $gcs->columns; my $pad = " " x (10 - $cols); say str, $pad, " |"; } generates this to show that it pads correctly no matter the normalization: crème | crème | brûlée | brûlée | =head2 ℞ 35: Unicode collation Text sorted by numeric codepoint follows no reasonable alphabetic order; use the UCA for sorting text. use Unicode::Collate; my $col = Unicode::Collate->new(); my @list = $col->sort(@old_list); See the I program from the L CPAN module for a convenient command-line interface to this module. =head2 ℞ 36: Case- I accent-insensitive Unicode sort Specify a collation strength of level 1 to ignore case and diacritics, only looking at the basic character. use Unicode::Collate; my $col = Unicode::Collate->new(level => 1); my @list = $col->sort(@old_list); =head2 ℞ 37: Unicode locale collation Some locales have special sorting rules. # either use v5.12, OR: cpan -i Unicode::Collate::Locale use Unicode::Collate::Locale; my $col = Unicode::Collate::Locale->new(locale => "de__phonebook"); my @list = $col->sort(@old_list); The I program mentioned above accepts a C<--locale> parameter. =head2 ℞ 38: Making C work on text instead of codepoints Instead of this: @srecs = sort { $b->{AGE} <=> $a->{AGE} || $a->{NAME} cmp $b->{NAME} } @recs; Use this: my $coll = Unicode::Collate->new(); for my $rec (@recs) { $rec->{NAME_key} = $coll->getSortKey( $rec->{NAME} ); } @srecs = sort { $b->{AGE} <=> $a->{AGE} || $a->{NAME_key} cmp $b->{NAME_key} } @recs; =head2 ℞ 39: Case- I accent-insensitive comparisons Use a collator object to compare Unicode text by character instead of by codepoint. use Unicode::Collate; my $es = Unicode::Collate->new( level => 1, normalization => undef ); # now both are true: $es->eq("García", "GARCIA" ); $es->eq("Márquez", "MARQUEZ"); =head2 ℞ 40: Case- I accent-insensitive locale comparisons Same, but in a specific locale. my $de = Unicode::Collate::Locale->new( locale => "de__phonebook", ); # now this is true: $de->eq("tschüß", "TSCHUESS"); # notice ü => UE, ß => SS =head2 ℞ 41: Unicode linebreaking Break up text into lines according to Unicode rules. # cpan -i Unicode::LineBreak use Unicode::LineBreak; use charnames qw(:full); my $para = "This is a super\N{HYPHEN}long string. " x 20; my $fmt = Unicode::LineBreak->new; print $fmt->break($para), "\n"; =head2 ℞ 42: Unicode text in DBM hashes, the tedious way Using a regular Perl string as a key or value for a DBM hash will trigger a wide character exception if any codepoints won’t fit into a byte. Here’s how to manually manage the translation: use DB_File; use Encode qw(encode decode); tie %dbhash, "DB_File", "pathname"; # STORE # assume $uni_key and $uni_value are abstract Unicode strings my $enc_key = encode("UTF-8", $uni_key, 1); my $enc_value = encode("UTF-8", $uni_value, 1); $dbhash{$enc_key} = $enc_value; # FETCH # assume $uni_key holds a normal Perl string (abstract Unicode) my $enc_key = encode("UTF-8", $uni_key, 1); my $enc_value = $dbhash{$enc_key}; my $uni_value = decode("UTF-8", $enc_value, 1); =head2 ℞ 43: Unicode text in DBM hashes, the easy way Here’s how to implicitly manage the translation; all encoding and decoding is done automatically, just as with streams that have a particular encoding attached to them: use DB_File; use DBM_Filter; my $dbobj = tie %dbhash, "DB_File", "pathname"; $dbobj->Filter_Value("utf8"); # this is the magic bit # STORE # assume $uni_key and $uni_value are abstract Unicode strings $dbhash{$uni_key} = $uni_value; # FETCH # $uni_key holds a normal Perl string (abstract Unicode) my $uni_value = $dbhash{$uni_key}; =head2 ℞ 44: PROGRAM: Demo of Unicode collation and printing Here’s a full program showing how to make use of locale-sensitive sorting, Unicode casing, and managing print widths when some of the characters take up zero or two columns, not just one column each time. When run, the following program produces this nicely aligned output: Crème Brûlée....... €2.00 Éclair............. €1.60 Fideuà............. €4.20 Hamburger.......... €6.00 Jamón Serrano...... €4.45 Linguiça........... €7.00 Pâté............... €4.15 Pears.............. €2.00 Pêches............. €2.25 Smørbrød........... €5.75 Spätzle............ €5.50 Xoriço............. €3.00 Γύρος.............. €6.50 막걸리............. €4.00 おもち............. €2.65 お好み焼き......... €8.00 シュークリーム..... €1.85 寿司............... €9.99 包子............... €7.50 Here's that program; tested on v5.14. #!/usr/bin/env perl # umenu - demo sorting and printing of Unicode food # # (obligatory and increasingly long preamble) # use utf8; use v5.14; # for locale sorting use strict; use warnings; use warnings qw(FATAL utf8); # fatalize encoding faults use open qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8 use charnames qw(:full :short); # unneeded in v5.16 # std modules use Unicode::Normalize; # std perl distro as of v5.8 use List::Util qw(max); # std perl distro as of v5.10 use Unicode::Collate::Locale; # std perl distro as of v5.14 # cpan modules use Unicode::GCString; # from CPAN # forward defs sub pad($$$); sub colwidth(_); sub entitle(_); my %price = ( "γύρος" => 6.50, # gyros "pears" => 2.00, # like um, pears "linguiça" => 7.00, # spicy sausage, Portuguese "xoriço" => 3.00, # chorizo sausage, Catalan "hamburger" => 6.00, # burgermeister meisterburger "éclair" => 1.60, # dessert, French "smørbrød" => 5.75, # sandwiches, Norwegian "spätzle" => 5.50, # Bayerisch noodles, little sparrows "包子" => 7.50, # bao1 zi5, steamed pork buns, Mandarin "jamón serrano" => 4.45, # country ham, Spanish "pêches" => 2.25, # peaches, French "シュークリーム" => 1.85, # cream-filled pastry like eclair "막걸리" => 4.00, # makgeolli, Korean rice wine "寿司" => 9.99, # sushi, Japanese "おもち" => 2.65, # omochi, rice cakes, Japanese "crème brûlée" => 2.00, # crema catalana "fideuà" => 4.20, # more noodles, Valencian # (Catalan=fideuada) "pâté" => 4.15, # gooseliver paste, French "お好み焼き" => 8.00, # okonomiyaki, Japanese ); my $width = 5 + max map { colwidth } keys %price; # So the Asian stuff comes out in an order that someone # who reads those scripts won't freak out over; the # CJK stuff will be in JIS X 0208 order that way. my $coll = Unicode::Collate::Locale->new(locale => "ja"); for my $item ($coll->sort(keys %price)) { print pad(entitle($item), $width, "."); printf " €%.2f\n", $price{$item}; } sub pad($$$) { my($str, $width, $padchar) = @_; return $str . ($padchar x ($width - colwidth($str))); } sub colwidth(_) { my($str) = @_; return Unicode::GCString->new($str)->columns; } sub entitle(_) { my($str) = @_; $str =~ s{ (?=\pL)(\S) (\S*) } { ucfirst($1) . lc($2) }xge; return $str; } =head1 SEE ALSO See these manpages, some of which are CPAN modules: L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L, L. The L CPAN module includes many programs to help with working with Unicode, including these programs to fully or partly replace standard utilities: I instead of I, I instead of I or I, I instead of I, I instead of I, I instead of I, and I instead of I. For exploring Unicode character names and character properties, see its I, I, and I programs. It also supplies these programs, all of which are general filters that do Unicode-y things: I and I; I and I; I and I; I, I, I, and I; and I, I, and I. Finally, see the published Unicode Standard (page numbers are from version 6.0.0), including these specific annexes and technical reports: =over =item §3.13 Default Case Algorithms, page 113; §4.2 Case, pages 120–122; Case Mappings, page 166–172, especially Caseless Matching starting on page 170. =item UAX #44: Unicode Character Database =item UTS #18: Unicode Regular Expressions =item UAX #15: Unicode Normalization Forms =item UTS #10: Unicode Collation Algorithm =item UAX #29: Unicode Text Segmentation =item UAX #14: Unicode Line Breaking Algorithm =item UAX #11: East Asian Width =back =head1 AUTHOR Tom Christiansen Etchrist@perl.comE wrote this, with occasional kibbitzing from Larry Wall and Jeffrey Friedl in the background. =head1 COPYRIGHT AND LICENCE Copyright © 2012 Tom Christiansen. This program is free software; you may redistribute it and/or modify it under the same terms as Perl itself. Most of these examples taken from the current edition of the “Camel Book”; that is, from the 4ᵗʰ Edition of I, Copyright © 2012 Tom Christiansen , 2012-02-13 by O’Reilly Media. The code itself is freely redistributable, and you are encouraged to transplant, fold, spindle, and mutilate any of the examples in this manpage however you please for inclusion into your own programs without any encumbrance whatsoever. Acknowledgement via code comment is polite but not required. =head1 REVISION HISTORY v1.0.0 – first public release, 2012-02-27