HTTP/2 200
server: nginx
content-type: text/plain;charset=UTF-8
content-encoding: gzip
content-security-policy: default-src 'self'; connect-src 'self' *.google-analytics.com; img-src 'self' data: www.google-analytics.com www.googletagmanager.com; script-src 'self' 'unsafe-inline' www.google-analytics.com www.googletagmanager.com; style-src 'self'; report-uri /csp-reports
accept-ranges: bytes
age: 0
date: Fri, 10 Oct 2025 21:11:04 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210087-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1760130663.294310,VS0,VE941
vary: Accept-Encoding
strict-transport-security: max-age=31557600
content-length: 31396
=head1 NAME
perlunicode - Unicode support in Perl
=head1 DESCRIPTION
If you haven't already, before reading this document, you should become
familiar with both L
and L.
Unicode aims to B-fy the en-B-ings of all the world's
character sets into a single Standard. For quite a few of the various
coding standards that existed when Unicode was first created, converting
from each to Unicode essentially meant adding a constant to each code
point in the original standard, and converting back meant just
subtracting that same constant. For ASCII and ISO-8859-1, the constant
is 0. For ISO-8859-5, (Cyrillic) the constant is 864; for Hebrew
(ISO-8859-8), it's 1488; Thai (ISO-8859-11), 3424; and so forth. This
made it easy to do the conversions, and facilitated the adoption of
Unicode.
And it worked; nowadays, those legacy standards are rarely used. Most
everyone uses Unicode.
Unicode is a comprehensive standard. It specifies many things outside
the scope of Perl, such as how to display sequences of characters. For
a full discussion of all aspects of Unicode, see
L.
=head2 Important Caveats
Even though some of this section may not be understandable to you on
first reading, we think it's important enough to highlight some of the
gotchas before delving further, so here goes:
Unicode support is an extensive requirement. While Perl does not
implement the Unicode standard or the accompanying technical reports
from cover to cover, Perl does support many Unicode features.
Also, the use of Unicode may present security issues that aren't
obvious, see L below.
=over 4
=item Safest if you C
In order to preserve backward compatibility, Perl does not turn
on full internal Unicode support unless the pragma
L>|feature/The 'unicode_strings' feature>
is specified. (This is automatically
selected if you S> or higher.) Failure to do this can
trigger unexpected surprises. See L below.
This pragma doesn't affect I/O. Nor does it change the internal
representation of strings, only their interpretation. There are still
several places where Unicode isn't fully supported, such as in
filenames.
=item Input and Output Layers
Use the C<:encoding(...)> layer to read from and write to
filehandles using the specified encoding. (See L.)
=item You must convert your non-ASCII, non-UTF-8 Perl scripts to be
UTF-8.
The L module has been deprecated since perl 5.18 and the
perl internals it requires have been removed with perl 5.26.
=item C still needed to enable L in scripts
If your Perl script is itself encoded in L,
the S> pragma must be explicitly included to enable
recognition of that (in string or regular expression literals, or in
identifier names). B> is needed.> (See L).
If a Perl script begins with the bytes that form the UTF-8 encoding of
the Unicode BYTE ORDER MARK (C, see L), those
bytes are completely ignored.
=item L scripts autodetected
If a Perl script begins with the Unicode C (UTF-16LE,
UTF16-BE), or if the script looks like non-C-marked
UTF-16 of either endianness, Perl will correctly read in the script as
the appropriate Unicode encoding.
=back
=head2 Byte and Character Semantics
Before Unicode, most encodings used 8 bits (a single byte) to encode
each character. Thus a character was a byte, and a byte was a
character, and there could be only 256 or fewer possible characters.
"Byte Semantics" in the title of this section refers to
this behavior. There was no need to distinguish between "Byte" and
"Character".
Then along comes Unicode which has room for over a million characters
(and Perl allows for even more). This means that a character may
require more than a single byte to represent it, and so the two terms
are no longer equivalent. What matter are the characters as whole
entities, and not usually the bytes that comprise them. That's what the
term "Character Semantics" in the title of this section refers to.
Perl had to change internally to decouple "bytes" from "characters".
It is important that you too change your ideas, if you haven't already,
so that "byte" and "character" no longer mean the same thing in your
mind.
The basic building block of Perl strings has always been a "character".
The changes basically come down to that the implementation no longer
thinks that a character is always just a single byte.
There are various things to note:
=over 4
=item *
String handling functions, for the most part, continue to operate in
terms of characters. C, for example, returns the number of
characters in a string, just as before. But that number no longer is
necessarily the same as the number of bytes in the string (there may be
more bytes than characters). The other such functions include
C, C, C, C, C, C,
C, C, and C.
The exceptions are:
=over 4
=item *
the bit-oriented C
E
=item *
the byte-oriented C/C C<"C"> format
However, the C specifier does operate on whole characters, as does the
C specifier.
=item *
some operators that interact with the platform's operating system
Operators dealing with filenames are examples.
=item *
when the functions are called from within the scope of the
S>> pragma
Likely, you should use this only for debugging anyway.
=back
=item *
Strings--including hash keys--and regular expression patterns may
contain characters that have ordinal values larger than 255.
If you use a Unicode editor to edit your program, Unicode characters may
occur directly within the literal strings in UTF-8 encoding, or UTF-16.
(The former requires a C, the latter may require a C.)
L gives other ways to place non-ASCII
characters in your strings.
=item *
The C and C functions work on whole characters.
=item *
Regular expressions match whole characters. For example, C<"."> matches
a whole character instead of only a single byte.
=item *
The C functionality has been removed. For similar functionality to
that, see S> and S>).
=item *
C reverses by character rather than by byte.
=item *
The bit string operators, C<& | ^ ~> and (starting in v5.22)
C<&. |. ^. ~.> can operate on bit strings encoded in UTF-8, but this
can give unexpected results if any of the strings contain code points
above 0xFF. Starting in v5.28, it is a fatal error to have such an
operand. Otherwise, the operation is performed on a non-UTF-8 copy of
the operand. If you're not sure about the encoding of a string,
downgrade it before using any of these operators; you can use
L|utf8/Utility functions>.
=back
The bottom line is that Perl has always practiced "Character Semantics",
but with the advent of Unicode, that is now different than "Byte
Semantics".
=head2 ASCII Rules versus Unicode Rules
Before Unicode, when a character was a byte was a character,
Perl knew only about the 128 characters defined by ASCII, code points 0
through 127 (except for under L>|perllocale>). That
left the code
points 128 to 255 as unassigned, and available for whatever use a
program might want. The only semantics they have is their ordinal
numbers, and that they are members of none of the non-negative character
classes. None are considered to match C<\w> for example, but all match
C<\W>.
Unicode, of course, assigns each of those code points a particular
meaning (along with ones above 255). To preserve backward
compatibility, Perl only uses the Unicode meanings when there is some
indication that Unicode is what is intended; otherwise the non-ASCII
code points remain treated as if they are unassigned.
Here are the ways that Perl knows that a string should be treated as
Unicode:
=over
=item *
Within the scope of S>
If the whole program is Unicode (signified by using 8-bit Bnicode
Bransformation Bormat), then all literal strings within it must be
Unicode.
=item *
Within the scope of
L>|feature/The 'unicode_strings' feature>
This pragma was created so you can explicitly tell Perl that operations
executed within its scope are to use Unicode rules. More operations are
affected with newer perls. See L.
=item *
Within the scope of S> or higher
This implicitly turns on S>.
=item *
Within the scope of
L>|perllocale/Unicode and UTF-8>,
or L>|perllocale> and the current
locale is a UTF-8 locale.
The former is defined to imply Unicode handling; and the latter
indicates a Unicode locale, hence a Unicode interpretation of all
strings within it.
=item *
When the string contains a Unicode-only code point
Perl has never accepted code points above 255 without them being
Unicode, so their use implies Unicode for the whole string.
=item *
When the string contains a Unicode named code point C<\N{...}>
The C<\N{...}> construct explicitly refers to a Unicode code point,
even if it is one that is also in ASCII. Therefore the string
containing it must be Unicode.
=item *
When the string has come from an external source marked as
Unicode
The L|perlrun/-C [numberElist]> command line option can
specify that certain inputs to the program are Unicode, and the values
of this can be read by your Perl code, see L.
=item * When the string has been upgraded to UTF-8
The function L|utf8/Utility functions>
can be explicitly used to permanently (unless a subsequent
C is called) cause a string to be treated as
Unicode.
=item * There are additional methods for regular expression patterns
A pattern that is compiled with the C<< /u >> or C<< /a >> modifiers is
treated as Unicode (though there are some restrictions with C<< /a >>).
Under the C<< /d >> and C<< /l >> modifiers, there are several other
indications for Unicode; see L.
=back
Note that all of the above are overridden within the scope of
C>; but you should be using this pragma only for
debugging.
Note also that some interactions with the platform's operating system
never use Unicode rules.
When Unicode rules are in effect:
=over 4
=item *
Case translation operators use the Unicode case translation tables.
Note that C, or C<\U> in interpolated strings, translates to
uppercase, while C, or C<\u> in interpolated strings,
translates to titlecase in languages that make the distinction (which is
equivalent to uppercase in languages without the distinction).
There is a CPAN module, C>, which allows you to
define your own mappings to be used in C, C, C,
C, and C (or their double-quoted string inlined versions
such as C<\U>). (Prior to Perl 5.16, this functionality was partially
provided in the Perl core, but suffered from a number of insurmountable
drawbacks, so the CPAN module was written instead.)
=item *
Character classes in regular expressions match based on the character
properties specified in the Unicode properties database.
C<\w> can be used to match a Japanese ideograph, for instance; and
C<[[:digit:]]> a Bengali number.
=item *
Named Unicode properties, scripts, and block ranges may be used (like
bracketed character classes) by using the C<\p{}> "matches property"
construct and the C<\P{}> negation, "doesn't match property".
See L for more details.
You can define your own character properties and use them
in the regular expression with the C<\p{}> or C<\P{}> construct.
See L for more details.
=back
=head2 Extended Grapheme Clusters (Logical characters)
Consider a character, say C. It could appear with various marks around it,
such as an acute accent, or a circumflex, or various hooks, circles, arrows,
I, above, below, to one side or the other, I. There are many
possibilities among the world's languages. The number of combinations is
astronomical, and if there were a character for each combination, it would
soon exhaust Unicode's more than a million possible characters. So Unicode
took a different approach: there is a character for the base C, and a
character for each of the possible marks, and these can be variously combined
to get a final logical character. So a logical character--what appears to be a
single character--can be a sequence of more than one individual characters.
The Unicode standard calls these "extended grapheme clusters" (which
is an improved version of the no-longer much used "grapheme cluster");
Perl furnishes the C<\X> regular expression construct to match such
sequences in their entirety.
But Unicode's intent is to unify the existing character set standards and
practices, and several pre-existing standards have single characters that
mean the same thing as some of these combinations, like ISO-8859-1,
which has quite a few of them. For example, C<"LATIN CAPITAL LETTER E
WITH ACUTE"> was already in this standard when Unicode came along.
Unicode therefore added it to its repertoire as that single character.
But this character is considered by Unicode to be equivalent to the
sequence consisting of the character C<"LATIN CAPITAL LETTER E">
followed by the character C<"COMBINING ACUTE ACCENT">.
C<"LATIN CAPITAL LETTER E WITH ACUTE"> is called a "pre-composed"
character, and its equivalence with the "E" and the "COMBINING ACCENT"
sequence is called canonical equivalence. All pre-composed characters
are said to have a decomposition (into the equivalent sequence), and the
decomposition type is also called canonical. A string may consist
as much as possible of precomposed characters, or it may consist of
entirely decomposed characters. Unicode calls these respectively,
"Normalization Form Composed" (NFC) and "Normalization Form Decomposed".
The C> module contains functions that convert
between the two. A string may also have both composed characters and
decomposed characters; this module can be used to make it all one or the
other.
You may be presented with strings in any of these equivalent forms.
There is currently nothing in Perl 5 that ignores the differences. So
you'll have to specially handle it. The usual advice is to convert your
inputs to C before processing further.
For more detailed information, see L of C<"L"> (letter) property (see
L below). Brackets are not
required for single letter property names, so C<\p{L}> is equivalent to C<\pL>.
More formally, C<\p{Uppercase}> matches any single character whose Unicode
C property value is C, and C<\P{Uppercase}> matches any character
whose C property value is C, and they could have been written as
C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively.
This formality is needed when properties are not binary; that is, if they can
take on more values than just C and C. For example, the
C property (see L below),
can take on several different
values, such as C, C, C, and others. To match these, one needs
to specify both the property name (C), AND the value being
matched against
(C, C, I). This is done, as in the examples above, by having the
two components separated by an equal sign (or interchangeably, a colon), like
C<\p{Bidi_Class: Left}>.
All Unicode-defined character properties may be written in these compound forms
of C<\p{I=I}> or C<\p{I:I}>, but Perl provides some
additional properties that are written only in the single form, as well as
single-form short-cuts for all binary properties and certain others described
below, in which you may omit the property name and the equals or colon
separator.
Most Unicode character properties have at least two synonyms (or aliases if you
prefer): a short one that is easier to type and a longer one that is more
descriptive and hence easier to understand. Thus the C<"L"> and
C<"Letter"> properties above are equivalent and can be used
interchangeably. Likewise, C<"Upper"> is a synonym for C<"Uppercase">,
and we could have written C<\p{Uppercase}> equivalently as C<\p{Upper}>.
Also, there are typically various synonyms for the values the property
can be. For binary properties, C<"True"> has 3 synonyms: C<"T">,
C<"Yes">, and C<"Y">; and C<"False"> has correspondingly C<"F">,
C<"No">, and C<"N">. But be careful. A short form of a value for one
property may not mean the same thing as the short form spelled the same
for another.
Thus, for the C> property, C<"L"> means
C<"Letter">, but for the L|/Bidirectional Character Types>
property, C<"L"> means C<"Left">. A complete list of properties and
synonyms is in L.
Upper/lower case differences in property names and values are irrelevant;
thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>.
Similarly, you can add or subtract underscores anywhere in the middle of a
word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space
is generally irrelevant adjacent to non-word characters, such as the
braces and the equals or colon separators, so C<\p{ Upper }> and
C<\p{ Upper_case : Y }> are equivalent to these as well. In fact, white
space and even hyphens can usually be added or deleted anywhere. So
even C<\p{ Up-per case = Yes}> is equivalent. All this is called
"loose-matching" by Unicode. The "name" property has some restrictions
on this due to a few outlier names. Full details are given in
L.
The few places where stricter matching is
used is in the middle of numbers, the "name" property, and in the Perl
extension properties that begin or end with an underscore. Stricter
matching cares about white space (except adjacent to non-word
characters), hyphens, and non-interior underscores.
You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
(C<^>) between the first brace and the property name: C<\p{^Tamil}> is
equal to C<\P{Tamil}>.
Almost all properties are immune to case-insensitive matching. That is,
adding a C regular expression modifier does not change what they
match. There are two sets that are affected.
The first set is
C,
C,
and C,
all of which match C under C matching.
And the second set is
C,
C,
and C,
all of which match C under C matching.
This set also includes its subsets C and C both
of which under C match C.
(The difference between these sets is that some things, such as Roman
numerals, come in both upper and lower case so they are C, but
aren't considered letters, so they aren't C's.)
See L for special considerations when
matching Unicode properties against non-Unicode code points.
=head3 B
Every Unicode character is assigned a general category, which is the "most
usual categorization of a character" (from
L).
The compound way of writing these is like C<\p{General_Category=Number}>
(short: C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up
through the equal or colon separator is omitted. So you can instead just write
C<\pN>.
Here are the short and long forms of the values the C property
can have:
Short Long
L Letter
LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}])
Lu Uppercase_Letter
Ll Lowercase_Letter
Lt Titlecase_Letter
Lm Modifier_Letter
Lo Other_Letter
M Mark
Mn Nonspacing_Mark
Mc Spacing_Mark
Me Enclosing_Mark
N Number
Nd Decimal_Number (also Digit)
Nl Letter_Number
No Other_Number
P Punctuation (also Punct)
Pc Connector_Punctuation
Pd Dash_Punctuation
Ps Open_Punctuation
Pe Close_Punctuation
Pi Initial_Punctuation
(may behave like Ps or Pe depending on usage)
Pf Final_Punctuation
(may behave like Ps or Pe depending on usage)
Po Other_Punctuation
S Symbol
Sm Math_Symbol
Sc Currency_Symbol
Sk Modifier_Symbol
So Other_Symbol
Z Separator
Zs Space_Separator
Zl Line_Separator
Zp Paragraph_Separator
C Other
Cc Control (also Cntrl)
Cf Format
Cs Surrogate
Co Private_Use
Cn Unassigned
Single-letter properties match all characters in any of the
two-letter sub-properties starting with the same letter.
C and C are special: both are aliases for the set consisting of everything matched by C, C, and C.
=head3 B
Because scripts differ in their directionality (Hebrew and Arabic are
written right to left, for example) Unicode supplies a C property.
Some of the values this property can have are:
Value Meaning
L Left-to-Right
LRE Left-to-Right Embedding
LRO Left-to-Right Override
R Right-to-Left
AL Arabic Letter
RLE Right-to-Left Embedding
RLO Right-to-Left Override
PDF Pop Directional Format
EN European Number
ES European Separator
ET European Terminator
AN Arabic Number
CS Common Separator
NSM Non-Spacing Mark
BN Boundary Neutral
B Paragraph Separator
S Segment Separator
WS Whitespace
ON Other Neutrals
This property is always written in the compound form.
For example, C<\p{Bidi_Class:R}> matches characters that are normally
written right to left. Unlike the
C> property, this
property can have more values added in a future Unicode release. Those
listed above comprised the complete set for many Unicode releases, but
others were added in Unicode 6.3; you can always find what the
current ones are in L. And
L
The world's languages are written in many different scripts. This sentence
(unless you're reading it in translation) is written in Latin, while Russian is
written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
Hiragana or Katakana. There are many more.
The Unicode C
