view Side-By-Side changes

draft-hoffman-i18n-terms-01.txt
draft-hoffman-i18n-terms-02.txt                           IMC & VPNC
January 17,
July 18, 2001
Expires in six months

          Terminology Used in Internationalization in the IETF

Status of this memo

This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC2026.

Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups. Note that other groups
may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.

Abstract

This document provides a glossary of terms used in the IETF when
discussing internationalization. The purpose is to help frame
discussions of internationalization in the various areas of the IETF and
to help introduce the main concepts to IETF participants.


1. Introduction

As [RFC2277] summarizes: "Internationalization is for humans. This means
that protocols are not subject to internationalization; text strings
are." Many protocols throughout the IETF use text strings that are
entered by, or are visible to, humans. It should be possible to make for anyone
can enter or read these text strings readable to anyone, strings, which means that the text must be
able to be entered in typical input methods and displayed in any human
language. This is the challenge of internationalization.

1.1 About this document

Internationalization is discussed in many working groups of the IETF.
However, few working groups have internationalization experts. When
designing or updating protocols, the question often comes up "should we
internationalize this" (or, more likely, "do we have to internationalize
this").

This document gives an overview of internationalization as it applies to
IETF standards work by covering lightly the many aspects of
internationalization and the vocabulary associated with those topics. It
is not meant to be a complete description of internationalization. The
definitions in this document are not normative for IETF standards;
however, they are useful and standards may make non-nomative reference
to this document after it becomes an RFC. Some of the definitions in
this document come from many earlier IETF documents and books.

As in many fields, there is disagreement in the internationalization
community on definitions for many words. The topic of language brings up
particularly passionate opinions for experts and non-experts alike. This
document attempts to define terms that will be most useful to the IETF
audience.

This document uses definitions from many documents that have been
developed outside the IETF. The primary documents used are:
- ISO/IEC 10646 [ISOIEC10646]
- The Unicode Standard [UNICODE]
- W3C Character Model [CHARMOD]
- IETF RFCs, including [RFC2277]

In the body of this document, the source for the definition is shown in
angle brackets, such as <ISOIEC10646>. Note that many definitions are
shown as <NONE>, which means that the definitions were crafted
originally for this document.

For some terms, there is a very early draft of commentary and examples after the document. Many definitions
here will likely change,
definitions. In those cases, the part before the angle brackets is the
definition that comes from the original source, and some the part after the
angle brackets is non-definition commentary.


2. Fundamental Terms

This section covers basic topics may be added. Discussion that are needed for almost anyone who
is involved with internationalization of IETF protocols. Many terms in
this document is encouraged. Information section are based on the mailing list for this
document can be found at <http://www.imc.org/ietf-i18n-terms/>.

1.2 Foundations for internationalization [IDN-REQ].

language

A language is a way that humans interact. The use of language occurs in
many forms, the most common of which are speech, writing, vocal, and visual.
Each language form is independent: some signing.
<NONE>

Some languages have a close relationship between the written and vocal spoken
forms, while others have a looser relationship. [RFC1766] and [RFC1766bis] discuss [RFC3066] discusses
languages in more detail.

script

A script is a collection set of symbols graphic characters used to represent textual
information in for the written form of one or more writing systems. A script can have many
attributes, such as the number
languages. <ISOIEC10646>

Examples of written characters scripts are Latin, Cyrillic, Greek, Arabic, and the rules for
combining written characters. Most IETF protocols that deal with
languages deal only with scripts, not spoken or visual forms. Han (the
ideographs used in writing Chinese, Japanese, and Korean). [RFC2277]
discusses scripts in more detail.

It is common for internationalization novices to mix up the terms
"language" and "script".  This can be a problem in protocols that
differentiate the two, such as mail content protocols. two. Almost all internationalized protocols deal with
scripts (the written systems), while fewer deal with languages. Many languages

A single name can be expressed using
different scripts.

grapheme, phoneme, alphabet

A grapheme is an abstract, atomic written entity of mean either a script. A phoneme
is an abstract, atomic spoken entity of language or a spoken language. An alphabet script; for example,
"Arabic" is both the name of a script that maps between graphemes language and phonemes.

internationalization

In the IETF, the verb "internationalize" means to add or improve the
handling name of international information in a protocol. Many protocols that
handle text only handle one script, the one that contains the letters script. Some
scripts are used in English text. Internationalizing such a protocol allows
the protocol to handle more scripts, hopefully all of for many languages; for example, the ones
useful to anyone Russian and
Bulgarian languages are written in the world.

localization

Internationalized applications Cyrillic script. Some languages
can handle a wide variety of languages.
Typical users only understand a small number of languages, so the
program must be tailored to interact with users in just the languages
they know. Localization involves not only changing expressed using different scripts; the Mongolian language
interaction, but also other relevant changes such as display of numbers,
dates, currency, and so on.

i18n, l10n

These are abbreviations for "internationalization" and "localization".
"18" is
written in both the number of characters between the "i" Mongolian and Cyrillic scripts. Further, some
languages are normally expressed with many scripts at the "n" in
"internationalization", and "10" is same time; for
example, the number of characters between Japanese language is normally expressed in the
"l" Han,
Katakana, and the "n" Hiragana scripts in "localization".


2. Fundamental Terms

This section covers basic topics that are needed for almost anyone who
is involved with internationalization a single string of IETF protocols. Many terms in
this section are based on [IDN-REQ].

characters

A text.

character is a

A member of a set of elements used for the organization, control, or
representation of data. In written form, <ISOIEC10646>

There are two definitions of the word "character":

- a language is
expressed in characters. The same set general description of characters can often be used
to express many languages.

In ISO 10646 [ISO10646], a text entity

- the encoded entity itself

When people talk about characters, they are mostly using the first
definition. Standards, however, mostly use the second definition.

A particular character is identified by its name, not by its shape. A
name may suggest a meaning, but the character may be used for
representing other meanings as well. A name may suggest a shape, but
that does not imply that just that is commonly used in print.

coded character

A coded character is a character together with its coded representation. <ISOIEC10646>

coded character set

A coded character set (CCS) is a set of unambiguous rules that establish
establishes a character set and the relationship between the characters
of the set and their coded representation. The specified set of characters in <ISOIEC10646>

character encoding form

A character encoding form is a CCS mapping from a character set definition
to the actual code units used to represent the data. <UNICODE>

transcoding

Transcoding is
often called the "repertoire". process of converting text data from one character
encoding form to another. Transcoders work only at the level of
character encoding and do not parse the text. Note: Transcoding may
involve one-to-one, many-to-one, one-to-many or many-to-many mappings.
Because some legacy mappings are glyphic, they may not only be
many-to-many, but also discontinuous: thus XYZ may map to yxz. <CHARMOD>

character encoding scheme

A character encoding scheme (CES) is a mapping from one or more coded character sets to a set of octets. encoding form plus byte
serialization. There are many character encoding schemes in Unicode,
such as UTF-8 and UTF-16. <UNICODE>

Some CESs are associated with a single CCS; for example, UTF-8 [RFC2279]
applies only to ISO ISO/IEC 10646. Other CESs, such as ISO 2022, are
associated with many CCSs.

charset

A charset is a method of mapping a sequence of octets to a sequence of
abstract characters. A charset is, in effect, a combination of one or
more CCS with a CES. Charset names are registered by the IANA according
to procedures documented in [RFC2278]. <NONE>

Many protocol definitions use the term "character set" in their
descriptions. The term "charset" is strongly preferred over the term
"character set" because "character set" has many other definitions in
other contexts and can ths be confusing.

internationalization

In the IETF, the verb "internationalize" means to add or improve the
handling of international information in a protocol. <NONE>

Many protocols that handle text only handle one script (often, the one
that contains the letters used in English text), or leaves the question
of what character set used up to local guesswork (which leads, of
course, to interoperability problems). Internationalizing such a
protocol allows the protocol to handle more scripts, hopefully all of
the ones useful to anyone in the world.

localization

A particular charset process of adapting an internationalized application platform or
application to a specific cultural environment. In localization, the
same semantics are preserved while the syntax may have
different glyphs depending on be changed.
[FRAMEWORK]

Localization is the act of tailoring an application for a different
language being used.

transfer encoding syntax

A transfer encoding syntax (TES) (sometimes called or script. Some internationalized applications can handle a transfer encoding
scheme) is
wide variety of languages. Typical users only understand a reversible transform small number
of already-encoded data represented languages, so the program must be tailored to interact with users in
one or more character encoding schemes. TESs
just the languages they know.

The major work of localization is translating the user interface and
documentation, not dealing with localization of protocols. Localization
involves not only changing the language interaction, but also other
relevant changes such as display of numbers, dates, currency, and so on.

Do not confuse "localization" with "locale", which is described later in
this document.

i18n, l10n

These are useful abbreviations for encoding
types "internationalization" and "localization".
<NONE>

"18" is the number of characters between the "i" and the "n" in
"internationalization", and "10" is the number of characters between the
"l" and the "n" in "localization".

multilingual

The term "multilingual" has many widely-varying definitions and thus is
not recommended for use in standards. Some of the definitions relate to
the ability to handle international characters; other definitions relate
to the ability to handle multiple charsets; and still others relate to
the ability to handle multiple languages. <NONE>

displaying and rendering text

To display text, a system puts characters on a visual display device
such as a screen or a printer. To render text, a system analyzes the
character data into an another format, usually for allowing new
types of data input to be transmitted over legacy protocols. Examples of TESs determine how to display the text. The terms
"display" and "render" are sometimes used interchangeably. <NONE>

Combining characters modify the display of the character (or, in some
cases, characters) that precede them. When rendering such text, the IETF include Base64 and quoted-printable (described
display engine must either find the glyph in more
detail the font that represents
the base character and all of the combining characters, or it must
render the combination itself. Such rendering can be straight-forward,
but it is sometimes complicated when the combining marks interact with
each other, such as when there are two combining marks that would appear
above the same character. Formatting characters can also change the way
that a renderer would display text.

Note that text might be rendered as audio and/or tactile output, such as
in Chapter 6). systems that have been designed for people with visual disabilities.


3. Standards Bodies and Standards

This section describes some of the standards bodies and standards that
appear in discussions of internationalization in the IETF. This is an
incomplete and possibly over-full list; listing too few bodies or
standards can be just as politically dangerous as listing too many. Note
that there are many other bodies that deal with internationalization;
however, none of them appear commonly in IETF standards work.

3.1 Standards bodies

ISO

The International Organization for Standardization has been involved
with standards for scripts characters since before the IETF was started. ISO is
a non-governmental group made up of national bodies. ISO has many
diverse standards in the international scripts characters area; the one that is
most used in the IETF is commonly referred to as "ISO "ISO/IEC 10646",
although its official name has more qualifications. ISO (The IEC is
International Electrotechnical Commission). ISO/IEC 10646 describes a
CCS that covers almost all known written characters in use today.

ISO

ISO/IEC 10646 is controlled by the group known as "ISO/IEC JTC 1/SC 2
WG2", often called "WG2" for short. ISO standards go through many steps
before being finished, and years often go by between changes to ISO ISO/IEC
10646. Information on WG2, and its work products, can be found at
<http://anubis.dkuug.dk/JTC1/SC2/WG2/>.

The standard can be purchased in both print and CD-ROM versions. It
comes in multiple parts, and is amended quite often. This leads to
difficulty in citing a specific instantiation of the standard in IETF
documents.

Unicode Consortium

The second important group for international character standards is the
Unicode Consortium. The Unicode Consortium is a trade association of
companies
companies, governments, and governments other groups interested in promoting the
Unicode Standard [Unicode3]. The Unicode Standard is a CCS whose
repertoire is identical to ISO ISO/IEC 10646. The Unicode Consortium has
added features to the base CCS which make it more useful in protocols,
such as defining attributes for each character.

The Unicode Consortium publishes addenda to the Unicode Standard as
Unicode Technical Reports. There
Unicode Technical Reports. There are many types of technical reports at
various stages of maturity. The Unicode Standard and affiliated
technical reports can be found at <http://www.unicode.org/>.

World Wide Web Consortium

Commonly known as the "W3C", this group created and maintains the
standard for XML, the markup language for text that has become very
popular. XML has always been fully internationalized so that there has
never needed to be a new version to handle international text.

local and regional standards organizations

Just as there are many types of technical reports at
various stages native CCSs and charsets, there are many local
and regional standards organizations to create and support them. Common
examples of maturity. The Unicode Standard these are ANSI (United States), and affiliated
technical reports can be found at <http://www.unicode.org/>

encodings CEN/ISSS (Europe).

3.2 Encodings and transformations transformation formats of ISO ISO/IEC 10646

Characters in the ISO ISO/IEC 10646 CCS can be expressed in many ways.
Encoding forms are direct addressing methods, while transformation
formats are methods for expressing encoding forms as bits on the wire. Two

Basic Multilingual Plane (BMP)

The BMP is the first 2^16 code points in ISO/IEC 10646. The BMP is also
called "plane 0". <NONE>

UCS-2 and UCS-4

UCS-2 and UCS-4 are the two encoding forms are defined for ISO 10646: ISO/IEC 10646.
UCS-2 addresses only the first 2^15 BMP. Because characters as 16-bit values. This range
is also called have been defined
outside of the "Basic Multilingual Plane" (BMP), and is also called
"plane 0". BMP, many would consider UCS-2 to be dead. UCS-4
addresses the entire range of 2^31 characters code points from ISO/IEC 10646 as
32-bit values.

Many transformation formats of the CCS are used in IETF standards. The
two most common are: <NONE>

UTF-8

UTF-8, a transformation format defined in [RFC2279], is the preferred
encoding for IETF protocols. Characters in the BMP are encoded as one,
two, or three octets.

UTF-16 (BE & LE), Characters outside the BMP are encoded as four
octets.

UTF-16, UTF-16BE, and UTF-16LE

UTF-16, UTF-16BE, and UTF-16LE, three transformation formats defined in
[RFC2781], is are not required by any IETF standards, and are thus used
much less often than UTF-8. Characters in the BMP are always encoded as
two octets, and many characters outside the BMP are encoded as four
octets.

native The three formats differ based on the order of the octets and
the presence of a special lead-in mark.

3.3 Native CCSs and charsets

Before ISO ISO/IEC 10646 was developed, many countries developed their own
CCSs and charsets. Many dozen of these are in common use on the Internet
today. Examples include ISO 8859-5 (Russia) for Cyrillic and Shift-JIS (Japan). for
Japanese scripts.

The official list of the registered charset tags names for use with IETF
protocols is maintained by IANA.

ASCII IANA and can be found at
<http://www.iana.org/assignments/character-sets>. The list contains
preferred names and aliases.

Probably the most well-known native CCS is ASCII [US-ASCII]. This CCS is
used in numerous IETF protocols that have not yet been
internationalized.

local and regional standards organizations

Just as there are many native CCSs and charsets, there are many local
and regional standards organizations to create and support them. Common
examples of these are ANSI (United States), JIS (Japan), GB (China), and
CEN/ISSS (Europe). been
internationalized.


4. Linguistic Character Issues

This section contains terms and topics that are commonly used in
linguistics
character handling and therefore are of concern to people
internationalizing protocols. These topics are standardized outside the
IETF.

composition

combining character

A member of an identified subset of the coded character set of ISO/IEC
10646 intended for combination with the preceding non-combining graphic
character, or with a sequence of combining characters preceded by a
non-combining character. <ISOIEC10646>

composite sequence

A sequence of graphic characters consisting of a non-combining character
followed by one or more combining characters. A graphic symbol for a
composite sequence generally consists of the combination of the graphic
symbols of each character in the sequence. A composite sequence is not a
character and decomposition therefore is not a member of the repertoire of ISO/IEC
10646. <ISOIEC10646>

In some CCSs, some characters consist of combinations of other
characters. For example, the letter "a with acute" might be a
combination of the two characters "a" and "combining acute". The rules
for combining two or more characters are called "composition", "composition rules", and
the rules for taking apart a character into other characters is called
"decomposition".
"decomposition rules". The results of composition is called a
"precomposed character"; the results of decomposition is called a
"decomposed character".

normalization

Normalization is the transformation of data to a normal form, for
example, to unify spelling. <UNICODE>

Note that the phrase "unify spelling" in the definition above does not
mean unifying different words with the same meaning (such as "color" and
"colour"). Instead, it means unifying different character sequences that
are intended to form the same composite characters (such as
"<a><n><combining tilde><o>" and "<a><n with tilde><o>").

The purpose of normalization is to allow two strings to be compared for
equivalence. The strings "<a><n><combining tilde><o>" and canonicalization

These "<a><n with
tilde><o>" would be shown identically on a text display device. If a
protocol designer wants those two strings to be considered equivalent
during comparison, the protocol must include normalization.

The terms "normalization" and "canonicalization" are often used interchangeably in internationalization.
interchangeably. Generally, they both mean to convert a string of one or
more characters into another string based on standardized rules. Some
CCSs allow multiple equivalent representations for a written string;
normalization selects one among multiple equivalent representations as a
base for reference purposes in comparing strings. In internationalized
text, these rules are usually based on decomposing combined characters
or composing characters with combining characters. [UTR15] describes the
process and many forms of normalization in detail. Normalization is
important when comparing strings to see if they are the same.

locale and region

Because languages differ from country to country (and even region to
region within a country), the locale of

case

Case is the user feature of internationalized
text can often be an important factor. Typically, the locale information
for a user includes certain alphabets where the language(s) used. Locale issues go beyond
character use, letters have two
distinct forms. These variants, which may differ markedly in shape and can include things such as
size, are called the display format for
currency, dates, uppercase letter (also known as capital or
majuscule) and times.

case

In many scripts, particularly those from the lowercase letter (also known as small or based on European alphabets,
there are two forms for letters: minuscule).
Case mapping is the association of the uppercase and lowercase. lowercase forms of
a letter. <UNICODE>

There is usually (but not always) a one-to-one mapping between the same
letter in the two cases. However, there are many examples of characters
which exist in one case but for which there is no corresponding
character in the other case. Case conversion mapping can even be dependant on
locale. Converting between the two cases text to have only one case is sometimes called "folding". "case folding".

sorting and collation

Collating is the process of ordering units of textual information.
Collation is usually specific to a particular language. It is also known
as alphabetizing. <UNICODE>

Many processes have a need to order strings in a consistent sequence
(sorted). For some CCS/CES combinations, there is an obvious sort order
that can be done without reference to the linguistic meaning of the
characters: the codepoint order is sufficient. That is, the codepoint
order is also the order that a person would use in sorting the
characters. For other CCS/CES (such as the ISO 2022 family) there is no such family), the
codepoint order that works well. would make no sense to a person and therefore is not
useful for sorting if the results will be displayed to a person.

Codepoint order is usually not how any human educated by a local school
system expects to see strings ordered; if one orders to the expectations
of a human, one has a localized sort. Sorting to codepoint order will
seem inconsistent if the strings are not normalized before sorting
because different representations of the same character will sort
differently. This problem may be smaller with a localized sort.

glyph

code table

A graphic character or glyph code table is a character, other than a control
function, that has a visual representation (normally handwritten,
printed, or displayed). It is table showing the characters allocated to the octets
in a recognizable abstract graphic symbol
which is independent code. <ISOIEC10646>

4.1 Types of any specific design. There characters

The following definitions of types of characters do not clearly
delineate each character into one type, nor do they allow someone to
accurately predict what types would apply to a particular character. The
definitions are useful for application designers to think about the many types
(sometimes confusing) properties of
glyphs, including text.

alphabetic characters, digits, punctuation,
diacritics, and symbols.

font

A collection of glyph images having

An informative Unicode property. Characters that are the same basic design.

code table or code page

A tabular representation primary units
of alphabets and/or syllabaries, whether combining or noncombining. This
includes composite characters that are canonical equivalents to a coded
combining character set, also showing the
coded representations.

code space

The numeric domain occupied by all bit combinations used for the coding sequence of a coded an alphabetic base character set.

4.1 Types plus one or
more combining characters: letter diagraphs; contextual variant of characters
alphabetic

Characters from the spoken part characters; ligatures of phonetic or syllabic scripts.
Examples include Latin letters a through z, Arabic letters, alphabetic characters; contextual
variants of ligatures; modifier letters; letterlike symbols that are
compatibility equivalents of single alphabetic letters; and Katakana
characters from Japanese.
miscellaneous letter elements. <UNICODE>

ideographic

Characters that, by themselves, represent words or concepts (as compared

Any symbol that primarily denotes an idea (or meaning) in contrast to phonetic sounds). Examples include a
sound (or pronunciation), for example, a symbol showing a telephone or
the Han characters used in Chinese, Japanese, and Korean. <UNICODE>

punctuation

Non-alphabetic characters

Characters that are used to delimit sounds, sentences, separate units of text, such as sentences and phrases. Examples include phrases,
thus clarifying the period, comma, meaning of the text. The use of punctuation marks is
not limited to prose; they are also used in mathematical and hyphen.

symbols

Characters scientific
formulae, for example. <UNICODE>

symbol

A character that represent pictures represents a picture or icons icon (as compared to ones one that
represent letters
represents a letter or punctuation). <NONE>

Examples of symbols include characters for arrows, faces, and geometric
shapes.

spacing [UNICODE] has a property that defines characters

Characters as symbols.

nonspacing character

A character that represent horizontal or vertical spaces is not a combining character but does not cause a
change in written text. position in display. <NONE>

Examples of non-spacing characters include the space character, the tab character, U+FEFF (the zero-width
no-break space), control characters such as U+0007 (bell), and the
paragraph mark.

diacritics

Characters
formatting characters such as U+200C (zero width non-joiner).

diacritic

A mark applied or attached to a symbol to create a new symbol that combine with alphabetic characters, usually
represents a modified or new value. They can also be marks applied to change
the spoken pronunciation a
symbol irrespective of whether it changes the base alphabetic character. Examples
include the combining acute accent, combining tilde, and combining ring
above.

combining character

Characters value of that visually change symbol. In
the characters that precede them.
Examples include combining diacritics, many Arabic alphabetic letters,
and many letters from latter case, the Indic scripts. diacritic usually represents an independent value
(for example, an accent, tone, or some other linguistic information).
Also called diacritical mark or diacritical. <UNICODE>

control character

Non-displaying

The 65 characters that cause changes in the systems in which
they are entered. Examples include the null character, the delete
character, ranges U+0000..U+001F and the bell character. U+007F..U+009F. They
are also known as control codes. <UNICODE>

formatting character

Non-displaying character

Characters that has are inherently invisible but that have an effect on the
surrounding characters. <UNICODE>

Examples of formatting characters include characters for specifying the
direction of text, letter-spacing characters, and characters for
specifying joining.

compatibility character

A graphic character included as a coded character of ISO/IEC 10646
primarily for compatibility with existing coded character sets.
<ISOIEC10646>


5. User interface for internationalized text

Although the IETF does not standardize user interfaces, many protocols
make assumptions about how a user will enter or see text that is used in
the protocol. Many protocols make inherent assumptions such as that text
will be typed on a standard (that is, US-centric) keyboard, or that text
will be displayed on a character-based monitor. Internationalization challenges assumptions like these, about the type
and it limitations of the input and output devices that may be used with
applications that use various protocols. It is therefore useful to
consider how users typically interact with internationalized text.

input methods

An input method is a mechanism for a person to enter text into an
application. <NONE>

Text can be entered into a computer in many ways. Keyboards are by far
the most common method device used, but many characters cannot be entered on
typical computer keyboards. keyboards in a single stroke. Many operating systems
come with system software that lets users input characters outside the
range of what is allowed by keyboards.

For example, there are dozens of different input method methods for Han
characters in Chinese, Japanese, and Korean. Some start with phonetic
input through the keyboard, while others use the number of strokes in
the character. Input methods are also needed for scripts that have many
diacritics, such as European characters that have two or three
diacritics on a single alphabetic character.

rendering rules

A rendering rule is an algorithm that a system uses to decide how to
display methods a string of text. <NONE>

Many scripts can be directly displayed with fonts, where each character
from an input stream can simply be copied from a font system and put on
the screen. screen or printed page. Other scripts need rules that are based on
the input stream in order to display text. render text for display.

Some examples of these display rendering rules include:

- Scripts such as Arabic (and many others), where the form of the letter
changes depending on the adjacent letters, whether the letter is
standing alone, at the beginning of a word, in the middle of a word, or
at the end of a word word. The rendering rules must choose between two or
more characters.

- Scripts such as the Indic scripts, where consonants may change their
form if they are adjacent to certain other consonants or may be
displayed in an order different from the way they are stored and
pronounced. The rendering rules must choose between two or more
characters.

- Arabic and Hebrew script, scripts, where the order of the characters displayed
can be changed by the bidirectional properties of the alphabetic
characters and with right-to-left and left-to-right ordering marks marks. The
rendering rules must choose the order that characters

Combining characters modify are displayed.

graphic symbol

A graphic symbol is the display visual representation of the a graphic character (or, in some
cases, characters) or
of a composite sequence. <ISOIEC10646>

glyph

A glyph is an abstract form that precede them. When rendering such text, represents one or more glyph images.
The term "glyph" is often a synonym for glyph image, which is the
actual, concrete image of a glyph representation having been rasterized
or otherwise imaged onto some display surface. In displaying character
data, one or more glyphs may be selected to depict a particular
character. These glyphs are selected by a rendering engine must either find during
composition and layout processing. <UNICODE>

glyph code

A glyph code is a numeric code that refers to a glyph. Usually, the character
glyphs contained in the a font are referenced by their glyph code. Glyph
codes are local to a particular font; that contains is, a different font
containing the base character and all same glyphs may use different codes. <UNICODE>

font

A font is a collection of glyphs used for the combining characters, or it must
render the combination itself. Such rendering can be straight-forward,
but it visual depiction of
character data. A font is sometimes complicated when the combining marks interact often associated with
each other, such as a set of parameters (for
example, size, posture, weight, and serifness), which, when there are two combining marks that would appear
above one character. set to
particular values, generate a collection of imagable glyphs. <UNICODE>

bidirectional scripts display

The process or result of mixing left-to-right oriented text and
right-to-left oriented text in a single line is called bidirectional
display. <UNICODE>

Most of the world's written languages are displayed left-to-right.
However, many widely-used written languages such as Hebrew and ones based on the
Hebrew or Arabic script scripts are displayed right-to-left. Right-to-left text
often confounds protocol writers because they have to keep thinking in
terms of the order of characters in a string in memory, and that order
might be different than what they see on the screen. (Note that some
scripts in some languages are written both horizontally and vertically.)

Bidirectional text can cause even more confusion because there are
formatting characters in ISO ISO/IEC 10646 which cause the order of display
of text to change. These formatting characters are needed so that one
can properly display right-to-left characters with left-to-right
characters at the same time without forcing the one of the strings to be
stored in reverse order.

It is common to see strings with text in both directions, such as
strings that include both text and numbers, or strings that contain a
mixture of scripts.

[Unicode3] has a long and incredibly detailed discussion
of algorithm for displaying
bidirectional text.

undisplayable characters

Some characters have character

A character that has no displayable form. <NONE>

For instance, the zero-width space (U+200B) cannot be displayed because
it takes up no horizontal space. Formatting characters such as those for
setting the direction of text are also undisplayable. Note, however,
that every character in [Unicode3] has a glyph associated with it, and
that the glyphs for undisplayable characters use are enclosed in a
dashed square as an indication that the actual character is
undisplayable


6. Text in current IETF protocols

Many IETF protocols started off being fully internationalized, while
others have been internationalized as they were revised. In this
process, IETF members have seen patterns in the way that many protocols
use text. This section describes some specific protocol interactions
with text.

protocol elements

[[ No definition yet. ]] <NONE>

Almost every protocol has named elements, such as "source port" in TCP.
In some protocols, the names of the elements (or text aliases for the
names) are transmitted within the protocol. For example, in SMTP, the
names of the verbs are part of the
command stream. The names of protocol elements are not normally seen
by end users.

on-the-wire encoding

Characters exist as codepoints in a charset. Before being transmitted
in a protocol, they must first be encoded. Similarly, when characters
are received in a transmission, they have been encoded, and a protocol
that needs to process the individual characters needs to decode them
before processing. command stream. The encoding and decoding used before and after
transmission is often called the "on-the-wire" (or sometimes just "wire")
format. names of protocol
elements are not normally seen by end users.

name spaces

[[ No definition yet. ]] <NONE>

Many items in Internet protocols use names to identify content. The
field of names for particular item is called its name space. The names
in a name space may be controlled centrally (such as by IANA) or may
have distributed control, such as the names in the DNS.

on-the-wire encoding

The encoding and decoding used before and after transmission is often
called the "on-the-wire" (or sometimes just "wire") format. <NONE>

Characters are identified by codepoints. Before being transmitted in a
protocol, they must first be encoded. Similarly, when characters are
received in a transmission, they have been encoded, and a protocol that
needs to process the individual characters needs to decode them before
processing.

parsed text

Text strings that searched for subparts. <NONE>

In some protocols, free text in text fields might be parsed. For
example, many mail user agents will parse the words in the text of the
Subject: field to attempt to thread based on the "Re:" prefix.

charset tagging identification

Specification of the charset used on a string of text. <NONE>

Protocols that allow more than one charset to be used on text that
is meant for human consumption usually require
that the text be tagged identified with the appropriate charset. Without this
tagging,
identification, a program looking at the text cannot definitively
discern the charset of the text. Charset identification is also called
"charset tagging".

language identification

Specification of the human language tagging used on a string of text. <NONE>

Some protocols allow text that is meant for machine processing to be tagged
identified with the language used in the text. Such tagging identification is
important for machine-processing of the text, such as by systems that "display"
render the text by speaking it. Language identification is also called
"language tagging".

MIME

MIME (Multipurpose Internet Mail Extensions) is a message format that
allows for textual message bodies and headers in character sets other
than US-ASCII in formats that require ASCII (most notably, RFC 822). 2822, the
standard for Internet mail headers). MIME is described in RFCs 2045
through 2049. <NONE>

transfer encoding syntax

A transfer encoding syntax (TES) (sometimes called a transfer encoding
scheme) is a reversible transform of already-encoded data represented in
one or more character encoding schemes. <NONE>

TESs are useful for encoding types of character data into an another
format, usually for allowing new types of data to be transmitted over
legacy protocols. Examples of TESs used in the IETF include Base64 and
quoted-printable.

Base64

Base64 is a transfer encoding syntax that allows binary data to be
represented by the ASCII characters A through Z, a through z, 0 through
9, +, /, and =. It is described defined in RFC 2045. <NONE>

quoted printable

The original design of the quoted

Quoted printable is a transfer encoding syntax was
to allow that allows strings that had some
have non-ASCII printable characters mixed in with mostly ASCII printable
characters to be somewhat human readable. It is described in RFC 2047.
<NONE>

The quoted printable syntax is generally considered to generally be a
failure at being readable. It is generally considered to be somewhat of jokingly referred to as "quoted
unreadable".

XML

XML (which is an approximate abbreviation for Extensible Markup
Language) is a
failure popular method for structuring text. XML text is
explicitly tagged with charsets. The specification for XML can be found
at being readable. <http://www.w3.org/XML/>. <NONE>

ASN.1 text formats

The ASN.1 data description language has many formats for text items. The
formats allow for different repertoires and different encodings. Some of
the formats that appear in IETF standards based on ASN.1 include
IA5String (all ASCII characters), PrintableString (most ASCII
characters, but missing many punctuation characters), BMPString
(characters from ISO ISO/IEC 10646 plane 0 in UTF-16BE format), UTF8String
(just as the name implies), and TeletexString (also called T61String;
the repertoire changes over time).

ASCII-compatible encoding (ACE)

Starting in 2000, 1996, many ASCII-compatible encoding schemes (which are
actually transfer encoding syntaxes) have been proposed as possible
solutions for internationalizing host names. Their goal is to be able to
encode any string of ISO ISO/IEC 10646 characters as legal DNS host names
(as described in STD 13). At the time of this writing, non no ACE has become
an IETF standard.


7. Other Common Terms In Internationalization

This is a hodge-podge of other terms that have appeared in
internationalization discussions in the IETF. It is likely that
additional terms will be added as this document matures.

locale and region

Because languages differ from country to country (and even region to
region within a country), the locale of the user of internationalized
text can often be an important factor. Typically, the locale information
for a user includes the language(s) used. <NONE>

Locale issues go beyond character use, and can include things such as
the display format for currency, dates, and times. Some locales
(especially the popular "C" and "POSIX" locales) do NOT include language
information.

Latin characters

"Latin characters" is a not-precise term for characters derived from historically
related to ancient Greek script and currently used throughout the world.
The base Latin characters make up the ASCII repertoire and have been
augmented by many single and multiple diacritics. diacritics and quite a few other
characters. ISO/IEC 10646 encodes the Latin characters in the ranges
U+0020..U+024F, U+1E00..U+1EFF, and other ranges. <NONE>

romanization

The transliteration of a non-Latin script into Latin characters. <NONE>

Because of the widespread use of Latin characters, people have tried to
represent many languages that are not based on a Latin repertoire in
Latin. This process is called "romanization". For example, there are two popular romanizations of Chinese:
Wade-Giles and Pinyin, the latter of which is by far more common today.
Most romanization systems are inexact and do not give perfect round trip
mappings between the native script and the Latin characters.

CJK characters and Han characters

The ideographic characters used in Chinese, Japanese, Korean, and
traditional Vietnamese scripts are often called "CJK" characters "CJK characters" after
the initial letters of the script names in English. They are also called
"Han" characters,
"Han characters", after the romanized translation of the term in Chinese
that is often used for these characters. <NONE>

Note that CJK and Han characters do not include the phonetic characters
of the Japanese or Korean alphabets.

translation

The process of converting one language to another, or one script to
another, is called translation. another. <NONE>

Most language translation systems are inexact and do not give one-to-one
round trip mappings between the languages.

transliteration

The process of converting one script to another. <NONE>

Most script translations transliterations are exact, and many have perfect round-trip
mappings.

regular expressions

Regular expressions are a language used to search for text within
strings, and possibly modify the text found with other text. <NONE>

Pattern matching for text involves being able to represent one or more
code points in an abstract notation, such as searching for all capital
Latin letters or all punctuation. The most common mechanism in IETF
protocols for naming such patterns is the use of regular expressions.
There is no single regular expression language, but there are numerous
very similar dialects.

private use characters

Many CCSs have

ISO/IEC 10646 code points that from U+E000 to U+F8FF, U+F0000 to U+FFFFD, and
U+100000 to U+10FFFD are defined as existing characters that
have available for private use. This refers to code
points of the standard whose interpretation is not defined semantics. specified by the
standard and whose use may be determined by private agreement among
cooperating users. <UNICODE>

The use of these "private use" characters is defined by the parties who
transmit and receive them, and is thus not appropriate for
standardization.

8. Security Considerations

Security is not discussed in this document.


9. References

[CHARMOD] Character Model for the World Wide Web 1.0, W3C,
http://www.w3.org/TR/charmod/

[FRAMEWORK] ISO/IEC TR 11017:1997(E). Information technology - Framework
for internationalization, prepared by ISO/IEC JTC 1/SC 22/WG 20.

[IDN-REQ] "Requirements of Internationalized Domain Names", work in
progress (draft-ietf-idn-requirements), Z. Wenzel and J. Seng.

[ISO10646]

[ISOIEC10646] ISO/IEC 10646-1:2000. International Standard --
Information technology -- Universal Multiple-Octet Coded Character Set
(UCS) -- Part
1: Architecture and Basic Multilingual Plane.

[RFC1766] "Tags for the Identification of Languages", RFC 1766, H.
Alvestrand.

[RFC1766bis] "Tags for the Identification of Languages", work in
progress (draft-alvestrand-lang-tag-v2), H. Alvestrand. -- Part 1: Architecture and Basic Multilingual Plane.

[RFC2277] "IETF Policy on Character Sets and Languages", RFC 2277, H.
Alvestrand.

[RFC2279] "UTF-8, a transformation format of ISO 10646", RFC 2279, F.
Yergeau.

[RFC2781] "UTF-16, an encoding of ISO 10646", RFC 2781, P. Hoffman and
F. Yergeau.

[Unicode3]

[RFC3066] "Tags for the Identification of Languages", RFC 3066, H.
Alvestrand.

[UNICODE] The Unicode Consortium, "The Unicode Standard -- Version 3.0",
ISBN 0-201-61633-5. Described at
<http://www.unicode.org/unicode/standard/versions/Unicode3.0.html>.

[US-ASCII]  Coded Character Set -- 7-bit American Standard Code for
Information Interchange, ANSI X3.4-1986.

[UTR15] "Unicode Normalization Forms", Unicode Technical Report #15, M.
Davis & M. Duerst.


10. Additional Interesting Reading

ALA-LC Romanization Tables, Randall Barry (ed.), ISBN 0844409405

Blackwell Encyclopedia of Writing Systems, Florian Coulmas, ISBN
063121481X

Unicode Standard version 3.0, Unicode Consortium,

The World's Writing Systems, Peter Daniels and William Bright, ISBN 0201616335
0195079930

Writing Systems of the World, Akira Nakanishi, ISBN 0804816549


11. Index

alphabetic -- 4.1
ASCII-compatible encoding (ACE) -- 6
ASN.1 text formats -- 6
Base64 -- 6
Basic Multilingual Plane (BMP) -- 3.2
bidirectional display -- 5
case -- 4
character -- 2
character encoding form -- 2
character encoding scheme -- 2
charset -- 2
charset identification -- 6
CJK characters and Han characters -- 7
code table -- 4
coded character -- 2
coded character set -- 2
combining character -- 4
compatibility character -- 4.1
composite sequence -- 4
control character -- 4.1
diacritic -- 4.1
displaying and rendering text -- 2
font -- 5
formatting character -- 4.1
glyph -- 5
glyph code -- 5
graphic symbol -- 5
i18n, l10n -- 2
ideographic -- 4.1
input methods -- 5
internationalization -- 2
ISO -- 3.1
language -- 2
language identification -- 6
Latin characters -- 7
local and regional standards organizations -- 3.1
locale and region -- 7
localization -- 2
MIME -- 6
multilingual -- 2
name spaces -- 6
nonspacing character -- 4.1
normalization -- 4
on-the-wire encoding -- 6
parsed text -- 6
private use -- 7
protocol elements -- 6
punctuation -- 4.1
quoted printable -- 6
regular expressions -- 7
rendering rules -- 5
romanization -- 7
script -- 2
sorting and collation -- 4
symbol -- 4.1
transcoding -- 2
transfer encoding syntax -- 6
translation -- 7
transliteration -- 7
UCS-2 and UCS-4 -- 3.2
undisplayable character -- 5
Unicode Consortium -- 3.1
UTF-16, UTF-16BE, and UTF-16LE -- 3.2
UTF-8 -- 3.2
World Wide Web Consortium -- 3.1
XML -- 6


A. Acknowledgements

The definitions in this document come from many sources, including a
wide variety of IETF documents.

James Seng contributed to the initial outline of this document. Harald
Alvestrand and Martin Duerst made extensive useful comments on an early
version. Others who contributed to the development include:

Jacob Palme
Susan Harris
Harald Alvestrand
Johan van Wingen
Yuri Demchenko
Peter Constable
Zita Wenzel


B. Changes between versions -01 and -02

Radical restructuring. Scrapped many terms that are not used in the IETF
context. Changed definitions to those from ISO and Unicode. Essentially
an almost complete rewrite.


C. Author Contact Information

Paul Hoffman
Internet Mail Consortium and VPN Consortium
127 Segre Place
Santa Cruz, CA  95060 USA
paul.hoffman@imc.org and paul.hoffman@vpnc.org
----