Supplementary Character Test Data

Following is the data you can use for testing. You can copy/paste the characters into a UTF-8 editor to insert into any of your test files.

The following information is provided:

• An image of the characters rendered on my machine.
• The string of characters implemented as 4-byte UTF-8.
• The string of characters implemented as Numeric Character References. They display as characters and copy/paste will provide the UTF-8 equivalents. However, if needed you have the NCR values in the page source.
• A table of the supplementary characters, one character per row, showing the Unicode Scalar Value, the character rendered from an NCR and from UTF-8 bytes. Th purpose of the table is to simplify debug and an analysis in the event one or more of the characters display incorrectly.
• A Background section explaining how the characters were chosen and why testing supplementary characters is a requirement.

Image of Characters on My Machine

4-Byte UTF-8

Numeric Character References (NCR)

The characters display properly on Windows 7 with either IE 8 or Firefox 3.6. I have the Chinese language pack installed, as well as a number of other fonts, so my system is not representative. If you have problems displaying the supplementary characters, let me know and I will provide additional information about fonts needed, etc.

Background

Supplementary Characters

In the Unicode Character Standard, Supplementary Characters are the characters assigned code points from U+10000 to U+10FFFF. In other words, these are the Unicode characters greater than U+FFFF.

• In UTF-8 these characters are each 4 bytes long.
• In UTF-16 these characters require 2 surrogates (16-bit units).

Supplementary Character Use Requirements

There are a number of characters in the Supplementary Planes that are frequently used in Asian markets. Therefore the Supplementary Planes are required to be supported. To identify useful test characters, I selected a set from IICORE.

IICORE

In 2005, the IRG (Ideographic Rapporteur Group) identified a set ideographs, called the Ideographic International Core (IICore). The 10,000 ideographs in the IICore are the most frequently used characters that would cover the vast majority of modern texts in all locales where ideographs are used. This collection is intended for use in devices with limited resources, such as mobile phones.

Test Characters

To have characters that are good for testing software support for the Supplementary Plane, I extracted the 62 characters from the IICORE that are in the Supplementary Plane. These characters have the properties that:

• Being in IICORE they are used frequently enough to be a minimum requirement for software supporting ideographs
• They are in the Supplementary Plane and will test support for code points above U+FFFF
• They are not "oddball" values. If using them uncovers a problem, fixing the problem is inherently justified.
• As these are among the most frequently used characters, it is likely that other software will support them as well. In particular, operating system, fonts, browsers, development platforms, etc. Otherwise, implementation and testing might be difficult.

Requirement for Testing Supplementary Characters

It is important to test software with characters from the Supplementary Plane.

• These characters use the maximum amount of memory (4 bytes in either UTF-8 or UTF-16). They are the most likely to trigger memory overruns if the software is not allocating the right amount of memory per character
• There are still many UTF-16 (or UCS-2) software components that do not support the surrogates required for Supplementary Plane characters.
• There are many UTF-8 software components that do not support 4 byte UTF-8 characters. For example, some legacy software presumes that the maximum byte length is 3 bytes for UTF-8.
• Some programming languages, such as Java, require additional or special coding to support surrogates.
• Some databases do not support 4-byte UTF-8 characters or UTF-16 surrogates. They may reject the characters, not allocate sufficient space in records, or corrupt the characters. Sorting may be incorrect.
• Some software may perform an incorrect conversion between UTF-8 and UTF-16. Instead of converting the 2 UTF-16 surrogates to 4 UTF-8 bytes, it incorrectly converts each surrogate to a 3 byte character.

Testing with supplementary characters can detect if there is code that does not provide the necessary support.

References

• Configure Microsoft Windows to support Unicode supplementary characters
• IICore 22 Mapping Table
• IICore Version 2.2
• ISO/IEC JTC1/SC2/WG2/IRG Ideographic Rapporteur Group
• Demonstrating Unicode Plane 1 (Supplementary) Characters Encoded in UTF-8
• Conversion Table: Surrogates to Scalar Value/UTF-32

Supplementary Characters

Top of page