Cheat Sheet:
Unicode-enabling Microsoft C/C++ Source Code

Contents

Initial Steps
I/O, Database
Stream I/O
BOM Values
Constants & Globals
Data Types
Platform String API
TCHAR String API
References

Here is a cheat sheet for converting Microsoft C or C++ source code to support Unicode. It does not attempt to explain much and presumes you are generally familiar with Microsoft's approach to Unicode. The goal is just to have a single place to look for names, correct spellings, etc. of relevant data types, functions, etc.

Also see: Cheat Sheet: Unicode-enabling Microsoft C/C++ Source Code in Chinese

I18nGuy Home Page XenCraft (Unicode Consulting)

Initial Steps for Unicode-enabling Microsoft C/C++ Source

Top of page

File I/O, Database, Transfer Protocol Considerations

Top of page

Stream I/O

Streams are difficult in Microsoft C++. You may run into 3 types of problems:

  1. Unicode filenames are not supported. The workaround is to use FILE * _wfopen and if needed, use the FILE handle in subsequent stream I/O.
    std::ifstream stm(_wfopen(pFilename, L"r"));
  2. Stream I/O will convert Unicode data from/to native (ANSI) code page on read/write, not UTF-8 or UTF-16. However the stream class can be modified to read/write UTF-8. You can implement a facet to convert between Unicode and UTF-8.
    codecvt <wchar_t, char_traits <wchar_t> >
  3. To read/write UTF-16 with stream I/O, use binary opens and binary I/O. To set binary I/O:
    _setmode( _fileno( stdin ), _O_BINARY );

    Also see the Microsoft run-time library reference: "Unicode Stream I/O in Text and Binary Modes".

Note: There aren't TCHAR equivalents for cout/wcout, cin/wcin, etc. You may want to make your own preprocessor definition for "tout", if you are compiling code both ways.

Top of page

Internationalization, Advanced Unicode, Platform and Other Considerations

Top of page

Unicode BOM Encoding Values

Encoding FormBOM Encoding
UTF-8EF BB BF
UTF-16
(big-endian)
FE FF
UTF-16
(little-endian)
FF FE
UTF-16BE, UTF-32BE
(big-endian)
No BOM!
UTF-16LE, UTF-32LE
(little-endian)
No BOM!
UTF-32
(big-endian)
00 00 FE FF
UTF-32
(little-endian)
FF FE 00 00
SCSU
(compression)
0E FE FF

The Byte Order Marker (BOM) is Unicode character U+FEFF. (It can also represent a Zero Width No-break Space.) The code point U+FFFE is illegal in Unicode, and should never appear in a Unicode character stream. Therefore the BOM can be used in the first character of a file (or more generally a string), as an indicator of endian-ness. With UTF-16, if the first character is read as bytes FE FF then the text has the same endian-ness as the machine reading it. If the character is read as bytes FF FE, then the endian-ness is reversed and all 16-bit words should be byte-swapped as they are read-in. In the same way, the BOM indicates the endian-ness of text encoded with UTF-32.

Note that not all files start with a BOM however. In fact, the Unicode Standard says that text that does not begin with a BOM MUST be interpreted in big-endian form.

The character U+FEFF also serves as an encoding signature for the Unicode Encoding Forms. The table shows the encoding of U+FEFF in each of the Unicode encoding forms. Note that by definition, text labeled as UTF-16BE, UTF-32BE, UTF-32LE or UTF-16LE should not have a BOM. The endian-ness is indicated in the label.

For text that is compressed with the SCSU (Standard Compression Scheme for Unicode) algorithm, there is also a recommended signature.

Top of page

Constant and Global Variables

ANSIWideTCHAR
EOFWEOF_TEOF
_environ_wenviron_tenviron
_pgmptr_wpgmptr_tpgmptr

Data Types

ANSIWideTCHAR
charwchar_t_TCHAR
_finddata_t_wfinddata_t_tfinddata_t
__finddata64_t__wfinddata64_t_tfinddata64_t
_finddatai64_t_wfinddatai64_t_tfinddatai64_t
intwint_t_TINT
signed charwchar_t_TSCHAR
unsigned charwchar_t_TUCHAR
charwchar_t_TXCHAR
 L_T or _TEXT
LPSTR
(char *)
LPWSTR
(wchar_t *)
LPTSTR
(_TCHAR *)
LPCSTR
(const char *)
LPCWSTR
(const wchar_t *)
LPCTSTR
(const _TCHAR *)
LPOLESTR
(For OLE)
LPWSTRLPTSTR
Top of page

Platform SDK String Functions

There are many Windows API that compile into ANSI or Wide forms, depending on whether the symbol UNICODE is defined. Modules that operate on both ANSI and Wide characters, need to be aware of this. Otherwise, using the Character Data Type-independent name requires no changes, just compile with the symbol UNICODE defined.

The following list is by no means all of the Character Data Type-dependent API, just some character and string related ones. Look in WinNLS.h for some code page and locale related API.

ANSIWideCharacter Data Type-
Independent Name
CharLowerACharLowerWCharLower
CharLowerBuffACharLowerBuffWCharLowerBuff
CharNextACharNextWCharNext
CharNextExACharNextExWCharNextEx
CharPrevACharPrevWCharPrev
CharPrevExACharPrevExWCharPrevEx
CharToOemACharToOemWCharToOem
CharToOemBuffACharToOemBuffWCharToOemBuff
CharUpperACharUpperWCharUpper
CharUpperBuffACharUpperBuffWCharUpperBuff
CompareStringACompareStringWCompareString
FoldStringAFoldStringWFoldString
GetStringTypeAGetStringTypeWGetStringType
GetStringTypeExAGetStringTypeExWGetStringTypeEx
IsCharAlphaAIsCharAlphaWIsCharAlpha
IsCharAlphaNumericAIsCharAlphaNumericWIsCharAlphaNumeric
IsCharLowerAIsCharLowerWIsCharLower
IsCharUpperAIsCharUpperWIsCharUpper
LoadStringALoadStringWLoadString
lstrcatAlstrcatWlstrcat
lstrcmpAlstrcmpWlstrcmp
lstrcmpiAlstrcmpiWlstrcmpi
lstrcpyAlstrcpyWlstrcpy
lstrcpynAlstrcpynWlstrcpyn
lstrlenAlstrlenWlstrlen
OemToCharAOemToCharWOemToChar
OemToCharBuffAOemToCharBuffWOemToCharBuff
wsprintfAwsprintfWwsprintf
wvsprintfAwvsprintfWwvsprintf
Top of page

TCHAR String Functions

Functions sorted by ANSI name, for ease of converting to Unicode.

ANSIWideTCHAR
_access_waccess_taccess
_atoi64_wtoi64_tstoi64
_atoi64_wtoi64_ttoi64
_cgets_cgetwscgetts
_chdir_wchdir_tchdir
_chmod_wchmod_tchmod
_cprintf_cwprintf_tcprintf
_cputs_cputws_cputts
_creat_wcreat_tcreat
_cscanf_cwscanf_tcscanf
_ctime64_wctime64_tctime64
_execl_wexecl_texecl
_execle_wexecle_texecle
_execlp_wexeclp_texeclp
_execlpe_wexeclpe_texeclpe
_execv_wexecv_texecv
_execve_wexecve_texecve
_execvp_wexecvp_texecvp
_execvpe_wexecvpe_texecvpe
_fdopen_wfdopen_tfdopen
_fgetchar_fgetwchar_fgettchar
_findfirst_wfindfirst_tfindfirst
_findnext64_wfindnext64_tfindnext64
_findnext_wfindnext_tfindnext
_findnexti64_wfindnexti64_tfindnexti64
_fputchar_fputwchar_fputtchar
_fsopen_wfsopen_tfsopen
_fullpath_wfullpath_tfullpath
_getch_getwch_gettch
_getche_getwche_gettche
_getcwd_wgetcwd_tgetcwd
_getdcwd_wgetdcwd_tgetdcwd
_ltoa_ltow_ltot
_makepath_wmakepath_tmakepath
_mkdir_wmkdir_tmkdir
_mktemp_wmktemp_tmktemp
_open_wopen_topen
_popen_wpopen_tpopen
_putch_putwch_puttch
_putenv_wputenv_tputenv
_rmdir_wrmdir_trmdir
_scprintf_scwprintf_sctprintf
_searchenv_wsearchenv_tsearchenv
_snprintf_snwprintf_sntprintf
_snscanf_snwscanf_sntscanf
_sopen_wsopen_tsopen
_spawnl_wspawnl_tspawnl
_spawnle_wspawnle_tspawnle
_spawnlp_wspawnlp_tspawnlp
_spawnlpe_wspawnlpe_tspawnlpe
_spawnv_wspawnv_tspawnv
_spawnve_wspawnve_tspawnve
_spawnvp_wspawnvp_tspawnvp
_spawnvpe_wspawnvpe_tspawnvpe
_splitpath_wsplitpath_tsplitpath
_stat64_wstat64_tstat64
_stat_wstat_tstat
_stati64_wstati64_tstati64
_strdate_wstrdate_tstrdate
_strdec_wcsdec_tcsdec
_strdup_wcsdup_tcsdup
_stricmp_wcsicmp_tcsicmp
_stricoll_wcsicoll_tcsicoll
_strinc_wcsinc_tcsinc
_strlwr_wcslwr_tcslwr
_strncnt_wcsncnt_tcsnbcnt
_strncnt_wcsncnt_tcsnccnt
_strncnt_wcsncnt_tcsnccnt
_strncoll_wcsncoll_tcsnccoll
_strnextc_wcsnextc_tcsnextc
_strnicmp_wcsnicmp_tcsncicmp
_strnicmp_wcsnicmp_tcsnicmp
_strnicoll_wcsnicoll_tcsncicoll
_strnicoll_wcsnicoll_tcsnicoll
_strninc_wcsninc_tcsninc
_strnset_wcsnset_tcsncset
_strnset_wcsnset_tcsnset
_strrev_wcsrev_tcsrev
_strset_wcsset_tcsset
_strspnp_wcsspnp_tcsspnp
_strtime_wstrtime_tstrtime
_strtoi64_wcstoi64_tcstoi64
_strtoui64_wcstoui64_tcstoui64
_strupr_wcsupr_tcsupr
_tempnam_wtempnam_ttempnam
_ui64toa_ui64tow_ui64tot
_ultoa_ultow_ultot
_ungetch_ungetwch_ungettch
_unlink_wunlink_tunlink
_utime64_wutime64_tutime64
_utime_wutime_tutime
_vscprintf_vscwprintf_vsctprintf
_vsnprintf_vsnwprintf_vsntprintf
asctime_wasctime_tasctime
atof_wtof_tstof
atoi_wtoi_tstoi
atoi_wtoi_ttoi
atol_wtol_tstol
atol_wtol_ttol
character compareMaps to macro or inline function_tccmp
character copyMaps to macro or inline function_tccpy
character lengthMaps to macro or inline function_tclen
ctime_wctime_tctime
fgetcfgetwc_fgettc
fgetsfgetws_fgetts
fopen_wfopen_tfopen
fprintffwprintf_ftprintf
fputcfputwc_fputtc
fputsfputws_fputts
freopen_wfreopen_tfreopen
fscanffwscanf_ftscanf
getcgetwc_gettc
getchargetwchar_gettchar
getenv_wgetenv_tgetenv
getsgetws_getts
isalnumiswalnum_istalnum
isalphaiswalpha_istalpha
isasciiiswascii_istascii
iscntrliswcntrl_istcntrl
isdigitiswdigit_istdigit
isgraphiswgraph_istgraph
islead (Always FALSE)(Always FALSE)_istlead
isleadbyte (Always FALSE)isleadbyte (Always FALSE)_istleadbyte
islegal (Always TRUE)(Always TRUE)_istlegal
isloweriswlower_istlower
isprintiswprint_istprint
ispunctiswpunct_istpunct
isspaceiswspace_istspace
isupperiswupper_istupper
isxdigitiswxdigit_istxdigit
mainwmain_tmain
perror_wperror_tperror
printfwprintf_tprintf
putcputwc_puttc
putcharputwchar_puttchar
puts_putws_putts
remove_wremove_tremove
rename_wrename_trename
scanfwscanf_tscanf
setlocale_wsetlocale_tsetlocale
sprintfswprintf_stprintf
sscanfswscanf_stscanf
strcatwcscat_tcscat
strchrwcschr_tcschr
strcmpwcscmp_tcscmp
strcollwcscoll_tcscoll
strcpywcscpy_tcscpy
strcspnwcscspn_tcscspn
strerror_wcserror_tcserror
strftimewcsftime_tcsftime
strlenwcslen_tcsclen
strlenwcslen_tcslen
strncatwcsncat_tcsncat
strncatwcsncat_tcsnccat
strncmpwcsncmp_tcsnccmp
strncmpwcsncmp_tcsncmp
strncpywcsncpy_tcsnccpy
strncpywcsncpy_tcsncpy
strpbrkwcspbrk_tcspbrk
strrchrwcsrchr_tcsrchr
strspnwcsspn_tcsspn
strstrwcsstr_tcsstr
strtodwcstod_tcstod
strtokwcstok_tcstok
strtolwcstol_tcstol
strtoulwcstoul_tcstoul
strxfrmwcsxfrm_tcsxfrm
system_wsystem_tsystem
tmpnam_wtmpnam_ttmpnam
tolowertowlower_totlower
touppertowupper_totupper
ungetcungetwc_ungettc
vfprintfvfwprintf_vftprintf
vprintfvwprintf_vtprintf
vsprintfvswprintf_vstprintf
WinMainwWinMain_tWinMain
Top of page

References