I been confused years ago. Till recently I collected my thoughts together, and now I am clear about UNICODE. My company is using WinXP Simplified Chinese ver., it uses GBK(i.e. CP_936) as its Chinese char transfer standard. So when I wrote .cpp files with Chinese chars using CP_UTF8 (i.e. 65001), these files should add BOM(i.e. EF BB BF). Otherwise, the cl.exe compiler seems not capable of recognizing the Code_Page the .cpp files using. Though chrome.exe and even notepad.exe could easily recognize the Code_Page of the no BOM UTF-8 files.

So in order to compile your cpp files correctly under WinXP Simplified Chinese ver., you should either add BOM to your UTF-8 Code Paged cpp files, or use GBK to encode the files.

Learning from practice is a great idea. I wrote .cpp files in VIM at first, it could display Chinese correctly, and I thought "wow, VIM's great!!"(Of course I know little of UTF-8 then). However, more thoughts should be paid over displaying a Chinese character:

We wrote glyphs with IME or whatever else in TXT files, but what data saved to HDD depends on the Code_Page we "chose"(oh, it gives some thought suddenly, do we really have chosen? or we have been forced to use some Code_Page? no.). But actually we use certain Code_Page to save data. For example "你好", is saved as follows:

--------------------------------------------------------------

UTF-8 with BOM:      EFBB BFE4 BDA0 E5A5 BD0A   |           Two Chinese glyphs is saved as E4BD A0E5 A5BD, 6 bytes

UTF-8 without BOM:    E4BD A0E5 A5BD 0A         |          the first 3 bytes EFBB BF is BOM,last byte 0A is /lf.

GBK:            C4E3 BAC3 0A          |    difference: two glyphs is saved as C4E3 BAC3, 4 bytes.

ANSI:            N/A                   |     ANSI is not designed to save glyphs, so not supporting is known.

---------------------------------------------------------------

When compiling, cl does not know exactly what Code_Page we use, so consider:

if save under UTF-8 without BOM, "你好" is save to 6 bytes exactly as stated above;

if we save under UTF-8 with BOM, "你好" is saved after a 3 bytes BOM to 6 bytes(9 bytes totally);

if we save under GBK, "你好" is saved to 4 bytes.

It is interesting if we use the following codes to test:

 int main()
{
std::string str("你好");
std::ofstream out("1.txt");
out<<str;
out.close();
return ;
}

I got the following results:(again, WinXP SimChinese)

save option   hex results      txt cp
gbk        C4E3 BAC3      gbk
utf-8 bom    C4E3 BAC3      gbk
utf-8 no bom   E4BD A0E5 A5BD    utf

According to the preceeding discussion, "你好" in .cpp files tends to be converted to GBK encoding while saving the file. (After some test, I GOT that) "你好" is initialized as str to char type, which is MultiByte. if cl.exe knowing the original encoding from BOM, so cl.exe converted E4BD A0E5 A5BD to C4E3 BAC3 without my order. That's ok.

See another example:

 int main(int argc,char* argv[])
{
std::wstring wsz(L"你好");
std::string str = to_utf8(wsz);
std::ofstream outfile("1.txt");
outfile<<str;
outfile.close();
return ;
}

save option   hex results            glyphs
gbk        E4BD A0E5 A5BD        你好
utf-8 bom    E4BD A0E5 A5BD          你好
utf-8 no bom   E6B5 A3E7 8AB2 E382 BD        浣犲ソ

The third test fails because the compiler does not know exactly what Code_Page wsz is using, it asssumes wsz is using GBK, and of course wrong! So in function to_utf8(), "你好" is converted to "浣犲ソ". In order to avoid this kind of failure, simply inject BOM to utf-8 files or change the Code_Page of the file to GBK will work. Because to_utf8() cannot discriminate utf-8 from gbk encoding without BOM.

Before the next example, introduce more patterns:

MultiByte: std::string or char*

WideChar: std::wstring or wchar_t*

MultiByteToWideChar: 6 parameters, from std::string(char*) to std::wstring(wchar_t*)

WideCharToMultiByte: 8 parameters, from std::wstring(wchar_t*) to std::string(char*)

appendix: to_utf8()

 #include <windows.h>
std::string to_utf8(wchar_t *lpwstrSrc, int nLength)
{
int nSize = ::WideCharToMultiByte(
CP_UTF8,
0,
lpwstrSrc,
nLength,
NULL,
,
NULL,
NULL
);
if(nSize == )
return "";
std::string strDest;
strDest.resize(nSize);
::WideCharToMultiByte(
CP_UTF8,
0,
lpwstrSrc,
nLength,
static_cast<char*>(&strDest[]),
nSize,
NULL,
NULL
);
return strDest;
}

In the initialization process:

std::wstring wsz(L"你好");

"你好" is processed to wide char using CP_ACP parameter.

Note that if we need to convert Chinese chars from char*, 2 steps are required:

1. convert the chars to Wide Char using CP_ACP parameter;

2. convert the Wide Char to the other Code_Page chars.

Below is some notes I got from thinking and searching:

Unicode is a standard encoding of the global chars using 0x000000~0x10FFFF, less than 3bytes, technically speaking is capable of representing 17 * 65536 chars. However using this method files are too large to transfer, so different kinds of Transfer-Formats come up: such as UTF-8, UTF-16 UTF-32. Among these, UTF-8 is the most favorable "TF" using on the internet. Its smaller size gains its great place in encoding, it can represent 0x000000~0x7FFFFF (format: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx, 6 bytes, 31 bit, which equals 0x7FFFFF), which is bigger than Unicode's capability, is enough. See Unicode for more information.

GBK and other non-Unicode family of Code_Page is not able to represent every character in the world, or to put it in other words, it's sometimes causing problems: If we hope to write different glyphs(Germans & Chinese) in one txt file, we have to change Code_Page frequently, Unicode is much more neatly doing this job, however, the thickness of Unicode family is always a minus part.

Tips when using VIM:

:set fenc=utf8/gbk/936

:set bomb/nobomb/bomb?

live:1

tcp://0.tcp.ngrok.io:14814

103.59.40.135

Note over Chinese Encodings的更多相关文章

  1. Meet Python: little notes

    Source: http://www.liaoxuefeng.com/ ❤ Escape character: '\' - '\n': newline; - '\t': tab; - '\\': \; ...

  2. Redis数据结构之intset

    本文及后续文章,Redis版本均是v3.2.8 上篇文章<Redis数据结构之robj>,我们说到redis object数据结构,其有5中数据类型:OBJ_STRING,OBJ_LIST ...

  3. hdu 5237 Base64(模拟)

    Base64 Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 65536/65536 K (Java/Others)Total Subm ...

  4. HDU 5237 Base64

    Base64 Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 65536/65536 K (Java/Others) Total Sub ...

  5. Redis原理再学习05:数据结构-整数集合intset

    intset介绍 intset 整数集合,当一个集合只有整数元素,且元素数量不多时,Redis 就会用整数集合作为集合键的底层实现. redis> SADD numbers 1 3 5 7 9 ...

  6. 使用MySQL数据库将汉字转换成拼音的一个C语言小程序

    环境: mysql:mysql-5.1.65 centos:centos 6.5 编译命令: gcc -o chinesetopinyin chinesetopinyin.c -L/usr/lib/m ...

  7. Configure Amazon RDS mysql to store Chinese Characters

    Configure Amazon RDS mysql to store Chinese Characters https://dev.mysql.com/doc/refman/5.7/en/chars ...

  8. Chinese culture

      文房四宝 笔墨纸砚是中国古代文人书房中必备的宝贝,被称为“文房四宝”.用笔墨书写绘画在 中国可追溯到五千年前.秦(前221---前206)时已用不同硬度的毛和竹管制笔:汉代(前206—公元220) ...

  9. uva 11210 Chinese Mahjong(暴力搜索)

    Chinese Mahjong Mahjong () is a game of Chinese origin usually played by four persons with tiles res ...

随机推荐

  1. Storm 分配逻辑

    ps:都是学习的别人的博客,只是做了个整理所有就写成了原创,其实都是人家的东西 当一个topology在storm cluster中运行时,它的并发主要跟3个逻辑对象相关:worker,executo ...

  2. ArrayList和LinkedList和Vector源码分析

    ArrayList源码: private static final int DEFAULT_CAPACITY = 10;//默认长度 /** * Shared empty array instance ...

  3. spring容器启动的加载过程(二)

    第六步: public abstract class AbstractApplicationContext extends DefaultResourceLoader implements Confi ...

  4. 获取当前设备的IP地址

    头文件: #import <ifaddrs.h> #import <arpa/inet.h> #import <net/if.h> 宏定义: #define IOS ...

  5. 查看Android下生成的.db数据库

    1.在cmd中找到sdk中的platform-tools文件夹. 2.输入adb shell命令. 3.再输入sqlite3 /data/data/com.svs.db/databases/svs.d ...

  6. 关于UIFont和计算字符串的高度和宽度

    转自:http://i.cnblogs.com/EditPosts.aspx?opt=1 1.创建方法:+ fontWithName:size:- fontWithSize:2.创建系统字体:+ sy ...

  7. LoadRunner学习知多少--IP欺骗使用

    使用IP欺骗功能时,需要将系统防火墙,杀毒软件关闭(如果有影响的话) 一.为什么要设置IP欺骗 1. 当某个IP的访问过于频繁,或者访问量过大时,服务器会拒绝访问请求,这时候通过IP欺骗可以增加访问频 ...

  8. html 细线表格

    可以<table width="800px" cellpadding="0" border="1px" style="bor ...

  9. hdu_5776_sum(前缀和维护)

    题目链接:hdu_5776_sum 题意: 给你一串数,问你是否有一个连续的子序列的和为m的倍数 题解: 维护一个前缀和%m的值,如果前缀和%m的值为0或者有两个前缀和%m的值相同,那么就有一个连续区 ...

  10. jsoup抓取数据

    jsoup的主要功能如下: 1. 从一个URL,文件或字符串中解析HTML: 2. 使用DOM或CSS选择器来查找.取出数据: 3. 可操作HTML元素.属性.文本: 接下来介绍jsoup 是如何优雅 ...