Defining how a sequence of bytes sits in a memory buffer or on disk can be challenging from time to time. Since everything that you’ll work with is a byte, it makes sense that we have an intuitive way to work with this information agnostic of the overlying type restrictions that the language will enforce on us.

In today’s post, I’m going to run through Python’s byte string packing and unpacking using the struct package.

Basics

From the Python documentation:

This module performs conversions between Python values and C structs represented as Python bytes objects. This can be used in handling binary data stored in files or from network connections, among other sources. It uses Format Strings as compact descriptions of the layout of the C structs and the intended conversion to/from Python values.

When working with a byte string in Python, you prefix your literals with b.

>>> b'Hello'
'Hello'

The ord function call is used to convert a text character into its character code representation.

>>> ord(b'H')
72
>>> ord(b'e')
101
>>> ord(b'l')
108

We can use list to convert a whole string of byte literals into an array.

>>> list(b'Hello')
[72, 101, 108, 108, 111]

The compliment to the ord call is chr, which converts the byte-value back into a character.

Packing

Using the struct module, we’re offered the pack function call. This function takes in a format of data and then the data itself. The first parameter defines how the data supplied in the second parameter should be laid out. We get started:

>>> import struct

If we pack the string 'Hello' as single bytes:

>>> list(b'Hello')
[72, 101, 108, 108, 111]
>>> struct.pack(b'BBBBB', 72, 101, 108, 108, 111)
b'Hello'

The format string b'BBBBB' tells pack to pack the values supplied into a string of 5 unsigned values. If we were to use a lower case b in our format string, pack would expect the byte value to be signed.

>>> struct.pack(b'bbbbb', 72, 101, 108, 108, 111)
b'Hello'

This only gets interesting once we send a value that would make the request overflow:

>>> struct.pack(b'bbbbb', 72, 101, 108, 129, 111)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
struct.error: byte format requires -128 <= number <= 127

The following tables have been re-produced from the Python documentation.

Byte order, size and alignment

Character Byte order Size Alignment
@ native native native
= native standard none
< little-endian standard none
> big-endian standard none
! network (= big-endian) standard none

Types

Format C Type Python type Standard size Notes
x pad byte no value    
c char bytes of length 1 1  
b signed char integer 1 (1),(3)
B unsigned char integer 1 (3)
? _Bool bool 1 (1)
h short integer 2 (3)
H unsigned short integer 2 (3)
i int integer 4 (3)
I unsigned int integer 4 (3)
l long integer 4 (3)
L unsigned long integer 4 (3)
q long long integer 8 (2), (3)
Q unsigned long long integer 8 (2), (3)
n ssize_t integer   (4)
N size_t integer   (4)
f float float 4 (5)
d double float 8 (5)
s char[] bytes    
p char[] bytes    
P void * integer   (6)

Unpacking

The direct reverse process of packing bytes into an array, is unpacking them again into usable variables inside of your python code.

>>> struct.unpack(b'BBBBB', struct.pack(b'BBBBB', 72, 101, 108, 108, 111))
(72, 101, 108, 108, 111)
>>> struct.unpack(b'5s', struct.pack(b'BBBBB', 72, 101, 108, 108, 111))
(b'Hello',)

Packing data with Python的更多相关文章

  1. 使用Python对Twitter进行数据挖掘(Mining Twitter Data with Python)

    目录 1.Collecting data 1.1 Register Your App 1.2 Accessing the Data 1.3 Streaming 2.Text Pre-processin ...

  2. python data analysis | python数据预处理(基于scikit-learn模块)

    原文:http://www.jianshu.com/p/94516a58314d Dataset transformations| 数据转换 Combining estimators|组合学习器 Fe ...

  3. Mining Twitter Data with Python

    目录 1.Collecting data 1.1 Register Your App 1.2 Accessing the Data 1.3 Streaming 2.Text Pre-processin ...

  4. Working with Binary Data in Python

    http://www.devdungeon.com/content/working-binary-data-python

  5. 7 Tools for Data Visualization in R, Python, and Julia

    7 Tools for Data Visualization in R, Python, and Julia Last week, some examples of creating visualiz ...

  6. 一句Python,一句R︱pandas模块——高级版data.frame

    先学了R,最近刚刚上手python,所以想着将python和R结合起来互相对比来更好理解python.最好就是一句python,对应写一句R. pandas可谓如雷贯耳,数据处理神器. 以下符号: = ...

  7. Python - 2. Built-in Collection Data Types

    From: http://interactivepython.org/courselib/static/pythonds/Introduction/GettingStartedwithData.htm ...

  8. A Complete Tutorial to Learn Data Science with Python from Scratch

    A Complete Tutorial to Learn Data Science with Python from Scratch Introduction It happened few year ...

  9. python接口测试(post,get)-传参(data和json之间的区别)

    python接口测试如何正确传参: POST 传data:data是python字典格式:传参data=json.dumps(data)是字符串类型传参 #!/usr/bin/env python3 ...

随机推荐

  1. “/”应用程序中的服务器错误。||分析器错误消息: 未能加载类型“WebApplication1._Default”

    环境VS2008 无法运行WEB项目,Winfrom程序OK. 新创建的WEB项目直接运行报下图错误. 尝试多种方法: 1,重新生成项目,运行.(失败) 2,重装VS2008(默认.完全.自定义)安装 ...

  2. JavaWeb开发中的分层思想(一)

    JavaWeb开发分层思想(一) 一.认识DAO.Service.Controller层 DAO(Data Access Object) 1.直接看英文意思就是"数据访问对象",也 ...

  3. python3 list合并

    1 t1=[x for x in range(5)] 2 t2=[x for x in range(5,10)] 3 4 #way1:通过方法extend(),直接修改列表,无返回值 5 # t1.e ...

  4. [源码解析] 分布式任务队列 Celery 之启动 Consumer

    [源码解析] 分布式任务队列 Celery 之启动 Consumer 目录 [源码解析] 分布式任务队列 Celery 之启动 Consumer 0x00 摘要 0x01 综述 1.1 kombu.c ...

  5. 别人 echo 、你也 echo ,是问 echo 知多少?-- Shell十三问<第三问>

    别人 echo .你也 echo ,是问 echo 知多少?-- Shell十三问<第三问> 承接上一章所介绍的 command line ,这里我们用 echo 这个命令加以进一步说明. ...

  6. java面试-生产环境服务器变慢,谈谈你的诊断思路

    1.uptime:查询linux系统负载 11:16:16 系统当前时间 up 64 days, 19:23 从上次启动开始系统运行的时间3 users 连接数量,同一用户多个连接的时候算多个load ...

  7. 【C/C++】面相对象开发之封装

    封装继承多态是面向对象程序开发的基础概念.是实现面向对象的基本要素. 封装 程序开发,最核心价值,是数据. 程序其实是读取数据,操作数据,保存数据等一系列操作. 那么经过良好组织过的数据,将使编程事半 ...

  8. OO_Unit2总结

    OO_Unit2总结 (1) 多线程协同控制设计策略 总体信号通讯策略 本单元作业,我采用的是生产者-消费者模式加类观察者模式. 通过分析指导书给出的需求,我将最终我要实现的程序简化为了"输 ...

  9. PAT B1033 旧键盘上的几个键又毁坏了,于是在输入一段文字时,对应得的字符就不会出现。

    题中可用的字母包括大小写(给出的坏键的字母,全为大写),数字,"_" "," "." "-" "+" ...

  10. 统计学习方法——实现AdaBoost

    Adaboost 适用问题:二分类问题 模型:加法模型 \[f(x)=\sum_{m=1}^{M} \alpha_{m} G_{m}(x) \] 策略:损失函数为指数函数 \[L(y,f(x))=ex ...