python标准库：collections和heapq模块

http://blog.csdn.net/pipisorry/article/details/46947833

python额外的数据类型。collections模块和heapq模块的主要内容。

集合库collection

collections模块介绍

Python拥有一些内置的数据类型，比如str, int, list, tuple, dict等， collections模块在这些内置数据类型的基础上，提供了几个额外的数据类型：
1.namedtuple(): 生成可以使用名字来访问元素内容的tuple子类
2.deque: 双端队列，可以快速的从另外一侧追加和推出对象
3.Counter: 计数器，主要用来计数
4.OrderedDict: 有序字典
5.defaultdict: 带有默认值的字典

Named tuples元素命名的元组(collections.namedtuple)

>>> Point = collections.namedtuple('Point', ['x', 'y'])
>>> p = Point(x=1.0, y=2.0)
>>> p
Point(x=1.0, y=2.0)
>>> p.x
1.0
>>> p.y
2.0

namedtuple主要用来产生可以使用名称来访问元素的数据对象，通常用来增强代码的可读性， 在访问一些tuple类型的数据时尤其好用。

举个例子

比如我们用户拥有一个这样的数据结构，每一个对象是拥有三个元素的tuple。使用namedtuple方法就可以方便的通过tuple来生成可读性更高也更好用的数据结构。
"""
from collections import namedtuple
websites = [
('Sohu', 'http://www.google.com/', u'张朝阳'),
('Sina', 'http://www.sina.com.cn/', u'王志东'),
('163', 'http://www.jbxue.com/', u'丁磊')
]
Website = namedtuple('Website', ['name', 'url', 'founder'])
for website in websites:
website = Website._make(website)
print website
# Result:
Website(name='Sohu', url='http://www.google.com/', founder=u'\u5f20\u671d\u9633')
Website(name='Sina', url='http://www.sina.com.cn/', founder=u'\u738b\u5fd7\u4e1c')

Website(name='163', url='http://www.jbxue.com/', founder=u'\u4e01\u78ca')

继承命名tuples

>>> class Point(collections.namedtuple("PointBase", ["x", "y"])):

... __slots__ = ()

... def __add__(self, other):

... return Point(x=self.x + other.x, y=self.y + other.y)

...

>>> p = Point(x=4.0, y=2.0)

>>> q = Point(x=2.0, y=3.0)

>>> p + q

Point(x=6.0, y=5.0)

deque双端队列

deque其实是 double-ended queue 的缩写，翻译过来就是双端队列，它最大的好处就是实现了从队列 头部快速增加和取出对象: .popleft(), .appendleft() 。
你可能会说，原生的list也可以从头部添加和取出对象啊？就像这样：
l.insert(0, v)
l.pop(0)

list对象的这两种用法的时间复杂度是 O(n) ，也就是说随着元素数量的增加耗时呈 线性上升。而使用deque对象则是 O(1) 的复杂度，所以当你的代码有这样的需求的时候， 一定要记得使用deque。
作为一个双端队列，deque还提供了一些其他的好用方法，比如 rotate 等。
举个栗子
使用deque的rotate方法来实现了一个无限循环的加载动画
import sys
import time
from collections import deque
fancy_loading = deque('>--------------------')
while True:
print '\r%s' % ''.join(fancy_loading),
fancy_loading.rotate(1)
sys.stdout.flush()
time.sleep(0.08)
# Result:
# 一个无尽循环的跑马灯
------------->-------

双向队列操作

>>> Q =collections.deque()

>>> Q.append(1)

>>> Q.appendleft(2)

>>> Q.extend([3,4])

>>> Q.extendleft([5,6])

>>> Q

deque([6,5,2,1,3,4])

>>> Q.pop()

4

>>> Q.popleft()

6

>>> Q

deque([5,2,1,3])

>>> Q.rotate(3)

>>> Q

deque([2,1,3,5])

>>> Q.rotate(-3)

>>> Q

deque([5,2,1,3])
限制长度的双向队列


>>> last_three = collections.deque(maxlen=3)

>>> foriinxrange(10):

... last_three.append(i)

... print", ".join(str(x)forxinlast_three)

...

0

0,1

0,1,2

1,2,3

2,3,4

皮皮Blog

Counter计数器 - Multisets运算

官方解释

It is kind of redundant, in that Collections already contains a class for doing this, called Counter. In this case, I need to first extract the second item from eachtuple, for which I can use a generator expression.

from collections import Counter order_count = Counter(menu_item for name, menu_item in order) print order_count   # output # Counter({ # 'Beer': 2, # 'Steak': 2, # 'Wine': 1, # 'Veggie Burger': 1 # })

view rawcounter.py hosted with ❤ by GitHub

Another, maybe better, example might be counting all the different lines that appear in a file. It becomes very simple.

with open('/some/file', 'r') as f: line_count = Counter(f)

举栗子

统计所有字符出现次数
from collections import Counter
s = '''A Counter is a dict subclass for counting hashable objects. It is an unordered collection where elements are stored as dictionary keys and their counts are stored as dictionary values. Counts are allowed to be any integer value including zero or negative counts. The Counter class is similar to bags or multisets in other languages.'''.lower()
c = Counter(s)
# 获取出现频率最高的5个字符
print c.most_common(5)
# Result:
[(' ', 54), ('e', 32), ('s', 25), ('a', 24), ('t', 24)]

Multisets运算

>>> A =collections.Counter([1,2,2])

>>> B =collections.Counter([2,2,3])

>>> A

Counter({2:2,1:1})

>>> B

Counter({2:2,3:1})

>>> A | B

Counter({2:2,1:1,3:1})

>>> A & B

Counter({2:2})

>>> A +B

Counter({2:4,1:1,3:1})

>>> A -B

Counter({1:1})

>>> B -A

Counter({3:1})
列表中出现最多的元素


>>> A =collections.Counter([1,1,2,2,3,3,3,3,4,5,6,7])

>>> A

Counter({3:4,1:2,2:2,4:1,5:1,6:1,7:1})

>>> A.most_common(1)

[(3,4)]

>>> A.most_common(3)

[(3,4), (1,2), (2,2)]

OrderedDict有序字典

在Python中，dict这个数据结构由于hash的特性，是无序的，这在有的时候会给我们带来一些麻烦， 幸运的是，collections模块为我们提供了OrderedDict，当你要获得一个有序的字典对象时，用它就对了。

>>> m = dict((str(x), x) for x in range(10))
>>> print ', '.join(m.keys())
1, 0, 3, 2, 5, 4, 7, 6, 9, 8
>>> m = collections.OrderedDict((str(x), x) for x in range(10))
>>> print ', '.join(m.keys())
0, 1, 2, 3, 4, 5, 6, 7, 8, 9
>>> m = collections.OrderedDict((str(x), x) for x in range(10, 0, -1))
>>> print ', '.join(m.keys())
10, 9, 8, 7, 6, 5, 4, 3, 2, 1

举个栗子
from collections import OrderedDict
items = (
('A', 1),
('B', 2),
('C', 3)
)
regular_dict = dict(items)
ordered_dict = OrderedDict(items)
print 'Regular Dict:'
for k, v in regular_dict.items():
print k, v
print 'Ordered Dict:'
for k, v in ordered_dict.items():
print k, v
# Result:
Regular Dict:
A 1
C 3
B 2
Ordered Dict:
A 1
B 2
C 3

Note:直接输出的OrderedDict形式如下：print(OrderedDict(reguale_slist))

OrderedDict([('GET', 'http://www.jobbole.com/login/?redirect=http://www.jobbole.com/ HTTP/1.1'), ('Host', 'www.jobbole.com'), ('Connection', 'keep-alive'),  ('Cookie', 'wordpress_test_cookie=WP+Cookie+check')...])

而不是dict的形式:print(dict(reguale_slist))

{'Referer': 'http://www.jobbole.com/', 'Host': 'www.jobbole.com', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate, sdch', ...}

defaultdict默认字典

我们都知道，在使用Python原生的数据结构dict的时候，如果用 d[key] 这样的方式访问， 当指定的key不存在时，是会抛出KeyError异常的。
但是，如果使用defaultdict，只要你传入一个默认的工厂方法，那么请求一个不存在的key时， 便会调用这个工厂方法使用其结果来作为这个key的默认值。

defaultdict字典元素类型初始化

collections.defaultdict([default_factory[,...]])

default_factory指定字典的value类型

from collections import defaultdict
>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...     d[k].append(v)...
>>> d.items()

[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

如果给default_dict传入int，则可以用来计数：
>>> s = 'mississippi'
>>> d = defaultdict(int)
>>> for k in s:
...     d[k] += 1
>>> d.items()

[('i', 4), ('p', 2), ('s', 4), ('m', 1)]

Note:defaultdict()参数设置成int，那么就设置字典值的初始值为0了

嵌套defaultdict

from collections import defaultdict
u_lt_ts_dict = defaultdict(lambda: defaultdict(list))

但是这个对象是不可序列化的，因为有lamba函数。也可以这样

from collections import defaultdict
from functools import partial
u_lt_ts_dict = defaultdict(partial(defaultdict, list))

这样就可以pickle了。

[Multiple levels of 'collection.defaultdict' in Python]

Using default dictionaries to represent simple trees

>>> import json
>>> tree = lambda: collections.defaultdict(tree)
>>> root = tree()
>>> root['menu']['id'] = 'file'
>>> root['menu']['value'] = 'File'
>>> root['menu']['menuitems']['new']['value'] = 'New'
>>> root['menu']['menuitems']['new']['onclick'] = 'new();'
>>> root['menu']['menuitems']['open']['value'] = 'Open'
>>> root['menu']['menuitems']['open']['onclick'] = 'open();'
>>> root['menu']['menuitems']['close']['value'] = 'Close'
>>> root['menu']['menuitems']['close']['onclick'] = 'close();'
>>> print json.dumps(root, sort_keys=True, indent=4, separators=(',', ': '))
{
    "menu": {
        "id": "file",
        "menuitems": {
            "close": {
                "onclick": "close();",
                "value": "Close"
            },
            "new": {
                "onclick": "new();",
                "value": "New"
            },
            "open": {
                "onclick": "open();",
                "value": "Open"
            }
        },
        "value": "File"
    }
}

(See https://gist.github.com/hrldcpr/2012250 for more on this.)

[一行Python代码实现树结构]

Mapping objects to unique counting numbers (collections.defaultdict)

>>> import itertools, collections
>>> value_to_numeric_map = collections.defaultdict(itertools.count().next)
>>> value_to_numeric_map['a']
0
>>> value_to_numeric_map['b']
1
>>> value_to_numeric_map['c']
2
>>> value_to_numeric_map['a']
0

举个栗子
from collections import defaultdict
members = [
# Age, name
['male', 'John'],
['male', 'Jack'],
['female', 'Lily'],
['male', 'Pony'],
['female', 'Lucy'],
]
result = defaultdict(list)
for sex, name in members:
result[sex].append(name)
print result
# Result:
defaultdict(<type 'list'>, {'male': ['John', 'Jack', 'Pony'], 'female': ['Lily', 'Lucy']})

[Python collections模块实例]

常用操作

>>> m =dict()

>>> m["a"]

Traceback (most recent call last):

File"<stdin>", line1,in<module>

KeyError: "a"

>>>

>>> m =collections.defaultdict(int)

>>> m["a"]

0

>>> m["b"]

0

>>> m =collections.defaultdict(str)

>>> m["a"]

""

>>> m["b"]+="a"

>>> m["b"]

"a"

>>> m =collections.defaultdict(lambda:"[default value]")

>>> m["a"]

"[default value]"

>>> m["b"]

"[default value]"
使用defaultdict代表tree


>>> importjson

>>> tree = lambda: collections.defaultdict(tree)

>>> root = tree()

>>> root["menu"]["id"]="file"

>>> root["menu"]["value"]="File"

>>> root["menu"]["menuitems"]["new"]["value"]="New"

>>> root["menu"]["menuitems"]["new"]["onclick"]="new();"

>>> root["menu"]["menuitems"]["open"]["value"]="Open"

>>> root["menu"]["menuitems"]["open"]["onclick"]="open();"

>>> root["menu"]["menuitems"]["close"]["value"]="Close"

>>> root["menu"]["menuitems"]["close"]["onclick"]="close();"

>>> printjson.dumps(root, sort_keys=True, indent=4, separators=(",",": "))

{

"menu": {

"id":"file",

"menuitems": {

"close": {

"onclick":"close();",

"value":"Close"

},

"new": {

"onclick":"new();",

"value":"New"

},

"open": {

"onclick":"open();",

"value":"Open"

}

},

"value":"File"

}

}

[查看更多]
映射对象到唯一的计数数字


>>> importitertools, collections

>>> value_to_numeric_map=collections.defaultdict(itertools.count().next)

>>> value_to_numeric_map["a"]

0

>>> value_to_numeric_map["b"]

1

>>> value_to_numeric_map["c"]

2

>>> value_to_numeric_map["a"]

0

>>> value_to_numeric_map["b"]

1
[collections — Container datatypes]
[collections — Container datatypes]
皮皮blog



heapq — Heap queue algorithm

最大 & 最小元素(heapq.nlargest and heapq.nsmallest)

>>> a =[random.randint(0,100)for__inxrange(100)]

>>> heapq.nsmallest(5, a)

[3,3,5,6,8]

>>> heapq.nlargest(5, a)

[100,100,99,98,98]
[其具体实现算法可以参考：编程之美之2.5 寻找最大的K个数]
[A fast way to find the largest N elements in an numpy array]
[How to get indices of N maximum values in a numpy array?]
[heapq — Heap queue algorithm]
皮皮blog

from:http://blog.csdn.net/pipisorry/article/details/46947833

ref: [Data Types]