Python之路Day9

摘要：

1. 协程
2. Select\Poll\Epoll异步IO与事件驱动
3. Python连接MySQL数据库操作
4. RabbitMQ队列
5. Redis\Memcached缓存
6. Paramiko
7. Twsited网络框架

1. 协程：

线程和进程的操作是由程序触发系统接口，最后的执行者是系统；而协程的操作者则是程序员。

协程存在的意义：对于多线程应用，CPU通过切片的方式来切换线程间的执行，线程切换时需要耗时（保存状态，下次继续）。协程则只使用一个线程，在一个线程中规定某个代码块执行顺序。

协程的适用场景：当程序中存在大量不需要CPU的操作时（IO），适用于协程。

from greenlet import greenlet

def test1():
    print(12)
    gr2.switch()
    print(34)
    gr2.switch()

def test2():
    print(56)
    gr1.switch()
    print(78)

gr1 = greenlet(test1)
gr2 = greenlet(test2)
gr1.switch()

output==>

12
56
34
78

import gevent

def foo():
    print("\033[31;1m fooooooooooooooooooooo\033[0m")
    gevent.sleep(1)
    print("\033[32;1m back to foo\033[0m")

def bar():
    print("\033[33;1m barrrrrrrrrrrrrrrr\033[0m")
    gevent.sleep(1)
    print("\033[36;1m back to bar\033[0m")

def exx():
    print("\033[37;1m exxxxxxxxxxxxxxxxxx\033[0m")
    gevent.sleep(1)
    print("\033[38;1m back to exx\033[0m")

gevent.joinall([
    gevent.spawn(foo),
    gevent.spawn(bar),
    gevent.spawn(exx),
])

output==>

 fooooooooooooooooooooo
 barrrrrrrrrrrrrrrr
 exxxxxxxxxxxxxxxxxx
 back to foo
 back to exx
 back to bar

遇到IO操作直接自动切换任务：

import gevent
from urllib.request import urlopen
from gevent import monkey
monkey.patch_all()

def f(url):
    print("GET:", url)
    resp = urlopen(url)
    data = resp.read()
    print("{} bytes received from {}.".format(len(data), url))

gevent.joinall([
    gevent.spawn(f, "https://www.baidu.com"),
    gevent.spawn(f, "https://www.yahoo.com"),
    gevent.spawn(f, "https://www.github.com"),
])

output==>

GET: https://www.baidu.com
GET: https://www.yahoo.com
GET: https://www.github.com
227 bytes received from https://www.baidu.com.
395547 bytes received from https://www.yahoo.com.
24748 bytes received from https://www.github.com.

1.2 IO多路复用：

I/O多路复用指：通过一种机制，可以监视多个描述符，一旦某个描述符就绪（一般是读就绪或者写就绪），能够通知程序进行相应的读写操作。

① Linux中的 select，poll，epoll 都是IO多路复用的机制。

② Python中的select

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "Q1mi"

"""
利用select实现的伪同时处理多个socket客户端请求：服务端
source:https://pymotw.com/3/select/
"""

import select
import socket
import queue
import sys

# 生成一个socket实例
server = socket.socket()
# 设置为非阻塞
server.setblocking(False)
# 配置IP和端口
ip_port = ("127.0.0.1", 4444)
# 绑定IP和端口
server.bind(ip_port)

# 监听五个连接
server.listen(5)

# 生成一个readable的列表,用于存放所有输入的信息
inputs = [server, ]

# 生成一个writeable的列表，用于存放所有输出的信息
outputs = []

# 生成一个消息队列
message_queue = {}

while inputs:
    print("\nWaiting for the next event.", file=sys.stderr)
    # select返回三个列表，readable中存放可以接收数据的连接，writeable存放可以发送数据的连接，exceptional存放发生错误的连接
    readable, writeable, exceptional = select.select(inputs, outputs, inputs)

    # 处理可以读取的连接
    for s in readable:
        # 如果server在inputs列表里，表示server已经准备好接收连接请求
        if s is server:
            # 接收连接请求
            conn, addr = s.accept()
            # 打印提示信息
            print("A new collection from:", addr, file=sys.stderr)
            # 将该连接设为非阻塞
            conn.setblocking(0)
            # 在inputs列表中放入该连接
            inputs.append(conn)
            # 为每个连接生成一个消息队列，用于存放我们要发送给该连接的数据
            message_queue[conn] = queue.Queue()
        # 不是server就是具体的连接
        else:
            # 接收数据
            data = s.recv(1024)
            # 如果数据存在
            if data:
                # 打印可能出现的错误信息
                print("Received {!r} from {}.".format(data, s.getpeername()), file=sys.stderr)
                # 将收到的数据存放到以该连接为key值的消息队列中
                message_queue[s].put(data)
                # 如果该连接不在可以输出消息的socket连接列表中
                if s not in outputs:
                    # 把该连接加入到可以输出消息的socket连接列表中
                    outputs.append(s)
            # 如果没有数据就关闭连接
            else:
                print("Closing {} ,after reading no data.".format(addr), file=sys.stderr)
                # 该连接如果存在于可以输出消息的socket连接列表中，就删除
                if s in outputs:
                    outputs.remove(s)
                # 也无需继续等待该连接的输入
                inputs.remove(s)
                # 关闭该连接
                s.close()
                # 从消息队列中删除该连接的消息
                del message_queue[s]

    # 处理输出的
    for s in writeable:
        try:
            next_msg = message_queue[s].get_nowait()
        # 如果消息队列为空
        except queue.Empty:
            # 打印提示信息
            print("Output queue for {} is empty.".format(s.getpeername()), file=sys.stderr)
            # 从可输出信息的socket连接列表中删除该连接
            outputs.remove(s)
        # 如果消息队列里有信息就将该信息发出。
        else:
            # 打印提示信息
            print("Sending {!r} to {}.".format(next_msg, s.getpeername()), file=sys.stderr)
            s.send(next_msg)
    # 处理异常情况
    for s in exceptional:
        # 打印提示信息
        print("Exceptional condition for {}.".format(s.getpeername()), file=sys.stderr)
        # 不再监听该连接是否可读
        inputs.remove(s)
        # 也不再监听是否可以对该连接发送数据
        if s in outputs:
            outputs.remove(s)
        # 关闭连接
        s.close()

        # 删除该连接的消息队列
        del message_queue[s]

利用select实现的伪同时处理多个socket客户端请求：服务端

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# __author__ = "Q1mi"

"""
利用select实现的伪同时处理多个socket客户端请求：客户端
source:https://pymotw.com/3/select/
"""

import socket
import sys

# 定义一个消息列表
message_list = [
    "This is the message.",
    "It will be sent ",
    "in parts.",
]

# 服务器的IP和端口信息
server_ip_port = ("127.0.0.1", 4444)

# 生成一堆socket实例
client_list = [
    socket.socket(),
    socket.socket(),
    # socket.socket(),
    # socket.socket(),
]

print("Connecting to {} port {}.".format(*server_ip_port), file=sys.stderr)

# 将socket连接实例分别去连接server端
for c in client_list:
    c.connect(server_ip_port)

# 将消息列表中的信息循环发送给server端
for message in message_list:
    outgoing_data = message.encode()
    for c in client_list:
        print("{}: sending {!r}.".format(c.getsockname(), outgoing_data), file=sys.stderr)
        c.send(outgoing_data)

    # 同时也接收server端返回的数据
    for c in client_list:
        data = c.recv(1024)
        print("{}: received {!r}.".format(c.getsockname(), data), file=sys.stderr)
        # 如果数据为空就关闭该连接
        if not data:
            print("Closing socket:{}".format(c.getsockname()), file=sys.stderr)
            c.close()

利用select实现的伪同时处理多个socket客户端请求：客户端

server端：
Waiting for the next event.
A new collection from: ('127.0.0.1', 12228)

Waiting for the next event.
A new collection from: ('127.0.0.1', 12229)
Received b'This is the message.' from ('127.0.0.1', 12228).

Waiting for the next event.
Received b'This is the message.' from ('127.0.0.1', 12229).
Sending b'This is the message.' to ('127.0.0.1', 12228).

Waiting for the next event.
Output queue for ('127.0.0.1', 12228) is empty.
Sending b'This is the message.' to ('127.0.0.1', 12229).

Waiting for the next event.
Output queue for ('127.0.0.1', 12229) is empty.

Waiting for the next event.
Received b'It will be sent ' from ('127.0.0.1', 12228).

Waiting for the next event.
Received b'It will be sent ' from ('127.0.0.1', 12229).
Sending b'It will be sent ' to ('127.0.0.1', 12228).

Waiting for the next event.
Output queue for ('127.0.0.1', 12228) is empty.
Sending b'It will be sent ' to ('127.0.0.1', 12229).

Waiting for the next event.
Output queue for ('127.0.0.1', 12229) is empty.

Waiting for the next event.
Received b'in parts.' from ('127.0.0.1', 12228).

Waiting for the next event.
Received b'in parts.' from ('127.0.0.1', 12229).
Sending b'in parts.' to ('127.0.0.1', 12228).

Waiting for the next event.
Output queue for ('127.0.0.1', 12228) is empty.
Sending b'in parts.' to ('127.0.0.1', 12229).

Waiting for the next event.
Output queue for ('127.0.0.1', 12229) is empty.

Waiting for the next event.
Closing ('127.0.0.1', 12229) ,after reading no data.

Waiting for the next event.
Closing ('127.0.0.1', 12229) ,after reading no data.

Waiting for the next event.
...
==============================================
client端：
Connecting to 127.0.0.1 port 4444.
('127.0.0.1', 12228): sending b'This is the message.'.
('127.0.0.1', 12229): sending b'This is the message.'.
('127.0.0.1', 12228): received b'This is the message.'.
('127.0.0.1', 12229): received b'This is the message.'.
('127.0.0.1', 12228): sending b'It will be sent '.
('127.0.0.1', 12229): sending b'It will be sent '.
('127.0.0.1', 12228): received b'It will be sent '.
('127.0.0.1', 12229): received b'It will be sent '.
('127.0.0.1', 12228): sending b'in parts.'.
('127.0.0.1', 12229): sending b'in parts.'.
('127.0.0.1', 12228): received b'in parts.'.
('127.0.0.1', 12229): received b'in parts.'.

结果

2. Select\Poll\Epoll异步IO与事件驱动

3.Python连接MySQL数据库操作

4.RabbitMQ

5.Redis\Memcached缓存

6.Paramiko

7.Twsited网络框架