Python学习路程day9

本节内容

Gevent协程
Select\Poll\Epoll异步IO与事件驱动
Python连接Mysql数据库操作
RabbitMQ队列
Redis\Memcached缓存
Paramiko SSH
Twsited网络框架

queue队列

queue is especially useful in threaded programming when information must be exchanged safely between multiple threads.

class queue.Queue(maxsize=0) #先入先出

class queue.LifoQueue(maxsize=0) #last in fisrt out
class queue.PriorityQueue(maxsize=0) #存储数据时可设置优先级的队列

Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.

The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form: (priority_number, data).

exception queue.Empty: Exception raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.

exception queue.Full: Exception raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.

Queue.qsize()

Queue.empty() #return True if empty

Queue.full() # return True if full

Queue.put(item, block=True, timeout=None): Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).

Queue.put_nowait(item): Equivalent to put(item, False).

Queue.get(block=True, timeout=None): Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).

Queue.get_nowait(): Equivalent to get(False).

Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.

Queue.task_done()

Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.

If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).

Raises a ValueError if called more times than there were items placed in the queue.

Queue.join() block直到queue被消费完毕

生产者消费者模型

 import time,random

 import queue,threading

 q = queue.Queue()

 def Producer(name):

   count = 0

   while count <20:

     time.sleep(random.randrange(3))

     q.put(count)

     print('Producer %s has produced %s baozi..' %(name, count))

     count +=1

 def Consumer(name):

   count = 0

   while count <20:

     time.sleep(random.randrange(4))

     if not q.empty():

         data = q.get()

         print(data)

         print('\033[32;1mConsumer %s has eat %s baozi...\033[0m' %(name, data))

     else:

         print("-----no baozi anymore----")

     count +=1

 p1 = threading.Thread(target=Producer, args=('A',))

 c1 = threading.Thread(target=Consumer, args=('B',))

 p1.start()

 c1.start()

协程

协程，又称微线程，纤程。英文名Coroutine。一句话说明什么是协程：协程是一种用户态的轻量级线程。

协程拥有自己的寄存器上下文和栈。协程调度切换时，将寄存器上下文和栈保存到其他地方，在切回来的时候，恢复先前保存的寄存器上下文和栈。因此：

协程能保留上一次调用时的状态（即所有局部状态的一个特定组合），每次过程重入时，就相当于进入上一次调用的状态，换种说法：进入上一次离开时所处逻辑流的位置。

协程的好处：

无需线程上下文切换的开销
无需原子操作锁定及同步的开销
方便切换控制流，简化编程模型
高并发+高扩展性+低成本：一个CPU支持上万的协程都不是问题。所以很适合用于高并发处理。

缺点：

无法利用多核资源：协程的本质是个单线程,它不能同时将单个CPU 的多个核用上,协程需要和进程配合才能运行在多CPU上.当然我们日常所编写的绝大部分应用都没有这个必要，除非是cpu密集型应用。
进行阻塞（Blocking）操作（如IO时）会阻塞掉整个程序

使用yield实现协程操作例子　　　　

 import time

 import queue

 def consumer(name):

     print("--->starting eating baozi...")

     while True:

         new_baozi = yield

         print("[%s] is eating baozi %s" % (name,new_baozi))

         #time.sleep(1)

 def producer():

     r = con.__next__()

     r = con2.__next__()

     n = 0

     while n < 5:

         n +=1

         con.send(n)

         con2.send(n)

         print("\033[32;1m[producer]\033[0m is making baozi %s" %n )

 if __name__ == '__main__':

     con = consumer("c1")

     con2 = consumer("c2")

     p = producer()

Greenlet

 #!/usr/bin/env python

 # -*- coding:utf-8 -*-

 from greenlet import greenlet

 def test1():

     print 12

     gr2.switch()

     print 34

     gr2.switch()

 def test2():

     print 56

     gr1.switch()

     print 78

 gr1 = greenlet(test1)

 gr2 = greenlet(test2)

 gr1.switch()

Gevent

Gevent 是一个第三方库，可以轻松通过gevent实现并发同步或异步编程，在gevent中用到的主要模式是Greenlet, 它是以C扩展模块形式接入Python的轻量级协程。 Greenlet全部运行在主程序操作系统进程的内部，但它们被协作式地调度。

 import gevent

 def foo():

     print('Running in foo')

     gevent.sleep(0)

     print('Explicit context switch to foo again')

 def bar():

     print('Explicit context to bar')

     gevent.sleep(0)

     print('Implicit context switch back to bar')

 gevent.joinall([

     gevent.spawn(foo),

     gevent.spawn(bar),

 ])

输出：

Running in foo

Explicit context to bar

Explicit context switch to foo again

Implicit context switch back to bar

同步与异步的性能区别

 import gevent

 def task(pid):

     """

     Some non-deterministic task

     """

     gevent.sleep(0.5)

     print('Task %s done' % pid)

 def synchronous():

     for i in range(1,10):

         task(i)

 def asynchronous():

     threads = [gevent.spawn(task, i) for i in range(10)]

     gevent.joinall(threads)

 print('Synchronous:')

 synchronous()

 print('Asynchronous:')

 asynchronous()

urllib

python 3.x: from urllib.request import urlopen

python 2.x: from urllib2 import urlopen

 from gevent import monkey; monkey.patch_all()

 import gevent

 #from urllib2 import urlopen

 from  urllib.request import urlopen

 def f(url):

     print('GET: %s' % url)

     resp = urlopen(url)

     data = resp.read()

     print('%d bytes received from %s.' % (len(data), url))

 gevent.joinall([

         gevent.spawn(f, 'https://www.python.org/'),

         gevent.spawn(f, 'https://www.yahoo.com/'),

         gevent.spawn(f, 'https://github.com/'),

 ])

在访问第一个python页面时，正常是等待接收返回的数据，但是此时会接着访问yahoo的页面，等待数据时，又会去返回github页面，相当于并发操作

通过gevent实现单线程下的多socket并发

server side

import sys

import socket

import time

import gevent

from gevent import socket,monkey

monkey.patch_all()

def server(port):

    s = socket.socket()

    s.bind(('0.0.0.0', port))

    s.listen(500)

    while True:

        cli, addr = s.accept()

        gevent.spawn(handle_request, cli)

def handle_request(s):

    try:

        while True:

            data = s.recv(1024)

            print("recv:", data)

            s.send(data)

            if not data:

                s.shutdown(socket.SHUT_WR)

    except Exception as  ex:

        print(ex)

    finally:

        s.close()

if __name__ == '__main__':

    server(8001)

client side 　

 import socket

 HOST = 'localhost'    # The remote host

 PORT = 8001           # The same port as used by the server

 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

 s.connect((HOST, PORT))

 while True:

     msg = bytes(input(">>:"),encoding="utf8")

     s.sendall(msg)

     data = s.recv(1024)

     #print(data)

     print('Received', repr(data))

 s.close()

论事件驱动与异步IO

事件驱动编程是一种编程范式，这里程序的执行流由外部事件来决定。它的特点是包含一个事件循环，当外部事件发生时使用回调机制来触发相应的处理。另外两种常见的编程范式是（单线程）同步以及多线程编程。

让我们用例子来比较和对比一下单线程、多线程以及事件驱动编程模型。下图展示了随着时间的推移，这三种模式下程序所做的工作。这个程序有3个任务需要完成，每个任务都在等待I/O操作时阻塞自身。阻塞在I/O操作上所花费的时间已经用灰色框标示出来了。

在单线程同步模型中，任务按照顺序执行。如果某个任务因为I/O而阻塞，其他所有的任务都必须等待，直到它完成之后它们才能依次执行。这种明确的执行顺序和串行化处理的行为是很容易推断得出的。如果任务之间并没有互相依赖的关系，但仍然需要互相等待的话这就使得程序不必要的降低了运行速度。

在多线程版本中，这3个任务分别在独立的线程中执行。这些线程由操作系统来管理，在多处理器系统上可以并行处理，或者在单处理器系统上交错执行。这使得当某个线程阻塞在某个资源的同时其他线程得以继续执行。与完成类似功能的同步程序相比，这种方式更有效率，但程序员必须写代码来保护共享资源，防止其被多个线程同时访问。多线程程序更加难以推断，因为这类程序不得不通过线程同步机制如锁、可重入函数、线程局部存储或者其他机制来处理线程安全问题，如果实现不当就会导致出现微妙且令人痛不欲生的bug。

在事件驱动版本的程序中，3个任务交错执行，但仍然在一个单独的线程控制中。当处理I/O或者其他昂贵的操作时，注册一个回调到事件循环中，然后当I/O操作完成时继续执行。回调描述了该如何处理某个事件。事件循环轮询所有的事件，当事件到来时将它们分配给等待处理事件的回调函数。这种方式让程序尽可能的得以执行而不需要用到额外的线程。事件驱动型程序比多线程程序更容易推断出行为，因为程序员不需要关心线程安全问题。

当我们面对如下的环境时，事件驱动模型通常是一个好的选择：

程序中有许多任务，而且…
任务之间高度独立（因此它们不需要互相通信，或者等待彼此）而且…
在等待事件到来时，某些任务会阻塞。

当应用程序需要在任务间共享可变的数据时，这也是一个不错的选择，因为这里不需要采用同步处理。

网络应用程序通常都有上述这些特点，这使得它们能够很好的契合事件驱动编程模型。

Select\Poll\Epoll异步IO　

http://www.cnblogs.com/alex3714/p/4372426.html

selectors模块

This module allows high-level and efficient I/O multiplexing, built upon the select module primitives. Users are encouraged to use this module instead, unless they want precise control over the OS-level primitives used.

 import selectors

 import socket

 sel = selectors.DefaultSelector()

 def accept(sock, mask):

     conn, addr = sock.accept()  # Should be ready

     print('accepted', conn, 'from', addr)

     conn.setblocking(False)

     sel.register(conn, selectors.EVENT_READ, read)

 def read(conn, mask):

     data = conn.recv(1000)  # Should be ready

     if data:

         print('echoing', repr(data), 'to', conn)

         conn.send(data)  # Hope it won't block

     else:

         print('closing', conn)

         sel.unregister(conn)

         conn.close()

 sock = socket.socket()

 sock.bind(('localhost', 10000))

 sock.listen(100)

 sock.setblocking(False)

 sel.register(sock, selectors.EVENT_READ, accept)

 while True:

     events = sel.select()

     for key, mask in events:

         callback = key.data

         callback(key.fileobj, mask)

数据库操作与Paramiko模块

SSHClient

用于连接远程服务器并执行基本命令

基于用户名密码连接：

 import paramiko

 # 创建SSH对象

 ssh = paramiko.SSHClient()

 # 允许连接不在know_hosts文件中的主机

 ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

 # 连接服务器

 ssh.connect(hostname='c1.salt.com', port=22, username='wupeiqi', password='')

 # 执行命令

 stdin, stdout, stderr = ssh.exec_command('df')

 # 获取命令结果

 result = stdout.read()

 # 关闭连接

 ssh.close()

基于公钥密钥连接：

 import paramiko

 private_key = paramiko.RSAKey.from_private_key_file('/home/auto/.ssh/id_rsa')

 # 创建SSH对象

 ssh = paramiko.SSHClient()

 # 允许连接不在know_hosts文件中的主机

 ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

 # 连接服务器

 ssh.connect(hostname='c1.salt.com', port=22, username='wupeiqi', key=private_key)

 # 执行命令

 stdin, stdout, stderr = ssh.exec_command('df')

 # 获取命令结果

 result = stdout.read()

 # 关闭连接

 ssh.close()

SFTPClient

用于连接远程服务器并执行上传下载

基于用户名密码上传下载

 import paramiko

 transport = paramiko.Transport(('hostname',22))

 transport.connect(username='wupeiqi',password='')

 sftp = paramiko.SFTPClient.from_transport(transport)

 # 将location.py 上传至服务器 /tmp/test.py

 sftp.put('/tmp/location.py', '/tmp/test.py')

 # 将remove_path 下载到本地 local_path

 sftp.get('remove_path', 'local_path')

 transport.close()

基于公钥密钥上传下载

 import paramiko

 private_key = paramiko.RSAKey.from_private_key_file('/home/auto/.ssh/id_rsa')

 transport = paramiko.Transport(('hostname', 22))

 transport.connect(username='wupeiqi', pkey=private_key )

 sftp = paramiko.SFTPClient.from_transport(transport)

 # 将location.py 上传至服务器 /tmp/test.py

 sftp.put('/tmp/location.py', '/tmp/test.py')

 # 将remove_path 下载到本地 local_path

 sftp.get('remove_path', 'local_path')

 transport.close()

 #!/usr/bin/env python

 # -*- coding:utf-8 -*-

 import paramiko

 import uuid

 class Haproxy(object):

     def __init__(self):

         self.host = '172.16.103.191'

         self.port = 22

         self.username = 'wupeiqi'

         self.pwd = ''

         self.__k = None

     def create_file(self):

         file_name = str(uuid.uuid4())

         with open(file_name,'w') as f:

             f.write('sb')

         return file_name

     def run(self):

         self.connect()

         self.upload()

         self.rename()

         self.close()

     def connect(self):

         transport = paramiko.Transport((self.host,self.port))

         transport.connect(username=self.username,password=self.pwd)

         self.__transport = transport

     def close(self):

         self.__transport.close()

     def upload(self):

         # 连接，上传

         file_name = self.create_file()

         sftp = paramiko.SFTPClient.from_transport(self.__transport)

         # 将location.py 上传至服务器 /tmp/test.py

         sftp.put(file_name, '/home/wupeiqi/tttttttttttt.py')

     def rename(self):

         ssh = paramiko.SSHClient()

         ssh._transport = self.__transport

         # 执行命令

         stdin, stdout, stderr = ssh.exec_command('mv /home/wupeiqi/tttttttttttt.py /home/wupeiqi/ooooooooo.py')

         # 获取命令结果

         result = stdout.read()

 ha = Haproxy()

 ha.run()

demo

数据库操作

Python 操作 Mysql 模块的安装

 linux：

     yum install MySQL-python

 window：

     http://files.cnblogs.com/files/wupeiqi/py-mysql-win.zip

SQL基本使用

1、数据库操作

 show databases;

 use [databasename];

 create database  [name];

2、数据表操作

 show tables;

 create table students

     (

         id int  not null auto_increment primary key,

         name char(8) not null,

         sex char(4) not null,

         age tinyint unsigned not null,

         tel char(13) null default "-"

     );

CREATE TABLE `wb_blog` (

    `id` smallint(8) unsigned NOT NULL,

    `catid` smallint(5) unsigned NOT NULL DEFAULT '',

    `title` varchar(80) NOT NULL DEFAULT '',

    `content` text NOT NULL,

    PRIMARY KEY (`id`),

    UNIQUE KEY `catename` (`catid`)

) ;

3、数据操作

 insert into students(name,sex,age,tel) values('alex','man',18,'')

 delete from students where id =2;

 update students set name = 'sb' where id =1;

 select * from students

4、其他

 主键

 外键

 左右连接

Python MySQL API

一、插入数据

 import MySQLdb

 conn = MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='mydb')

 cur = conn.cursor()

 reCount = cur.execute('insert into UserInfo(Name,Address) values(%s,%s)',('alex','usa'))

 # reCount = cur.execute('insert into UserInfo(Name,Address) values(%(id)s, %(name)s)',{'id':12345,'name':'wupeiqi'})

 conn.commit()

 cur.close()

 conn.close()

 print reCount

 import MySQLdb

 conn = MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='mydb')

 cur = conn.cursor()

 li =[

      ('alex','usa'),

      ('sb','usa'),

 ]

 reCount = cur.executemany('insert into UserInfo(Name,Address) values(%s,%s)',li)

 conn.commit()

 cur.close()

 conn.close()

 print reCount

批量插入数据

注意：cur.lastrowid

二、删除数据

 import MySQLdb

 conn = MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='mydb')

 cur = conn.cursor()

 reCount = cur.execute('delete from UserInfo')

 conn.commit()

 cur.close()

 conn.close()

 print reCount

三、修改数据

import MySQLdb

conn = MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='mydb')

cur = conn.cursor()

reCount = cur.execute('update UserInfo set Name = %s',('alin',))

conn.commit()

cur.close()

conn.close()

print reCount

四、查数据

 # ############################## fetchone/fetchmany(num)  ##############################

 import MySQLdb

 conn = MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='mydb')

 cur = conn.cursor()

 reCount = cur.execute('select * from UserInfo')

 print cur.fetchone()

 print cur.fetchone()

 cur.scroll(-1,mode='relative')

 print cur.fetchone()

 print cur.fetchone()

 cur.scroll(0,mode='absolute')

 print cur.fetchone()

 print cur.fetchone()

 cur.close()

 conn.close()

 print reCount

 # ############################## fetchall  ##############################

 import MySQLdb

 conn = MySQLdb.connect(host='127.0.0.1',user='root',passwd='',db='mydb')

 #cur = conn.cursor(cursorclass = MySQLdb.cursors.DictCursor)

 cur = conn.cursor()

 reCount = cur.execute('select Name,Address from UserInfo')

 nRet = cur.fetchall()

 cur.close()

 conn.close()

 print reCount

 print nRet

 for i in nRet:

     print i[0],i[1]

RabbitMQ队列　　

安装 http://www.rabbitmq.com/install-standalone-mac.html

安装python rabbitMQ module