《dive into python3》 笔记摘录

1、list can hold  arbitrary  objects and can expand dynamically as new items are added. A list is an  ordered  set of items.

2、A tuple is an 
immutable list. A tuple can not be changed in any way once it is created.

3、A set is an 
unordered “bag” 
of unique values. A single set can contain values of any immutable datatype.

4、A dictionary is an 
unordered set of key-value pairs. keys are 
unique and 
immutable

5、import os, glob, humansize

     

     metadata_list = [(f, os.stat(f)) for f in glob.glob('*test*.py')]

     metadata_dict = {f:os.stat(f) for f in glob.glob('*')}

humansize_dict = {os.path.splitext(f)[0]:humansize.approximate_size(meta.st_size) \


                              for f, meta in metadata_dict.items() if meta.st_size > 6000}

     

     a_set = {2**x for x in range(10)}

6、Bytes are not characters; bytes are bytes. Characters are an abstraction. A string is a sequence of those 

abstractions. In Python 3, all strings are 
immutable sequences of Unicode characters.The built-in len() 

function returns the length of the string, i.e. the 
number of characters. A string is like a tuple of characters.

An 
immutable sequence of numbers-between-0-and-255 is called a bytes object.

Each item in a string is a string, each item in a byte array is an integer.

aBuf = b'\xEF\xBB\xBF'

aBuf[-1] #191

aBuf[-1:] #b'\xbf'  byte array

7、To define a bytes object, use the 
b' ' “byte literal” syntax. Each byte within the byte literal can be an 

ASCII character or an encoded hexadecimal number from \x00 to \xff (0–255).To convert a bytes object into 

a mutable bytearray object, use the built-in bytearray() function.

8、bytes objects have a 
decode() method that takes acharacter encoding and returns a string, and strings 

have an 
encode() method that takes a characterencoding and returns a bytes object. 

9、'1MB = 1000{0.modules[humansize].SUFFIXES[1000][0]}'.format(sys)  #compound field names

10、“The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string.”

11、Within a replacement field, a colon (:) marks the start of the format specifier.

12、compact regular expressions

import re

s = '100 BROAD ROAD APT. 3'

 re.sub(r'\bROAD\b', 'RD.', s)

#  re.sub(b'\bROAD\b', 'RD.', s)

# search bytes

pattern = '^M{0,3}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$'

re.search(pattern, 'MDLV')

phonePattern = re.compile(r'(\d{3})\D*(\d{3})\D*(\d{4})\D*(\d*)$')

#Putting it all in parentheses means “match exactly three numeric digits, and then remember them as a #group that I can ask for later”. 

phonePattern.search('(800)5551212 ext. 1234').groups() 

('800', '555', '1212', '1234')

>>> phonePattern.search('800-555-1212').groups() 

('800', '555', '1212', '')

>>> phonePattern.search('work 1-(800) 555.1212 #1234')

('800', '555', '1212', '1234')

if phonePattern.match('800-555-1212'):

     do_something

if phonePattern.search('800-555-1212'):

     do_something

re.sub('([^aeiou])y$', r'\1ies', 'vacancy') 

'vacancies'

# \1, which means “hey, that first group you remembered? put it right here.” 

re.findall('[0-9]+', '16 2-by-4s in rows of 8') 

['16', '2', '4', '8']

re.findall('[A-Z]+', 'SEND + MORE == MONEY') 

['SEND', 'MORE', 'MONEY']

re.findall(' s.*? s', "The sixth sick sheikh's sixth sheep's sick.") 

# (.*?) means the shortest possible series of any character

[' sixth s', " sheikh's s", " sheep's s"] 
#doesn’t return overlapping matches. 

13、verbose regular expressions

Python allows you to do this with something called verbose regular expressions. A verbose regular expression

is different from a compact regular expression in two ways:

• Whitespace is ignored. Spaces, tabs, and carriage returns are not matched as spaces, tabs, and carriage

returns. They’re not matched at all. (If you want to match a space in a verbose regular expression, you’ll

need to escape it by putting a backslash in front of it.)

• Comments are ignored. A comment in a verbose regular expression is just like a comment in Python code:

it starts with a # character and goes until the end of the line. In this case it’s a comment within a multi-line

string instead of within your source code, but it works the same way.

pattern = '''

^                                    # beginning of string

M{0,3}                              # thousands - 0 to 3 Ms

(CM|CD|D?C{0,3})           # hundreds - 900 (CM), 400 (CD), 0-300 (0 to 3 Cs),

                                   # or 500-800 (D, followed by 0 to 3 Cs)

(XC|XL|L?X{0,3})           # tens - 90 (XC), 40 (XL), 0-30 (0 to 3 Xs),

                                   # or 50-80 (L, followed by 0 to 3 Xs)

(IX|IV|V?I{0,3})                # ones - 9 (IX), 4 (IV), 0-3 (0 to 3 Is),

                                   # or 5-8 (V, followed by 0 to 3 Is)

$                                    # end of string

'''

 re.search(pattern, 'M', re.VERBOSE) 

phonePattern = re.compile(r'''

                                   # don't match beginning of string, number can start anywhere

(\d{3})                          # area code is 3 digits (e.g. '800')

\D*                               # optional separator is any number of non-digits

(\d{3})                          # trunk is 3 digits (e.g. '555')

\D*                               # optional separator

(\d{4})                          # rest of number is 4 digits (e.g. '1212')

\D*                               # optional separator

(\d*)                          # extension is optional and can be any number of digits

$                               # end of string

''', re.VERBOSE)

phonePattern.search('work 1-(800) 555.1212 #1234').groups()

14、regular expressions

• ^ matches the beginning of a string.

• $ matches the end of a string.

• \b matches a word boundary.

• \d matches any numeric digit.

• \D matches any non-numeric character.

• x? matches an optional x character (in other words, it matches an x zero or one times).

• x* matches x zero or more times.

• x+ matches x one or more times.

• x{n,m} matches an x character at least n times, but not more than m times.

• (a|b|c) matches exactly one of a, b or c.

• (x) in general is a 
remembered group. You can get the value of what matched by using the groups() methodof the object returned by re.search.

15、This technique of using the values of outside parameters within a dynamic function is called 
closures. 

16、The 
with statement creates what’s called a context: when the with block ends, Python will automatically close the file, even if an exception is raised inside the with block.

There’s nothing file-specific about the with statement; it’s just a generic framework for creating runtime


contexts and telling objects that they’re entering and exiting a runtime context. If the object in question is a

stream object, then it does useful file-like things (like closing the file automatically). But that behavior is

defined in the stream object, not in the with statement. 

17、The first argument to the 
split() method is 
None, which means “split on any whitespace (tabs or spaces, it makes no difference).” The second argument is 3, which means “split on whitespace 3 times, then leave the rest of the line alone.” 

18、The presence of the 
yield x keyword in function body means that this is not a normal function. It is a special kind of function which generates values one at a time. You can think of it as a 
resumable function. Calling it will return a 
generator that can be used to generate successive values of x. The 
next() function takes a generator object and returns its next value.

"yield" pause a function, and "next()" resumes where it left off.

Generators are just a simple form of iterators. A function that yields values is a nice, compact way of building an 
iterator without building an iterator. File objects are iterators too! It’s iterators all the way down.

This is a useful idiom: pass a generator to the list() function, and it will iterate through the entire

generator (just like the 
for loop) and return a list of all the values.

The for loop will automatically call the next() function to get values from the 
generator and assign them to the for loop index variable.

19、‘
pass' is a Python reserved word that just means “move along, nothing to see here”. 

20、The first argument of every class method, including the 
__init__() method, is always a reference to the

current instance of the class. By convention, this argument is named 
self
.

21、An 
iterator is just a class that defines an 
__iter__() method. 

 Python Code 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

class Fib:      '''iterator that yields numbers in the Fibonacci sequence'''      def  __init__(self, maxn):         self.maxn = maxn def  __iter__(self):         self.a = self.b = return self def __next__(self):         fib = self.a          if fib > self.maxn:              raise  StopIteration         self.a, self.b = self.b, self.a + self.b          return fib for n  in Fib( ):      print(n, end= ' ')

After performing beginning-of-iteration initialization, the __iter__() method can return any object that implements a 
__next__() method. The __next__() method is called whenever someone calls 
next() on an iterator of an instance of a class.

iter(object) calls object.__iter__(), return an iterator object; 

next(iterator_object) calls iterator_object.__next__(), return a value;

 C++ Code 

1 2 3

for n in Fib( ):      print(n, end= ' ')

a for loop calls __init__() (if the object exists, ignore) and __iter__() once, but calls __next__() several times until encounter raise 
StopIteration exception. 

When the __next__() method raises a StopIteration exception, this signals to the caller that the iteration

is exhausted. Unlike most exceptions, this is not an error; it’s a normal condition that just means that the

iterator has no more values to generate. If the caller is a for loop, it will notice this StopIteration

exception and gracefully 
exit the loop. 

22、when the variable was  not defined within any method. It’s defined at the 
class level. It’s a 
class variable, and although you can access it just like an instance variable (self.rules_filename), it is shared across all instances of the same class.

23、A 
generator expression is like an anonymous function that yields values. The expression itself looks like a list comprehension, but it’s wrapped in 
parentheses instead of square brackets.

unique_characters = {'E', 'D', 'M', 'O', 'N', 'S', 'R', 'Y'}

 gen = (ord(c) for c in unique_characters)

If you like, you can iterate through all the possible values and return a tuple, list, or set, by passing the

generator expression to tuple(), list(), or set(). In these cases, you don’t need an extra set of

parentheses — just pass the “bare” expression ord(c) for c in unique_characters to the tuple()

function, and Python figures out that it’s a generator expression.

tuple(ord(c) for c in unique_characters) 

(69, 68, 77, 79, 78, 83, 82, 89)

24、The 
itertools.permutations() function doesn’t have to take a list. It can take any sequence — even a string.The permutations() function takes a sequence and a number, which is the number of items you want in each smaller group. The function returns an iterator.

The 
itertools.combinations() function returns an iterator containing all the possible combinations of the

given sequence of the given length.

 

The itertools.
groupby() function takes a sequence and a key function, and returns an iterator that

generates pairs. Each pair contains the result of key_function(each item) and another iterator containing

all the items that shared that key result.

The itertools.groupby() function only works if the input sequence is already sorted by the grouping function. 

The 
itertools.chain() function takes two iterators and returns an iterator that contains all the items

from the first iterator, followed by all the items from the second iterator. (Actually, it can take any number

of iterators, and it chains them all in the order they were passed to the function.)

25、
rstrip() string method to strip trailing whitespace from each line. (Strings also have an 
lstrip() method to strip leading whitespace, and a 
strip() method which strips both.)

26、zip()、tuple()、dict()、translate()、eval()

characters = ('S', 'M', 'E', 'D', 'O', 'N', 'R', 'Y') guess = ('1', '2', '0', '3', '4', '5', '6', '7')

tuple(zip(characters, guess)) 


(('S', '1'), ('M', '2'), ('E', '0'), ('D', '3'), ('O', '4'), ('N', '5'), ('R', '6'), ('Y', '7'))


 dict(zip(characters, guess)) 


{'E': '0', 'D': '3', 'M': '2', 'O': '4', 'N': '5', 'S': '1', 'R': '6', 'Y': '7'}

'SEND + MORE == MONEY'.translate(translation_table) 

'1053 + 2460 == 24507'

The second and third parameters passed to the 
eval() function act as the global and local namespaces for

evaluating the expression. 

The 
subprocess module allows you to run arbitrary shell commands and get the result as a Python string.

eval("__import__('subprocess').getoutput('rm -rf /')",  {"__builtins__":None}, {}) 

#error. the __import__() function is also a builtin function

27、Running the script runs 
unittest.main(), which runs each test case. Each test case is a method within a

class in xxxtest.py. There is no required organization of these test classes; they can each contain a single

test method, or you can have one class that contains multiple test methods. The only requirement is that

each test class must inherit from 
unittest.TestCase.

The unittest.TestCase class provides the 
assertRaises method, which takes the following arguments: the

exception you’re expecting, the function you’re testing, and the arguments you’re passing to that function. (If the function you’re testing takes more than one argument, pass them all to assertRaises, in order, and it

will pass them right along to the function you’re testing.)

28、It is important to understand that modules are only imported once, then cached. If you import an already-imported module, it does nothing. 

29、a file on disk is a sequence of bytes.The default encoding is platform dependent.

30、Python has a built-in function called 
open(). The open() function returns a 
stream object, which has methods and attributes for getting information about and manipulating a stream of characters.

31、Once you open a file (with the correct encoding), reading from it is just a matter of calling the stream

object’s 
read() method. The result is a string.The read() method can take an optional parameter, the number of characters to read.

32、The 
seek() and 
tell() methods always count 
bytes, but since you opened this file as text, the read() method counts 
characters. Chinese characters require multiple bytes to encode in UTF -8 .

33、The stream object file still exists; calling its 
close() method doesn’t destroy the object itself. But it’s

not terribly useful.Closed stream objects do have one useful attribute: the 
closed
 attribute will confirm that the file is closed.

34、Read a file one line at a time

 Python Code 

1 2 3 4 5

line_number = with  open( 'examples/favorite-people.txt', encoding= 'utf-8')  as a_file:       for a_line  in a_file:          line_number += print( '{:>4} {}'. format(line_number, a_line. rstrip()))

the stream object is also an 
iterator which spits out a single line every time you ask for a value.

The format specifier {:>4} means “print this argument right-justified within 4 spaces.” 

The 
rstrip() string method removes the trailing whitespace, including the carriage return characters.

35、Reading a “string” from a 
text file only works because you told Python what encoding to use to read a stream of bytes and convert it to a string. 

36、Opening a file in 
binary mode is simple but subtle. The only difference from opening it in text mode is that the mode parameter contains a 
'b' character. a binary stream object has no 
encoding attribute.

Reading a file in “binary” mode? You’ll get a stream of bytes. Fetching a web page? Calling a web API ? They return a stream of bytes, too.

37、Since you opened the file in binary mode, the read() method takes the number of 
bytes to read, not the number of characters.

38、As long as your functions take a stream object and simply call the object’s read() method, you can handle any 
input source that acts like a file, without specific code to handle each kind of input.

39、
io.StringIO lets you treat a string as a text file. There’s also a 
io.BytesIO class, which lets you treat a byte array as a binary file.

40、The 
gzip module lets you create a stream object for reading or writing a gzip-compressed file.

You should always open gzipped files in binary mode.

 

import gzip

with gzip.open('out.log.gz', mode='wb') as z_file: 

     z_file.write('A nine mile walk is no joke, especially in the rain.'.encode('utf-8'))

41、The print function adds a carriage return to the end of the string you’re printing, and calls 
sys.stdout.write.

42、Any class can be a 
context manager by defining two special methods: 
__enter__() and 
__exit__().

43、The ElementTree library is part of the Python standard library, in 
xml.etree.ElementTree

ElementTree represents XML elements as 
{namespace}localname.

In the ElementTree API, an element acts like a 
list. The items of the list are the element’s children.

XML isn’t just a collection of elements; each element can also have its own set of 
attributes（.attrib）. Once you have a reference to a specific element, you can easily get its attributes as a Python dictionary.

In a boolean context, ElementTree element objects will evaluate to False if they contain no children (i.e. if 
len(element) is 0). This means that if element.find('...') is not testing whether the find() method found a matching element; it’s testing whether that matching element has any child elements! To test whether the find() method returned an element, use 
if element.find('...') is not None

44、The time module contains a data structure 
(time_struct) to represent a point in time (accurate to one

millisecond) and functions to manipulate time structs. The 
strptime()
 function takes a formatted string an


converts it to a time_struct.

45、The 
dump() function in the 
pickle module takes a serializable Python data structure, serializes it into a

binary, Python-specific format using the latest version of the pickle protocol, and saves it to an open file.

The 
pickle.load() function takes a stream object, reads the serialized data from the stream, creates a new

Python object, recreates the serialized data in the new Python object, and returns the new Python object.

The 
pickle.dumps() function (note the 's' at the end of the function name) performs the same

serialization as the pickle.dump() function. Instead of taking a stream object and writing the serialized data

to a file on disk, it simply returns the serialized data.

The 
pickle.loads() function (again, note the 's' at the end of the function name) performs the same

deserialization as the pickle.load() function. Instead of taking a stream object and reading the serialized

data from a file, it takes a 
bytes object containing serialized data, such as the one returned by the

pickle.dumps() function.

46、Like the pickle module, the 
json module defines a dump() function which takes a Python data structure

and a writeable stream object. The dump() function serializes the Python data structure and writes it to the


stream object. Doing this inside a with statement will ensure that the file is closed properly when we’re


done.

JSON is a 
text-based format. Always open JSON files in text mode with a UTF -8 character encoding.

JSON doesn’t distinguish between 
tuples and lists; it only has a single list-like datatype, the array, and the json module silently converts both tuples and lists into JSON arrays during serialization. For most uses, you can ignore the difference between tuples and lists, but it’s something to keep in mind as you work with the json module.

47、The 
time.asctime() function will convert that nasty-looking time.struct_time into the string 'Fri Mar 27 22:20:42 2009'.

We can use the list() function to convert the bytes object into a list of integers. So b'\xDE\xD5\xB4\xF8' becomes [222, 213, 180, 248]. 

import customserializer 

 with open('entry.json', 'w', encoding='utf-8') as f: 

      json.dump(entry, f, default=customserializer.to_json)  
#shell 1

with open('entry.json', 'r', encoding='utf-8') as f:

     entry = json.load(f, object_hook=customserializer.from_json)      
#shell 2

48、The
 urllib.request.urlopen().read() method always returns a bytes object, not a string. Remember, bytes are 
bytes;characters are an abstraction. HTTP servers don’t deal in abstractions. If you request a resource, you get bytes. If you want it as a string, you’ll need to determine the character encoding and explicitly convert it to a string.

The 
response returned from the 
urllib.request.urlopen() function contains all the 
HTTP headers the

server sent back. download the actual data by calling 
response.read()

49、The primary interface to 
httplib2 is the 
Http object.Once you have an Http object, retrieving data is as simple as calling the 
request() method with the address of the data you want. This will issue an 
HTTP GET request for that URL . 

The request() method returns two values. The first is an 
httplib2.Response object, which contains all

the 
HTTP headers the server returned. For example, a status code of 200 indicates that the request was

successful.

 The content variable contains the 
actual data that was returned by the HTTP server. The data is returned

as a bytes object, not a string. If you want it as a string, you’ll need to determine the character encoding

and convert it yourself.

you should always create an 
httplib2.Http object with a directory name. Caching is the reason.

httplib2 allows you to add arbitrary HTTP headers to any outgoing request. In order to bypass all caches

(not just your local disk cache, but also any caching proxies between you and the remote server), add a 

no-cache header in the headers dictionary.

HTTP defines 
Last-Modified and
 Etag headers for this purpose. These headers are called 
validators. If the local cache is no longer fresh(expired), a client can send the validators with the next request to see if the data has actually changed. If the data hasn’t changed, the server sends back a 304 status code and no data. 

which caused httplib2 to look in its cache.

httplib2 sends the ETag validator back to the server in the 
If-None-Match header.

 httplib2 also sends the Last-Modified validator back to the server in the 
If-Modified-Since header.

50、Python comes with a utility function to URL -encode a dictionary: urllib.parse.
urlencode()

Store your username and password with the
 add_credentials() method. 

51、If Python sees an 
__init__.py file in a directory, it assumes that all of the files in that directory are part of the same 
module. The module’s name is the name of the directory. Files within the directory can reference other files within the same directory, or even within subdirectories.  But the entire collection of files is presented to other Python code as a single module — as if all the functions and classes were in a single .py file.

The __init__.py file doesn’t need to define anything; it can literally be an empty file. Or you can use it to define your main entry point functions. Or you put all your functions in it. Or all but one.

A directory with an __init__.py file is always treated as a 
multi-file module.

Without an __init__.py file, a directory is just a directory of unrelated .py files.

52、Within lists, tuples, sets, and dictionaries, whitespace can appear before and after commas with no ill effects.

53、In Python 2, the global 
file() function was an alias for the open() function, which was the standard way of opening text files for reading. In Python 3, the global file() function no longer exists, but the open() function still exists.

54、In Python 2, a string was an array of bytes whose character encoding was tracked separately. If you wanted Python 2 to keep track of the character encoding, you had to use a Unicode string 
(u'') instead. But in Python 3, a string is always what Python 2 called a Unicode string — that is, an array of Unicode characters

(of possibly varying byte lengths). 

摘自《dive into python3》