编程Tips集锦

以下是自己编程的一些小贴士，记录，总结提高自己。

1.python中集合类型的查找，尽量用dict or set类型。

dict和set类型，在python内部的实现都是使用hash映射，查找的时间复杂度是O(1)，比任何的查找算法都高效。

当在程序中使用到>1K次的查询，就应该开始考虑使用dict或set类型来进行数据的组织。

 #coding:utf-8
 from urllib.request import urlopen
 from bs4 import BeautifulSoup
 import re
 import string
 import operator
 import datetime

 commonWords = ["the", "be", "and", "of", "a", "in", "to", "have", "it", "i", "that", "for", "you", "he", "with", "on", "do", "say", "this", "they", "is", "an", "at", "but","we", "his", "from", "that", "not", "by", "she", "or", "as", "what", "go", "their","can", "who", "get", "if", "would", "her", "all", "my", "make", "about", "know", "will","as", "up", "one", "time", "has", "been", "there", "year", "so", "think", "when", "which", "them", "some", "me", "people", "take", "out", "into", "just", "see", "him", "your", "come", "could", "now", "than", "like", "other", "how", "then", "its", "our", "two", "more", "these", "want", "way", "look", "first", "also", "new", "because", "day", "more", "use", "no", "man", "find", "here", "thing", "give", "many", "well"]
 #若不注释，则为set类型，跑一遍程序，对比一下，则知优劣！
 #commonWords = set(commonWords)

 def isCommon(word):
     global commonWords
     if word in commonWords:
         return True
     return False

 def cleanText(input):
     input = re.sub('\n+', " ", input).lower()
     input = re.sub('\[[0-9]*\]', "", input)
     input = re.sub(' +', " ", input)
     input = re.sub("u\.s\.", "us", input)
     input = bytes(input, "UTF-8")
     input = input.decode("ascii", "ignore")
     return input

 def cleanInput(input):
     input = cleanText(input)
     cleanInput = []
     input = input.split(' ')
     for item in input:
         item = item.strip(string.punctuation)
         if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'):
             cleanInput.append(item)

     cleanContent = []
     for word in cleanInput:
         if not isCommon(word):
             cleanContent.append(word)
     return cleanContent

 def getNgrams(input, n):
     input = cleanInput(input)
     output = {}
     for i in range(len(input)-n+1):
         ngramTemp = " ".join(input[i:i+n])
         if ngramTemp not in output:
             output[ngramTemp] = 0
         output[ngramTemp] += 1
     return output

 def getFirstSentenceContaining(ngram, content):
     #print(ngram)
     sentences = content.split(".")
     for sentence in sentences:
         if ngram in sentence:
             return sentence
     return ""

 content = str(urlopen("http://pythonscraping.com/files/inaugurationSpeech.txt").read(), 'utf-8')

 print('Use the set as the format of common words.')
 print('Begin:',datetime.datetime.now())
 for i in range(50):
     ngrams = getNgrams(content, 2)
     sortedNGrams = sorted(ngrams.items(), key = operator.itemgetter(1), reverse = True)
 print('End:',datetime.datetime.now())
 print(sortedNGrams)

2.python往数据库插入数据

在插入数据之前，记得先进行一次查询，查看数据是否已经在数据库中。

一可以使程序更健壮，二也可顺便避免二次查询。

3.数据库在建表的时候，最后有索引

最近需要往数据库中插入上百万级的数据，十万级以后之后，数据库变得极慢，磁盘读写也是爆满！

后来，发现查询次数太多，重新建表，顺便加入索引。特别是unique index，我猜背后的实现机制是hash映射。

加入索引之后的数据库，大大减轻了磁盘的负担，查询速度几乎恒定，不过数据库的增大还是降低了读写的速度（实属情理之中）。

3.python字符串中转义字符的处理

python中\t所占位为4位，不是通常的8位。