C4.5算法

C4.5算法是对ID3算法的改进，在决策树的生成过程中，使用了信息增益率作为属性选择的方法，其具体的算法步骤如下：

输入：训练数据集D，特征集A，阈值e

输出：决策树T

1.如果D中所有实例属于同一类C，则置T为单结点树，并将C作为该结点的类，返回T

2.如果A=∅，则置T为单结点树，并将D中实例数最大的类C作为该结点的类，返回T

3.否则，计算A中各特征对D的信息增益率，选择信息增益率最大的特征Ak

4.如果Ak的信息增益率小于阈值e，则置T为单结点树，并将D中实例数最大的类C作为该结点的类，返回T

5.否则，对Ak的每一个可能值ai，依Ak=ai将D分割为子集若干非空Di，将属性Ak作为一个结点，其每个属性值ai作为一个分支，分别构建子结点，由结点及其子结点构成树T，返回T

6.对结点i，以Di为训练集，以A−{Ak}为特征集，递归地调用步骤(1)∼(5)得到子树Ti，返回Ti

通过上述算法步骤可以发现，ID3算法和C4.5算法步骤基本一致，唯一的变化就是，在第四步时将ID3算法中的信息增益，改成了C4.5算法中的信息增益率。其他步骤两种算法完全一致。

代码实现

# 加载数据
def loadDataSet(dataPath):
    dataset = []
    with open(dataPath) as file:
        lines = file.readlines()
        for line in lines:
            values = line.strip().split(' ')
            dataset.append(values)
    return dataset  

# 根据属性值，分割数据集
def splitDataSet(dataset, axis, value):
    retDataSet = []
    for featVec in dataset:
        if featVec[axis] == value:
            reducedFeatVec = featVec[:axis]
            reducedFeatVec.extend(featVec[axis+1:])
            retDataSet.append(reducedFeatVec)
    return retDataSet  

# 计算数据集的信息熵
def calShannonEnt(dataset):
    numEntries = len(dataset) * 1.0
    labelCounts = dict()
    for featVec in dataset:
        currentLabel = featVec[-1]
        if currentLabel not in labelCounts.keys():
            labelCounts[currentLabel] = 0
        labelCounts[currentLabel] += 1
    shannonEnt = 0.0
    for key in labelCounts:
        prob = labelCounts[key] / numEntries
        import math
        shannonEnt -= prob * math.log(prob, 2)
    return shannonEnt  

# 计算分割后的数据集相较于原数据集的信息增益
def InfoGain(dataset, axis, baseShannonEnt):
    featList = [example[axis] for example in dataset]
    uniqueVals = set(featList)
    newShannonEnt = 0.0
    numEntries = len(dataset) * 1.0
    for value in uniqueVals:
        subDataSet = splitDataSet(dataset, axis, value)
        ent = calShannonEnt(subDataSet)
        prob = len(subDataSet) / numEntries
        newShannonEnt += prob * ent
    infoGain = baseShannonEnt - newShannonEnt
    return infoGain  

# 计算属性的分裂信息值
def SplitInfo(dataset, axis):
    numEntries = len(dataset) * 1.0
    labelsCount = dict()
    ent = 0.0
    for featVec in dataset:
        value = featVec[axis]
        if value not in labelsCount:
            labelsCount[value] = 0
        labelsCount[value] += 1
    for key in labelsCount:
        prob = labelsCount[key] / numEntries
        import math
        ent -= prob * math.log(prob, 2)
    return ent  

# 计算属性的信息增益率
def GainRate(dataset, baseset, axis, baseShannonEnt):
    infoGain = InfoGain(dataset, axis, baseShannonEnt)
    splitInfo = SplitInfo(baseset, axis)
    return infoGain / splitInfo

# 根据信息增益率，来选择属性
def ChooseBestFeatureByGainRate(dataset, baseset):
    numFeature = len(dataset[0]) - 1
    baseShannonEnt = calShannonEnt(dataset)
    bestGainRate = 0.0
    bestFeature = -1
    for i in range(numFeature):
        gainRate = GainRate(dataset, baseset, i, baseShannonEnt)
        if gainRate > bestGainRate:
            bestGainRate = gainRate
            bestFeature = i
    return bestFeature

# 构建决策树
def createTree(dataset, baseset, labels):
    classList = [example[-1] for example in dataset]
    if classList.count(classList[0]) == len(classList):
        return classList[0]
    if len(dataset[0]) == 1:
        return majorityCnt(classList)
    bestFeature = ChooseBestFeatureByGainRate(dataset, baseset)
    bestFeatureLabel = labels[bestFeature]
    myTree = {bestFeatureLabel:{}}
    del(labels[bestFeature])
    featValues = [example[bestFeature] for example in dataset]
    uniqueVals = set(featValues)
    for value in uniqueVals:
        subLabels = labels[:]
        myTree[bestFeatureLabel][value] = \
         createTree(splitDataSet(dataset, bestFeature, value), baseset, subLabels)
    return myTree