使用weka进行Cross-validation实验

Generating cross-validation folds (Java approach)

文献：

http://weka.wikispaces.com/Generating+cross-validation+folds+%28Java+approach%29

This article describes how to generate train/test splits for cross-validation using
the Weka API directly.

The following variables are given:

Instances data =  ...;   // contains the full dataset we wann
create train/test sets from

int seed = ...;          // the seed for
randomizing the data

int folds = ...;         // the number of
folds to generate, >=2

 Randomize the data

First, randomize
your data:

Random rand = new Random(seed);   // create seeded number generator

randData = new
Instances(data);   // create copy of
original data

randData.randomize(rand);         // randomize data
with number generator

In case your data
has a nominal class and you wanna perform stratified cross-validation:

randData.stratify(folds);

 Generate the folds

 Single run

Next thing that we
have to do is creating the train and the test set:

for
(int n = 0; n < folds; n++) {

Instances train = randData.trainCV(folds, n);

Instances test = randData.testCV(folds, n);

// further
processing, classification, etc.

...

}

Note:

• the above code is used by the weka.filters.supervised.instance.StratifiedRemoveFolds filter
• the weka.classifiers.Evaluation class and the Explorer/Experimenter
  would use this method for obtaining the train set:

Instances train = randData.trainCV(folds, n, rand);

 Multiple runs

The example above
only performs one run of a cross-validation. In case you want to run 10 runs of
10-fold cross-validation, use the following loop:

Instances data = ...;  // our dataset again, obtained from
somewhere

int runs = 10;

for
(int i = 0; i < runs; i++) {

seed = i+1;  // every run gets a
new, but defined seed value

// see:
randomize the data

...

// see: generate
the folds

...

}

一个简单的小实验：

继续对上一节中的红酒和白酒进行分类。分类器没有变化，只是增加了重复试验过程

package assignment2;

import weka.core.Instances;

import weka.core.converters.ConverterUtils.DataSource;

import weka.core.Utils;

import weka.classifiers.Classifier;

import weka.classifiers.Evaluation;

import weka.classifiers.trees.J48;

import weka.filters.Filter;

import weka.filters.unsupervised.attribute.Remove;

import java.io.FileReader;

import java.util.Random;

public class cv_rw {

    public static Instances getFileInstances(String filename) throws Exception{

       FileReader frData =new FileReader(filename);

       Instances data = new Instances(frData);

       int length= data.numAttributes();

       String[] options = new String[2];

       options[0]="-R";

       options[1]=Integer.toString(length);

       Remove remove =new Remove();

       remove.setOptions(options);

       remove.setInputFormat(data);

       Instances newData= Filter.useFilter(data, remove);

       return newData;

    }

    public static void main(String[] args) throws Exception {

        // loads data and set class index

       Instances data = getFileInstances("D://Weka_tutorial//WineQuality//RedWhiteWine.arff");

//     System.out.println(instances);

       data.setClassIndex(data.numAttributes()-1);

        // classifier

//      String[] tmpOptions;

//      String classname;

//      tmpOptions     = Utils.splitOptions(Utils.getOption("W", args));

//      classname      = tmpOptions[0];

//      tmpOptions[0]  = "";

//      Classifier cls = (Classifier) Utils.forName(Classifier.class, classname, tmpOptions);

//

//      // other options

//      int runs  = Integer.parseInt(Utils.getOption("r", args));//重复试验

//      int folds = Integer.parseInt(Utils.getOption("x", args));

       int runs=1;

       int folds=10;

       J48 j48= new J48();

//     j48.buildClassifier(instances);

        // perform cross-validation

        for (int i = 0; i < runs; i++) {

          // randomize data

          int seed = i + 1;

          Random rand = new Random(seed);

          Instances randData = new Instances(data);

          randData.randomize(rand);

//        if (randData.classAttribute().isNominal())    //没看懂这里什么意思，往高手回复，万分感谢

//          randData.stratify(folds);

          Evaluation eval = new Evaluation(randData);

          for (int n = 0; n < folds; n++) {

            Instances train = randData.trainCV(folds, n);

            Instances test = randData.testCV(folds, n);

            // the above code is used by the StratifiedRemoveFolds filter, the

            // code below by the Explorer/Experimenter:

            // Instances train = randData.trainCV(folds, n, rand);

            // build and evaluate classifier

            Classifier j48Copy = Classifier.makeCopy(j48);

            j48Copy.buildClassifier(train);

            eval.evaluateModel(j48Copy, test);

          }

          // output evaluation

          System.out.println();

          System.out.println("=== Setup run " + (i+1) + " ===");

          System.out.println("Classifier: " + j48.getClass().getName());

          System.out.println("Dataset: " + data.relationName());

          System.out.println("Folds: " + folds);

          System.out.println("Seed: " + seed);

          System.out.println();

          System.out.println(eval.toSummaryString("=== " + folds + "-fold Cross-validation run " + (i+1) + "===", false));

        }

    }

}

运行程序得到实验结果：

=== Setup run 1 ===

Classifier:
weka.classifiers.trees.J48

Dataset:
RedWhiteWine-weka.filters.unsupervised.instance.Randomize-S42-weka.filters.unsupervised.instance.Randomize-S42-weka.filters.unsupervised.attribute.Remove-R13

Folds: 10

Seed: 1

=== 10-fold Cross-validation run
1===

Correctly Classified Instances        6415               98.7379 %

Incorrectly Classified
Instances        82                1.2621 %

Kappa statistic                          0.9658

Mean absolute error                      0.0159

Root mean squared error                  0.1109

Relative absolute error                  4.2898 %

Root relative squared error             25.7448 %

Total Number of Instances             6497