6
votes

I have an ARFF file containing 14 numerical columns. I want to perform a normalization on each column separately, that is modifying the values from each colum to (actual_value - min(this_column)) / (max(this_column) - min(this_column)). Hence, all values from a column will be in the range [0, 1]. The min and max values from a column might differ from those of another column.

How can I do this with Weka filters?

Thanks

4

4 Answers

8
votes

This can be done using

weka.filters.unsupervised.attribute.Normalize

After applying this filter all values in each column will be in the range [0, 1]

1
votes

That's right. Just wanted to remind about the difference of "normalization" and "standardization". What mentioned in the question is "standardization", while "normalization" assumes Gaussian distribution and normalizes by mean, and standard variation of each attribute. If you have an outlier in your data, the standardize filter might hurt your data distribution as the min, or max might be much farther than the other instances.

0
votes

Here is the working normalization example with K-Means in JAVA.

final SimpleKMeans kmeans = new SimpleKMeans();

final String[] options = weka.core.Utils
        .splitOptions("-init 0 -max-candidates 100 -periodic-pruning 10000 -min-density 2.0 -t1 -1.25 -t2 -1.0 -N 10 -A \"weka.core.EuclideanDistance -R first-last\" -I 500 -num-slots 1 -S 50");
kmeans.setOptions(options);

kmeans.setSeed(10);
kmeans.setPreserveInstancesOrder(true);
kmeans.setNumClusters(25);
kmeans.setMaxIterations(1000);

final BufferedReader datafile = new BufferedReader(new FileReader("/Users/data.arff");
Instances data = new Instances(datafile);

//normalize
final Normalize normalizeFilter = new Normalize();
normalizeFilter.setInputFormat(data);
data = Filter.useFilter(data, normalizeFilter);

//remove class column[0] from cluster
data.setClassIndex(0);
final Remove removeFilter = new Remove();
removeFilter.setAttributeIndices("" + (data.classIndex() + 1));
removeFilter.setInputFormat(data);
data = Filter.useFilter(data, removeFilter);

kmeans.buildClusterer(data);

System.out.println(kmeans.toString());

// evaluate clusterer
final ClusterEvaluation eval = new ClusterEvaluation();
eval.setClusterer(kmeans);
eval.evaluateClusterer(data);
System.out.println(eval.clusterResultsToString());

If you have CSV file then replace BufferedReader line above with below mentioned Datasource:

final DataSource source = new DataSource("/Users/data.csv");
final Instances data = source.getDataSet();
0
votes

In this case, we can use weka.filters.unsupervised.attribute.Normalize filter to normalize but if we want to normalize only some columns the following will be the best approach.

To apply normalize on selected columns

The unsupervised.attribute.PartitionedMultiFilter can be used for this task.
Thereby you have to configure the filters and ranges sections as per your need.
For Ex: If I want to normalize only on humidity attribute

Step 01 :
After adding the ParririonedMultiFilter -> Tap on filter text box -> choose Normalize from weka.filters.unsupervised.attribute.Normalize -> And edit the Normalize filter as of your need(by giving the scale and translation values)

enter image description here

Step 02:
Tap on ranges text box -> Delete the default filter( which is first-last) -> Then add the column number you want to filter -> Click ok -> Click on Apply
enter image description here

Now the filter will be added only to the selected(humidity) column.