scikit-learn decision tree important features order and max-depth choice relation

Question

Why does the order of feature importances change according to the max depth choice in the decision tree classifier?

I used the synthetic data, but I didn't share the code because it is unnecessary and long. I just wonder the logic behind that when I change the maximum depth, why does the order of important features change.

dec_tree_clf = tree.DecisionTreeClassifier(max_depth=4, random_state=23, criterion="entropy")   
dec_tree_clf.fit(X_data, y_data)

features    importance  
    z   0.267464  
    n   0.124694  
    y   0.094134  
    c   0.090750  
    i   0.084806

dec_tree_clf = tree.DecisionTreeClassifier(max_depth=3, random_state=23, criterion="entropy")   
dec_tree_clf.fit(X_data, y_data)

features    importance  
    z   0.350545  
    n   0.163426  
    c   0.118939    
    i   0.111149  
    b   0.106650

CutePoison CutePoison · Accepted Answer · 2020-02-12T10:04:53

From here you can see that the Gini importance "counts the times a feature is used to split a node, weighted by the number of samples it splits".

If you have a feature which splits a lot in a large tree it will naturally have fewer splits in a short tree, thus might change the Gini importance (aka the Feature Importance (It is the same)). Also the splits might be different in a tree with depth=3 compared to a tree with depth=4which then also changes the Gini importance.

scikit-learn decision tree important features order and max-depth choice relation

1 Answers