Features and Feature importance in Auto-Sklearn with One Hot Encoded Features

Question

I am trying to train an XGBoost model using Auto-Sklearn.

https://automl.github.io/auto-sklearn/stable/

The model trains fine, however, I require feature importance to refine the model and for reporting purposes.

autosklearn.classification.AutoSklearnClassifier does not have a function that can do this for me.

I am trying to get the features and feature importance scores from the underlying pipeline.

I have tried a few things using the details given in the below GitHub Issues.

1) https://github.com/automl/auto-sklearn/issues/524

2) https://github.com/automl/auto-sklearn/issues/224

I have also tried using the 'Trace' python module. This returned over 900,000 lines of code. Don't know where to start.

My code is a work in progress but looks like:

import pandas as pd
import numpy as np
import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
import datetime
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve, auc
from sklearn import preprocessing
from sklearn.preprocessing import LabelEncoder
import eli5 as eli
import pdb
df = pd.read_csv('titanic_train.csv')
df_target = df['Survived']
drop_Attbr = ['PassengerId', 'Name', 'Ticket', 'Cabin','Survived','Sex','Embarked']
df_labels = df.drop(drop_Attbr,axis=1)
feature_types = ['categorical'] +['numerical']+(['categorical']* 2)+['numerical']
df_train, df_test, y_train, y_test = train_test_split(df_labels, df_target, test_size=1/3, random_state=42)
automl = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=15,
        per_run_time_limit=5,
        ensemble_size=1,
        disable_evaluator_output=False,
        resampling_strategy='holdout',
        resampling_strategy_arguments={'train_size': 0.67},
        include_estimators=['xgradient_boosting']
    )
automl.fit(df_train, y_train,feat_type=feature_types)
y_hat = automl.predict(df_test)
a_score = sklearn.metrics.accuracy_score(y_test, y_hat)
print("Accuracy score "+str(a_score))

I am looking for a result like:

Feature 1 : Feature Importance score 1;
Feature 2 : Feature Importance score 2;
Feature 3 : Feature Importance score 3;
Feature 4 : Feature Importance score 4;
Feature 5 : Feature Importance score 5;

There is one more library in pythoin 'auto_ml'. You can install it by using 'pip install auto_ml'. It has a argument 'ml_for_analytics' if set at true can provide you a report which you are looking for. — Aman Raparia
I am presently using 'auto_ml'. We want to move our pipeline to Auto-Sklearn. — Aiman Akheel

Venkatachalam Venkatachalam · Accepted Answer · 2019-01-04T16:14:25

Try this!

for identifier in automl._automl._automl.model_:
    if identifier in automl.ensemble_.get_selected_model_identifiers():
        model = automl._automl._automl.models_[identifier].pipeline_._final_estimator()
        print(model.get_score())

Features and Feature importance in Auto-Sklearn with One Hot Encoded Features

1 Answers