Ideal classifiers in python to fit sparse high dimensional features (with hierarchical classification)

Question

This is my task:

I have a set of hierarchical classes (ex. "object/architecture/building/residential building/house/farmhouse")--and I've written two ways of classifying:

treating each class independently (using one model/classifier overall)
using a tree where each node represents a decision (the root represents "object/", and each level decreases generality), and a specific model/classifier for each node (here, I consider the c (usually 3) highest probabilities that come out of each node, and propagate the probabilities down (summing the log probs) to the leaves), and choose the highest.

I also had to introduce a way to incentivize going further down the tree (as it could stop at say object/architecture/building (if there is the corresponding training data)), and used an arbitrary trial-and-error process to decide specifically how (I don't feel comfortable with this).:
```
    if numcategories == 4:
        tempscore +=1
    elif numcategories ==5:
        tempscore +=1.3
    elif numcategories ==6:
        tempscore +=1.5
    elif numcategories >6:
        tempscore +=2
```

It is also important to note that I have around 290k training samples and ~150k (currently/mostly) boolean features (represented with 1.0 or 0.0)--although it's highly sparse, so I use scipy's sparse matrices. Also, there are ~6500 independent classes (though many less for each node in method 2)

With method 1, with scikit's sgdclassifier(loss=hinge), I get around 75-76% accuracy, and with linearsvc, I get around 76-77% (although it's 8-9 times slower).

However, for the second method (which I think can/should ultimately perform better) neither of these classifiers produce true probabilities, and while I've attempted to scale the confidence scores produced by their .decision_functions(), it didn't work well (accuracies of 10-25%). Thus, I switched to logisticregression(), which gets me ~62-63% accuracy. Also, NB based classifiers seem to perform substantially less well.

Ultimately, I have twoish questions:

Is there a better classifier (than scikit's logisticregression()) around implemented in python (could be scikit or mlpy/nltk/orange/etc) that can (i) handle sparse matrices, (ii) produce (something close to) probabilities, and (iii) work with multiclass classification?
Is there some way to handle method two better? 2.a. Specifically, is there some way to better handle incentivizing the classifier to produce results further down the tree?

(1) Is there a better classifier -- have you tried optimizing the parameters for the scikit-learn ones? Standardizing features, normalizing samples and penalty parameters can have a great effect on classification accuracy. As for (2), perhaps some custom structured learner would do substantially better. — Fred Foo
Hi larsmans-- (1) yes, but I'd like to use something better than logistic regression (I had good luck with linearsvc (much better than with logit) before I realized I needed something close to "true" probabilities) (2) agreed — user2553999

Gal175 Gal175 · Accepted Answer · 2019-04-24T13:17:39

A few ideas you could try:

Apply some embedding technique on your features, so as to avoid a large sparse matrix. However, it doesn't fit every case and also requires a lot of work
Use XGBoost with a custom loss function. In this loss function you can basically apply the logic you've described regarding the depth of the predicted class and provide an incentive for the model to get deeper classes predicted more often. In addition, a tree-based model will benefit you by taking into consideration the correlations between your features

Ideal classifiers in python to fit sparse high dimensional features (with hierarchical classification)

1 Answers