Which reinforcement algorithm to use for binary classification

Question

I am new to machine learning, but I've read a lot about Reinforcement Learning in the past 2 days. I have an application that fetches a list of projects (e.g. from Upwork). There is a moderator that manually accepts or rejects a project (based on some parameters explained below). If a project is accepted, I want to send a project proposal and if it is rejected, I'll ignore it. I am looking to replace that moderator with AI (among other reasons) so I want to know which Reinforcement Algorithm should I use for this.

Parameters: Some of the parameters that should decide whether the agent accepts or rejects the project are listed below. Assuming I only want to accept projects related to web development (specifically backend/server-side) here is how the parameters should influence the agent.

Sector: If the project is in related to IT sector it should have more chances of being accepted.
Category: If the project is in the Web Development category it should have more chances of being accepted.
Employer Rating: Employers having a rating of over 4 (out of 5) should have more chances of being accepted.

I thought Q-Learning or SARSA would be able to help me out but most of the examples that I saw were related to Cliff Walking problem where the states are dependent on each other which is not applicable in my case since each project is different from the previous one.

Note: I want the agent to be self-learning so that if in the future I start rewarding it for front-end projects too, it should learn that behavior. Therefore, suggesting a "pure" supervised learning algorithm won't work.

Edit 1: I would like to add that I have data (sector, category, title, employer rating etc.) of 3000 projects along with whether that project was accepted or rejected by my moderator.

If there is no dependence in states then you are not looking for an RL solution. Reinforcement Learning tackles sequential decision problems. Despite the final remark, you are in fact looking for supervised learning solution (or at the most - a bandit problem). — lejlot
@lejlot I looked into bandit problem, I think contextual bandit is more valid for my case since the state should also be taken into consideration before rewarding (towardsdatascience.com/…). But if I do implement CB, will the bandits be the different projects or should the state be a project? — Mujeeb
@lejlot Additionally, over time the bandit problem stores the success probability for each of its bandit, how does this map on my case? — Mujeeb

Rui Nian Rui Nian · Accepted Answer · 2018-08-15T16:24:27

your problem should easily be able to be solved using Q-learning. It just depends on how you design your problem. Reinforcement learning itself is a very robust algorithm that allows an agent to receive states from an environment, and then perform actions given those states. Depending on those actions, it will get rewarded accordingly. For your problem, the structure will look like this:

State

States: 3 x 1 matrix. [Sector, Category, Employer Rating]

The sector state are all integers, where each integer represents a different sector. For example, 1 = IT Sector, 2 = Energy, 3 = Pharmaceuticals, 4 = Automotives, etc.

The category state can also be all integers, where each integer represents a different category. Ex: 1 = Web Development, 2 = Hardware, 3 = etc.

Employer rating is again all integers between 1 - 5. Where the state represents the rating.

Action

Action: Output is an integer.

The action space would be binary. 1 or 0. 1 = Take the project, 0 = Don't take the project.

Reward

The reward provides feedback to your system. In your case, you would only evaluate the reward if the action = 1, i.e., you took the project. This will then allow your RL to learn how good of a job it did taking the project.

Reward would be a function that looks something like this:

def reward(states):
    sector, category, emp_rating = states
    rewards = 0
    if sector == 1:   # The IT sector
        rewards += 1

    if category == 1:   # The web development category
        rewards += 1

    if emp_rating = 5:   # Highest rating
        rewards += 2
    elif emp_rating = 4:   # 2nd highest rating
        rewards += 1

    return rewards

To enhance this reward function, you can actually give some sectors negative rewards, so the RL will actually receive negative rewards if it took those projects. I avoided that here to avoid the further complexity.

You can also edit the reward function in the future to allow your RL to learn new things. Such as making some sectors better than others, etc.

edit: yes, regards to lejlot's comment, it basically is a multi-armed bandit problem, where there is no sequential decision making. The setup of the bandit problem is basically the same as Q-learning minus the sequential part. All your concerned with is you have a project proposal (state), make a decision (action), and then your reward. It does not matter what happens next in your case.

Which reinforcement algorithm to use for binary classification

1 Answers