How to calculate precision and recall of a web service ranking algorithm?

Question

I want to calculate precision and recall of a web service ranking algorithm. I have different web services in a data base.

A customer specify some conditions in his/her search. According to the customer`s requirements, my algorithm should assign a score for each web service in data base and retrieve web services with highest scores.

I have searched the net and have read all the questions in this site about this topic, and know about precision and recall,but I dont know how to calculate them in my case. The most relevant search was in this link: http://ijcsi.org/papers/IJCSI-8-3-2-452-460.pdf

According to this article,

Precision = Highest rank score / Total rank score of all services

Recall= Highest rank score / Score of 2nd highest service

But, I think it is not true. Can you help me please?

Thanks a lot.

Answer 1

There is no such thing as "precision and recall for ranking". Precision and recall are defined for binary classification task and extended to multi label tasks. Rankings require different measures as this is much more complex problem. There are numerous ways to compute something similar to precision and recall, I will summarize some basic approaches for the precision, recall goes similarly:

• limit search algorithm to some K best results and count true positives as number of queries for which the desired results is in those K results. So precision is fraction of queries for which you can find relevant result in K best outputs
• very strict variation of the above, set K=1, meaning that results has to come "the best of all"
• assign weights to each position, so for example you can give 1/T "true positive" to each query where valid result vame T'th. In other words, if the valid result was not returned you assign 1/inf=0, if it was the first one on the list then 1/1=1, if second 1/2, etc. now precision is simply a mean of these scores

Answer 2

As lejlot pointed out, using "precision and recall for ranking" is weired to measure ranking performance. The definition of "precision" and "recall" is very "customized" in the referenced paper you pointed out.

It is a measure of the tradeoff between the precision and 
recall of the particular ranking algorithm. Precision is the  
accuracy of the ranks i.e. how well the algorithm has 
ranked the services according to the user preferences. 
Recall is the deviation between the top ranked service and 
the next relevant service in the list. Both these metrics are 
used together to arrive at the f-measure which then tests the 
algorithm efficiency. 

Probably the original author has some specific motivation to use such definition. Some usual metric for evaluating ranking algorithms include:

• Normalized discounted information gain or nDCG (used in a lot of kaggle competitions)
• Precision@K, Recall@K

This paper also listed a few common ranking measures.

Answer 3

This is what I could think of:

Recall could be fraction of getting user click for top 5 queries and precision could be getting the fraction of user getting the click in the first query when compared to rest of the queries. I don't know but it seems very vague to speak about precision and recall in such a scenario.