Hive factorial UDF

Question

I'm trying to find the factorial of a number in Hive. There's currently no Hive function to do that, so I've tried to write my own. This is my code:

package com.guy.hive.udf;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.LongWritable;
import org.apache.commons.math3.util.ArithmeticUtils;


public final class Factorial extends UDF {

public LongWritable evaluate(final LongWritable s){
        int n = (int) s.get();
        int fact = (int) ArithmeticUtils.factorial(n);
        return new LongWritable(fact);
    }
}

When I run this Hive query:

select factorial(c) from (select count(*) as c from test_table) ;

I get the exception:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.LongWritable com.vm.hive.udf.Factorial.evaluate(long)  on object com.vm.hive.udf.Factorial@37483748 of class com.vm.hive.udf.Factorial with arguments {39514210:java.lang.Long} of size 1

Can anyone help with this?

Stacktrace:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.LongWritable com.vm.hive.udf.Factorial.evaluate(org.apache.hadoop.io.LongWritable)  on object com.vm.hive.udf.Factorial@5faa5faa of class com.vm.hive.udf.Factorial with arguments {39514210:org.apache.hadoop.io.LongWritable} of size 1
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1030)
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
        at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:80)
        at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1052)
        at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1077)
        ... 10 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1006)
        ... 19 more
Caused by: org.apache.commons.math3.exception.MathArithmeticException: arithmetic exception
        at org.apache.commons.math3.util.ArithmeticUtils.factorial(ArithmeticUtils.java:317)
        at com.vm.hive.udf.Factorial.evaluate(Factorial.java:50)
        ... 24 more

[EDIT 1 - added imports to Java code.]

[EDIT 2 - added StackTrace

I'll try, but the parameter passed by Hive is a long: 39514210:java.lang.Long — Guy Needham
I'm interested in the answer to this as well, actually. It's gotten my curiosity. The error message does not seem to give much info. Is there no other message, stack trace, or Caused By clause? — Kon
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public org.apache.hadoop.io.LongWritable com.vm.hive.udf.Factorial.evaluate(org.apache.hadoop.io.LongWritable) on object com.vm.hive.udf.Factorial@1c6f1c6f of class com.vm.hive.udf.Factorial with arguments {39514210:org.apache.hadoop.io.LongWritable} of size 1 — Guy Needham
That's what I got with the LongWritable. There is a stack trace, I'll post it in the question. — Guy Needham

Kon Kon · Accepted Answer · 2014-11-20T16:31:29

I see your problem. The problem is not in Hive but in the ArithmeticUtils factorial method. See that it throws a MathArithmeticException? According to the docs, this will come when "the result is too large to be represented by a long."

This must be what's happening in your case. Try passing in a smaller number to the method.

Also, note that the factorial method is deprecated. The docs recommend using the CombinatoricsUtils.factorialLog(int) method instead.

Hive factorial UDF

1 Answers