OpenCL is normally used for processing large amounts of parallel operations(calculations etc). This is where OpenCL thrives and the throughput of your program is really increased by it.
Java is a general purpose language for writing applications running on various platforms/devices.
They have a different purpose as tools but you can use OpenCL from your Java program using JOCL. I also used a tool from AMD called Aparapi. because I wanted my application to carry out some calculations, a job that my ASUS AMD 6970 card seemed to get done faster than my 6 core AMD 1090T Phenom Black Edition.
Just for you to know what Aparapi is(taken from the google code site):
Aparapi allows Java developers to take advantage of the compute power
of GPU and APU devices by executing data parallel code fragments on
the GPU rather than being confined to the local CPU. It does this by
converting Java bytecode to OpenCL at runtime and executing on the
GPU, if for any reason Aparapi can't execute on the GPU it will
execute in a Java thread pool.
Despite so many similarities why and when should one prefer OpenCL
over Java for Heterogeneous Computing?
One would choose OpenCL for parallel processing tasks that would otherwise take a lot of time to be executed on the CPU. The disadvantage in this case would be the extra heat and power consumption costs. Bear also in mind that some tasks would execute faster on a CPU. It all depends on the application that most of the time would require trials on both types of chips in order to decide which one does the job best.
Why at all I chose OpenCL instead of Java?
One would choose OpenCL over Java for parallel processing oriented tasks. OpenCL is a tool built for this purpose. OpenCL is not something you would use for building enterprise or desktop applications. Java is a programming language for almost any domain running on many platforms. If one would know that using OpenCL would save them 24 hours of calculations over Java; of course Java would be sidelined and OpenCL would be the way to go or simply OpenCL could be used through a Java application.
If you say openCl has better performance then my immediate question
will be why it is so(since both generate intermediate code that need
to be converted into binaries for specific h/w).
It has better performance on specific applications since GPU have more cores than a CPU. Their frequency though is quite low so this means that they are better on parallel tasks rather than sequential. You have to bear in mind that OpenCL is used for basic calculations and not for building full flavor applications...at least as far as I know.