theano has different results on CPU and GPU

Question

I wrote a Theano machine learning program. But I got two absolutely different results between on CPU and on GPU.

Below is the log.(only a tiny part of log)

The loss function will quickly decrease and then converg to 0.2 on CPU.

However, the loss function will increase and finally become NaN on GPU.

What mistakes may be in my program? Or what should I take into attention? Thank you!

Framester Framester · Accepted Answer · 2016-03-29T10:48:57

Could it be that CPU is using float64 (double precision) and the GPU is using float32 (single precision)? Here you can look up the configuration flags: http://deeplearning.net/software/theano/library/config.html