Hope it should not work correctly, or you may have some issues.
It is not a good practice to use the lower version of a java driver to use against a higher version of mongo.
Always use the right version of java driver for the mongo version.
Also, You can use it in other way, (ie) you can use a higher version of a java driver for a lower version of mongo, but ensure that backward compatibility is provided for the mongo version.
Also Having different versions of Mongo for Primary and Secondary you will be facing the backward compatibility issues.
References:
http://docs.mongodb.org/master/release-notes/2.6-compatibility/
http://docs.mongodb.org/manual/release-notes/3.0-compatibility/