I know this question is a little old, but since I came to it with the same problem, I'm going to leave this here for future reference.
The solution I found was to manually compile OpenSSL with the installed cross-compiler and then manually install it to the cross-compilation libraries folder.
First, I installed the cross-compiler (I'm using Ubuntu 14.04). I installed both the C compiler and the C++ compiler. I also installed two cross-compiler toolchains, one with hard-floating point support (arm-linux-gnueabihf) and one without (arm-linux-gnueabi). Two directories are created (as noted on the question) /usr/arm-linux-gnueabi and /usr/arm-linux-gnueabihf, where the cross-compiled libraries should be installed to.
sudo apt-get install {gcc,g++}-arm-linux-gnueabi{,hf}
Secondly, cloned the OpenSSL Git repository and checked out the version I was interested in (1.0.2):
git clone https://github.com/openssl/openssl
git checkout OpenSSL_1_0_2 # or another version
Then, I configured the environment for cross-compilation and changed the installation directory (the prefix), and built the library following the instructions provided in the INSTALL file (and by forcing to use the specific cross-compilation toolchain):
export CROSS=arm-linux-gnueabi # or arm-linux-gnueabihf
export AR=${CROSS}-ar
export AS=${CROSS}-as
export CC=${CROSS}-gcc
export CXX=${CROSS}-g++
export LD=${CROSS}-ld
./Configure --prefix=/usr/${CROSS} os/compiler:${CC}
make
sudo make install
You can repeat the process and compile with both toolchains (arm-linux-gnueabi and arm-linux-gnueabihf).
Hope this helps.
