Cross-compiling Python outside of riscv-poky can be a bit tricky. By default, Python is built with its own copy of libffi that is bundled with the source distribution (in Modules/_ctypes/libffi). When building within poky, this libffi is overridden by the system's libffi (using the --with-system-ffi flag), which includes our libffi RISC-V port from this recipe:
https://github.com/riscv/riscv-poky/blob/master/meta-riscv/recipes-gnome/libffi/libffi_3.1.bb
It is likely that your build is trying to use the libffi that comes bundled with Python. The easiest solution might be to build Python within poky. However, if that is not an option, I would try to apply the libffi patch manually and see whether that solves the problem.