How do I return complex structures to Python when using Rust FFI?

Question

I am hoping to re-write some parts of a Python project in Rust to speed up things. The idea is to use Rust's FFI interface to connect to Python via ctypes. I am using Rust 1.10 nightly.

I need to return quite complex arrays/structures to Python. And this does not work properly. Unfortunately, I could not find any tutorial in the web for this kind of problem.

extern crate libc;
use libc::{c_int, c_char, size_t, c_float, c_double};
use std::ffi::{CStr, CString};
use std::str;
use std::vec::Vec;
use std::slice;
use std::mem;

#[repr(C)]
pub struct CPeak {
    // chemsc: *mut c_char, // adduct: c_char_p,
    mz_exp: c_float,
    mz_diff: c_float,
    intensity: c_float,
    resolution: c_float,
    noise: c_float,
    nb_frg: c_int,
    frgs: *mut CFrg,
}

#[repr(C)]
pub struct CFrg {
    mz_exp: c_float,
    mz_diff: c_float,
    intensity: c_float,
    resolution: c_float,
    noise: c_float,
}

#[no_mangle]
pub extern "C" fn standard_finder_get_standards(mut data: *mut *mut CPeak, mut len: *mut size_t) {
    // fill 'peaks' vector
    let peaks = find_standards(&standards.standards, "test.xml", 5.0, 10.0);

    // Copy contents of peaks (Rust structure) to CPeaks (C structure).
    // NOTE that CPeaks has a entry in the struct which is also a vector of
    // structs (frgs)
    let mut cpeaks: Vec<CPeak> = vec![];
    for peak in &peaks {

        // mk a vector frgs
        let mut frgs: Vec<CFrg> = vec![];
        if peak.frgs.len() > 0 {
            for frg in &peak.frgs {
                let f = CFrg {
                    mz_exp: frg.mz as c_float,
                    mz_diff: frg.mz_diff as c_float,
                    intensity: frg.intensity as c_float,
                    resolution: frg.resolution as c_float,
                    noise: frg.resolution as c_float,
                };
                frgs.push(f);
            }
        }
        frgs.shrink_to_fit();

        // mk a vector cpeaks
        cpeaks.push(CPeak {
            mz_exp: peak.mz as c_float,
            mz_diff: peak.mz_diff as c_float,
            intensity: peak.intensity as c_float,
            resolution: peak.resolution as c_float,
            noise: peak.resolution as c_float,
            nb_frg: peak.frgs.len() as c_int,
            frgs: frgs.as_ptr() as *mut CFrg, // <- add the frgs vector as c pointer (array)
        });
    }
    cpeaks.shrink_to_fit();
    unsafe {
        *data = cpeaks.as_ptr() as *mut CPeak;
        *len = cpeaks.len() as size_t;
    }
    mem::forget(cpeaks);
}

(Playground).

This code takes a Rust vector (peaks: Vec<Peak>) and copies its content to its C-structure array counterpart (cpeaks: Vec<CPeak>). Within CPeak is another pointer to an array of CFrg. The cpeaks vector is returned as call-by-reference to have space for an error return value.

When I try to read the data with Python I have two problems:

the first entry of cpeaks is empty or garbage
the CFrg entries are all garbage.

I guess that the problem is the lifetime of cpeaks which probably does not life long enough to be accessed in Python. However, I have no idea how to keep it alive. I tried Boxing it, but this did not help.

I think you also need to add mem::forget(frgs); after the cpeaks.push(...); line. — Dogbert
@Dogbert: This is true. Then it works, but cpeaks similar to frgs would be leaked because I would not be able to free it. — Ronny Herzog

9716278 9716278 · Accepted Answer · 2020-01-05T00:07:58

You can call rust code as a python module.

https://developers.redhat.com/blog/2017/11/16/speed-python-using-rust/

I'm not sure if this is what you were going for, but maybe it's worth a look.

How do I return complex structures to Python when using Rust FFI?

1 Answers