Move an iterator with a reference into a thread

Question

I have a type that represents a file. For simplicity lets say the type holds a buffer with the contents of the file.

There is also a method for building an iterator which holds a reference to the internal buffer.

The file-type is created on the main thread, but I need to "send" the iterator into the thread.

Let me show what I am trying to do

struct FileType {...}

let mut my_file_type = FileType::new("some_filename");
let mut my_iterator = my_file_type.iter();

external_library_object.start_process(move|_| {
    for _ in (0..10) {
        println!(my_iterator.next().unwrap());
    }
}

external_library_object (which by the way is cpal's device) is... well an object from a library I can't change. The closure parameter implements Send trait though.

This doesn't work because my_file_type doesn't live long enough. So I tried a combination of Mutex (so the iterator is mutable) and Arc(so we share the iterator between the 2 threads)

...

let mut my_file_type = FileType::new("some_filename");
let mut my_iterator = Arc::new(Mutex::new(my_file_type.iter()));

external_library_object.start_process(move|_| {
    let mut cloned = my_iterator.clone();
    for _ in (0..10) {
        println!(cloned.lock().unwrap().next().unwrap());
    }
}

But again, this doesn't work. I get an error similar to this:

xx |     let mut my_iterator = Arc::new(Mutex::new(my_file_type.iter()));
   |                                               ^^^^^^^^^^^^-------
   |                                                   |
   |                                                   borrowed value does not live long enough
   |                                                   argument requires that `my_file_type` is borrowed for `'static`
...
xx | }
   | - `my_file_type` dropped here while still borrowed

I am kind of stuck here. Is there a way to use the iterator which has a reference to other object(a buffer in this case) inside a different thread?

Edit: In my specific case the itertor is infinite and the closure executes multiple times. That is the reason I can't move the entire FileType inside the thread.

You could share the file between the threads rather than the iterator. Or make the iterator consume the file. — harmic
The problem with that is that the closure executes multiple times. Actually you just made me see the sample code is not as accurate. Let me update that — dospro
The issue you're running into is that my_iterator holds a reference to my_file_type, but my_file_type gets dropped at the end of the main thread and my_iterator might still exist in the secondary thread. You need the Arc(Mutex()) on my_file_type because that's the underlying data. You'll be forced to hold the mutex for as long as the iterator exists in either thread. — ddulaney

Aloso Aloso · Accepted Answer · 2021-01-02T22:14:29

This is tricky, because the process borrows an iterator, which in turn borrows a struct. Putting just the iterator into an Arc<Mutex<>> is not sufficient, because this pointer might still outlive the FileType it borrows. However, we can't put both the FileType and the iterator into a reference-counted struct, since that struct would then be self-referential.

The easiest solution to this that I can think of (short of using scoped threads) is to create an iterator that owns the FileType, and then put it into a Arc<Mutex<>> (minimal playground example):

let my_file_type = FileType::new("test!");
let my_iterator = Arc::new(Mutex::new(my_file_type.into_iter()));

external_library_object.start_process(move |_| {
    for _ in 0..10 {
        println!("{:?}", my_iterator.lock().unwrap().next().unwrap());
    }
});

You just have to implement an owned iterator for your FileType struct, if it doesn't exist yet.

Move an iterator with a reference into a thread

1 Answers