Write directly from a reader in Rust

Question

How does one stream data from a reader to a write in Rust?

My end goal is actually to write out some gzipped data in a streaming fashion. It seems like what I am missing is a function to iterate over data from a reader and write it out to a file.

This task would be easy to accomplish with read_to_string, etc. But my requirement is to stream the data to keep memory usage down. I have not been able to find a simple way to do this that doesn't make lots of buffer allocations.

use std::io;
use std::io::prelude::*;
use std::io::{BufReader};
use std::fs::File;
use flate2::read::{GzEncoder};
use flate2::{Compression};

pub fn gzipped<R: Read>(file: String, stream: R) -> io::Result<()> {
    let file = File::create(file)?;
    let gz = BufReader::new(GzEncoder::new(stream, Compression::Default));
    read_write(gz, file)
}

pub fn read_write<R: BufRead, W: Write>(mut r: R, mut w: W) -> io::Result<()> {
  // ?
}

Found a very similar question: How to read specific number of bytes from a stream. — Matthieu M.

Chris Emerson Chris Emerson · Accepted Answer · 2017-03-01T15:03:39

Your read_write function sounds exactly like io::copy. So this would be

pub fn gzipped<R: Read>(file: String, stream: R) -> io::Result<u64> {
    let mut file = File::create(file)?;
    let mut gz = BufReader::new(GzEncoder::new(stream, Compression::Default));
    io::copy(&mut gz, &mut file)
}

The only difference is that io::copy takes mutable references, and returns Result<u64>.

Write directly from a reader in Rust

1 Answers