I'm looking for a C++ library (for Linux, but preferably cross-platform) that will easily let me read audio files in some trivial format such as a raw byte stream, generate audio data in the same trivial format, and write audio files. I don't care what format the reading and writing happens in, as long as it's free (beer and speech) and commonly supported. I'm going to be reading, transforming, and generating many small audio clips in a very parallel program, so I need the library to be fast and thread-safe.
Additional features I'd like, but do not require, are simple DSP functions (FFT, filters), and recording/playback through the sound card.
A bit of Googling didn't turn anything up, but perhaps I'm using the wrong search terms. I've almost exclusively found libraries for use in apps that record or playback, and it's unclear if they have the ability to generate sound from raw bytes, and even if they do they seem like overkill for my purpose. I've been considering just writing my own library to manipulate WAV files, since they seem simple enough, but I'd rather not reinvent the wheel if I can avoid it.