You should check out SFML (Simple Fast Media Library). Although SDL is much more well-known due to its older age, SFML is far richer functionality- and design-wise, despite being only a few years old. It can essentially be described as a newer, more modern, and object-oriented SDL (it's C++; SDL is C).
To give you a brief overview of its features:
- Cross-platform (Windows, Mac OS X,
and Linux).
- Has modules for system-related stuff
(events, timers, etc), windowing,
video (hardware-accelerated OpenGL 2D
graphics), audio, and networking.
- Supports all essential and mainstream
image and audio file formats, with
the notable exception of MP3 due to
licensing issues (although this is
available as an extension).
- Also supports fonts (FreeType2 is
integrated) and many other things.
I use it for all my projects that need more than just a command line and I love it. I highly recommend it.
It should be able to provide a good base for your idea.