(disclaimer: I have never actually done this, but I see no technical reason it cannot be done. So YMMV)
If the decoder is a DMO filter, then it'll be a lot easier--you can talk to it through IMediaObject. This isn't really much different from how DirectShow uses DMOs; it simply wraps the DMO with another transform filter that handles the media type negotiation and sample passing, but there's nothing really stopping you from doing this in your own application.
There's one catch: for IMediaObject::ProcessInput and IMediaObject::ProcessOutput, you'll need your own buffer class that implements IMediaBuffer. But it's a pretty basic interface, so I don't think you'll have too much trouble implementing it. Here's a basic implementation.
For regular directshow filters, it's actually going to be a lot more difficult, because most DirectShow filters really depend on there being an external graph available (case in point: all of the directshow eventing sort of presumes the existence of this graph). If you really want to use a single DShow filter standalone, you'd probably have to wrap the entire filter graph, and then have a custom source filter to feed samples in. You could use the sample grabber (or a custom render filter) to yank samples out of the graph and expose to the rest of the application. (one sort of crazy idea would even be to wrap this graph in a DMO filter implementation, and then use IMediaObject to talk to it--this might be tricky, however)
Luckily most decoders tend to be implemented as DMO filters, so I think there's a strong likelihood that you can just use IMediaObject.