Wrap a stream of raw H264 NALUs into a container like MP4

Question

I have an application that sends raw h264 NALUs as generated from encoding on the fly using x264 x264_encoder_encode. I am getting them through plain TCP so I am not missing any frames. I need to be able to decode such a stream in the client using Hardware Acceleration in Windows (DXVA2). I have been struggling to find a way to get this to work using FFMPEG. Perhaps it may be easier to try Media Foundation or DirectShow, but they won't take raw H264. I either need to:

Change the code from the server application to give back an mp4 stream. I am not that experienced with x264. I was able to get raw H264 by calling x264_encoder_encode, by following the answer to this question: How does one encode a series of images into H264 using the x264 C API? How can I go from this to something that is wrapped in MP4 while still being able to stream it in realtime
I could at the receiver wrap it with mp4 headers and feed it into something that can play it using DXVA. I wouldn't know how to do this
I could find another way to accelerate it using DXVA with FFMPEG or something else that takes it in raw format.

An important restriction is that I need to be able to pre-process each decoded frame before displaying it. Any solution that does decoding and displaying in a single step would not work for me

I would be fine with either solution

The beauty and the speed of DXVA basically assume that video is presented to user and not read back into RAM from video memory. Be prepared that reading from video memory might appear unexpectedly slow and killing the whole idea of fast hardware assisted decoding for purposes other than presentation. — Roman R.
Yeah... I think I hace to accept that fact now. Is there anyway to preprocess the frames at some stage with DXVA. Perhaps writing a media foundation filter or something like that? — cloudraven

sipsorcery sipsorcery · Accepted Answer · 2012-05-07T04:58:59

I believe you should be able to use H.264 packets off the wire with Media Foundation. there's an example on page 298 of this book http://www.docstoc.com/docs/109589628/Developing-Microsoft-Media-Foundation-Applications# that use a HTTP stream with Media Foundation.

I'm only learning Media Foundation myself and am trying to do a similar thing to you, in my case I want to use H.264 payloads from an RTP packet, and from my understanding that will require a custom IMFSourceReader. Accessing the decoded frames should also be possible from what I've read since there seems to be complete flexibility in chaining components together into topologies.

Wrap a stream of raw H264 NALUs into a container like MP4

1 Answers