Streaming video from DirectX to HTML5

1
votes

I have a dynamic animated DirectX scene showing on screen. I want to stream that sequence of images to the browser, for an HTML5 client.

I'm currently encoding each frame as jpeg/base64 and sending it over a websocket. In the browser, I'm replacing the img's source to that encoded frame and the image gets updated.

Local desktop browsers get real-time changes, but performance is not great on Android devices. I'm not sure if that's due to frequent DOM updates or Websocket latency, but the image is laggy.

I thought of optimizing it by creating a video stream to load in the browser, so that I'll get performance boost from 1. hardware video decoding; 2. avoiding DOM manipulation and 3. avoiding websocket overhead.

I'm not sure if that's the way to go, and how to implement it (video novice). Can anyone help?

2
javascriptandroidc++htmlvideo
You mentioned the current solution is laggy. Is this meant to be interactive? I.e., does the user control the 3D scene via the web browser? - Multimedia Mike
Yes, via UI controls on screen (in-browser). But the native code may also trigger changes to the scene without any user intervention. - liorda
What's your bandwidth limitation? Is this on a wired gigabit network? Or between computers on opposite ends of the earth? What browsers are you targeting? Any and all, or just, e.g., Chrome? - Multimedia Mike
There are no bandwidth limitations I'm aware of. The server is running Windows 7 and the tablet is Nexus 10 with Android 4.4. The tablet is connected to a wifi hosted network interface created by the server. I'm targeting Chrome on both the desktop and tablet. - liorda
I've been meaning to put together an answer for you this past week. However, it's a big topic, as the existing answers have already indicated. I want to point out that at least 2 companies (OnLive and Gaikai) have spent lots and lots of money to do the exact thing you're hoping to accomplish here (albeit over the broader internet, which is why I was asking about your LAN constraint). - Multimedia Mike

2 Answers

2
votes

lol - I fittled with this issue once and it took me ages to understand the following:

If you need streaming, do streaming but do not fittle with images for following reasons:

  • browsers are allowed to load as many images at the same time as they want. While this serves html-pages well you can end up with extremely desynchronized image sequences if you try to simulate streaming.
  • videos gain factor 2-100 compression by building images on each other which converts into: you need lots and lots of more bandwidth if you simulate streaming based on images
  • html is not optimized for streaming which add lots of extra code, if the user should get even close to any streaming experience
  • most of what you need for streaming is already there (such as protocols are supported by browsers, codecs are installed on clients, browsers deal with all sort of plugins that transmit data to codecs and to make use of directx and open gl)

Problem is to get the whole story is something to be covered by a book which would overshoot the format of this post - however here are some starting points:

1
votes

If you already have each frame encoded as jpeg/base64 you can easily create an mp4 using the free ffmpeg tool. All moderm browsers have support for h264, and most of them if not all, with hardware acceleration, so you can start there:

If your images are written like:

image001.jpeg, image002.jpeg, ...

To create an mp4 from the images:

ffmpeg \
   -r 1/10 \      <-- A different image each 10 seconds
   -i image%03d.jpeg \
   -c:v libx264 \
   -r 25 \        <-- 25 images per second
   -pix_fmt yuv420p \
   newVideo.mp4

Notice that ffmpeg also let you capture directly from the screen or framebuffer, a pipe or many other sources as well as encoding to different bitrates, screensizes, ... apply filters -for example mixing two videos-, add audio tracks, ..