I'm almost sure that you can't do this in C#. I haven't been current with media stuff lately, but as of a few years ago, you had to do all of the video conversion stuff using Media Foundation (MF) in C++.
Media Foundation Programming Guide
I tried (and only half-succeeded) to make an MKV codec back when one didn't exist and that's what I had to turn to. These codecs are basically black boxes and MF will automatically pick the codec required for decoding video.
For writing your own codec (or making an existing codec work with MF), you'll have to turn to this tutorial. The problem is it uses WRL instead of the newer C++/WinRT. But oh, well.
Also, if you want to see my terrible attempt to make an MKV codec, go take a look here: https://github.com/limefrogyank/MKVSource
I was/am a novice at C++ and a complete noob at MF, too. But it worked... kind of.
Edited Feb 2, 2019
Just to be sure, have you seen this? Apparently, you can transcode video much more easily now using built-in tools:
https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/MediaTranscoding
But to incorporate your own codec, you can follow this tutorial to combine something you create using MF with UWP.
https://docs.microsoft.com/en-us/cpp/windows/wrl/walkthrough-creating-a-windows-store-app-using-wrl-and-media-foundation?view=vs-2017
