How to generate sample timestamps in an RTSP Source Filter?

Question

Actually, I'm already able to generate timestamps and it works with some filters (muxers) but since I want to be able to use GDCL MP4 Multiplexer Filter, I want to discuss my method of calculating sample time values.

Weird thing is, if I put Smart Tee Filter between my RTSP Source Filter and GDCL MP4 Multiplexer Filter everything seems to by working. I'm able to capture video streams correctly. But without Smart Tee Filter, i'm still able to capture video but this time with periodic glitches. As far as I know, Smart Tee Filter doesn't do much except giving double output pins, one with timestamps (capture pin) and one without timestamps (preview pin) and both pins share the same stream buffer. So, the thing came to my mind was somehow Smart Tee Filter is reorganizing things like timestamps. But if I don't generate timestamps, Smart Tee Filter doesn't generate those values, either. My current guess is, I'm calculating frame start times correctly but frame stop times incorrectly and Smart Tee Filter is recalculating frame stop times as they should be (this is just a guess of course).

I used to calculate start and stop times like the formula below.

startTime = now
timeDelta = now - previousFrame
endTime = startTime + timeDelta

This is not the exact formula but it was something close. And results were like below.

Media Time/Time: 0-1    0-0
Media Time/Time: 1-2    500028-1000056
Media Time/Time: 2-3    930053-1360078
Media Time/Time: 3-4    1610092-2290131
Media Time/Time: 4-5    2200126-2790160
Media Time/Time: 5-6    2900166-3600206
Media Time/Time: 6-7    3500200-4100234
Media Time/Time: 7-8    4240242-4980284
Media Time/Time: 8-9    4720270-5200298
Media Time/Time: 9-10   5350306-5980342
Media Time/Time: 10-11  5980342-6610378
Media Time/Time: 11-12  6610378-7240414
Media Time/Time: 12-13  7250414-7890450
Media Time/Time: 13-14  7880450-8510486
Media Time/Time: 14-15  8510486-9140522
Media Time/Time: 15-16  9140522-9770558
Media Time/Time: 16-17  9780559-10420596
Media Time/Time: 17-18  10410595-11040631
Media Time/Time: 18-19  11040631-11670667
Media Time/Time: 19-20  11680668-12320705
Media Time/Time: 20-21  12310704-12940740
Media Time/Time: 21-22  12940740-13570776
Media Time/Time: 22-23  13600778-14260816
Media Time/Time: 23-24  14220813-14840848
Media Time/Time: 24-25  14840849-15460885
Media Time/Time: 25-26  15480885-16120921
Media Time/Time: 26-27  16110921-16740957
Media Time/Time: 27-28  16740957-17370993
Media Time/Time: 28-29  17380994-18021031
Media Time/Time: 29-30  18011030-18641066
Media Time/Time: 30-31  18631065-19251100

These are Media Time and Time values (both in start-stop format) which are set by SetMediaTime() and SetTime() methods. The problem I've seen was some start/end times were overlapping in sequential frames. Of course that is caused by jitter factors etc. If I calculate the time delta depending on the previous frame and if next frame arrives earlier than expected, overlapping occurs. so, i made a little change in my code. my final code is like below.

        _FILETIME fileTime;
        GetSystemTimeAsFileTime(&fileTime);
        now = ((((__int64)fileTime.dwHighDateTime << 32) + fileTime.dwLowDateTime) - streamReader->rtpProtocolHandler->m_iReferenceTime);
        timeDelta = now - m_iFrameTimePrevious;
        m_iFrameTime = max(now, m_iFrameTime);
        m_iFrameTimePrevious = m_iFrameTime;
        REFERENCE_TIME rtStart = m_iFrameTime;
        REFERENCE_TIME rtStop;
        if(timeDelta > 0) {
            m_iFrameTime += timeDelta;
            rtStop = m_iFrameTime;
        } else {
            rtStop = rtStart;
        }
        pSample->SetTime(&rtStart, &rtStop);
        pSample->SetMediaTime(&m_iMediaTime, &(++m_iMediaTime));

And results are like this:

Media Time/Time: 0-1 0-0
Media Time/Time: 1-2 470027-940054
Media Time/Time: 2-3 940054-1380079
Media Time/Time: 3-4 1580091-2220128
Media Time/Time: 4-5 2220128-2810161
Media Time/Time: 5-6 2870164-3520200
Media Time/Time: 6-7 3520200-4110234
Media Time/Time: 7-8 4170239-4820278
Media Time/Time: 8-9 4820278-5420312
Media Time/Time: 9-10 5470313-6120348
Media Time/Time: 10-11 6120348-6720382
Media Time/Time: 11-12 6770387-7420426
Media Time/Time: 12-13 7420426-8060463
Media Time/Time: 13-14 8060463-8630494
Media Time/Time: 14-15 8630494-9140521
Media Time/Time: 15-16 9260530-9890566
Media Time/Time: 16-17 9890566-10490600
Media Time/Time: 17-18 10560604-11230642
Media Time/Time: 18-19 11230642-11840677
Media Time/Time: 19-20 11860679-12490716
Media Time/Time: 20-21 12490716-13090750
Media Time/Time: 21-22 13170754-13850792
Media Time/Time: 22-23 13850792-14450826
Media Time/Time: 23-24 14460827-15070862
Media Time/Time: 24-25 15070862-15680897
Media Time/Time: 25-26 15760902-16450942
Media Time/Time: 26-27 16450942-17060977
Media Time/Time: 27-28 17100978-17751014
Media Time/Time: 28-29 17751014-18171038
Media Time/Time: 29-30 18171040-18591066
Media Time/Time: 30-31 18771074-19371108

No overlapping this time but all the glitches remains. what am I doing wrong?

Please remember that my source filter works with other mp4 muxers and it also works with GDCL MP4 Muxer if I put Smart Tee filter in the middle.

Ralf Ralf · Accepted Answer · 2011-10-25T17:23:59

When using RTSP/RTP/RTCP I would recommend using the RTP timestamps for sample presentation times (offset to zero).

How can you be sure that each frame is exactly the same duration? In my experience, this has never been the case, whether using live sources or file-based media. Your use-case may of course differ.

Also, by calculating the timestamps you have no way of synchronising multiple RTP streams, say audio and video. That is one of the reasons for having RTCP sender reports.

How to generate sample timestamps in an RTSP Source Filter?

2 Answers