libpcap uses the functions pcap_compile and pcap_setfilter (described here with example code) to filter packets before starting the capture loop. The filter syntax is described in the man page for TCPDump
Filtering RTP packets presents a further challenge as they are not sent over a standard port and there is no other convenient method for detecting the protocol. The ports used for RTP are negotiated by the the control channel protocol (such as SIP) preceding the RTP setup. This example contains a SIP message where the line rport=5060 gives the UDP port used for RTP (see below).
Detection of RTP therefore requires inspecting the SIP protocol packets (or other signalling protocol to figure out the RTP ports to filter on.
SIP/2.0 200 OK
Call-ID: 29858147-465b0752@29858051-465b07b2
Contact: pel<sip:[email protected]:5060;line=7d36558f31367051>;q=0.500;expires=1200
CSeq: 6 REGISTER
From: <sip:[email protected]>;tag=659abf
P-Associated-URI: <sip:[email protected]>
To: <sip:[email protected]>;tag=00-04087-1701bae7-76fb74995
Via: SIP/2.0/UDP 192.168.1.2;received=80.230.219.70;rport=5060;branch=z9hG4bKnp6658824-465059f1192.168.1.2