How to write http layer sniffer

Question

I want to write an application layer sniffer (SMTP/ftp/http).

Based on my searchs, first (and perhaps hardest!) step is to reassemble the tcp stream of the sniffed connections.

Indeed, what I need is something like the "follow TCP stream" option of wireshark, but I need a tool which do it on live interface and automatically. As I know, Tshark can extract TCP streams data from the saved pcap files automatically (link) but not from live interfaces. Can Tshark do it on live interfaces???

As I know, TCPflow can do exactly what I want, however, it can not handle IP defragmentation and SSL connections (I want to analyse the SSL content in the case I have the server private key).

Finally, I also try bro network monitor. Although it provides the list of TCP connections (conn.log), I was not able to get TCP connections contents.

Any suggestion about mentioned tools or any other useful tool is welcome.

Thanks in advance, Dan.

Answer 1

perl Net::Inspect library might help you. It also comes with a tcpudpflow which can write tcp and udp flows into separate files, similar to tcpflow. It works on pcap files or can do live captures. The library handles IP fragmenting. It also comes with a httpflow tool to extract HTTP requests and responses (including decompression, chunked encoding..). It does not currently handle SSL.

As the author of this library I don't think that extracting TCP flows is the hardest part, the HTTP parser (exluding decompression, including chunked mode) is nearly twice as big than IP and TCP combined.

Answer 2

This example works for reassembling application data of a single protocol:

tshark -Y "tcp.dstport == 80" -T fields -d tcp.port==80,echo -e echo.data

It captures live http data, reassembles it, and outputs it as raw hex.

I can add a small script to parse the hex into ascii if you like.

I want to analyse the SSL content in the case I have the server private key

TL;DR: This can't be done with a capturing tool alone.

Why not: Because each SSL session generates a new secret conversation key, and you can't decrypt the session without this key. Having the private server key is not enough. The idea behind this is that if someone captures your SSL traffic, saves it, and then a year later he "finds" the private server key, then he still won't be able to decrypt your traffic.