How to decode TCP/IP packets in python?

Question

I get a tick by tick market data stream in a IP Address with a specified port in a TCP/IP packet format continuously. I want to log that in a csv file or work around the data stream for trading and data analysis.

Server sends all the packets in following format

typedef struct
{
CHAR cCompOrNot
SHORT nDataSize;
SHORT iNoOfPackets;
}ST_COMP_BATCH_HEADER
typedef struct
{
SHORT iCode;
SHORT iLen;
LONG lSeqNo;
} ST_INFO_HEADER;
typedef struct
{
.
.
}ST_DATA_INFO;
typedef struct
{
SHORT iCheckSum;
CHAR cEOT;
} ST_INFO_TRAILER;
typedef struct
{
ST_INFO_HEADER stInfoHdr;
ST_DATA_INFO stDataInfo;
ST_INFO_TRAILER stInfoTrailer;
.
}ST_DATA_PACKET

All the packets received from server consist of compress batch header. Compress batch header gives the information about the data packet compressed or not, number of packets in the following data packet and the total size of data packet. Client needs to decompress the data packet using LZO decompression algorithm. After decompression each data packet consists of ST_INFO_HEADER, which has the iCode field to identify the type of the packet. Using iCode field, data info packet is mapped to the respective data packet.

So is there any way that I can decode the data in those packets using some Python Library and work around.

Thanks in advance!

Have you looked into scapy or pyshark? These questions are off-topic for SO, FYI. — Michael Foukarakis

Noah Noah · Accepted Answer · 2015-12-01T12:42:24

Use the socket library. Here is a sample of code to connect to the tcp/ip port and parse the data directly from Python's site (set HOST static to desired IP):

# Echo server program
import socket

HOST = ''                 # Symbolic name meaning all available interfaces
PORT = 50007              # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print('Connected by', addr)
while True:
    data = conn.recv(1024)
    if not data: break
    conn.sendall(data)
conn.close()

For Python 2.x:

https://docs.python.org/2/library/socket.html

For Python 3.x:

https://docs.python.org/3/library/socket.html

LZO Library for Python:

https://pypi.python.org/pypi/python-lzo/1.08

How to decode TCP/IP packets in python?

1 Answers