Performance improvement in Tunneling HTTP over TCP

Question

I have a design requirement which requires HTTP communication over underlying TCP networking module

[HTTP_CLIENT<--->LOCAL_TCP_PSEUDOSERVER]---------TCP/IP--------[LOCAL_TCP_PSEUDOSERVER<--->HTTP_SERVER]

HTTP cllient talks to local running TCP pseudoserver
This pseudoserver communicates to TCP pseudoserver which in turn talks to HTTP_SERVER

The overall functionality works fine as expected, but performance is too low.

After some research, i would few solutions, below are the same

a) HTTP is request response packets have small data packets. This data has to travel over TCP, which enables the Nagle algorithm by default So we applied TCP_NODELAY to disable the Nagle algorithm so that the small data packets are delivered to the remote host without delay.

b) Using HTTP keep-alive. This will make the TCP socket open so that another request can be made without setting up a new connection.

Updated query: Inshort, how does http proxy server work so efficeintly?

FYI: Here is the info I referred

Optimizing HTTP: Keep-alive and Pipelining

Design issues - Sending small data segments over TCP with Winsock

Network Performance Effects of HTTP/1.1, CSS1, and PNG

HTTP Pipelining – Big in Mobile

The above experiment improved the performance at one level, but I suppose there is huge scope of improvement. Is the approach correct? Also, what other parameters may impact performance?

Update 2

consider A-B-E-F. A-B , E-F are local loopback connection, where A is http client, B is pseudo server. E is pseudo server, F is HTTP server. B-E is simple TCP socket communication.

A-B, E-F communicates is loopback and reuses same port

int on = 1; setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)); bind(s, (struct sockaddr *) &sin, sizeof (sin));

Performance is overall slow between A-F

Please share your valuable inputs

Many Thanks

It would be nice if you could explain what the performance penalty of your approach is (are we talking 5% or 50%?), where the penalty lies and what code you suspect to cause the penalty. In its current form, your question basically says "throw me all research you can find on HTTP or TCP/IP slowness", which is as far as SO is concerned not a real question. — CodeCaster
@CodeCaster if we directly communicate (without tcp), it takes x amount of time to send Y data. But if we did through TCP, it took 4 times. after applying tcp_nodelay and keep-alive, now it takes 2.5 times. I am not asking to throw all ur research. Just wanted to have few pointers. — Rohit
What is the nature of these 'pseudo-servers'? Did you write them, or are they standard products? — user207421
we wrote them, these are very simple servers, just listening on the http port and sending data to other end. everything in same private network. — Rohit
"now it takes 2.5 times" - as I said, what is "it"? Are there high-latency, low-throughput connections in between? Where does the delay occur? — CodeCaster

user207421 user207421 · Accepted Answer · 2013-04-24T08:27:23

You can have N layers of proxying without enduring N times the latency. Clearly you have got the design and implementation of your 'pseudo-servers' severely wrong. There's no reason to write your own, and most people who try it seem to make a complete hash of it judging by what I see here and in other comparable forums.

These 'pseudo-servers' are really just HTTP proxies, and there's no reason why you shouldn't use a proper one, such as Apache Squid or even Apache HTTPD.

If you post some code you may get some detailed comments.

Performance improvement in Tunneling HTTP over TCP

1 Answers