netfilter hook is not retrieving complete packet

Question

I'm writing a netfilter module, that deeply inspect the packet. However, during tests I found that netfilter module is not receiving the packet in full.

To verify this, I wrote the following code to dump packet retrieved on port 80 and write the result to dmesg buffer:

const struct iphdr *ip_header = ip_hdr(skb);
if (ip_header->protocol == IPPROTO_TCP)
{
    const struct tcphdr *tcp_header = tcp_hdr(skb);
    if (ntohs(tcp_header->dest) != 80)
    {
        return NF_ACCEPT;
    }

    buff = (char *)kzalloc(skb->len * 10, GFP_KERNEL);
    if (buff != NULL)
    {
        int pos = 0, i = 0;
        for (i = 0; i < skb->len; i ++)
        {
            pos += sprintf(buff + pos, "%02X", skb->data[i] & 0xFF);
        }

        pr_info("(%pI4):%d --> (%pI4):%d, len=%d, data=%s\n",
            &ip_header->saddr,
            ntohs(tcp_header->source),
            &ip_header->daddr,
            ntohs(tcp_header->dest),
            skb->len,
            buff
        );
        kfree (buff);
    }
}

In virtual machine running locally, I can retrieve the full HTTP request; On Alibaba cloud, and some other OpenStack based VPS provider, the packet is cut in the middle.

To verify this, I execute curl http://VPS_IP on another VPS, and I got the following output in dmesg buffer:

[ 1163.370483] (XXXX):5007 --> (XXXX):80, len=237, data=451600ED000040003106E3983D87A950AC11D273138F00505A468086B44CE19E80180804269300000101080A1D07500A000D2D90474554202F20485454502F312E310D0A486F73743A2033392E3130372E32342E37370D0A4163636570743A202A2F2A0D0A557365722D4167656E743A204D012000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000001E798090F5FFFF8C0000007B00000000E0678090F5FFFF823000003E00000040AE798090F5FFFF8C0000003E000000000000000000000000000000000000000000000000000000000000

When decoded, the result is like this

It's totally weird, everything after User-Agent: M is "gone" or zero-ed. Although the skb->len is 237, but half of the packet is missing.

Any ideas? Tried both PRE_ROUTING and LOCAL_IN, no changes.

Aleksey Aleksey · Accepted Answer · 2019-10-23T00:05:24

It appears that sometimes you are getting a linear skb, and sometimes your skb is not linear. In the latter case you are not reading the full data contents of an skb.

If skb->data_len is zero, then your skb is linear and the full data contents of the skb is in skb->data. If skb->data_len is not zero, then your skb is not linear, and skb->data contains just the the first (linear) part of the data. The length of this area is skb->len - skb->data_len. skb_headlen() helper function calculates that for convenience. skb_is_nonlinear() helper function tells in an skb is linear or not.

The rest of the data can be in paged fragments, and in skb fragments, in this order.

skb_shinfo(skb)->nr_frags tells the number of paged fragments. Each paged fragment is described by a data structure in the array of structures skb_shinfo(skb)->frags[0..skb_shinfo(skb)->nr_frags]. skb_frag_size() and skb_frag_address() helper functions help dealing with this data. They accept the address of the structure that describes a paged fragment. There are other useful helper functions depending on your kernel version.

If the total size of data in paged fragments is less than skb->data_len, then the rest of the data is in skb fragments. It's the list of skb which is attached to this skb at skb_shinfo(skb)->frag_list (see skb_walk_frags() in the kernel).

Please note that there may be that there's no data in the linear part and/or there's no data in the paged fragments. You just need to process data piece by piece in the order just described.

netfilter hook is not retrieving complete packet

1 Answers