AF-XDP: How do I get `ctx->data_meta` from kernel into user-space?

Question

I want to measure packet latency for my AF-XDP program. I was looking at this reference: https://github.com/xdp-project/xdp-project/blob/master/areas/arm64/xdp_for_tsn.org

and adapted it to this:

SEC("xdp_sock")
int xdp_sock_prog(struct xdp_md *ctx) {

    int index = ctx->rx_queue_index;

    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;

    const unsigned long long kstamp = bpf_ktime_get_ns();

    if(data - sizeof(unsigned long long) <= data_end) {
        ctx->data_meta = ctx->data - sizeof(unsigned long long);
        memcpy(&ctx->data_meta, &kstamp, sizeof(unsigned long long));
    }
    ...
}

Access in user-space then happens like this:

static bool process_packet(struct xsk_socket_info *xsk, uint64_t addr, uint32_t len) {

    uint8_t *pkt = xsk_umem__get_data(xsk->umem->buffer, addr);
    uint8_t *pkt_meta = xsk_umem__get_data(xsk->umem->buffer, addr - sizeof(unsigned long long));

    const unsigned long long kstamp = (uint64_t) pkt_meta[7] << 56 | (uint64_t) pkt_meta[6] << 48 | (uint64_t) pkt_meta[5] << 40
                                        | (uint64_t) pkt_meta[4] << 32 | (uint64_t) pkt_meta[3] << 24 | (uint64_t) pkt_meta[2] << 16
                                        | (uint64_t) pkt_meta[1] << 8  | (uint64_t) pkt_meta[0];
...
}

But I am not able to load the XDP-program into the kernel:

Load XDP program...
libbpf: load bpf program failed: Permission denied
libbpf: -- BEGIN DUMP LOG ---
libbpf: 
0: (bf) r6 = r1
1: (61) r1 = *(u32 *)(r6 +16)
2: (63) *(u32 *)(r10 -4) = r1
3: (61) r7 = *(u32 *)(r6 +4)
4: (61) r8 = *(u32 *)(r6 +0)
5: (85) call bpf_ktime_get_ns#5
6: (bf) r1 = r8
7: (07) r1 += -8
8: (2d) if r1 > r7 goto pc+3
 R0_w=inv(id=0) R1_w=pkt(id=0,off=-8,r=0,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=pkt_end(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0 fp-8=mmmm????
9: (63) *(u32 *)(r6 +8) = r0
invalid bpf_context access off=8 size=4
processed 10 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

libbpf: -- END LOG --

I mean, this shouldn't be possible - right? Because XDP is all about valid memory accesses and why would an access outside the defined borders data and data_end ever be valid?

Imho it could work if ctx->data_meta is left unchanged - but then I have the problem in user space because I don't know where data_meta is located. Is there any libbpf helper-function to gain access to packet meta data?

It could also be that the access is valid but my range check is wrong...

I might misunderstand, but on a super-quick glance, I can't see how you could write to ctx->data - <something> before adding headroom first, as this would be before the beginning of packet data and therefore out of range indeed. Note that the code you used as an example is described as “pseudo-code”, not necessarily something complete/functional. Does that help? — Qeole
I didn't know about headroom - what is that? How can I add it? — binaryBigInt
See this helper that can use room before data, e.g. to push new encapsulation headers. Note I didn't have time to read your question in details and I don't know if this is what you need at all. — Qeole
I tried this example: github.com/xdp-project/net-next/blob/master/samples/bpf/… but bpf_xdp_adjust_meta always fails (returning < 0). I guess my device driver doesn't support this? — binaryBigInt
Quite possible. The helper can return -EACCES, -EINVAL or -ENOTSUPP. The last case would indicate that your driver does not support XDP metadata. — Qeole

Qeole Qeole · Accepted Answer · 2020-03-03T11:09:49

The snippet you used is described in the tutorial as pseudo-code, it is not a complete and functional example. Let's have a look at the verifier log:

Load XDP program...
libbpf: load bpf program failed: Permission denied
libbpf: -- BEGIN DUMP LOG ---
libbpf: 
0: (bf) r6 = r1                 // r6 points to ctx
1: (61) r1 = *(u32 *)(r6 +16)   // [let's ignore that]
2: (63) *(u32 *)(r10 -4) = r1   // [and that]
3: (61) r7 = *(u32 *)(r6 +4)    // r7 points to ctx->data_end
4: (61) r8 = *(u32 *)(r6 +0)    // r8 points to ctx->data
5: (85) call bpf_ktime_get_ns#5 // r0 now contains timestamp
6: (bf) r1 = r8                 // r1 points to ctx->data
7: (07) r1 += -8                // r1 = data - sizeof(unsigned long long)
8: (2d) if r1 > r7 goto pc+3    // if(data-sizeof(unsigned long long)>data_end) jump;

Now for the fun part. We try to do:

        ctx->data_meta = ctx->data - sizeof(unsigned long long);
        memcpy(&ctx->data_meta, &kstamp, sizeof(unsigned long long));

And it produces:

 R0_w=inv(id=0) R1_w=pkt(id=0,off=-8,r=0,imm=0) R6_w=ctx(id=0,off=0,imm=0)
 R7_w=pkt_end(id=0,off=0,imm=0) R8_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0 fp-8=mmmm????
9: (63) *(u32 *)(r6 +8) = r0
invalid bpf_context access off=8 size=4

r6 points to ctx of type struct xdp_md, so r6 +8 is ctx->data - sizeof(unsigned long long). This is because, by default, ctx->data_meta points to ctx->data. So when you try to copy the timestamp to ctx->data_meta, you actually try to write it 8 bytes before the beginning of packet data. You don't have any specific space reserved there to put the metadata, so this is an out-of-bound access: the verifier complains that this is invalid.

But how can we create allocate some room for metadata, then? This is done thanks to the bpf_xdp_adjust_meta() BPF helper, as in the sample you referred to.

However, not all NIC drivers support XDP metadata. Some of them call function xdp_set_data_meta_invalid() internally, that sets ctx->data_meta to ctx->data + 1 instead of ctx->data. If bpf_xdp_adjust_meta(), called from the BPF program, detects that this adjustment has been performed, it errors out with -ENOTSUPP and refuses to adjust ctx->data_meta. Based in the discussion in the comments, it seems to be what is happening in your case.

If you do not have support for XDP metadata, you could try to encode the timestamp in another field of your packet. BPF maps might also be an option, but you would need one entry per packet, so not sure of the impact on memory/performance.

AF-XDP: How do I get `ctx->data_meta` from kernel into user-space?

1 Answers