No traffic emerges in DPDK RX/TX Callbacks Sample Application

Question

I am a freshman in the area of DPDK. I start with the sample applications given by the DPDK main page. I am stuck in the example --- DPDK RX/TX Callbacks Sample Application. The condition is that no traffic emerges (yet the output) for a long time. I guess the reason is that I am unable to send traffic to the DPDK port.

I first show my setup step by step. The nic info at the beginning

Then, I turn down eth2 and eth3 with

sudo ifconfig eth2 down
sudo ifconfig eth3 down

Next, using usertools/dpdk-setup.sh I build the DPDK source (i686-native-linuxapp-gcc), insert IGB UIO module, Setup hugepage mapping for non-NUMA systems, bind device to IGB UIO module. This is my device setting and hugepage list:

All preparation has done. I start to build the example application. Using the following commands

export RTE_SDK=~/dpdk/dpdk-18.08
export RTE_TARGET=i686-native-linuxapp-gcc
cd examples/rxtx_callbacks/
make

Everything is alright, and output compile info:

mininet@mininet-vm:~/dpdk/dpdk-18.08/examples/rxtx_callbacks$ make
  CC main.o
  LD rxtx_callbacks
  INSTALL-APP rxtx_callbacks
  INSTALL-MAP rxtx_callbacks.map

Then, I start to run the example application using:

sudo ./build/rxtx_callbacks -l1 -n4

The execute log is:

mininet@mininet-vm:~/dpdk/dpdk-18.08/examples/rxtx_callbacks$ sudo ./build/rxtx_callbacks -l1 -n4
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device 0000:00:08.0 on NUMA socket 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:09.0 on NUMA socket 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:0a.0 on NUMA socket 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:11.0 on NUMA socket 0
EAL:   probe driver: 8086:100f net_e1000_em
Port 0 MAC: 08 00 27 9d f9 5a
Port 1 MAC: 08 00 27 7f e8 8a

As time goes by, nothing happens. However, I find the execution result should be like this:

root@ubuntu:/home/chang/dpdk/examples/rxtx_callbacks/build# ./rxtx_callbacks -l 1 -n 4 
EAL: Detected 8 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.rte_unix
EAL: Probing VFIO support...
EAL: PCI device 0000:02:01.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:02:02.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:02:03.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
EAL: PCI device 0000:02:04.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:100f net_e1000_em
Port 0 MAC: 00 0c 29 f7 4d 25
Port 1 MAC: 00 0c 29 f7 4d 2f

Core 1 forwarding packets. [Ctrl+C to quit]
Latency = 629 cycles
Latency = 787 cycles
^C

I try to find the result, I add a printf in lcore_main() function as follows:

for (;;) {
        RTE_ETH_FOREACH_DEV(port) {
                struct rte_mbuf *bufs[BURST_SIZE];
                /* printf("receive a packet\n"); */
                const uint16_t nb_rx = rte_eth_rx_burst(port, 0,
                                bufs, BURST_SIZE);
                if (unlikely(nb_rx == 0))
                        continue;
                printf("send a packet\n");
                const uint16_t nb_tx = rte_eth_tx_burst(port ^ 1, 0,
                                bufs, nb_rx);
                if (unlikely(nb_tx < nb_rx)) {
                        uint16_t buf;

                        for (buf = nb_tx; buf < nb_rx; buf++)
                                rte_pktmbuf_free(bufs[buf]);
                }
        }
}

I find no packet has received, for it should pass the check --- if (unlikely(nb_rx == 0)). Therefore, I try to send packets by myself with pktgen provided by Linux (NOT DPDK). However fail. I will show my trail.

#!/bin/sh

modprobe pktgen

function pgset() {
    local result

    echo $1 > $PGDEV

    result=`cat $PGDEV | fgrep "Result: OK:"`
    if [ "$result" = "" ]; then
         cat $PGDEV | fgrep Result:
    fi
}

function pg() {
    echo inject > $PGDEV
    cat $PGDEV
}

# Config Start Here -----------------------------------------------------------


# thread config
# Each CPU has own thread. Two CPU example. We add eth1.

PGDEV=/proc/net/pktgen/kpktgend_0
  echo "Removing all devices"
 pgset "rem_device_all"
  echo "Adding eth1"
 pgset "add_device eth1"
  echo "Setting max_before_softirq 10000"
 pgset "max_before_softirq 10000"


# device config
# delay 0 means maximum speed.

CLONE_SKB="clone_skb 1000000"
# NIC adds 4 bytes CRC
PKT_SIZE="pkt_size 60"

# COUNT 0 means forever
#COUNT="count 0"
COUNT="count 10000000"
DELAY="delay 0"

PGDEV=/proc/net/pktgen/eth1
  echo "Configuring $PGDEV"
 pgset "$COUNT"
 pgset "$CLONE_SKB"
 pgset "$PKT_SIZE"
 pgset "$DELAY"
 pgset "dst 127.0.0.1"
 # here! the mac address given by the sample application
 pgset "dst_mac  08:00:27:9d:f9:5a" 

# Time to run
PGDEV=/proc/net/pktgen/pgctrl

 echo "Running... ctrl^C to stop"
 pgset "start"
 echo "Done"

# Result can be vieved in /proc/net/pktgen/eth1

Still, nothing happens. Maybe, the way to send packets to the dpdk port is wrong. Since dpdk ports are in layer 2, layer 3 tools, like ping, are useless. The sample application is stuck all the time, and I am desperate. Could you share some thoughts to solve my problem?

Best wishes.

EDIT1:

@Amedeo's reply works out, however, I am just able to send some packet (several packets not traffic flow) to RX/TX callback application. I have tried using pktgen provided by Linux, but the interface dtap0 cannot receive the traffic generated by pktgen. The following is my experiment course.

First, all preparation is same as above. However, I run the sample application with the command

 sudo ./build/rxtx_callbacks -l1 -n4 --vdev=net_tap0

rather than

 sudo ./build/rxtx_callbacks -l1 -n4

I can see a new nic item in ifconfig, after running the sample application.

Oh, I need to mention that I have turned down eth1 yet, such that eth1, eth2, and eth3 are for DPDK use. The execute output shows as follows:

EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: PCI device 0000:00:08.0 on NUMA socket 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:09.0 on NUMA socket 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:0a.0 on NUMA socket 0
EAL:   probe driver: 8086:100e net_e1000_em
EAL: PCI device 0000:00:11.0 on NUMA socket 0
EAL:   probe driver: 8086:100f net_e1000_em
rte_pmd_tap_probe(): Initializing pmd_tap for net_tap0 as dtap0
Port 0 MAC: 08 00 27 93 2b 19
Port 1 MAC: 08 00 27 9d f9 5a
Port 2 MAC: 08 00 27 7f e8 8a
Port 3 MAC: fe 02 cf 5a 23 ee

Core 1 forwarding packets. [Ctrl+C to quit]

The Port 3 with fe 02 cf 5a 23 ee is exactly the dtap0 nic in ifconfig. Now I try to inject traffic to dtap0 interface. However, I try to use pktgen provided by Linux, but still fail.

I first give the interace dtap0 an ip address using:

sudo dhclient dtap0

and get:

dtap0     Link encap:Ethernet  HWaddr fe:02:cf:5a:23:ee
          inet addr:10.0.2.16  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:73 errors:0 dropped:0 overruns:0 frame:0
          TX packets:92 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:6651 (6.6 KB)  TX bytes:7808 (7.8 KB)

Then, I write a shell script to generate traffic load from eth0 to dtap0(I wish). This is the script.

#!/bin/sh

#modprobe pktgen
modprobe pktgen

function pgset() {
    local result

    echo $1 > $PGDEV

    result=`cat $PGDEV | fgrep "Result: OK:"`
    if [ "$result" = "" ]; then
         cat $PGDEV | fgrep Result:
    fi
}

function pg() {
    echo inject > $PGDEV
    cat $PGDEV
}

# Config Start Here -----------------------------------------------------------


# thread config
# Each CPU has own thread. Two CPU exammple. We add eth1, eth2 respectivly.

PGDEV=/proc/net/pktgen/kpktgend_0
  echo "Removing all devices"
 pgset "rem_device_all"
  echo "Adding eth0"
 pgset "add_device eth0"
  echo "Setting max_before_softirq 10000"
 pgset "max_before_softirq 10000"


# device config
# delay 0 means maximum speed.

CLONE_SKB="clone_skb 1000000"
# NIC adds 4 bytes CRC
PKT_SIZE="pkt_size 60"

# COUNT 0 means forever
#COUNT="count 0"
COUNT="count 10000000"
DELAY="delay 0"

PGDEV=/proc/net/pktgen/eth0
  echo "Configuring $PGDEV"
 pgset "$COUNT"
 pgset "$CLONE_SKB"
 pgset "$PKT_SIZE"
 pgset "$DELAY"
 # ip address of interface dtap0
 pgset "dst 10.0.2.16"
 # the mac address of interface dtap0
 pgset "dst_mac  fe:02:cf:5a:23:ee"

# Time to run
PGDEV=/proc/net/pktgen/pgctrl

 echo "Running... ctrl^C to stop"
 pgset "start"
 echo "Done"

Running this script, I really found interface eth0 is producing outcoming packets (using nload or ifconfig). However, interface dtap0 has no incoming traffic. I also try iperf to send traffic, but dtap0 can handle neither TCP connection nor UDP ACK. iperf and ping are both unavailable.

Thanks for sharing any idea on my problem.

EDIT2:

Thanks to @Amedeo's share. I have figured out how to use pktgen provided by Linux to send packets to RX/TX Callback sample application.

All things for running RX/TX Callback sample application are same as EDIT1. I change the script for pktgen as follows:

#!/bin/sh

#modprobe pktgen
modprobe pktgen

function pgset() {
    local result

    echo $1 > $PGDEV

    result=`cat $PGDEV | fgrep "Result: OK:"`
    if [ "$result" = "" ]; then
         cat $PGDEV | fgrep Result:
    fi
}

function pg() {
    echo inject > $PGDEV
    cat $PGDEV
}

# Config Start Here -----------------------------------------------------------


# thread config
# Each CPU has own thread. Two CPU exammple. We add dtap0

PGDEV=/proc/net/pktgen/kpktgend_1
  # echo "Removing all devices"
 # pgset "rem_device_all"
  echo "Adding dtap0"
 pgset "add_device dtap0"
  echo "Setting max_before_softirq 10000"
 pgset "max_before_softirq 10000"


# device config
# delay 0 means maximum speed.

CLONE_SKB="clone_skb 1000000"
# NIC adds 4 bytes CRC
PKT_SIZE="pkt_size 60"

# COUNT 0 means forever
#COUNT="count 0"
COUNT="count 10000000"
DELAY="delay 0"

PGDEV=/proc/net/pktgen/dtap0
  echo "Configuring $PGDEV"
 pgset "$COUNT"
 pgset "$CLONE_SKB"
 pgset "$PKT_SIZE"
 pgset "$DELAY"
 # pgset "dst 192.168.211.101"
 # whatever mac address
 pgset "dst_mac ce:2a:23:42:ce:ff"

# Time to run
PGDEV=/proc/net/pktgen/pgctrl

 echo "Running... ctrl^C to stop"
 pgset "start"
 echo "Done"

What I need to do is only that send packets out of dtap0! And

dtap0 --- dpdk port --- rxtx_callbacks 

Everything goes right.

Answer 1

It is expected that you don't see traffic. The RX/TX Callbacks example is a generalization of the L2FWD example. This code receives traffic from one port and forwards it to the other. If there is no incoming traffic, the application is polling forever, trying to receive something and won't forward anything.

Since the 2 ports are using the DPDK drivers, they are not visible by the kernel. For this reason the usual tools (ifconfig, tcpdump, pktgen, etc.) don't see these interfaces.

You can bind to dpdk only one physical interface and use a tap interface as second interface. The tap interface is bound to the kernel, so you can easily inject traffic in it (e.g. using tcpreplay).

To use a tap interface you just need to add the parameter --vdev=net_tap0:

sudo ./build/rxtx_callbacks -l1 -n4 --vdev=net_tap0

this will create an interface dtap0 visible from ifconfig. All the traffic that enters this interface will be received by the rxtx_callbacks application and forwarded to the other physical port.

Source: https://doc.dpdk.org/guides/nics/tap.html

End-to-end experiment

You can also remove all the dpdk physical ports and use just 2 TAP ports. In this way you can have:

dtap0 --- dpdk port --- rxtx_callbacks --- dpdk port --- dtap1

and the 2 TAPs can ping each other through the rxtx_callbacks application.

The steps to reproduce this are:

1) Unbind all the physical interfaces from DPDK

2) Run rxtx_callbacks with 2 TAPs:

sudo ./rxtx_callbacks -l1 -n4 --vdev=net_tap0 --vdev=net_tap1

3) Now you have 2 TAP interfaces: dtap0 and dtap1. Move dtap1 to a different network namespace and assign IPs to dtap0 and dtap1:

sudo ip netns add ns1 
sudo ip link set dtap1 netns ns1
sudo ip netns exec ns1 ifconfig dtap1 11.0.0.2/24
sudo ifconfig dtap0 11.0.0.1/24

4) You can ping dtap1 from dtap0 (and vice versa):

sudo ping 11.0.0.2
sudo ip netns exec ns1 ping 11.0.0.1

5) Cleanup: move dtap1 to the default namespace and delete the namespace.

sudo ip netns exec ns1 ip link set dtap1 netns 1
sudo ip netns del ns1

Answer 2

The simplest solution which works great for a start is to use one host as a DPDK target and another host as a packet generator. Fast packet generator takes a lot of CPU resources, so running it on a DPDK target will definitely reduce the DPDK performance.

Sometimes another host is not available though, so the second simplest solution would be to connect back-to-back physical ethernet devices on the same host. In your case DPDK ports eth2 and eth3 should be physically connected using a crossover patch cord with pktgen ports eth0 and eth1.

In that case you will be able to use ping etc if you configure just one pktgen port with an ip address, i.e.:

# ifconfig eth2 down # for DPDK
# ifconfig eth3 down # for DPDK

# ifconfig eth0 172.16.0.1/24 up # for pktgen with an IP (TX)
# ifconfig eth1 up # for pktgen without IP (RX)

# arp -s 172.16.0.2 00:0c:29:f7:4d:25 # MAC of the connected DPDK port

Then the normal ping 172.16.0.2 should produce ICMP packets with DST IP 172.16.0.2, dst MAC 00:0c:29:f7:4d:25 and send them to eth0, which should be connected back-to-back to DPDK port.

To see the packets are arriving back on eth1, we should put the interface into a promiscuous mode. The tcpdump does so by default. So, if everything is up and running:

# tcpdump -eni eth1

should show the packets arriving back from DPDK. Once you have tested that, you can move to pktgen.

The solution with vdev described by @Amedeo should also work. You will not be able to push a lot of traffic through the vdevs though.