How to compile a DPDK application as a library

Question

I have a program using DPDK and I am compiling it using the Makefile provided in the examples.

If i compile the program as an APP (as describe here), all goes well. However, my code is part of a larger project, for which the use of a separate makefile causes a lot of troubles. So I bundled my code in a library, as described in the same page.

The program that calls the functions in the library (to initialize the EAL) is getting this error:

MBUF: error setting mempool handler
Cannot init mbuf pool

It seems that, when the application is compiled as a library, the EAL cannot be initialized correctly.

I report here the steps to reproduce the problem using the l2fwd example.

Background

I built DPDK from source as described here, and I have one ethernet interface bound to DPDK drivers:

$ $RTE_SDK/usertools/dpdk-devbind.py --status |head -n4
Network devices using DPDK-compatible driver
============================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe

Run the l2fwd example

Copy the example folder first and then run the example:

$ cp -r $RTE_SDK/examples/l2fwd $RTE_SDK/examples/l2fwd-lib/
$ cd $RTE_SDK/examples/l2fwd

$ make
CC main.o
LD l2fwd
INSTALL-APP l2fwd
INSTALL-MAP l2fwd.map

$ sudo ./build/l2fwd -l 0-3  -- -p 0x1
Port statistics ====================================
Statistics for port 0 ------------------------------
Packets sent:                        0
Packets received:                    0
Packets dropped:                     0
Aggregate statistics ===============================
Total packets sent:                  0
Total packets received:              0
Total packets dropped:               0
====================================================

It works!

Build the same example as a library

```
cd ../l2fwd-lib/
mv main.c l2fwd.c
```

Modify l2fwd.c, adding #include "l2fwd.h" on top and replacing

int main(int argc, char **argv)

with

int start(int argc, char **argv)

Create the header file l2fwd.h with the library interface:
```
int start(int argc, char **argv);
```

Modify the Makefile as described in the docs:

APP = l2fwd      --->   LIB = libl2fwd.a
SRCS-y := main.c --->   SRCS-y := l2fwd.c
include $(RTE_SDK)/mk/rte.extapp.mk   --->   include $(RTE_SDK)/mk/rte.extlib.mk

Compile the library:

$ make
CC l2fwd.o
AR libl2fwd.a
INSTALL-LIB libl2fwd.a

Write a program that uses the library. Create the main.c file with just these 2 lines:
```
#include "l2fwd.h"
int main (int argc, char **argv) { start(argc, argv); }
```

compile it (with all the needed libraries):

gcc -L build/lib/ -L $RTE_SDK/build/lib/ main.c -o main.o -l l2fwd -l dpdk -l numa -pthread -l dl

Finally, run it with the same parameters used earlier:

$ sudo ./main.o -l 0-3  -- -p 0x1
EAL: Detected 40 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
MAC updating enabled
EAL: Error - exiting with code: 1
  Cause: No Ethernet ports - bye

In this case, the EAL cannot be initialized properly (the ethernet port is still bound to the DPDK drivers).

Edit 1

According to @Andriy Berestovskyy, the linker --whole-archive option is needed when linking DPDK libraries. This solves the problem with the examples. However, my program is facing a different problem now. I need to use a custom build system, so I am linking the DPDK application as a library. During execution I get the error:

MBUF: error setting mempool handler
mempool/dpaa2: Not a valid dpaa2 buffer pool

Looks like it is using the dpaa2 driver, that is the wrong driver (my NIC is using igb_uio). Any hints on why this is happening? The same code worked when compiled as a DPDK app, so it is probably related to the linking process.

Edit 2

This error is due to the fact that I compiled DPDK with the CONFIG_RTE_BUILD_SHARED_LIB=y option. When DPDK is built as a shared library, the driver must be explicitly set using the -d EAL command line option. I re-compiled DPDK with CONFIG_RTE_BUILD_SHARED_LIB=n and the problem was solved.

Andriy Berestovskyy Andriy Berestovskyy · Accepted Answer · 2018-09-12T15:33:12

compile it (with all the needed libraries):

gcc -L build/lib/ -L $RTE_SDK/build/lib/ main.c -o main.o -l l2fwd -l dpdk -l numa -pthread -l dl

I guess the issue is in this step. DPDK uses link-level constructors (i.e. __attribute__((constructor))). See the definition of RTE_INIT() for example. Also there are callbacks etc. etc.

So to properly link with DPDK we have to:

either use an rte.app.mk in the Makefile (see Development Kit Build System)
or in case we need to use a custom build system, we need to link DPDK libraries after the --whole-archive option.

How to compile a DPDK application as a library

2 Answers