Running DPDK C program without root privileges

2
votes

I have C application that is using DPDK 19.11. Currently, the application is running with root permissions (using sudo command). In addition my application is running with huge pages (1GB).

Network devices using DPDK-compatible driver:

0000:02:00.0 'Ethernet Controller X710 for 10GbE backplane 1581' 
             drv=igb_uio unused=

I would like to run my application without the root permissions - get rid from "sudo" command. I change permission for those files/folders:

  • /sys/class/uio/uio*/device/resource*
  • /sys/class/uio/uio*/device/config
  • /dev/uio*
  • /dev/hugepages/*

when I run my application without "sudo"- I run in a problems with rte_eal_init function. I got this error:

EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available

My OS is Ubuntu 18.04, kernel 4.15.0-128-generic I noticed that at DPDK docs there is a remark about Running DPDK Applications Without Root Privileges - "since version 4.0, the kernel does not allow unprivileged processes to read the physical address information from the pagemaps file, making it impossible for those processes to be used by non-privileged users. In such cases, using the VFIO driver is recommended."

After reading comments I tried to use vfio-pci. I load the module using:

sudo modprobe vfio-pci enable_unsafe_noiommu_mode=1

I also changed permission for /dev/hugepages/* and /dev/vfio/* running with vfio-pci and sudo was successfully. when running without sudo i got the same error:

EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available

See also: https://doc.dpdk.org/guides/linux_gsg/enable_func.html#running-dpdk-applications-without-root-privileges

I wonder if someone has experience to run DPDK application without root with kernel 4.0 and above?

Also, as an alternative solution is to launch simple DPDK application with root privileges that will init DPDK. In parallel run another application without root privileges - this application will consume the packet and perform the business logic, is it possible?

thanks

2
dpdk
as I recollect I have run DPDK application on Linux Kernal 3.9 and 4.12 using DPDK 18.11 LTS and 19.11 LTS. I am not able to find any issues that you have faced or listed here. So have you tried running with necessary changes? The alternative solution you have mentioned will not work as long as you using Huge pages. Note: I am not recommending to use syscall to hide or skip sudo privellege checks to run as non privelleged user. - Vipin Varghese
are there any updates from your end, especially the error logs? I have to assume you have already modified huge page permission, NIc driver use, RUNTIME envormen. SO please share the specific error you have got while running. I am not marking the ticket as insufficent details yet - Vipin Varghese
As usal I notice https://stackoverflow.com/users/13121879/useme-alehosaini marking No Action Required. I humbly request useme-alehosain to justify why logs and DPDK log-level=8 is not required for debugging the issue? As mentioned in my first comment I have been able to run without sudo with DPDK 18.11 and 19.11. - Vipin Varghese
are there any updates from your end? - Vipin Varghese

2 Answers

1
votes

First, it makes sense to check if you really need to use the unsafe mode with vfio-pci. Perhaps you just need to add intel_iommu=on iommu=pt to the kernel parameters for making the device work safely, i.e.:

modprobe vfio-pci

I haven't used the unsafe mode so far, perhaps the kernel even unconditionally disallows mappings for the vfio device, if unsafe mode is enabled, for (obvious?) security reasons.

For running a dpdk application without root privileges you need to adjust the permissions of the right vfio device. For example, when the permissions look like this

# ls -l /dev/vfio/
total 0
crw-------. 1 root root 235,   0 2021-08-21 15:13 17
crw-rw-rw-. 1 root root  10, 196 2021-08-21 15:13 vfio

then /dev/vfio/17 is the device you've bound for dpdk, thus adjust its permission like this:

chown juser /dev/vfio/17

A user process doesn't need extra permission for mapping huge pages. You don't even have to mount the hugetblfs, if you supply the --in-memory option to your dpdk program.

However, some hugepages must be reserved by root, e.g. during system boot. Example:

echo 4096 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 8 >  /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages

Besides permissions, the default resource limits likely are too low. Especially, the memlock ones. If it's too low kernel logs something like this when starting the dpdk application:

kernel: vfio_pin_pages_remote: RLIMIT_MEMLOCK (65536) exceeded

And the dpdk application prints:

EAL:   cannot set up DMA remapping, error 12 (Cannot allocate memory)
EAL:   0000:05:00.1 DMA remapping failed, error 12 (Cannot allocate memory)

Increasing the limits fixes this issue, e.g.:

cat /etc/security/limits.d/24-memlock.conf                                         
# memlock unit: KiB
juerr hard memlock 16777216
juser soft memlock 1048576
0
votes

DPDK should detect if you should use IOVA VA or PA. Using the switch enable_unsafe_noiommu_mode=1 is telling DPDK that you have no iommu and that you will use IOVA PA.

The problem is, that running in PA mode requires root privileges as you need access to the physical address.

That dpdk.org document you cited should do the trick. I was able to get DPDK running without root privileges in 20.02 in a docker container. However, there was another problem with the software we were running on top of DPDK and its interaction with the hugepage backing.

In the end, we decided to still run DPDK as root, however, we limited the capabilities of the container to the bare minimum set needed to run DPDK.