linux rdma mellanox clarification

Question

I was going through the linux/drivers/net/ethernet/mellanox/mlx4/qp.c

Got a few questions. Would really appreciate if someone can clarify:

In the function, mlx4_qp_alloc_icm,

To allocate a QP, there are 2 paths taken:

1. using the ALLOC_RES virtual command

2. using the MAP_ICM

These paths are taken based on the return value of mlx4_is_func(dev). This is true for MASTER or SLAVE which I believe is Physical Function Driver/Virtual Function Driver. So for SRIOV, it covers all cases.

The MAP_ICM portion which gets executed as part of __mlx4_qp_alloc_icm never gets called??

Am I understanding it properly? Because as per my understanding ICM needs to be allocated for all the QPs.

Please help me in understanding this.

EDIT:

Adding relevant code:

320 static int mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn)
321 {
322         u64 param = 0;
323 
324         if (mlx4_is_mfunc(dev)) {
325                 set_param_l(&param, qpn);
326                 return mlx4_cmd_imm(dev, param, &param, RES_QP, RES_OP_MAP_ICM,
327                                     MLX4_CMD_ALLOC_RES, MLX4_CMD_TIME_CLASS_A,
328                                     MLX4_CMD_WRAPPED);
329         }
330         return __mlx4_qp_alloc_icm(dev, qpn);
331 }

http://lxr.free-electrons.com/source/drivers/net/ethernet/mellanox/mlx4/icm.c#L226

226 static int mlx4_MAP_ICM(struct mlx4_dev *dev, struct mlx4_icm *icm, u64 virt)
227 {
228         return mlx4_map_cmd(dev, MLX4_CMD_MAP_ICM, icm, virt);
229 }

Thanks so much.

Best Regards, Marc

Answer 1

The MAP_ICM command is privileged, and only the physical function driver is allowed to execute it. It is executed through __mlx4_qp_alloc_icm in the example quoted in the question when the driver is in "native" mode: neither handling a virtual function or a physical function that manages other virtual functions.

In SRIOV mode, when the driver manages a physical function that has some virtual functions, the __mlx4_qp_alloc_icm function is called only from within the resource tracker in resource_tracker.c. This module manages the allocation of ICM for all the functions it handles (virtual and physical) to prevent conflicts. The call to __mlx4_qp_alloc_icm specifically is in the qp_alloc_res function.