Vulkan multiple logical device from one physical device consequences

Question

When creating a logical device, we can specify multiple queue for one family queue index as we can see in the VkDeviceQueueCreateInfo, and we can pass an array of this struct in the VkDeviceCreateInfo.

So my first clue to, for instance, create a transfer queue and a graphics queue, is to use the same logical device with two different VkDeviceQueueCreateInfo at device creation.

But, can I create two logical devices from the same physical device with a different VkDeviceQueueCreateInfo (one for graphics, and one for transfer) ?

And if yes, what could be the benefit or the bad idea to make one or the other solution ?

"one for graphics, and one for transfer" Why would you want to do that? What problem are you looking to solve here? Do you plan to share memory between these devices? — Nicol Bolas
I want to implement the video memory transfer separated from graphics rendering. — Sébastien Bémelmans
In fact, I try to understand between the physical device, the logical device and a queue, where I can use multithreading (from the host). I'm pretty sure I must focus on queues to achieve this. — Sébastien Bémelmans
"I want to implement the video memory transfer separated from graphics rendering." That doesn't explain why you want to create multiple logical devices to do this. Is there some problem with using multiple queues (if available)? — Nicol Bolas
I don't want to specifically do multiple logical devices, but Vulkan seems to let it possible. And by doing my multithread design, I was wondering if I had to do this or not. — Sébastien Bémelmans

Nicol Bolas Nicol Bolas · Accepted Answer · 2021-02-27T16:41:36

Generally speaking, when assessing possible ways to do things in Vulkan, you should pick the way that seems to require the least stuff.

In this case, you're trying to select between multiple queues and multiple devices. Well, the multi-queue method obviously requires less stuff; you have one device with many queues (in theory), rather than many devices with many queues (one from each device). Same number of queues, but more devices. So pick the one with less.

The Vulkan API is not trying to trick you into taking the slower path. If using multiple devices with one queue per-device were the best option, then Vulkan wouldn't have multiple queues as an option at all.

To get into more detail, you say that you want to do memory transfers outside of graphics operations. OK, fine.

Physical devices do not have to provide multiple queues. Some devices provide exactly one queue family, which can have exactly one VkQueue created from it. Obviously if a device allows for multiple queues, you should use them. But if it only allows for one, then you might have reason to think that you should just create multiple devices and work that way.

Even in this case, do not do this.

Here's the thing: if a GPU could actually do multiple operations independently such that they overlap... they'd expose multiple queues. The fact that a physical device does not do so therefore suggests that independent execution of different operations simply is not possible at the GPU level.

This means that, even if you use multiple devices, the transfer and graphics operation will almost certainly execute in some order. That is, whichever vkQueueSubmit is issued first will be the one that does its work first.

So using multiple devices gives you no actual GPU execution overlap (in theory). You've gained nothing, and you've lost lost explicit control over the order in which these operations are issued.

Now, it may be that the execution of a transfer operation on the graphics queue will not inhibit the execution of rendering commands on that same queue. That is, transfer operations can start, then rendering commands can start while the transfer completes via DMA or something. So they start executing in order, but finish executing in any order.

Even if that's the case, working across devices doesn't give you any advantages here. As previously noted, you lose control over the order in which these commands are submitted. Graphics commands tend to hog the command queue, while a single transfer command could (on such a system) be processed and then execute in the background while processing unrelated commands. In such a case, it's important to send any transfer commands before graphics commands for a particular frame.

And if you have two devices, you have to have 2 vkQueueSubmit calls rather than one. And vkQueueSubmit calls are not known for being fast.

There are a host of other reasons not to try multi-device stuff for this. For example, if later rendering operations need access to the transferred data, this means you need external memory and external synchronization primitives to synchronize access between the devices. And so on.

Vulkan multiple logical device from one physical device consequences

1 Answers