Vulkan semaphores across queue families?

Question

I have two command buffers that I submit to two different queues from two different queue families:

When I submit them like this:

{
  VkSubmitInfo submitInfo = VK_STYPE;
  submitInfo.commandBufferCount = 1;
  submitInfo.pCommandBuffers = &commandBuffer;
  submitInfo.signalSemaphoreCount = 1;
  submitInfo.pSignalSemaphores = &semaphore;
  vkQueueSubmit(TransferQueue, 1, &submitInfo, VK_NULL_HANDLE);
}
{
  VkPipelineStageFlags waitDstStageMask = VK_PIPELINE_STAGE_ALL_COMMANDS_BIT;
  VkSubmitInfo submitInfo = VK_STYPE;
  submitInfo.commandBufferCount = 1;
  submitInfo.pCommandBuffers = &commandBuffer2;
  submitInfo.waitSemaphoreCount = 1;
  submitInfo.pWaitSemaphores = &semaphore;
  submitInfo.pWaitDstStageMask = &waitDstStageMask;
  vkQueueSubmit(GraphicsQueue, 1, &submitInfo, fence);
}

The code works without validation complaint.

When I change the submission order of them I do get validation complaint and runtime errors:

{
  VkPipelineStageFlags waitDstStageMask = VK_PIPELINE_STAGE_ALL_COMMANDS_BIT;
  VkSubmitInfo submitInfo = VK_STYPE;
  submitInfo.commandBufferCount = 1;
  submitInfo.pCommandBuffers = &commandBuffer2;
  submitInfo.waitSemaphoreCount = 1;
  submitInfo.pWaitSemaphores = &semaphore;
  submitInfo.pWaitDstStageMask = &waitDstStageMask;
  vkQueueSubmit(GraphicsQueue, 1, &submitInfo, fence);
}
{
  VkSubmitInfo submitInfo = VK_STYPE;
  submitInfo.commandBufferCount = 1;
  submitInfo.pCommandBuffers = &commandBuffer;
  submitInfo.signalSemaphoreCount = 1;
  submitInfo.pSignalSemaphores = &semaphore;
  vkQueueSubmit(TransferQueue, 1, &submitInfo, VK_NULL_HANDLE);
}

What I would have expected to happen in the second example, is that the first command would block in the queue waiting on the semaphore, until I submitted the second one that signals it.

What am I missing here? What rule does the second example break?

Nicol Bolas Nicol Bolas · Accepted Answer · 2020-11-18T14:29:12

The validation errors have text associated with them. That text would have pointed out that you cannot submit a batch containing a binary semaphore wait until the batch that signals the binary semaphore has been submitted.

So if you're using binary semaphores across queues in multiple threads, you have to use CPU synchronization tools to submit them in the correct order. This is one of the reasons why timeline semaphores exist: they don't have that limitation.

Vulkan semaphores across queue families?

1 Answers