Retrieve stacktrace of QEMU VM during mmio

Question

I am emulating some hardware in QEMU, which correspond to some drivers in the kernel of linux guest.

Right now, I can use memory_region_init_io to setup mmio regions so that whenever the kernel driver read/write to mmio address, I would get a callback.

How could I get the stack trace of the kernel that triggers mmio access within the callback? I want to know which line in the kernel driver triggers which mmio access.

I know that mmiotrace may be an option, but that trace occurs in guest kernel. Is there anyway I could achieve that with qemu-kvm.

static uint64_t mmio_read(void *opaque, hwaddr addr,
                              unsigned size) {
    /* Here, I want to get the stacktrace inside VM 
     * that caused this mmio read */
    printf("mmio_read: %lx[%u] returns %lx\n", addr, size, ret);
    return 0;
}
static void stream_dma_write(void *opaque, hwaddr addr,
                           uint64_t val, unsigned size) {
    /* Here, I want to get the stacktrace inside VM 
     * that caused this mmio write */
    printf("mmio_write: %lx[%u]=%lx \n", addr, size, val);
}

static const MemoryRegionOps mmio_ops {
    .read = mmio_read,
    .write = mmio_write,
}

void init_region(uintptr_t addr, size_t size) {
MemoryRegion *subregion = malloc(sizeof(MemoryRegion));
    memory_region_init_io(subregion, OBJECT(opaque), 
                &mmio_ops, NULL, "mmio-region", size);
    memory_region_add_subregion_overlap(get_system_memory(),
                addr, subregion, 100);
}

Peter Maydell Peter Maydell · Accepted Answer · 2020-06-19T10:25:29

Unfortunately QEMU doesn't provide anything really that would do this for you as an API you can call from within QEMU C code. There are a couple of problems:

QEMU doesn't continuously update all CPU state for each instruction, and in particular it does not update the PC value until it absolutely has to, because writing "add 4 to the PC field in the CPU state struct" all the time is expensive. So the current PC is not really conveniently accessible from a device MMIO read/write function.
QEMU doesn't have any code that knows how to do a guest stack backtrace. This is a comparatively complicated thing to do correctly (you'll find code for it in debuggers, of course).

I think if I were designing something for this purpose I'd probably try to provide a way for a device to trigger the guest to stop so that a target-architecture gdb attached to the QEMU gdbstub could examine registers and do backtraces. Then you could script the debugger if you wanted "print backtrace and continue guest execution".

That said, here are a couple of suggestions you could try:

If you're lucky, then setting watchpoints from a target-architecture gdb on the QEMU gdbstub for the addresses of the device's registers will let you get control in gdb when the guest does a device access so you can do a backtrace. I give this about a 50% chance of working, because I'm not sure how robust large-area watchpoint support is going to be; also you'll need to set the watchpoints on the virtual address the kernel mapped the device to, which might be tricky to determine.
My experience with writing device models has been that it's usually pretty obvious just by looking at the source code for the device driver what it was doing when it made an MMIO access to the device. You know which register was written to and with what value, which is often sufficient to narrow down which bit of the driver made the access. This does depend on the complexity of the hw and driver, of course.
Using QEMU's -d and -D options to log a combination of device-specific trace events and general guest CPU execution/control flow information is another trick I've found helpful in trying to work out what a guest was doing to a device.

Retrieve stacktrace of QEMU VM during mmio

1 Answers