How to implement mmap for PCI bar memory?

Question

I have a SRAM memory addressable at PCI bar 1. I would like to know how to correctly write mmap function in driver so that my user process can mmap the sram memory and read/write into it.

I wrote the following probe and mmap functions in driver to do the same:

The probe function:

static int probe(struct pci_dev * pcidev, const struct pci_device_id * id)
{
    int r,i;
    unsigned long start,end;
    int sram;

    r = pci_enable_device(pcidev);
    if(r < 0)
    {
        printk(KERN_ALERT DEVICE_NAME ": Enable of my_pci device fails %d\n",r); 
        return -1;
    }
    pci_set_master(pcidev);

    start = pci_resource_start(pcidev, 1);
    end = pci_resource_end(pcidev, 1);
    size = end - start + 1;

    sram_raw_base_address = start;
    sram_base_address = (unsigned long)ioremap_nocache(start,size);
    sram_size = size;
}

The mmap function is:

static int my_mmap(struct file *file, struct vm_area_struct *vma)
{   
    size_t size = vma->vm_end - vma->vm_start;
    unsigned long ioaddress, iosize;

    if(vma->vm_end < vma->vm_start)
    {
        return -EAGAIN;
    }

    /* mapping sram. */
    ioaddress = sram_base_address;
    iosize = sram_size;

    /* simple check to prevent mapping past io area. */
    if( (size > iosize) ||
        ((vma->vm_pgoff << PAGE_SHIFT) > iosize) ||
        ( ((vma->vm_pgoff << PAGE_SHIFT)+size) > iosize) )
    {
        return -EINVAL;
    }

    if (remap_pfn_range(vma,vma->vm_start,vmalloc_to_pfn(ioaddress+(vma->vm_pgoff << PAGE_SHIFT)),
            size,vma->vm_page_prot)) {
    return -EAGAIN;
    }

    vma->vm_ops = &vmops;
    vmops_open(vma);

    return 0;
}

Whenever I try to mmap the SRAM memory in my application, I get the following error in /var/log/syslog:

Jul 26 13:21:12 manik kernel: [86635.262485] my_pci my_mmap - called
Jul 26 13:21:12 manik kernel: [86635.262524] BUG: unable to handle kernel  paging request at 02530000
Jul 26 13:21:12 manik kernel: [86635.262549] IP: [<c11ab31d>] vmalloc_to_pfn+0xd/0x30
Jul 26 13:21:12 manik kernel: [86635.262569] *pdpt = 00000000050f8001 *pde = 0000000000000000
Jul 26 13:21:12 manik kernel: [86635.262584] Oops: 0000 [#1] SMP
Jul 26 13:21:12 manik kernel: [86635.262595] Modules linked in: qcomdl(POE) gdapdl(POE) gammadl(POE) aspdl(POE) ccce(POE) scard(POE) kmikohn(OE) fmon(OE) khcd(OE) kcid(OE) kspi(POE) kpwrfail(OE) mk7i_pci(OE) snd_usb_audio input_leds hid_multitouch joydev snd_usbmidi_lib intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm i2c_ocores snd_soc_rt5640 snd_hda_codec_hdmi irqbypass crc32_pclmul snd_soc_rl6231 snd_hda_intel snd_hda_codec lpc_ich ie31200_edac edac_core snd_hda_core shpchp snd_hwdep snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine 8250_fintek snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd snd_soc_sst_acpi soundcore 8250_dw dw_dmac soc_button_array dw_dmac_core elan_i2c spi_pxa2xx_platform i2c_designware_platform i2c_designware_core mac_hid parport_pc ppdev lp parport autofs4 nouveau mxm_wmi i915 wmi i2c_algo_bit ttm ahci libahci drm_kms_helper syscopyarea e1000e sysfillrect sysimgblt fb_sys_fops ptp drm pps_core sdhci_acpi video i2c_hid sdhci fjes hid_generic usbhid hid [last unloaded: mk7i_pci]
Jul 26 13:21:12 manik kernel: [86635.262938] CPU: 0 PID: 14025 Comm: test_sram Tainted: P        W IOE   4.4.0-31-generic #50
Jul 26 13:21:12 manik kernel: [86635.262950] Hardware name: ADLINK Technology Inc. Express-HL./SHARKBAY, BIOS 1.14 01/01/2013
Jul 26 13:21:12 manik kernel: [86635.262962] task: d0c05a00 ti: ca928000 task.ti: ca928000
Jul 26 13:21:12 manik kernel: [86635.262972] EIP: 0060:[<c11ab31d>] EFLAGS: 00010246 CPU: 0
Jul 26 13:21:12 manik kernel: [86635.262983] EIP is at vmalloc_to_pfn+0xd/0x30
Jul 26 13:21:12 manik kernel: [86635.262991] EAX: 02530000 EBX: c5617138 ECX: ee9d9000 EDX: fff1b000
Jul 26 13:21:12 manik kernel: [86635.263001] ESI: 02000000 EDI: f3200000 EBP: ca929e70 ESP: ca929e70
Jul 26 13:21:12 manik kernel: [86635.263010]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul 26 13:21:12 manik kernel: [86635.263019] CR0: 80050033 CR2: 02530000 CR3: 2dc02500 CR4: 001406f0
Jul 26 13:21:12 manik kernel: [86635.263028] Stack:
Jul 26 13:21:12 manik kernel: [86635.263034]  ca929e90 f0c566af f0c58260 024080c0 c5617138 c56171b4 b5321000 c56171b0
Jul 26 13:21:12 manik kernel: [86635.263063]  ca929ee4 c11a64f8 b7321000 000000fb 00000000 c6eac0c0 00000000 00000000
Jul 26 13:21:12 manik kernel: [86635.263091]  00000000 00000008 00000000 00000000 c56171b4 00002000 c6eac0c0 c5617138
Jul 26 13:21:12 manik kernel: [86635.263120] Call Trace:
Jul 26 13:21:12 manik kernel: [86635.263132]  [<f0c566af>] my_mmap+0x6f/0xf0 [mk7i_pci]
Jul 26 13:21:12 manik kernel: [86635.263145]  [<c11a64f8>] mmap_region+0x388/0x590
Jul 26 13:21:12 manik kernel: [86635.263155]  [<c11a6956>] do_mmap+0x256/0x360
Jul 26 13:21:12 manik kernel: [86635.263167]  [<c118d19a>] vm_mmap_pgoff+0x9a/0xc0
Jul 26 13:21:12 manik kernel: [86635.263177]  [<c11a4c54>] SyS_mmap_pgoff+0x194/0x230
Jul 26 13:21:12 manik kernel: [86635.263190]  [<c100393d>] do_fast_syscall_32+0x8d/0x150
Jul 26 13:21:12 manik kernel: [86635.263202]  [<c17b229c>] sysenter_past_esp+0x3d/0x61
Jul 26 13:21:12 manik kernel: [86635.263210] Code: 8b 80 c0 20 d2 c1 83 e0 fc 8d 1c d0 eb bb 31 db 89 d8 5b 5e 5f 5d c3 8d b4 26 00 00 00 00 55 89 e5 3e 8d 74 26 00 e8 d3 fe ff ff <8b> 10 5d c1 ea 19 c1 e2 04 8b 92 c0 20 d2 c1 83 e2 fc 29 d0 c1
Jul 26 13:21:12 manik kernel: [86635.263388] EIP: [<c11ab31d>] vmalloc_to_pfn+0xd/0x30 SS:ESP 0068:ca929e70
Jul 26 13:21:12 manik kernel: [86635.263406] CR2: 0000000002530000
Jul 26 13:21:12 manik kernel: [86635.269070] ---[ end trace f85e2d73d203ae00 ]---

clearly, the mmap function of the driver is failing in vmalloc_to_pfn call. I do not understand why.

I am using ioremap_nocache in probe function to create non-cacheable virtual addresses as I want the writes/read happening at PCI bar memory regions to be read/write at the memory hardware and not cpu cache. I hope I am doing it right.

Now, for mmap call I am using vmalloc_to_pfn to get the physical page frame number.

Is there something that I am doing wrong here ?

The arguments you've shown for ioremap_nocache are clearly wrong; it only takes two arguments -- a phys address and a size. I don't see how it can actually compile. — Gil Hamilton
Also, might be helpful if you were to first call vmalloc_to_pfn and printk the result to see what exactly you're passing to remap_pfn_range. — Gil Hamilton
thanks for pointing out the wrong arguments inioremap_nocache issue. well, vmalloc_to_pfn as per its documentation provides us the Physical Frame Number from the virtual address passed to it. the code worked with linux kernel 3.9.2 but not anymore with linux 4.4.0 — Monku
I don't know this for sure, but I suspect that vmalloc_to_pfn is not the right thing to call. There is an expectation that an address given to that function was allocated from kernel memory and has a struct page behind it. But I don't think PCI BARs have a struct page behind them. You probably need to instead construct a pfn directly from the pci_resource_start value. (Probably __phys_to_pfn but you should read Documentation/bus-virt-phys-mapping.txt in the kernel source for caveats about virt/phys/bus addresses too.) — Gil Hamilton
@GilHamilton My understanding was exactly the same. Having said that, I was confounded because this source code worked fine with kernel 3.9.2 but crashed with kernel 4.4.0. So I was like why in the god's name did it even work previously — Monku

ddbug ddbug · Accepted Answer · 2016-07-30T01:37:27

Do you need to write any code for this at all? Usermode can open the BAR of your device in sysfs (/sys/bus/pci/...) and mmap that. It works without any effort on your side.

How to implement mmap for PCI bar memory?

1 Answers