4
votes

As a first level test of my PCI driver I hoped I could gain access to the pci_iomap region via the /sys/bus/pci/devices/0000:01:00.0/resource0 file from my user application. The man page for mmap, the sample Program I found, and other posts seem to indicate that user process access should work. But some articles seem to indicate that the mmap call needs to be done from within the kernel via an ioctl accessor.

My Question is should mmap() of the PCI sysfs resource file work from application space?

When I run my code mmap returns what looks like a valid address but I get a Bus error when I try and access the virtual address. I believe my end device a PCI to Xilinx AXI bridge which is on the FPGA is functioning ok since I can R/W to it over a windows PCIe utilty (Win Driver) I am on an NXP LS1021A ARM7 processor with Linux ver 3.12.37.

Thanks Bill

Not that I want anyone to debug my code but what I am doing may be best explained by the code so I have included it also. I apologies if the pasted code doesn't display correctly. Hopefully it does.

I run the code below and get root@ls1021aiot:~# pcimem /sys/bus/pci/devices/0000:01:00.0/resource0 0 w

/sys/bus/pci/devices/0000:01:00.0/resource0 opened. Target offset is 0x0, page size is 4096 map mask is 0xFFF mmap(0, 4096, 0x3, 0x1, 3, 0x0) mmap(0, 4096, 0x3, 0x1, 3, 0x0) PCI Memory mapped 4096 byte region to map_base 0x76fb5000. PCI Memory mapped access 0x 76FB5000. Bus error

 /*
 * pcimem.c: Simple program to read/write from/to a pci device from userspace.
 *
 *  Copyright (C) 2010, Bill Farrow ([email protected])
 *
 *  Based on the devmem2.c code
 *  Copyright (C) 2000, Jan-Derk Bakker ([email protected])
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; either version 2 of the License, or
 * (at your option) any later version.
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 * 
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 *
 */

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>
#include <ctype.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <linux/pci.h>


#define PRINT_ERROR \
    do { \
        fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", \
        __LINE__, __FILE__, errno, strerror(errno)); exit(1); \
    } while(0)

#define MAP_SIZE 4096UL
#define MAP_MASK (MAP_SIZE - 1)

int main(int argc, char **argv) {
    int fd; 
    void *map_base, *virt_addr;
    uint32_t read_result, writeval;
    char *filename;
    off_t target;
    int access_type = 'w';

    if(argc < 3) {
        // pcimem /sys/bus/pci/devices/0001\:00\:07.0/resource0 0x100 w 0x00
        // argv[0]  [1]                                         [2]   [3] [4]
        fprintf(stderr, "\nUsage:\t%s { sys file } { offset } [ type [ data ] ]\n"
                "\tsys file: sysfs file for the pci resource to act on\n"
                "\toffset  : offset into pci memory region to act upon\n"
                "\ttype    : access operation type : [b]yte, [h]alfword, [w]ord\n"
                "\tdata    : data to be written\n\n",
                argv[0]);
        exit(1);
    }   
    filename = argv[1];
    target = strtoul(argv[2], 0, 0); 

    if(argc > 3)
        access_type = tolower(argv[3][0]);

    if((fd = open(filename, O_RDWR | O_SYNC)) == -1){
        PRINT_ERROR;
    }
    printf("%s opened.\n", filename);
    printf("Target offset is 0x%x, page size is %ld map mask is 0x%lX\n", (int) target, sysconf(_SC_PAGE_SIZE), MAP_MASK);
    fflush(stdout);

    /* Map one page */
#if 0
    //map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, (off_t) (target & ~MAP_MASK));
    //map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, target & ~MAP_MASK);
#endif
    printf("mmap(%d, %ld, 0x%x, 0x%x, %d, 0x%x)\n", 0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, (int) (target & ~MAP_MASK));
    map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, (target & ~MAP_MASK));
    if(map_base == (void *) -1){
       printf("PCI Memory mapped ERROR.\n");
        PRINT_ERROR;
        close(fd);
        return 1;
    }

    printf("mmap(%d, %ld, 0x%x, 0x%x, %d, 0x%x)\n", 0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, (int) (target & ~MAP_MASK));
    printf("PCI Memory mapped %ld byte region to map_base 0x%08lx.\n", MAP_SIZE, (unsigned long) map_base);
    fflush(stdout);

    virt_addr = map_base + (target & MAP_MASK);
    printf("PCI Memory mapped access 0x %08X.\n", (uint32_t ) virt_addr);
   switch(access_type) {
        case 'b':
                read_result = *((uint8_t *) virt_addr);
                break;  
        case 'h':
                read_result = *((uint16_t *) virt_addr);
                break;  
        case 'w':
                read_result = *((uint32_t *) virt_addr);
                        printf("READ Value at offset 0x%X (%p): 0x%X\n", (int) target, virt_addr, read_result);
                break;
        default:
                fprintf(stderr, "Illegal data type '%c'.\n", access_type);
                exit(2);
    }
    fflush(stdout);

    if(argc > 4) {
        writeval = strtoul(argv[4], 0, 0);
        switch(access_type) {
                case 'b':
                        *((uint8_t *) virt_addr) = writeval;
                        read_result = *((uint8_t *) virt_addr);
                        break;
                case 'h':
                        *((uint16_t *) virt_addr) = writeval;
                        read_result = *((uint16_t *) virt_addr);
                        break;
                case 'w':
                        *((uint32_t *) virt_addr) = writeval;
                        read_result = *((uint32_t *) virt_addr);
                        break;
        }
        printf("Written 0x%X; readback 0x%X\n", writeval, read_result);
        fflush(stdout);
    }

    if(munmap(map_base, MAP_SIZE) == -1) { PRINT_ERROR;}
    close(fd);
    return 0;
}
4
I compiled both my drivers and the pci_debug app for x86_64 ( linux 3.16.7) and they worked correctly. This leaves me to believe that I am missing something required for the the 32 bit arm processor (40bit internal addressing). The PCI device is mapped into the 0x40_0000_0000 address base. cat /proc/cpuinfo shows lpae feature.Bill Anderson
I forgot to mention the pci_debug app complied for the ARM processor Bus Errors like my test app did. The mmap returns void * to which I add an offset is this allowed in LPAE? Ill keep digging. ThanksBill Anderson

4 Answers

3
votes

You can take a look at pci_debug program code, available here. If you have bus error, it's maybe a FPGA design problem. Is your IP AXI bus accept 32bits/16bits/8bits access ? Be sure to access memory with right address (if 32bits address must divisible by 4, if 16 bits by 2).

2
votes

Still haven't found a fix for this on the NXP LS1021A processor, But I tested on a x86_64 machine and there the mmap worked to the same PCIe Edge target. The NXP ARMv7 processor I have is a 40 bit internal address space device but is a 32 bit processor. This must present a special case to the Virtual address tables which is not handled correct. The 2 PCIe root devices are mapped into the higher address space. I ended up just writing an IOCTL driver and used the read and write routines in kernel space to acccess the pci_iomap() pointer which I cached when the device is probed. NXP offered a suggestion which I quote below, but I still got the same Bus Error. If they ever solve it I will update.

" Regarding the mmap. Here is a response from the software team. use mmap64() to map 40bit phy_address to a 32 virt_addr or continue using mmap but add “-D_FILE_OFFSET_BITS=64” when compiling."

UPDATE. I received a new version of the toolchain from NXP and mmap64 was available in it. My take on this is that mmap64 support need only be used on 32bit systems when you want to access files and offsets larger > 4GB files. By replacing the calls to mmap with mmap64 the tests I used now run correctly.

0
votes

I had the same issue (bus error), and nothing I did worked.

Eventually, It was solved by using uio_pci_generic.

The driver seems to allow using mmap from userspace (without a driver, this can't be done).

More information can be found here:

https://github.com/rumpkernel/wiki/wiki/Howto:-Accessing-PCI-devices-from-userspace

0
votes

In case someone else lands here, note that you also want to look at the output of lspci or lshw for the resource of interest. Not all the resources are writable. I have certainly seen bus errors, but do have things working for my purpose. Have not typified all cases.