Linux kernel device driver to DMA from a device into user-space memory

Question

I want to get data from a DMA enabled, PCIe hardware device into user-space as quickly as possible.

Q: How do I combine \"direct I/O to user-space with/and/via a DMA

Answer 1

It is worth mention that driver with Scatter-Gather DMA support and user space memory allocation is most efficient and has highest performance. However in case we don't need high performance or we want to develop a driver in some simplified conditions we can use some tricks.

Give up zero copy design. It is worth to consider when data throughput is not too big. In such a design data can by copied to user by copy_to_user(user_buffer, kernel_dma_buffer, count); user_buffer might be for example buffer argument in character device read() system call implementation. We still need to take care of kernel_dma_buffer allocation. It might by memory obtained from dma_alloc_coherent() call for example.

The another trick is to limit system memory at the boot time and then use it as huge contiguous DMA buffer. It is especially useful during driver and FPGA DMA controller development and rather not recommended in production environments. Lets say PC has 32GB of RAM. If we add mem=20GB to kernel boot parameters list we can use 12GB as huge contiguous dma buffer. To map this memory to user space simply implement mmap() as

remap_pfn_range(vma,
    vma->vm_start,
    (0x500000000 >> PAGE_SHIFT) + vma->vm_pgoff, 
    vma->vm_end - vma->vm_start,
    vma->vm_page_prot)

Of course this 12GB is completely omitted by OS and can be used only by process which has mapped it into its address space. We can try to avoid it by using Contiguous Memory Allocator (CMA).

Again above tricks will not replace full Scatter-Gather, zero copy DMA driver, but are useful during development time or in some less performance platforms.