Xilinx AXI DMA Driver probe failed on ZynqMP Analog Devices' kernel

Hello everyone,

I am trying to implement a simple AXI DMA example using EK-U1-ZCU102-ES2-G, vivado 2017.2 and the 2017_R1 Analog Devices' kernel. Unfortunately, the Xilinx AXI DMA driver doesn't probe properly during the boot and leads to a kernel panic.

Here after is the boot log:

------------------------------------------
Xilinx Zynq MP First Stage Boot Loader
Release 2017.2   Oct 19 2017  -  09:35:44
NOTICE:  ATF running on XCZU9EG/silicon v4/RTL5.1 at 0xfffea000, with PMU firmware
NOTICE:  BL31: Secure code at 0x0
NOTICE:  BL31: Non secure code at 0x8000000
NOTICE:  BL31: v1.3(release):f9b244b
NOTICE:  BL31: Built : 09:35:17, Oct 19 2017


U-Boot 2016.07 (Dec 16 2016 - 15:04:11 -0700) Xilinx ZynqMP ZCU102 revB

I2C:   ready
DRAM:  4 GiB
EL Level:       EL2
Chip ID:        xczu9eg
MMC:   sdhci@ff170000: 0
Using default environment

In:    serial@ff000000
Out:   serial@ff000000
Err:   serial@ff000000
Bootmode: LVL_SHFT_SD_MODE1
SCSI:  SATA link 0 timeout.
SATA link 1 timeout.
AHCI 0001.0301 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
flags: 64bit ncq pm clo only pmp fbss pio slum part ccc apst
scanning bus for devices...
Found 0 device(s).
Net:   ZYNQ GEM: ff0e0000, phyaddr 12, interface rgmii-id

Warning: ethernet@ff0e0000 using MAC address from ROM
eth0: ethernet@ff0e0000
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Device: sdhci@ff170000
Manufacturer ID: 3
OEM: 5344
Name: SL16G
Tran Speed: 50000000
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 14.8 GiB
Bus Width: 4-bit
Erase Group Size: 512 Bytes
reading uEnv.txt
301 bytes read in 10 ms (29.3 KiB/s)
Loaded environment from uEnv.txt
Importing environment from SD ...
Running uenvcmd ...
reading Image
14477824 bytes read in 957 ms (14.4 MiB/s)
reading system.dtb
27256 bytes read in 18 ms (1.4 MiB/s)
Wrong Image Format for bootm command
ERROR: can't get kernel image!
reading system.dtb
27256 bytes read in 18 ms (1.4 MiB/s)
reading Image
14477824 bytes read in 957 ms (14.4 MiB/s)
## Flattened Device Tree blob at 04000000
   Booting using the fdt blob at 0x4000000
   Loading Device Tree to 000000000fff6000, end 000000000ffffa77 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.0 (collin@IODA_laptop) (gcc version 6.2.1 20161114 (Linaro GCC Snaps8
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] earlycon: cdns0 at MMIO 0x00000000ff000000 (options '115200n8')
[    0.000000] bootconsole [cdns0] enabled
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 128 MiB at 0x0000000078000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] percpu: Embedded 21 pages/cpu @ffffffc87ff7b000 s47384 r8192 d30440 u86016
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: enabling workaround for ARM erratum 845719
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 1034240
[    0.000000] Kernel command line: earlycon=cdns,mmio,0xFF000000,115200n8 console=ttyPS0,115200n8 rot
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.000000] software IO TLB [mem 0x73fff000-0x77fff000] (64MB) mapped at [ffffffc073fff000-ffffffc]
[    0.000000] Memory: 3918656K/4194304K available (9276K kernel code, 634K rwdata, 3672K rodata, 512)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffffff8000000000 - 0xffffff8008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffffff8008000000 - 0xffffffbebfff0000   (   250 GB)
[    0.000000]       .text : 0xffffff8008080000 - 0xffffff8008990000   (  9280 KB)
[    0.000000]     .rodata : 0xffffff8008990000 - 0xffffff8008d30000   (  3712 KB)
[    0.000000]       .init : 0xffffff8008d30000 - 0xffffff8008db0000   (   512 KB)
[    0.000000]       .data : 0xffffff8008db0000 - 0xffffff8008e4ea00   (   635 KB)
[    0.000000]        .bss : 0xffffff8008e4ea00 - 0xffffff8008eaf534   (   387 KB)
[    0.000000]     fixed   : 0xffffffbefe7fd000 - 0xffffffbefec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffffffbefee00000 - 0xffffffbeffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffffffbf00000000 - 0xffffffc000000000   (     4 GB maximum)
[    0.000000]               0xffffffbf00000000 - 0xffffffbf1dc00000   (   476 MB actual)
[    0.000000]     memory  : 0xffffffc000000000 - 0xffffffc880000000   ( 34816 MB)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GIC: Adjusting CPU interface base to 0x00000000f902f000
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 99.99MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x171015c90f, max_ids
[    0.000003] sched_clock: 56 bits at 99MHz, resolution 10ns, wraps every 4398046511101ns
[    0.008207] Console: colour dummy device 80x25
[    0.012473] Calibrating delay loop (skipped), value calculated using timer frequency.. 199.99 Bogo)
[    0.022835] pid_max: default: 32768 minimum: 301
[    0.027514] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    0.034087] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
[    0.041660] ASID allocator initialised with 65536 entries
[    0.047313] zynqmp_plat_init Power management API v0.3
[    0.052327] EFI services will not be available.
[    0.130138] Detected VIPT I-cache on CPU1
[    0.130163] CPU1: Booted secondary processor [410fd034]
[    0.194194] Detected VIPT I-cache on CPU2
[    0.194211] CPU2: Booted secondary processor [410fd034]
[    0.249507] Detected VIPT I-cache on CPU3
[    0.249524] CPU3: Booted secondary processor [410fd034]
[    0.249559] Brought up 4 CPUs
[    0.279846] SMP: Total of 4 processors activated.
[    0.284522] CPU features: detected feature: 32-bit EL0 Support
[    0.290319] CPU: All CPU(s) started at EL2
[    0.294391] alternatives: patching kernel code
[    0.299444] devtmpfs: initialized
[    0.306304] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785s
[    0.318565] xor: measuring software checksum speed
[    0.363219]    8regs     :  2303.000 MB/sec
[    0.403247]    8regs_prefetch:  2053.000 MB/sec
[    0.447615]    32regs    :  2830.000 MB/sec
[    0.487645]    32regs_prefetch:  2379.000 MB/sec
[    0.492072] xor: using function: 32regs (2830.000 MB/sec)
[    0.497497] pinctrl core: initialized pinctrl subsystem
[    0.503130] NET: Registered protocol family 16
[    0.523414] cpuidle: using governor menu
[    0.527334] Failed to initialise IOMMU /amba/smmu@fd800000
[    0.532840] vdso: 2 pages (1 code @ ffffff8008997000, 1 data @ ffffff8008db4000)
[    0.540044] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.547229] DMA: preallocated 256 KiB pool for atomic allocations
[    0.567070] reset_zynqmp reset-controller: Xilinx zynqmp reset driver probed
[    0.574641] ARM CCI_400_r1 PMU driver probed[    0.581767] zynqmp-pinctrl ff180000.pinctrl: zynqmpd
[    0.604397] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.678935] raid6: int64x1  gen()   407 MB/s
[    0.750868] raid6: int64x1  xor()   442 MB/s
[    0.822953] raid6: int64x2  gen()   673 MB/s
[    0.894987] raid6: int64x2  xor()   599 MB/s
[    0.967070] raid6: int64x4  gen()   984 MB/s
[    1.039069] raid6: int64x4  xor()   739 MB/s
[    1.111159] raid6: int64x8  gen()  1146 MB/s
[    1.183197] raid6: int64x8  xor()   746 MB/s
[    1.255243] raid6: neonx1   gen()   720 MB/s
[    1.327293] raid6: neonx1   xor()   732 MB/s
[    1.399330] raid6: neonx2   gen()  1166 MB/s
[    1.471381] raid6: neonx2   xor()  1035 MB/s
[    1.543467] raid6: neonx4   gen()  1505 MB/s
[    1.615492] raid6: neonx4   xor()  1182 MB/s
[    1.687572] raid6: neonx8   gen()  1586 MB/s
[    1.759610] raid6: neonx8   xor()  1221 MB/s
[    1.763686] raid6: using algorithm neonx8 gen() 1586 MB/s
[    1.769048] raid6: .... xor() 1221 MB/s, rmw enabled
[    1.773979] raid6: using intx1 recovery algorithm
[    1.779392] SCSI subsystem initialized
[    1.783117] usbcore: registered new interface driver usbfs
[    1.788437] usbcore: registered new interface driver hub
[    1.793716] usbcore: registered new device driver usb
[    1.798763] media: Linux media interface: v0.10
[    1.803221] Linux video capture interface: v2.00
[    1.807806] pps_core: LinuxPPS API ver. 1 registered
[    1.812715] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.i>
[    1.821807] PTP clock support registered
[    1.825706] EDAC MC: Ver: 3.0.0
[    1.829035] FPGA manager framework
[    1.832356] fpga-region fpga-full: FPGA Region probed
[    1.837352] Advanced Linux Sound Architecture Driver Initialized.
[    1.843576] Bluetooth: Core ver 2.22
[    1.846974] NET: Registered protocol family 31
[    1.851373] Bluetooth: HCI device and connection manager initialized
[    1.857690] Bluetooth: HCI socket layer initialized
[    1.862532] Bluetooth: L2CAP socket layer initialized
[    1.867562] Bluetooth: SCO socket layer initialized
[    1.872932] clocksource: Switched to clocksource arch_sys_counter
[    1.878892] VFS: Disk quotas dquot_6.6.0
[    1.882757] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.893820] NET: Registered protocol family 2
[    1.898294] TCP established hash table entries: 32768 (order: 6, 262144 bytes)
[    1.905515] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)
[    1.912358] TCP: Hash tables configured (established 32768 bind 32768)
[    1.918724] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    1.924727] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    1.931211] NET: Registered protocol family 1
[    1.935570] RPC: Registered named UNIX socket transport module.
[    1.941294] RPC: Registered udp transport module.
[    1.945962] RPC: Registered tcp transport module.
[    1.950632] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.957461] hw perfevents: enabled with armv8_pmuv3 PMU driver, 7 counters available
[    1.965638] futex hash table entries: 1024 (order: 5, 131072 bytes)
[    1.971751] audit: initializing netlink subsys (disabled)
[    1.977090] audit: type=2000 audit(1.920:1): initialized
[    1.982763] workingset: timestamp_bits=62 max_order=20 bucket_order=0
[    1.989620] NFS: Registering the id_resolver key type
[    1.994495] Key type id_resolver registered
[    1.998665] Key type id_legacy registered
[    2.002616] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    2.009286] jffs2: version 2.2. (NAND) (SUMMARY)  � 2001-2006 Red Hat, Inc.
[    2.020597] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    2.027810] io scheduler noop registered
[    2.031684] io scheduler deadline registered
[    2.035928] io scheduler cfq registered (default)
[    2.041120] nwl-pcie fd0e0000.pcie: Link is DOWN
[    2.045580] OF: PCI: host bridge /amba/pcie@fd0e0000 ranges:
[    2.051176] OF: PCI:   No bus range found for /amba/pcie@fd0e0000, using [bus 00-ff]
[    2.058883] OF: PCI:   MEM 0xe0000000..0xefffffff -> 0xe0000000
[    2.064756] OF: PCI:   MEM 0x600000000..0x7ffffffff -> 0x600000000
[    2.070987] nwl-pcie fd0e0000.pcie: PCI host bridge to bus 0000:00
[    2.077041] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.082488] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xefffffff]
[    2.089324] pci_bus 0000:00: root bus resource [mem 0x600000000-0x7ffffffff pref]
[    2.097043] pci 0000:00:00.0: PCI bridge to [bus 01-0c]
[    2.103034] xilinx-dpdma fd4c0000.dma: Xilinx DPDMA engine is probed
[   23.024927] INFO: rcu_sched detected stalls on CPUs/tasks:
[   23.030219]  3-...: (16 ticks this GP) idle=aab/140000000000000/0 softirq=345/347 fqs=2626
[   23.038517]  (detected by 1, t=5255 jiffies, g=-283, c=-284, q=259)
[   23.044747] Task dump for CPU 3:
[   23.047947] swapper/0       R  running task        0     1      0 0x00000002
[   23.054953] Call trace:
[   23.057383] [<ffffff800808530c>] __switch_to+0x8c/0xa0
[   23.062484] [<ffffff800845fcb4>] xilinx_dma_probe+0x484/0x8a8
[   23.068191] [<ffffff80084edad8>] platform_drv_probe+0x58/0xc0
[   23.073900] [<ffffff80084ebf7c>] driver_probe_device+0x1fc/0x2a8
[   23.079868] [<ffffff80084ec0d4>] __driver_attach+0xac/0xb0
[   23.085318] [<ffffff80084e9fcc>] bus_for_each_dev+0x64/0xa0
[   23.090853] [<ffffff80084eb768>] driver_attach+0x20/0x28
[   23.096130] [<ffffff80084eb2b8>] bus_add_driver+0x110/0x230
[   23.101666] [<ffffff80084ec8b8>] driver_register+0x60/0xf8
[   23.107115] [<ffffff80084eda10>] __platform_driver_register+0x40/0x48
[   23.113518] [<ffffff8008d53298>] xilinx_vdma_driver_init+0x18/0x20
[   23.119658] [<ffffff80080830b8>] do_one_initcall+0x38/0x128
[   23.125195] [<ffffff8008d30c94>] kernel_init_freeable+0x140/0x1e0
[   23.131250] [<ffffff800897b6a0>] kernel_init+0x10/0x100
[   23.136439] [<ffffff8008082e80>] ret_from_fork+0x10/0x50

Bloc Diagram of the AXI DMA example:

for the cross compilation:

    export CROSS_COMPILE=aarch64-linux-gnu-

    export ARCH=arm64

for the Zynq MP default kernel configuration:

    make adi_zynqmp_defconfig

such as described here: Building the ZynqMP / MPSoC Linux kernel and devicetrees from source [Analog Devices Wiki]


To implement the Xilinx DMA Driver, the kernel config file is set as following:

CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set
 
#
# DMA Devices
#
CONFIG_DMA_ENGINE=y
CONFIG_DMA_VIRTUAL_CHANNELS=y
CONFIG_DMA_OF=y
# CONFIG_AMBA_PL08X is not set
CONFIG_AXI_DMAC=y
# CONFIG_FSL_EDMA is not set
# CONFIG_INTEL_IDMA64 is not set
# CONFIG_MV_XOR_V2 is not set
# CONFIG_PL330_DMA is not set
CONFIG_XILINX_DMA_ENGINES=y
# CONFIG_XILINX_DMATEST is not set
# CONFIG_XILINX_VDMATEST is not set
# CONFIG_XILINX_CDMATEST is not set
CONFIG_XILINX_DPDMA=y
# CONFIG_XILINX_DPDMA_DEBUG_FS is not set
CONFIG_XILINX_FRMBUF=y
CONFIG_XILINX_DMA=y
CONFIG_XILINX_ZYNQMP_DMA=y
  • The Device tree is generated with SDK 2017.2:

pl.dtsi

/ {
    amba_pl: amba_pl@0 {
        #address-cells = <2>;
        #size-cells = <2>;
        compatible = "simple-bus";
        ranges ;
        axi_dma_0: dma@a0000000 {
            #dma-cells = <1>;
            clock-names = "s_axi_lite_aclk", "m_axi_sg_aclk", "m_axi_mm2s_aclk", "m_axi_s2mm_aclk";
            clocks = <&misc_clk_0>, <&misc_clk_0>, <&misc_clk_0>, <&misc_clk_0>;
            compatible = "xlnx,axi-dma-1.00.a";
            interrupt-parent = <&gic>;
            interrupts = <0 89 4 0 90 4>;
            reg = <0x0 0xa0000000 0x0 0x10000>;
            xlnx,addrwidth = <0x20>;
            xlnx,include-sg ;
            dma-channel@a0000000 {
                compatible = "xlnx,axi-dma-mm2s-channel";
                dma-channels = <0x1>;
                interrupts = <0 89 4>;
                xlnx,datawidth = <0x20>;
                xlnx,device-id = <0x0>;
            };
            dma-channel@a0000030 {
                compatible = "xlnx,axi-dma-s2mm-channel";
                dma-channels = <0x1>;
                interrupts = <0 90 4>;
                xlnx,datawidth = <0x20>;
                xlnx,device-id = <0x1>;
            };
        };
        psu_ctrl_ipi: PERIPHERAL@ff380000 {
            compatible = "xlnx,PERIPHERAL-1.0";
            reg = <0x0 0xff380000 0x0 0x80000>;
        };
        psu_message_buffers: PERIPHERAL@ff990000 {
            compatible = "xlnx,PERIPHERAL-1.0";
            reg = <0x0 0xff990000 0x0 0x10000>;
        };
        misc_clk_0: misc_clk_0 {
            compatible = "fixed-clock";
            #clock-cells = <0>;
            clock-frequency = <100000000>;
        };
    };
};

The Device tree is modified to add clock informations according to AXI DMA Driver Problem - Community Forums

The boot problem is still persistent when using clkc 71 instead of the generated misc_clk_0, as suggest in AXI DMA test failure - Community Forums

I think it can be linked to the clock or maybe the Analog Devices' kernel doesn't support Xilinx DMA driver.

Do you have any idea ?

Thanks for your help!

  • The ADI Linux kernel is based on linux-xlnx - however with some local modifications.

    We assume that these are only fixes - can you revert to the xilinx_dma driver and see if the issue persists?

    This is how you can do it in your tree:

    git remote add linux-xlnx github.com/.../linux-xlnx.git
    git fetch
    git checkout linux-xlnx/master drivers/dma/xilinx/xilinx_dma.c
    git checkout linux-xlnx/master include/linux/dma/xilinx_dma.h

    -Michael

  • Hi,

    I'm not sure that we can help you with this. You might want to contact Xilinx support abourt the Xilinx DMA.

    It looks like things get stuck and the CPU is spinning forever. In the probe function the only place I can see where this could happen is the xilinx_dma_reset() function. Maybe add a printk() before and after the xilinx_dma_poll_timeout() and see what happens.

    - Lars

  • Thanks for your answer, unfortunately the issue still persists updating the xilinx_dma driver to the linux-xlnx/master version.

    I also tried to boot with this version of the Xilinx Linux kernel, configured for zynqmp:

    make xilinx_zynqmp_defconfig

    but it leads to the same result.

  • Hi,

    Could be, if the address is wrong, or there is a bug in how the DMA handles the write. But we are not the experts for the Xilinx DMA, to get a better answer I recommend to contact the Xilinx support.

    - Lars

  • Hi,

    Thanks for your advice, the probe function get effectively stuck in the xilinx_dma_reset() function at:

    dma_ctrl_set(chan, XILINX_DMA_REG_DMACR, XILINX_DMA_DMACR_RESET);

    My first idea is that the bitstream could be loaded after the Xilinx DMA probe. The address of the Xilinx DMA register not being yet instantied, the writing in this register produces then a kernel panic at this step of the boot.

    I compiled then the kernel with the xilinx_dma driver as module. Once the boot done, I insured that the bitstream was loaded according to: Xilinx Wiki - Solution ZynqMP PL Programming . But when I insert the xilinx_dma module the OS get stuck again.

    Could it be a problem with the writing in the register itself ?