2010-10-01 16:26:27 /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94109

I am seeing the following CPLB miss on opening /dev/mem from user space. I compiled the kernel with MMU enabled, which to my surprise was not the default. While recompiling the kernel with MMU disabled (back to the default) I want to ask whether disabling the MMU is a prerequisite to use /dev/mem from the user space?

handle = open("/dev/mem", O_RDWR | O_SYNC);

printf("/dev/mem opened.\n");

fflush(stdout);

root:/mnt/read_fpga_cmd> ./fpga_cmd 0x20000000 4

/dev/mem opened.Data access CPLB miss

- Used by the MMU to signal a CPLB miss on a data access.

Deferred Exception context

CURRENT PROCESS:

COMM=fpga_cmd PID=144

CPU = 0

I also want to ask whether the flags used in "open(/dev/mem" and in mmap calls are explained anywhere on this Wiki, or should I try finding some other place for explanations. Are these flags universal, or are these specific to ucLinux? I mean O_RDWR, MAP_SHARED, etc.

QuoteReplyEditDelete

2010-10-01 16:31:14 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94110

there are very few things that are specific to Linux on a nommu processor. i can think of one mmap flag off the top of my head, but it isnt one meant for general usage. some mmap options like MAP_SHARED may not work because you dont have virtual memory, but those are documented in the wiki and in the linux Documentation folder.

otherwise, simply read the man page for whatever function you're using.

as for /dev/mem, i dont think we've really ever tested it before. we just tell people to access the async mem bus directly.

QuoteReplyEditDelete

2010-10-01 16:39:30 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94113

> we just tell people to access the async mem bus directly.

Accessing asynch bus fro user space is exactly my intention. I thought that the only way to read/write asynch memory is by mmap-int it via /dev/mem. If there is some other way then please tell me what it is. (But it has to be user space. I am not developing the kernel driver yet. That's future.)

QuoteReplyEditDelete

2010-10-01 16:44:00 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94114

i already said -- access it directly.

void *foo = 0x20000000;

memset(foo, 0, 0x100);

QuoteReplyEditDelete

2010-10-01 16:57:20 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94115

Mike: thanks! I will try. Do you have a rough idea of the performance of memset versus mmaped access?

QuoteReplyEditDelete

2010-10-01 17:11:57 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94117

"memset" vs "mmap" makes no sense. one is writing to a pointer, the other is creating a map and returning a pointer.

there is absolutely no performance difference between writing to a pointer returned by mmap() and hardcoding the pointer in your application on Blackfin systems.

QuoteReplyEditDelete

2010-10-01 18:31:24 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94124

Mike:

no difference is good because I can then choose whatever is more convenient. Then let me ask why are you advising memset versus direct write to a pointer? Is there a reason to prefer memset?

memset(foo, 0, 0x100);

*((unsigned short *) foo) = writeval;

QuoteReplyEditDelete

2010-10-01 18:36:50 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94125

you wanted an example of writing to a pointer. i gave you one.

QuoteReplyEditDelete

2010-10-01 22:32:27 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94126

Mike: thanks. Your help is apprecciated.

I boiled down the program as follows. I am still running it with MPU enabled, because the Wiki intro says MPU is beneficial. But it looks as if MPU is getting in the way of asynch memory access. Am I doing something wrong in my program, or is Blackfin MPU truly a piece of junk to be disabled and left dormant?

#include <stdio.h>

#include <string.h>

#include <errno.h>

#include <termios.h>

#include <sys/types.h>

#include <sys/mman.h>

int main(int argc, char **argv) {

void *virt_addr;

unsigned int read_result;

virt_addr = (void *) 0x21000000;

printf("Will read address %p: \n", virt_addr); fflush(stdout);

printf("\n"); fflush(stdout);

read_result = *( (unsigned int *) virt_addr);

printf("Value at address %p: 0x%04X\n", virt_addr, read_result);

fflush(stdout);

return 0;

}

root:/mnt/read_fpga_cmd> ./test

Will reaD addrass 0x210000t0: a access CPLB miss

- Used by the MMU to signal a CPLB miss on a data access.

Deferred Exception context

CURRENT PROCESS:

COMM=test PID=156

CPU = 0

TEXT = 0x030d4000-0x030d48b8 DATA = 0x032358b8-0x03235a24

BSS = 0x03235a24-0x032a0000 USER-STACK = 0x032bfec0

QuoteReplyEditDelete

2010-10-02 00:27:34 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94127

I recompiled the kernel with MPU disabled, loaded to the board, ran the test program, and the CPLD miss is gone. I am a little bit confused. On one hand, I fully subscribe to the idea that memory protection is a great thing. On the other hand, I want to see my programs running rather than crashing. So what is wrong with enabling the MPU on the Blackfin chip? Why is it standing in the way of asynch memory access?

root:/mnt/read_fpga_cmd> ./test

Will read address 0x21000000:

Value at address 0x21000000: 0xFFFFFFFF

QuoteReplyEditDelete

2010-10-02 15:12:26 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94148

the MPU works perfectly fine. changes specific to the async bank were introduced of late which probably broke user-mode access to the async bank when the MPU is enabled.

QuoteReplyEditDelete

2010-10-11 07:18:15 Re: /dev/mem causes CPLB miss

Sonic Zhang (CHINA)

Message: 94403

In MPU kernel, application is not allowed to access any address in the ASYNC memory region directly. The application should map the region to its memory space first.

QuoteReplyEditDelete

2010-10-11 11:37:42 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94405

Sonic:

thank you for the message. I tried /dev/mem and mmap, and I noticed that the mmaped address is the same as the physical address. It makes sense to me because Blackfin does not support virtual memory remapping, as far as I understand. It follows that using /dev/mem and mmap is superfluous and can be left out. Mike then suggested to use the physical address as is, without any call to mmap which has no effect anyway. I am doing just that and it works when MPU is disabled. There is a CPLB miss when the MPU is enabled.

My understanding of the situation is that CPLBs are not setup to cover the asynch memory. I do not know whether the CPLBs should cover this memory or not. In my case the asynch memory is accessing the FPGA, and therefore caching this memory is not possible. The data in FPGA registers can change any time and therefore cannot be cached.

At this point I am directly reading and writing the FPGA using pointers, whose values I supply on command line. I am not calling mmap prior to using these pointers. I would prefer to run with MPU enabled during the development, because I read that the MPU improves robustness of the code. But due to the CPLB miss using the MPU is not possible under 2009R1.1 which I am using.

I doubt I will be able to upgrade to 2010 tools due to lack of gigabit driver in these tools, as we discussed a while ago. So for now I am stuck with the 2009 tools. They seem to work just fine except for the MPU issue, which is minor at this point.

Thank you -- Wojtek

QuoteReplyEditDelete

2010-10-12 06:45:45 Re: /dev/mem causes CPLB miss

Sonic Zhang (CHINA)

Message: 94431

In 2009R1.1 there is no support for application to map an region in async memory in MPU mode. Unfortunitely, although we read authority is given to application if root romfs on NOR flash is enabled in MPU kernel, there is still no support for application to map and access the async memory.

function dcplb_miss() in file arch/blackfin/kernel/cplb-mpu/cplbmgr.c should be revised to set proper authority to cplb entry to async memory.

An alternative way is to write a driver to access your FPGA instead of do it in application.

QuoteReplyEditDelete

2010-10-12 20:48:21 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94444

that isnt true. look at the 2009R1 code cplb-mpu/cplbmgr.c:

if (addr >= ASYNC_BANK0_BASE && addr < ASYNC_BANK3_BASE + ASYNC_BANK3_SIZE

&& (status & FAULT_USERSUPV)) {

addr &= ~0x3fffff;

d_data &= ~PAGE_SIZE_4KB;

d_data |= PAGE_SIZE_4MB;

this logic was incorrectly merged into XIP specific stuff during the 2010R1 cycle by Barry. it isnt hard to fix.

QuoteReplyEditDelete

2010-10-12 21:01:30 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94445

Sonic:

the kernel driver is planned. The FPGA is performing data acquisition. It will interrupt Blackfin on PF1. The driver will be developed to handle the interrupt by reading the data from the FPGA. We will also explore using PPI for data transfer because it is also connected to the FPGA. All this is on our to do list.

At this point I am still testing the system and implementing basic diagnostics, which is best done in the user space. It is also possible that really simple and slow apps can stay in the user space forever. Things like downloading digital filter coefficients into memory-mapped registers do not need to be done in the kernel. I am very happy that I do not have to develop a driver in order to set a few bits in the FPGA. Having to do so would slow down my exploratory development quite significantly.

QuoteReplyEditDelete

2010-10-12 21:03:55 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94446

> it isnt hard to fix

Mike, I would fix it if I understood what it is doing ;-(

QuoteReplyEditDelete

2010-10-12 22:55:43 Re: /dev/mem causes CPLB miss

Sonic Zhang (CHINA)

Message: 94450

This stil doen't grant the application CPLB_USER_RD and CPLB_USER_WR.

QuoteReplyEditDelete

2010-10-12 22:57:40 Re: /dev/mem causes CPLB miss

Sonic Zhang (CHINA)

Message: 94451

And you can't simply grant the access authority as well, current_rwx_mask bits should be check accordingly.

QuoteReplyEditDelete

2010-10-12 23:11:53 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94452

hmm, i think you're right. i recall this stuff used to work (bootrom and async) from userspace. i'll have to build up 2009R1 and 2010R1 tomorrow to figure out what needs fixing.

QuoteReplyEditDelete

2010-10-12 23:14:13 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94453

i dont think so. the point of using the bitmask of addresses was that it was fast and did not require access to the process state (which could live anywhere and thus cause a CPLB miss of its own). the downside was that it isnt sufficient to cover all of memory. that is why the cplb manager has explicit memory checks for things above the 512MiB limit.

until we have a design for the higher addresses, granting user access to these regions is the only option.

QuoteReplyEditDelete

2010-10-13 07:09:16 Re: /dev/mem causes CPLB miss

Sonic Zhang (CHINA)

Message: 94464

Barry has already add the support for ASYNC memory address in the pages_rwx_mask bit array for each process when romfs XIP in NOR flash is enabled in MPU kernel.

See arch/blackfin/kernel/setup.c:646

QuoteReplyEditDelete

2010-10-13 12:20:02 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94484

and in doing so, broke async access when XIP stuff is disabled

QuoteReplyEditDelete

2010-10-13 12:23:32 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94485

btw, one fundamental question that hasnt been covered ... are all read accesses to the device impotent ? if you need to access a device FIFO via the async bus for example, this can only safely be done from kernel space. see the arch/blackfin/asm/io.h for details.

QuoteReplyEditDelete

2010-10-18 10:40:04 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94678

Mike:

excuse me for a delayed answer. I was busy with something else.... You asked: "are all read accesses to the device impotent ? if you need to access a device FIFO via the async bus for example, this can only safely be done from kernel space. see the arch/blackfin/asm/io.h for details."

I am not sure what you mean by "impotent". If it was a typo and you mean "important" then yes. The FPGA is mapped on AMS0 and AMS1 and it is the core of the design. Technically speaking, I am accessing the FPGA's block memories (BRAM) which are just like SRAM memory chips would have been, but embedded on-chip. Concerning FIFOs, in the future we may need to implement a FIFO in the FPGA and map it onto some location in the asynch memory. We are not doing it right now, but we may in the future, depending on application.

It just occurred to me that I have never seen a CPLB miss on AMS3, where the ether chip is mapped. Even when I I activated the MPU and I was seeing this CPLB miss, I was still executing my code over the ethernet and it worked just fine. (I mean, I was using the shared disk mounted over samba to execute the command.) It means that the asynch memory access was performed in AMS3 space without any problem. Does this prove that the entire traffic was handled under the kernel? Does this further mean that this CPLB miss only occurs in the user mode, when the MPU is activated? I guess it means so. I am pointing it out for completeness.

Accessing the FPGA is so important that I cannot give up on this under any circumstances. Furthermore, developing simple apps in user space is so infinitely more productive that I do not want to give it up, either.

Thank you and sorry again for the delay in answering. -- Wojtek

QuoteReplyEditDelete

2010-10-19 18:51:37 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94744

i didnt typo. if reads are destructive (like a FIFO), then you cannot safely do this from userspace.

as for the kernel accessing things, the Blackfin cpu has proper supervisor and user mode separation. supervisor getting access does not imply usermode gets it too.

QuoteReplyEditDelete

2010-10-20 21:02:26 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94786

Mike,

could you please explain why "if reads are destructive (like a FIFO), then you cannot safely do this from userspace"?

I understand that the memory cell cannot be cached if it is volatile. I heard that caching is not enabled for asynch spaces (is it a correct statement)? If true, then caching is not a problem.

But why a volatile variable cannot be read, and how would reading it under the kernel be any different? Here is my sample code:

short *virt_addr;

short read_result;

virt_addr = (short *) 0x21000000;

read_result = *( (short *) virt_addr);

I can see that there is no keyword "volatile" anywhere in this code. Perhaps I can tell the linker that "short fifo_location" has to be at a given address, and use the keyword "volatile". Not sure how it can be done, but I assume it can be done. Would it solve the problem?

volatile short fifo_location; /* tell the linker it is at address 0x21000000. Not sure how to do it */

read_result = fifo_location;

QuoteReplyEditDelete

2010-10-20 21:15:35 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94787

like i said, refer to the Blackfin io.h file

QuoteReplyEditDelete

2010-10-20 21:46:30 Re: /dev/mem causes CPLB miss

Wojtek Skulski (UNITED STATES)

Message: 94788

The location of the io.h is the following ("include" was missing). /home/uclinux/uClinux-dist/linux-2.6.x/arch/blackfin/include/asm

The file includes the comment "On the bfin architecture, we just read/write the memory location directly" from which I gather that the pointer stuff is fine. The code in the file seems to be just reading/writing the memory via pointers. (But I confess I do not understand the assembly wizardry in the three inline functions.)

In any case, I do not (yet) see any red flag concerning the FIFO access, or how this file would help understand why FIFOs would be a problem.

QuoteReplyEditDelete

2010-10-21 03:53:25 Re: /dev/mem causes CPLB miss

Mike Frysinger (UNITED STATES)

Message: 94810

__asm__ __volatile__ ( \

"cli %1;" \

"NOP; NOP; SSYNC;" \

"%0 = "#asm" [%2] "#asm_sign";" \

"sti %1;" \

you cant do those insns from user space