[#5662] Uncached DMA region no longer being uncached

Submitted By: Michael Hennerich

Open Date

2009-11-04 05:53:35 Close Date

2009-12-14 10:46:00

Priority:

Medium High Assignee:

Barry Song

Status:

Closed Fixed In Release:

N/A

Found In Release:

2010R1 Release:

trunk

Category:

Memory Board:

STAMP

Processor:

ALL Silicon Revision:

any

Is this bug repeatable?:

Yes Resolution:

Fixed

Uboot version or rev.:

trunk Toolchain version or rev.:

trunk

App binary format:

N/A

Summary: Uncached DMA region no longer being uncached

Details:

Uncached DMA region no longer being uncached.

The current code that polulates the DCPLB tables places a 4MB DCPLB at 0x03c00000

This covers the uncached region starting at 0x03f00000.

dma_alloc_init: dma_page @ 0x02680000 - 256 pages at 0x03f00000

root:/> cat /proc/cplbinfo/cpu0/dcplb

DCPLBs are enabled: 0x100f

Index Address Data Size U/RD U/WR S/WR Switch

0 0x00000000 00083 1K N N N N

1 0x00000000 3d09d 4M Y Y Y N

2 0x00400000 3d09d 4M Y Y Y N

3 0xff800000 3009f 4M Y Y Y N

4 0x02000000 3d09d 4M Y Y Y Y

5 0x02400000 3d09d 4M Y Y Y Y

6 0x03c00000 3009d 4M Y Y Y Y

7 0x02800000 3d09d 4M Y Y Y Y

8 0x00000000 00000 1K N N N Y

9 0x00000000 00000 1K N N N Y

10 0x00000000 00000 1K N N N Y

11 0x00000000 00000 1K N N N Y

12 0x00000000 00000 1K N N N Y

13 0x00000000 00000 1K N N N Y

14 0x00000000 00000 1K N N N Y

15 0x00000000 00000 1K N N N Y

root:/>

Follow-ups

--- Michael Hennerich 2009-11-04 06:38:19

Looks like this issue exists on 2009R1 as well ...

-Michael

--- Mike Frysinger 2009-11-04 20:10:50

seems to work on CM-BF537e. could you post your .config ?

DCPLBs are enabled: 0x100f

Index Address Data Size U/RD U/WR S/WR Switch

0 0x00000000 00083 1K N N N N

1 0x00000000 3109d 4M Y Y Y N

2 0xff800000 3009f 4M Y Y Y N

3 0x00400000 3109d 4M Y Y Y Y

4 0x01000000 3109d 4M Y Y Y Y

5 0x01400000 3109d 4M Y Y Y Y

6 0x20000000 3009d 4M Y Y Y Y

7 0x01f00000 2009d 1M Y Y Y Y

8 0x00c00000 3109d 4M Y Y Y Y

9 0x00800000 3109d 4M Y Y Y Y

10 0x01800000 3109d 4M Y Y Y Y

11 0x01c00000 2109d 1M Y Y Y Y

12 0x01d00000 2109d 1M Y Y Y Y

13 0x01e00000 2109d 1M Y Y Y Y

14 0x00000000 00000 1K N N N Y

15 0x00000000 00000 1K N N N Y

--- Mike Frysinger 2009-11-04 20:48:20

hrm, there does seem to be a bug if you poke the hole:

available = 0x003d8000-0x01eff000

DMA Zone = 0x01f00000-0x02000000

if i access 0x01eff001, we end up with:

8 0x01e00000 2009d 1M Y Y Y Y

that region should be cached. guess the question is why does the available

region end one page before the dma zone instead of at the dma zone ...

--- Mike Frysinger 2009-11-04 21:39:04

hmm, that 4k hole is because of your svn 2773 commit Michael ;). but the

changelog doesnt really indicate why it's needed.

#if !defined(CONFIG_MTD_UCLINUX)

/*In case there is no valid CPLB behind memory_end make sure we don't get

to close*/

memory_end -= SIZE_4K;

#endif

--- Graf Yang 2009-11-04 23:01:25

6 0x03c00000 3009d 4M Y Y Y Y

The '3009d' means none cached 4M page. So it is still be uncached.

--- Mike Frysinger 2009-11-04 23:23:10

true, but the nompu code should have generated multiple 1M mappings instead of a

single 4M mapping

--- Graf Yang 2009-11-05 00:01:05

If create the low and high bounds for the DMA_UNCACHED_REGION, the nompu code

will create 1M mappings one by one. Currently it only have high bounds.

But is it necessary?

--- Michael Hennerich 2009-11-05 03:58:19

Mike,

>hmm, that 4k hole is because of your svn 2773 commit Michael ;). but

>the changelog doesnt really indicate why it's needed.

>#if !defined(CONFIG_MTD_UCLINUX)

> /*In case there is no valid CPLB behind memory_end make sure we

>don't get to close*/

> memory_end -= SIZE_4K;

>#endif

IIRC it was a workaround for some HW Errors we've seen

- can't remember which one of the below Anomalies it was -

05000189 - False Protection Exceptions.

05000310 - False hardware errors caused by fetches at the boundary of reserved

memory.

But it was back when we used to have the ext2 fs MTD uclinux at the very top of

memory.

I could have sworn that the DCPLB indicated cacheable - must have been

cross-eyed

Also the bugs I've seen where typical ones when doing DMA on cached memory

without flush/inv.

Nevertheless - this 4MB DCPLD for a 1MB unchecked are is very bad.

It makes the top 3MB of memory which is typically heavily used un-cacheable.

BTW - the I've seen this on BF537-STAMP 1B un-cacheable area compiled for

silicon rev 0.2.

It's actually different when compiled for 0.3

-Michael

--- Mike Frysinger 2009-11-05 08:01:42

at the very least, the code should be checking to see if the DMA region is there

too before removing 4k -- the DMA region gives plenty of buffer space. and it

should have "if (ANOMALY_xxx)".

ignoring that, anomaly 310 also applies to L1 regions too and we dont handle

those. a better workaround for that would be to update the hardware error

handler like the anomaly sheet suggests -- ignore misses when the address in

question has a valid mapping. shouldnt be a lot of overhead here since hardware

errors rarely occur.

anomaly 189 should be handled already in the traps code since that applies to

edge of cplb mappings, not just edge of valid memory, and backing off the memory

limit wouldnt change the rest.

--- Barry Song 2009-11-20 02:45:21

BTW, is there anyone who can tell me why we need a reserved dma uncache area? It

seems it is not a common way. Why not get memory from normal area directly and

set uncacheable/cacheable at run-time? If blackfin DMA can access all memory

bus, why we need DMA zone?

And I am also strange why we call invalid/flush cache directly for DMA transfer

in many drivers. Why not we use dma_map_single families stream dma api?

--- Michael Hennerich 2009-11-20 03:46:45

>BTW, is there anyone who can tell me why we need a reserved dma uncache

>area? It seems it is not a common way. Why not get memory from normal

>area directly and set uncacheable/cacheable at run-time?

We would end up with the MPU case - where we cover the entire memory with 4k

CPLBs.

Memory would come from the common memory pool and some instance would need to

take care that DMA memory is set uncacheable.

Answering your question - For performance reasons...

>If blackfin DMA

>can access all memory bus, why we need DMA zone?

To avoid cache coherency problems, and to avoid Core flush/invalidate cycles.

Think about the framebuffer case -

User space asynchronously writes into some mmaped memory area.

While the PPI DMA runs in autobuffer mode.

>And I am also strange why we call invalid/flush cache directly for DMA

>transfer in many drivers. Why not we use dma_map_single families stream

>dma api?

It's up to the drivers author - if a driver is supposed to work on other

architectures, too.

You should use dma_map_single and friends.

But this doesn't change the fact that you need to flush/invalidate on every

DMA/Core access transition.

--- Barry Song 2009-11-20 04:12:52

>(enter your response here)

--- Barry Song 2009-11-20 04:18:56

>>If blackfin DMA

>>can access all memory bus, why we need DMA zone?

>To avoid cache coherency problems, and to avoid Core

>flush/invalidate cycles.

>Think about the framebuffer case -

>User space asynchronously writes into some mmaped memory area.

>While the PPI DMA runs in autobuffer mode.

I know the dma cache coherency issue. Here my question is actually the same

with the 1st one. I just suspect why we need a resevered area.

Since it is due to performance, I understand our current design now. If our

hardware has good support in MPU, we shouldn't reserve this kind of area in

fact.

>>And I am also strange why we call invalid/flush cache directly for DMA

>>transfer in many drivers. Why not we use dma_map_single

>families stream

>>dma api?

>It's up to the drivers author - if a driver is supposed to

>work on other architectures, too.

>You should use dma_map_single and friends.

>But this doesn't change the fact that you need to

>flush/invalidate on every DMA/Core access transition.

I didn't find we have fulfilled these APIs. To be compatible, do you think we

should start a task for it?

--- Sonic Zhang 2009-11-20 04:46:34

Because when init memory pages, kernel DMA zone should be located before the

NORMAL zone. But, in NOMMU system, the beginning of the physical memory should

be reserved for zero page and zero page should be part of the NORMAL zone. So,

there is no space for a DMA zone in between. In MMU system, there is no physical

memory for a zero page, it is fine to insert DMA zone before the NORMAL zone.

This is a limitation of the kernel memory zone design.

--- Barry Song 2009-11-20 04:57:09

>Because when init memory pages, kernel DMA zone should be located before

the

>NORMAL zone. But, in NOMMU system, the beginning of the physical memory

should

>be reserved for zero page and zero page should be part of the NORMAL zone.

So,

>there is no space for a DMA zone in between. In MMU system, there is no

physical

>memory for a zero page, it is fine to insert DMA zone before the NORMAL

zone.

>This is a limitation of the kernel memory zone design.

I think I am talking about different problems with you. The purpose of DMA ZONE

is for old ISA which can only access <16MB memory bus. Modern embedded

architecture has no this limit. So DMA memory is allocated from normal memory

and DMA zone size is 0. So my original thinking is deleting this zone on

blackfin.

--- Barry Song 2009-11-20 05:03:30

Or we can say all memory is in DMA ZONE for modern embedded architecture since

DMA can access all memory bus area.

--- Mike Frysinger 2009-11-20 15:01:22

we will still always have this uncached zone for people to leverage.

there is a tracker open already for filling out the dma_map_single and related

functions, but i dont think anyone has picked it up.

i dont think there's a problem marking all memory as being capable of DMA-ing

(dma zone), but i havent looked too much at the different zoning aspects.

--- Robin Getz 2009-11-20 16:21:08

Just to be pedantic -- You can not DMA into/out of L1 sratchpad - so it is not

"all memory bus area".

-Robin

--- Barry Song 2009-11-22 23:37:46

>Just to be pedantic -- You can not DMA into/out of L1 sratchpad - so it is

not

>?"all memory bus area".

That doesn't matter since L1 is not managed by kernel mm in blackfin. That will

not affect the kernel DMA ZONE layout.

--- Robin Getz 2009-11-23 06:55:43

Barry:

You didn't say "all kernel managed memory" - you said "all

memory".

Like I said - it is pedantic.

--- Barry Song 2009-11-23 22:00:40

>true, but the nompu code should have generated multiple 1M mappings instead

of a

>single 4M mapping

According to

MGR_ATTR static int dcplb_miss(int cpu)

{

...

addr1 = addr & ~(SIZE_4M - 1);

addr &= ~(SIZE_1M - 1);

d_data |= PAGE_SIZE_1MB;

if (addr1 >= base && (addr1 + SIZE_4M) <= eaddr) {

* This works because

* (PAGE_SIZE_4MB & PAGE_SIZE_1MB) == PAGE_SIZE_1MB.

d_data |= PAGE_SIZE_4MB;

addr = addr1;

}

/* Pick entry to evict */

idx = evict_one_dcplb(cpu);

...

}

The code actually creates 1MB entry for DMA uncache area if the area is

1MB/2MB. And it creates 4MB entry for 4MB DMA uncache area. Only when

ANOMALY_05000263 exists, the last dcplb bound will be >4MB less than ram_end,

then 4MB area is created for 1MB/2MB DMA area. But there is no problem due to

that memory is actually not useful.

--- Barry Song 2009-11-23 22:01:07

>true, but the nompu code should have generated multiple 1M mappings instead

of a

>single 4M mapping

According to

MGR_ATTR static int dcplb_miss(int cpu)

{

...

addr1 = addr & ~(SIZE_4M - 1);

addr &= ~(SIZE_1M - 1);

d_data |= PAGE_SIZE_1MB;

if (addr1 >= base && (addr1 + SIZE_4M) <= eaddr) {

* This works because

* (PAGE_SIZE_4MB & PAGE_SIZE_1MB) == PAGE_SIZE_1MB.

d_data |= PAGE_SIZE_4MB;

addr = addr1;

}

/* Pick entry to evict */

idx = evict_one_dcplb(cpu);

...

}

The code actually creates 1MB entry for DMA uncache area if the area is

1MB/2MB. And it creates 4MB entry for 4MB DMA uncache area. Only when

ANOMALY_05000263 exists, the last dcplb bound will be >4MB less than ram_end,

then 4MB area is created for 1MB/2MB DMA area. But there is no problem due to

that memory is actually not useful.

--- Barry Song 2009-11-24 00:02:33

Michael,

It seems not right you setup memory_end -= SIZE_4K to keep the last 4K in

memory_end useless since memory_end is usually not the boundary between valid

memory and reserved memory. If we want to use this way to workaround, maybe we

can leave 4K of ramend useless?

And have you once repeated this anomaly and how to repeat it? I use a loop to

read/write data before the real boundary(0x03ffffff), but never trigger any

External Memory Addressing Error.

-Barry

--- Michael Hennerich 2009-11-24 03:11:05

Barry,

I think we can remove the -4k stuff completely.

(Which will fix this bug)

We now have a workaround and the DMA Zone (like Mike said) acts as buffer too.

The whole thing is a leftover from the time we didn't know how to work around.

In addition the our rootfs these days used to be at the very top of memory.

-Michael

--- Mike Frysinger 2009-11-25 16:34:43

the anomaly 310 check needs to be in the "case VEC_HWERR" under

SEQSTAT_HWERRCAUSE_EXTERN_ADDR. then we jump to traps_done if we calculate that

the error was a false positive.

access_ok() is overkill. the anomaly text says it's at the boundary of

reserved memory, so simply use the two limits that are in play here -- the end

of kernel memory (physical_mem_end) and the end of L1 ISRAM.

but before we go banging our heads too much, we really should have a test case

that exercises the anomaly. otherwise we're going to spend a lot of time on

something that might ultimately be completely worth less. it should be checked

into the blkfin-test/ subdir too so we can play with it later. i'm attaching

the VDSP example that was used to reproduce the issue.

i also dont quite trust the anomaly text. it says "hardware error"

which is IVG5, not an exception, so the contents of the FAULT_ADDR MMRs wont be

valid. we should probably double check with the apps team to make sure this

wording is clarified.

--- Barry Song 2009-11-26 22:05:43

I agree 310 should be in the "case VEC_HWERR" under

SEQSTAT_HWERRCAUSE_EXTERN_ADDR. Then it seems 189 should be moved too? For the

two anomalies, we should confirm them then ignore the traps directly without

sending any signal.

I am strange I can't trigger 310 anomaly at memory end by both instruction

fetches and data fetches at all:

instruction fetches:

03ffffd0 <_trigger_boundary_anomaly>:

3ffffd0: e6 6f SP += -0x4; /* ( -4) */

3ffffd2: 13 60 R3 = 0x2 (X); /* R3=0x2(

2) */

3ffffd4: 01 60 R1 = 0x0 (X); /* R1=0x0(

0) */

3ffffd6: 0a 60 R2 = 0x1 (X); /* R2=0x1(

1) */

3ffffd8: 09 20 JUMP.S 0x3ffffea

<_trigger_boundary_anomaly+0x1a>;

3ffffda: 10 0c CC = R0 == 0x2;

3ffffdc: 10 18 IF CC JUMP 0x3fffffc

<_trigger_boundary_anomaly+0x2c>;

3ffffde: 00 0c CC = R0 == 0x0;

3ffffe0: 0e 14 IF !CC JUMP 0x3fffffc

<_trigger_boundary_anomaly+0x2c> (BP);

3ffffe2: 32 93 [SP] = R2;

3ffffe4: 30 91 R0 = [SP];

3ffffe6: 10 0d CC = R0 <= 0x2;

3ffffe8: 08 10 IF !CC JUMP 0x3fffff8

<_trigger_boundary_anomaly+0x28>;

3ffffea: 30 91 R0 = [SP];

3ffffec: 08 0c CC = R0 == 0x1;

3ffffee: f6 17 IF !CC JUMP 0x3ffffda

<_trigger_boundary_anomaly+0xa> (BP);

3fffff0: 33 93 [SP] = R3;

3fffff2: 30 91 R0 = [SP];

3fffff4: 10 0d CC = R0 <= 0x2;

3fffff6: fa 1f IF CC JUMP 0x3ffffea

<_trigger_boundary_anomaly+0x1a> (BP);

3fffff8: 26 6c SP += 0x4; /* ( 4) */

3fffffa: 10 00 RTS;

3fffffc: 31 93 [SP] = R1;

3fffffe: f3 2f JUMP.S 0x3ffffe4

<_trigger_boundary_anomaly+0x14>;

data fetches:

u16 *p = 0x03FFFFD0;

for(i=0;i<24;i++)

*p++ = 0;

p = 0x03FFFFD0;

for(i=0;i<24;i++)

a = *p++;

Both codes can run normally.

--- Barry Song 2009-11-27 03:35:12

I can repeat anomaly 310 at L1 instruction sram boundary by instruction fetch:

External Memory Addressing Error

dcplb_fault_addr:00000000

icplb_fault_addr:00000000

External Memory Addressing Error

Kernel OOPS in progress

HW Error context

CURRENT PROCESS:

COMM=modprobe PID=122 CPU=0

TEXT = 0x02a00040-0x02a49920 DATA = 0x02a49924-0x02a5b894

BSS = 0x02a5b894-0x02a5d214 USER-STACK = 0x02a65f6c

return address: [0xffa0ff0a]; contents of:

0xffa0fee0: 0c10 1880 0c18 187e 0c08 1403 6010 b1f0

0xffa0fef0: a1f0 0c10 1874 0c18 1872 0c08 1403 6010

0xffa0ff00: b1f0 a1f0 0c10 1868 0c18 [1866] 0c08 1403

0xffa0ff10: 6010 b1f0 a1f0 6408 b1f0 a1f0 0c10 1859

dcplb_fault_addr and icplb_fault_addr are both 0.

Files

Changes

Commits

Dependencies

Duplicates

Associations