write speed SD card sd-mode BF707

I am using a BF707 to write files to a micro sd card. My files a 4MB in size. My sd card clock is 50MHz and I have 4 data lines, so I should be able to write at least 5 files per second I thought. But in reality I reach only 2 files max and often I see the time increasing and I am writing only 1 fps.

I used the Analog Device rsi.c files, and the Fatfs the write in fat32 to the card.

Does any one know how I can improve the performance, or what is the reason why my performance varies som much and why is it even in best case so much lower than I had expected from a clock of 50MHz.