2010-10-01 10:21:24     KGDB debugging of kernel module

Document created by Aaronwu Employee on Aug 23, 2013
Version 1Show Document
  • View in full screen mode

2010-10-01 10:21:24     KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94107   

 

Hello All,

 

To give you some background we have developed a board based on  the BF537-Stamp board.  Currently we have modified u-boot for our board and setup uClinux to correctly load most of our requirements.  The last part we need to sort out is the zaptel drivers for COTS FXO and FXS boards (rowetel). 

 

I have loaded the zaptel drivers and we see nothing coming on the DTX line of the FXO or FXS board.  We see the clock is working, DRX is working.  We have decided the best way to debug this is to use KGDB.  I have set the system up to allow for break points following:

 

  docs.blackfin.uclinux.org/doku.php?id=linux-kernel:kgdb_2008r1.5

 

I can place a break point at sys_open and when i run the ls command I can step using DDD.  My question is:

 

How can I debug the zaptel and wcfxs drivers.  They are loaded during boot (or when i insmod them) and would like to attach to the process either during boot (this would be preferable if possible) otherwise i can attach just after insmod.  I am just not really sure on how the add-symbol-file works tried it but kept getting error that the file doesn't exist i am obviously doing something wrong here.

 

 

 

Any help will be appreciated.

 

Louis

QuoteReplyEditDelete

 

 

2010-10-01 15:13:52     Re: KGDB debugging of kernel module

Mike Frysinger (UNITED STATES)

Message: 94108   

 

post the actual commands you're running in gdb and the actual output you get back

 

the documentation you've found shows how to debug modules via kgdb.  i dont really have anything to add over that.

QuoteReplyEditDelete

 

 

2010-10-04 06:40:01     Re: KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94165   

 

Hello All,

 

Ok so i did some more reading and was able to make sense of the doc on the kgdb debugging.  I ran the script provided and got the following output as shown in scriptOutput.txt.  I then use this a plug it into the gdb debugger and get the output as shown in withstrsym.txt.

 

Because it gave errors on .strtab and .symtab I left them out during the add symbol file and got the output as shown in withoutstrsym.txt.

 

This seems to add the symbols correctly (but I am not sure because it doesn't give error which is why i think it adds it correctly).

 

I then try and add a breakpoint at the wcfxs_int function and I get the error function not defined.  What am I doing wrong here.  The only thing I think i could be doing wrong is the /path/to/module

 

Thanks in advance,

 

Louis

 

 

withoutstrsym.txt

withstrsym.txt

scriptOutput.txt

QuoteReplyEditDelete

 

 

2010-10-04 06:43:34     Re: KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94166   

 

Hello All,

 

On a another point.  When I add the wcfxs and zaptel modules, it tries to read the devices attached and does not get to the device, as I previously said we have nothing on the scope form the blackfin.  I can then attach to the module using gdb but at this point it has already been loaded and tried to read.  I need to break while it is trying to discover the devices.  Once I have insmod'ed the modules can i remove and insmod them again and then break on wcfxs_init?  Or how can I set up the debugger to attach before insmod'ed the module?

 

 

 

Kind regards,

 

Louis

QuoteReplyEditDelete

 

 

2010-10-04 10:10:15     Re: KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94169   

 

Hey All,

 

So I have been playing around with this for a while and saw that I am loading the vmlinux still at the beginning which I believe is incorrect.  I should only load the symbols of the module.  I have now loaded the modules and can place a break point(somewhat).

 

I place a break point at wcfxs_init and it goes to the correct c file and places a break point at line 2478 which is actually the line for wcfxs_cleanup.  So tried to place a break point at clean up and it puts it at the same line.  Next I tried to place a break point at wcfxs_init_one - this says it is not defined.  From the c code I see that these to functions (where I can put break points) are marked as the init and exit for the module.  Could this be the reason that it finds it?  Must I compile the module with debug mode or something like that.

 

Also I cannot test if these break points work (when i rmmod the drivers i get a kernel panic)

 

And I cannot re insmod the modules?  I.E how do i attach to the module before I insmod it so I can break as it calls the init fucntion.

 

 

 

I have attached the gdb output if you are interested.

 

 

 

Thanks Louis

 

somewhatworking.txt

QuoteReplyEditDelete

 

 

2010-10-05 10:44:10     Re: KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94192   

 

Hey All,

 

Just a follow up to see if anyone has any suggestions on what I can do to resolve this problem.  Didn't have time today to work anymore will continue tomorrow.

 

Thanks,

 

Louis

QuoteReplyEditDelete

 

 

2010-10-06 02:56:37     Re: KGDB debugging of kernel module

Mike Frysinger (UNITED STATES)

Message: 94217   

 

as soon as `insmod` returns, all __init sections have been freed.

 

you could trigger a break point manually in the code by calling:

    arch_kgdb_breakpoint();

QuoteReplyEditDelete

 

 

2010-10-07 03:47:02     Re: KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94309   

 

Hey,

 

I added the arch_kgdb_breakpoint function call, whoe ever when i try and build its gives me a compile error:

 

ERROR: "arch_kgdb_breakpoint" [drivers/zaptel/wcfxs.ko\ undefined!

 

I therefore assumed I am not included the correct h files so I grep'd for:

 

grep -ir "arch_kgdb_breakpoint" * and only found results in the wcfxs.c file where I placed it.  I am using uClinux 2008R1-RC8.  I have patch the kernel as explained in the doc I mentioned earlier.  Is there another patch I need to apply to get that function?

 

Thanks,

 

Louis

QuoteReplyEditDelete

 

 

2010-10-07 06:53:49     Re: KGDB debugging of kernel module

Louis Ruch (SOUTH AFRICA)

Message: 94314   

 

Hey Mike,

 

So I have forced the system to hit a breakpoint when i insmod the module.  Instead of using the arch_kgdb_breakpoint() function I found the code for what is called by that function and just include that line at the beginning of the init function:

 

asm("   ECPT 2;");

 

Now when I load the module (insmod) I get my printf saying "calling breakpoint" (I inserted a printk statement before and after).  And then the DDD does hit a break point however it is not in the correct location (source file).  I assume this is because the symbols and source are not matching up correctly. 

 

Like I said previously this is where my problem is: I need to insmod the module to get the symbols, however I need the symbols before I insmod to be able to step through the module init function.

 

My steps:

 

from ternimal run ddd - debugger bfin-elf-gdb

 

open program: select vmlinux (elf file)

 

target remote udp:192.168.0.159:6443

 

type 'c' to continue

 

insmod module and then i hit the breakpoint but with mismatched source and symbols.

 

Thanks for any help you can provide,

 

Louis

QuoteReplyEditDelete

 

 

2010-10-07 11:35:04     Re: KGDB debugging of kernel module

Mike Frysinger (UNITED STATES)

Message: 94316   

 

so patch the module loader code to printk the section & address so that you know where things are

QuoteReplyEditDelete

 

 

2010-10-09 06:45:14     Re: KGDB debugging of kernel module

Sonic Zhang (CHINA)

Message: 94357   

 

You best choice with kgdb is to select kgdb over uart kernel option, set kgdb uart port to free one other than you console and trap kernel in kgdb early in boot stage with boot parameter "kgdbwait".

 

Then, you can built you module into kernel image and set break point at the beginning.

QuoteReplyEditDelete

Outcomes