[#5067] TCP crash after two SYNs from same source port

Document created by Aaronwu Employee on Sep 3, 2013
Version 1Show Document
  • View in full screen mode

[#5067] TCP crash after two SYNs from same source port

Submitted By: Jon Kowal

Open Date

2009-04-14 08:14:06     Close Date

2009-04-14 08:55:09

Priority:

Medium     Assignee:

Nobody

Status:

Closed     Fixed In Release:

N/A

Found In Release:

2008R1.5-RC3     Release:

Category:

Networking     Board:

Custom

Processor:

BF537     Silicon Revision:

Is this bug repeatable?:

Yes     Resolution:

Out of Date

Uboot version or rev.:

    Toolchain version or rev.:

not sure (probably

App binary format:

N/A     

Summary: TCP crash after two SYNs from same source port

Details:

 

A customer of us managed to send two TCP SYNs with different sequence numbers but equal source port from his windows client application to our bf537 board. Those two network packets cause the following Kernel Exception (see below my text).

 

I managed to track it down to the call of csum_tcpudp_nofold() in tcp_v4_send_ack() in the linux kernel file net/ipv4/tcp_ipv4.c.

 

If I remove the call to csum_tcpudp_nofold(), no crash occurs but packets with invalid checksums are generated.

 

If I put a printk() right after the call, no crash occurs.

 

If I remove the inline attribute from the csum_tcpudp_nofold() function in include/asm-blackfin/checksum.h, no crash occurs. I used this as workaround to fix the bug for me, but it's surely no permanent fix.

 

So I assume there's something wrong with the assembler code in csum_tcpudp_nofold() that causes the crash whenever the function is compiled inline.

 

I attached a wireshark-log: The first two packets cause the problem. This log shows the behaviour after I implemented the workaround. So the blackfin didn't freeze. Before the workaround packet 5 would not be transmitted by the blackfin.

 

I attached the tool eth_send that I wrote to reproduce the network packets on a Linux PC. The tool sends packets to destination 192.168.100.32 port 50000 from source 192.168.100.5. You may have to change MAC adresses and TCP checksum (wireshark will calculate it for you) in the source to make it work.

 

Maybe this bug is related to #3614

 

 

----------

NULL pointer access (probably)

Kernel OOPS in progress

Defered Exception context

 

No Valid process in current context

return address: [0x00110aa6]; contents of:

0x00110a80:  030c  1002  514d  5155  030c  1002  514d  5145

0x00110a90:  030c  1002  514d  0000  6040  b3b5  b3f0  1807

0x00110aa0:  0000  0000  0000 [a058] e630  0010  e140  0019

0x00110ab0:  e100  adf0  e3f7  e654  e14a  001e  e10a  da08

 

SEQUENCER STATUS:               Not tainted

SEQSTAT: 00000027  IPEND: c030  SYSCFG: 0006

  HWERRCAUSE: 0x0

  EXCAUSE   : 0x27

  physical IVG14 asserted : <0xffa00af0> { _evt14_softirq + 0x0 }

  physical IVG15 asserted : <0xffa00e68> { _evt_system_call + 0x0 }

  logical irq   6 mapped  : <0xffa00258> { _timer_interrupt + 0x0 }

  logical irq  12 mapped  : <0x000de990> { _rx_handler + 0x0 }

  logical irq  13 mapped  : <0x000de9f0> { _tx_handler + 0x0 }

  logical irq  16 mapped  : <0x000c25a8> { _bfin_twi_interrupt_entry + 0x0 }

  logical irq  18 mapped  : <0x000a516c> { _bfin_serial_dma_rx_int + 0x0 }

  logical irq  19 mapped  : <0x000a4d14> { _bfin_serial_dma_tx_int + 0x0 }

  logical irq  24 mapped  : <0x000ae6f8> { _bf537mac_interrupt + 0x0 }

  logical irq  45 mapped  : <0x000df4d8> { _err_handler + 0x0 }

  logical irq  73 mapped  : <0x03639184> { :thales:_pca_irq_handler + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x001b1c40> /* unknown address */

RETX: <0x00110aa6> { _tcp_v4_send_ack + 0x11a }

RETS: <0x00110a60> { _tcp_v4_send_ack + 0xd4 }

PC  : <0x00110aa6> { _tcp_v4_send_ack + 0x11a }

DCPLB_FAULT_ADDR: <0x00000004> /* Maybe null pointer? */

ICPLB_FAULT_ADDR: <0x00110aa6> { _tcp_v4_send_ack + 0x11a }

 

PROCESSOR STATE:

R0 : 00000008    R1 : 00000001    R2 : 2064a8c0    R3 : 0564a8c0

R4 : 55da7261    R5 : 39c95780    R6 : 0000000f    R7 : 0016d016

P0 : 0000000a    P1 : 001b5158    P2 : 03349020    P3 : 00000000

P4 : 037ef940    P5 : 03349034    FP : 0318d7bc    SP : 001b1b64

LB0: ffa01db4    LT0: ffa01db4    LC0: 00000000

LB1: 000f81be    LT1: 000f81be    LC1: 00000000

B0 : 7e2dc1fd    L0 : 00000000    M0 : 00000000    I0 : 2064a8c0

B1 : 2064a8c0    L1 : 00000000    M1 : 00000000    I1 : 00000000

B2 : d6e5cd5a    L2 : 00000000    M2 : 00000000    I2 : ffff1a45

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 000000a0   A0.x: 00000000   A1.w: 000000a0   A1.x: 00000000

USP : 001b2000  ASTAT: 02002000

 

Hardware Trace:

   0 Target : <0x00004884> { _trap_c + 0x0 }

     Source : <0xffa00774> { _exception_to_level5 + 0xb4 }

   1 Target : <0xffa006c0> { _exception_to_level5 + 0x0 }

     Source : <0xffa0061c> { _ex_trap_c + 0x5c }

   2 Target : <0xffa005c0> { _ex_trap_c + 0x0 }

     Source : <0xffa0044a> { _ex_workaround_261 + 0x22 }

   3 Target : <0xffa00428> { _ex_workaround_261 + 0x0 }

     Source : <0xffa00814> { _trap + 0x28 }

   4 Target : <0xffa007ec> { _trap + 0x0 }

     Source : <0xffa00562> { _bfin_return_from_exception + 0xe }

   5 Target : <0xffa00554> { _bfin_return_from_exception + 0x0 }

     Source : <0xffa0043a> { _ex_workaround_261 + 0x12 }

   6 Target : <0xffa00428> { _ex_workaround_261 + 0x0 }

     Source : <0xffa00814> { _trap + 0x28 }

   7 Target : <0xffa007ec> { _trap + 0x0 }

     Source : <0x00110aa4> { _tcp_v4_send_ack + 0x118 }

   8 Target : <0x00110a96> { _tcp_v4_send_ack + 0x10a }

     Source : <0x00110a92> { _tcp_v4_send_ack + 0x106 }

   9 Target : <0x00110a8e> { _tcp_v4_send_ack + 0x102 }

     Source : <0x00110a8a> { _tcp_v4_send_ack + 0xfe }

  10 Target : <0x00110a86> { _tcp_v4_send_ack + 0xfa }

     Source : <0x00110a82> { _tcp_v4_send_ack + 0xf6 }

  11 Target : <0x00110a60> { _tcp_v4_send_ack + 0xd4 }

     Source : <0x0000d772> { _printk + 0x16 }

  12 Target : <0x0000d76e> { _printk + 0x12 }

     Source : <0x0000d626> { _vprintk + 0x1c2 }

  13 Target : <0x0000d61a> { _vprintk + 0x1b6 }

     Source : <0xffa00d08> { __common_int_entry + 0xd8 }

  14 Target : <0xffa00ca6> { __common_int_entry + 0x76 }

     Source : <0xffa00ad8> { _return_from_int + 0x58 }

  15 Target : <0xffa00ad8> { _return_from_int + 0x58 }

     Source : <0xffa00aae> { _return_from_int + 0x2e }

Stack from 001b1b44:

        00000000 ffa00778 001b456c 001b456c 001b4568 2064a8c0 001b2000 001b51b4

        00110aa6 0000c030 00000027 00000000 001b1c40 00110aa6 00110aa6 00110a60

        00000008 02002000 000f81be ffa01db4 000f81be ffa01db4 00000000 00000000

        000000a0 00000000 000000a0 00000000 00000000 d6e5cd5a 2064a8c0 7e2dc1fd

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 ffff1a45 00000000 2064a8c0 001b2000 0318d7bc 03349034 037ef940

 

Call Trace:

[<000e1ac0>] _sk_reset_timer+0x10/0x3c

[<00111a00>] _tcp_v4_reqsk_send_ack+0x24/0x2c

[<000016d0>] _copy_thread+0x1c/0x44

[<00113c8c>] _tcp_check_req+0x6c/0x290

[<000e4bd0>] _kfree_skbmem+0xc/0xb4

[<000e44c8>] _skb_checksum+0x38/0x228

[<0000a810>] _wake_up_process+0x0/0x10

[<001117a0>] _tcp_v4_do_rcv+0xdc/0x310

[<001138d4>] _tcp_v4_rcv+0x7a8/0x7dc

[<000fb390>] _ip_local_deliver+0xd4/0x1dc

[<000fb790>] _ip_rcv+0x2f8/0x378

[<000fb678>] _ip_rcv+0x1e0/0x378

[<0000994a>] ___activate_task+0x1a/0x34

[<000096e0>] ___wake_up_common+0x34/0x54

[<000e9730>] _netif_receive_skb+0x190/0x208

[<0001f718>] _hrtimer_wakeup+0x0/0x20

[<000eaeb2>] _process_backlog+0x7a/0x120

[<0000ae3a>] _scheduler_tick+0x26/0x84

[<000eafc6>] _net_rx_action+0x6e/0xf8

[<000119a0>] ___do_softirq+0x60/0xa8

[<0000ffff>] _do_exit+0x4d3/0x780

[<0000ffff>] _do_exit+0x4d3/0x780

[<00040000>] _sys_renameat+0x94/0x180

[<001c4a4c>] _start_kernel+0x218/0x26c

[<001c437c>] _unknown_bootoption+0x0/0x220

[<001c41cc>] _real_start+0x50/0x94

                                                                                                                                        

 

Follow-ups

 

--- Mike Frysinger                                           2009-04-14 08:18:40

there have been checksum fixes applied to the branch, so you should make sure to

use that rather than the exact release.

 

--- Michael Hennerich                                        2009-04-14 08:50:02

Yeah - this issue was fixed on svn trunk and the 2008R1 branch.

It was caused by a missing clobber of "CC" in our checksum

assembly statement.

 

Like Mike said update your tree from our release branch.

-Michael

 

--- Jon Kowal                                                2009-04-14 08:53:30

Yes, I just checked the branch and it's been fixed. Thanks!

 

Is it safe to use the branches for production releases?

 

--- Mike Frysinger                                           2009-04-14 08:55:09

yes ... we only commit fixes to branches as that's what they're for -- we do

development in trunk

 

 

 

    Files

    Changes

    Commits

    Dependencies

    Duplicates

    Associations

    Tags

 

File Name     File Type     File Size     Posted By

tcp_crash_after_2syns.pcap    application/cap    474    Jon Kowal

eth_send.c    text/x-csrc    3753    Jon Kowal

Outcomes