Comment enquêter sur une réinitialisation aléatoire sur le port client TCP connecté via une interface de bouclage à un serveur

10

J'ai une suite de tests qui initialise un serveur sur l'hôte local et exécute ensuite des milliers de tests de stress qui incluent le démarrage et l'arrêt des clients se connectant à ce serveur local sur la même machine. J'essaie de trouver la raison d'une TVD envoyée par le client au serveur.

Voici la séquence des événements:

  1. le client initialise SYN
  2. Le serveur renvoie SYN, ACK
  3. le client envoie RST (au lieu de ACK et d'établir la connexion)

Une observation est à de nombreuses reprises que les serveurs tournent autour de 99% du processeur bien qu'il revienne à la normale en quelques secondes / minutes.

Au départ, je pensais que cela pourrait être dû au fait que le client utilise un port d'une connexion précédemment fermée qui est dans l'état TIME_WAIT et donc lorsque le serveur envoie un SYN, ACK sur cette connexion client émet RST. Bien que j'étais sceptique quant à la raison pour laquelle le client devrait utiliser un port dans l'état TIME_WAIT, mais j'utilise l'ancien Fedora Core 4 et je pense qu'il n'a peut-être pas correctement implémenté TCP / IP :-).

Maintenant, je ne pense pas que ce soit la raison parce que tout le réglage effectué pour résoudre le problème ci-dessus n'a pas aidé à atténuer le problème. Pour info, voici les changements que j'ai essayés /etc/sysctl.conf:

net.ipv4.ip_local_port_range = 1024     65535
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_tw_reuse = 1

Encore un point, ce problème n'est pas cohérent. En fait, je ne peux reproduire ce problème qu'après avoir parcouru 15 à 35 cycles de redémarrage du serveur et d'exécution de tous les tests client.

J'ai manqué de toutes les raisons pour lesquelles je peux penser pourquoi cela pourrait se produire, alors faites-moi savoir si quelqu'un a des idées sur ce qui pourrait arriver.

Détails du paquet

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378209 1032719 1858.494749 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=0 WS=128

Frame 1032719: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378209000 EDT
    Epoch Time: 1403722185.378209000 seconds
    [Time delta from previous captured frame: 0.005893000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 1858.494749000 seconds]
    Frame Number: 1032719
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf489 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378222 1032720 1858.494762 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 navisphere > netview-aix-5 [SYN, ACK] Seq=0 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1587252 TSecr=1587252 WS=128

Frame 1032720: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378222000 EDT
    Epoch Time: 1403722185.378222000 seconds
    [Time delta from previous captured frame: 0.000013000 seconds]
    [Time delta from previous displayed frame: 0.000013000 seconds]
    [Time since reference or first frame: 1858.494762000 seconds]
    Frame Number: 1032720
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 0, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0xf7fa [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1587252, TSecr 1587252
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032719]
        [The RTT to ACK the segment was: 0.000013000 seconds]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:45.378228 1032721 1858.494768 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032721: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:45.378228000 EDT
    Epoch Time: 1403722185.378228000 seconds
    [Time delta from previous captured frame: 0.000006000 seconds]
    [Time delta from previous displayed frame: 0.000006000 seconds]
    [Time since reference or first frame: 1858.494768000 seconds]
    Frame Number: 1032721
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375927 1032722 1861.492467 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=0 WS=128

Frame 1032722: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375927000 EDT
    Epoch Time: 1403722188.375927000 seconds
    [Time delta from previous captured frame: 2.997699000 seconds]
    [Time delta from previous displayed frame: 2.997699000 seconds]
    [Time since reference or first frame: 1861.492467000 seconds]
    Frame Number: 1032722
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xf19b [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375950 1032723 1861.492490 ::1                   ::1                   TCP      94     navisphere  netview-aix-5 [TCP Previous segment lost] navisphere > netview-aix-5 [SYN, ACK] Seq=2997725 Ack=1 Win=32728 Len=0 MSS=16376 SACK_PERM=1 TSval=1588002 TSecr=1588002 WS=128

Frame 1032723: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375950000 EDT
    Epoch Time: 1403722188.375950000 seconds
    [Time delta from previous captured frame: 0.000023000 seconds]
    [Time delta from previous displayed frame: 0.000023000 seconds]
    [Time since reference or first frame: 1861.492490000 seconds]
    Frame Number: 1032723
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: Bad TCP]
    [Coloring Rule String: tcp.analysis.flags]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: navisphere (2162), Dst Port: netview-aix-5 (1665), Seq: 2997725, Ack: 1, Len: 0
    Source port: navisphere (2162)
    Destination port: netview-aix-5 (1665)
    [Stream index: 3374]
    Sequence number: 2997725    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 40 bytes
    Flags: 0x12 (SYN, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgement: Set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32728
    [Calculated window size: 32728]
    Checksum: 0x3414 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1588002, TSecr 1588002
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1032722]
        [The RTT to ACK the segment was: 0.000023000 seconds]
        [TCP Analysis Flags]
            [A segment before this frame was lost]
                [Expert Info (Warn/Sequence): Previous segment lost (common at capture start)]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:48.375958 1032724 1861.492498 ::1                   ::1                   TCP      74     netview-aix-5 navisphere netview-aix-5 > navisphere [RST] Seq=1 Win=0 Len=0

Frame 1032724: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
    Arrival Time: Jun 25, 2014 14:49:48.375958000 EDT
    Epoch Time: 1403722188.375958000 seconds
    [Time delta from previous captured frame: 0.000008000 seconds]
    [Time delta from previous displayed frame: 0.000008000 seconds]
    [Time since reference or first frame: 1861.492498000 seconds]
    Frame Number: 1032724
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP RST]
    [Coloring Rule String: tcp.flags.reset eq 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 20
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 1, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 1    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x04 (RST)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .1.. = Reset: Set
            [Expert Info (Chat/Sequence): Connection reset (RST)]
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 0
    [Calculated window size: 0]
    [Window size scaling factor: 128]
    Checksum: 0x4eea [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

Date & Time                No.     Time        Source                Destination           Protocol Length Source port Dest port Info
2014-06-25 14:49:54.376072 1032725 1867.492612 ::1                   ::1                   TCP      94     netview-aix-5 navisphere netview-aix-5 > navisphere [SYN] Seq=0 Win=32752 Len=0 MSS=16376 SACK_PERM=1 TSval=1589502 TSecr=0 WS=128

Frame 1032725: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
    Arrival Time: Jun 25, 2014 14:49:54.376072000 EDT
    Epoch Time: 1403722194.376072000 seconds
    [Time delta from previous captured frame: 6.000114000 seconds]
    [Time delta from previous displayed frame: 6.000114000 seconds]
    [Time since reference or first frame: 1867.492612000 seconds]
    Frame Number: 1032725
    Frame Length: 94 bytes (752 bits)
    Capture Length: 94 bytes (752 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:tcp]
    [Coloring Rule Name: TCP SYN/FIN]
    [Coloring Rule String: tcp.flags & 0x02 || tcp.flags.fin == 1]
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Destination: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:00:00_00:00:00 (00:00:00:00:00:00)
        Address: 00:00:00_00:00:00 (00:00:00:00:00:00)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: ::1 (::1), Dst: ::1 (::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 40
    Next header: TCP (0x06)
    Hop limit: 64
    Source: ::1 (::1)
    Destination: ::1 (::1)
Transmission Control Protocol, Src Port: netview-aix-5 (1665), Dst Port: navisphere (2162), Seq: 0, Len: 0
    Source port: netview-aix-5 (1665)
    Destination port: navisphere (2162)
    [Stream index: 3374]
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x02 (SYN)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...0 .... = Acknowledgement: Not set
        .... .... 0... = Push: Not set
        .... .... .0.. = Reset: Not set
        .... .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port navisphere]
        .... .... ...0 = Fin: Not set
    Window size value: 32752
    [Calculated window size: 32752]
    Checksum: 0xebbf [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (20 bytes)
        Maximum segment size: 16376 bytes
        TCP SACK Permitted Option: True
        Timestamps: TSval 1589502, TSecr 0
        No-Operation (NOP)
        Window scale: 7 (multiply by 128)

iptables-save production:

# Generated by iptables-save v1.3.0 on Thu Jun 26 10:15:33 2014
*filter
:FORWARD ACCEPT [0:0]
:INPUT ACCEPT [675:236200]
:OUTPUT ACCEPT [241:21540]
COMMIT
# Completed on Thu Jun 26 10:15:33 2014
fedora tcp tcpdump tcp-ip carolinien
la source
Le socket du client est-il en mode non bloquant? Est-il possible que le socket client soit fermé au moment où le serveur accepte? Avez-vous des horaires pour ces paquets? Un pcap peut-être?
Stéphane Chazelas
1
Vérifiez-vous la valeur de retour du connect()s? Est-il possible que vous fassiez deux non-blocage connect()ssur le même socket? Est-il possible que certains clients soient tués ou interrompus d'une manière ou d'une autre ( alarm()?) Au milieu d'un connect()? Pouvez-vous le reproduire avec le client sous strace -ffou autrace?
Stéphane Chazelas
1
Quelque chose de fantaisiste (nfqueue, limitation de connexion ...) dans la sortie de iptables-save? Possibilité qu'une certaine limite de ressources soit atteinte? Combien de clients simultanés? Faites-les bindà un port source spécifique?
Stéphane Chazelas
1
Serait mieux ajouté à la question. La question est plus de savoir si ces limites sont atteintes (ou probablement atteintes) dans votre test que ce qu'elles sont réellement.
Stéphane Chazelas
1
Je suppose que le stress exercé sur la pile entraîne une chute de connexion du côté client du délai d'attente à l'état semi-ouvert. Cette connexion est supprimée (ce qui signifie que l'application cliente doit la voir comme un ETIMEDOUT), et lorsque la synchronisation / l'acquittement est finalement traitée par la pile, il n'y a plus de connexions à associer, et donc elle est réinitialisée. Avez-vous essayé d'augmenter la taille du backlog syn ( sysctl -w net.ipv4.tcp_max_syn_backlog=???)?
didierc

Réponses:

1

Je veux juste conclure ce fil avec la solution adoptée en l'absence d'un correctif à la cause racine qui, je suppose, se trouve dans l'implémentation de la pile Fedora 4 TCP. Comme solution, j'ai simplement introduit une tentative de reconnexion immédiate au cas où la panne est due à ETIMEDOUT et le protocole utilisé est IPV6. Cela a résolu le problème pour moi et mon équipe pour de bon, avec un risque possible de retard supplémentaire de 3 minutes dans le signalement d'une erreur dans tout autre problème de connexion entraînant ETIMEDOUT. Ce n'est pas la solution réelle / idéale au problème, mais nous permet de passer à autre chose ... car cela n'affecte que notre suite de tests automatisés et n'est pas expédié au client. J'espère que quelqu'un qui connaît assez bien la mise en œuvre de la pile Fedora 4 TCP / IP résoudra ce mystère pour de bon.

carolinien
la source