Linux SysAdmin and Oracle/MySQL DBA: traceroute

Jephe Wu - http://linuxtechres.blogspot.com

Objective: use all kinds of open source free softwares to troubleshoot the Internet vpn slowness issue and pinpoint where is the packet loss router.
Environment: OpenBSD, Freebsd as vpn firewall. When accessing servers such as web servers, Oracle database servers through Internet vpn, we experiencd very slow connection.

Steps:

The following is the vpn network diagram:
10.0.5.x__10.0.5.1||1.2.3.4++++++++++++++++5.6.7.8||10.0.6.1__10.0.6.X

1. Check latency and packet loss from host 1.2.3.4 to 5.6.7.8
a. ping check if you are able to ping from 1.2.3.4 to 5.6.7.8, if yes, check the latency and packet loss rate.
The latency is not accurate because it might be due to icmp rate limiting confiugred by router, only tcp traffic is in priority.
b. if the ping is blocked, use tcptraceroute or traceroute -T (on CentOS 5) and tracetcp on Windows (http://tracetcp.sourceforge.net/), the latency here is more accurate as it's real tcp traffic from sender to receiver, although the return traffic is icmp TTL exceeded message.

How does tcptraceroute and tracetcp work?

When you issue command like 'tracetcp www.redhat.com:443', Wireshark captures the traffic below, for each hop, it will send 3 tcp packets and set TTL starting from 1. When the final destination reached and it gets ACK reply from destination host, it will immediately tear down the connection.

13    3.732592000    192.168.100.20    184.85.48.112    TCP    20527 > https [SYN] Seq=0 Win=16383 Len=0 (time to live is 1 in ip header)
14    3.734182000    192.168.100.1    192.168.100.20    ICMP    Time-to-live exceeded (Time to live exceeded in transit)
15    4.232399000    192.168.100.20    184.85.48.112    TCP    24043 > https [SYN] Seq=0 Win=16383 Len=0 (time to live is 1 in ip header)
16    4.241227000    192.168.100.1    192.168.100.20    ICMP    Time-to-live exceeded (Time to live exceeded in transit)
17    4.732323000    192.168.100.20    184.85.48.112    TCP    13233 > https [SYN] Seq=0 Win=16383 Len=0 (time to live is 1 in ip header)
18    4.735323000    192.168.100.1    192.168.100.20    ICMP    Time-to-live exceeded (Time to live exceeded in transit)

2. use mtr or pathping to check packet loss rate
You can install mtr (http://en.wikipedia.org/wiki/MTR_%28software%29) on Linux/FreeBSD/OpenBSD to check the packet loss rate for the trace path. There are also winmtr and pathping on Windows for similiar functionality.

MTR relies on ICMP Time Exceeded (type 11) packets coming back from routers, or ICMP Echo Reply packets when the packets have hit their destination host.

2082    191.558234    192.168.100.20    184.85.48.112    ICMP    Echo (ping) request
2083    191.590458    203.117.34.14    192.168.100.20    ICMP    Time-to-live exceeded (Time to live exceeded in transit)
2084    191.673071    192.168.100.20    184.85.48.112    ICMP    Echo (ping) request
2085    191.804462    192.168.100.20    184.85.48.112    ICMP    Echo (ping) request
2086    191.874170    198.32.176.127    192.168.100.20    ICMP    Time-to-live exceeded (Time to live exceeded in transit)
........
2090    191.804462    192.168.100.20    184.85.48.112    ICMP    Echo (ping) request
2091    174.139518    184.85.48.112    192.168.100.20    ICMP    Echo (ping) reply

Note: mtr will send icmp ping request with incremental TTL value starting from 1 to the destination host, by getting reply from each hop to get round trip time and packet loss rate.

3. the importance of having no/low packet loss and how to read the mtr report for packet loss
a. Packet loss kills throughput.
b. a slower connection with zero packet loss can easily outperform a faster connection with some packet loss
c. packet loss on the last hop, the desination, is what is most important; packet loss will happen on the return path which is totally different with the outgoing path.
d. sometimes routers in-between will not send ICMP "TTL expired in transit" messages, it will see 3 asterisk which is normal.
e. some routers may specifically block (or down-prioritize) ICMP echo requests, or might do the same where TTL=0. These routers (or the final destination) might show 100% packet loss
f. the router may also be programmed to limit the number of responses it sends to ICMP packets in an effort to mitigate DoS attacks
g. just because you see a hop with high loss doesn't mean it's slowing down "real" traffic; it may only be throwing away ICMP.

References:
a. http://help.rr.com/hmsfaqs/e_packetloss.aspx
2. http://library.linode.com/linux-tools/mtr/   --for how to read mtr report

4. how to configure vpn firewall for icmp traffic in OpenBSD or FreeBSD packet filter firewall

pass out log quick
pass in log quick on $ext inet proto icmp all icmp-type { echorep, timex, unreach }
pass in log quick on $ext inet proto udp from 1.2.3.4 to $ext keep state
pass in log quick on $ext inet proto icmp all icmp-type { echo } from 1.2.3.4 to $ext keep state

References:
ICMP filtering on the firewall -
http://www.richweb.com/icmp_filter

Jephe Wu - http://linuxtechres.blogspot.com

Objective: Understanding how tcp traceroute works and use it to make sysadmin life easier
Environment: Windows Vista, CentOS 5

Concepts:
1. What's TCP traceroute
According to http://michael.toren.net/code/tcptraceroute/. tcptraceroute is a traceroute implementation using TCP packets.

The more traditional traceroute(8) sends out either UDP or ICMP ECHO packets with a TTL of one, and increments the TTL until the destination has been reached. By printing the gateways that generate ICMP time exceeded messages along the way, it is able to determine the path packets are taking to reach the destination.

The problem is that with the widespread use of firewalls on the modern Internet, many of the packets that traceroute(8) sends out end up being filtered, making it impossible to completely trace the path to the destination. However, in many cases, these firewalls will permit inbound TCP packets to specific ports that hosts sitting behind the firewall are listening for connections on. By sending out TCP SYN packets instead of UDP or ICMP ECHO packets, tcptraceroute is able to bypass the most common firewall filters.

2. How to use it under Linux and Windows
You can use tcptraceroute(http://michael.toren.net/code/tcptraceroute/) under Linux and tracetcp(http://tracetcp.sourceforge.net/) under Windows.
Actually, under CentOS 5.5, traceroute command has many options which you can use to do tcp(-T) traceroute by default, you can also use it do tranditioanl udp(-U) traceroute or use icmp(-I) ping packets to do it like tracert on Windows.

3. How tcptraceroute or tracetcp works
it uses tcp syn package and set ttl as 1 as initial packet to send to network. Each hop will decrease ttl by 1, so each hop will generate a time exceed icmp packet back to the sender, those icmp packet includes the original packet information. For the next hop, the sender will use TTL 2 until the destination which will also send back TCP syn/ack reply to the sender.

4. examples - use traceroute to know network path
example 1: tcptraceroute to www.redhat.com
[root@linuxtest ~]# tcptraceroute www.redhat.com -p 443 -f 2
traceroute to www.redhat.com (118.214.80.112), 30 hops max, 40 byte packets
3 172.20.16.65 (172.20.16.65) 27.082 ms 27.637 ms 34.125 ms
4 172.26.16.1 (172.26.16.1) 38.854 ms 38.752 ms 38.627 ms
5 172.20.7.26 (172.20.7.26) 38.446 ms 38.280 ms 38.156 ms
6 172.20.7.82 (172.20.7.82) 38.031 ms 37.900 ms 37.753 ms
7 203.117.34.101 (203.117.34.101) 37.872 ms 41.769 ms 43.676 ms
8 203.117.34.6 (203.117.34.6) 54.960 ms 32.379 ms 34.262 ms
9 203.117.34.13 (203.117.34.13) 54.193 ms 30.849 ms 30.678 ms
10 203.117.34.1 (203.117.34.1) 62.952 ms 32.438 ms 41.922 ms
11 58.27.106.253 (58.27.106.253) 65.721 ms 46.337 ms 58.054 ms
12 a118-214.80-112.deploy.akamaitechnologies.com (118.214.80.112) 54.972 ms 63.682 ms 63.171 ms

C:\tracetcp>tracetcp www.redhat.com:443 -h 3

Tracing route to 184.85.48.112 [a184-85-48-112.deploy.akamaitechnologies.com] on
port 443
Over a maximum of 30 hops.
3       32 ms   50 ms   56 ms   172.20.16.65
4       34 ms   14 ms   33 ms   172.26.16.1
5       503 ms 14 ms   68 ms   172.20.7.34
6       43 ms   170 ms 25 ms   203.117.35.9
7       28 ms   86 ms   26 ms   203.117.34.2
8       216 ms 168 ms 99 ms   203.117.34.14
9       *       *       *       Request timed out.
10      Destination Reached in 211 ms. Connection established to 184.85.48.112
Trace Complete.

Let's look at the Wireshark details for above hop 3 packet:
filtering rule:
ip.addr == 192.168.100.102 and (icmp or tcp) and not tcp.port == 22
or
ip.addr == 192.168.100.20 and (icmp or tcp) and not tcp.port == 22

Internet Protocol, Src: 192.168.100.20 (192.168.100.20), Dst: 184.85.48.112 (184.85.48.112)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 40
    Identification: 0x1e44 (7748)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 3
        [Expert Info (Note/Sequence): "Time To Live" only 3]
            [Message: "Time To Live" only 3]
            [Severity level: Note]
            [Group: Sequence]
    Protocol: TCP (0x06)
    Header checksum: 0x4c0a [correct]
        [Good: True]
        [Bad : False]
    Source: 192.168.100.20 (192.168.100.20)
    Destination: 184.85.48.112 (184.85.48.112)
Transmission Control Protocol, Src Port: 30069 (30069), Dst Port: https (443), Seq: 0, Len: 0
    Source port: 30069 (30069)
    Destination port: https (443)
    [Stream index: 1569]
    Sequence number: 0    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgement: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port https]
                [Message: Connection establish request (SYN): server port https]
                [Severity level: Chat]
                [Group: Sequence]
        .... ...0 = Fin: Not set
    Window size: 16383
    Checksum: 0x054c [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

We then received the ICMP packet as follows:

Internet Protocol, Src: 172.20.16.65 (172.20.16.65), Dst: 192.168.100.20 (192.168.100.20)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 56
    Identification: 0xfed8 (65240)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 253
    Protocol: ICMP (0x01)
    Header checksum: 0xdd19 [correct]
        [Good: True]
        [Bad : False]
    Source: 172.20.16.65 (172.20.16.65)
    Destination: 192.168.100.20 (192.168.100.20)
Internet Control Message Protocol
    Type: 11 (Time-to-live exceeded)
    Code: 0 (Time to live exceeded in transit)
    Checksum: 0x97ea [correct]
    Internet Protocol, Src: 192.168.100.20 (192.168.100.20), Dst: 184.85.48.112 (184.85.48.112)
        Version: 4
        Header length: 20 bytes
        Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
            0000 00.. = Differentiated Services Codepoint: Default (0x00)
            .... ..0. = ECN-Capable Transport (ECT): 0
            .... ...0 = ECN-CE: 0
        Total Length: 40
        Identification: 0x1e44 (7748)
        Flags: 0x02 (Don't Fragment)
            0.. = Reserved bit: Not Set
            .1. = Don't fragment: Set
            ..0 = More fragments: Not Set
        Fragment offset: 0
        Time to live: 1
            [Expert Info (Note/Sequence): "Time To Live" only 1]
                [Message: "Time To Live" only 1]
                [Severity level: Note]
                [Group: Sequence]
        Protocol: TCP (0x06)
        Header checksum: 0x4e0a [correct]
            [Good: True]
            [Bad : False]
        Source: 192.168.100.20 (192.168.100.20)
        Destination: 184.85.48.112 (184.85.48.112)
    Transmission Control Protocol, Src Port: 30069 (30069), Dst Port: https (443)
        Source port: 30069 (30069)
        Destination port: https (443)
        Sequence number: 179559217


Finally, we received the SYN/ACK packet reply from www.redhat.com, as well as ICMP time exceed packet. So the tcp probe ends.

Internet Protocol, Src: 184.85.48.112 (184.85.48.112), Dst: 192.168.100.20 (192.168.100.20)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 44
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 55
    Protocol: TCP (0x06)
    Header checksum: 0x364a [correct]
        [Good: True]
        [Bad : False]
    Source: 184.85.48.112 (184.85.48.112)
    Destination: 192.168.100.20 (192.168.100.20)
Transmission Control Protocol, Src Port: https (443), Dst Port: 28408 (28408), Seq: 0, Ack: 1, Len: 0
    Source port: https (443)
    Destination port: 28408 (28408)
    [Stream index: 1595]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 24 bytes
    Flags: 0x12 (SYN, ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgement: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port https]
                [Message: Connection establish acknowledge (SYN+ACK): server port https]
                [Severity level: Chat]
                [Group: Sequence]
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0xcb85 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (4 bytes)
        Maximum segment size: 1460 bytes

example 2: detect transparent proxy in between
[root@linuxtest ~]# tcptraceroute www.redhat.com -f 2
traceroute to www.redhat.com (118.214.80.112), 30 hops max, 40 byte packets
3 172.20.16.65 (172.20.16.65) 16.943 ms 23.115 ms 31.587 ms
4 172.26.16.1 (172.26.16.1) 31.742 ms 31.969 ms 32.348 ms
5 172.20.7.26 (172.20.7.26) 43.759 ms 43.591 ms 43.662 ms
6 172.20.7.82 (172.20.7.82) 37.583 ms 38.229 ms 37.181 ms
7 a118-214.80-112.deploy.akamaitechnologies.com (118.214.80.112) 50.047 ms 49.993 ms 49.987 ms

C:\tracetcp>tracetcp www.redhat.com -h 3

Tracing route to 184.85.48.112 [a184-85-48-112.deploy.akamaitechnologies.com] on
port 80
Over a maximum of 30 hops.
3       39 ms   36 ms   27 ms   172.20.16.65
4       51 ms   34 ms   15 ms   172.26.16.1
5       50 ms   46 ms   68 ms   172.20.7.34
6       Destination Reached in 59 ms. Connection established to 184.85.48.112
Trace Complete.

Compare above port 80 output with port 443, we know there's transparent proxy in-between, it stops before reaching redhat.com

5. examples - use tracetcp and nc to detect open ports
[root@linuxtest ~]# nc -zv www.redhat.com 443
Connection to www.redhat.com 443 port [tcp/https] succeeded!

C:\tracetcp>tracetcp www.redhat.com -s 442 443
[184.85.48.112:442] 128        *       Request timed out.
[184.85.48.112:443] 128        Dest. in 210 ms. Port OPEN on 184.85.48.112

6. examples - use tracetcp or tcptraceroute to detect blocked ports
C:\tracetcp>tracetcp www.redhat.com:139

Tracing route to 184.85.48.112 [a184-85-48-112.deploy.akamaitechnologies.com] on
port 139
Over a maximum of 30 hops.
1       3 ms    2 ms    2 ms    192.168.1.1
2       *       *       *       Request timed out.
3       *       *       *       Request timed out.
4       *       *       *       Request timed out.
5       *       *
Terminate Event Occurred.

Note: above output shows after home router gateway, the ISP blocks port 139 at hop 2.
If you setup a server at home, use cable modem connection, you can test it from Internet if the port you are using is blocked by ISP or not.

7. Others
a. debug feature on tcptraceroute.
You can download the latest version, tcptraceroute 1.5beta7, the -d option is for debug output, very useful.

b. icmp code and type which should not be blocked
ICMP type 3, Destination Unreachable, especially code 4, "fragmentation needed but don't fragment bit set" (necessary for path MTU discovery)
ICMP type 11, time exceeded (so you can use traceroute from inside the network and get replies).

c. http://livenudefrogs.com/~anubis/icmp/#type3
icmp type 8 for echo request,
icmp type 0 for echo reply

icmp type 3 and 11 are important, should not be blocked on firewall. tcptraceroute will use them to get return packet.

Linux SysAdmin and Oracle/MySQL DBA

How to troubleshoot Internet connectivity issues

How to troubleshoot packet loss and latency for Internet VPN

TCP Traceroute Analysis

Total Pageviews

Bookmark

Search This Blog

Followers

keywords

All Posts

About me