TCP Traceroute Analysis

Jephe Wu - http://linuxtechres.blogspot.com

Objective: Understanding how tcp traceroute works and use it to make sysadmin life easier
Environment: Windows Vista, CentOS 5

Concepts:
1. What's TCP traceroute
According to http://michael.toren.net/code/tcptraceroute/. tcptraceroute is a traceroute implementation using TCP packets.

The more traditional traceroute(8) sends out either UDP or ICMP ECHO packets with a TTL of one, and increments the TTL until the destination has been reached. By printing the gateways that generate ICMP time exceeded messages along the way, it is able to determine the path packets are taking to reach the destination.

The problem is that with the widespread use of firewalls on the modern Internet, many of the packets that traceroute(8) sends out end up being filtered, making it impossible to completely trace the path to the destination. However, in many cases, these firewalls will permit inbound TCP packets to specific ports that hosts sitting behind the firewall are listening for connections on. By sending out TCP SYN packets instead of UDP or ICMP ECHO packets, tcptraceroute is able to bypass the most common firewall filters.

2. How to use it under Linux and Windows
You can use tcptraceroute(http://michael.toren.net/code/tcptraceroute/) under Linux and tracetcp(http://tracetcp.sourceforge.net/) under Windows.
Actually, under CentOS 5.5, traceroute command has many options which you can use to do tcp(-T) traceroute by default, you can also use it do tranditioanl udp(-U) traceroute or use icmp(-I) ping packets to do it like tracert on Windows.

3. How tcptraceroute or tracetcp works
it uses tcp syn package and set ttl as 1 as initial packet to send to network. Each hop will decrease ttl by 1, so each hop will generate a time exceed icmp packet back to the sender, those icmp packet includes the original packet information. For the next hop, the sender will use TTL 2 until the destination which will also send back TCP syn/ack reply to the sender.

4. examples - use traceroute to know network path
example 1:  tcptraceroute to www.redhat.com
[root@linuxtest ~]# tcptraceroute  www.redhat.com -p 443 -f 2
traceroute to www.redhat.com (118.214.80.112), 30 hops max, 40 byte packets
 3  172.20.16.65 (172.20.16.65)  27.082 ms  27.637 ms  34.125 ms
 4  172.26.16.1 (172.26.16.1)  38.854 ms  38.752 ms  38.627 ms
 5  172.20.7.26 (172.20.7.26)  38.446 ms  38.280 ms  38.156 ms
 6  172.20.7.82 (172.20.7.82)  38.031 ms  37.900 ms  37.753 ms
 7  203.117.34.101 (203.117.34.101)  37.872 ms  41.769 ms  43.676 ms
 8  203.117.34.6 (203.117.34.6)  54.960 ms  32.379 ms  34.262 ms
 9  203.117.34.13 (203.117.34.13)  54.193 ms  30.849 ms  30.678 ms
10  203.117.34.1 (203.117.34.1)  62.952 ms  32.438 ms  41.922 ms
11  58.27.106.253 (58.27.106.253)  65.721 ms  46.337 ms  58.054 ms
12  a118-214.80-112.deploy.akamaitechnologies.com (118.214.80.112)  54.972 ms  63.682 ms  63.171 ms

C:\tracetcp>tracetcp www.redhat.com:443 -h 3

Tracing route to 184.85.48.112 [a184-85-48-112.deploy.akamaitechnologies.com] on
 port 443
Over a maximum of 30 hops.
3       32 ms   50 ms   56 ms   172.20.16.65
4       34 ms   14 ms   33 ms   172.26.16.1
5       503 ms  14 ms   68 ms   172.20.7.34
6       43 ms   170 ms  25 ms   203.117.35.9
7       28 ms   86 ms   26 ms   203.117.34.2
8       216 ms  168 ms  99 ms   203.117.34.14
9       *       *       *       Request timed out.
10      Destination Reached in 211 ms. Connection established to 184.85.48.112
Trace Complete.

Let's look at the Wireshark details for above hop 3 packet:
filtering rule:
ip.addr == 192.168.100.102 and (icmp or tcp) and not tcp.port == 22
or
ip.addr == 192.168.100.20 and (icmp or tcp) and not tcp.port == 22

Internet Protocol, Src: 192.168.100.20 (192.168.100.20), Dst: 184.85.48.112 (184.85.48.112)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 40
    Identification: 0x1e44 (7748)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 3
        [Expert Info (Note/Sequence): "Time To Live" only 3]
            [Message: "Time To Live" only 3]
            [Severity level: Note]
            [Group: Sequence]
    Protocol: TCP (0x06)
    Header checksum: 0x4c0a [correct]
        [Good: True]
        [Bad : False]
    Source: 192.168.100.20 (192.168.100.20)
    Destination: 184.85.48.112 (184.85.48.112)
Transmission Control Protocol, Src Port: 30069 (30069), Dst Port: https (443), Seq: 0, Len: 0
    Source port: 30069 (30069)
    Destination port: https (443)
    [Stream index: 1569]
    Sequence number: 0    (relative sequence number)
    Header length: 20 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgement: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port https]
                [Message: Connection establish request (SYN): server port https]
                [Severity level: Chat]
                [Group: Sequence]
        .... ...0 = Fin: Not set
    Window size: 16383
    Checksum: 0x054c [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]

We then received the ICMP packet as follows:

Internet Protocol, Src: 172.20.16.65 (172.20.16.65), Dst: 192.168.100.20 (192.168.100.20)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 56
    Identification: 0xfed8 (65240)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 253
    Protocol: ICMP (0x01)
    Header checksum: 0xdd19 [correct]
        [Good: True]
        [Bad : False]
    Source: 172.20.16.65 (172.20.16.65)
    Destination: 192.168.100.20 (192.168.100.20)
Internet Control Message Protocol
    Type: 11 (Time-to-live exceeded)
    Code: 0 (Time to live exceeded in transit)
    Checksum: 0x97ea [correct]
    Internet Protocol, Src: 192.168.100.20 (192.168.100.20), Dst: 184.85.48.112 (184.85.48.112)
        Version: 4
        Header length: 20 bytes
        Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
            0000 00.. = Differentiated Services Codepoint: Default (0x00)
            .... ..0. = ECN-Capable Transport (ECT): 0
            .... ...0 = ECN-CE: 0
        Total Length: 40
        Identification: 0x1e44 (7748)
        Flags: 0x02 (Don't Fragment)
            0.. = Reserved bit: Not Set
            .1. = Don't fragment: Set
            ..0 = More fragments: Not Set
        Fragment offset: 0
        Time to live: 1
            [Expert Info (Note/Sequence): "Time To Live" only 1]
                [Message: "Time To Live" only 1]
                [Severity level: Note]
                [Group: Sequence]
        Protocol: TCP (0x06)
        Header checksum: 0x4e0a [correct]
            [Good: True]
            [Bad : False]
        Source: 192.168.100.20 (192.168.100.20)
        Destination: 184.85.48.112 (184.85.48.112)
    Transmission Control Protocol, Src Port: 30069 (30069), Dst Port: https (443)
        Source port: 30069 (30069)
        Destination port: https (443)
        Sequence number: 179559217

       
Finally, we received the SYN/ACK packet reply from www.redhat.com, as well as ICMP time exceed packet. So the tcp probe ends.

Internet Protocol, Src: 184.85.48.112 (184.85.48.112), Dst: 192.168.100.20 (192.168.100.20)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 44
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 55
    Protocol: TCP (0x06)
    Header checksum: 0x364a [correct]
        [Good: True]
        [Bad : False]
    Source: 184.85.48.112 (184.85.48.112)
    Destination: 192.168.100.20 (192.168.100.20)
Transmission Control Protocol, Src Port: https (443), Dst Port: 28408 (28408), Seq: 0, Ack: 1, Len: 0
    Source port: https (443)
    Destination port: 28408 (28408)
    [Stream index: 1595]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 24 bytes
    Flags: 0x12 (SYN, ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgement: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port https]
                [Message: Connection establish acknowledge (SYN+ACK): server port https]
                [Severity level: Chat]
                [Group: Sequence]
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0xcb85 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (4 bytes)
        Maximum segment size: 1460 bytes

example 2: detect transparent proxy in between
[root@linuxtest ~]# tcptraceroute  www.redhat.com -f 2
traceroute to www.redhat.com (118.214.80.112), 30 hops max, 40 byte packets
 3  172.20.16.65 (172.20.16.65)  16.943 ms  23.115 ms  31.587 ms
 4  172.26.16.1 (172.26.16.1)  31.742 ms  31.969 ms  32.348 ms
 5  172.20.7.26 (172.20.7.26)  43.759 ms  43.591 ms  43.662 ms
 6  172.20.7.82 (172.20.7.82)  37.583 ms  38.229 ms  37.181 ms
 7  a118-214.80-112.deploy.akamaitechnologies.com (118.214.80.112)  50.047 ms  49.993 ms  49.987 ms

 C:\tracetcp>tracetcp www.redhat.com -h 3

Tracing route to 184.85.48.112 [a184-85-48-112.deploy.akamaitechnologies.com] on
 port 80
Over a maximum of 30 hops.
3       39 ms   36 ms   27 ms   172.20.16.65
4       51 ms   34 ms   15 ms   172.26.16.1
5       50 ms   46 ms   68 ms   172.20.7.34
6       Destination Reached in 59 ms. Connection established to 184.85.48.112
Trace Complete.

Compare above port 80 output with port 443, we know there's transparent proxy in-between, it stops before reaching redhat.com

5. examples - use tracetcp and nc to detect open ports
[root@linuxtest ~]# nc -zv www.redhat.com 443
Connection to www.redhat.com 443 port [tcp/https] succeeded!

C:\tracetcp>tracetcp www.redhat.com -s 442 443
[184.85.48.112:442]  128        *       Request timed out.
[184.85.48.112:443]  128        Dest. in 210 ms. Port OPEN on 184.85.48.112

6. examples - use tracetcp or tcptraceroute to detect blocked ports
C:\tracetcp>tracetcp www.redhat.com:139

Tracing route to 184.85.48.112 [a184-85-48-112.deploy.akamaitechnologies.com] on
 port 139
Over a maximum of 30 hops.
1       3 ms    2 ms    2 ms    192.168.1.1
2       *       *       *       Request timed out.
3       *       *       *       Request timed out.
4       *       *       *       Request timed out.
5       *       *
Terminate Event Occurred.

Note: above output shows after home router gateway, the ISP blocks port 139 at hop 2.
If you setup a server at home, use cable modem connection, you can test it from Internet if the port you are using is blocked by ISP or not.

7. Others
a. debug feature on tcptraceroute.
You can download the latest version, tcptraceroute 1.5beta7, the -d option is for debug output, very useful.


b. icmp code and type which should not be blocked
ICMP type 3, Destination Unreachable, especially code 4, "fragmentation needed but don't fragment bit set" (necessary for path MTU discovery)
ICMP type 11, time exceeded (so you can use traceroute from inside the network and get replies).


c. http://livenudefrogs.com/~anubis/icmp/#type3
icmp type 8 for echo request,
icmp type 0 for echo reply

icmp type 3 and 11 are important, should not be blocked on firewall. tcptraceroute will use them to get return packet.