Archive

September 26, 2011 at 4:30 PM

Every time I talk to network engineers in any kind of support role, invariably the discussion turns to traceroute and the various misconceptions users at large have about its output. The engineers cringe when they see it coming, because the sender is, as often as not, convinced that he has a smoking gun, but in reality, just doesn't understand how the tool operates and is misinterpreting its output.

I'll confess that I was, for a very long time, guilty of this myself. It has only been in the last dozen years or so that I've changed my evil ways. In this post, I'm summing up what I've learned from other people, by reading and listening attentively, and by my own stunning victories and major embarrassments, in trying to diagnose and report routing and performance problems with this tool.

Rule #1: Understand how it works.

An alarming number of people I run into think that traceroute is some kind of one-packet network boomerang that magically traverses a path, collecting timing data and returning to its source. Cue the game-show buzzer. A traceroute is actually a series of ever-longer probes, each reaching progressively further out to the destination than the one before. Traceroute does its job iteratively, by sending packets out to the destination with specific, short "TTLs," or Time To Live values. All IP packets have a TTL, and every time a packet passes through a routing device (I'll just say "router" from now on, but it could be a firewall appliance, etc.), the router decrements the TTL as it forwards it. If a router decrements the TTL on a packet to zero, then that router drops the packet and sends an ICMP TTL Exceeded response to the originating router. The TTL's original intent was to prevent a packet from circulating in a network endlessly, but its (ab)use is key to how traceroute works. While most "normal" packets start at their origin with a high TTL (64 and 128 seem to be common defaults), traceroute starts with TTL=1 and works its way higher.

Let's say that we're trying to trace to a host that's some number of hops away; we don't yet know how many. The first thing traceroute does is send a packet on the network to the target host with a TTL of just one (TTL=1). That packet makes it out as far as the immediately adjacent router, which decrements the TTL to zero, throws up its hands in disgust, drops the packet, and builds and sends back an ICMP TTL Exceeded response. Traceroute sees that ICMP response, which bears the IP address of the reporting router, so from that traceroute has just discovered the first hop on the path. Traceroute then turns around and sends out another packet to the target IP address, this time with a TTL of two (TTL=2). It passes through the first router, which decrements the TTL to one in this case and hustles it along to another router. That second router decrements the TTL, which becomes zero, and it now throws its hands up in disgust and sends back an ICMP TTL Exceeded response. Second hop discovered. This continues with ever-increasing TTLs, until the packet reaches the actual target host address that traceroute is focused on. All along the way, traceroute is telling you what the IP address was of each hop, and how long the round trip took.

So, in a theoretical trace to a target that is 8 hops away, traceroute will actually do 8 separate probes, and that means over 70 hops total out and back are taken to complete the entire process. That's a lot of machination, and most implementations don't do it just once, they do it three times, and show you three sets of results for each hop.

Rule #2: The path traceroute shows you is the outbound path, not the return path, but the timing is the full round-tip time, not just the outbound time.

Packets in a network take whatever path each hop determines is the best next hop to reach a destination. Just because you come into a router on an interface, doesn't mean a return packet will go back on the same interface. This can radically affect timing, because the return path from hop 4 may be via one carrier, but when we're out at hop 8 it may determine that the best return path is a different carrier. The return path can be different from the outbound path. Think about how that can affect timing! For this reason, it's almost always useful to traceroute in both directions, at least, and very often useful to traceroute from as many places as possible.

Rule #3: It usually takes longer to get a reply packet from a router than it takes a router to pass a packet through to someplace else.

Most routers use fast switching/routing mechanisms that rely on a lot of cool tricks, often in hardware, to move packets through as quickly as possible and with as little handling as possible, particularly by the router's CPU. Because of this, the CPUs on routers are often designed as more supervisorial processors than they are actual high-rate routing engines, and are thus relatively slow processors. Any normal packet can probably move through the router very quickly, sometimes without the CPU being involved, as long as its TTL doesn't zero out. But when a packet's TTL hits zero, the CPU has to be called upon to generate the ICMP TTL Exceeded reply and send it. The time it takes to do that is often far greater than the time it takes to shuttle an unexpired packet across switching fabric between interfaces. Some security best-practices also recommend setting low priority to ICMP packet transit and generation, which helps to resist certain DoS attacks. So, when you see a traceroute that has a series of low-value timings for hops, followed a hop that has much higher timing, don't get excited because you've found "the slow router" in the middle. You may have just found a fast router with a modest CPU or strict QoS policies. This leads us to...

Rule #4: Real delays are cumulative.

Let's take the result sequence of 4ms, 6ms, 11ms, 4ms, 3ms, 25ms, 5ms, 7ms as an example. The big sin is that people focus on the 25ms at hop 6. Does that mean that that router is congested and causing a delay? Is it slowing things down? NO! Look at the next hop, 5ms, and the last hop, at 7ms. Any actual delay would be cumulative by definition, so if hop 6 was introducing a big delay, the round trip time for hops 7 and 8 would also be affected (high). But they're not; they're low. So all you've learned about hop 6 is that it probably just doesn't handle ICMP very quickly, and because hops 7 and 8 are pretty close in timing to hop 5, hop 6 actually passes packets through pretty fast. Conversely, if our example timing sequence was 4, 4, 11, 4, 3, 25, 27, 28, then we see an accumulating downstream delay and we might say that hop 6 is suspect (but see Rule #8).

Rule #5: If you want to know how fast you get to a destination from a source, just use ping.

Traceroute is a great tool for discovering the path between a source and destination, but it's not always a great timing tool. Because of all of the above rules, you should not fall into the trap of using traceroute alone for timing, and assuming that just because you see big numbers anywhere between the source and destination, you're seeing a problem. Use ping instead; it's a more honest representation of what "real" packets are doing.

Rule #6: Don't send one line of a traceroute on a trouble ticket.

If you understand rules 1-4, you understand why sending one line of a traceroute (or anything less than the full output, really) on a trouble ticket isn't useful or conclusive (if it's not clear, go back and re-read them). If you're sending a traceroute off as part of a trouble ticket, you're probably chasing a problem that you'd like to see fixed, so be a good citizen and make a full report. Even if you're wrong, you may get someone helpful who will tell you what you missed or give you some other things to try.

Rule #7: Not all traceroutes work alike.

Traceroutes in *nix default to UDP packets for probes, while Windows uses ICMP. Some systems default to, or are capable of, using TCP. Know what you are dealing with, because firewalls and routers respond to each differently. A lot of bad things can happen to a good packet just because it's misunderstood by some well-intending firewall. Know how your traceroute indicates to you that it never got a reply, or it got a deny or destination unreachable response. And don't be surprised to see very different timing results from side-by-side traceroutes out of different operating systems.

Rule #8: Understand physics, the speed of light, and engineering and business realities.

Reaching a router that is 50 miles away is a very different problem from traversing the continent and hitting New York from Emeryville. Although your traceroute may show that as less than a dozen hops, with huge geographic leaps (e.g. one router in San Jose, the next in Chicago), the reality is that there's a lot of fiber (or copper, still) and a lot of other pieces of equipment in between, and each introduces its own delays and adds to latency. Changes in network architecture and interfaces between peers (carriers) can cause packets to be queued. We have charts around the data center that show average RTTs for various distances, measured on actual equipment in real networks, vs theoretical/lightspeed/ideal. The bad news is that there's a huge difference, but the good news is that the average is a fairly consistent multiple of light speed, so "good" RTTs for a given distance are somewhat predictable, regardless of whose network you're on. Still, if you're in San Jose trying to load a web page from a server in New York, even though it may be 1GBps and 10GBps links the entire way between, that page is going to come up noticeably slower than the same page loaded from a nearby server.

Wrapping It Up

Traceroute is a great tool, but you have to understand what it's doing, and know more than a little about packet handling in routers to really make consistently effective use of it. In many cases, the casual user can identify a legitimate problem, but there are so many possible variables in the traffic-handling equation of the Internet, that you need to resist the trap of default thinking ("this looks wrong, so that must be it" and "I don't understand why it looks like this, so that must be it"). The output of traceroute can be inconsistent and potentially misleading because every device on the network effectively makes its own rules within the broad boundaries of what's acceptable. Doing your homework, preparing and studying your results carefully, and communicating them completely and clearly to your points of contact, will give you much greater success in reporting and resolving problems.