Hello. My name is Gabriel; I’m a network analyst here at Made4it, and today I’m going to talk about an interesting situation involving Google’s GGC CDN and traffic engineering that our consulting team handled a while back.

We received a call from a customer with a case where traffic from a local CDN GGC Google had a drastic decrease after a traffic engineering change. After extensive validations by the customer they were unable to find the root cause of this behavior and so they called our team.
We started there on 11/04 with a strange decrease in Inbound traffic on the interface with the local CDN. It was very succinct and represented 1Gbps less traffic on the interface.

With that information in hand, the first thing I did was go to the Made4Flow to help me understand what happened, using, of course, my favorite graph, the“Interface by ASN”
. I made sure to select a time range covering exactly the event from the 3rd to the 4th, specifically to understand which ASNs were no longer being “served” by the local cache.

With that in hand, I noticed 2 things right away:
– The “Orange” ASN is no longer being “served” by this CDN; this is evident from the clear drop in traffic shown in the graph…
– The “Blue” ASN(s) (a set of specific ASNs configured customarily in our software) also showed a significant decrease in the graph. The “Blue” ASN is the one hosting the cache.
The fact that ASN “Orange” is also a Made4it customer allowed us to conduct a more efficient test within its network. Since ASN “Blue” and ASN “Orange” are partners, they gave us complete freedom to troubleshoot and validate whatever was necessary on both sides.
After some checks, we used the content provider’s own tool to determine which“node/cache”was “serving” the traffic that was no longer coming through ASN “Blue’s” local CDN.
This traffic stopped coming from the “Azul” ASN’s CDN, but it has to be coming from somewhere, don’t you agree?

In the image above, two things caught my attention:
– A node in Guarulhos, part of this content provider’s network, is now delivering the traffic that was previously supposed to come from the local CDN of the “Azul” ASN
– x.x.x.0/25?

At the time, a very important piece of information about ads for this content provider’s CDNs came to mind. They obey a rule that says:
– Direct announcements (ASN hosting the cache): The node accepts up to /27 for direct influence on traffic engineering and eligibility for content delivery.
– Indirect announcements (ASNs adjacent to the ASN hosting the cache): the node accepts only up to /24 for direct influence on traffic engineering and eligibility for content delivery.
Great, for traffic from ASN “Azul,” I already had the possible cause of the problem. When they tried to get this local node to deliver more traffic, they ended up announcing /25 blocks to the cache node, causing the content provider’s network to “serve” this traffic via its nodes in Guarulhos.
However, we still have a decrease in traffic on the “Azul” ASN, what happened?
The “Blue” ASN in this case was actually three ASNs, a configuration that was custom-built within the Made4Flow software. Interestingly, two of these ASNs were “injecting” /25 prefixes into the CDN node, where the behavior observed was exactly the same as that reported above for the “Orange” ASN.

With all this information in hand, we set out to get to work and adjust the ads in accordance with the content provider’s requirements (which are documented) for its CDN. After making the necessary adjustments, here’s the result:

Conclusion:
We know that for a CDN node to function, there are several factors involved, and that it requires protocols—such as BGP, DNS, and TCP, among others—to work in an orchestrated manner… however, in this case, our problem was an unexpected behavior when an announcement was made to the CDN node outside the acceptable range.
When we announce the /25 blocks of ASNs adjacent to the ASN hosting the node/cache, the “player” began delivering traffic to that ASN/prefix via its CDN nodes in Guarulhos, resulting in an increase in indirect traffic on our client ASN “Azul” and a loss of performance and efficiency for the local node.

Users of this network were directly affected because their content began to be delivered via infrastructure located more than 1,000 km away, increasing latency, load times, and the overall user experience.
After adjustments in traffic engineering respecting the content provider’s documentation and good BGP practices, the node started to behave again as expected.
When designing traffic flow for CDNs, always follow the documentation provided by the specific CDN and use its tools to determine which “node” is “serving” the traffic. CDN providers typically also offer a portal for monitoring cache performance, complete with critical metrics and data; use these tools to your advantage.

Did you find this interesting and want to learn a little more about how the software we use in this process works, or would you like to talk to an expert to resolve issues like this on your network? Contact our team at
— Gabriel Henrique, a Network Analyst at Made4it, has been working with technology and ISPs for over 10 years.