Hello. My name is Gabriel; I’m a network analyst here at Made4it, and today I’m going to tell you about an interesting situation involving CDN and traffic engineering that we came across here on the consulting team a few days ago.
We received a call from a customer with a case where the traffic of a local CDN had a drastic decrease after a traffic engineering change. After extensive validations by the customer they were unable to find the root cause of this behavior and so they called our team.
We started there on 11/04 with a strange decrease in Inbound traffic on the interface with the local CDN. It was very succinct and represented 1Gbps less traffic on the interface.
With that information in hand, the first thing I did was go to the Made4Flow to help me understand what happened, using, of course, my favorite chart, the“Interface by ASN.”
I made sure to select a time range covering exactly the event from the 3rd to the 4th, specifically to understand which ASNs were no longer being “served” by the local cache.
With that in hand, I noticed 2 things right away:
– The “Laranja” ASN is no longer being “served” by this CDN; this is evident from the clear drop in traffic shown in the graph.
– The “Blue” ASN(s) (a set of specific ASNs custom configured in our software) also showed a significant decrease in the graph. The “Blue” ASN is who hosts the cache.
The fact that the “Orange” ASN is also a Made4it customer allowed us to conduct a more efficient test within its network. Since the “Blue” ASN and the “Orange” ASN are partners, they gave us complete freedom to troubleshoot and validate whatever was necessary on both sides.
After some checks, we used the content provider’s own tool to validate which “node/cache” was “serving” this traffic that stopped coming from the local CDN of the “Azul” ASN.
That traffic is no longer coming from the “Azul” ASN’s CDN, but it has to be coming from somewhere, don’t you agree?
In the image above, 2 things caught my attention:
– A node in Guarulhos, part of this content provider’s network, is now delivering the traffic that was previously supposed to come through the local CDN of the “Azul” ASN
– x.x.x.0/25?
At the time, a very important piece of information about ads for this content provider’s CDNs came to mind. They obey a rule that says:
– Direct ads (ASN hosting cache) node accepts up to /27 for direct influence on traffic engineering and content delivery eligibility.
– Indirect advertisements (ASNs adjacent to the ASN hosting the cache) the node only accepts up to /24 for direct influence on traffic engineering and content delivery eligibility.
Alright, for the ASN “Azul” traffic I already had the possible cause of the problem. When trying to induce this local node to deliver more traffic, they ended up announcing /25 blocks to the cache node, implying that the content provider network would “serve” this traffic via its nodes in Guarulhos.
However, we still have a decrease in traffic on the “Azul” ASN, what happened?
The “Blue” ASN in this case was actually three ASNs, a configuration that was custom-built within the Made4Flow software. Interestingly, two of these ASNs were “injecting” /25 prefixes into the CDN node, where the behavior observed was exactly the same as that reported above for the “Orange” ASN. With all this information in hand, we set out to get to work and adjust the announcements in accordance with what the content provider requires (and has documented) for its CDN. After making the necessary adjustments, here’s the result:
Conclusion:
We know that for a CDN node to work there are several factors and that it involves protocols working in an orchestrated way, such as BGP, DNS, TCP, among others… CDN out of acceptable range.
By announcing the /25 blocks of ASNs adjacent to the ASN that hosts the node/cache, the “player” started delivering traffic to that ASN/Prefix via its CDN nodes in Guarulhos, resulting in an increase in indirect traffic on our “Azul” ASN client and loss of performance and efficiency of the local node.
Users of this network were directly affected because their content began to be delivered via infrastructure located more than 1,000 km away, increasing latency, load times, and the overall user experience.
After adjustments in traffic engineering respecting the content provider’s documentation and good BGP practices, the node started to behave again as expected.
When designing traffic flow for CDNs, always follow the documentation provided by the specific CDN and use its tools to determine which “node” is “serving” the traffic. CDN providers typically also offer a portal for monitoring cache performance, complete with critical metrics and data; use these tools to your advantage.
– Gabriel Henrique, a Network Analyst at Made4it, has been working with technology and ISPs for over 10 years.
