Issues with local CDN traffic (Google GGC) and BGP ads

Hello. My name is Gabriel; I’m a network analyst here at Made4it, and today I’m going to talk about an interesting situation involving Google’s GGC CDN and traffic engineering that our consulting team handled a while back. We received a call from a customer with a case where traffic from a local CDN GGC Google had a drastic decrease after a traffic engineering change. After extensive validations by the customer they were unable to find the root cause of this behavior and so they called our team. We started there on 11/04 with a strange decrease in Inbound traffic on the interface with the local CDN. It was very succinct and represented 1Gbps less traffic on the interface. With that information in hand, the first thing I did was go to the Made4Flow to help me understand what happened, using, of course, my favorite graph, the“Interface by ASN”. I made sure to select a time range covering exactly the event from the 3rd to the 4th, specifically to understand which ASNs were no longer being “served” by the local cache. With that in hand, I noticed 2 things right away:– The “Orange” ASN is no longer being “served” by this CDN; this is evident from the clear drop in traffic shown in the graph…– The “Blue” ASN(s) (a set of specific ASNs configured customarily in our software) also showed a significant decrease in the graph. The “Blue” ASN is the one hosting the cache. The fact that ASN “Orange” is also a Made4it customer allowed us to conduct a more efficient test within its network. Since ASN “Blue” and ASN “Orange” are partners, they gave us complete freedom to troubleshoot and validate whatever was necessary on both sides. After some checks, we used the content provider’s own tool to determine which“node/cache”was “serving” the traffic that was no longer coming through ASN “Blue’s” local CDN. This traffic stopped coming from the “Azul” ASN’s CDN, but it has to be coming from somewhere, don’t you agree? In the image above, two things caught my attention:– A node in Guarulhos, part of this content provider’s network, is now delivering the traffic that was previously supposed to come from the local CDN of the “Azul” ASN– x.x.x.0/25? At the time, a very important piece of information about ads for this content provider’s CDNs came to mind. They obey a rule that says: – Direct announcements (ASN hosting the cache): The node accepts up to /27 for direct influence on traffic engineering and eligibility for content delivery. – Indirect announcements (ASNs adjacent to the ASN hosting the cache): the node accepts only up to /24 for direct influence on traffic engineering and eligibility for content delivery.Great, for traffic from ASN “Azul,” I already had the possible cause of the problem. When they tried to get this local node to deliver more traffic, they ended up announcing /25 blocks to the cache node, causing the content provider’s network to “serve” this traffic via its nodes in Guarulhos. However, we still have a decrease in traffic on the “Azul” ASN, what happened? The “Blue” ASN in this case was actually three ASNs, a configuration that was custom-built within the Made4Flow software. Interestingly, two of these ASNs were “injecting” /25 prefixes into the CDN node, where the behavior observed was exactly the same as that reported above for the “Orange” ASN. With all this information in hand, we set out to get to work and adjust the ads in accordance with the content provider’s requirements (which are documented) for its CDN. After making the necessary adjustments, here’s the result: Conclusion:We know that for a CDN node to function, there are several factors involved, and that it requires protocols—such as BGP, DNS, and TCP, among others—to work in an orchestrated manner… however, in this case, our problem was an unexpected behavior when an announcement was made to the CDN node outside the acceptable range.When we announce the /25 blocks of ASNs adjacent to the ASN hosting the node/cache, the “player” began delivering traffic to that ASN/prefix via its CDN nodes in Guarulhos, resulting in an increase in indirect traffic on our client ASN “Azul” and a loss of performance and efficiency for the local node. Users of this network were directly affected because their content began to be delivered via infrastructure located more than 1,000 km away, increasing latency, load times, and the overall user experience. After adjustments in traffic engineering respecting the content provider’s documentation and good BGP practices, the node started to behave again as expected. When designing traffic flow for CDNs, always follow the documentation provided by the specific CDN and use its tools to determine which “node” is “serving” the traffic. CDN providers typically also offer a portal for monitoring cache performance, complete with critical metrics and data; use these tools to your advantage. Did you find this interesting and want to learn a little more about how the software we use in this process works, or would you like to talk to an expert to resolve issues like this on your network? Contact our team at — Gabriel Henrique, a Network Analyst at Made4it, has been working with technology and ISPs for over 10 years.

How to Configure Blackhole on Cisco IOS-XE

Now that you know what a BGP Blackhole is (if you still don’t know, check out our article on RTBH – Blackhole). Now it’s time to configure it and be able to protect yourself from DDoS attacks. To summarize the Blackhole, it is a technique of sending a route to the “black hole” or simply making the router discard packets directed to that IP. With Blackhole you can also announce these attacked IPs to your suppliers/upstreams and thus stop the attacks. Now that I know what it is, now comes the question, how to blackhole my router? In today’s article we will show you how to configure Blackhole on Cisco routers To do the Blackhole manually we have some steps that are: Identify the attacked IP; Create the route to blackhole; Advertise this blackhole route via BGP to your carriers/upstreams. You can automate all of this with Made4Flow, already closing a direct session and not having to do manual work. Let’s go to the settings then 1 – Identify the attacked IP You can do this by analyzing NetFlow data—such as in Made4Flow—using the graphs and identifying, through the Raw Data Report, which IP address has the highest traffic and is likely the victim of the attack. Within Made4Flow, access, for example, the Interface Graph by Application, then, by clicking on the most used port, you can identify which IP is being attacked, or through Made4Flow, simply by accessing the Anti-DDoS module -> Active Anomalies. The attacked IP was: 200.189.56.55 (Example) 2) Create a route to Blackhole or Null0 After identifying the attacked IP via Made4Flow, now it’s time to create the route on your Cisco Router to effectively throw the IP to Blackhole or Null0. Let’s assume that the attacked IP is 200.200.200.1, let’s create the route as follows Commands applied: enableconfigure terminalip route 200.200.200.1 255.255.255.255 Null0 After applying the route pointing to Null0 this IP will STOP WORKING! You can check the route using the show command: If the route is showing as Null0 then you are already sending it to Blackhole. 3 – Advertise the IP in blackhole via BGP to your operators/upstreams After identifying and blackhole the route, you need to advertise via BGP to your carriers/upstreams. Note: Before setting up, it is always recommended to talk to your Operator/Upstream to find out which Blackhole BGP community is. The BGP session with your carrier needs to be established. For this we have a few steps: Configure your Carrier/Upstream Blackhole Community To configure the blackhole community to be used later, we need to run the following command: Commands: ip prefix-list BLACKHOLE permit 200.200.200.1/32route-map BLACKHOLE permitmatch ip address prefix-list BLACKHOLEset community 666:666 set community 666:666 In case it is necessary to add more communities, apply the same command changing the community name and number. Tip: Talk to your operator to find out which BGP blackhole community they use. Announce the attacked IP with the BlackHole community for Upstream To carry out the advertisement of the attacked IP with the blackhole community, it is necessary to carry out the following steps; Enter the Cisco router’s BGP configuration After that, we announce the attacked IP to our Upstream, using the command below; Commands: router bgp 65000neighbor 192.168.100.1 as 64700neighbor 192.168.100.1 route-map BLACKHOLE out Automating with Made4Flow With Made4Flow, it is possible to automate the blackhole announcement process of attacked IPs. For that we need: Configure the BGP session between the Edge Router and Made4Flow; To configure the BGP session between the Router and Made4Flow, you need to create a route-map and then the BGP session. To configure the route-map: Comando: route-map MADE4FLOW-IN permit 1000 set ip nex-hop 192.168.66.66 In this case it is necessary to add Next-hop manually on the router. Within Made4Flow, you can already advertise with the BGP community and the correct Next-hop if you prefer. Configure Made4Flow to send via Actions Within the Anti-DDoS Module, you can access the menu: Actions and Responses and configure the response to send the Blackhole with the correct BGP community: Configure the Router to Send to Carriers To configure to send to operators/upstreams you need to configure so that the BGP community is identified in the outbound Route-map match. For this we need to configure an ip community-filter The next step is to configure the Route-map of your operator/upstream, as in sending the blackhole, but now matching the community in the match, as in our configuration: Check if you receive from Made4Flow Command: show bgp ipv4 unicast neighbors 192.168.120.2 routes And if you are sending to the operator: Command: show bgp ipv4 unicast neighbors 192.168.1.1 advertised-routes Having made these settings, the automation of Made4Flow is ready. Upon receiving an attack, Made4Flow can now send this route to Blackhole. We have additional content on how to configure Blackhole, which you can find on our blog. If you have any questions, please contact our team of experts at . Leonardo Nascimento | Made4it Consultant

How to identify a illegal internet?

To start this article, I would like to describe what are the Clandestine Internet Providers, or also called “Illegal Internet”. Illegal Internet service providers (Gato Net) are typically set up by individuals or companies that subscribe to a residential Internet connection and generally do not have licenses from government authorities to resell Internet service. They usually subscribe to a residential connection and share it or sell it to their neighbors, friends, etc. Illegal internet service providers, in addition to being illegal (pursuant to Article 183 of Law No. 9,472 (General Telecommunications Law—LGT)), can pose risks to our networks, since they are most often set up by people who lack the technical knowledge necessary to maintain a network with proper configuration practices. In many cases, Gato Net ends up being the target of DDoS attacks, which have negative effects on its provider. If a customer is identified at your provider who is committing such practices, legal means can be used to stop such practices. However, how do we identify a customer who is using these practices to share or sell illegal internet? With the Made4graph tool, we were able to visualize the customers with the highest bandwidth usage: We can also check the customer’s real-time graph along with the complete usage extract, showing us the consumption in each period that he was connected: With this information we were able to make some comparisons with other customers who have the same amount of bandwidth contracted. If there is a big difference in usage between customers, we have an indication that this user is illegally sharing the Internet. Another observation we can make is by using the Customer History Chart; there, we’ll see a pattern similar to that of an ISP’s charts, where the heaviest traffic occurs between 7:00 p.m. and 10:00 p.m. It’s important for internet service providers to be proactive and maintain their networks so they don’t have to deal with even bigger problems! Would you like to see more chart options in the Made4Graph software? Try the demo version or schedule a presentation with the Made4it sales team! Gelso Baltazar | Consultant

What should you monitor on your network?

Monitoring provider networks is essential to ensure service delivery is never interrupted. There are several tools and protocols that can be used to perform this monitoring, but in this article we will focus on Zabbix with monitoring via SNMP v2.

How to configure Zabbix 6.0 LTS

Hey everyone, how’s it going? My name is Henrique; I’m a consultant at Made4it and part of the infrastructure and servers team. Today we’re going to learn a little about Zabbix, and I’ll give you an overview of what Zabbix is, how the tool works, and also show you how to install and configure a Zabbix server. For those who don’t know, Zabbix is ​​a totally open source monitoring solution for companies of any size. What does the Zabbix tool do? → Provides health monitoring of servers, networks, applications and services through parameters and metrics. → It also provides a flexible mechanism for sending customized notifications about detected alerts and events and we can send these notifications via email, Telegram, SMS and even opening a ticket in HelpDesk systems. → Reporting and data visualization based on stored data for analysis and decision making. → Supports polling and trapping ⇒ Polling: Zabbix sends requests through configured metrics ⇒ Trapping: Zabbix receives the information through the configured metrics → It has a graphical interface for viewing and configuring the monitoring (devices to be monitored → applications, servers and services) → Support is provided by Zabbix SIA and also by official partners; Made4it recently became an official Zabbix partner. Zabbix version 6.0 LTS Recently, Zabbix was updated to version 6.0 LTS, which has several new features such as: Kubernets Monitoring, Business Service at a completely different level, High Availability Cluster (HA Cluster) native, Advanced Service SLA Calculation Logic , Machine Learning. Follow the link for a deeper reading about the 6.0 LTS update: https://www.zabbix.com/br/whats_new_6_0#business_service_monitoring Requirements for installing Zabbix 6.0 LTS In this tutorial, we use a virtual machine running Debian 11 with the following specifications: 4 GB of RAM, 2 vCPUs, and 50 GB of disk space, which is more than enough for a simple installation for case studies. Our installation will be all-in-one, meaning everything will be on a single server (database, Zabbix Server, and Zabbix Frontend). Installation Here are the commands we used in the installation video: Now we’ve adjusted the Postgres ph_hba.conf file so that any change in the database will require the password that we just defined. Restart postgres Installation of Zabbix Server After that, we add our Zabbix user password in the configuration file. Search for #DBPassword= uncomment and add your password Installation of Zabbix Frontend Now, we tweak the Zabbix web config file Now we restart our services Done! Now all you have to do is access it via the WEB and finish the configuration. Installation and configuration via WEB → Go to http://IP_SERVIDOR/zabbix in your browser → Select the language you want, in this case, I selected “Portuguese (pt_BR)” → Checking the prerequisites, if everything is OK, proceed. → On the next screen, add: → Define a name for the server → Also select a “timezone”, in this case, we add America/Sao_Paulo → Choose a theme (we leave it as default) → Summary/check of settings from previous screens. → Installation completed! After completing the steps above, you will be redirected to the login screen. By default, the credentials are: → Completed installation and 100% functional Zabbix. You can follow the configuration video on our youtube channel and if you still have questions about the configuration or the tool, please contact our team. If you need assistance with installing or configuring Zabbix, please contact us!

Understand the main screens of Made4Graph!

Our Made4Graph software features several screens designed to help internet service providers manage their PPPoE customers. To better understand why Made4Graph is needed , check out this content where we explain what the tool is all about. Here, we’ll navigate through the tool to get to know each of the main screens better—how they work and what information they display.In Made4Graph, which is accessible via the web, the home screen provides an overview of the status of the network as a whole. When you open the software, you’ll see a graph showing the number of users connected across the entire network, and on the right, a “pie chart” showing the proportion of these users per PPPoE concentrator. And yes, it’s also compatible with Mikrotik (even if you still use Winbox or something else to manage it). Below, we also have information regarding the addressing of clients on the network. Made4Graph brings us very important data on the proportion of users with CGNAT IPv4 (RFC6598) or with public IP addressed in the CPE, in addition to IPv6. This data is of great value, especially for those who are deploying IPv6 on the network, guaranteeing and helping the future of the Internet. Scrolling down a little further, we have the exact number of clients registered per NAS (or BRAS/BNG/Access Concentrador, that is, whatever you prefer to call your authentication equipment) in addition to a “top 10 users disconnected in the last 24 hours”. In the disconnected customers report, Made4Graph highlights the customers who have disconnected the most in the last 24 hours, immediately flagging users who may be experiencing issues to support and the ISP’s Level 1 team. We know that customers with a high number of disconnections and reconnections within a short period of time are a strong indication of problems with the CPE or related equipment, allowing support to be proactive with these critical customers (by contacting them, opening a support ticket for verification, etc.), which increases the end customer’s confidence in the ISP’s availability and service quality. Imagine yourself as an internet user who notices that your connection is dropping frequently, and then—out of the blue—you receive a call from support saying they’ve noticed this and are asking for your permission to run a check. This is truly a major selling point for the ISP. One of the most used features of Made4Graph is the real time graph. With it, we can have real-time data on the end user’s bandwidth consumption in an intuitive graph and with action buttons such as “Release Bandwidth” and “Disconnect” the user. It is also possible on this screen to access a “Consumption History” of the referred customer. If you, for example, want to know what the user’s bandwidth consumption was in the last 24 hours, Made4Graph brings this possibility. The action buttons are widely used by ISPs in their support/N1 that makes the first contact with the customer, since they allow to quickly and effectively diagnose common problems either with a “Ping” (Made4Graph can directly ping the end user on the verification screen) or freeing up the user’s bandwidth so that he can fully test the “throughput” of his connection, already showing any possible bottleneck in the network. On the same screen, a little further down, we also have information about the TR069 module. For ISPs that implement Made4Graph with support for TR069 (and that have their CPEs managed by it) we have a general overview of the user’s CPE with data such as physical port status, wireless signal strength, devices connected to the router, among other information. Made4Graph is not limited to a graph visualization tool. We have a dedicated section for “reports” where, with his ability and overview of the network, he manages to bring data of great value to the ISP’s operational body. One of the most used by support staff is the “Lost Carrier” report. Here, Made4Graph reports an overview of users disconnected in a given time interval due to loss of communication between the NAS and the CPE at the customer’s home. We know that customers disconnected by “Lost Carrier” are often a strong indication of network instabilities, and having data from these users allows for a more predictive and proactive action by the ISP. Made4Graph is a great piece of software to help ISPs manage their access users better, not being restricted by manufacturer or protocol, as it is compatible with the main brands in the market and with PPPoE and IPoE, while unifying the management of users from across the ISP. It eliminates the need to configure access users and passwords on routers, interact with them via command lines and the like, reduces the complexity of the operation and verification of end users and brings data that allow more predictive and proactive actions with access clients. . Aligned with the TR069, it guarantees facilities in the management of the users’ CPE where a “reset” of a router or ONT is no longer a reason to move a technician to the user’s home since the management and configuration can be completely done remotely by their support team already in contact with the customer via communication tools. Made4Graph’s goal is to be the best technical support tool for ISPs customer support including end-to-end customer visibility, problem resolution, auto provisioning and automation support. It’s software made for ISP by people who understand ISP. You can access our demo version and learn a little more about our software at – Gabriel Henrique, a Network Analyst at Made4it, has been working with technology and ISPs for over 10 years.

Made4Graph, the tool your ISP needs to manage your users.

We know that every small ISP starts with Mikrotik equipment and that it has a very intuitive “graphic” interface (via its own application, Winbox). Many ISPs grow with these facilities in hand and end up using them for various operational and often administrative tasks.

Graphics on the subscriber’s invoice, will it become a law for all states?

In the state of Espírito Santo, a law was passed in October 2021, Espírito Santo Law No. 11,201, which requires all postpaid broadband providers to include a customer usage chart on the bill. In case you’re not familiar with it, we’ve written another article explaining this law in more detail; you can check it out by clicking this link. You might think this law applies only to the state of Espírito Santo, but given Brazil’s “modus operandi,” the other states in our country will likely follow the precedent already set by the state court and the Supreme Federal Court (STF). But what does the law say? Let’s highlight the main points here: All customers should see their average daily internet usage—based on data sent and received—displayed on their bill in the form of a usage graph. Simply put, a usage graph like those from Zabbix, Cacti, LibreNMS, PRTG, or Dude, showing interface usage in Mbps (megabits per second). This graph does not include the early morning hours, from midnight to 8 a.m. Fine of R$4,000.00 to R$15,000.00 per customer But now you might be thinking: I understand the law, I understand its consequences, but now, how can I solve this problem or how can I protect myself? We at Made4it have a solution that’s perfectly suited to solving this problem, Made4Graph, our software designed to assist the Support, NOC, and Call Center teams, provides the necessary information through real-time and historical usage charts for PPPoE or IPoE customers and also offers insights into customer behavior. Using the historical usage graph, you can access the information required by this law. Doing this manually—collecting each customer’s usage chart to include on their invoice—is impossible for the operation. With that in mind, our development team provides an API that allows you to integrate with your management system, such as ISP Integrator, IXC, RBX (Routerbox), SGP, Mk-auth, MKSolutions, TopsApp, HubSoft, ReceitaNet, or any other software you use—allowing you to easily automate your invoice issuance process so that invoices are automatically generated with your customer’s historical usage chart. Made4Graph also includes TR069, which allows you to manage customer routers, generate reports to help you be more proactive with customers, and perform many other functions. It’s very easy to learn more—just contact us via WhatsApp, and one of our consultants will assist you and answer all your questions. Click here to speak directly to our contact:43 98485-4013

DDoS attacks, how should ISPs protect themselves?

How should ISPs protect themselves from DDOS attacks?

What to do before a maintenance window?

In the day-to-day life of the Network and Internet area, it is very common for Internet Providers (ISPs) to have the need to carry out certain changes at controlled times, whenever there may be a considerable risk to the Network, due to some implementation, equipment migration , etc and we call this action the maintenance window