Question CPU Overload 100% on XUI only During Major Matches

[vicente113]

Hi everyone,

I'm facing a critical issue with my XUI panel. Everything works fine under normal conditions, but during major football events (especially Champions League matches), my main server’s CPU suddenly spikes right at kickoff.

Typically, 99% of my clients are already connected before the match starts, and CPU usage stays below 15%. But as soon as the game begins, the load jumps instantly, and the server becomes completely unresponsive. The only way to stop it is by immediately renaming either the portal.php or player_api.php file.

It’s clearly a targeted attack possibly a flood or brute-force. I’ve already implemented Fail2Ban to block abusive requests, but that hasn’t been enough.

Has anyone experienced a similar issue or found an effective way to mitigate this type of attack?

Thanks in advance for any advice.

 

[mmicha17]

Hi there.

Is your main behind cloudflare ? Behind a proxy?
Did you enabled any additional logs to see what is actually going on?

 

[vicente113]

Hi there.

Is your main behind cloudflare ? Behind a proxy?
Did you enabled any additional logs to see what is actually going on?

Yes mainserver runs behind CF and i have a script that monitors the access log in real time and automatically bans any IPs detected as flooding or exceeding a defined rate limit.

 

[mmicha17]

Yes mainserver runs behind CF and i have a script that monitors the access log in real time and automatically bans any IPs detected as flooding or exceeding a defined rate limit.

Personally i would start by analyzing those logs so that you get a general feeling of what is actually going on.

Since you are using CF, you can leverage WAF rules to block a lot of stuff such as unwanted countries, specific requests .e.t.c before even they reach your main.

In addition to that, you need to make sure that your server is only allowing access to cloudflare ips (since you are using the proxy mode).

Generally speaking, there are a lot of stuff you can do to actually fix this. But first, try to understand the bad requests that are flooding your server.
Are they handshake requests from mac scanners ? There are plenty stuff to deal with. Just start from the logs, this is where i would start from.

Cloudflare is a very powerful service. Hosting a service behind cloudflare combined with their WAF rules can make a lot of difference.

 

[vicente113]

Indeed, I have already configured Cloudflare rules to block many countries and specific types of requests.
Yes, the main server only accepts traffic from Cloudflare IPs and my own hostname. I have even disabled direct access using the server's IP address.

Personally i would start by analyzing those logs so that you get a general feeling of what is actually going on.

Since you are using CF, you can leverage WAF rules to block a lot of stuff such as unwanted countries, specific requests .e.t.c before even they reach your main.

In addition to that, you need to make sure that your server is only allowing access to cloudflare ips (since you are using the proxy mode).

Generally speaking, there are a lot of stuff you can do to actually fix this. But first, try to understand the bad requests that are flooding your server.
Are they handshake requests from mac scanners ? There are plenty stuff to deal with. Just start from the logs, this is where i would start from.

Cloudflare is a very powerful service. Hosting a service behind cloudflare combined with their WAF rules can make a lot of difference.

Thank you for your response.
Indeed, I have already configured Cloudflare rules to block many countries and specific types of requests.
Yes, the main server only accepts traffic from Cloudflare IPs and my own hostname. I have even disabled direct access using the server's IP address.

The handshake and get_profile.. requests are already controlled through Cloudflare and also limited by rate limiting. But the issue I experience only at the start of the match is clearly a real attack and I can’t even find any trace of it, not even in the logs.

This issue is truly strange I experience no problems for weeks or even over a month, except during major Champions League events, specifically when PSG is playing.

 

[mmicha17]

Well if you can't find something in the logs , then this sounds fishy.
Panel is cracked one?

Keep also in mind that it might also be a main server issue. Meaning, that there is a case where you are renting a server that is sold to you as a dedicated server, but in the end its just a vps. The reason i am saying this is because i have seen this happen twice over the last 8 months.

Providers selling dedicated servers, but instead they are vps and when there is a big traffic , the resources are limited.

I am not saying this is the case, but mentioning this as another possibility

Also, i am guessing that you do have a powerfull main to respond to the traffic isn't it ?

 

[vicente113]

This issue is not related to the number of connected clients or a sudden spike in demand. For example, last week, nearly 99% of clients were already connected before the event started, and CPU usage remained under 15% But as soon as the game begins, the load jumps instantly

I'm using a OVH dedicated server (32 threads, 128 GB RAM, NVMe), so hardware limitations are not the cause.

 

[maxnet]

The same thing is happening to me... I'm using version 1.5.13, but it didn't happen to me before when I had version 1.5.5.

There might be something wrong with version 1.5.13; it only happens to me at European events.

 

[mmicha17]

So hardware is a no then.

I always follow these rules in case i am having any kind of issues


1) Enable full logging on nginx.
2) Check what is the process that is taking high resources exactly on your main. Is it mysql ? It should be ideally. You can enable verbose logging as well.
3) Do you have redis enabled ?


There are plenty of cases where you can actually look.
The think is to catch it on real time.

Start by identifying why your cpu is high process wise and the isolate it accordingly. Step by step

 

[akawi11]

you need to optimize your main server to avoid see your clients traffic as D-D-o-s, just do not think someone attack your server
fail2ban for XCUI traffic in main server where XCUI working is bad idea, you just make it even worst

 

[waskip1]

you need to use WAF in cloudflare to filter out bad ip addresses and bad ASN from spamming your player_api.php url