Question App Contents Update Hangs and Panel Very slow - No Pattern

[jafo666]

Morning All,

Hoping someone could help or point in the right direction,

We are intermittently seeing the same issue, the initial thought was EPG or an influx of users logging in but after keeping an eye on it we don't think that is the cause.... Server CPU always stays nice and low. There doesn't seem to be a pattern to it.

Now we are wondering if its a Cloudflare/SSL issue? Also thought could it be a CRON issue but because there isn't a pattern not so sure.

The issue:

The app will get stuck on the updating contents screen, waiting for the "Live" and "EPG".



When this happens the panel also becomes very slow and unresponsive, it happens using different apps.

Users who are already using the app are not effected and can watch and change streams fine unless they manually click update contents.

So far we have tried:

New main server and XUI.ONE install
Different XUI.ONE versions 1.5.5 / 1.5.12 / 1.5.13 (Currently back on 1.5.13)
Removing EPG
Tweaking settings in the panel
Main and all LBs are on 20.04

It seems to be resolved with either:

Restarting Services
Regen Cache & Restart Streams


When using NXT there were no issues..... but moved back to XUI.ONE and now starting to contemplate moving over to ODIN because we are scratching our heads.....?

Any information will be most helpful

 

[julian]

Morning All,

Hoping someone could help or point in the right direction,

We are intermittently seeing the same issue, the initial thought was EPG or an influx of users logging in but after keeping an eye on it we don't think that is the cause.... Server CPU always stays nice and low. There doesn't seem to be a pattern to it.

Now we are wondering if its a Cloudflare/SSL issue? Also thought could it be a CRON issue but because there isn't a pattern not so sure.

The issue:

The app will get stuck on the updating contents screen, waiting for the "Live" and "EPG".

View attachment 5729

When this happens the panel also becomes very slow and unresponsive, it happens using different apps.

Users who are already using the app are not effected and can watch and change streams fine unless they manually click update contents.

So far we have tried:

New main server and XUI.ONE install
Different XUI.ONE versions 1.5.5 / 1.5.12 / 1.5.13 (Currently back on 1.5.13)
Removing EPG
Tweaking settings in the panel
Main and all LBs are on 20.04

It seems to be resolved with either:

Restarting Services
Regen Cache & Restart Streams


When using NXT there were no issues..... but moved back to XUI.ONE and now starting to contemplate moving over to ODIN because we are scratching our heads.....?

Any information will be most helpful

Hi, not a nice problem, this is what I would check. to rule out network or main problem. Also, Cloud flare has limits with free dns/proxy service and they don't want you to use it for streaming video content, only static content is permitted. Read the terms of use. They can and do limit or remove clients from time to time. If you have many clients and have to use CF proxy on main you must break them up into small cloudflare domain/dns/proxy groups to avoid having too many requests on one domain/dns.
1) Create another extra domain to main server without cloudflare proxy activated and check if the problem is replicated on it during the time of slowdown. that way you rule cloudflare out.
2) Continue with main server, use top command and look whats going on in processes, CPU Ram ext must stay low and cpu idle around 5% most of the time. There must be no large spikes in ram, and a consistent pattern, keep an eye on process ''mariadb'', which should also be consistent. Repeat the process for load balancers.
3) Use iftop (A powerfull network tool similar to top) and netstat. iftop will show you your network usage in real time as well as connection data in real time. Network usage must be consistent with with current load/demand. If network usage (incoming vs outgoing bandwidth) is much higher than expected you might be under ddos. or hacked. Check connection logs in os including ssh, look for ''connection successful'' from some IP you do not recognize, indicating un authorized entry (server hacked).
4) . Check if cpu or memory spike when the cache cron runs, if you are regularly getting token expired in logs increase cron time. also try, depending on number of users, to disable redis and cache.
5) Ping between main and lb servers and there must be very low packet loss (example no more then 1 in 5,000 packet loss or timeout). Do the same from some client location, that too has to be low packet loss. also ping your cloudflare url/ip(s) to rule out cloudflare problem.
6) If you are using some sort of vps for main ( some sort of public or private cloud multi core server) make sure your data center is delivering what they are promising in terms of cpu core and resources (most over subscribe to some extent). That's why a dedicated machine is always best.
7) Make sure Flood Limit and time are set to something reasonable
8) look for problems in vod or series were some content is located on some lb but set to main or some other server by mistake. example file movie "terminator" is located in "movies_server_1" but in the movie entry under the servers tab "main" is selected. That causes unpredictable behavior.
Hope this helps and good luck solving your problem

 

[jafo666]

So after lots of changes and testing it seems that the issue only happens when we have the EPG added, we have tried two different EPG providers, same issue.....

So yeah, it's a strange one!

If anyone has any ideas please let me know, otherwise I will update once we have done some more trial and error changes.

 

[julian]

So after lots of changes and testing it seems that the issue only happens when we have the EPG added, we have tried two different EPG providers, same issue.....

So yeah, it's a strange one!

If anyone has any ideas please let me know, otherwise I will update once we have done some more trial and error changes.

Hi. Download iftop and observe out going connections. Look for some ip with lots of abnormal traffic. Somebody could be trying to use you as epg source, downloading all your epg too often. May be try to find the user in logs and compare to iftop. In general settings > security > max simultaneous downloads set to something like 2. In general settings > api > append extra data default format. Also in same page disable epg api for restreamer. Try disabling epg api for all, just to test if problem goes away Hope it helps