Hey all,
So this has been a long time coming, but here's a new release. Okay it's still early access, I just needed to get something out there for the following reasons:
Now this release addresses a vulnerability in Xtream Codes where a person could restream your streams without a valid user ID, making them invisible. To address this I've enforced some security settings, also implemented a cron that checks for invalid streams and kills them. On top of this I've enforced mag locks and minimum password length of 8 characters for resellers. You CAN change these settings back, but they're forced on every update.
An Xtream UI flaw I've known about for a while is relating to XSS (Cross-Site Scripting), I planned to fix this months ago but never really got around to it, and didn't really anticipate how dangerous this could be so I've made much more of an effort this time around. I now believe this release to be fully patched against any XSS attacks and I've also reinforced any SQL statements to ensure they're correctly escaped to block SQL injection attacks. This is easily the most secure release yet.
I've removed auto-update entirely, however the current version can be checked by going to the Settings page. As well as this, I've also included a GeoLite2 update option. Head to the settings page and if an update is available you can click to install it. My server periodically downloads the GeoLite2 database direct from MaxMind so it's all automated.
On top of the above I've also reintegrated the database editor, however password protected with your MySQL password. To get this MySQL password, run the config.py file in the pytools folder and decrypt your configuration file. Store this somewhere so you can access the database when required.
Tables are updated automatically now so there's no need to click the Update Tables option etc. This will only work if permissions are correct, and Xtream UI will warn you if this isn't the case on the login screen. To fix permissions run the new permissions.sh file in /home/xtreamcodes/iptv_xtream_codes.
This may not seem like the biggest update in the world, but it's a very important one and marks my return to releasing new features. Release schedule will be set in advance, with the plan to be fortnightly. This will happen once 22 becomes official, integrating a few more features and bugfixes before that happens.
The rest of this release consists of various bugfixes, I've read through various posts and integrated what I could.
Next Release (22 Official):
I will be implementing further bugfixes along with completing the translation. The current release is about 80% translation mapped, once this is 100% I'll have it translated into various languages for 22 official.
Future Release (23 EA):
VPN Integration! I told you it was coming, and it will be. Okay okay, there was a delay, but still. It's coming.
The plan is to give Xtream UI users the ability to host their own VPN servers, with Xtream UI automatically installing and managing those servers for you. You click Install VPN, give it the IP and root SSH credentials and the installer will do it's thing.
VPN capability can be allocated to packages, with a separate option for max simultaneous connections. The VPN will only be authenticated during the period of the package. When a user is given access to the VPN, they authenticate to the API using their line username and password so they don't need an additional set of credentials, however the actual VPN authentication takes place with a secure certificate generated on the main panel. The VPN servers will have NFS access to the certificate folder to authenticate, and will use an API to communicate back to the main server to show usage statistics, upload, download, current users, CPU usage, memory usage, uptime etc.
VPN servers will be load balanced per country / city, will adhere to max connection per server limits and you'll be able to see throughput on the dashboard to ensure your servers aren't overloaded. Full logging is available in the interface.
As it's OpenVPN based, you can connect via the OpenVPN client for Windows, Android, iOS or many other devices. The users will be able to download their certificates from the panel (if the setting is enabled) or use an URL / QR Code. I've built a very basic Android app that's completely open source and will be provided free of charge, I want those more experienced with Java than I am to make improvements to it and release to the public, and the idea is that whoever wants it can rebrand it and utilise it.
Here's a quick demo of the basic app:
https://streamable.com/a9s51
I'll be providing my own API, but will also ensure Xtream UI is backwards compatible with other VPN API's such as DeployVPN.
Changelog for this Release:
Previously in R22 Early Access:
YOUTUBE TUTORIALS - THANKS emre1393:
https://www.youtube.com/playlist?list=P ... k3JH5U2ekn
TROUBLESHOOTING:
You'll notice most issues are down to permissions, you should probably take note.
PROCESS MONITOR - HIGH CPU / MEMORY USAGE
If you're having issues with high cpu or memory usage, you can list the processes that are causing issues using the process monitor function. Go to the dashboard and click the CPU / Memory Usage of an individual server or select Process Monitor from the settings cog dropdown to begin. It can take a few seconds to enumerate processes but will list anything that is being used by the Xtream Codes user. You can then see what individual streams or processes are causing you the biggest issues.
An example that can cause high CPU usage is having movies set up on one server, but the video files themselves are hosted on another server. This will cause XC to download those files using the system_api.php file to the server and attempt to process them upon completion. Doing this with hundreds of movies will cause you big issues. Best practice is to host the movies on the same server as the one encoding / symlinking them. Try to use symlink more often than not as it's the least intensive.
For high memory usage where you can't isolate the issue, try the following:
viewtopic.php?f=13&t=3014&p=14819
Notes from a trusted user who had 100% CPU and 100% Memory, managed to reduce this drastically using process monitor:
BACKUPS
If backups aren't working for you, run the following command:
TMDb ISN'T WORKING
Your database may not have updated correctly. Ensure the admin folder has the correct permissions by running the below command:
Now go to Settings -> Database, click Update Tables.
If it still doesn't work, ensure your TMDb API key is correct and active. If it definitely is, then your mysql user for XC probably doesn't have the right permissions to modify tables. Give the user all the available permissions and it try the above again.
WATCH FOLDERS
To set up folder watching, go to the Settings dropdown, Folder Watch and click Settings. You need to set up your Genre matching here. So firstly, click Update from TMDb to get the latest genres. You can attribute a category that you have created to each TMDb Genre. The first genre for each TV Show or Movie will be the one it selects.
The above is optional, this is for more accurate matching, however you can override these settings in the next step.
To set up a folder to be watched, click the + button on the Folder Watch page and fill out the required details. Make sure you select the correct type or you'll have a bunch of incorrectly allocated movies or episodes. The override settings allow you to select a category that the movie / tv show will default to, or a fallback incase it doesn't match the category allocation you may have set up in the first step. It also has various other options for those who want to customise their matches more.
If your scan doesn't seem to be running, check your crontab with the following command:
Look for watch_folders.php. If it isn't there, delete the crontab_refresh file from the tmp folder of xtreamcodes, and restart the service. If it still isn't there, maybe your crontab for xtreamcodes is immutable as it can't be changed. Up to you to figure that one out.
PROCESS MONITORING
So I've made a process killer that checks all PID's in the database against live PID's on the server, killing any it doesn't need. This could potentially help people running into CPU issues, or just help in general as XC isn't the best at killing PID's.
pid_monitor.zip
To install it, download the file from above and extract it here (on each server you want it running on, include LB's):
INSTALL
For manual update, download the release from the link below.
http://xtream-ui.com/releases/release_22c.zip
Update Script
So this has been a long time coming, but here's a new release. Okay it's still early access, I just needed to get something out there for the following reasons:
- To show I'm back
- To fix some security issues that could affect earlier releases
Now this release addresses a vulnerability in Xtream Codes where a person could restream your streams without a valid user ID, making them invisible. To address this I've enforced some security settings, also implemented a cron that checks for invalid streams and kills them. On top of this I've enforced mag locks and minimum password length of 8 characters for resellers. You CAN change these settings back, but they're forced on every update.
An Xtream UI flaw I've known about for a while is relating to XSS (Cross-Site Scripting), I planned to fix this months ago but never really got around to it, and didn't really anticipate how dangerous this could be so I've made much more of an effort this time around. I now believe this release to be fully patched against any XSS attacks and I've also reinforced any SQL statements to ensure they're correctly escaped to block SQL injection attacks. This is easily the most secure release yet.
I've removed auto-update entirely, however the current version can be checked by going to the Settings page. As well as this, I've also included a GeoLite2 update option. Head to the settings page and if an update is available you can click to install it. My server periodically downloads the GeoLite2 database direct from MaxMind so it's all automated.
On top of the above I've also reintegrated the database editor, however password protected with your MySQL password. To get this MySQL password, run the config.py file in the pytools folder and decrypt your configuration file. Store this somewhere so you can access the database when required.
Tables are updated automatically now so there's no need to click the Update Tables option etc. This will only work if permissions are correct, and Xtream UI will warn you if this isn't the case on the login screen. To fix permissions run the new permissions.sh file in /home/xtreamcodes/iptv_xtream_codes.
This may not seem like the biggest update in the world, but it's a very important one and marks my return to releasing new features. Release schedule will be set in advance, with the plan to be fortnightly. This will happen once 22 becomes official, integrating a few more features and bugfixes before that happens.
The rest of this release consists of various bugfixes, I've read through various posts and integrated what I could.
Next Release (22 Official):
I will be implementing further bugfixes along with completing the translation. The current release is about 80% translation mapped, once this is 100% I'll have it translated into various languages for 22 official.
Future Release (23 EA):
VPN Integration! I told you it was coming, and it will be. Okay okay, there was a delay, but still. It's coming.
The plan is to give Xtream UI users the ability to host their own VPN servers, with Xtream UI automatically installing and managing those servers for you. You click Install VPN, give it the IP and root SSH credentials and the installer will do it's thing.
VPN capability can be allocated to packages, with a separate option for max simultaneous connections. The VPN will only be authenticated during the period of the package. When a user is given access to the VPN, they authenticate to the API using their line username and password so they don't need an additional set of credentials, however the actual VPN authentication takes place with a secure certificate generated on the main panel. The VPN servers will have NFS access to the certificate folder to authenticate, and will use an API to communicate back to the main server to show usage statistics, upload, download, current users, CPU usage, memory usage, uptime etc.
VPN servers will be load balanced per country / city, will adhere to max connection per server limits and you'll be able to see throughput on the dashboard to ensure your servers aren't overloaded. Full logging is available in the interface.
As it's OpenVPN based, you can connect via the OpenVPN client for Windows, Android, iOS or many other devices. The users will be able to download their certificates from the panel (if the setting is enabled) or use an URL / QR Code. I've built a very basic Android app that's completely open source and will be provided free of charge, I want those more experienced with Java than I am to make improvements to it and release to the public, and the idea is that whoever wants it can rebrand it and utilise it.
Here's a quick demo of the basic app:
https://streamable.com/a9s51
I'll be providing my own API, but will also ensure Xtream UI is backwards compatible with other VPN API's such as DeployVPN.
Changelog for this Release:
- Forced security upgrades to fix Xtream Codes exploit.
- Patched all files against XSS exploits.
- Added GeoLite2 updater.
- Reintegrated database editor.
- Added user-agent to Live Connections page.
- Fixed search not being actioned on refresh.
- Added stream icons to stream page.
- Added EPG status indicator to stream page.
- Added settings option to disable auto-refresh by default.
- Ensured quotes " don't appear in bouquets in SQL. Can break things otherwise.
- Fixed any bouquet issues (I believe).
- Added noindex and nofollow to header to deter search engines from indexing.
- Updated NGINX to newer, faster version.
- Removed reseller API for now, the code may be insecure.
- Many many bugfixes.
- More but I forgot...
Previously in R22 Early Access:
- Fixed movies not showing in bouquet order.
- Fixed activity logs page.
- Added interactive connection statistics to dashboard plus cron. Enable in Settings.
- Added port selection to Install Load Balancer.
- Added ability to change port. Edit server, change the ports. Restart server afterwards.
- Added ability to reboot server instead of just restart services.
- Fixed newline in textareas.
- Changed year to appear in brackets instead of after a hyphen.
- Added option to extend sidebar in profile.
- Fixed various bugs.
- 70% translation completed... taking it's swweeeett time.
- Hidden expired MAG / Enigma passwords in reseller dashboard.
- Added current release to Settings page so you can stay up to date.
- Added advanced manual channel order.
- Added bouquet ordering.
- Partial localisation.
- Various bug fixes.
YOUTUBE TUTORIALS - THANKS emre1393:
https://www.youtube.com/playlist?list=P ... k3JH5U2ekn
TROUBLESHOOTING:
You'll notice most issues are down to permissions, you should probably take note.
PROCESS MONITOR - HIGH CPU / MEMORY USAGE
If you're having issues with high cpu or memory usage, you can list the processes that are causing issues using the process monitor function. Go to the dashboard and click the CPU / Memory Usage of an individual server or select Process Monitor from the settings cog dropdown to begin. It can take a few seconds to enumerate processes but will list anything that is being used by the Xtream Codes user. You can then see what individual streams or processes are causing you the biggest issues.
An example that can cause high CPU usage is having movies set up on one server, but the video files themselves are hosted on another server. This will cause XC to download those files using the system_api.php file to the server and attempt to process them upon completion. Doing this with hundreds of movies will cause you big issues. Best practice is to host the movies on the same server as the one encoding / symlinking them. Try to use symlink more often than not as it's the least intensive.
For high memory usage where you can't isolate the issue, try the following:
viewtopic.php?f=13&t=3014&p=14819
Notes from a trusted user who had 100% CPU and 100% Memory, managed to reduce this drastically using process monitor:
- Make sure pid_monitor is in crons.
- Clean up streams_sys using stream tools.
- Restart services.
- For Any high percentage vod - make sure its set to same server as the source and symlink is on.
- Any live streams that are high percentage that aren't being transcoded for a reason are likely starting and stopping too much: turn them direct or on demand.
BACKUPS
If backups aren't working for you, run the following command:
PHP:
sudo chown -R xtreamcodes:xtreamcodes /home/xtreamcodes/TMDb ISN'T WORKING
Your database may not have updated correctly. Ensure the admin folder has the correct permissions by running the below command:
PHP:
sudo chown -R xtreamcodes:xtreamcodes /home/xtreamcodes/Now go to Settings -> Database, click Update Tables.
If it still doesn't work, ensure your TMDb API key is correct and active. If it definitely is, then your mysql user for XC probably doesn't have the right permissions to modify tables. Give the user all the available permissions and it try the above again.
WATCH FOLDERS
To set up folder watching, go to the Settings dropdown, Folder Watch and click Settings. You need to set up your Genre matching here. So firstly, click Update from TMDb to get the latest genres. You can attribute a category that you have created to each TMDb Genre. The first genre for each TV Show or Movie will be the one it selects.
The above is optional, this is for more accurate matching, however you can override these settings in the next step.
To set up a folder to be watched, click the + button on the Folder Watch page and fill out the required details. Make sure you select the correct type or you'll have a bunch of incorrectly allocated movies or episodes. The override settings allow you to select a category that the movie / tv show will default to, or a fallback incase it doesn't match the category allocation you may have set up in the first step. It also has various other options for those who want to customise their matches more.
If your scan doesn't seem to be running, check your crontab with the following command:
PHP:
sudo crontab -e -u xtreamcodesLook for watch_folders.php. If it isn't there, delete the crontab_refresh file from the tmp folder of xtreamcodes, and restart the service. If it still isn't there, maybe your crontab for xtreamcodes is immutable as it can't be changed. Up to you to figure that one out.
PROCESS MONITORING
So I've made a process killer that checks all PID's in the database against live PID's on the server, killing any it doesn't need. This could potentially help people running into CPU issues, or just help in general as XC isn't the best at killing PID's.
pid_monitor.zip
To install it, download the file from above and extract it here (on each server you want it running on, include LB's):
PHP:
/home/xtreamcodes/iptv_xtream_codes/crons/INSTALL
For manual update, download the release from the link below.
http://xtream-ui.com/releases/release_22c.zip
Update Script
PHP:
apt-get install unzip e2fsprogs python-paramiko -y && chattr -i /home/xtreamcodes/iptv_xtream_codes/GeoLite2.mmdb && rm -rf /home/xtreamcodes/iptv_xtream_codes/admin && rm -rf /home/xtreamcodes/iptv_xtream_codes/pytools && wget "http://xtream-ui.com/releases/release_22c.zip" -O /tmp/update.zip -o /dev/null && unzip /tmp/update.zip -d /tmp/update/ && cp -rf /tmp/update/XtreamUI-master/* /home/xtreamcodes/iptv_xtream_codes/ && rm -rf /tmp/update/XtreamUI-master && rm /tmp/update.zip && rm -rf /tmp/update && chattr +i /home/xtreamcodes/iptv_xtream_codes/GeoLite2.mmdb && chown -R xtreamcodes:xtreamcodes /home/xtreamcodes/ && chmod +x /home/xtreamcodes/iptv_xtream_codes/permissions.sh && /home/xtreamcodes/iptv_xtream_codes/permissions.sh && /home/xtreamcodes/iptv_xtream_codes/start_services.sh
