1. xuione panel -> servers -> manage server -> edit specific server -> ssl certificate - uncheck enable ssl -> save.
^ it should work you dont need to mess with nginx config direct.
2. if u want to mess with nginx configuration - listen 443; is the ssl port.
make sure u dont have any 301 or 307 redirects to port 443 server block it config.
3. for some reason if ur using cloudflare -> site -> ssl -> edge certificate -> always use https ( make sure its not on ).
Update:
^ my bad i didnt undestand ur question. i dont think u can disable ssl only for panel. i might b wrong. wait for other answers.