A provider or a customer cannot log into the Operations Automation (OA) Control Panel. The Control Panel either is not accessible at all, or shows error messages such as:
An unforeseen error occurred
The corresponding service is not available at the moment, try it within 2-3 minutes.
This article describes basic troubleshooting steps when the Control Panel is down or giving an internal server error.
There may be several causes of Paralels Operations Automation Control Panel unavailability:
- Expired license
Below, you will find more details on checking for the above issues and troubleshooting them.
Important: in order to allow root cause analysis of each OA Control Panel outage, collect debug detail to be passed to the Odin Technical Support team. Use the attached oa_dump.bash script, launch it on OA Management Node, when CP is inaccessible. The script will produce a .tar.bz2 archive. Note: the OA core server should have
gdb package installed preliminarily.
Check the status of servers
Verify that all the servers listed below are up and running (the malfunctioning of any server below may affect the Control Panel). Start any servers that are down:
OA Management Node
OA Database server
OA UI (User Interface) server/cluster - the server where OA Control Panel packages are installed (package type = 'cp').
- Branding server(s)
- If your Odin Service Automation infrastructure is deployed inside Parallels Virtuozzo Containers (PVC), Parallels Server Bare Metal (PSBM), Parallels Cloud Server (PCS) or any other virtual environment, make sure that all required virtual servers are started on the corresponding hardware nodes.
Make sure all required services are running on the corresponding servers.
OA Management Node:
service pem status
- Windows-based server:
net start | find "PEM"
OA UI server:
Linux-based UI server:
service pemui status
- Windows-based UI server:
net start | find "PEMUI"
Standard Linux Shared Hosting server:
service pemhttpd status
Linux Shared Hosting NG server:
service httpd status
- Windows-based UI server:
Make sure the OA system database service is running on the corresponding server:
Linux-based installation - PostgreSQL server
- Windows-based installation - Microsoft SQL Server
If the PostgreSQL server does not start on the OA database server and shows a 'Permission denied' error message, see Odin Knowledgebase article #117810.
Analyze logs related to the problem with OA Control Panel:
- Apache or IIS logs on the branding servers
- poa-ui.log on the OA UI server
- poa.debug.log on the OA Management Node
Refer to the Knowledgebase article Odin Automation system logs location to find the required OA logs.
Analyze the part of the logs that are being appended to the log file when trying to log into the Control Panel. Use search in the Odin Knowledgebase for the error and warning messages you find in the logs.
Use the 'tail' utility to get the necessary part of the log. The 'tail' utility for Windows can be found in the Support Tools pack.
If OA servers are deployed as virtual environments, check that the proper amount of memory is allocated to the servers.
Refer to the hardware requirements in the corresponding installation guide. For example, see the Hardware Requirements for version 5.5.
If you are using Virtuozzo products, use the native PVC/PSBM/PCS tools to check and tune memory and other resource limits configured for virtual servers: vzlist, vzctl.
/proc/user_beancountersfor failed counters for specific resources and increase them if necessary.
For example, in case vzctl enters into container with OSA core throws:
# vzctl enter 10 Unable to fork: Cannot allocate memory
It means kmemsize is hit, as also mentioned in
/var/log/messages. To increase it:
# vzctl set 10 --save --kmemsize <softlimit>:<hardlimit>
Where the limits would be bigger than the current ones.
Check and tune (if necessary) the number of Linux threads on the OA UI server. Refer to Odin Knowledgebase article #118465 for more details.
Make sure there is enough free disk space on the OA Management Node, UI and Database servers.
In the case of a Linux-based installation, you also should check the number of disk inodes available on servers using the 'df -i' command.
On PVC, PSBM or PCS servers, the amount of disk inodes allocated for a container can be changed with the vzctl utility:
# vzctl set CT_ID --save --diskspace SOFTLIMIT:HARDLIMIT --diskinodes SOFTLIMIT:HARDLIMIT
See Virtuozzo Knowledgebase article #114019 for more details.
Make sure the OA UI and branding servers can access the OA Management Node and each other over the network (Backnet).
Check the HTTPS port is open in the firewall for external incoming connections to the branding servers.
- Check that ports 8080, 6308, 8352, 8354 are reachable on MN by UI node.
Refer to the Odin Automation Firewall Configuration Guide for the list of network ports to be open between different servers in a Odin Automation infrastructure.
If your servers are deployed as virtual environments, make sure the virtualization product's license is active and allows the required number of virtual environments to be launched. For PVC, PSBM or PCS servers, use the following command:
Pay attention to the following parameters of the license in the output of the command above - make sure the license is active and the number of allowed CPUs, containers and Virtual Machines has not been exceeded:
status="ACTIVE" expiration="unlimited" cpu_total=16 (2) ct_total="unlimited" (54) nr_vms="unlimited" (20)
Check that an active license is installed in the Odin Operations Automation system.
Check the license status in the OA Provider Control Panel at Configuration Director > License Manager > License.
Try restarting the following services to solve the problem with Control Panel availability:
OA UI service (
Apache or IIS on branding server
OA Management Node service (
- OA Database service (PostgreSQL or MS SQL Server)
Follow the instructions in Odin Knowledgebase article #4642: How to restart OA system services: UI, Management Node, Agents.
Finally, if all the above actions do not help, restart your physical servers or virtual environments (containers, Virtual Machines) where OA services are running.