Search Engine: Elastic

Article ID: 130687, created on Apr 20, 2017, last review on Apr 20, 2017

  • Applies to:
  • Operations Automation 7.0

Symptoms

2 failovers occurred for the same hardware node in short period of time (within few hours).

As a result, there are duplicated VEs across the cluster for the VEs that were located on the failed node. OA Control Panel also shows them with Duplicate detected status.

From Virtuozzo backend side the duplicates appear in INVALID state:

[root@vz01  ~]#  prlctl list -a | grep invalid
{f324eb4f-294e-484d-adb1-425c2eb3f9a2}  invalid      172.10.16.32    CT 1000021.server-1000024-1
{6bffed22-c2d6-48c4-aa60-6168eee8bad7}  invalid      172.10.16.37    CT 1000011.server-1000032-1

The following messages from /var/log/pa/vps.log on IM server indicate 2 recover attempts for the same VE:

2017-04-07 02:28:59,444 (4fc429c6-8c6d-4c1c-8d02-a5b21dcb1d11) DEBUG Pipeline [Shared executor thread #15 @1 @INTERACTIVE] - Running pipeline #2396 [(RECOVER_VE for: VeId[1000021.server-1000024-1), created at node [im1], step 1 ([RECOVER]), mode: EXEC, state:CALLBACK, reqIs: null]

2017-04-07 04:08:34,358 (6a629ed6-761b-49d4-992f-5357f7daa746) DEBUG Pipeline [ActiveMQ Session Task-2213] - pipeline #3262 [(RECOVER_VE for: VeId[1000021.server-1000024-1]), created at node [im1], step 1 ([RECOVER]), mode: EXEC, state:CALLBACK, reqIs: null] - invoked callback for reqId 7753

Cause

Consequent node failovers are not handled correctly in OACI HA cluster. The issue is acknowledged with ID CCU-14875.

Resolution

In order to remove the VEs with INVALID states, it is required to restart the Virtuozzo Dispatcher service on the nodes, where such VEs are present:

# service parallels-server restart  

After that, it is needed to use the following article to fix the state and location of all broken VEs manually:

Wrong location or status of Cloud Infrastructure Server in Control Panel

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 0871c0b47b3b86ae3b1af4c2942cd0ce 1941880841f714e458ae4dc3d9f3062d

Email subscription for changes to this article
Save as PDF