Article ID: 127598, created on Nov 28, 2015, last review on Nov 21, 2016

  • Applies to:
  • Operations Automation

Symptoms

There are failed tasks for VE backups in customer CP.

The tasks fail with timeout with no callback received from vm2vf daemon. In /var/log/IM/PACI-im.log the following error is found:

2015-11-23 15:21:19,589 () ERROR Vm2VfProviderBase [VM2VF callbacks watchdog thread] - No registered callback found: ActiveCallbackId [timestamp=2015-11-22 23:21:19.484, reqId=162737, corbaReqId=927068, info=backup ve [{246853d3-b2e6-4758-afb1-150bbcb3bc9c}] to the node [192.0.2.2], contextId=(95fd9507-9dd8-4ede-9502-631beafb9c85)]

The following pattern appears for the same issue on OA 7.0:

2016-11-21 16:11:18,054 (330f4970-6848-4b69-bed1-3d66704108c4) ERROR GenericVm2VfTask [Shared executor thread #7 @1 @INTERACTIVE] - timeout()
2016-11-21 16:11:18,054 (330f4970-6848-4b69-bed1-3d66704108c4) WARN  GenericVm2VfTask [Shared executor thread #7 @1 @INTERACTIVE] - VM2VF operation [BACKUP] (reqId=4070) finished with rc=-1 (0xFFFFFFFF)
2016-11-21 16:11:18,054 (330f4970-6848-4b69-bed1-3d66704108c4) WARN  GenericVm2VfTask [Shared executor thread #7 @1 @INTERACTIVE] - VM2VF operation [BACKUP] (reqId=4070) finished with rc=-1 (0xFFFFFFFF)

In /var/log/IM/PACI-vm2vf.log at the same time it is clear that the backup task is still in progress:

2015-11-23 15:21:40,678 (95fd9507-9dd8-4ede-9502-631beafb9c85) DEBUG NativeVm2VfCode [Thread-4272309] - [162737:927068] Callback invocation: in_progress(TIMEOUT: Dispatcher is still alive, 0)
2015-11-23 15:22:43,051 (95fd9507-9dd8-4ede-9502-631beafb9c85) INFO  NativeVm2VfCode [Thread-4272346] - [162737:927068] INF callback: job [__PrlSrv_CreateVmBackup] cbp 0x7f553ccdce08 index [162088] h type [PHT_EVENT/PET_DSP_EVT_BACKUP_PROGRESS_CHANGED] h/job [0x6f97d8/0x69fbac] (backup_action, backup.c, 492) @[common/generic_sdk_cb.c][366][generic_sdk_cb][51918])
2015-11-23 15:22:43,052 (95fd9507-9dd8-4ede-9502-631beafb9c85) DEBUG NativeVm2VfCode [Thread-4272347] - [162737:927068] Callback invocation: in_progress(PET_DSP_EVT_BACKUP_PROGRESS_CHANGED, 78)

Cause

The backup operation lasts longer than the task timeout.

Resolution

To solve the issue, find out if it is expected that the backup task lasts so long. Normal rate is 1 hour for a 100GB virtual machine. If the VM is indeed huge and requires more than 4 hours to be completed, edit the task timeout in /usr/local/share/PACI-im/IM-config.xml:

change

<vm2vf callback-timeout-min="240" log="true" mode="remote">

to

<vm2vf callback-timeout-min="360" log="true" mode="remote">

to increase the timeout to 6 hours

and restart the service:

# service PACI-im fast-restart

Note: on 7.0 the line related to vm2vf will contain mode="on-node" pattern instead

Search Words

ERROR ClusterImpl$Updater [Batch DB updater] - Failed to persist VE update

callback-timeout-min

clone failed

scheduled backup

backup Failed

Callback timeout detected:

creation in progress

cloning

No registered callback found:

backup fail

backup vm

x

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07

Email subscription for changes to this article
Save as PDF