Symptoms
An OACI VE backup experienced a failure, but the backup got created physically on the hardware node.
/var/log/pa/vps.log on OACI IM node contains these errors:
2017-07-25 09:41:00,835 (0a144b9a-efc2-4b69-a6b7-6f67ccc6e9a5) ERROR GenericVm2VfTask [Shared executor thread #4 @1
@BACKGROUND] - timeout()
2017-07-25 09:41:00,835 (0a144b9a-efc2-4b69-a6b7-6f67ccc6e9a5) WARN GenericVm2VfTask [Shared executor thread #4 @1
@BACKGROUND] - VM2VF operation [BACKUP] (reqId=926) finished with rc=-1 (0xFFFFFFFF)
The following log extract can be found in /var/log/pa/vps.vm2vf.log on the hardware node, where the VE is located:
2017-07-25 03:41:00,686 (0a144b9a-efc2-4b69-a6b7-6f67ccc6e9a5 backup_ve_cb) INFO Vm2VfApiOpsAdapter [ActiveMQ Session Task-612] - entering backup_ve_cb
...
2017-07-25 04:06:54,200 (0a144b9a-efc2-4b69-a6b7-6f67ccc6e9a5 backup_ve_cb callback) INFO BackupCallbackProxy [Thread-237908] - INF pthread_cond_timedwait [110]: cbp 0x7fd5c8019ac8 job 0x44572 [__PrlSrv_GetCommonPrefs_cbp] index [3136] (backup_action, backup.c, 492) @[common/generic_sdk_cb_tools.c][150][cbps_timeout_sleep][107385])
2017-07-25 04:06:54,200 (0a144b9a-efc2-4b69-a6b7-6f67ccc6e9a5 backup_ve_cb callback) INFO BackupCallbackProxy [Thread-237909] - INF PrlJob_GetStatus : job 0x44572 PJS_FINISHED @[common/generic_sdk_cb_tools.c][110][cbps_job_check_status_ex][107385])
2017-07-25 04:06:54,200 (0a144b9a-efc2-4b69-a6b7-6f67ccc6e9a5 backup_ve_cb callback) INFO BackupCallbackProxy [Thread-237910] - INF job [0x44572] finished already @[common/generic_sdk_cb_tools.c][156][cbps_timeout_sleep][107385])
After such backup failures, the used backup space is not reclaimed.
Cause
The backup operation got stuck in internal PrlSrv_GetCommonPrefs call to the Virtuozzo SDK interface, which led to incorrect result of the operation. The behavior is planned to be improved in CCU-17582.
Resolution
Remove the created backup and restart the vm2vf service on the node to remove similar stuck events:
# service PACI-vm2vf restart
In order to free up the backup space on the customer's subscription, it is necessary to contact Odin Technical Support and apply the following article for the failed backups: