Search Engine: Elastic

Article ID: 131279, created on Aug 10, 2017, last review on Aug 10, 2017

  • Applies to:
  • Operations Automation 7.0

Symptoms

Provider found one or several orphaned backups on OACI storage nodes. Recently OA environment suffered from vm2vf outages on Virtuozzo hosts.

In vps.log of IM one may find that initial backup request (scheduled or manual) failed after several minutes with the following message:

fast failure for backup_ve_cb after 240000 MILLISECONDS

That means that IM tried to connect to vm2vf for 4 minutes and then cancelled the operation due to vm2vf unavailability. However, backup pipeline survived the outage and when vm2vf became available, backup operation was completed with the following message:

Failed to retrieve pipeline #3202, callback backup_done will not be invoked

Cause

Even if vm2vf is not available in some particular moment of time, ActiveMQ stores scheduled messages for later send. When it finds vm2vf 'endpoint' online, it delivers the message and backup procedure is started. Upon its finish, result will not be pushed to IM database since operation timeout already expired, but physically backup will remain on Storage host.

This behavior will be improved in scope of CCU-15182.

Resolution

The issue here is vm2vf service instability on Virtuozzo host, which should be addressed at first priority. Issue with backups will be solved by stable work of vm2vf.

To perform a cleanup of orphaned backups, please contact Odin Technical Support.

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 0871c0b47b3b86ae3b1af4c2942cd0ce 1941880841f714e458ae4dc3d9f3062d

Email subscription for changes to this article
Save as PDF