Search Engine: Elastic

Article ID: 120906, created on Apr 5, 2014, last review on Dec 28, 2016

  • Applies to:
  • Operations Automation 6.0
  • Operations Automation 5.5
  • Operations Automation 5.4

Symptoms

A weekly OACI backup has been scheduled but it has not been created for a long period of time.

Cause

No free space on backup nodes or a OACI DB misconfiguration.

Resolution

Backups on OACI IM are run according to settings in /usr/local/share/PACI-im/IM-config.xml:

<!-- backup job details -->
<backups batchSize="10" intervalInMin="1" maxDurationInMin="200"/>

by default OACI runs backups in 10 VEs batches. It means that free space on Backup node is checked using the sum of 10 VEs that should be backed up.

In this case, try to calculate maximum size of the 10 biggest VEs:

im=# with topdisks as (select size from ve_storage where ve_ref in (select id from ve where template_id in (select id from templates where ((os_type in (select type from os_types where family='linux') and technology='CT')) or technology='VM')) order by size desc limit 10) select sum(size) from topdisks;
sum
-----
1625

In this example, ~1.65 T of free space is required for performing 10 biggest VEs backup batch. Please verify that Backup nodes have enough space.

As a workaround, change batchSize to "5", for example. In this case the backup task will run smaller batches that will require less space. Please note that any changes in IM-config.xml require PACI-IM service restart.

service PACI-im restart

Another kind of cause is a misconfiguration in OACI DB:

  1. Check backups_to_remove table - there should not be many backups. If there are, it is an indication of some problems with backups removal. In this case, OACI IM logs should be checked.
  2. Check active_backups table for long running backups (more than 1-2 days). Those are the indication of terracotta or dispatcher failures.

ac82ce33439a9c1feec4ff4f2f638899 caea8340e2d186a540518d08602aa065 5356b422f65bdad1c3e9edca5d74a1ae 2554725ed606193dd9bbce21365bed4e 5b048d9bddf8048a00aba7e0bdadef37 e12cea1d47a3125d335d68e6d4e15e07 956c448bddc7e1f3585373687602379f 6f1456866eed87488c0f02b298a741c0

Email subscription for changes to this article
Save as PDF