Article ID: 120906, created on Apr 5, 2014, last review on May 6, 2014

  • Applies to:
  • Operations Automation 5.5
  • Operations Automation 5.4

Symptoms

Customer schedules a weekly PACI backup, but it is not created for a long time

Cause

No free space on backup nodes, or misconfiguration in PACI DB

Resolution

Backups on PACI IM run according to settings in /usr/local/share/PACI-im/IM-config.xml:

<!-- backup job details -->
<backups batchSize="10" intervalInMin="1" maxDurationInMin="200"/>

by default PACI runs backups in 10 VEs batches. It means that free space on Backup node is checked using the sum of 10 VEs that should be backed up.

In this case try to calculate maximum size of the 10 biggest VEs:

im=# with topdisks as (select size from ve_storage where ve_ref in (select id from ve where template_id in (select id from templates where ((os_type in (select type from os_types where family='linux') and technology='CT')) or technology='VM')) order by size desc limit 10) select sum(size) from topdisks;
sum
-----
1625

In this example for performing 10 biggest VEs backup batch - 1.65 T of free space is required. Please verify that Backup nodes have enough space.

As a workaround, you may change batchSize to "5" for example - in this case backup task will ran smaller batches that will require less space. Please note that any changes in IM-config.xml require PACI-IM service restart.

Another kind of cause is misconfiguration in PACI DB:

  1. check backups_to_remove table - it shouldn't have many backups, if it is - indication of some problems with backups removal. In this case, PACI IM logs should be checked.
  2. check active_backups table - if you will notice any long running backups (more than 1-2 days) - indication of terracotta or dispatcher failures.

    to fix it - remove long running backups:

    im=# select id,ve_ref,started,ended,stage from active_backups;
      id  | ve_ref |            started            | ended |  stage
    ------+--------+-------------------------------+-------+---------
     6061 |    381 | 2013-08-02 12:00:01.276774+02 |       | STARTED
     6084 |    398 | 2013-08-03 06:02:03.397072+02 |       | STARTED
    (2 rows)
    
    im=# delete from active_backups;
    DELETE 2
    

    The info about running backups will be re-read on the next backup task run.

Search Words

Customer VM backup is not working

backup not running

PACI not backing up

backups not getting rotated

backups not getting removed

Backups

PACI

ac82ce33439a9c1feec4ff4f2f638899 caea8340e2d186a540518d08602aa065 5356b422f65bdad1c3e9edca5d74a1ae 2554725ed606193dd9bbce21365bed4e 5b048d9bddf8048a00aba7e0bdadef37 e12cea1d47a3125d335d68e6d4e15e07

Email subscription for changes to this article
Save as PDF