Get started
Solutions

Backup to Yandex Object Storage via Bacula on Centos 7

In these instructions, you'll learn how to create backups in Yandex Object Storage and recover data via Bacula on CentOS 7 in the Yandex.Cloud infrastructure.

Bacula consists of several components:

  • Bacula Director: Controls the backup and recovery process.
  • File Daemon: Provides access to backup files.
  • Storage Daemon: Reads and writes files to the hard disk.
  • Catalog: Maintains the file catalog used for backups. The catalog is stored in an SQL database.
  • Bacula Console: A management console for interacting with Bacula Director.

To back up and recover your data via Bacula:

  1. Before you start.
  2. Create a VM.
  3. Install Bacula and the additional components.
  4. Configure MariaDB.
  5. Configure backup storage.
  6. Configure the Bacula components.
  7. Create a backup.
  8. Check the backup.
  9. Run the recovery process.
  10. Check the recovered data.

If you no longer need the backup or recovered data, delete all related resources.

Before you start

  1. Go to the management console. Then log in to Yandex.Cloud or sign up if don't already have an account.
  2. On the billing page, make sure you linked a billing account, and it has the ACTIVE or TRIAL_ACTIVE status. If you don't have a billing account, create one.

If you have an active billing account, you can create or select a folder to run your VM in from the Yandex.Cloud page.

Learn more about clouds and folders.

The cost for backup and recovery includes:

Creating VMs

To create a VM:

  1. On the folder page in the management console, click Create resource and select Virtual machine.

  2. In the Name field, enter a name for the VM: bacula-vm.

  3. Select the availability zone to host the VM in.

  4. Under Images from Cloud Marketplace, click Select. Select a public CentOS 7 image.

  5. Under Computing resources, select:

    • Platform: Intel Cascade Lake.
    • Guaranteed vCPU share: 20%.
    • vCPU: 2.
    • RAM: 2 GB.

  6. In the Network settings section, select the network and subnet to connect the VM to. If you don't have a network or subnet, create them right on the VM creation page.

  7. In the Public address field, leave the Auto value to assign a random external IP address from the Yandex.Cloud pool, or select a static address from the list if you reserved one in advance.

  8. Specify data required for accessing the VM:

    • Enter the username in the Login field.

    • In the SSH key field, paste the contents of the public key file.

      You need to create a key pair for the SSH connection yourself. Learn how to connect to VMs via SSH.

    Alert

    The IP address and host name (FQDN) to connect to the VM are assigned on VM creation. If you selected No address in the Public address field, you won't be able to access the VM from the internet.

  9. Click Create VM.

Creating the VM may take several minutes.

When the VM is created, it is assigned a public IP address and hostname (FQDN). This data can be used for SSH access.

Create a bucket

To create a bucket for backups in Object Storage:

  1. Go to the Yandex.Cloud management console and select the folder where you want to perform the operations.

  2. On the folder page, click Create resource and select Bucket.

  3. In the Name field, enter a name for the bucket: bacula-bucket.

  4. In the Bucket access field, select Restricted.

  5. In the Storage class field, select Cold.

  6. Click Create bucket.

Create a service account

Create a service account and assign it the role editor.

Create static access keys

Create static access keys.

Make sure to immediately save the key_id ID and secret secret key. You will not be able to get the key value again.

Set up the AWS CLI

After bacula-vm switches to RUNNING:

  1. You can learn the IP address of the VM under Network on the VM page in the management console.

  2. Connect to the VM over SSH. You can use the ssh tool in Linux and macOS or PuTTY for Windows.

    The recommended authentication method when connecting over SSH is using a key pair. Don't forget to set up the created key pair: the private key must match the public key sent to the VM.

  3. Install the yum repository:

    $ sudo yum install epel-release -y

  4. Install pip:

    $ sudo yum install python-pip -y

  5. Install the AWS CLI:

    $ sudo pip install awscli --upgrade

  6. Set up the AWS CLI:

    $ sudo aws configure

    The command will request values for parameters:

    • AWS Access Key ID: Enter the key_id ID that you received when generating the static key.
    • AWS Secret Access Key: Enter the secret key that you received when creating the static key.
    • Default region name: ru-central1.
    • Default output format: json.

  7. Check that the /root/.aws/credentials file contains the correct key_id and secret values:

    $ sudo cat /root/.aws/credentials

  8. Check that the /root/.aws/config file contains the correct Default region name and Default output format values:

    $ sudo cat /root/.aws/config

Install Bacula and additional components

  1. Install the Bacula components:

    $ sudo yum install -y bacula-director bacula-storage bacula-console bacula-client

  2. Install MariaDB:

    $ sudo yum install -y mariadb-server

  3. Install s3fs to mount the Object Storage bucket on the server:

    $ sudo yum install -y s3fs-fuse

  4. Install the text editor nano:

    $ sudo yum install -y nano

Configure MariaDB

  1. Run MariaDB:

    $ sudo systemctl start mariadb

  2. Check that MariaDB is running:

    $ sudo systemctl status mariadb | grep Active

  3. Enable MariaDB to run at system startup:

    $ sudo systemctl enable mariadb

  4. Create database tables and configure access rights:

    $ /usr/libexec/bacula/grant_mysql_privileges
$ /usr/libexec/bacula/create_mysql_database -u root
$ /usr/libexec/bacula/make_mysql_tables -u bacula

  5. Configure the database security parameters:

    $ sudo mysql_secure_installation
    • Enter current password for root (enter for none): Press Enter to skip the field.
    • Set root password? [Y/n]: Enter Y, set the root password, and confirm it. You will need the password in the next step.
    • Remove anonymous users? [Y/n]: Press Enter to accept the default value.
    • Disallow root login remotely? [Y/n]: Press Enter to accept the default value.
    • Remove test database and access to it? [Y/n]: Press Enter to accept the default value.
    • Reload privilege tables now? [Y/n]: Press Enter to accept the default value.

  6. Log in to the DB command line and enter the root password created in the previous step:

    $ mysql -u root -p

  7. Create the password bacula_db_password for the bacula user:

    UPDATE mysql.user SET Password=PASSWORD('bacula_db_password') WHERE User='bacula';
FLUSH PRIVILEGES;
exit

  8. Enable the MySQL library for Bacula:

    $ sudo alternatives --config libbaccats.so

    Enter 1 to select MySQL:

      Selection    Command
-----------------------------------------------
   1           /usr/lib64/libbaccats-mysql.so
   2           /usr/lib64/libbaccats-sqlite3.so
*+ 3           /usr/lib64/libbaccats-postgresql.so

Enter to keep the current selection[+], or type selection number: 1

Configure backup storage

Prepare a backup folder

  1. Create the /tmp/bacula backup folder:

    $ sudo mkdir /tmp/bacula

  2. Configure access rights to /tmp/bacula:

    $ sudo chown -R bacula:bacula /tmp/bacula
$ sudo chmod -R 700 /tmp/bacula  
$ sudo semanage permissive -a bacula_t

Mount the bucket to the file system

Mount the bucket to the file system to upload copied files to Object Storage:

  1. Mount the bucket using s3fs:

    $ sudo s3fs bacula-bucket /tmp/bacula -o url=https://storage.yandexcloud.net -o use_path_request_style -o allow_other -o nonempty -o uid=133,gid=133,mp_umask=077
    • bacula-bucket: The name of the bucket in Object Storage.
    • uid=133: The ID of the bacula user from the /etc/passwd file.
    • gid=133: The ID of the bacula group from the /etc/passwd file.

  2. Check the access rights for the /tmp/bacula folder:

    $ sudo ls -la /tmp/bacula/

    The command output should look as follows:

    drwx------.  2 bacula bacula        31 Sep 18 09:16 .
drwxrwxrwt. 10 root   root         265 Sep 18 08:59 ..

  3. Check that the bacula user can create files in the /tmp/bacula folder:

    1. Temporarily enable the bash shell for the bacula user:

      $ sudo sed -i "/^bacula/ s@/sbin/nologin@/bin/bash@" /etc/passwd

    2. Create an arbitrary file in the /tmp/bacula folder:

      $ sudo runuser -l  bacula -c 'touch /tmp/bacula/test.test'

    3. Make sure that the file test.test was created in the /tmp/bacula folder:

      $ sudo ls -la /tmp/bacula | grep test.test

    4. On the catalog page in the management console, open Object Storage. Make sure the file test.test is visible in the bacula-bucket bucket.

    5. Delete the test file:

      $ sudo runuser -l  bacula -c 'rm -f /tmp/bacula/test.test'

    6. Disable the bash shell for the bacula user:

      $ sudo sed -i "/^bacula/ s@/bin/bash@/sbin/nologin@" /etc/passwd

Configure Bacula components

Configure Bacula Director

  1. Open the Bacula Director configuration file:

    $ sudo nano /etc/bacula/bacula-dir.conf

  2. In the Director configuration block, add the line DirAddress = 127.0.0.1 to set up a connection with Bacula Director:

    Director {                            # define myself
  Name = bacula-dir
  DIRport = 9101                # where we listen for UA connections
  QueryFile = "/etc/bacula/query.sql"
  WorkingDirectory = "/var/spool/bacula"
  PidDirectory = "/var/run"
  Maximum Concurrent Jobs = 1
  Password = "@@DIR_PASSWORD@@"         # Console password
  Messages = Daemon
  DirAddress = 127.0.0.1
}

  3. For your convenience, rename the task BackupClient1 to BackupFiles:

    Job {
  Name = "BackupFiles"
  JobDefs = "DefaultJob"
}

  4. In the RestoreFiles task configuration, add the line Where = /tmp/bacula-restores to make /tmp/bacula-restores a folder for restored files:

    Job {
  Name = "RestoreFiles"
  Type = Restore
  Client=bacula-fd
  FileSet="Full Set"
  Storage = File
  Pool = Default
  Messages = Standard
  Where = /tmp/bacula-restores
}

  5. Under the FileSet configuration section named Full Set under Include:

    • Add the compression = GZIP line to the Options section to enable compression during backup.
    • Specify File = / to back up the entire file system.
    	FileSet {
	  Name = "Full Set"
	  Include {
		Options {
		  signature = MD5
		  compression = GZIP
		}
		File = /
	  }
	  Exclude {
		File = /var/spool/bacula
		File = /tmp
		File = /proc
		File = /tmp
		File = /.journal
		File = /.fsck
	  }
	}

  6. You can find the internal IP address of the VM under Network on the VM page in the management console.

  7. In the Storage configuration block, enter the internal IP address of the VM in the Address field to set up a connection with Storage Daemon:

    Storage {
  Name = File
# Do not use "localhost" here
  Address = <VM internal IP address> # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "@@SD_PASSWORD@@"
  Device = FileStorage
  Media Type = File
}

  8. To connect to the DB, in the Catalog configuration block, enter the database password dbpassword = "bacula_db_password" created when configuring MariaDB:

    # Generic catalog service
Catalog {
  Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport =
  dbname = "bacula"; dbuser = "bacula"; dbpassword = "bacula_db_password"
}

  9. Save the file.

  10. Check that the bacula-dir.conf file doesn't contain any syntax errors:

    $ sudo bacula-dir -tc /etc/bacula/bacula-dir.conf

    If there aren't any error messages, the configuration is correct.

Configure Storage Daemon

  1. Open the Storage Daemon configuration file:

    $ sudo nano /etc/bacula/bacula-sd.conf

  2. You can find the internal IP address of the VM under Network on the VM page in the management console.

  3. In the Storage configuration block, enter the internal IP address of the VM in the SDAddress field to set up a connection with Storage Daemon:

    Storage {                             # definition of myself
  Name = BackupServer-sd
  SDPort = 9103                  # Director's port
  WorkingDirectory = "/var/spool/bacula"
  Pid Directory = "/var/run/bacula"
  Maximum Concurrent Jobs = 20
  SDAddress = <VM internal IP address> 
}

  4. In the Device configuration block, specify the Archive Device = /tmp/bacula folder for backups:

    Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /tmp/bacula 
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}

  5. Save the file.

  6. Check that the bacula-sd.conf file doesn't contain any syntax errors:

    $ sudo bacula-sd -tc /etc/bacula/bacula-sd.conf

    If there aren't any error messages, the configuration is correct.

Create passwords for Bacula components

Bacula Director, Storage Daemon, and File Daemon use passwords for inter-component authentication.

To set passwords for Bacula components:

  1. Generate passwords for Bacula Director, Storage Daemon, and File Daemon:

    $ DIR_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
$ SD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
$ FD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`

  2. Put the passwords in the configuration files:

    $ sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bacula-dir.conf
$ sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bconsole.conf
$ sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-sd.conf
$ sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-dir.conf
$ sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-dir.conf
$ sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-fd.conf

Run the Bacula components

  1. Run the Bacula components:

    $ sudo systemctl start bacula-dir
$ sudo systemctl start bacula-sd
$ sudo systemctl start bacula-fd

  2. Check that the Bacula components are running:

    $ sudo systemctl status bacula-dir
$ sudo systemctl status bacula-sd
$ sudo systemctl status bacula-fd

  3. Set up the Bacula components to launch at system startup:

    $ sudo systemctl enable bacula-dir
$ sudo systemctl enable bacula-sd
$ sudo systemctl enable bacula-fd

Create a backup

  1. Open Bacula Console:

    $ sudo bconsole

  2. Create a label to set up a backup profile:

    label

  3. Name the new volume MyVolume:

    Enter new Volume name: MyVolume

  4. Enter 2 to select the File pool:

    Defined Pools:
     1: Default
     2: File
     3: Scratch
Select the Pool (1-3): 2

  5. Run the backup process:

    run

    Select 1 to start the BackupFiles task:

    A job name must be specified.
The defined Job resources are:
    1: BackupFiles
    2: BackupCatalog
    3: RestoreFiles
Select Job resource (1-3): 1

    Enter yes to confirm the backup process launch:

    OK to run? (yes/mod/no): yes

  6. Check the backup status:

    status director

    Command output if the backup is running:

    Running Jobs:
Console connected at 12-Sep-19 07:22
 JobId Level   Name                       Status
======================================================================
     2 Full    BackupFiles.2019-09-12_07.22.56_03 is running

    Command output if the backup is complete:

    Running Jobs:
Console connected at 12-Sep-19 07:25
No Jobs running.
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
     2  Full     32,776    483.6 M  OK       12-Sep-19 07:24 BackupFiles

  7. Wait for the backup to complete and exit Bacula Console:

    exit

Check the backup

To make sure that the backup is complete:

  1. On the catalog page in the management console, open Object Storage.
  2. Open the bacula-bucket bucket.
  3. Make sure it contains MyVolume.

Run the recovery process

  1. Delete an arbitrary file, like the ping utility, to check the recovery:

    $ sudo rm -f /bin/ping

  2. Make sure that the ping utility is deleted:

    $ ping

    The command output should look as follows:

    bash: ping: command not found

  3. Log in to Bacula Console:

    $ sudo bconsole

  4. Run a full recovery:

    restore all

    Select 5 to start a recovery from the latest backup:

    To select the JobIds, you have the following choices:
    1: List last 20 Jobs run
    2: List Jobs where a given File is saved
    3: Enter list of comma separated JobIds to select
    4: Enter SQL list command
    5: Select the most recent backup for a client
    6: Select backup for a client before a specified time
    7: Enter a list of files to restore
    8: Enter a list of files to restore before a specified time
    9: Find the JobIds of the most recent backup for a client
    10: Find the JobIds for a backup for a client before a specified time
    11: Enter a list of directories to restore for found JobIds
    12: Select full restore to a specified Job date
    13: Cancel
Select item:  (1-13): 5

    Enter done to confirm the full recovery:

    You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.

cwd is: /
$ done

    Enter yes to confirm the recovery process launch:

    OK to run? (yes/mod/no): yes

  5. Check the recovery status:

    status director

    Command output if the recovery is running:

    Running Jobs:
Console connected at 12-Sep-19 07:25
 JobId Level   Name                       Status
======================================================================
     3         RestoreFiles.2019-09-12_07.27.42_05 is running

    Command output if the recovery is complete:

    Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
     2  Full     32,776    483.6 M  OK       12-Sep-19 07:24 BackupFiles
     3           32,776    1.136 G  OK       12-Sep-19 07:27 RestoreFiles

  6. Wait for the recovery to complete and exit Bacula Console:

    exit

Check the recovery

  1. Make sure that the /tmp/bacula-restores folder now contains the recovered data:

    $ sudo ls -la /tmp/bacula-restores

    The command output should look as follows:

    total 16
dr-xr-xr-x. 15 root   root    201 Sep 12 07:09 .
drwx------.  4 bacula bacula   35 Sep 12 07:09 ..
lrwxrwxrwx   1 root   root      7 Sep 12 07:27 bin -> usr/bin
dr-xr-xr-x   5 root   root   4096 Sep 12 07:01 boot
drwxr-xr-x   2 root   root      6 Sep 12 07:22 dev
drwxr-xr-x  79 root   root   8192 Sep 12 07:07 etc
drwxr-xr-x   3 root   root     18 Sep 12 07:01 home
lrwxrwxrwx   1 root   root      7 Sep 12 07:27 lib -> usr/lib
lrwxrwxrwx   1 root   root      9 Sep 12 07:27 lib64 -> usr/lib64
drwxr-xr-x   2 root   root      6 Apr 11  2018 media
drwxr-xr-x   2 root   root      6 Apr 11  2018 mnt
drwxr-xr-x   2 root   root      6 Apr 11  2018 opt
dr-xr-x---   3 root   root    217 Sep 12 07:21 root
drwxr-xr-x   2 root   root      6 Sep 12 07:22 run
lrwxrwxrwx   1 root   root      8 Sep 12 07:27 sbin -> usr/sbin
drwxr-xr-x   2 root   root      6 Apr 11  2018 srv
dr-xr-xr-x   2 root   root      6 Sep 12 07:22 sys
drwxr-xr-x  13 root   root    155 Mar  4  2019 usr
drwxr-xr-x  19 root   root    267 Sep 12 07:01 var

  2. Make sure that the ping utility is in the /tmp/bacula-restores folder:

    $ sudo ls -la /tmp/bacula-restores/bin/ping

    The command output should look as follows:

    -rwxr-xr-x 1 root root 66176 Aug  4  2017 /tmp/bacula-restores/bin/ping

  3. Copy the ping utility to the main file system:

    $ sudo cp /tmp/bacula-restores/bin/ping /bin/ping

  4. Make sure that ping works:

    $ sudo ping 127.0.0.1 -c 1

    The command output should look as follows:

    PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms

--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms

  5. Delete a copy of recovered files to free up disk space:

    $ sudo rm -rfd /tmp/bacula-restores/*

How to delete created resources

To stop paying for data storage, delete the VM and Object Storage bucket.

If you reserved a static public IP address specifically for this VM:

  1. Open the Virtual Private Cloud in your folder.
  2. Go to the IP addresses tab.
  3. Find the address you need, click ellipsis, and select Delete.