Backup to Yandex Object Storage via Bacula

Written by

Yandex Cloud

Getting started
Create a VM
Set up the AWS CLI
Install Bacula and additional components
Configure MariaDB
Set up the storage
- Prepare a backup folder
- Mount the bucket to the file system
Configure Bacula components
Create a backup
- Check the backup
Recover the files
- Check the recovered files
How to delete the resources you created

You can use Yandex Object Storage for VM backup and recovery via the Bacula utility.

Bacula consists of several components:

Bacula Director: Controls the backup and recovery process.
File Daemon: Provides access to backup files.
Storage Daemon: Reads and writes files to the hard disk.
Catalog: Maintains the file catalog used for backup. The catalog is stored in a MariaDB database.
Bacula Console: A management console for interacting with Bacula Director.

To set up backup and recovery via Bacula:

If you no longer need the resources you created, delete them.

Getting started

Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
On the Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder to run your infrastructure.

Learn more about clouds and folders.

Required paid resources

The cost for backup and recovery includes:

Fee for VM computing resources (see Yandex Compute Cloud pricing).
A fee for data storage in a bucket and operations with data (see Yandex Object Storage pricing).
Fee for using a dynamic or static public IP (see Yandex Virtual Private Cloud pricing).

Create a bucket

To create a bucket for backups in Object Storage:

Management console

Go to the Yandex Cloud management console and select the folder where you will perform the operations.
On the folder page, click Create resource and select Bucket.
In the Name field, enter a name for the bucket.
In the Bucket access field, select Restricted.
In the Storage class field, select Cold.
Click Create bucket.

Create a service account

Create a service account and assign it the editor role.

Create static access keys

Create static access keys.

Make sure to immediately save the ID key_id and secret key secret. You will not be able to get the key value again.

Create a VM

To create a VM:

Management console

In the management console, click Create resource and select Virtual machine.
In the Name field, enter the VM name bacula-vm.
Select an availability zone to place the VM in.
Under Image/boot disk selection, go to the Cloud Marketplace tab and select a public CentOS 7 image.
Under Computing resources, select:
- Platform: Intel Cascade Lake.
- Guaranteed vCPU share: 20%.
- vCPU: 2.
- RAM: 2 GB.
Under Network settings, select the network and subnet to connect the VM to. If there are no networks available, create one:
1. Select Create network.
2. In the window that opens, enter the network name and specify the folder to host the network.
3. (optional) To automatically create subnets, select the Create subnets option.
4. Click Create.
  
  Each network must have at least one subnet. If there is no subnet available, create one by selecting Add subnet.
Under Public address, keep Auto to assign your VM a random external IP address from the Yandex Cloud pool, or select a static address from the list if you reserved one in advance.
Enter the VM access information:
- Enter the username in the Login field.
- In the SSH key field, paste the contents of the public key file.
  
  You will need to create a key pair for the SSH connection yourself, see Creating an SSH key pair.
Click Create VM.
Wait for the VM to change to the RUNNING status.

Set up the AWS CLI

To set up the AWS CLI utility on your bacula-vm instance:

In the management console, go to the VM page and find out its public IP address.
Connect to the VM via SSH.

The recommended authentication method when connecting over SSH is using a key pair. Set up the generated key pair: the private key must match the public key sent to the VM.
Update the packages installed in the system. For this, in the terminal, run:
```
yum update -y
```
Install the AWS CLI:
```
yum install awscli -y
```
Set up the AWS CLI:
```
sudo aws configure
```
Specify the parameter values:
- AWS Access Key ID: The key_id that you received when generating the static key.
- AWS Secret Access Key: The secret key that you received when generating the static key.
- Default region name: ru-central1.
- Default output format: json.
Make sure that the /root/.aws/credentials file contains relevant values for key_id and secret:
```
sudo cat /root/.aws/credentials
```
Make sure that the /root/.aws/config file contains relevant values for Default region name and Default output format:
```
sudo cat /root/.aws/config
```

Install Bacula and additional components

Install the Bacula components:

sudo yum install -y bacula-director bacula-storage bacula-console bacula-client

Install MariaDB:
```
sudo yum install -y mariadb-server
```
Install the s3fs utility to mount the Object Storage bucket to the file system:
```
sudo yum install -y epel-release
sudo yum install -y s3fs-fuse
```
Install the text editor nano:
```
sudo yum install -y nano
```

Configure MariaDB

Run MariaDB:
```
sudo systemctl start mariadb
```

Check that MariaDB is running:

sudo systemctl status mariadb | grep Active

Enable MariaDB to run at system startup:
```
sudo systemctl enable mariadb
```

Create database tables and configure access rights:

/usr/libexec/bacula/grant_mysql_privileges
/usr/libexec/bacula/create_mysql_database -u root
/usr/libexec/bacula/make_mysql_tables -u bacula

Secure your database:
```
sudo mysql_secure_installation
```
For the following queries:
- Enter current password for root (enter for none): Press Enter to skip the field.
- Set root password? [Y/n]: Enter Y, set the root password, and confirm it. You will need the password in the next step.
- Remove anonymous users? [Y/n]: To accept the default value, press Enter.
- Disallow root login remotely? [Y/n]: To accept the default value, press Enter.
- Remove test database and access to it? [Y/n]: To accept the default value, press Enter.
- Reload privilege tables now? [Y/n]: To accept the default value, press Enter.
Log in to the DB command line and enter the root password created in the previous step:
```
mysql -u root -p
```

Create the password bacula_db_password for the bacula user:

UPDATE mysql.user SET Password=PASSWORD('bacula_db_password') WHERE User='bacula';
FLUSH PRIVILEGES;
exit

Enable the MySQL library for Bacula:

sudo alternatives --config libbaccats.so

Enter 1 to select MySQL:

  Selection    Command
-----------------------------------------------
   1           /usr/lib64/libbaccats-mysql.so
   2           /usr/lib64/libbaccats-sqlite3.so
*+ 3           /usr/lib64/libbaccats-postgresql.so

Enter to keep the current selection[+], or type selection number: 1

Set up the storage

Prepare a backup folder

Create the /tmp/bacula backup folder:
```
sudo mkdir /tmp/bacula
```

Set up access rights for the /tmp/bacula folder:

sudo chown -R bacula:bacula /tmp/bacula
sudo chmod -R 700 /tmp/bacula
sudo semanage permissive -a bacula_t

Mount the bucket to the file system

Use the s3fs utility to mount the bucket to upload backups to Object Storage. To do this, run the command below and specify the bucket name:
```
sudo s3fs <bucket_name> /tmp/bacula \
  -o url=https://storage.yandexcloud.net \
  -o use_path_request_style \
  -o allow_other \
  -o nonempty \
  -o uid=133,gid=133,mp_umask=077
```
Where:
- uid=133: The ID of the bacula user from the /etc/passwd file.
- gid=133: The ID of the bacula group from the /etc/passwd file.

Check the access rights for the /tmp/bacula folder:

sudo ls -la /tmp/bacula/

Result:

drwx------.  2 bacula bacula        31 Sep 18 09:16 .
drwxrwxrwt. 10 root   root         265 Sep 18 08:59 ..

Check that the bacula user can create files in the /tmp/bacula folder:
1. Temporarily enable the bash shell for the bacula user:
```
sudo sed -i "/^bacula/ s@/sbin/nologin@/bin/bash@" /etc/passwd
```
2. Create an arbitrary file in the /tmp/bacula folder:
```
sudo runuser -l  bacula -c 'touch /tmp/bacula/test.test'
```
3. Make sure that the file test.test was created in the /tmp/bacula folder:
```
sudo ls -la /tmp/bacula | grep test.test
```
4. In the management console, on the folder page, select Object Storage and make sure the test.test file is in the bucket.
5. Delete the test file:
```
sudo runuser -l  bacula -c 'rm -f /tmp/bacula/test.test'
```
6. Disable the bash shell for the bacula user:
```
sudo sed -i "/^bacula/ s@/bin/bash@/sbin/nologin@" /etc/passwd
```

Configure Bacula components

Configure Bacula Director

Open the Bacula Director configuration file:
```
sudo nano /etc/bacula/bacula-dir.conf
```

To set up a connection to Bacula Director, go to the Director configuration section and add the line DirAddress = 127.0.0.1:

...
Director {                              # define myself
  Name = bacula-dir
  DIRport = 9101                        # Specify the port (a positive integer) on which the Director daemon will listen for Bacula Console connections.
                                        # This same port number must be specified in the Director resource of the Console configuration file.
                                        # The default is 9101, so normally this directive need not be specified.
                                        # This directive should not be used if you specify the DirAddresses (plural) directive.
  QueryFile = "/etc/bacula/query.sql"
  WorkingDirectory = "/var/spool/bacula"
  PidDirectory = "/var/run"
  Maximum Concurrent Jobs = 1
  Password = "@@DIR_PASSWORD@@"         # Console password
  Messages = Daemon
  DirAddress = 127.0.0.1
}    
...

For your convenience, rename the task BackupClient1 to BackupFiles:

...
Job {
  Name = "BackupFiles"
  JobDefs = "DefaultJob"
}
...

To assign /tmp/bacula-restores as a folder for your recovered files, add the line Where = /tmp/bacula-restores to the RestoreFiles job configuration:

...
Job {
  Name = "RestoreFiles"
  Type = Restore
  Client=bacula-fd
  FileSet="Full Set"
  Storage = File
  Pool = Default
  Messages = Standard
  Where = /tmp/bacula-restores
}
...

Under the FileSet configuration section named Full Set under Include:

Add the compression = GZIP line to the Options section to enable compression during backup.
Specify File = / to back up the entire file system.

...
	FileSet {
	  Name = "Full Set"
	  Include {
		Options {
		  signature = MD5
		  compression = GZIP
		}
		File = /
	  }
	  Exclude {
		File = /var/spool/bacula
		File = /tmp
		File = /proc
		File = /tmp
		File = /.journal
		File = /.fsck
	  }
	}
...

In the management console, go to the VM page and find out its private IP address.

To set up an outbound connection to the Storage Daemon, in the Storage configuration section, enter the VM's internal IP address in the Address field:

...
Storage {
  Name = File
# Do not use "localhost" here
  Address = <internal_IP_address_of_the_VM>  # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "@@SD_PASSWORD@@"
  Device = FileStorage
  Media Type = File
}
...

To connect to the DB, in the Catalog configuration section, specify the database password dbpassword = "bacula_db_password" that you created when setting up MariaDB:

...
# Generic catalog service
Catalog {
  Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport =
  dbname = "bacula"; dbuser = "bacula"; dbpassword = "bacula_db_password"
}
...

Save the file.
Check that the bacula-dir.conf file contains no syntax errors:
```
sudo bacula-dir -tc /etc/bacula/bacula-dir.conf
```
If there aren't any error messages, the configuration is correct.

Configure Storage Daemon

Open the Storage Daemon configuration file:
```
sudo nano /etc/bacula/bacula-sd.conf
```

To set up an outbound connection to the Storage Daemon, in the Storage configuration section, enter the VM's internal IP address in the SDAddress field:

...
Storage {                                      # definition of myself
  Name = BackupServer-sd
  SDPort = 9103                                # Specifies port number on which the Storage daemon listens for Director connections. The default is 9103.
  WorkingDirectory = "/var/spool/bacula"
  Pid Directory = "/var/run/bacula"
  Maximum Concurrent Jobs = 20
  SDAddress = <internal_IP_address_of_the_VM>         # This directive is optional, and if it is specified,
                                               # it will cause the Storage daemon server (for Director and File daemon connections) to bind to the specified IP-Address,
                                               # which is either a domain name or an IP address specified as a dotted quadruple.
                                               # If this directive is not specified, the Storage daemon will bind to any available address (the default).
}
...

In the Device configuration block, specify the Archive Device = /tmp/bacula folder for backups:

...
Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /tmp/bacula
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}
...

Save the file.
Check that the bacula-sd.conf file doesn't contain any syntax errors:
```
sudo bacula-sd -tc /etc/bacula/bacula-sd.conf
```
If there aren't any error messages, the configuration is correct.

Create passwords for Bacula components

Bacula Director, Storage Daemon, and File Daemon use passwords for inter-component authentication.

To set passwords for Bacula components:

Generate passwords for Bacula Director, Storage Daemon, and File Daemon:

DIR_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
SD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
FD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`

Put the passwords in the configuration files:

sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bacula-dir.conf
sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bconsole.conf
sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-sd.conf
sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-dir.conf
sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-dir.conf
sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-fd.conf

Run the Bacula components

Run the Bacula components:

sudo systemctl start bacula-dir
sudo systemctl start bacula-sd
sudo systemctl start bacula-fd

Check that the Bacula components are running:

sudo systemctl status bacula-dir
sudo systemctl status bacula-sd
sudo systemctl status bacula-fd

Set up the Bacula components to launch at system startup:

sudo systemctl enable bacula-dir
sudo systemctl enable bacula-sd
sudo systemctl enable bacula-fd

Create a backup

Open Bacula Console:
```
sudo bconsole
```
To set up a backup profile, create a label:
```
label
```
Name the new volume MyVolume:
```
Enter new Volume name: MyVolume
```

To select the File pool, enter 2:

Defined Pools:
     1: Default
     2: File
     3: Scratch
Select the Pool (1-3): 2

Run the backup process:

run

To start the BackupFiles job, select 1:

A job name must be specified.
The defined Job resources are:
    1: BackupFiles
    2: BackupCatalog
    3: RestoreFiles
Select Job resource (1-3): 1

To confirm the startup, enter yes:

OK to run? (yes/mod/no): yes

Check the backup status:

status director

Result if the backup is running:

Running Jobs:
Console connected at 12-Sep-19 07:22
 JobId Level   Name                       Status
======================================================================
     2 Full    BackupFiles.2019-09-12_07.22.56_03 is running

Result if the backup is complete:

Running Jobs:
Console connected at 12-Sep-19 07:25
No Jobs running.
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name
====================================================================
     2  Full     32,776    483.6 M  OK       12-Sep-19 07:24 BackupFiles

Wait for the backup to complete and exit Bacula Console:
```
exit
```

Check the backup

To make sure that the backup is complete:

Management console

In the management console, on the folder page, select Object Storage.
Open the bucket.
Make sure it contains MyVolume.

Recover the files

To check the recovery process, delete an arbitrary file, for example, the ping utility:
```
sudo rm -f /bin/ping
```
Make sure that the ping utility is deleted:
```
ping
```
Result:
```
bash: ping: command not found
```
Log in to Bacula Console:
```
sudo bconsole
```

Run a full recovery:

restore all

Enter 5 to start recovering from the latest backup:

To select the JobIds, you have the following choices:
    1: List last 20 Jobs run
    2: List Jobs where a given File is saved
    3: Enter list of comma separated JobIds to select
    4: Enter SQL list command
    5: Select the most recent backup for a client
    6: Select backup for a client before a specified time
    7: Enter a list of files to restore
    8: Enter a list of files to restore before a specified time
    9: Find the JobIds of the most recent backup for a client
    10: Find the JobIds for a backup for a client before a specified time
    11: Enter a list of directories to restore for found JobIds
    12: Select full restore to a specified Job date
    13: Cancel
Select item:  (1-13): 5

To confirm full recovery, enter done:

You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.

cwd is: /
done

To confirm the recovery startup, enter yes:

OK to run? (yes/mod/no): yes

Check the recovery status:

status director

Here's the result if the recovery is in progress:

Running Jobs:
Console connected at 12-Sep-19 07:25
 JobId Level   Name                       Status
======================================================================
     3         RestoreFiles.2019-09-12_07.27.42_05 is running

Here's the result if the recovery is complete:

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name
====================================================================
     2  Full     32,776    483.6 M  OK       12-Sep-19 07:24 BackupFiles
     3           32,776    1.136 G  OK       12-Sep-19 07:27 RestoreFiles

Wait for the recovery to complete and exit Bacula Console:
```
exit
```

Check the recovered files

Make sure that the /tmp/bacula-restores folder now contains the recovered data:

sudo ls -la /tmp/bacula-restores

Result:

total 16
dr-xr-xr-x. 15 root   root    201 Sep 12 07:09 .
drwx------.  4 bacula bacula   35 Sep 12 07:09 ..
lrwxrwxrwx   1 root   root      7 Sep 12 07:27 bin -> usr/bin
dr-xr-xr-x   5 root   root   4096 Sep 12 07:01 boot
drwxr-xr-x   2 root   root      6 Sep 12 07:22 dev
drwxr-xr-x  79 root   root   8192 Sep 12 07:07 etc
drwxr-xr-x   3 root   root     18 Sep 12 07:01 home
lrwxrwxrwx   1 root   root      7 Sep 12 07:27 lib -> usr/lib
lrwxrwxrwx   1 root   root      9 Sep 12 07:27 lib64 -> usr/lib64
drwxr-xr-x   2 root   root      6 Apr 11  2018 media
drwxr-xr-x   2 root   root      6 Apr 11  2018 mnt
drwxr-xr-x   2 root   root      6 Apr 11  2018 opt
dr-xr-x---   3 root   root    217 Sep 12 07:21 root
drwxr-xr-x   2 root   root      6 Sep 12 07:22 run
lrwxrwxrwx   1 root   root      8 Sep 12 07:27 sbin -> usr/sbin
drwxr-xr-x   2 root   root      6 Apr 11  2018 srv
dr-xr-xr-x   2 root   root      6 Sep 12 07:22 sys
drwxr-xr-x  13 root   root    155 Mar  4  2019 usr
drwxr-xr-x  19 root   root    267 Sep 12 07:01 var

Make sure that the ping utility is in the /tmp/bacula-restores folder:

sudo ls -la /tmp/bacula-restores/bin/ping

Result:

-rwxr-xr-x 1 root root 66176 Aug  4  2017 /tmp/bacula-restores/bin/ping

Copy the ping utility to the main file system:

sudo cp /tmp/bacula-restores/bin/ping /bin/ping

Make sure that ping works:

sudo ping 127.0.0.1 -c 1

Result:

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms

--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms

To free up disk space, delete the copy of the recovered files:
```
sudo rm -rfd /tmp/bacula-restores/*
```

How to delete the resources you created

To stop paying for the resources you created:

Delete the VM.
Delete all objects from the Object Storage bucket:
Delete the bucket Object Storage.
Delete the static public IP if you reserved one.

Backup to Yandex Object Storage via Bacula

Getting startedGetting started

Required paid resourcesRequired paid resources

Create a bucketCreate a bucket

Create a service accountCreate a service account

Create static access keysCreate static access keys

Create a VMCreate a VM

Set up the AWS CLISet up the AWS CLI

Install Bacula and additional componentsInstall Bacula and additional components

Configure MariaDBConfigure MariaDB

Set up the storageSet up the storage

Prepare a backup folderPrepare a backup folder

Mount the bucket to the file systemMount the bucket to the file system

Configure Bacula componentsConfigure Bacula components

Configure Bacula DirectorConfigure Bacula Director

Configure Storage DaemonConfigure Storage Daemon

Create passwords for Bacula componentsCreate passwords for Bacula components

Run the Bacula componentsRun the Bacula components

Create a backupCreate a backup

Check the backupCheck the backup

Recover the filesRecover the files

Check the recovered filesCheck the recovered files

How to delete the resources you createdHow to delete the resources you created

Was the article helpful?

Getting started

Required paid resources

Create a bucket

Create a service account

Create static access keys

Create a VM

Set up the AWS CLI

Install Bacula and additional components

Configure MariaDB

Set up the storage

Prepare a backup folder

Mount the bucket to the file system

Configure Bacula components

Configure Bacula Director

Configure Storage Daemon

Create passwords for Bacula components

Run the Bacula components

Create a backup

Check the backup

Recover the files

Check the recovered files

How to delete the resources you created