Backup to Yandex Object Storage via Bacula on Centos 7
- Before you start
- Creating VMs
- Create a bucket
- Create a service account
- Create static access keys
- Set up the AWS CLI
- Install Bacula and additional components
- Configure MariaDB
- Configure backup storage
- Configure Bacula components
- Create a backup
- Check the backup
- Run the recovery process
- Check the recovery
- How to delete created resources
In these instructions, you'll learn how to create backups in Yandex Object Storage and recover data via Bacula on CentOS 7 in the Yandex.Cloud infrastructure.
Bacula consists of several components:
- Bacula Director: Controls the backup and recovery process.
- File Daemon: Provides access to backup files.
- Storage Daemon: Reads and writes files to the hard disk.
- Catalog: Maintains the file catalog used for backups. The catalog is stored in an SQL database.
- Bacula Console: A management console for interacting with Bacula Director.
To back up and recover your data via Bacula:
- Before you start.
- Create a VM.
- Install Bacula and the additional components.
- Configure MariaDB.
- Configure backup storage.
- Configure the Bacula components.
- Create a backup.
- Check the backup.
- Run the recovery process.
- Check the recovered data.
If you no longer need the backup or recovered data, delete all related resources.
Before you start
- Go to the management console. Then log in to Yandex.Cloud or sign up if don't already have an account.
- On the billing page, make sure you linked a billing account, and it has the
ACTIVE
orTRIAL_ACTIVE
status. If you don't have a billing account, create one.
If you have an active billing account, you can create or select a folder to run your VM in from the Yandex.Cloud page.
Learn more about clouds and folders.
Required paid resources
The cost for backup and recovery includes:
- A fee for a continuously running VM (see Yandex Compute Cloud pricing).
- A fee for data storage (see Yandex Object Storage pricing).
- A fee for using a dynamic or static external IP address (see Yandex Virtual Private Cloud pricing).
Creating VMs
To create a VM:
-
On the folder page in the management console, click Create resource and select Virtual machine.
-
In the Name field, enter a name for the VM:
bacula-vm
. -
Select the availability zone to host the VM in.
-
Under Images from Cloud Marketplace, click Select. Select a public CentOS 7 image.
-
Under Computing resources, select:
- Platform: Intel Cascade Lake.
- Guaranteed vCPU share: 20%.
- vCPU: 2.
- RAM: 2 GB.
-
In the Network settings section, select the network and subnet to connect the VM to. If you don't have a network or subnet, create them right on the VM creation page.
-
In the Public address field, leave the Auto value to assign a random external IP address from the Yandex.Cloud pool, or select a static address from the list if you reserved one in advance.
-
Specify data required for accessing the VM:
-
Enter the username in the Login field.
-
In the SSH key field, paste the contents of the public key file.
You need to create a key pair for the SSH connection yourself. Learn how to connect to VMs via SSH.
Alert
The IP address and host name (FQDN) to connect to the VM are assigned on VM creation. If you selected No address in the Public address field, you won't be able to access the VM from the internet.
-
-
Click Create VM.
Creating the VM may take several minutes.
When the VM is created, it is assigned a public IP address and hostname (FQDN). This data can be used for SSH access.
Create a bucket
To create a bucket for backups in Object Storage:
-
Go to the Yandex.Cloud management console and select the folder where you want to perform the operations.
-
On the folder page, click Create resource and select Bucket.
-
In the Name field, enter a name for the bucket:
bacula-bucket
. -
In the Bucket access field, select Restricted.
-
In the Storage class field, select Cold.
-
Click Create bucket.
Create a service account
Create a service account and assign it the role editor
.
Create static access keys
Create static access keys.
Make sure to immediately save the key_id
ID and secret
secret key. You will not be able to get the key value again.
Set up the AWS CLI
After bacula-vm
switches to RUNNING
:
-
You can learn the IP address of the VM under Network on the VM page in the management console.
-
Connect to the VM over SSH. You can use the
ssh
tool in Linux and macOS or PuTTY for Windows.The recommended authentication method when connecting over SSH is using a key pair. Don't forget to set up the created key pair: the private key must match the public key sent to the VM.
-
Install the yum repository:
$ sudo yum install epel-release -y
-
Install
pip
:$ sudo yum install python-pip -y
-
Install the AWS CLI:
$ sudo pip install awscli --upgrade
-
Set up the AWS CLI:
$ sudo aws configure
The command will request values for parameters:
AWS Access Key ID
: Enter thekey_id
ID that you received when generating the static key.AWS Secret Access Key
: Enter thesecret
key that you received when creating the static key.Default region name
:ru-central1
.Default output format
:json
.
-
Check that the
/root/.aws/credentials
file contains the correctkey_id
andsecret
values:$ sudo cat /root/.aws/credentials
-
Check that the
/root/.aws/config
file contains the correctDefault region name
andDefault output format
values:$ sudo cat /root/.aws/config
Install Bacula and additional components
-
Install the Bacula components:
$ sudo yum install -y bacula-director bacula-storage bacula-console bacula-client
-
Install MariaDB:
$ sudo yum install -y mariadb-server
-
Install
s3fs
to mount the Object Storage bucket on the server:$ sudo yum install -y s3fs-fuse
-
Install the text editor
nano
:$ sudo yum install -y nano
Configure MariaDB
-
Run MariaDB:
$ sudo systemctl start mariadb
-
Check that MariaDB is running:
$ sudo systemctl status mariadb | grep Active
-
Enable MariaDB to run at system startup:
$ sudo systemctl enable mariadb
-
Create database tables and configure access rights:
$ /usr/libexec/bacula/grant_mysql_privileges $ /usr/libexec/bacula/create_mysql_database -u root $ /usr/libexec/bacula/make_mysql_tables -u bacula
-
Configure the database security parameters:
$ sudo mysql_secure_installation
Enter current password for root (enter for none)
: Press Enter to skip the field.Set root password? [Y/n]
: EnterY
, set the root password, and confirm it. You will need the password in the next step.Remove anonymous users? [Y/n]
: Press Enter to accept the default value.Disallow root login remotely? [Y/n]
: Press Enter to accept the default value.Remove test database and access to it? [Y/n]
: Press Enter to accept the default value.Reload privilege tables now? [Y/n]
: Press Enter to accept the default value.
-
Log in to the DB command line and enter the
root
password created in the previous step:$ mysql -u root -p
-
Create the password
bacula_db_password
for thebacula
user:UPDATE mysql.user SET Password=PASSWORD('bacula_db_password') WHERE User='bacula'; FLUSH PRIVILEGES; exit
-
Enable the MySQL library for Bacula:
$ sudo alternatives --config libbaccats.so
Enter
1
to select MySQL:Selection Command ----------------------------------------------- 1 /usr/lib64/libbaccats-mysql.so 2 /usr/lib64/libbaccats-sqlite3.so *+ 3 /usr/lib64/libbaccats-postgresql.so Enter to keep the current selection[+], or type selection number: 1
Configure backup storage
Prepare a backup folder
-
Create the
/tmp/bacula
backup folder:$ sudo mkdir /tmp/bacula
-
Configure access rights to
/tmp/bacula
:$ sudo chown -R bacula:bacula /tmp/bacula $ sudo chmod -R 700 /tmp/bacula $ sudo semanage permissive -a bacula_t
Mount the bucket to the file system
Mount the bucket to the file system to upload copied files to Object Storage:
-
Mount the bucket using
s3fs
:$ sudo s3fs bacula-bucket /tmp/bacula -o url=https://storage.yandexcloud.net -o use_path_request_style -o allow_other -o nonempty -o uid=133,gid=133,mp_umask=077
bacula-bucket
: The name of the bucket in Object Storage.uid=133
: The ID of thebacula
user from the/etc/passwd
file.gid=133
: The ID of thebacula
group from the/etc/passwd
file.
-
Check the access rights for the
/tmp/bacula
folder:$ sudo ls -la /tmp/bacula/
The command output should look as follows:
drwx------. 2 bacula bacula 31 Sep 18 09:16 . drwxrwxrwt. 10 root root 265 Sep 18 08:59 ..
-
Check that the
bacula
user can create files in the/tmp/bacula
folder:-
Temporarily enable the
bash
shell for thebacula
user:$ sudo sed -i "/^bacula/ s@/sbin/nologin@/bin/bash@" /etc/passwd
-
Create an arbitrary file in the
/tmp/bacula
folder:$ sudo runuser -l bacula -c 'touch /tmp/bacula/test.test'
-
Make sure that the file
test.test
was created in the/tmp/bacula
folder:$ sudo ls -la /tmp/bacula | grep test.test
-
On the catalog page in the management console, open Object Storage. Make sure the file
test.test
is visible in thebacula-bucket
bucket. -
Delete the test file:
$ sudo runuser -l bacula -c 'rm -f /tmp/bacula/test.test'
-
Disable the
bash
shell for thebacula
user:$ sudo sed -i "/^bacula/ s@/bin/bash@/sbin/nologin@" /etc/passwd
-
Configure Bacula components
Configure Bacula Director
-
Open the Bacula Director configuration file:
$ sudo nano /etc/bacula/bacula-dir.conf
-
In the
Director
configuration block, add the lineDirAddress = 127.0.0.1
to set up a connection with Bacula Director:Director { # define myself Name = bacula-dir DIRport = 9101 # where we listen for UA connections QueryFile = "/etc/bacula/query.sql" WorkingDirectory = "/var/spool/bacula" PidDirectory = "/var/run" Maximum Concurrent Jobs = 1 Password = "@@DIR_PASSWORD@@" # Console password Messages = Daemon DirAddress = 127.0.0.1 }
-
For your convenience, rename the task
BackupClient1
toBackupFiles
:Job { Name = "BackupFiles" JobDefs = "DefaultJob" }
-
In the
RestoreFiles
task configuration, add the lineWhere = /tmp/bacula-restores
to make/tmp/bacula-restores
a folder for restored files:Job { Name = "RestoreFiles" Type = Restore Client=bacula-fd FileSet="Full Set" Storage = File Pool = Default Messages = Standard Where = /tmp/bacula-restores }
-
Under the
FileSet
configuration section namedFull Set
underInclude
:- Add the
compression = GZIP
line to theOptions
section to enable compression during backup. - Specify
File = /
to back up the entire file system.
FileSet { Name = "Full Set" Include { Options { signature = MD5 compression = GZIP } File = / } Exclude { File = /var/spool/bacula File = /tmp File = /proc File = /tmp File = /.journal File = /.fsck } }
- Add the
-
You can find the internal IP address of the VM under Network on the VM page in the management console.
-
In the
Storage
configuration block, enter the internal IP address of the VM in theAddress
field to set up a connection with Storage Daemon:Storage { Name = File # Do not use "localhost" here Address = <VM internal IP address> # N.B. Use a fully qualified name here SDPort = 9103 Password = "@@SD_PASSWORD@@" Device = FileStorage Media Type = File }
-
To connect to the DB, in the
Catalog
configuration block, enter the database passworddbpassword = "bacula_db_password"
created when configuring MariaDB:# Generic catalog service Catalog { Name = MyCatalog # Uncomment the following line if you want the dbi driver # dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport = dbname = "bacula"; dbuser = "bacula"; dbpassword = "bacula_db_password" }
-
Save the file.
-
Check that the
bacula-dir.conf
file doesn't contain any syntax errors:$ sudo bacula-dir -tc /etc/bacula/bacula-dir.conf
If there aren't any error messages, the configuration is correct.
Configure Storage Daemon
-
Open the Storage Daemon configuration file:
$ sudo nano /etc/bacula/bacula-sd.conf
-
You can find the internal IP address of the VM under Network on the VM page in the management console.
-
In the
Storage
configuration block, enter the internal IP address of the VM in theSDAddress
field to set up a connection with Storage Daemon:Storage { # definition of myself Name = BackupServer-sd SDPort = 9103 # Director's port WorkingDirectory = "/var/spool/bacula" Pid Directory = "/var/run/bacula" Maximum Concurrent Jobs = 20 SDAddress = <VM internal IP address> }
-
In the
Device
configuration block, specify theArchive Device = /tmp/bacula
folder for backups:Device { Name = FileStorage Media Type = File Archive Device = /tmp/bacula LabelMedia = yes; # lets Bacula label unlabeled media Random Access = Yes; AutomaticMount = yes; # when device opened, read it RemovableMedia = no; AlwaysOpen = no; }
-
Save the file.
-
Check that the
bacula-sd.conf
file doesn't contain any syntax errors:$ sudo bacula-sd -tc /etc/bacula/bacula-sd.conf
If there aren't any error messages, the configuration is correct.
Create passwords for Bacula components
Bacula Director, Storage Daemon, and File Daemon use passwords for inter-component authentication.
To set passwords for Bacula components:
-
Generate passwords for Bacula Director, Storage Daemon, and File Daemon:
$ DIR_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` $ SD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33` $ FD_PASSWORD=`date +%s | sha256sum | base64 | head -c 33`
-
Put the passwords in the configuration files:
$ sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bacula-dir.conf $ sudo sed -i "s/@@DIR_PASSWORD@@/${DIR_PASSWORD}/" /etc/bacula/bconsole.conf $ sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-sd.conf $ sudo sed -i "s/@@SD_PASSWORD@@/${SD_PASSWORD}/" /etc/bacula/bacula-dir.conf $ sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-dir.conf $ sudo sed -i "s/@@FD_PASSWORD@@/${FD_PASSWORD}/" /etc/bacula/bacula-fd.conf
Run the Bacula components
-
Run the Bacula components:
$ sudo systemctl start bacula-dir $ sudo systemctl start bacula-sd $ sudo systemctl start bacula-fd
-
Check that the Bacula components are running:
$ sudo systemctl status bacula-dir $ sudo systemctl status bacula-sd $ sudo systemctl status bacula-fd
-
Set up the Bacula components to launch at system startup:
$ sudo systemctl enable bacula-dir $ sudo systemctl enable bacula-sd $ sudo systemctl enable bacula-fd
Create a backup
-
Open Bacula Console:
$ sudo bconsole
-
Create a label to set up a backup profile:
label
-
Name the new volume
MyVolume
:Enter new Volume name: MyVolume
-
Enter
2
to select theFile
pool:Defined Pools: 1: Default 2: File 3: Scratch Select the Pool (1-3): 2
-
Run the backup process:
run
Select
1
to start theBackupFiles
task:A job name must be specified. The defined Job resources are: 1: BackupFiles 2: BackupCatalog 3: RestoreFiles Select Job resource (1-3): 1
Enter
yes
to confirm the backup process launch:OK to run? (yes/mod/no): yes
-
Check the backup status:
status director
Command output if the backup is running:
Running Jobs: Console connected at 12-Sep-19 07:22 JobId Level Name Status ====================================================================== 2 Full BackupFiles.2019-09-12_07.22.56_03 is running
Command output if the backup is complete:
Running Jobs: Console connected at 12-Sep-19 07:25 No Jobs running. ==== Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 2 Full 32,776 483.6 M OK 12-Sep-19 07:24 BackupFiles
-
Wait for the backup to complete and exit Bacula Console:
exit
Check the backup
To make sure that the backup is complete:
- On the catalog page in the management console, open Object Storage.
- Open the
bacula-bucket
bucket. - Make sure it contains
MyVolume
.
Run the recovery process
-
Delete an arbitrary file, like the
ping
utility, to check the recovery:$ sudo rm -f /bin/ping
-
Make sure that the
ping
utility is deleted:$ ping
The command output should look as follows:
bash: ping: command not found
-
Log in to Bacula Console:
$ sudo bconsole
-
Run a full recovery:
restore all
Select
5
to start a recovery from the latest backup:To select the JobIds, you have the following choices: 1: List last 20 Jobs run 2: List Jobs where a given File is saved 3: Enter list of comma separated JobIds to select 4: Enter SQL list command 5: Select the most recent backup for a client 6: Select backup for a client before a specified time 7: Enter a list of files to restore 8: Enter a list of files to restore before a specified time 9: Find the JobIds of the most recent backup for a client 10: Find the JobIds for a backup for a client before a specified time 11: Enter a list of directories to restore for found JobIds 12: Select full restore to a specified Job date 13: Cancel Select item: (1-13): 5
Enter
done
to confirm the full recovery:You are now entering file selection mode where you add (mark) and remove (unmark) files to be restored. No files are initially added, unless you used the "all" keyword on the command line. Enter "done" to leave this mode. cwd is: / $ done
Enter
yes
to confirm the recovery process launch:OK to run? (yes/mod/no): yes
-
Check the recovery status:
status director
Command output if the recovery is running:
Running Jobs: Console connected at 12-Sep-19 07:25 JobId Level Name Status ====================================================================== 3 RestoreFiles.2019-09-12_07.27.42_05 is running
Command output if the recovery is complete:
Terminated Jobs: JobId Level Files Bytes Status Finished Name ==================================================================== 2 Full 32,776 483.6 M OK 12-Sep-19 07:24 BackupFiles 3 32,776 1.136 G OK 12-Sep-19 07:27 RestoreFiles
-
Wait for the recovery to complete and exit Bacula Console:
exit
Check the recovery
-
Make sure that the
/tmp/bacula-restores
folder now contains the recovered data:$ sudo ls -la /tmp/bacula-restores
The command output should look as follows:
total 16 dr-xr-xr-x. 15 root root 201 Sep 12 07:09 . drwx------. 4 bacula bacula 35 Sep 12 07:09 .. lrwxrwxrwx 1 root root 7 Sep 12 07:27 bin -> usr/bin dr-xr-xr-x 5 root root 4096 Sep 12 07:01 boot drwxr-xr-x 2 root root 6 Sep 12 07:22 dev drwxr-xr-x 79 root root 8192 Sep 12 07:07 etc drwxr-xr-x 3 root root 18 Sep 12 07:01 home lrwxrwxrwx 1 root root 7 Sep 12 07:27 lib -> usr/lib lrwxrwxrwx 1 root root 9 Sep 12 07:27 lib64 -> usr/lib64 drwxr-xr-x 2 root root 6 Apr 11 2018 media drwxr-xr-x 2 root root 6 Apr 11 2018 mnt drwxr-xr-x 2 root root 6 Apr 11 2018 opt dr-xr-x--- 3 root root 217 Sep 12 07:21 root drwxr-xr-x 2 root root 6 Sep 12 07:22 run lrwxrwxrwx 1 root root 8 Sep 12 07:27 sbin -> usr/sbin drwxr-xr-x 2 root root 6 Apr 11 2018 srv dr-xr-xr-x 2 root root 6 Sep 12 07:22 sys drwxr-xr-x 13 root root 155 Mar 4 2019 usr drwxr-xr-x 19 root root 267 Sep 12 07:01 var
-
Make sure that the
ping
utility is in the/tmp/bacula-restores
folder:$ sudo ls -la /tmp/bacula-restores/bin/ping
The command output should look as follows:
-rwxr-xr-x 1 root root 66176 Aug 4 2017 /tmp/bacula-restores/bin/ping
-
Copy the
ping
utility to the main file system:$ sudo cp /tmp/bacula-restores/bin/ping /bin/ping
-
Make sure that
ping
works:$ sudo ping 127.0.0.1 -c 1
The command output should look as follows:
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data. 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.016 ms --- 127.0.0.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms
-
Delete a copy of recovered files to free up disk space:
$ sudo rm -rfd /tmp/bacula-restores/*
How to delete created resources
To stop paying for data storage, delete the VM and Object Storage bucket.
If you reserved a static public IP address specifically for this VM:
- Open the Virtual Private Cloud in your folder.
- Go to the IP addresses tab.
- Find the address you need, click , and select Delete.