Creating test VMs using GitLab CI
- Before you start
- 1. Create a VM for the test application
- 2. Prepare a VM with the test application
- 3. Check that the test application is working
- 4. Take a snapshot of the VM disk
- 5. Create a virtual machine with GitLab
- 6. Configure GitLab
- 7. Configure Runner
- 8. Configure CI
- 9. Check how the application works on the VM created using CI
You can use Yandex.Cloud to automate routine actions, such as running a specific script after each commit to the master
branch in a git repository. In the example below, a VM is created and tested after each commit.
To configure CI for VM disk snapshots:
- Create a VM for the test application: create a new virtual machine that will be used to make a disk snapshot for creating new virtual machines via CI.
- Prepare the VM with the test application: install the web server and the set of components for the test application on the VM. Write a test application that will reverse the words in a text sent to the server.
- Check how the application works: send a test request to check the server settings and the results.
- Create a snapshot of the VM disk: create a snapshot of the VM disk that CI will use to create new virtual machines.
- Create a VM with GitLab: create a virtual machine with GitLab, where the repository will store the CI settings and the functional testing script.
- Configure GitLab: create a repository for files and get the necessary configuration parameters.
- Configure Runner: set up the Runner utility for performing tasks.
- Configure CI: set the CI configuration by specifying the necessary parameters for commands and testing.
- Check how the application works on the VM created using CI: make sure that VMs created with CI and the snapshot are created and the test application is running.
Before you start
Before creating a VM:
- Go to the Yandex.Cloud management console and select the folder where you want to perform the operations.
- Make sure the selected folder has a network with a subnet that the VM can be connected to. To do this, select Virtual Private Cloud on the folder page. If the list contains a network, click on its name to see the list of subnets. If there aren't any networks or subnets, create them.
1. Create a VM for the test application
Create a VM where the test application will be installed, the set of components required for it to work, and a web server:
-
On the folder page of the management console, click Create resource and select Virtual machine.
-
In the Name field, enter the VM name:
ci-tutorial-test-app
. -
Select the availability zone to locate the VM in.
-
Select a public image of Ubuntu 18.04.
-
In the Computing resources section, select the following configuration:
- Platform: Intel Cascade Lake.
- Guaranteed vCPU share: 5%.
- vCPU: 2.
- RAM: 1 GB.
-
In the Network settings section, select the subnet to connect the VM to when creating it.
-
Specify data required for accessing the VM.
- Enter the username in the Login field.
- In the SSH key field, copy the contents of the public key file.
You need to create a key pair for SSH connection yourself. To generate keys, use third-party tools, such asssh-keygen
utilities on Linux and macOS or PuTTYgen on Windows.
-
Click Create VM.
Creating the VM may take several minutes. When the VM's status changes to RUNNING
, you can begin configuring it.
When a VM is created, it is assigned an IP address and hostname (FQDN). This data can be used for SSH access.
2. Prepare a VM with the test application
On the created VM, install the set of components required for the test application and a web server for processing requests. The application will be written in Python 2.
-
In the Network section on the VM page of the management console, find the VM's public IP address.
-
Connect to the VM over SSH. You can use the
ssh
tool on Linux and macOS and PuTTY for Windows.$ ssh <Login>@<Public_IP_of_the_VM>
-
Run the
apt update
command to update lists of available packages. -
Install the necessary packages (jq JSON processor, git client, PIP package manager, virtualenv virtual environment management system, the set of header files for Python C API, and the nginx web server):
$ sudo apt-get --yes install jq git python-pip virtualenv python-dev nginx-full
-
Create the folder where the application will be located and make the user you used for connecting to the VM the owner of the folder:
$ sudo mkdir /srv/test-app $ sudo chown -R $USER /srv/test-app
-
Go to the folder and create a virtualenv virtual environment in it:
$ cd /srv/test-app $ virtualenv test-venv
-
Activate the virtual environment:
$ . test-venv/bin/activate
-
Install the Flask framework and the uWSGI web server in the virtual environment:
$ pip install flask uwsgi
-
Deactivate the virtual environment:
$ deactivate
-
Create the
api.py
file in the/srv/test-app
folder:$ touch api.py
-
Open the
api.py
file using any text editor, and insert the Python code that:- Accepts a text string as input in the
text
parameter. - Writes each word from the passed string in the opposite order.
- Returns a response:
- In JSON format, if the client application can accept JSON.
- In plain text if the client application does not accept JSON.
# api.py import json from flask import Flask, request, make_response as response app = Flask(__name__) @app.route("/") def index(): text = request.args.get('text') json_type = 'application/json' json_accepted = json_type in request.headers.get('Accept', '') if text: words = text.split() reversed_words = [word[::-1] for word in words] if json_accepted: res = response(json.dumps({'text': reversed_words}), 200) else: res = response(' '.join(reversed_words), 200) else: res = response('text not found', 501) res.headers['Content-Type'] = json_type if json_accepted else 'text/plain' return res
- Accepts a text string as input in the
-
Create the
wsgi.py
file in the/srv/test-app
folder:$ touch wsgi.py
-
Open the
wsgi.py
file using any text editor and insert the code that runs the test application:# wsgi.py from api import app if __name__ == "__main__": app.run()
-
Create the
test-app.ini
file in the/srv/test-app
folder:$ touch test-app.ini
-
Open the
test-app.ini
file using any text editor and insert the configuration of the uWSGI server:#test-app.ini [uwsgi] module = wsgi:app master = true processes = 1 socket = test-app.sock chmod-socket = 660 vacuum = true die-on-term = true
-
Assign the
www-data
user as the owner of the/srv/test-app
folder and the files it contains:$ sudo chown -R www-data:www-data /srv/test-app
-
Prepare the service to start your uWSGI server. To do this, make the
/etc/systemd/system/test-app.service
file look like this:#/etc/systemd/system/test-app.service [Unit] Description=uWSGI instance to serve test API After=network.target [Service] User=www-data Group=www-data WorkingDirectory=/srv/test-app Environment="PATH=/srv/test-app/test-venv/bin" ExecStart=/srv/test-app/test-venv/bin/uwsgi --ini test-app.ini [Install] WantedBy=multi-user.target
-
Specify the settings of the new virtual server in the nginx configuration by making the
/etc/nginx/sites-available/test-app.conf
file look like this:#/etc/nginx/sites-available/test-app.conf server { #server_name test-app.yandex www.test-app.yandex; listen 80; location /test/ { include uwsgi_params; uwsgi_param SCRIPT_NAME /test; uwsgi_modifier1 30; uwsgi_pass unix:/srv/test-app/test-app.sock; } }
-
Create a symbolic link that points to the nginx configuration file
test-app.conf
:$ sudo ln -s /etc/nginx/sites-available/test-app.conf /etc/nginx/sites-enabled/
-
Delete the symbolic link that points to the default nginx configuration:
$ sudo unlink /etc/nginx/sites-enabled/default
-
Add the service to the system autorun list:
$ sudo systemctl enable test-app.service
3. Check that the test application is working
To make sure the test application is running and the web server is configured correctly, send a simple request.
-
In the browser bar, enter the URL for testing the web server and application:
http://<Public_IP_of_the_VM>/test/?text=hello_world
-
If everything is working, the screen will display text with reversed words from the
text
parameter.
4. Take a snapshot of the VM disk
To easily transfer the created application and web server configuration to VMs created with CI, you need to take a snapshot of the test VM disk.
- In the Yandex.Cloud management console, select the folder where the VM was created.
- Choose Compute Cloud.
- Find the
ci-tutorial-test-app
VM and select it. - Click Stop.
- In the window that opens, click Stop.
- After stopping the VM, select the Disks tab.
- In the disk row, click and select Create snapshot.
- In the window that opens, enter a name for the snapshot:
test-app-snap
. - Click Create snapshot.
5. Create a virtual machine with GitLab
One of the ways to set up CI in Yandex.Cloud is to take advantage of a public image with GitLab pre-installed. GitLab includes a set of tools for managing git repositories and configuring CI.
-
On the folder page of the management console, click Create resource and select Virtual machine.
-
In the Name field, enter the VM name:
ci-tutorial-gitlab
. -
Select the availability zone to locate the VM in.
-
Under Images, click Choose.
-
In the window that opens, open the DevTools tab.
-
Choose the GitLab image.
-
In the Computing resources section, select the following configuration:
- Guaranteed vCPU share: 100%.
- vCPU: 2.
- RAM: 2 GB.
-
In the Network settings section, select the subnet to connect the VM to when creating it.
-
Specify data required for accessing the VM.
- Enter the username in the Login field.
- In the SSH key field, copy the contents of the public key file.
You need to create a key pair for SSH connection yourself. To generate keys, use third-party tools, such asssh-keygen
utilities on Linux and macOS or PuTTYgen on Windows.
-
Click Create VM.
Creating the VM may take several minutes. When the VM's status changes to RUNNING
, you can begin configuring it.
When a VM is created, it is assigned an IP address and hostname (FQDN). This data can be used for SSH access.
6. Configure GitLab
To configure GitLab and prepare CI, you must create a new repository and enter the necessary parameters for authorization in CI.
-
In a browser, open the GitLab admin panel for the created VM. To do this, open a link that looks like
http://<VM-public-IP-address>
. -
Set the administrator password.
-
Log in as
root
with the administrator password. -
Click Create a project.
-
Set the project name:
gitlab-test
. -
Click Create project.
-
Get an OAuth token from Yandex OAuth. To do this, follow the link and click Allow.
-
In the browser, open a link that looks like
http://<VM-public-IP-address>/root
. -
Select the
gitlab-test
project. -
On the screen that opens, choose the Settings tab on the left. In the menu that opens, select CI/CD.
-
Under Variables, click Expand.
-
Create a new variable:
- For the variable name, specify
YC_OAUTH
. - Specify the OAuth token you received as the value of the variable.
- Click Save variables.
- For the variable name, specify
-
Under Runners, click Expand.
-
Under Set up a specific Runner automatically, you will see the server address to connect to and the token for registering the server in the project. Use these values when registering Runner.
7. Configure Runner
Runner is a tool for performing tasks that a user creates. You need to install Runner on the VM and register it in GitLab. In order for Runner to perform tasks, prepare additional components: install the Yandex.Cloud CLI and create a test to check the created VM.
-
Connect to the VM with GitLab over SSH.
$ ssh <Login>@<Public_IP_of_the_GitLab_VM>
-
Add the new repository to the package manager:
$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
-
Install the Runner that will run your CI scripts:
$ sudo apt-get -y install gitlab-runner
-
Register the Runner:
- In the step
Please enter the gitlab-ci coordinator URL
, specify the address of the GitLab server. - In the step
Please enter the gitlab-ci token for this runner
, specify the Runner token. - In the step
Please enter the gitlab-ci description for this runner
, enter the descriptiongitlab test runner
. - In the step
Please enter the gitlab-ci tags for this runner
, don't type anything. Press Enter. - In the step
Please enter the executor
, specifyshell
.
$ sudo gitlab-runner register Runtime platform arch=amd64 os=linux pid=8197 revision=3afdaba6 version=11.5.0 Running in system-mode. Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/): http://<ci-gitlab-IP-address>/ Please enter the gitlab-ci token for this runner: <Runner-token> Please enter the gitlab-ci description for this runner: [ci-tutorial-gitlab]: gitlab test runner Please enter the gitlab-ci tags for this runner (comma separated): Registering runner... succeeded runner=wZqzyy9s Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, kubernetes, docker, docker-ssh, shell, ssh, parallels: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
- In the step
-
Install the Yandex.Cloud CLI so that the CI script can create VMs:
$ curl https://storage.yandexcloud.net/yandexcloud-yc/install.sh --output install.sh $ sudo bash install.sh -n -i /opt/yandex-cloud
-
To perform functional testing, install the
pytest
package:$ sudo apt-get install python-pytest
-
Create the
test.py
file with the functional testing script:-
Open the home page of the
gitlab-test
repository. -
Click + and choose New file.
-
In the window that opens, give the file the name
test.py
. -
Copy the following code to the file body:
# test.py import pytest import json import socket as s @pytest.fixture def hostname(request): with open("instance-creation.out", "r") as fd: fqdn = json.loads(fd.read()).get("fqdn") return fqdn @pytest.fixture def socket(request): _socket = s.socket(s.AF_INET, s.SOCK_STREAM) def socket_teardown(): _socket.close() request.addfinalizer(socket_teardown) return _socket def test_server_connect(socket, hostname): socket.connect((hostname, 80)) assert socket
-
Write any commit message and click Commit changes.
-
8. Configure CI
You need to define the configuration for CI.
-
Open the home page of the
gitlab-test
repository:http://<Public_IP_of_the_GitLab_VM>/root/gitlab-test
-
Click Set up CI/CD. You will see the screen for adding a new file.
-
GitLab automatically gives the file the name
.gitlab-ci.yml
— don't change it. Copy the following configuration to the file:#.gitlab-ci.yml stages: - build - test build: stage: build variables: snapshot_name: test-app-snap folder_id: <folder-id> subnet_name: <subnet-name> script: - export instance_name="ci-tutorial-test-app-$(date +%s)" - export PATH="/opt/yandex-cloud/bin:${PATH}" - yc config set token $YC_OAUTH - yc compute instance create --format json --name $instance_name --folder-id $folder_id --zone ru-central1-c --network-interface subnet-name=$subnet_name,nat-ip-version=ipv4 --create-boot-disk name=$instance_name-boot,type=network-ssd,size=15,snapshot-name=$snapshot_name,auto-delete=true --memory 1 --cores 1 --hostname $instance_name > instance-creation.out - sleep 30 artifacts: when: on_success paths: - instance-creation.out expire_in: 10 mins test_external: stage: test script: - py.test test.py > pytest-external.out artifacts: paths: - pytest-external.out expire_in: 1 hour
-
In the
snapshot_name
field, enter a name for the snapshot of the first VM.
In thefolder_id
field, specify the ID of the folder where the VMs are created.
In thesubnet_name
field, specify the name of the subnet that the VMs will connect to. To find the name in the management console, open the appropriate folder and go to the Virtual Private Cloud page. -
Click Commit changes.
9. Check how the application works on the VM created using CI
After the commit, you need to make sure that CI worked correctly. A new VM should appear in the folder where the test application and web server are deployed.
To check the created VM:
-
Open the Yandex.Cloud management console.
-
In the folder where the VMs were created, open the Compute Cloud service.
-
If everything was configured correctly, the list of VMs should include a new VM with a name like
ci-tutorial-test-app-1543910277
. -
Select the VM you created and copy the public address of the created VM.
-
In the browser, open a link in this format:
http://<public_IP_address_of_the_created_VM>/test/?text=hello_world
-
The application created in the previous steps should also work on the created VM by returning reversed words from the
text
parameter.