Automating image builds using Jenkins and Packer

This scenario describes how to set up automated VM image builds using Jenkins and Packer.
Jenkins lets you to set up the continuous delivery of image configuration changes to Packer to build VM image used in Yandex.Cloud.

The images built can be used to create a cloud infrastructure. In this scenario, Terraform is used to deploy virtual machines from the created images.

To install and configure Jenkins, Packer, GitHub, and Terraform to interact with one another:

1. Create a service account
2. Create a VM with Jenkins
3. Install Packer
4. Configure Jenkins
5. Set up a Jenkins job
6. Configure a GitHub repository
7. Create an image using Jenkins
8. Deploy the images using Terraform

If you no longer need the created VM and images, delete them.

Before you start

Before deploying your applications, sign up for Yandex.Cloud and create a billing account:

1. Go to the management console. Then log in to Yandex.Cloud or sign up if don't already have an account.
2. On the billing page, make sure you linked a billing account, and it has the ACTIVE or TRIAL_ACTIVE status. If you don't have a billing account, create one.

If you have an active billing account, you can create or select a folder to run your VM in from the Yandex.Cloud page.

Learn more about clouds and folders.

• Install the Yandex.Cloud command line interface.
• Install Terraform and configure it to work with Yandex.Cloud.
• Configure the Git client. If you are running Windows, use Git Bash.
• Create a repository branch with examples in your GitHub account.
• Prepare an SSH key to access the virtual machines.
• Create a cloud network and subnet in the selected availability zone.

Required paid resources

The cost of this infrastructure includes:

• A fee for continuously running VMs (see pricingYandex Compute Cloud).
• A fee for storing created images (see Yandex Compute Cloud pricing).
• A fee for using dynamic public IP addresses (see Yandex Virtual Private Cloud pricing).

Create a service account

Jenkins uses service accounts to perform actions in your cloud and folder. To create a service account:

1. Get the IDs of the folder and cloud by running yc config list.

2. Create a service account and pass its ID to the environment variable using the following commands:

   $ yc iam service-account create --name <username>
   $ yc iam key create --service-account-name <username> -o <username.json>
   $ SERVICE_ACCOUNT_ID=$(yc iam service-account get --name <username> --format json | jq -r .id)
   

3. Assign the service account the admin role for the folder where operations will be performed:

   $ yc resource-manager folder add-access-binding <folder_name> --role admin --subject serviceAccount:$SERVICE_ACCOUNT_ID
   

Create a VM with Jenkins

Jenkins will get VM image configuration changes from GitHub and then use Packer to create images in the cloud.

To create a virtual machine with Jenkins:

1. On the folder page in the management console, click Create resource and select Virtual machine.

2. In the Name field, enter a name for the VM: jenkins-tutorial.

3. Select the availability zone to host the VM in.

4. Under Images from Cloud Marketplace, click Select. In the window that opens, select the Jenkins image.

5. Under Disks, enter 15 GB for the size of the boot disk:

6. Under Computing resources:

   • Choose a platform: Intel Cascade Lake.
   • Specify the number of vCPUs and amount of RAM:
     • vCPU: 2.
     • Guaranteed vCPU share: 5%.
     • RAM: 2 GB.

7. Under Network settings, click Add network and choose the subnet to connect the virtual machine to. Under Public IP, assign a public address to the VM automatically or select a reserved address.

8. Under Access, specify the data required to access the VM:

   • Enter the username in the Login field.
   • In the SSH key field, paste the contents of the public key file.
     You need to create a key pair for SSH connections on your own.

9. Click Create VM.

Install Packer

Packer lets you create virtual machine images with parameters specified in the configuration file.

1. Download the Packer distribution for Linux.

2. Upload Packer to the VM you created:

   $ scp packer_<Packer version>_linux_amd64.zip <Login>@<Public IP address of your VM>:~/
   

3. Connect to the VM over SSH. You can use the ssh utility in Linux or macOS, or PuTTY in Windows.

4. Create a new folder, move the Packer executables there, and unpack the archive:

   $ sudo mkdir /opt/yandex-packer/
   $ sudo mv packer_<Packer version>_linux_amd64.zip /opt/yandex-packer/
   $ unzip packer_<Packer version>_linux_amd64.zip
   

5. All Jenkins system activity is performed on behalf of the jenkins user. Grant this user rights to launch Packer:

   $ sudo chmod u+x /opt/yandex-packer/packer*
   $ sudo chown jenkins:jenkins /opt/yandex-packer/packer*
   

Configure Jenkins

To build images based on configurations from GitHub, configure Jenkins:

1. Connect to the VM over SSH. You can use the ssh utility in Linux or macOS, or PuTTY in Windows.

2. Open the password file needed to launch the configuration and copy the password:

   $ sudo cat /var/lib/jenkins/secrets/initialAdminPassword
   

3. Open http://<public IP address of VM with Jenkins> in the browser. Open the Jenkins management console.

4. Enter the copied password in the Administrator password field and click Continue.

5. Choose Select plugins to install.

6. You will need the following plugins to run this scenario:

   • Pipeline — The plugin for getting the source code from the VCS and assembling, testing, and deploying it.
   • Git — The plugin for working with git repositories.
   • Credentials Binding — The plugin for creating environment variables with authentication data.

7. Click Install. The selected components start to install.

8. When the installation is complete, you are prompted to create an administrator account. Fill in the form fields and click Save and Continue.

9. You are prompted to create a URL for Jenkins. Leave the URL as http://<VM public IP address>/. Click Save and finish.

10. Click Start using Jenkins to complete the installation and go to the Jenkins administration panel.

Set up a Jenkins task

Enter your Yandex.Cloud authorization data and create a task to download changes from the github repository so that Jenkins can build images. The authorization data is used in the variables stored in the Packer configuration files.

1. Open the Jenkins administration panel.
2. Select Credentials.
3. Under Stores scoped to Jenkins, click the Global link.
4. Get the ID of the subnet where images will be built by running the yc vpc subnet list command.
5. Click Add credentials. Specify the following parameters:
   1. In the Kind list, click Secret text.
   2. In the Scope list, leave Global.
   3. In the Secret field, enter the ID of your folder.
   4. In the Id field, enter YC_FOLDER_ID. Click OK.
6. Create another secret with the following parameters:
   1. Kind: Secret text.
   2. Scope: Global.
   3. Secret: ID of the subnet where the Jenkins VM is hosted.
   4. ID: YC_SUBNET_ID.
7. Create another secret with the following parameters:
   1. Kind: Secret file.
   2. Scope: Global.
   3. File: The <username>.json file from step 1.
   4. ID: YC_ACCOUNT_KEY_FILE.
8. Go back to the main page of the administration panel and select New item.
9. Enter a name for the task, like jenkins-tutorial, and select Pipeline as the task type. Click OK.
10. In the window that opens, select the GitHub hook trigger for GITScm polling checkbox. This option lets you run the build every time you push to the master branch of your git repository.
11. Under Pipeline, select Pipeline script from SCM from the Definition list.
12. In the SCM list, select Git.
13. In the Report URL field, enter the URL of your GitHub fork.
14. In the Script path field, enter jenkins-packer/Jenkinsfile.
15. Leave the other fields unchanged and click Save.

Configure the GitHub repository

In the GitHub repository settings, set up a webhook to run builds in Jenkins and add the public SSH key for authorization.

Set up a webhook

1. Open the fork of your GitHub repository in the browser.
2. Click the Settings tab.
3. Select Webhooks and click Add webhook.
4. In the Payload URL field, enter http://<VM public IP address>/github-webhook/.
5. Click Add webhook.

Add an SSH key to GitHub

1. Click your GitHub avatar. In the menu that opens, select Settings.
2. Click SSH and GPG keys.
3. Click New SSH key.
4. In the Title field, enter a name for your key.
5. Paste your SSH key into the Key box.
6. Click Add SSH key.

Create an image using Jenkins

The image is built in Jenkins automatically after you push to the master branch of your GitHub repository.

1. Clone the examples repository fork you created while getting started to your computer:

   $ git clone git@github.com:<GitHub login>/examples.git
   

2. Make changes to the Packer templates hosted in the jenkins-packer/packer/ folder. You can find the Packer template documentation on the developer's website. In the image_family and source_image_family parameters, specify the families of images to be built by Jenkins. For more information about families, see Image families.

3. Make changes to the Jenkinsfile Pipeline description file for Jenkins located in the root of the repository. For the Pipeline documentation, see the developer's website.

4. Upload the changes to Github:

   $ git add -A
   $ git commit -m "Build update"
   $ git push
   

5. Open the Jenkins administration panel and check the task status.

6. If all the settings are correct, the image assembly is launched. You can see the result in the build logs.

Then there are three new images in Compute Cloud under Images:

• Debian: The basic image with the latest updates.
• nginx: An image with the nginx web server, based on the Debian image.
• Django: An image with the Django framework, based on the Debian image.

Deploy the images

Once the images are created, you can use them to create your virtual machines. Create a test infrastructure using Terraform:

1. In the folder with the fork, go to the directory with the Terraform files:

   $ cd examples/jenkins-packer/terraform
   

2. Rename the terraform.tfvars_example file:

   mv terraform.tfvars_example terraform.tfvars
   

3. Fill in the file fields with the applicable values. Read the Terraform documentation on the developer's website.

4. Initialize the Terraform provider by running terraform init.

5. Execute terraform plan -var-file="terraform.tfvars". Check the configuration created.

6. Execute terraform apply and confirm that you want to create the infrastructure by entering yes in the terminal.

This will create:

1. A cloud network.
2. Subnets in all availability zones.
3. Virtual machines from images created by Packer. Virtual machines with nginx get public IP addresses. All virtual machines are connected to subnets.

Delete the created resources

If you no longer need the VMs and images:

• Delete the created VMs.
• Delete the created images.

To delete the resources created with Terraform, run terraform destroy.