Preview
Yandex Managed Service for Apache Airflow^™

A service for managing the Apache Airflow^™ data processing flow orchestrator in Yandex Cloud infrastructure.

The service is offered free of charge and is at the Preview stage.

One-click deployment

Deploy Apache Airflow components in just a few minutes, with product settings already optimized for the selected cluster size.

Secure access

Users are authorized through the Yandex Cloud infrastructure with an IAM role verification step.

Automating data processing

Use familiar tools to process data and prepare results automatically using Yandex Cloud services integrated with Yandex Managed Service for Apache Airflow.

Data Proc integration

Use computing resources more efficiently by automatically creating a temporary Data Proc cluster, which is deleted after data processing is completed.

We handle most of the database maintenance

Processes

Yandex Managed Service for Apache Airflow

Apache Airflow self‑installation

Data access differentiation

Choice of Airflow desktop environment

Deployment of virtual machines

Network setup

OS and software installation

Database update

Data replication configuration*

Data warehouse and equipment security

Integration with Yandex Cloud services

Independent control

Yandex Cloud control

Getting started

Create a Yandex Managed Service for Apache Airflow cluster.

Get started in the Apache Airflow web interface.

Create a cluster

FAQ

How does Apache Airflow differ from other process orchestrators?

Apache Airflow has several features that make it a unique and powerful tool for automating tasks, planning and managing workflows (workflow orchestration). The main features that set Apache Airflow apart are:

Support for a variety of data sources and plugins. Apache Airflow features more than 150 integrations with data storage and processing services, including Yandex Cloud services.
Scalability. Apache Airflow supports the dynamic creation of computing resources to perform tasks, and is capable of adapting to current loads.
Open source and an active community. Apache Airflow is an open-source project. The community of developers and users provides support and continuous updates for the tool.
Dependency identification. Apache Airflow allows you to explicitly define dependencies between tasks, which gives you control over how they are completed. This is useful when orchestrating complex processes.
Monitoring and logging. Apache Airflow provides tools for monitoring and logging tasks. You can easily monitor tasks’ status and progress, and analyze logs to identify errors and improve performance.
Customizability. You can customize Apache Airflow to suit your needs by creating your own operators and expanding functionality with custom plugins.