Configuring networks for Data Proc
According to the network concept in Yandex Cloud, hosts without public IP addresses in Data Proc clusters can't access resources outside of the VPC virtual network. To interact with nodes from other networks, Yandex Cloud service interfaces, and internet nodes, you need to set up a public IP address for the host or use egress NAT for the subnet.
The ability to set up egress NAT for VPC subnets is at the Preview stage. If this feature isn't available to you for now, the management console lets you fill out a request to access it.
To enable egress NAT for a subnet:
- Log in to management console.
- Select VPC in the proper folder.
- Click on the line of the subnet you need.
- Click in the subnet line and select Enable egress NAT.
Using a NAT instance and static routes
When using a NAT instance, all traffic passes through an additional VM (the NAT instance):
- You can monitor all outbound traffic, and when necessary, you can also deploy a VPN between the Yandex Cloud subnet and the resources you need.
- This creates additional expenses on the NAT instance and the bandwidth limit on a single VM port can become substantial.
To use a NAT configuration, you need two virtual subnets: one subnet hosts the Data Proc clusters and the second hosts the VM with the public IP address.
Examples of subnets:
dataproc-net— Subnet hosting the Data Proc clusters with CIDR
dataproc-nat-net— Subnet for the NAT instance with CIDR
To get access from the
dataproc-net subnet to the external resources:
dataproc-nat-netnetwork, create a virtual machine based on the NAT instance image with a public IP address.
Copy the internal IP address of this VM.
On the subnet network page, create a routing table named
Add a static route to the routing table:
- Destination prefix —
- Next hop — Internal IP address of the NAT instance.
- Destination prefix —
On the network page, link the
natrouting table to the
Enable routing on the NAT instance by adding the following lines to the file
net.ipv4.ip_forward = 1 net.ipv4.conf.all.accept_redirects = 1 net.ipv4.conf.all.send_redirects = 1
To enable execution of
/etc/rc.localon boot, use the commands:
$ sudo systemctl enable rc-local $ sudo touch /etc/rc.local $ sudo chmod 755 /etc/rc.local
Add the following code to the end of
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
Restart the VM:
$ sudo reboot -f
To check whether you configured NAT correctly, run the following command on the NAT instance:
$ curl ifconfig.co
If the configuration is correct, the command outputs the public IP address of the NAT instance.