Configuring the Cilium network policy controller
This scenario shows the implementation of L3/L4 and L7 network policies
To use the Cilium network policy controller in a cluster:
- Install and configure the Hubble networking and security observability platform.
- Create a test environment.
- Create an L3/L4 network policy.
- Create an L7 network policy.
Getting started
-
Create a service account and grant it the roles of
k8s.tunnelClusters.agent
andvpc.publicAdmin
. -
Create a security group and add rules allowing service traffic within the cluster and access to the Kubernetes API.
-
Create a Kubernetes cluster with any suitable configuration.
When creating it, specify the service account and security group prepared in advance. Under Cluster network settings, select Enable tunnel mode.
-
Create a node group of any suitable configuration. When creating it, specify the security group prepared in advance.
-
Configure security groups for the Managed Service for Kubernetes cluster and its node groups.
Warning
The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
-
Install kubectl
and configure it to work with the created cluster.
Install and configure Hubble
-
Create a file named
hubble-ui.yaml
with a specification of resources required for Hubble:hubble-ui.yaml--- # Source: cilium/templates/hubble-ui-serviceaccount.yaml apiVersion: v1 kind: ServiceAccount metadata: name: "hubble-ui" namespace: kube-system --- # Source: cilium/templates/hubble-ui-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: hubble-ui-envoy namespace: kube-system data: envoy.yaml: | static_resources: listeners: - name: listener_hubble_ui address: socket_address: address: 0.0.0.0 port_value: 8081 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: auto stat_prefix: ingress_http route_config: name: local_route virtual_hosts: - name: local_service domains: ["*"] routes: - match: prefix: "/api/" route: cluster: backend prefix_rewrite: "/" timeout: 0s max_stream_duration: grpc_timeout_header_max: 0s - match: prefix: "/" route: cluster: frontend cors: allow_origin_string_match: - prefix: "*" allow_methods: GET, PUT, DELETE, POST, OPTIONS allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout max_age: "1728000" expose_headers: grpc-status,grpc-message http_filters: - name: envoy.filters.http.grpc_web - name: envoy.filters.http.cors - name: envoy.filters.http.router clusters: - name: frontend connect_timeout: 0.25s type: strict_dns lb_policy: round_robin load_assignment: cluster_name: frontend endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 8080 - name: backend connect_timeout: 0.25s type: logical_dns lb_policy: round_robin http2_protocol_options: {} load_assignment: cluster_name: backend endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 8090 --- # Source: cilium/templates/hubble-ui-clusterrole.yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hubble-ui rules: - apiGroups: - networking.k8s.io resources: - networkpolicies verbs: - get - list - watch - apiGroups: - "" resources: - componentstatuses - endpoints - namespaces - nodes - pods - services verbs: - get - list - watch - apiGroups: - apiextensions.k8s.io resources: - customresourcedefinitions verbs: - get - list - watch - apiGroups: - cilium.io resources: - "*" verbs: - get - list - watch --- # Source: cilium/templates/hubble-ui-clusterrolebinding.yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hubble-ui roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: hubble-ui subjects: - kind: ServiceAccount namespace: kube-system name: "hubble-ui" --- # Source: cilium/templates/hubble-ui-service.yaml kind: Service apiVersion: v1 metadata: name: hubble-ui labels: k8s-app: hubble-ui namespace: kube-system spec: selector: k8s-app: hubble-ui ports: - name: http port: 80 targetPort: 8081 type: ClusterIP --- # Source: cilium/templates/hubble-ui-deployment.yaml kind: Deployment apiVersion: apps/v1 metadata: namespace: kube-system labels: k8s-app: hubble-ui name: hubble-ui spec: replicas: 1 selector: matchLabels: k8s-app: hubble-ui template: metadata: annotations: labels: k8s-app: hubble-ui spec: securityContext: runAsUser: 1001 serviceAccount: "hubble-ui" serviceAccountName: "hubble-ui" containers: - name: frontend image: "quay.io/cilium/hubble-ui:v0.7.9@sha256:e0e461c680ccd083ac24fe4f9e19e675422485f04d8720635ec41f2b********" imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: http resources: {} - name: backend image: "quay.io/cilium/hubble-ui-backend:v0.7.9@sha256:632c938ef6ff30e3a080c59b734afb1fb7493689275443faa1435f71********" imagePullPolicy: IfNotPresent env: - name: EVENTS_SERVER_PORT value: "8090" - name: FLOWS_API_ADDR value: "hubble-relay:80" ports: - containerPort: 8090 name: grpc resources: {} - name: proxy image: "docker.io/envoyproxy/envoy:v1.18.2@sha256:e8b37c1d75787dd1e712ff389b0d37337dc8a174a63bed9c34ba7335********" imagePullPolicy: IfNotPresent ports: - containerPort: 8081 name: http resources: {} command: ["envoy"] args: ["-c", "/etc/envoy.yaml", "-l", "info"] volumeMounts: - name: hubble-ui-envoy-yaml mountPath: /etc/envoy.yaml subPath: envoy.yaml volumes: - name: hubble-ui-envoy-yaml configMap: name: hubble-ui-envoy
-
Create resources:
kubectl apply -f hubble-ui.yaml
Command result:
serviceaccount/hubble-ui created configmap/hubble-ui-envoy created clusterrole.rbac.authorization.k8s.io/hubble-ui created clusterrolebinding.rbac.authorization.k8s.io/hubble-ui created service/hubble-ui created deployment.apps/hubble-ui created
-
Check that the Hubble pods' status changes to
Running
. To do this, run the following command:kubectl get pod -A -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase \ | grep hubble | grep -v certs
Command result:
hubble-relay-6b9c774ffc-2jm7t Running hubble-ui-95d74d44c-7jpvl Running
-
Download the Hubble client to the local computer:
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt) && \ curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz" && \ curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz.sha256sum" && \ sha256sum --check hubble-linux-amd64.tar.gz.sha256sum && \ tar zxf hubble-linux-amd64.tar.gz
-
Configure Hubble port forwarding onto the local computer:
kubectl port-forward -n kube-system svc/hubble-ui 12000:80
Note
Port forwarding is stopped if you close the terminal window or abort the command.
-
To open the Hubble UI in a browser, follow this link.
Create a test environment
-
Create a file named
http-sw-app.yaml
with a specification of resources for test applications:http-sw-app.yaml--- apiVersion: v1 kind: Service metadata: name: deathstar spec: type: ClusterIP ports: - port: 80 selector: org: empire class: deathstar --- apiVersion: apps/v1 kind: Deployment metadata: name: deathstar spec: replicas: 2 selector: matchLabels: org: empire class: deathstar template: metadata: labels: org: empire class: deathstar spec: containers: - name: deathstar image: docker.io/cilium/starwars --- apiVersion: v1 kind: Pod metadata: name: tiefighter labels: org: empire class: tiefighter spec: containers: - name: spaceship image: docker.io/tgraf/netperf --- apiVersion: v1 kind: Pod metadata: name: xwing labels: org: alliance class: xwing spec: containers: - name: spaceship image: docker.io/tgraf/netperf
-
Create applications:
kubectl create -f http-sw-app.yaml
Command result:
service/deathstar created deployment.apps/deathstar created pod/tiefighter created pod/xwing created
-
Make sure the pods and services you created are working:
kubectl get pods,svc
Command result:
NAME READY STATUS RESTARTS AGE pod/deathstar-c74d84667-6x4gx 1/1 Running 1 7d pod/deathstar-c74d84667-jrdsp 1/1 Running 0 7d pod/tiefighter 1/1 Running 0 7d pod/xwing 1/1 Running 0 7d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/deathstar ClusterIP 10.4.207.162 <none> 80/TCP 7d service/kubernetes ClusterIP 10.4.0.1 <none> 443/TCP 8d
-
Find out the name of the pod serving the Cilium controller:
kubectl -n kube-system get pods -l k8s-app=cilium
Command result:
NAME READY STATUS RESTARTS AGE cilium-67c4p 1/1 Running 0 42h
In this example, the Cilium controller pod name is
cilium-67c4p
. If there are multiple nodes in the created node group, the Cilium controller pod is run on each of them. In this case, use the name of any of the pods. -
Make sure that network policies are disabled: the
POLICY (ingress) ENFORCEMENT
andPOLICY (egress) ENFORCEMENT
columns are set toDisabled
:kubectl -n kube-system exec <Cilium_controller_pod_name> -- cilium endpoint list
-
Make sure the
xwing
andtiefighter
applications have access to thedeathstar
API and return theShip landed
string, because the network policies are not activated:kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing && \ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Command result:
Ship landed Ship landed
Create an L3/L4 network policy
Apply an L3/L4 network policy to disable the xwing
pod's access to deathstar
. Access rules for the tiefighter
pod remain unchanged.
For access differentiation, the following Kubernetes labels are assigned to pods when creating them:
org: empire
for thetiefighter
pod.org: alliance
for thexwing
pod.
The L3/L4 network policy only allows org: empire
labeled pods to access deathstar
.
-
Create a file named
sw_l3_l4_policy.yaml
with a specification of the policy:sw_l3_l4_policy.yaml--- apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "rule1" spec: description: "L3-L4 policy to restrict deathstar access to empire ships only" endpointSelector: matchLabels: org: empire class: deathstar ingress: - fromEndpoints: - matchLabels: org: empire toPorts: - ports: - port: "80" protocol: TCP
-
Create a policy:
kubectl create -f sw_l3_l4_policy.yaml
-
Make sure that the status of the policy for the
k8s:class=deathstar
object isEnabled
:kubectl -n kube-system exec <Cilium_controller_pod_name> -- cilium endpoint list
-
Check that the
deathstar
service is available for thetiefighter
pod:kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Command result:
Ship landed
-
Make sure the
xwing
pod can't access thedeathstar
service:kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Press Ctrl + C to abort the command. The network policy has denied this pod access to the service.
-
Learn how the policy works:
-
To view the policy specification and status, run the command:
kubectl describe cnp rule1
-
To check if the pods can connect to
deathstar
, open the Hubble UI. To do this, follow the link.
-
Create an L7 network policy
In this part of the scenario, we will change the access policy for the tiefighter
pod:
- Access to the
deathstar.default.svc.cluster.local/v1/exhaust-port
API method will be disabled. - Access to the
deathstar.default.svc.cluster.local/v1/request-landing
API method remains unchanged.
Access for the xwing
pod remains unchanged. This pod can't access deathstar
.
-
Check that the
tiefighter
pod can access thedeathstar.default.svc.cluster.local/v1/exhaust-port
method in the current policy version:kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Command result:
Panic: deathstar exploded goroutine 1 [running]: main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa) /code/src/github.com/empire/deathstar/ temp/main.go:9 +0x64 main.main() /code/src/github.com/empire/deathstar/ temp/main.go:5 +0x85
-
Create a file named
sw_l3_l4_l7_policy.yaml
with a specification of the policy:sw_l3_l4_l7_policy.yaml--- apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "rule1" spec: description: "L7 policy to restrict access to specific HTTP call" endpointSelector: matchLabels: org: empire class: deathstar ingress: - fromEndpoints: - matchLabels: org: empire toPorts: - ports: - port: "80" protocol: TCP rules: http: - method: "POST" path: "/v1/request-landing"
-
Create a new policy:
kubectl apply -f sw_l3_l4_l7_policy.yaml
-
Check that the
tiefighter
pod can access thedeathstar.default.svc.cluster.local/v1/request-landing
method:kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Command result:
Ship landed
-
Make sure that access to the
deathstar.default.svc.cluster.local/v1/exhaust-port
method is disabled for thetiefighter
pod:kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Command result:
Access denied
-
Check that the
xwing
pod can't accessdeathstar
:kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Press Ctrl + C to abort the command.
-
Learn how the policy works:
-
To view the updated policy specification and status, run the command:
kubectl describe cnp rule1
-
To check if the pods can connect to
deathstar
, open the Hubble UI. To do this, follow the link.
-
Delete the resources you created
Delete the resources you no longer need to avoid paying for them:
- Delete the Kubernetes cluster.
- If static public IP addresses were used for cluster and node access, release and delete them.