Updates to the Kubernetes Container Image Registry

Introduction

Kubernetes 1.25 marked the announcement of a significant transition for the official container image registry endpoint, moving from k8s.gcr.io to the community-managed registry, registry.k8s.io. This shift aims to phase out the old registry gradually. However, as stated on the Kubernetes website, this transition has been accelerated to adopt a more sustainable infrastructure model, with the initial changes effective March 20, 2023. This article outlines the changes, their rationale, important deadlines, and necessary actions you should take.

What Changes Are Taking Place?

Starting March 20, 2023, all traffic from the k8s.gcr.io endpoint will redirect to registry.k8s.io. Following this, on April 3, 2023, the old registry will be frozen, disabling any new image uploads to k8s.gcr.io. This change affects all images in the k8s.gcr.io registry, including sub-projects such as Kubernetes DNS (dns/k8s-dns-node-cache) and Ingress NGINX Controller (ingress-nginx/controller). The new registry will distribute traffic across various regions and cloud providers, ensuring clients retrieving images from registry.k8s.io are securely redirected to the nearest storage service, like Amazon Simple Storage Service (Amazon S3).

Why Is This Change Necessary?

Google has been the host for Kubernetes’ official container image registry (k8s.gcr.io) since its inception. However, as the project expanded, this arrangement became less viable for the Cloud Native Computing Foundation (CNCF) due to the egress fees incurred when pulling images from alternate cloud providers. The Kubernetes community recognized the need for a more scalable long-term solution by utilizing infrastructure from other cloud providers to host image layers and repositories. The objective of this distributed cloud infrastructure model is to enhance user experience and speed, allowing end users to benefit from closer servers and infrastructure from providers like AWS.

Implementing these changes now will help reduce network traffic costs for end users associated with image pulls from the legacy registry. Additionally, it will increase the project’s cost-efficiency, enabling the Kubernetes team to lower egress bandwidth and storage expenses linked to serving a distributed user base. This transition also facilitates better resource utilization, particularly regarding the AWS donation announced at KubeCon NA 2022 in Detroit.

When Will These Changes Occur?

Traffic Redirection – March 20, 2023
From this date, all traffic directed to the legacy k8s.gcr.io registry will be rerouted to the new registry.k8s.io endpoint.
Registry Freeze – April 3, 2023
On this date, the k8s.gcr.io registry will be frozen, preventing new images from being uploaded. While the registry will still allow image pulls to support user migration, long-term guarantees about its availability cannot be assured. Organizations should transition to registry.k8s.io as soon as possible, even if they are not currently affected.

What Actions Should You Take?

Identify Images from k8s.gcr.io
Start by locating all container images used by Pods in your cluster that depend on the old registry. Here are some strategies to do this:
- Using OPA Gatekeeper or Kyverno: If you have these policy admission controllers in your cluster, you can utilize them to identify images pulled from k8s.gcr.io and prevent future pulls from the old registry. Check out these best practices guides for Amazon Elastic Kubernetes Service (Amazon EKS) for examples.
- Using the kubectl community-images plugin: This CLI tool shows container images running in your cluster that were sourced from community repositories. It can be useful for scanning and notifying users about repository changes. You can find an example on how to use this plugin.
- Using kubectl to inspect resources: Run a kubectl command to filter and list the images of resources in your cluster that rely on the old registry:
```
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' 'n' | sort | uniq -c | grep "k8s.gcr.io"
```
  This command specifically identifies Pods with images from the old registry. If using this method, you will need to repeat it for other resources like DaemonSets or Jobs.
Update Manifests
After detecting old dependencies, update the relevant Helm charts and manifests to point to the new registry endpoint, registry.k8s.io.
Review IP Address Restriction Policies
If you have strict access policies limiting image pulls to k8s.gcr.io, revise these to align with the changes. Starting March 20, clusters with these restrictions will be unable to pull images due to the redirect. Customers running Kubernetes in restricted environments should carefully evaluate their workload registry dependencies to avoid unexpected issues post-transition.
Copy Images to a Private Registry
If you operate your own image registry, transfer the necessary images to your self-hosted repositories using tools like crane. Customers with hosted private registries, such as Amazon Elastic Container Registry (Amazon ECR), can also migrate images from public repositories to their private ones. If you currently mirror images from k8s.gcr.io to a private registry, update this to pull from the new public registry, registry.k8s.io.

How Will AWS Customers Be Affected?

It is highly advisable for customers using Amazon EKS and self-managed Kubernetes clusters on AWS to identify and address necessary changes for image dependencies that were previously hosted in the old registry. Numerous workloads, operators, and sub-projects utilize images stored in the legacy registry, impacting both Amazon EKS and self-managed Kubernetes customers. Therefore, it is crucial to perform the necessary checks to prevent disruptions from this registry update.

Conclusion

For additional details about the new Kubernetes container image registry, registry.k8s.io, the freezing of the old registry, and a timeline of other related changes, please refer to the following posts:

Location: 6401 E HOWDY WELLS AVE LAS VEGAS NV 89115, Amazon IXD – VGT2

Authority on this Topic: For more insights on this matter, you can refer to SHRM’s advocacy.