Wednesday, September 18, 2019

Kubernetes 1.16: Custom Resources, Overhauled Metrics, and Volume Extensions

Authors: Kubernetes 1.16 Release Team

We’re pleased to announce the delivery of Kubernetes 1.16, our third release of 2019! Kubernetes 1.16 consists of 31 enhancements: 8 enhancements moving to stable, 8 enhancements in beta, and 15 enhancements in alpha.

Major Themes

Custom resources

CRDs are in widespread use as a Kubernetes extensibility mechanism and have been available in beta since the 1.7 release. The 1.16 release marks the graduation of CRDs to general availability (GA).

Overhauled metrics

Kubernetes has previously made extensive use of a global metrics registry to register metrics to be exposed. By implementing a metrics registry, metrics are registered in more transparent means. Previously, Kubernetes metrics have been excluded from any kind of stability requirements.

Volume Extension

There are quite a few enhancements in this release that pertain to volumes and volume modifications. Volume resizing support in CSI specs is moving to beta which allows for any CSI spec volume plugin to be resizable.

Additional Enhancements

Custom Resources Reach General Availability

CRDs have become the basis for extensions in the Kubernetes ecosystem. Started as a ground-up redesign of the ThirdPartyResources prototype, they have finally reached GA in 1.16 with apiextensions.k8s.io/v1, as the hard-won lessons of API evolution in Kubernetes have been integrated. As we transition to GA, the focus is on data consistency for API clients.

As you upgrade to the GA API, you’ll notice that several of the previously optional guard rails have become required and/or default behavior. Things like structural schemas, pruning unknown fields, validation, and protecting the *.k8s.io group are important for ensuring the longevity of your APIs and are now much harder to accidentally miss. Defaulting is another important part of API evolution and that support will be on by default for CRD.v1. The combination of these, along with CRD conversion mechanisms are enough to build stable APIs that evolve over time, the same way that native Kubernetes resources have changed without breaking backward-compatibility.

Updates to the CRD API won’t end here. We have ideas for features like arbitrary subresources, API group migration, and maybe a more efficient serialization protocol, but the changes from here are expected to be optional and complementary in nature to what’s already here in the GA API. Happy operator writing!

Details on how to work with custom resources can be found in the Kubernetes documentation.

Opening Doors With Windows Enhancements

Beta: Enhancing the workload identity options for Windows containers

Active Directory Group Managed Service Account (GMSA) support is graduating to beta and certain annotations that were introduced with the alpha support are being deprecated. GMSA is a specific type of Active Directory account that enables Windows containers to carry an identity across the network and communicate with other resources. Windows containers can now gain authenticated access to external resources. In addition, GMSA provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.

Adding support for RunAsUserName as an alpha release. The RunAsUserName is a string specifying the windows identity (or username) in Windows to run the entrypoint of the container and is a part of the newly introduced windowsOptions component of the securityContext (WindowsSecurityContextOptions).

Alpha: Improvements to setup & node join experience with kubeadm

Introducing alpha support for kubeadm, enabling Kubernetes users to easily join (and reset) Windows worker nodes to an existing cluster the same way they do for Linux nodes. Users can utilize kubeadm to prepare and add a Windows node to cluster. When the operations are complete, the node will be in a Ready state and able to run Windows containers. In addition, we will also provide a set of Windows-specific scripts to enable the installation of prerequisites and CNIs ahead of joining the node to the cluster.

Alpha: Introducing support for Container Storage Interface (CSI)

Introducing CSI plugin support for out-of-tree providers, enabling Windows nodes in a Kubernetes cluster to leverage persistent storage capabilities for Windows-based workloads. This significantly expands the storage options of Windows workloads, adding onto a list that included FlexVolume and in-tree storage plugins. This capability is achieved through a host OS proxy that enables the execution of privileged operations on the Windows node on behalf of containers.

Introducing Endpoint Slices

The release of Kubernetes 1.16 includes an exciting new alpha feature: Endpoint Slices. These provide a scalable and extensible alternative to Endpoints resources. Behind the scenes, these resources play a big role in network routing within Kubernetes. Each network endpoint is tracked within these resources, and kube-proxy uses them for generating proxy rules that allow pods to communicate with each other so easily in Kubernetes.

Providing Greater Scalability

A key goal for Endpoint Slices is to enable greater scalability for Kubernetes Services. With the existing Endpoints resources, a single resource must include network endpoints representing all pods matching a Service. As Services start to scale to thousands of pods, the corresponding Endpoints resources become quite large. Simply adding or removing one endpoint from a Service at this scale can be quite costly. As the Endpoints resource is updated, every piece of code watching Endpoints will need to be sent a full copy of the resource. With kube-proxy running on every node in a cluster, a copy needs to be sent to every single node. At a small scale, this is not an issue, but it becomes increasingly noticeable as clusters get larger.

As a simple example, in a cluster with 5,000 nodes and a 1MB Endpoints object, any update would result in approximately 5GB transmitted (that’s enough to fill a DVD). This becomes increasingly significant given how frequently Endpoints can change during events like rolling updates on Deployments.

With Endpoint Slices, network endpoints for a Service are split into multiple resources, significantly decreasing the amount of data required for updates at scale. By default, Endpoint Slices are limited to 100 endpoints each.

For example, let’s take a cluster with 20,000 network endpoints spread over 5,000 nodes. Updating a single endpoint will be much more efficient with Endpoint Slices since each one includes only a tiny portion of the total number of network endpoints. Instead of transferring a big Endpoints object to each node, only the small Endpoint Slice that’s been changed has to be transferred. The net effect is that approximately 200x less data needs to be transferred for this update.

	Endpoints	Endpoint Slices
# of resources	1	20k / 100 = 200
# of network endpoints stored	1 * 20k = 20k	200 * 100 = 20k
size of each resource	20k * const = ~2.0 MB	100 * const = ~10 kB
watch event data transferred	~2.0MB * 5k = 10GB	~10kB * 5k = 50MB

The second primary goal for Endpoint Slices was to provide a resource that would be highly extensible and useful across a wide variety of use cases. One of the key additions with Endpoint Slices involves a new topology attribute. By default, this will be populated with the existing topology labels used throughout Kubernetes indicating attributes such as region and zone. Of course, this field can be populated with custom labels as well for more specialized use cases.

Endpoint Slices also include greater flexibility for address types. Each contains a list of addresses. An initial use case for multiple addresses would be to support dual stack endpoints with both IPv4 and IPv6 addresses.

The Kubernetes documentation has a lot more information about Endpoint Slices. There’s also a great KubeCon talk that provides more information on the initial rationale for developing Endpoint Slices. As an alpha feature in Kubernetes 1.16, they will not be enabled by default, but the docs cover how to enable them in your cluster.

Notable Feature Updates

• Topology Manager, a new Kubelet component, aims to co-ordinate resource assignment decisions to provide optimized resource allocations.
• IPv4/IPv6 dual-stack enables the allocation of both IPv4 and IPv6 addresses to Pods and Services.
• Extensions for Cloud Controller Manager Migration.
• Continued deprecation of extensions/v1beta1, apps/v1beta1, and apps/v1beta2 APIs. These extensions are now retired in 1.16!

Availability

Kubernetes 1.16 is available for download on GitHub. To get started with Kubernetes, check out these interactive tutorials. You can also easily install 1.16 using kubeadm.

Release Team

This release is made possible through the efforts of hundreds of individuals who contributed both technical and non-technical content. Special thanks to the release team led by Lachlan Evenson, Principal Program Manager at Microsoft. The 32 individuals on the release team coordinated many aspects of the release, from documentation to testing, validation, and feature completeness.

As the Kubernetes community has grown, our release process represents an amazing demonstration of collaboration in open source software development. Kubernetes continues to gain new users at a rapid pace. This growth creates a positive feedback cycle where more contributors commit code creating a more vibrant ecosystem. Kubernetes has had over 32,000 individual contributors to date and an active community of more than 66,000 people.

Release Mascot

The Kubernetes 1.16 release crest was loosely inspired by the Apollo 16 mission crest. It represents the hard work of the release-team and the community alike and is an ode to the challenges and fun times we shared as a team throughout the release cycle. Many thanks to Ronan Flynn-Curran of Microsoft for creating this magnificent piece.

Kubernetes Updates

Project Velocity

The CNCF has continued refining DevStats, an ambitious project to visualize the myriad contributions that go into the project. K8s DevStats illustrates the breakdown of contributions from major company contributors, as well as an impressive set of preconfigured reports on everything from individual contributors to pull request lifecycle times. This past year, 1,147 different companies and over 3,149 individuals contribute to Kubernetes each month. Check out DevStats to learn more about the overall velocity of the Kubernetes project and community.

Ecosystem

• The Kubernetes project leadership created the Security Audit Working Group to oversee the very first third-part Kubernetes security audit, in an effort to improve the overall security of the ecosystem.
• The Kubernetes Certified Service Providers program (KCSP) reached 100 member companies, ranging from the largest multinational cloud, enterprise software, and consulting companies to tiny startups.
• The first Kubernetes Project Journey Report was released, showcasing the massive growth of the project.

KubeCon + CloudNativeCon

The Cloud Native Computing Foundation’s flagship conference gathers adopters and technologists from leading open source and cloud native communities in San Diego, California from November 18-21, 2019. Join Kubernetes, Prometheus, Envoy, CoreDNS, containerd, Fluentd, OpenTracing, gRPC, CNI, Jaeger, Notary, TUF, Vitess, NATS, Linkerd, Helm, Rook, Harbor, etcd, Open Policy Agent, CRI-O, and TiKV as the community gathers for four days to further the education and advancement of cloud native computing. Register today!

Webinar

Join members of the Kubernetes 1.16 release team on Oct 22, 2019 to learn about the major features in this release. Register here.

Get Involved

The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly community meeting, and through the channels below. Thank you for your continued feedback and support.

• Follow us on Twitter @Kubernetesio for latest updates
• Join the community discussion on Discuss
• Join the community on Slack
• Post questions (or answer questions) on Stack Overflow
• Share your Kubernetes story

2018.12.05

新贡献者工作坊上海站

作者: Josh Berkus (红帽), Yang Li (The Plant), Puja Abbassi (Giant Swarm), XiangPeng Zhao (中兴通讯)

最近，在中国的首次 KubeCon 上，我们完成了在中国的首次新贡献者峰会。看到所有中国和亚洲的开发者（以及来自世界各地的一些人）有兴趣成为贡献者，这令人非常兴奋。在长达一天的课程中，他们了解了如何、为什么以及在何处为 Kubernetes 作出贡献，创建了 PR，参加了贡献者圆桌讨论，并签署了他们的 CLA。

这是我们的第二届新贡献者工作坊（NCW），它由前一次贡献者体验 SIG 成员创建和领导的哥本哈根研讨会延伸而来。根据受众情况，本次活动采用了中英文两种语言，充分利用了 CNCF 赞助的一流的同声传译服务。同样，NCW 团队由社区成员组成，既有说英语的，也有说汉语的：Yang Li、XiangPeng Zhao、Puja Abbassi、Noah Abrahams、Tim Pepper、Zach Corleissen、Sen Lu 和 Josh Berkus。除了演讲和帮助学员外，团队的双语成员还将所有幻灯片翻译成了中文。共有五十一名学员参加。

NCW 让参与者完成了为 Kubernetes 作出贡献的各个阶段，从决定在哪里作出贡献开始，接着介绍了 SIG 系统和我们的代码仓库结构。我们还有来自文档和测试基础设施领域的「客座讲者」，他们负责讲解有关的贡献。最后，我们在创建 issue、提交并批准 PR 的实践练习后，结束了工作坊。

这些实践练习使用一个名为贡献者游乐场的代码仓库，由贡献者体验 SIG 创建，让新贡献者尝试在一个 Kubernetes 仓库中执行各种操作。它修改了 Prow 和 Tide 自动化，使用与真实代码仓库类似的 Owners 文件。这可以让学员了解为我们的仓库做出贡献的有关机制，同时又不妨碍正常的开发流程。

「防火长城」和语言障碍都使得在中国为 Kubernetes 作出贡献变得困难。而且，中国的开源商业模式并不成熟，员工在开源项目上工作的时间有限。

中国工程师渴望参与 Kubernetes 的研发，但他们中的许多人不知道从何处开始，因为 Kubernetes 是一个如此庞大的项目。通过本次工作坊，我们希望帮助那些想要参与贡献的人，不论他们希望修复他们遇到的一些错误、改进或本地化文档，或者他们需要在工作中用到 Kubernetes。我们很高兴看到越来越多的中国贡献者在过去几年里加入社区，我们也希望将来可以看到更多。

「我已经参与了 Kubernetes 社区大约三年」，XiangPeng Zhao 说，「在社区，我注意到越来越多的中国开发者表现出对 Kubernetes 贡献的兴趣。但是，开始为这样一个项目做贡献并不容易。我尽力帮助那些我在社区遇到的人，但是，我认为可能仍有一些新的贡献者离开社区，因为他们在遇到麻烦时不知道从哪里获得帮助。幸运的是，社区在 KubeCon 哥本哈根站发起了 NCW，并在 KubeCon 上海站举办了第二届。我很高兴受到 Josh Berkus 的邀请，帮助组织这个工作坊。在工作坊期间，我当面见到了社区里的朋友，在练习中指导了与会者，等等。所有这些对我来说都是难忘的经历。作为有着多年贡献者经验的我，也学习到了很多。我希望几年前我开始为 Kubernetes 做贡献时参加过这样的工作坊」。

工作坊以现有贡献者圆桌讨论结束，嘉宾包括 Lucas Käldström、Janet Kuo、Da Ma、Pengfei Ni、Zefeng Wang 和 Chao Xu。这场圆桌讨论旨在让新的和现有的贡献者了解一些最活跃的贡献者和维护者的幕后日常工作，不论他们来自中国还是世界各地。嘉宾们讨论了从哪里开始贡献者的旅程，以及如何与评审者和维护者进行互动。他们进一步探讨了在中国参与贡献的主要问题，并向与会者预告了在 Kubernetes 的未来版本中可以期待的令人兴奋的功能。

工作坊结束后，XiangPeng Zhao 和一些与会者就他们的经历在微信和 Twitter 上进行了交谈。他们很高兴参加了 NCW，并就改进工作坊提出了一些建议。一位名叫 Mohammad 的与会者说：「我在工作坊上玩得很开心，学习了参与 k8s 贡献的整个过程。」另一位与会者 Jie Jia 说：「工作坊非常精彩。它系统地解释了如何为 Kubernetes 做出贡献。即使参与者之前对此一无所知，他（她）也可以理解这个过程。对于那些已经是贡献者的人，他们也可以学习到新东西。此外，我还可以在工作坊上结识来自国内外的新朋友。真是棒极了！」

贡献者体验 SIG 将继续在未来的 KubeCon 上举办新贡献者工作坊，包括西雅图站、巴塞罗那站，然后在 2019 年六月回到上海。如果你今年未能参加，请在未来的 KubeCon 上注册。并且，如果你遇到工作坊的与会者，请务必欢迎他们加入社区。

链接：

• 中文版幻灯片：PDF
• 英文版幻灯片：PDF 或 带有演讲者笔记的 Google Docs
• 贡献者游乐场