Wednesday, September 18, 2019
Kubernetes 1.16: Custom Resources, Overhauled Metrics, and Volume Extensions
Authors: Kubernetes 1.16 Release Team
We’re pleased to announce the delivery of Kubernetes 1.16, our third release of 2019! Kubernetes 1.16 consists of 31 enhancements: 8 enhancements moving to stable, 8 enhancements in beta, and 15 enhancements in alpha.
Major Themes
Custom resources
CRDs are in widespread use as a Kubernetes extensibility mechanism and have been available in beta since the 1.7 release. The 1.16 release marks the graduation of CRDs to general availability (GA).
Overhauled metrics
Kubernetes has previously made extensive use of a global metrics registry to register metrics to be exposed. By implementing a metrics registry, metrics are registered in more transparent means. Previously, Kubernetes metrics have been excluded from any kind of stability requirements.
Volume Extension
There are quite a few enhancements in this release that pertain to volumes and volume modifications. Volume resizing support in CSI specs is moving to beta which allows for any CSI spec volume plugin to be resizable.
Additional Enhancements
Custom Resources Reach General Availability
CRDs have become the basis for extensions in the Kubernetes ecosystem. Started as a ground-up redesign of the ThirdPartyResources prototype, they have finally reached GA in 1.16 with apiextensions.k8s.io/v1, as the hard-won lessons of API evolution in Kubernetes have been integrated. As we transition to GA, the focus is on data consistency for API clients.
As you upgrade to the GA API, you’ll notice that several of the previously optional guard rails have become required and/or default behavior. Things like structural schemas, pruning unknown fields, validation, and protecting the *.k8s.io group are important for ensuring the longevity of your APIs and are now much harder to accidentally miss. Defaulting is another important part of API evolution and that support will be on by default for CRD.v1. The combination of these, along with CRD conversion mechanisms are enough to build stable APIs that evolve over time, the same way that native Kubernetes resources have changed without breaking backward-compatibility.
Updates to the CRD API won’t end here. We have ideas for features like arbitrary subresources, API group migration, and maybe a more efficient serialization protocol, but the changes from here are expected to be optional and complementary in nature to what’s already here in the GA API. Happy operator writing!
Details on how to work with custom resources can be found in the Kubernetes documentation.
Opening Doors With Windows Enhancements
Beta: Enhancing the workload identity options for Windows containers
Active Directory Group Managed Service Account (GMSA) support is graduating to beta and certain annotations that were introduced with the alpha support are being deprecated. GMSA is a specific type of Active Directory account that enables Windows containers to carry an identity across the network and communicate with other resources. Windows containers can now gain authenticated access to external resources. In addition, GMSA provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.
Adding support for RunAsUserName as an alpha release. The RunAsUserName is a string specifying the windows identity (or username) in Windows to run the entrypoint of the container and is a part of the newly introduced windowsOptions component of the securityContext (WindowsSecurityContextOptions).
Alpha: Improvements to setup & node join experience with kubeadm
Introducing alpha support for kubeadm, enabling Kubernetes users to easily join (and reset) Windows worker nodes to an existing cluster the same way they do for Linux nodes. Users can utilize kubeadm to prepare and add a Windows node to cluster. When the operations are complete, the node will be in a Ready state and able to run Windows containers. In addition, we will also provide a set of Windows-specific scripts to enable the installation of prerequisites and CNIs ahead of joining the node to the cluster.
Alpha: Introducing support for Container Storage Interface (CSI)
Introducing CSI plugin support for out-of-tree providers, enabling Windows nodes in a Kubernetes cluster to leverage persistent storage capabilities for Windows-based workloads. This significantly expands the storage options of Windows workloads, adding onto a list that included FlexVolume and in-tree storage plugins. This capability is achieved through a host OS proxy that enables the execution of privileged operations on the Windows node on behalf of containers.
Introducing Endpoint Slices
The release of Kubernetes 1.16 includes an exciting new alpha feature: Endpoint Slices. These provide a scalable and extensible alternative to Endpoints resources. Behind the scenes, these resources play a big role in network routing within Kubernetes. Each network endpoint is tracked within these resources, and kube-proxy uses them for generating proxy rules that allow pods to communicate with each other so easily in Kubernetes.
Providing Greater Scalability
A key goal for Endpoint Slices is to enable greater scalability for Kubernetes Services. With the existing Endpoints resources, a single resource must include network endpoints representing all pods matching a Service. As Services start to scale to thousands of pods, the corresponding Endpoints resources become quite large. Simply adding or removing one endpoint from a Service at this scale can be quite costly. As the Endpoints resource is updated, every piece of code watching Endpoints will need to be sent a full copy of the resource. With kube-proxy running on every node in a cluster, a copy needs to be sent to every single node. At a small scale, this is not an issue, but it becomes increasingly noticeable as clusters get larger.
As a simple example, in a cluster with 5,000 nodes and a 1MB Endpoints object, any update would result in approximately 5GB transmitted (that’s enough to fill a DVD). This becomes increasingly significant given how frequently Endpoints can change during events like rolling updates on Deployments.
With Endpoint Slices, network endpoints for a Service are split into multiple resources, significantly decreasing the amount of data required for updates at scale. By default, Endpoint Slices are limited to 100 endpoints each.
For example, let’s take a cluster with 20,000 network endpoints spread over 5,000 nodes. Updating a single endpoint will be much more efficient with Endpoint Slices since each one includes only a tiny portion of the total number of network endpoints. Instead of transferring a big Endpoints object to each node, only the small Endpoint Slice that’s been changed has to be transferred. The net effect is that approximately 200x less data needs to be transferred for this update.
| Endpoints | Endpoint Slices | |
| # of resources | 1 | 20k / 100 = 200 |
| # of network endpoints stored | 1 * 20k = 20k | 200 * 100 = 20k |
| size of each resource | 20k * const = ~2.0 MB | 100 * const = ~10 kB |
| watch event data transferred | ~2.0MB * 5k = 10GB | ~10kB * 5k = 50MB |
The second primary goal for Endpoint Slices was to provide a resource that would be highly extensible and useful across a wide variety of use cases. One of the key additions with Endpoint Slices involves a new topology attribute. By default, this will be populated with the existing topology labels used throughout Kubernetes indicating attributes such as region and zone. Of course, this field can be populated with custom labels as well for more specialized use cases.
Endpoint Slices also include greater flexibility for address types. Each contains a list of addresses. An initial use case for multiple addresses would be to support dual stack endpoints with both IPv4 and IPv6 addresses.
The Kubernetes documentation has a lot more information about Endpoint Slices. There’s also a great KubeCon talk that provides more information on the initial rationale for developing Endpoint Slices. As an alpha feature in Kubernetes 1.16, they will not be enabled by default, but the docs cover how to enable them in your cluster.
Notable Feature Updates
- Topology Manager, a new Kubelet component, aims to co-ordinate resource assignment decisions to provide optimized resource allocations.
- IPv4/IPv6 dual-stack enables the allocation of both IPv4 and IPv6 addresses to Pods and Services.
- Extensions for Cloud Controller Manager Migration.
- Continued deprecation of extensions/v1beta1, apps/v1beta1, and apps/v1beta2 APIs. These extensions are now retired in 1.16!
Availability
Kubernetes 1.16 is available for download on GitHub. To get started with Kubernetes, check out these interactive tutorials. You can also easily install 1.16 using kubeadm.
Release Team
This release is made possible through the efforts of hundreds of individuals who contributed both technical and non-technical content. Special thanks to the release team led by Lachlan Evenson, Principal Program Manager at Microsoft. The 32 individuals on the release team coordinated many aspects of the release, from documentation to testing, validation, and feature completeness.
As the Kubernetes community has grown, our release process represents an amazing demonstration of collaboration in open source software development. Kubernetes continues to gain new users at a rapid pace. This growth creates a positive feedback cycle where more contributors commit code creating a more vibrant ecosystem. Kubernetes has had over 32,000 individual contributors to date and an active community of more than 66,000 people.
Release Mascot
The Kubernetes 1.16 release crest was loosely inspired by the Apollo 16 mission crest. It represents the hard work of the release-team and the community alike and is an ode to the challenges and fun times we shared as a team throughout the release cycle. Many thanks to Ronan Flynn-Curran of Microsoft for creating this magnificent piece.
Kubernetes Updates
Project Velocity
The CNCF has continued refining DevStats, an ambitious project to visualize the myriad contributions that go into the project. K8s DevStats illustrates the breakdown of contributions from major company contributors, as well as an impressive set of preconfigured reports on everything from individual contributors to pull request lifecycle times. This past year, 1,147 different companies and over 3,149 individuals contribute to Kubernetes each month. Check out DevStats to learn more about the overall velocity of the Kubernetes project and community.
Ecosystem
- The Kubernetes project leadership created the Security Audit Working Group to oversee the very first third-part Kubernetes security audit, in an effort to improve the overall security of the ecosystem.
- The Kubernetes Certified Service Providers program (KCSP) reached 100 member companies, ranging from the largest multinational cloud, enterprise software, and consulting companies to tiny startups.
- The first Kubernetes Project Journey Report was released, showcasing the massive growth of the project.
KubeCon + CloudNativeCon
The Cloud Native Computing Foundation’s flagship conference gathers adopters and technologists from leading open source and cloud native communities in San Diego, California from November 18-21, 2019. Join Kubernetes, Prometheus, Envoy, CoreDNS, containerd, Fluentd, OpenTracing, gRPC, CNI, Jaeger, Notary, TUF, Vitess, NATS, Linkerd, Helm, Rook, Harbor, etcd, Open Policy Agent, CRI-O, and TiKV as the community gathers for four days to further the education and advancement of cloud native computing. Register today!
Webinar
Join members of the Kubernetes 1.16 release team on Oct 22, 2019 to learn about the major features in this release. Register here.
Get Involved
The simplest way to get involved with Kubernetes is by joining one of the many Special Interest Groups (SIGs) that align with your interests. Have something you’d like to broadcast to the Kubernetes community? Share your voice at our weekly community meeting, and through the channels below. Thank you for your continued feedback and support.
- Follow us on Twitter @Kubernetesio for latest updates
- Join the community discussion on Discuss
- Join the community on Slack
- Post questions (or answer questions) on Stack Overflow
- Share your Kubernetes story
0001.01.01
title: “ 如何在AWS上部署安全，可审计，可复现的k8s集群 “ date: 2016-04-15 slug: kubernetes-on-aws_15
url: /blog/2016/04/Kubernetes-On-Aws_15
今天的客座文章是由Colin Hom撰写，CoreOS的基础架构工程师。CoreOS致力于推广谷歌的基础架构模式（Google’s Infrastructure for Everyone Else， #GIFEE），让全世界的容器都能在CoreOS Linux, Tectonic 和 Quay上安全运行。
加入到我们的柏林CoreOS盛宴，这是一个开源分布式系统主题的会议，在这里可以了解到更多关于CoreOS和Kubernetes的信息。
在CoreOS, 我们一直都是在生产环境中大规模部署Kubernetes。今天我们非常兴奋地想分享一款工具，它能让你的Kubernetes生产环境大规模部署更加的轻松。Kube-aws这个工具可以用来在AWS上部署可审计，可复现的k8s集群，而CoreOS本身就在生产环境中使用它。
也许今天，你更多的可能是用手工的方式来拼接Kubernetes组件。但有了这个工具之后，Kubernetes可以流水化地打包、交付，节省时间，减少了相互间的依赖，更加快捷地实现生产环境的部署。
借助于一个简单的模板系统，来生成集群配置，这么做是因为一套声明式的配置模板可以版本控制，审计以及重复部署。而且，由于整个创建过程只用到了AWS CloudFormation 和 cloud-init，你也就不需要额外用到其它的配置管理工具。开箱即用！
如果要跳过演讲，直接了解这个项目，可以看看kube-aws的最新发布，支持Kubernetes 1.2.x。如果要部署集群，可以参考[文档]](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html).
为什么是kube-aws？安全，可审计，可复现
Kube-aws设计初衷有三个目标。
安全 : TLS 资源在嵌入到CloudFormation JSON之前，通过AWS 秘钥管理服务加密。通过单独管理KMS密钥的IAM 策略，可以将CloudFormation栈的访问与TLS秘钥的访问分离开。
可审计 : kube-aws是围绕集群资产的概念来创建。这些配置和账户资产是对集群的完全描述。由于KMS被用来加密TLS资产，因而可以无所顾忌地将未加密的CloudFormation栈 JSON签入到版本控制服务中。
可重复 : –export 选项将参数化的集群定义打包成一整个JSON文件，对应一个CloudFormation栈。这个文件可以版本控制，然后，如果需要的话，通过现有的部署工具直接提交给CloudFormation API。
如何开始用kube-aws
在此基础之上，kube-aws也实现了一些功能，使得在AWS上部署Kubernetes集群更加容易，灵活。下面是一些例子。
Route53集成 : Kube-aws 可以管理你的集群DNS记录，作为配置过程的一部分。
cluster.yaml
externalDNSName: my-cluster.kubernetes.coreos.com
createRecordSet: true
hostedZone: kubernetes.coreos.com
recordSetTTL: 300
现有VPC支持 : 将集群部署到现有的VPC上。
cluster.yaml
vpcId: vpc-xxxxx
routeTableId: rtb-xxxxx
验证 : kube-aws 支持验证 cloud-init 和 CloudFormation定义，以及集群栈会集成用到的外部资源。例如，下面就是一个cloud-config，外带一个拼写错误的参数：
userdata/cloud-config-worker
#cloud-config
coreos:
flannel:
interrface: $private\_ipv4
etcd\_endpoints: {{ .ETCDEndpoints }}
$ kube-aws validate
> Validating UserData… Error: cloud-config validation errors: UserDataWorker: line 4: warning: unrecognized key “interrface”
考虑如何起步？看看kube-aws 文档！
未来的工作
一如既往，kube-aws的目标是让生产环境部署更加的简单。尽管我们现在在AWS下使用kube-aws进行生产环境部署，但是这个项目还是pre-1.0，所以还有很多的地方，kube-aws需要考虑、扩展。
容错 : CoreOS坚信 Kubernetes on AWS是强健的平台，适于容错、自恢复部署。在接下来的几个星期，kube-aws将会迎接新的考验：混世猴子（Chaos Monkey）测试 - 控制平面以及全部！
零停机更新 : 更新CoreOS节点和Kubernetes组件不需要停机，也不需要考虑实例更新策略（instance replacement strategy）的影响。
有一个github issue来追踪这些工作进展。我们期待你的参与，提交issue，或是直接贡献。
想要更多地了解Kubernetes，来柏林CoreOS盛宴看看，- 五月 9-10, 2016
– Colin Hom, 基础架构工程师, CoreOS