apiserver源码分析——处理请求

前言

上一篇说道k8s-apiserver如何启动，本篇则介绍apiserver启动后，接收到客户端请求的处理流程。如下图所示



认证与授权一般系统都会使用到，认证是鉴别访问apiserver的请求方是谁，一般情况下服务端是需要知晓客户端是谁方可接受请求，除了允许匿名访问这种场景，同时认证也为后续的授权提供基础。授权是为了判断当前请求的客户端是否具备请求当前资源的权限，具备则放行让其继续往后走，否则拒绝本次请求。准入控制器为请求处理流程提供了一个扩展的口，它提供了两个回调的钩子，能让用户在资源持久化前再额外对资源的值作改动或者验证，如果验证出错同样可以终止整个处理流程。最后对资源的变更会持久化到Etcd。

本篇以创建pod为例，探索apiserver如何处理。

Authentication

请求到达apiserver后第一个是需要进行认证，辨别请求来源的身份。认证方式的配置在上一篇讲述构建genericConfig的时候有提及，在执行buildGenericConfig函数时调用s.Authentication.ApplyTo配置

代码位于/pkg/kubeapiserver/options/authentication.go

func (o *BuiltInAuthenticationOptions) ApplyTo(authInfo *genericapiserver.AuthenticationInfo,.....) error {
	//创建出authenticatorConfig
	authenticatorConfig, err := o.ToAuthenticationConfig()
	//对authenticatorConfig字段进行设置
	...
	//创建出Authenticator
	authInfo.Authenticator, openAPIConfig.SecurityDefinitions, err = authenticatorConfig.New()
}

ApplyTo先创建出认证相关配置authenticatorConfig，然后初始化部分认证方式的Provider，最终调用authenticatorConfig.New方法将按照认证的配置信息构造出一个Authenticator，传递给authInfo.Authenticator

Authenticator.New方法如下所示，定义了两个数组用于存放启用的authenticators和token类的authenticators，根据Config的配置信息按需启用认证方式，再将token类的authenticators转换成普通的authenticators。最终将这个authenticator传递给一个Wrapper类型UnionAuthenticator返回

代码位于/pkg/kubeapiserver/authenticator/config.go

func (config Config) New() (authenticator.Request, *spec.SecurityDefinitions, error) {
	var authenticators []authenticator.Request
	var tokenAuthenticators []authenticator.Token
	//各种认证方式的初始化操作
	...

	if len(tokenAuthenticators) > 0 {
		// Union the token authenticators
		tokenAuth := tokenunion.New(tokenAuthenticators...)
		// Optionally cache authentication results
		if config.TokenSuccessCacheTTL > 0 || config.TokenFailureCacheTTL > 0 {
			tokenAuth = tokencache.New(tokenAuth, true, config.TokenSuccessCacheTTL, config.TokenFailureCacheTTL)
		}
		authenticators = append(authenticators, bearertoken.New(tokenAuth), websocket.NewProtocolAuthenticator(tokenAuth))
		securityDefinitions["BearerToken"] = &spec.SecurityScheme{
			SecuritySchemeProps: spec.SecuritySchemeProps{
				Type:        "apiKey",
				Name:        "authorization",
				In:          "header",
				Description: "Bearer Token authentication",
			},
		}
	}

	if len(authenticators) == 0 {
		if config.Anonymous {
			return anonymous.NewAuthenticator(), &securityDefinitions, nil
		}
		return nil, &securityDefinitions, nil
	}

	authenticator := union.New(authenticators...)

	authenticator = group.NewAuthenticatedGroupAdder(authenticator)

}

在这里简单列举一下上述提到的多种认证类型，包括9种，分别是：BasicAuth,TokenAuth,BootstrapToken,OIDC,RequesHeader,WebhookTokenAuth,Anonymous,ClientCA,ServiceAccountAuth。鄙人为了方便记忆分别将他们归为3类

• token类：TokenAuth，BootstrapToken，WebhookTokenAuth，OIDC
• 证书类：ClientCA，ServiceAccountAuth
• 其他类：BasicAuth，RequesHeader，Anonymous

由于篇幅原因各种配置类型的特点则不展开介绍

特别地提及一下，pod里面访问apiserver一般用的是ServiceAccountAuth；在进行apiserver-aggregrate双向认证的时候会用到clientCA；往k8s添加新节点时kubelet会用到BootstrapToken

认证在请求过程是一个HandlerChain串起来的，每个handler函数的构建时都会里层的handler函数，待本层handler处理完毕后才会执行里层的handler，这样一层层执行最后才执行到真正的请求响应逻辑，如Pod创建

回归到上篇介绍的buildGenericConfig函数，一开始调用了 genericapiserver.NewConfig，NewConfig创建Config结构时给BuildHandlerChainFunc字段传入DefaultBuildHandlerChain这个函数

代码位于 /vendor/k8s.io/apiserver/pkg/server/config.go

func NewConfig(codecs serializer.CodecFactory) *Config {
	return &Config{
		Serializer:                  codecs,
		BuildHandlerChainFunc:       DefaultBuildHandlerChain,
		...
	}
}

func DefaultBuildHandlerChain(apiHandler http.Handler, c *Config) http.Handler {
	handler := genericapifilters.WithAuthorization(apiHandler, c.Authorization.Authorizer, c.Serializer)
	if c.FlowControl != nil {
		handler = genericfilters.WithPriorityAndFairness(handler, c.LongRunningFunc, c.FlowControl)
	} else {
		handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc)
	}
	handler = genericapifilters.WithImpersonation(handler, c.Authorization.Authorizer, c.Serializer)
	handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc)
	failedHandler := genericapifilters.Unauthorized(c.Serializer)
	failedHandler = genericapifilters.WithFailedAuthenticationAudit(failedHandler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithAuthentication(handler, c.Authentication.Authenticator, failedHandler, c.Authentication.APIAudiences)
	handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true")
	handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc, c.RequestTimeout)
	handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup)
	handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver)
	if c.SecureServing != nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 {
		handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance)
	}
	handler = genericapifilters.WithAuditAnnotations(handler, c.AuditBackend, c.AuditPolicyChecker)
	handler = genericapifilters.WithWarningRecorder(handler)
	handler = genericapifilters.WithCacheControl(handler)
	handler = genericapifilters.WithRequestReceivedTimestamp(handler)
	handler = genericfilters.WithPanicRecovery(handler)
	return handler
}

DefaultBuildHandlerChain函数就是上面构建HandlerChain串的地方，查看genericapifilters.WithAuthentication定义，代码位于/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go

func WithAuthentication(handler http.Handler, auth authenticator.Request, failed http.Handler, apiAuds authenticator.Audiences) http.Handler {
	if auth == nil {
		klog.Warningf("Authentication is disabled")
		return handler
	}
	return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		authenticationStart := time.Now()

		if len(apiAuds) > 0 {
			req = req.WithContext(authenticator.WithAudiences(req.Context(), apiAuds))
		}
		resp, ok, err := auth.AuthenticateRequest(req)
		defer recordAuthMetrics(resp, ok, err, apiAuds, authenticationStart)
		if err != nil || !ok {
			if err != nil {
				klog.Errorf("Unable to authenticate the request due to an error: %v", err)
			}
			failed.ServeHTTP(w, req)
			return
		}

		if !audiencesAreAcceptable(apiAuds, resp.Audiences) {
			err = fmt.Errorf("unable to match the audience: %v , accepted: %v", resp.Audiences, apiAuds)
			klog.Error(err)
			failed.ServeHTTP(w, req)
			return
		}

		// authorization header is not required anymore in case of a successful authentication.
		req.Header.Del("Authorization")

		req = req.WithContext(genericapirequest.WithUser(req.Context(), resp.User))
		handler.ServeHTTP(w, req)
	})
}

代码中auth.AuthenticateRequest就是执行认证逻辑的地方，如果认证失败则会返回返回失败。认证成功会把请求头中Authorization去掉，再调用里层的handler函数handler.ServeHTTP(w, req)

func (authHandler *unionAuthRequestHandler) AuthenticateRequest(req *http.Request) (*authenticator.Response, bool, error) {
	var errlist []error
	for _, currAuthRequestHandler := range authHandler.Handlers {
		resp, ok, err := currAuthRequestHandler.AuthenticateRequest(req)
		if err != nil {
			if authHandler.FailOnError {
				return resp, ok, err
			}
			errlist = append(errlist, err)
			continue
		}

		if ok {
			return resp, ok, err
		}
	}

	return nil, false, utilerrors.NewAggregate(errlist)
}

它就是遍历了所有启用的认证方式，只有一个成功了就可以了。

Authorization

与认证的类似，授权方式的配置也是在buildGenericConfig函数中，调用BuildAuthorizer函数创建，返回时将authorizer.Authorizer赋予给genericConfig.Authorization.Authorizer

buildGenericConfig最终调用authorizationConfig.New完成Authorizer的创建，代码位于 /pkg/kubeapiserver/authorizer/config.go

func (config Config) New() (authorizer.Authorizer, authorizer.RuleResolver, error) {
	if len(config.AuthorizationModes) == 0 {
		return nil, nil, fmt.Errorf("at least one authorization mode must be passed")
	}

	var (
		authorizers   []authorizer.Authorizer
		ruleResolvers []authorizer.RuleResolver
	)

	for _, authorizationMode := range config.AuthorizationModes {
		// Keep cases in sync with constant list in k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes/modes.go.
		switch authorizationMode {
		case modes.ModeNode:
			graph := node.NewGraph()
			node.AddGraphEventHandlers(
				graph,
				config.VersionedInformerFactory.Core().V1().Nodes(),
				config.VersionedInformerFactory.Core().V1().Pods(),
				config.VersionedInformerFactory.Core().V1().PersistentVolumes(),
				config.VersionedInformerFactory.Storage().V1().VolumeAttachments(),
			)
			nodeAuthorizer := node.NewAuthorizer(graph, nodeidentifier.NewDefaultNodeIdentifier(), bootstrappolicy.NodeRules())
			authorizers = append(authorizers, nodeAuthorizer)
			ruleResolvers = append(ruleResolvers, nodeAuthorizer)

		case modes.ModeAlwaysAllow:
			alwaysAllowAuthorizer := authorizerfactory.NewAlwaysAllowAuthorizer()
			authorizers = append(authorizers, alwaysAllowAuthorizer)
			ruleResolvers = append(ruleResolvers, alwaysAllowAuthorizer)
		case modes.ModeAlwaysDeny:
			alwaysDenyAuthorizer := authorizerfactory.NewAlwaysDenyAuthorizer()
			authorizers = append(authorizers, alwaysDenyAuthorizer)
			ruleResolvers = append(ruleResolvers, alwaysDenyAuthorizer)
		case modes.ModeABAC:
			abacAuthorizer, err := abac.NewFromFile(config.PolicyFile)
			if err != nil {
				return nil, nil, err
			}
			authorizers = append(authorizers, abacAuthorizer)
			ruleResolvers = append(ruleResolvers, abacAuthorizer)
		case modes.ModeWebhook:
			webhookAuthorizer, err := webhook.New(config.WebhookConfigFile,
				config.WebhookVersion,
				config.WebhookCacheAuthorizedTTL,
				config.WebhookCacheUnauthorizedTTL,
				config.CustomDial)
			if err != nil {
				return nil, nil, err
			}
			authorizers = append(authorizers, webhookAuthorizer)
			ruleResolvers = append(ruleResolvers, webhookAuthorizer)
		case modes.ModeRBAC:
			rbacAuthorizer := rbac.New(
				&rbac.RoleGetter{Lister: config.VersionedInformerFactory.Rbac().V1().Roles().Lister()},
				&rbac.RoleBindingLister{Lister: config.VersionedInformerFactory.Rbac().V1().RoleBindings().Lister()},
				&rbac.ClusterRoleGetter{Lister: config.VersionedInformerFactory.Rbac().V1().ClusterRoles().Lister()},
				&rbac.ClusterRoleBindingLister{Lister: config.VersionedInformerFactory.Rbac().V1().ClusterRoleBindings().Lister()},
			)
			authorizers = append(authorizers, rbacAuthorizer)
			ruleResolvers = append(ruleResolvers, rbacAuthorizer)
		default:
			return nil, nil, fmt.Errorf("unknown authorization mode %s specified", authorizationMode)
		}
	}

	return union.New(authorizers...), union.NewRuleResolvers(ruleResolvers...), nil
}

函数一开始也是创建了一个authorizers的数组，用于存放启用的授权方式。遍历config.AuthorizationModes，对对应的授权方式进行实例化。最后调用union.New(authorizers...)，以一个unionAuthzHandler作为支持的所有授权方式的wrapper返回回去。

授权方式有6种，分别是AlwaysAllow,AlwaysDeny,RBAC,ABAC,Node,Webhook。其中最常用的就是RBAC,k8s里面给sa绑定role和clusterrole进行授权的就是这个RBAC。

授权逻辑跟前文介绍认证一样通过HandlerChain串起来，同样在DefaultBuildHandlerChain函数中被加到HandlerChain中，调用了genericapifilters.WithAuthorization函数，代码位于/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go

func WithAuthorization(handler http.Handler, a authorizer.Authorizer, s runtime.NegotiatedSerializer) http.Handler {
	if a == nil {
		klog.Warningf("Authorization is disabled")
		return handler
	}
	return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		ctx := req.Context()
		ae := request.AuditEventFrom(ctx)

		attributes, err := GetAuthorizerAttributes(ctx)
		if err != nil {
			responsewriters.InternalError(w, req, err)
			return
		}
		authorized, reason, err := a.Authorize(ctx, attributes)
		// an authorizer like RBAC could encounter evaluation errors and still allow the request, so authorizer decision is checked before error here.
		if authorized == authorizer.DecisionAllow {
			audit.LogAnnotation(ae, decisionAnnotationKey, decisionAllow)
			audit.LogAnnotation(ae, reasonAnnotationKey, reason)
			handler.ServeHTTP(w, req)
			return
		}
		if err != nil {
			audit.LogAnnotation(ae, reasonAnnotationKey, reasonError)
			responsewriters.InternalError(w, req, err)
			return
		}

		klog.V(4).Infof("Forbidden: %#v, Reason: %q", req.RequestURI, reason)
		audit.LogAnnotation(ae, decisionAnnotationKey, decisionForbid)
		audit.LogAnnotation(ae, reasonAnnotationKey, reason)
		responsewriters.Forbidden(ctx, attributes, w, req, reason, s)
	})
}

处理函数中，先调用GetAuthorizerAttributes获取认证后得到的user信息以及请求资源的相关信息requestInfo，统一放到attributes，再调用授权的方法 a.Authorize。同样它也是一个接口，它与认证时类似，先调用一个unionAuthzHandler的wrapper,在这个wrapper里遍历各个启用的authorizer。只要里面有一个allow或deny的结果就立马返回，代码位于/vendor/k8s.io/apiserver/pkg/authorization/union/union.go

func (authzHandler unionAuthzHandler) Authorize(ctx context.Context, a authorizer.Attributes) (authorizer.Decision, string, error) {
	var (
		errlist    []error
		reasonlist []string
	)

	for _, currAuthzHandler := range authzHandler {
		decision, reason, err := currAuthzHandler.Authorize(ctx, a)

		if err != nil {
			errlist = append(errlist, err)
		}
		if len(reason) != 0 {
			reasonlist = append(reasonlist, reason)
		}
		switch decision {
		case authorizer.DecisionAllow, authorizer.DecisionDeny:
			return decision, reason, err
		case authorizer.DecisionNoOpinion:
			// continue to the next authorizer
		}
	}

	return authorizer.DecisionNoOpinion, strings.Join(reasonlist, "\n"), utilerrors.NewAggregate(errlist)
}

AdmissionWebhook

AdmissionWebhook是准入控制器，它作为k8s-apiserver对外暴露的一种扩展方式，主要针对增删改资源时对暴露两个hook点。一个是Mutate，可修改提交上来的资源；另一个是Validate,是对提交上来的资源进行验证。当然Mutate里面也可以包含验证操作。但是本篇不对这两种准入控制器的使用实例作介绍。

准入控制器的配置在buildGenericConfig函数中，通过调用s.Admission.ApplyTo方法进行配置。经过两层调用后到达AdmissionOptions.ApplyTo执行实际的创建逻辑，即： s.Admission.ApplyTo->a.GenericAdmission.ApplyTo。代码位于 /vendor/k8s.io/apiserver/pkg/server/options/admission.go

func (a *AdmissionOptions) ApplyTo(
	c *server.Config,
	informers informers.SharedInformerFactory,
	kubeAPIServerClientConfig *rest.Config,
	features featuregate.FeatureGate,
	pluginInitializers ...admission.PluginInitializer,
) error {
	if a == nil {
		return nil
	}

	// Admission depends on CoreAPI to set SharedInformerFactory and ClientConfig.
	if informers == nil {
		return fmt.Errorf("admission depends on a Kubernetes core API shared informer, it cannot be nil")
	}

	pluginNames := a.enabledPluginNames()
	//获取各个准入控制器的provider
	pluginsConfigProvider, err := admission.ReadAdmissionConfiguration(pluginNames, a.ConfigFile, configScheme)
	if err != nil {
		return fmt.Errorf("failed to read plugin config: %v", err)
	}

	clientset, err := kubernetes.NewForConfig(kubeAPIServerClientConfig)
	if err != nil {
		return err
	}
	genericInitializer := initializer.New(clientset, informers, c.Authorization.Authorizer, features)
	initializersChain := admission.PluginInitializers{}
	pluginInitializers = append(pluginInitializers, genericInitializer)
	initializersChain = append(initializersChain, pluginInitializers...)
	//将准入控制器集合串成一个admissionChain，再外面包一个Wrapper,类似于之前处理认证与授权一样的方式
	admissionChain, err := a.Plugins.NewFromPlugins(pluginNames, pluginsConfigProvider, initializersChain, a.Decorators)
	if err != nil {
		return err
	}
	//又在外面套一个可统计指标的wrapper
	c.AdmissionControl = admissionmetrics.WithStepMetrics(admissionChain)
	return nil
}
//代码位于 /vendor/k8s.io/apiserver/pkg/admission/plugins.go
func (ps *Plugins) NewFromPlugins(pluginNames []string, configProvider ConfigProvider, pluginInitializer PluginInitializer, decorator Decorator) (Interface, error) {
	handlers := []Interface{}
	mutationPlugins := []string{}
	validationPlugins := []string{}
	for _, pluginName := range pluginNames {
		pluginConfig, err := configProvider.ConfigFor(pluginName)
		if err != nil {
			return nil, err
		}

		plugin, err := ps.InitPlugin(pluginName, pluginConfig, pluginInitializer)
		if err != nil {
			return nil, err
		}
		if plugin != nil {
			if decorator != nil {
				handlers = append(handlers, decorator.Decorate(plugin, pluginName))
			} else {
				handlers = append(handlers, plugin)
			}

			if _, ok := plugin.(MutationInterface); ok {
				mutationPlugins = append(mutationPlugins, pluginName)
			}
			if _, ok := plugin.(ValidationInterface); ok {
				validationPlugins = append(validationPlugins, pluginName)
			}
		}
	}
	if len(mutationPlugins) != 0 {
		klog.Infof("Loaded %d mutating admission controller(s) successfully in the following order: %s.", len(mutationPlugins), strings.Join(mutationPlugins, ","))
	}
	if len(validationPlugins) != 0 {
		klog.Infof("Loaded %d validating admission controller(s) successfully in the following order: %s.", len(validationPlugins), strings.Join(validationPlugins, ","))
	}
	return newReinvocationHandler(chainAdmissionHandler(handlers)), nil
}

准入控制器除了自定义的，从上述代码中也可以观察到也有内置的，内置的准入控制器大概有30+种。

但是准入控制器的调用却不像认证与授权那样在调用DefaultBuildHandlerChain时加入到handler调用链中，它是每个增删改的实际处理函数中被调用，GenericConfig的AdmissionControl字段也是在初始化GenericServer的时候传递给后者的同名字段

registerResourceHandlers方法

延续上篇介绍apiserver启动流程时，调用installer.Install方法，创建了webservice，api中各个URL的路由注册，实现了对应地址的handler,这个handler是通过registerResourceHandlers，方法篇幅即长（约900行），包含了对一个资源的增删改查各种请求的处理，对其只能分段介绍。代码位于/vendor/k8s.io/apiserver/pkg/endpoints/installer.go

这个方法有三个入参

• 代表URL的path
• 资源存储相关的类storage
• 用于存放路由的go-rest对象webservice

先从path以及APIInstaller对象中获取group,version,kind,分辨这种资源是cluster scope还是namespace scope的

	admit := a.group.Admit

	optionsExternalVersion := a.group.GroupVersion
	if a.group.OptionsExternalVersion != nil {
		optionsExternalVersion = *a.group.OptionsExternalVersion
	}

	resource, subresource, err := splitSubresource(path)
	if err != nil {
		return nil, err
	}

	group, version := a.group.GroupVersion.Group, a.group.GroupVersion.Version

	fqKindToRegister, err := GetResourceKind(a.group.GroupVersion, storage, a.group.Typer)
	if err != nil {
		return nil, err
	}

	versionedPtr, err := a.group.Creater.New(fqKindToRegister)
	if err != nil {
		return nil, err
	}
	defaultVersionedObject := indirectArbitraryPointer(versionedPtr)
	kind := fqKindToRegister.Kind
	isSubresource := len(subresource) > 0

	// If there is a subresource, namespace scoping is defined by the parent resource
	namespaceScoped := true
	if isSubresource {
		parentStorage, ok := a.group.Storage[resource]
		if !ok {
			return nil, fmt.Errorf("missing parent storage: %q", resource)
		}
		scoper, ok := parentStorage.(rest.Scoper)
		if !ok {
			return nil, fmt.Errorf("%q must implement scoper", resource)
		}
		namespaceScoped = scoper.NamespaceScoped()

	} else {
		scoper, ok := storage.(rest.Scoper)
		if !ok {
			return nil, fmt.Errorf("%q must implement scoper", resource)
		}
		namespaceScoped = scoper.NamespaceScoped()
	}

接着是一系列的判定操作，根据当前这个storage是否有实现对应接口来判定能否提供对应服务，如 创建操作。这个结果会影响后面是否添加对应操作请求的路由

	creater, isCreater := storage.(rest.Creater)

然后就创建对应请求的Options，如CreateOptions。这个用于在后面创建路由时作为参数，平时使用client-go时也要传入metav1包的CreateOption,ListOption,DeleteOption等，就是这个参数了。

	versionedCreateOptions, err := a.group.Creater.New(optionsExternalVersion.WithKind("CreateOptions"))
	if err != nil {
		return nil, err
	}

下一步按照资源类型是cluster scope还是namespace scope来将支持的操作类型组成action集合，这个action集合的动作则是对应http的请求方法，如创建的

		actions = appendIf(actions, action{"POST", resourcePath, resourceParams, namer, false}, isCreater)

往后就是遍历action集合，为各个操作绑定路由，将其添加到路由集合中，如创建的

		case "POST": // Create a resource.
			var handler restful.RouteFunction
			if isNamedCreater {
				handler = restfulCreateNamedResource(namedCreater, reqScope, admit)
			} else {
				handler = restfulCreateResource(creater, reqScope, admit)
			}
			handler = metrics.InstrumentRouteFunc(action.Verb, group, version, resource, subresource, requestScope, metrics.APIServerComponent, deprecated, removedRelease, handler)
			if enableWarningHeaders {
				handler = utilwarning.AddWarningsHandler(handler, warnings)
			}
			article := GetArticleForNoun(kind, " ")
			doc := "create" + article + kind
			if isSubresource {
				doc = "create " + subresource + " of" + article + kind
			}
			route := ws.POST(action.Path).To(handler).
				Doc(doc).
				Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
				Operation("create"+namespaced+kind+strings.Title(subresource)+operationSuffix).
				Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
				Returns(http.StatusOK, "OK", producedObject).
				// TODO: in some cases, the API may return a v1.Status instead of the versioned object
				// but currently go-restful can't handle multiple different objects being returned.
				Returns(http.StatusCreated, "Created", producedObject).
				Returns(http.StatusAccepted, "Accepted", producedObject).
				Reads(defaultVersionedObject).
				Writes(producedObject)
			if err := AddObjectParams(ws, route, versionedCreateOptions); err != nil {
				return nil, err
			}
			addParams(route, action.Params)
			routes = append(routes, route)

最后才把这些路由添加到webservice中

	for kubeVerb := range kubeVerbs {
		apiResource.Verbs = append(apiResource.Verbs, kubeVerb)
	}

回头看创建POST路由时，同样按照资源是否命名空间级别的创建赌赢的handler,后面则是go-restful创建路由的代码

pod是属于命名空间级别的资源，进入restfulCreateNamedResource函数，经过三层调用到达createHandler函数，调用链如下

restfulCreateNamedResource->handlers.CreateNamedResource->createHandler

createHandler大概逻辑如下

• 从请求中获取资源的namespace,name，GVK等信息
• 从RequestScope中获取资源的反序列化器，将body的数据反序列化为runtimeObject
• 执行mutating准入控制器
• 调用storage的create，同时传入Validate准入控制器，准备持久化到Etcd
• 将处理结果写到响应

代码位于/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go

func createHandler(r rest.NamedCreater, scope *RequestScope, admit admission.Interface, includeName bool) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		timeout := parseTimeout(req.URL.Query().Get("timeout"))
		//从请求中获取资源的namespace,name，GVK等信息
		namespace, name, err := scope.Namer.Name(req)
		gv := scope.Kind.GroupVersion()

		//从RequestScope中获取资源的反序列化器，将body的数据反序列化为runtimeObject
		decoder := scope.Serializer.DecoderToVersion(s.Serializer, scope.HubGroupVersion)

		body, err := limitedReadBody(req, scope.MaxRequestBodyBytes)

		obj, gvk, err := decoder.Decode(body, &defaultGVK, original)

		//调用storage的create，同时传入Validate准入控制器，准备持久化到Etcd
		requestFunc := func() (runtime.Object, error) {
			return r.Create(
				ctx,
				name,
				obj,
				rest.AdmissionToValidateObjectFunc(admit, admissionAttributes, scope),
				options,
			)
		}
		result, err := finishRequest(timeout, func() (runtime.Object, error) {
			if scope.FieldManager != nil {
				liveObj, err := scope.Creater.New(scope.Kind)
				if err != nil {
					return nil, fmt.Errorf("failed to create new object (Create for %v): %v", scope.Kind, err)
				}
				obj = scope.FieldManager.UpdateNoErrors(liveObj, obj, managerOrUserAgent(options.FieldManager, req.UserAgent()))
			}
			//执行mutating准入控制器
			if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && mutatingAdmission.Handles(admission.Create) {
				if err := mutatingAdmission.Admit(ctx, admissionAttributes, scope); err != nil {
					return nil, err
				}
			}
			result, err := requestFunc()
			// If the object wasn't committed to storage because it's serialized size was too large,
			// it is safe to remove managedFields (which can be large) and try again.
			if isTooLargeError(err) {
				if accessor, accessorErr := meta.Accessor(obj); accessorErr == nil {
					accessor.SetManagedFields(nil)
					result, err = requestFunc()
				}
			}
			return result, err
		})

		//将处理结果写到响应
		//如果创建成功的结果按照请求来源时的格式序列化，写到响应体里面
		transformResponseObject(ctx, scope, trace, req, w, code, outputMediaType, result)
	}
}

由此段代码可得，Mutate 准入控制器要比Validate 准入控制器先执行

继续追r.Create方法调用，r.Create==>namedCreaterAdapter.Create-->c.Creater.Create

到Creater.Create是一个接口的调用，这里实现太多，无法单纯通过goland去找到实现。但这个Creater已经是storage的一个接口，在目录中找pod的storage相关定义在 /pkg/registry/core/pod/storage/storage.go中

对应的结构定义如下

type REST struct {
	*genericregistry.Store
	proxyTransport http.RoundTripper
}

它继承于genericregistry.Store，自身并没有再去实现Creater接口了

genericregistry.Store的定义在/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go

所实现的Create方法大概包含下面步骤

• 调用了validate准入控制器验证资源
• 生成name,key等信息用于后续持久化到Etcd
• 创建一个新的空的资源用于成功时返回结果
• 调用storage的Create,准备持久化到Etcd

func (e *Store) Create(ctx context.Context, obj runtime.Object, createValidation rest.ValidateObjectFunc, options *metav1.CreateOptions) (runtime.Object, error) {
	if err := rest.BeforeCreate(e.CreateStrategy, ctx, obj); err != nil {
		return nil, err
	}
	// at this point we have a fully formed object.  It is time to call the validators that the apiserver
	// handling chain wants to enforce.
	//调用了validate准入控制器验证资源
	if createValidation != nil {
		if err := createValidation(ctx, obj.DeepCopyObject()); err != nil {
			return nil, err
		}
	}

	//生成name,key等信息用于后续持久化到Etcd
	name, err := e.ObjectNameFunc(obj)
	if err != nil {
		return nil, err
	}
	key, err := e.KeyFunc(ctx, name)
	if err != nil {
		return nil, err
	}
	qualifiedResource := e.qualifiedResourceFromContext(ctx)
	ttl, err := e.calculateTTL(obj, 0, false)
	if err != nil {
		return nil, err
	}
	//创建一个新的空的资源用于成功时返回结果
	out := e.NewFunc()
	//调用storage的Create,准备持久化到Etcd
	//如果持久化成功，out里面就会填上持久化后的所有信息到里面
	if err := e.Storage.Create(ctx, key, obj, out, ttl, dryrun.IsDryRun(options.DryRun)); err != nil {
		err = storeerr.InterpretCreateError(err, qualifiedResource, name)
		err = rest.CheckGeneratedNameError(e.CreateStrategy, err, obj)
		if !apierrors.IsAlreadyExists(err) {
			return nil, err
		}
		if errGet := e.Storage.Get(ctx, key, storage.GetOptions{}, out); errGet != nil {
			return nil, err
		}
		accessor, errGetAcc := meta.Accessor(out)
		if errGetAcc != nil {
			return nil, err
		}
		if accessor.GetDeletionTimestamp() != nil {
			msg := &err.(*apierrors.StatusError).ErrStatus.Message
			*msg = fmt.Sprintf("object is being deleted: %s", *msg)
		}
		return nil, err
	}
	if e.AfterCreate != nil {
		if err := e.AfterCreate(out); err != nil {
			return nil, err
		}
	}
	if e.Decorator != nil {
		if err := e.Decorator(out); err != nil {
			return nil, err
		}
	}
	return out, nil
}

持久化到Etcd

从e.Storage.Create经过两层调用到达store.Create方法，因为有可能包含dryRun,如果dryRun就不需要持久化到Etcd，在这里将看到

• 将资源转换成无版本类型,即__internal版本
• 再将资源转换成适合存储的格式
• 调用Etcd检查资源是否已经存在了
• 不存在才调用Put把资源存进去
• 成功了才从etcd的响应中把存储结果反序列化成传进来时的格式

代码位于 /vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go

func (s *store) Create(ctx context.Context, key string, obj, out runtime.Object, ttl uint64) error {
	if version, err := s.versioner.ObjectResourceVersion(obj); err == nil && version != 0 {
		return errors.New("resourceVersion should not be set on objects to be created")
	}
	if err := s.versioner.PrepareObjectForStorage(obj); err != nil {
		return fmt.Errorf("PrepareObjectForStorage failed: %v", err)
	}
	//将资源转换成无版本类型
	data, err := runtime.Encode(s.codec, obj)
	if err != nil {
		return err
	}
	key = path.Join(s.pathPrefix, key)

	opts, err := s.ttlOpts(ctx, int64(ttl))
	if err != nil {
		return err
	}

	//再将资源转换成适合存储的格式
	newData, err := s.transformer.TransformToStorage(data, authenticatedDataString(key))
	if err != nil {
		return storage.NewInternalError(err.Error())
	}

	startTime := time.Now()
	//检查资源是否已经存在了
	txnResp, err := s.client.KV.Txn(ctx).If(
		notFound(key),
	).Then(
	//不存在才调用Put把资源存进去
		clientv3.OpPut(key, string(newData), opts...),
	).Commit()
	metrics.RecordEtcdRequestLatency("create", getTypeName(obj), startTime)
	if err != nil {
		return err
	}
	if !txnResp.Succeeded {
		return storage.NewKeyExistsError(key, 0)
	}

	//转换响应结果
	if out != nil {
		putResp := txnResp.Responses[0].GetResponsePut()
		return decode(s.codec, s.versioner, data, out, putResp.Header.Revision)
	}
	return nil
}

至此，资源已落库，创建请求已完毕，apiserver也将结果响应给客户端。

小结

本篇衔接前一篇apiserver的启动流程，讲述了认证器，授权器，准入控制器如何被配置的，如果根据APIGroupInfo映射好的storage创建处理请求的handler。当一个请求来的时候如何执行认证操作，授权操作，接着经过Mutate准入控制器和Validate准入控制器等一系列校验，最终转换资源的版本，调用Etcd客户端将资源持久化，也将结果响应回给客户端。

如有兴趣,可阅读鄙人“k8s源码之旅”系列的其他文章

kubelet源码分析——kubelet简介与启动

kubelet源码分析——启动Pod

kubelet源码分析——关闭Pod

scheduler源码分析——调度流程

apiserver源码分析——启动流程

apiserver源码分析——处理请求