k8s（openshift） 部署istio1.1

准备工作：

openshift 默认不允许UID为0的容器运行，要先授权scc以便安装istio

# oc adm policy add-scc-to-user anyuid -z istio-ingress-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z default -n istio-system
# oc adm policy add-scc-to-user anyuid -z prometheus -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-egressgateway-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-citadel-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-ingressgateway-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-cleanup-old-ca-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-mixer-post-install-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-mixer-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-pilot-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-sidecar-injector-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-galley-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-security-post-install-account -n istio-system

下载istio包

# curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.1.6 sh -

下载Helm工具

# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz

# tar -zvxf helm-v2.13.1-linux-amd64.tar.gz
# cp linux-amd64/* /usr/bin/

安装istio：

初始化，向Kubernetes api-server提交CDR

# kubectl create namespace istio-system
# helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -　　

验证CDR是否提交成功，数量为53

# kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l

安装核心组件

# helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

尝试注入:

istio组件需要privileged权限，否则无法创建Pod

# oc adm policy add-scc-to-user privileged -z default -n dev

openshift注入设置，配置Webhook和证书签名

# vim /etc/origin/master/master-config.patch
admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission

# cd /etc/origin/master/
# cp -p master-config.yaml master-config.yaml.prepatch
# oc ex config patch master-config.yaml.prepatch -p "$(cat master-config.patch)" > master-config.yaml
# master-restart api
# master-restart controllers

自动注入（默认配置）：

给namespace绑定注入标签，即使是手动注入也要绑定标签

# oc label  namespace dev istio-injection=enabled
# oc get namespace -L istio-injection
NAME                                STATUS    AGE       ISTIO-INJECTION
app-storage                         Active    21h
default                             Active    21h
dev                                 Active    5h        enabled

关闭特殊Pod的自动注入，比如OpenShift Builds完全没必要注入istio

修改istio-system下的ConfigMap istio-sidecar-injector，加入以下内容

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-sidecar-injector
data:
  config: |-
    policy: enabled
    neverInjectSelector:
      - matchExpressions:
        - {key: openshift.io/build.name, operator: Exists}
      - matchExpressions:
        - {key: openshift.io/deployer-pod-for.name, operator: Exists}
    template: |-
      initContainers:
...

手动注入：

修改istio-system下的ConfigMap istio-sidecar-injector，关闭自动注入

policy: disabled

修改需要注入的Deployment配置

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ignored
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: ignored
        image: tutum/curl
        command: ["/bin/sleep","infinity"]

如果sidecar.istio.io/inject=false  即使policy: enabled 也不会注入

排错：

• Pod无法创建

检查scc privileged 是否给当前空间的default用户授权

• 无法创建openshift Deployment 或者 Builds

Error creating deployer pod: pods "nginx-20-deploy" is forbidden: unable to validate against any pod security policy: [spec.initContainers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000080000, 1000089999] spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: "NET_ADMIN": capability may not be added spec.containers[1].securityContext.securityContext.runAsUser: Invalid value: 1337: must be in the ranges: [1000080000, 1000089999]]

直接排除这些系统Pod或者授权scc

# oc adm policy add-scc-to-user privileged -z deployer -n dev
# oc adm policy add-scc-to-user privileged -z builder -n dev

• Pod能成功创建但是istio-init容器一直是CrashLoopBackOff

这是因为istio-init容器需要特权模式，需要修改容器模板 istio-system/configmap/istio-sidecar-injector

- name: istio-init
   securityContext:
      privileged: true 

• istio注入后容器不能访问外部网络

这是因为istio默认劫持所有流量，需要把外部网络地址排除掉，最简单的方式就是只包含k8s内部网络

修改istio-system/configmap/istio-sidecar-injector

    - "-i"
    - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges`  "172.30.0.0/,10.128.0.0/"  ]]"
    - "-x"
    - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges`  ""  ]]"