物理机Jenkins接入K8s环境

前言

本次记录物理机部署Jenkins，k8s弹性伸缩agent节点供部署项目。

安装 K8S 插件

登录 Jenkins，系统管理→ 插件管理 → 搜索 kubernetes，选择第二个 Kubernetes，点击 安装，安装完成后重启 Jenkins 。

申请 K8S 凭据

因为 Jenkins 服务器在 kubernetes 集群之外，所以我们准备以下文件才能从外面连接到 kubernetes 集群。

登录 Jenkins，系统管理→ 证书管理 → 全局凭据→添加凭据

配置云kubernetes连接K8S集群的验证文件

kubectl config view --raw -o json

把获取到的数据，分别生成ca.crt client.crt client.key

echo "LS0tLS1CRUdJTiBDRVJUSUZJVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EZ3lNekEzTXpnd05WaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS09JClJOMmpweXJLK2M1ZktoMVpWbE1KdVppUXEzMFNVRzU5dU51SEJxaUJKVnZzd0syWFhxNXZZOUc1KzEzUTJ2TisKYTIrTm4wcDlDT0xtSitJZVd1V3VMQXZUdUx4RlRPZEd0MUJsQ3FGSkR5RGd2S3hUaHUwVHVCdEp1eUpuNDVBRgpLSTk2UysrRjRpSWU2aW1VSzgydXM3WUdBNjVOSkJFUnlZN1hTQVMyTWlJbUJDSnFmYkVlbVFTRDJXcm1iYlhsCk1Cdmd6VUFGUUdvbUFOYTNmQklQM2QvWWxGZTZqNlJDTmhUSGhqak1BUGk3WUpwcEpzTjZCTDBWYVQ3MTVEYVoKTVI3blo2cVdmMk5RS3k2bGV3MjZxUERzaVFWaCtZS1NxVDdzaWVXSm11a0F3UEpoY2lJL2tGUmd0eXYxSGozWAp4WVNQU2RsOURoUXp0WGVFckljQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZDN2xFcldSQ3QxVUtqWFAxMENpN2pacVpOcEJNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBR3cwR01hcWY5MUlZYjJ4NDVqYgpueVZ6ZkZDSWZuTnA4R1JBbmFCUFpWVFM0ZGJnQ0dUTHZvN3pWUWtDendwMGJaeC9nT0t5ek5hakw0bU5MOEw2CjIxdnYxRzVzV29TTTFDbDh6clM1NVNKTjZwWnlnYUgzSkNFTlB3cDlIb0xEQjdka3gzS2dOZjJQWGJRM1ZRQjQKZ3dWTVFTbmhMRCtLV3hPV3phYjRDdnMra2tEVm81aEY5cVRuTitxY1hKM3A5UG1YaEkvQjVhd2UxME5PUTdDcwpyTzVUYzJKS1dhZUV2ZGx0MmZ6dEQ3Lyt5UlNLRWxML3AzczFtUVlCTkllMWh1anFnS3hVamFKZE40UmJwUk9wCk9pVmVGMmhmVVNycUNtQldhSjY5b2p0cEhUSUtY1MFAKUE5jPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==" |base64 -d > ca.crt

echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJVENDQWdtZMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTlaUmw0bytSZnZBMUNmWUwKbTU1R3hmMFpnYWoySnc4b1hsWUVVU2VJcC82VU14SS9XUXhNWTVxNFFXcUdXYVNDZXV3cHhqTFU2MXlkcmVOcApjeFM4TUpIQXJ6QXJ1cGhhb1FiZkdqdkNHRldHMEtPcUlNcGhMemtiTmRJSTNBOThjV1kwWTBRcnNQaEtHaFVQCmd3ejJYWFhQWFFLSVlLSWN1cGRnbWJKeUpPakpEU01ZQUJJQjZqU3VabkFQeTVaTzNJMTBwT0JNSXJKTVhCY3EKYk1LZDNKRkl5Tm5jNFdpckd2cks2Q2xWUXJYUzY5bDd0c2lVWDhvekZLdEd2ZVllSTA0SmpJWWdQOG5CdnJ2dApMSGpibksrM1pTNUVmV0pQQmtZc0VxK3RtODBtcDRkSzhpVkFZbUVEeVZnS2JoWVZYckpjemcyN3lnZEY1Y3dzCllzU1dUUUlEQVFBQm8xWXdWREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RBWURWUjBUQVFIL0JBSXdBREFmQmdOVkhTTUVHREFXZ0JRdTVSSzFrUXJkVkNvMXo5ZEFvdTQyYW1UYQpRVEFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBSmNwRUlNc0FGT2tzOXdvVzliNmVSc2lyNkpKRFRKMGhTeFp6ClR0Q2RxdGRZT252ZG82RVRQVVlkMkQxU0dJbHcxQ3JLSHF1OEREb0F4K1RqRjhoYjlobkpNTzZsWUlsWFNtM3YKcDcrd0RSTncyZlRjS2pxWStHSWZxRE9WQUlqREFyaWJFVmprK2pkRW8vU3hBOWc4YjJySzRtZTV0Z1JrSFZuQwozNUZodG1JUkhlbnJNMEV5c3JibDFOdGdTdktQLzJ4a0EwaGp5ckNib1puWjhOVk1RTXk5ZURMRmhjMDNqb0tLCmNKSFdNVzdEZVpGVlBLZGdqYXVMUStObkQwUTVkOUdGK1p3cHNtMTVDd1RRZzVUQSt5cW9CNE0vbTJzelRuRXIKNjlNUG5WckxNdmJGVlNDQjJybTYmZiSHJhZWhKNkFMV3BPRDllNHZWN0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==" |base64 -d > client.crt

echo "LS0tLS1CRUdJTiBSU9RYmZHanZDR0ZXRzBLT3EKSU1waEx6a2JOZejRSSFpyMUEKS2NnSW95VWsvUEFnVm8xQVA1Rm1mcHl3SGNNT3JXYktRT25JbG5FbWFLRWZNSGo0eVRSOGlJdUpNcUV0a0NiYwpyclZyUVdVR2VIY1FaaERkdkE0cWtFNXJVNlo1aXBaYjBoaXUvTTBRUGlkR1VGbkVBTlZLdzgrbGJCYjV6a3BwCnpOQW1YYnAvc0Nha29VZ1kzWjB6cDNueEZ0eWpPbXFJdlZlaGlVRStvMlBxaytyVmFaN0g0THlKWlY1RUZoTWMKeEhKTFVGMENnWUVBK1pybWdaZEJib0twKzBlS0g4SFlyRmtKZmU5bS9heU9KUmh0TFR0NjJtaitlaTRTS09IMgp3ekdySmoreEtaeGkwNGs0eGZDNG81VEI2YUhvZDFNS0lRbWRabTgvWE1kOEUxNVUweE4rZkF0S0diL09rd1ExCmx2Z3RvWHpDTVpaTUkyeXUwTnc4OWhCOXR0STNjcDlOTU1yb1dNMTVqRFdoZ0s5enMwaE85bThDZ1lFQSs5OFkKajhjNHJrNjZnd0RjY3Q0dmZGaUU4VjdwdlF0MDJUd0xFZGw4YkM0OXRNY3pTYjAzR0xuL0UzanFlWkJJTlpOTQpacG1oVHhFNFMxejZwaC80VXp2VUVBMVRFdTcrS0xEelF3TlRTWXJmdkpYYXc5eXZOeVRNUGM4L3ZsVWZSV1BGCllKbW9WU0toN1UxYVNhY1BFQTB5TVVIS3pZcW93SFo0VTRwMC9RTUNnWUVBbWlwQVFMNUpYWlk2b3NiMjJ0S0IKbjBHSENVOTZUeWZxeTZLZ25CVm10Y3V2a1lzdkVDMHdtakx0UUx5S0VlZFRhdEFRVHNGMWFTT1IrVmxWR1k0aQpiWEZnMUVuVjJ0b2pHbHRqRE5PM2hhbjdaSmdCWVBCeW9RSThQTWV2c2F6ZzdlaE11UmV1VjFTc01PVDU5a1BLCkMxN256eUZwOTEyUW1NakZMM0J6eE9VQ2dZRUFuT3ZuQXBVTmRGUHFrZm4wWHNDYzl5eXpUeXBpRi96T1JSWEsKazhsYllvOEF0T095eVl0UWkyQStVbVJ3T1dLeUNtY0dINW94elBBZWtaMlJML1VWSk1idnhMVmpIWHF2VFJHblg2WnRBQ0FsRHptczVXYVpGblViVnNjdTRqSFVXYTRBMXZwWnZvPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=" |base64 -d > client.key

生成client P12认证文件cert.pfx,用于配置Jenkins连接k8s集群的认证凭据

生成Client P12认证文件cert.pfx，并下载至本地:

openssl pkcs12 -export -out ./cert.pfx -inkey ./client.key -in ./client.crt -certfile ./ca.crt
Enter Export Password:123456
Verifying - Enter Export Password:123456

上传cert.pfx到Jenkins页面，输入刚才设置的密码。

配置 K8S 集群的对接

登录 Jenkins，点击 系统管理 → 系统配置 → Manager Nodes and Clouds

填写以下内容

• Kubernetes 地址：kubernetes服务地址，也就是 apiserver 的地址，一般是master 节点 NodeIP+6443 端口

• Kubernetes 服务证书 key：刚才生成的ca.crt 文件的内容

• 凭据：刚才创建的 certificate 凭据

• Jenkins 地址：Agent 连接 Jenkins Master 的地址

启用50000代理端口

创建一个pipeline脚本构建项目

定义参数

def label = "jenkinsagent-${JOB_NAME}-${BUILD_NUMBER}"
def GIT_ADDR = "http://git地址/special_provider.git"

定义选项参数和动态选项参数

properties([
    [$class: 'RebuildSettings', autoRebuild: false, rebuildDisabled: false],
    buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '6')),
    parameters([
        choice(choices: ['uat','prod'], description: '选择命名空间', name: 'NAMESPACE'),
        choice(choices: ['sp-oms-webapi','sp-provider','sp-webapi'], description: '选择发布项目', name: 'project'),
        [
            $class: 'CascadeChoiceParameter',
            choiceType: 'PT_RADIO',
            filterLength: 1,
            filterable: false,
            name: 'POMDIR',
            description: '选择项目目录',
            randomName: 'choice-parameter-9129716595374480',
            referencedParameters: 'project',
            script: [
                $class: 'GroovyScript',
                fallbackScript: [
                    classpath: [ ],
                    sandbox: false,
                    script: 'return["ERROR"]'
                ],
                script: [
                    classpath: [ ],
                    sandbox: false,
                    script: '''\
                       if(project.equals("sp-oms-webapi")){
                           return ["sp-oms-webapi:selected"]
                       }else if(project.equals("sp-provider")){
                           return ["sp-rpc-provider:selected"]
                       }else if(project.equals("sp-webapi")){
                           return ["sp-webapi:selected"]
                       }
                    '''
                ]
            ]
        ]
    ])
])

配置Jenkins的pod模板

podTemplate(
    label: label, cloud: 'kubernetes',
    containers:[
        containerTemplate(name: 'jnlp', image: 'hubar地址/jenkins-slave-maven:v1',ttyEnabled:true)
    ],
    volumes:[
        hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
        hostPathVolume(mountPath: '/etc/localtime', hostPath: '/etc/localtime'),
        hostPathVolume(mountPath: '/usr/bin/docker',hostPath: '/usr/bin/docker'),
        hostPathVolume(mountPath: '/usr/bin/kubectl',hostPath: '/usr/bin/kubectl'),
        nfsVolume(mountPath: '/maven/LocalStore',serverAddress: '10.0.2.85',serverPath: '/maven/LocalStore', readOnly: false),
    ]
)

执行拉代码、编译、部署

{
    node(label) {
        stage('checkout'){
            checkout([$class: 'GitSCM',
                branches: [[name: 'release']],
                doGenerateSubmoduleConfigurations: false,
                extensions: [],
                submoduleCfg: [],
                userRemoteConfigs: [[credentialsId: 'ee2e4a25-0f67-4b1a-99e6-98749064f85c', url: "${GIT_ADDR}"]]
            ])
        }

        stage('complie'){
            ansiColor('xterm') {
                echo "\u001B[34m 执行mvn编译 \u001B[0m"
            }
            container('jnlp'){
                def build_version = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
                sh label: '', script: "mvn -q -B -f ${POMDIR}/pom.xml -DskipTests=true clean install -Daliyun=central -P$NAMESPACE docker:build -Dbuild_version=release-$build_version -Dbuild_tag=$BUILD_TAG"
            }
        }

        stage('deploy'){
            container('jnlp'){
                withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'db6edcf0-d9a4-4b9b-b92e-898e64665ed3', namespace: '$NAMESPACE', serverUrl: 'https://kubernetes.default.svc.cluster.local') {
                    sh label: '', script: "kubectl apply -f ${POMDIR}/target/docker/k8s/deployment.yml"
                }
            }
            script{
                buildName "#${BUILD_NUMBER}-${BUILD_USER}"
                buildDescription "选择项目:${project} 环境:${NAMESPACE}"
            }
        }
    }
}