Online learning

github source

Kubeflow实战系列

Prepare

了解ksonnet初探Google之Kubeflow (采用的是0.8.0)
install depinstall ksKubeflow 安利:在 Kubernetes 上进行机器学习
ksonnet功能类似helm kubernetes应用的包管理工具。 可以对比存裸部署sparkKubeflow 入门Jupyter notebook入门教程(上) - CSDN博客 Jupyter notebook教程more)
Docker 入门教程
最新的安装方式:
KSURL="https://github.com/kubeflow/kubeflow/releases/latest"
LASTURL=""
LASTURL=$(curl $KSURL -s -L -I -o /dev/null -w '%{url_effective}')
if (test $? -ne ); then
LASTURL=$(wget -O /dev/null --content-disposition $KSURL >& |awk '/^Location: /{print $2}')
if [ $? -ne ]; then
echo "Error: Can not find the latest ksonnet version by wget."
fi
fi LATES=${LASTURL##*/}
echo $LATES mkdir kubeflow_$LATES
cd kubeflow_$LATES
export KUBEFLOW_TAG=$LATES curl https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/download.sh | bash

echo "Please run:"
echo "export KUBEFLOW_SRC=`pwd`"
echo "export KFAPP=~/kfapp"

cat > `pwd`/kubeflow.env <<EOL
export KUBEFLOW_TAG=$LATES
export KUBEFLOW_SRC=`pwd`
export KFAPP=~/kfapp
EOL

echo "Please run the commnad before doing any kubeflow setting up:"
echo "source `pwd`/kubeflow.env"

------------------------------------------------
以下是陈旧的安装方式源代码安装ksonnet脚本(ksonnet的变化很大, 经常出问题。 不建议源码安装)
go get -u github.com/golang/dep/cmd/dep
go get github.com/ksonnet/ksonnet # Build and install binary under shortname `ks` into $GOPATH/bin
cd $GOPATH/src/github.com/ksonnet/ksonnet
sudo -E -H make install
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig

安装最最新发布版的ksonnet

KSURL="https://github.com/ksonnet/ksonnet/releases/latest"
LASTURL=""
LASTURL=$(curl $KSURL -s -L -I -o /dev/null -w '%{url_effective}')
if (test $? -ne ); then
LASTURL=$(wget -O /dev/null --content-disposition $KSURL >& |awk '/^Location: /{print $2}')
if [ $? -ne ]; then
echo "Error: Can not find the latest ksonnet version by wget."
fi
fi
LATES=""
if [ "$LASTURL" = "$KSURL" ]; then
echo "Error: Not find the latest ksonnet version by curl."
else
if test -n "$LASTURL"; then
echo "OK: Find the latest ksonnet version."
else
echo "Error: Not find the latest ksonnet version by wget."
fi
fi
LATES=${LASTURL##*/}
if [[ $LATES =~ ^v[-.]{,}$ ]]; then
LATES=${LATES#v}
fi
wget https://github.com/ksonnet/ksonnet/releases/download/v$LATES/ks_${LATES}_linux_amd64.tar.gz GOROOT=/usr/local
tar  -C $GOROOT/bin -xzf ks_${LATES}_linux_amd64.tar.gz --wildcards --no-anchored 'ks'
ln -s $GOROOT/bin/ks_${LATES}_linux_amd64/ks $GOROOT/bin/ks
# tar --extract --file=ks_0..0_linux_amd64.tar.gz ks_0..0_linux_amd64/ks

Using Kubeflow (翻译) (中文) (latest EN)

(中文简要介绍)

ks init my-kubeflow
sudo chown $USER:$USER ~/my-kubeflow -R
cd my-kubeflow
ks registry add kubeflow \
github.com/google/kubeflow/tree/master/kubeflow
ks pkg install kubeflow/core
ks pkg install kubeflow/tf-serving
ks pkg install kubeflow/tf-job
ks generate core kubeflow-core --name=kubeflow-core

check  kubectl contexts:

kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://localhost:6443/
name: local-up-cluster
contexts:
- context:
cluster: local-up-cluster
user: local-up-cluster
name: local-up-cluster
current-context: local-up-cluster
kind: Config
preferences: {}
users:
- name: local-up-cluster
user:
client-certificate-data: REDACTED
client-key-data: REDACTED

Jupyter启动之后运行tf的example

Troubleshooting

1.

$ sudo -E -H ks pkg install kubeflow/core
ERROR GET https://api.github.com/repos/google/kubeflow/commits/master: 403 API rate limit exceeded for 134.134.139.74. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 26m21s]

 403 API rate limit of 60 still exceeded

issue 1254

issues 195

   generate one   GITHUB_TOKEN

GTOKEN=801c4f41553571107afa06d459c0910900aeb5fb
GTOKEN=5d67c71ea35d85a5459e6e05a3075363d438b8c7
export GITHUB_TOKEN=$GTOKEN
echo "GITHUB_TOKEN=$GTOKEN" >> ~/.profile

2.  Can not ks apply default -c kubeflow-core with ks 0.9.5

rollback to v0.8.0

@seekyiyi I have the same problem using v0.9.1(or dev build) version of ksonnet, but I use v0.8.0 can be solved.

centos 下安装go

git clone https://github.com/jamesarems/gocentos.git
cd gocentos
sh InstallGO.sh
source /etc/profile
go get -u github.com/golang/dep/cmd/dep
echo 'GOROOT=`go env |grep "GOROOT" |cut -d "=" -f2`
GOROOT=${GOROOT#\"}
GOROOT=${GOROOT%\"}
echo $GOROOT GOPATH=`go env |grep GOPATH |cut -d "=" -f `
GOPATH=${GOPATH%\"}
GOPATH=${GOPATH#\"}
echo $GOPATH
declare -x GOROOT
declare -x GOPATH
export PATH=${PATH}:${GOROOT}/bin:${GOPATH}/bin' >> ~/.bashrc source ~/.bashrc
echo $GOPATH
go get github.com/ksonnet/ksonnet
cd $GOPATH/src/github.com/ksonnet/ksonnet
make install
export KUBECONFIG=/root/config
ks init my-kubeflow

tensorflow 学习

get redirect URL

wget -O /dev/null --content-disposition https://github.com/ksonnet/ksonnet/releases/latest --server-response 2>&1 |   awk '(NR==1){SRC=$3;} /^  Location: /{DEST=$2} END{ print SRC, DEST}'

https://github.com/ksonnet/ksonnet/releases/latest https://github.com/ksonnet/ksonnet/releases/tag/v0.11.0

setup local registry (git hub)

Sharing a local registry with minikube(推荐)

在Kubernetes集群上部署高可用Harbor镜像仓库

docker 私有仓库Harbor

11.部署私有 docker registry

Kubernetes镜像仓库-Harbor的Helm部署

在kubernetes 集群上搭建docker 私有仓库Harbor

Get latest Github app version:

# cat github-latest.sh

#!/bin/bash
URL=$
APPURL=${URL%/releases/latest*}
APP=${APPURL##*/}
LASTURL=""
LASTURL=$(curl $URL -s -L -I -o /dev/null -w '%{url_effective}')
if (test $? -ne ); then
LASTURL=$(wget -O /dev/null --content-disposition $URL >& |awk '/^Location: /{print $2}')
if [ $? -ne ]; then
echo "Error: Can not find the latest $APP version by wget."
fi
fi
LATES=""
if [ "$LASTURL" = "$URL" ]; then
echo "Error: Not find the latest $APP version by curl."
else
if test -n "$LASTURL"; then
echo "OK: Find the latest $APP version."
else
echo "Error: Not find the latest $APP version by wget."
fi
fi
LATES=${LASTURL##*/}
if [[ $LATES =~ ^v[-.]{,}$ ]]; then
LATES=${LATES#v}
fi
echo "Version:$LATES"
exit
URL=https://github.com/goharbor/harbor/releases/latest
V=`./github-latest.sh $URL|grep Version: |cut -d ":" -f2` BIN=https://storage.googleapis.com/harbor-releases/harbor-offline-installer-v$V.tgz wget $BIN

harbor 官网使用, 使用k8s部署

doker official setup registry

k8s tensorflow的更多相关文章

  1. TensorFlow的开源与Hadoop的开源

    最近看TensorFlow代码的时候,用Git pull下来最新的master一看,哇好多的更新,然后点击去之前看到一半的cc文件继续看,好多地方都改变了.但是一看Git log,有好多巨大的comm ...

  2. Tensorflow serving with Kubernetes

    1.Build docker image 由于自己build镜像总是遇到问题,此处暂时借用dockerhub上的一个镜像 docker.io/mochin/tensorflow-serving 将此镜 ...

  3. kubeflow 创建tensorflow过程

    online deployable ,install k8s 代码 Kubeflow有三个核心组件 TFJob Operator 和 Controller: 作为Kubernetes的扩展,来简化分布 ...

  4. 『TensorFlow』卷积层、池化层详解

    一.前向计算和反向传播数学过程讲解

  5. 使用NNI的scikit-learn以及Tensorflow分析

    一.NNI简介 NNI (Neural Network Intelligence) 是自动机器学习(AutoML)的工具包. 它通过多种调优的算法来搜索最好的神经网络结构和(或)超参,并支持单机.本地 ...

  6. K8S全栈容器服务如何助力企业云化创新?

    容器编排管理平台Kubernetes在实践两年多后,市场主导地位被正式确定,随着首批认证服务商的宣布,围绕着容器的应用编排部署服务已然成熟,Kubernetes开始在商业场景为企业创造价值.华为云在K ...

  7. Tensorflow Serving介绍及部署安装

    TensorFlow Serving 是一个用于机器学习模型 serving 的高性能开源库.它可以将训练好的机器学习模型部署到线上,使用 gRPC 作为接口接受外部调用.更加让人眼前一亮的是,它支持 ...

  8. 从零开始入门 K8s | 可观测性:监控与日志

    作者 | 莫源  阿里巴巴技术专家 一.背景 监控和日志是大型分布式系统的重要基础设施,监控可以帮助开发者查看系统的运行状态,而日志可以协助问题的排查和诊断. 在 Kubernetes 中,监控和日志 ...

  9. 一文看懂 K8s 日志系统设计和实践

    上一篇中我们介绍了为什么需要一个日志系统.为什么云原生下的日志系统如此重要以及云原生下日志系统的建设难点,相信DevOps.SRE.运维等同学看了是深有体会的.本篇文章单刀直入,会直接跟大家分享一下如 ...

随机推荐

  1. Python的Matplotlib库简述

    Matplotlib 库是 python 的数据可视化库import matplotlib.pyplot as plt 1.字符串转化为日期 unrate = pd.read_csv("un ...

  2. ASP.NET Core 启动流程图

    简洁描述: 一   WebHostBuilder.Build() =>1注入公共的实例 2创建WebHost实例 3注入自定义实例 4创建IServer 5添加中间件(_components集合 ...

  3. 数据加密之RijndaelManaged加密

    #region RijndaelManaged加密 /// <summary> /// 加密数据 /// </summary> /// <param name=" ...

  4. [提权] 脏牛漏洞 Dirty COW CVE-2016-5195 2.6.22 < 3.9 (x86/x64)

    /* * (un)comment correct payload first (x86 or x64)! * * $ gcc cowroot.c -o cowroot -pthread * $ ./c ...

  5. 深入解析Java反射(1) - 基础

    深入解析Java反射(1) - 基础 最近正筹备Samsara框架的开发,而其中的IOC部分非常依靠反射,因此趁这个机会来总结一下关于Java反射的一些知识.本篇为基本篇,基于JDK 1.8. 一.回 ...

  6. JSP知识点

    1.九大内置对象: request HttpServletRequest类的实例 response HttpServletResponse类的实例 out PrintWriter类的实例,用于把结果输 ...

  7. 列表选择模式:单选、按shift、按shift或ctrl

    2018-10-29 21:23:16 开始写 import java.awt.BorderLayout; import java.awt.EventQueue; import javax.swing ...

  8. Python - 1. Built-in Atomic Data Types

    From:http://interactivepython.org/courselib/static/pythonds/Introduction/GettingStartedwithData.html ...

  9. Swift之关键字使用(I)

    static和class的使用 static 使用 在非class的类型(包括enum和struct)中,一般使用static来描述类型作用域.在这个类型中,我们可以在类型范围中声明并使用存储属性,计 ...

  10. mitmproxy 中间人攻击的小玩笑

    import mitmproxy.http from mitmproxy import ctx, http class Joker: def request(self, flow: mitmproxy ...