使用s3-sftp-proxy 暴露minio s3 数据为sftp 访问

尽管s3 很不错，但是ftp 也有自己存在的价值，以下是一个简单的通过s3-sftp-proxy 暴露minio s3 数据为ftp 的访问方式

环境准备

• docker-compose 文件

version: "3"

services: 

  minio:

    image: minio/minio

    command: server /export

    ports:

    - "9000:9000"

    environment:

    - "MINIO_ACCESS_KEY=dalongdemo"

    - "MINIO_SECRET_KEY=dalongdemo"

  s3-proxy:

    build: ./

    image: dalongrong/s3-sftp-proxy

    command: -debug

    ports: 

    - "10022:10022"

    volumes: 

    - "./host_key:/app/host_key"

    - "./s3-sftp-proxy.example.toml:/app/s3-sftp-proxy.example.toml"

• s3-sftp-proxy 项目说明
  项目fork 了https://github.com/moriyoshi/s3-sftp-proxy 添加了go mod 的支持同时添加了docker 镜像
  go mod 转换处理

 

git clone https://github.com/moriyoshi/s3-sftp-proxy.git

cd s3-sftp-proxy 

go mod init 

go mod tidy

 

本地构建

go  build 

docker 镜像通过多阶段构建

FROM golang:alpine as build

WORKDIR /app

ENV GOPROXY=https://mirrors.aliyun.com/goproxy/

RUN apk update && apk add wget unzip build-base git bzr mercurial gcc 

RUN git clone https://github.com/moriyoshi/s3-sftp-proxy.git

RUN cd s3-sftp-proxy && go mod init && go mod tidy && go build 

​

FROM alpine:latest

LABEL EMAIL="1141519465@qq.com"

LABEL AUTHOR="dalongrong"

WORKDIR /app

ENV BIND=:10022

ENV CONFIG=/app/s3-sftp-proxy.example.toml

EXPOSE 10022

COPY entrypoint.sh /app/entrypoint.sh

RUN chmod +x /app/entrypoint.sh

ENV PATH=$PATH:/usr/local/bin

COPY s3-sftp-proxy.example.toml /app/s3-sftp-proxy.example.toml

COPY --from=build /app/s3-sftp-proxy/s3-sftp-proxy /usr/local/bin/s3-sftp-proxy

ENTRYPOINT ["./entrypoint.sh"]

 

• s3-sftp-proxy demo 配置文件
  s3-sftp-proxy.example.toml 文件

host_key_file = "./host_key"

[buckets.test]

endpoint = "http://minio:9000"

s3_force_path_style = true

disable_ssl = true

bucket = "demoapp"

region = "us-east-1"

max_object_size = 65536

writable = true

readable = true

listable = true

auth = "test"

keyboard_interactive_auth = false

​

[buckets.test.credentials]

aws_access_key_id = "dalongdemo"

aws_secret_access_key = "dalongdemo"

​

[auth.test]

type = "inplace"

​

[auth.test.users.user01]

password = "test"

[auth.test.users.user02]

password = "test"

 

 

启动&&测试

• 启动

docker-compose up -d

• 效果

• sftp 连接

sftp -P 10022 user01@localhost

使用goreleaser 构建以及发布

• goreleaser 说明
  goreleaser 是一个很不错的golang 软件构建 工具，可以快速的构建支持跨平台的二进制软件包，同时还支持rpm，deb 包的制作
• 初始化goreleaser 项目
  goreleaser 提供了方便的cli 命令可以快速生成构建配置文件

 

goreleaser init

• 修改文件,添加多平台支持
  .goreleaser.yml

 

.# This is an example goreleaser.yaml file with some sane defaults.

# Make sure to check the documentation at http://goreleaser.com

project_name: s3-sftp-proxy

before:

  hooks:

    # you may remove this if you don't use vgo

    - sh goproxy.sh

    - go mod download

builds:

- env:

  - CGO_ENABLED=0

  goos:

  - freebsd

  - windows

  - darwin

  goarch:

  - amd64

  - arm

  - arm64

archive:

  replacements:

    darwin: Darwin

    linux: Linux

    windows: Windows

    386: i386

    amd64: x86_64

checksum:

  name_template: 'checksums.txt'

changelog:

  sort: asc

  filters:

    exclude:

    - '^docs:'

    - '^test:'

 

说明
添加了一个hooks，方便支持goproxy 的加速构建

• 本地构建
  注意goreleaser 是基于tag 的构建管理，你需要包含tag 分支可以使用git tag 进行操作

goreleaser release --skip-publish --rm-dist

效果

• github release 支持
  goreleaser 可以快速的发布构建好的二进制文件到github，但是提前是需要 GITHUB_TOKEN ,可以直接在github 创建
  添加 GITHUB_TOKEN 环境变量

export GITHUB_TOKEN=xxxxxxxxxxxx

发布

goreleaser release --rm-dist

效果

   • releasing using goreleaser 0.101.0...

   • loading config file file=.goreleaser.yml

   • RUNNING BEFORE HOOKS

      • running sh goproxy.sh

      • running go mod download

   • GETTING AND VALIDATING GIT STATE

      • releasing v1.9, commit e7fa48b35d11406c2a7d8945cfa450865264aba3

   • PARSING TAG      

   • SETTING DEFAULTS 

      • LOADING ENVIRONMENT VARIABLES

      • SNAPSHOTING      

      • GITHUB RELEASES  

      • PROJECT NAME     

      • ARCHIVES         

      • BUILDING BINARIES

      • LINUX PACKAGES WITH NFPM

      • SNAPCRAFT PACKAGES

      • CALCULATING CHECKSUMS

      • SIGNING ARTIFACTS

      • DOCKER IMAGES    

      • ARTIFACTORY      

      • S3               

      • HOMEBREW TAP FORMULA

      • SCOOP MANIFEST   

   • SNAPSHOTING      

      • pipe skipped error=not a snapshot

   • CHECKING ./DIST  

      • --rm-dist is set, cleaning it up

   • WRITING EFFECTIVE CONFIG FILE

      • writing config=dist/config.yaml

   • GENERATING CHANGELOG

      • writing changelog=dist/CHANGELOG.md

   • LOADING ENVIRONMENT VARIABLES

   • BUILDING BINARIES

      • building binary=dist/darwin_amd64/s3-sftp-proxy

      • building binary=dist/windows_amd64/s3-sftp-proxy.exe

      • building binary=dist/freebsd_amd64/s3-sftp-proxy

      • building binary=dist/freebsd_arm_6/s3-sftp-proxy

   • ARCHIVES         

      • creating archive=dist/s3-sftp-proxy_1.9_Darwin_x86_64.tar.gz

      • creating archive=dist/s3-sftp-proxy_1.9_Windows_x86_64.tar.gz

      • creating archive=dist/s3-sftp-proxy_1.9_freebsd_x86_64.tar.gz

      • creating archive=dist/s3-sftp-proxy_1.9_freebsd_armv6.tar.gz

   • LINUX PACKAGES WITH NFPM

   • SNAPCRAFT PACKAGES

      • pipe skipped error=no summary nor description were provided

   • CALCULATING CHECKSUMS

      • checksumming file=s3-sftp-proxy_1.9_Darwin_x86_64.tar.gz

      • checksumming file=s3-sftp-proxy_1.9_freebsd_x86_64.tar.gz

      • checksumming file=s3-sftp-proxy_1.9_freebsd_armv6.tar.gz

      • checksumming file=s3-sftp-proxy_1.9_Windows_x86_64.tar.gz

   • SIGNING ARTIFACTS

      • pipe skipped error=artifact signing is disabled

   • DOCKER IMAGES    

      • pipe skipped error=docker section is not configured

   • PUBLISHING       

      • S3               

         • pipe skipped error=s3 section is not configured

      • HTTP PUT         

         • pipe skipped error=put section is not configured

      • ARTIFACTORY      

         • pipe skipped error=artifactory section is not configured

      • DOCKER IMAGES    

      • SNAPCRAFT PACKAGES

      • GITHUB RELEASES  

         • creating or updating release repo=rongfengliang/s3-sftp-proxy-docker tag=v1.9

         • release updated url=https://github.com/rongfengliang/s3-sftp-proxy-docker/releases/tag/v1.9

         • uploading to release file=dist/checksums.txt name=checksums.txt

         • uploading to release file=dist/s3-sftp-proxy_1.9_Darwin_x86_64.tar.gz name=s3-sftp-proxy_1.9_Darwin_x86_64.tar.gz

         • uploading to release file=dist/s3-sftp-proxy_1.9_freebsd_x86_64.tar.gz name=s3-sftp-proxy_1.9_freebsd_x86_64.tar.gz

         • uploading to release file=dist/s3-sftp-proxy_1.9_Windows_x86_64.tar.gz name=s3-sftp-proxy_1.9_Windows_x86_64.tar.gz

         • uploading to release file=dist/s3-sftp-proxy_1.9_freebsd_armv6.tar.gz name=s3-sftp-proxy_1.9_freebsd_armv6.tar.gz

      • HOMEBREW TAP FORMULA

         • pipe skipped error=brew section is not configured

      • SCOOP MANIFEST   

         • pipe skipped error=scoop section is not configured

   • release succeeded after 26.53s

参考资料

https://github.com/moriyoshi/s3-sftp-proxy
https://github.com/rongfengliang/s3-sftp-proxy-docker
https://goreleaser.com/
https://github.com/goreleaser/goreleaser