java实现简单的oss存储

oss

工作中需要用到文件上传，之前使用的是本地文件系统存储方式，后来重构为支持多个存储源的方式，目前支持三种方式：local、seaweedfs、minio

存储介质

seaweedfs

seaweedfs是一款go语言开发的轻量级、高性能的存储服务器。

https://github.com/chrislusf/seaweedfs

# 启动 master
docker run -d \
-p 9333:9333 \
-p 19333:19333 \
-v /home/mining/report-cloud/docker/seaweedfs/master/data:/data \
--name seaweedfs-master \
chrislusf/seaweedfs:1.53 \
master -ip=master

# 启动 volume
docker run -d \
-p 8080:8080 \
-p 18080:18080 \
-v /home/mining/report-cloud/docker/seaweedfs/volume01/data:/data \
--name seaweedfs-volume \
--link seaweedfs-master:master \
chrislusf/seaweedfs:1.53 \
volume -max=5 -mserver="master:9333" -port=8080

访问web地址：http://localhost:9333/

简单使用：

# 申请空间
curl http://localhost:9333/dir/assign
#
{"count":1,"fid":"4,017f52110b","url":"127.0.0.1:8080","publicUrl":"localhost:80
80"}
# 推送文件，4,017f52110b表示第4个volume，017f52110b表示文件的唯一标识
curl -F file=@/home/seaweedfs/balance.png http://127.0.0.1:8080/4,017f52110b

minio

MinIO是与Amazon S3 API兼容的高性能对象存储服务器，提供了人性化的管理页面

https://github.com/minio/minio

docker run --name report-minio \
-p 19000:9000 \
-e "MINIO_ACCESS_KEY=qweasdzxc" \
-e "MINIO_SECRET_KEY=1234567890" \
-v /home/mining/report-cloud/docker/minio:/data \
-d minio/minio:latest server /data

# web 管理页面
http://localhost:19000/minio/login
# 创建一个 bucket 为 report
# 通过URL直接访问文件需要设置权限，参考下面博客
# https://blog.csdn.net/iKaChu/article/details/105809957
# 访问格式为
http://localhost:19000/bucket名/对象名

Java代码

maven依赖

<!-- oss 存储的依赖 -->
<!-- seaweedfs 依赖 -->
<dependency>
  <groupId>org.lokra.seaweedfs</groupId>
  <artifactId>seaweedfs-client</artifactId>
  <version>0.7.3.RELEASE</version>
</dependency>

<!-- minio 依赖 -->
<dependency>
  <groupId>io.minio</groupId>
  <artifactId>minio</artifactId>
  <version>3.0.10</version>
</dependency>

代码

配置类

FileTag 枚举类

用来标记使用哪种存储介质

/**
 * 文件上传的tag
 */
public enum FileTag {
    TYPE_LOCAL,
    TYPE_MINIO ,
    TYPE_SEAWEEDFS;
}

FileUploadConfigLocal

本地存储配置类

public class FileUploadConfigLocal {
    private String parentPath;
    // 省略 get set
}

FileUploadConfigSeaweedfs

seaweedfs存储配置类

public class FileUploadConfigSeaweedfs {
    private String host;
    private int port = 9333;
    private String publicUrl;
    private int connectTimeout = 60; // 60 s
    // 省略 get set
}

FileUploadConfigMinio

minio存储配置类

public class FileUploadConfigMinio {
    private String domain; // url 前缀
    private String accesKey;
    private String secretKey;
    private String bucket; // 必须事先创建好
    // 省略 get set
}

FileUploadConfig

spring application.yaml 配置类

import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties( prefix = "oss-config")
public class FileUploadConfig {
    FileTag tag;
    FileUploadConfigLocal local;
    FileUploadConfigSeaweedfs seaweedfs;
    FileUploadConfigMinio minio;
    // 省略 get set
}

配置文件

application.yaml

# 对象存储配置
oss-config:
  tag: TYPE_LOCAL # 使用哪种存储介质
  local:
    parent-path: D:\dev\2019-06-19\upload
  minio:
    domain: http://localhost:19000
    bucket: report
    acces-key: qweasdzxc
    secret-key: 1234567890
  seaweedfs:
    host: localhost
    port: 9333
    publicUrl: http://localhost:8080
    connectTimeout: 3000 # 5min

FileUploadInfoVO文件上传类

public class FileUploadInfoVO {

    private String fileName; // 文件名
    private String suffix; // 文件后缀
    private String contentType; // 文件类型
    private Long size; // 文件大小
    private String fid; // 文件唯一ID，同时也是保存在服务器上的文件名
    private boolean delete; // 是否删除，默认false
    private String description;
    private String url; // 完整的url
    private String tag; // 标签，比如
    private String uploadTime;
    private String deleteTime;
    // 省略 get set
}

业务逻辑类

FileStorageService 文件存储接口

/**
 * 文件存储的接口
 */
public interface FileStorageService {
    FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception ;
    FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception ;
    FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception ;
}

本地存储实现

@Service("FileStorageServiceLocal")
public class FileStorageServiceLocal implements FileStorageService {

    @Autowired
    FileUploadConfig fileUploadConfig;

    @PostConstruct
    public void init() {
        FileUtils.checkAndCreateDir(fileUploadConfig.getLocal().getParentPath());
    }

    @Override
    public FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        String filePath = fileUploadConfig.getLocal().getParentPath() + File.separator + fileUploadInfoVO.getFid();
        FileUtils.copyFileFromInputStream(in, filePath);
        fileUploadInfoVO.setUrl(filePath+"."+fileUploadInfoVO.getSuffix());
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        String filePath = fileUploadConfig.getLocal().getParentPath() + File.separator + fileUploadInfoVO.getFid();
        FileUtils.copyFileFromBytes(bytes, filePath);
        fileUploadInfoVO.setUrl(filePath);
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception {
        File file = new File(fileUploadInfoVO.getUrl());
        if(file.exists()) {
            file.delete();
        }
        return fileUploadInfoVO;
    }
}

Seaweedfs存储实现

@Service("FileStorageServiceSeaweedfs")
public class FileStorageServiceSeaweedfs implements FileStorageService {

    @Autowired
    FileUploadConfig fileUploadConfig;

    FileTemplate fileTemplate;

    @PostConstruct
    public void init() {
        FileUploadConfigSeaweedfs seaweedfs = fileUploadConfig.getSeaweedfs();
        FileSource fileSource = new FileSource();
        fileSource.setHost(seaweedfs.getHost());
        fileSource.setPort(seaweedfs.getPort());
        fileSource.setConnectionTimeout(seaweedfs.getConnectTimeout());//5min
        try {
            fileSource.startup();
        } catch (IOException e) {
            throw new RuntimeException("创建seaweedfs连接失败，原因是：" + e.getMessage());
        }
        fileTemplate = new FileTemplate(fileSource.getConnection(), seaweedfs.getPublicUrl());
    }

    @Override
    public FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        FileHandleStatus fileHandleStatus = fileTemplate.saveFileByStream(fileUploadInfoVO.getFileName(), in, ContentType.create(fileUploadInfoVO.getContentType(), "utf-8"));
        fileUploadInfoVO.setFid(fileHandleStatus.getFileId());
        fileUploadInfoVO.setUrl(String.format("%s/%s", fileUploadConfig.getSeaweedfs().getPublicUrl(), fileHandleStatus.getFileId()));
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        return upload(new ByteArrayInputStream(bytes), fileUploadInfoVO);
    }

    @Override
    public FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception {
        fileTemplate.deleteFile(fileUploadInfoVO.getFid());
        return fileUploadInfoVO;
    }
}

Minio存储实现

@Service("FileStorageServiceMinio")
public class FileStorageServiceMinio implements FileStorageService {

    @Autowired
    FileUploadConfig fileUploadConfig;

    @Override
    public FileUploadInfoVO upload(InputStream in, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        FileUploadConfigMinio minio = fileUploadConfig.getMinio();
        String domain = minio.getDomain();
        String bucket = minio.getBucket();
        String fileName = fileUploadInfoVO.getFileName();
        MinioClient minioClient = new MinioClient(domain, minio.getAccesKey(), minio.getSecretKey());
        minioClient.putObject("report", fileName, in, fileUploadInfoVO.getSize(), fileUploadInfoVO.getContentType());
        fileUploadInfoVO.setUrl(String.format("%s/%s/%s", domain, bucket, fileName));
        return fileUploadInfoVO;
    }

    @Override
    public FileUploadInfoVO upload(byte[] bytes, FileUploadInfoVO fileUploadInfoVO) throws Exception {
        return upload(new ByteArrayInputStream(bytes), fileUploadInfoVO);
    }

    @Override
    public FileUploadInfoVO delete(FileUploadInfoVO fileUploadInfoVO) throws Exception {
        FileUploadConfigMinio minio = fileUploadConfig.getMinio();
        MinioClient minioClient = new MinioClient(minio.getDomain(), minio.getAccesKey(), minio.getSecretKey());
        minioClient.removeObject(minio.getBucket(), fileUploadInfoVO.getFileName());
        return fileUploadInfoVO;
    }
}

存储工厂类实现

用来根据 FileTag 枚举类获得对应的具体实现

@Service
public class FileStorageFactory {

    Map<FileTag, FileStorageService> fileStorageServiceMap;

    @Qualifier("FileStorageServiceLocal")
    @Autowired
    FileStorageService fileStorageServiceLocal;

    @Qualifier("FileStorageServiceSeaweedfs")
    @Autowired
    FileStorageService fileStorageServiceSeaweedfs;

    @Qualifier("FileStorageServiceMinio")
    @Autowired
    FileStorageService fileStorageServiceMinio;

    @PostConstruct
    public void init() {
        fileStorageServiceMap = new HashMap<>();
        fileStorageServiceMap.put(FileTag.TYPE_LOCAL, fileStorageServiceLocal);
        fileStorageServiceMap.put(FileTag.TYPE_SEAWEEDFS, fileStorageServiceSeaweedfs);
        fileStorageServiceMap.put(FileTag.TYPE_MINIO, fileStorageServiceMinio);
    }

    public FileStorageService get(FileTag type) {
        FileStorageService fileStorageService = fileStorageServiceMap.get(type);
        return fileStorageService == null ? fileStorageServiceMap.get(FileTag.TYPE_LOCAL) : fileStorageService;
    }
}

业务中实现

这里根据具体的业务，来选择具体的实现逻辑

eg：

@Autowired
private FileStorageFactory fileStorageFactory;
public FileUploadInfoVO put(MultipartFile multipartFile) throws Exception {
  FileUploadInfoVO vo = new FileUploadInfoVO();
  // 填充一些信息（省略）
  // 获得文件上传处理器
  FileStorageService fileStorageService = fileStorageFactory.get(FileTag.TYPE_LOCAL);
  fileUploadInfoVO = fileStorageService.upload(multipartFile.getInputStream(), vo);
  // ... 业务逻辑
}

这样，配置文件中可以使用FileTag来控制文件存储的介质，从而实现一个简单的对象存储服务器。