在多数据源中对部分数据表使用shardingsphere进行分库分表

背景

近期在项目中需要使用多数据源，其中有一些表的数据量比较大，需要对其进行分库分表；而其他数据表数据量比较正常，单表就可以。

项目中可能使用其他组的数据源数据，因此需要多数据源支持。

经过调研多数据源配置比较方便。在该项目中分库分表的策略比较简单，仅根据一个字段分就可以，因此分库分表方案选用比较流行方便易用的 sharding-jdbc

需要实现的目标是根据学生姓名字段 student_name 进行分表，但是不需要分库。数据表是student_hist0 - student_hist9

引入 sharding-jdbc maven 依赖

<dependency>

    <groupId>org.apache.shardingsphere</groupId>

    <artifactId>sharding-jdbc-core</artifactId>

    <version>4.1.1</version>

</dependency>

数据源配置文件

spring:

  application:

    name: student-service-provider

  jackson:

    date-format: yyyy-MM-dd HH:mm:ss

    time-zone: GMT+8

    defaultPropertyInclusion: non_null

    deserialization:

      FAIL_ON_UNKNOWN_PROPERTIES: false

  #对返回的时间进行格式化

  datasource:

    hikari:

      student:

        url: jdbc:mysql://127.0.0.1:3306/student_service?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&useTimezone=true&serverTimezone=GMT%2

        username: root

        password: root123

      log1:

        url: jdbc:mysql://127.0.0.1:3306/log1?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false

        username: root

        password: root123

      log2:

        url: jdbc:mysql://127.0.0.1:3306/log2?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false

        username: root

        password: root123

配置多数据源代码

DataSourceProperties 数据源

import com.zaxxer.hikari.HikariDataSource;

import lombok.Data;

import org.springframework.boot.context.properties.ConfigurationProperties;

import org.springframework.context.annotation.Configuration;

@Data

@Configuration

@ConfigurationProperties(prefix = "spring.datasource.hikari")

public class DataSourceProperties {

    private HikariDataSource student;

	private HikariDataSource log1;

	private HikariDataSource log2;

}

DynamicDataSource 动态数据源

import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;

public class DynamicDataSource extends AbstractRoutingDataSource {

    /*

    当前据源名称

     */

    private static final ThreadLocal<String> dataSourceContextHolder = new ThreadLocal<>();

    /*

    设置数据源名称

     */

    public static void setDataSourceName(String dataSourceName) {

        dataSourceContextHolder.set(dataSourceName);

    }

    /*

    获取据源名称

     */

    public static String getDataSourceName() {

        return dataSourceContextHolder.get();

    }

    /*

    清除当数据源名称

     */

    public static void clearDataSource() {

        dataSourceContextHolder.remove();

    }

    @Override

    protected Object determineCurrentLookupKey() {

        return getDataSourceName();

    }

}

MultiDataSource 多数据源标记

import java.lang.annotation.ElementType;

import java.lang.annotation.Retention;

import java.lang.annotation.RetentionPolicy;

import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)

@Target({ElementType.METHOD})

public @interface MultiDataSource {

    String value() default DataSourceConfig.DEFAULT_DATASOURCE_NAME;

}

重点来了，因为是根据表的某一个字段进行分表，该字段是一个字符串类型，因此需要想根据字符串的一致性hash码算出在哪张表上。在sharding-jdbc需要实现 PreciseShardingAlgorithm 类

例如：想要在student.student_hist 表中根据学生姓名进行分表，逻辑表是student_hist，真实表是 student_hist0 - student_hist9

DataSourceConfig.SHARD_MMS_DATASOURCE_TABLE_COUNT=10

import org.apache.shardingsphere.api.sharding.standard.PreciseShardingAlgorithm;

import org.apache.shardingsphere.api.sharding.standard.PreciseShardingValue;

import java.util.Collection;

public class PreciseNodeIdShardingAlgorithm implements PreciseShardingAlgorithm<String> {

    @Override

    public String doSharding(Collection<String> collection, PreciseShardingValue<String> preciseShardingValue) {

        for (String tbnm : collection) {

            if (tbnm.endsWith("hist" + (getHash(preciseShardingValue.getValue()) % DataSourceConfig.SHARD_MMS_DATASOURCE_TABLE_COUNT))) {

                return tbnm;

            }

        }

        throw new UnsupportedOperationException();

    }

	private static int getHash(String str) {

        final int p = 16777619;

        int hash = (int) 2166136261L;

        for (int i = 0; i < str.length(); i++)

            hash = (hash ^ str.charAt(i)) * p;

        hash += hash << 13;

        hash ^= hash >> 7;

        hash += hash << 3;

        hash ^= hash >> 17;

        hash += hash << 5;

        // 如果算出来的值为负数则取其绝对值

        if (hash < 0)

            hash = Math.abs(hash);

        return hash;

    }

}

多数据源装配 DataSourceConfig 。需要指定默认数据源，当不需要使用分表的表时就使用默认的数据源，否则指定需要分表的数据源。

在配置分表策略时如果不需要分库，可以不进行设置 tableRuleConfiguration.setDatabaseShardingStrategyConfig();

import org.apache.shardingsphere.api.config.sharding.ShardingRuleConfiguration;

import org.apache.shardingsphere.api.config.sharding.TableRuleConfiguration;

import org.apache.shardingsphere.api.config.sharding.strategy.StandardShardingStrategyConfiguration;

import org.apache.shardingsphere.shardingjdbc.api.ShardingDataSourceFactory;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.context.annotation.Bean;

import org.springframework.context.annotation.Configuration;

import org.springframework.context.annotation.Primary;

import org.springframework.jdbc.datasource.DataSourceTransactionManager;

import org.springframework.transaction.PlatformTransactionManager;

import javax.sql.DataSource;

import java.sql.SQLException;

import java.util.HashMap;

import java.util.Map;

import java.util.Properties;

@Configuration

public class DataSourceConfig {

    public static final String DEFAULT_DATASOURCE_NAME = "defaultDataSource";

    public static final String MMS_DATASOURCE_NAME = "mmsDatasource";

    public static final String SHARD_MMS_DATASOURCE_NAME = "shardMmsDatasource";

    public static int SHARD_MMS_DATASOURCE_TABLE_COUNT = 10;

    @Autowired

    private DataSourceProperties properties;

    @Primary

    @Bean(name = "dynamicDataSource")

    public DataSource dynamicDataSource() {

        DynamicDataSource dynamicDataSource = new DynamicDataSource();

        // 默认数据源

        dynamicDataSource.setDefaultTargetDataSource(properties.getMms());

        // 配置多数据源

        Map<Object, Object> dsMap = new HashMap();

        dsMap.put(DEFAULT_DATASOURCE_NAME, properties.getStudent());

        dsMap.put(MMS_DATASOURCE_NAME, properties.getStudent());

        dsMap.put(SHARD_MMS_DATASOURCE_NAME, buildShardDatasources());

        dynamicDataSource.setTargetDataSources(dsMap);

        return dynamicDataSource;

    }

    public DataSource buildShardDatasources() {

        // 配置多数据源

        Map<String, DataSource> dsMap = new HashMap();

        dsMap.put("shardMms", properties.getMms());

        TableRuleConfiguration stuHisTableRuleConfig = new TableRuleConfiguration("student_hist", "shardMms.student_hist${0.." + (SHARD_MMS_DATASOURCE_TABLE_COUNT - 1) + "}");

//        tableRuleConfiguration.setDatabaseShardingStrategyConfig();

        stuHisTableRuleConfig.setTableShardingStrategyConfig(new StandardShardingStrategyConfiguration("student_name", new PreciseNodeIdShardingAlgorithm()));

        ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();

        shardingRuleConfig.getTableRuleConfigs().add(stuHisTableRuleConfig);

        try {

            Properties properties = new Properties();

            properties.setProperty("sql.show", "true");

            return ShardingDataSourceFactory.createDataSource(dsMap, shardingRuleConfig, properties);

        } catch (SQLException throwables) {

            throwables.printStackTrace();

            throw new IllegalArgumentException();

        }

    }

    @Bean

    public PlatformTransactionManager transactionManager() {

        return new DataSourceTransactionManager(dynamicDataSource());

    }

}

多数据源切换 DataSourceAspect ,需要使用多数据源切换时，需要在service方法上使用标注方法 MultiDataSource 并指定数据源。

import lombok.extern.slf4j.Slf4j;

import org.aspectj.lang.ProceedingJoinPoint;

import org.aspectj.lang.annotation.Around;

import org.aspectj.lang.annotation.Aspect;

import org.aspectj.lang.annotation.Pointcut;

import org.aspectj.lang.reflect.MethodSignature;

import org.springframework.context.annotation.Configuration;

import org.springframework.core.annotation.Order;

import java.lang.reflect.Method;

@Aspect

@Configuration

@Slf4j

@Order(1)

public class DataSourceAspect {

    //切入点，service 中所有注解方法

    @Pointcut("execution(* com.huitong..service..*.*(..)) && @annotation(com.huitong.app.config.datasource.MultiDataSource)")

    public void dataSourceAspect() {

    }

    @Around("dataSourceAspect()")

    public Object around(ProceedingJoinPoint joinPoint) throws Throwable {

        MethodSignature signature = (MethodSignature) joinPoint.getSignature();

        Method method = signature.getMethod();

        MultiDataSource ds = method.getAnnotation(MultiDataSource.class);

        if (ds != null) {

            DynamicDataSource.setDataSourceName(ds.value());

        }

        try {

            return joinPoint.proceed();

        } finally {

            DynamicDataSource.clearDataSource();

        }

    }

}

参考文献：