Java Spring Batch 中多个数据源的使用

Use of multiple DataSources in Spring Batch

提问by Ahmed Bhaila

I am trying to configure a couple of datasources within Spring Batch. On startup, Spring Batch is throwing the following exception:

我正在尝试在 Spring Batch 中配置几个数据源。在启动时，Spring Batch 抛出以下异常：

To use the default BatchConfigurer the context must contain no more thanone DataSource, found 2

To use the default BatchConfigurer the context must contain no more thanone DataSource, found 2

Snippet from Batch Configuration

批处理配置的片段

@Configuration
@EnableBatchProcessing 
public class BatchJobConfiguration {

    @Primary
    @Bean(name = "baseDatasource")
    public DataSource dataSource() {
         // first datasource definition here
    }
    @Bean(name = "secondaryDataSource")
    public DataSource dataSource2() {
         // second datasource definition here
    }
    ...
}

Not sure why I am seeing this exception, because I have seen some xml based configuration for Spring batch that declare multiple datasources. I am using Spring Batch core version 3.0.1.RELEASE with Spring Boot version 1.1.5.RELEASE. Any help would be greatly appreciated.

不知道为什么我会看到这个异常，因为我看到了一些声明多个数据源的 Spring 批处理的基于 xml 的配置。我正在使用 Spring Batch 核心版本 3.0.1.RELEASE 和 Spring Boot 版本 1.1.5.RELEASE。任何帮助将不胜感激。

采纳答案by vanarchi

AbstractBatchConfigurationtries to lookup BatchConfigurerin container first, if it is not found then tries to create it itself - this is where IllegalStateExceptionis thrown where there is more than one DataSourcebean in container.

AbstractBatchConfiguration首先尝试BatchConfigurer在容器中查找，如果未找到，则尝试自己创建它——这是在容器IllegalStateException中有多个DataSourcebean 的地方抛出的地方。

The approach to solving the problem is to prevent from creation the DefaultBatchConfigurerbean in AbstractBatchConfiguration. To do it we hint to create DefaultBatchConfigurerby Spring container using @Componentannotation:

解决问题的方法是防止DefaultBatchConfigurer在AbstractBatchConfiguration. 为此，我们提示DefaultBatchConfigurer使用@Component注释由 Spring 容器创建：

The configuration class where @EnableBatchProcessingis placed we can annotate with @ComponentScanthat scan the package that contains the empty class that is derived from DefaultBatchConfigurer:

@EnableBatchProcessing放置的配置类我们可以注释@ComponentScan，扫描包含派生自的空类的包DefaultBatchConfigurer：

package batch_config;
...
@EnableBatchProcessing
@ComponentScan(basePackageClasses = MyBatchConfigurer.class)
public class MyBatchConfig {
    ...
}

the full code of that empty derived class is here:

该空派生类的完整代码在这里：

package batch_config.components;
import org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer;
import org.springframework.stereotype.Component;
@Component
public class MyBatchConfigurer extends DefaultBatchConfigurer {
}

In this configuration the @Primaryannotation works for DataSourcebean as in the example below:

在此配置中，@Primary注释适用于DataSourcebean，如下例所示：

@Configuration
public class BatchTestDatabaseConfig {
    @Bean
    @Primary
    public DataSource dataSource()
    {
        return .........;
    }
}

This works for the Spring Batch version 3.0.3.RELEASE

这适用于 Spring Batch 版本 3.0.3.RELEASE

The simplest solution to make @Primaryannotation on DataSourcework might be just adding @ComponentScan(basePackageClasses = DefaultBatchConfigurer.class)along with @EnableBatchProcessingannotation:

为了使简单的解决方案@Primary上标注DataSource的工作可能只是增加@ComponentScan(basePackageClasses = DefaultBatchConfigurer.class)连同@EnableBatchProcessing注释：

@Configuration
@EnableBatchProcessing
@ComponentScan(basePackageClasses = DefaultBatchConfigurer.class)
public class MyBatchConfig {

回答by Dayong

First, create a custom BatchConfigurer

首先，创建一个自定义的 BatchConfigurer

@Configuration
@Component
public class TwoDataSourcesBatchConfigurer implements BatchConfigurer {

    @Autowired
    @Qualifier("dataSource1")
    DataSource dataSource;

    @Override
    public JobExplorer getJobExplorer() throws Exception {
        ...
    }

    @Override
    public JobLauncher getJobLauncher() throws Exception {
        ...
    }

    @Override
    public JobRepository getJobRepository() throws Exception {
        JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
        // use the autowired data source
        factory.setDataSource(dataSource);
        factory.setTransactionManager(getTransactionManager());
        factory.afterPropertiesSet();
        return factory.getObject();
    }

    @Override
    public PlatformTransactionManager getTransactionManager() throws Exception                      {
        ...
    }

}

Then,

然后，

@Configuration
@EnableBatchProcessing
@ComponentScan("package")
public class JobConfig {
    // define job, step, ...
}

回答by PeterSan

You must provide your own BatchConfigurer. Spring does not want to make that decision for you

您必须提供自己的 BatchConfigurer。Spring 不想为你做那个决定

@Configuration
@EnableBatchProcessing
public class BatchConfig {

     @Bean
      BatchConfigurer configurer(@Qualifier("batchDataSource") DataSource dataSource){
        return new DefaultBatchConfigurer(dataSource);
      }

...

回答by saiD

If I may add to the above question, the implications of having 2 transaction contexts one for each DS. How to integrate the XA transaction with Batch step as we would need to ensure the TXN management at step level? Requirement is like in a batch step we need the following.

如果我可以补充上述问题，那么每个 DS 有 2 个事务上下文的含义。如何将 XA 事务与 Batch 步骤集成，因为我们需要确保步骤级别的 TXN 管理？要求就像在批处理步骤中我们需要以下内容。

1. read from DS 1 -- jpaItemReader
2. write to DS2 - JPAItemwriter
3. read from DS2 - JPAItemreader
4. write to Ds1 - JPAItemwriter
5. Commit all txns Step completed.

1. 从 DS 1 读取——jpaItemReader
2. 写入 DS2 - JPAItemwriter
3. 从 DS2 读取 - JPAItemreader
4. 写入 Ds1 - JPAItemwriter
5. 提交所有 txns 步骤已完成。

回答by imarchuang

I would like to provide a solution here, which is very similar to the one answered by @vanarchi, but I managed to put all the necessary configurations into one class.

我想在这里提供一个解决方案，它与@vanarchi 的回答非常相似，但我设法将所有必要的配置放在一个类中。

For the sake of completeness, the solution here assumes that primary datasource is hsql.

为了完整起见，这里的解决方案假设主数据源是 hsql。

@Configuration
@EnableBatchProcessing
public class BatchConfiguration extends DefaultBatchConfigurer {

@Bean
@Primary
public DataSource batchDataSource() {

    // no need shutdown, EmbeddedDatabaseFactoryBean will take care of this
    EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
    EmbeddedDatabase embeddedDatabase = builder
            .addScript("classpath:org/springframework/batch/core/schema-drop-hsqldb.sql")
            .addScript("classpath:org/springframework/batch/core/schema-hsqldb.sql")
            .setType(EmbeddedDatabaseType.HSQL) //.H2 or .DERBY
            .build();
    return embeddedDatabase;
}

@Override
protected JobRepository createJobRepository() throws Exception {
    JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
    factory.setDataSource(batchDataSource());
    factory.setTransactionManager(transactionManager());
    factory.afterPropertiesSet();

    return (JobRepository) factory.getObject();
}

private ResourcelessTransactionManager transactionManager() {
    return new ResourcelessTransactionManager();
}

//NOTE: the code below is just to provide developer an easy way to access the in-momery hsql datasource, as we configured it to the primary datasource to store batch job related data. Default username : sa, password : ''
@PostConstruct
public void getDbManager(){
    DatabaseManagerSwing.main(
            new String[] { "--url", "jdbc:hsqldb:mem:testdb", "--user", "sa", "--password", ""});
}

}

}

THREE key points in this solution:

此解决方案的三个关键点：

1. This class is annotated with @EnableBatchProcessingand @Configuration, as well as extended from DefaultBatchConfigurer. By doing this, we instruct spring-batch to use our customized batch configurer when AbstractBatchConfigurationtries to lookup BatchConfigurer;
2. Annotate batchDataSource bean as @Primary, which instruct spring-batch to use this datasource as its datasource of storing the 9 job related tables.
3. Override protected JobRepository createJobRepository() throws Exceptionmethod, which makes the jobRepository bean to use the primary datasource, as well as use a different transactionManager instance from the other datasource(s).

1. 这个类用@EnableBatchProcessing和注释@Configuration，以及从 扩展DefaultBatchConfigurer。通过这样做，我们指示 spring-batch 在AbstractBatchConfiguration尝试查找时使用我们定制的批处理配置器BatchConfigurer；
2. 将 batchDataSource bean 注释为@Primary，指示 spring-batch 使用此数据源作为其存储 9 个作业相关表的数据源。
3. 覆盖protected JobRepository createJobRepository() throws Exception方法，它使 jobRepository bean 使用主数据源，并使用与其他数据源不同的 transactionManager 实例。

回答by user3474985

The simplest solution is to extend the DefaultBatchConfigurer and autowire your datasource via a qualifier:

最简单的解决方案是扩展 DefaultBatchConfigurer 并通过限定符自动装配您的数据源：

@Component
public class MyBatchConfigurer extends DefaultBatchConfigurer {

    /**
     * Initialize the BatchConfigurer to use the datasource of your choosing
     * @param firstDataSource
     */
    @Autowired
    public MyBatchConfigurer(@Qualifier("firstDataSource") DataSource firstDataSource) {
        super(firstDataSource);
    }
}

Side Note (as this also deals with the use of multiple data sources): If you use autoconfig to run data initialization scripts, you may notice that it's not initializing on the datasource you'd expect. For that issue, take a look at this: https://github.com/spring-projects/spring-boot/issues/9528

旁注（因为这也涉及多个数据源的使用）：如果您使用 autoconfig 来运行数据初始化脚本，您可能会注意到它没有在您期望的数据源上进行初始化。对于那个问题，看看这个：https: //github.com/spring-projects/spring-boot/issues/9528