Java Spring-batch 流/一步后拆分

Spring-batch flow / split after a step

提问by lulu

I am building a spring-batch solution that contains the following process:

我正在构建一个包含以下过程的 spring-batch 解决方案：

step 1 : split a list into multiple lists step 2 : process each sub-list step 3 : merge sub-lists

第 1 步：将列表拆分为多个列表 第 2 步：处理每个子列表 第 3 步：合并子列表

The generated sub-lists can be processed in parallel, and according to the spring-batch documentation this is supported. Sadly I can only find spring-batch example jobs that start with parallel steps, not examples that start out sequentially.

生成的子列表可以并行处理，根据 spring-batch 文档，这是支持的。遗憾的是，我只能找到以并行步骤开始的 spring-batch 示例作业，而不是按顺序开始的示例。

The following job will not compile. Spring gives me an error: 'cannot resolve step2'

以下作业将无法编译。Spring 给我一个错误：'无法解决 step2'

<batch:job id="webServiceJob2">
    <batch:step id="step1" next="step2"></batch:step>
    <batch:split id="step2" next="step3"></batch:split>
    <batch:step id="step3"></batch:step>
</batch:job>

So how can I configure a job to first run a single step, than run a number of steps in parallel, and then run a last single step?

那么如何将作业配置为首先运行单个步骤，而不是并行运行多个步骤，然后运行最后一个步骤？

采纳答案by Pablo Lozano

I've stumbled upon this question asking about how split works, and maybe this answer arrives a bit (one year) late, but here I go...

我偶然发现了这个问题，询问 split 是如何工作的，也许这个答案来得有点晚（一年），但我走了......

The issue there is "split" is not a step by itself, but you were naming (and referencing) it as it was:

“拆分”的问题本身并不是一个步骤，而是您按原样命名（和引用）它：

<batch:job id="webServiceJob2">
    <batch:step id="step1" next="step2"></batch:step>
    <batch:split id="step2" next="step3"></batch:split> <!-- This is not a step -->
    <batch:step id="step3"></batch:step>
</batch:job>

The correct syntax would be:

正确的语法是：

<batch:job id="webServiceJob2">
    <batch:step id="step1" next="step2"></batch:step>
    <batch:split id="split_step2" next="step3">
        <flow> 
             <step id="step2_A_1" ... next="step2_A_2"/>
             <step id="step2_A_2" ... />
        </flow>
        <flow> 
             <step id="step2_B_1" ... />
        </flow>
    </batch:split>
    <batch:step id="step3"></batch:step>
</batch:job>

But this is not what you want to achieve, because by splitdeclarations you have to set in compile time the exact number of parallel steps that will be executed, and the purpose of split is using different steps in each flow instead calling several times the same one.

但这不是您想要实现的，因为通过split声明，您必须在编译时设置将执行的确切并行步骤数，而 split 的目的是在每个流程中使用不同的步骤，而不是多次调用相同的步骤.

You should check the documentation about Scaling and Parallel processes, the partition stepseems a good candidate for your requirements.

您应该查看有关Scaling 和 Parallel processes的文档，分区步骤似乎很适合您的要求。

回答by jaco0646

Parallel steps would indicate a different step for each sub-list, which I don't think is what you want.
A single Multi-threaded Stepseems more appropriate.
As documented, you start by defining a TaskExecutorbean, which will process each chunkin a separate thread. Since TaskExecutors are fairly simple to use, you could also invoke the TaskExecutor on your own. In this case, your step can be multi-threaded without Spring Batch needing to know about it.

并行步骤将指示每个子列表的不同步骤，我认为这不是您想要的。
单个多线程步骤似乎更合适。
如文档所述，您首先定义一个TaskExecutorbean，它将在单独的线程中处理每个块。由于 TaskExecutor 使用起来相当简单，您也可以自己调用 TaskExecutor。在这种情况下，您的步骤可以是多线程的，而无需 Spring Batch 了解它。

回答by DBK

Of course you can have a split in the middle of a job! Here is the example from Spring Batch In Action (2012).

当然，您可以在工作中进行拆分！这是 Spring Batch In Action (2012) 中的示例。

<batch:job id="importProductsJob">
  <batch:step id="decompress" next="readWrite">
    <batch:tasklet ref="decompressTasklet"/>
  </batch:step>
  <batch:split id="readWrite" next="moveProcessedFiles">
    <batch:flow>
      <batch:step id="readWriteBookProduct"/>
    </batch:flow>
    <batch:flow>
      <batch:step id="readWriteMobileProduct"/>
    </batch:flow>
  </batch:split>
  <batch:step id="moveProcessedFiles">
    <batch:tasklet ref="moveProcessedFilesTasklet" />
  </batch:step>
</batch:job>