Pipeline execution modes
When you add a pipeline to a workflow, you specify an execution mode for the pipeline. A pipeline's execution mode specifies where and how the pipeline processes documents:
- Workflow-Agent mode: Document processing for the pipeline is distributed among the system instances that the workflow is allowed to run on. This distribution allows a pipeline to process multiple document batches at the same time.
This is the default execution mode for all pipelines you add to workflows.
- Preprocessing mode: Document processing for the pipeline occurs on a single instance and multiple document batches are not processed at the same time. However, this can actually increase pipeline performance in some situations.
When to use Workflow-Agent mode
Use the Workflow-Agent mode when:
- Your pipeline uses archive expansion stages, or if your data source contains small-sized archives files.
- Your pipeline does not use the Read Lines stage.
When to use Preprocessing mode
Consider using the Preprocessing mode when:
Your pipeline uses the Read Lines stage to create new documents from each line in a document such as a log file or comma-separated value (CSV) file.
For an example, see Parsing and indexing CSV and log files.
Setting pipeline execution modes
You set a pipeline's execution mode when you add it to a workflow.
To change the execution mode of a pipeline already added to a workflow, you need to remove the pipeline and then add it again.
For more information, see Adding pipelines to a workflow and Removing pipelines from a workflow.
How workflow recursion settings interact with pipeline execution modes
In a workflow all Preprocessing pipelines are grouped together and precede all Workflow-Agent pipelines.
When the Workflow Agent Recursion setting is enabled, new documents that originate within a Workflow-Agent pipeline are sent to the beginning of the first Workflow-Agent pipeline.
When the Preprocessing Recursion setting is enabled, new documents that originate within a Preprocessing pipeline are sent to the beginning of the first Preprocessing pipeline.
For information on recursion settings, see Task settings.