The Split Pipeline operation allows the user to split a single pipeline into two separate pipelines connected by a SQLstream native stream. This allows the developer to easily refactor his or her streaming application code.
The minimum requirement for using Split Pipeline is that the pipeline must have at least two steps (excluding any route-to-sink step).
To demonstrate how Split Pipeline works:
Create a copy of the Sydney Buses gallery application.
Open the “ingest” pipeline by clicking the pencil icon near the top right corner. You should see a Script panel in the bottom left of the window, something like this pipeline guide, which has 10 script steps (the last one is the route to sink):
Pick any one of the script steps amd click the three vertical dots at the right. As long as you are not on the last script step or the route to sink you should be offered “Split to pipeline” as one of the menu options:
Let us choose step 6 of the pipeline. Click on the three dots and then click the “Split to pipeline” option. A dialog box will appear where you must enter a name for the “Sink” (stream) and for the new “Pipeline”. Let us enter “categorized_buses” for the sink stream and “ingest_continued” for the pipeline.
Note: Sink names and Pipeline names share the same namespace within the StreamLab project. Duplicate names are not allowed. StreamLab will tell you if the name is missing clashes with another object.
Click the “Split Pipeline” button. When StreamLab has finished you should see the first 6 steps remain in the pipeline; subsequent steps have been moved. Step 7 has been added as a route to sink to the new “categorized_buses” stream:
All later steps (and any existing route to sink) in the “ingestion” pipeline will be moved to the new pipeline (named “ingest_continued” in the image). A new SQLstream stream (named “categorized_buses” in the image) will be created which will be the sink for the “ingest” pipeline and the source for the new “ingest_continued” pipeline.
When you open the “ingest_continued” pipeline you can see the steps that were numbered 7,8,9 plus the route to sink step 10 of the “ingest” pipeline are now steps numbered 1,2,3 plus route to sink step 4 of the “ingest_continued” pipeline.