(Added in StreamLab version 7.3.0)
This template creates a UNION ALL of two streams (for a streaming pipeline) or two tables (for a finite SQL pipeline).
The parameters are:
Parameter | Type | Meaning | Default |
---|---|---|---|
Union Source | List of Values | The stream or table to be UNIONed with the current pipeline | First in list |
Merge identical columns? | Boolean | Attempt to map same-named input columns into one output column | True |
Suffix for duplicated columns | Text | Suffix to append to any duplicated columns | ‘_U’ |
The result contains all unique named columns and by default maps identically named columns together.
So the output will consist of:
Given a pipeline with columns A,B,C,D,E and a secondary stream with columns A,B,D,F,G the output would consist of:
Columns: Sources | A | B | C | D | E | F | G |
---|---|---|---|---|---|---|---|
Pipeline | a1 | b1 | c1 | d1 | e1 | null | null |
Secondary | a2 | b2 | null | d2 | null | f2 | g2 |
Sometimes the same column name means different things in the two source streams. In this case the user may set “Merge Identical Columns” to false, and will set a suffix to add to any duplicated column names. For the same incoming data we would now see:
Columns: Sources | A | A_U | B | B_U | C | D | D_U | E | F | G |
---|---|---|---|---|---|---|---|---|---|---|
Pipeline | a1 | null | b1 | null | c1 | d1 | null | e1 | null | null |
Secondary | null | a2 | null | b2 | null | null | d2 | null | f2 | g2 |
The user can then wrangle individual columns as needed in subsequent pipeline steps.