Union

(Added in StreamLab version 7.3.0)

This template creates a UNION ALL of two streams (for a streaming pipeline) or two tables (for a finite SQL pipeline).

The parameters are:

Parameter Type Meaning Default
Union Source List of Values The stream or table to be UNIONed with the current pipeline First in list
Merge identical columns? Boolean Attempt to map same-named input columns into one output column True
Suffix for duplicated columns Text Suffix to append to any duplicated columns ‘_U’

The result contains all unique named columns and by default maps identically named columns together.

So the output will consist of:

  • All the columns of the input pipeline
  • Any additional columns from the added stream

Given a pipeline with columns A,B,C,D,E and a secondary stream with columns A,B,D,F,G the output would consist of:

Columns: Sources A B C D E F G
Pipeline a1 b1 c1 d1 e1 null null
Secondary a2 b2 null d2 null f2 g2

Sometimes the same column name means different things in the two source streams. In this case the user may set “Merge Identical Columns” to false, and will set a suffix to add to any duplicated column names. For the same incoming data we would now see:

Columns: Sources A A_U B B_U C D D_U E F G
Pipeline a1 null b1 null c1 d1 null e1 null null
Secondary null a2 null b2 null null d2 null f2 g2

The user can then wrangle individual columns as needed in subsequent pipeline steps.