Adding an External Stream Sink

This sink writes data in CSV (Comma Separated Value), JSON, XML, or BSON to a File System, AMQP, Network Socket, HTTP, WebSocket, Amazon Kinesis, Kafka, MongoBB, or Snowflake destination.

  • First, select a sink output type.
  • Configure this sink in the top pane. For example, for a socket you need to enter a server name and port.
  • Start by specifying the s-Server schema and stream names for the sink's stream. By default, StreamLab uses the Routing Data to a Sink for the sink. You can also route guide data to the sink later by selecting the Route to Sink button in a guide.
  • Next, fill in the list of columns and their SQL types. Click the + icon to add another column. Give the column a name, and select its type from the pulldown menu. You can use the Clipboard to copy column names and types from another form.
  • Use the output selector to choose an output and fill in the parameters to connect to that source. (This sink uses the Extensible Common Data Adapter to support writing to a number of different output media.)
  • Click the blue title to save the project. The title returns to white after you click it.
  • Click the Go Up arrow to exit the Add CSV Sink page.

Local vs. Remote

Remote Adapter In this mode the output is accessed directly from s-Server. Files are local to the s-Server host, and sockets and other network connections are relative to the s-Server host. When this mode is enabled, StreamLab submits your configuration for this sink directly to your instance of s-Server.

Local Adapter In this mode you can access sinks via a host running the ECDA agent. The agent is a Java application running on a host other than the s-Server host, configured to read data from your instance of s-Server. When this mode is enabled, StreamLab configures the stream that the remote agent will read from. To configure the remote agent,use the Export Properties button to get the contents of the properties file, and paste this into a new file on the remote agent server (for example, mysource.props).

From a command shell on the remote host, run

%$SQLSTREAM_HOME/../clienttools/EcdaAgent.commondataagent.sh --output --props mysource.props.

Note: $SQLSTREAM_HOME refers to the installation directory for s-Server, such as /opt/sqlstream/<VERSION>/s-Server. Everything you've specified here to configure the sink will now apply within the remote agent, so file names will be local to that host, and sockets and other network connections will be relative to that host.

CSV Options

The character used to separate columns defaults to comma and can be changed via the Column Separators(s) field. You can also change the Row Separator, which defaults to 000A. You can also change the Character Encoding and decide if you want a CSV header containing the column names to be written.

XML Options

You control the XML format by specifying information in the fields below Character Encoding:

  • Document elements indicates a list of elements, separated by slashes, to use as the root of the XML document.
  • Row elements indicates a list of elements, separated by slashes, to add for each value output to the document.
  • Data elements indicates a list of elements, separated by slashes, to add for each value output to the document.
  • Data attributes indicates the name of an attribute to add to the output.

If no value for DATA_ELEMENTS is specified, then the column name is used as an element name (not an <attribute> name). So a column named foo would be in an XML element named /batch/row/foo if no values were specified.