Using StreamLab

StreamLab 2.3 is a browser-based interface for parsing, analyzing, and displaying data streams.You can now read and write to CSV, XML, and JSON files from StreamLab. StreamLab 2.3 also lets you read and write to external data sources, including Oracle, PostgreSQL, MySQL, Teradata, and Microsoft SQL Server.StreamLab 2.3 also lets you create sinks. Sinks can serve as data sources for other StreamLab projects. StreamLab 2.3 also adds a number of new analytics, including the join command, distinct, pivot, window, partition window, and GroupRank.

Guide sections

For a tutorial on using new StreamLab features, please see http://thyrd.org/sqlstream/tutorials.html. This tutorial walks you through the process of installing s-Server, installing StreamLab, and creating a PostgreSQL table. It then walks you through the process of using StreamLab to analyze a log file (in this case, data on the location and speeds of regional buses), create visualizations based on this analysis, and then store this data in a PostgreSQL database.

This page features the following subtopics:

StreamLab Overview

StreamLab is a web application development environment that automatically generates streaming SQL, which are known as “StreamApps.” Using StreamLab’s graphical interface, you can set up input streams for data, apply built-in analytics to this data, and create sinks for data for output to external locations.

StreamLab lets you:

To use StreamLab, you need one of the following browsers: Microsoft Internet Explorer 11 (or later), Google Chrome 31 (or later), Mozilla Firefox 17 (or later), or Apple Safari 7 (or later).

StreamLab includes demonstration applications in the StreamLab gallery. These include an application based on global weather sensors, an application based on Meetup data, and an application based on bus data from Sydney, Australia.

StreamLab uses its Scrutinizer to make guesses about incoming data, and in turn offer suggestions on this data. You can manage the Scrutinizer to affect how it performs.

StreamLab Menu

The StreamLab menu lets you switch between StreamLab projects and dashboards, as well as managing your StreamLab account and project settings. It is available in the upper right corner of StreamLab.

Using the Clipboard

Many of the tables used in forms in StreamLab now have a clipboard icon in the upper left corner. Pressing this opens a dialog with the contents of the table, as text in CSV format. The user can then copy, paste, or edit this; when saved, the table will be updated. To use the clipboard:

  • In a form containing rows, click the clipboard icon.
  • In the dialog box that opens, you can copy, paste, or edit the text. To update the columns, close the dialog box.

StreamLab Workflow

Before you can analyze and present data in StreamLab, that data needs to be made available in s-Server. Local log files are an exception to this rule, in that you can add these to StreamLab directly through the StreamLab interface.

The workflow for StreamLab includes the following steps:

  1. Once you have set up StreamLab and it is communicating with s-Server, StreamLab developers add sources to StreamLab. Some of these sources may exist in s-Server. Others can be created with local log files.
  2. StreamLab developers then use these sources to create pipeline guides in order to create views of data in s-Server.
  3. StreamLab developers may develop sinks from these pipeline guides–sinks can serve as sources for other pipeline guides, or feed data to an external source.
  4. StreamLab developers generate dashboards from these views and share them with others.

StreamLab Technical Details

StreamLab communicates with the s-Server webAgent over Websockets to get stream data, using Node, a JavaScript integration platform. webAgent, in turn, communicates with s-Server over JDBC.

StreamLab SQL Overview

For all projects, you can view the SQL generated by the project, as well as the SQL for individual pages. To do so, click the SQL Script Log button in the upper right corner of StreamLab.

The s-Server SQL Log window lists the following:

  • This project’s SQL script. SQL for all script items in the entire project, such as those from columns parsed with Regex, renamed columns, split columns, extracted columns, and so on. See Viewing SQL Generated by Guide for more details.
  • This project’s removal script. The removal script consists of the commands to remove everything created by the project from s-server’s catalog, and can be exported separately, or executed to clean up after a project.

Removing SQL Objects Generated by StreamLab

You can remove all or part of a project by using the Removal Script. The removal script consists of the commands to remove everything created by the project from s-server’s catalog, and can be exported separately, or executed to clean up after a project.

To run a project’s Removal Script:

  1. Click the View Log button in the upper right corner of the Guides page:

​ The Goals and Scripts page opens. 2. Click the The Project’s Removal Script tab. ​ The Removal Script page opens.

  1. Click Execute.

The Removal Script runs.

Goals Overview

Goals guide you through the process of getting a script ready to execute. If a page or project has unmet goals, it will return errors.   For all projects, you can goals for the entire project and for each page. To do so, click the Goals and Scripts button in the upper right corner of StreamLab (/images/sl/sl_guide_goals_script_button.png). The Goals and Scripts window lists the following:

  • Goals for this page. All goals, met and unmet, for the current page.
  • Goals for the project. All goals, met and unmet, for the current project. By clicking the arrow next to a goal, you can open the page containing the goal.

StreamLab Artifacts

StreamLab generates a set of artifacts in its repository. You can move project and dashboard files from one installation of StreamLab to another. Older versions of StreamLab may not run projects generated in newer versions of StreamLab. With some exceptions, new versions of StreamLab will run projects generated in older versions.

To more a StreamLab project, you simply copy files from one repository to another. You can also move StreamLab dashboards to either a machine running s-Dashboard or a machine running another instance of StreamLab.

Moving Repository Files

Files are located in the following directory:

<installation>/repository/_users/user/projects/<projectname>.rose

To move files, copy files from this location to the same location on another user’s machine. Once you do so and start StreamLab, these projects will be available on the Projects page.

Moving Dashboard Files

Dashboard files are located in the following location:

<installation>/repository/user/Dashboards/autogen/

These are the dashboards the user has created via StreamLab. Each dashboard is two files, *.json and *.json.rose.

  • The *.json file can be copied to a machine running s-Dashboard, to be served as a dashboard.
  • To copy dashboards between StreamLab repositories, you need to copy both the *.json file and the corresponding *.json.rose file. If need to rename these files, rename the both with the same name.