Running SQLstream on Docker

You can run Guavus SQLstream as a Docker container, as either a Complete, Slim or Micro installation. The Complete version includes a Cover Page that lets you manage installed applications.

Docker containers are like lightweight Virtual Machines, but instead of bundling an entire operating system, they bundle the code, a runtime, libraries, environment variables, and config files required to make Guavus SQLstream work. Using Docker, you can do everything you could do with a "bare metal" installation of s-Server and StreamLab.

You can run Docker on Linux, Mac, and Windows systems. For more information, see https://docs.docker.com/engine/installation/.

We recommend running the Guavus SQLstream Docker container on a system with at least 2 CPU cores, 4GB RAM, and 4GB of available disk space. All Guavus SQLstream Containers are 64-bit and require a 64-bit host OS and a Docker version that supports 64-bit containers.

sqlstream/complete

We have designed the Guavus SQLstream Complete container for users that are getting started with Guavus SQLstream. It includes an Ubuntu server with s-Server, WebAgent, s-Dashboard, StreamLab, and the Guavus SQLstream Cover Page installed, as well as Apache Kafka, PostgreSQL, and a set of demonstration applications with sample data. These are the same components that are included in the trial VM appliances (for Parallels, Hyper-V and VirtualBox / VMware) and for the cloud appliances (for the AWS AMI, AWS Marketplace AMI, and Azure).

To run the Guavus SQLstream Complete container, make sure Docker is installed and enter the following command:

docker run -d -p 80:80 -p 5590:5590 -p 5595:5595 -p 5580:5580 -p 5560:5560 -p 9092:9092 -p 2181:2181 sqlstream/complete

(The -p command publishes needed ports to your host machine, so that you can connect with Guavus SQLstream components in the container.)

NOTES:

  • The 9092 and 2181 ports are used by Kafka and Zookeeper respectively; if you don't need to access these from outside the container, you may omit them from the list of ports.
  • If you already use port 80, you can map the coverpage to another port using -p 8080:80.

sqlstream/slim

The Guavus SQLstream Slim image contains only s-Server and WebAgent, and includes all standard ECD plugins. This version only needs to map ports 5570 (s-Server) and 5580 (WebAgent).

We have designed this image for integrating with orchestrators like Kubernetes, mass deployments, autoscaling and highly available environments. sqlstream/slim replaces the former sqlstream/minimal image.

To run the Guavus SQLstream Slim Docker container, make sure Docker is installed and enter the following command:

docker run -d -p 5580:5580 -p 5570:5570 sqlstream/slim

(The -p command publishes needed ports to your host machine, so that you can connect with Guavus SQLstream components in the container.)

sqlstream/micro

The Guavus SQLstream Micro image also contains only s-Server and WebAgent. It is by far the smallest of the sqlstream images. To save space No ECD plugins are included. This version only needs to map ports 5570 (s-Server) and 5580 (WebAgent).

We have designed this image for integrating with orchestrators like Kubernetes, mass deployments, autoscaling and highly available environments.

To run the Guavus SQLstream Micro Docker container, make sure Docker is installed and enter the following command:

docker run -d -p 5580:5580 -p 5570:5570 sqlstream/micro

(The -p command publishes needed ports to your host machine, so that you can connect with Guavus SQLstream components in the container.)

It is very likely that you will want to include at least a couple of the standard ECD plugins when you run a container based on sqlstream/micro. You will need to include one or more source plugins, parsers, formatters and sink plugins. There are a number of ways to achieve this:

You can also include your own user-defined ECD plugins using similar techniques - see Installing an ECD plugin in a Docker container.

sqlstream/minimal

The Guavus SQLstream Minimal image has been deprecated and replaced by sqlstream/slim since version 8.0.8. If you have been using sqlstream/minimal, you can safely switch to using sqlstream/slim. Please update your deployment scripts accordingly.

sqlstream/development

The Guavus SQLstream Development image has now been deprecated - the last available version is 7.2.0 (January 2021). If you have been using sqlstream/development, you can safely switch to sqlstream/complete.

Tailing Docker Messages

You can follow the docker logs (tail log messages) by first using docker ps to get a container id or generated name:

$ docker run -d -p 5580:5580 -p 5570:5570 sqlstream/slim
450fdbd61f94699bf025572714a2f9a56bec435bb8bb206f0dcaa83ec1e7b547

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
450fdbd61f94        sqlstream/slim      "/bin/bash -c /etc/i…"   4 seconds ago       Up 3 seconds        0.0.0.0:5570->5570/tcp, 0.0.0.0:5580->5580/tcp   vigilant_panini

Then use either the container id or the name (450fdbd61f94 or vigilant_panini in this example) with docker logs. Using the --f switch results in tailing the logs forever (until you cancel using Ctrl-C).

    docker logs -f <containerid-or-name>

To avoid the need for the docker ps step you may name your containers using the --name switch for docker run:

$ docker run --name mytestcontainer -d -p 5580:5580 -p 5570:5570 sqlstream/slim
cb63b9e4b2503f1e05e4d2c7f2d9fe097311e257cb9cf1ba5f590b318cd199af

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
cb63b9e4b250        sqlstream/slim     "/bin/bash -c /etc/i…"   6 seconds ago       Up 5 seconds        0.0.0.0:5570->5570/tcp, 0.0.0.0:5580->5580/tcp   mytestcontainer

$ docker logs -f mytestcontainer
SQLSTREAM_HOME=/opt/sqlstream/7.0.6.20069-55f2203c/s-Server
SQLSTREAM_CATALOG_DIR=/opt/sqlstream/7.0.6.20069-55f2203c/s-Server/catalog
SQLSTREAM_TRACE_DIR=/var/log/sqlstream
...

For more information on viewing container logs, see https://docs.docker.com/engine/reference/commandline/logs/.

Using the Guavus SQLstream Container Cover Page

If it does not open automatically, to access the Guavus SQLstream Cover Page, open your browser and go to the following address:

http://localhost/

The Guavus SQLstream Container Cover Page displays. Note: The message at the top of the page will differ depending on the container in which you have installed Guavus SQLstream.

This page lets you do the following:

  1. Confirm that the following Guavus SQLstream components are running (you can also stop these from this page): s-Server, WebAgent, s-Dashboard, StreamLab
  2. Register your Guavus SQLstream Container, which installs as a free version limited to processing 1G of data per day. Once you have registered, SQLstream will send you a 60-day trial license for Guavus SQLstream's Enterprise Edition, which lets you process unlimited data. When your 60-day trial license expires, this instance will revert to the 1GB/day limit.
  3. Run the Mochi Demonstration application. Using s-Dashboard, the Mochi demonstration application simulates clusters of failed logins at a bank, either by phone or web, as well as withdrawals or debits using the same customer id number.
  4. Start a stream of simulated bus location data, see this data being used in a StreamLab application, and view a tutorial on how to integrate this data on your own.
  5. Download and install SQLstream s-Studio and other client tools. See Installing SQLstream s-Studio and Installing SQLstream Client Tools for more details. For both s-Studio and sqlLine, you will need the JDBC address for the s-Server instance in the Guavus SQLstream Docker Container. By default, this address is jdbc:sqlstream:sdp://localhost:5570.

Copying plugins into an image derived from sqlstream/micro

When using sqlstream/micro some ECD plugins will always be needed - at the very least a source plugin and a parser - even if there is no output.

The approach described here allows you to build your own image derived from sqlstream/micro, containing only the plugins that you need - which will be copied from sqlstream/slim.

To make this as simple as possible, we generate a Dockerfile using the build.sh script below:

# build.sh
# This script generates a Dockerfile and builds an image based on sqlstream/micro
# 
# SQLSTREAM_VERSION, SQLSTREAM_BUILD and SQLSTREAM_PLUGINS are defaulted but can be preset in the environment

# Set these enviromment variables to determine source images
: ${SQLSTREAM_VERSION:=8.0.8}

# we need to know the full build - reference the plugin image for that - used for source and target directories to copy
: ${SQLSTREAM_BUILD:=${SQLSTREAM_VERSION}.20037-a235d63cd}

# what plugins to include - space separated list
: ${SQLSTREAM_PLUGINS="file csv json"}

# what to call this image (which will be tagged with $SQLSTREAM_VERSION)
thisimage=microfilecsvjson


# SQLSTREAM_BASE_IMAGE - the image to which plugins are being added
SQLSTREAM_BASE_IMAGE=sqlstream/micro:$SQLSTREAM_VERSION

# SQLSTREAM_PLUGIN_IMAGE - the image containing the plugin directories
SQLSTREAM_PLUGIN_IMAGE=sqlstream/slim:$SQLSTREAM_VERSION

# SQLSTREAM_PLUGIN_DIR - the directory for the plugins (source and target assumed to be the same)
SQLSTREAM_PLUGIN_DIR=/opt/sqlstream/$SQLSTREAM_BUILD/s-Server/plugin

# generate a docker file that copies all the required plugins
(
    echo FROM $SQLSTREAM_BASE_IMAGE

    # copy all required plugins from minimal (could use complete)

    # set ourselves in the plugin directory
    echo WORKDIR $SQLSTREAM_PLUGIN_DIR
    echo RUN mkdir $SQLSTREAM_PLUGINS

    # would be nice to be able to do this as a loop driving from a list of plugins
    for p in $SQLSTREAM_PLUGINS
    do
        echo COPY --from=$SQLSTREAM_PLUGIN_IMAGE $SQLSTREAM_PLUGIN_DIR/$p/ ./$p/
    done

    echo WORKDIR /home/sqlstream
    echo COPY --chown=sqlstream:sqlstream /loadandgo.sh .

    # Run script which installs the plugins before calling /entrypoint.sh

    echo ENTRYPOINT /home/sqlstream/loadandgo.sh
) > Dockerfile


# actually build the image

docker build -t $thisimage:$SQLSTREAM_VERSION .

The build script also depends on loadandgo.sh which installs the required plugins before calling the standard entrypoint script */entrypoint.sh":

# loadandgo.sh
# Installs any plugins found in the SQLSTREAM_HOME/plugin directory

. /etc/sqlstream/environment
cd $SQLSTREAM_HOME/plugin
for d in */installEcdaPlugin.sql
do
    # TODO check for no directories
    echo "... running $d"
    $SQLSTREAM_HOME/bin/sqllineEngine < $d
done

cd /home/sqlstream

# Now launch the standard entrypoint

exec /entrypoint.sh

Once you have created your image, you can run it in the same way you would run a regular SQLstream image:

docker run -d -p 5580:5580 -p 5570:5570 microfilecsvjson:7.2.4