We run a multi-tenant cloud platform on which our beta users deploy their apps. There are two different sources of logs:

  1. Logs from the runtime (executing Storyscript)
  2. Logs from each microservice spawned by the runtime as a K8s Pod

Google Stackdriver as our first approach

We used Google Stackdriver as a quick and dirty solution to fulfil our logging needs, but had to move away from it soon due to the following shortcomings:

  1. Unable to follow logs via an API
  2. Retrieving logs took almost 5 seconds (although we used the right indexes)

This made for a poor experience in attempting to debug an app deployed on the Storyscript Cloud.

At Storyscript, we care about delivering a flawless experience.

The Elastic (ELK) approach

Due to the issues presented above, we were forced to move, as users were finding it difficult to iterate fast enough.

We setup the standard Elastic stack, where each component fulfilled different needs:

  • Elasticsearch - support for tail -n equivalent (retrieve last N log entries for an app)
  • Filebeat - collect logs from all Pods
  • Logstash - Enrich logs with app_id and other transformations
    • Logstash Websocket Output Plugin - to support tail -f equivalent (follow/stream an app's logs)
  • Kibana (not used right now, but helpful during initial setup)
The Elastic architecture to solve our logging needs

The main component here was the logstreamer module. This is our home-grown solution which authenticates and authorises users to view and follow a stream of logs from their apps.

When a client connection is established to logstreamer, it hits elasticsearch to retrieve the last N logs, and then forwards any new log entries received directly from logstash. Under the hood, logstreamer connects to logstash via a websocket connection, and receives all logs processed. It's upto logstreamer to forward these logs appropriately to end users.

With our own setup, users are now able to retrieve their app logs in less than 1 second (most cases), and are able to follow their app's logs as well. This improved their time to iterate considerably.

Curious to see how logstreamer works? Our platform is entirely open source, including logstreamer.

Engineered along with Anukul Sangwan.