Skip to main content

Architecture

Welcome to Responsive​

Responsive is designed to modernize the Kafka Streams architecture and streamline mission-critical production deployments. We built the platform around these four principles:

Separate Storage & Compute​

The primary cause of production outages can be traced back to the management of embedded RocksDB stores.

Responsive offloads the work of managing a distributed database to proven storage systems so that Kafka Streams can focus on what it's best at: stream processing.

Automate Operations​

Correctly sizing and configuring Kafka Streams deployments is a challenge even for the most veteran engineers.

Responsive takes a different approach: measure the bottlenecks and automatically address them as necessary.

Complete the Product​

Most teams that have Kafka Streams deployments which last the test of time tend to come up with a suite of bootstrapped tooling to help them operate it.

Responsive provides that tooling and more out-of-the-box so that you don't have to build it.

Your Code, Your Tools, and Your VPC​

Underpinning Responsive is the belief that you own your application. You can deploy it how you like, run it with your CI/CD tools and monitor it with your observability stack. We just make all that easier.

And one more thing: when you write Responsive code, you are writing Kafka Streams code (and vice versa). That means you are never locked in to a commercial offering.

Architecture Overview​

The first two principles, separating storage from compute and automating operations, require changing the architecture of Kafka Streams to look like this:

architecture

Let's dig into how this all fits together. The entire ecosystem consists of the following "players":

  1. A Kafka Cluster. Responsive is agnostic to the Kafka Cluster -- it can run self-hosted in your VPC, in some Cloud Provider (such as Confluent, Redpanda or Warpstream) or even locally for testing.
  2. Your Kafka Streams Applications. Represented in the diagram above as java boxes, these applications run on your hardware in your network. To take advantage of the Responsive autoscaling capabilities, these apps must run on an orchestration framework such as Kubernetes.
  3. The Responsive Operator. Responsive provides a Helm chart to deploy the operator inside your network. This component manages your Kafka Streams Applications, configuring and scaling up and down based on signals it polls from the Responsive Controller.
  4. The Responsive Controller. Responsive deploys a multi-tenant Controller in our VPC. This Controller collects metrics from your Kafka Streams Applications and executes user-defined policies, the results of which are polled by the operator.
  5. The Storage Backend. Instead of embedding RocksDB stores, ResponsiveKafkaStreams offloads this state to a natively distributed database. Today we support MongoDB and anything Cassandra-compatible such as ScyllaDB.

Further Reading​

  • Read more about the history and design of our Storage Engine
  • Learn about how Operator and Controller work together to provide Autoscaling
  • Get started by following our Quickstart