Data Streaming Platform

Enterprise-grade Apache Kafka and Confluent Platform deployment on AWS — a fully automated, highly available, KRaft-mode streaming infrastructure for real-time data pipelines and event-driven architectures.

Why Choose Data Streaming Platform?

High-Availability Kafka Cluster

Three-node cluster spanning multiple availability zones ensures continuous operation and fault tolerance for mission-critical data streams.

ZooKeeper-Free KRaft Mode

Modern KRaft consensus protocol eliminates ZooKeeper dependency, reducing operational complexity and improving cluster resiliency.

Fully Automated Deployment

Infrastructure provisioning and service configuration are fully automated with Terraform and Ansible using the official cp-ansible collection.

Secure Private Networking

Kafka brokers run in private subnets with no public exposure; SSM port forwarding and VPC peering provide controlled application access.

Complete Confluent Platform

Schema Registry, Kafka Connect, ksqlDB, REST Proxy, and Control Center deployed alongside brokers for a complete streaming ecosystem.

Scalable for Production

Environment-specific configurations scale from development (t3.xlarge, 50GB) to production (r5.xlarge, 500GB) with a single configuration change.

Core Features

Apache Kafka (KRaft)

Multi-broker Kafka cluster in KRaft mode with internal DNS-based bootstrap servers across all availability zones

Schema Registry

Centralised schema management for Avro, Protobuf, and JSON Schema with compatibility enforcement

Kafka Connect

Distributed connector framework for ingesting and exporting data to databases, cloud storage, and SaaS systems

ksqlDB

SQL-based stream processing for real-time aggregations, joins, filters, and enrichment of Kafka topics

Control Center

Web-based monitoring and management UI for cluster health, consumer lag, topic inspection, and alerting

Kafka REST Proxy

RESTful interface for producing and consuming messages without a native Kafka client

Modules & Components

1Kafka Brokers & Controllers

Core Kafka cluster with KRaft-mode controllers for leader election and metadata management

3-Node KRaft Cluster
Multi-AZ Broker Placement
Internal DNS Bootstrap (port 9092)
Encrypted EBS Data Volumes
Controller Quorum Management
Topic Replication Configuration

2Schema Registry

Centralised schema store for enforcing data contracts across producers and consumers

Avro / Protobuf / JSON Schema
Schema Compatibility Levels
Subject Namespacing
REST API (port 8081)
Schema Evolution Management
Producer & Consumer Integration

3Kafka Connect

Scalable connector platform for data integration between Kafka and external systems

Distributed Connector Workers
Source & Sink Connectors
REST Management API (port 8083)
Connector Configuration Management
Dead Letter Queue Support
Custom Connector Deployment

4ksqlDB

Stream processing engine for real-time SQL queries on Kafka topics

Streaming SQL Queries
Persistent Queries
Aggregations & Windowing
Stream-Table Joins
REST API (port 8088)
Pull & Push Queries

5Control Center

Centralised observability and management for the entire Confluent Platform

Cluster Health Monitoring
Consumer Lag Tracking
Topic & Schema Browser
Connector Management UI
Alerting & Notifications
Web UI (port 9021)

6Access & Integration

Secure connectivity patterns for applications and operational access

SSM Port Forwarding
VPC Peering Integration
socat Port Relay (Broker Discovery)
Kafka REST Proxy (port 8082)
Cross-VPC DNS Resolution
Private Subnet Isolation

Transform Your Business Today

Join hundreds of companies leveraging Data Streaming Platform to streamline operations and drive growth.