Learn the differences between an event-driven streaming platform like Apache Kafka and middleware like Message Queues (MQ), Extract-Transform-Load (ETL), and Enterprise Service Bus (ESB). Including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.
This post shares my slide deck and video recording. I discuss the differences between Apache Kafka as Event Streaming Platform and integration middleware. Learn if they are friends, enemies, or frenemies.
Problems of Legacy Middleware
Extract-Transform-Load (ETL) is still a widely-used pattern to move data between different systems via batch processing. Due to its challenges in today’s world where real-time is the new standard, an Enterprise Service Bus (ESB) is used in many enterprises as integration backbone between any kind of microservice, legacy application, or cloud service to move data via SOAP/REST Web Services or other technologies.
Stream Processing is often added as its own component in the enterprise architecture for correlation of different events to implement contextual rules and stateful analytics. Using all these components introduces challenges and complexities in development and operations. For more info ETL Testing Certification
Apache Kafka: An Open Source Event Streaming Platform
This session discusses how teams in different industries solve these challenges by building a native event streaming platform from the ground up instead of using ETL and ESB tools in their architecture. This allows us to build and deploy independent, mission-critical streaming, real-time applications, and microservices.
The architecture leverages distributed processing and fault-tolerance with fast failover, no-downtime, rolling deployments, and the ability to reprocess events, so you can recalculate output when your code changes. Integration and Stream Processing are still key functionality but can be realized in real time natively instead of using additional ETL, ESB or Stream Processing tools. Learn more from ETL Testing Training
A concrete example architecture shows how to build a complete streaming platform leveraging the widely-adopted open-source framework Apache Kafka to build a mission-critical, scalable, highly performant streaming platform. Messaging, integration, and stream processing are all built on top of the same strong foundation of Kafka; deployed on-premise, in the cloud, or in hybrid environments. In addition, the open source Confluent projects, based on top of Apache Kafka, add additional features like a Schema Registry, additional clients for programming languages like Go or C, or many pre-built connectors for various technologies.
To get in-depth knowledge, enroll for a live free demo on ETL Testing Online Training
MongoDB Programming 