Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 06:50 on June 21, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 20.6.2021

This is the 226th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

AsyncAPI, CloudEvents, OpenTelemetry: Which Event-Driven Specs Should Your DevOps Include? by Jesse Menning
6 Eye-Opening Benefits of Real-Time Data by Devin Partida
Designing Domain Events by Deepak Narayana Rao
Designing your APIs with AsyncAPI (Part 1) by Jonas Lagoni

Kafka

Streaming ETL and Analytics on Confluent with Maritime AIS Data by Robin Moffatt
Detecting Patterns of Behaviour in Streaming Maritime AIS Data with Confluent by Robin Moffatt
Are We There Yet? The Query Your Database Can’t Answer by Michael Drogalis
Kafka Monthly Digest – May 2021 by Mickael Maison
Why developers use Confluent to manage Apache Kafka by Tim Berglund
Why RudderStack Used Postgres Over Apache Kafka for Streaming Engine by RudderStack
Eventual Consistency with Spring for Apache Kafka: Part 1 of 2 by Gary A. Stafford
Eventual Consistency with Spring for Apache Kafka: Part 2 of 2 by Gary A. Stafford
How to Integrate Apache Kafka into your Web-App using AtomicKafka by David Behmoaras
Setting up AWS Lambda with an Apache Kafka cluster within a VPC by James Beswick
Retrying Apache Kafka Messages With A Dedicated Java Micro-service by Baron Chan
Event-Driven APIs and Schema Governance for Apache Kafka by Hugo Guerrero
Apache Kafka in the Insurance Industry by Kai Waehner
Serverless Event Driven Systems with Confluent Cloud and AWS Lambda by Adam McQuistan
Optimizing Kafka broker configuration by Paul Mellor
Introducing Health+ with Confluent Platform 6.2 by Hasan Jilani
How to Better Manage Apache Kafka by Creating Kafka Messages from within Control Center by Rinka Yoshida
How to Better Manage Apache Kafka by Exporting Kafka Messages via Control Center by Rinka Yoshida
How to Better Manage Apache Kafka with Improved Topic Inspection via Last-Produced Timestamp by Rinka Yoshida
How to Better Manage Apache Kafka by Removing Residue Data with Control Center Cleanup Script by Rinka Yoshida
Gently Down the Stream by Mitch Seymour
Real time anomaly detection with Apache Kafka and Python by Rodrigo Arenas
Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka by Guozhang Wang

Redis Streams

Redis Streams in Action: Part 1 (Intro and overview) by Abhishek Gupta
Redis Streams in Action — Part 2 (Tweets consumer app) by Abhishek Gupta
Redis Streams in Action — Part 3 (Tweets processor app) by Abhishek Gupta

Flink

Build a Streaming SQL Pipeline with Apache Flink and Apache Kafka by Francesco Tisiot

Pulsar

Four Reasons Why Apache Pulsar is Essential to the Modern Data Stack by DataStax
5 More Reasons to Choose Apache Pulsar Over Apache Kafka by Chris Bartholomew
DataStax launches beta of Astra Streaming service by Tony Baer
Apache Pulsar Launches 2.8: Unified Messaging and Streaming With Transactions by Matteo Merli & Sijie Guo
Why developers should use Apache Pulsar by Patrick McFadin

StreamSets

Snowpark + StreamSets = Powerful Data Engineering for Snowflake Data Cloud, Simplified by Judy Ko

New Presentations

Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies? by Kai Waehner
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka by Kai Waehner & Gianluca Natali
Why Cloud-Native Kafka Matters: 4 Reasons to Stop Managing it Yourself by Greg Murphy
Can Apache Kafka Replace a Database? – The 2021 Update by Kai Waehner
Making Kafka Cloud Native by Jay Kreps
Battle-tested event-driven patterns for your microservices architecture by Natan Silnitsky
Application modernization patterns with apache kafka, debezium, and kubernetes by Bilgin Ibryam
How Adobe uses Structured Streaming at Scale by Yeshwanth Vijayakumar

New Videos

Message-driven microservices with Kafka and Micronaut by Graeme Rocher
Machine Learning Inference in Flink with ONNX by Colin Jermain
Bootiful Stream Processing! by Viktor Gamov

New Podcasts

Adopting OpenTelemetry in Confluent and Beyond by Confluent Streaming Audio (#161)
Confluent Platform 6.2 | What’s New in This Release + Updates by Confluent Streaming Audio (#162)
Boosting Security for Apache Kafka with Confluent Cloud Private Link ft. Dan LaMotte by Confluent Streaming Audio (#163)
Streaming Annotated May 2021

New Releases

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 20:04 on May 26, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 26.5.2021

This is the 225th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

Streaming Annotated Monthly – May 2021 by Antón
How Event-driven Technologies Can Transform the Shipping Industry by Tom Fairbairn
Kafka vs RabbitMQ vs AWS SNS/SQS: Which Broker to Choose? by Team Aspecto

Kafka

Kafka Summit Europe 2021 Recap by Tim Berglund
Confluent CLI Launches Exciting New Features and an Intuitive UI by David Hyde
Best Practices for Apache Kafka® – 5 Tips Every Developer Should Know by Bill Bejeck
Tips for designing payloads by Lorna Mitchell
Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka® by Guozhang Wang, Lei Chen, Ayusman Dikshit, Jason Gustafson, Boyang Chen, Matthias J. Sax, John Roesler, Sophie Blee-Goldman, Bruno Cadonna, Apurva Mehta, Varun Madan, and Jun Rao
Introducing Cluster RBAC, Audit Logs, and BYOK for Enterprise-Grade Security by Vahid Fereydouny & Greg Murphy
Deploying an Apache Kafka mock service with Microcks by Hugo Guerrero
Error Handling Patterns for Apache Kafka Applications by Gerardo Villeda
Low Latency Data Streaming with Apache Kafka and Cloud-Native 5G Infrastructure by Kai Waehner
Modernize Your Business with Confluent’s Connector Portfolio by Aditya Chidurala & Kevin Chao
Kafka & Confluent at the Edge: Interview with Kai Waehner by Dasha Korotkykh
Announcing ksqlDB 0.18.0 by Zara Lim
Continuous Deployment of Confluent with Ansible Tower by Domenic Bove

Flink

Streaming Market Data with Flink SQL Part II: Intraday Value-at-Risk by Patrick Angeles & Krishnen Vytelingum

New Presentations

Stream processing with ksqlDB and Apache Kafka by Keith Resar
REST the Events – REST APIs for Event-Driven Architecture by Mark Teehan

New Videos

Kafka Summit Europe 2021 Keynote: How to Build the Data Mesh Foundation by Zhamak Dehghani
How to Deploy Apache Pulsar Using Helm Charts by Axel Sirota
Ask the Experts: How to Build Modern Applications with Confluent on Azure by Michael Cooper & Stephanie Lemus & Alicia Moniz & Dan Rosanova
Engaging Database Partials with Apache Kafka for Distributed System Consistency ft. Pat Helland by Confluent Streaming Audio (#159)
Running Apache Kafka Efficiently on the Cloud ft. Adithya Chandra by Confluent Streaming Audio (#160)
Harnessing the power of Redis and Apache Kafka to crunch high-velocity time series data by Abhishek Gupta

New Podcasts

Ep.90 ‘Apache Kafka, Real-Time Stream Processing and ksqlDB’ Drill to Detail with Special Guest Michael Drogalis
Engaging Database Partials with Apache Kafka for Distributed System Consistency ft. Pat Helland by Confluent Streaming Audio (#159)
Running Apache Kafka Efficiently on the Cloud ft. Adithya Chandra by Confluent Streaming Audio (#160)

New Releases

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 21:42 on May 17, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 17.5.2021

This is the 224th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

5 Factors to Consider When Choosing a Stream Processing Engine by Dinesh Chandrasekhar

Kafka

Apache Kafka Storage Architecture by Kadir Alan
In the event-driven galaxy, which metadata matters most? by Yiannis Glampedakis
Announcing the MongoDB Atlas Sink and Source Connectors in Confluent Cloud by Nathan Nam
Comparison of Open Source Apache Kafka vs Vendors including Confluent, Cloudera, Red Hat, Amazon MSK by Kai Waehner
Deploying Kafka with Let’s Encrypt certificates by Jakub Scholz
Kafka API is the De Facto Standard API for Event Streaming like Amazon S3 for Object Storage by Kai Waehner
Using kafka-merge-purge to Deal with Failure in an Event-Driven System at FLYERALARM by Marvin Roesch
Introducing Confluent for Kubernetes by Rohit Bakhshi
Monitoring of an Underwater Drone by Felipe de Pontes Adachi
Kafka Monthly Digest – April 2021 by Mickael Maison
Resiliency and Disaster Recovery with Kafka by Engin Yoeyen
Exploring ZooKeeper-less Kafka by Gunnar Morling

Pulsar

Function Mesh – Simplify Complex Streaming Jobs in Cloud by Neng Lu & Rui Fu

Flink

Introduction to Apache Flink by Aiven Team
SQL Query Optimization with Ververica Platform 2.4 by Nico Kruber

StreamSets

StreamSets Transformer pipeline to Dataproc in 5 mins by Rishi Jain

New Presentations

5 lessons learned for Successful Migration to Confluent Cloud by Natan Silnitsky
How ksqlDB works by Michael Drogalis
On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data by Robin Moffatt
Why MQTT and Kafka are a match made in heaven by Margaretha Erber & Dominik Obermaier
Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30 min by Florian Hussonnois

New Videos

Real-time streaming now faster, easier with Confluent Cloud on Azure by Scott Hanselman & Rob Caron
Kafka Summit Europe 2021 Keynote: Making Kafka Cloud Native (Confluent) by Jay Kreps
Golang Microservices: Events Streaming using Apache Kafka by Mario Carrion
Apache Kafka for Oil & Gas, Smart Grid, and Energy Utilities by Kai Waehner
Stateful stream processing with Apache Flink by Kundan Kumar

New Podcasts

ksqlDB Database for Streaming Events by Michael Drogalis
The Truth About ZooKeeper Removal and the KIP-500 Release in Apache Kafka ft. Jason Gustafson and Colin McCabe by Confluent Streaming Audio (#158)

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 21:36 on May 4, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 4.5.2021

This is the 223rd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

Internal consistency in streaming systems by scattered-thoughts.net
Event-driven APIs — Understanding the Principles by Dunith Dhanushka

Kafka

Kafka Visualization by Softwaremill.com
Three simple ideas to make your life easier with Kafka by Antón Rodríguez Yuste
Kafka No Longer Requires ZooKeeper by Giorgos Myrianthous
Streaming ETL with Confluent: Routing and Fan-Out of Apache Kafka Messages with ksqlDB by Li Dai

Flink

Streaming Market Data with Flink SQL Part I: Streaming VWAP by Patrick Angeles & Krishnen Vytelingum

New Presentations

REST the Events – REST APIs for Event-Driven Architecture by Mark Teehan
The best practices for building Event-Driven APIs with Kafka by Iddo Gino

New Videos

Resilient Edge Infrastructure for IoT Using Apache Kafka ft. Kai Waehner by Confluent Streaming Audio
Workshop: Red Hat OpenShift Streams for Apache Kafka by Redhat
Building streaming applications using a managed Kafka service by Evan Shortiss
Data Management and Digital Transformation with Apache Kafka at Van Oord by Confluent Streaming Audio
Set Your Sql Server Data Free With Kafka by Niels Berglund

New Podcasts

Data Management and Digital Transformation with Apache Kafka at Van Oord by Confluent Streaming Audio (#156)
Resilient Edge Infrastructure for IoT Using Apache Kafka ft. Kai Waehner by Confluent Streaming Audio (#157)

New Releases

Apache Flink 1.13.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 20:37 on April 27, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 28.4.2021

This is the 222nd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

Anatomy of an Event Streaming Platform — Part 2 by Dunith Dhanushka
Digital Transformation And Event-Driven Architecture, Part Two: The Tools It Takes To Modernize Your Supply Chain With EDA by Denis King
What is Streaming Analytics? by Laura Chu

Kafka

Snowflake – Near Real-Time Ingestion from RDBMS using Debezium and Kafka by Karthik Venkatraman
In the event-driven galaxy, which metadata matters most? by Yiannis Glampedakis
Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients by Allison Walther
Leader election and Sharding Practices at Wix microservices by Natan Silnitsky
Announcing ksqlDB 0.17.0 by Leah Thomas
How to Survive a Kafka Outage by Jakub Korab
Kafka weather station by Igor De Souza

Spark

What’s New in Apache Spark™ 3.1 Release for Structured Streaming by Yuanjian Li, Shixiong Zhu & Bo Zhang

StreamSets

Data Engineering with a Purpose: Tackling Climate Change by Judy Ko

New Videos

Powering Microservices Using Apache Kafka on Node.js with KafkaJS at Klarna by Confluent Streaming Audio (#155)
From Zero to Hero with Kafka Connect by Robin Moffatt
Machine Learning & Apache Kafka – the next generation of intell. software by Kai Waehner

New Podcasts

Apache Kafka 2.8 – ZooKeeper Removal Update (KIP-500) and Overview of Latest Features by Confluent Streaming Audio (#154)
Powering Microservices Using Apache Kafka on Node.js with KafkaJS at Klarna ft. Tommy Brunn by Confluent Streaming Audio (#155)

New Releases

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 19:02 on April 20, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 20.4.2021

This is the 221st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

Kafka

Kafka producer and consumer written with Kotlin by Dan Lanky
Setting Up Secure Networking in Confluent with Azure Private Link by Sunil Patil
Kafka Monthly Digest – March 2021 by Mickael Maison
Getting started with Kafka and Rust: Part 1 by Abhishek Gupta
Event-Driven Architecture with Apache Kafka for .NET Developers Part 1 – Event Producer by Rahul Rai
The Best Books to Learn Apache Kafka by John L. Watson
Improving TCO and ROI with Confluent’s Cloud-Native Service for Apache Kafka by Lyndon Hedderly
Confluent and Elastic Partner to Deliver Optimized Search and Real-Time Analytics by Jay Kreps
Bridge between Edge / On Premise and Multi-Cloud with Apache Kafka by Kai Waehner
How We Process One Billion Events Per Day With Kafka by Jonathan Ohms
Anatomy of an Event Streaming Platform — Part 1 by Dunith Dhanushka
Understanding Kafka Topic Partitions by Dunith Dhanushka
Data streaming with Apache Kafka — a guide for data engineers by Bartosz Mikulski
You Can Replace Kafka with a Database by Emil Koutanov
Easily Discover and Test Kafka Topics with RapidAPI by Srivatsan Srinivasan
Data streaming with Apache Kafka — a guide for data engineers by Bartosz Mikulski
Debuting a Modern C++ API for Apache Kafka by Kenneth Jia & Benedek Thaler
Apache Kafka for the Connected World — Vehicles, Factories, Cities, Digital Services by Kai Waehner
Building the Confluent UI with React Hooks – Benefits and Lessons Learned by Mi Li
Event Streams Are Nothing Without Action by Bernd Ruecker
Beyond the Quickstart: Running Apache Kafka as a Service on Kubernetes by Gwen Shapira
What’s New in Apache Kafka 2.8 by John Roesler

Flink

Detecting Image Similarity in (Near) Real-time Using Apache Flink by Pinterest Engineering
The Impact of Disks on RocksDB State Backend in Flink: A Case Study by Jun Qin
Stateful Functions 3.0.0: Remote Functions Front and Center by Igal Shilman & Tzu-Li (Gordon) Tai

StreamSets

Load Change Data Capture data from PostgreSQL to Amazon Redshift using StreamSets by Dash Desai

New Presentations

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and gRPC by Natan Silnitsky

New Videos

Formula 1 telemetry processing using Kafka Streams by Paolo Patierno & Thomas Cooper
Connecting Azure Cosmos DB with Apache Kafka – Better Together ft. Ryan CrawCour by Confluent
A Gentle Introduction to Stream Processing by Nicolas Frankel
Streaming with Structure by Katherine Stanley
April Updates on Apache Pulsar by Sijie Guo
ESPBench: The Enterprise Stream Processing Benchmark by Guenter Hesse

New Podcasts

Resurrecting In-Sync Replicas with Automatic Observer Promotion ft. Anna McDonald by Confluent Streaming Audio (#151)
Automated Cluster Operations in the Cloud ft. Rashmi Prabhu by Confluent Streaming Audio (#152)
Connecting Azure Cosmos DB with Apache Kafka – Better Together ft. Ryan CrawCour by Confluent Streaming Audio (#153)

New Releases

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 07:01 on April 6, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 06.4.2021

This is the 220th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

AsyncAPI, CloudEvents, OpenTelemetry: Which Event-Driven Specs Should Your DevOps Include? by Jesse Menning
Real-time Data Infrastructure at Uber by Yupeng Fu & Chinmay Soman

Kafka

Apache Kafka and MQTT (Part 3 of 5) – Manufacturing 4.0 and Industrial IoT by Kai Waehner
Apache Kafka and MQTT (Part 4 of 5) – Mobility Services and Transportation by Kai Waehner
Apache Kafka and MQTT (Part 5 of 5) – Smart City and 5G by Kai Waehner
Kafka & Confluent at the Edge: Interview with Kai Waehner by Dasha Korotkykh
Why You Need to Move to a Cloud-Native Apache Kafka Service Right Now by Josh Rosenberg
Building an open source Scala gRPC/REST HTTP Proxy for Kafka (I) by Tom Lous
Building an open source Scala gRPC/REST HTTP Proxy for Kafka (II) by Tom Lous
How to Run Confluent on Windows in Minutes by Jim Galasyn
Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana by Abhishek Walia
Integrating Azure and Confluent: Real-Time Search Powered by Azure Cache for Redis and Spring Cloud by Ramya Oruganti & Abhishek Gupta
How to Run Confluent on Windows in Minutes by Jim Galasyn
Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper by Ben Stopford & Ismael Juma
How to monitor containerized Kafka with Elastic Observability by Jamie Smith
Loading CSV data into Confluent Cloud using the FilePulse connector by Robin Moffatt
Source Kafka connect by Igor Souza
Sink Kafka connect by Igor Souza
Building your own Kafka Connect image with a custom resource by Jakub Stejskal
Understanding JSON Schema Compatibility by Robert Yokota
A great day out with Apache Kafka by Gunnar Morling & Hans-Peter Grashl
A bash script to deploy ksqlDB queries automagically by Robin Moffatt
KRaft aka KIP-500 and Raspberry PI by Igor Souza

Flink

How to Build and Debug a Flink Pipeline Based in Event Time by Anton Rodriguez
Apache Flink Continuous Deployment by Akhlaq Malik

StreamSets

Smart Data Pipelines: Tools, Techniques, and Key Concepts by StreamSets
3 Important Trends in Data Integration by Sean Anderson

New Presentations

Apache Kafka and MQTT – Overview, Comparison, Use Cases, Architectures by Kai Waehner & Dominik Obermaier
Event-Driven Applications: Apache Kafka and Python by Francesco Tisiot

New Videos

Apache Flink x Pulsar Virtual Meetup: Streaming SQL at Uber and Facebook by Girish Galiga
Building a Telegram Bot with Apache Kafka, Go & ksqlDB by Robin Moffatt
Build Event-Driven Microservices with Apache Kafka®, Kotlin, and Ktor by Viktor Gamov & Anton Arhipov

New Podcasts

Smooth Scaling and Uninterrupted Processing with Apache Kafka ft. Sophie Blee-Goldman by Confluent Streaming Audio (#149)
Building Real-Time Data Pipelines with Microsoft Azure, Databricks, and Confluent by Confluent Streaming Audio (#150)

New Releases

Apache Pulsar 2.7.1

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 13:51 on March 20, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 20.3.2021

This is the 219th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

Once more there is a lot of content, as I was quite busy the past few weeks and so it is compilation of the almost 3 weeks!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

Capturing Every Change From Shopify’s Sharded Monolith by John Martin & Adam Bellemare
How to develop a digital transformation strategy based on Event-Driven Architecture by Tom Fairbairn
Streaming Annotated Monthly – March 2021 by Antón
Process your iPhone’s sensor data with an IoT Spreadsheet and MQTT by Kristian Raue

Kafka

Real-Time Locating System with Kafka by Igor De Souza
Azkarra Streams 0.9.0 release by Florian Hussonnois
Quick profiling of data in Apache Kafka using kafkacat and visidata by Robin Moffatt
Kafka Monthly Digest – February 2021 by Mickael Maison
OCI Dashboard with Smashing/Dashing and Kafka by Igor De Souza
Simplifying Apache Kafka Multi-Cluster Management Using Control Center and Cluster Registry by Shailesh Panwar
Connected vehicles & Self-driving cars with Kafka by Igor De Souza
App Modernization and Hybrid Cloud Architectures with Apache Kafka by Kai Waehner
How to Tune RocksDB for Your Kafka Streams Application by Bruno Cadonna & Dhruba Borthakur
Energy switch and smart home with Kafka by Igor De Souza
Visualizing Kafka by Timothy Stepro
Apache Kafka and MQTT (Part 1 of 5) – Overview and Comparison by Kai Waehner
Apache Kafka and MQTT (Part 2 of 5) – V2X and Connected Vehicles by Kai Waehner
Kafka Connect JDBC Sink deep-dive: Working with Primary Keys by Robin Moffatt
Kafka Connect – SQLSyntaxErrorException: BLOB/TEXT column … used in key specification without a key length by Robin Moffatt
Using ksqlDB to process data ingested from ActiveMQ with Kafka Connect by Robin Moffatt
Creating your managed Kafka shortlist by Olivier de Garrigues
Under the Hood of Real-Time Analytics with Apache Kafka and Pinot by Chinmay Soman
Integrating Azure and Confluent: Ingesting Data to Azure Cosmos DB through Apache Kafka by Alicia Moniz & Siva Mullapudi
Securing the Infrastructure of Confluent with HashiCorp Vault by Tim Ellis
Integrating Apache Kafka Clients with CNCF Jaeger at Funding Circle Using OpenTelemetry by Nacho Munoz & Samir Hafez
Understanding Non-Key Joins With the Quarkus Extension for Kafka Streams by Anisha Mohanty
Data Driven Development for Stream Processing – Part 1 of 4: Introduction + Foundational Concepts by Caito S
Data Driven Development for Stream Processing – Part 2 of 4: Anti-Patterns and The “Metrics-Driven Metrics” Cycle by Caito S
Learning with LiveStreams: Cloud-Native Apache Kafka and Serverless Stream Processing by Viktor Gamov
Introducing Azkarra Streams V0.9 by Florian Hussonnois
Contrasting Kafka with Akka by Emil Koutanov
Apache Kafka MirrorMaker 2 (MM2) Part 1: Theory by Paul Brebner
Apache Kafka MirrorMaker 2 (MM2) Part 2: Practice by Paul Berebner

Streamsets

Load Data from S3 to Snowflake and Use TensorFlow Model by Dash Desai

Streamsets

Building Riviera: A Declarative Real-Time Feature Engineering Framework by Allen Wang & Kunal Shah

New Presentations

Can Apache Kafka Replace a Database? by Kai Waehner
Pragmatic Guide to Apache Kafka®’s Exactly Once Semantics by Gwen Shapira
Evolving from Messaging to Event Streaming by Perry Krol & Mic Hussey
Kafka in the Service Mesh, Not just API’s in disguise! by Johnny Mirza

New Videos

How to stream data from MySQL to Apache Kafka® by Viktor Gamov
App Modernization and Hybrid Cloud Architectures with Apache Kafka by Kai Waehner
Streaming Concepts & Introduction to Flink – [BONUS] Exactly Once Fault Tolerance Guarantees by Alexander Fedulov
Kafka Connect JDBC sink deep-dive: Working with Primary Keys by Robin Moffatt

New Podcasts

Gamified Fitness at Synthesis Software Technologies Using Apache Kafka and IoT by Confluent Streaming Audio (#146)
The Human Side of Apache Kafka and Microservices ft. SPOUD by Confluent Streaming Audio (#147)
Event-Driven Architecture – Common Mistakes and Valuable Lessons ft. Simon Aubury by Confluent Streaming Audio (#148)

New Releases

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 20:02 on March 2, 2021
Tags: kafka ( 228 ), kafka-streams ( 156 ), nifi ( 218 ), spark ( 133 ), spark-streaming ( 199 ), Stream Processing ( 221 )

Last Week in Stream Data Integration & Stream Analytics – 2.3.2021

This is the 218th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

Saga Orchestration for Microservices Using the Outbox Pattern by Gunnar Morling

Kafka

From Oracle to Google Big Query by Kafka by Gleb
Apache Kafka and SAP Integration with the Kafka Connect ODP Source Connector by Kai Waeher & Ronald Schertle
How to Manage Secrets for Confluent Platform with Kubernetes and HashiCorp Vault by Amit Gupta
Containerized Testing with Kerberos and SSH by Daniel Osvath
Loading delimited data into Kafka – quick & dirty (but effective) by Robin Moffatt
Apache Kafka for the Connected World – Vehicles, Factories, Cities, Digital Services by Kai Waehner
Kafka at the edge with Raspberry PI by Igor De Souza
To Pull or to Push Your Data with Kafka Connect? That Is the Question by Vitalii Rudenskyi

Streamsets

How To Load Data Into Google BigQuery on Dataproc and AutoML by Dash Desai
Optimizing Re-use for Data Quality to Scale Data Pipelines by James Parham

New Presentations

Apache Kafka for Automotive Industry, Mobility Services & Smart City by Kai Waehner
Schemas, streams, and grocery stores by Alexei Zenin
Evolving from Messaging to Event Streaming by Perry Krol & Mic Hussey
Journey to Event Driven Architecture by Suraj Pillai

New Videos

Serverless Kotlin and Apache Kafka® with Google Cloud by Viktor Gamov & James Ward

New Releases

Debezium 1.5.0.Beta1

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!