Last Week in Stream Data Integration & Stream Analytics – 4.5.2021

This is the 223rd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• Internal consistency in streaming systems by scattered-thoughts.net
• Event-driven APIs — Understanding the Principles by Dunith Dhanushka

Kafka

• Kafka Visualization by Softwaremill.com
• Three simple ideas to make your life easier with Kafka by Antón Rodríguez Yuste
• Kafka No Longer Requires ZooKeeper by Giorgos Myrianthous
• Streaming ETL with Confluent: Routing and Fan-Out of Apache Kafka Messages with ksqlDB by Li Dai

Flink

• Streaming Market Data with Flink SQL Part I: Streaming VWAP by Patrick Angeles & Krishnen Vytelingum

New Presentations

• REST the Events – REST APIs for Event-Driven Architecture by Mark Teehan
• The best practices for building Event-Driven APIs with Kafka by Iddo Gino

New Videos

• Resilient Edge Infrastructure for IoT Using Apache Kafka ft. Kai Waehner by Confluent Streaming Audio
• Workshop: Red Hat OpenShift Streams for Apache Kafka by Redhat
• Building streaming applications using a managed Kafka service by Evan Shortiss
• Data Management and Digital Transformation with Apache Kafka at Van Oord by Confluent Streaming Audio
• Set Your Sql Server Data Free With Kafka by Niels Berglund

New Podcasts

• Data Management and Digital Transformation with Apache Kafka at Van Oord by Confluent Streaming Audio (#156)
• Resilient Edge Infrastructure for IoT Using Apache Kafka ft. Kai Waehner by Confluent Streaming Audio (#157)

New Releases

• Apache Flink 1.13.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 28.4.2021

This is the 222nd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• Anatomy of an Event Streaming Platform — Part 2 by Dunith Dhanushka
• Digital Transformation And Event-Driven Architecture, Part Two: The Tools It Takes To Modernize Your Supply Chain With EDA by Denis King
• What is Streaming Analytics? by Laura Chu

Kafka

• Snowflake – Near Real-Time Ingestion from RDBMS using Debezium and Kafka by Karthik Venkatraman
• In the event-driven galaxy, which metadata matters most? by Yiannis Glampedakis
• Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients by Allison Walther
• Leader election and Sharding Practices at Wix microservices by Natan Silnitsky
• Announcing ksqlDB 0.17.0 by Leah Thomas
• How to Survive a Kafka Outage by Jakub Korab
• Kafka weather station by Igor De Souza

Spark

• What’s New in Apache Spark™ 3.1 Release for Structured Streaming by Yuanjian Li, Shixiong Zhu & Bo Zhang

StreamSets

• Data Engineering with a Purpose: Tackling Climate Change by Judy Ko

New Videos

• Powering Microservices Using Apache Kafka on Node.js with KafkaJS at Klarna by Confluent Streaming Audio (#155)
• From Zero to Hero with Kafka Connect by Robin Moffatt
• Machine Learning & Apache Kafka – the next generation of intell. software by Kai Waehner

New Podcasts

• Apache Kafka 2.8 – ZooKeeper Removal Update (KIP-500) and Overview of Latest Features by Confluent Streaming Audio (#154)
• Powering Microservices Using Apache Kafka on Node.js with KafkaJS at Klarna ft. Tommy Brunn by Confluent Streaming Audio (#155)

New Releases

• Apache Kafka 2.6.2
• ksqlDB 0.17

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 20.4.2021

This is the 221st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

Kafka

• Kafka producer and consumer written with Kotlin by Dan Lanky
• Setting Up Secure Networking in Confluent with Azure Private Link by Sunil Patil
• Kafka Monthly Digest – March 2021 by Mickael Maison
• Getting started with Kafka and Rust: Part 1 by Abhishek Gupta
• Event-Driven Architecture with Apache Kafka for .NET Developers Part 1 – Event Producer by Rahul Rai
• The Best Books to Learn Apache Kafka by John L. Watson
• Improving TCO and ROI with Confluent’s Cloud-Native Service for Apache Kafka by Lyndon Hedderly
• Confluent and Elastic Partner to Deliver Optimized Search and Real-Time Analytics by Jay Kreps
• Bridge between Edge / On Premise and Multi-Cloud with Apache Kafka by Kai Waehner
• How We Process One Billion Events Per Day With Kafka by Jonathan Ohms
• Anatomy of an Event Streaming Platform — Part 1 by Dunith Dhanushka
• Understanding Kafka Topic Partitions by Dunith Dhanushka
• Data streaming with Apache Kafka — a guide for data engineers by Bartosz Mikulski
• You Can Replace Kafka with a Database by Emil Koutanov
• Easily Discover and Test Kafka Topics with RapidAPI by Srivatsan Srinivasan
• Data streaming with Apache Kafka — a guide for data engineers by Bartosz Mikulski
• Debuting a Modern C++ API for Apache Kafka by Kenneth Jia & Benedek Thaler
• Apache Kafka for the Connected World — Vehicles, Factories, Cities, Digital Services by Kai Waehner
• Building the Confluent UI with React Hooks – Benefits and Lessons Learned by Mi Li
• Event Streams Are Nothing Without Action by Bernd Ruecker
• Beyond the Quickstart: Running Apache Kafka as a Service on Kubernetes by Gwen Shapira
• What’s New in Apache Kafka 2.8 by John Roesler

Flink

• Detecting Image Similarity in (Near) Real-time Using Apache Flink by Pinterest Engineering
• The Impact of Disks on RocksDB State Backend in Flink: A Case Study by Jun Qin
• Stateful Functions 3.0.0: Remote Functions Front and Center by Igal Shilman & Tzu-Li (Gordon) Tai

StreamSets

• Load Change Data Capture data from PostgreSQL to Amazon Redshift using StreamSets by Dash Desai

New Presentations

• Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and gRPC by Natan Silnitsky

New Videos

• Formula 1 telemetry processing using Kafka Streams by Paolo Patierno & Thomas Cooper
• Connecting Azure Cosmos DB with Apache Kafka – Better Together ft. Ryan CrawCour by Confluent
• A Gentle Introduction to Stream Processing by Nicolas Frankel
• Streaming with Structure by Katherine Stanley
• April Updates on Apache Pulsar by Sijie Guo
• ESPBench: The Enterprise Stream Processing Benchmark by Guenter Hesse

New Podcasts

• Resurrecting In-Sync Replicas with Automatic Observer Promotion ft. Anna McDonald by Confluent Streaming Audio (#151)
• Automated Cluster Operations in the Cloud ft. Rashmi Prabhu by Confluent Streaming Audio (#152)
• Connecting Azure Cosmos DB with Apache Kafka – Better Together ft. Ryan CrawCour by Confluent Streaming Audio (#153)

New Releases

• Debezium 1.5.0.Final
• Streamsheets 2.3
• Apache Kafka 2.8.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 06.4.2021

This is the 220th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• AsyncAPI, CloudEvents, OpenTelemetry: Which Event-Driven Specs Should Your DevOps Include? by Jesse Menning
• Real-time Data Infrastructure at Uber by Yupeng Fu & Chinmay Soman

Kafka

• Apache Kafka and MQTT (Part 3 of 5) – Manufacturing 4.0 and Industrial IoT by Kai Waehner
• Apache Kafka and MQTT (Part 4 of 5) – Mobility Services and Transportation by Kai Waehner
• Apache Kafka and MQTT (Part 5 of 5) – Smart City and 5G by Kai Waehner
• Kafka & Confluent at the Edge: Interview with Kai Waehner by Dasha Korotkykh
• Why You Need to Move to a Cloud-Native Apache Kafka Service Right Now by Josh Rosenberg
• Building an open source Scala gRPC/REST HTTP Proxy for Kafka (I) by Tom Lous
• Building an open source Scala gRPC/REST HTTP Proxy for Kafka (II) by Tom Lous
• How to Run Confluent on Windows in Minutes by Jim Galasyn
• Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana by Abhishek Walia
• Integrating Azure and Confluent: Real-Time Search Powered by Azure Cache for Redis and Spring Cloud by Ramya Oruganti & Abhishek Gupta
• How to Run Confluent on Windows in Minutes by Jim Galasyn
• Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper by Ben Stopford & Ismael Juma
• How to monitor containerized Kafka with Elastic Observability by Jamie Smith
• Loading CSV data into Confluent Cloud using the FilePulse connector by Robin Moffatt
• Source Kafka connect by Igor Souza
• Sink Kafka connect by Igor Souza
• Building your own Kafka Connect image with a custom resource by Jakub Stejskal
• Understanding JSON Schema Compatibility by Robert Yokota
• A great day out with Apache Kafka by Gunnar Morling & Hans-Peter Grashl
• A bash script to deploy ksqlDB queries automagically by Robin Moffatt
• KRaft aka KIP-500 and Raspberry PI by Igor Souza

Flink

• How to Build and Debug a Flink Pipeline Based in Event Time by Anton Rodriguez
• Apache Flink Continuous Deployment by Akhlaq Malik

StreamSets

• Smart Data Pipelines: Tools, Techniques, and Key Concepts by StreamSets
• 3 Important Trends in Data Integration by Sean Anderson

New Presentations

• Apache Kafka and MQTT – Overview, Comparison, Use Cases, Architectures by Kai Waehner & Dominik Obermaier
• Event-Driven Applications: Apache Kafka and Python by Francesco Tisiot

New Videos

• Apache Flink x Pulsar Virtual Meetup: Streaming SQL at Uber and Facebook by Girish Galiga
• Building a Telegram Bot with Apache Kafka, Go & ksqlDB by Robin Moffatt
• Build Event-Driven Microservices with Apache Kafka®, Kotlin, and Ktor by Viktor Gamov & Anton Arhipov

New Podcasts

• Smooth Scaling and Uninterrupted Processing with Apache Kafka ft. Sophie Blee-Goldman by Confluent Streaming Audio (#149)
• Building Real-Time Data Pipelines with Microsoft Azure, Databricks, and Confluent by Confluent Streaming Audio (#150)

New Releases

• Apache Pulsar 2.7.1

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 20.3.2021

This is the 219th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

Once more there is a lot of content, as I was quite busy the past few weeks and so it is compilation of the almost 3 weeks!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• Capturing Every Change From Shopify’s Sharded Monolith by John Martin & Adam Bellemare
• How to develop a digital transformation strategy based on Event-Driven Architecture by Tom Fairbairn
• Streaming Annotated Monthly – March 2021 by Antón
• Process your iPhone’s sensor data with an IoT Spreadsheet and MQTT by Kristian Raue

Kafka

• Real-Time Locating System with Kafka by Igor De Souza
• Azkarra Streams 0.9.0 release by Florian Hussonnois
• Quick profiling of data in Apache Kafka using kafkacat and visidata by Robin Moffatt
• Kafka Monthly Digest – February 2021 by Mickael Maison
• OCI Dashboard with Smashing/Dashing and Kafka by Igor De Souza
• Simplifying Apache Kafka Multi-Cluster Management Using Control Center and Cluster Registry by Shailesh Panwar
• Connected vehicles & Self-driving cars with Kafka by Igor De Souza
• App Modernization and Hybrid Cloud Architectures with Apache Kafka by Kai Waehner
• How to Tune RocksDB for Your Kafka Streams Application by Bruno Cadonna & Dhruba Borthakur
• Energy switch and smart home with Kafka by Igor De Souza
• Visualizing Kafka by Timothy Stepro
• Apache Kafka and MQTT (Part 1 of 5) – Overview and Comparison by Kai Waehner
• Apache Kafka and MQTT (Part 2 of 5) – V2X and Connected Vehicles by Kai Waehner
• Kafka Connect JDBC Sink deep-dive: Working with Primary Keys by Robin Moffatt
• Kafka Connect – SQLSyntaxErrorException: BLOB/TEXT column … used in key specification without a key length by Robin Moffatt
• Using ksqlDB to process data ingested from ActiveMQ with Kafka Connect by Robin Moffatt
• Creating your managed Kafka shortlist by Olivier de Garrigues
• Under the Hood of Real-Time Analytics with Apache Kafka and Pinot by Chinmay Soman
• Integrating Azure and Confluent: Ingesting Data to Azure Cosmos DB through Apache Kafka by Alicia Moniz & Siva Mullapudi
• Securing the Infrastructure of Confluent with HashiCorp Vault by Tim Ellis
• Integrating Apache Kafka Clients with CNCF Jaeger at Funding Circle Using OpenTelemetry by Nacho Munoz & Samir Hafez
• Understanding Non-Key Joins With the Quarkus Extension for Kafka Streams by Anisha Mohanty
• Data Driven Development for Stream Processing – Part 1 of 4: Introduction + Foundational Concepts by Caito S
• Data Driven Development for Stream Processing – Part 2 of 4: Anti-Patterns and The “Metrics-Driven Metrics” Cycle by Caito S
• Learning with LiveStreams: Cloud-Native Apache Kafka and Serverless Stream Processing by Viktor Gamov
• Introducing Azkarra Streams V0.9 by Florian Hussonnois
• Contrasting Kafka with Akka by Emil Koutanov
• Apache Kafka MirrorMaker 2 (MM2) Part 1: Theory by Paul Brebner
• Apache Kafka MirrorMaker 2 (MM2) Part 2: Practice by Paul Berebner

Streamsets

• Load Data from S3 to Snowflake and Use TensorFlow Model by Dash Desai

Streamsets

• Building Riviera: A Declarative Real-Time Feature Engineering Framework by Allen Wang & Kunal Shah

New Presentations

• Can Apache Kafka Replace a Database? by Kai Waehner
• Pragmatic Guide to Apache Kafka®’s Exactly Once Semantics by Gwen Shapira
• Evolving from Messaging to Event Streaming by Perry Krol & Mic Hussey
• Kafka in the Service Mesh, Not just API’s in disguise! by Johnny Mirza

New Videos

• How to stream data from MySQL to Apache Kafka® by Viktor Gamov
• App Modernization and Hybrid Cloud Architectures with Apache Kafka by Kai Waehner
• Streaming Concepts & Introduction to Flink – [BONUS] Exactly Once Fault Tolerance Guarantees by Alexander Fedulov
• Kafka Connect JDBC sink deep-dive: Working with Primary Keys by Robin Moffatt

New Podcasts

• Gamified Fitness at Synthesis Software Technologies Using Apache Kafka and IoT by Confluent Streaming Audio (#146)
• The Human Side of Apache Kafka and Microservices ft. SPOUD by Confluent Streaming Audio (#147)
• Event-Driven Architecture – Common Mistakes and Valuable Lessons ft. Simon Aubury by Confluent Streaming Audio (#148)

New Releases

• Apache Flink 1.12.2
• Debezium 1.5.0.Beta2

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 2.3.2021

This is the 218th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• Saga Orchestration for Microservices Using the Outbox Pattern by Gunnar Morling

Kafka

• From Oracle to Google Big Query by Kafka by Gleb
• Apache Kafka and SAP Integration with the Kafka Connect ODP Source Connector by Kai Waeher & Ronald Schertle
• How to Manage Secrets for Confluent Platform with Kubernetes and HashiCorp Vault by Amit Gupta
• Containerized Testing with Kerberos and SSH by Daniel Osvath
• Loading delimited data into Kafka – quick & dirty (but effective) by Robin Moffatt
• Apache Kafka for the Connected World – Vehicles, Factories, Cities, Digital Services by Kai Waehner
• Kafka at the edge with Raspberry PI by Igor De Souza
• To Pull or to Push Your Data with Kafka Connect? That Is the Question by Vitalii Rudenskyi

Streamsets

• How To Load Data Into Google BigQuery on Dataproc and AutoML by Dash Desai
• Optimizing Re-use for Data Quality to Scale Data Pipelines by James Parham

New Presentations

• Apache Kafka for Automotive Industry, Mobility Services & Smart City by Kai Waehner
• Schemas, streams, and grocery stores by Alexei Zenin
• Evolving from Messaging to Event Streaming by Perry Krol & Mic Hussey
• Journey to Event Driven Architecture by Suraj Pillai

New Videos

• Serverless Kotlin and Apache Kafka® with Google Cloud by Viktor Gamov & James Ward

New Releases

• Debezium 1.5.0.Beta1

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 15.2.2021

This is the 217th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• Why the UK Government’s COVID Vaccine Data Needs to be Real-Time & Event-Driven by Jamil Ahmed
• Part I. Designing Next-Gen Event-Driven application powered by Stateful Functions by Tymur Yarosh

Kafka

• 6 Things You Need to Know About Kafka Before Using it in a System Design Interview by Eileen Pangu
• How to Write a Connector for Kafka Connect – Deep Dive into Configuration Handling by Igor Buzatovic
• Automatic Observer Promotion Brings Fast and Safe Multi-Datacenter Failover with Confluent Platform 6.1 by Sanjana Kaundinya
• Introducing Confluent Platform 6.1 by Hasan Jilani
• Announcing ksqlDB 0.15 by Rohan Desai
• 42 Things You Can Stop Doing Once ZooKeeper Is Gone from Apache Kafka by Yeva Byzek
• Fully Reactive Stream Processing with Apache Kafka and Project Reactor by Sage Pierce
• Oracle CDC Source Premium Connector is Now Generally Available by Nathan Nam
• Event Streaming with Kafka as Foundation for a Smart City by Kai Waehner
• Exploring Data Pipelines in Apache Kafka with Streams Explorer by Salomon Popp
• Keys in ksqlDB, Unlocked by Victoria Xia
• Apache Kafka in the Airline, Aviation and Travel Industry by Kai Waehner
• Oracle to Kafka — Playing with Confluent’s new Oracle CDC Source Connector in Docker by Simon Aubury
• Lessons Learned from Running Apache Kafka at Scale at Pinterest by Ambud Sharma & Vahid Hashemian
• How to Build a Distributed Big Data Pipeline Using Kafka and Docker by Vignesh Perumal

Spark Streaming

• Apache Spark Structured Streaming-Output Sinks (3 of 6) by Neeraj Bhadani

Flink

• How to natively deploy Flink on Kubernetes with High-Availability (HA) by Yang Wang
• How to join streams in Apache Flink by Aris Koliopoulos

New Presentations

• Battle Tested Event-Driven Patterns for your Microservices Architecture by Natan Silnitsky

New Videos

• Streaming Processing and Testing with Spring Cloud Stream by Viktor Gamov
• Kafka Tutorials: Window Final Result by Loïc Divad
• ksqlDB HOWTO: Filtering by Robin Moffatt
• ksqlDB HOWTO: Schema Manipulation by Robin Moffatt
• ksqlDB HOWTO: Joins by Robin Moffatt
• ksqlDB HOWTO: Stateful Aggregates by Robin Moffatt
• ksqlDB HOWTO: Integration with other systems by Robin Moffatt
• ksqlDB HOWTO: Reserialising data in Apache Kafka by Robin Moffatt
• ksqlDB HOWTO: Split and Merge Kafka Topics by Robin Moffatt
• ksqlDB HOWTO: Handling Time by Robin Moffatt
• Mosquitto 2.0 webinar by Roger Light
• How to Count Messages in a Kafka Topic by Viktor Gamov

New Podcasts

• Confluent Platform 6.1 | What’s New in This Release + Updates by Confluent Streaming Audio (#143)
• Integrating Spring Boot with Apache Kafka ft. Viktor Gamov by Confluent Streaming Audio (#144)
• Becoming Data Driven with Apache Kafka and Stream Processing ft. Daniel Jagielski by Confluent Streaming Audio (#145)

New Releases

• ksqlDB 0.15.0
• Confluent 6.1.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 8.2.2021

This is the 216th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• Database vs Stream Processing — What it means for the Future of Data Management by Edward Huang

Kafka

• Real-time monitoring of Formula 1 telemetry data on Kubernetes with Grafana, Apache Kafka, and Strimzi by Paolo Patierno
• Kafka Monthly Digest – January 2021 by Mickael Maison
• Announcing the Confluent Community Forum by Robin Moffatt
• Infrastructure Checklist for Apache Kafka at the Edge by Kai Waehner
• 8 Years of Event Streaming with Apache Kafka by Tobias Kaymak
• Consuming Avro Data from Apache Kafka Topics and Schema Registry with Databricks and Confluent Cloud on Azure by Angela Chu & Gianluca Natali & Caio Moreno
• Fixing Debezium Connectors when they break on production by Aman Garg
• A Fault-tolerant Kafka Replication Tool across Multiple Datacenters that scales by Ning Zhang
• Docker free Kafka integration tests by Francesco Nobilia
• Omnichannel Retail and Customer 360 in Real Time with Apache Kafka by Kai Waehner
• 4 Steps for Kafka Rebalance – Notes From the Field by Wix Engineering
• Node.js ❤️ Apache Kafka – Getting Started with KafkaJS by Tommy Brunn

Spark Streaming

• Apache Spark Structured Streaming — Input Sources (2 of 6) by Neeraj Bhadani

Streamsets

• Data Preparation: Getting spark ready with StreamSets Data Collector by Rishi Jain
• CDC Pipeline From Mysql to Snowflake by Rishi Jain

New Presentations

• Kafka Tutorials – Window Final Result by Loic David

New Videos

• Stream Processing with Spring Cloud Stream and Kafka Streams | Livestreams 019 by Viktor Gamov
• Stream Processing & Streamsheets explained in 2 minutes! by Cedalo

New Podcasts

• Building a Microservices Architecture with Apache Kafka at Nationwide Building Society ft. Rob Jackson by Confluent Streaming Audio (#142)

New Releases

• Confluent Kafka Python 1.6.0
• Debezium 1.5.0.Alpha1 Released

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 1.2.2021

This is the 215th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

Was quite busy the past weeks so I might have missed some posts. But at least here is a new edition of my blog series 🙂

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• A Guide for the Developer’s Journey to Event-Driven Development by  Tamimi Ahmad
• Building Real-Time ML Pipelines with a Feature Store by Adi Hirschtein
• Event-Driven Architecture Can Clean Up Your Noisy Machine Learning Labels by Aline Guisky

Kafka

• Property Based Testing Confluent Server Storage for Fun and Safety by Lucas Bradstreet & Matthew Wong
• Helpful Tools for Apache Kafka Developers by Dave Klein
• How Apache Kafka Enables Podium to “Ship It and See What Happens” by Dan Conger
• Confluent and Microsoft Announce Strategic Alliance by Jay Kreps
• Powering Microservices at SEI Investments with Event Streaming by Steven Sermarini
• Event Streaming Across Networks and Corporate Firewalls Using PubNub and Confluent Platform by Stephen Blum & Dan Rosanova
• Better to Be Wrong Than Vague: Apache Kafka and Software Architecture Predictions for 2021 by Tim Berglund
• Apache Kafka Meets Table Football by Gaetan Collaud
• Running a self-managed Kafka Connect worker for Confluent Cloud by Robin Moffatt
• Introducing seamless integration between Microsoft Azure and Confluent Cloud by Ramya Oruganti
• Building a data lake: from batch to real-time using Kafka by Andrey Fadeev
• Use Cases for Apache Kafka in Retail by Kai Waehner
• Application of Kafka and Spark in Transaction Processing by Rushi Pradhan
• Combining strict order with massive parallelism using Kafka by Emil Koutanov
• A Hybrid Streaming Architecture for Smart Retail Stores with Apache Kafka by Kai Waehner

Flink

• Flink setup for development (and some IntelliJ Idea cool tricks) by Antón Rodríguez Yuste

Spark Streaming

• Apache Spark Structured Streaming — First Streaming Example (1 of 6) by Neeraj Bhadani

Streamsets

• 3 AWS Data Pipelines to Impress Your Boss by Thomas Ganka

New Presentations

• State-of-the-art Patterns for Efficient, Resilient, and Highly Available Global Kafka by Sanjana Kaundinya & Luke Knepper
• Streaming Data from Scylla to Kafka by Tim Berglund
• Apache Kafka for Smart Grid, Utilities and Energy Production by Kai Waehner

New Videos

• Kafka Streams Interactive Queries and gRPC by Viktor Gamov
• Kafka as a Platform: The Ecosystem from the Ground Up by Robin Moffatt

New Podcasts

• Scaling Developer Productivity with Apache Kafka ft. Mohinish Shaikh by Confluent Streaming Audio (#139)
• Distributed Systems Engineering with Apache Kafka ft. Guozhang Wang by Confluent Streaming Audio (#140)
• Examining Apache Kafka Performance Metrics ft. Alok Nikhil by Confluent Streaming Audio (#141)

New Releases

• Kafka 2.6.1
• Debezium 1.4.1.Final

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 11.1.2020

Happy New Year!

This is the 214th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

• The Big Little Guide to Message Queues by Sudhir Jonathan

Kafka

• Apache Kafka is NOT Hard Real Time BUT Used Everywhere in Automotive and Industrial IoT by Kai Waehner
• Implementing mTLS and Securing Apache Kafka at Zendesk by Joy Nag
• Kafka Connect – Deep Dive into Single Message Transforms by Robin Moffatt
• Streaming data into Kafka S01/E04 — Parsing log files using Grok Expressions by Florian Hussonnois
• Defining custom topic properties with Kafka Connect source connectors by Robin Moffatt
• Real Time Locating System (RTLS) with Apache Kafka for Transportation and Logistics by Kai Waehner
• Optimizing Kafka consumers by Paul Mellor
• Considerations when moving your Apache Kafka to the cloud by John Hammink
• Avro, Kafka and the Schema Registry: Clearing Things Up by David Hettler
• How Replication and ISR work in Apache Kafka by Conduktor
• Change Data Analysis with Debezium and Apache Pinot by Kenny Bastani
• Kafka As A Database? Yes Or No by David Xiang
• Putting Apache Kafka to REST: Confluent REST Proxy 6.0 by Rigen Bezerra de Melo

Flink

• What’s New in the Pulsar Flink Connector 2.7.0 by Jianyun Zhao & Jennifer Huang

New Presentations

• Event Driven Architecture with Quarkus,Kafka, Kubernetes by Jeremy Davis
• Microservices Architecture – Cloud Native Apps by Araf Karsh Hamid

New Videos

• Becoming an Event-Driven Enterprise by Tech Chat (#86)

New Podcasts

• Event Streaming Trends and Predictions for 2021 ft. Gwen Shapira, Ben Stopford, and Michael Noll by Confluent Streaming Audio (#137)
• Change Data Capture and Kafka Connect on Microsoft Azure ft. Abhishek Gupta by Confluent Streaming Audio (#138)

New Releases

• Debezium 1.4.0
• Apache Beam 2.27.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!