Last Week in Stream Data Integration & Stream Analytics – 19.1.2020

This is the 181st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

General

• Introducing Apache Druid by Imply

Apache Kafka / Kafka Streams / Confluent Platform

• Streams and Tables in Apache Kafka: A Primer by Michael Noll
• Streams and Tables in Apache Kafka: Topics, Partitions, and Storage Fundamentals by Michael Noll
• Streams and Tables in Apache Kafka: Processing Fundamentals with Kafka Streams and ksqlDB by Michael Noll
• Streams and Tables in Apache Kafka: Elasticity, Fault Tolerance, and Other Advanced Concepts by Michael Noll
• Sneak Peak: Confluent Schema Registry & Protobuf by Matt Howlett
• Kafka Connect Healthchecks by Engineering Alice
• Changing the Logging Level for Kafka Connect Dynamically by Robin Moffatt
• What is Apache Kafka and what are Kafka use cases? by Maria Wachal
• Who and why uses Apache Kafka? by Mical Matloka
• How to integrate Dapr and Azure Event Hubs using Kafka bindings by Abhishek Gupta
• Introduction to Event Streaming with Kafka and Kafdrop by Emil Koutanov
• Apache Kafka and max.in.flight.requests.per.connection by Bartosz Konieczny
• Custom Partitioner in Kafka Using Scala: Take Quick Tour! by Swapnil Gosavi

Apache Flink

• Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System by Alexander Fedulov
• Announcing support for Apache Flink with the GA of Cloudera Streaming Analytics by Dinesh Chandrasekhar

New Presentations

• How to build 1000 microservices with Kafka and thrive by Natan Silnitsky
• Kafka on Kubernetes by Uma Mukkar
• Real time data processing and model inferncing platform with Kafka streams by Navinder Singh
• Tale of two streaming frameworks by Karthik Deivasigamni

New Videos

• Modern Data Architectures with Kafka and Cassandra by Cliff Gilmore
• How to Manage State in Apache Kafka by Ben Abramson
• You too can build event-driven applications with Apache Kafka made serverless by Ricardo Ferreira
• Running Apache Flink and Apache Beam on Kubernetes by Micah Wylde
• Stream processing for the masses with Beam, Python and Flink by Enrico Canzonieri
• Advanced Stream Processing on the Edge by Eduardo Silva

New Podcasts

• Making Apache Kafka Connectors for the Cloud ft. Magesh Nandakumar by Confluent Streaming Audio

New Releases

• Apache Beam 2.17.0
• Confluent Platform 5.4.0
• Strimzi Kafka Operator 0.16.0
• Debezium 1.1.0.Alpha1

 

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 10.1.2020

This is the 180th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

General

• How to Ensure Security of Your Real-Time Data Analytics Pipelines by Rostyslav Demush
• What is Stream Data Integration? by Katherine Rincon

Apache Kafka / Kafka Streams / Confluent Platform

• What’s New in Azkarra Streams v0.5 by Florian Hussonnois
• Using Strimzi with Amazon Network load balancers by Jakub Scholz
• Learnings from using Kafka Connect – Debezium – PostgreSQL by Stephane Derosiaux
• Grafana Dashboard For Monitoring Debezium MySQL Connector by Bhuvanesh
• Streaming messages from RabbitMQ into Kafka with Kafka Connect by Robin Moffatt
• Pipeline to the Cloud – Streaming On-Premises Data for Cloud Analytics by Robin Moffatt
• Apache Kafka as a Service with Confluent Cloud Now Available on GCP Marketplace by Ricardo Ferreira
• The Power Of Streaming ETL — Flatten JSON With ksqlDB by Bhuvanesh
• Exploring ksqlDB window start time by Robin Moffatt
• Create Kafka Streams applications faster than ever before with Azkarra Streams by Florian Hussonnois

New Videos

• Practical Change Data Streaming Use Cases with Apache Kafka & Debezium by Gunnar Morling

New Podcasts

• Location Data and Geofencing with Apache Kafka ft. Guido Schmutz by Confluent Streaming Audio

New Releases

• Azkarra Streams 0.5

 

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 31.12.2019

Happy New Year! This is the last blog post for 2019. Wish you all a successful 2020!

This is the 179th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

General

• Real-Time Dashboard With MongoDB by Artyom Keydunov
• Hybrid Cloud for Streaming & Real-Time Analytics by RTInsightsTeam

Apache Kafka / Kafka Streams / Confluent Platform

• Detecting and Analysing SSH Attacks with ksqlDB by Robin Moffatt
• Analysing network behaviour with ksqlDB and MongoDB by Robin Moffatt
• Realtime Data in Apache Druid — Choosing the Right Strategy by Kartik Khare
• Towards a MongoDB-backed Apache Kafka Streams State Store by Hans-Peter Grahsl
• Envoy protocol filter for Kafka, meshed by Balint Molnar & Zsolt Varga
• Do not reinvent the wheel — use Kafka Connect by Adam Kaczmarek
• Exploring ksqlDB with Twitter Data by Robin Moffatt
• Pull Queries and Connector Management Added to ksqlDB (KSQL) Event Streaming Database for Kafka by Jan Stenberg
• Apache Kafka Producer Improvements with the Sticky Partitioner by Justine Olshan
• Monitor Debezium MySQL Connector With Prometheus And Grafana  by Bhuvanesh
• How to Setup Strimzi Kafka Operator with Mutual TLS and Simple Authorization by Anoop L
• Apache Kafka is the New Black at the Edge in Industrial IoT, Logistics and Retailing by Kai Waehner
• The Easiest Way to Install Apache Kafka and Confluent Platform – Using Ansible by Justin Manchester
• Streaming Now: Debezium 1.0 Final Is Out by Gunnar Morling

Apache Pulsar

• Apache Pulsar adapter for SAP PO by Peter Ha
• Pulsar Community Weekly Update by Stream Native

Apache Flink

• Apache Flink for Continuous Insight Based on Event Streams by ODSC

Spark

• Extending state store in Structured Streaming – reprocessing and limits by Bartosz Konieczny

Apache NiFi

• Modern Apache NiFi Load Balancing by Tim Spann

New Presentations

• Building Event Driven (Micro)services with Apache Kafka by Guido Schmutz
• LEAD: A Formal Specification for Event Processing by Anas Al Bassit

New Videos

• Modern Stream Processing Engines Compared by Jacek Laskowski
• Dropbox + Ending Endless Rebalances by A. Sophie Blee-Goldman & Guozhang Wang
• Real Time Analytics with Druid, Apache, Spark, and Kafka by Daria Litvinov
• Kafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replication by Ran Tavory

New Podcasts

• Jay Kreps on Enterprise Integration Architecture with a Kafka Event Log by Software Engineering Radio
• Multi-Cloud Monitoring and Observability with the Metrics API by Confluent Streaming Audio
• Apache Kafka and Apache Druid – The Perfect Pair ft. Rachel Pedreschi by Confluent Streaming Audio

New Releases

• Kafka Connect MongoDB Version 1.4.0
• Debezium 1.0.0

 

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 18.12.2019

This is the 178th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

General

• DBLog: A Generic Change-Data-Capture Framework by Netflix Technology Blog
• Gwen Shapira on the Power of Event-Driven Architecture by Jennifer Riggins

Apache Kafka / Kafka Streams / Confluent Platform

• Apache Kafka as Digital Twin in Industrial IoT (IIoT) by Kai Wähner
• Change Data Capture (CDC) With Embedded Debezium and SpringBoot by
• Kafdrop by Emil Koutanov
• Transferring Avro Schemas Across Schema Registries with Kafka Connect by Jordan Moore
• Vertically scaling Kafka consumers by Jesse Yates
• Running Kafka on Azure Kubernetes Service by Agraj Mangal
• What’s New in Apache Kafka 2.4 by Manikumar Reddy
• Kafka Donuts – on KSQLDB – #2 by Charl Lamprecht
• Jaeger OpenTracing with Kafka, KSQL, Kafka Connect and Kafka Streams by Andy Muir
• Testing Kafka Streams Using TestInputTopic and TestOutputTopic by Jukka Karvanen
• Is Kafka the Next Big Thing in the Banking and Financial Sector? by Rebecca Sampson

Apache Flink

• Running Apache Flink on Kubernetes with KUDO by Gerred Dillon

StreamSets

• StreamSets Transformer: Natural Language Processing in PySpark by Dash Desai
• Announcing StreamSets Data Collector 3.12.0 and StreamSets Data Collector Edge 3.12.0 by Dash Desai

New Presentations

• Location Analytics – Real-Time Geofencing using Apache Kafka (updated for ksqlDB) by Guido Schmutz
• Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka by Guido Schmutz
• Removing performance bottlenecks with Kafka Monitoring and topic configuration by Neelam
• Real time analytics in Azure IoT by Sam Vanhoutte

New Videos

• Apache Kafka 2.4 – Overview of Latest Features, Updates, and KIPs by Tim Berglund
• Building stream processing applications with Apache Kafka using ksql by Robin Moffatt

New Podcasts

• Cloud-Native Patterns with Cornelia Davis by Confluent Streaming Audio
• Apache Kafka 2.4 – Overview of Latest Features, Updates, and KIPs by Confluent Streaming Audio
• Ask Confluent #16: ksqlDB Edition by Confluent Streaming Audio
• Apache Kafka and Apache Druid – The Perfect Pair ft. Rachel Pedreschi by Confluent Streaming Audio

New Releases

• StreamSets Data Collector 3.12.0
• Pulsar 2.4.2
• Apache Kafka 2.4.0
• Apache Flink 1.8.3
• Debezium 1.0.0.CR1

 

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 4.12.2019

This is the 177th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Confluent introduces ksqlDB event streaming database by Sean Michael Kerner
• Common mistakes made when configuring multiple Kafka Connect workers by Robin Moffatt
• Understanding Kafka partition assignment strategies and how to write your own custom assignor by Florian Hussonnois
• Will Users Tune Into ‘Application Playlists’ In Future? by Adrian Bridgwater
• Snowflake Launches Kafka Connector by Isaac Kunen
• Kafka Donuts – on KSQLDB – #1 by Charl Lamprecht
• Conquering Hybrid Cloud with Replicated Event-Driven Architectures by Rick Spurgeon
• Introducing Azkarra Streams : The first micro-framework for Apache Kafka Streams by Florian Hussonnois
• Apache Kafka in the Automotive Industry by Kai Waehner
• Stream Processing with Spring Cloud Stream and Apache Kafka Streams. Part 1 – Programming Model by Soby Chacko
• Conquering Hybrid Cloud with Replicated Event-Driven Architectures by Rick Spurgeon
• Providing Timely, Reliable, and Consistent Travel Information to Millions of Deutsche Bahn Passengers with Apache Kafka and Confluent Platform by Axel Löhn & Uwe Eisele
• Integrating Apache Kafka With Python Asyncio Web Applications by Matt Howlett
• Service Mesh and Cloud-Native Microservices with Apache Kafka, Kubernetes and Envoy, Istio, Linkerd by Kai Waehner
• Kafka Monthly Digest: November 2019 by Mickael Maison
• Introduction to Event Streaming with Kafka and Kafdrop by Emil Koutanov

Apache Flink

• Streaming ETL With Apache Flink — Part 2 by Preetdeep Kumar

Apache NiFi

• Exploring Apache NiFi 1.10: Parameters and Stateless Engine by Tim Spann
• Secured NiFi cluster with Terraform on the Google Cloud Platform by Pierre Villard
• Adding NiFi and Kafka to Cloudera Data Platform by Michael Kohs

New Presentations

• Kongo:Building a Scalable Streaming IoT Application using Apache Kafka by Paul Brebner

New Videos

• Building Event Streaming Applications with Serverless Kafka by Ricardo Ferreira
• Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph by Jeff Carpenter

New Podcasts

• Machine Learning with Kafka Streams, Kafka Connect, and ksqlDB by Streaming Audio
• Real-Time Payments with Clojure and Apache Kafka by Streaming Audio

 

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 21.11.2019

This is the 176th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

General

• Local-first software: you own your data, in spite of the cloud by Adrian Colyer

Apache Kafka / Kafka Streams / Confluent Platform

• Processing guarantees in Kafka by Andy Bryant
• Tales from running Kafka Streams in Production by Pedro Tavareλ
• Zeebe ❤️ Kafka by Bernd Ruecker
• Using Confluent Platform to Complete a Massive Cloud Provider Migration and Handle Half a Million Events Per Second by Oguz Kayral
• Introducing ksqlDB by Jay Kreps
• Confluent Reveals ksqlDB, a Streaming Database Built on Kafka by Alex Woodie
• Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect by Robin Moffatt
• Kafka Streams and ksqlDB Compared – How to Choose by Dani Traphagen
• Running Apache Kafka on Windows 10 by Bivás Biswas
• Kafka Gotchas by Emil Koutanov

Apache Spark

• Streaming a Kafka topic in a Delta table on S3 using Spark Structured Streaming by Kees C. Bakker

Apache Flink

• Announcing Flink Community Packages by Robert Metzger

Streamsets

• StreamSets Transformer: Design Patterns For Slowly Changing Dimensions by Dash Desai

New Presentations

• Practical Change Data Streaming Use Cases With Apache Kafka and Debezium by Gunnar Morling
• On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data by Robin Moffatt
• The Changing Face of ETL: Event-Driven Architectures for Data Engineers by Robin Moffatt
• Streaming Visualization by Guido Schmutz
• Location Analytics Real-Time Geofencing using Kafka by Guido Schmutz
• Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka by Guido Schmutz
• What is Apache Kafka? Why is it so popular? Should I use it? by Guido Schmutz
• A guide through the Azure Messaging services by Eldert Grootenboer
• Best Practices for Streaming IoT Data with MQTT and Apache Kafka by Kai Waehner & Dominik Obermaier

New Videos

• Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, and DSE Graph by Jeff Carpenter
• From Zero to Hero with Kafka Connect by Robin Moffatt
• On Track with Apache Kafka by Robin Moffatt
• Implementing Enterprise Apache Kafka platform to enable Machine Learning by Nitiin Tyagi & Peter Buckley

New Podcasts

• Securing the Cloud with VPC Peering by Confluent Streaming Audio
• Installing Apache Kafka with Ansible by Confluent Streaming Audio
• Announcing ksqlDB by Confluent Streaming Audio

New Releases

• Debezium 1.0.0.Beta3 Released

 

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 12.11.2019

This is the 175th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading!

News and Blog Posts

General

• Why event-driven is business’s best friend by Vladimir Elvov

Apache Kafka / Kafka Streams / Confluent Platform

• How to Use Single Message Transforms in Kafka Connect by Chris Matta
• Deploy a Kafka Cluster with Terraform and Ansible by Marcos Lomboglia
• Apache Kafka Rebalance Protocol, or the magic behind your streams applications by Florian Hussonnois
• An Opinionated Approach to Developing Event-Driven Microservice Applications with Kafka and Web-Sockets (Part 1 of 4) by Orhan Tuncer
• An Opinionated Approach to Developing Event-Driven Microservice Applications with Kafka and Web-Sockets (Part 2 of 4) by Orhan Tuncer
• An Opinionated Approach to Developing Event-Driven Microservice Applications with Kafka and Web-Sockets (Part 3 of 4) by Orhan Tuncer
• Running Dockerised Kafka Connect worker on GCP by Robin Moffatt
• Proper Kubernetes Health Check for a Kafka Streams Application by
• Analytics on Kafka Event Streams Using Druid, Elasticsearch, and Rockset by

Apache Pulsar

• Introduction to Apache Pulsar — Concepts, Architecture & Java Clients by Florian Hussonnois
• Apache Pulsar — One Cluster for the Entire Enterprise Using Multi Tenancy by Karthikeyan Palanivelu

Apache Flink

• Consuming Kafka Messages From Apache Flink  by Dursun Koç

Streamsets

• Ingest data into Azure Synapse Analytics (formerly SQL DW) with StreamSets Cloud by Pat Patterson

New Presentations

• FaaS composition using Kafka and CloudEvents by Neil Avery
• Kafka as an event store – is it good enough? by Guido Schmutz
• Simplified Machine Learning Architecture with an Event Streaming Platform (Apache Kafka + TensorFlow I/O) by Kai Waehner
• Flowable Business Processing from Kafka Events by Joram Barrez
• Stream processing with Apache Flink by Timo Walther
• Connect Data and Devices with Apache NiFi by Bill Farmer & Ryan Hendrickson & Elli Schwarz

New Videos

• The Changing Face of ETL: Event-Driven Architectures for Data Engineers by Robin Moffatt
• No More Silos: Integrating Databases and Apache Kafka by Robin Moffatt
• A Deep Dive into Apache Kafka This is Event Streaming, not just messaging by Kate Stanley & Andrew Dunnings
• Apache Kafka Event streaming platform for .NET developers by Viktor Gamov
• Running Kafka in Kubernetes: a practical guide by Kaherine Stanley
• Building stream processing applications with Apache Kafka by Ugo Landini
• Flowable business processing from Kafka events by Joram Barrez & Tijs Rademakers
• Streaming Machine Learning with Apache Kafka and TensorFlow by Kai Waehner
• Reactive Services in practice with Webflux, Kafka and DynamoDB by Cleber Silveira & Vinicius Soares
• How to build stream data pipeline with Apache Kafka and Spark Structured Streaming by Takanori Aoki
• Expose your Kafka with GraphQL by Gerard Klijs
• Apache Kafka: Advice from the Trenches or how to Successfully Fail by Pere Urbon Bayes
• Raising the performance bar for stream processing with Apache Storm 2.0 by Roshan Naik
• Stream Processing beyond Streaming Data by Stephan Ewen
• Event-driven business automation powered by cloud native Java by Mario Fusco & Maciej Swiderski & Edoardo Vacchi
• Apache Pulsar 101: architecture, concepts & benchmarks by Quentin Adam & Steven Le Roux
• 7 Reasons why your microservices should use Event Sourcing & CQRS by Hugh Mckee
• Taming IoT Data: Making Sense of Sensors with SQL Streaming by Hans-Peter Grahsl

New Podcasts

• LinkedIn Kafka with Nacho Solis by Software Engineering Daily

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!