Last Week in Stream Data Integration & Stream Analytics – 20.9.2019

This is the 169th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Why Event Stream Processing Is Leading the New Big Data Era by

Apache Kafka / Kafka Streams / Confluent Platform

• Apache Kafka Rebalance Protocol for the Cloud: Static Membership by Boyang Chen
• Built-In Multi-Region Replication with Confluent Platform 5.4-preview by David Arthur
• Why do you need Apache Kafka for your cloud migration by Slim Baltagi
• Reflections on Event Streaming as Confluent Turns Five – Part 1 by Tim Berglund
• Reflections on Event Streaming as Confluent Turns Five – Part 2 by Tim Berglund
• The Rise of Managed Services for Apache Kafka by Ricardo Ferreira
• The economic viability of the open-source model for Apache Kafka and Confluent by Stéphane Maarek

Apache Flink

• The State Processor API: How to Read, write and modify the state of Flink applications by Seth Wiesman & Fabian Hueske
• Apache Flink on YARN with Kerberos Authentication by Ana Esguerra

StreamSets

• With new solutions, StreamSets aims to simplify data integration and Apache Spark by Maria Deutscher
• StreamSets Transformer Extensibility: Spark and Machine Learning by Dash Desai

New Presentations

• Building stream processing applications for Apache Kafka using KSQL by Robin Moffatt
• Keeping your Caches Hot with Apache Kafka and the Connector API by Ricardo Ferreira
• Streaming Machine Learning with Python, Jupyter, TensorFlow, Apache Kafka and KSQL by Kai Waehner
• Stream processing for the masses with beam, python and flink by Enrico Canzonieri
• Interactive Querying of Streams Using Apache Pulsar by Jerry Peng

New Videos

• Unleashing Apache Kafka and TensorFlow in Public Cloud and Hybrid Architectures by Kai Waehner
• Microservices, Events, and Breaking the Data Monolith with Kafka by Cornelia Davis & Ben Stopford
• Bringing Observability to Your Stream Processing by Viktor Gamov
• Temporal Databases for Streaming Architectures by Jeremy Taylor & Jon Pither

New Podcasts

• Jay Kreps on the Last 10 Years of Apache Kafka and Event Streaming by Confluent Streaming Audio
• Should You Run Apache Kafka on Kubernetes? ft. Balthazar Rouberol by Confluent Streaming Audio
• KIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason Gustafson by Confluent Streaming Audio
• Kafka Applications with Tim Berglund by SE Daily

New Releases

• Confluent Platform 5.3.1

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Advertisements

Last Week in Stream Data Integration & Stream Analytics – 12.9.2019

This is the 168th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Data Ingestion: Just Because You Can, Doesn’t Mean You Should by Conor Reynolds

Apache Kafka / Kafka Streams / Confluent Platform

• Apache Kafka, KSQL and Apache PLC4X for IIoT Data Integration and Processing by Kai Waehner
• Implementing the Outbox Pattern with CDC using Debezium by Thorben Janssen
• Hacking Strimzi for Cruise Control by Kyle Liberti
• How to Use Schema Registry and Avro in Spring Boot Applications by Viktor Gamov
• Introducing Derivative Event Sourcing by Anna McDonald
• Role-Based Access Control (RBAC) for Kafka Connect by Yeva Byzek
• SQL Server Change Stream by Davide Mauri
• Inca — Message Tracing and Loss Detection For Streaming Data @Netflix by Allen Wang
• CDC pipeline with Red Hat AMQ Streams and Red Hat Fuse by Sadhana Nandakumar
• Using the Zeebe Kafka Connector to Orchestrate, Integrate, and More by Bernd Rücker
• Crash course Apache Kafka using the Confluent Cloud by Tim van Baarsen
• Strimzi Apache Kafka Operator joins the CNCF by Strimzi Developer
• Six Ways Your Kafka Design will Fail by Miles Garnsey
• Kafka Operator for Kubernetes to establish a Cloud-Native Apache Kafka Platform by Kai Waehner
• Kafka Streams, the Clojure way by Dave Martin
• 10 Apache Kafka best practices for data management pros by George Lawton

Apache Flink

• Flink Community Update – September’19 by Marta Paes

Apache Pulsar

• A Look at Multi-Topic Subscriptions with Apache Pulsar by Jack Vanlightly

StreamSets

• StreamSets Announces Beta of StreamSets Cloud by A.R. Guess
• StreamSets Transformer is Here by Sean Anderson

New Presentations

• On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data by Robin Moffatt
• Improving the Observability of Cassandra, Kafka and Kubernetes applications with Prometheus and OpenTracing by Paul Brebner

New Videos

• Ask Confluent #15: Attack of the Zombie Controller by Gwen Shapira
• Unifying streaming and message queue with Apache Kafka by Mingmin Chen

New Podcasts

• Ask Confluent #15: Attack of the Zombie Controller by Streaming Audio
• Streaming Audio: Connecting to Apache Kafka with Neo4j by Streaming Audio

New Releases

• Apache Pulsar 2.4.1
• Debezium 0.10.0.CR1
• Kafkacat 1.5.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 31.8.2019

This is the 167th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

This time it is actually for two weeks, as I was too busy to release one earlier. From next week I hope to get back to a weekly cycle. As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Why event stream processing is leading the new ‘big data’ era by InformationAge
• Druid and SuperSet for real-time monitoring at scale by Brachi Packter
• Performance Comparison Between Apache Pulsar and Kafka: Latency by Chris Bartholomew
• How Real-Time Data is Automating the Temperature-Controlled Supply Chain by Gisli Herjolfsson

Apache Kafka / Kafka Streams / Confluent Platform

• Understanding Spark Streaming with Kafka and Druid by Daria Litvinov
• Top 10 Reasons to Attend Kafka Summit by Tim Berglund
• Kafka Streams: Topology and Optimizations by Stéphane Derosiaux
• Shoulder Surfers Beware: Confluent Now Provides Cross-Platform Secret Protection by Yeva Byzek
• InfluxDB and Kafka: How InfluxData Uses Kafka in Production by Anais Dotis-Georgiou
• Kafka Connect Improvements in Apache Kafka 2.3 by Robin Moffatt
• A Guide to the Confluent Verified Integrations Program by Jeff Bean
• Outsized Impact: How Confluent’s Engineering Team is Building the Infrastructure for Real-Time Event Streaming by Job Portraits
• Introduction to Topic Log Compaction in Apache Kafka by Nihit Saxena
• Building Transactional Systems Using Apache Kafka by Michael Seifert
• Apache Kafka and Machine Learning for Real Time Supply Chain Optimization in IIoT by Kai Waehner
• Using Apache Kafka to implement event-driven microservices  by Víctor Gil
• Kafka Basics: Tables vs Streams by Edward Loveall
• MySQL CDC with Apache Kafka and Debezium by Kushal Yellam
• Confluent Cloud Schema Registry is Now Generally Available by Nathan Nam
• Does Kafka really guarantee the order of messages? by Kamil Charłampowicz
• Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) — Friends, Enemies or Frenemies? by Kai Waehner
• Using Graph Processing for Kafka Stream Visualizations by David Allen
• Kafka | IoT Ecosystem ::Cluster; Performance Metrics; Sensorboards & OBD-II by Victor Martins

Spark Streaming

• Understanding Spark Streaming with Kafka and Druid by Daria Litvinov

Apache Pulsar

• One simple chart: Who is interested in Apache Pulsar? by Ben Lorica

StreamSets

• Automating Kerberos KeyTab Generation for Kubernetes-based Deployments by Mark Brooks

New Presentations

• IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale by Kai Waehner & Graham Ganssie
• How to Build an Apache Kafka® Connector by Jeff Bean & Todd McGrath & Lisa Sensmeier
• Stream Processing in 2019 by Chris Horder
• Kick your database to the curb by Bill Bejeck
• Thoughts on kafka capacity planning by Jamie Alquiza
• Adding Avro to your Kafka streams to meet your messaging needs by Erik Tank

New Videos

• IIoT with Kafka and Machine Learning for Supply Chain Optimization In Real Time at Scale by Kai Waehner & Graham Ganssie

New Podcasts

• Helping Healthcare with Apache Kafka and KSQL ft. Ramesh Sringeri by Confluent Streaming Audio
• Contributing to Open Source with the Kafka Connect MongoDB Sink ft. Hans-Peter Grahsl by Confluent Streaming Audio
• Teaching Apache Kafka Online with Stéphane Maarek by Confluent Streaming Audio

New Releases

• Debezium 0.10.0.Beta4
• Apache Flink 1.9.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 12.8.2019

This is the 166th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

This time it is actually for two weeks, as I was too busy to release one earlier. From next week I hope to get back to a weekly cycle. As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• The Future of Data Engineering by Chris Riccomini
• Blueshift: Scaling real-time campaign analytics with Apache Druid  by Anuraj Pandey
• What can you do with Event Processing? by Mariah Tauer

Apache Kafka / Kafka Streams / Confluent Platform

• How Kafka Enables Real-Time Stream Processing, Part 1 by André Melo
• How Kafka Enables Real-Time Stream Processing, Part 2 by André Melo
• Kafka Operator for Kubernetes – Confluent Operator to establish a Cloud-Native Apache Kafka Platform by Kai Waehner
• Introducing Confluent Platform 5.3 by Gaetan Castelein
• De-de-de-de-duplicating events with Kafka Streams by Jaroslaw Kijanowski
• Kafka Connect – Source Connectors: A detailed guide to connecting to what you love by Rufus Nash
• Building Shared State Microservices for Distributed Systems Using Kafka Streams by Nitzan Gilkis
• KSQL UDFs and UDAFs Made Easy by Mitch Seymour
• How to use Kafka Streams to process events from one topic to another? by Łukasz Chrząszcz
• Create an unbundled database with Crux and Confluent Cloud Kafka in 5 minutes by Jeremy Taylor
• Announcing Tutorials for Apache Kafka by Michael Drogalis
• How can I learn Confluent Kafka? by Stéphane Maarek

Apache Pulsar

• Build a Priority-based Push Notification System Using Apache Pulsar at GeTui by StreamNative

StreamSets

• Announcing StreamSets Data Collector 3.10.0 and StreamSets Data Collector Edge 3.10.0 by Dash Desai
• How DataOps is Adding Value to Data Lakes by Sean Anderson
• A New Definition of DataOps by John Schmidt

New Presentations

• Confluent Operator as Cloud-Native Kafka Operator for Kubernetes by Kai Waehner
• Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Were Afraid to Ask by Matthias J. Sax
• Monitor Kafka Like a Pro by Gwen Shapria & Xavier Léauté
• Getting Superman X-Ray Vision: Bringing Observability to your Stream Processing by Ricardo Ferreira

New Videos

• Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Were Afraid to Ask by Matthias J. Sax

New Podcasts

• Transparent GDPR Encryption with David Jacot by Confluent Streaming Audio
• In Control of Kafka with Dan Norwood by Confluent Streaming Audio
• How to Convert Python Batch Jobs into Kafka Streams Applications with Rishi Dhanaraj by Confluent Streaming Audio
• What’s New in This Release by Confluent Streaming Audio
• Connecting Apache Cassandra to Apache Kafka with Jeff Carpenter from DataStax by Confluent Streaming Audio

New Releases

• Apache Beam 2.14.0
• StreamSets Data Collector 3.10.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 26.7.2019

This is the 165th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

This time it is actually for two weeks, as I was too busy to release one earlier. From next week I hope to get back to a weekly cycle. As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Five Advantages of Log-Based Change Data Capture by Gunnar Morling
• Communicating Data Changes Across Service Boundaries… Safely! by Hans-Peter Grahsl

Apache Kafka / Kafka Streams / Confluent Platform

• Autoscaling Kafka Streams applications with Kubernetes by Jaroslaw Kijanowski
• Streaming Cassandra at WePay – Part 1 by Joy Gao
• Streaming Cassandra at WePay – Part 2 by Joy Gao
• Bust the Burglars – Machine Learning with TensorFlow and Apache Kafka by Erik-Berndt Scheper
• Open Sourcing Brooklin: Near Real-Time Data Streaming at Scale by Celia Kung
• Getting Started with the MongoDB Connector for Apache Kafka and MongoDB by Robert Walters
• Kafka Streams, Jigsaw and Docker walk into a bar by Jaroslaw Kijanowski
• Bridging the HTTP protocol to Apache Kafka by Paolo Patierno
• Why I Can’t Wait for Kafka Summit San Francisco by Gwen Shapira
• Fault Tolerance in Distributed Systems: Tracing with Apache Kafka and Jaeger by Aaron Burk
• An introduction to event-driven architecture and Apache Kafka by Neil McAllister

Apache Pulsar

• Sentiment Analysis of Tweets using Apache Pulsar by Jerry Peng
• Apache Pulsar as One Storage System for Both Real-time and Historical Data Analysis by Yijie Shen

Apache Flink

• Apache Pulsar Flink Connector by Eren Avşaroğulları
• Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing by Nico Kruber & Piotr Nowojski

StreamSets

• Enhanced Error Diagnostics in StreamSets Data Collector 3.9.0 by Pat Patterson
• A New Definition of DataOps by John Schmidt

New Presentations

• From Zero to Hero with Kafka Connect by Robin Moffatt
• Top Ten Kafka® Configs by Bill Bejeck
• Real-time processing of large amounts of data by Perry Krol

New Videos

• From Zero to Hero with Kafka Connect by Robin Moffatt
• End-to-End Integration: IoT Edge to Confluent Cloud by Kai Wähner
• Ask Confluent #14: In Control of Kafka by Dan Norwood
• Kubernetes as a Streaming Data Platform with Kafka, Spark, and Scala by Gerard Maas
• View from Apache Flink on Evolution & Outlooks for the Modern Stateful Stream Processor by Tzu-Li Tai

New Releases

• Apache Pulsar 2.4.0
• Debezium 0.10.0.Beta3

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 10.7.2019

This is the 164th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Server-Sent Events (SSE): A Conceptual Deep Dive by
• Event-Driven Architectures: A Simple Analogy by Kevin Larsen
• AoA and AoD: Real-Time Location Services Drive IoT Value by Fabio Belloni

Apache Kafka / Kafka Streams / Confluent Platform

• Lessons learned from running Kafka at Datadog by Emily Chang
• KSQL and Registered topics by Jaroslaw Kijanowski
• KSQL in Football: FIFA Women’s World Cup Data Analysis by Francesco Tisiot
• Kafka Monthly Digest: June 2019 by Mickael Maison
• Analysis of streaming data with Kafka by Rob Cowell
• Persistent storage improvements by Jakub Scholz
• Deploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams by Stewart Bryson

Apache NiFi

• NiFi & NiFi Registry on the Google Cloud Platform with Cloud Source Repositories by Pierre Villard

Apache Flink

• On Stream Processing and Apache Flink. Q&A with Fabian Hueske by Fabian Hueske

New Videos

• Real Time Investment Alerts using Apache Kafka at ING Bank by Marcos Maia & Tim v Baarsen
• Deep Learning For Real Time Streaming Data With Kafka And Tensorflow by Yong Tang
• How to use Apache Kafka to transform a batch pipeline into a Real-Time-One? by Stephane Maarek
• The internals of stateful stream processing in Spark Structured Streaming by Jacek Laskowski

New Podcasts

• Change Data Capture with Debezium ft. Gunnar Morling by Confluent Streaming Audio

New Releases

• Apache Pulsar 2.4.0

Upcoming Events

• 9.7.2019 (Moutain View, US) – Apache Druid Bay Area Meetup @ Intuit (Meetup)
• 11.7.2019 (Utrecht, NL) – From Zero to Hero with Kafka Connect (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 2.7.2019

This is the 163st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Design Considerations for Real-Time Operational Data Architectures for Industry 4.0 by Mahendraprabu Sundarraj

Apache Kafka / Kafka Streams / Confluent Platform

• A real showcase of Kafka at Wirecard Brazil by Cauê Ferreira
• An overview of the Kafka Connect RabbitMQ Source Connector by Danny Kay
• Building a Scalable Search Architecture by Pere Urbón-Bayes
• What’s New in Apache Kafka 2.3 by Colin McCabe
• High Availability Kafka Service in Microsoft Azure Cloud by Ada Yan
• Microservices, Apache Kafka, and Domain-Driven Design by Kai Waehner
• Building Kafka Streams applications with Quarkus and Eclipse MicroProfile by Gunnar Morling
• Designing the .NET API for Apache Kafka by Matt Howlett
• Building zero-latency data lake using Change Data Capture by Ofir Ventura
• Optimizing Your Apache Kafka Deployment by Yeva Byzek
• Kafka Listeners – Explained by Robin Moffatt
• Change data capture from Oracle to Kafka with triggers by Janne Keskitalo
• Kafka Connectors Without Kafka by Dawid Kublik

Apache Flink

• A Practical Guide to Broadcast State in Apache Flink by Fabian Hueske

New Slides

• Data Integration with Apache Kafka: What, Why, How by Pat Patterson
• Streaming database events over Kafka using Debezium by Clement Demonchy & Job Teaser
• The Changing Face of ETL: Event-Driven Architectures for Data Engineers by Robin Moffatt
• Testing Kafka based applications by Nikita Zhevnitskiy
• Integrating Databases and Apache Kafka by Robin Moffatt
• Deep Dive on Apache Flink State by Seth Wiesman
• Streaming ETL in Practice with Oracle, Apache Kafka, and KSQL by Robin Moffatt

New Videos

• What’s new in Kafka 2.3 by Stephane Maarek
• Apache Kafka 2.3 | What’s New in this Release + Updates and KIPs by Tim Berglund
• From zero to hero with Apache Kafka’s connect API by Robin Moffatt

New Podcasts

• Rolling Kafka Upgrades and Confluent Cloud by Gwen Shapira

New Releases

• Apache Kafka 2.3.0
• StreamSets Data Collector 3.9.1

Upcoming Events

• 1.7.2019 (Paris, FR) – Apache Beam meetup 1 at Criteo (Meetup)
• 2.7.2019 (Vienna, AU) – Apache Kafka® & New Ways of Stream Processing with KSQL (Meetup)
• 3.7.2019 (Munich, GE) – From Zero to Hero with Kafka® Connect (Meetup)
• 4.7.2019 (Istanbul, TU) – Strimzi: Distributed Streaming with Apache Kafka in Kubernetes (Meetup)
• 9.7.2019 (Moutain View, US) – Apache Druid Bay Area Meetup @ Intuit (Meetup)
• 11.7.2019 (Utrecht, NL) – From Zero to Hero with Kafka Connect (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!