Last Week in Stream Data Integration & Stream Analytics – 10.7.2019

This is the 164th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Server-Sent Events (SSE): A Conceptual Deep Dive by
• Event-Driven Architectures: A Simple Analogy by Kevin Larsen
• AoA and AoD: Real-Time Location Services Drive IoT Value by Fabio Belloni

Apache Kafka / Kafka Streams / Confluent Platform

• Lessons learned from running Kafka at Datadog by Emily Chang
• KSQL and Registered topics by Jaroslaw Kijanowski
• KSQL in Football: FIFA Women’s World Cup Data Analysis by Francesco Tisiot
• Kafka Monthly Digest: June 2019 by Mickael Maison
• Analysis of streaming data with Kafka by Rob Cowell
• Persistent storage improvements by Jakub Scholz
• Deploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams by Stewart Bryson

Apache NiFi

• NiFi & NiFi Registry on the Google Cloud Platform with Cloud Source Repositories by Pierre Villard

Apache Flink

• On Stream Processing and Apache Flink. Q&A with Fabian Hueske by Fabian Hueske

New Videos

• Real Time Investment Alerts using Apache Kafka at ING Bank by Marcos Maia & Tim v Baarsen
• Deep Learning For Real Time Streaming Data With Kafka And Tensorflow by Yong Tang
• How to use Apache Kafka to transform a batch pipeline into a Real-Time-One? by Stephane Maarek
• The internals of stateful stream processing in Spark Structured Streaming by Jacek Laskowski

New Podcasts

• Change Data Capture with Debezium ft. Gunnar Morling by Confluent Streaming Audio

New Releases

• Apache Pulsar 2.4.0

Upcoming Events

• 9.7.2019 (Moutain View, US) – Apache Druid Bay Area Meetup @ Intuit (Meetup)
• 11.7.2019 (Utrecht, NL) – From Zero to Hero with Kafka Connect (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Advertisements

Last Week in Stream Data Integration & Stream Analytics – 2.7.2019

This is the 163st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Design Considerations for Real-Time Operational Data Architectures for Industry 4.0 by Mahendraprabu Sundarraj

Apache Kafka / Kafka Streams / Confluent Platform

• A real showcase of Kafka at Wirecard Brazil by Cauê Ferreira
• An overview of the Kafka Connect RabbitMQ Source Connector by Danny Kay
• Building a Scalable Search Architecture by Pere Urbón-Bayes
• What’s New in Apache Kafka 2.3 by Colin McCabe
• High Availability Kafka Service in Microsoft Azure Cloud by Ada Yan
• Microservices, Apache Kafka, and Domain-Driven Design by Kai Waehner
• Building Kafka Streams applications with Quarkus and Eclipse MicroProfile by Gunnar Morling
• Designing the .NET API for Apache Kafka by Matt Howlett
• Building zero-latency data lake using Change Data Capture by Ofir Ventura
• Optimizing Your Apache Kafka Deployment by Yeva Byzek
• Kafka Listeners – Explained by Robin Moffatt
• Change data capture from Oracle to Kafka with triggers by Janne Keskitalo
• Kafka Connectors Without Kafka by Dawid Kublik

Apache Flink

• A Practical Guide to Broadcast State in Apache Flink by Fabian Hueske

New Slides

• Data Integration with Apache Kafka: What, Why, How by Pat Patterson
• Streaming database events over Kafka using Debezium by Clement Demonchy & Job Teaser
• The Changing Face of ETL: Event-Driven Architectures for Data Engineers by Robin Moffatt
• Testing Kafka based applications by Nikita Zhevnitskiy
• Integrating Databases and Apache Kafka by Robin Moffatt
• Deep Dive on Apache Flink State by Seth Wiesman
• Streaming ETL in Practice with Oracle, Apache Kafka, and KSQL by Robin Moffatt

New Videos

• What’s new in Kafka 2.3 by Stephane Maarek
• Apache Kafka 2.3 | What’s New in this Release + Updates and KIPs by Tim Berglund
• From zero to hero with Apache Kafka’s connect API by Robin Moffatt

New Podcasts

• Rolling Kafka Upgrades and Confluent Cloud by Gwen Shapira

New Releases

• Apache Kafka 2.3.0
• StreamSets Data Collector 3.9.1

Upcoming Events

• 1.7.2019 (Paris, FR) – Apache Beam meetup 1 at Criteo (Meetup)
• 2.7.2019 (Vienna, AU) – Apache Kafka® & New Ways of Stream Processing with KSQL (Meetup)
• 3.7.2019 (Munich, GE) – From Zero to Hero with Kafka® Connect (Meetup)
• 4.7.2019 (Istanbul, TU) – Strimzi: Distributed Streaming with Apache Kafka in Kubernetes (Meetup)
• 9.7.2019 (Moutain View, US) – Apache Druid Bay Area Meetup @ Intuit (Meetup)
• 11.7.2019 (Utrecht, NL) – From Zero to Hero with Kafka Connect (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 20.6.2019

This is the 162nd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Clickstream Analysis – An Open Source Architecture by Mike McLaughlin & Peter Marshall

Apache Kafka / Kafka Streams / Confluent Platform

• Streaming Data from the Universe with Apache Kafka by Maria Patterson
• How to Connect KSQL to Confluent Cloud using Kubernetes with Helm by Mark Plascencia
• Introducing Kafka Backup by Anatoly Zelenin

StreamSets

• From Zero to Production ETL in 30 minutes with StreamSets by Pat Patterson

New Slides

• Keeping your Data Close and your Caches Hotter by Viktor Gamov
• From Zero to Hero with Kafka Connect by Robin Moffatt
• Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt
• Implementing Distributed Tracing ‘Like a Boss’ in your Apache Kafka Deployments by Ricardo Ferreira
• Location Analytics – Real Time Geofencing using Apache Kafka by Guido Schmutz
• Connecting Apache Kafka to Cash by Lyndon Hedderly
• Kafka Operations Deep Dive by Gwen Shapira
• Architecting Microservices Applications with Instant Analytics by Tim Berglund & Rachel Pedreschi
• DataOps on Streaming Data: From Kafka to InfluxDB via Kubernetes Native Flows by Christos Kotsis

New Videos

• Ditching the Overhead: Moving Apache Kafka Workloads into Amazon MSK by Damian Wylie

New Podcasts

• Why Kafka Connect? by Streaming Audio
• Confluent, Event-First Thinking and Streaming Real-Time Analytics by Drill To Detail

New Releases

• Debezium 0.10.0.Beta1 Released

Upcoming Events

• 19.6.2019 (online) – Architecting Microservices Applications with Instant Analytics (Confluent Webinar)
• 20.6.2019 (Bern, CH) – Kafka & it’s friends (Meetup)
• 20.6.2019 (online) – Building a Cloud Data Warehouse on Microsoft Azure with StreamSets (StreamSets Webinar)
• 20.6.2019 (Boston, US) – Confluent Confab
• 25.6.2019 (Brabrand, DK) – Cloud Native Streaming with Apache Flink and a Fairytale of Databases (Meetup)
• 1.7.2019 (Paris, FR) – Apache Beam meetup 1 at Criteo (Meetup)
• 3.7.2019 (Munich, GE) – From Zero to Hero with Kafka® Connect (Meetup)
• 4.7.2019 (Istanbul, TU) – Strimzi: Distributed Streaming with Apache Kafka in Kubernetes (Meetup)
• 11.7.2019 (Utrecht, NL) – From Zero to Hero with Kafka Connect (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 12.6.2019

This is the 161st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Building streaming pipelines for production - Part 2 by Andrew Stevenson

Apache Kafka / Kafka Streams / Confluent Platform

• What happened in the Kafka community in May 2019? by Mickael Maison
• Kafka is Coming – Previewing The Official MongoDB Connector for Apache Kafka by Scott L’Hommedieu
• Debezium’s Newsletter 01/2019 by Chris Cranford
• Reliable, Fast Access to On-Chain Data Insights by Jendrik Poloczek
• Spring for Apache Kafka Deep Dive – Part 4: Continuous Delivery of Event Streaming Pipelines by Ilayaperumal Gopinathan

Apache Flink

• A Deep-Dive into Flink’s Network Stack by Nico Kruber

StreamSets

• Announcing StreamSets Data Collector 3.9.0 and StreamSets Data Collector Edge 3.9.0 by Dash Desai

New Slides

• Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes by Kamela Dasika & Michael Ng
• Better Together: Apache Cassandra and Apache Kafka by Jeff Carpenter & Cedrick Lunven
• Best Practices for Streaming IoT Data to Apache Kafka® by Domnik Obermaier
• Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka by Tim van Baarsen
• Event Sourcing, Stream Processing and Serverless by Ben Stopford
• Event-Driven Messaging and Actions using Apache Flink and Apache NiFi by Dave Torok

New Videos

• Real Time Investment Alerts @ ING using Apache Kafka & Spring Kafka by Tim van Baarsen
• Better Together: Apache Cassandra and Apache Kafka by Jeff Carpenter & Cedrick Lunven
• Real-time Analytics at PayPal by Shobana Neelakantan & Sanjeev Koranga

New Releases

• Kafka HQ 0.8.0
• StreamSets Data Collector 3.9.0
• Pulsar Express 0.4.1

Upcoming Events

• 12.6.2019 (New York, US) – Druid NYC Meetup @ Tumblr (Meetup)
• 12.6.2019 (Stockholm, SE) – Streaming at King, Migrating batch-ETLs to streaming Flink (Meetup)
• 13.6.2019 (London, UK) – Intelligent Engineering – Making sense of BIG data (Meetup)
• 18.6.2019 (Olso, NO) – From Zero to Hero with Kafka® Connect (Meetup)
• 19.6.2019 (online) – Architecting Microservices Applications with Instant Analytics (Confluent Webinar)
• 20.6.2019 (Bern, CH) – Kafka & it’s friends (Meetup)
• 20.6.2019 (online) – Building a Cloud Data Warehouse on Microsoft Azure with StreamSets (StreamSets Webinar)
• 20.6.2019 (Boston, US) – Confluent Confab
• 25.6.2019 (Brabrand, DK) – Cloud Native Streaming with Apache Flink and a Fairytale of Databases (Meetup)
• 1.7.2019 (Paris, FR) – Apache Beam meetup 1 at Criteo (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 4.6.2019

This is the 160th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Event-sourcing at Nordstrom: Part 2 by Rob Gruhl
• Streaming in the Cloud With Camel and Strimzi by Josh Reagan
• One SQL to Rule Them All:An Efficient and Syntactically Idiomatic Approach toManagement of Streams and Tables
• One SQL to Rule Them All:An Efficient and Syntactically Idiomatic Approach toManagement of Streams and Tables by Edmon Begoli & Tyler Akidau & Fabian Hueske & Julian Hyde & Kathryn Knight & Kenneth Knowles

Apache Kafka / Kafka Streams / Confluent Platform

• Spring for Apache Kafka Deep Dive – Part 3: Apache Kafka and Spring Cloud Data Flow by Ilayaperumal Gopinathan
• Create a CDC Event Stream From Oracle Database to Kafka With GoldenGate by
• Lenses on Microsoft Azure HDInsight by Andrew Stevenson
• Amazon Managed Streaming for Apache Kafka (MSK) – Now Generally Available by Danilo Poccia
• Anomalia Machina 10 – Final Results: Massively Scalable Anomaly Detection with Apache Kafka and Cassandra by Paul Brebner
• Twint: Loading tweets into Kafka and Neo4j by Mark Needham

Apache Flink

• Stream processing with High Cardinality and Large State at Klaviyo by Ning Shi & Seed Zeng

Apache Storm

• Apache Storm 2.0 Improvements by Kishor Patil & Bobby Evans

StreamSets

• Low-latency Inference Using Databricks ML in StreamSets by Dash Desai
• Preview and Snapshot Features in StreamSets Data Collector by Dash Desai

New Slides

• Apache Kafka: advice from the trenches or how to successfully fail! by Pere Urbon-Bayes
• Ditching the overhead – Moving Apache Kafka workloads into Amazon MSKDitching the overhead – Moving Apache Kafka workloads into Amazon MSK by Damian Wylie
• Seamless Guest Experience with Kafka Streams by Himani Arora & Suresh Mulukaladu & Ram Indukuri
• Scalable serverless architectures using event-driven design by Eric Johnson
• Using Apache Pulsar to Provide Real-Time IoT Analytics on the Edge by David Kjerrumgaard
• Everything You Wanted to Know About Apache Kafka But You Were Too Afraid to Ask! by Riccardo Ferreira

New Videos

• Event-driven Graph Analytics using Neo4j and Apache Kafka by Neo4J
• Swim AI – Moving from near real time to real time stream processing by Greg Holland
• 99 Ways to Enrich Streaming Data with Apache Flink by Konstantin Knauf

New Postcasts

• Schema Registry Made Simple by Confluent Cloud by Streaming Audio

New Releases

• Debezium 0.10.0.Alpha2
• Storm 2.0.0
• Apache Kafka 2.2.1
• Apache Pulsar 2.3.2

Upcoming Events

• 5.6.2019 (Barcelona, ES) – 10 ways to deploy Apache Kafka® and have fun along the way (Meetup)
• 12.6.2019 (New York, US) – Druid NYC Meetup @ Tumblr (Meetup)
• 12.6.2019 (Stockholm, SE) – Streaming at King, Migrating batch-ETLs to streaming Flink (Meetup)
• 13.6.2019 (London, UK) – Intelligent Engineering – Making sense of BIG data (Meetup)
• 20.6.2019 (Bern, CH) – Kafka & it’s friends (Meetup)
• 20.6.2019 (online) – Building a Cloud Data Warehouse on Microsoft Azure with StreamSets (StreamSets Webinar)
• 20.6.2019 (Boston, US) – Confluent Confab
• 25.6.2019 (Brabrand, DK) – Cloud Native Streaming with Apache Flink and a Fairytale of Databases (Meetup)
• 1.7.2019 (Paris, FR) – Apache Beam meetup 1 at Criteo (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 29.5.2019

This is the 159th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Develop IoT Apps with Confluent Kafka, KSQL, Spring Boot & Distributed SQL by Sid Choudhury
• Kafka Summit London 2019 Session Videos by Tim Berglund
• Scylla and Confluent Integration for IoT Deployments by Maheedhar Gunturu
• What’s the proper Kubernetes health check for a Kafka Streams application?What’s the proper Kubernetes health check for a Kafka Streams application? by Jaroslaw Kijanowski
• Accessing Kafka: Part 5 – Ingress by Jakub Scholz
• Using Unravel for End-to-End Monitoring of Kafka by George Demarest
• 17 Ways to Mess Up Self-Managed Schema Registry by Yeva Byzek
• Did I beat Ben? Race tracking with Kafka KSQL, MQTT & Kibana by Simon Aubury
• Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations by Stewart Bryson
• The SHAPE architecture: scaling the core using Apache Kafka by Richard’s Technotes

Apache Flink

• State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink by Fabian Hueske & Andrey Zagrebin
• An Introduction to Temporal Data Handling With Apache Flink by Marta Paes

Apache Pulsar

• Tutorial for Using Debezium Connectors With Apache Pulsar by Jia Zhai

StreamSets

• Ingesting Data from Apache Kafka to TimescaleDB by Pat Patterson

New Slides

• Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamSets Open Source Community by Pat Patterson
• Slides of Kafka Summit London 2019 by Confluent
• Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications by Sean Glover
• Neo4j Graph Streaming Services with Apache Kafka by Michael Hunger & Andrea Santurbano
• Building Event-Driven (Micro) Services with Apache Kafka by Guido Schmutz
• Confluent Kafka and KSQL: Streaming Data Pipelines Made Easy by Kairo Tavares
• Introduction to KSQL: Streaming SQL for Apache Kafka® by Tom Green
• Using Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL and KSQL by Neil Buesing
• Introduction to Stream Processing by Guido Schmutz
• Stream Processing in 2019 by Chris Horder
• Flink SQL in Action by Fabian Hueske
• Streaming Data from Cassandra into Kafka by Andrew Prudhomme & Abrar Sheikh
• Introduction to Apache NiFi by Timothy Spann
• Tracking crime as it occurs with Apache Phoenix, Apache HBase and Apache NiFi by Timothy Spann

New Videos

• Videos of Kafka Summit London 2019 by Confluent
• Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications by Sean Glover
• Kicking Your Database to the Curb and Fun with Kafka! by Bill Bejeck
• Architecting Streaming Analytics Apps for Rapid Business Value by Sid Choudhury
• Real-time Fraud Detection for Southeast Asia’s Leading Mobile Platform by Aravind Srinivasan
• FaaS is Not Only the Serverless: Stream Processing with Serverless by Jun Makishi & Kensaku Komatsu

New Podcasts

• Why is Stream Processing Hard? ft. Michael Drogalis by Confluent Streaming Audio

Upcoming Events

• 30.5.2019 (Seattle, US) – Running Flink applications in Kinesis Analytics (Meetup)
• 5.6.2019 (Barcelona, ES) – 10 ways to deploy Apache Kafka® and have fun along the way (Meetup)
• 12.6.2019 (New York, US) – Druid NYC Meetup @ Tumblr (Meetup)
• 12.6.2019 (Stockholm, SE) – Streaming at King, Migrating batch-ETLs to streaming Flink (Meetup)
• 13.6.2019 (London, UK) – Intelligent Engineering – Making sense of BIG data (Meetup)
• 20.6.2019 (Bern, CH) – Kafka & it’s friends (Meetup)
• 20.6.2019 (online) – Building a Cloud Data Warehouse on Microsoft Azure with StreamSets (StreamSets Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Data Integration & Stream Analytics – 21.5.2019

This is the 158th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Capturing Real-Time Data from the “Point of Work” by Joe McKendrick
• NHL Skates Toward Real-time IoT by Michael Vizard

Comparison

• 7 Reasons We Choose Apache Pulsar Over Apache Kafka by Chris Bartholomew

Apache Kafka / Kafka Streams / Confluent Platform

• Streaming Machine learning pipeline for Sentiment Analysis using Apache APIs: Kafka, Spark and Drill – Part 1 by Carol McDonald
• Streaming Machine learning pipeline for Sentiment Analysis using Apache APIs: Kafka, Spark and Drill Part 2 by Carol McDonald
• On Kafka, Clouds and The Phoenix Project by Gwen Shapira
• Schemas, Contracts, and Compatibility by Gwen Shapira

Apache Spark

• Structured Streaming with PySpark by Todd Birchard

New Slides

• Deep Dive Into / Internals of Kafka Streams by Jacek Laskowski
• Local-first software: Returning data ownership to users by Martin Kleppmann
• Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt
• The Awakening of the New Event-Driven (Beast) by Viktor Gamov
• The Foundations of Multi-DC Kafka by Jakub Korab
• Everything You Always Wanted to Know About Kafka’s Rebalance Protocol but Were Afraid to Ask by Matthias J. Sax
• KSQL in Practice by Almog Gavra
• Presentations from Kafka Summit London 2019 by Confluent
• Strimzi – Where Apache Kafka meets OpenShift by José Román Martín Gil
• Stream processing and managing real-time data by  Javier Ramirez & Prakash Sethuraman
• Apache Kafka and KSQL: Stream Processing for Everyone! by Patrick Druley

New Videos

• Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt

New Podcasts

• Chris Riccomini on the History of Apache Kafka and Stream Processing by Streaming Audio
• Ask Confluent: Machine Learning with Kai Waehner by Streaming Audio
• Testing Kafka Streams Applications with Viktor Gamov by Streaming Audio

New Releases

• Conduktor

Upcoming Events

• 21.5.2019 (online) – GCP for Apache Kafka® Users: Stream Ingestion and Processing (Confluent Webinar)
• 23.5.2019 (Karlsruhe, GE) – Apache Kafka and KSQL (Meetup)
• 23.5.2019 (Boston, US) – Kicking Your Database to the Curb and Fun with Kafka! (Meetup)
• 29.5.2019 (Madrid, ES) – Tips and Tricks about Apache Kafka® in the Cloud for Java Developers (Meetup)
• 30.5.2019 (Seattle, US) – Running Flink applications in Kinesis Analytics (Meetup)
• 5.6.2019 (Barcelona, ES) – 10 ways to deploy Apache Kafka® and have fun along the way (Meetup)
• 12.6.2019 (New York, US) – Druid NYC Meetup @ Tumblr (Meetup)
• 20.6.2019 (Bern, CH) – Kafka & it’s friends (Meetup)
• 20.6.2019 (online) – Building a Cloud Data Warehouse on Microsoft Azure with StreamSets (StreamSets Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!