gschmutz 17:56 on October 26, 2020
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Data Integration & Stream Analytics – 26.10.2020

This is the 207th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

Kafka

Cloud-Like Flexibility and Infinite Storage with Confluent Tiered Storage and FlashBlade from Pure Storage by Marc Selwan
Optimizing Kafka producers by Paul Mellor
Kafka — Core Concepts & Terminologies by Animesh Gaitonde
Real Time APIs in the Context of Apache Kafka by Robin Moffatt
Top 5 Things Every Kafka Developer Should Know by Bill Bejeck
Publishing Events to Kafka using an Outbox Pattern by Swapnil Desai
Intrusion Detection with ksqlDB by Maxime Ribera & Göraud Dugé De Bernonville
How Krake Makes Floating Workloads on Confluent Platform by Mirko Kämpf & Matthias Goerens
Lessons Learned from Evolving a Risk Management Platform to Event Streaming by Daniel Jagielski
Towards a Graphical Debezium User Interface by Gunnar Morling
Kafka pitfalls – Q&A with a Kafka Architect by Jaroslaw Kijanowski
Monolith to Microservices with Kafka Streaming Data Connector by Michael Bogan
How Call Analytics Pipeline works, Part 1: Kafka by Nicholas Yi
The InfoQ eMag: Real-Time APIs: Designing, Operating, and Observing by InfoQ
Kafka consumer Issues: Fixing JVM Garbage Collection Problems by Lior Durahly
Apache Kafka on Kubernetes with Strimzi — Part 1: Creating and Deploying a Strimzi Kafka Cluster by Sina Nourian
Apache Kafka on Kubernetes with Strimzi — Part 2: Creating Producer and Consumer apps using Go and Scala and deploying on Kubernetes by Sina Nourian
Apache Kafka on Kubernetes with Strimzi — Part 3: Monitoring our Strimzi Kafka Cluster with Prometheus and Grafana by Sina Nourian
Bounding ksqlDB Memory Usage by Rohan Desai

Apache Flink

Flink on PaaSTA: Yelp’s new stream processing platform runs on Kubernetes by Antonio Verardi
From Aligned to Unaligned Checkpoints – Part 1: Checkpoints, Alignment, and Backpressure by Arvid Heise & Stephan Ewen

New Presentations

Twitter’s Apache Kafka® Adoption Journey by Ming Liu & Yiming Zang

New Videos

Twitter’s Apache Kafka® Adoption Journey by Ming Liu & Yiming Zang
Building your First Connector for Kafka Connect by Ricardo Ferreira

New Podcasts

Building an Apache Kafka Center of Excellence Within Your Organization by Confluent Streaming Audio

New Releases

Strimzi 0.20.0

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 20:29 on October 13, 2020
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Data Integration & Stream Analytics – 13.10.2020

This is the 206th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

Introduction to Event Stream Processing using Eclipse Streamsheets by Philip Ackermann

Kafka

ETL Batch Processing With Kafka? by Tomasz Kaszuba
Sync MySQL to PostgreSQL using Debezium and KafkaConnect by Sumant Rana
Transactional Outboxes by Sachin Kumar
How to save Confluent Kafka schema registry with cURL by Lukas Futera
Solving My Weird Kafka Rebalancing Problems & Explaining What Is Happening and Why? by Benjamin Feldmann
Streaming XML messages from IBM MQ into Kafka into MongoDB by Robin Moffatt
Ingesting XML data into Kafka – Introduction by Robin Moffatt
Ingesting XML data into Kafka – Option 1: The Dirty Hack by Robin Moffatt
Ingesting XML data into Kafka – Option 2: Kafka Connect plus Single Message Transform by Robin Moffatt
Ingesting XML data into Kafka – Option 3: Kafka Connect FilePulse connector by Robin Moffatt
Introducing Cluster Linking in Confluent Platform 6.0 by Brian Byrne & Rajini Sivaram
Restoring Balance to the Cluster: Self-Balancing Clusters in Confluent Platform 6.0 by Steve Rodrigues
Real Time Machine Learning at Scale using SpaCy, Kafka & Seldon Core by Alejandro Saucedo
Kafka Monthly Digest – September 2020 by Mickael Maison
On the importance of load testing Kafka by Evan Rosenstein
Project Metamorphosis Month 6: Secure Apache Kafka in Confluent Cloud by Dan Rosanova
Simplify Kafka at Scale with Confluent Tiered Storage by Joshua Robinson
Kafka record tracing by Cosmin Mircea
Raspberry Pi Kafka cluster by Igor Souza
Getting Started with Kafka Connect for New Relic by Rahul Bhattacharya & Gavin Johnson
Building a Lambda Architecture with Druid and Kafka Streams by Jeremy Plichta
How Real-Time Materialized Views Work with ksqlDB, Animated by Michael Drogalis
Orchestrating Multi-Brokers Kafka Cluster through CLI Commands by Gautam Goswami
ksqlDB – How to model a variable number of fields in a nested value (STRUCT) by Robin Moffatt

Apache Flink

Stateful Functions Internals: Behind the scenes of Stateful Serverless by Tzu-Li (Gordon) Tai

New Presentations

New Features in Confluent Platform 6.0 / Apache Kafka 2.6 by Kai Waehner
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka by Kai Waehner
Event Driven Architecture: Mistakes, I’ve made a few… by Simon Aubry
Apache Kafka Architectures and Fundamentals by Henrik Janzon
Choose Right Stream Storage: Amazon Kinesis Data Streams vs MSK by Kim Sungmin

New Videos

I don’t always test my streams, but when I do I do it in production by Viktor Gamov
Kafka as a Platform: The Ecosystem from the Ground Up by Robin Moffatt

New Podcasts

Episode 20: Kafka Streams by Quarkus Insights
Creating Your Own Kafka Improvement Proposal (KIP) as a Confluent Intern ft. Leah Thomas by Confluent Streaming Audio
Confluent Platform 6.0 | What’s New in This Release + Updates by Confluent Streaming Audio

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!

gschmutz 18:27 on October 2, 2020
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Data Integration & Stream Analytics – 2.10.2020

This is the 205th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As usual, find below the new blog articles, presentations, videos and software releases from last week. Happy reading and stay safe!

News and Blog Posts

General

From Monolith to Event-Driven: Finding Seams in Your Future Architecture by Jayson Go
Be Vigilant about Time Order in Event-Based Data Processing by Mingwei Li

Kafka

How to Make Kafka Producer/Consumer Production-Ready by Shivanshu Goyal
Managing Kafka Connectors at scale using Kafka Connect Manager by Sandeep Mehta
Infrastructure Modernization with Google Anthos and Apache Kafka by Josh Treichel & Vivian Fialho
What’s new in Azkarra Streams 0.8 The micro-framework that takes your Kafka Streams apps to the next level! by Florian Hussonnois
Apache Kafka DevOps with Kubernetes and GitOps by Rick Spurgeon
Kafka and XML Messages – Transformation, Connector, Middleware by Kai Waehner
Data Ingestion into Azure Data Explorer using Kafka Connect by Abhishek Gupta
Building a Machine Learning Logging Pipeline with Kafka Streams at Twitter by Peilin Yang
How Real-Time Stream Processing Works with ksqlDB, Animated by Michael Drogalis
Introducing Confluent Platform 6.0 by Ganesh Srinivasan
ksqlDB 0.12.0 Introduces Real-Time Query Upgrades and Automatic Query Restarts by Alan Sheinberg
Build a data streaming pipeline using Kafka Streams and Quarkus by Kapil Shukla
Insights from the first Strimzi survey by Tom Bentley
Making Sense of Change Data Capture Pipelines for Postgres with Debezium Kafka Connector by Cemal Turkoglu
Using KCache with a Persistent Cache by Rayokota
Setting key value when piping from jq to kafkacat by Robin Moffatt
MongoDB Connector for Apache Kafka 1.3 Available Now by Robert Walters
Accelerate Data Warehousing by Streaming Data with Confluent Cloud into Amazon Redshift by Jobin George, Joseph Morais, and Michael Coughlin
ksqlDB Meets Java: An IoT-Inspired Demo of the Java Client for ksqlDB by Victoria Xia

Apache Pulsar

Announcing MQTT-on-Pulsar: Bringing Native MQTT Protocol Support to Apache Pulsar by Penghui Li & Xiaolong Ran

StreamSets

Using IAM Roles with StreamSets Data Collector on AWS Elastic Kubernetes Service by Mark Brooks

New Presentations

Kafka Summit 2020: Event Streaming Everywhere by Confluent
Bridge Your Kafka Streams to Azure by Derk Van Ogtrop & Johanna De Castro-Dee & Ben Echols
Building a real-time data processing pipeline using Apache Kafka, Kafka Connect, Elasticsearch and Kibana by Paul Brebner
Apache Kafka® and Analytics in a Connected IoT World by Kai Waehner
Real time stock processing with Apache NiFi, Apache Flink and Apache Kafka by Timothy Spann
Building Physical Industrial IoT Models with Kafka by Oliver Thamm

New Videos

Kafka Summit 2020: Event Streaming Everywhere by Confluent
Risk Management in Retail with Stream Processing by Daniel Jagielski
Livestreams 009: Live QA, Confluent Cloud Quick Start, ksqlDB, and Project Reactor by Victor Gamov

New Podcasts

Using Event Modeling to Architect Event-Driven Information Systems ft. Bobby Calderwood by Confluent Streaming Audio

New Releases

Please let me know if that is of interest. Please tweet your projects, blog posts, and presentations & videos to @gschmutz to get them listed in next week’s edition!