Last Week in Stream Data Integration & Stream Analytics – 23.4.2019

This is the 154th edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

Log Compacted Topics in Apache Kafka by Seyed Morteza Mousavi
Accessing Kafka: Part 1 – Introduction by Jakub Scholz
Accessing Kafka: Part 2 – Node ports by Jakub Scholz
Kafka Summit New York 2019 Session Videos by Tim Berglund
Dawn of DevOps: Managing and Evolving Schemas with Confluent Control Center by Yeva Byzek
Reshaping Entire Industries with IoT and Confluent Cloud by Priya Shivakumar
Kafka-Streams: a road to autoscaling via Kubernetes by Loïc Divad
Bridge to Cloud by Yeva Byzek
ML in KSQL by Christopher Harris
Data Stream Processing for Newbies with Kafka, KSQL, and Postgres by Maria Patterson
Game of Thrones Series 8: Real Time Sentiment Scoring with Apache Kafka, KSQL, Google’s Natural Language API and Python by Francesco Tisiot
Kafka Connect from MySQL to Couchbase with Debezium by Matthew Groves
12 Programming Languages Walk into a Kafka Cluster… by Yeva Byzek

Apache Flink

Build and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics for Java Applications by Steffen Hausmann
Apache Flink’s Application to Season of Docs by Konstantin Knauf
Towards a Unified Data Processing Framework: Batch as a Special Case of Streaming With Apache Flink by Fabian Hueske & Aljoscha Krettek

Presentations

Collection of Presentations from Kafka Summit 2019 by Confluent
Druid Ecosystem @ Yahoo! by Niketh Sabbineni
Scalable, Mission-Critical Machine Learning Infrastructures with Apache Kafka by Kai Waehner
Mirus Reliable, high performance replication for Apache Kafka by Paul Davidson
Apache Kafka – Patterns anti-patterns by Florent Ramiere
An Introduction to KSQL & Kafka Streams Processing with Ticketmaster by Dani Traphagen & Chris Smith
Detecting row patterns with Flink SQL by Dawid Wysakowicz

New Videos

Collection of Videos from Kafka Summit 2019 by Confluent

New Podcasts

Upcoming Events

23.4.2019 (online) – An Introduction to KSQL & Kafka Streams Processing with Ticketmaster (Confluent Webinar)
23.4.2019 (New York, US) – Facial recognition, drug discovery; NoSQL databases; ingesting streaming data! (Meetup)
24.4.2019 (Amsterdam, NL) – Kafka, Microservices and more from Confluent and Camunda (Meetup)
25.4.2019 (London, UK) – Managing Flink on Kubernetes & How to contribute to Apache Flink (Meetup)
25.4.2019 (Zurich, CH) – Apache Kafka® Meetup in Zurich (Meetup)
29.4.2019 (Berlin, GE) – Confluent and Visual Meta talk Kafka (Meetup)
29.4.2019 (Munich, GE) – Kafka, Microservices and more from Confluent and Camunda (Meetup)
16.5.2019 (online) – Driving Digital Transformation by Streaming IoT Data with Apache Kafka (Confluent Webinar)
23.5.2019 (Karlsruhe, GE) – Apache Kafka and KSQL (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 16:08 on April 16, 2019
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Data Integration & Stream Analytics – 15.4.2019

This is the 153rd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Bringing the best of open source to Google Cloud customers by Chris DiBona

Apache Kafka / Kafka Streams / Confluent Platform

Stream Data Processing with Apache Kafka and TensorFlow by Open Data Science
Data Stream Processing for Newbies with Kafka, KSQL, and Postgres by Maria Patterson
Announcing Confluent Cloud for Apache Kafka as a Native Service on Google Cloud Platform by Jay Kreps
From Apache Kafka to Amazon S3: Exactly Once by Konstantine Karantasis
Monitoring Data Replication in Multi-Datacenter Apache Kafka Deployments by Yeva Byzek
Not just for Processing: How Kafka Streams as a Distributed Database Boosted our Reliability and Reduced Maintenance by Nitzan Gilkis
Kafka-Streams: a road to autoscaling via Kubernetes by Loïc Divad

Apache Flink

How to Run Apache Flink Effectively On YARN? by Kartik Khare
Using Apache Flink for Smart Cities: the case of Warsaw by Piotr Wawrzyniak

Apache NiFi

Deploying NiFi workflows on Google Cloud Run by Pierre Villard

Presentations

Confluent Cloud for Apache Kafka by Gaetan Castelein
How To Use Kafka and Druid to Tame Your Router Data by Rachel Pedreschi
How is Kafka so Fast? by Ricardo Paiva & Hervé Rivière
Microservices, Kafka Streams and KafkaEsque by Patrick Schuh & Patrik Kleindl
Cost Effectively and Reliably Aggregating Billions of Messages Per Day Using Kafka by Chunky Gupta & Osman Sarood
Using Kafka and Azure Event hub together for streaming Big data by Sergii Bielskyi

New Videos

GCP for Apache Kafka Users: Stream Ingestion and Processing by Ricardo Ferreira & Karthi Thyagarajan
Advances in Stream Analytics by Sergei Sokolenko & Kir Titievsky & Thomas Weise
Drinking from the Stream: How to Use Messaging Platforms for Scalability&Performance by Mark Heckler
Streaming Analytics: Generating Real Value From Real Time by Evren Eryurek & Sergey Chernyakov & Rémi Aubert
SQL on Kafka with Presto by Itamar Syn-Hershko

New Podcasts

Using Terraform and Confluent Cloud by Confluent

New Releases

Upcoming Events

17.4.2019 (Los Angeles, US) – Taming Apache Kafka with StreamSets & Kubernetes (Streamsets Workshop)
18.4.2019 (Frankfurt, GE) – Orchestrate Apache Kafka® on Kubernetes & more (Meetup)
18.4.2019 (Bellevue, US) – Seattle Apache Kafka Meetup (Meetup)
18.4.2019 (Raleigh, US) – Streams Must Flow: Deployment fault-tolerant stream processing application (Meetup)
23.4.2019 (online) – An Introduction to KSQL & Kafka Streams Processing with Ticketmaster (Confluent Webinar)
23.4.2019 (New York, US) – Facial recognition, drug discovery; NoSQL databases; ingesting streaming data! (Meetup)
24.4.2019 (Amsterdam, NL) – Kafka, Microservices and more from Confluent and Camunda (Meetup)
25.4.2019 (London, UK) – Managing Flink on Kubernetes & How to contribute to Apache Flink (Meetup)
29.4.2019 (Berlin, GE) – Confluent and Visual Meta talk Kafka (Meetup)
29.4.2019 (Munich, GE) – Kafka, Microservices and more from Confluent and Camunda (Meetup)
16.5.2019 (online) – Driving Digital Transformation by Streaming IoT Data with Apache Kafka (Confluent Webinar)
23.5.2019 (Karlsruhe, GE) – Apache Kafka and KSQL (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 21:19 on April 8, 2019
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Data Integration & Stream Analytics – 8.4.2019

This is the 152nd edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

Kafka on Kubernetes — a good fit? by Johann Gyger
Introducing Confluent Platform 5.2 by Mau Barra
KSQL: What’s New in 5.2 by Robin Moffatt
Kafka Monthly Digest: March 2019 by Mickael Maison
Putting Events in Their Place with Dynamic Routing by Yeva Byzek

Spark Streaming

Spark Structured Streaming Joins by Sylvester Daniel

Apache Pulsar

Streamlio Heads to the Cloud With New Version of Apache Pulsar Fast-Data Platform by Sue Walsh

Apache NiFi

Advanced XML Processing With Apache NiFi 1.9.1 by Tim Spann

Presentations

How Zhaopin built its Event Center using Apache Pulsar by Penghui Li & Sijie Guo
No More Silos: Integrating Databases and Apache Kafka by Robin Moffatt
Keep your Data Close and your Caches Hotter using Apache Kafka, KSQL and Connect by Ricardo Ferreira
Kafka as an Event Store – is it Good Enough? by Guido Schmutz
Strimzi: How Apache Kafka has fallen in love with Kubernetes by Paolo Patierno
Future of Apache Flink Deployments: Containers, Kubernetes and More by Till Rohrmann
Where is Confluent going? by Tony Baer
What is Apache Kafka and What is a Streaming Platform? by Hellmar Becker
Use Apache Gradle to Build and Automate KSQL and Kafka Streams by Stewart Bryson
Lessons Learned Building a Connector Using Kafka Connect by Katherine Stanley & Andrew Schofield
Flexible Authentication Strategies with SASL/OAUTHBEARER by Michael Kaminski & Ron Dagostino
Machine Learning on Streaming Data using Kafka, Beam, and TensorFlow by Mikhail Chrestkha & Stephane Maarek
Bulletproof Kafka with Fault Tree Analysis by Andrey Falko
Developing high frequency indicators using real time tick data on apache superset and druid by Emre Tokel, Kerem Başol, M. Yağmur Şahin Zekeriya Besiroglu & Komtas Bilgi Yonetimi
Building Scalable and Extendable Data Pipeline for Call of Duty Games by Yaroslav Tkachenko
Event Driven Architecture with a RESTful Microservices Architecture by Kyle Bendickson
Real Time Streaming Data with Kafka and TensorFlow by Yong Tang
Making Sense of Your Event-Driven Dataflows by Jorge Esteban Quilcate Otoya
Events Everywhere by Jay Kreps
Exactly Once Semantics Revisited by Jason Gustafson
Streaming Design Patterns Using Alpakka Kafka Connector by Sean Glover
Business Growth Is Fueled By Your Event-Centric Digital Strategy by Walid Aly-Hassan
KSQL and Security: The Current State of Affairs by Victoria Xia
Troubleshooting as Your Kafka Clusters Grow by Krunal Vora
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kafka Streams by Alex Leung and Danica Fine, Bloomberg L.P.
Kafka on Kubernetes: Does it really have to be “The Hard Way”? by Viktor Gamov & Michael Ng
Complex Event Flows in Distributed Systems by Bernd Ruecker
Zen and the Art of Streaming Joins – The What, When and Why by Nick Dearden
Maintaining Consistency for a Financial Event-Driven Architecture by Iago Borges
Data Transformations on Ops Metrics using Kafka Streams by Srividhya Ramachandran
Streaming Ingestion of Logging Events at Airbnb by Hao Wang
Kafka Streams at Scale by Deepak Goyal
The Migration to Event-Driven Microservices by Adam Bellemare
From a Million to a Trillion Events Per Day: Stream Processing in Ludicrous Mode by Shrijeet Paliwal
Elastic Data Processing with Apache Flink and Apache Pulsar by Sijie Guo
Flink Forward San Francisco 2019: Creating millions of user sessions using Complex Event Processing by Prem Santosh

New Videos

Kafka Summit NYC 2019 Keynote by Jay Kreps
Kafka Summit 2019 Keynote by James Watters

New Podcasts

Magnus Edenhill on librdkafka 1.0 by Tim Berglund & Gwen Shapira
Ben Stopford on Microservices and Event Streaming by Tim Berglund & Gwen Shapira

New Releases

Upcoming Events

11.4.2019 (Stockholm, SE) – Deep Anomaly Detection at Swedbank and Streaming at King (Meetup)
11.4.2019 (online) – No More Silos: Integrating Databases into Apache Kafka® (Confluent Webinar)
18.4.2019 (Frankfurt, GE) – Orchestrate Apache Kafka® on Kubernetes & more (Meetup)
18.4.2019 (Bellevue, US) – Seattle Apache Kafka Meetup (Meetup)
24.4.2019 (Amsterdam, NL) – Kafka, Microservices and more from Confluent and Camunda (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 00:19 on April 2, 2019
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Data Integration & Stream Analytics – 1.4.2019

This is the 151st edition of my blog series blog series around Stream Data Integration and Stream Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Online Event Processing by Martin Kleppmann & Alastair R. Beresford & Boerge Svingen
The Software Architects’ Newsletter by InfoQ
Building Real-Time Analytics APIs at Scale by Sylvain Friquet
Efficient Window Aggregation with General Stream Slicing by Jonas Traub & Philipp Grulich & Alejandro Rodríguez Cuéllar & Sebastian Breß & Asterios Katsifodimos & Tilmann Rabl & Volker Mark
DATA is the New OIL: Mine it Well to Tap your Customers in Real-Time by Neha Sood

Apache Kafka / Kafka Streams / Confluent Platform

Should I use Confluent or Kafka? by Kai Waehner
The Importance of Distributed Tracing for Apache-Kafka-Based Applications by Jorge Quilcate
Tutorial: An End-to-end Streaming Analytics Stack for Network Telemetry Data by Eric Graham
Consuming Messages Out of Apache Kafka in a Browser by Joseph Rea

Spark Streaming

Spark Streaming vs. Structured Streaming by Anuj Saxena

Apache Flink

Process big data transaction stream in near real-time with Apache Flink by IOTA

StreamSets

Building a Slack Slash Command as a Microservice Pipeline by Pat Patterson

Presentations

Introduction to leveraging Azure Event Hubs from Apache Kafka by Nikolai Blackie
Beyond the brokers – A tour of the Kafka ecosystem by Damien Gasparina
Brooklin Mirror Maker – How and why we moved away from Kafka Mirror Maker by Shun-ping Chiu

New Videos

Taming IoT Data: Making Sense of Sensors with SQL Streaming by Hans-Peter Grashl
Streaming Visualisation by Guido Schmutz
What’s New in Kafka 2.2? by Stephane Maarek
Streaming data processing with Apache Spark by Roksolana Diachuk

New Releases

Upcoming Events

2.4.2019 (San Francisco, US) – Apache Druid Bay Area Meetup @ Unity Technologies (Meetup)
3.4.2019 (New York, US) – Fraud Detection with Apache Kafka and KSQL (Meetup)
4.4.2019 (New York, US) – Stream Processing Beyond Streaming with Apache Flink, by Co-Creator of Flink (Meetup)
4.4.2019 (Seattle, US) – Exactly Once Processing with Flink and Pravega (Meetup)
11.4.2019 (Stockholm, SE) – Deep Anomaly Detection at Swedbank and Streaming at King (Meetup)
18.4.2019 (Frankfurt, GE) – Orchestrate Apache Kafka® on Kubernetes & more (Meetup)
24.4.2019 (Amsterdam, NL) – Kafka, Microservices and more from Confluent and Camunda (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!