Last Week in Stream Processing & Analytics – 14.1.2018

This is the 140th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Exposing Remote JMX in your Kafka Setup by Jendrik Poloczek
• How to Run Kafka on Openshift, the Enterprise Kubernetes, With AMQ Streams by
• Stream data with Apache Kafka into the IBM Db2 Event Store by Loic Julien
• Updated Apache Kafka Benchmarking by Ben Slater

Apache Flink

• Towards achieving a 10x compile-time improvement in a Flink codebase by Aris Koliopoulos
• Alibaba Acquires Apache Flink Backer data Artisans by Alex Woodie

Spark Streaming

• Apache Spark 2.4.0 features – watermark configuration by Bartosz Konieczny

StreamSets

• Ingesting Data from Relational Databases to Cassandra with StreamSets by Todd McGrath

Presentations

• Streams, Tables, and Time in KSQL by Matthias Sax
• Big data processing with PubSub, Dataflow, and BigQuery by Thuyen Ho
• Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL by Sönke Liebau
• The Open Source SQL Streaming Engine for Apache Kafka by Kai Waehner

New Releases

• StreamSets Data Collector 3.7.1

Upcoming Events

• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 22.1.2019 (Menlo Park, US) – CDC and Building a Streaming Analytics Stack with Kafka and Druid (Meetup)
• 23.1.2019 (Washington, US) – Apache NiFi in Washington (Meetup)
• 24.1.2019 (New York, US) – Unifying Messaging, Queuing, Streaming & Light Weight Compute with Apache Pulsar (Meetup)
• 29.1.2019 (New York, US) – Druid NYC Meetup @ Datadog (Meetup)
• 5.2.2019 (Vancouver, CA) – How to Structure your Data Pipeline and New Kafka Features You Might Not Know
• 13.2.2019 (Melbourne, AU) – Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications (Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Advertisements

Last Week in Stream Processing & Analytics – 9.1.2018

This is the 139th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• With Real-Time, What Data Center Management Will Look Like in 2019 by Kevin McNamara
• Data Insights Engine @ MakeMyTrip by Aditya Banerjee
• From Real-Time to the Edge: The Top IoT Trends for the Coming Year by Michael Kanellos

Apache Kafka / Kafka Streams / Confluent Platform

• Building Secure and Governed Microservices With Kafka Streams by George Vetticaden
• Easy Ways to Generate Test Data in Kafka by Yeva Byzek
• Kafkawize by Muralidhar Basani
• Using GridGain With Kafka Connector by Akmal Chaudhri
• Storm-Kafka Integration With Configurations and Code by Rinu Gour
• Kafka Monthly Digest: December and 2018 in Review by Mickael Maison
• Kappa architecture at NTT: Building a streaming analytics stack with Druid and Kafka by Paolo Lucente
• Building microservices with kafka streams by Andreas Schroeder
• Getting Your Feet Wet with Stream Processing – Part 1: Tutorial for Developing Streaming Applications by Yeva Byzek

Apache Flink

• German startup data Artisans sells to Alibaba for €90 million by Andrii Degeler
• Alibaba’s data Artisans acquisition breathes new life into Apache Flink by Tony Baer

Presentations

• Apache flink 1.7 and Beyond by Till Rohrmann

Videos

• Who’s Tweeting about #DataSciCon by Viktor Gamov
• Distributed systems for stream processing: Apache Kafka and Spark Streaming by Alena Hall

New Releases

• Wallaroo 0.6.1
• StormCrawler 1.13

Upcoming Events

• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)
• 9.1.2019 (Munich, GE) – Building Microservices with Kafka Streams: Beyond Kafka-for-Messaging (Meetup)
• 10.1.2019 (Leeds, UK) – Running a cluster in the Cloud and ATM Fraud detection with Kafka and KSQL (Meetup)
• 10.1.2019 (Berlin, GE) – Streams, Tables, and Time in KSQL (Meetup)
• 10.1.2019 (Frankfurt, GE) – Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL (Meetup)
• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 23.1.2019 (Washington, US) – Apache NiFi in Washington (Meetup)
• 29.1.2019 (New York, US) – Druid NYC Meetup @ Datadog (Meetup)
• 5.2.2019 (Vancouver, CA) – How to Structure your Data Pipeline and New Kafka Features You Might Not Know
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)d

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 3.1.2018

Happy New Year!

This is the 138th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Event Sourcing with AWS Lambda by Paul Bourdel
• Apps Depend on Real-Time Streaming Data, Here’s How to Manage Them by Shivnath Babu

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka Streams: Is it the Right Stream Processing Engine for You? by
• Anomalia Machina 5 – Application Monitoring with Prometheus: Massively Scalable Anomaly Detection with Apache Kafka and Cassandra by Paul Brebner
• Interpreting Kafka’s Exactly-Once Semantics by Rahul Agarwal

Presentations

• Real-time Analytics with Apache Kafka by Mohammad Mehdi Yalollahi & Alihossein Shahabi

Videos

• The Magical Group Coordination Protocol of Apache Kafka by Gwen Shapira
• Unlimited Kafka Messages by Maor Mordehay
• Fast Data apps with Alpakka Kafka connector and Akka Streams by Sean Glover
• Stream Processing Design Patterns by Andreas Markmann

New Releases

• Apache Pulsar 2.2.1

Upcoming Events

• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)
• 10.1.2019 (Leeds, UK) – Running a cluster in the Cloud and ATM Fraud detection with Kafka and KSQL (Meetup)
• 10.1.2019 (Berlin, GE) – Streams, Tables, and Time in KSQL (Meetup)
• 10.1.2019 (Frankfurt, GE) – Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL (Meetup)
• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 25.12.2018

Merry Christmas!

This is the 137th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Top 7 Trends in Streaming Data for 2019 by Ori Rafael
• Is Your Stream Processor Obese? by Srinath Perera

Apache Kafka / Kafka Streams / Confluent Platform

• 12 Days of Tech Tips by Gwen Shapira
• How to Govern Data using Kafka & Avro by Stéphane Derosiaux
• Topology-Based Event Correlation With Apache Spark Streaming by
• Monitoring Kafka Streams Microservices with Hortonworks Streams Messaging Manager (SMM) by George Vetticaden
• Introducing Hive-Kafka integration for real-time Kafka SQL queries by George Vetticaden & Slim Bouguerra
• Kafka Performance Tuning — Ways for Kafka Optimization by Rinu Gour
• Kafka-Docker: Steps To Run Apache Kafka Using Docker by Rinu Gour
• Chris Splinter, PM – Developer Solutions Introducing the DataStax Apache Kafka™ Connector by Chris Splinter
• Real-Time Sysmon Processing via KSQL and HELK — Part 1: Initial Integration by Roberto Rodriguez

Apache Pulsar

• Apache Pulsar Architecture: Designing for Streaming Performance and Scalability by Sijie Guo

Spark Streaming

• Topology-Based Event Correlation With Apache Spark Streaming by Arseniy Tashoyan

StreamSets

• Five Ways to Scale Kafka with StreamSets by Pat Patterson

Presentations

• Bust the burglars: Machine Learning with TensorFlow and Apache Kafka by Erik-Berndt Scheper
• State Schema Evolution for Apache Flink® Applications by Tzu-Li Tai
• Kafka from Go by Javier Sanz Olivera
• Enterprise wide publish subscribe with Apache Kafka by Johan Louwers
• Hortonworks DataFlow (HDF) 3.3 – Taking Stream Processing to the Next Level by Dinesh Chandrasekhar
• Data Integration and Real-Time Data Processing with Spring Boot by Sabby Anandan
• Apache Flink 1.7 and Beyond by Till Rohrmann

Videos

• Under 10 seconds from source to target – Real-Time Data Streaming in the Insurance Industry by Christian Nicoll & Christopher Knauf & Kai Waehner
• Bust the burglars: Machine Learning with TensorFlow and Apache Kafka by Erik-Berndt Scheper

New Releases

• Druid 0.13.0
• Debezium 0.9.0.Beta2
• Apache Flink 1.7.1
• Apache Flink 1.6.3
• Strimzi 0.9.0

Upcoming Events

• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)
• 10.1.2019 (Leeds, UK) – Running a cluster in the Cloud and ATM Fraud detection with Kafka and KSQL (Meetup)
• 10.1.2019 (Berlin, GE) – Streams, Tables, and Time in KSQL (Meetup)
• 10.1.2019 (Frankfurt, GE) – Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL (Meetup)
• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 17.12.2018

This is the 136th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Deep Dive into KSQL Deployment Options by Hojjat Jafarpour
• Confluent Cloud: Security Overview and Recommendations by Gwen Shapira
• One for the Road by Dan Taylor
• Kafka Streams Interactive Queries by Abhishek Gupta
• Streaming data from Oracle into Kafka (December 2018) by Robin Moffatt
• Accessing Kafka on host machine from minikube pods by Abhishek Gupta
• License Changes for Confluent Platform by Jay Kreps
• A Quick Comment On Bryan Cantrill’s Blog On Licensing by Jay Kreps
• Apache Kafka Security | Need and Components of Kafka by Rinu Gour
• Southpaw – Streaming Left Joins with Change Data Capture by Morrigan Jones
• Our real-time data processing – part 2 by Bartosz Łoś
• Democratizing Analytics Within Kafka With 3 New Access Patterns by George Vetticaden
• Docker Tips and Tricks with KSQL and Kafka by Robin Moffatt
• HDP 3.1 Released! All The Kafka! by Tim Spann
• New Features of Kafka 2.1 by Stéphane Maarek

Spark Streaming

• Apache Spark and Developing Applications Using Spark Streaming by Radhakrishnan Ramakrishnan

StreamSets

• Getting Started with Apache Pulsar and Data Collector by Pat Patterson

NiFi / HDF

• What’s new in Hortonworks DataFlow 3.3 by Dinesh Chandrasekhar
• Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector by Tim Spann

Presentations

• ATM Fraud detection with Kafka and KSQL by Robin Moffatt
• Polylog: A Log-Based Architecture for Distributed Systems by Morrigan Jones & Eric Weaver
• IoT data streaming with Apache Kafka by Paolo Patierno
• Cloud-Native Streaming Platform: Running Apache Kafka on PKS (Pivotal Container Service) by Viktor Gamov
• What’s new in Apache Spark 2.4 by Xiao Li
• Demystifying Data-Intensive Systems On Kubernetes by Alena Hall

Videos

• The IoT data river: processing the flood with Apache Kafka Streams by Paolo Patierno
• Demystifying Data-Intensive Systems On Kubernetes by Alena Hall

New Releases

• Landoop Lenses 2.0
• Apache Beam 2.9.0

Upcoming Events

• 18.12.2018 (online) – How-To Build Real-Time Dashboards (MemSQL Webinar)
• 19.12.2018 (online) – Hortonworks DataFlow (HDF) 3.3 – Taking Stream Processing to the Next Level (Hortonworks Webinar)
• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)
• 10.1.2019 (Leeds, UK) – Running a cluster in the Cloud and ATM Fraud detection with Kafka and KSQL (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 9.12.2018

This is the 135th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Real-Time Machine Learning With TensorFlow in Data Collector by Dash Desai
• By 2025, nearly 30 percent of data generated will be real-time, IDC says by Stephanie Condon
• With Real-Time Analytics, Alexa Will Soon Know You Better by Seeta Hariharan

Apache Kafka / Kafka Streams / Confluent Platform

• Time at the Bar Chart by Dan
• Kafka’s Beer Festival by Dan
• How Pinterest runs Kafka at scale by Yu Yang
• Queryable Kafka Topics with Kafka Streams by Robert Schmid
• Kafka Go client: No Producer error during connection ? by Abhishek Gupta
• An Honest Review of AWS Managed Apache Kafka: Amazon MSK by Stéphane Maarek
• Big Data Spain Talk About KSQL, the Streaming SQL Engine for Apache Kafka by Kai Waehner
• Kafka Connect CLI tricks by Robin Moffatt
• Optimizing Read Access to Sharded MongoDB Collections utilizing Apache Kafka Connect by Hans-Peter Grahsl
• Twitter’s Kafka adoption story by Philip Su
• What happened in the Kafka community in November 2018? by Mickael Maison
• Kafka Streams – Is it the right Stream Processing engine for you? by George Vetticaden
• Kafka Streams and KSQL with Minimum Privileges by Gwen Shapira
• Building Secure and Governed Microservices with Kafka Streams by George Vetticaden
• Kafka Summit 2019 Call for Papers, Tracks and Office Hours by Tim Berglund

Apache Pulsar

• Simplifying Zhaopin’s Event Center with Apache Pulsar by Penghui Li

Samza

• Samza 1.0: Stream Processing at Massive Scale by Jagadish Venkatraman

Apache Flink

• Stream processing with Apache Flink and Minio by Nitish Tiwari
• 5 “baby” steps to develop a Flink application by

  Marta Paes and Markos Sfikas

StreamSets

• Low-latency Inference Using Databricks ML in StreamSets by Dash Desai

Presentations

• Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner
• Build the future Scale in / scale out with Kafka-Streams and Kubernetes by Loic Divad
• Large scale stream processing with Apache Flink by Nikolay Stoitsev
• Intro to Kapacitor for Alerting and Anomaly Detection by Noah Crowley
• KSQL: Open Source Streaming for Apache Kafka by Kim Berglund
• Visualizing Streams by Francesco Tisiot
• Embrace the Anarchy : Apache Kafka’s Role in Modern Data Architectures  by Robin Moffatt
• No More Silos: Integrating Databases and Apache Kafka by Robin Moffatt
• Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt
• Serverless Stream Processing Pipeline Best Practices by Roy Ben-Alta & Eyal Levi
• Stream Processing Live Traffic Data with Kafka Streams bx Tim Ysewyn & Tom Van den Bulck
• Serverless Stream Processing Tips & Tricks by Rajeev Chakrabarti & Girish Sood
• Bridge OLTP and Stream Processing with Amazon Kinesis, AWS Lambda, & MongoDB Atlas by Eliot Horowitz
• Building Event Driven (Micro)services with Apache Kafka by Guido Schmutz
• Introduction to Stream Processing by Guido Schmutz
• Spark (Structured) Streaming vs. Kafka Streams by Guido Schmutz

Videos

• Kafka Consumers and Streams Failover Explained ft. Matthias Sax by Ask Confluent
• Rethinking stream processing with Kafka Streams and KSQL by Viktor Gamov by Viktor Gamov
• Introduction to Kafka Streaming by Lindsey Dew & Omnia Ibrahim
• Integrating Machine Learning, Reactive Microservices, and Akka with Kafka by Andrew Bonham
• Fast Data apps with the Alpakka Kafka connector by Sean Glover
• Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming by Gerard Maas
• Processing Fast Data with Apache Spark: The Tale of Two Streaming APIs by Gerard Maas
• Introduction to Confluent Operator for Kubernetes by Viktor Gamov
• State of the Apache NiFi Ecosystem & Community by Aldrin Piri
• Taking The Pain out of Deploying Streaming Applications by Craig Blitz & Gerard Maas
• Streaming pipelines in a batch world: The journey from batch to stream by Jeffrey Sewell
• Fast Data apps with the Alpakka Kafka connector by Sean Glover
• Mastering the Microservices, Fast Data, & Hybrid Cloud Trifecta by Edward Hsu
• Deeper insights from streaming data by Dean Wampler & Steven Astorino

New Releases

• Apache Flink 1.7.0
• Wallaroo 0.6.0

Upcoming Events

• 10.12.2018 (Frankfurt, GE) – ATM Fraud Detection with Apache Kafka and KSQL (Meetup)
• 13.12.2018 (Atlanta, US) – The present state of operational analytics and Apache Druid (Meetup)
• 13.12.2018 (online) – Real-Time Data Streaming in the Insurance Industry (Confluent / Attunity webinar)
• 18.12.2018 (online) – How-To Build Real-Time Dashboards (MemSQL Webinar)
• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 26.11.2018

This is the 134th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Using Apache Kafka to Drive Cutting-Edge Machine Learning by Kai Waehner
• KCache: An In-Memory Cache Backed by Kafka by Robert Yokota
• Streaming in the Clouds: Where to Start by Priya Shivakumar
• Big Data Spain Talk About KSQL, the Streaming SQL Engine for Apache Kafka by Kai Waehner
• Democratizing Analytics within Kafka with Three New Access Patterns by George Vetticaden
• 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments by Yeva Byzek
• Getting Familiar with Azure Event Hubs for Apache Kafka by Steef-Jan Wiggers

Spark Streaming

• Scaling Spark Streaming for Logging Event Ingestion by Hao Wang and Liyin Tang

Apache Flink

• Data Streaming with Flink by Oleksandr Nitavskyi
• 7 reasons to use stream processing & Apache Flink in the IoT industry by Jakub Piasecki

StreamSets

• Flexible Data Pipelines: 5-Minute Interview with Pat Patterson, Technical Director, StreamSets by Bryce Merkl Sasaki

Apache NiFi

• Real-Time Stock Processing With Apache NiFi and Apache Kafka, Part 1 by Tim Spann

Presentations

• KSQL – Stream Processing simplified! by Guido Schmutz
• Kafka, KSQL & YugaByte DB: Building Scalable & Reliable Streaming Apps The Easy Way by Bogdan Matican & Bharat Baddepudi
• Meet Apache Kafka : data streaming in your hands by Paolo Patierno
• Deploying Kafka Streams Applications with Docker and Kubernetes by Gwen Shapira & Matthias J. Sax
• Stream Processing in Action by Nirmal Fernando
• Streaming Graph Partitioning by Zainab Abbas & Vasiliki Kalavri & Paris Carbone & Vladimir Vlassov
• Spline: Data Lineage For Spark Structured Streaming  by Marek Novotny & Vaclav Kosar
• Time and ordering in streaming distributed systems  by Zhenzhong Xu
• Using Event Streams in Serverless Applications by Jonathan Dee
• Heterogeneous job processing with Apache Kafka by Hua Chu
• Stream Processing with Hazelcast Jet by Rafał Leszko
• Patterns for Building Streaming Apps by Sriskandarajah Suhothayan

Videos

• Kafka at PayPal: Enabling 400 billion messages a day by Kevin Lu & Maulin Vasavada & Na Yang
• Stream Processing with Hazelcast Jet by Rafał Leszko
• The Rise of Streaming SQL by Sriskandarajah Suhothayan
• Patterns for Building Streaming Apps by Sriskandarajah Suhothayan

New Releases

• Apache Kakfa 2.1.0
• Debezium 0.9.0.Beta1

Upcoming Events

• 27.11.2018 (online) – Stream into Kafka Series: Monitoring and Protecting Data in Kafka (StreamSets Webinar)
• 27.11.2018 (Prague, CZ) – Streaming applications – from basic to advance usage (Meetup)
• 27.11.2018 (online) – Stream into Kafka Series: Monitoring and Protecting Data in Kafka (StreamSets Webinar)
• 28.11.2018 (Atlanta, US) – Stream Processing Best Practices – In the Cloud and with Apache Ignite (Meetup)
• 29.11.2018 (online) – Run Apache Kafka on Kubernetes with Red Hat AMQ streams (Redhat Webinar)
• 6.12.2018 (Cambridge, US) – StreamSets Boston User Group (Meetup)
• 6.12.2018 (online) – Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming (Lightbend Webinar)
• 7.12.2018 (Sao Paulo, BR) – Apache Kafka® Rethinked: From Messaging to a Streaming Data Platform (Meetup)
• 10.12.2018 (Frankfurt, GE) – ATM Fraud Detection with Apache Kafka and KSQL (Meetup)
• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!