Last Week in Stream Processing & Analytics – 28.3.2018

This is the 108th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Cloud + Streaming Analytics + Data Science = Five Big Data Trends Now by Anand Venugopal
• Understanding the SMACK stack for big data by Steven Vaughan-Nichols

Apache Kafka / Kafka Streams / Confluent Platform

• KSQL in Action: Enriching CSV Events with Data from RDBMS into AWS by Robin Moffatt
• Apache Kafka and Confluent Enterprise Reference Architecture by Gwen Shapira
• Tutorial: Real-Time Database Integration with Apache Kafka via Change Data Capture by Steve Wilkes
• Streaming Data from MySQL into Kafka with Kafka Connect and Debezium by Robin Moffatt
• What even is Apache Kafka by Victor Arias

Spark Streaming

• Azure Event Hubs integration with Apache Spark now generally available by Sabee Grewal
• Introducing Low-latency Continuous Processing Mode in Structured Streaming in Apache Spark 2.3 by Joseph Torres, Michael Armbrust, Tathagata Das & Shixiong Zhu

Apache Flink

• Full Stream Ahead! dA Platform is Generally Available and Ready for Download by

  Kostas Tzoumas, Stephan Ewen & Robert Metzger

Apache Pulsar

• A developer’s introduction to Apache Pulsar Functions by Sanjeev Kulkarni

Streamsets

• Using StreamSets to Tame and Simply Apache Kafka by StreamSets
• Case Study: Cox Automotive Solves Data Drift and ETL Challenges by Amber Lee Dennis
• A Fun Example of Streaming Data into Minecraft by Pat Patterson

Apache NiFi / HDF

• Integrating lucene-geo-gazetteer for Geo Enrichment With Apache NiFi by Tim Spann

Presentations

• Spark (Structured) Streaming vs. Kafka Streams – two stream processing platforms compared by Guido Schmutz
• Meet Apache Kafka : data streaming in your hands by Paolo Patierno
• Introduction to Apache Kafka, Confluent … and why they matter by Paolo Castagna
• Ingesting streaming data into Graph Database by Guido Schmutz
• Distributed Kafka Architecture, Taboola Scale by Tal Sliwowicz & Lior Chaga
• Case Study: Stream Processing on AWS using Kappa Architecture by Joey Bolduc-Gilbert
• Stream-Native Processing with Pulsar Functions by Sanjeev Kulkarni & Sijie Guo
• KSQL – Open-source streaming for Apache Kafka by Florent Ramiere
• Rethinking Stream Processing with Apache Kafka by Florent Ramiere
• Apache Flink: Better, Faster & Uncut by Piotr Nowojski
• Getting Started with AWS Lambda Serverless Computing by Chris Muns
• Enhancing Spark – increase streaming capabilities of your applications by Tomasz Mirowski
• Amazon Kinesis – Building Serverless real-time solution by Roy Ben-Alta & Sefi Itzkovich

Videos

• Use Apache Kafka to Add Streaming Analytics to Your Application by Björn Rost
• Venice with Apache Kafka & Samza by Gaojie Liu
• Embracing Database Diversity with Kafka and Debezium by Frank Lyaruu
• Distributed Kafka Architecture, Taboola Scale by Tal Sliwowicz & Lior Chaga

New Releases

• Debezium 0.7.5
• StormCrawler 1.8

Upcoming Events

• 28.3.2018 (Manchester, UK) – Mesosphere and Confluent at our first Kafka Meetup in Manchester (Meetup)
• 28.3.2018 (Frankfurt, GE) – IoT Stream Processing and Analytics in Real-time (Meetup)
• 17.4.2018 (Laurel, US) – Huge steps for Apache NiFi – Talks on NiFi in Finance and Powerful New Features  (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 19.3.2018

This is the 107th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Events and Commands: Two Faces of the Same Coin? by Tareq Abedrabbo
• Why Streaming Analytics is Critical to Real-time and Transformation by RTInisights Team

Apache Kafka / Kafka Streams / Confluent Platform

• No More Silos: How to Integrate your Databases with Apache Kafka and CDC by Robin Moffatt
• Using Apache Kafka for Real-Time Event Processing by
• Tuning the Kafka Connect Cassandra Source (part 2) by Mike B
• Rethinking Stream Processing With Apache Kafka, Kafka Streams, and KSQL by Kai Waehner
• Kafka Monthly Digest #1: January and February 2018 by Mickael Maison

Spark Streaming

• Streaming Data Pipeline to Transform, Store, and Explore With Kafka, Spark, and Drill by Carol McDonald

Apache Flink

• New in Flink in February 2018 by Till Rohrmann & Aljoscha Krettek & Timo Walther & Michael Winters

Streamsets

• Streaming tool from StreamSets eyes data in motion for GDPR by Jack Vaughan

Apache NiFi / HDF

• Scala Spark Integration With Apache NiFi by Tim Spann
• Creating HTML From PDF, Excel, or Word With Apache NiFi and Apache Tika by Tim Spann
• Versioned DataFlows with Apache NiFi 1.5 and Apache NiFi Registry 0.1.0 by Andrew Lim
• Setting Up a Secure Apache NiFi Registry by Andrew Lim
• Setting Up a Secure NiFi to Integrate with a Secure NiFi Registry by Andrew Lim

Presentations

• Embracing Database Diversity with Kafka and Debezium by Frank Lyaruu
• Building a company-wide data pipeline on Apache Kafka – engineering for 150 billion messages per day by Yuto Kawamura
• How to build leakproof stream processing pipelines with Apache Kafka and Apache Spark by Jordan Hambleton & Guru Medasani

Videos

• Stream processing and real-time data pipelines by Vladimir Schreiner
• Processing Streaming Data using Apache Spark, Storm and Kafka by Tim Renner
• Structured Streaming: What No One Is Talking About by Himanshu Gupta

Upcoming Events

• 21.3.2018 (Sunnyvale, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
• 21.3.2018 (Sunnyvale, US) – Data Operations: a New Discipline for Data in Motion (Meetup)
• 22.3.2018 (online) – Integrating the Enterprise with a Streaming Data Approach (Gigaom Webinar)
• 26.3.2018 (Santa Clara, US) – Advantages of Streaming Architecture in Management of Machine Learning Models (Meetup)
• 28.3.2018 (Manchester, UK) – Mesosphere and Confluent at our first Kafka Meetup in Manchester (Meetup)
• 17.4.2018 (Laurel, US) – Huge steps for Apache NiFi – Talks on NiFi in Finance and Powerful New Features  (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 13.3.2018

This is the 106th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Stream-Relational Processing Platforms by Robert Yokota

Apache Kafka / Kafka Streams / Confluent Platform

• Introduction to Apache Kafka Security by Stéphane Maarek
• Creating DDD aggregates with Debezium and Kafka Streams by Hans-Peter Grahsl, Gunnar Morling
• Using Apache Kafka for Real-Time Event Processing at New Relic by Amy Boyle
• KSQL February Release: Streaming SQL for Apache Kafka by Neil Avery
• KSQL: Topic … does not conform to the requirements by Robin Moffatt
• Streaming data from Kafka into Elasticsearch by Robin Moffatt
• Installing the Python Kafka library from Confluent – troubleshooting some silly errors… by Robin Moffatt
• Why Do We Need Streaming ETL? by Robin Moffatt
• Unit Testing Kafka Streams by Jesse Anderson
• Getting started with the Kafka Connect Cassandra Source by Mike B
• MongoDB and Data Streaming: Implementing a MongoDB Kafka Consumer by Andrew Morgan

Spark Streaming

• Introducing Stream-Stream Joins in Apache Spark 2.3 by Tathagata Das and Joseph Torres

Apache Pulsar

• Introducing Pulsar Functions by Jerry Peng, Sanjeev Kulkarni, Sijie Guo

StreamSets

• Creating Data Pipelines for IoT with StreamSets by Ian Downard
• StreamSets Balances Streaming Data Demands for Security, Access by Alex Woodie

Apache NiFi / HDF

• Ingesting Flight Data ADS-B USB Receiver With Apache NiFi 1.5+ by Tim Spann

Presentations

• Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL by Kai Waehner
• Integrating Apache Kafka and Elastic Using the Connect Framework by Confluent
• Streaming ETL to Elastic with Apache Kafka and KSQL by Nick Dearden
• Why and how to leverage the power and simplicity of SQL on Apache Flink by Fabian Hueske
• Introducing Spark Structured Streaming by Jose Carlos Garcia Serrano
• Kafka’s principles and protocols by Minh Nguyen
• Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

Videos

• Introduction to Apache Kafka as Event-Driven Open Source Streaming Platform by Kai Waehner
• Apache Pulsar -Distributed Pub/Sub as scale by Joe Francis
• Apache Pulsar – fast, durable, flexible pub/sub messaging by Matteo Merli
• Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz
• Kafka Streams and Protobuf stream processing at trivago by Clemens Valiente
• Not less, Not more. Exactly Once Large-Scale Stream Processing in Action by Paris Carbone
• The Database Unbundled: Commit Logs in an Age of Microservices by Tim Berglund
• Streaming Architecture by Tugudal Grall
• SMACK 2.0: Emerging Data Pipelines Panel by IBM Code
• Intelligently Collecting Data at the Edge Intro to Apache NiFi and MiNiFi by Andy LoPresto

New Releases

• Debezium 0.7.4

Upcoming Events

• 14.3.2018 (Leeds, UK) – Kafka Meetup in Leeds (Meetup)
• 14.3.2018 (Kontich, BE) – Cloudera Training Tapas: Hands-on with StreamSets (Meetup)
• 15.3.2018 (online) – Empowering Streams through KSQL: Part 3: Deploying and Operating KSQL (Confluent Webinar)
• 17.4.2018 (Laurel, US) – Huge steps for Apache NiFi – Talks on NiFi in Finance and Powerful New Features  (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 5.3.2018

This is the 105th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Microservices: It’s All About the Events by Joab Jackson
• Architectural Considerations When Building a Suite of Scalable, Interconnected Healthcare Apps by Simon Black

Apache Kafka / Kafka Streams / Confluent Platform

• ETL with Kafka by Akhlaq Malik
• Big, fast, easy data with KSQL by Michael Noll
• IoT for Smart Homes and trillions of messages from Kafka to Elasticsearch by Angelos Petheriotis
• Building a Confluent Platform Data Pipeline on DC/OS by Alex Ly
• Landoop Lenses Promises to Ease Application Development for Kafka Streams by Alex Handy
• Event sourcing using Kafka by Adam Warski
• Squeezing the firehose: getting the most from Kafka compression by Ivan Babrou

Pulsar / Heron

• Leaving the Nest: Heron donated to Apache Software Foundation by Maosong Fu

Apache Flink

• An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!) by Piotr Nowojski & Mike Winters

StreamSets

• Introducing StreamSets Data Protector by Clarke Patterson

Apache NiFi / HDF

• HORTONWORKS DATAFLOW (HDF) 3.1 BLOG SERIES PART 6: INTRODUCING APACHE MINIFI C++ AGENT by Marc Parisi
• Gentle Introduction to Apache NiFi for Data Flow… and Some Clojure by Nicolas Modrzyk

Presentations

• KSQL- Streaming Sql for Kafka by Neha Bhardwaj
• Spring Cloud Stream with Kafka by David Kiss
• Part 02: Live Coding a KSQL Application by Robin Moffatt
• Running Apache NiFi with Apache Spark : Integration Options by Timothy Spann

Videos

• Part 02: Live Coding a KSQL Application by Robin Moffatt
• Understanding Spark Structured Streaming by Himanshu Gupta

Upcoming Events

• 6.3.2018 (Madrid, ES) – An Introduction to Spark Streaming (Meetup)
• 8.3.2018 (online) – HDF 3.1 Series – Part 3: Technical Deep-Dive on New Flow Management Features (Hortonworks Webinar)
• 14.3.2018 (Leeds, UK) – Kafka Meetup in Leeds (Meetup)
• 14.3.2018 (Kontich, BE) – Cloudera Training Tapas: Hands-on with StreamSets (Meetup)
• 15.3.2018 (online) – Empowering Streams through KSQL: Part 3: Deploying and Operating KSQL (Confluent Webinar)
• 17.4.2018 (Laurel, US) – Huge steps for Apache NiFi – Talks on NiFi in Finance and Powerful New Features  (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!