Last Week in Stream Processing & Analytics – 16.7.2018

This is the 121th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Change data capture as a gateway to a streaming platform by Gwen Shapira
• Solving the Challenges of IoT Analytics by Jon Bock

Apache Kafka / Kafka Streams / Confluent Platform

• Model Serving: Stream Processing vs. RPC / REST with Java, gRPC, Apache Kafka, TensorFlow by Kai Waehner
• Kafka 1.0 on HDInsight lights up real time analytics scenarios by Dhruv Goel
• Apache Mesos and Kafka Streams for Highly Scalable Microservices by Kai Waehner

Apache Pulsar

• Apache Pulsar — Gentle Introduction by Karthikeyan Palanivelu

Streamsets

• StreamSets Data Collector Installation Using tarball (SDC) by Ark

Apache NiFi

• DataFlow com Apache Nifi + Docker + Rabbit + Kibana by Alexandre Gasparino

Presentations

• Lessons Learned from Buildiing an Apache Kafka Managed Service by Nitin Kumar
• Azure Event Hubs and Event Hubs for Kafka Ecosystems by Dan Rosanova & Shubha Vijayasarathy

Videos

• Processing Streaming Data with KSQL by Tim Berglund
• Dhalion: towards self-regulating stream processing by Avrilia Floratou & Ashvin Agrawal
• Make streaming processing towards ANSI SQL by Shaoxuan Wang
• Real-time data streams with Apache Kafka and Spark by Alena Hall
• Using Spark Streaming and NiFi for the next generation of ETL in the enterprise by Darryl Dutton & Kenneth Poon
• The Database Unbundled: Commit Logs in an Age of Microservices by Tim Berglund
• Foundations of streaming SQL: stream & table theory by Anton Kedin
• Join Semantics in Kafka Streams by Himani Arora
• Curing the Kafka Blindness by George Vetticaden
• Visualizing Kafka Security by Vipin Rathor
• Bootstrapping state in Apache Flink by Gregory Fee
• How TMW Systems uses NiFi and SAM to create sub-second transportation visibility by Krishna Potluri & Donnie Wheat
• Using NIFI to simplify data flow & streaming use cases @ Mastercard by Vishal Anand
• Sharing metadata across the data lake and streams by Alan F. Gates
• DataSketch based aggregations and windowing in a streaming query system by Akshai Sarma & Michael Natkovich
• Experience Trimble’s Live Blockchain—Based System powered by Apache Kafka and Apache NiFi by Timothy Leonard & Donnie Wheat

New Releases

• Apache Flink 1.5.1

Upcoming Events

• 18.7.2018 (Madrid, SP) – All about Kafka: origins, ecosystem and future directions (Meetup)
• 24.7.2018 (Berlin, GE) – Kafka, KSQL & IoT with Crate and Confluent (Meetup)
• 25.7.2018 (Krakow, PL) – Stream processing and build streaming data pipelines with Apache Kafka & KSQL (Meetup)
• 1.8.2018 (online) – Unleashing Apache Kafka® and TensorFlow in the Cloud (Confluent Webinar)
• 2.10.2018 (New Jersey, US) – Apache NiFi and Blockchain (Meetup)
• 9.10.2018 (Munich, GE) – Confluent Streaming Event 2018 in München (Confluent Event)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Advertisements

Last Week in Stream Processing & Analytics – 6.7.2018

This is the 120th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Is Event Streaming the new Big Thing for Finance? by Ben Stopford
• Processing streams of data with Apache Kafka and Spark: ingestion, processing, reaction, examples  by Alena Hall

Apache Kafka / Kafka Streams / Confluent Platform

• June Preview Release: Packing Confluent Platform with the Features You Requested! by Hojjat Jafarpour & Ran Ma
• This Week in Numbers: Apache Kafka’s Metamorphosis by Lawrence Hecht

Streamsets

• Using Docker Wrong: My Journey to a Better Container by Pat Patterson
• Preview and Snapshot Features in StreamSets Data Collector by Dash Desai

Apache NiFi

• Mastering Apache NiFi For Free by Tanmay Deshpande
• Ingesting Infura REST APIs to Access the Ethereum Network with Apache NiFi by Tim Spann

Presentations

• Designing Events First Microservices by Jonas Boner
• Data all over the place! How SQL and Apache Calcite bring sanity to streaming and spatial data by Julian Hyde
• Stream Processing and Real-Time Data Pipelines by Vladimir Schreiner
• The Rise of Streaming SQL and Evolution of Streaming Applications by Srinath Perera
• Event-time stream processing with Akka Streams by Nadav Wiener
• KSQLOpen-source streaming for Apache Kafka by Florent Ramiere
• Join Semantics in Kafka Streams by Himani Arora
• A Functional Approach to Architecture – Kafka & Kafka Streams by Kevin Mas Ruiz & Alex Gravanov
• Streaming ETL in Practice with PostgreSQL, Apache Kafka, and KSQL by Robin Moffatt

Videos

• Stream All the Things!! by Dean Wampler
• Deep Learning at Extreme Scale (in the Cloud) with the Apache Kafka Open Source Ecosystem by Kai Waehner
• DevOps at ING Analytics: data engineering, data operations by Taco Bakker

New Releases

• Debezium 0.8.0.CR1
• StreamSets Data Collector 3.3.1

Upcoming Events

• 10.7.2018 (Sunnyvale, US) – Apache Pulsar 2.0 – Functions, Connectors and Use cases (Meetup)
• 10.7.2018 (Austin, US) – Optimizing and indexing streams with Kafka for low latency queries (Meetup)
• 12.7.2018 (Bellevue, US) – Apache Kafka Meetup (Meetup)
• 12.7.2018 (Seattle, US) – Seattle Apache Kafka Meetup (Meetup)
• 18.7.2018 (Madrid, SP) – All about Kafka: origins, ecosystem and future directions (Meetup)
• 24.7.2018 (Berlin, DE) – Kafka, KSQL & IoT with Crate and Confluent (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 28.6.2018

This is the 119th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• We ❤️ syslogs: Real-time syslog processing with Apache Kafka and KSQL—Part 3: Enriching events with external data by Robin Moffatt
• 2018 Apache Kafka® Report by Confluent
• Introducing Confluent Hub by Tim Berglund
• Recommendations for Deploying Apache Kafka® on Kubernetes by Gwen Shapira
• How to write a Kotlin DSL – e.g. for Apache Kafka by Peter-Josef Meisch
• Stream Processing Made Easy With Confluent Cloud and KSQL by Yeva Byzek

Streamsets

• Streaming Extreme Data Made Simple with Kinetica and StreamSets by Pat Patterson
• Using StreamSets and MapR Together in Docker by Ian Downard

Apache Pulsar

• Distributed Messaging Framework Apache Pulsar 2.0 Supports Schema Registry and Topic Compaction by Srini Penchikala

Apache NiFi

• XML Record Writer in Apache NiFi 1.7.0 by Tim Spann
• Apache NiFi Registry 0.2.0  by Bryan Bende

Presentations

• KSQL – the Apache Kafka Streaming SQL Engine – Live Demo using Delimited, JSON and Avro data by Kai Waehner
• Streaming ETL in Practice with Oracle, Apache Kafka, and KSQL by Robin Moffatt
• Continuous Processing in Structured Streaming by Jose Torres
• How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning by Kai Waehner
• Structured Streaming Data Pipeline Using Kafka, Spark, and MapR-DB by Carol McDonald
• Visualizing Kafka Security by Vipin Rathor
• Autonomous workload rebalancing in Kafka by Indrajeet Kumar
• Apache Kafka : Monitoring vs Alerting by Ratish Ravindran
• Data Driven Enterprise with Apache Kafka by Bill McLaren
• Make streaming processing towards ANSI SQL by Shaxoxuan Wang

Videos

• Continuous Processing in Structured Streaming by Jose Torres
• Stateful & Reactive Stream Processing Applications Without a Database by Hans-Peter Grahsl
• Event Stream Processing into the ElasticStack by Henry Stamerjohann

New Releases

• Apache NiFi 1.7.0

Upcoming Events

• 28.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)
• 24.7.2018 (Berlin, DE) – Kafka, KSQL & IoT with Crate and Confluent (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 21.6.2018

This is the 118th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Democratizing Stream Processing with Apache Kafka and KSQL by Michael Noll
• Analysing Network Data with Apache Kafka, KSQL, and Elasticsearch by Robin Moffatt
• Siphon: Streaming data ingestion with Apache Kafka by Thomas Alex

Streamsets

• Streaming Extreme Data Made Simple with Kinetica and StreamSets by Mathew Hawkins
• Extract Data from Google Analytics using StreamSets Data Collector by Pat Patterson

Presentations

• Data Pipeline with Kafka by Mole Wong
• Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python by Tim Spann
• Interactive real-time dashboards on data streams using Kafka, Druid, and Superset by Nishant Bangarwa
• Streaming Transformations – Putting the T in Streaming ETL by Nick Dearden
• Navigating Idiosyncrasies of IoT Development by Marc Parisi

Videos

• Distributed stream processing for everyone with KSQL by Michael Noll
• Event stream processing using Kafka Streams by Fredrik Vraalsen
• Kapacitor Stream Processing by Nathanial Cook

New Releases

• Debezium 0.8.0-beta1

Upcoming Events

• 23.6.2018 (Athens, GR) – Athens Big Data Meetup 2018 (Meetup)
• 25.6.2018 (London, UK) – Streaming ETL with Kafka & Drones with APIs for Developers (Meetup)
• 28.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 13.6.2018

This is the 118th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Microservices, events, and upside-down databases by Sam Newman
• Top 10 Strategic Technology Trends for 2018: Event-Driven Model by David W. Cearley, Yefim V. Natis, Mike J. Walker, Brian Burke

Comparison

• A comparison between RabbitMQ and Apache Kafka by Piyush Sinha

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka Streams work allocation by Andy Bryant
• Getting started with Apache Kafka in Python by Adnan
• Big, Fast, Easy Data: Distributed Stream Processing for Everyone with KSQL, the Streaming SQL Engine for Apache Kafka by Michael Noll

Streamsets

• Franck Pachot Change Data Capture from Oracle with StreamSets Data Collector by Franck Pachot

Presentations

• Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL by Robin Moffatt
• Top 10 data engineering mistakes by Lars Albertsson
• Lessons learned from building an Apache Kafka Managed Service by Ben Slater
• A Deep Dive into Stateful Stream Processing in Structured Streaming by Tathagata Das
• Why is My Stream Processing Job Slow?  by Xavier Léauté & Gwen Shapira

Videos

• Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL by Robin Moffatt

New Releases

• Apache Storm 1.1.3
• Apache Storm 1.2.2

Upcoming Events

• 13.6.2018 (St. Louis, US) – Apache Heron (Meetup)
• 13.6.2018 (Budapest, HU) – Big Data Meetup – BudapestData edition (Meetup)
• 14.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)
• 19.6.2018 (online) – Stream Processing and IoT Leveraging Apache Kafka (Confluent Webinar)
• 20.6.2018 (online) – Part 03: Streaming Transformations – Putting the T in Streaming ETL (Confluent Webinar)
• 23.6.2018 (Athens, GR) – Athens Big Data Meetup 2018 (Meetup)
• 25.6.2018 (London, UK) – Streaming ETL with Kafka & Drones with APIs for Developers (Meetup)
• 28.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 6.6.2018

This is the 117th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka Monitoring With Burrow by Gaurav Garg
• How Debezium & Kafka Streams Can Help You Write CDC Solution by Neenad Ingole
• Apache Kafka, .NET and Protobuf Serialized Data by Matt Howlett
• Using the Lenses SQL Jdbc driver with Apache Spark by Stephen Samuel
• Data Governance and GDPR with Apache Kafka by Antonios Chalkiopoulos
• Use Apache Kafka to Transform a Batch Pipeline Into a Real-Time One, Part 1 by Stephane Maarek
• Use Apache Kafka to Transform a Batch Pipeline Into a Real-Time One, Part 2 by Stephane Maarek
• Building a Custom Kafka Connect Connector by Randhir Singh
• May Preview Release: Advancing KSQL and Schema Registry by Rohan Desei & Ran Ma

Apache Pulsar

• Announcing Apache Pulsar 2.0 by Matteo Merli
• The Pulsar Schema Registry by Dave Rusek
• Pulsar topic compaction by Ivan Kelly

Apache NiFi / HDF

• Apache NiFi + Apache OpenNLP With Organizations and Flow Files by Tim Spann

Presentations

• Processing IoT data with Apache Kafka, KSQL, and Machine Learning by Robin Moffatt
• Capital One Delivers Risk Insights in Real Time with Stream Processing by Jeff Sharpe and Ravi Dubey
• Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL by Robin Moffatt
• RealTime Recommendations @Netflix by Eliot Chow & Nitin Sharma

Videos

• Ask Confluent #1 | Kubernetes, Confluent Operator, Kafka, KSQL, and more by Gwen Shapira & Tim Berglund
• Kafka and The Service Mesh by Gwen Shapira
• RabbitMQ vs Kafka by Jack Vanlightly
• Building event sourced systems with Kafka Streams by Amitay Horwitz
• Embrace the Anarchy : Apache Kafka’s Role in Modern Data by Robin Moffatt
• Tech Time 6/4/2018 – Kafka Summit 2018 Roundup by Gianlo Faglio
• Building Event Driven MicroServices with Apache Kafka by Guido Schmutz
• Reactive stream processing with Akka Streams by Johan Andren
• Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark  by Tathagata Das
• The journey from queues to data pipeline streams by Shlomi Shemesh

Upcoming Events

• 6.6.2018 (online) – Part 02: Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL (Confluent Webinar)
• 12.6.2018 (Atlanta, US) – Crossing the Streams: Stream Processing for Spring developers (Meetup)
• 13.6.2018 (St. Louis, US) – Apache Heron (Meetup)
• 13.6.2018 (Budapest, HU) – Big Data Meetup – BudapestData edition (Meetup)
• 14.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)
• 20.6.2018 (online) – Part 03: Streaming Transformations – Putting the T in Streaming ETL (Confluent Webinar)
• 25.6.2018 (London, UK) – Streaming ETL with Kafka & Drones with APIs for Developers (Meetup)
• 28.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 30.5.2018

This is the 116th edition of my blog series blog series around Stream Processing and Analytics!

This week was crazy busy so I’m really late with this week’s edition.

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• How Debezium & Kafka Streams Can Help You Write CDC Solution by Neenad Ingole
• Querying Debezium Change Data Events With KSQL by Jiri Pechanec
• TWiGCP — “Managed Kafka on GCP, billing notifications, and Dialogflow and GKE enterprise enhancements” by Alexis
• Oracle Big Data Cloud, Event Hub and Analytics Cloud Data Lake Edition pt.2 : So Just What is OAC Data Lake Edition, and WTF Are “Data Engineers”? by Mark Rittman
• Using KSQL, Apache Kafka, a Raspberry Pi and a software defined radio to find the plane that wakes my cat by Simon Aubury
• Kafka Connect: Elasticsearch source connector by Dario Balinzo

StreamSets

• Ingest Game-Streaming Data from the Twitch API by Pat Patterson

Apache NiFi / HDF

• Parsing Web Pages for Images With Apache NiFi by Tim Spann
• 2 minutes to spare: Apache NiFi on Mac by icorda
• Building an IIoT system using Apache NiFi, MiNiFi, C2 Server, MQTT and Raspberry Pi by Abdelkrim Hadjidj

Presentations

• The Future of ETL Isn’t What It Used to Be by Gwen Shapira
• Functional Programming with Kafka Streams and Scala by Joan
• Unlocking the world of stream processing with KSQL, the streaming SQL engine for Apache Kafka by Michael Noll
• No More Silos: Integrating Databases and Apache Kafka by Robin Moffatt
• Embrace the Anarchy : Apache Kafka’s Role in Modern Data Architectures by Robin Moffatt
• Swisscom Firehose – Kafka as a Service @ Swisscom by David Jacot
• Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

Videos

• Stateful, Stateless and Serverless – Running Apache Kafka® on Kubernetes by Joe Beda & Gwen Shapira
• So You’ve Got Yourself a Kafka: Event-Powered Rails Services by Stella Cotton

New Releases

• Apache Flink 1.5.0
• Pulsar 2.0.0-rc1-incubating

Upcoming Events

• 6.6.2018 (online) – Part 02: Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL (Confluent Webinar)
• 13.6.2018 (St. Louis, US) – Apache Heron (Meetup)
• 14.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)
• 20.6.2018 (online) – Part 03: Streaming Transformations – Putting the T in Streaming ETL (Confluent Webinar)
• 28.6.2018 (Hamilton, US) – Predictive Maintenance with IoT, Apache NIFi, MiniFi and Blockchain (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!