Last Week in Stream Data Integration & Stream Analytics – 28.1.2019

This is the 142nd edition of my blog series blog series around Stream Data Integration and Stream Analytics! I have slightly changed the title to better reflect the content of by weekly blog series.

According to Gartner’s “Market Guide for Event Stream Processing”, Stream Data Integration and Stream Analytics are the two subsegments of the overall Stream Processing domain. I really like this definition, as it easily allows to categorize products either into one of these two segments.

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• 4 things CIOs should know about event-centric data strategy by Stephan Ewen
• Ten Things Companies Want from a Modern Data Architecture by Wayne Eckerson
• Choosing a Stream Processing framework. Spark Streaming or Kafka Streams or Alpakka Kafka? by Unmesh Joshi

Apache Kafka / Kafka Streams / Confluent Platform

• Confluent Raises a $125M Series D Funding Round by Jay Kreps
• Using Apache Kafka Command Line Tools with Confluent Cloud by Ricardo Ferreira
• Kafka is your escape hatch by Chris Riccomini
• How to embrace event-driven graph analytics using Neo4j and Apache Kafka by Ljubica Lazarevic
• What to consider for painless Apache Kafka integration by Adi Polak
• Install Confluent Kafka Platform onto Amazon EKS using Helm Chart by Haimo Zhang
• Apache Kafka Topics: Architecture and Partitions by Anjita Agrawal
• How to Prepare for the Confluent Certified Developer for Apache Kafka (CCDAK) exam by Stéphane Maarek
• Apache Kafka Load Testing Using JMeter by Rinu Gour

Apache Flink

• How Trackunit leverages Flink to process real-time data from industrial IoT devices by Lasse Nedergaard
• Oceanus: A one-stop platform for real time stream processing powered by Apache Flink by Big Data Team at Tencent

Apache NiFi

• Creating An Email Bot in Apache NiFi  by Timothy Spann

StreamSets

• Encrypt and Decrypt Data in Dataflow Pipelines by Pat Patterson
• How to Bulk Load Amazon RedShift from Relational Databases with StreamSets by Todd McGrath
• Snowflake + StreamSets: DataOps Accelerating Cloud Adoption by Mike Carley
• Data Collecting for Snowflake by Pat Patterson

Presentations

• IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT by Kai Waehner
• Lies Enterprise Architects Tell by Gwen Shapira
• Ingesting IoT data in Food Processing by Stefan Frehse
• Beyond the brokers – A Tour of the Kafka Ecosystem by Florent Ramiere
• Building event sourced systems with Kafka Streams by Amitay Horwitz
• How Criteo is managing one of the largest Kafka Infrastructure in Europe by Ricardo Paiva
• Blockchain and Apache NiFi by Timothy Spann
• Lightweight and scalable IoT Messaging with MQTT by Dominik Obermaier

New Videos

• Stream Processing Design Patterns by Andreas Markmann

New Releases

• Debezium 0.9.0.CR1 Released

Upcoming Events

• 29.1.2019 (New York, US) – Druid NYC Meetup @ Datadog (Meetup)
• 5.2.2019 (Vancouver, CA) – How to Structure your Data Pipeline and New Kafka Features You Might Not Know
• 5.2.2019 (Stockholm, SE) – Apache Kafka in the Cloud for Busy Software Engineers (Meetup)
• 12.2.2019 (Boston, US) – What is Apache Kafka and Wayfair’s Journey with Apache Kafka
• 13.2.2019 (Melbourne, AU) – Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications (Meetup)
• 13.2.2019 (Zagreb, HR) – Building real-time Kafka pipelines and Kafka security
• 14.2.2019 (online) – Bridge to Cloud: Using Apache Kafka to Migrate to GCP (Confluent Webinar)
• 19.2.2019 (Bristol, UK) – Apache Kafka and KSQL in Action! (Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)
• 5.3.2019 (Brentwood, US) – Tips and Tricks about Apache Kafka in the Cloud for Java Developers (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 21.1.2019

This is the 141st edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Scaling a Distributed Stream Processor in a Containerized Environment by Sarangan Janakan & Miyuru Dayarathna
• The Three Components of a Big Data Data Pipeline by Jesse Anderson

Apache Kafka / Kafka Streams / Confluent Platform

• Getting Your Feet Wet with Stream Processing – Part 2: Testing Your Streaming Application by Yeva Byzek
• Fun with Confluent Schema Registry Extensions by Robert Yokota
• Amazon Managed Streaming for Kafka (MSK) with Apache Spark on Qubole by Chandan Prakash and Vikram Agrawal
• The Need for a Stream Registry — Intro by Rene Parra
• Migrating Kafka’s Zookeeper With No Downtime by Toby Cole
• Effects of Batch Size, Acknowledgments, and Compression on Kafka Throughput by Stuart Eudaly
• Apache Kafka & KSQL & TensorFlow for Data Scientists via Python & Jupyter Notebook by Kai Waehner

Apache Flink

• How to manage your RocksDB memory size in Apache Flink by Apache Flink
• 4 characteristics of Timers in Apache Flink to keep in mind by Gordon Tai

Spark Streaming

• Building a Data Pipeline with Kafka, Spark Streaming and Cassandra by Kumar Chandrakant

Apache NiFi

• Monitoring Nifi with Data Dog by NiFi.rocks
• Developing a Custom Apache Nifi Reporting Task by Phillip Grenier
• Reading Kerberos KDC Logs and Parsing Them for Events via Apache NiFi 1.8 – Part 1 by Timothy Spann

StreamSets

• StreamSets Helps You Build Your Data Warehouse in the Cloud by Sean Anderson
• Building a Real-Time Bike-Share Data Pipeline with StreamSets, Kafka and MapD by Jowanza Joseph
• Working with Streamsets to ingest and transform real-time data with a Cloudera cluster by Clearpeaks

Presentations

• Crossing the Streams: Rethinking Stream Processing with Kafka Streams and KSQL by Viktor Gamov
• ATM Fraud Detection with Apache Kafka and KSQL by Robin Moffatt
• Deep Learning at Extreme Scale in the Cloud with Apache Kafka and TensorFlow by Kai Waehner
• Driving Business Transformation with Real-Time Analytics Using Apache Kafka and KSQL by Nick Dearden & John Thuma & Thomas Clarke

New Videos

• Stream Processing at Lyft with Flink and Beam by Micah Wylde & Sherin Thomas

Upcoming Events

• 22.1.2019 (Menlo Park, US) – CDC and Building a Streaming Analytics Stack with Kafka and Druid (Meetup)
• 23.1.2019 (Washington, US) – Apache NiFi in Washington (Meetup)
• 24.1.2019 (New York, US) – Unifying Messaging, Queuing, Streaming & Light Weight Compute with Apache Pulsar (Meetup)
• 29.1.2019 (New York, US) – Druid NYC Meetup @ Datadog (Meetup)
• 5.2.2019 (Vancouver, CA) – How to Structure your Data Pipeline and New Kafka Features You Might Not Know
• 13.2.2019 (Melbourne, AU) – Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications (Meetup)
• 19.2.2019 (Bristol, UK) – Apache Kafka and KSQL in Action! (Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 14.1.2019

This is the 140th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Apache Kafka / Kafka Streams / Confluent Platform

• Exposing Remote JMX in your Kafka Setup by Jendrik Poloczek
• How to Run Kafka on Openshift, the Enterprise Kubernetes, With AMQ Streams by
• Stream data with Apache Kafka into the IBM Db2 Event Store by Loic Julien
• Updated Apache Kafka Benchmarking by Ben Slater

Apache Flink

• Towards achieving a 10x compile-time improvement in a Flink codebase by Aris Koliopoulos
• Alibaba Acquires Apache Flink Backer data Artisans by Alex Woodie

Spark Streaming

• Apache Spark 2.4.0 features – watermark configuration by Bartosz Konieczny

StreamSets

• Ingesting Data from Relational Databases to Cassandra with StreamSets by Todd McGrath

Presentations

• Streams, Tables, and Time in KSQL by Matthias Sax
• Big data processing with PubSub, Dataflow, and BigQuery by Thuyen Ho
• Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL by Sönke Liebau
• The Open Source SQL Streaming Engine for Apache Kafka by Kai Waehner

New Releases

• StreamSets Data Collector 3.7.1

Upcoming Events

• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 22.1.2019 (Menlo Park, US) – CDC and Building a Streaming Analytics Stack with Kafka and Druid (Meetup)
• 23.1.2019 (Washington, US) – Apache NiFi in Washington (Meetup)
• 24.1.2019 (New York, US) – Unifying Messaging, Queuing, Streaming & Light Weight Compute with Apache Pulsar (Meetup)
• 29.1.2019 (New York, US) – Druid NYC Meetup @ Datadog (Meetup)
• 5.2.2019 (Vancouver, CA) – How to Structure your Data Pipeline and New Kafka Features You Might Not Know
• 13.2.2019 (Melbourne, AU) – Using Apache Cassandra and Apache Kafka to Scale Next Gen Applications (Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 9.1.2019

This is the 139th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• With Real-Time, What Data Center Management Will Look Like in 2019 by Kevin McNamara
• Data Insights Engine @ MakeMyTrip by Aditya Banerjee
• From Real-Time to the Edge: The Top IoT Trends for the Coming Year by Michael Kanellos

Apache Kafka / Kafka Streams / Confluent Platform

• Building Secure and Governed Microservices With Kafka Streams by George Vetticaden
• Easy Ways to Generate Test Data in Kafka by Yeva Byzek
• Kafkawize by Muralidhar Basani
• Using GridGain With Kafka Connector by Akmal Chaudhri
• Storm-Kafka Integration With Configurations and Code by Rinu Gour
• Kafka Monthly Digest: December and 2018 in Review by Mickael Maison
• Kappa architecture at NTT: Building a streaming analytics stack with Druid and Kafka by Paolo Lucente
• Building microservices with kafka streams by Andreas Schroeder
• Getting Your Feet Wet with Stream Processing – Part 1: Tutorial for Developing Streaming Applications by Yeva Byzek

Apache Flink

• German startup data Artisans sells to Alibaba for €90 million by Andrii Degeler
• Alibaba’s data Artisans acquisition breathes new life into Apache Flink by Tony Baer

Presentations

• Apache flink 1.7 and Beyond by Till Rohrmann

Videos

• Who’s Tweeting about #DataSciCon by Viktor Gamov
• Distributed systems for stream processing: Apache Kafka and Spark Streaming by Alena Hall

New Releases

• Wallaroo 0.6.1
• StormCrawler 1.13

Upcoming Events

• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)
• 9.1.2019 (Munich, GE) – Building Microservices with Kafka Streams: Beyond Kafka-for-Messaging (Meetup)
• 10.1.2019 (Leeds, UK) – Running a cluster in the Cloud and ATM Fraud detection with Kafka and KSQL (Meetup)
• 10.1.2019 (Berlin, GE) – Streams, Tables, and Time in KSQL (Meetup)
• 10.1.2019 (Frankfurt, GE) – Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL (Meetup)
• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 23.1.2019 (Washington, US) – Apache NiFi in Washington (Meetup)
• 29.1.2019 (New York, US) – Druid NYC Meetup @ Datadog (Meetup)
• 5.2.2019 (Vancouver, CA) – How to Structure your Data Pipeline and New Kafka Features You Might Not Know
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)d

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 3.1.2019

Happy New Year!

This is the 138th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Event Sourcing with AWS Lambda by Paul Bourdel
• Apps Depend on Real-Time Streaming Data, Here’s How to Manage Them by Shivnath Babu

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka Streams: Is it the Right Stream Processing Engine for You? by
• Anomalia Machina 5 – Application Monitoring with Prometheus: Massively Scalable Anomaly Detection with Apache Kafka and Cassandra by Paul Brebner
• Interpreting Kafka’s Exactly-Once Semantics by Rahul Agarwal

Presentations

• Real-time Analytics with Apache Kafka by Mohammad Mehdi Yalollahi & Alihossein Shahabi

Videos

• The Magical Group Coordination Protocol of Apache Kafka by Gwen Shapira
• Unlimited Kafka Messages by Maor Mordehay
• Fast Data apps with Alpakka Kafka connector and Akka Streams by Sean Glover
• Stream Processing Design Patterns by Andreas Markmann

New Releases

• Apache Pulsar 2.2.1

Upcoming Events

• 9.1.2019 (Munich, GE) – Apache Flink Meetup (Meetup)
• 10.1.2019 (Leeds, UK) – Running a cluster in the Cloud and ATM Fraud detection with Kafka and KSQL (Meetup)
• 10.1.2019 (Berlin, GE) – Streams, Tables, and Time in KSQL (Meetup)
• 10.1.2019 (Frankfurt, GE) – Serving the Real-Time Data Needs of an Airport with Kafka Streams and KSQL (Meetup)
• 17.1.2019 (New York, US) – Kafka: Crossing the Streams + Data Ops ( Meetup)
• 28.2.2019 (Bern, Switzerland) – 1. Meetup zu Kafka & Strimzi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!