Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 16:22 on May 23, 2017
Tags: aws ( 3 ), flink ( 92 ), IoT ( 10 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics – 22.5.2017

This is the 67th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Anomaly Detection in Telecommunications Using Complex Streaming Data by Ted Dunning
The Future of Event Stream Processing by Michael Drogalis
How to avoid downtime and disruption when moving data by David Richards
The Anatomy of an IoT Solution by Hans de Visser

Apache Kafka / Kafka Streams / Confluent Platform

Reading Data Securely From Apache Kafka to Apache Spark by Mark Grover
Kafka Topic Design Checklist by Jesse Anderson
Building Real-Time data pipelines with Kafka Connect by Slim Baltagi
Trialing Single Message Transformations in Kafka Connect by Mike Donovan

Spark Streaming

Taking Apache Spark’s Structured Structured Streaming to Production by Bill Chambers & Michael Armbrust
Running Streaming Jobs Once a Day For 10x Cost Savings by Burak Yavuz & Tyson Condie

Apache Flink

A Year of Blink at Alibaba: Apache Flink in Large Scale Production by Xiaowei Jiang

AWS Kinesis

Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight by Karan Desai

Apache NiFi

Monitoring NiFi – Scripted Reporting Task by Pierre Villard
Monitoring NiFi – Ambari & Grafana by Pierre Villard
Monitoring NiFi – Workflow SLA by Pierre Villard
Monitoring NiFi – Site2Site reporting tasks by Pierre Villard
Monitoring NiFi – Logback configuration by Pierre Villard
Record-oriented Data with NiFi by Mark Payne

New Presentations

Real Time Analytics – Stream Processing by Mahesh Madushanka
Streaming Analytics for IoT-Oriented Applications by Adrian Bowles
Geo-Distributed Big Data and Analytics by Ted Dunning
Top 5 mistakes when writing Streaming applications by Mark Grover & Ted Malaska
Fast Cars, Big Data How Streaming can help Formula 1 by Carol McDonald
Stateful streaming data pipelines by Chandni Singh & Timothy Farkas
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform by Chrix Finne & Bob Lehmann
Microservices interaction at scale using Apache Kafka by Ivan Ursul
Capture the Streams of Database Changes by Randall Hauch
Stream Processing Metamorphosis – A Kafka’s tale by João Vazão Vasques
Event processing with Apache Storm by Nuno Caneco

New Podcasts

Apache Kafka, Streaming Data Integration and Schema Registry by Mark Rittman with Gwen Shapira

Upcoming Events

22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
30.5.2017 (Frankfurt, GE) – First Kafka Meetup (Meetup)
7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:33 on May 17, 2017
Tags: beam ( 25 ), confluent ( 8 ), flink ( 92 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics – 17.5.2017

This is the 66th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Fast Cars, Big Data – How Streaming Data Can Help Formula 1 by Carol McDonald
Thoughts on Building a Successful Streaming Analytics Platform by George Vetticaden
A Digital Reference Architecture for the Industrial Internet Of Things (IIoT) by Vamsi Chemitiganti
Introducing DevHigh5 Project Pravega: Storage reimagined for a streaming world by Salvatore DeSimone

Apache Kafka / Kafka Streams / Confluent Platform

Kafka Summit New York: The Rise of the Streaming Platform by Clarke Patterson
Reading data securely from Apache Kafka to Apache Spark by Mark Grover
Installing and Running Kafka on an AWS Instance by Sahil Sawhney
Managing Spark and Kafka Pipelines by Larry Murdock
Getting Started with the Kafka Streams API using Confluent Docker Images by Michael Noll
Kafka Consumer Architecture – Consumer Groups and subscriptions by Cloudurable
The new version of Kafka Topics UI is now available! by Christina Daskalaki
Startup Confluent makes Kafka cloud play to ease streaming deployment by Jack Vaughan
Securing Apache Kafka with Kerberos by Colm O hEigeartaigh
Apache Kafka – Introduction & Installation by Rudra Narayan Garnaik
Visualizing the flow of data as it moves through a Kafka cluster by James Cheng

Spark Streaming

How to Get Started with Spark Streaming and MapR Streams Using the Kafka API by Carol McDonald

Apache Beam

Apache Beam publishes the first stable release by Davor Bonaci

Apache Flink

Introducing Official Docker Images for Apache Flink by Patrick Lucas & Ismaël Mejía

StreamSets

Visualizing and Analyzing Salesforce Data with Neo4j by Pat Patterson
Calling External Libraries from the JavaScript Evaluator by Pat Patterson

Apache NiFi

Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster by Roger Carhuatocto
Apache Nifi Sensitive Value Encryption by Kya
Record based processors in Apache NiFi 1.2 by Haimo Liu

New Presentations

Intro to Apache Kafka by Jason Hubbard
Stream processing using Kafka by Himani Arora & Prabhat Kashyap
Rethinking Services With Stateful Streams by Ben Stopford
Connect S3 with Kafka leveraging Akka Streams by Seiya Mizuno
Stream Processing with MicroProfile and Apache Kafka by Matthias Wessendorf
Real-time Data Pipelines with SAP and Apache Kafka by Carole Gunst
Processing IoT Data with Apache Kafka by Confluent
Functional streams with Kafka – A comparison between Akka-streams and FS2 by Luis Reis & Rui Batista
Open Blueprint for Real-Time Analytics with In-Stream Processing (ISP) by Victoria Livschitz
Dataservices – Processing Big Data The Microservice Way by Josef Adersberger

New Video

Scalable Stream Processing: A Survey of Storm, Samza, Spark and Flink by Felix Gessert
Improve your App UX with one model processing insights from batch and streams by Robert Kubis
Apache Ignite: Real Time Processing of IoT Generated Streaming Data by Denis Magda
Rethinking Microservices with Stateful Streams by Ben Stopford
Fast Cars, Big Data – How Streaming Can Help Formula 1 by Tugdual Grall

New Releases

Apache Beam 2.0.0

Upcoming Events

17.5.2017 (Mountain View, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
18.5.2017 (online) – Conquering Data in Motion with a Modern Ingestion Platform (Cloudera Webinar)
22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:42 on May 10, 2017

Last week in Stream Processing & Analytics – 10.5.2017

This is the 65th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Four Keys to Enterprise IoT Architecture by Joe McKendrick
Part 1: Hortonworks Thoughts on Building a Successful Streaming Analytics Platform by George Vetticaden
Real-Time Data Analysis: Where the Jobs Are by Joe McKendrick

Comparison

Apache Kafka vs RabbitMQ: Fit For Purpose / Decision Tree by Slim Baltagi

Apache Kafka / Kafka Streams / Confluent Platform

Pepper-Box Kafka Load Generator by Satish Bhor
Watermarks, Tables, Event Time, and the Dataflow Model by Eno Thereska
2017 Apache Kafka Report by Confluent
The 2017 Apache Kafka Survey: Streaming Data on the Rise by Luanne Dauber
Kafka ‘Massively Simplifies’ Data Infrastructure, Report Says by Alex Woodie
Robust Usage of Apache Kafka on Heroku by Heroku
The Many Meanings of Event-Driven Architecture: Kafka Edition by Jesse Anderson
Confluent Now Offers Apache Kafka as a Service with Confluent Cloud by Sue Walsh
Announcing Confluent Cloud: Apache Kafka as a Service by Neha Narkhede
Joins on stateful stream processing using Kafka Streams’ KTables and GlobalKTables by Daniel Lebrero
Optimizing your Kafka Deployment by Yeva Byzek
The Continued Rise of Apache Kafka by RedMonk
Apache Kafka Startup Confluent Launches A Cloud Service With Bold Words For AWS by Alex Konrad
Apache Kafka – Usage Patterns by Gagan

Spark Streaming

Event-time Aggregation and Watermarking in Apache Spark’s Structured Streaming by Tathagata Das

Apache Flink

Flink Forward San Francisco 2017 Recap: On the State of Stream Processing with Apache Flink by dataArtisans
Continuous Queries on Dynamic Table by Fabian Hueske, Shaoxuan Wang, and Xiaowei Jiang

StreamSets

StreamSets Named a 2017 Cool Vendor in Data Management by Gartner
StreamSets benefits from focus on continuous data integration by Matt Aslett

Heron

Heron Going Exactly-Once by Sanjeev Kulkarni

Apache NiFi

My first experiences with Apache NiFi by Marcel-Jan Krijgsman
Speeding up Apache NiFi compilation by Richards Technotes
A New Era of Open-Source Streaming by Tim Spann
Apache NiFi 1.2.0 – Component Versioning by Bryan Bende
IoT: Ingesting GPS Data from Raspberry PI Zero Wireless with USB GPS Device by Timothy Spann

New Presentations

Big Data Architectural Patterns and Best Practices on AWS by Amazon AWS
Red Hat JBoss AMQ and Apache Kafka : which to use ? by Christian Posta & Paolo Patierno
What is apache Kafka? by Kenny Gorman
Kafka Tutorial – introduction to the Kafka streaming platform by Cloudurable
Azure real-time analytics and kappa architecture with Kafka and Cassandra clusters by Vitalii Bondarenko
Challenges of a multi tenant kafka service by Thomas Alex
From Three Nines to Five Nines – A Kafka Journey by Allen Wang
Pinot: Near Realtime Analytics @ Uber by Xiang Fu
Stream Processing with SMACK by Tamás István Ujj
Oracle Stream Analytics – Industry Examples by Prabhu Thukkaram
Apache Kafka – Scalable Message-Processing and more ! by Guido Schmutz
Introducing Exactly Once Semantics To Apache Kafka by Jason Gustafson, Guozhang Wang, Sriram Subramaniam & Apurva Mehta
Introduction to Kafka Streams with a Real-Life Example by Alexis Seigneurin
Cloud Native Data Streaming Microservices With Spring Cloud and Kafka by Marius Bogoevici

New Video

Visual VM with Kafka by Patrick Bailey
Comparison of Apex and Spark Streaming by Bhupesh Chawada

New Podcast

Data Intensive Appications with Martin Kleppmann by SE Daily

New Releases

Upcoming Events

16.5.2017 (online) – Geo-Distributed Big Data and Analytics (MapR Webinar)
16.5.2017 (Copenhagen, DK) – Meet Gwen Shapira, the Master of Apache Kafka! (Meetup)
18.5.2017 (online) – Conquering Data in Motion with a Modern Ingestion Platform (Cloudera Webinar)
23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:54 on May 1, 2017
Tags: kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics – 1.5.2017

This is the 64th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

A Big Data Reference Architecture for IoT by Tim Spann & Vamsi Chemitiganti

Apache Kafka / Kafka Streams / Confluent Platform

Hello Kafka Streams by Vishnu Viswanath
Test Strategy for Samza/Kafka Services by Devi Sridharan
Understanding When to use RabbitMQ or Apache Kafka by Pieter Humphrey
How Apache Kafka Works and Why by Stefan Groschupf

StreamSets

Create a Custom Expression Language Function for StreamSets Data Collector by Pat Patterson

Apache NiFi

Ingesting GPS Data From Onion Omega2 Devices with Apache NiFi by Tim Spann
Ingesting iBeacon Data via BLE to MQTT WiFi Gateways via Apache NiFi by Tim Spann
Advanced Apache NiFi Flow Techniques by Tim Spann

Azure Stream Analytics

Data driven troubleshooting of Azure Stream Analytics jobs by Samartha Chandrashekar

New Presentations

What will be new in Apache NiFi 1.2.0 by Koji Kawamura
Event Driven Services Part 2: Building Event-Driven Services with Apache Kafka 23 views by Ben Stopford
Putting the Micro into Microservices with Stateful Stream Processing by Ben Stopford
Schema Registry & Stream Analytics Manager by Harsha Chintalapani
Using Akka Streams For Real Time Decision Making by Dustin Lyons
Twitter Sentiment Analysis Example by Edureka
Spark Tutorial For Beginners by Edureka
Bridging Batch and Streaming Data Integration by Shirshanka Da
Apache Storm by Nguyen Quang

New Video

Introduction to Apache Kafka by James Ward
Using Akka Streams For Real Time Decision Making by Dustin Lyons

Upcoming Events

6.5.2017 (Fremont, US) – Hands-on workshop in Real Time Stream Processing and Analytics (Meetup)
9.5.2017 (London, UK) – Reactive in Practice (CakeSolutions Event)
16.5.2017 (online) – Geo-Distributed Big Data and Analytics (MapR Webinar)
23.4.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!