Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 07:43 on November 22, 2016
Tags: apex ( 18 ), confluent ( 8 ), flink ( 92 ), hdf ( 20 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

2016

This is the 41st installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Apache BigData Europe Conference Summary by Marcin Kuthand
Small Scale Stream Processing Kata (Part 1): Thread Pools by Tomasz Nurkiewicz
From Big Data to Fast Data in Four Weeks or How Reactive Programming is Changing the World – Part 1 by Michael Zeltser
From Big Data to Fast Data in Four Weeks or How Reactive Programming is Changing the World – Part 2 by Michael Zeltser
Understanding Apache Ecosystem: stream processing by Lorenzo Toscano
A fast data architecture whizzes by traditional data management tools by Paul Korzeniowski

Apache Kafka

Announcing Confluent 3.1 with Apache Kafka 0.10.1.0 by Sriram Subramanian
Unit Testing Kafka Consumers by Jesse Anderson
Confluent Releases 3.1 with Apache Kafka by Sue Walsh
Partner Development Guide for Kafka Connect by Confluent
Spring Kafka Producer/Consumer sample by Biju Kunjummen
Apache Kafka Streams and JCache with BlazingCache by Enrico Olivelli
39th meetup on Apache Kafka and Apache Ignite by Wim van Leuven
Oracle Service Bus: A quickstart for the Kafka transport by Maarten Smeets

Apache Storm

Carrier Payments Big Data Pipeline using Apache Storm by Mansi Mahendru

Apache Flink

Apache Flink Shell Command Tutorial – A Quickstart For beginners by Data Flair

Apache NiFi / Hortonworks Data Flow (HDF)

Routing Logs Through Apache NiFi to Phoenix, HDFS and Offsite to Sumologic for Log Analytics by Timothy Spann

New Presentations

Reliable Data Intestion in BigData / IoT by Guido Schmutz
Introduction to Streaming Analytics by Guido Schmutz
Streaming operational data with Kafka by David Tucker & David Ostrovsky
Demystifying Stream Processing with Apache Kafka by Neha Narkhede
Design Patterns for working with Fast Data in Kafka by Ian Downard
Architecture Big Data open source S.M.A.C.K by Julien Anguenot
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And Kafka by Akara Sucharitakul
Powering Microservices with Docker, Kubernetes, and Kafka by Andrew Morgan
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi by Timothy Spann
HDF Powered by Apache NiFi Introduction by Milind Pandit
Next Gen Big Data Analytics with Apache Apex by Thomas Weise
Introduction to Apache Apex by Thomas Weise
Building Streaming Applications with Apache Apex by Chinmay Kolhatkar & Thomas Weise
Twitter heron – Evolution or Revolution? by Grzegorz Kolpuc
Data Microservices with Spring Cloud Stream, Task, and Data Flow by Toshiaki Maki
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Services by Kai Wähner
Stream Application Development with Apache Kafka by Matthias J. Sax
RSEP-QL: A Query Model to Capture Event Pattern Matching in RDF Stream Processing Languages by Daniele Dell’Aglio

New Videos

Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And Kafka by Akara Sucharitakul
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Services by Kai Wähner
Demystifying Stream Processing with Apache Kafka by Neha Narkhede

New Podcasts

Drill to Detail – Streamsets, Data-in-Motion and Data Drift by Mark Ritmann with Pat Patterson

New Releases

Upcoming Events

11/22/2016 (online) – Digital Transformation to Data-Driven Real-Time Enterprise (Cloudera Webinar)
11/22/2016 (Austin, US) – Sensor Data Analytics Acceleration with Apache Edgent (Meetup)
11/24/2016 (online) – Real-Time Data Ingestion and Streaming Analytics with Oracle and Hortonworks (Hortonworks Webinar)
11/29/2016 (Berlin, GE) – Apache Flink Meetup @ data Artisans (Meetup)
11/30/2016 (Tel Aviv, IL) – Introducing StreamSets – Creating Scalable and Resilient Data Pipelines (DataZone Event)
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/5/2016 (New York, US) – Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi (Meetup)
12/6/2016 (Lisbon, PT) – A Deep-dive into Structured Streaming (Meetup)
12/8/2016 (New York, US) – Streaming Analytics in a Flash – presented by Cask Data (Meetup)
12/8/2016 (Singapore, SG) – Stream Processing and TensorFlow 101 (Google Meetup)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:51 on November 14, 2016
Tags: flink ( 92 ), hdf ( 20 ), kafka ( 248 ), nifi ( 238 ), samza ( 8 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 11/14/2016

This is the 40th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

In 2017 IoT will drive continuous big data operations by Rick Bilodeau
Real-Time Election Predictions: VoteCastr’s Experiment by Sue Walsh
The 3 Keys to Digital Transformation by Jack Norris
North America Streaming Analytics Market (2016-2022) – Number of Connected Devices will Grow to 50 Billion – Research and Markets by Yahoo Finance

Comparison

15 Questions and Answers from Apache NiFi, Kafka & Storm: Better Together by Haimo Liu & Bryan Bende

Apache Apex

Throughput, Latency, and Yahoo! Performance Benchmarks. Is there a winner? by DataTorrent

Apache Kafka

Unit Testing Kafka by Jesse Anderson
Movin’ On Up: Apache Kafka Gets a Promotion to the Enterprise by Christy Wilson

Spark Streaming

Structured Streaming In Apache Spark A new high-level API for streaming by Matei Zaharia, Tathagata Das, Michael Armbrust & Reynold Xin

Apache Flink

Processing Rabbitmq’s Stream with “Apache Flink” by Shekhar Singh

StreamSets

More Than One Third of the Fortune 100 Have Downloaded StreamSets Data Collector by Rick Bilodeau
Ingesting Streaming Data from JMS into HDFS and Solr using StreamSets by Rick Bilodeau

Apache NiFi / Hortonworks Data Flow (HDF)

Edge Intelligence for IoT with Apache MiNiFi by Anna Yong & Joe Percivall
How to Harness the Power of Data-in Motion: Webinar Series by Anna Yong

New Presentations

Apache Samza Past, Present and Future by Kartik Paramasivam
Apache Kafka, and the Rise of Stream Processing by Guozhang Wang
Apache Nifi, Kafka and Storm Better Together by Bryan Bende & Haimo Liu
Data Streaming with Apache Kafka & MongoDB by Andrew Morgan & David Tucker
Data Loss and Duplication in Kafka by Jayesh Thakrar
Apache Kafka – Stream Processing made easy by Daan Gerits
What’s New in Spark 2? by Eyal Ben Ivri
Real-Time Analytics with MemSQL and Spark by Neil Dahlke
Stream processing by Ashic Mahtab
Developing a Streaming Application Using Apache Apex by Pramod Immaneni
Building Streaming Applications with Apache Apex by Chinmay Kolhatkar & Thomas Weise
Scaling up Near real-time Analytics @ Uber and LinkedIn by Chinmay Soman & Yi Pan
Analytics on AWS:Structured, Unstructured and Streaming by Russell Nash
Apache NiFi Crash Course Intro by Rafael Coss

New Videos

Stream processing by Ashic Mahtab
Apache Kafka, Storm and NiFi: Better Together by Bryan Bende & Haimo Liu
Distributed Commit Logs with Apache Kafka by James Ward
Apache Samza Past, Present and Future by Kartik Paramasivam
Google Dataflow: The New Open Model for Batch and Stream Processing by Sara Robinson & Robert Kubis
Cloud Native Streaming and Event-Driven Microservices by Marius Bogoevici
Reactive Streams principles applied in Akka Streams by Eric Loots
Oracle Streams Analytics platform – Is it time to reverse traditional analytics? Oracle Partner Webcast

New Releases

Spark 1.6.3

Upcoming Events

11/20/2016 (Mountain View, US) – November Apache Kafka Meetup (Meetup)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
11/17/2016 (Mountain View, US) – Pulsar – Distributed pub-sub messaging & Apache NiFi in action (Meetup)
11/21/2016 (Columbia, US) – Using Apache NiFi at Capital One & The Apache Way (Meetup)
11/22/2016 (online) – Digital Transformation to Data-Driven Real-Time Enterprise (Cloudera Webinar)
11/29/2016 (Berlin, GE) – Apache Flink Meetup @ data Artisans (Meetup)
11/30/2016 (Tel Aviv, IL) – Introducing StreamSets – Creating Scalable and Resilient Data Pipelines (DataZone Event)
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/5/2016 (New York, US) – Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi (Meetup)
12/6/2016 (Lisbon, PT) – A Deep-dive into Structured Streaming (Meetup)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:25 on November 7, 2016
Tags: flink ( 92 ), hdf ( 20 ), ignite, kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 11/7/2016

This is the 39th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Lambda Architecture with Apache Spark by Taras Matyashovskyy

Comparison

Comparison of Open Source IoT Integration Frameworks by Kai Wähner

Apache Kafka

Confluent Contributions to the Apache Kafka™ Client Ecosystem by Jay Kreps
Apache Kafka foundation of modern data stream processing by Jakub Stransky
Highlights in the Apache Kafka and Stream Processing Community | November 2016 by Gwen Shapira

Apache Storm / Heron

Apache Storm Resource Contention Resolution Strategies by Constantin Stanca

Spark Streaming

OpenShift Commons Big Data SIG #2: Running Apache Spark Natively on Kubernetes with OpenShift by Diane Mueller

Apache Flink

Now run real-time stream processing at scale with Apache Flink on Amazon EMR by Amazon AWS
Use Apache Flink on Amazon EMR by Craig Foster

StreamSets

Tame unruly big data flows with StreamSets by Arvind Pabhakar

Apache NiFi / Hortonworks Data Flow (HDF)

Using Apache NiFi to ingest SNMP tables into Avro by Michal Klempa
Easy custom logging of diverse sources in mere seconds by Greg Keys
Running Apache Pig Scripts from Apache NiFi and Storing the Results in HDFS by Timothy Spann

Apache Ignite

Fire up big data processing with Apache Ignite by InfoWorld

New Presentations

Better Together: Fast Data with Apache Spark and Apache Ignite by Christos Erotocritou
Introduction to Kafka Cruise Control by Jiangjie Qin & Efe Gencer
Data Pipelines with Apache Kafka by Ben Stopford
How We Built an Event Time Merge of Two Kafka Streams with Spark Streaming by Sebastian Schröder
Streaming, Database & Distributed Systems Bridging the Divide by Ben Stopford
React Fast by Processing Streaming Data in Real-Time by Kobi Biton
Lambda Architecture with Spark in the IoT by Bas Geerdink
Online Learning with Structured Streaming by Ram Sriharsha & Vlad Feinberg
Sessionization with Spark streaming by Ramunas Urbonas
Elastic Streaming – Spark Streaming + Dynamic Provisioning + Dynamic Allocation by Neelesh Shastry & Shaun Klopfstein
Paddling Up the Stream by Miklos Christine
Spark Streaming at Bing Scale by Kaarthik Sivashanmugam
Online Learning with Structured Streaming by Ram Sriharsha & Vlad Feinberg
Fast Data: Selecting The Right Streaming Technologies For Data Sets That Never End by Dean Wampler
Building Real-Time Data Pipelines with Kafka, Spark, and MemSQL by Gary Orenstein
Edge Intelligence IoT MiniFi by Joe Percivall & Anna Yong
Kafka Streams: The Stream Processing Engine of Apache Kafka by Eno Thereska
IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets) by Kay Wähner
Streaming Outlier Analysis for Fun and Scalability by Casey Stella

New Videos

Online Learning with Structured Streaming by Ram Sriharsha & Vlad Feinberg
Finding Outliers in Streaming Data: A Scalable Approach by Casey Stella
Elastic Streaming – Spark Streaming + Dynamic Provisioning + Dynamic Allocation by Neelesh Shastry & Shaun Klopfstein
Spark Streaming at Bing Scale by Kaarthik Sivashanmugam
Building Data Pipelines with Spark and StreamSets by Pat Patterson
Paddling Up the Stream by Miklos Christine
Lambda Architecture with Spark in the IoT by Bas Geerdink
How We Built an Event Time Merge of Two Kafka Streams with Spark Streaming by Sebastian Schröder
Better Together: Fast Data with Apache Spark and Apache Ignite by Christos Erotocritou
IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets) by Kay Wähner
Running Apache Spark Natively on Kubernetes with OpenShift by Diane Mueller

Upcoming Events

11/8/2016 (online) – Apache Kafka, Storm and Apache NiFi: Better Together (Hortonworks Webinar Data-in-Motion Series)
11/8/2016 (online) – Data Streaming with Apache Kafka & MongoDB (MongoDB Webinar)
11/9/2016 (Santa Clara, US) – Analyze Traffic Patterns with Apache NiFi (Meetup)
11/9/2016 (Mountain View, US) – Realtime Anomaly Detection w/ Flink at Mux, and Connecting Flink to the World (Meetup)
11/9/2016 (Palo Alto) – Big Data App Meetup 11/09 (Meetup)
11/20/2016 (Mountain View, US) – November Apache Kafka Meetup (Meetup)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
11/21/2016 (Columbia, US) – Using Apache NiFi at Capital One & The Apache Way (Meetup)
11/30/2016 (Tel Aviv, IL) – Introducing StreamSets – Creating Scalable and Resilient Data Pipelines (DataZone Event)
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!