Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:51 on November 14, 2016
Tags: flink ( 92 ), hdf ( 20 ), kafka ( 248 ), nifi ( 238 ), samza, spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

2016

This is the 40th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

In 2017 IoT will drive continuous big data operations by Rick Bilodeau
Real-Time Election Predictions: VoteCastr’s Experiment by Sue Walsh
The 3 Keys to Digital Transformation by Jack Norris
North America Streaming Analytics Market (2016-2022) – Number of Connected Devices will Grow to 50 Billion – Research and Markets by Yahoo Finance

Comparison

15 Questions and Answers from Apache NiFi, Kafka & Storm: Better Together by Haimo Liu & Bryan Bende

Apache Apex

Throughput, Latency, and Yahoo! Performance Benchmarks. Is there a winner? by DataTorrent

Apache Kafka

Unit Testing Kafka by Jesse Anderson
Movin’ On Up: Apache Kafka Gets a Promotion to the Enterprise by Christy Wilson

Spark Streaming

Structured Streaming In Apache Spark A new high-level API for streaming by Matei Zaharia, Tathagata Das, Michael Armbrust & Reynold Xin

Apache Flink

Processing Rabbitmq’s Stream with “Apache Flink” by Shekhar Singh

StreamSets

More Than One Third of the Fortune 100 Have Downloaded StreamSets Data Collector by Rick Bilodeau
Ingesting Streaming Data from JMS into HDFS and Solr using StreamSets by Rick Bilodeau

Apache NiFi / Hortonworks Data Flow (HDF)

Edge Intelligence for IoT with Apache MiNiFi by Anna Yong & Joe Percivall
How to Harness the Power of Data-in Motion: Webinar Series by Anna Yong

New Presentations

Apache Samza Past, Present and Future by Kartik Paramasivam
Apache Kafka, and the Rise of Stream Processing by Guozhang Wang
Apache Nifi, Kafka and Storm Better Together by Bryan Bende & Haimo Liu
Data Streaming with Apache Kafka & MongoDB by Andrew Morgan & David Tucker
Data Loss and Duplication in Kafka by Jayesh Thakrar
Apache Kafka – Stream Processing made easy by Daan Gerits
What’s New in Spark 2? by Eyal Ben Ivri
Real-Time Analytics with MemSQL and Spark by Neil Dahlke
Stream processing by Ashic Mahtab
Developing a Streaming Application Using Apache Apex by Pramod Immaneni
Building Streaming Applications with Apache Apex by Chinmay Kolhatkar & Thomas Weise
Scaling up Near real-time Analytics @ Uber and LinkedIn by Chinmay Soman & Yi Pan
Analytics on AWS:Structured, Unstructured and Streaming by Russell Nash
Apache NiFi Crash Course Intro by Rafael Coss

New Videos

Stream processing by Ashic Mahtab
Apache Kafka, Storm and NiFi: Better Together by Bryan Bende & Haimo Liu
Distributed Commit Logs with Apache Kafka by James Ward
Apache Samza Past, Present and Future by Kartik Paramasivam
Google Dataflow: The New Open Model for Batch and Stream Processing by Sara Robinson & Robert Kubis
Cloud Native Streaming and Event-Driven Microservices by Marius Bogoevici
Reactive Streams principles applied in Akka Streams by Eric Loots
Oracle Streams Analytics platform – Is it time to reverse traditional analytics? Oracle Partner Webcast

New Releases

Spark 1.6.3

Upcoming Events

11/20/2016 (Mountain View, US) – November Apache Kafka Meetup (Meetup)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
11/17/2016 (Mountain View, US) – Pulsar – Distributed pub-sub messaging & Apache NiFi in action (Meetup)
11/21/2016 (Columbia, US) – Using Apache NiFi at Capital One & The Apache Way (Meetup)
11/22/2016 (online) – Digital Transformation to Data-Driven Real-Time Enterprise (Cloudera Webinar)
11/29/2016 (Berlin, GE) – Apache Flink Meetup @ data Artisans (Meetup)
11/30/2016 (Tel Aviv, IL) – Introducing StreamSets – Creating Scalable and Resilient Data Pipelines (DataZone Event)
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/5/2016 (New York, US) – Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi (Meetup)
12/6/2016 (Lisbon, PT) – A Deep-dive into Structured Streaming (Meetup)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:19 on October 31, 2016
Tags: beam ( 25 ), flink ( 92 ), hdf ( 20 ), heron ( 23 ), kafka ( 248 ), nifi ( 238 ), samza, spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 10/31/2016

This is the 38th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Is it time you started thinking about event stream processing? Is it time you started thinking about event stream processing? by Henrikki Hervonen

Comparison

Hadoop, Storm, Samza, Spark, and Flink: Big Data Frameworks Compared by Justin Ellingwood

Apache Kafka

Unifying Stream Processing and Interactive Queries in Apache Kafka by Damian Guy & Michael Noll & Neha Narkhede
Build an ETL Pipeline With Kafka Connect via JDBC Connectors by Saikrishna Teja Bobba
How to Persist Kafka Data as JSON in NoSQL Storage Using MapR Streams and MapR-DB by Ian Downard

Apache Storm / Heron

Big Data in the Linode Cloud: Streaming Data Processing with Apache Storm by Phil Zona
Developing Utility Bolts for Apache Storm by Spiro Michaylov
Apache Storm Topology Tuning Approach for Squirrels by Constantin Stanca
What’s new in StormCrawler 1.2 by Julien Nioche

Spark Streaming

Containers and microservices find home in Hadoop ecosystem by Jack Vaughan
Setting Up an Apache Spark Powered Recommendation Engine by David Wilmer

Apache Flink

Windowing data in Big Data Streams – Spark, Flink, Kafka, Akka by Adam Warski
Microservices and Stream Processing Architecture at Zalando Using Apache Flink by Srini Penchikala

Apache NiFi / Hortonworks Data Flow (HDF)

Guide to the New Features of Hortonworks DataFlow 2.0 by Haimo Liu & Bryan Bende
NiFi and HTTP Post Configuration by Tomas Zezula

New Presentations

Simplifying Big Data in Apache Spark 2.0 by Matei Zaharia
Building Data Pipelines with Spark and StreamSets by Pat Patterson
Multi Tier, Multi-Tenant, Multi-Problem Kafka by Todd Palino
Data integration with Apache Kafka by David Tucker
Updating materialized views and caches using Kafka by Zach Cox
How we built an event-time merge of two Kafka Streams with Spark Streaming by Ralf Sigmund & Sebastian Schröder
Guide to New Features of Hortonworks DataFlow 2.0 by Haimo Liu & Bryan Bende
When your Stream is the System of Record by Will Ochandarena
In-Stream Processing Service Blueprint- Reference Architecture for real-time Big Data Applications Anton Ovchinnikov
Real-time Streamng Data on AWS – Deep Dive & Best Practices Using Amazon Kinesis, Spark Streaming, AWS Lambda and Amazon EMR by Radhika Ravirala
Building a Streaming Data Platform on AWS by Radhika Ravirala
Real-Time Streaming: Intro to Amazon Kinesis by Roy Ben-Alta
Apache NiFi 1.0 in Nutshell by Koji Kawamura & Arti Wadhwani
Online Model Updating with Spark Streaming by Keira Zhou
Storing SysDig Streams with Apache NiFi by Tim Spann

New Videos

Simplifying Distributed Systems Using Apache Kafka by ObjectPartners
Multi-Tier, Multi-Tenant, Multi-Problem Kafka by Todd Palino
Real-time Data Processing using AWS Lambda by Tara Walker
The Evolution of Massive Scale Data Processing by Taylor Akidau
New Features in Samza 0.10.0 by Navina Ramesh
Performance benchmarking for Samza by Tao Feng

New Podcasts

Frances Perry on Apache Beam

New Releases

Upcoming Events

11/2/2016 (Mountain View, US) – Stream Processing Meetup @ LinkedIn (Meetup)
11/3/2016 (online) – Edge intelligence for IoT with Apache MiNiFi (Hortonworks Webinar Data-in-Motion Series)
11/3/2016 (San Francisco, US) – Ingest at Intuit and an Intro to StreamSets Dataflow Performance Manager (Meetup)
11/8/2016 (online) – Apache Kafka, Storm and Apache NiFi: Better Together (Hortonworks Webinar Data-in-Motion Series)
11/9/2016 (Santa Clara, US) – Analyze Traffic Patterns with Apache NiFi (Meetup)
11/9/2016 (Mountain View, US) – Realtime Anomaly Detection w/ Flink at Mux, and Connecting Flink to the World (Meetup)
11/20/2016 (Mountain View, US) – November Apache Kafka Meetup (Meetup)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 17:51 on October 25, 2016
Tags: beam ( 25 ), flink ( 92 ), hdf ( 20 ), nifi ( 238 ), samza, storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 10/25/2016

This is the 37th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Stream Processing and Lambda Architecture Challenges by Alexandre Rodrigues
Anomaly Detection in Telecommunications Using Complex Streaming Data by Ted Dunning
To stream or not to stream? by Felix Liao

Apache Kafka

How Apache Kafka is powering a real-time data revolution by Deb Nicholson
Announcing Apache Kafka 0.10.1.0 by Jason Gustafson
Released Mocked Streams for Apache Kafka by Jendrik Poloczek
Apache Kafka for Real-Time Retail at Walmart Labs by Anil Kumar

Apache Storm / Heron

Complex event processing (CEP) with Apache Storm and Apache Ignite by Shamim Ahmed Bhuiyan

Apache Samza

Announcing the release of Apache Samza 0.11.0 by Apache Samza

Apache Flink

Flink: Worth a Second Look by Moshe Kranc

Apache Beam

Testing Unbounded Pipelines in Apache Beam by Thomas Groh

StreamSets

Creating a Custom Processor for StreamSets Data Collector by Pat Patterson
StreamSets Data Collector – New Package Manager in Action by Guido Schmutz
Additional Stage Libraries with dockerized StreamSets by Guido Schmutz
Contributing to the StreamSets Data Collector Community by Pat Patterson

Apache NiFi / Hortonworks Data Flow (HDF)

Real World Use Cases of Real-Time DataFlows in Record Time by Anna Yong & Haimo Liu

New Presentations

Distributed stream processing with Apache Kafka by Jay Kreps
Design Patterns for working with Fast Data by Ian Downard
Log Ingestion: Kafka to Impala using Apex by Shubham Pathak
Intro to Apache Apex @ Women in Big Data by Pramod Immaneni
Introduction to Real-Time Data Processing by Yogi Devendra
Writing an Apache Apex Application by Akshay Gore
Windowing in Apache Apex by Yogi Devendra
Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid by Tony Ng
Confluent Platform 3.1 Introduction by Hans Jespersen
Reactive Integrations with Akka Streams by Johan Andrén
Harnessing Data-in-Motion with Hortonworks DataFlow A Paradigm Shift to Business as Usual: Real World Use Cases of Real-Time DataFlows in Record Time by Anna Yong Product & Haimo Liu
In-Stream Processing Service Blueprint, Reference architecture for real-time Big Data applications by Anton Ovchinnikov
Spark Streaming into context by David Martinez Rego
Streaming Engines for Big Data by Stavros Kontopoulos
IoT & Cortana Analytics by Arun Kumar

New Videos

Monitoring Kafka and Understanding Consumer Lag by David Brinegar & Sachin Agarwal
Kafka @Scale in Cloud by Allen Wang

New Releases

Upcoming Events

10/25/2016 (Chicago, US) – Big Data Streaming Platform Ecosystem (Meetup)
10/25/2016 (Madision, US) – Real-time Analytics on Medical Device Data with Kudu and Streamsets (Meetup)
10/26/2016 (online) – How-To Guide for New Features of Hortonworks DataFlow 2.0 (Hortonworks Webinar Data-in-Motion Series)
10/27/2016 (online) – Data Integration with Apache Kafka (Confluent Online Talk series)
11/2/2016 (Mountain View, US) – Stream Processing Meetup @ LinkedIn (Meetup)
11/3/2016 (online) – Edge intelligence for IoT with Apache MiNiFi (Hortonworks Webinar Data-in-Motion Series)
11/3/2016 (San Francisco, US) – Ingest at Intuit and an Intro to StreamSets Dataflow Performance Manager (Meetup)
11/8/2016 (online) – Apache Kafka, Storm and Apache NiFi: Better Together (Hortonworks Webinar Data-in-Motion Series)
11/9/2016 (Santa Clara, US) – Analyze Traffic Patterns with Apache NiFi (Meetup)
11/20/2016 (Mountain View, US) – November Apache Kafka Meetup (Meetup)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 05:29 on September 20, 2016
Tags: beam ( 25 ), flink ( 92 ), hdf ( 20 ), heron ( 23 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), samza, spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 9/19/2016

This is the 32nd installment of my blog series around Stream Processing and Analytics.

This week I’m at Oracle Open World 2016 in San Francisco and it was not easy to find time to finish this article!

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Building a successful enterprise grade IoT platform by Mark Lochbihler
Streaming analytics goes mainstream by Preetam Kumar
Internet of Things Propels Streaming Analytics into the Mainstream by Karl-Heinz Streibich
The strong adoption of streaming analytics by Preetam Kumar & James Kobielus
All Data As Streams: The Rise of Streaming Analytics for Real-time and Beyond by Daniel Gutierrez

Comparison

Scalable Stream Processing: A Survey of Storm, Samza, Spark and Flink by Wolfram Wingerath

Spark Streaming

Alarm Flooding Control with Event Clustering Using Spark Streaming by Mawazo

Apache Kafka

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | September 2016 by Gwen Shapira
Solving the First and Last Mile Problem With Kafka Part 1 by Jesse Anderson
Serializing Debezium events with Avro by Randall Hauch
Apache Kafka for GCP users: connectors for Pub/Sub, Dataflow and BigQuery by Kir Titievsky

Apache Flink

Flink: Worth a Second Look by Moshe Kranc

Apache NiFi / Hortonworks Data Flow (HDF)

Announcing Hortonworks DataFlow (HDF) 2.0 GA! by Haimo Liu
Integrating Apache NiFi and Apache Kafka by Bryan Bende
Creating a Kibana dashboard of Twitter data pushed to Elasticsearch with NiFi by Michael Young
Google Vision & Apache NiFi – Making Advanced Computer Vision Feasible by Jeremy Dyer
Hortonworks DataFlow 2.0 Gets a Fresh Face by Rob Moran

New Presentations

Dynamic Scaling: How Apache Flink Adapts to Changing Workloads by Till Rohrmann
RBea: Scalable Real-Time Analytics at King by Gyula Fóra
Taking a look under the hood of Apache Flink’s relational APIs by Fabian Hueske
Spark Stream Processing + Kafka by Satendra Kumar
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations by Chaitanya Chebolu
Faster and Furiouser – Flink at Speed by Ted Dunning
Joining Infinity – Windowless Stream Processing with Flink by Sanjar Akhmedov
Evaluating Streaming Framework Performance for a Large-Scale Aggregation Pipeline by Ron Crocker
Connecting Apache Flink to the World – Reviewing the streaming connectors by Robert Metzger
Flink and Beam by Maximilian Michels
Building a real time Tweet map with Flink in six weeks by Matthias Kricke
The Stream Processor as a Database- Building Online Applications directly on Streams by Jamie Grier
Robust Stream Processing with Apache Flink by Jamie Grier
Spark Stream Processing + Kafka by Satendra Kumar
Declarative stream processing with StreamSQL and CEP by Fabien Hueske & Till Rohrmann
Deep Dive into Unbounded Data Processing Systems by Monal Daxini
Cloud scale event processing pipeline by Monal Daxini
What is FastData & How the SMACK stack plays a major role by implementing a Fast Data strategy by Stavros Kontopoulos
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with Apache Kafka by Jun Rao
Our Kafka Journey to 0.10 by Suneet Grover
Real-time Data Integration with Kafka and Cassandra by Ewen Cheslack-Postava
Data Pipelines with Kafka Connect by Kaufman Ng
Rise of Intermediate APIs – Beam and Alluxio by Eric Anderson
Evaluating Streaming Framework Performance for a Large-Scale Aggregation Pipeline by Ron Crocker

New Videos

Building a real-time analytics pipeline with BigQuery and Cloud Dataflow by Eric Schmidt
Data Streaming with Apache Kafka & MongoDB by Andrew Morgan & David Tucker
Real-time Data Integration with Kafka and Cassandra by Ewen Cheslack-Postava
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations by Chaitanya Chebolu

New Releases

Upcoming Events

9/20/2016 (online) – Easy Big Data Ingestion into Microsoft Azure HDInsight (Streamsets Webinar)
9/20/2016 (Omaha, US) – Machine Leaning in the Lambda Architecture: Simultaneous Speed and Batch ML (Meetup)
9/21/2016 (Warsaw, PL) – Apache Spark Streaming with Apache NiFi and Apache Kafka (Meetup)
9/22/2016 (New York, US) – Twitter Heron in Practice (Meetup)
9/22/2016 (online) – Accelerating Real-time Data Ingest into Hadoop (Hortonworks Webinar)
9/27/2016 (online) – Introduction to Streaming Data and Processing with Apache Kafka (Confluent Online Talk series)
10/4/2016 (Austin, US) – Leveraging Kafka, Kafka Streams and Kafka Connect for Fast Data Streams, Analytics and Machine Learning using Scala (Reactive Summit 2016)
10/5/2016 (online) – Harness Your Dataflow Operations with StreamSets Dataflow Performance Manager (StreamSets Webinar)
10/6/2016 (online) – Deep Dive into Kafka (Confluent Online Talk series)
10/13/2016 (London, UK) – Distributed stream processing with Apache Kafka (Meetup)
10/27/2016 (online) – Data Integration with Apache Kafka (Confluent Online Talk series)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:39 on September 13, 2016
Tags: apex ( 18 ), confluent ( 8 ), flink ( 92 ), hdf ( 20 ), heron ( 23 ), kafka ( 248 ), nifi ( 238 ), samza, spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 9/13/2016

This is the 31st installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Fast Data Architectures For Streaming Applications: free O’Reilly report by Dean Wampler by Markus Eisele
Yahoo open-sources Pulsar, a low-latency alternative to Apache Kafka by Mike Wheatley

Comparison

Samza vs Spark Streaming by Jae Hyeon Bae

Spark Streaming

How to Get Started with Spark Streaming and MapR Streams Using the Kafka API by Carol McDonald
AVRO Data Processing in Spark Streaming + Kafka by Chandan Prakash

Apache Kafka

Announcing Confluent Platform 3.0.1 & Apache Kafka 0.10.0.1 by Neha Narkhede
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection? by Neha Narkhede
Announcing the Certified DataStax Connector for Confluent Platform by David Tucker
Confluent and DataStax Partner to Enable Streaming Applications based on Apache Kafka and Apache Cassandra by Confluent
Kafka Connect Cassandra Sink: The Perfect Match by Andrew Stevenson
Using logdump to Troubleshoot the Oracle GoldenGate for Big Data Kafka Handler by Robin Moffatt

Apache Flink

Real-time Streaming Attribution Using Apache Flink by Vishnu Viswanath

Apache Apex

Apache Apex for Real Time Stream Processing by Akshay Harale & Saurabh Agrawal

StreamSets

Ingesting Drifting Data into Hive and Impala by Pat Patterson
StreamSets Data Collector in Action at IBM Ireland by Pat Patterson
Introducing StreamSets DPM – Operational Control of Your Data in Motion by Girish Pancha

Apache NiFi / Hortonworks Data Flow (HDF)

Apache NiFi 1.0.0 – Zero-Master Clustering by Mark Payne
Reading OpenData JSON and Storing into Phoenix Tables by Timothy Spann
Change Data Capture using NiFi by Satish Bomma
Reading Sensor Data from Remote Sensors on Raspberry Pis by Timothy Spann
New HDF Product Integration Certification Program by Anna Yong & Mark Lochbihler

New Presentations

Adaptive Data Cleansing with StreamSets and Cassandra by Pat Patterson
The Avant-garde of Apache NiFi by Joe Percivall
Apache Kafka Security by Manikumar Reddy
Real-Time Relevance Rabobank’s Event-driven Architecture using Apache Kafka by Jeroen van Disseldorp
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application by Munagala V. Ramanath
Intro to Apache Apex – Next Gen Platform for Ingest and Transform by Pramod Immaneni
Apache Flink Training Workshop by Tzu-Li (Gordon) Tai
Log Event Stream Processing In Flink Way by George Lai
Data Streaming with Apache Kafka & MongoDB by Andrew Morgan & David Tucker
Streaming SQL by Julian Hyde

New Videos

Intro to Apex (Next Gen Hadoop) and Storm Comparison by Devendra Tagare
Ingesting Drifting Data into Hive and Impala by Pat Patterson

New Releases

New Books

Fast Data Architectures For Streaming Applications by Dean Wampler
Introduction to Apache Flink by Ellen Frieden & Kostas Tzoumas
Apache Flink in Action (MEAP) by Sameer B. Wadkar & Hari Rajaram

Upcoming Events

9/14/2016 (Princeton, US) – Data Pipelines with Kafka Connect (Meetup)
9/15/2016 (online) – Sneak Peek: Confluent Platform 3.1 (Confluent Webinar)
9/15/2016 (Mountain View, US) – September Apache Kafka Meetup (Meetup)
9/19-22/2016 (online) – Building Distributed Pipelines for Data Science using Kafka, Spark, and Cassandra (O’Reilly Online Training)
9/20/2016 (online) – Easy Big Data Ingestion into Microsoft Azure HDInsight (Streamsets Webinar)
9/20/2016 (Omaha, US) – Machine Leaning in the Lambda Architecture: Simultaneous Speed and Batch ML (Meetup)
9/21/2016 (Warsaw, PL) – Apache Spark Streaming with Apache NiFi and Apache Kafka (Meetup)
9/22/2016 (New York, US) – Twitter Heron in Practice (Meetup)
9/22/2016 (online) – Accelerating Real-time Data Ingest into Hadoop (Hortonworks Webinar)
10/4/2016 (Austin, US) – Leveraging Kafka, Kafka Streams and Kafka Connect for Fast Data Streams, Analytics and Machine Learning using Scala (Reactive Summit 2016)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 06:41 on August 17, 2016
Tags: beam ( 25 ), flink ( 92 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), samza, storm ( 46 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 8/15/2016

This is the 27th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Streaming Analytics: A story of many tales by Teradata
Microservices: Real Architectural Patterns by Camille Fournier

Apache Kafka / Kafka Streams

The top sessions from this year’s Kafka Summit are… by Luanne Dauber
More Than Just a Schema Store by Chia-Chi Lin
Kafka Streams – Not Looking at Facebook by Timothy Renner
Running Apache Kafka on Windows without Cygwin by Gopal Tiwari
Tuning Handbook of Apache Kafka! by Kumar Chinnakali
Kafka Connect – Import Export for Apache Kafka by Jarosław Kijanowski
Data Reprocessing with Kafka Streams: Resetting a Streams Application by Matthias J Sax

Spark Streaming

Testing Spark Streaming: Integration Testing With Docker Compose by Tomaz Tekavec

Apache Storm

Storm as a Metaphor by Hugo Picado

Apache NiFi / Hortonworks Data Flow (HDF)

Three Things To Know About HDF 2.0 by Haimo Liu & Kanishk Mahajan
Nifi 1.0.0 Beta UI Introduction by Henry Sowell
Apache NiFi 1.0.0 – Cluster setup by Pierre Villard
Apache NiFi 1.0.0-BETA! by James Barney

New Presentations

Building Event-Driven Systems with Apache Kafka by Brian Ritchie
Apache NiFi and Stream Processing by Dhruv Kumar
Motivation for and Introduction to Beam and Dataflow by Eric Anderson
Cloud Native Streaming and Event-Driven Microservices by Marius Bogoevici
Introduction to Amazon Kinesis Analytics by Roger Barga & Ryan Nienhuis
Deep Dive and Best Practices for Real-Time Streaming Applications by Adi Krishnan
SQL on Streams powered by Apache Flink and Apache Calcite by Radu Tudoran
Reactive Kafka by Rajini Sivaram

New Videos

Evolving Big Data ETL in the Face of Data Drift by Arvind Prabhakar & Pat Patterson

New Release

Upcoming Events

8/17/2016 (Sunnyvale, US) – 53rd Bay Area Hadoop User Group (HUG) Meetup (Meetup)
8/18/2016 (New York, US) – Apache NiFi – MiNiFi: Taking Dataflow Management to the Edge (Meetup)
8/18/2016 (San Jose, US) – Ingest and Extract Big Data Natively in Hadoop with Apache Apex (Meetup)
8/18/2016 (online) – The Business Impact of Fast Data Analytics (Hortonworks Webinar)
8/23/2016 (Mountain View, US) – Stream Processing Meetup @ LinkedIn (Meetup)
8/25/2016 (San Francisco, US) – Focusing on Ingest into Hive and Streaming with Kafka (Meetup)
8/31/2016 (Palo Alto, US) – Heron@Twitter (Meetup)
9/1/2016 (online) – Microservices in the Apache Kafka Ecosystem (Confluent Webinar)
9/8/2016 (San Francisco, US) – Adaptive Data Cleansing with StreamSets and Cassandra (Cassandra Summit)
9/13/2016 (online) – Data Streaming with Apache Kafka & MongoDB (MongoDB webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 17:24 on February 13, 2016
Tags: flink ( 92 ), kafka ( 248 ), nifi ( 238 ), samza, spark-streaming ( 219 ), storm ( 46 ), stream-explorer ( 2 )

This week in Stream Processing & Streaming Analytics 2/13/2016

Update 14.2: Have added Google Dataflow / Apache Beam

Inspired by “This Week in Cassandra”, I will start collecting and documenting once a week the latest news in the world of “Stream Processing & Streaming Analytics” platforms and framework. I will look at new projects/subprojects, blog posts and new and upcoming features around Open Source, Oracle, IBM and others.

So far it’s not planned to do that in a live discussion, as the folks at DataStax do. But maybe that could be something for the future.

Before starting with the news of the last week, here are some classical / evergreens around the topic of Stream Processing / Streaming Analytics:

The world beyond batch: Streaming 101 by Tyler Akidau August 5, 2015
The world beyond batch: Streaming 102 by Tyler Akidau January 20, 2016
Stream Processing, Event Sourcing, Reactive, CEP … and making sense of it all by Martin Kleppmann (Confluent)
Some Important Streaming Algorithms you should know about by Ted Dunning (MapR)
Apache Kafka, Samza and the Unix Philosophy of Distributed Data by Martin Kleppmann (Confluent)
Real-Time full-text search with Luwak and Samza by Martin Kleppmann (Confluent)
The Log: What every software engineer should know about real-time data’s unifying abstraction by Jay Kreps (LinkedIn Engineering)
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing by Tyler Akidau et. al.

News and Blog Posts

General

Big Data Analytics Forecast : The Streaming Future by Anuj Tripathi
Data Lake vs. Data Streams by Krishna Balaram Singh

Comparisons

Help choosing a streaming analytics platform: Apache Storm comparison to Azure Stream Analytics by Jeff Stokes (Microsoft Azure)
Stream Processing Everywhere – What to use? by Jim Scott (MapR)
Comparison of Apache Stream Processing Frameworks: Part 1 by Petr Zapletal (Cake Solutions)
A Comprehensive Analysis – Data Processing Part Deux: Apache Spark vs Apache Storm by Rassul Fazelat
Dataflow/Beam & Spark: A Programming Model Comparison by Tyler Akidau & Frances Perry (Apache Beam)
Google Dataflow, Hortonworks DataFlow – What’s in a name? Are they the same? by Haimo Liu (Hortonworks)

Apache Storm

Windowing and State checkpointing in Apache Storm by aiyer (Hortonworks Forum)

Apache Spark Streaming

Faster Stateful Stream Processing in Spark Streaming by Tathagata Das and Shining Zhu (Databricks)
Querying Amazon Kinesis Streams Directly with SQL and Spark Streaming by Arno Abeyaratne (Amazon Kinesis)

Apache Samza

Stream Processing with Apache Samza – Current and Future by Kartik Parmasivam (LinkedIn Engineering)

Apache Flink

The Importance of Apache Flink in Processing Streaming Data by Kaushik Pal (Techopedia)
Extending the Yahoo! Streaming Benchmark by Jamie Grier (Data Artisans)
How Apache Flink enables new streaming applications by Uruk Celebi and Kostas Tzoumas (Data Artisans)
The Essential Guide to Streaming-first Processing with Apache Flink by Fabian Hueske (MapR)

Google Dataflow / Apache Beam

Dataflow and open source – proposal to join the Apache Incubator by Google Cloud Platform Blog

Apache NiFi

Understanding how to use Process Groups and Remote Process Groups by mclark
How do I distribute Data across a NifI Cluster by bbende
Monitoring an S3 bucket in Apache NiFi by Adam Lamar

Apache Kafka / Confluent Platform

Apache Kafka Security 101 by Ismael Juma (Confluent)
Apache Kafka is booming, but should you use it? by Matt Asay
Introducing the Kafka Consumer: Getting Started with the new Apache Kafka 0.9 Consumer Client by Jason Gustafson (Confluent)
Getting started with sample programs for Apache Kafka 0.9 by Tugdual Grall (MapR)
The Rise and Rise of Apache Kafka by Ryan Fintan

Oracle Stream Xplorer / Oracle Event Processing

Connecting Stream Explorer and BAM by Wesley Souza

Others

StreamAnalytix 2.0 adds support for Spark by Andrew Brust (ZDNet)

New Presentations

Apache Spark – Introduction to Spark Streaming and Deep Dive by Paranth Thiruvengadam and Sachin Aggarwal (IBM)
Integrating Apache NiFi and Apache Flink by Bryan Bende (Hortonworks)
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications! by Tugdual Grall (MapR)
Real-Time Analytics with Apache Cassandra by Guido Schmutz (Trivadis)
Prefix Time Series and Apache Apex by Venkatesh Sivasubramanian

New Releases / Components

Upcoming Events

2/16/2016 (New York City, US) – Spark Summit East
2/16/2016 (Chicago, US) – Modern Data Architecture using Flink and Hadoop (Meetup)
2/17/2016 (Munich, Germany) – The 2 Worlds of Big Data: Batch & Stream – use cases, challenges and solutions (Meetup)
2/17/2016 (Austin, US) – Invited Speaker Tim Renner Presents on Streaming with Storm, Spark and Kafka (Meetup)
2/17/2016 (San Jose, US) – Apex&Geode Integration; Ingestion: Kafka to Hadoop/Apex&Geode (Meetup)
2/17/2016 (Sunnyvale, US) – Stream Processing Architecture and Applications – Apache Apex (Meetup)
2/22/2016 (Warshow, Poland) – Spark in Action – Spark Streaming (Meetup)
2/24/2016 (Sao Paulo, Brazil) – Introduction to Apache Flink: What, How, Why, Who, Where? (Meetup)
2/25/2016 (Toronto, Canada) – The Emerging Fast Data Architecture by Dean Wampler (Meetup)
2/25/2016 (Online) – Online Meetup – Real-time Anomaly Detection Architecture using Kafka (Meetup)
2/25/2016 (Seattle, US) – Accelerating Data Ingestion & Streaming Analytics with HDF/NiFi (Meetup)
3/1/2016 (Washington, US) – Extending the Yahoo! Streaming Benchmark & Winning Twitter Hack-Week with Flink (Meetup)
4/14/2016 (Chicago, US) – Jay Kreps: Apache Kafka and the Confluent Platform – Overview and Roadmap (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!