Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:06 on April 26, 2017
Tags: apex ( 18 ), azure-stream-analytics, flink ( 92 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), streamsets ( 74 )

Last week in Stream Processing & Analytics – 26.4.2017

This is the 63rd edition of my blog series blog series around Stream Processing and Analytics! A few days later than planed, but I had some busy days with speaking on Apache Kafka at Oracle Code Berlin on Monday and today we held our Trivadis/DataStax Hands-On workshop in Zurich. So next week will be again on time (Monday), I hope 😉

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Comparison

Event time processing in Apache Spark and Apache Flink by Matthias Niehoff

Apache Kafka / Kafka Streams / Confluent Platform

Kafka channels the big data firehose by Tony Baer
Introducing Kasper: The Kafka Stream Processing Library Built For Go by Nicolas Maquet
Benchmarking Apache Kafka with Aiven – Part 1: Write throughput (AWS vs Azure vs DigitalOcean vs Google Cloud vs UpCloud) by Aiven

Spark Streaming

Fault tolerance in Apache Spark – Reliable Spark Streaming by Data Flair
Real-Time Weather Event Processing With HDF, Spark Streaming, and Solr by Balaji Kandregula
Spark Streaming – under the hood by veejayendraa
Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2 by Kunal Khamar & Tyson Condie & Michael Armbrust

Apache Apex

One Year Anniversary of Apache Apex by Thomas Weise

Apache Flink

Build a Real-time Stream Processing Pipeline with Apache Flink on AWS by Steffen Hausmann

StreamSets

StreamSets Data Collector v2.5 Adds IoT, Spark, Performance and Scale by Kirit Basu
Make sense of stream processing by Pat Patterson
Creating a Custom Multithreaded Origin for StreamSets Data Collector by Pat Patterson

Apache Nifi

Basics of Apache Nifi: 1 by Malcom Thirus
How to send data from Apache NIFI to HANA by Franco Lezana

Azure Stream Analytics

Announcing Azure Stream Analytics on edge devices (preview) by Santosh Balasubramanian

New Presentations

Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema
Kafka used at scale to deliver real-time notifications by Sergio Nunes
Apache Storm In Retail Content by Karthik Deivasigamani
Storm worker redesign by Roshan Naik
From Three Nines to Five Nines – A Kafka Journey by Allen Wang
Microsoft challenges of a multi tenant kafka service by Nitin Kumar
The Rise of Real-Time by Neha Narkhede
Apache Kafka – Scalable Message-Processing and more ! by Guido Schmutz
Building Streaming Applications with Apache Storm 1.1 by Hugo Louro
Kafka Tutorial – basics of the Kafka streaming platform by Jean-Paul Azar
Bootstrapping Microservices with Kafka, Akka and Spark by Alex Silva

New Videos

Building Faster Streaming Applications with Apache Storm 1.1 by Suresh Srinivas & Hugo Louro & Karthik Deivasigamani
Putting the Micro into Microservices with Stateful Stream Processing by Ben Stopford
StreamING models, how ING adds models … by Erik de Nooij
Real-Time Analytics On Data Streams – A New Era For Big Data And IoT by Jeremy Hillier
Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema

New Podcasts

Data science for real-time streaming analytics by Roger Rea

New Releases

StreamSets 2.5

Upcoming Events

25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)
27.4.2017 (Charlotte, US) – Apache NiFi – An Uber Data Flow Tool (Meetup)
16.5.2017 (online) – Geo-Distributed Big Data and Analytics (MapR Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:55 on March 6, 2017
Tags: akka ( 3 ), apex ( 18 ), azure-stream-analytics, flink ( 92 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics – 6.03.2017

This is the 56th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Unlocking the real value of the IoT by Anne-Lindsay Beall
Building the Ideal Stack for Real-Time Analytics by Mason Hooten
Real-Time Stream Processing as Game Changer in a Big Data World by Hossein Eslambolchi
The 3 keys to digital transformation by Jack Norris
Stream processing and the IBM Open Platform by Andrea Braida

Apache Kafka / Kafka Streams / Confluent Platform

Confluent 3.2 with Apache Kafka 0.10.2. Now Available by Chrix Finne
Kafka Connect Elasticsearch: Consuming and Indexing with Kafka Connect by Rafal Kuć
Configuring and Running Apache Kafka in IBM BigInsights by Nisanth Simon
Monitoring Consumer Offsets in #Kafka by Jason Bell

Apache Flink

Running Apache Flink® Everywhere: Flink on DC/OS and Apache Mesos by Till Rohrmann

StreamSets

StreamSets with Docker – an example HDFS integration by Dominic Ketley
Announcing StreamSets Data Collector ver 2.4.0.0 by Kirit Basu
Read and Write JSON to MapR DB with StreamSets Data Collector by Rupal Shah
Drift Synchronization with StreamSets Data Collector and Azure Data Lake by Pat Patterson

Apache NiFi / Hortonworks HDF

Working with S3 Compatible Data Stores via Apache NiFi by Timothy Spann
Using MiniFi to read data from a Sense HAT on a Raspberry Pi 3 by Timothy Spann
Cool Projects in Big Data, Machine Learning, and Apache NiFi by Timothy Sapnn

Azure Stream Analytics

Announcing real-time Geospatial Analytics in Azure Stream Analytics by Samartha Chandrashekar

New Presentations

Kafka’s Role in Implementing Oracle’s Big Data Reference Architecture by Robin Moffatt
StreamSets and Spark by Hari Shreedharan
Introduction to data flow management using apache nifi by Anshuman Ghosh
End-to-End “Exactly-Once” processing using Apache Apex by Ajay Gupta
Real-Time Stream Processing on EMR: Apache Flink vs. Apache Spark Streaming by Keith Steward
In memory analytics with apache spark and hazelcast by Viktor Gamov
Akka-chan’s Survival Guide for the Streaming World by Konrad Malawski
Apache Flink’s Table & SQL API – unified APIs for batch and stream processing by Timo Walther

New Videos

StreamSets and Spark by Hari Shreedharan
Building an Open Source Streaming Analytics Stack with Kafka and Druid by Fangjin Yang
Streaming all the things with Akka Streams by Johan Andrén
Real-time Stream Processing on EMR: Apache Flink vs. Spark Streaming by Keith Steward
Secure IoT Command, Control, and Exfil with Apache MiNiFi by Andy LoPresto
End-to-End “Exactly-Once” processing using Apache Apex by Ajay Gupta

New Releases

New Books

Kafka Streams in Action by William Bejeck

Upcoming Events

7.3.2017 (Santa Monica, US) – Stream processing with R in AWS (Meetup)
7.3.2017 (Austin, US) – Stream All the Things! (Meetup)
14.3.2017 (Palo Alto, US) – Apache Kafka Meetup at Strata San Jose (Meetup)
15.3.2017 (Paris, FR) – Apache Flink: Use Cases (Meetup)
21.3.2017 (online) – Getting Started with Streaming Data and Stream Processing with Apache Kafka (KPI Webinar)
23.3.2017 (San Francisco, US) – Bay Area Apache Spark Meetup @ Intel in Santa Clara (Meetup)
28.3.2017 (Princeton, US) – Apache NiFi: Ingesting Enterprise Data @ Scale (Meetup)
29.3.2017 (online) – Part 2: Building Event-Driven Services with Apache Kafka (Confluent Webinar)
5.4.2017 (London, UK) – Realtime stream processing with coroutines (Meetup)
25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 18:04 on August 2, 2016
Tags: azure-stream-analytics, kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), Oracle ( 9 ), oracle stream analytics ( 11 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 8/2/2016

This is the 25th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Comparison

Big data brawlers: 4 challengers to Spark by Serdar Yegulalp
Apache Kafka and Google Cloud Pub/Sub by Jesse Anderson

Apache Beam

Apache Beam: The Case for Unifying Streaming APIs by Andrew Psaltis

Apache Kafka / Kafka Streams

Design and Deployment Considerations for Deploying on AWS by Alex Loddengaard
Kafka Connect JDBC – Oracle – Number of groups must be positive by Robin Moffatt
Streaming MySQL tables in real-time to Kafka by Prem Santosh & Udaya Shankar
Confluent and Mesosphere Drive the “Container 2.0” Era With Enterprise-Grade Kafka on DC/OS by Market Wired
Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | August 2016 by Gwen Shapira
How Apache Kafka takes streaming data mainstream by Matt Asay
The dog ate my schema… or what is your excuse not to use the Schema Registry with Apache Kafka? by Jaroslaw Kijanowski
Apache Kafka Connect MQTT Source Tutorial by howtoprogram

Spark Streaming

Structured Streaming In Apache Spark by Matei Zaharia, Tathagata Das, Michael Armbrust and Reynold Xin
Continuous Applications: Evolving Streaming in Apache Spark 2.0 by Matei Zaharia

Streamsets

Standard Deviations on Cassandra – Rolling Your Own Aggregate Function by Pat patterson

Apache NiFi / Hortonworks Data Flow (HDF)

Using Apache Flume Sources and Sinks with Apache NiFi 0.70 by Timothy Spann
Crash Courses on Spark, IoT, Hadoop, NiFi, DataScience, and More! by Timothy Spann
Accessing Facebook Page Data from Apache NiFi by Timothy Spann
Use NiFi to Lessen the Friction of Moving Data by Hays Hutton

Oracle Stream Analytics

Stream Analytics and Processing with Kafka and Oracle Stream Analytics by Robin Moffatt

New Presentations

Apache Beam: The Case for Unifying Streaming APIs by Andrew Psaltis
Handson with Twitter Heron by Karthik Ramasamy
Apache Apex: Stream Processing Architecture and Applications by Thomas Weise
Getting started with azure event hubs and stream analytics services by Vladimir Bychkov
Apache Beam and Google Cloud Dataflow by Szabolcs Feczak
Introduction to Kafka connect by Himani Arora
Building a fully-automated Fast Data Platform by Bernd Zuther

New Videos

Apache Beam: The Case for Unifying Streaming APIs by Andrew Psaltis

New Release

Upcoming Events

8/2/2016 (online) – Real-Time Data Streaming from Oracle to Apache Kafka (Confluent Webinar)
8/2(2016 (online) – Streaming Analytics for Real-Time Action – Best Practices for Getting Started (TDWI Webinar)
8/3/2016 (Moutain View, US) – Building a Real-time Streaming Platform Using Kafka Streams and Kafka Connect (Meetup)
8/9/2016 (online) – Evolving Big Data ETL in the Face of Data Drift (StreamSets Webinar)
8/17/2016 (Sunnyvale, US) – 53rd Bay Area Hadoop User Group (HUG) Meetup (Meetup)
8/18/2016 (New York, US) – Apache NiFi – MiNiFi: Taking Dataflow Management to the Edge (Meetup)
8/23/2016 (Mountain View, US) – Stream Processing Meetup @ LinkedIn (Meetup)
8/25/2016 (San Francisco, US) – Focusing on Ingest into Hive and Streaming with Kafka (Meetup)
8/31/2016 (Palo Alto, US) – Heron@Twitter (Meetup)
9/8/2016 (San Francisco, US) – Adaptive Data Cleansing with StreamSets and Cassandra (Cassandra Summit)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:29 on June 13, 2016
Tags: azure-stream-analytics, beam ( 25 ), flink ( 92 ), kafka ( 248 ), oracle stream analytics ( 11 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 6/13/2016

This is the 18th installment of my blog series around Stream Processing and Analytics.

There were two conferences last week with quite a lot of talks around stream processing: the Spark Summit in San Francisco and the Berlin Buzzwords.
Berlin Buzzwords did a good job in recording the sessions and all of them are already available and the ones talking about Stream Processing listed below.

Last week I have done some work on Oracle Stream Analytics and made the Docker support available.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Real-Time Analytics: Six Steps For Fast, Precise Decision-Making by Roy Schulte
Beyond Real-time Data Applications – Whiteboard Walkthrough by Ellen Friedman

Comparison

Top 18 Open Source and Commercial Stream Analytics Platforms by predictiveanalyticstoday

Apache Storm

Getting Started With Azure HDInsight: Create Apache Storm Cluster by Kuppurasu Nagaraj

Apache Flink

Data Streaming Architecture with Apache Flink by Srini Penchikala

Apache Spark Streaming

Spark Summit keynote explores structured streaming, innovation in deep learning by Marlene Den Bleyker
Spark Streaming Testing with Scala Example by Todd McGrath

Apache Kafka

Kafka Connect Sink for PostgreSQL from JustOne Database by Duncan Pauly
Interest grows in Apache Kafka, edge of network for IoT by Marlene Den Bleyker
‘Real-Time’ Online Machine Learning with Apache Kafka and Sklearn [plus other buzz words] by Bryan Travis Smith

Apache Beam / Google Dataflow

Understanding timing in Cloud Dataflow pipelines by Robert Burke

Apache NiFi / Hortonworks HDF

Testing ExecuteScript processor scripts by Matt Burgess
Data Injestion Apache Nifi for Data Collection by Don Jernigan

StreamSets

Ingesting MQTT Traffic into Riak TS via RabbitMQ and StreamSets by Pat Patterson

Concord

Concord Leverages Mesos for High Performance Stream Processing by Susan Hall

Oracle Stream Analytics

Providing Oracle Stream Analytics 12c environment using Docker by Guido Schmutz
Real-Time Data Streaming & Exploration with Oracle GoldenGate & Oracle Stream Analytics by Issam Hijazi
Accessing and Analyzing Twitter Feeds with Oracle Stream Analytics (Part 1) by John Featherly

Microsoft Stream Analytics

How To Build a Real-Time IoT Dashboard with Azure IoT Hub, Azure Stream Analytics and Power BI by John Gallant

New Presentations

A Data Streaming Architecture with Apache Flink by Robert Metzger
A Deep Dive into Structured Streaming by Tathagata Das
So you think you can Stream? by Parkash Chockalingam
Structuring Spark: Dataframes, Datasets And Streaming by Michael Armbrust
The Stream Processor as the Database by Stephan Ewen
Building Realtim Data Pipelines with Kafka Connect and Spark Streaming by Guozhang Wang
Spark streaming , Spark SQL by Jerry Jung
Streaming Analytics & CEP – Two sides of the same coin? by Till Rohrmann
Continuous Processing with Apache Flink by Stephan Ewen
Graphs as Streams: Rethinking Graph Processing in the Streaming Era by Vasia Kalavri
Architectual Comparison of Apache Apex and Spark Streaming by Thomas Weise
Application development and data in the emerging world of stream processing by Neha Narkhede
Introducing Kafka Streams, the new stream processing library of Apache Kafka by Michael Noll

New Videos

Fundamentals of stream processing with Apache Beam by Tyler Akidau
What we’ve learned from developing a stream processing platform at scale by Amit Sela
Architectual Comparison of Apache Apex and Spark Streaming by Thomas Weise
An Introduction to Azure Stream Analytics by Ashish Bhatia
Data Streaming for Connected Devices with Azure Stream Analytics by Juan Manuel Servera
Twitter Heron at Scale by Hakka Labs
Streaming Analytics & CEP – Two sides of the same coin? by Till Rohrmann
Scaling Yelp’s Logging Pipeline with Apache Kafka by Enrico Canzonieri
Real Time Marketing with Kafka, Storm, Cassandra and a pinch of Spark by Volker Janz
Application development and data in the emerging world of stream processing by Neha Narkhede
Help I need a stream processor – learning how to chose between Spark, Flink, Samza, and Storm by Andrew Psaltis
Graphs as Streams: Rethinking Graph Processing in the Streaming Era by Vasia Kalavri
Google Dataflow: The new open model for batch and stream processing by Felipe Hoffa
SMACK Stack – Data done Right by Stefan Siprell
A Data Streaming Architecture with Apache Flink by Robert Metzger
Leveraging blockchain technologies for the internet of things by Sam Bessalah
Introducing Kafka Streams, the new stream processing library of Apache Kafka by Michael Noll
Fast Analytics on Fast Data by Todd Lipcon
Fast Cars, Big Data – How Streaming Can Help Formula 1 by Ted Dunning

Upcoming Events

6/13/2016 (Amsterdam, NL) – GOTO Night: Stream Processing with Apache Flink and Mining Github (Meetup)
6/13/2016 (New York, US) – Apache Beam (Stream Processing @ Scale Track at QCon New York)
6/13/21016 (Dublin, IR) – Production Quality Data Science. Building Rapid Ingestion Data Pipelines (Meetup)
6/14/2016 (Garden City, US) – Moving data gracefully with Apache NiFi
6/14/2016 (Lausanne, CH) – Apache Kafka A high-throughput distributed messaging system (Meetup)
6/15/2016 (Mountain View, US) – Stream Processing Meetup @ LinkedIn (Meetup)
6/15/2016 (London, UK) – Hortonworks Dataflow (HDF) Meetup London (Meetup)
6/16/2016 (Garden City, US) – Moving data gracefully with Apache NiFi (Meetup)
6/21/2016 (San Jose, US) – Apache Bigtop & Apache Apex (Meetup)
6/21/2016 (online) – A New Paradigm for Managing Data in Motion (Webinar)
6/22/2016 (online) – Comparing Open Source Big Data Ingest Options (Streamsets Webinar)
6/27/2016 (San Jose, US) – Building Big Data applications with Apache Beam and Apache Apex (Meetup)
7/5/2016 (San Francisco, US) – Building (and running) Netflix’s Data Pipeline using Apache Kafka (Meetup)
7/6/2016 (Atlanta, US) – StreamSets, For The Coding Minimalist In All of Us (Meetup)
7/14/2016 (Princeton, US) – Apache NiFi (Meetup)
7/19/2016 (Munich, GE) – Apache Apex: Stream Processing Architecture and Applications (Meetup)
7/19/2016 (Taipei, TW) – Stream Processing with Apache Flink (Meetup)
8/18/2016 (New York, US) – Apache NiFi – MiNiFi: Taking Dataflow Management to the Edge (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 21:50 on May 16, 2016
Tags: apex ( 18 ), azure-stream-analytics, beam ( 25 ), flink ( 92 ), kafka ( 248 ), nifi ( 238 ), raspberryPi, spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 5/16/2016

This is the 14th installment of my blog series around Stream Processing and Analytics.

If you have not been able to attend the Kafka Summit in San Francisco last week, then you might like the tweet from Confluent Inc.

#kafkasummit session videos are posted on https://t.co/pCHbD4dpNv. Thanks to all the speakers! @apachekafka #streamprocessing

— Confluent Inc (@ConfluentInc) May 13, 2016

I listed most of the presentations below in the section “New Presentations” but I did not do it for the videos. Find them as mentioned in the tweet here. I have already start watching some of the presentations and it is really worth it! Will list my top 3 sessions next week.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Beyond ETL: Real-time, Streaming Architectures by Sean Anderson
Stream Processing vs. Continuous PCAP: The Big Shift in Monitoring Architectures by Raja Mukerji
3 things you need to know about event stream processing by Alison Bolen
The Value Proposition of Streaming Analytics to the Enterprise by Daniel D. Gutierrez
Ideal Messaging Capabilities for Streaming Data by Ellen Friedman

Comparison

Spark Vs Flink – Which of the two will win by Blue Pi

Apache Spark Streaming

Spark 2.0 Will Be Faster, Easier for App Development, and Tackle Streaming Data by Ian Murphy
Preview of Apache Spark 2.0 now on Databricks Community Edition by Reynold Xin
How-to: Build a Prediction Engine using Spark, Kudu, and Impala by Richard Williamson
Realtime Tweets Analysis using Spark Streaming with Scala by Harini Janakiraman
How to Integrate Custom Data Sources Into Apache Spark by Nicolas A Perez
Receiving AVRO Messages through KAFKA in a Spark Streaming Scala by Timothy Spann

Apache Flink

Apache Flink: The Next Gen Big Data Analytics Framework For Stream And Batch Data Processing by eureka

Apache Kafka

Big data messaging with Kafka, Part 2 by Sunil Patil
Just Enough Kafka for the Elastic Stack, Part 1 by Suyog Rao
Elastic Applications with Apache Kafka by Eslam Nawara

Apache NiFi / Hortonworks HDF

SolR, NiFi, Twitter and CDH 5.7 by Hadoop and Beyond
Running NiFi on Raspberry Pi. Best Practices by Andrew Grande
Comparison of NiFi to Python for streaming application by Vasilis Vagias

Microsoft Azure Stream Analytics

Troubleshooting Azure Stream Analytics jobs with SELECT INTO by Ken Wang
Azure Stream Analytics Publishing to Power BI Reaches General Availability by Kent Weare
Integrating Azure Machine Learning With Azure Stream Analytics to Predict Customer Churn by Rahul Rai

New Presentations

Distributed Real-Time Stream Processing: Why and How 2.0 by Peter Zapletal
Streaming SQL with Apache Calcite by Julian Hyde
Real-Time Event & Stream Processing on MS Azure by Khalid M. Salama
The Rise of Real-Time by Jay Kreps
Fundamentals of Stream Processing with Apache Beam by Tyler Akidau & Frances Perry
Advanced Streaming Analytics with Apache Flink and Apache Kafka by Stephan Ewen
Introducing Kafka Streams: Large-scale Stream Processing with Kafka by Neha Narkhede
When it Absolutely, Positively, Has to be There: Reliability Guarantees in Kafka by Gwen Shapira & Jeff Holoman
Kafka At Scale in the Cloud by Allen Wang
Kafka Connect: Real-time Data Integration at Scale with Apache Kafka by Ewen Cheslack-Postava
The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus by Rimas Silkaitis
Leveraging Kafka for Big Data in Real Time Bidding, Analytics, ML & Campaign Management for Globally Distributed Data Flows by Helena Edelson
Simplifying Event Streaming: Tools for Location Transparency and Data Evolution by Paul Osman
Towards A Stream Centered Enterprise by Gabriel Commeau
Kafka and Stream Processing, Taking Analytics Real-time by Mike Spicer
Never at Rest – IoT and Data Streaming at British Gas Connected Homes by Paul Makkar
Generating Real-time Recommendations with NiFi, Kafka, and Spark by Chris Fregly
Kafka, the “DialTone for Data”: Building a self-service, scalable, streaming analytics system @ HomeAway by Rene Parra
Siphon – Near Real Time Databus Using Kafka by Eric Boyd & Nitin Kumar
Securing Kafka by Jun Rao
Spark Streaming Data Pipelines by Carol McDonald
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex by Venkatesh Sivasubramanian
Big Data Streams Architectures. Why? What? How? by Anton Nazaruk
Ingesting Healthcare Data by Micah Whitacre
Make Streaming IoT Analytics Work for You by Kanishk Mahajan & Dhruv Kumar
Apache NiFi Deep Dive by Bryan Bende
Lambda Architecture with Apache Spark by Taras Matyashovsky

New Videos

Reactive Stream Processing with Akka by Abheek Das Gupta
The Raise of Real-Time by Jay Kreps
Building an Event-oriented Data Platform with Kafka by Eric Sammer
Conquering All Your Stream Processing Needs with Kafka and Spark by Tathagata Das
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex by Venkatesh Sivasubramanian
Emergence of Real-Time Analytics: Real-time Analysis of Customer Financial Activities With Apache Flink by Srinivas Palthepu
Untangling Healthcare with Spark and Dataflow by Ryan Brush
Google Dataflow : The new open model for batch and stream processing by Sara Robinson & Felipe Hoffa
Make Streaming IoT Analytics Work For You: The Devil is in the Details by Kanishk Mahajan & Dhruv Kumar
Water Conservation with Apache Quarks, Raspberry Pi and IBM Streams by Apache Quarks
Hangout: Apache Quarks on Raspberry Pi by Apache Quarks

New Releases

Upcoming Events

5/17/2016 (Mountain View, US) – May Kafka Meetup (Meetup)
5/17/2016 (Berlin, GE) – AWS User Group Meetup @ Lieferando (Meetup)
5/20/2016 (Madrid, SP) – Workshop Apache Flink (Meetup)
5/20/2016 (online) – Spark Streaming Made Easy! (StreamAnalytix Webinar)
5/24/2016 (online) – Apache Kafka and Real Time Stream Processing (Gluent Webinar)
5/26/2016 (online) – Productionizing your Streaming Jobs (Databricks Webinar)
6/1-3/2016 (London, UK) – Real-time conference sessions (Strata + Hadoop World)
6/1/2016 (London, UK) – Apache Spark Users In London (Meetup)
6/5-7/2016 (Berlin, GE) – Berlin Buzzwords
6/6/2016 (San Francisco, US) – Spark Meetup At Spark Summit (Meetup)
6/13/2016 (Amsterdam, NL) – GOTO Night: Stream Processing with Apache Flink and Mining Github (Meetup)
6/13/2016 (New York, US) – Apache Beam (Stream Processing @ Scale Track at QCon New York)
7/5/2016 (San Francisco, US) – Building (and running) Netflix’s Data Pipeline using Apache Kafka (Meetup)
8/18/2016 (New York, US) – Apache NiFi – MiNiFi: Taking Dataflow Management to the Edge (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:02 on March 21, 2016
Tags: azure-stream-analytics, kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 3/21/2016

This is the 6th installment of my blog series around Stream Processing and Analytics.

This week I would like to quote two articles, one from Forbes Tech and another from Information Week.

In the 17 Predictions about the future of Big Data … Forbes compiles 17 predictions from well known analysts, mentioning fast data as one of the next trends:

“real-time streaming insights into data will be the hallmarks of data winners going forward”
“autonomous agents and things will continue to be a huge trends”
“Fast Data and actionable data will replace big data”

According to this article, Forrester defines streaming analytics software as technology that “can filter, aggregate, enrich, and analyze a high throughput of data from multiple disparate live data sources and in any data format to identify simple and complex patterns to visualize business in real-time, detect urgent situations, and automate immediate actions.“.Forrester also identified key vendors and technology for streaming analytics, which are Apache Spark Streaming, Apache Storm, Data Torrent, IBM, Informatica, SAP, Software AG, SQLstream, Strim (WebAction), TIBCO, and Vitria.

I’m wondering why Oracle’s Stream Explorer product is not on Forrester’s radar. Especially because it can be used for both of the two Big Data’s priorities: Streaming Analytics and Self Service.

As usual, just find what I have noticed last week:

News and Blog Posts

General

7 conditions for stream processing by Dana Sandu
Six Best Practices for Real-Time Analytics by Christy Pettey (Gartner)
Making IoT Payoff With Big Data Streaming Analytics by John Haddad
The IoT is a natural ecosystem for streaming analytics by Timothy McGovern
Batch vs. Stream Processing by Luis Claudio Rodrigues da Silveira
Big Data’s Priorities: Streaming Analytics, Self-Service by Jessica Davis
Closed Loop: Machine Learning and Real Time Stream Processing by Kai Wähner
Business Goes for Streaming Analytics by the Zorba Research Blog
17 Predictions About The Future Of Big Data Everyone Should Read by Bernard Marr

Comparison

Streaming Big Data: Storm, Spark and Samza by BlogT

Apache Spark Streaming

The Art of Aggregation With Kafka and Spark: Part 1 by Patrick Shields
The Art of Aggregation With Kafka and Spark: Part 2 by Patrick Shields

Apache NiFi / Hortonworks DataFlow

ExecuteScript – JSON-to-JSON Revisited (with Jython) by Matt Burgess
Twitter Tweets – Apache Nifi combined with MongoDb, Groovy, Velocity and Highcharts by Uwe Geercken

Apache Kafka

Message Hub: Apache Kafka as a Service by Rajini

Goggle Cloud Dataflow / Apache Beam

Clarifying & Formalizing Runner Capabilities by Frances Perry & Tyler Akidau
Apache Beam Capability Matrix by Apache Beam documentation

Hortonworks DataFlow / Apache NiFi

StreamAnalytix from @ImpetusTech Selected for @Hortonworks DataFlow

StreamSets

Visualizing Apache Log Data… with Minecraft! by Pat Patterson
What’s the Biggest Lot in the City of San Francisco? by Pat Patterson
New Release of StreamSets Data Collector Takes on Data Drift by Datafloq
StreamSets Achieves Certification With MapR Converged Data Platform With Extended Support for MapR Streams by StreamSets blog
How Trend Micro Uses StreamSets – An Interview with the Threat Research Team by Kirit Basu

IBM Quarks

Data Streaming at the Edge: IBM and Apache Quarks by Jusin Grammens

Microsoft Azure Stream Analytics

Innovative Azure Stream Analytics customer use cases by Siddhika Nevrekar
Stream Analytics Data Lake Store output by Jeff Stokes
SQL Data Warehouse as output of Azure Stream Analytics by Kati Iceva

SAS Event Stream Processing

Hortonworks HDP and SAS Event Stream Processing together, using YARN by Mark Lochbihler

New Presentations

Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spark Streaming by Evan Chan and Helena Edelson
Introduction to Apache Beam by Jean-Baptisste Onofré
Real time data viz with Spark Streaming, Kafka and D3.js by Ben Laird

New Videos

Building Real-Time Data Pipelines with Spark Streaming, Kafka, and Cassandra by Nik Rouda and Nanda Vijaydev

New Releases / Components

none

Upcoming Events

3/23/2016 (Chicago, US) – Real-Time Analytics on Customer Financial Activities With Apache Flink (Flink Meetup)
3/23/2016 (Online) – How to Effectively Manage IoT DataFlows with HDF (Hortonworks Webinar)
3/24/2016 (Hanover, US) – March NiFi Meetup (Meetup)
3/29/2016 (Chicago, US) – Battle Royale: Hadoop vs. Spark vs. Google Dataflow (ITA Tech Talk)
4/6/2016 (Berlin, UK) – Big Data & Real Time Analytics at Idealo.de ! (Meetup)
4/13/2016 (London, UK) – Real time search and insights with Apache Kafka (Meetup)
4/14/2016 (Chicago, US) – Jay Kreps: Apache Kafka and the Confluent Platform – Overview and Roadmap (Meetup)
4/18/2016 (Munich, GE) – Batch- & Stream-Processing mit Google Dataflow (Smart Data Developer Conference)
4/20/2016 (London, UK) – Kafka, Samza and HBase at the Home Office / Kafka as a Service on IBM’s Bluemix (Unified Log London Meetup)
4/21/2016 (Leidschendam, NL) – Apache Kafka: A guide to integrate and build data processing pipelines (Meetup)
4/25/2016 (San Francisco, US) – Kafka Summit
5/3/2016 (Flemington, US) – Apache NiFi – Deep Dive (Meetup)
5/10/2016 (New York, US) – The Big Heist: Attempting to steal distributed systems from complexity and chaos (Scala Days)
6/1-3/2016 (London, UK) – Real-time conference sessions (Strata + Hadoop World)
6/5-7/2016 (Berlin, GE) – Berlin Buzzwords
6/13/2016 (Amsterdam, NL) – GOTO Night: Stream Processing with Apache Flink and Mining Github (Meetup)
6/13/2016 (New York, US) – Apache Beam (Stream Processing @ Scale Track at QCon New York)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!