striim | Enjoy IT - SOA, Java, Event-Driven Computing and Integration

This is the 10th installment of my blog series around Stream Processing and Analytics.

Two days later than planned, was traveling and had again trouble with my power supply 😦

So what happened in the world of Stream Processing? For me the most interesting news last week was the release of Storm 1.0.

Apache Storm 1.0 released!https://t.co/fXWKposkt5

— P. Taylor Goetz (@ptgoetz) April 12, 2016

I’m a storm user for more than 3 years now and this is really a significant release that delivers several features that pertain to enterprise readiness, operational simplicity and ease of use. I really like that Storm now has native Windowing and State Management Support, Automatic Back Pressure Support and the new connectors for Cassandra, Elasticsearch and Kafka.

Nathan Marz, the founder and creator of Storm also tweeted about it:

From humble beginnings at a tiny startup more than 5 years ago, Storm has matured to have a big impact across the whole world

— Nathan Marz (@nathanmarz) April 12, 2016

And Ian Hellström already updated his stream processing overview chart with Storm 1.0.0.

apache-streaming6

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Streaming Analytics Will Transform The Internet Of Things Into The Internet Of Analytics by Rowan Curran
Hadoop Summit 2016: Data in motion governs Hortonworks roadmap by Brian McKenna
Amazing Big Data At NASA: Real Time Analytics 150 Million Miles From Earth by Bernard Marr
15 “True” Streaming Analytics Platforms For Real-Time Everything by Mike Gualtieri
Case Studies: How are Enterprises Using Streaming Analytics by insideBIGDATA
Streaming Analytics Explained by Andrew Brust
Making Sense of Stream Processing – A Must Read by Leeor Engel
Streaming Analytics: Predictions of the Future by BI News
Real-Time Event Streaming: What Are Your Options? by Ankur Desai
Counting in Streams: A Hierarchy of Needs by Kostas Tzoumas
Streaming Data Who’s Who: Kafka, Kinesis, Flume, and Storm by Jake Dolezal

Apache Beam

Apache Beam wants to be uber-API for big data by Ian Pointer

Apache Storm

Tutorial: Deploying Apache Storm on Docker Swarm by Baqend Tech Blog
Storm 1.0.0 released
Apache Storm Reaches 1.0, Brings Improved Performance, Many New Features by Sergio De Simone
Apache Storm 1.0 packs a punch by Sendur Yegulalp
Apache Storm New UI Features by Kishorkumar Patil
Apache Storm 1.0 milestone includes native streaming window API by Christina Mulligan

Apache Spark Streaming

Real-Time Global Anomaly Detection in IoT with EMC Elastic Cloud Storage (ECS) by Claudio Fahey
Real-Time Global Anomaly Detection in IoT with EMC Elastic Cloud Storage (ECS) – Part 2 by Claudio Fahey
Real-Time Global Anomaly Detection in IoT with EMC Elastic Cloud Storage (ECS) – Part 3 by Claudio Fahey
Spark Streaming added to IBM BigInsights examples project by Chris Snow
Real-Time Aggregation on Streaming Data Using Spark Streaming and Kafka by Anant Asthana
About an example using kafka, spark-streaming, mongodb and twitter by Aironman

Apache Flink

Counting in streams: A hierarchy of needs by Kostas Tzoumas

Apache Kafka

Join Us For The Inaugural Stream Data Hackathon by Ewen Cheslack-Postava
How We Monitor and Run Kafka At Scale by Rajiv Kurian
Decoupling the Data Pipeline with Kafka – A (Very) Simple Real Life Example by Robin Moffatt

Apache NiFi / Hortonworks DataFlow

Inspecting your NiFi DistributedMapCacheServer with Groovy by Matt Burgess
OAuth 1.0A with Apache NiFi (Twitter API example) by Pierre Villard

Apache Metron

Kicking off the Apache Metron Tech Preview 1 Blog Series by George Vetticaden & James Sirota
Apache Metron User Personas and Core Functional Themes by George Vetticaden & James Sirota
Apache Metron for Big Data Cyber Security Analytics by Hortonworks

Striim

Real-Time Financial Transaction Monitoring by Steve Wilkes

StreamSets

Ingesting JSON Data Into Apache Kudu with StreamSets Data Collector by Pat Patterson
Announcing Data Collector ver 1.3.0.0 by Kirit Basu
Startup Spotlight: StreamSets’ Big Data Integration by Loraine Lawson
Visualize Apache Log Data in Minecraft with StreamSets Data Collector by Pat Patterson

IBM Quarks

Quarks: Sending events from Raspberry Pi to Watson IoT Platform by Samanha Chan

New Presentations

How to Build Data Pipelines for Real-Time Applications with SMACK & Apache Kafka by Patrick McFadin
Large-Scale Stream Processing in the Hadoop Ecosystem by Gyula Fora & Marton Balassi
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data Streaming by Michael Rainey
Spark Streaming + Twitter : Analytics to one’s taste by jpaniego
Unified Stream & Batch Processing with Apache Flink by Ufuk Celebi
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident by Julian Hyde
Streaming Data Processing with Apache Storm by Rishabh Jain
Apache NiFi in the Hadoop Ecosystem by Bryan Bende
Fast Data Intelligence in the IoT – real-time data analytics with Spark by Bas Geerdink
Introduction to Real-Time Data Processing by Yogi Devendra

New Videos

Ingest and Stream Processing What will you choose by Pat Patterson & Ted Malaska
Apache NiFi in the Hadoop Ecosystem by Bryan Bende
Large-Scale Stream Processing in the Hadoop Ecosystem by Gyula Fora & Marton Balassi
The Future of Apache Storm by Taylor Goetz
Streaming outlier analysis for fun and scalability by Casey Stella
Unified Stream and Batch Processing with Apache Flink by Ufuk Celebi
Querying the Internet of Things Streaming SQL on Kafka Samza and Storm Trident by Julian Hyde
StreamSets Data Collector with Apache Kafka and Kudu by Pat Patterson
Real time Search on Terabytes of Data Per Day Lessons Learned by Joey Echeverria
Telematics with Hadoop and Nifi by Adam Morton & Simon Elliston Ball
High Performance Stream Processing by Stephane Maldini, Glen Renfo & David Turanski
Introduction to Apache Quarks by Kathy Saunders, Dan Debrunner, Will Marshall, Susan CLine and Kathey Marsden
Distributed Real-Time Stream Processing by Petr Zapletal
Overview of Apache Flink the 4G of Big Data Analytics Frameworks by Slim Baltagi

New Releases / Components

Upcoming Events

4/19/2016 (Chicago, US) – Apache Flink 1.0: A new era for Real-Time Streaming Analytics (Meetup)
4/19/2016 (Santa Clara, US) – Mining MySQL’s Binary Log with Apache Kafka and Kafka Connect (Percona Live)
4/20/2016 (London, UK) – Kafka, Samza and HBase at the Home Office / Kafka as a Service on IBM’s Bluemix (Unified Log London Meetup)
4/20/2016 (Copenhagen, DK) – High performance data flow with a GUI, and guts (Meetup)
4/21/2016 (Austin, (US) – Accelerating Data Ingestion & Streaming Analytics with HDF/NiFi (Meetup)
4/21/2016 (Leidschendam, NL) – Apache Kafka: A guide to integrate and build data processing pipelines (Meetup)
4/21/2016 (Palo Alto, US) – Data in Motion: Simplifying Security & Building Custom Integrations (Meetup)
4/25/2016 (San Francisco, US) – Kafka Summit
4/26/2016 (New York, US) – Building Data Pipelines for Solr with Apache NiFi (Meetup)
4/27/2016 (Laurel, US) – Apache NiFi: Because It Aint Data Science Without the Data (Meetup)
4/28/2016 (Renens, CH) – Spark Streaming: Dealing with State (Meetup)
5/4/2016 (online) – How to Achieve High Throughput for Real-Time Applications with SMACK, Apache Kafka and Spark Streaming (Datastax Webinar)
5/9/2016 (Vancouver, CA) – Streams Track (Big data 2016 conference)
5/10/2016 (New York, US) – The Big Heist: Attempting to steal distributed systems from complexity and chaos (Scala Days)
5/10/2016 (Vancouver, CA) – Using Kafka and Kudu for Fast, Low-Latency SQL Analytics on Streaming Data (Big data 2016 conference)
5/10/2016 (Flemington, US) – Apache NiFi – Deep Dive (Meetup)
5/16-20/2 (San Francisco, US) – Pileplines By the Bay (Pipelines By the Bay Conference)
6/1-3/2016 (London, UK) – Real-time conference sessions (Strata + Hadoop World)
6/5-7/2016 (Berlin, GE) – Berlin Buzzwords
6/13/2016 (Amsterdam, NL) – GOTO Night: Stream Processing with Apache Flink and Mining Github (Meetup)
6/13/2016 (New York, US) – Apache Beam (Stream Processing @ Scale Track at QCon New York)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!