Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

2016

This is the 45th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

The Data Dichotomy: Rethinking the Way we Treat Data and Services by Ben Stopford
Interview with Ted Dunning of MapR Expert Interview, Part 1: Ted Dunning of MapR on Advantages and Use Cases of Apache Flink and Apex by Christy Wilson
Expert Interview, Part 2: Ted Dunning of MapR on Apache Storm’s Advantages Over New Kids Flink and Apex by Chris
For the Love of Lambda Architectures (or how a full picture is better than a puzzle) by Dana Sandu
The Difference Between Streaming and Batch Processing by Dana Sandu

Apache Kafka

Kafka Connect on MapR by Ankur Desai

Apache Apex

Machine Learning on Apache Apex with Apache SAMOA by DataTorrent

Apache Beam

Apache Beam Regex by Jesse Anderson

Storm / Heron

Julien Nioche on StormCrawler, Open-Source Crawler Pipelines Backed by Apache Storm by Alexandra Rodrigues

StreamSets

Continuous Data Integration with StreamSets Data Collector and Spark Streaming on Databricks by Pat Patterson
Creating Custom Origin for Streamsets by Yogesh

Apache NiFi / Hortonworks Data Flow (HDF)

Decyphering error messages in Apache NiFi by Timothy Spann

Oracle GoldenGate

GoldenGate for Big Data 12.3 is released ! by Thomas Vengal
Oracle GoldenGate Studio 12.2.1.2.6 Now Available by Soneij Nisharahmed

New Presentations

Streaming in Practice – Putting Apache Kafka in Production by Roger Hoover
Billing the Cloud – Real world Stream Processing by Pierre-Yves Richard
Drone Data Flowing Through Apache NiFi by Timothy Spann
Apache Storm Tutorial by Davide Mazza
Stream Processing as a Foundational Paradigm and Apache Flink’s Approach to It by Stephan Ewen
Streaming datasets for personalization by Shriya Arora
Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Analytics by Aljoscha Krettek

New Videos

Apache Kafka and the Rise of Stream Processing by Guozhang Wang
Interactive data analysis with Spark streaming by Shashidhar Datamantra
End to End Streaming ML Recommendation Pipeline Spark 2 0, Kafka, TensorFlow Workshop by Chris Fregly
Stream processing with Apache Kafka on DC/OS running on Azure by Matias Pihlquist
Streaming in Practice – Putting Apache Kafka in Production by Roger Hoover
Stream Processing as Game Changer for Big Data and IoT by Kai Wähner
Building a Spark Streaming application by Petar Zecevic

Upcoming Events

1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:47 on December 12, 2016
Tags: beam ( 25 ), hdf ( 20 ), heron ( 23 ), IoT ( 10 ), kafka ( 170 ), nifi ( 160 ), storm ( 46 ), Stream Processing ( 163 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 12/12/2016

This is the 44th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

CEP Patterns for Stream Analytics by Sibanjan Das

Apache Kafka

Kafka Connect and Kafka REST API on MapR: Streaming Just Became a Whole Lot Easier! by Ankur Desai
Enterprise Streaming Multi-Datacenter Replication using Apache Kafka by Ewen Cheslack-Postava

Apache Beam

Beam’s Pico WordCount by Jesse Anderson

Apache Flink

November 2016 in Review: Flink Forward 2017, Amazon EMR + Google Dataproc, and kicking off the Flink training series in Germany by Michael Winters

Storm / Heron

Twitter’s real-time stack: Processing billions of events with Heron and DistributedLog by Ofir Shalev

StreamSets

Running Apache Spark Code in StreamSets Data Collector by Pat Patterson
Creating a StreamSets Spark Transformer by Pat Patterson
Creating a Custom Origin for StreamSets Data Collector by Pat Patterson

Apache NiFi / Hortonworks Data Flow (HDF)

Announcing the Availability of Hortonworks DataFlow (HDF) 2.1 by Wei Wang & Haimo Liu

New Presentations

ETL is Dead; Long-Live Streams by Neha Narkhede
Leveraging Mainframe Data for Modern Analytics by Jordan Martz & David Tucker & Keith Reid
Real-Time Streaming with Kafka – for data scientists by Jennifer Rawlins
Stream Processing as Game Changer for Big Data and Internet of Things by Kai Wähner
CEP – simplified streaming architecture by Mathieu Dumoulin
Streaming Architecture to Connect Everything (Including Hybrid Cloud) by Mathieu Dumoulin
Continuous Application with Structured Streaming 2.0 by Jules S. Damji
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi by Bryan Bende
Better together: Fast Data with Apache Spark™ and Apache Ignite™ by Micahel Griggs

New Videos

ETL is Dead; Long-Live Streams by Neha Narkhede
Apache Kafka, Stream Processing, and Microservices by Jay Kreps
Stream-1st Architecture, Apache Flink & Other Emerging Technologies by Ellen Friedman
Containers, DevOps, Microservices, & Kafka Tools by Jonathan Owens & Jose Fernandez
Stream Processing use cases and applications with Apache Apex by Thomas Weise
Big Data Analytics, Machine Learning, Real Time Processing by Kai Wähner

New Releases

Upcoming Events

12/14/2016 (online) – How to Manage Continuous Dataflows for a Customer 360 Application (StreamSets Webinar)
12/14/2016 (online) – Counteract Cyberattacks and Fraud with Streaming Analytics (Striim Webinar)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)
12/16(2016 (Kansas City, US) – Kansas City Big Data Group – Big Data Pipelines/Flows, Use case driven comparison: Apache NiFi & StreamSets (Meetup)
12/17/2016 (Fremont, US) – Spark Saturday – Hands-on Workshop with Apache Spark 2.x on Databricks (Meetup)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 21:26 on December 6, 2016
Tags: flink ( 92 ), heron ( 23 ), IoT ( 10 ), kafka ( 170 ), nifi ( 160 ), storm ( 46 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 12/06/2016

This is the 43rd installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

5 architectural principles for building big data systems on AWS by Conner Forrest
Benchmarking Commit Logs by Brave New Geek
Big data pipeline drives change in Hadoop architecture, development by Jack Vaughan
Big Data week in review: Integrated features rule by Andrew Brust
10 Big Data and IoT Predictions for 2017 by Joyce Wells and Stephanie Simone

Apache Kafka

KIP-98 – Exactly Once Delivery and Transactional Messaging by Jason Gusta
kafka-avro-console-producer – Error registering Avro schema / io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException by Robin Moffatt
Apache Kafka, Data Pipelines, and Functional Reactive Programming with Node.js by Chris Castle
Kafka Connect for Azure IoT Hub by Sam George
Use Apache Kafka with Apache Storm on HDInsight by Larry Franks
Microsoft Azure IoT Connects to Apache Kafka to Manage Device Links by Pedro Hernandez
Quick Recipe for #Kafka Streams in #Clojure by Jason Bell
Log Compaction – Highlights in the Apache Kafka and Stream Processing Community – December 2016 by Apurva Mehta.

Spark Streaming

Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming by insideBIGDATA

Storm / Heron

Storm vs. Heron – Part 2 – Why Heron? A developer’s view by Christian Hollinger

StreamSets

Upgrading From Apache Flume to StreamSets Data Collector by Pat Patterson
Announcing Data Collector ver 2.2.0.0 by Kirit Basu

Apache NiFi / Hortonworks Data Flow (HDF)

Scaling up/down a NiFi cluster by Pierre Villard
Apache NiFi 1.1.0 – Secured cluster setup by Pierre Villard
Data routing, transformation, and system mediation in Big Data & IoT scenarios with Apache NiFi by Roger Carhuatocto

New Presentations

Apache Flink Community Updates November 2016 by Robert Metzger
A Practical Guide to Selecting a Stream Processing Technology by Michael Noll
Graph Stream Processing : spinning fast, large scale, complex analytics by Paris Carbone
A look at Flink 1.2 by Stefan Richter
Online Tweet Sentiment Analysis with Apache Spark by Davide Nardone
Fast Data processing with kafka, rfx and redis by Trieu Nguyen
Introduction to Apache NiFi And Storm by Jungtaek Lim
Apache Kafka – Scalable Message-Processing and more ! by Guido Schmutz
IoT Architecture – Are Traditional Architectures Good Enough or do we Need New Approaches? by Guido Schmutz
Deep Dive of Kafka to HDFS/Hadoop Ingestion App Template by Chaitanya Chebolu
Architecting next generation big data platform by Mark Grover & Ted Malaska & Jonathan Seidman

New Videos

Graph Stream Processing : spinning fast, large scale, complex analytics by Paris Carbone
A Practical Guide to Selecting a Stream Processing Technology by Michael Noll
How Stream-First Architecture & Emerging Technologies Provide a Competitive Edge by Ellen Friedmann
Distributed Commit Logs with Apache Kafka by James Ward
Getting it Right Exactly Once: Principles for Streaming Architectures by Darryl Smith
Real-time Data Processing Using AWS Lambda by Cecila Deng & Andreas Fritz & Marco Pierleoni
JustGiving: Serverless Data Pipelines, ETL & Stream Processing by Richard Freeman
Twitter Heron on YARN/REEF by Ashvin Agrawal & Avrillia Floratou & Karthik Ramasamy
Analyzing Streaming Data in Real-time with Amazon Kinesis Analytics by Allan MacInnis
Beeswax: Building a Real-Time Streaming Data Platform on AWS by Ryan Nienhuis & Ram Kumar
IoT Analytics: Insights for a Connected World by Ryan Burke & Bo Mangels & Aubry Wing & Michael Garcia

New Releases

Upcoming Events

12/6/2016 (Lisbon, PT) – A Deep-dive into Structured Streaming (Meetup)
12/7/2016 (London, UK) – Structured logging: The Future of log management? (Meetup)
12/8/2016 (New York, US) – Streaming Analytics in a Flash – presented by Cask Data (Meetup)
12/8/2016 (Atlanta, US) – Building Event Data Pipelines with Kafka and Hadoop (Meetup)
12/8/2016 (Singapore, SG) – Stream Processing and TensorFlow 101 (Google Meetup)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)
12/17/2016 (Fremont, US) – Spark Saturday – Hands-on Workshop with Apache Spark 2.x on Databricks (Meetup)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!