Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:06 on April 26, 2017
Tags: apex, azure-stream-analytics ( 7 ), flink ( 92 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), streamsets ( 74 )

Last week in Stream Processing & Analytics – 26.4.2017

This is the 63rd edition of my blog series blog series around Stream Processing and Analytics! A few days later than planed, but I had some busy days with speaking on Apache Kafka at Oracle Code Berlin on Monday and today we held our Trivadis/DataStax Hands-On workshop in Zurich. So next week will be again on time (Monday), I hope 😉

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Comparison

Event time processing in Apache Spark and Apache Flink by Matthias Niehoff

Apache Kafka / Kafka Streams / Confluent Platform

Kafka channels the big data firehose by Tony Baer
Introducing Kasper: The Kafka Stream Processing Library Built For Go by Nicolas Maquet
Benchmarking Apache Kafka with Aiven – Part 1: Write throughput (AWS vs Azure vs DigitalOcean vs Google Cloud vs UpCloud) by Aiven

Spark Streaming

Fault tolerance in Apache Spark – Reliable Spark Streaming by Data Flair
Real-Time Weather Event Processing With HDF, Spark Streaming, and Solr by Balaji Kandregula
Spark Streaming – under the hood by veejayendraa
Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2 by Kunal Khamar & Tyson Condie & Michael Armbrust

Apache Apex

One Year Anniversary of Apache Apex by Thomas Weise

Apache Flink

Build a Real-time Stream Processing Pipeline with Apache Flink on AWS by Steffen Hausmann

StreamSets

StreamSets Data Collector v2.5 Adds IoT, Spark, Performance and Scale by Kirit Basu
Make sense of stream processing by Pat Patterson
Creating a Custom Multithreaded Origin for StreamSets Data Collector by Pat Patterson

Apache Nifi

Basics of Apache Nifi: 1 by Malcom Thirus
How to send data from Apache NIFI to HANA by Franco Lezana

Azure Stream Analytics

Announcing Azure Stream Analytics on edge devices (preview) by Santosh Balasubramanian

New Presentations

Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema
Kafka used at scale to deliver real-time notifications by Sergio Nunes
Apache Storm In Retail Content by Karthik Deivasigamani
Storm worker redesign by Roshan Naik
From Three Nines to Five Nines – A Kafka Journey by Allen Wang
Microsoft challenges of a multi tenant kafka service by Nitin Kumar
The Rise of Real-Time by Neha Narkhede
Apache Kafka – Scalable Message-Processing and more ! by Guido Schmutz
Building Streaming Applications with Apache Storm 1.1 by Hugo Louro
Kafka Tutorial – basics of the Kafka streaming platform by Jean-Paul Azar
Bootstrapping Microservices with Kafka, Akka and Spark by Alex Silva

New Videos

Building Faster Streaming Applications with Apache Storm 1.1 by Suresh Srinivas & Hugo Louro & Karthik Deivasigamani
Putting the Micro into Microservices with Stateful Stream Processing by Ben Stopford
StreamING models, how ING adds models … by Erik de Nooij
Real-Time Analytics On Data Streams – A New Era For Big Data And IoT by Jeremy Hillier
Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema

New Podcasts

Data science for real-time streaming analytics by Roger Rea

New Releases

StreamSets 2.5

Upcoming Events

25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)
27.4.2017 (Charlotte, US) – Apache NiFi – An Uber Data Flow Tool (Meetup)
16.5.2017 (online) – Geo-Distributed Big Data and Analytics (MapR Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 16:42 on April 11, 2017
Tags: apex, flink ( 92 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics – 10.4.2017

This is the 61st edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Lloyds Banking Group prepares for open banking by shifting towards realtime data feeds by Scott Carey
Machine Learning, Microservices, Streaming Data, IoT, Cloud & Containers by Ellen Friedman
Architecting and building end-to-end streaming applications by Ben Lorica
How Real Time Stream Processing Changes Everything About Big Data by Shelly Kramer
Firing on All Cylinders: The 2017 Big Data Landscape by Matt Turck
The Next Data Revolution: Intelligent Real-Time Decisions by Ion Stoica
Beyond Big Data Basics: In-Stream Processing Cures Batch Processing Blues by Robin Miller

Apache Kafka / Kafka Streams / Confluent Platform

New Quick Start deploys Confluent Platform on the AWS Cloud by AWS blog
Python Kafka Client Benchmarking by Activision Game Science
Kafka REST Proxy – Performance Tuning for MapR Streams by Mathieu Dumoulin
Kafka Connect with Couchbase by David Maier
Running Kafka Streams applications against a containerized Kafka cluster with Confluent Open Source by Michal Noll
Kafka Summit NYC: Streaming Pipelines Track – What to expect by Becket Qin & Gwen Shapira

Spark Streaming

Real-Time End-to-End Integration with Apache Kafka in Apache Spark’s Structured Streaming by Sunil Sitaula

Apache Storm

Video Tutorial – StormCrawler + Elasticsearch + Kibana by Julien Nioche

Apache Flink

Continuous Queries on Dynamic Tables by Fabian Hueske
Flink Aims to Simplify Stream Processing by Alex Woodie
Apache Flink: Does the world need another streaming engine? by Tony Baer

Apache NiFi

Data pipelines with NiFi as a Storm alternative by Samra Kasim

New Presentations

Dealing With Drift – Building an Enterprise Data Lake by Pat Patterson
Stream Processing with Apache Kafka and .NET by Matt Howlett
Simplify Governance of Streaming Data by Gwen Shapira
Implementation of the Lambda Architecture using Apache Apex, Kafka & Impala by Tim Zhu & Sunil Parmar
Kafka pours and Spark resolves by Alexey Zinoviev
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi by Aldrin Piri
IoT pipeline for anomaly detection at ml kitchen by Mathieu Dumolin & Mateusz Dymczyk
The Data Dichotomy: Rethinking Data and Services wih Streams by Ben Stopford
Streaming SQL by Julian Hyde
Real-Time Visualizations to Apex Apps by Sasha Parfenov
IoT Stream Processing – Processing Data Streams from Internet of Things by Subu Balakrishnan
Tibco streaming analytics overview and roadmap by Lou Bajuk
Build a Real-time Streaming Data Visualization System with Amazon Kinesis Analytics by Allan Macinnis
Storm/Spark Streaming vs. Apache Apex Comparison by Devendra Tagare
Machine Learning in the IoT with Apache NiFi by Michael Bironneau
Introduction to Streaming with Apache Flink by Tugdual Grall

New Podcasts

Jay Kreps on streaming data, the power of Kafka and a whole lot more by ArchiTECHt
Architecting and Building End-to-End Streaming Applications by O’Reilly Radar

Upcoming Events

11.4.2017 (online) – Real-World Best Practices for Big Data Ingestion (Make BigData Work Webinar)
11.4.2017 (Palo Alto, US) – Streaming Data at Uber & A Streaming platform architecture based on Apache Kafka Streaming Data at Uber & A Streaming platform architecture based on Apache Kafka (Meetup)
18.4.2017 (Zurich, CH) – Apache Flink Zurich Kickoff (Meetup)
19.4.2017 (London, UK) – Transitioning from ETL to Streaming – Neha Narkhede (Meetup)
20.4.2017 (London, UK) – Transitioning from ETL to Streaming – Neha Narkhede (Meetup)
25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:09 on April 3, 2017
Tags: apex, flink ( 92 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics – 3.4.2017

This is the 60th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

How big, fast data is transforming business intelligence by Jon Bruner

Apache Kafka / Kafka Streams / Confluent Platform

Applying Kafka Streams to the Purchase Transaction Flow by bbejeck
A Change Data Capture Pipeline From PostgreSQL to Kafka by Jeff Klukas
Securing the Confluent Schema Registry for Apache Kafka by Stephane Maarek
Benchmarking Kafka Performance Part 1: Write Throughput by Aiven
Kafka for the Oracle DBA by Chris Lawless

Apache Storm

Apache Storm 1.1.0 Released by Taylor Goetz

Apache Flink

From Streams to Tables and Back Again: An Update on Flink’s Table & SQL API by Timo Walther

Apache NiFi

Getting Familiar: the ExecuteScript Processor – Groovy by Nifi.rocks
Making Spark and Kafka Data Pipelines Manageable with Tuning by Larry Murdock
Monitor Apache NiFi with Apache NiFi by Timothy Spann
Collect data from OPC UA protocol by wsalazar

New Presentations

User Behavior Analysis with Session Windows and Apache Kafka’s Streams API by Michael Noll
Building Event-Driven Services with Apache Kafka by Ben Stopford
What’s new with Apache Spark’s Structured Streaming? by Miklos Christine
Easy, scalable, fault tolerant stream processing with structured streaming by Tathagata Das
Streaming Analytics in Uber by Danny Yuan
How To Get Monitoring Right For Streaming & Fast Data Systems Built With Spark, Mesos, Akka, Cassandra & Kafka by Paul Jasek
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application by Munagala Ramanath
Apache NiFi: Ingesting Enterprise Data At Scale by Timothy Spann

New Videos

Bay Area Apache Spark Meetup @ Intel
Reactive Kafka with Akka Streams by Scala Matsuri

New Releases

Upcoming Events

4.4.2017 (Charlotte, US) – Kafka for Real-Time Data Ingestion with .NET (Meetup)
5.4.2017 (London, UK) – Realtime stream processing with coroutines (Meetup)
5.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO
6.4.2017 (San Francisco, US) – Building a Stream Processor for event-driven applications (Meetup)
6.4.2017 (Stockholm, SE) – Simple realtime analytics with Spark & Build the simplest data pipeline (Meetup)
11.4.2017 (online) – Real-World Best Practices for Big Data Ingestion (Make BigData Work Webinar)
11.4.2017 (Palo Alto, US) – Streaming Data at Uber & A Streaming platform architecture based on Apache Kafka Streaming Data at Uber & A Streaming platform architecture based on Apache Kafka (Meetup)
18.4.2017 (Zurich, CH) – Apache Flink Zurich Kickoff (Meetup)
25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 21:46 on March 27, 2017
Tags: apex, flink ( 92 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics – 27.03.2017

This is the 59th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Distributed Stream Processing Frameworks for Fast & Big Data by
Matthias Niehoff
Fog Computing Architecture Will Set IoT on Better Path by Steve Jennis
Applying the Kappa architecture in the telco industry by Nicolas Seyvet & Ignacio Mulas Viela
At the Speed of Light by Keith Manthey

Apache Kafka / Kafka Streams / Confluent Platform

Apache Kafka on the Oracle Cloud: My First experiences with Oracle Event Hub Cloud Service by Lucas Jellema
Implementing The Schema Registry by Callum Leahy
Flafka: Big Data Solution for Data Silos by Randall V Shane
Kafka with Docker: A Docker introduction by Nikolaos Georgiou
Kafka Streams state stores… by Abhishek

Apache Storm

Taking control of your Apache Storm cluster with tag-aware scheduling by Panos Kasteas
Storm vs. Hadoop by Suresh Rajagopal

Apache Flink

Queryable State in Apache Flink® 1.2.0: An Overview & Demo by Michael Winters & Ufuk Celebi

Apache NiFi

Mass provisioning of Kismet and Apache MiNiFi in Raspberry Pi using Ansible by Roger Carhuatocto

New Presentations

Spark Streaming + Kafka Best Practices by Brandon O’Brien
Design and Implementation of Spark Streaming Connectors by Nan Zhu
Spark and MapR Streams: A Motivating Example by Ian Downard
Data Pipelines Made Simple with Apache Kafka by Ewen Cheslack-Postava
The Rise of Real Time: Apache Kafka and the Streaming Revolution by Jay Kreps
Multi-Datacenter Kafka by Gwen Shapira
JustGiving – Serverless Data Pipelines, API, Messaging and Stream Processing by Luis Gonzalez
A primer on building real time data-driven products by Lars Albertsson
Getting Started with IoT using Azure, Windows IoT Core, and Raspberry Pi by Chris Pietschmann
Smart Partitioning in Apache Apex by Tushar Gosavi
Ingesting Drone Data into Big Data Platforms by Timothy Spann
Building a Real-Time Data Science Service for Mobile Advertising by Robin Li

New Videos

New Releases

Upcoming Events

28.3.2017 (Princeton, US) – Apache NiFi: Ingesting Enterprise Data @ Scale (Meetup)
29.3.2017 (online) – Part 2: Building Event-Driven Services with Apache Kafka (Confluent Webinar)
30.3.2017 (San Francisco, US) – Stream Processing with Apache Kafka and .NET (Meetup)
5.4.2017 (London, UK) – Realtime stream processing with coroutines (Meetup)
5.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO
18.4.2017 (Zurich, CH) – Apache Flink Zurich Kickoff (Meetup)
25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:55 on March 6, 2017
Tags: akka ( 3 ), apex, azure-stream-analytics ( 7 ), flink ( 92 ), kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics – 6.03.2017

This is the 56th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Unlocking the real value of the IoT by Anne-Lindsay Beall
Building the Ideal Stack for Real-Time Analytics by Mason Hooten
Real-Time Stream Processing as Game Changer in a Big Data World by Hossein Eslambolchi
The 3 keys to digital transformation by Jack Norris
Stream processing and the IBM Open Platform by Andrea Braida

Apache Kafka / Kafka Streams / Confluent Platform

Confluent 3.2 with Apache Kafka 0.10.2. Now Available by Chrix Finne
Kafka Connect Elasticsearch: Consuming and Indexing with Kafka Connect by Rafal Kuć
Configuring and Running Apache Kafka in IBM BigInsights by Nisanth Simon
Monitoring Consumer Offsets in #Kafka by Jason Bell

Apache Flink

Running Apache Flink® Everywhere: Flink on DC/OS and Apache Mesos by Till Rohrmann

StreamSets

StreamSets with Docker – an example HDFS integration by Dominic Ketley
Announcing StreamSets Data Collector ver 2.4.0.0 by Kirit Basu
Read and Write JSON to MapR DB with StreamSets Data Collector by Rupal Shah
Drift Synchronization with StreamSets Data Collector and Azure Data Lake by Pat Patterson

Apache NiFi / Hortonworks HDF

Working with S3 Compatible Data Stores via Apache NiFi by Timothy Spann
Using MiniFi to read data from a Sense HAT on a Raspberry Pi 3 by Timothy Spann
Cool Projects in Big Data, Machine Learning, and Apache NiFi by Timothy Sapnn

Azure Stream Analytics

Announcing real-time Geospatial Analytics in Azure Stream Analytics by Samartha Chandrashekar

New Presentations

Kafka’s Role in Implementing Oracle’s Big Data Reference Architecture by Robin Moffatt
StreamSets and Spark by Hari Shreedharan
Introduction to data flow management using apache nifi by Anshuman Ghosh
End-to-End “Exactly-Once” processing using Apache Apex by Ajay Gupta
Real-Time Stream Processing on EMR: Apache Flink vs. Apache Spark Streaming by Keith Steward
In memory analytics with apache spark and hazelcast by Viktor Gamov
Akka-chan’s Survival Guide for the Streaming World by Konrad Malawski
Apache Flink’s Table & SQL API – unified APIs for batch and stream processing by Timo Walther

New Videos

StreamSets and Spark by Hari Shreedharan
Building an Open Source Streaming Analytics Stack with Kafka and Druid by Fangjin Yang
Streaming all the things with Akka Streams by Johan Andrén
Real-time Stream Processing on EMR: Apache Flink vs. Spark Streaming by Keith Steward
Secure IoT Command, Control, and Exfil with Apache MiNiFi by Andy LoPresto
End-to-End “Exactly-Once” processing using Apache Apex by Ajay Gupta

New Releases

New Books

Kafka Streams in Action by William Bejeck

Upcoming Events

7.3.2017 (Santa Monica, US) – Stream processing with R in AWS (Meetup)
7.3.2017 (Austin, US) – Stream All the Things! (Meetup)
14.3.2017 (Palo Alto, US) – Apache Kafka Meetup at Strata San Jose (Meetup)
15.3.2017 (Paris, FR) – Apache Flink: Use Cases (Meetup)
21.3.2017 (online) – Getting Started with Streaming Data and Stream Processing with Apache Kafka (KPI Webinar)
23.3.2017 (San Francisco, US) – Bay Area Apache Spark Meetup @ Intel in Santa Clara (Meetup)
28.3.2017 (Princeton, US) – Apache NiFi: Ingesting Enterprise Data @ Scale (Meetup)
29.3.2017 (online) – Part 2: Building Event-Driven Services with Apache Kafka (Confluent Webinar)
5.4.2017 (London, UK) – Realtime stream processing with coroutines (Meetup)
25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:57 on February 6, 2017
Tags: apex, beam ( 25 ), flink ( 92 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics – 06.02.2017

This is the one year anniversary edition of my blog series blog series around Stream Processing and Analytics! 🙂

From this week onward, I’ll be updating the following two lists with all the presentations and videos I mention each week:

Starting with this edition, it is now Europe first 😉 I’m changing all date formats to the “European rule”. It is definitely less confusing for me!

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Drinking from the industrial IoT data fire hose by Andy Oram
An Interview With the SMACK Stack by Harmeet Singh
Stream Processing Myths Debunked by Kostas Tzoumas & Michael Winters
Main “streaming” of streaming analytics by Debjani Deb
Oracle lists top 10 Big Data predictions for 2017 by Infotech Lead
TensorFlow on the Edge, Part 1 of 5 by Tim Spann

Spark Streaming

How to Shutdown a Spark Streaming Job Gracefully by Lan Jiang

Apache Kafka / Kafka Streams

Processing billions of events/day by Nikita Salnikov-Tarnovski
Perfecting Lambda Architecture with Oracle Data Integrator (and Kafka / MapR Streams) by Issam Hijazi
An inside look at why Apache Kafka adoption is exploding by Matt Asay

Apache Flink

Using Apache Flink With Java 8 by Ivan Mushketyk
Going with the stream: Unbounded data processing with Apache Flink by George Anadiotis
Announcing Apache Flink 1.2.0 by Robert Metzger

Apache Beam

Media recap of the Apache Beam graduation by Davor Bonaci

Apache NiFi / Hortonworks HDF

NiFi and OAuth 2.0 to request WordPress API by Pierre Villard
Adding Stanford CoreNLP To Big Data Pipelines (Apache NiFi 1.1/HDF 2.1) Part 1 of 2 by Timothy Spann
Adding Stanford CoreNLP To Big Data Pipelines (Apache NiFi 1.1/HDF 2.1) Part 2 of 2 by Timothy Spann
Open NLP Example Apache NiFi Processor by Timothy Spann

StreamSets

Announcing Data Collector ver 2.3.0.0 by Kirit Basu
Replicating Relational Databases with StreamSets Data Collector by Pat Patterson

New Presentations

Distributed Stream Processing with Apache Kafka by Jay Krebs
Big Data Processing by Stefan Papp
Streaming Transformations Using Oracle Data Integration by Michael Rainey
Introducing Apache Kafka’s Streams API by Michael Noll
Building Kafka-powered Activity Stream by Olesksiy Holubyev
Apache Apex (Next Gen Hadoop) vs. Storm – Comparison and Migration Outline by Devendra Tagare
Kafka’s Role in Implementing Oracle’s Big Data Reference Architecture on the Big Data Appliance by Robin Moffatt
Not Less, Not More: Exactly Once, Large-Scale Stream Processing in Action by Paris Carbone

New Videos

Reactive Kafka by Rajini Sivaram
IoT im Big-Data-Umfeld mit Vert.x-MQTT, Kafka und Apache Spark by Jochen Mader & Jens Deters (in german)
Real-Time Dashboards for Apache Apex (Next Gen Hadoop) apps by Sasha Parfenov
Anomaly Detection Using Metrics and Exception Logs | Whiteboard Walkthrough by Ted Dunning
Streaming Data Analytics with Apache Spark Streaming by Data Gurus

New Releases

New Code Samples

Flink Event Time Example

Upcoming Events

7.2.2017 (Minneapolis, US) – Apache Kafka and Kafka Streams (Meetup)
9.2.2017 (New York, US) – Crunching streams of data, an introduction to Akka streams (Meetup)
10.2.2017 (London, UK) – Intro to Apache Apex, the Next Gen Hadoop Platform for Ingest & Transform (TechWrap Meetup)
16.2.2017 (Mountain View, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
16.2.2017 (Berlin, GE) – Apache Flink Meetup @ResearchGate (Meetup)
21.2.2017 (Karlsruhe, GE) – Let’s talk about Apache Flink 1.2 – and put it in a container! (Meetup)
21.2.2017 (Chicago, US) – Hands on Apache Flink workshop! (Meetup)
22.2.2017 (London, UK) – A tutorial: Get your hands on implementing a Flink app (Meetup)
23.2.2017 (Chicago, US) – Building streaming data applications using Kafka*[Connect + Core + Streams] (Meetup)
2.3.2017 (Amsterdam, NL) – Apache Flink’s Stateful Operators And Table SQL Api (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 20:30 on January 31, 2017
Tags: apex, flink ( 92 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 01/30/2017

This is the 51st installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Towards a realtime streaming architecture by Alice Kaerast
Streaming hot: Real-time big data architecture matters by George Anadiotis
Real-time big data analytics brings change to data management by Jack Vaughan
Streaming Analytics for Chain Monitoring by Roberto Zicari

Comparison

Stream processing and the IBM Open Platform by Andrea Braida
Analytics on Fast Data: Main-Memory Database Systems versus Modern Streaming Systems by Adreas Kipf et. al.

Spark Streaming

Stream Processing – Simple Get-Start with Kafka and Spark by visualskyrim

Apache Kafka / Kafka Streams

Transforming and Aggregating Kafka Messages With Kafka Streams by Basanth Roy
Comparing Confluent Platform with Traditional Messaging Middleware by Hans Jespersen
Transforming and Aggregating Kafka Messages With Kafka Streams by Basanth Roy
Confluent Delivers Upgrades to Clients, Kafka Streams, Brokers and Apache Kafka™ 0.10.1.1 by Ewen Cheslack-Postava
Kafka – Rewind Consumer Offsets by Jorge Esteban Quilcate Otoya
How to Create Serializers With Kafka by Prabhat Kashyap

Apache Flink

Streaming Windows in Apache Flink – A Comprehensive Guide by DF HD Team
Apache Flink® User Survey 2016 Results, Part 2 by Michael Winters

Apache Beam

In a way, Apache Beam is the glue that connects many big data systems together by Kypriani Sinaris

Apache NiFi / Hortonworks HDF

Could We Finally Have a Secure IoT, Courtesy of the NSA? by Joel Hans
How to send file from localhost to AWS EC2 using NiFi Site-to-Site? by ijokarumawak
Syslog User login Analytics using Apache NiFi and Zeppelin by zhoussen
Why NiFi? by Doug Mengistu
Cool Projects in Big Data, Machine Learning, and Apache NiFi by Tim Spann

New Presentations

SQL on Apache Apex by Chinmay Kolhatkar
Apache Kafka lessons learned @PAYBACK by Maxim Shelest
The hardest part of microservices: your data by Christian Posta
Recipes for building resilient cross-DC data pipeline with Kafka by Suneet Grover
State of the Streaming Platform 2017 What’s new in Apache Kafka and the Confluent Platform by David Tucker
Kafka Connect, Scalable, Fault tolerant ETL for streams by Andrew Stevenson
Lightbend Fast Data Platform by Dean Wampler
What no one tells you about writing a streaming app by Mark Grover & Ted Malaska
Extracting Insights from Data at Twitter by Prasad Wagle
Real-Time Processing Using AWS Lambda by Cecilia Deng
Real-time Dashboards for Apache Apex by Sasha Parfenov
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application by Munagala Ramanath
Storm over Gearpump by Tianlun Zhang

New Videos

SQL on Apache Apex by Chinmay Kolhatkar
Reactive Kafka by Rajini Sivaram
Stream processing with Apache Kafka on DC/OS running on Azure by Michael Hausenblas
Deploying Apache Ignite and GridGain by Christos Erotocritou & Rachel Pedreschi

New Releases

Streamparse 3.4.0

Upcoming Events

1/31/2017 (online) – Enabling Next Gen Analytics with Azure Data Lake (StreamSets Webinar)
1/31/2017 (London, UK) – Microservices, The Data Dichotomy: Rethinking Data & Services with Streams (Meetup)
2/1/2017 (Fremont, US) – Understanding Big Data Streaming and Apache Flink (Meetup)
2/2/2017 (Sunnyvale, US) – Cloud Network Engineering (Meetup)
2/7/2017 (Minneapolis, US) – Apache Kafka and Kafka Streams (Meetup)
2/9/2017 (New York, US) – Crunching streams of data, an introduction to Akka streams (Meetup)
2/16/2017 (Mountain View, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
2/22/2016 (London, UK) – A tutorial: Get your hands on implementing a Flink app (Meetup)
3/2/2017 (Amsterdam, NL) – Apache Flink’s Stateful Operators And Table SQL Api (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 08:07 on January 17, 2017
Tags: apex, kafka ( 248 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 01/16/2017

This is the 49th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Better Complex Event Processing at Scale Using a Microservices-based Streaming Architecture (Part 1) by Mathieu Dumoulin
Real-time Smart City Traffic Monitoring Using Microservices-based Streaming Architecture (Part 2) by Mathieu Dumoulin
Apache Beam and Spark: New coopetition for squashing the Lambda Architecture? by Tony Baer
Streaming Analytics for Chain Monitoring by Nataliono Busa

Comparison

Kafka vs. MapR Streams: Why MapR? by Ian Downard

Spark Streaming

Getting Started with Spark Streaming, Python, and Kafka by Robin Moffatt
Data Processing and Enrichment in Spark Streaming with Python and Kafka by Robin Moffatt
Apache Spark 2.1 Improves Structured Streaming by David Ramel

Apache Kafka

Apache Kafka: Getting Started by Gwen Shapira
Scaling Kafka with Docker Containers by Jorge Quilcate
Performance considerations when using Apache Kafka with SSL/TLS by Emanuel Muhr
Importing JSON into Hadoop via Kafka by Nuria Ruiz & Andrew Otto

Apache Storm

Installing Apache Storm on Windows by Aikansh Manchanda

Apache Apex

SQL on Apache Apex by Sidhartha Bardoloye

Apache Beam

The Apache Software Foundation Announces Apache® Beam™ as a Top-Level Project by Apache Software Foundation
Apache Beam graduates from incubation: Try it today on Google Cloud Dataflow by Frances Perry
Release 0.4.0 adds a runner for Apache Apex by Thomas Weise
Apache Beam established as a new top-level project by Davor Bonaci

Apache NiFi / Hortonworks HDF

Ingest Remote Camera Images from Raspberry Pi via MQTT and FTP in Apache NiFi by Timothy Spann
Creating An Email Bot in Apache NiFi by Timothy Spann

New Presentations

Introduction to Kafka Streams with a Real-Life Example by Alexis Seigneurin
Spring for Apache Kafka by Gary Russell
Stream Processing as a Foundational Paradigm and Apache Flink’s Approach to It by Stephan Ewen

New Videos

Spring for Apache Kafka by Gary Russell
Top 5 IoT Use Cases by Vijay Raja & Dave Shuman

New Releases

StormCrawler 1.3

Upcoming Events

1/17/2017 (Amsterdam, NL) – Sensor Data Ingestion and Processing with NiFi and Spark (Meetup)
1/17/2017 (Chicago, US) – Fast Data: Selecting The Right Streaming Technologies For Never Ending Data Sets (Meetup)
1/17/2017 (online) – Solving the Really Big Tech Problems with IoT (HP Webinar)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)
1/18/2017 (Milano, IT) – TensorFlow & TensorFrames with Apache Spark + A Deep-dive into Structured Streaming (Spark Hub)
1/18/2017 (Brussels, BE) – Running Kafka in production and Streamsets (Meetup)
1/19/2017 (New York, US) – Fast Data / Event-Driven Architecture (Meetup)
1/23/2017 (Zurich, CH) – 19th Swiss Big Data User Group Meeting (Meetup)
1/25/2017 (Munich, GE) – Apache Kafka with Jay Kreps (Meetup)
1/31/2017 (online) – Enabling Next Gen Analytics with Azure Data Lake (StreamSets Webinar)
2/1/2017 (Fremont, CA) – Understanding Big Data Streaming and Apache Flink (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 17:23 on January 3, 2017
Tags: apex, fastdata, flink ( 92 ), kafka ( 248 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 01/03/2017

Happy New Year!

This is the 1st edition in 2017 and the 47th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Big Data’s 2017: Can more meta thinking free us from current malaise? by Andrew Brust
A Simple Wardley Map of Data Flow by James Urquhart
Why Distributed Stream Processing must have a Persistent Local/Embedded Store like RocksDB by Chandan Parkash
Streaming analytics in a digitally industrialized world by Tugdual Grall
WebSockets as a Real-Time Data in Motion Flow by Tim Spann
2016: The Year in Big Data by Tim Spann
Big Data 2017: The future is cloudy by Tony Baer

Spark Streaming

Operationalizing Spark Streaming (Part 1) by Brandon O’Brien
Simple Spark Streaming & Kafka Example in a Zeppelin Notebook by Henning Kropp

Apache Kafka

Dimensioning Kafka consumers by Davor Poldrugo
End-To-End Data Security Using Apache Kafka by Chandani Patel

Apache Storm

The Battle of the Crawlers : Apache Nutch vs StormCrawler by Julien Nioche

Apache Flink

Apache Flink: A New Wave to Real-time Stream Processing by Md. Rezaul Karim

New Presentations

SAMOA: A Platform for Mining Big Data Streams by Nicolas Kourtellis
Fast Data Architecture by Knoldus
FFWD: latency-aware event stream processing via domain-specific load-shedding policies by R. Brondolin, M. Ferroni, M. D. Santambrogio
Providence Future of Data Meetup – Apache Metron Open Source Cybersecurity Platform by Carolyn Duby

New Videos

Ingest from DB to HDFS with Real-Time Dashboards – Big Data App Template by Yogi Devendra
Stream processing with Apache Kafka on DC/OS running on Azure by Michael Hausenblas

New Releases

Upcoming Events

1/7/2017 (Istanbul) – A Deep-dive into Structured Streaming / Predictive Analytics with SparkR (Spark Hub Meetup)
1/11/2017 (online) – Top 5 IoT Use Cases (Cloudera Webinar)
1/11/2017 (Washington, US) – Introduction to Kafka Streams with a Real-Life Example (Meetup)
1/13/2017 (San Francisco, US) – Processing IoT data with Apache Kafka (#SFO Tech Talk)
1/17/2017 (Amsterdam, NL) – Sensor Data Ingestion and Processing with NiFi and Spark (Meetup)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)
1/25/2017 (Munich, GE) – Apache Kafka with Jay Kreps (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!