Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 07:07 on February 28, 2018
Tags: flink ( 92 ), hdf, kafka ( 248 ), ksql ( 6 ), nifi ( 238 ), pulsar ( 16 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last Week in Stream Processing & Analytics – 27.2.2018

This is the 104th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

4 Ways Real-time Big Data Improves Pharma’s Bottom Line by Kayla Matthews
Real-time Analytics Requires Modern IT Infrastructure by Michael Vizard

Apache Kafka / Kafka Streams / Confluent Platform

5 Machine Learning Trends for 2018 Combined With Apache Kafka Ecosystem by Kai Waehner
Securing KSQL by Yeva Byzek
Kafka SSL – Custom Principal Builder by Pierre Villard
Using Apache Kafka for Asynchronous Communication in Microservices by Wale Martins
From Riak to Kafka: Part II by Stephen Ball

Pulsar

How Apache Pulsar ensures no messages lost and no messages duplicated by Ivan Kelly
Effectively-once semantics in Apache Pulsar by Matteo Merli

Streamsets

Managing Data Operations on the Edge: StreamSets by Pat Patterson

Apache NiFi / HDF

HDP 2.6.4-HDF 3.1: Apache Spark Streaming Integration With Apache NiFi 1.5 by Tim Spann
Apache Spark Structured Streaming Integration With Apache NiFi 1.5: Scala Edition by Tim Spann
Hortonworks DataFlow (HDF) 3.1 blog series part 5: Introducing Apache NiFi-Atlas integration by Haimo Liu & Koji Kawamura
Apache Deep Learning 101: Using Apache MXNet on an HDF Node by Tim Spann

Presentations

Monitoring Apache Kafka by Gwen Shapira
MQTT. Kafka. InfluxDB. SQL. IoT Harmony by Stefano Bocutiu
Applying Machine Learning to IOT: End to End Distributed Pipeline for Real-Time Uber Data Using Apache APIs: Kafka, Spark, HBase by Carol MacDonald
Let Your Data Flow with Apache NiFi by Bryan Bende
Unlock Value from Big Data with Apache NiFi and Streaming CDC by Jordan Martz & Mark Payne
Hortonworks DataFlow (HDF) 3.1 – Redefining Data-In-Motion with Modern Data Architecture by Dinseh Chandrasekhar & Haimo Liu & Guruditta Golani
Streaming Pipelines in Kubernetes Using Apache Pulsar, Heron and BookKeeper by Karthik Ramasamy
Distributed crypto-currency trading with Apache Pulsar by David Kjerrumgaard

Videos

Webinar – Real Time Customer Experience for the Right Now Enterprise featuring Forrester Research by Brendan Witcher
Kapacitor Stream Processing by Nathaniel Cook
Real-Time Data Processing with Apache Flink by Joerg Schad
Real-Time, Data-Rich Apps with the “SMACK Stack” by Achim Nierbeck & Joerg Schad
Building a Real-Time Streaming Platform with Oracle, Apache Kafka, and KSQL by Gwen Shapira
Integrating Apache Spark Structured Streaming with Apache NiFi via Apache Kafka by Tim Spann

Podcasts

Kafka Design Patterns with Gwen Shapira

New Releases

Apache Beam 2.3.0

Upcoming Events

1.3.2018 (Mexico City, MX) – Nuestro primer Kafka Meetup en Ciudad de México (Meetup)
1.3.2018 (San Diego, US) – Building a Real-Time Streaming Platform with Gwen Shapira (Meetup)
5.3.2018 (Seattle, US) – Flink SQL by dataArtisans and Pravega by Dell EMC (Meetup)
6.3.2018 (Madrid, ES) – An Introduction to Spark Streaming (Meetup)
15.3.2018 (online) – Empowering Streams through KSQL: Part 3: Deploying and Operating KSQL (Confluent Webinar)
17.4.2018 (Laurel, US) – Huge steps for Apache NiFi – Talks on NiFi in Finance and Powerful New Features (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 21:59 on February 13, 2018
Tags: hdf, kafka ( 248 ), kafka-connect ( 8 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 )

Last Week in Stream Processing & Analytics – 13.2.2018

This is the 102nd edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Migrating Batch ETL to Stream Processing: A Netflix Case Study with Kafka and Flink by Daniel Bryant
How Analytics Got the Eagles and the Patriots to the Superbowl by Sue Walsh
Study Finds More Reliance on Real-Time Streaming Analytics by Michael Vizard

Apache Kafka / Kafka Streams / Confluent Platform

Inventory Microservice example with Kafka Streams and CQRS by Sanjay Acharya
How to Write a User Defined Function (UDF) for KSQL by Kai Waehner
Confluent and Apache Kafka in 2017 by Jay Kreps
Toward a Functional Programming Analogy for Microservices by Bobby Calderwood
Custom Partitioning and Analysis Using Kafka SQL Windowing by
Rathnadevi Manivannan
Tuning the Kafka Connect Cassandra Source (part 2) by The Agile Jedi
When Streams Fail: Implementing a Resilient Apache Kafka Cluster at Goldman Sachs by Daniel Bryant

Spark Streaming

Introduction to Spark Structured Streaming – Part 15: Meetup Talk on Time and Window API by Madhukara Phatak

Apache NiFi / HDF

HDF 3.1 blog series part 2: Introducing the NiFi registry by Haimo Liu
HDF 3.1 Blog Series Part 3: Kafka 1.0 Support with Powerful HDF Integrations by George Vetticaden
FoD Paris Jan 18′ – NiFi Registry and workflow monitoring with a use case by Pierre Villard
HDF 3.1: Executing Apache Spark via ExecuteSparkInteractive via Apache Livy (Apache NiFi 1.5+) Part 1 of 2 by Timothy Spann
HDF 3.1: Executing Apache Spark via ExecuteSparkInteractive via Apache Livy (Apache NiFi 1.5+) Part 2 of 2 by Timothy Spann
Schema and Spark Fun With HDF 3.1 by Timothy Spann

Presentations

Machine Learning Trends of 2018 combined with the Apache Kafka Ecosystem by Kai Waehner
Kafka streams – From pub/sub to a complete stream processing platform by Paolo Castagna
Confluent and Elastic by Paolo Castagna
HDF 3.1 : An Introduction to New Features by Timothy Spann
Analyzing Streaming Data in Real-time with Amazon Kinesis by Sebastien Linsolas
Analyzing Streams by Darin Briskman
Structured Streaming in Spark by Vikram Agrawal
Streaming Microservices with Akka Streams and Kafka Streams by Dean Wampler

Videos

Building Large-scale Production Systems for Latency-sensitive Applications by Shadi Noghabi
Streaming Microservices with Akka Streams and Kafka Streams by Dean Wampler
What’s New in the Upcoming Apache Spark 2.3 Release? by Reynold Xin

New Releases

StreamSets 3.1.0.0

Upcoming Events

15.2.2018 (Sunnyvale, US) – Learn about unified queuing and streaming at Yahoo scale with Apache Pulsar (Meetup)
15.2.2018 (Ottawa, CA) – Building Multi-Region & Multi-Cloud Services with Kafka (Meetup)
15.2.2018 (Stockholm, SE) – Kafka Meetup with Confluent and Forefront Consulting (Meetup)
20.2.2018 (online) – Empowering Streams through KSQL: Part 2: Live Coding a KSQL Application (Confluent Webinar)
20.2.2018 (San Mateo, US) – Monitoring Apache Kafka with Gwen Shapira Alice Richardson (Meetup)
21.2.2018 (online) – Unlock Value from Big Data with Apache NiFi and Streaming CDC (Hortonworks Webinar)
22.2.2018 (Boulder, US) – Messaging,storage,or both: The real-time story of Pulsar & Apache DistributedLog (Meetup)
1.3.2018 (Mexico City, MX) – Nuestro primer Kafka Meetup en Ciudad de México (Meetup)
15.3.2018 (online) – Empowering Streams through KSQL: Part 3: Deploying and Operating KSQL (Confluent Webinar)
17.4.2018 (Laurel, US) – Huge steps for Apache NiFi – Talks on NiFi in Finance and Powerful New Features (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 21:59 on June 19, 2017
Tags: apex ( 18 ), beam ( 25 ), flink ( 92 ), hdf, IoT ( 10 ), kafka ( 248 ), kafka-streams ( 176 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last Week in Stream Processing & Analytics – 19.6.2017

This is the 71st edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

IBM Throws In with Hortonworks’ Hadoop by Alex Woodie
The future of Fast and Big Data Technologies by Dana Sandu
Real-Time Customer Service Will Soon Be the Rule by Joe McKendrick
Connected Intelligence in the IoT Edge by Rajeev Kozhikkattuhodi

Comparison

Streaming in Spark, Flink, and Kafka by Shivangi Gupta
Spark Streaming vs. Kafka Stream by Mahesh Chand Kandpa

Apache Kafka / Kafka Streams / Confluent Platform

Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks by Raghav Mohan
Simplifying your microservices architecture with Kafka and KafkaStreams by Dan Lebrero

Spark Streaming

Apache Spark DStream by DataFlair
Exploring Spark Structured Streaming by Mahesh Chand Kandpal

Apache Beam

Beam 2.0 Q and A by Jesse Anderson

Apache NiFi / Hortonworks HDF

Developing Streaming Analytics Applications without writing a single line of code for Hortonworks DataFlow by Wei Wang

New Presentations

Stateful Stream Processing with Apache Flink by Tzu-L Gordon
The History and Future of the Blurring of Stream Processing & OLTP by John Hugg
From Batch to Streaming ET(L) with Apache Apex by Thomas Weise
Complex Event Processing with Flink: the state of FlinkCEP by Kostas Kloudas
Stream Analytics with SQL on Apache Flink by Fabian Hueske
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT by Denis Magda
Stream All Things—Patterns of Modern Data Integration by Gwen Shapira
Connecting Apache Kafka With Mule ESB by Jitendra Bafna
Introducing Exactly Once Semantics in Apache Kafka by Matthias J. Sax

New Videos

Rethinking Stream Processing with Apache Kafka by Michael Noll
From Batch to Streaming ET(L) with Apache Apex by Thomas Weise
Integration Patterns for Big Data Applications by Michael Häusler
Experiences running Flink at Very Large Scale by Stephan Ewen
Challenges of Monitoring Distributed Systems by Nenad Bozic
Design Patterns for Calculating User Profiles in Real Time by Igor Mazor
The History and Future of the Blurring of Stream Processing & OLTP by John Hugg
Using Apache Beam to create a unified benchmarking framework by Ismaël Mejía
A Big Data Streaming Recipe by Konstantin Gregor
Balancing Heroes and Pokemon in Real Time by Fokko Driesprong & Vincent Warmerdam
Containerizing Distributed Pipes by Hagen Tönnies
Update on the t-digest: Finding Faults in Real Data by Ted Dunning
Mechanics of Data Pipelines by Sean Braithwaite
Hops: Multi-Tenancy and Streaming-First in an open-source SaaS platform by Jim Dowling
Complex Event Processing with Flink: the state of FlinkCEP by Kostas Kloudas
Stream Analytics with SQL on Apache Flink by Fabian Hueske
Hands-on with Apache NiFi and MiNiFi by Andrew Psaltis

Upcoming Events

20.6.2017 (Düsseldorf, GE) – Big Data & Stream Processing with Apache Spark
22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
22.6.2017 (Zurich, CH) – Data stream processing at PSI (Meetup)
27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)
28.6.2017 (Munich, GE) – Rethinking Stream Processing with Apache Kafka (Meetup)
3.7.2017 (Warsaw, PL) – Apache Flink – Stateful Stream Processing (dataArtisans) & CEP (GetInData) (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 19:00 on February 27, 2017
Tags: beam ( 25 ), flink ( 92 ), hdf, kafka ( 248 ), kafka-connect ( 8 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics – 27.02.2017

This is the 55th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Hazelcast Launches an Open Source In-Memory Stream Processing Engine by Susan Hall
Evaluation Guide to Streaming Analytics by evam

Spark Streaming

Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1 by Burak Yavuz, Michael Armbrust, Tathagata Das & Tyson Condie
Streaming Wikipedia edits with Spark and Clojure by Joel Wilsson

Apache Kafka / Kafka Streams / Confluent Platform

Streaming databases in realtime with MySQL, Debezium, and Kafka by Chris Riccomini
Join us for Kafka Summit Hackathon in New York City by Michael Noll
Connecting ICS and Apache Kafka via REST Proxy API by Ricardo Ferreira
Apache Kafka: Multiple ways for Produce or Push Message to Kafka topics by Harmeet Singh
Apache Kafka: Multiple ways for Consume or Read messages from Kafka Topic by Harmeet Singh
Create a VM Image With Apache Kafka Configured Using Vagrant and Ansible by Andy Boyle

Apache Storm

How to Run Scala-Implemented Storm Topologies on HDInsight by Emmanouil Gkatziouras
Word Count With Storm and Scala by Emmanouil Gkatziouras

Apache Flink

Stream Processing with Apache Flink by Fabien Hueske & Vasiliki Kalavri

StreamSets

Running Scala Code in StreamSets Data Collector by Pat Patterson

Apache NiFi / Hortonworks HDF

List/Fetch pattern and Remote Process Group in Apache NiFi by Pierre Villard

New Presentations

The Data Dichotomy- Rethinking the Way We Treat Data and Services by Ben Stopford
Containerizing Distributed Pipes by Hagen Toennies
Kafka as message broker by Haluan Irsad
Part 1: Lambda Architectures: Simplified by Apache Kudu by Michael Crutcher
Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic Databases by Alex Gutow & Ryan Lippert
Kick-Start with SMACK Stack by Sandeep Purohit
Reactive Fast Data & the Data Lake with Akka, Kafka, Spark by Todd Fritz
Scalable Real-time Complex Event Processing at Uber by Shuyi Chen
Streaming all the things with akka streams by Johan Andren
Apex & Geode: In-memory streaming, storage & analytics by Ashish Tadose
Complex Event Processing with Esper by Ted Won
Streaming Data Analytics with Amazon Kinesis Firehose and Redshift by Ray Zhu

New Videos

The Data Dichotomy- Rethinking the Way We Treat Data and Services by Ben Stopford
Asynchronous Processing and Multithreading in Apache Samza by Xinyu Liu
Developing Fast Data Architectures with Streaming Applications by Karl Wehden
Batching to Streaming Analytics at Optimizely by Mike Davis & Hao Xia
SSD Benchmarks on Kafka by Mingmin Chen

New Releases

New Books

Stream Processing with Apache Flink by Fabian Hueske & Vasiliki Kalavri
Spark: The Definitive Guide – Big data processing made simple by Bill Chambers & Matei Zaharia
Learning Apache Flink by Tanmay Deshpande

Upcoming Events

1.3.2017 (Phoenix, US) – Bio-manufacturing Optimization using Apache NiFi, Kafka and Spark (Meetup)
1.3.2017 (Leeds, UK) – Kafka’s Role in Implementing Oracle’s Big Data Reference Architecture (YoDB#7)
2.3.2017 (Amsterdam, NL) – Apache Flink’s Stateful Operators And Table SQL Api (Meetup)
2.3.2017 (San Francisco, US) – Streaming with MapR and StreamSets Data Collector (Meetup)
7.3.2017 (Santa Monica, US) – Stream processing with R in AWS (Meetup)
7.3.2017 (Austin, US) – Stream All the Things! (Meetup)
21.3.2017 (online) – Getting Started with Streaming Data and Stream Processing with Apache Kafka (KPI Webinar)
23.3.2017 (San Francisco, US) – Bay Area Apache Spark Meetup @ Intel in Santa Clara (Meetup)
28.3.2017 (Princeton, US) – Apache NiFi: Ingesting Enterprise Data @ Scale (Meetup)
29.3.2017 (online) – Part 2: Building Event-Driven Services with Apache Kafka (Confluent Webinar)
25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 23:19 on January 9, 2017
Tags: flink ( 92 ), hdf, kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 01/09/2017

This is the 48th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

A Better Way to Build a Fraud Detector: Streaming Data and Microservices Architecture by Ted Dunning
8 data trends on our radar for 2017 by Ben Lorica
The Impact of Data Grids in IoT by Jason Stamper
Applying Machine Learning to Real Time Streaming Analytics by Robert Waywell

Spark Streaming

Monitoring Real-Time Uber Data Using Spark Machine Learning, Streaming, and the Kafka API (Part 2) by Carol McDonald
Big Data Processing with Apache Spark – Part 3: Spark Streaming by Srini Penchikala

Apache Kafka

Kafka Avro Scala Example by Sushil Kumar Singh
Using Kafka with JUnit by Biju Kunjummen
Log Compaction: Highlights in the Apache Kafka and Stream Processing Community – January 2017 by Gwen Shapira
Proof of concept using KafkaStreams and KTables by Daniel Lebrero
How to Build a Custom Flogo Adapter by Kai Waehner
Microservices messaging on Oracle Cloud using Apache Kafka by Abhishek Gupta
Managing IoT devices with Kafka and MQTT by Marcello Vitaletti

Apache Storm

Join: Storm in a Teacup, Continued… by Ernestas Vaiciukevičius

Apache Samza

Apache Flink

Monitoring Wikipedia Edit Streams with Apache Flink and Packaging the Application with Dependencies by Rezaul Karim
Comprehensive Guide on Apache Flink Ecosystem Components by Data Flair

Apache NiFi / Hortonworks HDF

Clojure on Apache NiFi by Goran Jovic
Apache Nifi Installation on Ubuntu by Dhamodharan N

New Presentations

Open Source IoT Project Flogo – Building a Custom Apache Kafka Connector by Kai Waehner

New Releases

Mocked Streams 1.1.0

Upcoming Events

1/11/2017 (online) – Top 5 IoT Use Cases (Cloudera Webinar)
1/11/2017 (Washington, US) – Introduction to Kafka Streams with a Real-Life Example (Meetup)
1/12/2017 (Mountain View, US) – Tech Talk: Processing IoT data with Apache Kafka (Meetup)
1/13/2017 (San Francisco, US) – Processing IoT data with Apache Kafka (#SFO Tech Talk)
1/17/2017 (Amsterdam, NL) – Sensor Data Ingestion and Processing with NiFi and Spark (Meetup)
1/17/2017 (Chicago, US) – Fast Data: Selecting The Right Streaming Technologies For Never Ending Data Sets (Meetup)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)
1/18/2017 (Milano, IT) – TensorFlow & TensorFrames with Apache Spark + A Deep-dive into Structured Streaming (Spark Hub)
1/18/2017 (Brussels, BE) – Running Kafka in production and Streamsets (Meetup)
1/19/2017 (New York, US) – Fast Data / Event-Driven Architecture (Meetup)
1/23/2017 (Zurich, CH) – 19th Swiss Big Data User Group Meeting (Meetup)
1/25/2017 (Munich, GE) – Apache Kafka with Jay Kreps (Meetup)
1/31/2017 (online) – Enabling Next Gen Analytics with Azure Data Lake (StreamSets Webinar)
2/1/2017 (Fremont, CA) – Understanding Big Data Streaming and Apache Flink (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 18:38 on December 26, 2016
Tags: apex ( 18 ), aws ( 3 ), flink ( 92 ), hdf, nifi ( 238 ), spark ( 153 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 12/26/2016

Happy Holidays!

This is the Christmas edition and therefore rather compact! It’s 46th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Big Data Industry Predictions for 2017 by Daniel Gutierrez
The Future of Streaming Analytics in Financial Services by Mark Palmer
Big data: Why online gambling should bet on streaming analytics by Jay Patani
Embedded analytics to feel widest impact of machine learning projects by Jack Vaughan

Spark Streaming

24/7 Spark Streaming on YARN in Production by Bernhard Schäfer

Apache Flink

Apache Flink in 2016: Year in Review by Mike Winters
Apache Flink: The Next Gen Big Data Analytics Framework by Sibanjan Das

StreamSets

Calling External Java Code from Script Evaluators by Pat Patterson
Building an Amazon SQS Custom Origin for StreamSets Data Collector by Pat Patterson

Apache NiFi / Hortonworks Data Flow (HDF)

Apache NiFi 1.x Cheatsheet by Tim Spann
An Example WebSocket Application in Apache NiFi 1.1 by Timothy Spann
Submitting Spark Jobs From Apache NiFi Using Livy by Timothy Spann
Trigger SonicPi Music Via Apache NiFi by Timothy Spann
Running DNS and Domain Scanning Tools From Apache NiFi by Timothy Spann
Adding A Custom Processor to NiFi : LinkProcessor by Timothy Spann
Drones (UAV) Data Ingest Methods by Timothy Spann

New Presentations

Event Driven Architecture by Benjamin Joyen-Conseil
Deep dive of Deduplication using Apache Apex and RTS by Bhupesh Chawda
Building a Real-Time Streaming Data Platform on AWS by Ryan Nienhuis & Ram Kumar Rengaswamy
Structured streaming in Spark by Giri R Varatharajan

New Releases

Apache Flink 1.1.4

Upcoming Events

1/7/2017 (Istanbul) – A Deep-dive into Structured Streaming / Predictive Analytics with SparkR (Spark Hub Meetup)
1/11/2017 (online) – Top 5 IoT Use Cases (Cloudera Webinar)
1/11/2917 (Washington, US) – ntroduction to Kafka Streams with a Real-Life Example (Meetup)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)
1/25/2017 (Munich, GE) – Apache Kafka with Jay Kreps (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:47 on December 12, 2016
Tags: beam ( 25 ), hdf, heron ( 23 ), IoT ( 10 ), kafka ( 248 ), nifi ( 238 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 12/12/2016

This is the 44th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

CEP Patterns for Stream Analytics by Sibanjan Das

Apache Kafka

Kafka Connect and Kafka REST API on MapR: Streaming Just Became a Whole Lot Easier! by Ankur Desai
Enterprise Streaming Multi-Datacenter Replication using Apache Kafka by Ewen Cheslack-Postava

Apache Beam

Beam’s Pico WordCount by Jesse Anderson

Apache Flink

November 2016 in Review: Flink Forward 2017, Amazon EMR + Google Dataproc, and kicking off the Flink training series in Germany by Michael Winters

Storm / Heron

Twitter’s real-time stack: Processing billions of events with Heron and DistributedLog by Ofir Shalev

StreamSets

Running Apache Spark Code in StreamSets Data Collector by Pat Patterson
Creating a StreamSets Spark Transformer by Pat Patterson
Creating a Custom Origin for StreamSets Data Collector by Pat Patterson

Apache NiFi / Hortonworks Data Flow (HDF)

Announcing the Availability of Hortonworks DataFlow (HDF) 2.1 by Wei Wang & Haimo Liu

New Presentations

ETL is Dead; Long-Live Streams by Neha Narkhede
Leveraging Mainframe Data for Modern Analytics by Jordan Martz & David Tucker & Keith Reid
Real-Time Streaming with Kafka – for data scientists by Jennifer Rawlins
Stream Processing as Game Changer for Big Data and Internet of Things by Kai Wähner
CEP – simplified streaming architecture by Mathieu Dumoulin
Streaming Architecture to Connect Everything (Including Hybrid Cloud) by Mathieu Dumoulin
Continuous Application with Structured Streaming 2.0 by Jules S. Damji
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi by Bryan Bende
Better together: Fast Data with Apache Spark™ and Apache Ignite™ by Micahel Griggs

New Videos

ETL is Dead; Long-Live Streams by Neha Narkhede
Apache Kafka, Stream Processing, and Microservices by Jay Kreps
Stream-1st Architecture, Apache Flink & Other Emerging Technologies by Ellen Friedman
Containers, DevOps, Microservices, & Kafka Tools by Jonathan Owens & Jose Fernandez
Stream Processing use cases and applications with Apache Apex by Thomas Weise
Big Data Analytics, Machine Learning, Real Time Processing by Kai Wähner

New Releases

Upcoming Events

12/14/2016 (online) – How to Manage Continuous Dataflows for a Customer 360 Application (StreamSets Webinar)
12/14/2016 (online) – Counteract Cyberattacks and Fraud with Streaming Analytics (Striim Webinar)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)
12/16(2016 (Kansas City, US) – Kansas City Big Data Group – Big Data Pipelines/Flows, Use case driven comparison: Apache NiFi & StreamSets (Meetup)
12/17/2016 (Fremont, US) – Spark Saturday – Hands-on Workshop with Apache Spark 2.x on Databricks (Meetup)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 07:43 on November 22, 2016
Tags: apex ( 18 ), confluent ( 8 ), flink ( 92 ), hdf, kafka ( 248 ), nifi ( 238 ), spark-streaming ( 219 ), storm ( 46 ), Stream Processing ( 241 ), streaming-analytics ( 84 )

Last week in Stream Processing & Analytics 11/21/2016

This is the 41st installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Apache BigData Europe Conference Summary by Marcin Kuthand
Small Scale Stream Processing Kata (Part 1): Thread Pools by Tomasz Nurkiewicz
From Big Data to Fast Data in Four Weeks or How Reactive Programming is Changing the World – Part 1 by Michael Zeltser
From Big Data to Fast Data in Four Weeks or How Reactive Programming is Changing the World – Part 2 by Michael Zeltser
Understanding Apache Ecosystem: stream processing by Lorenzo Toscano
A fast data architecture whizzes by traditional data management tools by Paul Korzeniowski

Apache Kafka

Announcing Confluent 3.1 with Apache Kafka 0.10.1.0 by Sriram Subramanian
Unit Testing Kafka Consumers by Jesse Anderson
Confluent Releases 3.1 with Apache Kafka by Sue Walsh
Partner Development Guide for Kafka Connect by Confluent
Spring Kafka Producer/Consumer sample by Biju Kunjummen
Apache Kafka Streams and JCache with BlazingCache by Enrico Olivelli
39th meetup on Apache Kafka and Apache Ignite by Wim van Leuven
Oracle Service Bus: A quickstart for the Kafka transport by Maarten Smeets

Apache Storm

Carrier Payments Big Data Pipeline using Apache Storm by Mansi Mahendru

Apache Flink

Apache Flink Shell Command Tutorial – A Quickstart For beginners by Data Flair

Apache NiFi / Hortonworks Data Flow (HDF)

Routing Logs Through Apache NiFi to Phoenix, HDFS and Offsite to Sumologic for Log Analytics by Timothy Spann

New Presentations

Reliable Data Intestion in BigData / IoT by Guido Schmutz
Introduction to Streaming Analytics by Guido Schmutz
Streaming operational data with Kafka by David Tucker & David Ostrovsky
Demystifying Stream Processing with Apache Kafka by Neha Narkhede
Design Patterns for working with Fast Data in Kafka by Ian Downard
Architecture Big Data open source S.M.A.C.K by Julien Anguenot
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And Kafka by Akara Sucharitakul
Powering Microservices with Docker, Kubernetes, and Kafka by Andrew Morgan
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi by Timothy Spann
HDF Powered by Apache NiFi Introduction by Milind Pandit
Next Gen Big Data Analytics with Apache Apex by Thomas Weise
Introduction to Apache Apex by Thomas Weise
Building Streaming Applications with Apache Apex by Chinmay Kolhatkar & Thomas Weise
Twitter heron – Evolution or Revolution? by Grzegorz Kolpuc
Data Microservices with Spring Cloud Stream, Task, and Data Flow by Toshiaki Maki
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Services by Kai Wähner
Stream Application Development with Apache Kafka by Matthias J. Sax
RSEP-QL: A Query Model to Capture Event Pattern Matching in RDF Stream Processing Languages by Daniele Dell’Aglio

New Videos

Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And Kafka by Akara Sucharitakul
Streaming Analytics Comparison of Open Source Frameworks, Products, Cloud Services by Kai Wähner
Demystifying Stream Processing with Apache Kafka by Neha Narkhede

New Podcasts

Drill to Detail – Streamsets, Data-in-Motion and Data Drift by Mark Ritmann with Pat Patterson

New Releases

Upcoming Events

11/22/2016 (online) – Digital Transformation to Data-Driven Real-Time Enterprise (Cloudera Webinar)
11/22/2016 (Austin, US) – Sensor Data Analytics Acceleration with Apache Edgent (Meetup)
11/24/2016 (online) – Real-Time Data Ingestion and Streaming Analytics with Oracle and Hortonworks (Hortonworks Webinar)
11/29/2016 (Berlin, GE) – Apache Flink Meetup @ data Artisans (Meetup)
11/30/2016 (Tel Aviv, IL) – Introducing StreamSets – Creating Scalable and Resilient Data Pipelines (DataZone Event)
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/5/2016 (New York, US) – Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi (Meetup)
12/6/2016 (Lisbon, PT) – A Deep-dive into Structured Streaming (Meetup)
12/8/2016 (New York, US) – Streaming Analytics in a Flash – presented by Cask Data (Meetup)
12/8/2016 (Singapore, SG) – Stream Processing and TensorFlow 101 (Google Meetup)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)
1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

gschmutz 22:51 on November 14, 2016
Tags: flink ( 92 ), hdf, kafka ( 248 ), nifi ( 238 ), samza ( 8 ), spark-streaming ( 219 ), Stream Processing ( 241 ), streaming-analytics ( 84 ), streamsets ( 74 )

Last week in Stream Processing & Analytics 11/14/2016

This is the 40th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

In 2017 IoT will drive continuous big data operations by Rick Bilodeau
Real-Time Election Predictions: VoteCastr’s Experiment by Sue Walsh
The 3 Keys to Digital Transformation by Jack Norris
North America Streaming Analytics Market (2016-2022) – Number of Connected Devices will Grow to 50 Billion – Research and Markets by Yahoo Finance

Comparison

15 Questions and Answers from Apache NiFi, Kafka & Storm: Better Together by Haimo Liu & Bryan Bende

Apache Apex

Throughput, Latency, and Yahoo! Performance Benchmarks. Is there a winner? by DataTorrent

Apache Kafka

Unit Testing Kafka by Jesse Anderson
Movin’ On Up: Apache Kafka Gets a Promotion to the Enterprise by Christy Wilson

Spark Streaming

Structured Streaming In Apache Spark A new high-level API for streaming by Matei Zaharia, Tathagata Das, Michael Armbrust & Reynold Xin

Apache Flink

Processing Rabbitmq’s Stream with “Apache Flink” by Shekhar Singh

StreamSets

More Than One Third of the Fortune 100 Have Downloaded StreamSets Data Collector by Rick Bilodeau
Ingesting Streaming Data from JMS into HDFS and Solr using StreamSets by Rick Bilodeau

Apache NiFi / Hortonworks Data Flow (HDF)

Edge Intelligence for IoT with Apache MiNiFi by Anna Yong & Joe Percivall
How to Harness the Power of Data-in Motion: Webinar Series by Anna Yong

New Presentations

Apache Samza Past, Present and Future by Kartik Paramasivam
Apache Kafka, and the Rise of Stream Processing by Guozhang Wang
Apache Nifi, Kafka and Storm Better Together by Bryan Bende & Haimo Liu
Data Streaming with Apache Kafka & MongoDB by Andrew Morgan & David Tucker
Data Loss and Duplication in Kafka by Jayesh Thakrar
Apache Kafka – Stream Processing made easy by Daan Gerits
What’s New in Spark 2? by Eyal Ben Ivri
Real-Time Analytics with MemSQL and Spark by Neil Dahlke
Stream processing by Ashic Mahtab
Developing a Streaming Application Using Apache Apex by Pramod Immaneni
Building Streaming Applications with Apache Apex by Chinmay Kolhatkar & Thomas Weise
Scaling up Near real-time Analytics @ Uber and LinkedIn by Chinmay Soman & Yi Pan
Analytics on AWS:Structured, Unstructured and Streaming by Russell Nash
Apache NiFi Crash Course Intro by Rafael Coss

New Videos

Stream processing by Ashic Mahtab
Apache Kafka, Storm and NiFi: Better Together by Bryan Bende & Haimo Liu
Distributed Commit Logs with Apache Kafka by James Ward
Apache Samza Past, Present and Future by Kartik Paramasivam
Google Dataflow: The New Open Model for Batch and Stream Processing by Sara Robinson & Robert Kubis
Cloud Native Streaming and Event-Driven Microservices by Marius Bogoevici
Reactive Streams principles applied in Akka Streams by Eric Loots
Oracle Streams Analytics platform – Is it time to reverse traditional analytics? Oracle Partner Webcast

New Releases

Spark 1.6.3

Upcoming Events

11/20/2016 (Mountain View, US) – November Apache Kafka Meetup (Meetup)
11/17/2016 (online) – Demystifying Stream Processing with Apache Kafka (Confluent Online Talk series
11/17/2016 (Mountain View, US) – Pulsar – Distributed pub-sub messaging & Apache NiFi in action (Meetup)
11/21/2016 (Columbia, US) – Using Apache NiFi at Capital One & The Apache Way (Meetup)
11/22/2016 (online) – Digital Transformation to Data-Driven Real-Time Enterprise (Cloudera Webinar)
11/29/2016 (Berlin, GE) – Apache Flink Meetup @ data Artisans (Meetup)
11/30/2016 (Tel Aviv, IL) – Introducing StreamSets – Creating Scalable and Resilient Data Pipelines (DataZone Event)
12/1/2016 (online) – A Practical Guide to Selecting a Stream Processing Technology (Confluent Online Talk series)
12/5/2016 (New York, US) – Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi (Meetup)
12/6/2016 (Lisbon, PT) – A Deep-dive into Structured Streaming (Meetup)
12/8/2016 (Vancouver, CA) – Connecting All Things with Apache Kafka (Meetup)
12/15/2016 (online) – Streaming in Practice: Putting Apache Kafka in Production (Confluent Online Talk series)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!