Last week in Stream Processing & Analytics – 22.5.2017

This is the 67th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Anomaly Detection in Telecommunications Using Complex Streaming Data by Ted Dunning
• The Future of Event Stream Processing by Michael Drogalis
• How to avoid downtime and disruption when moving data by David Richards
• The Anatomy of an IoT Solution by Hans de Visser

Apache Kafka / Kafka Streams / Confluent Platform

• Reading Data Securely From Apache Kafka to Apache Spark by Mark Grover
• Kafka Topic Design Checklist by Jesse Anderson
• Building Real-Time data pipelines with Kafka Connect by Slim Baltagi
• Trialing Single Message Transformations in Kafka Connect by Mike Donovan

Spark Streaming

• Taking Apache Spark’s Structured Structured Streaming to Production by Bill Chambers & Michael Armbrust
• Running Streaming Jobs Once a Day For 10x Cost Savings by Burak Yavuz & Tyson Condie

Apache Flink

• A Year of Blink at Alibaba: Apache Flink in Large Scale Production by Xiaowei Jiang

AWS Kinesis

• Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight by Karan Desai

Apache NiFi

• Monitoring NiFi – Scripted Reporting Task by Pierre Villard
• Monitoring NiFi – Ambari & Grafana by Pierre Villard
• Monitoring NiFi – Workflow SLA by Pierre Villard
• Monitoring NiFi – Site2Site reporting tasks by Pierre Villard
• Monitoring NiFi – Logback configuration by Pierre Villard
• Record-oriented Data with NiFi by Mark Payne

New Presentations

• Real Time Analytics – Stream Processing by Mahesh Madushanka
• Streaming Analytics for IoT-Oriented Applications by Adrian Bowles
• Geo-Distributed Big Data and Analytics by Ted Dunning
• Top 5 mistakes when writing Streaming applications by Mark Grover & Ted Malaska
• Fast Cars, Big Data How Streaming can help Formula 1 by Carol McDonald
• Stateful streaming data pipelines by Chandni Singh & Timothy Farkas
• Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform by Chrix Finne & Bob Lehmann
• Microservices interaction at scale using Apache Kafka by Ivan Ursul
• Capture the Streams of Database Changes by Randall Hauch
• Stream Processing Metamorphosis – A Kafka’s tale by João Vazão Vasques
• Event processing with Apache Storm by Nuno Caneco

New Podcasts

•  Apache Kafka, Streaming Data Integration and Schema Registry by Mark Rittman with Gwen Shapira

Upcoming Events

• 22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
• 23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
• 23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
• 24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
• 30.5.2017 (Frankfurt, GE) – First Kafka Meetup (Meetup)
• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 17.5.2017

This is the 66th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Fast Cars, Big Data – How Streaming Data Can Help Formula 1 by Carol McDonald
• Thoughts on Building a Successful Streaming Analytics Platform by George Vetticaden
• A Digital Reference Architecture for the Industrial Internet Of Things (IIoT) by Vamsi Chemitiganti
• Introducing DevHigh5 Project Pravega: Storage reimagined for a streaming world by Salvatore DeSimone

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka Summit New York: The Rise of the Streaming Platform by Clarke Patterson
• Reading data securely from Apache Kafka to Apache Spark by Mark Grover
• Installing and Running Kafka on an AWS Instance by Sahil Sawhney
• Managing Spark and Kafka Pipelines by Larry Murdock
• Getting Started with the Kafka Streams API using Confluent Docker Images by Michael Noll
• Kafka Consumer Architecture – Consumer Groups and subscriptions by Cloudurable
• The new version of Kafka Topics UI is now available! by Christina Daskalaki
• Startup Confluent makes Kafka cloud play to ease streaming deployment by Jack Vaughan
• Securing Apache Kafka with Kerberos by Colm O hEigeartaigh
• Apache Kafka – Introduction & Installation by Rudra Narayan Garnaik
• Visualizing the flow of data as it moves through a Kafka cluster by James Cheng

Spark Streaming

• How to Get Started with Spark Streaming and MapR Streams Using the Kafka API by Carol McDonald

Apache Beam

• Apache Beam publishes the first stable release by Davor Bonaci

Apache Flink

• Introducing Official Docker Images for Apache Flink by Patrick Lucas & Ismaël Mejía

StreamSets

• Visualizing and Analyzing Salesforce Data with Neo4j by Pat Patterson
• Calling External Libraries from the JavaScript Evaluator by Pat Patterson

Apache NiFi

• Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster by Roger Carhuatocto
• Apache Nifi Sensitive Value Encryption by Kya
• Record based processors in Apache NiFi 1.2 by Haimo Liu

New Presentations

• Intro to Apache Kafka by Jason Hubbard
• Stream processing using Kafka by Himani Arora & Prabhat Kashyap
• Rethinking Services With Stateful Streams by Ben Stopford
• Connect S3 with Kafka leveraging Akka Streams by Seiya Mizuno
• Stream Processing with MicroProfile and Apache Kafka by Matthias Wessendorf
• Real-time Data Pipelines with SAP and Apache Kafka by Carole Gunst
• Processing IoT Data with Apache Kafka by Confluent
• Functional streams with Kafka – A comparison between Akka-streams and FS2 by Luis Reis & Rui Batista
• Open Blueprint for Real-Time Analytics with In-Stream Processing (ISP) by Victoria Livschitz
• Dataservices – Processing Big Data The Microservice Way by Josef Adersberger

New Video

• Scalable Stream Processing: A Survey of Storm, Samza, Spark and Flink by Felix Gessert
• Improve your App UX with one model processing insights from batch and streams by Robert Kubis
• Apache Ignite: Real Time Processing of IoT Generated Streaming Data by Denis Magda
• Rethinking Microservices with Stateful Streams by Ben Stopford
• Fast Cars, Big Data – How Streaming Can Help Formula 1 by Tugdual Grall

New Releases

• Apache Beam 2.0.0

Upcoming Events

• 17.5.2017 (Mountain View, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
• 18.5.2017 (online) – Conquering Data in Motion with a Modern Ingestion Platform (Cloudera Webinar)
• 22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
• 23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
• 23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
• 24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 26.4.2017

This is the 63rd edition of my blog series blog series around Stream Processing and Analytics! A few days later than planed, but I had some busy days with speaking on Apache Kafka at Oracle Code Berlin on Monday and today we held our Trivadis/DataStax Hands-On workshop in Zurich. So next week will be again on time (Monday), I hope 😉

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Comparison

• Event time processing in Apache Spark and Apache Flink by Matthias Niehoff

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka channels the big data firehose by Tony Baer
• Introducing Kasper: The Kafka Stream Processing Library Built For Go by Nicolas Maquet
• Benchmarking Apache Kafka with Aiven – Part 1: Write throughput (AWS vs Azure vs DigitalOcean vs Google Cloud vs UpCloud) by Aiven

Spark Streaming

• Fault tolerance in Apache Spark – Reliable Spark Streaming by Data Flair
• Real-Time Weather Event Processing With HDF, Spark Streaming, and Solr by Balaji Kandregula
• Spark Streaming – under the hood by veejayendraa
• Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2 by Kunal Khamar & Tyson Condie & Michael Armbrust

Apache Apex

• One Year Anniversary of Apache Apex by Thomas Weise

Apache Flink

• Build a Real-time Stream Processing Pipeline with Apache Flink on AWS by Steffen Hausmann

StreamSets

• StreamSets Data Collector v2.5 Adds IoT, Spark, Performance and Scale by Kirit Basu
• Make sense of stream processing by Pat Patterson
• Creating a Custom Multithreaded Origin for StreamSets Data Collector by Pat Patterson

Apache Nifi

• Basics of Apache Nifi: 1 by Malcom Thirus
• How to send data from Apache NIFI to HANA by Franco Lezana

Azure Stream Analytics

• Announcing Azure Stream Analytics on edge devices (preview) by Santosh Balasubramanian

New Presentations

• Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema
• Kafka used at scale to deliver real-time notifications by Sergio Nunes
• Apache Storm In Retail Content by Karthik Deivasigamani
• Storm worker redesign by Roshan Naik
• From Three Nines to Five Nines – A Kafka Journey by Allen Wang
• Microsoft challenges of a multi tenant kafka service by Nitin Kumar
• The Rise of Real-Time by Neha Narkhede
• Apache Kafka – Scalable Message-Processing and more ! by Guido Schmutz
• Building Streaming Applications with Apache Storm 1.1 by Hugo Louro
  
• Kafka Tutorial – basics of the Kafka streaming platform by Jean-Paul Azar
• Bootstrapping Microservices with Kafka, Akka and Spark by Alex Silva

New Videos

• Building Faster Streaming Applications with Apache Storm 1.1 by Suresh Srinivas & Hugo Louro & Karthik Deivasigamani
  
• Putting the Micro into Microservices with Stateful Stream Processing by Ben Stopford
• StreamING models, how ING adds models … by Erik de Nooij
• Real-Time Analytics On Data Streams – A New Era For Big Data And IoT by Jeremy Hillier
• Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema

New Podcasts

• Data science for real-time streaming analytics by Roger Rea

New Releases

• StreamSets 2.5

Upcoming Events

• 25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
• 25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
• 25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
• 25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
• 25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
• 27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
• 27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)
• 27.4.2017 (Charlotte, US) – Apache NiFi – An Uber Data Flow Tool (Meetup)
• 16.5.2017 (online) – Geo-Distributed Big Data and Analytics (MapR Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 18.4.2017

This is the 62nd edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Multi-Master Replication For Geo-Distributed Data: It’s more than you think by Ellen Friedman
• Understanding Indicators of Attack (IOAs): The Power of Event Stream Processing in CrowdStrike Falcon by Dan Brown
• Stream processing and messaging systems for the IoT age by Ben Lorica

Apache Kafka / Kafka Streams / Confluent Platform

• Creating a Data Pipeline with Kafka Connect API – from Architecture to Operations by Alexandra Wang
• Streaming Spring Boot Application Logs to ELK Stack — Part 1 by ksadayamuthu
• Streaming Spring Boot Application Logs to Apache Kafka — ELK(K) Stack — Part 2 by ksadayamuthu
• How we embraced the new release of Apache Kafka by Ning Zhang
• Streaming Data with Apache Kafka by Tyler Keenan
• Kafka Security Mechanism (SASL/PLAIN) by Bharat Viswanadham
• Stories from the Front: Lessons Learnt from supporting Apache Kafka by Dustin Cote

Apache Storm

• Important Production bugs and fixes for Storm and Kafka integration by Adrianos Dadis

StreamSets

• Transform Data in StreamSets Data Collector by Pat Patterson
• Stream Me Up (to the Cloud), Scotty by Mike Fuller
• How to Create a Custom StreamSets Origin That Supports Parallel Execution by

New Presentations

• SPaaS Stream Processing with Flink by Monal Daxini
• Apache Kafka Lesson Learned by Guozhang Wang
• A Streaming Platform Architecture Based on Apache Kafka by Ewen Cheslack-Postava
• Apache Kafka Best Practices by Manikumar Reddy
• Apache Kafka & Usecases within Search System @ Walmartlabs by Snehal Nagmote
• Debunking Common Myths in Stream Processing by Robert Metzger & Ufuk Celebi
• Unifying Stream SQL and CEP for Declarative Stream Processing with Apache Flink by Till Rohrmann
• A Benchmark Suite for Distributed Stream Processing Systems by Claudio Geyer
• Dataflow with Apache NiFi by Aldrin Piri
• Apex Runner for Apache Beam by Tushar Gosav
• Building Stream-Analytic Ecosystems for Enterprises by Raman Ganesh
• Amazon Kinesis Platform – The Complete Overview by Roy Ben-Alta &Damian Wylie
• Comparison of Apache Apex and Spark Streaming by Bhupesh Chawda
• The past, present and future of Apache Storm by Taylor Goetz
• A Special Case in Streaming by Bhupesh Chawda
• Using Flink and Queryable State to Buffer High Frequency Time Series Data by Joe Olson
• The Stream Processor as a Database: Building Online Applications directly on Streams by Ufuk Celebi
• Convergence of real-time analytics & data-driven applications by Stephan Ewen
• Real Time Analytics in the real World – Challenges and Lessons at Uber by Chinmay Soman
• Streaming Deep Learning Scenarios with Flink by Dean Wampler
• Runtime Improvements in Blink for Large Scale Streaming at Alibaba by Feng Wang & Zhijiang Wang
• Experiences with Streaming vs Micro-Batch for Online Learning by Swaminathan Sundararaman
• Extending Flink’s Streaming APIs by Kostas Kloudas
• Table & SQL API – unified APIs for batch and stream processing by Timo Walther
• StreamING models, how ING adds models at runtime to catch fraudsters by Erik de Nooij
• Building a Real-Time Anomaly-Detection System with Flink @ Mux by Scott Kidder
• Blinks Improvements to Flink SQL And TableAPI by Shaoxuan Wang & Xiaowei Jiang
• Apache Flink – The latest and greatest by Jamie Grier
• AthenaX : Uber’s streaming processing platform on Flink by Bill Liu & Haohui Mai

New Videos

• Real-World Best Practices for Big Data Ingestion by Michael Ferguson & Kirit Basu
• Stream Processing with Flink at Netflix by Monal Daxini
• Debunking Common Myths in Stream Processing by Robert Metzger & Ufuk Celebi
• Apache Kafka Best Practices by Manikumar Reddy
• Approaches to achieving real time ingestion & analysis of security events by Sagar Gaikwad
• Unified, Efficient and Portable Data Processing with Apache Beam by Davor Bonaci
• Unifying Stream SQL and CEP for Declarative Stream Processing with Apache Flink by Till Rohrmann
• Spark Streaming and suicidal tendencies by Youen Chéné & Aurélien Vandel
• Machine Learning in the IoT with Apache NiFi by Michael Bironneau
• Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi by Aldrin Piri
• The past, present and future of Apache Storm by Taylor Goetz
• Building a real-time Bank by Andrew McCall
• Using Flink and Queryable State to Buffer High Frequency Time Series Data by Joe Olson
• The Stream Processor as a Database: Building Online Applications directly on Streams by Ufuk Celebi
• Convergence of real-time analytics & data-driven applications by Stephan Ewen
• Real Time Analytics in the real World – Challenges and Lessons at Uber by Chinmay Soman
• Streaming Deep Learning Scenarios with Flink by Dean Wampler
• Runtime Improvements in Blink for Large Scale Streaming at Alibaba by Feng Wang & Zhijiang Wang
• Experiences with Streaming vs Micro-Batch for Online Learning by Swaminathan Sundararaman
• Extending Flink’s Streaming APIs by Kostas Kloudas
• Table & SQL API – unified APIs for batch and stream processing by Timo Walther
• StreamING models, how ING adds models at runtime to catch fraudsters by Erik de Nooij
• Building a Real-Time Anomaly-Detection System with Flink @ Mux by Scott Kidder
• Blinks Improvements to Flink SQL And TableAPI by Shaoxuan Wang & Xiaowei Jiang
• Apache Flink – The latest and greatest by Jamie Grier
• AthenaX : Uber’s streaming processing platform on Flink by Bill Liu & Haohui Mai

New Podcasts

• Stream processing and messaging systems for the IoT age by Ben Lorica

Upcoming Events

• 19.4.2017 (London, UK) – Transitioning from ETL to Streaming – Neha Narkhede (Meetup)
• 19.4.2017 (Lisbon, PT) – First Lisbon Apache Kafka Meetup (Meetup)
• 20.4.2017 (London, UK) – Transitioning from ETL to Streaming – Neha Narkhede (Meetup)
• 20.4.2017 (Sunnyvale, US) – Building Streaming Applications with Apache Storm 1.1 (Meetup)
• 21.4.2017 (Budapest, HU) – Stream processing with R in AWS (Meetup)
• 25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
• 25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
• 25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
• 25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
• 25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
• 27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
• 27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 10.4.2017

This is the 61st edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Lloyds Banking Group prepares for open banking by shifting towards realtime data feeds by Scott Carey
• Machine Learning, Microservices, Streaming Data, IoT, Cloud & Containers by Ellen Friedman
• Architecting and building end-to-end streaming applications by Ben Lorica
• How Real Time Stream Processing Changes Everything About Big Data by Shelly Kramer
• Firing on All Cylinders: The 2017 Big Data Landscape by Matt Turck
• The Next Data Revolution: Intelligent Real-Time Decisions by Ion Stoica
• Beyond Big Data Basics: In-Stream Processing Cures Batch Processing Blues by Robin Miller

Apache Kafka / Kafka Streams / Confluent Platform

• New Quick Start deploys Confluent Platform on the AWS Cloud by AWS blog
• Python Kafka Client Benchmarking by Activision Game Science
• Kafka REST Proxy – Performance Tuning for MapR Streams by Mathieu Dumoulin
• Kafka Connect with Couchbase by David Maier
• Running Kafka Streams applications against a containerized Kafka cluster with Confluent Open Source by Michal Noll
• Kafka Summit NYC: Streaming Pipelines Track – What to expect by Becket Qin & Gwen Shapira

Spark Streaming

• Real-Time End-to-End Integration with Apache Kafka in Apache Spark’s Structured Streaming by Sunil Sitaula

Apache Storm

• Video Tutorial – StormCrawler + Elasticsearch + Kibana by Julien Nioche

Apache Flink

• Continuous Queries on Dynamic Tables by Fabian Hueske
• Flink Aims to Simplify Stream Processing by Alex Woodie
• Apache Flink: Does the world need another streaming engine? by Tony Baer

Apache NiFi

• Data pipelines with NiFi as a Storm alternative by Samra Kasim

New Presentations

• Dealing With Drift – Building an Enterprise Data Lake by Pat Patterson
• Stream Processing with Apache Kafka and .NET by Matt Howlett
• Simplify Governance of Streaming Data by Gwen Shapira
• Implementation of the Lambda Architecture using Apache Apex, Kafka & Impala by Tim Zhu & Sunil Parmar
• Kafka pours and Spark resolves by Alexey Zinoviev
• Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi by Aldrin Piri
• IoT pipeline for anomaly detection at ml kitchen by Mathieu Dumolin & Mateusz Dymczyk
• The Data Dichotomy: Rethinking Data and Services wih Streams by Ben Stopford
• Streaming SQL by Julian Hyde
• Real-Time Visualizations to Apex Apps by Sasha Parfenov
• IoT Stream Processing – Processing Data Streams from Internet of Things by Subu Balakrishnan
• Tibco streaming analytics overview and roadmap by Lou Bajuk
• Build a Real-time Streaming Data Visualization System with Amazon Kinesis Analytics by Allan Macinnis
• Storm/Spark Streaming vs. Apache Apex Comparison by Devendra Tagare
• Machine Learning in the IoT with Apache NiFi by Michael Bironneau
• Introduction to Streaming with Apache Flink by Tugdual Grall

New Podcasts

• Jay Kreps on streaming data, the power of Kafka and a whole lot more by ArchiTECHt
• Architecting and Building End-to-End Streaming Applications by O’Reilly Radar

Upcoming Events

• 11.4.2017 (online) – Real-World Best Practices for Big Data Ingestion (Make BigData Work Webinar)
• 11.4.2017 (Palo Alto, US) – Streaming Data at Uber & A Streaming platform architecture based on Apache KafkaStreaming Data at Uber & A Streaming platform architecture based on Apache Kafka (Meetup)
• 18.4.2017 (Zurich, CH) – Apache Flink Zurich Kickoff (Meetup)
• 19.4.2017 (London, UK) – Transitioning from ETL to Streaming – Neha Narkhede (Meetup)
• 20.4.2017 (London, UK) – Transitioning from ETL to Streaming – Neha Narkhede (Meetup)
• 25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
• 25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
• 25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
• 25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
• 25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
• 27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
• 27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 27.03.2017

This is the 59th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Distributed Stream Processing Frameworks for Fast & Big Data by

  Matthias Niehoff

• Fog Computing Architecture Will Set IoT on Better Path by Steve Jennis
• Applying the Kappa architecture in the telco industry by Nicolas Seyvet & Ignacio Mulas Viela
• At the Speed of Light by Keith Manthey

Apache Kafka / Kafka Streams / Confluent Platform

• Apache Kafka on the Oracle Cloud: My First experiences with Oracle Event Hub Cloud Service by Lucas Jellema
• Implementing The Schema Registry by Callum Leahy
• Flafka: Big Data Solution for Data Silos by Randall V Shane
• Kafka with Docker: A Docker introduction by Nikolaos Georgiou
• Kafka Streams state stores… by Abhishek

Apache Storm

• Taking control of your Apache Storm cluster with tag-aware scheduling by Panos Kasteas
• Storm vs. Hadoop by Suresh Rajagopal 

Apache Flink

• Queryable State in Apache Flink® 1.2.0: An Overview & Demo by Michael Winters & Ufuk Celebi

Apache NiFi

• Mass provisioning of Kismet and Apache MiNiFi in Raspberry Pi using Ansible by Roger Carhuatocto

New Presentations

• Spark Streaming + Kafka Best Practices by Brandon O’Brien
• Design and Implementation of Spark Streaming Connectors by Nan Zhu
• Spark and MapR Streams: A Motivating Example by Ian Downard
• Data Pipelines Made Simple with Apache Kafka by Ewen Cheslack-Postava
• The Rise of Real Time: Apache Kafka and the Streaming Revolution by Jay Kreps
• Multi-Datacenter Kafka by Gwen Shapira
• JustGiving – Serverless Data Pipelines, API, Messaging and Stream Processing by Luis Gonzalez
• A primer on building real time data-driven products by Lars Albertsson
• Getting Started with IoT using Azure, Windows IoT Core, and Raspberry Pi by Chris Pietschmann
• Smart Partitioning in Apache Apex by Tushar Gosavi
• Ingesting Drone Data into Big Data Platforms by Timothy Spann
• Building a Real-Time Data Science Service for Mobile Advertising by Robin Li

New Videos

New Releases

• Apache Flink 1.1.5
• StormCrawler 1.4

Upcoming Events

• 28.3.2017 (Princeton, US) – Apache NiFi: Ingesting Enterprise Data @ Scale (Meetup)
• 29.3.2017 (online) – Part 2: Building Event-Driven Services with Apache Kafka (Confluent Webinar)
• 30.3.2017 (San Francisco, US) – Stream Processing with Apache Kafka and .NET (Meetup)
• 5.4.2017 (London, UK) – Realtime stream processing with coroutines (Meetup)
• 5.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO
• 18.4.2017 (Zurich, CH) – Apache Flink Zurich Kickoff (Meetup)
• 25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
• 27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 20.03.2017

This is the 58th edition of my blog series blog series around Stream Processing and Analytics!

Every week I’m also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Rheos by Connie Yang
• Sports Follies Exemplify Need for Instant Analysis of Streaming Data by Girish Pancha
• If you care about Big Data, you care about Stream Processing by Arpan Shah
• What Does Latency Mean for Stream Processing? by Thomas Oppong
• Event stream processing: Ethical considerations by David Loshin
• Hadoop at Strata: Not Exactly ‘Failure,’ But It Is Complicated by Alex Woodie

Apache Kafka / Kafka Streams / Confluent Platform

• Confluent Delivers First-ever Certification Program for Apache Kafka by Confluent
• Microservices in the Apache Kafka™ Ecosystem by Confluent
• Building a distributed Runtime for Interactive Queries in Apache Kafka with Vert.x by Florian Trossbach
• Stream Oracle data to Kafka; Q&A with the Dbvisit Replicate Connector for Kafka – Part 5 by Mike Donovan
• Configuring and Running Apache Kafka in IBM BigInsights by Simon Nisanth
• Kafka, why and how we have used it? by Sapna Upreti

Twitter Heron

• Optimizing Twitter Heron by Karthik Ramasamy

Spark Streaming

• Do You Know all about Apache Spark Streaming? Read This. by Santosh Bakliwal

New Presentations

• Anomaly detection in real-time data streams using Heron by Arun Kejariwal & Karthik  Ramasamy
• Monitoring Apache Kafka with Confluent Control Center by Nick Dearden
• Distributed stream processing with Apache Kafka by Jay Kreps
• Lessons from a year of supporting Apache Kafka by Dusin Cote & Ryan Pridgeon
• Kafka In and Out by Mohammed Fazuluddin
• Building a healthy data ecosystem around Kafka and Hadoop by Shirshanka Das
• Microservices with Kafka by Laszlo Robert Albert
• Achieving Real-time Ingestion and Analysis of Security Events through Kafka and Storm by Kevin Mao
• Harnessing Data within Hadoop in the Connected Brewery: Kafka, Spark Streaming, and Kudu by Jason Hubbard
• Developing streaming applications with apache apex by David Yan
• Anomaly detection in real-time data streams using Heron by Arun Kejariwal
• From Rivulets to Rivers: Elastic Stream Processing in Heron by Bill Graham & Ashvin Agrawal & Avrila Floratou
• Spark Tips & Tricks by Jason Hubbard
• Spark Streaming – What is Spark Streaming in pictures by Yogesh Kumar

New Videos

• HDInsight: Streaming Petabytes of IoT Data in Real-Time by Raghav Mohan
• Distributed Streaming Platform Using Apache Kafka by Manivannan Sugumar
• Generalized Streaming Pipelines with Spark Streaming by Rafi Kurlansik

Upcoming Events

• 21.3.2017 (online) – Getting Started with Streaming Data and Stream Processing with Apache Kafka (KPI Webinar)
• 21.3.2017 (Vienna, AU) – Tableau and Structured Streaming in Spark (Meetup)
• 21.3.2017 (Chicago, US) – What’s changed with Apache Spark’s Structure Streaming? (Meetup)
• 22.3.2017 (Chicago, US) – Flink Forward San Fran Sneak Peak Double Feature! (Meetup)
• 22.3.2017 (Brighton, US) – How Apache Kafka can change your life (Meetup)
• 23.3.2017 (San Francisco, US) – Bay Area Apache Spark Meetup @ Intel in Santa Clara (Meetup)
• 23.3.2017 (Paris, FR) – Meet Kafka, the distributed log (Meetup)
• 23.3.2017 (Bellevue, US) – Spark Streaming @ Expedia and Implementing Spark Streaming Connector (Meetup)
• 28.3.2017 (Princeton, US) – Apache NiFi: Ingesting Enterprise Data @ Scale (Meetup)
• 29.3.2017 (online) – Part 2: Building Event-Driven Services with Apache Kafka (Confluent Webinar)
• 30.3.2017 (San Francisco, US) – Stream Processing with Apache Kafka and .NET (Meetup)
• 5.4.2017 (London, UK) – Realtime stream processing with coroutines (Meetup)
• 5.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO
• 18.4.2017 (Zurich, CH) – Apache Flink Zurich Kickoff (Meetup)
• 25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
• 27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!