Last Week in Stream Processing & Analytics – 19.6.2017

This is the 71st edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• IBM Throws In with Hortonworks’ Hadoop by Alex Woodie
• The future of Fast and Big Data Technologies by Dana Sandu
• Real-Time Customer Service Will Soon Be the Rule by Joe McKendrick
• Connected Intelligence in the IoT Edge by Rajeev Kozhikkattuhodi

Comparison

• Streaming in Spark, Flink, and Kafka by Shivangi Gupta
• Spark Streaming vs. Kafka Stream by Mahesh Chand Kandpa

Apache Kafka / Kafka Streams / Confluent Platform

• Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks by Raghav Mohan
• Simplifying your microservices architecture with Kafka and KafkaStreams by Dan Lebrero

Spark Streaming

• Apache Spark DStream by DataFlair
• Exploring Spark Structured Streaming by Mahesh Chand Kandpal

Apache Beam

• Beam 2.0 Q and A by Jesse Anderson

Apache NiFi / Hortonworks HDF

• Developing Streaming Analytics Applications without writing a single line of code for Hortonworks DataFlow by Wei Wang

New Presentations

• Stateful Stream Processing with Apache Flink by Tzu-L Gordon
• The History and Future of the Blurring of Stream Processing & OLTP by John Hugg
• From Batch to Streaming ET(L) with Apache Apex by Thomas Weise
• Complex Event Processing with Flink: the state of FlinkCEP by Kostas Kloudas
• Stream Analytics with SQL on Apache Flink by Fabian Hueske
• Apache Spark and Apache Ignite: Where Fast Data Meets the IoT by Denis Magda
• Stream All Things—Patterns of Modern Data Integration by Gwen Shapira
• Connecting Apache Kafka With Mule ESB by Jitendra Bafna
• Introducing Exactly Once Semantics in Apache Kafka by Matthias J. Sax

New Videos

• Rethinking Stream Processing with Apache Kafka by Michael Noll
• From Batch to Streaming ET(L) with Apache Apex by Thomas Weise
• Integration Patterns for Big Data Applications by Michael Häusler
• Experiences running Flink at Very Large Scale by Stephan Ewen
• Challenges of Monitoring Distributed Systems by Nenad Bozic
• Design Patterns for Calculating User Profiles in Real Time by Igor Mazor
• The History and Future of the Blurring of Stream Processing & OLTP by John Hugg
• Using Apache Beam to create a unified benchmarking framework by Ismaël Mejía
• A Big Data Streaming Recipe by Konstantin Gregor
• Balancing Heroes and Pokemon in Real Time by Fokko Driesprong & Vincent Warmerdam
• Containerizing Distributed Pipes by Hagen Tönnies
• Update on the t-digest: Finding Faults in Real Data by Ted Dunning
• Mechanics of Data Pipelines by Sean Braithwaite
• Hops: Multi-Tenancy and Streaming-First in an open-source SaaS platform by Jim Dowling
• Complex Event Processing with Flink: the state of FlinkCEP by Kostas Kloudas
• Stream Analytics with SQL on Apache Flink by Fabian Hueske
• Hands-on with Apache NiFi and MiNiFi by Andrew Psaltis

Upcoming Events

• 20.6.2017 (Düsseldorf, GE) – Big Data & Stream Processing with Apache Spark
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 22.6.2017 (Zurich, CH) – Data stream processing at PSI (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)
• 28.6.2017 (Munich, GE) – Rethinking Stream Processing with Apache Kafka (Meetup)
• 3.7.2017 (Warsaw, PL) – Apache Flink – Stateful Stream Processing (dataArtisans) & CEP (GetInData) (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Advertisements

Last week in Stream Processing & Analytics – 13.6.2017

This is the 70th edition of my blog series blog series around Stream Processing and Analytics!

This weeks edition is really large, as both the videos from Spark Summit SF and Kafka Summit NY talks have been published last week.

Last weekend I have watched some of the presentations of Kafka Summit NY, obviously I was not able to watch them all, but what I have seen so far is definitely worth the time! Thanks to both Confluent and Databricks for providing the recordings of all the sessions!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Stream Processing for Enterprise IT by Robin Miller
• Stream processing in e-commerce by Alexander Dean

Apache Kafka / Kafka Streams / Confluent Platform

• Introduction to Apache Kafka™ for Python Programmers by Matt Howlett
• Load Testing Kafka With Ranger by Matija Gobec
• An Introduction to Kafka Streams by Sangeet Aguli
• Announcing Data Collector ver 2.6.0.0 by Kirit Basu
• Configuring Kafka Connect to log REST HTTP messages to a separate file by Robin Moffatt
• A super quick comparison between Kafka and Message Queues by Hendrik Swanepoel

Spark Streaming

• What’s the big deal with event-by-event streaming in Spark’s 2.2 release?

Heron

• Heron on DC/OS by John Crawford

Apache Flink

• Apache Flink on DC/OS (and Apache Mesos) by Till Rohrmann
• Apache Flink® at MediaMath: Rescaling Stateful Applications in Production by Stefan Richter & Seth Wiesman and Michael Winters

Apache NiFi / Hortonworks HDF

• Real-Time SQL On Event Streams by Mark Payne
• Announcing the general availability of Hortonworks DataFlow 3.0 by Wei Wang
• Dynamic IIoT Dataflows for Demand Based Surveillance via Apache NiFi by Chris Herrera

StreamSets

• Introducing the Data Collector Support Bundle by Wagner Camarao
• Announcing Data Collector ver 2.6.0.0 by Kirit Basu

New Presentations

• Connect K of SMACK:pykafka, kafka-python or? by Shuhsi Lin
• Easy, Scalable, Fault-Tolerant Stream Processing with Structured Streaming in Apache Spark by Michael Armbrust & Tathagata Das
• Etl is Dead; Long Live Streams by Neha Narkhede
• Building Data Product Based on Apache Spark at Airbnb by Jingwei Lu & Liyin Tang
• SSR: Structured Streaming on R for Machine Learning by Felix Cheung
• StreamSets and spark by Hari Shreedharan
• How to implement a real-time streaming data analytics engine by Juantomás García
• Governance using Apache Atlas: Why and How by Vimal Sharma
• Structured Streaming for Columnar Data Warehouses by Jack Gudenkauf
• Fast Data at ING – the why, what and how of the streaming analytics platform in a financial enterprise by Bas Geerdink
• Real-time Machine Learning Analytics Using Structured Streaming and Kinesis Firehose by Caryl Yuhas & Myles Baker
• The History and Future of the Blurring of Stream Processing & OLTP by John Hugg

New Videos

• Building Continuous Application with Structured Streaming and Real Time Data Source by Nan Zhu & Arijit Tarafdar
• Scalable Monitoring Using Apache Spark and Friends by Utkarsh Bhatnagar
• Dynamic DDL: Adding Structure to Streaming Data on the Fly by David Winters & Hao Zou
• Real-Time Machine Learning Analytics Using Structured Streaming and Kinesis Firehose by Caryl Yuhas & Myles Backer
• A Practical Approach to Building a Streaming Processing Pipeline for an Online Advertising Platform by Yifan Wang & Amit Ramesh
• SSR: Structured Streaming on R for Machine Learning by Felix Cheung
• The Top Five Mistakes Made When Writing Streaming Applications by Mark Grover & Ted Malaska
• Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling by Jim Dowling
• Structured Streaming Demo by Michael Armbrust
• Easy, Scalable, Fault-Tolerant Stream Processing with Structured Streaming in Apache Spark (part1, part2) by Michael Armbrust & Tathagata Das
• Expanding Apache Spark Use Cases in 2.2 and Beyond by Matei Zaharia &Tim Hunter & Michael Armbrust
• How to Write Batch or Streaming Data Pipelines with Apache Beam in 15 mins by James Malone
• Structured Streaming for Columnar Data Warehouses by Jack Gudenkauf
• Introducing Exactly Once Semantics in Apache Kafka by Apurva Mehta
• Data Processing at LinkedIn with Apache Kafka by Joel Koshy & Kartik Paramasivam
• The Rise of the Streaming Platform by Jay Kreps
• The Real-time Event Driven Bank: A Kafka Story by Ferd Scheepers
• Stream It Together: 3 Realities of Modern Programming by Neha Narkhede
• Kafka in the Enterprise: What if it Fails? by Anton Gorshkov
• Apache Kafka Core Internals: A Deep Dive by Jun Rao
• How to Lock Down Apache Kafka and Keep Your Streams Safe by Rajini Sivaram
• Running Hundreds of Kafka Clusters with 5 People by Thomas Crayford
• Hardening Kafka for New Use Cases with Venice by Matthew Wise
• Portable Streaming Pipelines with Apache Beam by France Perry
• The Best Thing Since Partitioned Bread: Rethinking Stream Processing with Apache Kafka’s new Streams API by Michael Noll
• Microservices with Kafka: An Introduction to Kafka Streams with a Real-Life Example by Alexis Seigneurin
• Hanging Out With Your Past Self In VR: Time-Shifted Avatar Replication Using Kafka Streams by Greg Fodor
• Building Advanced Streaming Applications using the Latest from Apache Flink and Kafka by Jamie Grier
• The Data Dichotomy: Rethinking Data & Services with Streams by Benjamin Stopford
• Easy, Scalable, Fault-tolerant Stream Processing with Kafka and Spark’s Structured Streaming by Tathagata Das
• Scalable Real-time Complex Event Processing at Uber by Shuyi Chen
• Capture the Streams of Database Changes by Randall Hauch
• Billions of Messages a Day – Yelp’s Real-time Data Pipeline by Justin Cunningham
• California Schemin’! How the Schema Registry has Ancestry Basking in Data by Chris Sanders
• Achieving Predictability And Compliance With The Data Distribution Hub at Bank of New York Mellon by Rajesh Kaveti
• Every Message Counts: Kafka as a Foundation for Highly Reliable Logging at Airbnb by Youssef Francis
• The Source of Truth: Why The New York Times Stores Every Piece of Content Ever Published in Kafka by Boerge Svingen
• Single Message Transformations Are Not the Transformations You’re Looking For by Ewen Cheslack-Postava
• Cloud Native Data Streaming Microservices with Spring Cloud and Kafka by Marius Bogoevici
• Etl is Dead; Long Live Streams by Neha Narkhede

New Releases

• StreamSets 2.6.0.0
• Apache NiFi 1.3.0

Upcoming Events

• 14.6.2017 (Online) – Real-time Data Pipelines with Attunity and Apache Kafka (Attunity Webinar)
• 20.6.2017 (Düsseldorf, GE) – Big Data & Stream Processing with Apache Spark
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 22.6.2017 (Zurich, CH) – Data stream processing at PSI (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)
• 28.6.2017 (Munich, GE) – Rethinking Stream Processing with Apache Kafka (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 5.6.2017

This is the 69th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• MQTT: The Nerve System of IoT by Abhinaya Balaji
• Data Sketching by Graham Cormode
• Real-Time Stream Processing for Internet of Things by Asena Atilla Saunders

Comparison

• Amazon Kinesis vs. Apache Kafka For Big Data Analysis by Christopher Low

Apache Kafka / Kafka Streams / Confluent Platform

• Build Services on a Backbone of Events by Ben Stopford
• The Future of ETL Isn’t What It Used To Be by Gwen Shapira
• How Pandora Uses Kafka by Alex Woodie
• Running Apache Kafka on Docker by Suci Lin

Spark Streaming

• Performance Tuning of an Apache Kafka/Spark Streaming System – Telecom Case Study by Mathieu Dumoulin
• Making Apache Spark the Fastest Open Source Streaming Engine by Michael Armbrust

Storm

• Securing Apache Storm – part IV by Colm O hEigeartaigh
• Securing Apache Storm – part III by Colm O hEigeartaigh
• Securing Apache Storm – part II by Colm O hEigeartaigh
• Securing Apache Storm – part I by Colm O hEigeartaigh

Heron

• Heron on DC/OS by John Crawford

Apache Flink

• Apache Flink 1.3.0 and the Evolution of Stream Processing with Flink by dataArtisans

Apache NiFi

• Real-Time SQL On Event Streams by Mark Payne

Apache Ignite

• Getting Started With Apache Ignite (Part 2) by Akmal Chaudhri

New Presentations

• Introducing Confluent Cloud: Apache Kafka as a Service by Clarke Patterson
• Flink meet DC/OS – Deploying Apache Flink at Scale by Elizabeth K. Joseph & Ravi Yadav
• Spark Streaming – Large-scale near-real-time stream processing by Venkateswaran Kandasamy
• Partitioning for Scalability: Real-Time Streaming Big Data Applications by Vikram Patil
• Apache Flink and IoT: How Stateful Event-Time Processing Enables Accurate Analytics by Aljoscha Krettek
• Event-driven applications with streams and snapshots (Apache Flink) by Stephan Ewen
• Kappa architecture in the telecom industry by Nicolas Seyvet
• Hazelcast Jet – Riding the Jet Streams by Christoph Engelbert
• Portable stateful big data processing in Apache Beam by Aljoschka Krettek
• Realizing the promise of portability with Apache Beam by Tyler Akidau
• Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu by Jeremy Beard
• Structured Streaming Using Spark 2.1 by Yaswanth Yadlapalli
• Dimensions Computation With Apache Apex by Devendra Tagare
• Big & Fast Data for Identity & Telemetry services by Softserve

New Videos

• Introducing Confluent Cloud: Apache Kafka as a Service by Clarke Patterson
• Stream All Things: Patterns of Modern Data Integration by Gwen Shapira
• Stateful stream processing with Apache Spark by Yuriy Taras
• Unified processing with the Samza High-level API by Yi Pan
• Streaming Data Pipelines with Brooklin by Samarth Shetty
• Kafka at half the price by Dong Lin
• How Superset and Druid Power Real-Time Analytics at Airbnb by Hakka Labs
• Partitioning for Scalability: Real-Time Streaming Big Data Applications by Vikram Patil

New Podcast

• New-World BI with BigQuery, Kafka, Streamsets and Looker by Mark Rittman & Steward Bryson

New Releases

• Apache Flink 1.3.0

Upcoming Events

• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 8.6.2017 (Online) – Load Testing Streaming Platforms (Neoty Webinar)
• 8.6.2017 (Utrecht, NL) – Kafka June Meetup (Meetup)
• 8.6.2017 (Online) – ETL is Dead; Long Live Streams (Confluent Webinar)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 13.6.2017 (Berlin, GE) – Rethinking Stream Processing with Apache Kafka (Meetup)
• 14.6.2017 (Online) – Real-time Data Pipelines with Attunity and Apache Kafka (Attunity Webinar)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 22.6.2017 (Zurich, CH) – Data stream processing at PSI (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)
• 28.6.2017 (Munich, GE) – Rethinking Stream Processing with Apache Kafka (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 30.5.2017

This is the 68th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• View Counting at Reddit by Krishnan Chandra
• To Stream or Not to Stream? by Roger Rea

Comparison

• Kafka Connect vs StreamSets: advantages and disadvantages? by Slim Baltagi

Apache Kafka / Kafka Streams / Confluent Platform

• Reliability with Kafka by Brian Anderson & Gwen Shapira
• Log Compaction – Highlights in the Apache Kafka and Stream Processing Community – May 2017 by Yeva Byzek
• Apache Kafka Streams to Build Highly Scalable Machine Learning Microservices by Kai Wähner

Spark Streaming

• Running Streaming Jobs Once a Day For 10x Cost Savings by Burak Yavuz and Tyson Condie

Apache Storm

• What’s new in StormCrawler 1.5 by Julien Nioche

Apache Flink

• A Year of Blink at Alibaba: Apache Flink® in Large Scale Production by dataArtisans

Apache NiFi

• Working with the Official NiFi Docker Image by NiFi

Apache Ignite

• Getting Started With Apache Ignite (Part 1) by Akmal Chaudhri

New Presentations

• Apache Kafka Streams + Machine Learning / Deep Learning by Kai Wähner
• Will it Scale? The Secrets behind Scaling Stream Processing Applications by Navina Ramesh
• Creating Docker-based Container Clusters with clusterdock by Pat Patterson
• Apache Kafka – an introduction by Joris Meijer
• Rabo Alerts: Real-Time notifications using Apache Kafka by Jeroen van Disseldorp
• Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, Streams vs. Databases by Michael Noll
• Developers guide for building real-time clickstream pipeline with Snowplow Apach Kafka and BigQuery by Evaldas Miliauskas
• Spark streaming + kafka 0.10 by Joan Viladrosa
• Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark by Burak Yavu
• Structured streaming for machine learning by Seth Hendrickson
• Five Early Challenges Of Building Streaming Fast Data Applications by Craig Blitz
• Couchbase and Kafka: scalable data streaming by Tyler Mitchell & David Tucker
• The Rise of Real Time by Neha Narkhede by Neha Narkhede
• Routing Trillions of Events Per Day @Twitter by Gary Steelman

New Videos

• Functional Streams with Kafka by Luis Reis & Rui Batista at LX Scala by Luis Reis & Rui Batista
• Couchbase and Kafka: scalable data streaming by Tyler Mitchell & David Tucker
• Building Robust and Scalable Data Pipelines with Kafka by Sam Kimbrel
• Five Early Challenges Of Building Streaming Fast Data Applications by Craig Blitz
• Explore Spark 2.0 and structured streaming in Microsoft Azure HDInsight by Maxim Lukiyanov
• Data Stream Processing: Can We Finally Forget The Batches? by Dominik Wagenknecht

New Releases

• StormCrawler 1.5a

Upcoming Events

• 30.5.2017 (Frankfurt, GE) – First Kafka Meetup (Meetup)
• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 8.6.2017 (Online) – Load Testing Streaming Platforms (Neoty Webinar)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 22.5.2017

This is the 67th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Anomaly Detection in Telecommunications Using Complex Streaming Data by Ted Dunning
• The Future of Event Stream Processing by Michael Drogalis
• How to avoid downtime and disruption when moving data by David Richards
• The Anatomy of an IoT Solution by Hans de Visser

Apache Kafka / Kafka Streams / Confluent Platform

• Reading Data Securely From Apache Kafka to Apache Spark by Mark Grover
• Kafka Topic Design Checklist by Jesse Anderson
• Building Real-Time data pipelines with Kafka Connect by Slim Baltagi
• Trialing Single Message Transformations in Kafka Connect by Mike Donovan

Spark Streaming

• Taking Apache Spark’s Structured Structured Streaming to Production by Bill Chambers & Michael Armbrust
• Running Streaming Jobs Once a Day For 10x Cost Savings by Burak Yavuz & Tyson Condie

Apache Flink

• A Year of Blink at Alibaba: Apache Flink in Large Scale Production by Xiaowei Jiang

AWS Kinesis

• Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight by Karan Desai

Apache NiFi

• Monitoring NiFi – Scripted Reporting Task by Pierre Villard
• Monitoring NiFi – Ambari & Grafana by Pierre Villard
• Monitoring NiFi – Workflow SLA by Pierre Villard
• Monitoring NiFi – Site2Site reporting tasks by Pierre Villard
• Monitoring NiFi – Logback configuration by Pierre Villard
• Record-oriented Data with NiFi by Mark Payne

New Presentations

• Real Time Analytics – Stream Processing by Mahesh Madushanka
• Streaming Analytics for IoT-Oriented Applications by Adrian Bowles
• Geo-Distributed Big Data and Analytics by Ted Dunning
• Top 5 mistakes when writing Streaming applications by Mark Grover & Ted Malaska
• Fast Cars, Big Data How Streaming can help Formula 1 by Carol McDonald
• Stateful streaming data pipelines by Chandni Singh & Timothy Farkas
• Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform by Chrix Finne & Bob Lehmann
• Microservices interaction at scale using Apache Kafka by Ivan Ursul
• Capture the Streams of Database Changes by Randall Hauch
• Stream Processing Metamorphosis – A Kafka’s tale by João Vazão Vasques
• Event processing with Apache Storm by Nuno Caneco

New Podcasts

•  Apache Kafka, Streaming Data Integration and Schema Registry by Mark Rittman with Gwen Shapira

Upcoming Events

• 22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
• 23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
• 23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
• 24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
• 30.5.2017 (Frankfurt, GE) – First Kafka Meetup (Meetup)
• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 17.5.2017

This is the 66th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Fast Cars, Big Data – How Streaming Data Can Help Formula 1 by Carol McDonald
• Thoughts on Building a Successful Streaming Analytics Platform by George Vetticaden
• A Digital Reference Architecture for the Industrial Internet Of Things (IIoT) by Vamsi Chemitiganti
• Introducing DevHigh5 Project Pravega: Storage reimagined for a streaming world by Salvatore DeSimone

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka Summit New York: The Rise of the Streaming Platform by Clarke Patterson
• Reading data securely from Apache Kafka to Apache Spark by Mark Grover
• Installing and Running Kafka on an AWS Instance by Sahil Sawhney
• Managing Spark and Kafka Pipelines by Larry Murdock
• Getting Started with the Kafka Streams API using Confluent Docker Images by Michael Noll
• Kafka Consumer Architecture – Consumer Groups and subscriptions by Cloudurable
• The new version of Kafka Topics UI is now available! by Christina Daskalaki
• Startup Confluent makes Kafka cloud play to ease streaming deployment by Jack Vaughan
• Securing Apache Kafka with Kerberos by Colm O hEigeartaigh
• Apache Kafka – Introduction & Installation by Rudra Narayan Garnaik
• Visualizing the flow of data as it moves through a Kafka cluster by James Cheng

Spark Streaming

• How to Get Started with Spark Streaming and MapR Streams Using the Kafka API by Carol McDonald

Apache Beam

• Apache Beam publishes the first stable release by Davor Bonaci

Apache Flink

• Introducing Official Docker Images for Apache Flink by Patrick Lucas & Ismaël Mejía

StreamSets

• Visualizing and Analyzing Salesforce Data with Neo4j by Pat Patterson
• Calling External Libraries from the JavaScript Evaluator by Pat Patterson

Apache NiFi

• Apache NiFi and TLS Toolkit Ansible Roles to create a multi-node secure NiFi cluster by Roger Carhuatocto
• Apache Nifi Sensitive Value Encryption by Kya
• Record based processors in Apache NiFi 1.2 by Haimo Liu

New Presentations

• Intro to Apache Kafka by Jason Hubbard
• Stream processing using Kafka by Himani Arora & Prabhat Kashyap
• Rethinking Services With Stateful Streams by Ben Stopford
• Connect S3 with Kafka leveraging Akka Streams by Seiya Mizuno
• Stream Processing with MicroProfile and Apache Kafka by Matthias Wessendorf
• Real-time Data Pipelines with SAP and Apache Kafka by Carole Gunst
• Processing IoT Data with Apache Kafka by Confluent
• Functional streams with Kafka – A comparison between Akka-streams and FS2 by Luis Reis & Rui Batista
• Open Blueprint for Real-Time Analytics with In-Stream Processing (ISP) by Victoria Livschitz
• Dataservices – Processing Big Data The Microservice Way by Josef Adersberger

New Video

• Scalable Stream Processing: A Survey of Storm, Samza, Spark and Flink by Felix Gessert
• Improve your App UX with one model processing insights from batch and streams by Robert Kubis
• Apache Ignite: Real Time Processing of IoT Generated Streaming Data by Denis Magda
• Rethinking Microservices with Stateful Streams by Ben Stopford
• Fast Cars, Big Data – How Streaming Can Help Formula 1 by Tugdual Grall

New Releases

• Apache Beam 2.0.0

Upcoming Events

• 17.5.2017 (Mountain View, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
• 18.5.2017 (online) – Conquering Data in Motion with a Modern Ingestion Platform (Cloudera Webinar)
• 22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
• 23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
• 23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
• 24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics – 26.4.2017

This is the 63rd edition of my blog series blog series around Stream Processing and Analytics! A few days later than planed, but I had some busy days with speaking on Apache Kafka at Oracle Code Berlin on Monday and today we held our Trivadis/DataStax Hands-On workshop in Zurich. So next week will be again on time (Monday), I hope 😉

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

Comparison

• Event time processing in Apache Spark and Apache Flink by Matthias Niehoff

Apache Kafka / Kafka Streams / Confluent Platform

• Kafka channels the big data firehose by Tony Baer
• Introducing Kasper: The Kafka Stream Processing Library Built For Go by Nicolas Maquet
• Benchmarking Apache Kafka with Aiven – Part 1: Write throughput (AWS vs Azure vs DigitalOcean vs Google Cloud vs UpCloud) by Aiven

Spark Streaming

• Fault tolerance in Apache Spark – Reliable Spark Streaming by Data Flair
• Real-Time Weather Event Processing With HDF, Spark Streaming, and Solr by Balaji Kandregula
• Spark Streaming – under the hood by veejayendraa
• Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2 by Kunal Khamar & Tyson Condie & Michael Armbrust

Apache Apex

• One Year Anniversary of Apache Apex by Thomas Weise

Apache Flink

• Build a Real-time Stream Processing Pipeline with Apache Flink on AWS by Steffen Hausmann

StreamSets

• StreamSets Data Collector v2.5 Adds IoT, Spark, Performance and Scale by Kirit Basu
• Make sense of stream processing by Pat Patterson
• Creating a Custom Multithreaded Origin for StreamSets Data Collector by Pat Patterson

Apache Nifi

• Basics of Apache Nifi: 1 by Malcom Thirus
• How to send data from Apache NIFI to HANA by Franco Lezana

Azure Stream Analytics

• Announcing Azure Stream Analytics on edge devices (preview) by Santosh Balasubramanian

New Presentations

• Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema
• Kafka used at scale to deliver real-time notifications by Sergio Nunes
• Apache Storm In Retail Content by Karthik Deivasigamani
• Storm worker redesign by Roshan Naik
• From Three Nines to Five Nines – A Kafka Journey by Allen Wang
• Microsoft challenges of a multi tenant kafka service by Nitin Kumar
• The Rise of Real-Time by Neha Narkhede
• Apache Kafka – Scalable Message-Processing and more ! by Guido Schmutz
• Building Streaming Applications with Apache Storm 1.1 by Hugo Louro
  
• Kafka Tutorial – basics of the Kafka streaming platform by Jean-Paul Azar
• Bootstrapping Microservices with Kafka, Akka and Spark by Alex Silva

New Videos

• Building Faster Streaming Applications with Apache Storm 1.1 by Suresh Srinivas & Hugo Louro & Karthik Deivasigamani
  
• Putting the Micro into Microservices with Stateful Stream Processing by Ben Stopford
• StreamING models, how ING adds models … by Erik de Nooij
• Real-Time Analytics On Data Streams – A New Era For Big Data And IoT by Jeremy Hillier
• Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema

New Podcasts

• Data science for real-time streaming analytics by Roger Rea

New Releases

• StreamSets 2.5

Upcoming Events

• 25.4.2017 (online) – Part 3: Putting the Micro into Microservices with Stateful Stream Processing (Confluent Webinar)
• 25.4.2017 (online) – Deploying Hazelcast IMDG and Apache Kafka for IoT Stream Processing and Analytics (Hazelcast Webinar)
• 25.4.2017 (online) – Real-time data processing with Apache Flink and DC/OS (Mesosphere Webinar)
• 25.4.2017 (Washington, US) – Kafka Connect and Kafka Streams with Jay Kreps, Confluent co-founder & CEO (Meetup)
• 25.4.2017 (Atlanta, US) – Getting Started with Structured Streaming (SparkHub Meetup)
• 27.4.2017 (Munich, GE) – 3rd Flink Meetup Munich: Flink 1.3, Queryable State & Spark Streaming (Meetup)
• 27.4.2017 (Palo Alto, US) – Criteo Labs Tech Talks: Data Stream Processing @ Scale (Meetup)
• 27.4.2017 (Charlotte, US) – Apache NiFi – An Uber Data Flow Tool (Meetup)
• 16.5.2017 (online) – Geo-Distributed Big Data and Analytics (MapR Webinar)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!