Last Week in Stream Processing & Analytics – 14.8.2017

This is the 77th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Complete event trend detection in high-rate data streams by Adrian Colyer
• How to Check the Vital Signs of Real-Time Patient Monitoring by Kevin Jordan
• Don’t Have Real-Time Analytics? Your Insights Are Already Too Late by Shelby Blitz
• Apache Metron Streaming Architecture Stellar: “the Excel of Cybersecurity” by Casey Stella
• How To Conquer Your Dataflow Chaos by Arvind Prabhakar
• Lessons from LinkedIn: Faster Insights Through a Unified Data Ecosystem by Kat Campise

Apache Kafka / Kafka Streams / Confluent Platform

• So, you want to build a Kafka Connector? Source edition by Chris Riccomini
• The Simplest Useful Kafka Connect Data Pipeline In The World … or Thereabouts by Robin Moffat
• Kafka Architecture by Jean-Paul Azar
• What Is Kafka?  by Jean-Paul Azar
• Kafka Producer in Java  by Jean-Paul Azar
• Why Kafka Streams didn’t work for us? – Part 1 by Alexis Seigneurin
• Why Kafka Streams didn’t work for us? – Part 2 by Alexis Seigneurin
• Why Kafka Streams didn’t work for us? – Part 3 by Alexis Seigneurin
• Cassandra to Kafka Data Pipeline (Part 1) by Vladimir Vajda
• Dresses, Stream Processing and Apache Kafka by Cristina Negrean
• How to Publish Data Collector Tables to Apache Kafka by Soniya

Amazon AWS

• Building a Real World Evidence Platform on AWS by Aaron Friedman

Spark

• Apache Spark’s Structured Streaming with Amazon Kinesis on Databricks by Jules Damji
• Apache Spark Streaming by Shreya Gupta
• Spark Streaming Kerberized Kafka by Talend Community

Apache Flink

• Apache Flink: Stream Processing by lakshaysuri
• An Introduction to Apache Flink by Dursun Koç

Apache Storm

• Analyzing Twitter Data With Apache Storm, Power BI, and Azure SQL Server by Nischal S

Apache NiFi / Hortonworks HDF

• Running SQL on FlowFiles using QueryRecord Processor (Apache NiFi 1.2+) by Andrew Lim
• TensorFlow and NiFi: Big Data AI Sandwich by Tim Spann
• Apache NiFi on Hortonworks HDF Verses … Microsoft Flow? by Ian Roseman
• MiNiFi: Transmitting and Storing From Devices to Apache Hive by Tim Spann
• Creating Word Clouds From DataFlows With Apache NiFi and Python by Tim Spann

New Presentations

• NiFi Best Practices for the Enterprise by Greg Keys
• Building Streaming Data Applications Using Apache Kafka by Slim Baltagi
• Monitoring Big Data Hadoop Apex Apps by Ajay Gupta
• Hadoop made fast – Why Virtual Reality Needed Stream Processing to Survive by Greg Fodor & Gehrig Kunz
• Introducing Hazelcast Jet by Rahul Gupta
• HDF 3.0 Deep Dive by Aldrin Piri

New Releases

• Apache Pulsar 1.19.0-incubating

Upcoming Events

• 15.8.2017 (Milwaukee, US) – Sensors, Spark and Kafka: Applied Machine Learning (Meetup)
• 15.8.2017 (Los Angeles, US) – Kafka Connect Implementation at GumGum and more (Meetup)
• 16.8.2017 (online) – Pandora Plays Nicely Everywhere with Real-Time Data Pipelines (Confluent Webinar)
• 16.8.2017 (online) – Data Warehouse Modernization: Accelerating Time-to-Action (MapR Webinar)
• 16.8.2017 (online) – Tungsten Replicator for Kafka – Real-time Data Loading from Oracle and MySQL into Apache Kafka (Continuent Webinar)
• 24.8.2017 (San Francisco, US) – Visualizing and Analyzing Salesforce Data with StreamSets and Neo4j (Meetup)
• 27.8.2017 (online) – Embrace Streaming Analytics and Transform Your Business (AWS Webinar)
• 30.8.2017 (Dublin, IR) – The SMACK Stack (Meetup)
• 16.9.2017 (McLean, US) – Spark Saturday DC (Meetup)
• 18.9.2017 (Zurich, CH) – First Kafka Meetup Zurich (Meetup)
• 19.9.2017 (online) – Real-time marketing analytics with stream processing (O’Reilly Webcast)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

 

Last week in Stream Processing & Analytics – 22.5.2017

This is the 67th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Anomaly Detection in Telecommunications Using Complex Streaming Data by Ted Dunning
• The Future of Event Stream Processing by Michael Drogalis
• How to avoid downtime and disruption when moving data by David Richards
• The Anatomy of an IoT Solution by Hans de Visser

Apache Kafka / Kafka Streams / Confluent Platform

• Reading Data Securely From Apache Kafka to Apache Spark by Mark Grover
• Kafka Topic Design Checklist by Jesse Anderson
• Building Real-Time data pipelines with Kafka Connect by Slim Baltagi
• Trialing Single Message Transformations in Kafka Connect by Mike Donovan

Spark Streaming

• Taking Apache Spark’s Structured Structured Streaming to Production by Bill Chambers & Michael Armbrust
• Running Streaming Jobs Once a Day For 10x Cost Savings by Burak Yavuz & Tyson Condie

Apache Flink

• A Year of Blink at Alibaba: Apache Flink in Large Scale Production by Xiaowei Jiang

AWS Kinesis

• Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight by Karan Desai

Apache NiFi

• Monitoring NiFi – Scripted Reporting Task by Pierre Villard
• Monitoring NiFi – Ambari & Grafana by Pierre Villard
• Monitoring NiFi – Workflow SLA by Pierre Villard
• Monitoring NiFi – Site2Site reporting tasks by Pierre Villard
• Monitoring NiFi – Logback configuration by Pierre Villard
• Record-oriented Data with NiFi by Mark Payne

New Presentations

• Real Time Analytics – Stream Processing by Mahesh Madushanka
• Streaming Analytics for IoT-Oriented Applications by Adrian Bowles
• Geo-Distributed Big Data and Analytics by Ted Dunning
• Top 5 mistakes when writing Streaming applications by Mark Grover & Ted Malaska
• Fast Cars, Big Data How Streaming can help Formula 1 by Carol McDonald
• Stateful streaming data pipelines by Chandni Singh & Timothy Farkas
• Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform by Chrix Finne & Bob Lehmann
• Microservices interaction at scale using Apache Kafka by Ivan Ursul
• Capture the Streams of Database Changes by Randall Hauch
• Stream Processing Metamorphosis – A Kafka’s tale by João Vazão Vasques
• Event processing with Apache Storm by Nuno Caneco

New Podcasts

•  Apache Kafka, Streaming Data Integration and Schema Registry by Mark Rittman with Gwen Shapira

Upcoming Events

• 22.5.2017 (Los Angeles, US) – Apache Kafka & the Riseof real-time (Meetup)
• 23.5.2017 (online) – Deploy and Scale a Real-Time Streaming Data Platform with Confluent on DC/OS (Mesosphere Webinar)
• 23.5.2017 (London, UK) – Stream Analytics with SQL on Apache Flink (Meetup)
• 24.5.2017 (online) – Simplifying Real-Time Architectures for IoT with Apache Kudu (Cloudera Webinar)
• 30.5.2017 (Frankfurt, GE) – First Kafka Meetup (Meetup)
• 7.6.2017 (San Francisco, US) – Integrated Dataflow Processing with Spark and StreamSets (Meetup)
• 12.6.2017 (Columbia, US) – Latest developments in Apache NiFi and MiNiFi (Meetup)
• 22.6.2017 (Canberra, Australia) – Data-in-Motion: Recent advances in Apache projects for streaming data (Meetup)
• 27.6.2017 (London, UK) – Stateful Stream Processing (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics 12/26/2016

Happy Holidays!

This is the Christmas edition and therefore rather compact! It’s 46th installment of my blog series around Stream Processing and Analytics.

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Big Data Industry Predictions for 2017 by Daniel Gutierrez
• The Future of Streaming Analytics in Financial Services by Mark Palmer
• Big data: Why online gambling should bet on streaming analytics by Jay Patani
• Embedded analytics to feel widest impact of machine learning projects by Jack Vaughan

Spark Streaming

• 24/7 Spark Streaming on YARN in Production by Bernhard Schäfer

Apache Flink

• Apache Flink in 2016: Year in Review by Mike Winters
• Apache Flink: The Next Gen Big Data Analytics Framework by Sibanjan Das

StreamSets

• Calling External Java Code from Script Evaluators by Pat Patterson
• Building an Amazon SQS Custom Origin for StreamSets Data Collector by Pat Patterson

Apache NiFi / Hortonworks Data Flow (HDF)

• Apache NiFi 1.x Cheatsheet by Tim Spann
• An Example WebSocket Application in Apache NiFi 1.1 by Timothy Spann
• Submitting Spark Jobs From Apache NiFi Using Livy by Timothy Spann
• Trigger SonicPi Music Via Apache NiFi by Timothy Spann
• Running DNS and Domain Scanning Tools From Apache NiFi by Timothy Spann
• Adding A Custom Processor to NiFi : LinkProcessor by Timothy Spann
• Drones (UAV) Data Ingest Methods by Timothy Spann

New Presentations

• Event Driven Architecture by Benjamin Joyen-Conseil
• Deep dive of Deduplication using Apache Apex and RTS by Bhupesh Chawda
• Building a Real-Time Streaming Data Platform on AWS by Ryan Nienhuis & Ram Kumar Rengaswamy
• Structured streaming in Spark by Giri R Varatharajan

New Releases

• Apache Flink 1.1.4

Upcoming Events

• 1/7/2017 (Istanbul) – A Deep-dive into Structured Streaming / Predictive Analytics with SparkR (Spark Hub Meetup)
• 1/11/2017 (online) – Top 5 IoT Use Cases (Cloudera Webinar)
• 1/11/2917 (Washington, US) – ntroduction to Kafka Streams with a Real-Life Example (Meetup)
• 1/18/2017 (London, UK) – Instrumenting Apache Kafka (Meetup)
• 1/18/2017 (Palo Alto, US) – Apache Kafka Meetup with Walmart Labs and Confluent (Meetup)
• 1/25/2017 (Munich, GE) – Apache Kafka with Jay Kreps (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!