Last Week in Stream Processing & Analytics – 11.12.2017

This is the 94th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Comparing Pulsar and Kafka: how a segment-based architecture delivers better performance, scalability, and resilience by Sijie Guo

Apache Kafka / Kafka Streams / Confluent Platform

• Validating Topic Configurations in Apache Kafka by Florian Trossbach
• Kafka stream processing via Lenses SQL, scale with Kafka – part 3 by Andrew Stevenson
• Handling GDPR: How does a log forget? by Ben Stopford
• If You Want To Access Kafka From Hive, Then Read This by Melliyal Annamalai

Apache Pulsar

• Talking Apache Pulsar with Matteo and Sijie from Streamlio by Jhon Masschelein

Spark Streaming

• Monitoring Spark Streaming with InfluxDB and Grafana by Christian Gügi

StreamSets

• Generate your Avro Schema – Automatically! by Pat Patterson

New Presentations

• What is Kafka & why is it Important? by Lucas Jellema
• Improving Kafka at-least-once performance at Uber by Ying Zheng
• Scan and go with the flow: how I met Kafka by Matteo Ferroni
• Stream Processing using Samza SQL by Srinivasulu Punuru
• Unified Stream Processing at Scale with Apache Samza by Jake Maes
• Anomaly Detection and Spark Implementation by Maxim Shkarayev & Anand Venugopal & Punit Shah
• Netflix Keystone SPaaS by Monal Daxini 
• Serverless Architecture and Best Practices by Itzik Paz
• How Netflix Monitors Applications in Near Real-time with Amazon Kinesis by John Bennett & Roy Ben-Alta
• Dealing with Drift: Building an Enterprise Data Lake by Pat Patterson
• You Can’t Search Without Data by Bryan Bende
• Nifi by Julio Castro
• Design patterns and best practices for data analytics with Amazon EMR by Jonathan Fritz & Anya Bida
• Regain Control Thanks To Prometheus by Guillaume Lefevre & Etienne Coutaud
• Building IoT applications with Apache Spark and Apache Bahir by Luciano Resende

New Videos

• Improving Kafka at-least-once performance at Uber by Ying Zheng
• Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema
• A journey from batch to streaming with Kafka Streams by Sander Uiterkamp & Jeroen Resoort
• Stream Processing using Samza SQL by Srinivasulu Punuru

Upcoming Events

• 12.12.2017 (online) – Living on the Edge Ultra Lightweight Data Movement for IoT (StreamSets Webinar)
• 18.12.2017 (Frankfurt, GE) – Real-Time Data-Ingest with Apache NiFi for IoT and Streaming Analytics (Meetup)
• 19.12.2017 (Melbourne, AU) – First Kafka Meetup (Meetup)
• 17.1.2018 (Princeton, US) – Advanced Apache NiFi Flows and Updates (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Advertisements

Last Week in Stream Processing & Analytics – 3.12.2017

This is the 93rd edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Comparing Pulsar and Kafka: unified queuing and streaming by Sijie GUo

Apache Kafka / Kafka Streams / Confluent Platform

• Better late then never : Time to replace your micro-service architecture with Kafka… by Mukesh Kumar
• ‘Go’ with Kafka & MySQL on Oracle Cloud by Abhishek Gupta
• Confluent Cloud: Enterprise-Ready, Hosted Apache Kafka is Here! by Neha Narkhede
• Introducing Confluent Platform 4.0 by Sriram Subramanian
• Toward a Functional Programming Analogy for Microservices by Bobby Calderwood
• Running Kafka Streams applications in AWS by Ian Duffy & Nina Hanzlikova
• Apache Kafka and GDPR Compliance by Antonios Chalkiopoulos
• Handling GDPR: How to make Kafka Forget by Ben Stopford

Heron

• The Heron Stream Processing Engine on Google Kubernetes Engine by Chris Kellogg

Apache Flink

• Apache Flink 1.4 Lays Groundwork for Unified Stream and Batch Processing by Alex Handy
• Real-time applications are going places by George Anadiotis

Spark Streaming

• Apache Spark – Kafka Integration for Real-time Data Processing with Scala by Omal Perera

Apache NiFi

• Integrating Apache Spark 2.x Jobs with Apache NiFi 1.4+ by Timothy Spann
• Nifi Alienvault Open Threat Exchange connection by Conor O’Callaghan

StreamSets

• Announcing StreamSets Data Collector Edge by Kirit Basu
• Announcing StreamSets Data Collector version 3.0 by Kirit Basu
• StreamSets Delivers Ultralight Open Source Ingestion for Edge Devices by insideBIGDATA

New Presentations

• Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford
• Introduction to Apache Kafka and Confluent… and why they matter by Confluent
• Kafka Connect & Kafka Streams/KSQL – the ecosystem around Kafka by Guido Schmutz
• ProtectWise optimizes performance of Cassandra and Kafka workloads with Amazon EBS by Gene Stevens & Robert Tarrall & Andrey Zaychikov
• Cell based security in Kafka by Tibor Várkony
• Divide, distribute and conquer stream v. batch by Victor Gamov
• Combining Batch and Stream Processing to Get the Best of Both Worlds 40 views by Rajeev Srinivasan & Ujjawal Ratan
• Expedia Flies with DynamoDB Lightning Fast Stream Processing for Travel Analytics by Brandon O’Brien
• Big Data Architectural Patterns and Best Practices on AWS by Siva Raghupathy
• Don’t Wait Until Tomorrow How to Use Streaming Data to Gain Real-time Insights into Your Business by Jeremy Zigmond & Allan MacInnis
• Analyzing Streaming Data in Real Time with Amazon Kinesis by Ryan Nienhuis
• Getting Started with Serverless Computing Using AWS Lambda by Chris Munns & Nicky Joshi
• Embrace the Anarchy : Apache Kafka’s Role in Modern Data Architectures by Robin Moffatt
• Building a Real-Time Streaming Platform with Oracle, Apache Kafka, and KSQL by Robin Moffatt

New Videos

• Netflix Keystone SPaaS: Real-time Stream Processing as a Service by Monal Daxini
• Combining Batch and Stream Processing to Get the Best of Both Wo by Rajeev Srinivasan & Ujjawal Ratan
• Getting Started with Serverless Computing Using AWS Lambda by Chris Munns & Nicky Joshi
• Expedia Flies with DynamoDB Lightning Fast Stream Processing for Travel Analytics by Brandon O’Brien
• Building High-Throughput Serverless Data Processing Pipelines by Cecilia Deng
• Bridge to the Cloud: Using Apache Kafka to Migrate to AWS by Jay Krebs

New Releases

• StormCrawler 1.7
• Confluent Platform 4.0
• StreamSets Data Collector 3.0

Upcoming Events

• 4.12.2017 (Sunnyvale, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)
• 6.12.2017 (Manchester, UK) – An evening of Apache Kafka (Meetup)
• 19.12.2017 (Melbourne, AU) – First Kafka Meetup (Meetup)
• 17.1.2018 (Princeton, US) – Advanced Apache NiFi Flows and Updates (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 21.11.2017

This is the 91st edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Why Event Driven? by Conor Clifford

Apache Kafka / Kafka Streams / Confluent Platform

• Apache Kafka Streaming, count on SQL by Stefan Bocutiu
• Transactions in Apache Kafka by Apurva Mehta & Jason Gustafson
• Sizing your Event Hub (Apache Kafka) Cluster on Oracle Cloud by Kunal Rupani
• Kafka on Rails: Using Kafka with Ruby on Rails — Part 1 — Kafka basics and its advantages by Maciej Mensfeld
• Deploying Kafka-Dependent Scala Microservices With Docker by Saumitra Srivastav
• Cassandra to Kafka Data Pipeline (Part 2) by Vladimir Vajda
• Spring Cloud Stream and Apache Kafka based Microservices on Oracle Cloud by Abhishek Gupta
• Installing Oracle GoldenGate for Big Data 12.3.1 with Kafka Connect and Confluent Platform by Robin Moffatt

Apache Spark Streaming

• Getting Streaming data from Kafka with Spark Streaming using Python by Mukesh Kumar

Apache Pulsar

• A Developer’s Introduction to the Pulsar Streaming Messaging System by Matteo Merli

StreamSets

• Bulk Loading Zone by Mike Fuller

Azure Stream Analytics

• Azure Stream Analytics now available on IoT Edge by Jean-Sebastien Brunner

New Presentations

• Metrics are Not Enough: Monitoring Apache Kafka by Gwen Shapira
• Apache Spark Streaming + Kafka 0.10: an Integration Story by Joan Viladrosa Riera
• Data Stream Processing – Concepts and Frameworks by Matthias Niehoff
• Modern Stream Processing With Apache Flink by Till Rohrmann
• Enabling Real-Time Business with Change Data Capture by Rupal Shah & Audrey Egan
• Modern Stream Processing With Apache Flink by Till Rohrmann
• Unified Stream Processing at Scale with Apache Samza by Jacob Maes
• Apache Flink & Kudu: a connector to develop Kappa architectures by Nacho García Fernández
• Developing Java Streaming Applications with Apache Storm by Lester Martin

New Videos

• Unified Data Management: The Best of Data Lakes, Data Warehouses and Streaming by Jason Pohl & Bill Chambers
• Event Bus as Backbone for Decoupled Microservice Choreography by Lucas Jellema
• A journey from batch to streaming with Kafka Streams by Sander Schoot Uiterkamp & Jeroen Resoort

New Releases

• Confluent 3.3.1
• Heron 0.17.0
• Debezium 0.6.2

Upcoming Events

• 29.11.2017 (Berne, US) – Apache Kafka Skalierbare Nachrichtenverarbeitung und mehr! (Java User Group Switzerland)
• 29.11.2017 (Milano, IT) – First Italian Kafka meetup – An introduction by Confluent and a production case (Meetup)
• 30.11.2017 (Online) – A Tale of Two APIs: Using Spark Streaming In Production (Lightbend Webinar)
• 4.12.2017 (Sunnyvale, US) – Stream Processing with Apache Kafka & Apache Samza (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 30.10.2017

This is the 88th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Making Apache Spark the Most Versatile and Fast Data Platform Ever by Pierce Lamb
• Better Machine Learning Model Management: A Streaming Approach by Ted Dunning & Ellen Friedman
• IOT: Streaming Analytics is an Enabling Technology by Avi Rosenthal
• Best-Ever Algorithm Found for Huge Streams of Data by Kevin Hartnett

Apache Kafka / Kafka Streams / Confluent Platform

• How to use Apache Kafka to transform a batch pipeline into a real-time one by Stéphane Maarek
• Running Kafka Streams Applications in AWS by Ian Duffy & Nina Hanzlikova
• Reattaching Kafka EBS in AWS by Andrey Dyachkov
• Data Streaming With WildFly Swarm and Apache Kafka by Ken Finnigan
• The London Kafka Summit 2018 Call for Papers is Open! by Tim Berglund
• Confluent Releases KSQL, a Distributed Streaming SQL Engine for Apache Kafka by Srini Penchikala by Srini Penchikala

Apache Pulsar / Heron / Streamlio

• Strata Presentation: “Messaging, storage, or both” (Part 1) by Matteo Merli

Apache Spark Streaming

• Databricks Delta: A Unified Management System for Real-time Big Data by Michael Armbrust, Bill Chambers and Matei Zaharia
• Making Apache Spark the Most Versatile and Fast Data Platform Ever by Pierce Lamb
• Databricks Puts ‘Delta’ at the Confluence of Lakes, Streams, and Warehouses by Alex Woodie

Apache Flink

• StreamING Machine Learning Models: How ING Adds Fraud Detection Models at Runtime with Apache Flink by Erik de Nooij

Apache NiFi

• Tales from the Spark Summit: Exploring Apache Nifi by Richard Conway

Streamsets

• Evolving Avro Schemas with Apache Kafka and StreamSets Data Collector by Pat Patterson
• How to Convert Apache Sqoop™ Commands Into StreamSets Data Collector Pipelines by Clarke Patterson
• Getting Started with Cloudera’s Cybersecurity Solution (feat. StreamSets, Arcadia Data and Centrify) by Rick Bilodeau

New Presentations

• Common Patterns of Multi Data-Center Architectures with Apache Kafka by Gwen Shapira
• Apache Kafka + Apache Mesos + Kafka Streams – Highly Scalable Streaming Microservices by Kai Waehner
• Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, Streams vs. Databases by Michael Noll
• Disaster Recovery Plans for Apache Kafka by Gwen Shapira
• Easy, scalable, fault tolerant stream processing with structured streaming by Tathagata Das
• Apache Kafka Streams – Building distributed fault-tolerant stream processing by Nimrod Ticotzner
• Streaming ETL in Kafka for Everyone with KSQL by Hojjat Jafarpour
• Stream Processing @ Lyft by Jamie Grier
• Intelligently Monitor And Rapidly Troubleshoot Streaming Fast Data Applications With OpsClarity by David Brinegar
• Event stream processing using Kafka streams by Fredrik Vraalsen
• Apache spark streaming + kafka 0.10 an integration story by Joan Viladrosa Riera
• Deep dive into stateful stream processing in structured streaming by Tathagata Das
• Stateful Streaming Applications with Apache Spark by Burak Yavuz
• Deep Learning and Streaming in Apache Spark 2.x by Matei Zaharia

New Videos

• Common Patterns of Multi Data-Center Architectures with Apache Kafka by Gwen Shapira
• Apache Kafka and the Rise of the Streaming Platform by Neha Narkhede
• Apache Kafka + Apache Mesos = Highly Scalable Streaming Microservices by Kai Waehner
• Real-Time Machine Learning with TensorFlow, Kafka and MemSQL by Bryan Offut
• Easy, scalable, fault tolerant stream processing with structured streaming by Tathagata Das
• Deep Learning and Streaming in Apache Spark 2.x by Matei Zaharia,
• Apache Spark Streaming Programming Techniques You Should Know by Gerard Maas
• Apache Spark Structured Streaming Helps Smart Manufacturing by Xiaochang Wu & Hao Cheng
• Monitoring Structured Streaming Applications Using Web UI by Jacek Laskowski

New Releases

• Debezium 0.6.1

Upcoming Events

• 1.11.2017 (San Francisco, US) – Learn About Stream & Batch Processing with Apache Beam (Meetup)
• 7.11.2017 (Berlin, GE) – Kafka Meetup at The Shuttle (Meetup)
• 7.11.2017 (online) – Apache Kafka Made Dead Easy Modern Data Ingestion Best Practices (Streamsets Webinar)
• 11.11.2017 (Freemont, US) – Hands-on Workshop: Apache Pulsar (Meetup)
• 15.11.2017 (London, UK) – OPENING KEYNOTE: The Rise of the Streaming Platform (Big Data London)
• 15.11.2017 (London, UK) – Using GPUs for Fast Data and Converged BI + AI (Big Data London)
• 15.11.2017 (London, UK) – Stream All the Things!! (Big Data London)
• 15.11.2017 (London, UK) – The Streaming Transformation (Big Data London)
• 16.11.2017 (London, UK) – Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka (Big Data London)
• 16.11.2017 (London, UK) – Unifying Big Data Workloads in Apache Spark (Big Data London)
• 16.11.2017 (London, UK) – Processing Fast Data with Apache Spark: The Tale of Two APIs (Big Data London)
• 29.11.2017 (Berne, US) – Apache Kafka Skalierbare Nachrichtenverarbeitung und mehr! (Java User Group Switzerland)
• 29.11.2017 (Milano, IT) – First Italian Kafka meetup – An introduction by Confluent and a production case (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 16.10.2017

This is the 86th edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Forrester: 3 ways IoT can drive business value by Fredric Paul
• Event-Driven Architecture by Herberto Graca
• How IoT Changes the World Of Data by Manuvir Das
• Streaming Data Pipelines with Brooklin by Samarth Shetty
• 3 Ways Real-Time Analytics Is Revolutionizing the Finance Industry by Dana Meschiany

Apache Kafka / Kafka Streams / Confluent Platform

• Getting Started Analyzing Twitter Data in Apache Kafka through KSQL by Robin Moffatt
• Keeping the Web API Layer in Kafka With a REST Proxy by Kin Lane
• Event First Development – Moving Towards Kafka Pipeline Applications by Conor Clifford
• Real-Time Activity Tracking With Kafka by Chamath Kirinde
• Event Sourcing Microservices with Kafka by Daniel Battaglia
• Describe and execute Kafka stream topologies with Lenses SQL by Stefan Bocutiu
• Kafka Racing: Know the Circuit by Nikola Ivancevic
• Using Kafka Streams API for predictive budgeting by Boyang Chen

Apache Pulsar / Heron / Streamlio

• Geo-replication in Apache Pulsar, part 2: patterns and practices by Sijie Guo
• Exactly once is NOT exactly the same by Jerry Peng

Apache Spark Streaming

• Benchmarking Structured Streaming on Databricks Runtime Against State-of-the-Art Streaming Systems by Burak Yavuz
• Using Spark SQL and Spark Streaming together by Anatoliy Plastini
• Zome: Real-time Merchant Home Page with Spark Streaming & Kafka by Sudhir Hasbe

Apache Flink

• Uber Introduces Open Source AthenaX, A Streaming SQL Platform Powered By Apache Flink by Michael Winters
• Introducing AthenaX, Uber Engineering’s Open Source Streaming Analytics Platform  by Haohui Mai, Bill Liu, & Naveen Cherukuri
• Stream Processing With Apache Flink by Ivan Mushketyk

Apache NiFi

• Apache NiFi – Secure Connections to Apache Solr  by Bryan Bende

Streamsets

• API Response Tracking With StreamSets, Elasticsearch, and Kibana by Rathnadevi Manivannan

New Presentations

• Reliability Guarentees in Apache Kafka by Gwen Shapira
• Introduction to Structured streaming by Madhukara Phatak
• Comparative Evaluation of Spark and Flink Stream Processing by Ehab Qadah
• Serverless Streaming Data Processing using Amazon Kinesis Analytics by Amazon Web Services
• Design Streaming Fast Data Applications with Spark, Akka, Kafka and Cassandra on Mesos & DC/OS by Craig Pottinger
• Introduction to Heron by Karthik Ramasamy
• Introducing Venice by Felix GV & Yan Yan
• The Past, Present and Future of Big Data @LinkedIn by Suja Viswesan
• World of Tanks Experience of Using Kafka by Levon Avakyan
• Big Data: Let’s SMACK by Joerg Schad
• Building real time Data Pipeline using Spark Streaming by Veeramani Moorthy

New Videos

• Reliability Guarentees in Apache Kafka by Gwen Shapira
• Building streaming data applications using Kafka*[Connect + Core + Streams] by Slim Baltagi
• Enabling Scalable IOT Applications by Adam Mollenkopf
• Big Data: Let’s SMACK by Joerg Schad
• Building real time Data Pipeline using Spark Streaming by Veeramani Moorthy
• Storm/Spark Streaming vs. Apex Comparison by Devendra Tagare

New Releases

• Apache 1.20.0-incubating
• Ueber AthenaX

Upcoming Events

• 16.10.2017 (New York, US) – Exactly-once Semantics In Apache Kafka (Meetup)
• 17.10.2017 (online) – Common Patterns of Multi Data-Center Architectures with Apache Kafka (Confluent Webinar)
• 17.10.2017 (Raleigh, US) – Real-world deployments with Apache Kafka (Meetup)
• 18.10.2017 (London, UK) – Stream SQL and Realtime Applications with Apache Flink (Meetup)
• 19.10.2017 (Dublin, IR) – Let’s talk Kafka Streams (Meetup)
• 24.10.2017 (online) – Disaster Recovery Plans for Apache Kafka (Confluent Webinar)
• 24.10.2017 (online) – Metrics are Not Enough: Monitoring Apache Kafka and Streaming Applications (Confluent Webinar)
• 7.11.2017 (Berlin, GE) – Kafka Meetup at The Shuttle (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 3.10.2017

This is the 84th edition of my blog series blog series around Stream Processing and Analytics!

It’s the one, I write during my visit at OpenWorld and JavaOne 2017, so due to the limited time for collecting the information, it might be not complete at all. But I hope I have been able to cover at least some of what is happening 🙂

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• What Net Neutrality Means for Data and Streaming by Lindsey Patterso
• Four Tenets of the Modern Application Platform by Florian Leibert
• Data Streaming in the API Landscape by Kin Lane
• Analytic Algorithms May Deliver Bad Data at Real-Time Speeds by Joe McKendrick
• Real-Time Computers on Wheels (What Used to be Called Cars) by Joe McKendrick
• Mesosphere DC/OS Brings Large-Scale Real-Time Processing to Geospatial Data by Scott M. Fulton
• What Is Realtime? by Daniel Kadvany

Comparison

• Apache Flink vs. Apache Spark by Ivan Mushketyk

Apache Kafka / Kafka Streams / Confluent Platform

• How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka by Kai Waehner
• Monitoring Kafka: The Real-Time Ops Challenge by Jason Bloomberg
• Comparison API for Apache Kafka by Artem Rukavytsia
• Deep Learning in Real Time with TensorFlow, H2O.ai and Kafka Streams by Kai Waehner

Apache Pulsar

• Multi-tenant messaging with Apache Pulsar by Matteo Merli & Sijie Guo
• Geo-Replication in Apache Pulsar Part 1: Concepts and Features by Sijie Guo

Apache NiFi

• Apache NiFi – Authentication with OpenID Connect by Bryan Bende

Spark Streaming

• IoT With Amazon Kinesis and Spark Streaming on Qubole by Dharmesh (Dash) Desai

Streamsets

• Data Quality Checks With StreamSets Using Drift Rules by Rathnadevi Manivannan

Oracle Stream Analytics

• New! Oracle Stream Analytics floats to Cloud based on Big Data Spark Streaming by Robin Smith

New Presentations

• Foundations of streaming SQL or how I learned to love stream & table theory by Tyler Akidau
• Kafka and Debezium at trivago | Code.Talks 2017 edition by Mario Mueller
• Apache Kafka™’s Role in Implementing Oracle’s Big Data Reference Architecture by Robin Moffatt
• Stream Analytics with SQL on Apache Flink by Fabian Hueske
• Apache Kafka Scalable Message Processing and more! by Guido Schmutz
• What’s this Stream Processing stuff anyway? by Gwen Shapira & Robin Moffatt
• Deploying Confluent Platform for Production by Gwen Shapira
• Riding the Streaming Wave DIY style by Konstantine Karantasis
• PostgreSQL + Kafka: The Delight of Change Data Capture by Jeff Klukas
• Streaming Solutions for Real time problems by Abhishek Gupta
• Data Guarantees and Fault Tolerance in Streaming Systems by Roshan Naik
• ETL Microservices using Kafka for Fast Big Data DataTorrent AppFactory by Yogi Devendra
• Data-In-Motion Unleashed by Andrew Psaltis
• Faster Data Analytics with Apache Spark using Apache Solr by Kiran Chitturi
• Apache NiFi: latest developments for flow management at scale by Abdelkrim Hadjidj
• Data Guarantees and Fault Tolerance in Streaming Systems by Roshan Naik
• Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning4J, H2O by Kai Waehner

New Videos

• “EoS in Kafka: Listen up, I will only say this once!” by Jason Gustafson
• Rethinking Microservices with Stateful Streams by Ben Stopford
• Scaling Validation and Quality of Streaming Data Products at Twitter by Kelly Kaoudis
• Inside Twitter’s Realtime Delivery of Tweets for Enterprise Customers by Lisa White
• Keeping Time in Real Systems by Kavya Joshi

New Releases

• Apache NiFi 1.4.0

Upcoming Events

• 19.10.2017 (Dublin, IR) – Let’s talk Kafka Streams (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last Week in Stream Processing & Analytics – 26.9.2017

This is the 83rd edition of my blog series blog series around Stream Processing and Analytics!

As every week I was also updating the following two lists with the presentations/videos of the current week:

• Presentations from SlideShare
• Videos from YouTube

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

• Using Avro for Big Data and Data Streaming Architectures: An Introduction by Jean-Paul Azar
• Document worth reading: “Resource Elasticity for Distributed Data Stream Processing: A Survey and Future Directions” by Michael Laux
• The Rise of Realtime Infrastructure as a Service (IaaS)  by Justin Baker & Jeffrey Lee
• QDS Business Edition: The power of Autonomous Data Platform for Everyone by Dharmesh Desai

Apache Kafka / Kafka Streams / Confluent Platform

• Apache Kafka™ talks at Oracle OpenWorld, JavaOne, and Oak Table World 2017 by Robin Moffatt
• How Streams Relate to Database Tables in Kafka by Bill Bejeck
• Disaster Recovery for Multi-Datacenter Apache Kafka Deployments by Yeva Byzek
• Setting up Oracle Event Hub (Apache Kafka) Cloud Service and Pub & Sub from local Node Kafka client by Lucas Jellema
• Streaming data to a downstream database by Jiri Pechanec
• Comparing Confluent Platform with Traditional Messaging Middleware by Hans Jespersen
• Learning w/ Diagrams: Intro to Kafka by Diana Hsieh

Spark Streaming

• Spark Streaming: IoT With Amazon Kinesis and Visualizing With Qubole Notebooks by Jeffrey Ellin
• Basic Example for Spark Structured Streaming and Kafka Integration by Ayush Tiwari
• Spark Streaming: IoT with Amazon Kinesis and Visualizing with Qubole Notebooks by Dharmesh Desai

Oracle GoldenGate

• GoldenGate Cloud Service (GGCS): How to provision GGCS in Oracle Public Cloud (OPC) via RESTful API by Mike Papio

New Presentations

• Look Ma, no Code! Building Streaming Data Pipelines with Apache Kafka by Robin Moffatt
• Big Data – in the cloud or rather on-premises? by Guido Schmutz
• End-to-end streaming analytics on Google Cloud Platform by Javier Ramirez
• Kafka as a message queue by Adam Warski
• Building Stream Processing Applications with Apache Kafka’s Exactly-Once Processing Guarantees by Matthias Sax
• Exactly-once Stream Processing with Kafka Streams by Guozhang Wang
• Smarter internet of things with stream and event processing by András Jankó & István Papp & István Ráth
• Divide, Distribute and Conquer: Stream v. Batch by Viktor Gamov
• Streaming Analytics Manager – SAM by Priyank Shah
• Spark Structured Streaming by Revin Chalil
• Structured Streaming with Apache Spark by Dataya Nolja
• Build Smart Systems with AWS IOT , Analytics & AI by Neel Mitra
• Hadoop – Apache Storm Real Time Processing by Duy Viec
• Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi and TensorFlow by Timothy Spann
• The Power of Intelligent Flows Real-Time IOT Botnet Classification with Apache NiFi by Andre Fucs de Miranda & Andy LoPresto
• KSQL: Streaming SQL for Kafka by Neil Avery
• Unified Batch & Stream Processing with Apache Samza by Navina Ramesh

New Videos

• Smarter internet of things with stream and event processing by András Jankó & István Papp & István Ráth
• Divide, Distribute and Conquer: Stream v. Batch by Viktor Gamov
• Autopiloting #realtime Stream Processing in Heron by Karhik Ramasamy
• Applying Geospatial Analytics at a Massive Scale using Kafka, Spark and Elasticsearch on DC/OS by Adam Mollenkopf
• Deploying Kafka on DC/OS by Kaufmann Ng
• Using Mesosphere Enterprise DC/OS to Deliver Real-Time Microservices to a Global Fleet of Ships by Eli Tsinovoi
• Real-Time Event Processing and Handling Stateful Applications on Apache Mesos by Balajee Nagarajan & Venkatesh Sivasubramanian
• Streaming Data Pipelines on Apache Mesos: Lessons Learned by Dean Wampler
• KSQL: Streaming SQL for Kafka by Neil Avery

New Releases

• Debezium 0.6

Upcoming Events

• 26.9.2017 (New York, US) – Distributed Deep Learning with TensorFlow in Docker containersDistributed Deep Learning with TensorFlow in Docker containers (Meetup)
• 27.9.2017 (New York, US) – Crossing The Streams: Rethinking Stream Processing with KStreams and KSQL (Meetup)
• 19.10.2017 (Dublin, IR) – Let’s talk Kafka Streams (Meetup)

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!