Oracle Stream Analytics (OSA): the new Oracle Stream Explorer

A few days ago, Oracle released the new version of Oracle Stream Explorer and renamed it to Oracle Stream Analytics (OSA). This new version is an impressive release with over 15 new major features! It really deserves the name change.

Enhanced Patterns Library

The existing patterns have been enhanced substantially  now including Spatial, Statistical, General industry and Anomaly detection through streaming machine learning.

patterns

New Geo-spatial pattern

This pattern can be used to analyze streams containing geo-location data and determine how events relate to pre-defined geo-fences in your maps.

map.png

Integrated Expression Builder

The Expression Builder allows to add calculated/derived fields on the Live Output Stream of an exploration, an important step towards the “streaming Excel sheet” idea of Oracle Stream Analytics.

expression.png

It provides the ability to apply and insert mathematical and statistical calculations into the active live output stream. Once a new expression has been defined and validated, a column will be added next to the column of relevance. This new column can then be used in subsequent filters and explorations.

Support for Business Rules in Explorations

The Business Rules section of the Stream Analytics canvas provides the ability to apply the more traditional IF-THEN-ELSE constraints and clauses on the various properties of the event shape.

business-rule.png

This capability enables the user to combine both streaming query analytics using temporal criteria together with a collection of business rules that can randomly effect the information in existing or new columns.

New streaming end point connections/targets

Oracle Stream Analytics supports new Event Stream sources and targets, such as MQTT, Apache Kafka and Twitter.

connection

Especially Kafka gets more and more important in modern Big Data architectures so I’m really pleased to see it available now.

We can now use Oracle GoldenGate for immediately capturing changes on any database table (CDC = change data capture), send these captured change events into Kafka using  GoldenGate for BigData  and consume it from OSA to apply streaming analytics on it.

Scaling-Out with Spark Streaming

An OEP server is no longer the only runtime option. With Oracle Stream Analytics you can deploy and execute streaming applications to a Spark Streaming infrastructure.

The figure below shows how you can select one of the two possible runtime environments (Spark grayed-out because not yet configured on my environment).

spark

Better Insights with Catalog Topology Viewer and Navigation

Topology is a graphical representation of the connected entities. The topology illustrates the dependencies and connections between the entities. The Topology Viewer helps in identifying the dependencies that a selected entity has on other entities. Understanding the dependencies helps you in being cautious while deleting or undeploying an entity.

topology

 

I’m really pleased with this new release and I’m looking forward to see more enhancements and improvements in future releases. As already mentioned, the product really deserves the name change, but I also hope it’s the last one for the next couple of years😉. Oracle Stream Analytics simplifies stream processing and will enable Self Service Streaming Analytics applications for business people.

Find more information on Oracle Stream Analytics in the Documentation available here.

Stay tuned for an update on the Docker support I already had for Stream Explorer. I’m currently in the progress of updating it for Oracle Stream Analytics so you can quickly setup your own playground environment.

 

Last week in Stream Processing & Analytics 5/2/2016

This is the 12th installment of my blog series around Stream Processing and Analytics.

The most important event last week was probably the first Kafka Summit being held in San Francisco. At the summit, Confluent shared results from a recent survey that clearly shows the rise of Kafka across the enterprise and the growth of stream processing:

  • According to the results, Apache Kafka is most commonly used for stream processing (72% of respondents). In addition, nearly seven of ten (68%) Kafka users surveyed say they plan to incorporate more stream processing over the next 6 to 12 months. Of those:
    • 74% say they will add it to new applications in development now
    • 69% will add it to existing applications
    • 60% will build net-new applications using stream processing and Kafka

The survey also reveals how Kafka is being used:

  • Kafka powers a wide variety of applications today based on survey respondents, including application monitoring (60%), data warehousing (51%) and asynchronous applications (47%) to system monitoring (39%), recommendation/decisioning engines (35%), security/fraud detection (26%), Internet of Things applications (20%) and dynamic pricing applications (12%), to name a few.

We have been using Kafka for almost 3 years now and I have never regretted  choosing it. Unfortunately the journey to California for just that one day was a bit too far for me to attend, but I read that most/some of the presentation should be available on video later. Find below the presentations which are already available through SlideShare.

So that’s it for this week. As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Apache Storm

Apache Spark Streaming

Apache Flink

Apache Apex

Apache Kafka

Apache NiFi / Hortonworks DataFlow

Striim

StreamSets

StreamingAnalytix

IBM Streaming Analytics

Microsoft Azure Stream Analytics

MapR

New Presentations

New Videos

Upcoming Events

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics 4/25/2016

This is the 11th installment of my blog series around Stream Processing and Analytics.

First two interesting tweets I found last week. The first one by Steve Wilkes brings it straight to the point:

The second one by Neha Narkhede reveals some impressive metrics about the usage of Kafka @ LinkedIn. 1.4 trillion messages a day on 1400 brokers. Kafka is really a game changer!

Last but not least I would like to quote from Mark Palmer’s latest article on 8 Predictions for the Internet of Analytics which I really enjoyed reading:

  • Streaming analytics will become a fundamental topic in computer science. Forrester’s Streaming Analytics Wave defines a set of computer science criteria to define streaming analytics: time windowing, aggregation, correlation, and integration with interactive analytics. These fundamentals are not well understand by the computer science community, are not yet taught in school, and are therefore not yet well known.
  • Data streams will be as important as data lakes. Data lakes contain data at rest; data streams contain data in motion. But most IT applications today are designed around data at rest. In the coming decade, data streams will become as important as data at rest.
  • Streaming analytics and traditional analytics will become increasingly intertwined. In order to apply analytics to streams, you need to know what to look for. Traditional analytics help you look through the rearview mirror at the past, and predict important conditions. Streaming analytics are about looking forward, through your windshield, looking at real-time conditions, and acting.

 

So that’s it for this week. As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Comparison

Apache Beam

Apache Storm

Apache Spark Streaming

Apache Flink

Apache Apex

Apache Kafka

Apache NiFi / Hortonworks DataFlow

Apache Metron

StreamSets

New Presentations

New Videos

New Podcasts

New Releases / Components

Upcoming Events

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics 4/18/2016

This is the 10th installment of my blog series around Stream Processing and Analytics.

Two days later than planned, was traveling and had again trouble with my power supply😦

So what happened in the world of Stream Processing? For me the most interesting news last week was the release of Storm 1.0.

I’m a  storm user for more than 3 years now and this is really a significant release that delivers several features that pertain to enterprise readiness, operational simplicity and ease of use. I really like that Storm now has native Windowing and State Management Support, Automatic Back Pressure Support and the new connectors for Cassandra, Elasticsearch and Kafka.

Nathan Marz, the founder and creator of Storm also tweeted about it:

And Ian Hellström already updated his stream processing overview chart with Storm 1.0.0.

apache-streaming6

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Apache Beam

Apache Storm

Apache Spark Streaming

Apache Flink

Apache Kafka

Apache NiFi / Hortonworks DataFlow

Apache Metron

Striim

StreamSets

IBM Quarks

New Presentations

New Videos

New Releases / Components

Upcoming Events

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics 4/11/2016

This is the 9th installment of my blog series around Stream Processing and Analytics.

First I have to mention a blog article I have somehow missed last month. It nicely compares the various streaming frameworks available from the Apache software foundation.

 

Last week Forrester published it’s updated Forrester Wave for Big Data Streaming Analytics products. Forrester Research defines Big Data Streaming Analytics as

Software that can filter, aggregate, enrich, and analyze a high throughput of data from multiple, disparate live data sources and in any data format to identify simple and complex patterns to provide applications with context to detect opportune situations, automate immediate actions, and dynamically adapt.

Here the Leaders, the Strong Performers and Contenders as seen by Forrester:

cez8rzywwaadrg8

Source: The Forrester Wave: Big Data Streaming Analytics, Q1 16

 

As usual, find below the new blog articles, presentations, videos and software releases from last week:

News and Blog Posts

General

Apache Storm

Apache Spark Streaming

Apache Flink

Apache Kafka

Apache NiFi / Hortonworks DataFlow

StreamSets

New Presentations

New Videos

New Releases / Components

Upcoming Events

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics 4/4/2016

This is the 8th installment of my blog series around Stream Processing and Analytics. It’s two days later, due to some technical difficulties. Forgot power adapter at home on Monday and I’m traveling😉 87 EUR and one day later I’m back in business:-)

As expected, there were quite a lot of topics around stream processing and streaming analytics at the Strata conference last week.

Jay Krebs and Neha Narkhede from Confluent both mentioned it on Twitter:

And Jack Vaughan summarized it in his blog article:  “Moving streams of data is a must in many modern applications. As a result, streaming analytics applications with Spark Streaming, Kafka and other components are coming to the big data forefront.

Definitely very interesting times ahead:-)

As usual, just find what I have noticed last week:

News and Blog Posts

General

Comparison

Apache Storm

Apache Flink

Apache Kafka

Goggle Cloud Dataflow / Apache Beam

MapR Streams

Apache NiFi / Hortonworks DataFlow

Oracle Stream Explorer

StreamSets

New Presentations

New Podcasts

New Videos

New Books

New Releases / Components

Upcoming Events

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!

Last week in Stream Processing & Analytics 3/28/2016

This is the 7th installment of my blog series around Stream Processing and Analytics.

Due to the Easter Weekend there were not so many posts last week, guess that will change this week with Strata Hadoop San Jose conference just starting today.

Personally I have started giving Streamsets a try, after having played a little bit with Apache NiFi before. Both offer a nice solution for the handling of data flows, and I’m not yet in a position to judge which one does it better. But there some obvious differences and I guess it’s not wrong to say that the combination of both would offer a really good solution.

As usual, just find what I have noticed last week:

News and Blog Posts

General

Apache Storm

Apache Spark Streaming

Apache Flink

Apache Kafka

Goggle Cloud Dataflow / Apache Beam

Apache NiFi / Hortonworks DataFlow

StreamSets

New Presentations

New Videos

New Books

New Releases / Components

Upcoming Events

Please let me know if that is of interest. Please tweet your projects, blog posts, and meetups to @gschmutz to get them listed in next week’s edition!