RT Platform – EMindfactorY

Real-Time Analytics Platform

With our “Real-Time Analytics” platform, we offer you a solution that allows you to receive, process and analyze your data as streams in real time. Together with AWS’s IoT solution, our platform can cover a majority of streams of sensor data, services , social media, and so on.

In the platform, we have integrated leading technologies such as Mesos, Kafka, Cassandra and Spark streaming, which are also aligned with each other and build on industry standards such as Java. This also gives you support and development security and allows you to benefit from new developments.

Since we implement the platform in your “private cloud”, you have complete control over your data and high flexibility over your processes. Yet you still enjoy the benefits of a cloud-based solution such as low investment costs, high scalability, maximum mobility and consumption-based fees.

Components

Cassandra

Cassandra is one of the leading distributed and highly available No-SQL databases used by well-known companies such as Cisco, Credit Suisse, Disney, Ebay, Hp and many more. is in operation.

Cassandra is known for high availability and high throughput characteristics, and it is capable of handling enormous write loads and surviving cluster node failures. With respect to the CAP theorem, Cassandra provides configurable consistency and availability for operations.

In terms of data processing, Cassandra is linearly scalable (increased loads can be met by increasing the number of nodes in a cluster) and it is capable of cross-data center replication (XDCR). XDCR offers a number of interesting use cases for:

geo-distributed data centers: data specific to the region or closer to the customer.
Data center data migration: recovering from outages or moving data to a new datacenter.
Separate operational and analytics workloads: Separate clusters can be set up for write-intensive and analytics-intensive applications.

Cassandra is subject to the Apache 2.0 license.

Spark

Spark is a very fast engine for distributed, large-scale data processing and is now used as a quasi-standard in most platforms (SAP, Microsoft, IBM, etc.) for Big Data Analytics.

In this process, according to the “Map-Reduce” principle, the data processing task is broken down into several parts by the Spark master and sent to the distributed machines (so-called workers). These work through the subtasks and send the results to the master, which combines the partial results into an overall result. A Spark cluster can consist of a number of distributed machines (so-called workers), depending on the requirements.

Spark has replaced the leading technology of recent years, ,Hadoop MapReduce” and executes programs up to 100x faster than ,Hadoop MapReduce in memory (In Memory) or 10x faster on disk. In addition, handling is easier, programming is more efficient, and modules for various analytics tasks are already integrated into Spark.

spark-stack

Cassandra is subject to the Apache 2.0 license.

Kafka

Kafka is a high throughput, low latency distributed messaging system. It serves as a buffer for incoming and outgoing streams. Apache Kafka is often used with Spark because it can be distributed across multiple nodes and provides interfaces on the servers. It is characterized by the following features:

Speed: A single Kafka Broker can perform hundreds of megabytes of reads and writes per second.
Scalability: It can be expanded elastically and transparently, without downtime.
Fail-safe: Messages are retained on servers to prevent replicated data loss within the cluster.
Distributed by Design: Kafka has a modern cluster-centric design that ensures strong durability and fault tolerance.

Cassandra is subject to the Apache 2.0 license.

Mesos

Mesos is a cluster resource management system that provides efficient resource sharing and isolation in distributed applications or “frameworks”. Since the distributed servers are considered as one system, resources can be used more efficiently as they are allocated as needed.

With the increasing number of services, servers and applications, the management and monitoring of the system becomes more complex. Mesos reduces complexity and, in addition, built-in services such as Marathon facilitate fault-tolerant execution of long-running applications. This is especially important for streaming real-time services.

Mesos natively supports Spark, Kafka and Cassandra and is characterized by high usability. Mesos is in use at well-known companies such as Apple, Ebay, Verizon, etc.

Cassandra is subject to the Apache 2.0 license.

AWS IoT/Kinesis

AWS IoT provides secure, two-way communication between Internet-connected things (such as sensors, actuators, embedded devices, or smart devices) and the AWS cloud. This way you can collect telemetry data from multiple devices, store it and analyze the data. You can also create applications that allow users to control these devices from their phones or tablets.

Our platform uses AWS IoT in particular due to security requirements.

EM Streaming Server

With EM streaming server, they can receive web socket streams and forward them to Kafka, and receive messages from Kafka and forward them as web sockets streams to web client. Our solution runs on the Java Virtual Machine and is characterized by speed and resilience. In addition, we offer real-time web solutions in which the data can be visualized in real time.

The EM streaming server is configured individually, according to your requirements, and the streaming interfaces and the web application are programmed.

Real-Time Processing

Streaming data is data that is generated continuously and from a variety of data sources. The data recordings are usually sent simultaneously and in small packets (kilobyte range). This data must be processed sequentially and incrementally on a per-record basis or in sliding time windows. Examples of streaming data are

Sensors from industrial equipment and machines send data to a real-time application to monitor production.
A financial company monitors financial transactions in real time for anomalies and fraud.
Evaluation of stock market performance and share performance using extra real-time data.
Online store evaluates visitor activity and the number of clicks in real time.
A real estate mobile app sends users suggestions of potential properties to view nearby, based on their location.

Unlike batch processing, the latency required for stream processing is in the range of seconds or milliseconds. This places special demands on both the processing and storage of the data. In addition, the system must be fail-safe and scalable.

Our real-time analytics platform is designed to meet the requirements and ensure that streaming data is processed with the required latency. With Kafka, Spark and Cassandra, we are building on technologies that have already proven themselves in many demanding real-time applications.

This post is also available in: German