Analytic Platform – EMindfactorY

Big Data Analytics Platform

With our “Big Data Analytics” platform, we offer you a technical solution that helps you store your valuable data securely, process it reliably and analyze it quickly. Our Analytics Suite with the modules Process Management, Data Management, ETL and Analytics & Reporting supports you in this, but is not necessary for the operation of the platform.

We built our platform on industry-standard Java and integrated leading technologies, such as Cassandra and Spark. These are perfectly coordinated with each other and offer you support and development security.

Since we implement the platform in your “private cloud”, you have complete control over your data and high flexibility over your processes. Yet you still enjoy the benefits of a cloud-based solution such as low investment costs, high scalability, maximum mobility and consumption-based fees.

Components

Cassandra

Cassandra is one of the leading distributed and highly available No-SQL databases used by well-known companies such as Cisco, Credit Suisse, Disney, Ebay, Hp and many more. is in operation.

Cassandra is known for high availability and high throughput characteristics, and it is capable of handling enormous write loads and surviving cluster node failures. With respect to the CAP theorem, Cassandra provides configurable consistency and availability for operations.

In terms of data processing, Cassandra is linearly scalable (increased loads can be met by increasing the number of nodes in a cluster) and it is capable of cross-data center replication (XDCR). XDCR offers a number of interesting use cases for:

geo-distributed data centers: data specific to the region or closer to the customer.
Data center data migration: recovering from outages or moving data to a new datacenter.
Separate operational and analytics workloads: Separate clusters can be set up for write-intensive and analytics-intensive applications.

Cassandra is subject to the Apache 2.0 license.

Spark

Spark is a very fast engine for distributed, large-scale data processing and is now used as a quasi-standard in most platforms (SAP, Microsoft, IBM, etc.) for Big Data Analytics.

In this process, according to the “Map-Reduce” principle, the data processing task is decomposed into several parts by the Spark master and sent to the distributed machines (so-called workers). These work through the subtasks and send the results to the master, which combines the partial results into an overall result. A Spark cluster can consist of a number of distributed machines (so-called workers), depending on the requirements.

Spark has replaced the leading technology of recent years, ,Hadoop MapReduce” and executes programs up to 100x faster than ,Hadoop MapReduce in memory (In Memory) or 10x faster on disk. In addition, handling is easier, programming is more efficient, and modules for various analytics tasks are already integrated into Spark.

spark-stack

Spark is subject to the Apache 2.0 license.

Zeppelin

Zeppelin is a web-based notebook that enables interactive data analysis. Zeppelin notebooks can be used to perform the following tasks:

Data Ingestion
Data Discovery
Data Analytics
Data Visualization & Collaboration

Zeppelin interpreter concept allows that a variety of data processing backend can be integrated. The integration of Spark and Cassandra is particularly excellent. In addition, Zeppelin supports many interpreters such as Java, Scala, Python, R JDBC and Shell.

Cassandra is subject to the Apache 2.0 license.

EM Connector

The EnterpriseMind Connector establishes the connection between your “Big Data Analytics” platform and the Analytics Suite. It supports the following functions:

Creating tables according to the schemas from the Data Management module
Capture and query data
Perform ETL jobs
Monitoring the processing of ETL jobs

The EM connector serves as an interface. No data from the EM platform is stored in the Analytics Suite. Data analyses are performed directly from the Zeppelin server.

Big Data Analytics

Big Data is characterized by the following three features.

Volume: Large amounts of data are generated
Variety: It is about different data types and data sources
Velocity: The data is generated and processed at high speed

Big Data solutions can provide new insights, especially in areas where a lot of data has been generated, but the potential has not yet been exploited. Companies can gain competitive advantages, generate potential savings and create new business areas by analyzing Big Data. Examples of Big Data application include.

Fraud detection: detection of irregularities in business transactions and transcation
Smart metering: Enables intelligent network and resource control
Smart Billing: Building flexible billing systems
Predective Mainatance : Reduction of downtime of machines and equipment

However, the technical development of processor performance cannot keep pace with the amount of data to be processed. With the requirements for speed, the success of No-SQL databases as distributed systems began since the early 2000s. This involves creating multiple copies of the database and distributing them across multiple systems. The databases can then be queried in parallel, which increases throughput. The disadvantage of these systems lies in the so-called CAP theorem, which states that only two goals can be achieved simultaneously.

Consistency (C consistency): The consistency of the stored data. In distributed systems with replicated data, it must be ensured that all replicas of the manipulated data set are also updated after a transaction is completed.
Availability (A availability): Availability in the sense of acceptable response times. All requests to the system are always answered.
Failure tolerance (P partition tolerance): The failure tolerance of the computer/server networks. The system continues to work even in case of loss of messages, individual network nodes or partition of the network.

Cassandra’s focus in Analytics is on availability and failure tolerance, where consistency can be improved at the expense of the other two goals.

The second disadvantage is the limited query capability. In contrast to SQL databases, it must already be determined during table design how the data is to be queried. Otherwise, it requires an analytics engine like Spark that can evaluate and analyze the data. Only with an analytics engine can the added value of Big Data be generated through advanced analytics.

Our “Big Data Analytics” platform is designed to meet the requirements of Big Data and ensures that your data is secure, highly available and can be analyzed according to your business needs. With Spark and Cassandra, we are building on technologies that have already proven themselves in many demanding Big Data applications.

This post is also available in: German