Endor Development Update- July / August 2019

Hello from the Endor team!

As we move quickly towards our goal of creating a self-serve prediction ecosystem, based on external data, we remind ourselves daily that we have set the bar pretty high and must now deliver. Our next stop is the H2 2019 milestone:

THE THREE ‘V’ HURDLES

The Endor protocol is designed to process huge amounts of data. As such, the ecosystem must address and adapt to the three big-data challenges, also known as the three Vs — Volume, Variety and Velocity.

Volume refers to the amount of data, variety refers to the types of data, and velocity refers to the speed of data processing. According to the 3V model, the true challenge of big-data management is the constant expansion of all three properties at once.

VOLUME

In order to process and analyze vast amounts of data, we integrated Amazon EMR into the Endor protocol. Amazon EMR is a managed cluster platform that simplifies the management of big-data frameworks (such as Apache Hadoop and Apache Spark) on AWS. Additionally, we began using Amazon EMR to transform and move large amounts of data into and out of other AWS data stores and databases, such as Amazon S3.

VARIETY

When it comes to variety, it’s a whole new ballgame. We all know that data comes in many forms — from call records, through Facebook posts, tweets and web logs, to mobile data and much more. And each data type is completely different from any other. Long gone are the days of simple rows, columns and database joins. Today, every application creates a different data type, and most of them are unstructured, meaning they don’t easily fit into predefined fields.

Take, for example, user activity data. A company may need to sift through thousands, even millions of email messages, Twitter posts // tweets and internal user transactions, and fuse all the relevant data to gain one unified overview. And no two datasets are going to be exactly the same. Each will include a common identifier (name, email address, etc.), event timestamp and event properties, but none will fit one data scheme that can be easily mapped.

The Endor protocol fuses together the different data sources in order to create a single, enriched picture that can later be used for prediction generation. But, while unifying the data provides the basis for predictions, we also preserve data sensitivity, as we work on fully-tokenized data that does not contain meaningful semantics.

VELOCITY

To illustrate the final V, velocity, Let’s take Facebook data as an example. According to Facebook’s Fabric Aggregator, its distributed network system, velocity is the speed at which data flows into the platform. And at Facebook, this speed represents a tsunami of data coming in daily. 22 billion posts a month may seem like a lot of data. But what really blows people’s mind is this… Every 60 seconds on Facebook, 510,000 comments are posted, 293,000 statuses are updated, and 136,000 photos are uploaded. So in comparison, within a few months, 22 billion posts will seem like a drop in the ocean. Facebook must process it all, file it, and somehow, be able to retrieve it at a later date.

Here’s another example of high-velocity data generation. If you’re running a marketing campaign and you want to know how people feel about your brand, you could license Twitter data from Gnip (acquired by Twitter) and gain access to a constant stream of tweets, which you could then evaluate using sentiment analysis. A Twitter feed like this is often called “the firehose” because of the intensity and speed of data produced.

In order to deal with such velocity, we’ve created a hyper-elastic infrastructure, with the ability to spawn thousands of servers, and distribute the processing load between them. This ability to auto-scale the infrastructure, opening and closing servers as needed, is a key component when processing vast amounts of data, while still carefully managing the processing budget.

That’s it for now… but we’ve already set our sights on the next milestone, which includes you, our community, and expanding the ecosystem. So, stay tuned — we’ve got much more up our sleeves.

Onwards and upwards,

The Endor Team

EDR is listed on the following exchanges：

Upbit , Bitrrex ，Hotbit , BitForex ，Houbi korea , CoinTiger ，Bilaxy，Trade.io , Coinall , KuCoin , Idex , DigiFinex , P2PB2B , OAX , CoinBene , LATOKEN , BitMart , Coinbit , ABCC .