Cloud Based Load Testing using TFS

What is Cloud Based Load Testing?
It goes without saying that performance testing your application not only gives you the confidence that the application will work under heavy levels of stress but also gives you the ability to test how scalable the architecture of your application is. It is important to know how much is too much for your application!

Working with various clients in the industry, it has been realized that the biggest barriers in Load Testing & Performance Testing adoption are,

1. High infrastructure and administration cost that comes with this phase of testing
2. Time taken to procure & set up the test infrastructure
3. Finding use for this infrastructure investment after completion of testing

Endurance Testing
To test for endurance, Azure diagnostics is used to begin with, but later started using Cerebrata Azure Diagnostics Manager to capture the metrics of the machine under test. Currently Microsoft Load Test service does not support metrics from the machine under test.

 Herez an use case of quote generation engine with an expected fixed user load and ran the test for very long duration such as over 48 hours and observed the affect of the long running test on the Azure infrastructure.

Threshold Testing
Another use case to step load the quote generation engine by incrementing user load with different variations of incremental user load per minute till the application crashed out and forced an IIS reset.

It depicts the threshold testing as above.

There a few limits on the usage of Microsoft Cloud based Load Test service that you can read at http://blogs.msdn.com/b/visualstudioalm/archive/2013/06/26/load-testing-with-team-foundation-service-launching-preview-and-early-adoption-program.aspx

Dell EMC

Computer-maker Dell Inc struck a deal on Monday to buy data storage company EMC Corp for $67 billion in cash & stock with 19% premium. The acquisition, the year's third-largest in all sectors, top in technology sector.

The deal should help privately held Dell, the world's No. 3 computer maker, diversify from a stagnant consumer PC market and give it greater scale in the more profitable and faster-growing market for cloud-based data services.

The deal will be financed through a combination of new equity from Dell's owners - founder and Chief Executive Michael Dell, its investment firm MSD Partners, private equity firm Silver Lake and Singapore state-owned investor Temasek Holdings - as well as the issuance of the tracking stock, new debt and cash on hand.

Michael Dell, with the help of Silver Lake, took the PC maker private in $25 billion deal two years ago.

Dell first approached EMC in October 2014 following speculation over a deal between Hewlett-Packard and EMC collapsing and Elliott attacking the company, the source said.

Michael Dell then met EMC Chief Executive Joe Tucci at the World Economic Forum Annual Meeting 2015 in Davos in January, the source added. Negotiations between Dell and EMC intensified in the last two months.

The transaction is expected to close between May and October 2016, the companies said.  The combined company aims to be a leading enterprise player in the following markets:

• Servers and Storage fro the Enterprise Market
• Virtualization via VMware;
• Converged infrastructure (EMC owns VCE);
• Hybrid cloud and cloud computing
• And security via RSA, which is owned by EMC.

Airbnb Airpal

We know that Airbnb is a popular website for people to list, find, and rent lodging. It has over 1,500,000 listings in 34,000 cities and 190 countries.

Recently, Airbnb  launched Big Data tool Airpal, a web-based query execution tool that leverages Facebook’s PrestoDB to facilitate data analysis by authoring queries and retrieving results simple for users.

Key features of Airpal:

• optional access controls for users
• ability to search and find tables
• see metadata, partitions, schemas, and sample rows
• write queries in an easy-to-read editor
• submit queries through a web interface
• track query progress
• get the results back through the browser as a CSV
• create new Hive table based on the results of a query
• save queries once written
• searchable history of all queries run within the tool

Requirements are:

• Java 7 or higher
• MySQL database
• Presto 0.77 or higher
• S3 bucket (to store CSVs)
• Gradle 2.2 or higher

On keeping with the spirit of Presto, they have tried to make it simple to install Airpal by providing a local storage option for people who would like to test it out without any overhead or cost.

For more detailed information, visit the GitHub page here: https://github.com/airbnb/airpal

Coursera Architecture

Coursera is a venture backed for-profit educational technology company that offers massive open online courses (MOOCs).

It works with top universities and organizations to make some of their courses available online, and offers courses in physics, engineering, humanities, medicine, biology, social sciences, mathematics, business, computer science, digital marketing, data science and other subjects.  Itz an online educational startup with over 14 million learners across the globe to offer more than 1000 courses from over 120 top universities.

At Coursera, Amazon Redshift is used as primary data warehouse because it provides a standard SQL interface and has fast and reliable performance. AWS Data Pipeline is used to extract, transform, and load (ETL) data into the warehouse. Data Pipeline provides fault tolerance, scheduling, resource management and an easy-to-extend API for ETL processing.

Dataduct is a Python-based framework built on top of Data Pipeline that lets users create custom reusable components and patterns to be shared across multiple pipelines. This boosts developer productivity and simplifies ETL management.

At Coursera, 150+ pipelines were executed to pull the data from 15 data sources such as Amazon RDS, Cassandra, log streams, and third-party APIs. 300+ tables are loaded every day into Amazon Redshift, processing several terabytes of data. Subsequent pipelines push data back into Cassandra to power our recommendations, search, and other data products.

The attached image below illustrates the data flow at Coursera.

ScyllaDB

ScyllaDB is the world's fastest NoSQL column store database, which is written in C++. Itz fully compatible with Apache Cassandra at 10x throughput and jaw dropping low latency.

Scylla will work with existing Cassandra command line CQL clients. However, mixed clusters of Scylla and Cassandra nodes are not supported. A Scylla node cannot join a Cassandra cluster, and a Cassandra node cannot join a Scylla cluster.

To share the benchmark between Scylla and Cassandra, both throughput on a single multi core server is evaluated with the Hardward specification:
2x Xeon E5-2695v3: 2.3GHz base, 35M cache,
14 core -> 28 core with HT
64GB RAM
2x 400GB Intel NVMe P3700 SSD
Intel Ethernet CNA XL710-QDA1

In terms of software Scylla 0.8 & Cassandra 3.0, is enabled as TestBed.

In the attached image, average throughput for the test is presented as lines, latency as bars.

Scylla’s measured latency of less than 1 ms for the 99th percentile is significantly lower than Cassandra’s, while providing significantly higher throughput (the single client machine could not fully load the server).  The lack of garbage collection means that there are no externally imposed latency sources, so Scylla latency can be brought even lower.


Scylla’s order of magnitude improvement in performance opens a wide range of possibilities.  Instead of designing a complex data model to achieve adequate performance, use a straightforward data model, eliminate complexity, and finish your NoSQL project in less time with fewer bugs.

MapReduce Authors

We know that MapReduce is the ice breaker for the traditional computing model by introducing Scale Out technology in the easy way. Google has a separate research web page on Google's MapReduce at http://research.google.com/archive/mapreduce.html . Authors are Sanjay Ghemawat & Jeff Dean from Google Inc.

As the research scholar, I liked the motivation of their research paper - Large Scale Data Processing. It was achievable with super computing on earlier days. But key difference is parallel execution of hundreds or thousands of CPUs, with commodity box and easy mode.  More over, MapReduce provides:

1. Automatic parallelization and distribution
2. Fault-tolerance
3. I/O scheduling
4. Status and monitoring

Fault-tolerance is handled via re-execution.  On worker failure:

• Detect failure via periodic heartbeats
• Re-execute completed and in-progress map tasks
• Re-execute in progress reduce tasks
• Task completion committed through master

Data Locality Optimization, Skipping Bad Records and Compression of intermediate data are their few refinement technique to boost the performance on large scale data.

In their research paper, the use case was listed in August 2004 with the below metric:

• Number of jobs 29,423
• Average job completion time 634 secs
• Machine days used 79,186 days
• Input data read 3,288 TB
• Intermediate data produced 758 TB
• Output data written 193 TB
• Average worker machines per job 157
• Average worker deaths per job 1.2
• Average map tasks per job 3,351
• Average reduce tasks per job 55
• Unique map implementations 395
• Unique reduce implementations 269
• Unique map/reduce combinations 

Amazing and game changing methodology with easiness, as the result of great minds research from Google.  Herez an opportunity for me to highlight the authors of MapReduce - Sanjay Ghemawat & Jeff Dean

MapReduce Execution

On submission of the job by the User, Hadoop initiates the Job Tracker process at Master Node.  Internally, the execution undergoes 3 major tasks/steps as below:

1. Map 

A map task is created that runs the user-supplied map function on each record. The map function takes a key-value pair as input and produces zero or more intermediate key-value pairs. Map tasks are executed in parallel by various machines across the cluster.  

Mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the Hadoop administrator. The intermediate data is cleaned up  after the Hadoop Job completes.

2. Shuffle and Sort

It is preformed by the reducers (reduce tasks). Each reducer is assigned one of the partitions on which it should work. This is a flurry of network copies between each reducer in the cluster so it can get the partition (intermediate key-value) data it was assigned to work on.

The output of the reduce is normally stored in HDFS for reliability. This step uses a lot of bandwidth between servers and benefits from very fast networking like 10G.

3. Reduce

After the partition data has been copied we can start performing a merge sort of the data. A merge sort takes a number of sorted items and merges them together to form a fully sorted list. 

   

Each reducer produces a separate output file, usually in HDFS. Each reducer output file usually named part-, where  is the number of the reduce task within the job. The output format of the file is specified by the author of the MapReduce job. The number of reducers is defined by the developer.

Attached image depicts the execution flow of Hadoop job.