Apache Phoenix

Apache Phoenix is an efficient SQL skin for Apache HBase that has created a lot of buzz. Many companies are successfully using this technology, including Salesforce.com, where Phoenix first started.

Internally, Phoenix takes your SQL query, compiles it into a series of native HBase API calls, and pushes as much work as possible onto the cluster for parallel execution. It automatically creates a metadata repository that provides typed access to data stored in HBase tables. Phoenix’s direct use of the HBase API, along with coprocessors and custom filters, results in performance on the order of milliseconds for small queries, or seconds for tens of millions of rows.

Regardless of these helpful features, Phoenix is not a drop-in RDBMS replacement. There are some limitations:

1. Phoenix doesn’t support cross-row transactions yet.
2. Itz query optimizer and join mechanisms are less sophisticated than most COTS DBMSs.
3. As secondary indexes are implemented using a separate index table, they can get out of sync with the primary table (although perhaps only for very short periods.) These indexes are therefore not fully-ACID compliant.
4. Multi-tenancy is constrained—internally, Phoenix uses a single HBase table.

Unlike Impala, however, Phoenix is intended to operate exclusively on HBase data; its design and implementation are heavily customized to leverage HBase features including coprocessors and skip scans.

Top-3 differences between Impala and Phoenix

1. The main goal of Phoenix is to provide a high-performance relational database layer over HBase for low-latency applications. Impala’s primary focus is to enable interactive exploration of large data sets by providing high-performance, low-latency SQL queries on data stored in popular Hadoop file formats. Hive is mainly concerned with providing data warehouse infrastructure, especially for long-running batch-oriented tasks.
2. Phoenix is a good choice, for example, in CRUD applications where you need the scalability of HBase along with the facility of SQL access. In contrast, Impala is a better option for strictly analytic workloads and Hive is well suited for batch-oriented tasks like ETL.
3. Phoenix is comparatively lightweight since it doesn’t need an additional server.

Next Cloudera release CDH 5.5 will be packaged with Apache Phoenix to leverage. I'm waiting !!!

Amazon Physical Store

In the recent Wall Street Journal (WSF), Amazon.com Inc. plans to open a store in the middle of New York City.

Itz the first brick-and-mortar outlet in its 20-year history and an experiment to provide the type of face-to-face experience found at traditional retailers.  Physical store is contradictory strategy of online Amazon store.

Operating stores also carries risks. Until now, Amazon largely has avoided some costs associated with retailing, including leases, paying employees and managing inventory in hundreds of stores. Those expenses could imperil the company’s already thin profit margins.

If it is successful, however, the New York location could presage a roll-out to other U.S. cities, according to the people familiar with the company’s thinking.

Amazon took some inspiration from a trial by the U.K.’s Home Retail Group PLC, allowing customers to order eBay goods online and pick them up in its Argos stores. By year’s end Argos expects to provide the service at 650 stores from 65,000 eBay sellers.

Other primarily online retailers have opened physical storefronts, including clothier Bonobos Inc., eyeglasses purveyor Warby Parker, and subscription beauty-products service Birchbox.

One philosophy is striking in my mind.  Life is a circle, which will repeat/rotate the trends.

Virtusa acquires Polaris

Today, Virtusa announces definitive agreement to acquire a majority interest in Polaris Consulting & Services, Ltd.

Virtusa and Polaris collectively will have approximately 18,000 employees, generating $826 million of pro forma revenue for the twelve months ended September 30, 2015.

Upon the closing of the Polaris transaction, the combination of Virtusa and Polaris would create a leading global provider of IT services and solutions to the banking and financial services industry segment. The acquisition would combine Virtusa’s deep domain expertise in consumer and retail banking with Polaris’ proven strength in corporate and investment banking. This combination would provide an end-to-end portfolio of differentiated solutions to the global banking and financial services industry segment, improving the combined entity’s competitive position, and expanding its addressable market.

Virtusa expects to realize over $100 million of cumulative revenue synergies over the next three fiscal years from the business combination.

Ref: http://www.thehindubusinessline.com/info-tech/virtusa-to-acquire-53-of-polaris-consulting/article7846701.ece

​

Cloud Based Load Testing using TFS

What is Cloud Based Load Testing?
It goes without saying that performance testing your application not only gives you the confidence that the application will work under heavy levels of stress but also gives you the ability to test how scalable the architecture of your application is. It is important to know how much is too much for your application!

Working with various clients in the industry, it has been realized that the biggest barriers in Load Testing & Performance Testing adoption are,

1. High infrastructure and administration cost that comes with this phase of testing
2. Time taken to procure & set up the test infrastructure
3. Finding use for this infrastructure investment after completion of testing

Endurance Testing
To test for endurance, Azure diagnostics is used to begin with, but later started using Cerebrata Azure Diagnostics Manager to capture the metrics of the machine under test. Currently Microsoft Load Test service does not support metrics from the machine under test.

 Herez an use case of quote generation engine with an expected fixed user load and ran the test for very long duration such as over 48 hours and observed the affect of the long running test on the Azure infrastructure.

Threshold Testing
Another use case to step load the quote generation engine by incrementing user load with different variations of incremental user load per minute till the application crashed out and forced an IIS reset.

It depicts the threshold testing as above.

There a few limits on the usage of Microsoft Cloud based Load Test service that you can read at http://blogs.msdn.com/b/visualstudioalm/archive/2013/06/26/load-testing-with-team-foundation-service-launching-preview-and-early-adoption-program.aspx

Dell EMC

Computer-maker Dell Inc struck a deal on Monday to buy data storage company EMC Corp for $67 billion in cash & stock with 19% premium. The acquisition, the year's third-largest in all sectors, top in technology sector.

The deal should help privately held Dell, the world's No. 3 computer maker, diversify from a stagnant consumer PC market and give it greater scale in the more profitable and faster-growing market for cloud-based data services.

The deal will be financed through a combination of new equity from Dell's owners - founder and Chief Executive Michael Dell, its investment firm MSD Partners, private equity firm Silver Lake and Singapore state-owned investor Temasek Holdings - as well as the issuance of the tracking stock, new debt and cash on hand.

Michael Dell, with the help of Silver Lake, took the PC maker private in $25 billion deal two years ago.

Dell first approached EMC in October 2014 following speculation over a deal between Hewlett-Packard and EMC collapsing and Elliott attacking the company, the source said.

Michael Dell then met EMC Chief Executive Joe Tucci at the World Economic Forum Annual Meeting 2015 in Davos in January, the source added. Negotiations between Dell and EMC intensified in the last two months.

The transaction is expected to close between May and October 2016, the companies said.  The combined company aims to be a leading enterprise player in the following markets:

• Servers and Storage fro the Enterprise Market
• Virtualization via VMware;
• Converged infrastructure (EMC owns VCE);
• Hybrid cloud and cloud computing
• And security via RSA, which is owned by EMC.

Airbnb Airpal

We know that Airbnb is a popular website for people to list, find, and rent lodging. It has over 1,500,000 listings in 34,000 cities and 190 countries.

Recently, Airbnb  launched Big Data tool Airpal, a web-based query execution tool that leverages Facebook’s PrestoDB to facilitate data analysis by authoring queries and retrieving results simple for users.

Key features of Airpal:

• optional access controls for users
• ability to search and find tables
• see metadata, partitions, schemas, and sample rows
• write queries in an easy-to-read editor
• submit queries through a web interface
• track query progress
• get the results back through the browser as a CSV
• create new Hive table based on the results of a query
• save queries once written
• searchable history of all queries run within the tool

Requirements are:

• Java 7 or higher
• MySQL database
• Presto 0.77 or higher
• S3 bucket (to store CSVs)
• Gradle 2.2 or higher

On keeping with the spirit of Presto, they have tried to make it simple to install Airpal by providing a local storage option for people who would like to test it out without any overhead or cost.

For more detailed information, visit the GitHub page here: https://github.com/airbnb/airpal

Coursera Architecture

Coursera is a venture backed for-profit educational technology company that offers massive open online courses (MOOCs).

It works with top universities and organizations to make some of their courses available online, and offers courses in physics, engineering, humanities, medicine, biology, social sciences, mathematics, business, computer science, digital marketing, data science and other subjects.  Itz an online educational startup with over 14 million learners across the globe to offer more than 1000 courses from over 120 top universities.

At Coursera, Amazon Redshift is used as primary data warehouse because it provides a standard SQL interface and has fast and reliable performance. AWS Data Pipeline is used to extract, transform, and load (ETL) data into the warehouse. Data Pipeline provides fault tolerance, scheduling, resource management and an easy-to-extend API for ETL processing.

Dataduct is a Python-based framework built on top of Data Pipeline that lets users create custom reusable components and patterns to be shared across multiple pipelines. This boosts developer productivity and simplifies ETL management.

At Coursera, 150+ pipelines were executed to pull the data from 15 data sources such as Amazon RDS, Cassandra, log streams, and third-party APIs. 300+ tables are loaded every day into Amazon Redshift, processing several terabytes of data. Subsequent pipelines push data back into Cassandra to power our recommendations, search, and other data products.

The attached image below illustrates the data flow at Coursera.