Zeppelin

Apache Zeppelin is an open source GUI which creates interactive and collaborative notebooks for data exploration using Spark. You can use Scala, Python, SQL (using Spark SQL), or HiveQL to manipulate data and quickly visualize results.

Zeppelin notebooks can be shared among several users, and visualizations can be published to external dashboards. Zeppelin uses the Spark settings on your cluster and can use Spark’s dynamic allocation of executors to let YARN estimate the optimal resource consumption.

To run the prediction analysis, you need to create notebooks that generate prediction % and are scheduled to run daily. As part of the prediction analysis, we needed to connect to multiple data sources, like MySQL and Vertica for data ingestion and error rate generation. This enabled us to aggregate data across multiple dimensions, thus exposing underlying issues and anomalies at a glance.

Using Zeppelin, we applied many A/B models by replaying our raw data in AWS S3 to generate different prediction reports, which in turn helped us move in the right direction and provide better forecasting.

Zeppelin helps us to turn the huge amounts of raw data, often from across different data stores, into consumable information with useful insights.

Slide share reference is available at http://www.slideshare.net/prajods/big-data-visualization-with-apache-spark-and-zeppelin

Amazon Lambda

What is AWS Lambda?
AWS Lambda is a compute service where you can upload your code to AWS Lambda and the service can run the code on your behalf using AWS infrastructure.

AWS Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging.

All you need to do is supply your code in one of the languages that AWS Lambda supports (currently Node.js, Java, and Python)

Best Use Cases
AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.

Itz an ideal compute platform for many application scenarios, best 5 use cases are listed as below:

Production Site

Teletext.io is well known as the serverless start-up, entirely built around AWS, but leveraging only the Amazon API Gateway, Lambda functions, DynamoDb, S3 and Cloudfront.

This is not only a really scalable solution with an almost infinite peak capacity, but also a very cheap solution (< $90K) as per their publication.

Conclusion

On back of my mind, Martin Fowlers' Infrastructure As Code, is bumping.  He said "Infrastructure as code is the approach to defining computing and network infrastructure through source code that can then be treated just like any software system."

Awesome experience on learning the emerging technology and its business value.

I/O 2016

As I wrote in my weekly blog @ last month, Google I/O has just completed with some exciting industry updates.  Letz see what are those new offers?

Google I/O 2016, Alphabet’s annual developer conference, has wrapped up for the year between May 18-20 at the Shoreline Amphitheater in Mountain View, California. This year, the announcements came thick and fast but many were not quite ready for general demoing or product release, so we’ll have to wait a little longer to see them in action.

Herez few major announcements from Google I/O 2016, interested to me.

Google Assistant
Assistant is a kind of revamped version of Google Now. Itz a beefed-up voice assistant that can help you do much more than ever before.

It can now provide users with more natural two-way conversations and do much more than just schedule events and perform quick Google searches.

Google Home
Home is the central hub for connecting Google Assistant with your connected home, smart devices and more. Itz a direct competitor to the Amazon Echo.

Like Echo, Google Home integrates a built-in bluetooth speaker and microphone all in small, sleek package. Home is meant to be your hub to all thing Google, right from your living room, kitchen, or wherever you place.

Google Allo
Allo is a new telephone number-based messaging app, which has 3 main aspects as self-expression, Assistant integration, and security privacy.

As Allo features end-to-end encryption, incognito chats, private notifications and expiring chats, it will be the “first home for Google Assistant”.

Google Duo
Duo is a new cross-platform video calling app, companion of Allo. Duo lets you initiate fast one to one video calls.

Duo is a companion video chat app that allows you to see the person calling you – in a live video preview – before you even answer the call.

Daydream VR (Virtual Reality)
Itz Google’s vision for an affordable mobile virtual reality platform. The first Daydream-ready devices will arrive later this year.

Android N will feature a VR Mode for putting chip sets into “performance mode”, add head-tracking algorithms, support sub-20 ms latency on mobile devices and render incoming messages and calls in 3D to appear in stereo.

On the software front, Daydream provides a standard VR interface for mobile and adds a VR category to the Play Store.

Google Chrome OS
Contrary to the rumors that Chrome OS would be folded into Android, Google officially denied the claim.

After tackling wearables, TVs, and autos, Google is in need of a real computer operating system. So, Chrome OS will live on in some form, even if it’s just functionality integrated into Android.

Android "N"
Itz the code name of an upcoming release of the Android OS (Operating System). It has few key features like multi window support, direct reply notification, data saver, picture in picture & no device flashing. Release schedule of N will be:

Watch List
Complete set of Google I/O 2016 videos are available at YouTube series. Enjoy it at https://www.youtube.com/playlist?list=PLOU2XLYxmsILe6_eGvDN3GyiodoV3qNSC

Stay tuned for more on all the development from Google I/O

VB Silver Anniversary

This week is turned to be to mark the 25th (“Silver Anniversary”) considering that Visual Basic (VB) 1st debuted to the world. A celebration so comprehensive it is also a marathon.  VB has the journey across each era, from VB 1. to VB6 to the early days of VB.NET to Roslyn

Despite much of the hate VB has gotten over the years, it served an insanely important purpose in the rise of internal business software. Itz amazed at how much of their world still runs on VB6 applications. Something like 1/3 of insurance software still does.

The transition from VB6 to VB.NET was a really sad one as it lost a lot of people - .NET is a lot more difficult than it was.  The result is that an entire group of people simply stopped making software and now we have businesses running on applications that are more than 20 years old that some random person in the company threw together over a week.

There's a huge gulf between building a VB6 app and throwing together a web app today and despite much of the progress that  has been made, we've taken some big steps back in terms of accessibility.

The difference between VB.NET and C# is pretty superficial, but I sincerely hope that someday people can experience the magic that something like VB6 offered.  A lot of people got their start in programming using Visual Basic.

Google IO

Google I/O is an annual developer conference at which the company announces new hardware and software, hosts educational sessions pertaining to its various products and services, and broadly tries to build hype and warm feelings in the hearts of creators and fans who watch online.

The event begins with a keynote. This year, the host will be Google CEO Sundar Pichai. We expect him and his colleagues to make a number of announcements, like giving an official name to its new mobile operating system, revealing more of Google's plans for augmented and virtual reality, and detailing how Android would work on the desktop.

How to watch

Starting time: 
San Francisco: 10AM / New York: 1PM / London: 6PM / Berlin 7PM / Moscow: 8PM / Beijing: 1AM (May 19th) / Tokyo: 2AM (May 19th) / Sydney 3AM (May 19th).

Sessions live-streaming : 
You can also watch sessions throughout the two-day event right here.  Google's live stream is available via the company's own website.
https://events.google.com/io2016/schedule?filters=Live%20streamed#day1/

Live tweeting: 
Follow @Verge on Twitter for the latest headlines and specs as they emerge.

Stay tuned for more updates from google

Incremental Design

Last Saturday (May 7), I took part in Design Workshop, conducted by ThoughtWorks Chennai division.

Theme of the workshop was "Incremental Design", which is considered as the chance to validate the participant's software design/coding skills.  This session was hosted by GM Sivsu with around 100 coding participants.

Over a period of 4 hours, the participants found out how Incremental Design works by solving the given Retail use case.  It is allowed to have either pair with a fellow participant or work solo on a given problem statement in any programming language of your choice.  A series of check points during the course of the workshop will ensure that we pause, observe and learn to design the right way.

Personally, I learnt the below Top 5 key items on solving the given problem statement.
1. Understanding the User
2. Functional Coverage
3. Stability
4. Defining Core features
5. Extension

1. Understanding the User
Usually Personas are created and validated through User Research in the field.

2. Functional Coverage
Functionality is covered with the basic/strong test cases and the matching result set.

3. Stability
At any point of failure cases, App should be stable with graceful way of exit/error.

4. Defining Core features
Inevitably when building a product, teams are constrained by time, money, resources, etc and can’t build everything they want to at once. At the end of the workshop the solutions are compared with the different feature ideas they had come up with, to select the best.

5. Extension
As the core theme of Incremental design, the later change requests should be adoptable to extend further.

With the interesting hands-on mode of learning, signing off this week activity.

Microsoft Azure

In the latest Forrester Wave report for Big Data Hadoop Cloud Solutions, Microsoft Azure came on top beating both Google and Amazon Web Services.

Microsoft specifically was called out for having a cloud-first strategy that is paying off. Tiffany Wissner, Senior Director Product Marketing, Data Platform mentioned the following in a recent blog post,

Our goal is to make big data processing and analytics simpler and more accessible to bring big data to everybody. We do this through the Cortana Intelligence Suite which offers a fully managed big data and advanced analytics solution in the cloud.

Within Cortana Intelligence is the Azure Data Lake which includes HDInsight, a managed Hadoop service that runs Hortonworks Data Platform, Data Lake Analytics, a new service built on Apache YARN that dynamically scales your big data jobs, and Data Lake Store, a single repository to capture data of any size, type, and speed. We also offer IaaS images in Azure Marketplace that deploy any third party Hadoop distribution (Hortonworks, Cloudera, and MapR).

After 37-criteria evaluation, Forrester recognized Microsoft Azure as a leader in their Big Data Hadoop Cloud Solutions.

Forrester Ref: https://www.forrester.com/report/The+Forrester+Wave+Big+Data+Hadoop+Cloud+Solutions+Q2+2016/-/E-RES126541