Feature Flag Driven Development

With 25 years of industry experience, it is so exciting to sail through the positive transformations of Information Technology (IT). This week blog, I'm going to cover the engineering process in stead of usual technology topic.

On start of my career, Waterfall process was ruling the industry. In the world of waterfall, you will typically see one continuous build that culminates in a single deploy.  After this deploy, you’ll receive feedback and fix some bugs, but you will likely need to restart the process for any major feature releases.

Waterfall matures into Agile, Continuous Delivery, Test Driven, Lean, Feature Flag Driven development process. Diagram depicts Agile vs FDD.

Emerging industry trend is Feature Flag Driven Development (FDD), which is completely aligned with the recent model - "Collaboration". Microsoft is the best role model for this transformation.  They were closed technology company during inception; but promoting strongly on "Collaborative" Open Source model.

Feature flag driven development allows you to quickly release iterations of your features to market, receive feedback, improve, and redeploy.  It allows you to roll out features to small segments of your users in order to mitigate risk all while receiving valuable feedback.  More importantly, your team will converge and collaborate based on real market feedback and make the necessary improvements to drive the product forward.

Leader moves life forward with the sacrifices, who jump in and get their hands dirty actually doing it.  I'm blessed to be part of FDD transformation using my current work assignment.

Ref: https://martinfowler.com/bliki/FeatureToggle.html

Best Article Award

Today, I'm pleased to receive the best article award from CodeProject.  Thanks for your support and encouragement.

Article Ref: http://www.codeproject.com/Articles/1120056/Evolution-of-Microsoft-Bot-Framework

My Inbox has the below note:

Hi Ganesan Senthilvel,

Congratulations! You have won CodeProject Best "Everything Else" Article of March 2017 Second Prize.

Type: Article
Location: Evolution of Microsoft Bot Framework 

Again - thanks for your contribution to the CodeProject community.

Best regards,

Sean and the CodeProject Team

---

CodeProject

Note: This message has been sent from an unattended email box.

Algorithm Badge

As per my famous quote, IT professionals need to breadth day in and day out with software coding.

•    Chef can't survive without Food
•    Carpenter without Wood
•    Software without Code

In the process of Continuous learning, evaluation & sharing, Industry provides so many free platforms on your choice:

1. Hands-on interactive learning on emerging technology (using CodeAcademy, etc.)
2. Learnt skill Validation (using HackerRank, TopCoder, LeetCode, InterviewBit, etc.)
3. Bench marking with Industry Certification (Microsoft-.NET, Oracle-Java, Cloudera-Big Data, etc.)
4. Sharing the (l)earning (CodeProject, Pulse, Blogspot, Wordpress, etc.)

I'm delighted to share the first milestone @ HackerRank.  Yeps.  Badge (Level-1 Completion) on my favorite area - Algorithm.

Just a recap.  HackerRank focuses on competitive programming challenges using problem statements and contests in different domains of Computer Science. Inspiring to solve interesting and challenging programming problems using itz platform.

HackerRank tech domain validates the depth & breadth of your technical skill.  Each problem statement is enjoyable and motivating to gear further. Interesting part is the completion of all test cases within 3 seconds :-

Software Professionals will generally agree that learning the latest and hottest set of technologies will help you advance your career and avoid getting left behind.  In addition, most of us know that specialization in one technology stack triumphs shallow knowledge in many others.

In spite of the industry saturation and high business/cost challenges, Software companies are always in high demand for the quality hands-on IT professionals.

With this motivation & interest, let me continue my passionate coding path.  Happy Coding all IT professonals !!!

No Shared Resource

As I was pretty busy in one of the big data project, I'm kind of busy in Production move and related activities.

One of the key lesson learnt is not to use network access storage like NAS/SAN.  It is highly recommended to use the local storage with the key factor 'data locality'. You can understand this concept by understanding how MapReduce and HDFS works in Big Data world.

As you know, HDFS is a distributed file system. Think of it as RAID over the Internet. What if you had a 10GB file and 10 servers, and each disk could burst 1GBps? Assume your Ethernet is also 10GB. Well, if you read the file from one server at one time and got back 1GBps, it would take 10 seconds. What if you could read all 10GB at once? In essence, that is what HDFS allows: a burst from your cluster that's bigger than the burst you could get from any individual node.

As another point, MapReduce is built by Google with the divide and conquer concept. The given problem is broken into small independent chunk and sent to each node, as part of MapReduce algorithm.  On completion of the calculated answers in parallel, the result is sent back and combined as reduced concept.  If your application add in network hops and latency along with multiple nodes contending for the same resources, then you loose the power of high performance by choosing hadoop in shared platform.

As the last point, the application doesn't have the priority control on the shared resource like NAS/SAN, by default.  So, the app loose its performance benchmark even though it is running under MapReduce concept.

The conclusion is not to use the shared resource, on building the high performance, massively parallel processing MapReduce methodology.

Facebook Presto

Potentially raising the bar on SQL scalability, Facebook has released open source SQL query engine called Presto that was built to work with petabyte-sized data warehouses.

Currently, more than 1,000 Facebook employees use Presto daily to run 30,000 interactive queries, involving over a petabyte of processing. The company has scaled the software to run on a 1000-node cluster

Unlike Hive, Presto does not use MapReduce, which involves writing results back to disk. Instead, Presto compiles parts of the query on the fly and does all of its processing in memory. As a result, Facebook claims Presto is 10 times better in terms of CPU efficiency and latency than the Hive and MapReduce combo.
.
Now, Facebook wants other data-driven organizations to use, and it hopes, refine Presto. The company has posted the software’s source code at https://github.com/Facebook/presto. The software is already being tested by a number of other large Internet services, namely AirBnB and Dropbox.

Big Data Expo

Big Data Expo is the single most effective event for you to learn how to use you own enterprise data – processed in the cloud – most effectively to drive value for your business.

A recent Gartner report predicts that the volume of enterprise data overall will increase by a phenomenal 650% over the next five years.

These two unstoppable enterprise IT trends, Cloud Computing and Big Data, will converge in Silicon Valley at Big Data Expo 2013 Silicon Valley – being held November 4 - 7, 2013, at the Santa Clara Convention Center in Santa Clara, CA.

It offers in Silicon Valley a vast selection of technical and strategic breakout sessions, General Sessions, Industry Keynotes, our signature discussion "Power Panels" and a bustling Expo floor complete with two busy Demo Theaters so that as a delegate you can kick the tires of solutions and offerings, and discuss one-on-one with all the leading Cloud and Big Data players what they are offering and how to make use of it in your particular situation.

Just as Big Data and Cloud solutions will be side by side on the Expo floor, so they are in the conference program. We're including below a sampler of the breakouts you can look forward to in the Big Data part of the overall technical program...and if you look at the welter of company logos at the very bottom you'll get a foretaste of the hundreds of companies in attendance on the Expo floor.

The surest way to help yourself and your company obtain maximum value from your enterprise data and to understand why Big Data is going mainstream is by attending Big Data Expo 2013 Silicon Valley

BigData Elite

The fund, called Big Data Elite, is backed by celebrity investors like Ron Conway, one of Silicon Valley’s most prominent angel investors, and Andreessen Horowitz, the $2.5 billion venture capitalist firm, as well as Social+Capital Partnership and Anand Rajaraman, whose social media start-up was bought by Wal-Mart Stores in 2011.

In an announcement on Thursday, Big Data Elite described itself as a venture lab and early stage fund that will offer a six-month program beginning January 2014. The fund will choose 10 start-ups or individuals from a list of 20. Those chosen will work a Big Data Elite’s offices in San Francisco and will have access to advisers who work at Facebook, Zynga, Netflix, LinkedIn, Riot Games and a handful of other companies that rely heavily on data analysis.

Companies like Netflix, Google and Amazon rely heavily on the data its customers provide through their Internet searches, and over recent years they have poured large sums of money into data teams of computer scientists who sift through search and transaction data to find patterns.

“Big Data is behind every sector,” Mr. Venios said. “By 2016, the budget spend for Big Data will be larger in corporate business development than information technology,” Mr. Venios added, citing data from McKinsey & Company.

The fund will invest $150,000 to $1 million in each participant, in exchange for equity — typically 6 percent initially — and convertible notes.

QueryIO

QueryIO is a Hadoop-based SQL and Big Data Analytics solution, used to store, structure, analyze and visualize vast amounts of structured and unstructured Big Data. It is especially well suited to enable users to process unstructured Big Data, give it a structure and support querying and analysis of this Big Data using standard SQL syntax.

QueryIO enables you to leverage the vast and mature infrastructure built around SQL and relational databases and utilize it for your Big Data Analytics needs.

QueryIO builds on Apache Hadoop's scalability and reliability and enhances basic Hadoop by adding data integration services, cluster management and monitoring services as well as big data querying and analysis services. It makes it easy to query and analyze big data across hundreds of commodity Compute+Store cluster nodes and petabytes of data in an easy and logical manner.

Highly funded startup

Riding the Big Data tidal wave, MongoDB has raised $150 million in financing to accelerate its global expansion. The round was led by an unnamed financial services company and attracted industry heavy hitters such as Salesforce and Intel.

Now the most valuable startup in New York, MongoDB offers a highly scalable document-oriented database for analytical and fast-changing operational workloads. The platform has been downloaded over 5 million times, and deployed by SAP, MTV Networks, Foursquare and other big name companies.

As part of its aggressive growth strategy, MongoDB will expand its partner ecosystem and target new markets. The company also plans to double its headcount to 600 over the next 12 months.

Originally known as 10gen, the SQL vendor rebranded in August to “get back into alignment” with its flagship product. Shortly afterwards, rival MemSQL updated its distributed in-memory database to support JSON, a popular syntax for processing semi-structured data.  It also improves index scan performance and expands support for SQL.

Hadoop on Windows

Hortonworks, a leading contributor to and provider of enterprise Apache Hadoop, today announced the general availability of Hortonworks Data Platform 1.3 (HDP) for Windows, a 100-percent open source data platform powered by Apache Hadoop.  HDP 1.3 for Windows is the only Apache Hadoop-based distribution certified to run on Windows Server 2008 R2 and Windows Server 2012, enabling Microsoft customers to build and deploy Hadoop-based analytic applications. This release is further demonstration of the deep engineering collaboration between Microsoft and Hortonworks.

New functionality in HDP 1.3 for Windows includes HBase 0.94.6.1, Flume 1.3.1, ZooKeeper 3.4.5 and Mahout 0.7.0. These new capabilities enable customers to exploit net new types of data to build new business applications as part of their modern data architecture.

Hortonworks Data Platform 1.3 for Windows is now available for download at: http://hortonworks.com/download/

Intel Big Data

Intel, increasingly customizing server chips for customers, is now tuning chips for workloads in big data.

Software is becoming an important building block in chip design, and customization will help applications gather, manage and analyze data a lot quicker, said Ron Kasabian, general manager of big data solutions at Intel.

The plan includes developing accelerators or cores for big-data type workloads. For example, Intel is working with Chinese company Bocom to implement the Smart City project, which tries to solve counterfeit license plate problems in China by recognizing plates, car makes and models. The project involves sending images through server gateways, and Intel is looking to fill software gaps by enhancing the silicon.

A lot of research is also taking place at Intel labs on stream processing and graph analytics as the company designs chips and tweaks software.

BigData Ayasdi

Ayasdi’s Insight Discovery platform highlights include automatic discoveries from complex data and operationalization of end-to-end analytic workflow tied to complex and expensive business problems. It computes across hundreds to millions of attributes to automatically find similarity amongst data points and surface hidden patterns and anomalies in the data. Users can also deploy an API to automate integration across in-house and third party applications to fully operationalize the end-to-end analytic workflow.

 Topology isn't new, but using it to analyze and understand big data is a novel idea -- one that could empower business users to find value in very large data sets without having to consult data scientists or write algorithms or models.

That's the promise of Ayasdi, a Palo Alto-based startup that uses topological data analysis to quickly glean meaning from big data.The company says its approach to analyzing big data is unique. Based on the work of Stanford University mathematics professor and Ayasdi cofounder Gunnar Carlsson, Ayasdi's enterprise-focused tools apply the abstruse concepts of topology to quickly identify relevant patterns in data.

HCatalog

HCatalog is a table and storage management layer for Hadoop that enables users with different data processing tools – Pig, MapReduce, and Hive – to more easily read and write data on the grid. HCatalog’s table abstraction presents users with a relational view of data in the Hadoop distributed file system (HDFS) and ensures that users need not worry about where or in what format their data is stored – RCFile format, text files, or sequence files

HCatalog supports reading and writing files in any format for which a SerDe (serializer-deserializer) can be written. By default, HCatalog supports RCFile, CSV, JSON, and SequenceFile formats. To use a custom format, you must provide the InputFormat, OutputFormat, and SerDe.
 

IBM Big Data training

IBM hopes to help create the next generation of “big-data” specialists through a series of partnerships with universities around the world, as well as influence the curriculum.

Nine new agreements announced Wednesday involve Georgetown University, George Washington University, Rensselaer Polytechnic Institute, the University of Missouri, and Northwestern University in the U.S. IBM is also beginning big-data programs at Dublin City University, Mother Teresa Women’s University in India, the National University of Singapore, and the Philippines’ Commission on Higher Education.

They will result in a variety of programs, including a master of science degree in the business analytics track at George Washington University; an undergraduate course titled “Big Data Analytics” at the University of Missouri; and a center for business analytics at the National University of Singapore.

In its announcement, IBM cited U.S. Bureau of Labor statistics that found there will be a 24 percent rise in demand for people with “data analytics skills” over the next eight years.

While companies are managing to fill big data positions, there’s a caveat. “They are finding the candidates but a lot of what they’re doing is poaching candidates from other companies,” spokesman said. “One of the reasons I would expect IBM is making these partnerships to make sure there’s enough engineers to meet the demand they’re seeing.”

Ideal data scientist

FICO, a leading predictive analytics and decision management software company, today released an infographic showing the characteristics of a good data scientist — what a Harvard Business Review article called the “sexiest job of the 21st century.”

The rise of Big Data has fueled demand for data scientists. Indeed.com reported that job postings for analytic scientists jumped 15,000 percent between the summer of 2011 and 2012. McKinsey & Company predicted the U.S. will see a 50- to 60-percent shortfall in analytic scientists by 2018.

“There’s more demand than ever for data scientists, but at the same time we demand more from job candidates,” said Dr. Andrew Jennings, chief analytics officer at FICO and head of FICO Labs. “FICO has been hiring data scientists — or analysts, as we used to call them — since 1956. We’ve learned that excellent math skills alone just aren’t enough. We want someone who can solve problems for businesses, and explain their insights to people who don’t have a Ph.D. in operations research.”

The FICO infographic identifies eight characteristics of a good data scientist. These include the ability to tease out insights from data, communicate with business users and focus on the practical applications of their work.

Social Intelligence

No matter what industry a business operates in, data is now being used more than ever before to gain an advantage. Social is only one of the newest layers in this big data bonanza, and some companies that were early adopters are starting to mature their models into Social Intelligence.

Enterprises have an average of 178 social media accounts, the report found, and an array of departments and executives are increasingly active there. However, when it comes to things like customer relationship management, analytics and market research, social data is mostly isolated. This leads to disjointed efforts across a company, and doesn’t allow for a strategic, holistic view to be put into place.

It’s becoming a roadblock as companies seek to really tap into social data insights, so companies need to develop a common framework for social data collection and integration. Not doing so could result in poorer customer experiences, and of course, missed opportunities.

Altimeter collected input from 34 enterprise organizations on how to integrate social data, and how to build holistic systems that scale for its report.

Big Data Stream

Stream computing is a new paradigm necessitated by new data-generating scenarios, such as the ubiquity of mobile devices, location services, and sensor pervasiveness. A crucial need has emerged for scalable computing platforms and parallel architectures that can process vast amounts of generated streaming data.

In static data computation (the left-hand side of attached diagram), questions are asked of static data. In streaming data computation (the right-hand side), data is continuously evaluated by static questions.

Let me give a simple example.  In financial trading platform, applications are written traditionally to analyse the historical records in the batch mode.  Meaning, we preserved the data in the data ware house.  Based on the user request/query, the result is produced/returned back to the consumer.  It is the first use case.

With big data streaming technology, the requests (like market trend of IT stocks) are pre built  On the arrival/streaming of the data, the results are published to the prescribed subscriber/consumer.  Isn't it too cool to taste the technology?

Storm at Yahoo

Yahoo! is enhancing its web properties and mobile applications to provide its users personalized experience based on interest profiles. To compute user interest, we process billions of events from our over 700 million users, and analyze 2.2 billion content every day. Since users' change interest over time, we need to update user profiles to reflect their current interests.

Enabling low-latency big-data processing is one of the primary design goals of Yahoo!’s next-generation big-data platform. While MapReduce is a key design pattern for batch processing, additional design patterns will be supported over time. Stream/micro-batch processing is one of design patterns applicable to many Yahoo! use cases.

Yahoo! big-data platform enables Hadoop applications and Storm applications to share data via shared storage such as HBase.  Yahoo! engineering teams are developing technologies to enable Storm applications and Hadoop applications to be hosted on a single cluster.

Hunk

Splunk is getting on board with a new Hadoop-based application it cheekily calls Hunk. Hunk takes Splunk’s popular analytics platform and puts it to work on data stored in Hadoop. Businesses that use Hadoop can now use this for exploration and visualization of data. Itz key features are:

 

• Splunk Virtual Index: Splunk virtual index technology enables the “seamless use” of the entire Splunk technology stack, including the Splunk Search Processing Language (SPL), for interactive exploration, analysis and visualization of data stored anywhere, as if it was stored in a Splunk software index.
• Explore data in Hadoop from one place: Hunk is designed for interactive data exploration across large, diverse data sets on top of Hadoop.
• Interactive analysis of data in Hadoop: Hunk enables users to drive deep analysis, detect patterns, and find anomalies across terabytes and petabytes of data.
• Create custom dashboards: Hunk users can combine multiple charts, views and reports into role-specific dashboards which can be viewed and edited on laptops, tablets or mobile devices.

 

Splunk has 5,600 customers, which includes half of the Fortune 100. It says the new Hunk product will target both new and existing customers, as long as they use Hadoop.

 

Citrix hypervisor open source

IT departments reevaluating their server virtualization spending have another free option to choose from.

Citrix Systems Inc. today released its XenServer 6.2 hypervisor to the open source community. All of the features previously in the XenServer Platinum Edition will be available at no charge.

Citrix had made parts of XenServer code open source in the past, as part of various cloud computing initiatives, but this marks the first time the entire product has been made available to the community.

Competitor  VMware pricing has come under fire, but Microsoft's Hyper-V -- not XenServer -- has been the biggest beneficiary. More than 22% of respondents to TechTarget's 2013 Data Center and Reader's Choice survey identified Microsoft as their primary server virtualization hypervisor -- up from 13% last year. XenServer's numbers actually dropped slightly, from 4.1%in 2013 to 3.3% in 2012, while VMware's tumbled from 73% to 60%.

Meanwhile, cloud computing has created an inflection point where organizations are reconsidering how many workloads they need to run in-house and how much money they need to spend to do so.