Redshift Spectrum

Redshift Spectrum helps to run SQL queries against data in an Amazon S3 data lake as easily as you analyze data stored in Amazon Redshift. It achieves without loading data or resizing the Amazon Redshift cluster based on growing data volumes.

Redshift Spectrum separates compute and storage to meet workload demands for data size, concurrency, and performance.  It scales processing across thousands of nodes, so results are fast, even with massive datasets and complex queries. It is possible to query open file formats that you already use—such as Apache Avro, CSV, Grok, ORC, Apache Parquet, RCFile, RegexSerDe, SequenceFile, TextFile, and TSV—directly in Amazon S3, without any data movement.

Top 3 performance features are:

1. Short Query Acceleration - speed up execution of queries such as reports, dashboards, and interactive analysis
2. Results Caching - deliver sub-second response times for queries that are repeated, such as dashboards, visualizations, and those from BI tools
3. Late Materialization - reduce the amount of data scanned for queries with predicate filters by batching and factoring in the filtering of predicates before fetching data blocks in the next column

AWS Summit video at https://www.youtube.com/watch?v=gchd2sDhSuY

MongoDB ACID Tackle

MongoDB is making the ACID feature of 4.0 available for testing in public beta.

With the most recent release improving the robustness of BI queries, database writes, schema validation, and Python support, the upcoming 4.0 release will tackle ACID.

Ref: https://www.mongodb.com/transactions?jmp=hero

Amazon Spheres

Amazon’s Spheres, the centerpiece of the retail juggernaut’s $4 billion urban campus, was opened to employees on Jan'18 end.

Amazon has built an architecturally ambitious focal point and symbol of its status as Seattle’s largest employer and one of the most dynamic entities in corporate America.

The flora inside the domes is the work of a team of Amazon horticulturists who were charged with scouring the globe for interesting plants and growing them in a 40,000-square-foot greenhouse in Woodinville.

Four years of growth later, and supplemented by a few older plants acquired from other conservatories, the Spheres feel like a mature, if unusually well-manicured, jungle.  Back to nature, now !!

C# Nullable

Technology is fulfilled when it adds business value.

Nullable data type helps to process optional business parameter.  As an example, App shouldn't store '0' in 'manufacturing year' as optional/default integer; it leads to wrong business context.

Current issue of Microsoft magazine illustrates tech concept using C# 

Ref: https://msdn.microsoft.com/en-us/magazine/mt829270

Angular 3 year support

AngularJS is planning one more significant release, version 1.7, and on July 1, 2018 it will enter a 3 year Long Term Support period.

AngularJS is an extremely stable framework for building web applications, and has been used by millions of developers across the web. Angular is its successor and uses the same philosophies such as declarative templates and dependency injection. 

Many developers are still using AngularJS, and that the migration process to Angular takes time and energy.

All AngularJS applications that work now, will continue to work in the future. All published versions of AngularJS, on npm, bower, CDNs, etc will continue to be available.

Redshift ETL

In ETL world, Amazon Redshift is revolutionary to make the developer life simple.  It is used to calculate daily, weekly, and monthly aggregations, which are then unloaded to S3, where they can be further processed and made available for end-user reporting using a number of different tools, including Redshift Spectrum and Amazon Athena.

The proposed ETL process has 4 key steps to execute:

Step 1:  Extract from the RDBMS source to a S3 bucketIn this ETL process, the data extract job fetches change data every 1 hour and it is staged into multiple hourly files.

Step 2: Stage data to the Amazon Redshift table for cleansing
Ingesting the data can be accomplished using a JSON-based manifest file. Using the manifest file ensures that S3 eventual consistency issues can be eliminated and also provides an opportunity to deduce any files if needed.

Step 3: Transform data to create daily, weekly, and monthly datasets and load into target tables
Data is staged in the “stage_tbl” from where it can be transformed into the daily, weekly, and monthly aggregates and loaded into target tables.

Step 4: Unload the daily dataset to populate the S3 data lake bucketThe transformed results are now unloaded into another S3 bucket, where they can be further processed and made available for end-user reporting using a number of different tools, including Redshift Spectrum and Amazon Athena.

India Economics 2018

Last 3 years, Indian politics has different perspective on the current Prime Minister Mr. Narendra Modi.  Forget about all the criticism against his initiatives.  In 4 members family, we used to have difference of opinions; think about 1.4 billion people with different culture, language, religion, etc.

Being an individual from my motherland, he inspires fellow citizen that every thing is possible in life from poor boy to prime minister of the largest powerful democratic nation.

Being industry leader, I've 2 dimension to review (technical; not political) his effort namely leadership skill and business domain.

(1) Leadership Skill
As Warren says "Leadership is the capacity to translate vision into reality".  Mr. Prime minister demonstrated strong leadership skill with courageous execution on few long pending initiatives like Global Sales Tax (GST), Demonetization, Unique identity (Aadhar) roll out, Citizens welfare on Health policy, Digital Innovation, etc.

(2) Business Domain
Indian Prime Minister prepares this week to address global business and political leaders in Davos, Switzerland, as his country passes France and the U.K. to become the world’s fifth-largest economy, underscoring the South Asian nation’s drive for recognition as a great power.

Some economists calculate that India’s gross domestic product jumped into the top five last quarter as it continued to outgrow every country in Europe—and for that matter most of the rest of the world. It has been reflected in Wall Street Journal (WSJ).
Ref: https://www.wsj.com/articles/davos-offers-modi-stage-to-push-muscular-vision-for-india-1516552979

Big salute to all the contributors.  Jai Hind !