AWS Trasnfer FEDRAMP

 

Federal Risk and Authorization Management Program (FedRAMP) is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services.

AWS Transfer Family provides fully managed file transfers over SFTP, FTPS, and FTP for Amazon S3 and Amazon EFS. The Federal Risk and Authorization Management Program (FedRAMP) is a US government-wide program that delivers a standard approach to the security assessment, authorization, and continuous monitoring for cloud products and services

AWS Transfer Family is now authorized as FedRAMP Moderate in US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon) and as FedRAMP High in GovCloud(US-West) and GovCloud(US-East).

 US Federal agencies and commercial customers working with the US Federal government can now utilize AWS Transfer Family to run sensitive and highly regulated file transfer workloads.
 
 AWS compliance reference is available at https://aws.amazon.com/compliance/services-in-scope/

AWS Log4j Vulnerability

We know Log4j security vulnerability hits the world recently and everyone is reacting to resolve immediately.  AWS is not an exception and their fixes are listed below.

More information about the Java hot patch is available at https://aws.amazon.com/blogs/security/open-source-hotpatch-for-apache-log4j-vulnerability/

Amazon Connect
Amazon Connect has been updated to mitigate the issues identified in CVE-2021-44228. 

Amazon Chime
Amazon Chime SDK services have been updated to mitigate the issues identified in CVE-2021-44228 and CVE-2021-45046.

Amazon EMR
CVE-2021-44228 impacts Apache Log4j versions between 2.0 and 2.14.1 when processing inputs from untrusted sources. EMR clusters launched with EMR 5 and EMR 6 releases include open source frameworks such as Apache Hive, Apache Flink, HUDI, Presto, and Trino, which use these versions of Apache Log4j. When you launch a cluster with EMR’s default configuration, it does not process inputs from untrusted sources. Many customers use the open source frameworks installed on their EMR clusters to process and log inputs from untrusted sources. Therefore, AWS recommends that you apply the solution described here.

Amazon Fraud Detector
Amazon Fraud Detector services have been updated to mitigate the issues identified in CVE-2021-44228.

Amazon Kendra
Amazon Kendra has been updated to mitigate CVE-2021-44228.

Amazon Lex
Amazon Lex has been updated to mitigate the issues identified in CVE-2021-44228.

Amazon Macie
The Amazon Macie service has been updated to mitigate the issues identified in CVE-2021-44228.

Amazon Macie Classic
The Amazon Macie Classic service has been updated to mitigate the issues identified in CVE-2021-44228.

Amazon Monitron
Amazon Monitron has been updated to mitigate the issues identified in CVE-2021-44228.

Amazon RDS
Amazon RDS and Amazon Aurora have been updated to mitigate the issues identified in CVE-2021-44228.

Amazon Rekognition
Amazon Rekognition services have been updated to mitigate the issues identified in CVE-2021-44228.

Amazon VPC
Amazon VPC, including Internet Gateway and Virtual Gateway services, have been updated to mitigate the Log4j issue referenced in CVE-2021-44228.

AWS AppSync
AWS AppSync has been updated to mitigate the issues identified in CVE-2021-44228 and CVE-2021-45046.

AWS Certificate Manager
AWS Certificate Manager services have been updated to mitigate the issues identified in CVE-2021-44228.

AWS Service Catalog
AWS Service Catalog has been updated to mitigate the issues identified in CVE-2021-44228.

AWS Systems Manager
AWS Systems Manager service has been updated to mitigate the issues identified in CVE-2021-44228. The Systems Manager agent itself is not affected by this issue.

Software Resilience

 

This week blog is based on my work experience in AWS.  On Tuesday 7 Dec 2021, our products got affected due to AWS multi hours global outage. 

The first phase of outage began at approximately 15:35 UTC (7:35 am PT), when multiple Amazon sites and services began to show significant performance degradation. While site loading appeared to mostly normalize by 16:50 UTC (8:50 am PT), we observed AWS API service failures that caused API transactions to experience dramatically higher completion times or simply time out.

Second wave of the outage lasted for over 7 hours, not fully resolving until approximately 0:44 UTC (4:44 pm PT) . We managed our products with Disaster Recovery (DR) strategy.

Amazon confirmed that service issues with AWS main US-East-1 region, located in Northern Virginia, were causing problems for its warehouse and delivery network.

On reading the various industry news, it was an opportunity to learn AWS influences in the industry.

AWS controlled 33% of the global cloud infrastructure market in the second quarter, according to Synergy Research Group, followed by Microsoft at 20% and Google at 10%.

As the result, key industry observations are

• brought down many popular websites and services.
• some of Amazon’s delivery operations ground to a halt, and third-party sellers couldn’t ship products.
• Colleges that rely on software to host content had to postpone exams during finals week.
  

As software engineer, personal learning/improvement (recollecting golden words of my mentor Suresh) - resilience is to re-adapt to any crisis situation intelligently

ECS container health

 Amazon Elastic Container Service (Amazon ECS) now provides customers enhanced visibility into the health of their compute infrastructure.  It helps customers improve application resiliency.

Customers running containerized workloads using Amazon ECS on Amazon Elastic Compute Cloud  (Amazon EC2) or on-premises with Amazon ECS Anywhere can now query the health status of the container runtime (i.e Docker) for their container instances directly from the Amazon ECS API.

mazon ECS automatically monitors the container runtime for responsiveness on customers’ behalf. Customers can use the ECS Describe-Instances API with the include Health Status option to view the health information for their Amazon ECS Tasks.

Customers can view the instance health status for all their Amazon ECS container instances running version 1.57.0 of the Amazon ECS container Agent or higher.

Athena CPU cost

 Last week, AWS launched a new featured in Amazon Athena.  Now, it displays the computational cost of your queries alongside their execution plans.

With the release of the EXPLAIN ANALYZE statement, Athena can now execute your specified query and return a detailed breakdown of its execution plan along with the CPU usage of each stage and the number of rows processed.

In addition to understanding a query’s execution plan, you can now see the time spent within each operator to better assess the performance profiles of query clauses and their chosen ordering. With row input and output counts, you can also validate the impact of query predicates, especially over large datasets.

Administrators will also find the scanned data counts useful in planning the financial impact of their users’ workloads and identifying queries that could benefit from further optimization or that should be governed to control costs using Athena’s data usage controls.

Babelfish for Aurora

Babelfish for PostgreSQL is an open source project available under the Apache 2.0 and PostgreSQL licenses. It provides the capability for PostgreSQL to understand queries from applications written for Microsoft SQL Server.

Babelfish understands the SQL Server wire-protocol and T-SQL, the Microsoft SQL Server query language, so you don’t have to switch database drivers or re-write all of your application queries. With Babelfish, applications currently running on SQL Server can now run directly on PostgreSQL with fewer code changes.

The source code of open source project Babelfish is available at https://www.babelfishpg.org/getstarted/#source

This allows users to leverage Babelfish on their own PostgreSQL servers. Babelfish includes support for stored procedures, save points, static cursors, nested transactions, the variant data type and much more.

Last week, AWS announced Babelfish for Aurora PostgreSQL as Generally Available (GA).

Fault Injection Simulator

AWS Fault Injection Simulator is a fully managed fault injection service that makes it easier for teams to discover an application’s weaknesses at scale in order to improve performance, observability, and resiliency.

Fault Injection Simulator simplifies the process of setting up and running controlled fault injection experiments across a range of AWS services so teams can build confidence in their application behavior.

With Fault Injection Simulator, teams can quickly set up experiments using pre-built templates that generate the desired disruptions. Fault Injection Simulator provides the controls and guardrails that teams need to run experiments in production, such as automatically rolling back or stopping the experiment if specific conditions are met. 

With a few clicks in the console, teams can run complex scenarios with common distributed system failures happening in parallel or building sequentially over time, enabling them to create the real world conditions necessary to find hidden weaknesses.

Documentation ref: https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#fis-actions-reference-fis

LaunchConfig vs. LaunchTemplate

 It's been a while to push for the migration of AWS terraform from LaunchConfig to LaunchTemplate.  

Just wondering why it's so important.  Here's the answer to the question.

 

So, it's highly recommended to leverage LaunchTemplate in any of your terraform at the earliest.

Windows2022 in AWS

 Microsoft has just released its most recent Windows Server platform - WindowsServer2022.

Why?

1. Advanced multi-layered security - numerous security enhancements with Secured-core server and secured connectivity.
2. Hybrid capabilities with Azure - hybrid and multi-cloud approach to digitally transform the businesses
3. Flexible application platform -  scalability for 48TB of memory and 2,048 logical cores running on 64 physical sockets

What?
Minimum system requirements are:

• 1.4 GHz 64-bit processor
• Compatible with x64 instruction set
• Supports NX and DEP
• Supports CMPXCHG16b, LAHF/SAHF and PrefetchW
• Supports Second Level Address Translation (EPT or NPT)
• 512 MB (2 GB for Server with Desktop Experience installation option)
• ECC (Error Correcting Code) type or similar technology, for physical host deployments
• An Ethernet adapter capable of at least 1 gigabit per second throughput
• Compliant with the PCI Express architecture specification
• UEFI 2.3.1c-based system and firmware that supports secure boot
• Trusted Platform Module 2.0

AWS AMI
Last week, Amazon announced to create and manage Microsoft Windows Server 2022 AMIs (Amazon Machine Image) providing a reliable and quick way to launch Windows Server 2022 on AWS EC2 instances. Ref: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/windows-ami-version-history.html

Conclusion
This event depicts the collaboration and seamless cross platform integration in a faster and easier way in cloud technologies.  Happy Cloud!

Make500

 On BlogSpot alone (beyond other publications), I've written over 500 passionate weekly blog posts.

Appreciate almighty and every human being for Make500 opportunity to me, on logging into 50th life year.

Details are inked at https://www.linkedin.com/pulse/make500-ganesan-senthilvel

SSM Execution Timeout

 

Problem Statement

One of core business servers didn't get boot up on scaling out new instance.  The next box is stale and didn't proceed.  The business impact - no new deployments are possible to release further.

Why

On deep analysis, it has been found that the bootstrap execution got stuck after 60 minutes.  Question is how do we resolve this?

One clue is timeout settings in send-command of aws ssm document.  There are two types namely delivery and execution timeout; both with default of 3600 seconds.

Total timeout is equal to the value of delivery timeout plus execution timeout. If execution timeout isn't required by the SSM document, then total timeout is equal to the value of delivery timeout plus default execution timeout.

How

Let us review the purpose of two timeout parameters. 

If Systems Manager receives an execution timeout reply from SSM Agent on a target, then Systems Manager marks the command invocation as executionTimeout.

If Run Command doesn't receive a document terminal response from SSM Agent, the command invocation is marked as deliveryTimeout.

To fix this bootstrap stale state, the below terraform code is built programmatically

resource "aws_ssm_document" "TestServer-ssmCommand" {
  name          = "TestServer-Execute-Userdata-Prod"
  document_type = "Command"

  content = <<DOC
  {
    "schemaVersion": "2.0",
    "description": "Downloads and executes the userdata for Test Server",
    "parameters": {},
    "mainSteps": [
      {
        "action": "aws:runShellScript",
        "name": "runShellScript",
        "inputs": {
            "timeoutSeconds": 4500,
            "runCommand": [
              "sudo yum install dos2unix -y",
              "sudo yum install aws-cli -y ",
              ......
            ]
        }
    ]
  }
}

Key take away is timeoutSeconds property in mainSteps->inputs section of ssm_document object.

Conclusion

Thus the reported booting timeout issue is resolved to meet the business expectation.  Technology needs to enable the business.

Hudson FilePath Pipeline

Problem Statement

Last week, one of build servers start failing every jobs.  The business impact is not to proceed the entire build and deployment processes of every release.

Every jenkins build jobs report the below exceptions without any clue.  
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] sh
Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node
[Pipeline] End of Pipeline
org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing
......
Finished: FAILURE
 

Why

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.  It is an automated expression for getting right versioned software in the underlying build server.
 

There are two types of pipelines in Jenkins namely declarative and scripted.  Declarative pipeline is a more recent feature of Jenkins pipeline that provides better syntactical features compared to scripted pipeline syntax.
 

How

On troubleshooting the build job failures, it has been found that some software configuration/setup got changed.  

The resolution steps are applied in two steps processes
1. Upgraded the required pipeline plugins as in screenshot
2. Restarted the build server after plugins upgrades.
 

There are two ways of build server restart namely cold and warm. As we know, cold restart is the complete physical restart of the build server.  In warm restart, click on "Restart from Stage" of classic UI panel once pipeline has completed.
 

Conclusion

Thus the repored build server error is resolved to meet the business expectation.  Technology needs to enable the business.

Lifecycle Hooks

 

Problem Statement

One of the monster production box holds the core business functionalities with the complete set of data and services. 

During the deployment, the scale-in process kick starts only after an hour.  The business impact is to await for the new scale-out services, after an hour of the effectiveness.

Why

A lifecycle hook provides a specified amount of time (one hour by default) to complete the lifecycle action before the instance transitions to the next state.

It allows to control what happens when Amazon EC2 instances are launched and terminated as they are scale out and in.

How

On troubleshooting the mess-up in lifecycle hook, it has been found the error in instance_terminate lifecycle transition.  As the resolution, new lifecycle hook is created for instance_terminate with the right heartbeat timeout and auto scaling default result with 'Continue' option.

This fix is applied in two methods - at AWS console manually and at terraform programmatically.

resource "aws_autoscaling_lifecycle_hook" "app" {
    depends_on             = ....
    autoscaling_group_name = ....
    name                   = ....
    default_result         = "CONTINUE"
    heartbeat_timeout      = 600
    lifecycle_transition   = "autoscaling:EC2_INSTANCE_TERMINATING"
    notification_target_arn = ....
    role_arn                = ....
}

Conclusion

Thus the reported scale-in delay problem is resolved to meet the business expectation.  Technology needs to enable the business.

Custom Widgets for CloudWatch

 

Last week, Amazon CloudWatch announced the immediate availability of custom widgets.  This new feature is used to gain operational visibility and agility by customizing the content of your CloudWatch dashboard such as adding visualizations, displaying information from multiple data sources or adding controls like buttons to take remediation actions.

Custom widgets can help you to correlate trends over time and spot issues more easily by displaying related data from different sources side by side on CloudWatch dashboards. Custom widgets allow you to extend your CloudWatch dashboards’ out of the box capabilities including line, bar and pie charts with rich, business specific visualizations that represent the operational health and performance of your workloads.

Technically, custom widgets has 3 steps process as below:

1. CloudWatch dashboard calls the Lambda function containing the widget code. It passes in any custom defined parameters in the widget.
2. The Lambda function returns a string of HTML, JSON, or Markdown. Markdown is returned as JSON {"markdown":"markdown content"}
3. The dashboard displays the returned HTML or JSON.

A set of templates and a sample library is available at https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/add_custom_widget_samples.html

Toyota Connected AWS

 

Toyota Connected North America (TC) is a technology/big data company that partners with Toyota Motor Corporation and Toyota Motor North America to develop products that aim to improve the driving experience for Toyota and Lexus owners.

Couple of days ago, AWS published a technical whitepaper to design, build, refine, and deploy this collision assistance product with Serverless on AWS services. The associated architecture is highlighted here.

In the aftermath of an accident, Collision Assistance offers Toyota and Lexus drivers instructions to help them navigate a post-collision situation. This includes documenting the accident, filing an insurance claim, and transitioning to the repair process.

Gartner AWS 2021

 

Last month, Gartner Names AWS a 2021 Magic Quadrant Leader. Gartner post ref: https://www.gartner.com/doc/reprints?id=1-271OE4VR&ct=210802&st=sb&refid=em_inv1_gartner_mq_leader

In this report, Gartner briefed the reason to position AWS as a Leader.  It has the details to learn how a Leader in this report is defined and dive deep into the benefits. Thus, public cloud can bring to any organizations in the industry.

Cloud adoption continues to increase as agile development, rapid deployment, and unlimited scale become the new normal for customers of all industries, sizes, and geographies.

In Gartner's second evaluation covering both cloud infrastructure and platform services, AWS is evaluated as a Leader placed highest in both axes of measurement, Ability to Execute and Completeness of Vision.

MemoryDB for Redis

 

Last week, AWS announced MemoryDB for Redis, whch is a new Redis-compatible, durable, in-memory database service to deliver ultra-fast performance.

It is best suit for modern applications with microservices architectures. Industry metric says more than 13 trillion requests per day to support peaks of over 160 million requests per second.

Amazon MemoryDB is compatible with Redis, a popular open source data store. Redis is well proven industry product as most loved database for five consecutive years by Stack Overflow.

Amazon MemoryDB stores data durably across multiple Availability Zones (AZs) using a Multi-AZ transactional log to enable fast failover, database recovery, and node restarts. Delivering both in-memory performance and Multi-AZ durability, Amazon MemoryDB can be used as a high-performance primary database for your microservices applications eliminating the need to separately manage both a cache and durable database.

Technical documentations are available at https://docs.aws.amazon.com/memorydb/index.html

CloudWatch Usage Metric

Last week, Amazon CloudWatch logs started to support CloudWatch usage metrics, which enables the customer to monitor the CloudWatch Logs API Usage. It is not only monitoring but also alarms notification.  

Using usage metrics, the system alarms are configured to otify on breaching the rule like CloudWatch Logs API service quota.  The process flow can be visualized of the usage on CloudWatch dashboards.

 CloudWatch Logs Usage Metrics technical documentation at https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch-Logs-Monitoring-CloudWatch-Metrics.html#CloudWatchLogs-Usage-Metrics

Multus CNI

 This week, Amazon Elastic Kubernetes Service (EKS) announced to support the Multus Container Networking Interface (CNI) plugin, enabling pods running in EKS clusters to attach multiple network interfaces in support of advanced networking configurations.

Multus CNI is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces. This is accomplished by Multus acting as a "meta-plugin", a CNI plugin that can call multiple other CNI plugins.

Amazon EKS provides a highly available managed Kubernetes service that is available in all global AWS regions, and supported in edge locations like AWS Local Zones and AWS Outposts. Using Multus with EKS enables advanced networking across these environments to run containerized network functions that deliver high quality content to end users.

Github site details at https://github.com/k8snetworkplumbingwg/multus-cni

AWS Cloudwatch

 

This week blog is based on the life experiences between mathematics and computers.

Last week, Amazon cloud platform (AWS) announced support for the trimmed mean statistic on CloudWatch metrics.

Details are published at https://www.linkedin.com/pulse/maths-computers-ganesan-senthilvel

Textract

 Amazon Textract is a machine learning service to extract text, handwriting and data from scanned documents automatically.  It goes beyond simple optical character recognition (OCR) to identify, understand, and extract data from forms and tables.

Last month, AWS made an accuracy enhancement update to the handwriting extraction feature with an improved accuracy of handwritten transcriptions, specifically for numerals, dates, phone numbers, and website address across many documents in finance, healthcare, legal, public sector, and others. Textract now more accurately detects handwriting within documents such as checks, medical forms, travel forms and more.

As part of the AWS Free Usage Tier, you can get started with Amazon Textract for free. New customers can analyze up to 1,000 pages per month using the Detecting Document Text API and up to 100 pages per month using the Analyze Document API.

Kendra Crawler

 

Amazon Kendra is an intelligent search service powered by machine learning, enabling organizations to provide relevant information to customers and employees, when they need it.

Amazon Kendra recently introduce the web crawler to index and search webpages.

Critical information can be scattered across multiple data sources in an enterprise, including internal and external websites. Kendra web crawler helps to index documents made available on websites (HTML, PDF, MS Word, MS PowerPoint, and Plain Text) and search for information across this content using Kendra Intelligent Search.

The Kendra web crawler honors access rules in robots.txt, and customers using the Kendra web crawler will need to ensure they are authorized to index those webpages in order to return search results for end users.

Ref: https://aws.amazon.com/kendra/

Cloud Architect

 

Today, got my second cloud certification in AWS (Amazon Web Service) as 2021 self learning goal.  

It's follow-up of last year cloud work at https://ganesansenthilvel.blogspot.com/2020/06/cloud-practitioner.html

AWS Certified Solutions Architect exam is a testimony of the snatched role opportunity as Cloud Architect.  

This certification is an acid test to me by Amazon for the below abilities

• Design and deploy dynamically scalable, highly available, fault-tolerant, and reliable applications
• Select appropriate AWS services to design and deploy an application based on given requirements
• Migrate complex, multi-tier applications on AWS
• Design and deploy enterprise-wide scalable operations on AWS
• Implement cost-control strategies

Since 2008, it's my 13th certification year in the relevant career opportunities like .NET, Big Data, IoT, SAFe, Blockchain, Cloud, etc.

Continuous Learning is the minimum requirement for success in any field.

Athena Parameterized Query

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Recently, AWS worked on architecture improvements.  As the result, Amazon Athena leverages the re-usability, simplification, and security benefits of parameterized queries.

Last Thursday, analysts can save time and eliminate errors by converting queries that have frequently modified criteria, such as date filters or aggregation periods, into a parameterized query that serves multiple use cases. Application developers can use them to safeguard against SQL injection risks and simplify application integrations that generate SQL based on a user’s selections.

As an example, there is a query for retail sales metrics where the filters for product category, region, and date can vary from one execution to the next. Instead of manually modifying the logic in your SQL code each time you run the query, you can use variables for product category, region, and date whose values are provided when executing the query.

Parameterized query is available in SELECT and INSERT INTO queries through the Athena console, API, and SQL clients using Athena’s ODBC or JDBC drivers. Ref: https://docs.aws.amazon.com/athena/latest/ug/querying-with-prepared-statements.html

Cloud Watch design

 

AWS CloudWatch is a vital instrumentation service for all resources of AWS.  By design, it has two core components

1. Metric
2. Alarm

In AWS Metric model, a dimension is metrics metadata in the form of a name/value pair. Metrics can have up to ten dimensions. When you set dimensions, AWS services send both data and metadata to CloudWatch. Dimensions can be useful for filtering data and aggregating statistics.

CloudWatch Alarms feature allows you to watch CloudWatch metrics and to receive notifications when the metrics fall outside of the levels (high or low thresholds) that you configure. You can attach multiple Alarms to each metric and each one can have multiple actions.

Workflow Studio

 

AWS Step Functions allow you to build scalable, distributed applications using state machines. As of now, building workflows on Step Functions required you to learn and understand Amazon State Language (ASL).

Now, AWS provides a low-code visual tool namely Workflow Studio. It helps you learn Step Functions through a guided interactive interface and allows you to prototype and build workflows faster.

Workflow Studio is great for developers who are new to Step Functions, because it reduces the time to build their first workflow and provides an accelerated learning path where developers learn by doing. It helps developers who are experienced in building workflows, because they can now develop them faster using a visual tool. For example, you can use Workflow Studio to do prototypes of the workflows and share them with your stakeholders quickly.

The attached screeshot shows how easy is to create a state machine using Workflow Studio.

Multi region KMS

This month, AWS Key Management Service (AWS KMS) introduced multi-Region keys.  It is a new capability that lets you replicate keys from one AWS Region into another. With multi region keys, it is easy to move the encrypted data between Regions without having to decrypt and re-encrypt with different keys in each Region. 

Multi region keys are supported for client-side encryption in the AWS Encryption SDK, AWS S3 Encryption Client, and AWS DynamoDB Encryption Client.  The process of creating a CMK in AWS KMS, is depicted in the attached diagram with seven steps process.

This new service simplify any process that copies protected data into multiple Regions, such as disaster recovery/backup, DynamoDB global tables, or for digital signature applications that require the same signing key in multiple Regions.

AWS KMS makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and in your applications. AWS KMS is a secure and resilient service that uses hardware security modules that have been validated under FIPS 140-2.

Proton GA

 

Last week, AWS announced the general availability (GA) of AWS Proton. It is the first fully managed delivery service for container and serverless applications, which is designed to provide platform teams the management tools, governance, and visibility needed to provide consistent standards and best practices when managing deployments, while helping to increase developer productivity and innovation.

Using infrastructure as code, platform operators can create a template that defines and configures everything needed to provision, deploy, and monitor a service.

Two key features got launched. First, AWS Proton supports multi-account infrastructures. With this new capability, platform operators can use AWS Proton to help configure and manage their architecture securely across multiple AWS accounts. The second feature release supports IAM condition context keys in AWS Proton APIs. With this additional layer of control, operators can designate which developers can create services based on template characteristics.

Product page at https://aws.amazon.com/proton/

App Runner

 Last week, AWS announced the general availability of AWS App Runner.

App Runner is a fully managed container application service that makes it easy for customers without any prior containers or infrastructure experience to build, deploy, and run containerized web applications and APIs in just a few clicks. 

As an end user, it is not required to be tech savvy.  You simply provide source code, a container image, or deployment pipeline; rest will be done by App Runner.  It automatically builds and deploys the web application, load balances traffic, scales on demand, and monitors application health.

Out of the box, App Runner is built for web scale, so there’s no need to re-platform or re-architect as the business grows. It makes it simpler to rapidly deliver innovative solutions and business value.

Documentation is available with three sections (1) Developer Guide, (2) API Reference and (3) Release Notes.  Ref: https://docs.aws.amazon.com/apprunner/

Data Prediction

Indian Institute of Technology (IIT) Kanpur built the Covid prediction data model for India's second wave. 

Our scientists are working on the SUTRA model for charting the trajectory of COVID-19. They have been working on a mathematical model to predict the spread of the virus.

It is important to note that a mathematical model can only predict future with some certainty so long as virus dynamics and its transmissibility don’t change substantially over time. Mathematical models can also provide a mechanism to predicting alternate scenarios corresponding to various policy decisions such as non-pharmaceutical interventions.

 It is quite accurate and in sync with the current date. Ref: https://www.sutra-india.in/

CloudFront functions

 

Amazon CloudFront Functions is ideal for lightweight CloudFront content delivery network customization.  It can run on every request to enable high scale and latency sensitive operations like HTTP header manipulations, URL rewrites/redirects, and cache key normalization. For example, you can use CloudFront Functions to rewrite requests to language specific versions of your site based on the Accept-Language header of the incoming request. You can also use CloudFront Functions to validate custom tokens to authorize incoming requests.

As these functions run at all of CloudFront’s edge locations, they can scale instantly to millions of requests per second with minimal latency overhead.  It is built for lightweight HTTP(S) transformations and manipulations, allowing you to deliver richer, more personalized content with low latency to your customers.

CloudFront Functions is natively built-in to CloudFront, allowing you to easily build, test, and deploy viewer request and viewer response functions entirely within CloudFront.

Few useful sample code repository is available at https://github.com/aws-samples/amazon-cloudfront-functions

Goodbye Python2.7

 

Python is one of the powerful language recently used in the industry.  Why? Because, it is dynamically typed with garbage collected concepts. Also, it supports multiple programming paradigms like structured, procedural, object-oriented and functional programming.

Coming back to AWS Cloud world, Python plays a vital role.  

Few key dates and updates of AWS Lambda Python 2.7 deprecation

• January 1, 2020 - Python Software Foundation (PSF) officially sunset Python 2.7
  
  
• July 15, 2021 - AWS Lambda will end support for Python 2.7
• You will not be able to create new Python 2.7 functions after this date
  
  
• September 30, 2021 - Unable to update existing Python 2.7 functions
• Invokes to existing functions will not be affected
• Existing function runtime can still be changed
  
  
• Resources:
• AWS Blog: https://aws.amazon.com/blogs/compute/announcing-end-of-support-for-python-2-7-in-aws-lambda/
• PSF Migration Guide: https://docs.python.org/3/howto/pyporting.html

It is high time to take an immediate action against the existing Python 2.7 Lambda functions in your firm.

AWS CloudShell

 As we know, there are 3 ways to connect AWS infrastructure.  They are:

1. AWS Console (web based interface)
2. AWS CLI (Command Line Interface)
3. AWS API (Application Program Interface)

Non technical users prefer to use AWS Console.  Scripting engineers are used to leverage CLI.  Third option API is used by application engineers.

Recently, start of the year, AWS CloudShell is available to use as attached.

AWS CloudShell is a browser-based, pre-authenticated shell that the user can launch directly from the AWS Management Console. It runs any AWS CLI commands against AWS services using your preferred shell (Bash, PowerShell, or Z shell). Cool thing is no need to download or install command line tools.

Happy Cloud Access in simple way!

Request Response Measure

 

In continuation of the last blog,  today we are going to talk about 2 key metric of any system.  They are

1. Request time (input)
2. Response time (output)

It is essential to indicate the performance of the underlying system.  

In AWS cloud world, the input request comes as user/system input to our processing engine.  AWS ELB (in simple term cluster manager) is the middle layer to route the concurrent bulk requests in an evenly distributed load.  The actual processing engine is hosted in EC2 instance.

Here, request processing time is the time taken from user request to ELB.  It is measured in every load balancer log records along with the back end processing time between ELB and actual EC2 server.

Response processing time is measure as time taken to return the response from ELB.

Thus, the fundamental request and response time elements are integral part of ELB logs in AWS cloud systems.

ELB Log Format

 

AWS Elastic Load Balancing (ELB) helps us to build systems that are highly scalable and highly reliable. It is possible to distribute traffic automatically across a dynamically-sized collection/cluster of Amazon EC2 instances.

Elastic Load Balancers provides the accessibility for the entire transactions log. After enabling and configuring this feature of ELB, log files can be delivered to the Amazon S3 bucket of the configuring choice.

ELB Log files are generated in a plain-text format, one line per request. Each line contains a total of twelve fields (as attached here). These fields are self explanatory about their holding information at ELB infrastructure level.

 In long run, each log records are highly beneficial for the business instrumentation.

Athena EXPLAIN

 Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

In any SQL code, EXPLAIN statement displays execution plans chosen by the optimizer for SELECT , UPDATE , INSERT , and DELETE statements. A statement execution plan is the sequence of operations that the database performs to run the statement.

This week, Amazon Athena introduced the support of the execution plan for the submitted queries. When used in the Athena console, the EXPLAIN statement provides a detailed breakdown of a query's execution plan. Users can analyze the execution plan to identify and reduce query complexity and improve run time.

EXPLAIN can also be used to validate SQL syntax prior to execution. When EXPLAIN is used, Athena does not execute the underlying query. This allows analysts to tune their query without waiting for the results to be returned or incurring costs for data scanned.

 Usage reference is at https://docs.aws.amazon.com/athena/latest/ug/athena-explain-statement.html

Interactive EC2 Serial Console

 

Long pending issue / cause
On early 2010's, there were few users request about a use case where the console output was “Continue to wait; or Press S to skip mounting or M for manual recovery.”

Cause was due to the absence of interactive console feature to discover - "when boot failed, why SSH daemon did not start, errors configuring the firewall or network which blocked all access, broken networking on the instance, or denial-of-service attacks.

Alternative Solutions
There was an alternative solution by the system administrator. With an inaccessible EC2, it is possible to stop the instance, detach the storage, mount the storage on a working instance, and edit or recover the files from there. This is not always possible, though.

If VM uses instance type storage, this cannot be detached. It also requires interruption of service.

If a VM uses ephemeral storage, then the question is why do troubleshoot VM rather than simply deleting it and creating a new one?

When do we need?
Most AWS users never need this feature because SSH access does not often fail, and the range of use cases is relatively narrow.  But, still, it is worth troubleshooting feature.

Generally Available
This month, AWS interactive EC2 Serial Console is generally available. EC2 Serial Console provides a simple and secure way to troubleshoot boot and network connectivity issues by establishing a connection to the serial port of an instance. This access can be used for interactive troubleshooting.

AWS DevOps monitoring

 

AWS provides a set of flexible developer tools that enable customers to host code, build, test, and deploy applications. Each tool provides customers with separate activity and usage metrics.

With AWS DevOps Monitoring Dashboard solution, you can capture and analyze metrics across these AWS developer tools and view them from a single dashboard. The solution automates the setup of DevOps dashboards in your AWS pipeline environment so you can quickly start measuring mean time to recover, change failure rate, deployment frequency, deployment status, and code change volume.

The solution will begin collecting raw data from the AWS Code Services in your environment. This data is ingested into Amazon S3 and SQL functions query the data to calculate key metrics such as deployment frequency and mean time to recover. You can view the metrics as dashboards in AWS QuickSight, or your preferred visualization tool. You can use the dashboard information to track development velocity as well as other key operational metrics and identify areas for continued improvement.

This week, AWS DevOps monitoring dashboard solution is generally available now.

Amazon Comprehend

 Amazon Comprehend is a natural language processing (NLP) service that uses machine learning to find insights and relationships in text. No machine learning experience required.

Amazon Comprehend now supports identification of text documents that contain personally identifiable information (PII). You can use Amazon Comprehend’s Contains PII API synchronously to discover documents that contain PII, to set up alarms and control access on documents with sensitive information. Amazon Comprehend’s machine learning models find documents that contain PII information such as social security numbers, credit card numbers, and email addresses and allow you to target the PII of your choice.

Amazon Comprehend provides pre-trained models for recognizing entities, key phrases, sentiments, and other common elements in a document. You can also build custom models with Amazon Comprehend to recognize custom entities and classify documents.

EKS performance

 Amazon Elastic Kubernetes Service (EKS) has reduced new cluster creation time by 40%, enabling you to create an EKS cluster in 9 minutes or less, on average.  

Amazon EKS is a managed service that makes it easy for you to run Kubernetes on AWS. Reduced cluster creation time means you can now quickly test new features and iterate on your application infrastructure faster than before. This is especially useful if you have adopted continuous integration and continuous deployment mechanisms that require frequent cluster creation thus improving agility for your teams.

 It was announced couple of days ago by AWS team.

AWS Shield Advanced

 

AWS Shield Advanced now supports tagging of protected resources and protection groups. It is possible for tagging to restrict the ability to create or modify protections to sensitive resources via IAM policies, or to organize and track your AWS Shield Advanced costs at the tag level. 

Resource tagging allows you to define custom names for protected application resources, such as Amazon Elastic Compute Cloud (EC2), Elastic Load Balancing (ELB), Amazon CloudFront, AWS Global Accelerator and Amazon Route 53.  

To associate tags with Shield Advanced protections, log into the Shield Console and navigate to the protected resources tab. From there you can add, edit, or delete tags from existing protected resources or protection groups. Tags can also be added when creating new protections or protection groups through the creation wizard, or through the Shield Advanced API.  

Resource tagging is available to AWS Shield Advanced subscribers at no additional cost. Shield Advanced Developer guide reference is at https://docs.aws.amazon.com/waf/latest/developerguide/ddos-overview.html

AWS Event Driven

 

An event-driven architecture uses events to trigger and communicate between decoupled services and is common in modern applications built with microservices. An event is a change in state, or an update

Event-driven architectures have three key components: event producers, event routers, and event consumers. A producer publishes an event to the router, which filters and pushes the events to consumers. Producer services and consumer services are decoupled, which allows them to be scaled, updated, and deployed independently.

In the given diagram, an event-driven architecture is briefed for an e-commerce site. This architecture enables the site to react to changes from a variety of sources during times of peak demand, without crashing the application or over-provisioning resources.

Many customers are choosing to build event-driven application architectures – those in which subscriber or target services automatically perform work in response to events triggered by publisher or source services. This pattern can enable development teams to operate more independently so they can release new features faster, while also making their applications more scalable.

Technically, AWS covers the basics of event-driven design, using Amazon EventBridge, Amazon SNS, Amazon SQS, AWS Lambda and more.

AWS Digital Labs

 

Last week, AWS launched new digital course - Getting Started with DevOps on AWS. This course explores the basics of developing, delivering, and maintaining high-quality secure applications and services at high velocity on AWS. 

The course covers the philosophies, practices, and tools used to implement a DevOps environment on AWS, while the lab gives you practical experience with the technologies discussed in the course.

By taking this course, you can learn about DevOps methodologies, the key AWS DevOps services, and Amazon’s own transformation journey to DevOps. The lab will present a use case that shows how a company, department, or team can leverage DevOps to increase the quality, speed, and security of their applications. Additional learning about the key DevOps services for automating the continuous integration and continuous delivery (CI/CD) process.

While the course is offered free of charge, the lab costs 10 credits (at 1 USD per credit) on amazon.qwiklabs.com

AWS Workspace on Covid

 

We knew 2020 was highly challenging year across the world due to Covid virus spread.  Entire world was forced to work remotely.

Traditionally we have supplied our software engineers with desktop machines rather than laptops in order to get maximum computing power for the money spent. With the pandemic they all went home and took their desktop machines with them, and have been working from there ever since. As the world is recovering back with the potential for a return to office.

The current remote situation leads to one solution to consider Amazon Workspaces, which is a desktop EC2 instance connected to your network via Amazon's Virtual Private Cloud, and streamed to a client.

It would indeed be more flexible to have your own development environment directly within the cloud. Earlier, AWS shared free tier version to explore/experience AWS Workspaces.  It seems there is another extension of the Free Tier ongoing up until 31st of July 2021.

Technical insights are available at https://aws.amazon.com/workspaces/faqs/

AWS ELB 504 error

Last couple of weeks, had trouble to resolve the mysterious occurrences of AWS ELB 504 errors in our product development.  Identification of root causes, is not straight forward and difficult to trace and debug the time out error in production.

In general, 504 Gateway Timeout is caused by the using the Elastic Load Balancer (ELB) address. When the ELB is unable to reach the underlying url of the request page during the short process-intensive period required for the database setup, the ELB serves the user a 504 error.

Our problem was not straight case with few challenges to understand.  Got a recent/relevant blog which is the exact match of our current production scenarios.  Recommended solution is that all relevant backend timeouts (not just explicit CF keepalive timeouts) must be larger than the ELB’s idle timeout.

Ref: https://sigopt.com/blog/the-case-of-the-mysterious-aws-elb-504-errors/

AWS PrivateLink S3

AWS PrivateLink provides private connectivity between VPCs, AWS services, and your on-premises networks, without exposing your traffic to the public internet. AWS PrivateLink makes it easy to connect services across different accounts and VPCs to significantly simplify your network architecture.

Now, Amazon S3 supports AWS PrivateLink, providing direct access to S3 via a private endpoint within your virtual private network. Simplify your network architecture by connecting to S3 from on-premises or in AWS using private IP addresses in your Virtual Private Cloud (VPC), eliminating the need to use public IPs, configure firewall rules, or configure an Internet Gateway to access S3 from on-premises.

AWS Transfer Family

AWS Transfer Family provides fully managed Secure File Transfer Protocol (SFTP), File Transfer Protocol (FTP) over TLS, and FTP support for Amazon Simple Storage Service (S3), enabling you to seamlessly migrate your file transfer workflows to AWS.

Last month, AWS announced the file transfer support to Amazon Elastic File System (EFS) file systems as well as Amazon S3. This feature enables you to easily and securely provide your business partners access to files stored in Amazon EFS file systems. With this launch, you now have the option to store the transferred files in a fully managed file system and reduce your operational burden, while preserving your existing workflows that use SFTP, FTPS, or FTP protocols.

When Amazon EFS is selected as the data store for your AWS Transfer Family server, the transferred files are readily available to your business-critical applications running on Amazon Elastic Compute Cloud (EC2), as well as to containerized and serverless applications run using AWS services such as Amazon Elastic Container Service (ECS), Amazon Elastic Kubernetes Service (EKS), AWS Fargate, and AWS Lambda. 

Happy Cloud-ing !

AWS Personalize

Amazon Personalize enables developers to build applications with the same machine learning (ML) technology used by Amazon.com for real-time personalized recommendations – no ML expertise required.

It is a fully managed machine learning service that goes beyond rigid static rule based recommendation systems and trains, tunes, and deploys custom ML models to deliver highly customized recommendations to customers across industries such as retail and media and entertainment.

Amazon Personalize provisions the necessary infrastructure and manages the entire ML pipeline, including processing the data, identifying features, using the best algorithms, and training, optimizing, and hosting the models. You will receive results via an Application Programming Interface (API) and only pay for what you use, with no minimum fees or upfront commitments.

All data is encrypted to be private and secure, and is only used to create recommendations for your users.

Federated AWS EKS

 

Federated Amazon EKS Clusters on AWS is a new AWS Solutions Implementation that automates the deployment and federation of two Amazon Elastic Kubernetes Service (Amazon EKS) clusters across multiple AWS Regions, configuring highly available, low latency, and easily scalable applications.

Over the last few years, Kubernetes has gained an increasing popularity for automating application deployment, scaling, and management, and while it has enabled more and more users, it also is taxing to properly configure its consistent use to deploy applications globally and to manage lots of clusters.

It leverages Kubefed to automate the deployment and federation of multiple well-architected Amazon EKS clusters across AWS Regions. It also deploys the pre-configured bastion host and all necessary dependencies of tools and techniques.

Lambda Millisecond Billing

 

Top 5 Features of AWS Lambda during AWS ReInvent 2020     

1. One millisecond billing - helps to pay for value with millisecond granularity
2. Larger function sizes - supports the allocation up to 10GB of memory, 3X increase
3. Container Image Support - benefits sub-second auto-scaling, high availability and native integrations
4. CloudWatch Lambda Insights - collects and summarizes the performance metrics
5. Streaming analytics - lightweight solution for stream aggregations: count, max, average, etc.

 

Here, I want to highlight the top feature 'one millisecond billing' .

With the recent release, AWS Lambda reduces the efficient billing granularity for the customers from 100ms to 1ms.  Eventually, it generates the largest cost benefits even for the short-lived functions.

Effective with December 2020 billing cycle, the customer can save up to 70% on Lambda functions