AWS Notes

Подведение итогов reCap re:Invent 2021 - бизнес-трек:

https://www.youtube.com/watch?v=E6CgUapKl9o

#reInvent

​​Terraform module for setting up OIDC federation between AWS and Github Actions/Gitlab CI:

https://github.com/marco-lancini/utils/tree/main/terraform/aws-oidc-ci

module "ci_oidc" {
source = "./aws-oidc-ci"
allow_github = true
github_org = "<org_name>"
github_repos = [
"<repo2_name>:*",
# To restrict to a branch,
# replace `*` with `ref_type:branch:ref:main`
"<repo1_name>:ref_type:branch:ref:main",
]
}

#Terraform

​​How to send repeated notifications for CloudWatch Alarms:

https://aws.amazon.com/blogs/mt/how-to-enable-amazon-cloudwatch-alarms-to-send-repeated-notifications/

1️⃣ CloudWatch alarm is triggered and goes into the ALARM state.
2️⃣ CloudWatch alarm sends the first alarm notification to the associated SNS alarm actions.
3️⃣ CloudWatch Alarms service sends an alarm state change event which triggers the EventBridge rule.
4️⃣ With a match event, the EventBridge rule invokes the Step Function target.
5️⃣ Once the Step Function starts execution, it first enters a Wait state.
6️⃣ The Step Function enters the Lambda Invocation task. The Lambda invocation task:
▫️ Checks if the alarm has the specific tag key and value (e.g., RepeatedAlarm:true). If not, the function exits
▫️ Checks the alarm’s current state by performing a DescribeAlarms API with the alarm name.
▫️ Publishes the existing alarm’s status returned from the DescribeAlarms API call to all the SNS topics subscribed on the alarm
▫️ Returns the alarm’s current state together with the original received event back to the Step Function.
7️⃣ The Choice state checks the alarm state returned by the Lambda function and directs the workflow to go back to the Wait state if the alarm state is ‘ALARM’ otherwise it ends the step function’s execution.
8️⃣ The repeated notification for an alarm within the workflow above stops when:
▫️ The alarm transitions to a non-ALARM state.
▫️ The alarm is deleted.
▫️ A specific tag is removed from the alarm.

#CloudWatch

​​В AWS сертификации теперь есть бэджи, которые можно получить для какого-то направления.

https://aws.amazon.com/training/badges/

Получить их можно бесплатно, на текущий момент есть лишь блок по хранилищам данных.

AWS Learning badges are credentials that offer these benefits:
🔸 AWS digital badges demonstrate your knowledge and skills for specific AWS Cloud topics, such as Object Storage
🔸 AWS digital badges are shareable on social networks to help you stand out to recruiters and prospective hiring managers
🔸 AWS digital badges are free to earn and share

#AWS_certification

#машины_aws

Одно из моих любимых занятий - ковырять нелюбимые инструменты, чтобы не любить их экспертно и за дело. Один из таких инструментов - CodeDeploy - до неприятия муторный, сложный и противный.

Чего только стоит необходимость запускать Shell скрипты из spec-файла. Да, даже если весь скрипт состоит из одной команды.

Предлагаю разделить мою нелюбовь и начать эту неделю с мультирегионального развертывания приложений с помощью CodePipeline и CodeDeploy. Да еще и на виртуальные машины, чтоб жизнь совсем уж медом не казалась.

​​Как добавить дополнительную информацию об AWS аккаунте в Security Hub:

https://aws.amazon.com/blogs/security/how-to-enrich-aws-security-hub-findings-with-account-metadata/

По умолчанию в Security Hub показывается лишь AWS ID, что становится совершенно бесполезным, как только у вас 10+ аккаунтов (всех по айдишнику не запомнишь).

Ситуацию можно исправить с помощью Лямбды, которая будет тригериться на каждый эвент Security Hub findings, обрабатывая его через через EventBridge.

В результате можно легко увидеть имя AWS аккаунта (см. картинку). Я бы ещё добавил туда и тэги.

#SecurityHub

​​S3 Replication vs AWS Datasync vs S3 Batch Operations vs S3 CopyObject API:

https://aws.amazon.com/blogs/storage/considering-four-different-replication-options-for-data-in-amazon-s3/

#S3

​​Weekly Summary on AWS (January 23-29)

🔸 Chime SDK + background replacement
🔸 Cloud Map API + IPv6
🔸 DocumentDB
➖ 1-month free trial (t3.medium, 750 hours / 30M IOs / 5GB)
➖ $geoIntersects
➖ $geoWithin
➖ $mergeObjects
➖ $reduce
🔸 EC2 X2iezn instances (32:1 ratio of memory to vCPU)
🔸 ECS Anywhere + ECS Exec and Amazon Linux 2
🔸 EFS Replication
🔸 Fraud Detector + prediction explanations
🔸 FSx for ONTAP + increase storage capacity
🔸 GuardDuty + EKS
🔸 Lightsail CDN + Lightsail Container Services
as origin
🔸 MWAA (Airflow) + version 2.2
🔸 PrivateLink + CloudWatch metrics
🔸 QuickSight + comparative and cumulative date/time calculations
🔸 RDS for PostgreSQL 14.1, 13.5, 12.9, 11.14, 10.19, and 9.6.24
🔸 RDS Performance Insights + query execution plan
🔸 S3 File Gateway + schedule-based network bandwidth throttling
🔸 SageMaker Autopilot
➖ 100 GB datasets (before - 10GB)
➖ Apache Parquet file format
🔸 Textract + single page PDF documents & JPEG 2000 encoded images
🔹 Panorama Appliances + available for purchase

#AWS_week

Python While Loop.

​​S3 console — generating a presigned URL:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html#ShareObjectPreSignedURLConsole

The credentials that you can use to create a presigned URL include:

🔸 IAM instance profile: Valid up to 6 hours

🔸 STS: Valid up to 36 hours when signed with permanent credentials, such as the credentials of the AWS account root user or an IAM user

🔸 IAM user: Valid up to 7 days when using AWS Signature Version 4

#S3 #AWS_Console

​​Gitlab Runner on EC2

https://aws.amazon.com/blogs/devops/deploy-and-manage-gitlab-runners-on-amazon-ec2/

This article demonstrated how to utilize IaC to efficiently conduct various administrative tasks associated with a Gitlab Runner.
▪️ We deployed Gitlab Runner consistently and quickly across multiple accounts.
▪️ We utilized IaC to enforce guardrails and best practices, such as tracking Gitlab Runner configuration changes, terminating the Gitlab Runner gracefully, and autoscaling the Gitlab Runner to ensure best performance and minimum cost.

#Gitlab

​​EFS Replication:

https://aws.amazon.com/blogs/aws/new-replication-for-amazon-elastic-file-system-efs/

🔸 Once configured, replication begins immediately. All replication traffic stays on the AWS global backbone, and most changes are replicated within a minute, with an overall Recovery Point Objective (RPO) of 15 minutes for most file systems. Replication does not consume any burst credits and it does not count against the provisioned throughput of the file system.

🔸 EFS tracks modifications to the blocks (currently 4 MB) that are used to store files and metadata, and replicates the changes at a rate of up to 300 MB per second. Because replication is block-based, it is not crash-consistent; if you need crash-consistency you may want to take a look at AWS Backup.

🔸 You pay the usual storage fees for the original and replica file systems and any applicable cross-region or intra-region data transfer charges.

#EFS

Zero-day уязвимости в AWS CloudFormation и AWS Glue.

В середние января Orca Security (израильский стартап в области облачной кибербезопасности с офисом разработке в Минске) опубликовал отчёты о двух найденных критических уязвимостях в инфраструктуре AWS:

1. Ability to gain control plane access to a CloudFormation host and retrieve its AWS credentials:
https://orca.security/resources/blog/aws-cloudformation-vulnerability/

2. Cross-account access via AWS Glue:
https://orca.security/resources/blog/aws-glue-vulnerability/

Обе уязвимости были полностью устранены через несколько дней после сообщения.

Позже были опубликованы Security Bulletins:
https://aws.amazon.com/security/security-bulletins/AWS-2022-001/
https://aws.amazon.com/security/security-bulletins/AWS-2022-002/

​​Build an observability solution using managed AWS services and the OpenTelemetry standard:

https://aws.amazon.com/blogs/mt/build-an-observability-solution-using-managed-aws-services-and-the-opentelemetry-standard/

We centralized the metrics, traces, and logs collected from workloads running in various AWS accounts using:
▫️ ADOT (AWS Distro for OpenTelemetry)
▫️ Amazon Managed Grafana
▫️ Amazon Managed Service for Prometheus
▫️ Amazon OpenSearch Service.
To visualize these metrics, traces, logs, and to show correlation, we setup:
▫️ OpenSearch dashboard
▫️ Grafana workspace with Amazon Managed Grafana.
This provided us with a native integration with Amazon Managed Service for Prometheus.
We also leveraged a hub-and-spoke architecture for solution scalability.

#observability

​​Using Amazon Cognito to Authenticate Players for a Game Backend Service:

https://aws.amazon.com/blogs/gametech/using-amazon-cognito-to-authenticate-players-for-a-game-backend-service/

A: Game client make REST API call to unauthenticated endpoint to invoke Login Lambda function with username and password in JSON body.
B: Login Lambda function uses username and password to authenticate with Amazon Cognito user pool and obtains IdToken.
C: Login Lambda function sends IdToken back to game client through the API Gateway.
D: Game client makes a REST API call to Amazon API Gateway which will validate the IdToken with the Cognito authorizer. API Gateway will then invoke the backend service Lambda function.

#Cognito

New AWS white paper with guidance for implementing WAF

#aws #security

​​Weekly Summary on AWS (January 16-22)

🔸 CloudWatch Application Insights + Microsoft AD & SharePoint
🔸 DRS (AWS Elastic Disaster Recovery) + failback
🔸 DynamoDB + ReturnConsumedCapacity
🔸 FinSpace + Spark 3.1.2
🔸 FSx for NetApp ONTAP + CloudWatch
🔸 EMR
• Spark SQL to insert data into and update Glue Data Catalog
• Iceberg
🔸 GuardDuty + detection of EC2 credentials used from another AWS account
🔸 Location Service
• Matrix routing
• Request-based pricing for all customer use cases
🔸 Migration Hub Strategy Recommendations + Babelfish for Aurora PostgreSQL
🔸 MQ + RabbitMQ 3.8.26
🔸 Security Hub
• Trusted Advisor adds 111 checks
• Integration with AWS Health
🔸 SSM Automation
• Execute runbooks from Slack
• Outbound webhooks for 3-d party tools
🔸 SageMaker Pipelines
• EMR integration for large scale data processing
• Parallelism Configuration
🔸 AWS Client VPN + banner text and maximum session duration
🔹 Wavelength Zones + Charlotte, Detroit, Los Angeles and Minneapolis

#AWS_week