AWS notes for self reference

!! Disclamer this info may be obsolete or redundant in the future.

IAM (Identity and Access Management)

IAM is eventually consistent
Federating existing users
- If users have identities in a corporate network: if directory is compatible with SAML, directory can be configured for SSO with the console; if not, we can write an identity broker app to provide SSO with the console
- If users have web identities: use cognito for federation
Federated users do not have permanent identities in IAM; role are created for this; when a federated user signs into AWS, it is assigned a role which defines the things it can do
Principal defines who is granted the permissions in a policy; it can be * (wild card for all), a user or a resource (using ARNs) there will be a global entrypoint; we can also specify the region in the sign in url
Cloudtrail logs for signin: if we signin directly to a console, the logs appear in the regional/global endpoint. E.g. the console homepage support regions so the Cloudtrail logs will appear in a specific region; however if we use S3,
We cannot switch role if signed in as an AWS root account holder
Switching roles using the console only with accounts which do not require an external ID. When working with a third party AWS account, we need an ExternalID and switching role is possible only through the SDK and the CLI using aws sts assume-role
Use the "Access Advisor" to identify what resources are actually accessed by a Role, User, Group
Can rename an user only using the API/CLI
If the MFA device stops working
- For an IAM user, contact admin to deactivate MFA
- For the root account, contact AWS to secure temporary credentials
Enable MFA on API calls
- For MFA protection, must add MFA conditions to policies. If the policy doesn't include MFA conditions, the policy does not enforce the use of MFA. For cross account delegation, the same is true for the role's trust policy which must contain the MFA conditions
- Perform these tasks
  - Enable MFA device for the user
  - Create policy which check whether user authenticated using MFA
  - User calls AssumeRole or GetSessionToken passing the device identifier and one time password generated by the device
  - If valid, AWS returns temporary credentials
  - User uses the temp credentials to access AWS services which supports temp credentials (not all services do so)
- Choosing between GetSessionToken and AssumeRole
  - Credential returned by AssumeRole is used to call APIs that access resources in the same or other account.
  - Credential returned by GetSessionToken is used to call APIs that access resources in the same account as the IAM user who makes the request. The temporary credentials returned by AssumeRole do not include MFA information in the context, so we cannot check individual API calls for MFA. This is why we must use GetSessionToken to restrict access to resources protected by resource based policies
- Cannot use MFA-API with root account
- GetFederationToken, AssumeRole With SAML and AssumeRole With WebIdentity does not support MFA
- AssumeRole and GetSessionToken can be called without MFA, but the session info for the temp credentials doesn't indicate that the user authenticated with MFA
We can upload a certificate to IAM
If a user/service is assigned a role, access keys are dynamically created and provided to the user/service
we cannot specify * in Principal of a role's trust policy
Federation is creating a trust between an external identity and AWS.
A principal is an entity in AWS that can perform actions and access resources. A principal can be an AWS account (root user), an IAM user or a role.
Resource based policies: With some AWS services, we can attach a policy directly to a resource (instead of using a role as a proxy). we can use them to grant Principals in other AWS account access to the resource
When setting up cross account access with 3rd parties, external ID is the ID that the 3rd party uses to uniquely associate us with our role. The external ID is specified in the trusting account's trust policy as a condition
Identity federation types
- Federating users of mobile/web apps using Cognito (with web identity federators)
- Federating users directly with a web identity federator
- Federating users using SAML 2.0
- Federating users using an identity broker application. This broker app authenticates users, requests temporary credentials from users from AWS which the users use to access AWS resources. The app calls GetFederationToken or AssumeRole to obtain temporary credentials, depending on how to manage the policies for users and when the credentials expire.
  - The broker app should have permission to request STS for creating temporary credentials
  - Cognito itself is an identity broker which does the federation
Web identity federation with Cognito
- App is registered with IDP
- Use cognito to define the IAM roles that the app needs
- Users signin via the IDP and the app gets an oauth token
- App can exchange the auth token with a cognito token asking cognito
- App can exchange the cognito token with the access credentials asking STS
- App uses the credentials to access the AWS services
Web identity federation without cognito
- App is registered with IDP to get a unique ID
- If the IDP is compatible with OIDC (open id connect), then create an identity provider entity in IAM. Cognito and Google are built in, so this step is not needed for them
- Create one or more roles for each IDP the app supports
  - Who can assume the role (trust policy); in trust policy assign the IDP name as Principal and include a condition that matches the IDP assigned app ID
  - What permissions will the app's users have (permission policy)
- Authenticate users via the IDP and the app gets an oauth token
- Call AssumeRoleWithWebIdentity passing the token and the ARN to role created for the IDP; apps gets temporary access credentials
- App uses the credentials; cache credentials for up to 1 hour
When creating policies for roles for web identity federation, users can be identified as ${www.amazon.com:user_id}. This can be used in the policies (with a version)
SAML based federation
- In organization's IDP, register AWS as a service provider by using the SAML metadata document available at HTTPs://signin.aws.amazon.com/static/saml-metadata.xml
  - Using organization's IDP, generate an equivalent metadata xml file that can describe our IDP as an Identity provider to AWS. Along with other details, it must contain the keys that AWS can use to validate authentication responses (assertions) from our organizations
- Create SAML identity provider in IAM and upload the metadata
- In IAM create one or more roles with the trust policy containing the SAML provider as the Principal and other permissions in the permission policy
- In organization's IDP, define assertions that maps users or groups in our organization to IAM roles. If IDP enables SSO to the AWS console, can configure the maximum duration of the console sessions
  - The AWS implementation of SAML 2.0 federation does not support encrypted SAML assertions between the identity provider and AWS. However the traffic between the 2 can be over an encrypted channel
- Client app requests the IDP to authenticate user. IDP returns a SAML assertion
- Client app call AssumeRoleWithSAML to STS, passing the role ARN, ARN of the SAML identity and the SAML assertion
- STS returns temporary credentials
- Use ${saml:namequalifier}/${saml:sub}
SAML 2.0 federated users to access AWS console
- User browses to organization's portal provided by the IDP which handles the exchange of trust between the organization and the AWS
- The portal verifies the user identity using the AD
- The portal generates an SAML auth response containing the assertions to identify the user and include user attributes. Can configure IDP to include SessionDuration attribute in the SAML assertion to define how long the console session should last
- The client browser is redirected to AWS SSO endpoint and posts the SAML assertion to STS
- The endpoint require temporary credentials for the user and creates a console signin url that uses those credentials
- AWS sends the sign-url back to client browser as a redirect
- The client browser is redirected to the AWS console. if the assertion maps to multiple roles, the user must first select the role
Using external id is beneficial for cross account 3rd party setup because if somehow someone has access to trusted account role ARN (to trusting account), they cannot assume the role without the external ID
For service roles, Principal contains one or more names of the service which assume that role
For a user to pass a role to an AWS service, e.g. starting EC2 with a specific role to do something, the user must have iam:PassRole, iam:GetRole access
Resource based policies apply the policy directly on the resource instead of using a role as a proxy
- So the user doesn't need to assume a role and give up current privileges, however not all AWS services provide resource based policies
- Eg. Account B has created a resource based role giving access to a specific S3 bucket with the principal as account A. Now admin of account A can create Role to access the S3 bucket in account B using the account B Role arn, for certain users, Groups in account A. If account B, given * access, admin in account A can limit (by cannot increase, in case account B gave read-only access) the permissions in the account A Role
STS is a global service (us-east-1) with a single endpoint HTTPs://sts.amazonaws.com; though we can disable certain regions and use a region specific endpoint
For a user-based policy, the Principal isn't explicitly specified. The Principal is the user to which the policy is assigned
For resource based policy, the Principal should be explicitly specified; who can access the resource e.g. the root account
IAM policies can be inline or managed
- An inline policy is a policy that's embedded in a principal entity; i.e. a policy is an inherent part of the principal entity
Resource based policies are associated with the resource S3, SNS, SQS, Lambda, Glacier, VPC, KMS, OpsWorks, ElasticSearch, IOT and aren't managed
When Cloudtrail is used to log events in each separate region into different trails, the global events (such as IAM user creation) is added to each trail. When all the trails are aggregated it will contain duplicate global events. This can be prevented by using a separate trail for global events and disable global events in all other trails
Advantage of federated users over IAM users
- Access keys are automatically rotated
- All credentials can be managed in a single place

S3 (Simple Storage service)

No need to change API when changing from standard to standard IA
Lifecycle transitions
- Archives to standard IA or glacier after specified time
- Expires or deletes objects after a specified time
Can directly put objects to Standard IA
Use versioning and lifecycle policies to implement recycle bin; i.e. delete non-current versions after a specified amount of time
Parallelize GETs using range-based GETs
Parallelize LIST when we need a sequential list of our keys
For timeouts, use connection pooling and perform keep-alive to avoid handshake

Cloudfront

Origin server stores the original, definitive version of the object
Objects can be anything that can be served over HTTP or a supported version of Adobe RTMP, the protocol used by the Adobe Flash Media Server
Each object stays in the edge location for 24 hours before it expires; min 0 secs; no upper bound
Objects are cached in the edge location only when accessed
Can configure a header in the responses from the origin to exclude some responses from being cached
To serve Adobe flash multimedia content, create a RTMP distribution which must use a S3 bucket as the origin
Can create an alternate domain name for subdomain using a wildcard; must begin with *. e.g. *.example.com
Cannot have duplicate and overlapping non-wildcard domain names; can have overlaps between wildcard and non-wildcard domain names
When the S3 bucket is moved to a different region, to force a faster update, change any cloudfront attribute and save
For the bucket to work with Cloudfront, the name must conform to DNS naming convention
When the bucket is not configured as a website endpoint (.S3.amazonaws.com), we can
- Configure CF to use SSL
- Use a origin access identity to require that users access the content only using CF and not via S3
- Update the content using POST and PUT request to CF
RTMP distributions and private content aren't supported for custom origins
For custom origins
- Log the X-Amz-Cf-Id header entries on all servers; CF requires this information for debugging
Can associate a WAF ACL with the distribution
Can use CF Geo Restriction to whitelist or blacklist users from specified countries
RTMP distributions
- Media files must be served as RTMP dist stored in a S3 bucket; the media player must also be provided, which can be in a Web distribution, S3 or custom origin
- CF uses Adobe Flash Media Server 3.5 as a streaming server and streams the media files using RTMP protocol
- To play the media file, must configure the media player with the correct path to the file; this path can be the path to the RTMP distribution
- Adobe media server uses crossdomain.xml to define which domains can access the content; CF supplies a default version which allows all domains and cannot be overridden
To specify how long CF caches objects, configure the origin to add a Cache-control max-age directive to the objects and set an appropriate age value
Caching behavior
- Based on query string params
  - Actions
    - Don't forward query string to origin; CF does not cache based on query string params
    - Forward all query strings; cache based on all params;
    - Forward specific query strings; cache based on specific params
  - All params and their values represents a single object can causes CF to reach the origin even if the same objects are returned for some configs of params and values
  - For RTMP distribution we cannot cache based on query params because query params are removed when reaching the origin
  - S3 and some HTTP servers don't return different object based of params; so disable query params for these cases
  - Best practices
    - Cache based on only those params for which origin returns different versions of the object
    - Always list params in the order and use the same case for param name and values
    - Don't use param names that conflict with signed urls; signed urls have their own params which takes precedence
- Based on cookies
  - Forward cookies to origin and cache based on cookie values
  - Not possible for RTML distribution
  - S3 and some HTTP servers do not process cookies
  - Actions
    - Forward all cookies to the origin; when the origin returns a response, CF caches the response and the cookies and cookie values in the viewer request. (if origin sends more cookies, no caching based on the new cookies). CF returns the object and the all cookies
    - Forward a whitelist of specified cookies; CF removes unlisted cookies and sends to origin and caches based on the values of whitelisted cookies only even if the origin returns new cookies. CF returns the object with the whitelisted cookies and new cookies returns by the origin
    - Don't forward cookies to origin; doesn't cache objects based on cookie values; CF removes the cookies header and removes the Set-cookie header from responses
- Based on request headers
  - Cache separate version of a object based on request header values; disabled by default
  - Not possible for RTMP distribution
  - Actions
    - Forward only the default headers; CF doesn't cache object based on the values of the header
    - Forward all headers to origin; CF doesn't cache the object associated with this cache behavior; instead it sends all requests to the origin
    - Forward a whitelist of specified headers; CF caches based on the values of the headers; however it also forwards the headers that it forwards by default
A viewer request can contain Accept-Encoding: gzip to make Cloudfront compress files of certain types. The origin can compress files not supported by CF. If the origin compresses the file. CF would know about it based on the Content-Encoding and not compress it further
When a user requests the root URL, CF can return a pre-configured root object such as index.html
CF supports "Range GETs" for larger files to improve the efficiency of partial downloads and recovery from partially failed transfers
- S3 supports Range GETs along with some HTTP server like Apache, IIS
- If the origin doesn't support Range GET, CF fetches the complete object and then manages the range gets itself
- Can return custom page for various error conditions and status code like 4xx range and 5xx range
- Along with the custom page CF can also return a different status code (to the one returned by origin) to the client
- By default when the origin return HTTP 4xx or 5xx, CF caches the response for 5 mins. This can be changed using the "Error caching minimum TTL" value
Instead of updating existing objects with the same name, use versioned names so that CF can pickup the new changes quicker
Can adjust the TTL for objects using CacheControl
Object can be manually invalidated at the edge locations to make CF pick up a new version from the origin before the TTL expires
Custom header
- CF can include custom headers when it forwards a request to the origin
- Custom headers can be used to identify cDNS when more than one is used against the same origin

Import-Export

256-bit encryption
Doesn't write data directly to snowball; Data is written via the snowball client which encrypts the data before it reaches the appliance
Keys are managed by KMS and never sent to the snowball
Using the snowball client (without required the appliance) we can test how quickly the upload will be
Can setup notifications via the console to keep track of progress

Storage Gateway

Service connecting an on-premise software appliance with cloud based storage
Appliance installed on-premise; appear as a new drive on windows and a filesystem in linux
Use cases
- Backup and archive
- Disaster recovery
- Data migration or mirroring
- Replace or expand on-premise storage
In on-premise, an application server connects to an AWS storage gateway appliance which then connects to AWS storage gateway backend over internet (HTTP(s)) or DX. The backend integrates with S3, Glacier and EBS snapshots
- For writes
  - App writes data to Gateway using the iSCSI block protocol
  - Gateway stores blocks locally and async compresses and securely uploads the changed data
  - Backend takes compressed data and stores in S3, Glacier or EBS with AES 256 encryption
- For reads
  - App reads data from Gateway using the iSCSI block protocol
  - Gateway returns requested data from local storage if found
  - Data not in local storage is requested from backend
  - Backend fetches compressed data from S3, Glacier, EBS
  - Gateway receives data, decompresses it, stores locally and send to application
Configurations
- Gateway stored volumes (ISCSI block storage)
  - Low latency for all data with point-in-time backups in AWS
  - Primary data is on-premise
  - Async backups to AWS
- Gateway cached volumes (ISCSI block storage)
  - Low latency for frequently accessed data with all data in AWS
  - Primary data on AWS
  - Frequently accessed data cached on-premise
  - Point in time backups stored as EBS snapshots
- Gateway virtual tape library (ISCSI virtual tape storage)
  - Replacement for on-premises physical tape infra for backup and archive
  - Virtual tapes stored in AWS
  - Frequently accessed data cached on premise
  - Unlimited number of tapes in virtual tape shelf
Can use Cloudwatch alarms for monitoring disk spaces on the storage volumes
Can create a snapshot of a storage volume and create a new volume based on the snapshot
Can take a snapshot of an EBS volume and bring it to a gateway; can reuse snapshots of one volume type with another; snapshots are interchangeable between all 3 formats
Use system resource check on the VM to check virtual system resources available to the gateway; gateway will not start if minimum resources aren't provisioned
27 CW metrics for gateways, volumes and tapes
Changed data stored both in cache and upload buffer; writes are throttled as the upload buffer gets close to full; use metrics UploadBufferPercentUsed and CachePercentDirty
Before creating the storage volume, must create the volumes for upload buffer and cache
TimeSinceLastRecoveryPoint can be used to understand RPO
Storage gateway is integrated with CloudTrail

SNS (Simple Notification service)

Pub-sub based message service
Supported consumers
- HTTP
- Email
- Mobile Push
- SMS
- SQS queue
- Lambda
Mobile SDK can send message to SNS
Use direct publish to send notification to single endpoint via publish(msg, endpointARN); available only from mobile endpoints/apps
For multiple destinations, use topics to which one of more consumers subscribe
SNS sends out the message at fast rates, however to throttle the requests and control when the notifications are sent out use an SQS queue
For each mobile push, each device has a global endpoint ARN which are notified using Apple/Android mobile push. the workflow can be
- User install the app
- User authenticates using amazon cognito to get temporary account credentials
- User subscribes
Using SNS as a messaging bus
- Decouple our architecture using SNS
- Use SQS for reliable storage of messages for immediate or delayed processing

SQS (Simple Queue Service)

Messages can contain 256 KB of Text in any format
Messages larger than 256 KB can use the sdk to use S3 for storing the larger payloads
- Specify whether message are always stored on S3 or only when size exceeds 256KB
- Send a message that refers a single S3 bucket object
- Get/Delete message object from a S3 bucket
SQS locks the message during processing, keeping from other parts of the system from processing the same message continuously
Each queue can have different configurations
A delay queue can be used such that delivery of all messages enqueued will be postponed for that duration of time
- Use DelaySeconds attribute; from 0 to 15 mins
- Different from VT (Visibility Timeout), because message is hidden when added to queue; with VT message is hidden when read from the queue
- For delay on individual message rather than the whole queue, use message timers
- If message sent with DelaySeconds to 45, it will be visible after 45 secs; message timers
By default, the queue retains a message for 4 days, can configure to retain the message for 14 days
When using short polling, SQS samples a subset of the redundant SQS servers (based on a random distribution) and returns messages from only those servers. so all messages may not be returned
Each queue must have a name which must be unique across all queues in our account. SQS creates a queue URL to be used when interacting with the queue. This is based on the AWS account no and the queue name
Each message has a unique ID generated by SQS. To delete the message or change the message visibility, we need the message's receipt handle instead of the id
Every time we fetch a message, SQS returns a receipt handle. This is used to delete the message. so to delete the message we always need to fetch the message first. Each time a message is fetched, a different receipt handle is sent
Message should be explicitly deleted from the queue after processing. max of 120000 inflight messages. message shows up in the queue if the message is not deleted within the visibility period
Default VT is 30 seconds
If queue contains message which takes varying times to process, create multiple queues with diff VTs. From the central queue, forward the message to the right queue based on processing times
Other the queue's VT, a message's VT can also be changed during processing. To terminate message's VT, set VT to 0
A dead letter queue is a queue which other queues can target for message that cannot be processed successfully. Can isolate the message in the dead letter to find out why the processing failed
- Should reside in the same region as other queues
- The redrive policy redirect message to the dead letter queue after the source queue fails to process the message a specified no of times
- Using "Maximum receives" setting, message can be put in the dead letter queue. This defines the no of times the message can be received before being sent to the dead letter queue
- Viewing the message in the console count to the maximum receives setting and if matches, the message can be sent to the dead letter queue
Each message can have up to 10 metadata attributes with Name, type and Value fields. sent along with the message body
- Type: string, number, binary or custom (Number.byte, Number.int ...) The part after the . is the custom option
Long polling allows the SQS service to wait until a message is available in the queue before sending a response
- Can avoid false empty response, sent by short polling; long polling queries all servers instead of a random subset
- WaitTimeSeconds parameter; max value 20 secs
- Billed the same as short polling
Can integrate with JMS
If we need to extend the VT for longer than 12 hours, consider using SWF
SQS supports regional endpoints and 2 queue with the same name in different region is completely independent
Can share queue with other based on access control policies
- Can provide access to anonymous users; set Principal to *
Can send/delete a batch of message using SendMessageBatch; However the total size of all messages in the batch cannot exceed 256KB
- Perform client side buffering using the SDK
Each EC2 instance consumer should be running many threads to improve throughput

VPC (Virtual Private Cloud)

NAT (Network address translation) maps multiple private IP addresses to a single public IP address
By default, an instance in a non-default VPC is not assigned a public IP address
NACLs control Inbound and Outbound traffic for subnet(s)
A VPN connection connecting AWS with on-premise consists of VGW (virtual gateway) on the AWS side and A customer gateway (CW) on the customer site over an IPsec VPN tunnel. A VPN only subnet can be created with appropriate route table entries where instances in this subnet talk to instances on-premise via the VPN connection.
If an instance in a public subnet needs to communicate with the internet, it needs a public IP address or an elastic IP address
CIDR blocks
- A CIDR block of a subnet can be the same as the VPC's. In that case, only one subnet is allowed
- CIDR blocks of subnets cannot overlap
- 10.0.0.0 - network address
- 10.0.0.1 - reserved by AWS for the VPC router
- 10.0.0.2 - DNS server
- 10.0.0.3 - reserved by AWS for future use
- 10.0.0.255 - Network broadcast address, not supported now
Each subnet must be associated with a route table; by default it is associated with the main route table, which can be changed
Each subnet must be associated with NACL; by default, it is associated with the VPCs NACL, which can be changed
If a VPC's IP address prefix overlaps with one of the on-premise network prefixes, any traffic to the network's prefix from the VPC is dropped
SGs are stateful - responses to allowed inbound traffic are allowed to flow outbound regardless of outbound rules and vice versa
The default NACL, by default allows all inbound/outbound traffic
A custom NACL, by default denies all inbound/outbound traffic
NACLs are stateless - responses to allowed inbound traffic are subject to the rules of outbound traffic and vice versa
Flow logs
- Feature that enables we to capture the IP traffic going to and from network interfaces in our VPC
- Stored using Cloudwatch logs
- Can create for a VPC, subnet or a network interface and the type of traffic, e.g. accepted traffic, rejected traffic or all traffic. If setup for a VPC, all network interfaces within the VPC is monitored
- Do not capture real-time log streams for the interfaces
- Can create flow logs for network interfaces created by AWS services like ELB, RDS, ElasticCache using the EC2 console or the EC2 API
- Limitations
  - Cannot create for a peered VPC unless the peered VPC is in the same account
  - Cannot tag a flow log
  - After a flow log is created, its configuration cannot be changed
An instance can have multiple secondary addresses which can assigned and reassigned; the primary IP addr is fixed and unassigned only when the instance is terminated
EIPs (Elastic IPs) are charged when they are not associated with any instance, or attached with stopped instance or unattached network interface; or when additional EIPs are associated with an instance
Route table
- Every Route table contains a local route to enable communication within the VPC which cannot be modified or deleted
A NAT gateway supports TCP, UDP and ICMP protocols; cannot assign a SG to a NAT gateway
- Cannot send traffic over VPC endpoints, VPN connections, AWS Direct connect or VPC peering connections
DHCP options set
- Used to assign a domain name to EC2 instances
- By default, instances get a ec2-* DNS name; this can be changed by using domain-name-servers (up to 4) and domain-name attributes of the DHCP option set
- By default, the domain NS if AmazonProvidedDNS which is running in the +2 VPC address
VPC endpoints
- This allows to create a private connection between the VPC and another AWS service without requiring access over the internet, through a NAT device, a VPN connection or Direct connect
- Supported with the same region; currently only for S3
- Highly scalable, redundant, HA; not bandwidth constraints
- Once endpoint created for S3, must be added to the subnet route table
- Cannot have 2 endpoints for the same service in a single route table
VGW provides 2 VPN endpoints for automatic failover
For dedicated VPC, cannot run EC2 instance in default or shared mode
For default VPC, can run EC2 instances in dedicated or single tenant mode
- EBS volume doesn't work on single tenant dedicated hardware
VPN
- Can create multiple customer gateways pointing (say from different customer locations) to the same VGW
- Routing options
  - Static routing: the VPC route table must explicitly know where to route traffic; should perform live checks for failovers
  - Dynamic routing: for devices which support BGP, device advertises the routes using BGP to the VGW; this is better for failover scenarios when one VPN tunnel goes down, can advertise route for a working tunnel
- VGW provides 2 VPN endpoints/tunnels using unique VGW IP addresses; can create 2 cgw utilizing both endpoints for failover scenarios

VPC peering

AWS uses existing infra to create a peering connection; it is neither a gateway or a VPN connection and does not rely on a single piece of physical hardware; there is no single point of failure
CIDR blocks cannot overlap
Cannot create across regions
Placement groups can span peered VPC's, but we do not get full bisection bandwidth between instances
Update route tables in both VPC's to enable flow of traffic using private ip addresses
May need to change SG's
By default, if an instance in a peered vpc addresses an instance in our vpc using public DNS hostname, the hostname resolves to instance's public ip address; to resolve to private ip address, enable DNS resolution in the peering connection
Edge to edge routing via a VPN connection/direct connect isn't supported
EC2 auto recovery
- Applies only to system status checks
- Limited to c3, c4, m3, r3, and t2 instance types
- Cannot use local instance store
- Cannot be dedicated instances
- Use EC2ActionAccess IAM role
VPC S3 endpoint is made up of prefix list which are static but the IP addresses behind the prefix lists could change
- Can use prefix lists in security groups
- Policies
  - Add policy on the endpoint controlling what bucket to access and how to access
  - Can add policy on the S3 bucket itself controlling what endpoint and VPC the request is coming from
- Only one route per route table to an endpoint
- Endpoints and bucket must be in the same region
- DNS enabled on the vpc for the prefix list resolution
- Source IPs to S3 will be private

Direct connect

AWS VPN features
- Static or dynamic (BGP)
- Static requires routes (IP prefixes/cidr block) to be specified
- Dynamic VPN supports max-prefixes of 100
AWS VPN requirements
- Connections are initiated from the VGW
- Security associations using a pre-shared key
- IPsec security assoc in tunnel mode
- AES 128-bit encryption
- Fragment IP packets before encryption
On the AWS VGW side, 2 endpoints in 2 different AZs are provided; tunnel is the VPN connection using the VGW endpoint
- 1 unique SA pair per tunnel, 1 inbound and 1 outbound
- 2 unique pair for 2 endpoints/tunnel: so 4 SAs
- Instead on having one CGW, can have 2 CGW using the same VGW i.e 4 tunnels, 2 for each CGW (resilient dynamic VPN)
BGP is a TCP based protocol on port 179
- The client announces new IP prefixes to VGW over BGP; AWS dynamically updates the route tables with the IP prefixes by enabling "route propagation"
Only one VGW can be attached to a VPC at one time

Direct connect

Multiple AWS accounts can share a connection
Inter region (available in US) allow to connect to one region using direct connect and then access other regions via that region
Uses BGP to exchange routing information over a VLAN
Can use Direct connect partner to handle the connectivity from the Direct Connect location to our network
1 GB or 10 GB connectivity
Private virtual interfaces are used to access the VPC but no access to VPC S3 endpoints or transitive VPC peering; multiple private VIFs to a VGW can be added for resilience
Public virtual interface are used to access non-vpc services like Dynamodb
Use dual DX for hardware resiliency
we can have a hardware VPN connection over DX public VIF

EBS (Elastic Block Storage)

Block storage as a service
EBS vols attaches to any EC2 instance in the same AZ
Definitions
- IOPS - input/output operations per sec (number)
- Throughput - R/W rate to storage (MB/s)
- Latency - delay between request and completion (ms)
- Capacity - volume of data that can be stored (GB)
- Block Size - size of each i/0 (KB)
Prewarming not needed for newly created volumes
Volumes restored from snapshots
- EBS implements lazy loading from S3 so we can use the vol right away
- In some cases, may want to read the entire vol to accelerate loading of data from S3 and reduce latency
Volume types
- GP2 ssd
  - IOPS baseline - 100 - 10,000 (3/GB) hence we get higher IOPS for bigger vol sizes
  - IOPS burst - 30 min for 3,000
    - From 0 to 1 TB (3000 iops) the vol has the capability to burst up to 3000 IOPS. For vols greater than 1 TB, no bursts because the baseline IOPS is greater than 3000 IOPS
  - Throughput - up to 160 MB/s
  - Latency - <10 ms
  - Performance consistency - 99%
  - PIOPS is mostly targeted for production workloads; gp2 can be used for dev/test/env
  - Instead of using a 1TB gp2 vol, can use 2 500 GB gp2 vol to double the burst capacity to 6k iops
- IOPS
  - 100 - 20,000 ; diff between gp2 is that for very small amount of vol size can get higher iops; customer provisions the IOPS and they are independent of the size of the volume
  - Throughput - up to 320 MB/s
  - Latency - <10 ms
  - Performance consistency - 99.9%
  - Mission critical workloads and tier 1 databases
- Magnetic
  - IOPS - typically 100
  - Throughput - 40-90 MB/s
  - Latency - 10-40 ms, write 2-10 ms
  - Best for infrequently accessed data
Always use gp2 volumes for root volumes
Migrating to gp2 volumes
- Change vol type during launch
- Use EBS snapshots, the snapshot has no relation to the volume type underneath
- we may be able to resize the file system
- Use gp2 sizing guide
Snapshots are zone independent can be clone into any AZ, can be copied efficiently between regions
Snapshots are more durable than EBS volumes because they are stored in S3 with 11 9's of durability
When the snapshot API returns as "creating", we don't need to wait for the snapshot to be created
EBS optimized instances
- Dedicated network bandwidth allocated to EBS traffic
- Most instance families support the ebs-optimized flag
- Now support up to 4 GB/s
- Available by default on newer instance types
- EC2 *.8xlarge instances support 10 GB/s network
- Max IOPS per node supported is ~48,000 IOPS for 16k
Encryption
- Can encrypt a EBS volume with a master key which gets associated with the volume
- Any snapshot created will use the same key
To turn the instance into a EBS optimized instance, change the instance attribute and reboot
All EBS metrics in CloudWatch are prefixed with Volume

EC2 (Elastic Cloud Compute)

Placement groups (PG)

Instance types which supports enhanced networking
- c3 - compute optimized - HPC, on-demand batch processing, distributed analytics
- r3 - memory optimized - high performance DBs, distributed memory caches, in-memory analytics
- i2 - storage optimized - nosql DBs, scale-out transactional DBs, cluster file systems
PG are instructions to EC2 to provide better networking between instances in the group; EC2 will place the instances in physical proximity to one another to improve network bandwidth and throughput
PGs do not span AZs
Only certain instance types can be launched in a PG (e.g instances with enhanced networking)
Running instances cannot be moved into a PG (terminate and relaunch)
Best practices
- Launch all PG instances at once, more choices for EC2 to find a section of the AZ for the PG. So PGs might not be the best fit for horizontally scalable web services
- Use homogeneous instance types in the PG

Enhanced networking

For instances with EN, packets originating from the eth0 and eth1 interface do not go through the virtualization layers (as the instances without EN do). Instead they have a direct connection with the physical layer using a virtual function
Uses SR-IOV
The AMI needs to setup with the virtual function driver to support EN.
Can use the ethtool to find the driver; vif for instances without EN.
AMI/instance support for SR-IOV
- c3, r3, i3 instance types
- HVM virtualization type
- Specific kernel version: if the kernel is old, update the kernel and reboot the machine
- Specific VF driver
Things needed for EN (the instance side and EC2 side)
- The AMI needs to support it with the right driver etc (instance side)
- The instance type should support EN (instance side)
- The enhanced networking attribute must be enabled for the instance; once this is turned on, cannot be disabled (ec2 side)
For instance store backed instance which cannot be stopped
- Update OS; install driver etc
- Create an AMI off the running instance
- Register AMI as SR-IOV instance explicitly using the register-image command
- Launch new instance with the AMI

EC2 instances deep dive

c4 instances enables P-state and C-state controls

ELB (Elastic Load Balancer)

The SSL and the TLS protocols use an x.509 certificate to authenticate both the client and the backend application
A certificate must be specified when the HTTPS listener is created for the load balancer
ELB uses SSL negotiations configuration also known as security policy to negotiate SSL connections between a client and the load balancer
Enable keep-alive when enabled it allows the load balancer to re-use connections to the backend instance.
If we specify that the HTTPS listener sends requests to the instances on port 80, the LB terminates the requests and the communication from the LB to the instances is not encrypted. if the listener sends requests to the instances on port 443, communication between the ELB and the instances is encrypted. In this case, we can enable authentication on the instances. This ensures that the LB communicates with only those instances whose public key matches the key specified for the ELB
To be sure that the lb closes the connections to the instances, make sure that the value set on our instance for the keep-alive time is greater than the idle timeout set on the ELB
For each request made by the client, the lb opens 2 connections; one with the client and the other with the target; a default idle timeout of 60 seconds is applied on each connection. If neither sends data before the timeout ends, the connection is closed

Application Load Balancer

Supports HTTP and HTTPS
Can use websockets with both HTTP and HTTPS listeners
Can send 128 requests in parallel using one HTTP/2 connection, can't use the server push feature of HTTP/2
Can log access to a S3 bucket logging several request parameters
- The logs contain the TLS version no
Can attach SGs to ELBs to control traffic
ELB uses a least connection algorithm i.e it chooses an instance with the least active connections
- Beware of blackholing traffic. So if an instance return 500 but does it quickly, most request will go to that instance; use health checks
Can trigger an auto scaling event based on some of the ELB metrics; 13 CW metrics provided at 1 min granularity
- HealthyHostCount the count of the no of instances in each AZ; view at the zonal dimension
- Latency measures the time elapsed in seconds after the request leaves the lb until the response is received; debug individual requests using access logs
- SurgeQueue and spillovers
  - Count of the no of requests that could not be sent to back-end instances
  - Queue up to 1024 request per LB node, after which spillover happens and LB returns 503
  - Caused by not being able to open connections to the backend instances
  - Normally a sign of underscaled instances
Access logs can be used to find the slowest requests
Health checks
- Support for tcp and HTTP health checks
- Customize frequency and failure thresholds
- Must return a 200 response
Idle timeouts allow for connections to be closed by the LB when no longer in use; length of time that an idle connection should be kept open; default to 60 but can be set between 1 to 3600 sec
Cross zone LB
- Absorbs impact of DNS caching
- Eliminates imbalances in back-end instance utilization

Route 53

There is no charge for the DNS queries that R53 routes to Cloudfront, EB, ELB or S3
AWS supports DNSSEC for domain registration but does not support DNSSEC for DNS service. DNSSEC is used to prevent DNS spoofing and man in the middle attacks
When a hosted zone is created, AWS automatically creates for NS records and a SOA record for the zone.
NS records identify the name servers that we give to our registrar or our DNS service so that queries are routed to AWS route 53 DNS servers
The 4 NS are called delegation set. Reusable delegation sets can be created where the same delegation set can be used for more than one hosted zones
Can create multiple hosted zones with the same name, but with different NS records; the registrar can refer to only one delegation set
Cannot change the NS associated with an existing hosted zone. Can specify a reusable delegation set when creating the hosted zone
The SOA record set identifies the base DNS information about the domain. The SOA record contains the host (one of the DNS name server) which created the SOA record
Private hosted zone
- To use a VPC in a private hosted zone, enableDnsSupport and enableDnsHostNames must be enabled for the VPC
- Can associate heath checks only with failover record sets
- Can use split view DNS to configure internal and external versions
- Can associate a VPC with multiple private hosted zones with non-overlapping domain names; e.g test.eg.com and eg.com overlaps
- For overlapping private and public hosted zones, say a.com and b.a.com (private), DNS requests from within EC2 instances in a VPC is handled as follows:
  - Priority to private zones; so look for an exact match or for a parent domain. e.g. c.b.a.com goes to b.a.com. If no match is found, the request is sent to the public DNS resolver and handled as a normal DNS query
  - Then a record set is searched in b.a.com. If none is found, the client gets a NXDOMAIN
- The VPC DNS server is always at VPC network range plus 2. e.g. for 10.0.0.0/16 its 10.0.0.2
- Failover routing policy is not available for private zones
- Can't create alias record sets for cloudfront distributions in a private zone
Latency based record sets can be created only for EC2 instances and ELBs
For geolocation based record sets, there needs to be a "default" geolocation record set to handle cases where the IP address cannot be mapped to any location or for queries that come from a location for which there isn't any geo record set. Otherwise R53 returns a "no answer" when the default record set is missing
CName v/s alias records
- CName queries are charged whereas alias records aren't
- Cannot create a CName record at the zone apex
- CName redirects queries for a domain name regardless of the record type. R53 follows the pointer in the alias record only if the record type also matches
- CName record can point to any DNS record; alias record can point to the few in AWS
- CName record is visible in the answer section of a reply from the DNS server, alias records are visible only in the console and via the api
- CName record is followed by a recursive resolver; alias record set is followed only inside R53 which means that both the alias record and the target must exist inside R53
Record types
- Can't create CName record for zone apex
- If we create CName record for a subdomain, can't create any other record for that subdomain
Cannot enable health check for a record set based on Cloudfront distribution
For Beanstalk alias records, if the ELB is down, health checks are tested on any available resources, otherwise the ELB takes care of it if there are more than 1 instance. If there is a single instance, no additional setup is needed
Do not create R53 health checks for EC2 instances behind an ELB. R53 can use the ELB health checks. If the ELB is down, R53 does the health check on any available resources
Can create only one latency record set for each EC2 region
Can't create non-latency record set with the same name and type
For any latency record tagged with a region, this record set will respond to queries within that region irrespective of the latency
For failover based record set, a primary and secondary failover record set must be created
Can't create non-geolocation record set with the same name and type as existing geolocation record set
Traffic flow can be used to create traffic policies for public hosted zones only
No, charges for traffic policies but we incur a charge for policy records
DNS constraints
- Maximum response size: responses sent over UDP is limited to 512 bytes. any remaining bytes are truncated. if the resolver support EDNS0 and advertises this option to R53, up to 4096 bytes is supported
- Authoritative section processing: for successful response; R53 appends the NS record sets for the domain in the authority section of the response. For names that are not found (NXDOMAIN), R53 appends the start of the SOA record set
For Cloudfront alias record set, routing policy should be simple and evaluate target health should be 'no'
R53 can route traffic to a AWS workmail domain only for public hosted zones
R53 can create health checks for
- Endpoint
- Other health checks which monitor either an Endpoint or an alarm
- Cloudwatch Alarm
Health Check
- Some endpoints require that HTTPs requests include the host name in the client_hello message during the SSL handshake. In such cases Enable SNI must be checked on the health check
- To perform the health checks on the endpoint, the firewall/sg settings must be updated to allow inbound access from the R53 health checkers. The list of health checker ips can be obtained from the forum
- For HTTP/HTTPs health checks, R53 health checkers must be able to connect within 4 sec and then receive 200 or 3** status code within 2 seconds
- For TCP connection must happen with 10 sec
- The health checkers report the status to the DNS servers
- If more than 18% of the R53 health checkers around the world report a healthy status, R53 responds to the resource sets with the associated health checks. If 18% or less, R53 ignores the associated record sets
- Explicit health checks are required for isolated EC2 instances acting as web servers. A health for the instances behind an ELB isn't required. It is generally more useful for R53 to perform health checks on the health of the underlying resources
- Health checks gates are applicable at 2 layers
  - If an explicit health check is associated, that is first checked. if it is failing, the record set is ignored at that point. R53 doesn't look at the target health even when enabled
  - If "Evaluate target health" is enabled, R53 checks the health of the target before qualifying the record set
- It is recommended to associate health checks with all non-alias record sets
- "Evaluate target health" is applicable only for alias target record sets
- If "Evaluate target health" is disabled, R53 selects a target even if the associated health check if failing. Hence it is recommended to enable this always.
Failover
- Active-Active: all resources to be available all the time and serves requests; if any becomes unhealthy R53 routes requests to the healthy ones
  - For weighted record set, specify non-0 weights to all record sets; if every health fails, R53 choose one with the highest weight
  - For latency record set; assign health checks to all;
- Active-Passive: only primary resources serves requests; if a primary resource becomes unhealth, remaining primary resources handles the requests; once all primary resources are unhealthy, the secondsary resources are used
  - For weighted record set, specify non-0 for some, and 0 for others
  - Not possible with latency based record set
  - Failover record is recommended with a primary & secondary record set
    - If primary working, use primary
    - If primary fails, use secondary
    - If both fail i.e. secondary has a health check which failed, use primary. If secondary does not have a health check, secondary is always used if primary fails
- Active-Active-Passive and other mixed configs using alias and non-alias record sets
R53 doesn't check the health when the record sets are created and the DNS query is made. Heath check are done only when health checks are created. This process is different from the DNS query resolution. During DNS query resolution, any record set whose associated health check has failed, will not be used to serve requests
- If health checks aren't associated with a record set, R53 will consider the resource associated with the record set to be healthy and continue to route request to it (unless using an alias record, with internal health checks)
R53 health checker metrics HTTP://docs.aws.amazon.com/Route53/latest/DeveloperGuide/metrics_dimensions.html
Tutorials HTTP://docs.aws.amazon.com/Route53/latest/DeveloperGuide/TutorialWeightedFTMR.html
- R53 permits 3 levels of recursion of alias record set and record sets
Recommended bulk domain transfer order to ensure availability
- Export DNS to R53 (zone files)
- Delegate to R53 (name servers)
- Transfer domains to R53
Private DNS
- Can't be used as the authoritative server
- Can't be reached from outside the VPC
  - For this to work a "forwarder" can be put in place which can forward queries from a on-premise resolver to the +2 VPC resolver to the R53 authority and back in the same flow; spin up an EC2 instance and run unbound as the forwarder
  - Use the AWS Active directory as the forwarder; this is the recommended option
  - Recommended architecture
    - Manage all non-ad names in private DNS. The AD gets a NS too. so if the name is not present in the AD NS, it falls through to the private DNS. so benefits include
      - API endpoint in R53
      - Zones can cross regions with private DNS using the AD as non-ad names will fall thru
- Delegation back via NS records doesn't work
- When on premise contains data (record sets) not present in the private DNS; for requests originating in the VPC with missing record sets in the +2 resolver:
  - Loop the query through on premises; private DNS asks the on premise resolve for the data
  - Copy the data into private DNS
  - Setup conditional forwarding in AD

RDS (Relational Database Service)

DB SG controls access to a DB instance not in a VPC, VPC SG controls access to a DB instance in a VPC
DB engine configuration is maintained in parameter groups
When a DB snapshot is restored, the default SG and PG is applied, therefore we must save the original SG and PG used when the snapshot was created; once the instance is restored, change the SG and PG
Can restore a snapshot to a different edition of the DB engine
If client app is caching the DNS data of the DB instances, set a TTL of less than 30 secs, because the underlying IP address of the DB instance can change after a failover
By default, we can have up to 40 DB instances; only 10 sql oracle/server instance with license-included model
There are DB instance classes with burst modes
RDS supports regional endpoint; default endpoint is the us-west-2 oregon region
Multi-AZ deployments for postgres, mysql, postgres and mariadb used AWS tech; use SQL server mirroring for SQL server instances
Synchronous replication of primary to replica in another AZ
Mutli-AZ option can be specified when creating the db. When converting a single-AZ db to a multi-AZ db, RDS first creates a snapshot; this could result in downtime and degraded performance
Failover happens under the following conditions
- AZ outage
- Primary db instance fails
- DB instance's server type is changed
- OS of the DB instance is being patched
- Manual failover using "reboot with failover"
Automated backups can be done during user defined backup windows. In multi-AZ, backups are done from the replica. can do a point-in-time recovery up to 5 mins
- Automated backup retention period is from 1 to 35 days; an outage will occur is period is changed from 0 to non-zero value and vice versa
- Can disable automated backups
- DB snapshots can be copied to in the same region or to a different region; aurora db cluster cannot be copied to a different region
- To retain an automated backup for a longer period, copy it to a manual snapshot
- Can create a copy of the snapshot and encrypt with a kms key; can restore to get an encrypted database instance;
- Cannot copy encrypted snapshots from other regions
- Can share manual snapshots with other AWS accounts
Manual snapshots aren't deleted when the DB instance is deleted, but automated backups are deleted
Read replicas are asynchronously updated and can reduce load requirements
- Can be created in the same region as the primary or in a different region (for MySql and MariaDb)
Maintenance updates can be deferred, applied or scheduled
- For multi-AZ maintenance updated, update standby -> promote standby to primary -> update previous primary to standby
- For mutli-AZ database engine changes, both the primary and the standby are updated at the same time. DB engine for the entire multi-AZ deployment is shutdown
When updating DB engine versions, major version changes has to manually applied (non backward compatible changes)
Modifications to the DB instance can be applied immediately or during a maintenance window
DB instance names can be renamed
Can change the storage types, which could cause some outage
Can increase the allocated storage without any outage; cannot decrease the allocated storage
Option groups are used to specify database specific features that make it easier to manage data and databases
Can use DB reserved instances
Provides enhanced monitoring for certain DB instance types can provide more lower level DB metrics
- Standard monitoring reports only the metrics that the hypervisor can see
- Enhanced monitoring uses on instance agent to report additional metrics
Can use RDS subscriptions and events to be notified of any changes to the database
Can download the database log files
Encryption
- Can enable encrypt only on new database creation
- Encryption cannot be removed
- If master is encrypted, RR*(Read replica) must be encrypted
- Unencrypted snapshots cannot be restored to encrypted DB
- Cannot restore encrypted Mysql to Aurora and vice versa
- Cannot copy encrypted snapshot to another region as KMS key are tied to an AWS region
In Aurora the read replicas acts as the failover targets
Scaling up or down in RDS can be done by changing the db instance type and apply
Can setup auto scaling: cloudwatch metric -> cloudwatch alarm -> SNS -> lambda function to change the database instance size

Elastic Beanstalk

Application: logical collection of beanstalk Components including environments, versions and environment configurations. It is conceptually similar to a folder
Application version refers to a specific, labelled iteration of the application code stored in S3
Environment is a version deployed onto AWS resources. Each environment can run a single application, however many environments can be run at the same time
Environment configuration defines the collection of parameters and configuration about how an env and its associated resource behave. When the config changes, beanstalk applies the changes to existing resources or deletes and creates new ones
Configuration template is used to create unique env configs
Tiers, one env cannot support 2 env tiers because both env require its own ASG but beanstalk supports only 1 ASG per env
- Web server env tier
  - R53, ELB, AZ, ASG, SG, EC2 instances
  - A host manager runs on each EC2 instance responsible for
    - Deploy the app
    - Aggregating events and metrics for retrieval
    - Generating instance level events
    - Monitoring app log files for errors, monitoring the app server
    - Patching instance Components
    - Rotating log files and pushing to S3
- Worker tier (background processing tasks)
  - ELB, ASG, SG, EC2, SQS
  - Each instance has a daemon which can pull message from the SQS queue and submit to the Beanstalk App
Roles
- Service role for Beanstalk to access other AWS services (like Cloudwatch) on our behalf
- Instance profile role applied to instance for e.g. pushing logs to S3
When using HTTPs, by default encryption is terminated at the load balancer and the traffic between the load balancer and EC2 instances are unencrypted
Supported platforms: single/multi/preconfigured Docker, Go, Java SE, Java with Tomcat, .NET with IIS, Node.js, PHP, Python, Ruby
App source bundle requirements
- Have to be a .war or .jar file
- Cannot exceed 512 MB
- Not include a parent folder or top-level directory (subdirectories are fine)
- For worker app with periodic background tasks (include cron.yaml file)
Can create separate env for dev, test and prod use and can deploy any version of the app to any env
To update the platform config, another compatible platform must be available; with the same arch, os name, and platform name
Can swap the CNAME of the current env with another env in the same tier
When logs are requested, the host manager in the EC2 instance uploads the logs to a S3 bucket. These logs are automatically deleted after 15 mins; can configure the instance to upload logs to S3 after they have been rotated locally
"Managed Updates" shows information about upcoming and completed managed platform updates and instance replacement. This lets we configure the instances to the latest platform version in a maintainence window
Under software settings, a proxy server can be configured to serve static assets and reduce load on the application
Beanstalk env can be created using the HTTPs API by passing env parameters and the auth params
A launch now URL can be shared which contains prepopulated env configurations. This is useful when having a common configuration settings for all env
Compose Environments API
- Can be used to group multiple environments within a single project and defining dependencies between env.
- Each env within the project is in a specific folder with the environment manifest yaml defining the env config
- Beanstalk creates dependent environments only after the dependencies are up and running
- EnvironmentLinks can be used to link 2 envs. Say a front end environment manifest has an attribute of EnvironmentLinks defined as WORKQUEUE="worker". Beanstalk look for an env named worker and injects an env var into the frontend env where WORKQUEUE is set to the URL of the SQS queue
When deploying a new version of the application, the env can be configured to use "immutable updates" which means that instead of deploying to an existing instance, new instances will be started. In an immutable update, a new ASG is created and both ASGs can serve traffic, until the health checks pass on the new ASG after which the old ASG is removed.
Deployment methods
- All at once existing instances
- Rolling, existing instances, traffic is served by old and new version; app servers needs to be restarted so instances can go out of service
- Rolling with additional batch, new and existing, when full capacity is needed.
  - Launch a new batch of instances (with the new version) prior to taking any instance out of service; on completion, the additional batch is terminated
- Immutable new instances is a new ASG; traffic served by old and new instances
- Blue/Green; new instances; swap URL
Healthy thresholds can be used to lower the thresholds at which an instance is considered healhty during rolling updates and immutable updates; this is the count of instances which reports healthy
If a health check URL isn't configured, the instance will pass the health check as soon it can accept a TCP connection; if set, the ELB expect a 200 response
Rolling deployment happens in batches, i.e. a batch of instances is deregistered from the ELB and then updated/created and then reattached to the ELB
- If the deployment fails after one or more batches have been successfully deployed, the completed batches run the new version and pending batches run the old ones. we have to manually terminate the old instances. New instances are then deployed with the new versions
- Can specify a delay between batches, which give time for the instance to bootstrap and serve requests
- With health based checks, beanstalk waits for the instances in a batch to pass health checks. health checks can be:
  - Basic: ELB health checks
  - Enhanced health monitoring: along with ELB check, app logs, env's other resources are also monitored
A config file can be created within the .ebextensions folder to define the deployment method using the aws:elasticbeanstalk:command namespace
blue/green deployments require that the database runs outside of the environment
- Clone environment
- Swap urls
Rolling updates happen when env settings (say VPC) are changed which causes the instances to be replaced
Rolling deployments happen when a new version of the app is deployed
Update policies are setup separately for application deployments and configuration updates
Immutable updates
- If fails, new instances bundles logs to S3 before being terminated. These logs lasts for 1 hour compared to the standard logs which survives for 15 mins
- Cannot perform immutable updates including source code changes with resource configuration changes. If source code changes are require resource configuration changes use blue/green deployments
Managed platform updates perform immutable environment updates
An environment can be a load-balancing, autoscaling env or a single instance env
Worker env
- The daemon fetches a message from the queue and POSTs at a predefined URL. If the application responds with a 200 ok to acknowledge that it has recevied and processed, the daemon remove the message from the queue. If different status code is sent, the daemon puts the message back in the queue after the configured ErrorVisibilityTimeout period. If there is no response, the daemon puts in back into the queue after InactivityTimeout period
- Beanstalk supports dead letter queues, Can't be disabled and if queue is created by beanstalk, can set MaxRetries to a max value of 100. If using an external queue, set MaxReceiveCount SQS option to 100
- Create a cron.yaml file for periodic tasks
Environment configuration
- Precedence
  - Settings directly applied to the environment: during env creation via the console
  - Saved configurations: launched using a saved configuration in console
  - Configuration files (.ebextensions)
    - Create files with .config extension
  - Default values
- If multiple configuration options are defined in each layer, the precedence applies. Say setting A is specified during env creation and also specified in configuration files; the setting set during creation takes precedence. If that setting is removed from the console, configuration file settings take precedence; if the option is removed from the configuration file, the default config value is applied
- To make a configuration file change take effect, a new version must be deployed
- Configuration options
  - Before env creation
    - Configuration files in .ebextension directory (in yaml or json)
    - Saved configuration which has been created from the existing running environment (via console or api). This can be stored under ./.elasticbeanstalk/saved_configs/<name>.cfg.yml and then use with the --cfg option with the eb cli tool. Configuration can also be stored in S3. When saved via the console, it is stored in S3. The EB CLI can refer to the saved config by name, if not found locally, the eb S3 bucket is searched.
    - A JSON doc can be directly specified with the EB cli commands
    - Can specify configs in EB cli configs under ./.elasticbeanstalk/config.yml
  - Custom options can be specified in a eb config file using aws:elasticbeanstalk:customoption: namespace
- .ebextensions
  - Has several sections in the config file for doing various operations similar to the cfn-init file, useful for bootstrapping
  - Can have multiple config processed alphabetically
  - option_settings key in the file can be used to define eb config for the env
  - Similar to cfn-init, has other sections like packages, groups, users, commands, services to help in bootstrapping
  - Can use the Resources key to define resources using cfn templates. Options (including custom options) defined in the option_settings key can be retrieved in the cfn templates in the Resources key using Fn::GetOptionSetting intrinsic function
  - Can use cfn functions like Ref. e.g. to get the current regions AWS::Region. .config file can Ref the resources created in another .ebextension config in the Resources section
  - When using the cli + git, eb uses the last commit and ignores any uncommitted changes
- Environment manifest env.yaml defines the environment settings and uses the same format as Saved configurations
For enhanced heath reporting, EB install a health agent in the EC2 instances
Beanstalk sets a default TTL of 60 secs
Use R53 health check on EBS targets
EB automatically tags env with environment name and id
Troubleshooting
- HTTP HEAD request fails: health check not successful

DynamoDB

EC - Eventually consistent SC - Strongly consistent PT - Provisioned throughput

All data stored in SSDs and automatically replicated across multiple AZs in a region
Components: tables, items, attributes; item size limit of 400KB
Attributes can be scalar or nested; supports up to 32 levels of nesting
- Scalar (string, number, boolean, binary, null)
- Document (list, map)
- Set (multiple scalar values of the same type); unique order not preserved
Primary key
- Partition/Hash key: input to a hash function which determines the partition in which the items will be stored
- Partition/Hash and sort/range: partition key determines the physical partition and sort key determines the sort order. Items with the same partition key are stored in the same partition
Secondary indexes
- Global sec idx(index): index with different partition and sort key
- Local sec idx: index with same partition key but different sort key
- Up to 5 global and local idx
- All writes to the base table is automatically copied to the sec idxs
- Global sec idxs are eventually consistent
Streams
- Captures data modifications in near real time and in the order it happened
- If a new item is added, captures the image of the entire item
- If an item is updated, before and after image
- If an item is deleted, before deletion image
- Save for 24 hours, after which they are deleted
- Can trigger events based on streams. e.g. invoke a lambda function to send an email when a new customer is added; also used for data replication across and within regions; data analysis
API HTTP://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.API.html
- BatchWriteItem does not support item updates
Eventually consistent reads by default; can specify ConsistentRead param to get operations (GetItem, Query, Scan) for strongly consistent reads. So the read behavior is not at the table level
Provisioned throughput: Capacity units
- RCU: 1 RCU for 1 SC read/sec; 2 EC read/sec; for items up to 4 KB in size
- WCU: 1 WCU for 1 write/sec; for items up to 1 KB in size. for tables with sec idxs, 1 wcu is needed for each idx
- When througput exceeded, throttled with 400 error with ProvisionedThroughputExceededException
- When querying multiple items, throughput required is based on cumulative data retrieved
- Can purchase reserved rcu/wcu
Mandatory parameters when creating a table
- Tablename; key schema (partition key and sort key if any), attribute definition of the primary key, provisioned throughput
Use ProjectExpression with GetItem to fetch only certain attributes for an item
With Query use KeyConditionExpression to specify conditions on the primary key(s), use FilterExpression to specify conditions on non-pk and ExpressionAttributeValues for the values for the placeholder keys in the other 2 expressions
Unlike SQL database, we can run queries directly on the index
UpdateItem is like an upsert operation which create new items when it does not exist; otherwise update exist item's attributes. Support atomic counters or attributes of type Number which can be incremented or decremented
When an item is inserted, we must specify primary key values of the base table and primary key values of any secondary indexes
When decided the capacity units for the table, the following needs to be considered
- Item size
- Expected read and write request rates
- Consistency (EC or SC)
- Local secondary indexes; these can be created only during table creation; global Secondary index can be created later with separate capacity units. Local Secondary index consume the capacity units set for the table
Update UpdateTable to increase the PT for the table. When the table is being updated, the table status changes from AVAILABLE to UPDATING. But the table remains fully available
Size of an item is the sum of the lengths of its attribute names and values. (reduce attribute name length)
Size of null or bool attribute length is (length of attribute + 1 byte)
Attribute of type list or map requires 3 bytes of overhead, regardless of contents; (length of attr name + sum(lenght of attribute vals) + 3 bytes)
Read operations
- BatchGetItem can return up to 100 items; however each item size is rounded to the next 4 KB
- Query returns multiple items; all items returned are treated as a single read operation wherein dyn computes the total size of all items and rounds to the next 4KB boundary.
- Scan reads all item in a table; dyn considers the size of the items that are evaluated; not the size returned
- Even if a subset of attributes is returned, it has no effect on the size calculations
Write operations
- Putitem add a new item or replaces an existing one with the same pk. item size that matters is the larger of the two, before and after
- Updateitem consume full amount even if a single attribute is changed. item size that matters is the larger of before and after
- Deleteitem consumes the size of item deleted
- Batchwriteitem - writes up to 25 items; processes each item as an individual putitem or deleteitem
- For conditional writes, dyn consumes atleast wcu based on the item which already exists.
ListTables returns a maximum of 100 tables; after that results are paginated
Guidelines for working with tables
- Use a partition key with large no of distinct values so that the items are distributed evently across all partitions
- Request items fairly uniformly and as randomly as possible
- Randomize with partition key suffix; all random suffix from a range when writing data, e.g 1 to 200; while reading data, read when the suffix values, the application then has to merge all results returned for pks with suffix between 1 to 200
- Using the random suffix key, we wouldn't know which partition an item will be written to. In such cases we can use an internal hash function, say based on characters of the input key.
- Allocation
  - A single partition can support a max of 3000 rcu and 1000 wcu and a max of 10GB data
  - Num of partitions = ceiling[rcu/3000 + wcu/1000] and then the PT is equally divided between these partitions
  - Partition split
    - Can occur in response to increased provisioned settings and increased storage requirements.
    - when PT is changed, dyn always "doubles" the no of partitions and data is evenly distributed across the new partitions and each partition's rcu/wcu is recalculated. PT across all partitions is equal
    - When data in a partition gets more than 10 GB, 2 new partitions are added. Data from the other partition is evenly distributed between the 2 new and then the old partition is deleted. Dyn is fully available during this operation. PT of other partitions isn't affected however the PT of the deleted partition is shared between the 2 new partitions. (pt old / 2) so PT across all partitions is unequal
  - As the num of partitions increases, each partition has lower PT; however the overall PT for the table remains the same
- Dynamodb partitions has burst capacity but use that sparingly and don't design our app based on that
- When uploading data with pk having multiple values, instead of loading all items of one PK, then all items of other PKs, load one item of each PK and then continue. this will keep dyn busy and can achieve more throughput performance
- Understand access patterns to TS data; create separate monthly, weekly tables if old data isn't frequently accessed. For HOT data, create a separate table with more PT.
- Deleting the whole table is more efficient then delete each item 1 by 1
- Use a caching solution in front of dyn to cache popular items; the application first looks into the cache
- When reducing the PT for a table, Dyn does not reduce the no of partitions; each partition will now have lower PT; think about this upfront
Working with items
- With GetItem, must specify the entire PK with partition key and sort key (if any)
- Using the ReturnValues attribute, can return the before/after value of the item for an update/put/delete operation; possible values (ALL_OLD/NEW, UPDATED_OLD/NEW)
- Use ReturnConsumedCapacity to know how many capacity units were consumed for the operation; possible values : TOTAL (base + global idx), INDEXES (global idx), NONE
- Batch operations
  - BatchGetItem; up to 16 MB and a max of 100 items; can retrieve items from multiple tables
  - BatchWriteItem; up to 16 MB and max of 25 put or delete requests; cannot update; can write to multiple tables
  - a batch operation doesn't fail unless all operations fail; its returns the failed ops for retry
- Atomic counter used to incr or decr an attribute without interfering with other write ops; performed in the order received; not idempotent
- Use conditional writes to avoid data inconsistency using there are multiple writes; these writes are idempotent;
- When using projection expresion to fetch specific attributes, use document path syntax to fetch specific elements from document type attributes; e.g. MyMap.attribute[50].foo
- When using reserved names for attributes, or names with dots for attributes (dyn allows it), say: 'a.b.c', we cannot use it in a query because dyn interprets this as a document path. Use the #name syntax to indidcate a placeholder.
  - Expression attr names begins with #
  - Expression attr values begins with :
Patterns
- When the size of the item to be stored is greater than 400KB, break the item into chunks with the chunks stored in a different table to improve write throughput
- Store large attribute values in S3
- Compress large attribute values
- Use multiple tables to support varied access patterns; say one table for catalog, another for availability instead of putting everything in one table
- Use 1 to many tables instead of large set attributes
Query and scans
- Use ScanIndexForward to reverse the sort order
- A single query/scan can fetch up to 1 MB of data prior to any filtering based on the FilterExpression. a non-null LastEvaluatedKey in the response can be used to paginate results and use that values as the ExclusiveStartKey in the next request to page through 1 MB increments
- FilterExpression does not allow key attributes; we cannot define a filter expression based on a partition key or a sort key
- Use Limit parameter to restrict the size of the resultset returned
- Dyn calculates the amount of consumed PT based on the item size and not on the total data returned by the application. Hence the capacity units consumed are the same if we fetch one or more attributes in an item
- ScannedCount: total no of item that were queried/scanned before applying any filters; Count: total no of items returned in the response
- A scan operation can only read one partition at a time. use multiple threads to read different Segments of the data using TotalSegment no of worker threads
- Guidelines
  - A single scan or query operation can retrieve 1mb of data, this might result in the no of read operations with bursts of reads thereby consuming all rcus. Instead of having these bursts, reduce the page size using the Limit for the query and scan operation, so the data is fetched in controlled rates avoiding the bursts. If possible, run scans on isolated table specific for the scan operations
  - Use parallel scans
Secondary indexes
- Tables with secondary indexes can be created sequentially only
- Up to 5 global and local secondary indexes
- Attributes can be projected using KEYS_ONLY, INCLUDE and ALL
- Global
  - Called global, because queries on the index can span all data in a table, across all partitions
  - No size restrictions
  - Can be deleted at any time
  - Support EC only
  - Can request only those attributes which are projected onto the index
  - The PK does not have to be one of the existing key attributes of the table; the index (partition and sort if any) keys has to be a top level string, binary or number type
  - Unlike the base table, the key values may not be unique.
  - Only keeps track of data items where the key attributes exists; e.g. if an item is added to the table, but one or key value (in the index) is missing, that item is not added to the idx
  - ValidationException: when the data types of the attributes of the item inserted does not match the data type of the key values in the sec index
  - If an indexed key attribute is changed in the base table, this causes 2 write ops on the index, one to delete and one to create a new one
  - Index creation phases
    - Resource allocation
    - Backfilling - copying items from the base table to the index; dyn does reads on the base table using an internal system not affecting the PT; backfilling on a large table will take longer. monitor these metrics
      - OnlineIndexPercentageProgress; OnlineIndexConsumedWriteCapacity; OnlineIndexThrottleEvents
      - During backfilling we cannot add or delete other indexes in the table
      - During the backfilling phase, there may be index key violations. e.g.
      - Data type mismatch for the pk
      - The max size of the attr value of the partition key is 2048bytes and for the sort key is 1024 bytes. The value of a non-key attribute in the base table which is now a key in the index, if greater the max sizes.
      - Use the violation detector tool which can run in 2 modes
      - Detection : reports the detection
      - Correction: corrects the violations based on set rules
  - Take advantage of sparse indexes to efficiently locate items with uncommon attributes; e.g. Secondary index can contain items with specific attributes only; due to the sparse nature, items with missing attributes won't be written to the dynamodb index
  - Use the Secondary index as an eventually consistent read replica
    - Can be used for an internal monitoring tool
    - Can be used to restrict access to data based on IAM policies
- Local
  - Called local, because every partition of a local sec index is scoped to a table partition that has the same partition key value
  - For each partition key value, the total size of the indxed items must be 10GB or less
  - Supports EC and SC
  - Can request attributes not projected onto the index, dyn will fetch from the base table
  - Data in this index is stored in the same partition key as the table but with a diff sort key
  - The sort key should contain exactly one scalar index
  - The sort key of the source table is projected onto the index as a non-scalar attribute
  - The sort key must be a string, number or binary
  - For tables with local sec idx, there is a 10 GB limit per partition key value
  - The sort key does not need to be unique, unlike the base table; Query will then return multiple items with the same partition & sort keys
  - Also a sparse index like a global sec idx
  - Item collections (only in local idx): items in the source table and all the local sec index which share the same partition key. The following ops can return information about item collections
    - BatchWriteItem
    - DeleteItem
    - PutItem
    - DeleteItem
  - The max size of any item collection is 10 GB. ItemCollectionSizeLimitExceededException
  - The table and index data for each item collection is stored in the same partition
Stream
- Change record near real time only once, in order, for 24 hours
- Organized and grouped in shards; each shard acts as a container for multiple stream records and contains information required for accessing and iterating over these records
- Shards are ephemeral and one shard can be split into multiple shards to achieve higher parallelism
- If we disable a stream in a table, the data in the stream will continue to be readable for 24 hours
- Integrate with kinesis
  - App -> DynamoDB API
  - App -> KCL -> DynamoDb streams adapter -> dynamodb streams low-level API
Auth and Access Control
- Can setup ARN rules on tables, indexes and stream as resources
- Can use conditions on IAM policy for fine grained control on table access; say see certain attributes only, see certain items only
- Use web identity federation
  - Register the app with the auth provider
  - Create a role for the app to access DynamoDb
  - App signs in the user with auth provider to get a token
  - App calls STS to get temporary credentials to assume the role
  - App calls DynamoDb
Do not mix hot and cold data; keep a separate table for cold data per time period and give greater PT to hot table
Use composite keys - concatenate 2 attributes
Right sharding - add random values to key

Cloudwatch

Metrics cannot be deleted, but they expire in 14 days if no new data is published to them
Namespaces are containers for metrics. There isn't a default namespace. Data points should have a associated namespace when put into CloudWatch(CW)
Dimensions are name, value pairs used to categories a metric. More like tags for a metric. Must specify a dimension when creating a metric
CW can aggregate metrics across dimensions for metrics created by AWS services. This is not possible for custom metrics
Up to 10 dimensions for a metric
CW treats each unique combination of dimensions as a separate metric. When fetching the metric, all the dimensions should be specified
Statistics are metric data aggregations over a period of time
CW can be fed pre-aggregated data using StatisticSet which can feed the sum, min, max and sample count to CW
Alarms invoke actions for sustained state changes only. Actions aren't invoked because they are in a particular state, the state must have changed and been maintained for a specified no of periods. Alarm actions should reside in the same region as the alarm
Alarm history is available for 14 days
Cannot aggregate data across regions. Metrics are completely separate between regions
Aggregated statistics are only available if detailed monitoring is enabled
Although we can publish metrics with granular timestamps, cloudwatch aggregates the data to a minimum granularity of 1 min
Alarm actions
- For ASG, the alarm continues to invoke the action for every period that the alarm remains in that state
- For SNS, after the action is invoked once, no additional actions are needed
Some AWS resources do not send metric data to CW under certain situations. Say an EBS volume not attached to any instance
The EC2ActionsAccess IAM role enables AWS to perform alarm actions on our behalf. Such alarm actions can be set of a Amazon EC2-per instance metric with InstanceId dimension
If we are using STS generated temp credentials, we cannot recover an EC2 instance using alarm actions
Alarm actions
- SNS notification
- Auto scaling action
- EC2 action (Recover, Stop, Terminate, Reboot)
  - Reboot action is for instance health checks. rebooting doesn't start a new instance billing hour StatusCheckFailed_Instance
  - Recover action is for system health checks StatusCheckFailed_System
    - c3, c4, m3, m4, r3, t2, and x1 instance types
    - Instances in a VPC with shared tenancy
    - Instances that use EBS volumes
    - Instances retain the public ip address after recovery
Cloudwatch events
Near real time stream of system events that describe changes in AWS resources
Components
- Events: generated by changes in the AWS resource state, Cloudtrail or app generated scheduled events
- Rules: match incoming events and route them to one or more targets for processing; rules aren't processed in any particular order
- Targets: actions; lambda functions, SNS topics, SQS queues, streams, built in targets.
  - A single rule can specify multiple targets, processed in parallel
Need events:* and iam:PassRole in policies
Can create rules and targets in a specific region and then can send events to a region specific endpoint
Scenarios
- Log the state of an EC2 instance (when changed) using a lambda func
- Take schedule EBS snapshots using built-in target
- Log an AWS API call using a lambda func
- Log the state of an ASG using a lambda func
- Relay events (EC2 instance state changes) to a kinesis stream
- Run a lambda function on a schedule
Events are represented in json, and events pattern are a subset of that json (along with the nesting, values etc) if it has to match
Events are posted using PutEvents; can post multiple events as long as the event entry size is less than 256KB
For running the actions, cloudwatch needs to have the right roles
- For lambda, SQS, SNS cloudwatch relies on resource based policies
- For Kinesis stream, cloudwatch relies on IAM roles
BuiltIn targets include: create a snapshot, reboot/terminate/stop an instance
Cloudwatch logs
Terms
- Log Event: record of some activity recorded by an application or resource being monitored
- Log Streams: sequence of log events generated by the same source
- Log Groups: groups of log streams that have the same retention policies, monitoring and access control settings
- Metric Filters: filters used by cloudwatch to extract information from the events to transform into a cloudwatch data metric; metric filters are assigned to a log group and all of the filters assigned to a log group are applied to all the streams within the group
- Retention Settings: how long will cloudwatch keep the logs; expired log events are deleted automatically
An agent is installed in the EC2 instance and an agent configuration defines where to look for the logs and in what format
Filter pattern: what pattern to look in the log; metric name: what metric to relate to; metric namespace: the destination namespace/dimension for the metric; metric value: for each match for the filter pattern, what value to record?
Realtime processing of log data using subscriptions
- Log events from cloudwatch can be fed to other services in real time
- Components
  - Log group name: whose log data will be streamed
  - Filter pattern: to filter specific log events to be streamed
  - Destination arn: arn of the destination resource
  - Role: role required for cloudwatch to post to the destination
Cross account log data sharing using subscriptions
- Cross account subscriptions using lambda isn't supported
- Log data sender: cloudwatch logs sender; needs to know about the destination subscription
- Log data recipient: a destination that encapsulates kinesis stream and lets cw logs know
- Components
  - Destination name
  - Target ARN: kinesis stream ARN
  - Role ARN: role in recipient for cloudwatch logs in sender account to write to
  - Access policy: IAM policy defining who can write to the destination (in sender acc)
Log data can be exported to S3
- Log data can take up to 12 hours to be available for export
Log data can be stream into elasticsearch

CloudTrail

Can deliver events to a log group in cloudwatch logs
Types
- A trail that applies to all regions (default): the same trail is created in all regions. For notifs, one SNS topic is enough for all regions; events for all regions can be sent to a single log group
- A trail that applies to one region: we specify a bucket that receives events only from one region; the bucket can be in any region; individual trails in specific regions can deliver logs to a single bucket
Logs are encrypted using S3 sse
Logs are delivered within 15 mins of the API call
Publishes new log files, usually every 5 mins
Can inform a SNS topic when log files are delivered
A trail is a configuration that enabled logging of AWS API activity and related events in our account
Can prevent API calls from global services like IAM and STS to be logged
Can create multiple trails per region
When using a region specific STS endpoint, trails in that region delivers only the STS events in that region; for most services, events are sent to the region where the event happened.
- For global services IAM, STS and CloudFront events are sent to all trails with "Include global services" option enabled.
- OpsWorks and R53 events are logged only in us-east-1 trail
- Global service events are always delivered to trails that have "Apply trails to all regions" enabled, irrespective of "Include global services" option.
- To avoid duplicate event when using multiple trails, disable "Include global services" in single region trails
Cloudtrail endpoint has the format: cloudtrail..amazonaws.com
Cloudtrail log file has the format: <account_id>CloudTrail.
Can encrypt the S3 trail logs using a KMS key
Can enable trail log validation to know if the log file was updated/modified/deleted
Logging can be disable for a trail, but old logs in S3 will still be accessible
Can create 5 trails per region; a trail that logs activity from all regions counts as 1 trail per region
Cloudwatch logs have an event size limitation of 256KB. Cloudtrail logs exceeding that size will not be sent to cloudwatch; break down large requests into smaller requests
Based on the cloudwatch logs, we can create alarms looking for specific API requests in the cloudwatch logs; e.g checking $.eventName = DeleteSecurityGroup
Cloudtrail logs from multiple AWS accounts can be fed into a single S3 bucket; since the logs are namespaced by account numbers; the IAM policies need to be in place
- Note that the policies allow Cloudtrail service in the other accounts to write to the bucket, not users in the other accounts
By default logs are encrypted with S3 managed encryption keys; this can be overridden by KMS managed encryption keys
Logs are validated using hash keys and digest files (delivered on an hourly basis); the digest files are compared with the actual files; using custom policies we can restrict access to the digest files; custom algorithms can be implemented for log file integrity validations
From the cli, "create trail" and "start logging" are 2 separate actions
Cloudtrail events are available in the console for the last 7 days
we can encrypt log files from multiple accounts using one KMS key
- If log files are encrypted, to read the files we need decrypt permissions as well

Kinesis

Streams
Collect and process large streams of data records in real time using kinesis stream applications
Stream apps run on EC2 instances using the kinesis stream client and reads data off the streams and processes it
Use cases
- Accelerated log and data feed intake and processing
- Real time metrics and reporting
- Real time data analytics
- Complex stream processing
Sub second delay between data put in stream and when it is available to be consumed
Terms
- Kinesis streams - an ordered sequence of data records. Each record in stream has a seq no. The data records in the stream are distributed into shards
- Data record - each record in the stream is composed of a sequence no, partition key and data blob, which is a seq of bytes. Stream does not inspect, interpret or change the data blob in any way. Data blob can be 1Mb is size
- Retention period - length of time data records are available in the stream after being added. default is 24 hrs and can be increased to 7 days
- Shards - uniquely identified group of data records in a stream. A stream has one or more shards each with a fixed capacity. 5 transactions per/sec for reads to a max of data read rate of 2 MB/sec. 1000 rec/sec for writes up to a max of total write rate of 1 MB/sec (including partition keys). The total capacity of a stream is the sum of the capacities of the shards
  - Can increase/decrease the no of shards; pricing based on the no of shards
- A partition key is used to group data records by shards
- Each data record has a uniquely generated partition key
num_of_shards = max(incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB/2000)
To put data into the stream, specify the stream name, partition key and the data blob to be added
- Partition key is choosen based on app logic and decides which shard the data record is written to
Each consumer is known as a kinesis stream applications reading data from a stream
- Each consumer reads from shard using a shard iterator which determines the position in the stream from which the consumer will read
- When the consumer reads, it gets a batch of data records to operate on
- Each consumer must have a unique name that is scoped by the AWS account and the region used by the application
- A shard iterator returned by GetShardIterator times out after 5 minutes if we haven't used it
The no. of partition keys should be greater than the no of shards in most cases; a high no of partition keys relative to shards allow the stream to distribute data records evenly across the shards in our stream
KPL - kinesis producer lib used for writing producers
KCL - kinesis client lib used for writing consumers
Amazon kinesis agent publishes custom metrics to help if the stream is working properly
Streams can be tagged for cost tracking
Support multiple consumers and replayable reads; if a consumer goes down, they can replay the streaming data
Fault tolerance: checkpoint and replay
Ensure a high cardinality for partition keys with respect to shards to prevent a hot shard problem. Generate random partition keys
Non-Random app generated/maintained partition keys can be used for streaming map-reduce. All records with the same partition key ends up in the same shard
- Partition key per billing customer; per device id; per stock symbol
- Consumers can then work on specific shard, knowing its contents
Pre-Batch before small puts to improve efficiency; in SDK use AsyncProducer (SynchronousQueue or ArrayBlockingQueue)
To find hot shards
- Log hashkey values generated by our partition keys
- Log shard ids
- Determine which shard receive the most traffic and then call split shard
KCL can spin up multiple record processors and manages state information in a dynamo db table
PUT consumers in a ASG
Use metrics GetRecords.IteratorAge and GetRecords.Latency to find out why a consumer is running slow
Use checkpoint to avoid duplicate processing of streams. Once checkpointed, we can say that the data has been processed and does not need to be replayed
Firehose
Managed service for delivering real time streaming data to apps such as S3, redshift, elasticsearch
Write producers to send data to firehouse which is then streamed realtime to the destinations
The streamed data to S3 can be compressed, encrypted
Amazon kinesis agent can be used in the producer or use the AWS SDK
For redshift, data is first copied to S3 and then copied to redshift

OpsWorks

Lifecycle events include setup, configure, deploy, undeploy and shutdown
Each layer has recipes for each lifecycle event
If an instance is shared across multiple layers, multiple recipes can be run on the instance for specific layer and their events
Instance types
- 24/7 instances are started manually and run until we stop them
- Time-Based instances are run by OPS at scheduled times
- Load-Based instances are managed by OPS based on metrics. only available for linux based stacks
- Supports autohealing. if an agent stops communicating with Opsworks, OPS stops and restarts the instance
- Can integrate instance created outside of OPS (including onpremise) into OPS by registering the instances with OPS
2 separate roles are used
- Service role for the Opsworks service
- EC2 instance profile role for the instance running in Opsworks
if auto healing is enabled, Opsworks automatically restarts the layer's instances in case of a failure
Workflow
- Create the stack
- Create the layer
- Specify the app
- Add and launch the instance
- Deploy the app on the instance
  - Setup the recipes to be run for the various events
  - Deploy the app
After a restart, instance store backend instances take the same time as they did during the initial start, and performs the same tasks including package installation. The restart for EBS backed volumes are faster because the setup recipes don't run reinstalling packages that are already installed on the root volume
Instance
- Types
  - 24/7 - started manually and run until they are manually stopped
  - Time based instances - automatically started and stopped on a user-specific schedule
  - Load based instances - automatically started and stopped when a metric threshold is crossed
- Recommendation is use to use a mix of all 3 types of instances based on the app behavior
  - Say 24/7 instance to handle the base load
  - Time based instances - for varying loads during the day
  - Load based instances - for the additional spike; only run when the current capacity isn't enough
Each stack has granular permission levels which can be used to set who does what actions
Managing and deploying apps and cookbooks
- Goals
  - All instances of the stack must be running the same code/cookbook unless doing A/B testing
  - Deploying an update should not interrupt the site's operations
- Maintaining consistency
  - Deployment mode
    - When instances are started, they use the latest code
    - For online instances, we must manually deploy the current code or cookbook
  - Use S3 archives
  - Use tagged git branches; instead of just the master branch
- Deploying code to online instances
  - Rolling deployment
    - Has to done manually
    - (De)registration with the ELB also has to done manually
  - Use separate stacks for dev, staging and prod and promote code manually from one stack to another
  - Blue/Green deployment
- All of the above (stoping/starting/deployment) has to done/managed by the user
Can use Berkshelf to package the one or more cookbooks locally using chef's cli and push the zip file to a S3 bucket and use that in the stack
Security updates
- By default Opsworks install sec updates during setup and after the instances finished booting. It does not install updates after the instance is online. user must manage it
- Choices
  - Create and start new instances; delete old ones
  - For chef 11 and older, run Update dependencies command
- Can disable auto update on boot, but the user needs to add scripts to run the update via the recipes
By default, Opsworks create SG's to allow ssh access (all) and TCP, UDP and ICMP access from other Opsworks layers
- If the default SG get deleted, can do the following which makes Opsworks create the default sgs
  - Create a new stack with the same name which creates the def sgs. Then delete the new stack
  - Make changes to stack settings in the console
  - Start a stack instance
The stack default subnet can be overridden when creating the instance
When creating the stack, can choose a specific Opsworks agent or an auto-updated opswork agent; can be overridden when creating the instance
Custom JSON defines attributes that are merged into the stack configuration and deployment attributes that are installed in every machine
- The data is stored in json files in well known locations
- Chef recipes can reference the json attributes using node syntax
Cannot change region or VPC id after a stack is created; can change OS but not from windows to linux and vice versa
Can clone a stack but that does not create any instances; instances have to be manually added
After the stack is created, various commands can be created using "Run commands"
- Update cookbook: update the cookbooks in all the instances (fetch from repo); does not run the recipes
- Execute recipes: run one or more recipes
- Run setup or configure recipes on the instances
- Install/Update dependencies : chef 11 or before
- Upgrade OS
To shutdown a stack, all instances, app and layers must be removed first
Layers
- Each layer in a stack must have atleast one instance
- Each instance in a stack must be a member of atleast one layer except for registered instances
- An instance can be part of multiple layers; recipes for different stages/events run on each layer
- Instance shutdown timeout can be used to perform cleanup tasks before terminating the instance
- Can control whether OW assigns public ip or eip to instances
- Autohealing is set at the layer level
  - If the Opsworks agent fails to send heartbeats, instance is marked as failed
  - An instance can be part of multiple layers; if any of the layers has autohealing disabled, Opsworks does not heal the instance if it fails
  - Once healed, OW runs the lifecycle event on all the instances in the stack
  - If a volume was attached when the instance failed, OW reattaches the volume
  - When enabled, use cli, console or API to stop instances; else the instance is healed
- ELB
  - An ELB must be created outside of OW and then attached a layer; one ELB per layer; one layer per ELB
  - Once attached, any existing ELB instances are deregistered and then layer instances are attached
  - If connection draining is enabled, shutdown lifecycle event is delayed until connections are completely drained
- RDS instance must be previously created and then a new layer should be created
- Starting from chef 12, we must use custom recipes to install packages on layers that are running different operations systems
Instances
- Instance can be created by OW or existing instances (even on-premise) can be registered with OW
- Supported os: amazon linux, centos, ubuntu, redhat
- To enable proxy server support, an instance must have /etc/environment file that provides the settings for HTTP and HTTPs traffic
- Custom ami's are better for load based instances because the instance can boot up quicker
- Chef recipes are more flexible and allow updates on running instances
- OW uses cloud-init to install the OW agent on the instance and the agent runs the instance's setup recipes followed by the deploy recipes. After the instance is online, the agent runs the configure recipe for each instance in the stack, including the newly added instance
- When creating a instance AMI running in OW, first ssh into the instance and disable and remove the OW agent and then create the AMI using cli
- The AMI can be locked to a specific version of the amazon linux using repo_releasever value in user_data
- Time and load based instances distribute, scale and balance applications over multiple availability zones within a region
- Load based scaling starts/stops new instances based on load metrics; auto healing starts a new instance if it existing instances stops responding irrespective of load metrics; load metrics are averaged over all instances
- Load based scaling does not create new instances; it starts and stops only those instances that we have created; hence provision enough load based instances to handle load
- Using "after scaling, ignore metrics" option, we can specify the amount of time in mins that OW should ignore metrics and suppress additional scaling events
- Linux instances created outside of OW can be added to a stack
  - Can run stack commands on these instances
  - Can add instances to an existing layer and becomes part of the lifecycle events
  - Install the agent into the instance (from workstation or sshing into the instance) and register with a stack
  - Provides a way to specify which users can access the instance and what operations they are allowed to perform
  - Every stack can have one or more users and each user can have a ssh key. Based on the permission policies, a user can perform certain actions (show, deploy, manage, deny etc) or ssh into the instance with his SSH and have sudo access. sudo access can be disabled for a user
While deploying an app, the deploy recipe can run on one or more instances
Deploy action consists of the following commands
- Deploy/Undeploy
- Rollback
- Start/stop/restart web server
When a database instance is associated with an app; OW creates a stack configuration and deployment attributes file on each instance that include the database connection information i.e on each instance that the app is deployed. The app can then read the config file to known the database params.
- Can also write a custom recipe to extract the database info from the config file to a format more appropriate for the application
When environment variable are specified for an app, all env vars are part of the app's deploy attributes; can implement a custom recipe that reads the env vars from the deploy attributes and generates say a YAML file with the env vars
To pass data to applications use "Custom JSON"; the data then appears with the deploy attributes and can be used by the recipes
Except for node js app, can run multiple applications on the same app server
can enable SSL for an application
Lifecycle events
- OW runs the custom recipe for an event after running the built-in ones
- Setup
  - Run after the instance has finished booting
  - Can be triggered manually using the stack command
- Configure
  - Occurs in all instances when one of the following happens
    - An instance enters or leaves the online state
    - We associate or disassociate an elastic ip with an instance
    - We attach or detach a load balancer
  - OW responds to the configure event by running each layer's configure recipes, which update the instances configuration to reflect the current set of online instances
  - Can be triggered manually using the stack command
- Deploy
  - Occurs when the deploy command is run to deploy an app to one or more instances
  - Can run deploy on other instances so they can, for eg update their configuration about the new change
  - Setup runs deploy; it runs the deploy recipes after setup is complete
- Undeploy
  - When we delete an app or run an undeploy command to remove an app from app servers
- Shutdown
  - When we tell OW to shut an instance down (before terminating); teardown actions can be done here
- Rebooting an instance does not trigger any lifecycle events
- For each lifecycle event, ow install a set of configuration files on each instances that contains the current stack state and for deploy events, information about the deployment.
- Starting and stopping large no of instances can trigger a large no of configure events, to avoid unnecessary processing, ow responds to the last event only.
- Each layer (Opsworks, ECS, RDS) have their own built-in recipes; not all have undeploy recipes
- Recipes can be run manually using the stack command
Resource managment
- Resources page enables managment of AWS account resources in the context of an Opsworks stack
- After registering a resource with a stack, we can attach it to stack's instances
  - Assign EBS vols to an instance
  - Assign eip to an instance
  - Assign RDS instances to an app
Security
- OW integrates with IAM to provide role based access to ow and its functionalities
Chef 12
- Two separate chef runs: one to install ow specific agents, configuration etc and the other to run just the custom cookbooks
- No longer supports built-in cookbooks for app server environments php, node js, rail.. no namespace conflicts between built-in cookbooks and custom ones
- Berkshelf is no longer installed in instances; instead use berkshelf to package locally and push to S3
- Raid configs for EBS vols are no longer supported; instead use piops
- Stack settings are no longer available as chef attributes, so stack settings can no longer be accessed through the chef object. Stack settings are available as chef data bags and accessed only thru chef search
Set Chef::Log.level = :debug for controlling chef logs

AWS Opsworks under the hood

Architecture
- Opsworks backend which takes care of running instance, dbs, autoscaling, autohealing, ELB etc. A managed service...
- Opsworks agent which uses chef to do the in-instance provisioning. Once the agent is running, the backend tells the agent to run specific commands using chef recipes. The agent then uploads the logs and status back to the backend.
- The chef client is running on zero mode.
- The backend server is HA but not the real chef server which is different than the usual chef client-server model
- Opsworks is a push model, different than the pull model in the usual setup
Stack change/lifecycle events are completely described using JSON
Events
- Setup
  - Sent when the instance boots; user data on steroids
  - Includes the deploy event; deploying the jar files, restart svcs etc
  - Used for initial installation of software and services
- Configure
  - Sent to all instances when any instance enters or leaves online state
  - Used for making sure that the configuration is up to date; say a new instance is added, update the ELB to include the new node
- Deploy
  - Sent when we deploy via ui/api; part of each setup
  - Use for custom deployment
- Undeploy
  - Sent via ui/API when apps are deleted
  - Used to remove apps from running instances
- Shutdown
  - Sent when an instance is shut down; invoked before the instance is terminated
  - Configurable timeout
  - Use for clean shutdown; connection draining; flush caches etc
Since chef 12.x, Opsworks recipes and custom recipes are run into 2 isolated chef runs
AWS Opsworks layers mapped as roles
Encrypted data bags aren't supported; alternative solutions are
- App env vars with values defined as protected
- Upload encrypted JSON to S3
- Use instance profiles
Opsworks can be used to manage instances in on-premise or hybrid environments
Existing instances running outside of Opsworks can be included in Opsworks by running the agent and including them in one or more layers in an Opsworks stack

Directory-Service

Simple AD is a highly available directory service in AWS
- Highly available and runs in a VPC
- Must specify two subnets; a domain controller is put in both of these subnets; used for failover; the unhealthy one is automatically recovered
- Need to create a DHCP option set in the VPC pointing the Simple AD domain name and the two instances in two subnets as the name servers. Any instances with DHCP enabled can discover these new AD domains.
- After setting up the directory, we can domain join any instances in the VPC to the directory.
  - After joining the instance, users can ssh into the instance using credentials stored in the AD
- Maximum no of users supported is 5000
- Samba 4 compatibility
- Windows powershell cmdlets not supported
- Only 2 domain controller
- Schema extensions not supported
- No LDAP-S integration
- No MFA
AD connector
- Need to open ports for DNS, Kerberos and LDAP in the customer side
- Makes LDAP requests back to the on-premise directory service
- Similar to Simple AD, this is also highly available in a VPC with two subnets
AD connector federation
- Create IAM roles
- Map users in AD to groups
- AD users log in via access URL
  - Log in using AD credentials
  - AD connector proxies the LDAP and kerberos requests over VPN
  - Assume role into AWS managment console
- Just a proxy - no user information is cached

CloudFormation

If a stack cannot be deleted, any remaining resource is retained until the stack can be successfully deleted.
The CF console does not provide a dropdown list to search for AMIs. If only verifies if the ID is a valid AMIs
By default, the console list input parameters alphabetically by their logical id. For custom ordering and grouping of parameters, use AWS::CloudFormation::Interface.
Stack policies can be used to prevent unintentional updates to resources during a stack update.
Deleted stacks are retained and viewable for 90 days.
The validate-template command only checks if the template if valid JSON or YAML.
If the stack update is cancelled, the stack is rolled back to the configuration which existed prior to initiating the stack update.
When a stack policy is set, all updates are denied by default. Explicit allows must be added to the stack policy to allow updates to resources. A DENY statement overrides an ALLOW statement.
To update protected resources, a specific stack policy must be explicitly specified. This policy is applicable only for the current policy.

Billing

Cost allocation tags aren't enabled by default in the billing reports in S3
Types of reports
- Monthly report
- Detailed billing report
- Cost allocation report
- Detailed w/ resources and tags
Turn off unused instances
Spot instances use cases
- Stateless web/app server fleets
- Amazon EMR
- Continuous integration
- HPC
- Grid computing
- Media rendering/transcoding
DynamoDB optimization
- Use caching to save read capacity units
- Better table design
- Use SQS to buffer over-capacity writes

Cost optimization at scale

Turn off non-production workloads outside of business hours
Auto scaling in production workloads
Right sizing - use the cheapest available instance to sustain the requirements; stop using previous generation instances
Reserved capacity
- Buy reserved capacity to cover off always on resources
- Leverage RI utilization to up to 95% utilization like moving around ri's across AZs
Key to achieve cost optimization at scale is by using automation tools; these tools should do the following
- Autotag resources
  - Use AWS config to raise alert for untagged resources
  - Tags should always be created at creation time because use that information we can account for the costs accurately
- Identify "always on" non prod; a automated report listing these resource will be very handy
- Identify instance to down size
- Recommends RI's to purchase
- Dashboard the status and outcomes
- Prepare reports on savings and share with team
Automation
- Allocate costs by tag and account; makes teams responsible for the cost of their resources
- Turn off non-prod instances daily; group non-prod resources into their own accounts
- Change instance size quickly instead of doing things manually
- Move underutilized RI's
Get the right metrics in place for everyone to aim for that; all of these metrics weighted by ECU
- % of instances turned off daily
- % of instances right sized
- % of always on resources covered by RI
- % RI utilization
Tag types
- Cost center
- Responsible owner/sme/lead
- Service (what service in the arch/company)
- Name
Next steps
- Setup a cloud competency center
- Bring in the right tools
- Use metrics to reinforce behavior
- Use partners to accelerate

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
LICENSE		LICENSE
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

AWS notes for self reference

!! Disclamer this info may be obsolete or redundant in the future.

IAM (Identity and Access Management)

S3 (Simple Storage service)

Cloudfront

Import-Export

Storage Gateway

SNS (Simple Notification service)

SQS (Simple Queue Service)

VPC (Virtual Private Cloud)

VPC peering

Direct connect

EBS (Elastic Block Storage)

EC2 (Elastic Cloud Compute)

ELB (Elastic Load Balancer)

Route 53

RDS (Relational Database Service)

Elastic Beanstalk

DynamoDB

Cloudwatch

CloudTrail

Kinesis

OpsWorks

Directory-Service

CloudFormation

Billing

About

Releases

Packages

License

avinashadev/aws-notes

Folders and files

Latest commit

History

Repository files navigation

AWS notes for self reference

!! Disclamer this info may be obsolete or redundant in the future.

IAM (Identity and Access Management)

S3 (Simple Storage service)

Cloudfront

Import-Export

Storage Gateway

SNS (Simple Notification service)

SQS (Simple Queue Service)

VPC (Virtual Private Cloud)

VPC peering

Direct connect

EBS (Elastic Block Storage)

EC2 (Elastic Cloud Compute)

ELB (Elastic Load Balancer)

Route 53

RDS (Relational Database Service)

Elastic Beanstalk

DynamoDB

Cloudwatch

CloudTrail

Kinesis

OpsWorks

Directory-Service

CloudFormation

Billing

About

Topics

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Packages