Update outdated default image for AWS/GCP #75

guessi · 2020-09-11T15:56:53Z

Fix all regions have corresponding default AMI mapping.
Refresh outdated default images for AWS & GCP.
Change default instance type with latest T3 series.
Support for AWS new region: ap-southeast-3, Asia Pacific (Jakarta)

AMI information comes from upstream Ubuntu Cloud Image Finder

https://cloud-images.ubuntu.com/locator/

Or alternative, retrieve from scripts:

AWS:

curl -fsSL https://cloud-images.ubuntu.com/locator/releasesTable | \
    awk -F '"' '/Amazon.*focal.*amd64.*20220706*/{print$4" "$18}' | \
    sed 's/\>//;s/\<\/a\>//' | \
    column -t | \
    sort

GCP:

curl -fsSL https://cloud-images.ubuntu.com/locator/releasesTable | \
    awk -F '"' '/Google.*amd64.*ubuntu-2004-focal-v20220712.*/{print$4" "$16}' | \
    column -t | \
    sort

Last Update: 2022.07.16

mtb-xt · 2020-09-30T23:11:40Z

@bradrydzewski @tboerger is there any chance we could merge this? It would be awesome if we could have AWS SSM agent in our drone agent instances...

ademariag · 2020-12-16T11:16:01Z

Any chance that we could push this at one point?

guessi · 2021-01-16T17:32:58Z

rebase, and update to current latest images

ademariag · 2021-01-22T11:39:07Z

@bradrydzewski @tboerger is there any chance we could merge this?

tboerger · 2021-01-25T09:10:12Z

I can't merge on this repo, it's up to @bradrydzewski

ademariag · 2021-03-16T21:26:38Z

Is this project dead?
I don't see any updates lately and lots of valuable contributions are not being accepted. @bradrydzewski could you add more maintainers perhaps from the community?

guessi · 2021-06-12T07:30:47Z

FYI, just update the default images available on AWS/GCP

guessi · 2022-01-10T02:18:15Z

Two years later, it finally got approved 🎉

For long-term consideration, it feels like we need an auto discovery mechanism for discovering new image id for specific requirement of the platform. Those image ids are now 6 month old 👶 🍼 since my last commit.

Let me know if image ids update is required before it get merged.

guessi · 2022-01-12T16:46:56Z

FYI, just refreshed AMI IDs and squash commits, also I added support for AWS new region: ap-southeast-3 (Asia Pacific, Jakarta).

bradrydzewski · 2022-01-14T15:18:29Z

@guessi thanks, can you please provide the link where you are sourcing the AMI list? For security purposes, we need to manually verify before we can merge (sorry, this is a security requirement, trust but verify ...)

guessi · 2022-01-14T15:22:13Z

@bradrydzewski sure, security always first !!!

Let me know where should I put the link,
put it here as PR description or as comment in code?

Update: I've update the PR description for where the AMI info comes from.

bradrydzewski · 2022-01-14T15:41:59Z

I noticed these images are being labeled as hvm-ssd but the autoscaler does not use ssd disk types by default. Have you tested any of these new amis with default settings to ensure there is no risk of regression?

needs internal testing before merge

guessi · 2022-01-14T16:07:20Z

@bradrydzewski I personally only use SSD nowadays and I have used these image as source for a long time :-)

If this is really a concern, we can alternatively retrieve image id from awscli instead, which might be much reliable for user?

aws ssm get-parameters \
    --names /aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id

From trustworthy source mentioned in Ubuntu Official Documentation[1].

or

aws ec2 describe-images \
    --owners aws-marketplace \
    --filters "Name=owner-id,Values=679593333241" "Name=name,Values=*ubuntu-focal-20.04-amd64*" \
    --query "sort_by(Images, &CreationDate)[-1:].[Name, ImageId]" \
    --region us-east-1 \
    --output text

Where "679593333241" is Canonical's ID, where you can find how to verify it is released by Canonical here[2].

Or even more better, retrieve from AWS API.

Ref:

bradrydzewski · 2022-01-14T16:18:31Z

I have no problem with ssd or even ssd by default, I just want to make sure that if someone runs the autoscaler with default options it will work out of the box and not error. If we use images labeled hvm-ssd do we need to make any changes to volume type or will everything just work? I am asking because I do not use AWS so the answer to this is non-obvious to me.

guessi · 2022-01-14T16:32:27Z

Your concern is not exist, since hvm-ssd have been used for quite long time.

To verify that we can easily find existed image ids for us-east-1[1], us-west-1[2] and you will find that current images are hvm-ssd already.

AWS US-EAST-1

$ aws ec2 describe-images --region us-east-1 --image-ids "ami-43a15f3e" --output json | grep "ImageLocation" # image id for us-east-1
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180306",

AWS US-WEST-1

% aws ec2 describe-images --region us-west-1 --image-ids "ami-925144f2" --output json | grep "ImageLocation" # image id for us-west-1
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180306",

[1] Image ID for us-east-1: https://github.com/drone/autoscaler/blob/810c77f/drivers/amazon/util.go#L42
[2] Image ID for us-west-1: https://github.com/drone/autoscaler/blob/810c77f/drivers/amazon/util.go#L47

** 810c77f is current master commit hash.

bradrydzewski · 2022-01-17T14:41:21Z

you will find that current images are hvm-ssd already.

thanks! I think this is the context I was missing

guessi · 2022-02-18T18:29:05Z

Just update once again after month.

Notable changes: - Refresh AMIs for AWS/GCP - Support for AWS new region support: ap-southeast-3, Asia Pacific (Jakarta)

guessi · 2022-07-16T09:17:30Z

Umm... looks like I need to resolve conflict and bump the image version again. Let me handle it.

guessi · 2022-07-16T09:45:57Z

drivers/amazon/util.go

+	"us-gov-west-1": "ami-0cbaf57cea1d72aec",
+
+	// AWS China: Ubuntu Server 20.04 LTS
+	// Upstream release version: 20210720


Current latest version available from upstream Ubuntu. Yes, it's "2021", but it's latest.

guessi · 2022-07-16T09:52:34Z

Just update the the image id for AWS and GCP once again.

At the moment of I created this PR (Sep 11, 2020) the latest version available was Ubuntu Server 20.04 LTS and 2 years later, it's now 2022, Ubuntu Server 22.04 LTS was released on Apr 22, 2022.

Feel free to let me know if it is needed to update to Ubuntu Server 22.04 LTS.

maxgruebneraeroqual · 2022-07-24T03:40:02Z

We've run into an issue with builds that was sorted by switching to Ubuntu 22.04 AMIs.

https://community.harness.io/t/drone-autoscaler-amis-too-old-for-net-6-application/12249/5

Further, we've just got an email from AWS about the 16.04 Ubuntu AMIs being deprecated.

tphoney · 2022-08-30T10:05:39Z

thanks for the continued work on this PR !

solid work !!

tboerger approved these changes Oct 8, 2020

View reviewed changes

bradrydzewski previously approved these changes Jan 7, 2022

View reviewed changes

bradrydzewski self-requested a review January 14, 2022 15:56

bradrydzewski approved these changes Jul 14, 2022

View reviewed changes

guessi added 4 commits July 16, 2022 17:16

Update outdated default image for AWS/GCP (2022-01)

a892244

Notable changes: - Refresh AMIs for AWS/GCP - Support for AWS new region support: ap-southeast-3, Asia Pacific (Jakarta)

Fixed missing line after rebase

8588d76

Update outdated default image for AWS (2022-02-19)

aac975c

Update outdated default image for GCP (2022-02-19)

d455fa1

guessi added 2 commits July 16, 2022 17:42

Update outdated default image for AWS (2022-07-16)

5546374

Update outdated default image for GCP (2022-07-16)

3cbdcd0

guessi commented Jul 16, 2022

View reviewed changes

tphoney merged commit 79ea07e into drone:master Aug 30, 2022

guessi deleted the refresh-default-images branch August 30, 2022 15:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update outdated default image for AWS/GCP #75

Update outdated default image for AWS/GCP #75

guessi commented Sep 11, 2020 •

edited

Loading

mtb-xt commented Sep 30, 2020

ademariag commented Dec 16, 2020

guessi commented Jan 16, 2021

ademariag commented Jan 22, 2021

tboerger commented Jan 25, 2021

ademariag commented Mar 16, 2021

guessi commented Jun 12, 2021

guessi commented Jan 10, 2022

guessi commented Jan 12, 2022 •

edited

Loading

bradrydzewski commented Jan 14, 2022

guessi commented Jan 14, 2022 •

edited

Loading

bradrydzewski commented Jan 14, 2022 •

edited

Loading

guessi commented Jan 14, 2022

bradrydzewski commented Jan 14, 2022

guessi commented Jan 14, 2022

bradrydzewski commented Jan 17, 2022

guessi commented Feb 18, 2022

guessi commented Jul 16, 2022 •

edited

Loading

guessi Jul 16, 2022

guessi commented Jul 16, 2022

maxgruebneraeroqual commented Jul 24, 2022

tphoney commented Aug 30, 2022

Update outdated default image for AWS/GCP #75

Update outdated default image for AWS/GCP #75

Conversation

guessi commented Sep 11, 2020 • edited Loading

mtb-xt commented Sep 30, 2020

ademariag commented Dec 16, 2020

guessi commented Jan 16, 2021

ademariag commented Jan 22, 2021

tboerger commented Jan 25, 2021

ademariag commented Mar 16, 2021

guessi commented Jun 12, 2021

guessi commented Jan 10, 2022

guessi commented Jan 12, 2022 • edited Loading

bradrydzewski commented Jan 14, 2022

guessi commented Jan 14, 2022 • edited Loading

bradrydzewski commented Jan 14, 2022 • edited Loading

guessi commented Jan 14, 2022

bradrydzewski commented Jan 14, 2022

guessi commented Jan 14, 2022

bradrydzewski commented Jan 17, 2022

guessi commented Feb 18, 2022

guessi commented Jul 16, 2022 • edited Loading

guessi Jul 16, 2022

Choose a reason for hiding this comment

guessi commented Jul 16, 2022

maxgruebneraeroqual commented Jul 24, 2022

tphoney commented Aug 30, 2022

guessi commented Sep 11, 2020 •

edited

Loading

guessi commented Jan 12, 2022 •

edited

Loading

guessi commented Jan 14, 2022 •

edited

Loading

bradrydzewski commented Jan 14, 2022 •

edited

Loading

guessi commented Jul 16, 2022 •

edited

Loading