Auto-configure Network Interface Card (NIC) throughput

[jamesbornholt]

Remove the need for customers to manually configure Mountpoint for Amazon S3 to achieve high throughout for the instance type they are using

A few of us now have misconfigured the connector by not specifying target throughput on EC2 instances with large NICs. This seems like something we should be able to autodiscover and just set correctly by default.

The CRT S3 client has the beginnings of this interface (awslabs/aws-c-s3#70), but as far as I can tell it needs to be manually invoked and only carries data for c5n.18xlarge right now.

[sean-smith]

I wrote a script to auto-detect this based on ec2 metadata & describe instance api. Would love to see this integrated in somehow:

#!/bin/bash

# get network throughput from ec2 instance
instance_type=$(curl -s http://169.254.169.254/latest/meta-data/instance-type)
region=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep region|awk -F\" '{print $4}')
network=$(aws ec2 --region ${region} describe-instance-types --instance-types ${instance_type} --query "InstanceTypes[].[NetworkInfo.NetworkPerformance]" --output text | grep -o '[0-9]\+')

# Mount S3 Bucket
mkdir -p ${1}
mount-s3 --throughput-target-gbps ${network} ${2} ${1}

[eslrahc-swa]

Hey there, we are implementing this via calling crt imds client.

[sean-smith]

Cool - keep in mind you'll likely need to filter down. IMDS returns values like:

$ aws ec2 describe-instance-types --filters "Name=instance-type,Values=c5.*" --query "InstanceTypes[].[InstanceType, NetworkInfo.NetworkPerformance]" --output table
-------------------------------------
|       DescribeInstanceTypes       |
+--------------+--------------------+
|  c5.4xlarge  |  Up to 10 Gigabit  |
|  c5.xlarge   |  Up to 10 Gigabit  |
|  c5.12xlarge |  12 Gigabit        |
|  c5.24xlarge |  25 Gigabit        |
|  c5.9xlarge  |  10 Gigabit        |
|  c5.2xlarge  |  Up to 10 Gigabit  |
|  c5.large    |  Up to 10 Gigabit  |
|  c5.metal    |  25 Gigabit        |
|  c5.18xlarge |  25 Gigabit        |
+--------------+--------------------+

I added a regex and filter to get down to just the number.

$ instance_type=c5n.18xlarge
$ region=us-east-1
$ aws ec2 --region ${region} describe-instance-types --instance-types ${instance_type} --query "InstanceTypes[].[NetworkInfo.NetworkPerformance]" --output text | grep -o '[0-9]\+'
100

[jamesbornholt]

Added in #184!

[ec2-user@ip-172-31-17-77 data]$ TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-type
m6idn.16xlarge
[ec2-user@ip-172-31-17-77 data]$ ./mount-s3 bornholt-test-bucket ~/mnt -f
2023-04-27T01:19:26.070099Z  INFO mount_s3: target network throughput 100 Gbps
2023-04-27T01:19:26.410418Z  WARN mount_s3: bucket bornholt-test-bucket is in region us-west-2, not us-east-1. redirecting...
2023-04-27T01:19:26.531456Z  INFO mount_s3: successfully mounted "/home/ec2-user/mnt"
2023-04-27T01:19:26.531570Z  INFO mountpoint_s3::fuse::session: fuse worker 0 is thread ID 1030411

[sean-smith]

Sweeet!