Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update DevEx branch #133

Merged
merged 6 commits into from
Sep 19, 2023
Merged

update DevEx branch #133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
15791a7
explicit region_name in boto3 client init
MartinKlefas Sep 14, 2023
2a46311
properly sanitize url substring
MartinKlefas Sep 15, 2023
19f98db
update upload test case to include s3 region_name
MartinKlefas Sep 15, 2023
36f20ff
Bump nest-asyncio from 1.5.7 to 1.5.8
dependabot[bot] Sep 18, 2023
e46c921
Merge pull request #132 from runpod/dependabot/pip/nest-asyncio-1.5.8
justinmerrell Sep 18, 2023
d47d9c5
Merge pull request #130 from MartinKlefas/fix-s3-region-issue
justinmerrell Sep 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ fastapi[all] == 0.103.1
urllib3 >= 1.26.6

# Testing Requirements
nest_asyncio == 1.5.7
nest_asyncio == 1.5.8
19 changes: 18 additions & 1 deletion runpod/serverless/utils/rp_upload.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,24 @@
from boto3.s3.transfer import TransferConfig
from botocore.config import Config
from tqdm_loggable.auto import tqdm
from urllib.parse import urlparse

logger = logging.getLogger("runpod upload utility")
FMT = "%(filename)-20s:%(lineno)-4d %(asctime)s %(message)s"
logging.basicConfig(level=logging.INFO, format=FMT, handlers=[logging.StreamHandler()])

def extract_region_from_url(endpoint_url):
parsed_url = urlparse(endpoint_url)
# AWS/backblaze S3-like URL
if '.s3.' in endpoint_url:
return endpoint_url.split('.s3.')[1].split('.')[0]
# DigitalOcean Spaces-like URL
elif parsed_url.netloc.endswith('.digitaloceanspaces.com'):
return endpoint_url.split('.')[1].split('.digitaloceanspaces.com')[0]
else:
# Additional cases can be added here
return None


# --------------------------- S3 Bucket Connection --------------------------- #
def get_boto_client(
Expand Down Expand Up @@ -57,12 +70,16 @@ def get_boto_client(
secret_access_key = os.environ.get('BUCKET_SECRET_ACCESS_KEY', None)

if endpoint_url and access_key_id and secret_access_key:
# Extract region from the endpoint URL
region = extract_region_from_url(endpoint_url)

boto_client = bucket_session.client(
's3',
endpoint_url=endpoint_url,
aws_access_key_id=access_key_id,
aws_secret_access_key=secret_access_key,
config=boto_config
config=boto_config,
region_name=region
)
else:
boto_client = None
Expand Down
3 changes: 2 additions & 1 deletion tests/test_serverless/test_utils/test_upload.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ def test_get_boto_client(self):
endpoint_url=bucket_creds['endpointUrl'],
aws_access_key_id=bucket_creds['accessId'],
aws_secret_access_key=bucket_creds['accessSecret'],
config=unittest.mock.ANY
config=unittest.mock.ANY,
region_name=None
)

def test_get_boto_client_environ(self):
Expand Down