Merge pull request #12 from elfkuzco/http-timeouts

configure resources with timeouts

Author: benoit74

CHANGELOG.md

@@ -3,9 +3,15 @@
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html), 
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html),
 as of 0.8.2
 
+## [Unreleased]
+
+### Added
+
+- configure resources with user-specified timeouts
+
 ## [0.9.0]
 
 ### Added
@@ -40,7 +46,6 @@ as of 0.8.2
 
 - Fixed `delete_object()` which contained a typo.
 
-
 ## [0.8.1]
 
 - Added `download_matching_file()` and `download_matching_fileobj()` to download a key
@@ -61,54 +66,53 @@ as of 0.8.2
 
 ## [0.5]
 
-* fixed `allow_public_downloads_on()` when not specifying bucket
-* using less-specified boto3 dependency version to prevent urllib dep version clash with requests
+- fixed `allow_public_downloads_on()` when not specifying bucket
+- using less-specified boto3 dependency version to prevent urllib dep version clash with requests
 
 ## [0.4]
 
-* progress report uses humanfriendly if available
-* added `get_wasabi_compliance()` to retrieve an XML bucket or object compliance
+- progress report uses humanfriendly if available
+- added `get_wasabi_compliance()` to retrieve an XML bucket or object compliance
 
 ## [0.3]
 
-* added `has_object_matching()` to check an object with multiple key/value pairs
+- added `has_object_matching()` to check an object with multiple key/value pairs
 
 ## [0.2]
 
-* more flexible requirements for requests
-* removed some debug prints
-* don't fail if `AWS_PROFILE` environ is present
-* added `s3_test_url` script to test an URL
-* added `delete_object()` shortcut
+- more flexible requirements for requests
+- removed some debug prints
+- don't fail if `AWS_PROFILE` environ is present
+- added `s3_test_url` script to test an URL
+- added `delete_object()` shortcut
 
 ## [0.1]
 
-* Initial version featuring:
- * URL parsing
- * Visual transfer progress hook
- * wasabi detection
- * configured S3 resource (`.resource`)
- * configured S3 client (`.client`)
- * requests auth module (`.aws_auth`)
- * credentials check (list bucket, read, write)
- * configured generic service (`iam` for ex.)
- * direct bucket names
- * direct bucket access
- * test bucket existence
- * bucket creation
- * test object exitence
- * direct object head access
- * direct object stats access
- * test object with etag exitence
- * test object with meta existence
- * set policy on bucket (allow external downloads)
- * set wasabi auto delete on bucket (retention)
- * set wasabi auto delete on object (retention extension)
- * delete bucket (with wasabi force)
- * wasabi rename bucket
- * wasabi object rename (with prefix)
- * set wasabi compliance
- * direct object download
- * object download url
- * validate file etag (mono and multipart)
-
+- Initial version featuring:
+- URL parsing
+- Visual transfer progress hook
+- wasabi detection
+- configured S3 resource (`.resource`)
+- configured S3 client (`.client`)
+- requests auth module (`.aws_auth`)
+- credentials check (list bucket, read, write)
+- configured generic service (`iam` for ex.)
+- direct bucket names
+- direct bucket access
+- test bucket existence
+- bucket creation
+- test object exitence
+- direct object head access
+- direct object stats access
+- test object with etag exitence
+- test object with meta existence
+- set policy on bucket (allow external downloads)
+- set wasabi auto delete on bucket (retention)
+- set wasabi auto delete on object (retention extension)
+- delete bucket (with wasabi force)
+- wasabi rename bucket
+- wasabi object rename (with prefix)
+- set wasabi compliance
+- direct object download
+- object download url
+- validate file etag (mono and multipart)

src/kiwixstorage/VERSION

@@ -1 +1 @@
-0.9.0
+0.9.1

src/kiwixstorage/__init__.py

@@ -2,15 +2,15 @@
 # -*- coding: utf-8 -*-
 # vim: ai ts=4 sts=4 et sw=4 nu
 
-""" Kiwix/OpenZIM S3 interface
+"""Kiwix/OpenZIM S3 interface
 
-    helpers for S3 storage, autoconf from URL + Wasabi (wasabisys.com) extras
+helpers for S3 storage, autoconf from URL + Wasabi (wasabisys.com) extras
 
-    Goal is mainly to provide a configured s3.client and s3.resource from an URL
-    Users could limit usage to this and use boto3 directly from there.
+Goal is mainly to provide a configured s3.client and s3.resource from an URL
+Users could limit usage to this and use boto3 directly from there.
 
-    A few additional wrappers are in place to simplify common actions.
-    Also, non-S3, wasabi-specific features are exposed directly. """
+A few additional wrappers are in place to simplify common actions.
+Also, non-S3, wasabi-specific features are exposed directly."""
 
 import os
 import io
@@ -26,6 +26,7 @@
 
 import boto3
 import botocore
+from botocore.config import Config
 import requests
 from aws_requests_auth.aws_auth import AWSRequestsAuth
 
@@ -155,7 +156,6 @@ class NotFoundError(Exception):
 
 
 class KiwixStorage:
-
     BUCKET_NAME = "bucketname"
     KEY_ID = "keyid"
     SECRET_KEY = "secretaccesskey"  # nosec
@@ -187,6 +187,12 @@ def __init__(self, url, **kwargs):
         self._resource = self._bucket = None
         self._params = {}
         self._parse_url(url, **kwargs)
+        # set configuration params with defaults being the same as if it were missing
+        # https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
+        self._config = Config(
+            connect_timeout=kwargs.get("connect_timeout", 60),
+            read_timeout=kwargs.get("read_timeout", 60),
+        )
 
     def _parse_url(self, url, **kwargs):
         try:
@@ -260,6 +266,7 @@ def get_resource(self):
                 aws_access_key_id=self.params.get(self.KEY_ID),
                 aws_secret_access_key=self.params.get(self.SECRET_KEY),
                 endpoint_url=self.params.get(self.ENDPOINT_URL),
+                config=self._config,
             )
         except Exception as exc:
             raise AuthenticationError(str(exc))
@@ -282,6 +289,7 @@ def get_service(self, service_name, use_default_region=True):
                     if self.is_wasabi and use_default_region
                     else self.region
                 ),
+                config=self._config,
             )
         except Exception as exc:
             raise AuthenticationError(str(exc))