Functionality to create storage buckets in GCP and AWS #291

Fryyyyy · 2021-04-19T02:16:48Z

As part of the work for #135 - first we need to be able to create buckets.

codecov-commenter · 2021-04-19T02:20:11Z

Codecov Report

Merging #291 (551f8c6) into master (e4448c6) will increase coverage by 0.26%.
The diff coverage is 67.08%.

@@            Coverage Diff             @@
##           master     #291      +/-   ##
==========================================
+ Coverage   65.39%   65.66%   +0.26%     
==========================================
  Files          14       32      +18     
  Lines         968     2222    +1254     
==========================================
+ Hits          633     1459     +826     
- Misses        335      763     +428

Flag	Coverage Δ
nosetests	`65.66% <67.08%> (+0.26%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...cloudforensics/providers/azure/internal/network.py	`24.52% <24.52%> (ø)`
libcloudforensics/providers/gcp/forensics.py	`34.78% <28.28%> (-30.22%)`	⬇️
libcloudforensics/providers/aws/internal/kms.py	`40.00% <40.00%> (ø)`
...ibcloudforensics/providers/gcp/internal/compute.py	`51.01% <45.94%> (-11.34%)`	⬇️
libcloudforensics/providers/aws/forensics.py	`56.16% <53.06%> (-13.61%)`	⬇️
...cs/providers/gcp/internal/compute_base_resource.py	`37.33% <54.28%> (+0.82%)`	⬆️
libcloudforensics/providers/aws/internal/log.py	`73.07% <55.55%> (-4.20%)`	⬇️
libcloudforensics/providers/gcp/internal/common.py	`56.89% <57.44%> (+0.57%)`	⬆️
...bcloudforensics/providers/gcp/internal/function.py	`40.54% <62.50%> (+8.28%)`	⬆️
...loudforensics/providers/azure/internal/resource.py	`65.38% <65.38%> (ø)`
... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8184367...551f8c6. Read the comment docs.

…ensics-utils into create_storage

Fryyyyy · 2021-04-19T04:59:27Z

Not sure what's up with this current linting error ?

giovannt0

Few nits, otherwise LGTM

giovannt0 · 2021-04-19T08:28:56Z

libcloudforensics/providers/aws/internal/s3.py

+          ACL=acl,
+          CreateBucketConfiguration={
+              'LocationConstraint': region or self.aws_account.default_region
+          })  # type: Dict[str, Any]


CreateBucketConfiguration={ # type: Dict[str, Any] 'LocationConstraint': region or self.aws_account.default_region })

Should fix the mypy problem

giovannt0 · 2021-04-19T08:31:22Z

libcloudforensics/providers/aws/internal/s3.py

+      self,
+      name: str,
+      region: Optional[str] = None,
+      acl: Optional[str] = 'private') -> Dict[str, Any]:


The Optional type hint can be removed here since it's assigned a default value.
In general, for named parameters that default to None, we use the Optional type hint. All other cases, we don't.

giovannt0 · 2021-04-19T08:32:23Z

libcloudforensics/providers/aws/internal/s3.py

+    https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl  # pylint: disable=line-too-long
+
+    Returns:
+      Dict: An API operation object an S3 bucket.


Typo/missing word(s) in the sentence?

giovannt0 · 2021-04-19T08:33:15Z

libcloudforensics/providers/gcp/internal/storage.py

-                        gcs_path: str,
-                        user_project: Optional[str] = None) -> Dict[str, Any]:
+  def GetObjectMetadata(
+      self, gcs_path: str,


Add a new line after self param

giovannt0 · 2021-04-19T08:33:44Z

libcloudforensics/providers/gcp/internal/storage.py

@@ -98,8 +98,7 @@ def GetObjectMetadata(self,
    response = request.execute()  # type: Dict[str, Any]
    return response

-  def GetBucketACLs(self,
-                    bucket: str,
+  def GetBucketACLs(self, bucket: str,


Add a new line after self param

giovannt0 · 2021-04-19T08:34:16Z

libcloudforensics/providers/gcp/internal/storage.py

+      self,
+      bucket: str,
+      labels: Optional[Dict[str, str]] = None,
+      predefinedacl: Optional[str] = 'private',


See comment above for the use of Optional
Same below

Fryyyyy · 2021-04-20T01:54:00Z

libcloudforensics/providers/aws/internal/s3.py

+      self,
+      name: str,
+      region: Optional[str] = None,
+      acl: Optional[str] = 'private') -> Dict[str, Any]:


Fryyyyy · 2021-04-20T01:54:35Z

libcloudforensics/providers/aws/internal/s3.py

+    https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl  # pylint: disable=line-too-long
+
+    Returns:
+      Dict: An API operation object an S3 bucket.


Fryyyyy · 2021-04-20T01:54:55Z

libcloudforensics/providers/aws/internal/s3.py

+          ACL=acl,
+          CreateBucketConfiguration={
+              'LocationConstraint': region or self.aws_account.default_region
+          })  # type: Dict[str, Any]


Fryyyyy · 2021-04-20T01:55:58Z

libcloudforensics/providers/gcp/internal/storage.py

-                        gcs_path: str,
-                        user_project: Optional[str] = None) -> Dict[str, Any]:
+  def GetObjectMetadata(
+      self, gcs_path: str,


Fryyyyy · 2021-04-20T01:56:33Z

libcloudforensics/providers/gcp/internal/storage.py

@@ -98,8 +98,7 @@ def GetObjectMetadata(self,
    response = request.execute()  # type: Dict[str, Any]
    return response

-  def GetBucketACLs(self,
-                    bucket: str,
+  def GetBucketACLs(self, bucket: str,


Fryyyyy · 2021-04-20T01:56:49Z

libcloudforensics/providers/gcp/internal/storage.py

+      self,
+      bucket: str,
+      labels: Optional[Dict[str, str]] = None,
+      predefinedacl: Optional[str] = 'private',


Fryyyyy · 2021-04-21T10:08:39Z

:( still the mypy error, @giovannt0

giovannt0 · 2021-04-21T13:25:10Z

@Fryyyyy this works locally:

bucket = client.create_bucket(
          Bucket=name,
          ACL=acl,
          CreateBucketConfiguration={
              'LocationConstraint': region or self.aws_account.default_region
          })  # type: Dict[str, Any]
      return bucket

libcloudforensics/providers/aws/internal/s3.py

libcloudforensics/providers/gcp/internal/storage.py

tests/providers/aws/internal/test_s3.py

tests/providers/gcp/internal/test_storage.py

Fryyyyy added 7 commits April 13, 2021 21:57

GCS Create

1b9bd0f

Add test for GCS

46160a1

Create S3 Bucket

5da2fed

CLI tool for S3

1a2371c

S3 create bucket test

2547e64

Fix comment

582f78e

Small linting thing

88fe5be

Fryyyyy requested review from giovannt0, hacktobeer, sa3eed3ed and tomchop as code owners April 19, 2021 02:16

google-cla bot added the cla: yes label Apr 19, 2021

Merge branch 'master' into create_storage

9d60be5

Fryyyyy added 2 commits April 18, 2021 22:27

Pylint fixes

3eb0387

Merge branch 'create_storage' of https://github.com/Fryyyyy/cloud-for…

cb067d1

…ensics-utils into create_storage

giovannt0 requested changes Apr 19, 2021

View reviewed changes

Fixes from review

7e20a21

Fryyyyy commented Apr 20, 2021

View reviewed changes

Merge branch 'master' into create_storage

551f8c6

tomchop requested changes Apr 22, 2021

View reviewed changes

Review fixes

2322c2e

giovannt0 approved these changes Apr 23, 2021

View reviewed changes

Merge branch 'master' into create_storage

6ccbb2e

tomchop self-requested a review April 26, 2021 12:13

tomchop approved these changes Apr 26, 2021

View reviewed changes

tomchop merged commit d15bc2a into google:master Apr 26, 2021

Fryyyyy deleted the create_storage branch April 27, 2021 00:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Functionality to create storage buckets in GCP and AWS #291

Functionality to create storage buckets in GCP and AWS #291

Fryyyyy commented Apr 19, 2021

codecov-commenter commented Apr 19, 2021 •

edited

Loading

Fryyyyy commented Apr 19, 2021

giovannt0 left a comment

giovannt0 Apr 19, 2021

Fryyyyy Apr 20, 2021

giovannt0 Apr 19, 2021

giovannt0 Apr 19, 2021

Fryyyyy Apr 20, 2021

giovannt0 Apr 19, 2021

Fryyyyy Apr 20, 2021

giovannt0 Apr 19, 2021

Fryyyyy Apr 20, 2021

giovannt0 Apr 19, 2021

Fryyyyy Apr 20, 2021

Fryyyyy Apr 20, 2021

Fryyyyy Apr 20, 2021

Fryyyyy Apr 20, 2021

Fryyyyy Apr 20, 2021

Fryyyyy Apr 20, 2021

Fryyyyy Apr 20, 2021

Fryyyyy commented Apr 21, 2021

giovannt0 commented Apr 21, 2021

Functionality to create storage buckets in GCP and AWS #291

Functionality to create storage buckets in GCP and AWS #291

Conversation

Fryyyyy commented Apr 19, 2021

codecov-commenter commented Apr 19, 2021 • edited Loading

Codecov Report

Fryyyyy commented Apr 19, 2021

giovannt0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fryyyyy commented Apr 21, 2021

giovannt0 commented Apr 21, 2021

codecov-commenter commented Apr 19, 2021 •

edited

Loading