adobe · afranken · Aug 15, 2023 · Aug 8, 2023 · Aug 14, 2023 · Aug 15, 2023
diff --git a/.github/workflows/codeql.yml → .github/workflows/codeql-analysis.yml b/.github/workflows/codeql.yml → .github/workflows/codeql-analysis.yml
diff --git a/README.md b/README.md
@@ -1,33 +1,39 @@
 [![Latest Version](https://img.shields.io/maven-central/v/com.adobe.testing/s3mock.svg?maxAge=3600&label=Latest%20Release)](https://search.maven.org/#search%7Cga%7C1%7Cg%3Acom.adobe.testing%20a%3As3mock)
-[![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/adobe/s3mock/)
 ![Maven Build](https://github.com/adobe/S3Mock/workflows/Maven%20Build/badge.svg)
-[![Java17](https://img.shields.io/badge/MADE%20with-Java17-RED.svg)](#Java)
-[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#Kotlin)
+[![Docker Hub](https://img.shields.io/badge/docker-latest-blue.svg)](https://hub.docker.com/r/adobe/s3mock/)
 [![Docker Pulls](https://img.shields.io/docker/pulls/adobe/s3mock)](https://hub.docker.com/r/adobe/s3mock)
-[![GitHub stars](https://img.shields.io/github/stars/adobe/S3Mock.svg?style=social&label=Star&maxAge=2592000)](https://github.com/adobe/S3Mock/stargazers/)
+[![Java17](https://img.shields.io/badge/MADE%20with-Java17-RED.svg)](#Java)
+[![Kotlin](https://img.shields.io/badge/MADE%20with-Kotlin-RED.svg)](#Kotlin)  
 [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/7673/badge)](https://bestpractices.coreinfrastructure.org/projects/7673)
 [![OpenSSF Scorecard](https://img.shields.io/ossf-scorecard/github.com/adobe/S3Mock?label=openssf%20scorecard&style=flat)](https://api.securityscorecards.dev/projects/github.com/adobe/S3Mock)
+[![GitHub stars](https://img.shields.io/github/stars/adobe/S3Mock.svg?style=social&label=Star&maxAge=2592000)](https://github.com/adobe/S3Mock/stargazers/)
 
 <!-- TOC -->
   * [S3Mock](#s3mock)
   * [Changelog](#changelog)
   * [Supported S3 operations](#supported-s3-operations)
-  * [File System Structure](#file-system-structure)
-    * [Root-Folder](#root-folder)
-    * [Buckets](#buckets)
-    * [Objects](#objects)
-    * [Multipart Uploads](#multipart-uploads)
   * [Usage](#usage)
-    * [Configuration](#configuration)
+    * [Usage of AWS S3 SDKs](#usage-of-aws-s3-sdks)
+      * [Path-style vs Domain-style access](#path-style-vs-domain-style-access)
+      * [Presigned URLs](#presigned-urls)
+    * [Usage of AWS CLI](#usage-of-aws-cli)
+    * [Usage of plain HTTP](#usage-of-plain-http)
+    * [S3Mock configuration options](#s3mock-configuration-options)
     * [S3Mock Docker](#s3mock-docker)
       * [Start using the command-line](#start-using-the-command-line)
       * [Start using the Fabric8 Docker-Maven-Plugin](#start-using-the-fabric8-docker-maven-plugin)
       * [Start using Testcontainers](#start-using-testcontainers)
+      * [Start using Docker compose](#start-using-docker-compose)
     * [S3Mock Java](#s3mock-java)
       * [Start using the JUnit4 Rule](#start-using-the-junit4-rule)
       * [Start using the JUnit5 Extension](#start-using-the-junit5-extension)
       * [Start using the TestNG Listener](#start-using-the-testng-listener)
       * [Start programmatically](#start-programmatically)
+  * [File System Structure](#file-system-structure)
+    * [Root-Folder](#root-folder)
+    * [Buckets](#buckets)
+    * [Objects](#objects)
+    * [Multipart Uploads](#multipart-uploads)
   * [Build & Run](#build--run)
     * [Java](#java)
     * [Kotlin](#kotlin)
@@ -150,77 +156,80 @@ Of these [operations of the Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/
 | [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html)                                                           | :white_check_mark: |                        |
 | [WriteGetObjectResponse](https://docs.aws.amazon.com/AmazonS3/latest/API/API_WriteGetObjectResponse.html)                                           | :x:                |                        |
 
-## File System Structure
-S3Mock stores Buckets, Objects, Parts and other data on disk.  
-This lets users inspect the stored data while the S3Mock is running.  
-If the config property `retainFilesOnExit` is set to `true`, this data will not be deleted when S3Mock is shut down.
+## Usage
 
-| :exclamation: FYI                                                                                                                                                                                                                                   |
-|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| While it _may_ be possible to start S3Mock on a root folder from a previous run and have all data available through the S3 API, the structure and contents of the files are not considered Public API, and are subject to change in later releases. |
-| Also, there are no automated test cases for this behaviour.                                                                                                                                                                                         |
+### Usage of AWS S3 SDKs
 
-### Root-Folder
-S3Mock stores buckets and objects a root-folder.
+S3Mock can be used with any of the available AWS S3 SDKs.
 
-This folder is expected to be empty when S3Mock starts. See also FYI above.
-```
-/<root-folder>/
-```
+The [Integration Tests](integration-tests) contain various examples of how to use the S3Mock with the AWS SDK for Java v1 and v2 in Kotlin.
+The modules below [testsupport](testsupport) contain examples in Java.
 
-### Buckets
-Buckets are stored as a folder with their name as created through the S3 API directly below the root:
-```
-/<root-folder>/<bucket-name>/
-```
-[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) is stored in a file in the bucket directory, serialized as JSON.  
-[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) contains the "key" -> "uuid" dictionary for all objects uploaded to this bucket among other data.
-```
-/<root-folder>/<bucket-name>/bucketMetadata.json
-```
-### Objects
+`S3Client` or `S3Presigner` instances are created here:
+* Kotlin: [Integration Test base class](integration-tests/src/test/kotlin/com/adobe/testing/s3mock/its/S3TestBase.kt)
+* Java: [S3Mock unit test starter base class](testsupport/common/src/main/java/com/adobe/testing/s3mock/testsupport/common/S3MockStarter.java)
 
-Objects are stored in folders below the bucket they were created in.
-A folder is created that uses the Object's UUID assigned in the [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) as a name.
-```
-/<root-folder>/<bucket-name>/<uuid>/
-```
-Object data is stored below that UUID folder.
+#### Path-style vs Domain-style access
 
-Binary data is always stored in a file `binaryData`
-```
-/<root-folder>/<bucket-name>/<uuid>/binaryData
-```
+AWS S3 SDKs usually use domain-style access by default. Configuration is needed for path-style access.
 
-[Object metadata](server/src/main/java/com/adobe/testing/s3mock/store/S3ObjectMetadata.java) is serialized as JSON and stored as `objectMetadata.json`
-```
-/<root-folder>/<bucket-name>/<uuid>/objectMetadata.json
+S3Mock currently only supports path-style access (e.g., `http://localhost:9090/bucket/someKey`).
+
+Domain-style access to buckets (e.g., `http://bucket.localhost:9090/someKey`) does not work.
+This is due to the fact that the domain `localhost` is special and does not allow for subdomain access without modifications to the operating system.
+
+#### Presigned URLs
+
+S3 SDKs can be used to [create presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html), the S3 API supports access through those URLs. 
+
+S3Mock will accept presigned URLs, but it *ignores all parameters*.  
+For instance, S3Mock does not verify the HTTP verb that the presigned uri was created with, and it does not validate whether the link is expired or not.
+
+S3 SDKs can be used to create presigned URLs pointing to S3Mock if they're configured for path-style access. See the "Usage of..." section above for links to examples on how to use the SDK with presigned URLs.
+
+### Usage of AWS CLI
+
+S3Mock can be used with the AWS CLI. Setting the `--endpoint-url` enables path-style access.
+
+Examples:
+
+Create bucket 
+```shell
+aws s3api create-bucket --bucket my-bucket --endpoint-url=http://localhost:9090
 ```
 
-[Object ACL](server/src/main/java/com/adobe/testing/s3mock/dto/AccessControlPolicy.java) is serialized as XML and stored as `objectAcl.xml`
+Put object 
+```shell
+aws s3api put-object --bucket my-bucket --key my-file --body ./my-file --endpoint-url=http://localhost:9090
 ```
-/<root-folder>/<bucket-name>/<uuid>/objectAcl.xml
+
+Get object 
+```shell
+aws s3api get-object --bucket my-bucket --key my-file --endpoint-url=http://localhost:9090 my-file-output
 ```
 
-### Multipart Uploads
+### Usage of plain HTTP
 
-Multipart Uploads are created in a bucket using object keys and an uploadId.  
-The object is assigned a UUID within the bucket (stored in [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java)).  
-The [Multipart upload metadata](server/src/main/java/com/adobe/testing/s3mock/store/MultipartUploadInfo.java) is currently not stored on disk.
+As long as the requests work with the S3 API, they will work with S3Mock as well.
 
-The parts folder is created below the object UUID folder named with the `uploadId`:
-```
-/<root-folder>/<bucket-name>/<uuid>/<uploadId>/
-```
+Examples:
 
-Each part is stored in the parts folder with the `partNo` as name and `.part` as a suffix.
+Create bucket
+```shell
+curl --request PUT "http://localhost:9090/my-test-bucket/"
 ```
-/<root-folder>/<bucket-name>/<uuid>/<uploadId>/<partNo>.part
+
+Put object
+```shell
+curl --request PUT --upload-file ./my-file http://localhost:9090/my-test-bucket/my-file
 ```
 
-## Usage
+Get object
+```shell
+curl --request GET http://localhost:9090/my-test-bucket/my-file
+```
 
-### Configuration
+### S3Mock configuration options
 
 The mock can be configured with the following environment variables:
 
@@ -284,6 +293,24 @@ To use the [`S3MockContainer`](testsupport/testcontainers/src/main/java/com/adob
 </dependency>
 ```
 
+#### Start using Docker compose
+
+Create a file `docker-compose.yml`
+
+```yaml
+services:
+  s3mock:
+    image: adobe/s3mock:3.1.0
+    environment:
+      - initialBuckets=bucket1
+    ports:
+      - 9090:9090
+```
+
+Start with `docker compose up -d`
+
+Stop with `docker compose down`
+
 ### S3Mock Java
 
 `S3Mock` Java libraries are released and published to the Sonatype Maven Repository and subsequently published to
@@ -346,7 +373,7 @@ To use the JUnit5 Extension, use the following Maven artifact in `test` scope:
 
 #### Start using the TestNG Listener
 
-The example [`S3MockListenerXMLConfigurationTest`](testsupport/testng/src/test/java/com/adobe/testing/s3mock/testng/S3MockListenerXmlConfigurationTest.java) demonstrates the usage of the `S3MockListener`, which can be configured as shown in [`testng.xml`](testsupport/testng/src/test/resources/testng.xml). The listener bootstraps S3Mock application before TestNG execution starts and shuts down the application just before the execution terminates. Please refer to [`IExecutionListener`](https://jitpack.io/com/github/cbeust/testng/main/javadoc/org/testng/IExecutionListener.html)
+The example [`S3MockListenerXMLConfigurationTest`](testsupport/testng/src/test/java/com/adobe/testing/s3mock/testng/S3MockListenerXmlConfigurationTest.java) demonstrates the usage of the `S3MockListener`, which can be configured as shown in [`testng.xml`](testsupport/testng/src/test/resources/testng.xml). The listener bootstraps the S3Mock application before TestNG execution starts and shuts down the application just before the execution terminates. Please refer to [`IExecutionListener`](https://github.com/testng-team/testng/blob/master/testng-core-api/src/main/java/org/testng/IExecutionListener.java) from the TestNG API.
 
 To use the TestNG Listener, use the following Maven artifact in `test` scope:
 
@@ -371,6 +398,75 @@ Include the following dependency and use one of the `start` methods in `com.adob
 </dependency>
 ```
 
+## File System Structure
+
+S3Mock stores Buckets, Objects, Parts and other data on disk.  
+This lets users inspect the stored data while the S3Mock is running.  
+If the config property `retainFilesOnExit` is set to `true`, this data will not be deleted when S3Mock is shut down.
+
+| :exclamation: FYI                                                                                                                                                                                                                                   |
+|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| While it _may_ be possible to start S3Mock on a root folder from a previous run and have all data available through the S3 API, the structure and contents of the files are not considered Public API, and are subject to change in later releases. |
+| Also, there are no automated test cases for this behaviour.                                                                                                                                                                                         |
+
+### Root-Folder
+S3Mock stores buckets and objects a root-folder.
+
+This folder is expected to be empty when S3Mock starts. See also FYI above.
+```
+/<root-folder>/
+```
+
+### Buckets
+Buckets are stored as a folder with their name as created through the S3 API directly below the root:
+```
+/<root-folder>/<bucket-name>/
+```
+[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) is stored in a file in the bucket directory, serialized as JSON.  
+[BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) contains the "key" -> "uuid" dictionary for all objects uploaded to this bucket among other data.
+```
+/<root-folder>/<bucket-name>/bucketMetadata.json
+```
+### Objects
+
+Objects are stored in folders below the bucket they were created in.
+A folder is created that uses the Object's UUID assigned in the [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java) as a name.
+```
+/<root-folder>/<bucket-name>/<uuid>/
+```
+Object data is stored below that UUID folder.
+
+Binary data is always stored in a file `binaryData`
+```
+/<root-folder>/<bucket-name>/<uuid>/binaryData
+```
+
+[Object metadata](server/src/main/java/com/adobe/testing/s3mock/store/S3ObjectMetadata.java) is serialized as JSON and stored as `objectMetadata.json`
+```
+/<root-folder>/<bucket-name>/<uuid>/objectMetadata.json
+```
+
+[Object ACL](server/src/main/java/com/adobe/testing/s3mock/dto/AccessControlPolicy.java) is serialized as XML and stored as `objectAcl.xml`
+```
+/<root-folder>/<bucket-name>/<uuid>/objectAcl.xml
+```
+
+### Multipart Uploads
+
+Multipart Uploads are created in a bucket using object keys and an uploadId.  
+The object is assigned a UUID within the bucket (stored in [BucketMetadata](server/src/main/java/com/adobe/testing/s3mock/store/BucketMetadata.java)).  
+The [Multipart upload metadata](server/src/main/java/com/adobe/testing/s3mock/store/MultipartUploadInfo.java) is currently not stored on disk.
+
+The parts folder is created below the object UUID folder named with the `uploadId`:
+```
+/<root-folder>/<bucket-name>/<uuid>/<uploadId>/
+```
+
+Each part is stored in the parts folder with the `partNo` as name and `.part` as a suffix.
+```
+/<root-folder>/<bucket-name>/<uuid>/<uploadId>/<partNo>.part
+```
+
 ## Build & Run
 
 To build this project, you need Docker, JDK 17 or higher, and Maven: