-
Notifications
You must be signed in to change notification settings - Fork 24
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #70 from carlspring/issue-22
Issue-22: Set up a project documentation site using mkdocs and publish it to github.io
- Loading branch information
Showing
46 changed files
with
2,563 additions
and
183 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,6 +4,7 @@ target/ | |
tmp/ | ||
*.versionsBackup | ||
.m2/ | ||
site/ | ||
|
||
# Java Files # | ||
*.jar | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
For our code of conduct, please check [here](https://s3fs.carlspring.org/contributing/code-of-conduct.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
For details on how to contribute, please check [here](https://s3fs.carlspring.org/contributing/index.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,196 +1,57 @@ | ||
An **Amazon AWS S3** FileSystem Provider **JSR-203** for Java 7 (NIO2) | ||
# General | ||
|
||
Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any amount of data, at any time. | ||
NIO2 is the new file management API, introduced in Java version 7. | ||
This project provides a first API implementation, little optimized, but "complete" to manage files and folders directly on Amazon S3. | ||
This is an implementation of an **Amazon AWS S3** `FileSystem` provider using **[JSR-203]** (a.k.a. NIO2) for Java 8. | ||
|
||
[![Build Status](https://travis-ci.org/Upplication/Amazon-S3-FileSystem-NIO2.svg?branch=master)](https://travis-ci.org/Upplication/Amazon-S3-FileSystem-NIO2/builds) [![Coverage Status](https://coveralls.io/repos/Upplication/Amazon-S3-FileSystem-NIO2/badge.png?branch=master)](https://coveralls.io/r/Upplication/Amazon-S3-FileSystem-NIO2?branch=master) [![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.upplication/s3fs/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.upplication/s3fs) | ||
Amazon Simple Storage Service provides a fully redundant data storage infrastructure for storing and retrieving any | ||
amount of data, at any time. | ||
|
||
#### How to use | ||
[NIO2] is the new file management API, introduced in Java version 7. | ||
|
||
##### Download from Maven Central | ||
This project provides a complete API implementation, for managing files and folders directly in Amazon S3. | ||
|
||
```XML | ||
<dependency> | ||
<groupId>com.upplication</groupId> | ||
<artifactId>s3fs</artifactId> | ||
<version>2.2.2</version> | ||
</dependency> | ||
``` | ||
[![Master Build Status][master-build-status-badge]][master-build-status-link] | ||
[![Docs][master-docs-badge]][master-docs-link] | ||
[![License][license-badge]][license-link] | ||
[![Help Contribute to Open Source][codetriage-badge]][codetriage-link] | ||
[![GitHub issues by-label][good-first-issue-badge]][good-first-issue-link] | ||
[![GitHub issues by-label][help-wanted-badge]][help-wanted-link] | ||
[![GitHub issues by-label][hacktoberfest-badge]][hacktoberfest-link] | ||
[![GitHub issues by-label][stackoverflow-badge]][stackoverflow-link] | ||
|
||
# Compatibility | ||
|
||
And add to your META-INF/services/java.nio.file.spi.FileSystemProvider (create if not exists yet) a new line like this: com.upplication.s3fs.S3FileSystemProvider. | ||
We support both JDK 8 and 11. | ||
|
||
##### S3FileSystem and AmazonS3 settings | ||
## Documentation | ||
|
||
All settings for S3FileSystem and for the underlying AmazonS3 connector library can be set through System properties or environment variables. | ||
Possible settings can be found in com.upplication.s3fs.AmazonS3Factory. | ||
You can check out our documentation [here](https://s3fs.carlspring.org). | ||
|
||
#### Using service locator and system vars | ||
[<--# Links -->]: # | ||
|
||
Check that s3fs_access_key and s3fs_secret_key system vars are present with the correct values to have full access to your amazon s3 bucket. | ||
[NIO2]: https://jcp.org/en/jsr/detail?id=203 | ||
[JSR-203]: https://jcp.org/en/jsr/detail?id=203 | ||
|
||
Use this code to create the fileSystem and set to a concrete endpoint. | ||
[<--# Badges -->]: # | ||
[master-build-status-link]: https://github.com/carlspring/s3fs-nio2/actions?query=branch%3Amaster | ||
[master-build-status-badge]: https://github.com/carlspring/s3fs-nio2/workflows/Build%20and%20test%20workflow/badge.svg | ||
|
||
```java | ||
FileSystems.newFileSystem("s3:///", new HashMap<String,Object>(), Thread.currentThread().getContextClassLoader()); | ||
``` | ||
[master-docs-link]: https://carlspring.github.io/s3fs-nio2 | ||
[master-docs-badge]: https://img.shields.io/badge/docs-current-brightgreen.svg | ||
|
||
##### Using service locator and amazon.properties in the classpath | ||
[license-link]: https://opensource.org/licenses/Apache-2.0 | ||
[license-badge]: https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg | ||
|
||
Add to your resources folder the file amazon.properties with the content: | ||
s3fs_access_key=access-key | ||
s3fs_secret_key=secret-key | ||
[codetriage-link]: https://www.codetriage.com/carlspring/s3fs-nio2 | ||
[codetriage-badge]: https://www.codetriage.com/carlspring/s3fs-nio2/badges/users.svg | ||
|
||
Use this code to create the fileSystem and set to a concrete endpoint. | ||
[good-first-issue-link]: https://github.com/carlspring/s3fs-nio2/issues?q=is%3Aissue+is%3Aopen+label%3A%22good%20first%20issue%22 | ||
[good-first-issue-badge]: https://img.shields.io/github/issues-raw/carlspring/s3fs-nio2/good%20first%20issue.svg?label=good%20first%20issue | ||
|
||
```java | ||
FileSystems.newFileSystem("s3:///", new HashMap<String,Object>(), Thread.currentThread().getContextClassLoader()); | ||
``` | ||
[help-wanted-link]: https://github.com/carlspring/s3fs-nio2/issues?q=is%3Aissue+is%3Aopen+label%3A%22help%20wanted%22 | ||
[help-wanted-badge]: https://img.shields.io/github/issues-raw/carlspring/s3fs-nio2/help%20wanted.svg?label=help%20wanted&color=%23856bf9& | ||
|
||
##### Using service locator and programatically authentication | ||
[hacktoberfest-link]: https://github.com/carlspring/s3fs-nio2/issues?q=is%3Aissue+is%3Aopen+label%3A%22hacktoberfest%22 | ||
[hacktoberfest-badge]: https://img.shields.io/github/issues-raw/carlspring/s3fs-nio2/hacktoberfest.svg?label=hacktoberfest&color=orange | ||
|
||
Create a map with the authentication and use the fileSystem to create the fileSystem and set to a concrete endpoint. | ||
|
||
```java | ||
Map<String, ?> env = ImmutableMap.<String, Object> builder() | ||
.put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY, "access key") | ||
.put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY, "secret key").build() | ||
FileSystems.newFileSystem("s3:///", env, Thread.currentThread().getContextClassLoader()); | ||
``` | ||
|
||
Complete settings lists: | ||
|
||
* s3fs_access_key | ||
* s3fs_secret_key | ||
* s3fs_request_metric_collector_class | ||
* s3fs_connection_timeout | ||
* s3fs_max_connections | ||
* s3fs_max_retry_error | ||
* s3fs_protocol | ||
* s3fs_proxy_domain | ||
* s3fs_proxy_host | ||
* s3fs_proxy_password | ||
* s3fs_proxy_port | ||
* s3fs_proxy_username | ||
* s3fs_proxy_workstation | ||
* s3fs_socket_send_buffer_size_hint | ||
* s3fs_socket_receive_buffer_size_hint | ||
* s3fs_socket_timeout | ||
* s3fs_user_agent | ||
* s3fs_amazon_s3_factory | ||
* s3fs_signer_override | ||
* s3fs_path_style_access | ||
|
||
##### Set endpoint to reduce data latency in your applications | ||
|
||
```java | ||
// Northern Virginia or Pacific Northwest | ||
FileSystems.newFileSystem("s3://s3.amazonaws.com/", env, Thread.currentThread().getContextClassLoader()); | ||
// Northern Virginia only | ||
FileSystems.newFileSystem("s3://s3-external-1.amazonaws.com/", env, Thread.currentThread().getContextClassLoader()); | ||
// US West (Oregon) Region | ||
FileSystems.newFileSystem("s3://s3-us-west-2.amazonaws.com/", env, Thread.currentThread().getContextClassLoader()); | ||
// US West (Northern California) Region | ||
FileSystems.newFileSystem("s3://s3-us-west-1.amazonaws.com/", env, Thread.currentThread().getContextClassLoader()); | ||
// EU (Ireland) Region | ||
FileSystems.newFileSystem("s3://s3-eu-west-1.amazonaws.com/", env, Thread.currentThread().getContextClassLoader()); | ||
``` | ||
|
||
For a complete list of available regions look at: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region | ||
|
||
##### How to use in Apache MINA | ||
|
||
```java | ||
public FileSystemFactory createFileSystemFactory(String bucketName) throws IOException, URISyntaxException { | ||
FileSystem fileSystem = FileSystems.newFileSystem(new URI("s3:///"), env, Thread.currentThread().getContextClassLoader()); | ||
String bucketPath = fileSystem.getPath("/" + bucketName); | ||
|
||
return new VirtualFileSystemFactory(bucketPath); | ||
} | ||
``` | ||
|
||
##### How to use in Spring | ||
|
||
Add to classpath and configure: | ||
|
||
```java | ||
@Configuration | ||
public class AwsConfig { | ||
|
||
@Value("${upplication.aws.accessKey}") | ||
private String accessKey; | ||
|
||
@Value("${upplication.aws.secretKey}") | ||
private String secretKey; | ||
|
||
@Bean | ||
public FileSystem s3FileSystem() throws IOException { | ||
Map<String, String> env = new HashMap<>(); | ||
env.put(com.upplication.s3fs.AmazonS3Factory.ACCESS_KEY, accessKey); | ||
env.put(com.upplication.s3fs.AmazonS3Factory.SECRET_KEY, secretKey); | ||
|
||
return FileSystems.newFileSystem(URI.create("s3:///"), env, Thread.currentThread().getContextClassLoader()); | ||
} | ||
} | ||
``` | ||
|
||
Now you can inject in any spring component: | ||
|
||
```java | ||
@Autowired | ||
private FileSystem s3FileSystem; | ||
``` | ||
|
||
##### What is new 2.0.0 | ||
|
||
* Preserve URI with end slash #76 | ||
* Removed META-INF/services/java.nio.file.spi.FileTypeDetector #78 | ||
* Bucket are filestores and root directories for a bucket is the bucket itself. | ||
* getFileName for a root Path is ```null``` | ||
* Added new method toURL to S3Path and can be customized with the property s3fs_path_style_access #83 | ||
* Improved S3Path Tests | ||
|
||
#### Features: | ||
|
||
* Copy and create folders and files | ||
* Delete folders and files | ||
* Copy paths between different providers | ||
* Walk file tree | ||
* Works with virtual s3 folders (not really exists and are element's subkeys) | ||
* List buckets for the client | ||
* Multi endpoint fileSystem | ||
|
||
#### Roadmap: | ||
|
||
* Performance issue (slow querys with virtual folders, add multipart submit...) | ||
* Disallow upload binary files with same name as folders and vice versa | ||
|
||
#### Out of Roadmap: | ||
|
||
* Watchers | ||
|
||
#### How to contribute | ||
|
||
Clone the github repository: | ||
|
||
```java | ||
git clone https://github.com/Upplication/Amazon-S3-FileSystem-NIO2.git | ||
``` | ||
|
||
To run the tests: | ||
|
||
First, you must copy the file `src/test/resources/amazon-test-sample.properties` and paste in the same directory with the name amazon-test.properties. In your copy you must edit all the keys: | ||
|
||
``` | ||
bucket_name=/your-bucket-name for test | ||
# http://docs.aws.amazon.com/general/latest/gr/rande.html | ||
s3fs_secret_key= your secret key for test | ||
s3fs_access_key=your access key for test | ||
``` | ||
|
||
Thats all, now you can run the test with the command: `mvn test` or `mvn integration-test -Pintegration-tests` | ||
|
||
#### LICENSE: | ||
|
||
Amazon S3 FileSystem NIO2 is released under the MIT License. | ||
[stackoverflow-link]: https://stackoverflow.com/tags/s3fs-nio2/ | ||
[stackoverflow-badge]: https://img.shields.io/badge/stackoverflow-ask-orange.svg |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
## Documentation | ||
|
||
This directory contains the S3FS NIO2 documentation. | ||
|
||
## Pre-requisites | ||
|
||
* [Docker][docker-install] | ||
* [Docker Compose][docker-compose-install] | ||
|
||
## Getting started | ||
|
||
1. `docker-compose up` | ||
2. Open [`http://localhost:8000`](http://localhost:8000) | ||
|
||
## Debugging | ||
|
||
1. Build image | ||
``` | ||
docker-compose build | ||
``` | ||
|
||
2. Log into the container | ||
``` | ||
docker-compose run --rm -p 8000:8000 -v $(pwd)/../:/workspace --entrypoint /bin/sh mkdocs | ||
``` | ||
|
||
2. Manually start the server (i.e. test plugins, cwd should be ./docs!) | ||
``` | ||
mkdocs serve -a 0.0.0.0:8000 | ||
``` | ||
|
||
## Notes | ||
|
||
* Imported snippets (`--8<-- "./filename"`) are using relative paths to `./docs` | ||
(i.e. `./content/something` would be searched for in `./docs/content/something`) | ||
|
||
## Used tools and extensions | ||
|
||
| Tool | Documentation | Sources | | ||
| ------------------------------- | ------------------------------------- | ----------------------------------- | | ||
| mkdocs | [documentation][mkdocs] | [Sources][mkdocs-src] | | ||
| mkdocs-material | [documentation][mkdocs-material] | [Sources][mkdocs-material-src] | | ||
| pymdown-extensions | [documentation][pymdown-extensions] | [Sources][pymdown-extensions-src] | | ||
| mdx_gh_links | [documentation][mdx_gh_links] | [Sources][mdx_gh_links] | | ||
| mkdocs-redirects | [documentation][mkdocs-redirects] | [Sources][mkdocs-redirects] | | ||
| mkdocs-markdownextradata-plugin | [documentation][mkdocs-markdownextradata-plugin] | [Sources][mkdocs-markdownextradata-plugin] | | ||
| mkdocs-pom-parser-plugin | [documentation][mkdocs-pom-parser-plugin] | [Sources][mkdocs-pom-parser-plugin] | | ||
| mkdocs-minify-plugin | [documentation][mkdocs-minify-plugin] | [Sources][mkdocs-minify-plugin] | | ||
|
||
|
||
[docker-install]: https://docs.docker.com/get-docker/ "Docker Installation" | ||
[docker-compose-install]: https://docs.docker.com/compose/install/ "Docker Compose Installation" | ||
|
||
[mkdocs]: https://www.mkdocs.org "Mkdocs" | ||
[mkdocs-src]: https://github.com/mkdocs/mkdocs "Mkdocs - Sources" | ||
|
||
[mkdocs-material]: https://squidfunk.github.io/mkdocs-material/ "Material for MkDocs" | ||
[mkdocs-material-src]: https://github.com/squidfunk/mkdocs-material "Material for MkDocs - Sources" | ||
|
||
[pymdown-extensions]: https://facelessuser.github.io/pymdown-extensions "PyMdown Extensions" | ||
[pymdown-extensions-src]: https://github.com/facelessuser/pymdown-extensions "PyMdown Extensions - Sources" | ||
|
||
|
||
[mdx_gh_links]: https://github.com/Python-Markdown/github-links/ | ||
[mkdocs-redirects]: https://github.com/datarobot/mkdocs-redirects | ||
[mkdocs-markdownextradata-plugin]: https://github.com/rosscdh/mkdocs-markdownextradata-plugin | ||
[mkdocs-pom-parser-plugin]: https://github.com/steve-todorov/mkdocs-pom-parser-plugin "A pom.xml file parser for mkdocs" | ||
|
||
[mkdocs-minify-plugin]: https://github.com/byrnereese/mkdocs-minify-plugin |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.