Update aws sdk + s3 improvements #4410

icewind1991 · 2017-04-20T15:16:10Z

requires: add aws sdk 3rdparty#51
move ask sdk from files_external/3rdparty to 3rdparty
update aws sdk
adjust files_external s3 and object store s3 to work with the updated sdk
improve performance for s3 external storage

external storage upload performance comparison: blackfire #

Xuanwo · 2017-04-25T03:42:32Z

As we talked in #3910 , is there any plan to build a new app and move aws sdk into it, so that we can support more object storage backend?

icewind1991 · 2017-04-25T12:17:14Z

You can already create an app to add support for an object store backend, simply have the app define the class implementing the backend.

Since the object store config specifies the class to use directly there is no need to do any further "hooking up" of things

codecov · 2017-06-07T15:37:08Z

Codecov Report

Merging #4410 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4410      +/-   ##
============================================
+ Coverage     53.36%   53.38%   +0.01%     
+ Complexity    22576    22535      -41     
============================================
  Files          1409     1410       +1     
  Lines         87338    87229     -109     
  Branches       1340     1340              
============================================
- Hits          46611    46565      -46     
+ Misses        40727    40664      -63

Impacted Files	Coverage Δ	Complexity Δ
lib/private/RichObjectStrings/Validator.php	`78.26% <0%> (-3.89%)`	`13% <0%> (ø)`
lib/private/AppFramework/Http/Request.php	`90.86% <0%> (-1.82%)`	`96% <0%> (-52%)`
lib/private/Group/Group.php	`87.62% <0%> (-1.27%)`	`51% <0%> (-1%)`
apps/files_external/lib/Lib/Storage/AmazonS3.php	`0% <0%> (ø)`	`110% <0%> (+18%)`	⬆️
lib/private/legacy/template/functions.php	`11.11% <0%> (ø)`	`0% <0%> (ø)`	⬇️
...te/Authentication/LoginCredentials/Credentials.php	`100% <0%> (ø)`	`2% <0%> (-2%)`	⬇️
...s/files_external/lib/Lib/Storage/StreamWrapper.php	`0% <0%> (ø)`	`14% <0%> (-5%)`	⬇️
lib/private/Files/ObjectStore/S3ObjectTrait.php	`85.71% <0%> (ø)`	`3% <0%> (?)`
lib/private/Security/IdentityProof/Signer.php	`80.64% <0%> (+0.64%)`	`7% <0%> (ø)`	⬇️
lib/private/Files/ObjectStore/S3.php	`50% <0%> (+50%)`	`2% <0%> (-3%)`	⬇️
... and 1 more

icewind1991 · 2017-06-07T15:43:32Z

This is ready for review imo

nickvergessen · 2017-06-08T12:54:02Z

hmm couldnt the deps be in the files_externals composer instead of the global one?

icewind1991 · 2017-06-09T10:28:38Z

I moved them to the global one since it is also used by the object store backend which is in core

MorrisJobke · 2017-06-19T21:00:26Z

@icewind1991 Could you please fix the updated 3rdparty

icewind1991 · 2017-06-20T11:26:48Z

rebased

MorrisJobke · 2017-08-01T11:43:36Z

I resolved the conflict and updated the 3rdparty module.

MorrisJobke · 2017-08-01T11:52:40Z

I get this when I run the installation:

{
    "app": "files_external",
    "level": 3,
    "message": {
    	"Exception":"Aws\\S3\\Exception\\S3Exception",
    	"Message":"Error executing \"CreateBucket\" on \"http://abc.127.0.0.1:4567/\"; AWS HTTP error: cURL error 6: Couldn't resolve host name",
    	"Code":0,
    	"Trace":"
    		#0 3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\\Psr7\\Request), Object(Aws\\Command), Array)
    		#1 3rdparty/guzzlehttp/promises/src/Promise.php(203): Aws\\WrappedHttpHandler->Aws\\{closure}(Array)
    		#2 3rdparty/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\\Promise\\Promise::callHandler(2, Array, Array)
    		#3 3rdparty/guzzlehttp/promises/src/RejectedPromise.php(40): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}(Array)
    		#4 3rdparty/guzzlehttp/promises/src/TaskQueue.php(47): GuzzleHttp\\Promise\\RejectedPromise::GuzzleHttp\\Promise\\{closure}()
    		#5 3rdparty/guzzlehttp/promises/src/Promise.php(234): GuzzleHttp\\Promise\\TaskQueue->run()
    		#6 3rdparty/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()
    		#7 3rdparty/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()
    		#8 3rdparty/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()
    		#9 3rdparty/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()
    		#10 3rdparty/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()
    		#11 3rdparty/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()
    		#12 3rdparty/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
    		#13 3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(59): GuzzleHttp\\Promise\\Promise->wait()
    		#14 3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(78): Aws\\AwsClient->execute(Object(Aws\\Command))
    		#15 lib/private/Files/ObjectStore/S3ConnectionTrait.php(102): Aws\\AwsClient->__call('createBucket', Array)
    		#16 lib/private/Files/ObjectStore/S3ObjectTrait.php(63): OC\\Files\\ObjectStore\\S3->getConnection()
    		#17 lib/private/Files/ObjectStore/ObjectStoreStorage.php(367): OC\\Files\\ObjectStore\\S3->writeObject('urn:oid:4', Resource id #778)
    		#18 lib/private/Files/Storage/Wrapper/Wrapper.php(350): OC\\Files\\ObjectStore\\ObjectStoreStorage->touch('files/welcome.t...', 1501588075)
    		#19 lib/private/Files/Storage/Wrapper/Availability.php(375): OC\\Files\\Storage\\Wrapper\\Wrapper->touch('files/welcome.t...', NULL)
    		#20 lib/private/Files/Storage/Wrapper/Wrapper.php(350): OC\\Files\\Storage\\Wrapper\\Availability->touch('files/welcome.t...', NULL)
    		#21 lib/private/Files/View.php(1127): OC\\Files\\Storage\\Wrapper\\Wrapper->touch('files/welcome.t...')
    		#22 lib/private/Files/View.php(550): OC\\Files\\View->basicOperation('touch', '/admin/files/we...', Array, NULL)
    		#23 lib/private/Files/Node/Folder.php(181): OC\\Files\\View->touch('/admin/files/we...')
    		#24 lib/private/legacy/util.php(425): OC\\Files\\Node\\Folder->newFile('welcome.txt')
    		#25 lib/private/legacy/util.php(395): OC_Util::copyr('/srv/projects/s...', Object(OC\\Files\\Node\\Folder))
    		#26 lib/private/User/Session.php(486): OC_Util::copySkeleton('admin', Object(OC\\Files\\Node\\Folder))
    		#27 lib/private/User/Session.php(361): OC\\User\\Session->prepareUserLogin(true)
    		#28 lib/private/User/Session.php(547): OC\\User\\Session->completeLogin(*** sensitive parameters replaced ***)
    		#29 lib/private/User/Session.php(326): OC\\User\\Session->loginWithPassword(*** sensitive parameters replaced ***)
    		#30 lib/private/Setup.php(406): OC\\User\\Session->login(*** sensitive parameters replaced ***)
    		#31 core/Command/Maintenance/Install.php(92): OC\\Setup->install(Array)
    		#32 3rdparty/symfony/console/Command/Command.php(256): OC\\Core\\Command\\Maintenance\\Install->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
    		#33 3rdparty/symfony/console/Application.php(818): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
    		#34 3rdparty/symfony/console/Application.php(186): Symfony\\Component\\Console\\Application->doRunCommand(Object(OC\\Core\\Command\\Maintenance\\Install), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
    		#35 3rdparty/symfony/console/Application.php(117): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
    		#36 lib/private/Console/Application.php(170): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
    		#37 console.php(92): OC\\Console\\Application->run()
    		#38 occ(11): require_once('/srv/projects/s...')
    		#39 {main}",
    	"File":"3rdparty/aws/aws-sdk-php/src/WrappedHttpHandler.php",
    	"Line":192
    },
    "method": "--",
    "remoteAddr": "",
    "reqId": "6L06558ZNHfIqjOZYMY9",
    "time": "2017-08-01T11:47:57+00:00",
    "url": "--",
    "user": "admin",
    "userAgent": "--",
    "version": "13.0.0.2"
}

This is the config:

'objectstore' =>
		array (
			'class' => 'OC\\Files\\ObjectStore\\S3',
			'arguments' =>
				array (
					'bucket' => 'abc',
					'key' => '123',
					'secret' => 'abc',
					'hostname' => '127.0.0.1',
					'port' => '4567',
					'use_ssl' => false,
					'use_path_style' => 'true',
				),
		),

And I'm using fakeS3

icewind1991 · 2017-08-31T13:29:16Z

@MorrisJobke should be fixed

MorrisJobke

And please also update the autoloader then this can be merged.

MorrisJobke · 2017-09-18T12:04:42Z

apps/files_external/lib/Lib/Storage/AmazonS3.php

@@ -394,7 +456,7 @@ public function touch($path, $mtime = null) {
 		$fileType = $this->filetype($path);


Line 453: phan says:

apps/files_external/lib/Lib/Storage/AmazonS3.php:453 PhanUndeclaredClassConstant Reference to constant RFC1123 from undeclared class \Aws\Common\Enum\DateFormat

Signed-off-by: Robin Appelman <robin@icewind.nl>

karlitschek · 2017-09-19T12:08:41Z

Can we backport this?

icewind1991 added the 2. developing Work in progress label Apr 20, 2017

icewind1991 added this to the Nextcloud 12.0 milestone Apr 20, 2017

icewind1991 mentioned this pull request Apr 20, 2017

Upgrade internal S3 library #2315

Closed

MorrisJobke modified the milestones: Nextcloud 13, Nextcloud 12.0 May 4, 2017

icewind1991 force-pushed the update-aws-sdk branch from ca44024 to 1a401b6 Compare June 7, 2017 15:36

icewind1991 added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Jun 7, 2017

icewind1991 changed the title ~~[wip] Update aws sdk~~ Update aws sdk + s3 improvements Jun 7, 2017

icewind1991 mentioned this pull request Jun 7, 2017

add aws sdk nextcloud/3rdparty#51

Merged

icewind1991 requested review from rullzer, nickvergessen and MorrisJobke June 7, 2017 15:43

icewind1991 force-pushed the update-aws-sdk branch from 1a401b6 to 7acacd8 Compare June 20, 2017 11:23

icewind1991 force-pushed the update-aws-sdk branch from 7acacd8 to cc3a499 Compare July 21, 2017 13:21

MorrisJobke force-pushed the update-aws-sdk branch from cc3a499 to 905d63e Compare August 1, 2017 11:43

MorrisJobke added 2. developing Work in progress and removed 3. to review Waiting for reviews labels Aug 1, 2017

icewind1991 force-pushed the update-aws-sdk branch from 905d63e to f5b0a8b Compare August 31, 2017 13:10

MorrisJobke force-pushed the update-aws-sdk branch from bf03293 to 1853959 Compare September 18, 2017 09:25

MorrisJobke added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Sep 18, 2017

MorrisJobke requested changes Sep 18, 2017

View reviewed changes

MorrisJobke added 2. developing Work in progress 3. to review Waiting for reviews and removed 4. to release Ready to be released and/or waiting for tests to finish 2. developing Work in progress labels Sep 18, 2017

icewind1991 added 11 commits September 18, 2017 15:16

remove non composer aws sdk

cf1da57

Signed-off-by: Robin Appelman <robin@icewind.nl>

update aws sdk and move it to 3rdparty

dad18ba

Signed-off-by: Robin Appelman <robin@icewind.nl>

reuse object read/write/delete logic in s3 implementations

d706071

Signed-off-by: Robin Appelman <robin@icewind.nl>

more efficient write only fopen

9b5accf

Signed-off-by: Robin Appelman <robin@icewind.nl>

more efficient permissions for s3 external storage

4bbc1cd

Signed-off-by: Robin Appelman <robin@icewind.nl>

more efficient is_dir

d567820

Signed-off-by: Robin Appelman <robin@icewind.nl>

add stat cache for s3 external storage

cc9dd42

Signed-off-by: Robin Appelman <robin@icewind.nl>

disable part files for s3 external storage

9154b27

Signed-off-by: Robin Appelman <robin@icewind.nl>

fix path style

d526969

Signed-off-by: Robin Appelman <robin@icewind.nl>

merge vendor js

3eaf662

Signed-off-by: Robin Appelman <robin@icewind.nl>

fix undefined constants

91f44f5

Signed-off-by: Robin Appelman <robin@icewind.nl>

icewind1991 force-pushed the update-aws-sdk branch from 1b21238 to 3bf1495 Compare September 18, 2017 13:16

icewind1991 added 2 commits September 18, 2017 17:20

update autoloader

e7f3c12

Signed-off-by: Robin Appelman <robin@icewind.nl>

show diff in autoload checker

f0c7b8f

Signed-off-by: Robin Appelman <robin@icewind.nl>

icewind1991 force-pushed the update-aws-sdk branch from e9583a5 to f0c7b8f Compare September 18, 2017 15:20

icewind1991 merged commit 1995c69 into master Sep 18, 2017

icewind1991 deleted the update-aws-sdk branch September 18, 2017 15:47

icewind1991 mentioned this pull request Oct 3, 2017

[12] Update aws sdk + s3 improvements #6737

Merged

1 task

This was referenced Dec 18, 2017

Support of Swift V3 Auth aka Identity v3 API for Swift Object Storage #1504

Closed

Upgrade PHP-opencloud to support Swift DLO #2474

Closed

Nextcloud 12.0.4 and 13 doesn't support S3 Backend using V2 authentication #7569

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update aws sdk + s3 improvements #4410

Update aws sdk + s3 improvements #4410

icewind1991 commented Apr 20, 2017 •

edited by MorrisJobke

Loading

Xuanwo commented Apr 25, 2017

icewind1991 commented Apr 25, 2017

codecov bot commented Jun 7, 2017 •

edited

Loading

icewind1991 commented Jun 7, 2017

nickvergessen commented Jun 8, 2017

icewind1991 commented Jun 9, 2017

MorrisJobke commented Jun 19, 2017

icewind1991 commented Jun 20, 2017

MorrisJobke commented Aug 1, 2017

MorrisJobke commented Aug 1, 2017

icewind1991 commented Aug 31, 2017

MorrisJobke left a comment

MorrisJobke Sep 18, 2017

karlitschek commented Sep 19, 2017

		@@ -394,7 +456,7 @@ public function touch($path, $mtime = null) {
		$fileType = $this->filetype($path);

Update aws sdk + s3 improvements #4410

Update aws sdk + s3 improvements #4410

Conversation

icewind1991 commented Apr 20, 2017 • edited by MorrisJobke Loading

Xuanwo commented Apr 25, 2017

icewind1991 commented Apr 25, 2017

codecov bot commented Jun 7, 2017 • edited Loading

Codecov Report

icewind1991 commented Jun 7, 2017

nickvergessen commented Jun 8, 2017

icewind1991 commented Jun 9, 2017

MorrisJobke commented Jun 19, 2017

icewind1991 commented Jun 20, 2017

MorrisJobke commented Aug 1, 2017

MorrisJobke commented Aug 1, 2017

icewind1991 commented Aug 31, 2017

MorrisJobke left a comment

Choose a reason for hiding this comment

MorrisJobke Sep 18, 2017

Choose a reason for hiding this comment

karlitschek commented Sep 19, 2017

icewind1991 commented Apr 20, 2017 •

edited by MorrisJobke

Loading

codecov bot commented Jun 7, 2017 •

edited

Loading