Fix typo that cause AWS auth to fail. #7

acinader · 2016-07-20T14:29:34Z

No description provided.

acinader · 2016-07-20T14:29:46Z

PS its now running......

flovilmart · 2016-07-20T14:30:04Z

Oh! That's scary! You sure you should do that that early in the morning ?

acinader · 2016-07-20T14:30:30Z

meh

JeremyPlease · 2016-07-20T17:31:06Z

@acinader Thanks for patching up all my code! 😀

acinader · 2016-07-20T17:35:30Z

team effort ftw!

flovilmart · 2016-07-20T17:36:03Z

BTW how is it going on your dev app?

acinader · 2016-07-20T17:47:10Z

@flovilmart glad you asked.

-- it ran all the way through, so that is good.
-- when i tested the app, i got 403 on all of the images
-- Here's the steps of my test and why i think that it failed, and what i am going to test next.

I created my test environment by

dump db and create new test db from prod data
use parse.com to clone the app (dev)
run file-util with cloned (dev) file, master, app keys. fails with a 403 on each image
switch to using prod file, master and app key for file-util, but dev db
it runs through no problem
run parse server using dev keys. fail with 403 on images
run parse server using prod keys but dev db. fails with 403 on images.

Good news I didn't screw up anything irreversibly!

So I deleted the dev db and I am now loading a new one with fresh data so I can test again.

I really have no idea how file, master, app key interact with the file/filename in the db or what really happened, so I am going to have to put some time into grokking it.

In any event, I'm pretty sure this is all just user error?

flovilmart · 2016-07-20T17:53:18Z

403 on the images? still hosted on parse.com or uploaded on S3? maybe that's your bucket read policy that's borked?

Before running the script, we should maybe add a small file on the adatper, and try to fetch it without the access keys to ensure they are publicly accessible?

acinader · 2016-07-20T18:03:42Z

uploaded to s3. i deleted everything (hasty.) will try again and report
back. it shouldn't be the read policy since its using the s3Adapter and
that is working just fine for parse-server. but if it fails again, I'll
investigate better this time.

On Wed, Jul 20, 2016 at 1:53 PM, Florent Vilmart notifications@github.com
wrote:

403 on the images? still hosted on parse.com or uploaded on S3? maybe
that's your bucket read policy that's borked?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAqwnIjUCTaTKHQpw4xhoXlSYKFPuKcqks5qXmCPgaJpZM4JQ1iN
.

Arthur Cinader Jr.
Founder & EVP, Product Development
arthur@popsugar.com
415.645.3545

111 Sutter St.
San Francisco, CA 94104

flovilmart · 2016-07-20T18:06:01Z

where did you get the 403? From parse-server? what were the URL's?

acinader · 2016-07-20T18:12:04Z

no, from s3.

http://files.parsetfss.com/7aeac95f-b7f4-41fd-9174-8f4a0f77bc53/tfss-d5902a51-af87-4349-8968-40ae33b29c7a-70C57CBF-120C-42AC-9627-9855F1800328.jpg

ok, and as i am looking at it now, i see the file key in there. I think my
setup was the problem. adding the additional parse.com app instance with
its keys was all unnecessary. All i need is a seperate db with the same
keys.

id suggest you not waste your time thinking about it, i think i see the
problem.....

On Wed, Jul 20, 2016 at 2:06 PM, Florent Vilmart notifications@github.com
wrote:

where did you get the 403? From parse-server? what were the URL's?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAqwnKaWgqgRL89b8nv19xcHLGwpiwQNks5qXmOJgaJpZM4JQ1iN
.

Arthur Cinader Jr.
Founder & EVP, Product Development
arthur@popsugar.com
415.645.3545

111 Sutter St.
San Francisco, CA 94104

JeremyPlease · 2016-07-20T18:12:13Z

@acinader
Did you use the "clone app" functionality in the Parse hosted dashboard? If so, that actually doesn't transfer files to the new app. I ran into this a long time ago, but api.parse.com attempts to serve the files in the new cloned app with a different file key which then results in 403.

Take this url as an example
http://files.parsetfss.com/75b8d55e-cb49-4169-8aa8-51c3252d7031/tfss-59bbc9c1-34ee-4ed2-b9b9-c889929b548e-photo.jpeg

The url parts are broken into:
files.parsetfss.com/FILE_KEY/tfss-UNIQUE_KEY-FILENAME
In the new cloned app the FILE_KEY changes in the served file urls but parse.com doesn't actually copy the files to the new path.

flovilmart · 2016-07-20T18:17:04Z

id suggest you not waste your time thinking about it, i think i see the
problem.....

acinader · 2016-07-20T18:23:04Z

@JeremyPlease yeah, that's it all right, and i kinda saw what was going on, but didn't realize the implications.

OK, so i checked the source code and I'm pretty sure that I'm safe, but one question before i f'up my world ;)

in the config, i use parse.com as the server url, but it is never used for writes.
all writing is done to the mongo db uri. the parse server is just used to get the schema.

so i am ok using my master and app id keys, nothing will get changed.

the ONLY thing I have to be careful of is that I am using a backup db and not our prod db.

I'm like 99.9999% sure i've got this right, but.... ;) i.e. the only Parse. string is for getting the schema.

flovilmart · 2016-07-20T18:27:05Z

Maybe one last thingy thing... The Content-Type of the files? is that set anywhere?

JeremyPlease · 2016-07-20T18:28:55Z

@acinader I'll give you that extra 0.0001% 😉

Your understanding is absolutely right. The applicationId, masterKey, and serverUrl are all only used for reads (fetching the schema and all objects). The mongoURL is used for all writes. So, as long as you put in a backup db there, then you're good to go (I used a bunch of localhost mongo urls in my testing).

@flovilmart The content type is read from server response when fetching the file and then passed as 3rd argument in filesAdapter.createFile.

flovilmart · 2016-07-20T18:29:19Z

👍

flovilmart · 2016-07-20T18:29:39Z

@JeremyPlease would be awesome to add tests on that :)

JeremyPlease · 2016-07-20T18:30:30Z

Tests would indeed be awesome 😁 Maybe this weekend.

flovilmart · 2016-07-20T18:36:30Z

Just before we release a proper NPM package.

acinader · 2016-07-20T18:38:37Z

ok. running again. optimistic. will take a couple of hours.

Processing 637/27956

flovilmart · 2016-07-20T19:08:12Z

I'll open a branch for the ci configuration and code coverage, so we'll be all set

acinader · 2016-07-20T20:53:51Z

ok, so it ran, and all image requests are still failing with a 403.

BUT! all of the images are on s3 with the correct content-type and acl.

so that's good.

when i look at the records in dashboard (my own dashboard running on my box against my parse-server), the url's for files look like this:

http://files.parse.com/7aeac95f-b7f4-41fd-9174-8f4a0f77bc53/bec56de9-bd4f-48ac-9018-d3695b18cebf-FA9F6705-5905-4C31-8AAA-D9A5694E885C.jpg

and they 403.

BUT!!!!

https://parse-image-bucket.s3.amazonaws.com/epic-dev/bec56de9-bd4f-48ac-9018-d3695b18cebf-FA9F6705-5905-4C31-8AAA-D9A5694E885C.jpg

works just fine, so it is clearly just MY config error

i've just started poking around in the s3 file adapter, but i have run out of time for today. Seems like it is certainly a config of my parse-server or s3FileAdapter that I don't have right. I'm confident i'll track it down in the morning as I see the critical section, I just ran out of time. Wanted to give you feedback before I check out for the day, but I don't need any help at this time.
it looks like there may be a bug, but i haven't tested it yet (maybe it works, but looks wrong to me - in any event, i'll test it in the morning):
https://github.com/parse-server-modules/parse-server-s3-adapter/blob/master/index.js#L169

thanks for the help guys.

JeremyPlease · 2016-07-20T21:12:39Z

@acinader
I'm pretty sure that you're actually experiencing an issue with the parse-file-utils renaming files.

https://github.com/ParsePlatform/parse-server/blob/master/src/Controllers/FilesController.js#L63-L76

I wrote the _nonParseFileName function in transfer.js to simply remove "tfss-" from the filename. However, it looks like that results in a filename that matches the legacy file regex.

Example:

original url: http://files.parsetfss.com/75b8d55e-cb49-4169-8aa8-51c3252d7031/tfss-0e4d6293-5f8b-4382-be05-30ddb1cc686f-photo.jpeg
filename: tfss-0e4d6293-5f8b-4382-be05-30ddb1cc686f-photo.jpeg
new filename after renaming: 0e4d6293-5f8b-4382-be05-30ddb1cc686f-photo.jpeg
new url: http://files.parse.com/75b8d55e-cb49-4169-8aa8-51c3252d7031/0e4d6293-5f8b-4382-be05-30ddb1cc686f-photo.jpeg

Then, Parse Server runs this code and determines that the new filename is hosted on Parse's legacy file server 😖

I'll submit a PR with this fix. Unfortunately, you'll need to reset your database and try the migration again to test.

Addresses issue mentioned in comments of PR parse-server-modules#7

Fix typo that cause AWS auth to fail.

6bf9d6f

flovilmart merged commit 5ea04ec into parse-server-modules:master Jul 20, 2016

JeremyPlease added a commit to JeremyPlease/parse-files-utils that referenced this pull request Jul 20, 2016

Fix Parse file renaming so tfss- files don't become legacy files

7e57f61

Addresses issue mentioned in comments of PR parse-server-modules#7

JeremyPlease mentioned this pull request Jul 20, 2016

Fix Parse file renaming so tfss- files don't become legacy files #9

Merged

Fix typo that cause AWS auth to fail. #7

Fix typo that cause AWS auth to fail. #7

Uh oh!

Conversation

acinader commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

JeremyPlease commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

JeremyPlease commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

JeremyPlease commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

JeremyPlease commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

flovilmart commented Jul 20, 2016

Uh oh!

acinader commented Jul 20, 2016

Uh oh!

JeremyPlease commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

flovilmart commented Jul 20, 2016 •

edited

Loading

flovilmart commented Jul 20, 2016 •

edited

Loading

JeremyPlease commented Jul 20, 2016 •

edited

Loading

acinader commented Jul 20, 2016 •

edited

Loading

JeremyPlease commented Jul 20, 2016 •

edited

Loading

JeremyPlease commented Jul 20, 2016 •

edited

Loading