Fixes #24: Allow images to be uploaded when creating a new bug report. #679

miketaylr · 2015-08-18T20:49:54Z

There's a lot going on here, including:

New upload blueprint route (we'll make more use of this in the future when we add comment uploads and screenshot uploads).
Adding a <input type=file> for our bug report form.
Image uploading to the server.
Use Flask-Uploads for image uploading.
Use Flask-WTF for form handling and image type validation.
Clientside validation for image types.
Functional tests for clientside file validation.
Unit tests for new upload rote.

Thx to @magsout for his help with frontend, as always.

r? @karlcow

…file]

(The image/* filter accepts way more stuff than we want. See https://dxr.mozilla.org/mozilla-central/source/toolkit/content/filepicker.properties)

…put from being too big.

tests/test_uploads.py

+    def testBadUploads(self):
+        # Loop over some files and the status codes that we are expecting
+        for filename, status_code in \
+                (('foo.xxx', 415), ('foo', 415), ('foo.rb', 415)):


karlcow · 2015-08-19T06:53:14Z

I would love if @magsout could review the JavaScript. upload seems to me a bit too dangerous to just me thinking the JS is fine. I'll focus on the python code.

miketaylr · 2015-08-19T16:12:55Z

I would love if @magsout could review the JavaScript. upload seems to me a bit too dangerous to just me thinking the JS is fine.

Sounds good. The JS doesn't really do much with respect to uploads. Basically it's just duplicating the file validation on the client side that Flask-WTF is doing on the server side--so people don't have to submit an issue and get an error back (if they have JS enabled). But a sanity-check on checkImageTypeValidity in bugform.js and the tests would be welcome.

webcompat/api/uploads.py

+
+# limit image uploads to 4MB
+patch_request_class(app, 4 * 1024 * 1024)
+


webcompat/api/uploads.py

+
+from flask import abort
+from flask import Blueprint
+from flask import render_template


karlcow · 2015-08-20T04:29:59Z

@miketaylr
I imported locally to test a couple of things.
Test at webcompat/webcompat-tests#272

I noticed something. First of all there are two nested uploads folder. Is it what you want?

→ find uploads
uploads
uploads/uploads
uploads/uploads/751a802ba7b56cd1f5d1170fda195a06_1433859324.jpg

Second, I would encourage to set a kind of dated space for the images inside upload. Something like

uploads/2015/08/image_hash_key.jpg

The reason is that the image are not stored into a database but in the filesystem. Filesystems have a tendency to become slow when the number of individual items grow into one unique directory. Having a dated space helps to create a partition that will reduce the number of images in one directory. It makes also less necessary to have a long hash_key, but that's secondary.

tests/test_uploads.py

+            self.assertEqual(rv.status_code, status_code)
+
+
+class TestingFileStorage(FileStorage):


miketaylr · 2015-08-20T16:00:25Z

I noticed something. First of all there are two nested uploads folder. Is it what you want?

Oh, we can fix that. That's now actually how it's laid out on the server - just a goof up for the localhost stuff.

miketaylr · 2015-08-20T16:00:41Z

Second, I would encourage to set a kind of dated space for the images inside upload.

Agreed, let me work on that today.

[ci skip]

karlcow · 2015-08-20T21:34:25Z

r+ from me for the python side.
I don't know if you want to wait for @magsout r+ on JS.

Thanks a lot @miketaylr for your patience and the code. 💯
This is a very nice feature. I think there's an additional item which might come from this, a separate issue. Design work for the bug submitting form.

miketaylr · 2015-08-20T21:44:02Z

Thanks! Let's wait a day or two to see if @magsout can take a look. No real rush for this feature.

I think there's an additional item which might come from this, a separate issue. Design work for the bug submitting form.

Cool. Also I need to file bugs (or find the ones that exist) for: adding upload form to issues/comments, drag & drop of images, API upload support for an arbitrary issue (then the stuff around screenshots).

magsout · 2015-08-25T05:04:30Z

lgtm

karlcow · 2015-08-25T07:21:32Z

Thanks @magsout !

And let's pull it with love.

Fixes #24: Allow images to be uploaded when creating a new bug report.

magsout · 2015-08-25T07:44:57Z

Good job everyone ❤️

miketaylr · 2015-08-25T14:36:01Z

Thanks for review @magsout and @karlcow! Will deploy at some point today.

Mike Taylor and others added 21 commits August 14, 2015 13:29

Don't swallow the details of the exception.

fd93070

Issue #24. Add an upload blueprint + route.

a8d9c09

Issue #24. Serve images from Flask -- but only for local development.

6cce652

Issue #24. Update config.py.example with new config settings.

df49e0b

Issue #24. Hook up image uploading with bug reporting.

59b51ed

Issue #24. Tweak upload form label and add accept attr to input[type=…

35dd685

…file]

Issue #24. Be less liberal and only accept: jpeg, png, gif and bmp.

0300844

(The image/* filter accepts way more stuff than we want. See https://dxr.mozilla.org/mozilla-central/source/toolkit/content/filepicker.properties)

Issue #24. Display errors in template for image uploading.

174636d

Issue #24. Only try to add an image to the issue... if there's an image.

4d2c359

Issue #24. Clean up the BugForm consructor object.

6f593c6

Issue #24. Show loader when uploading an image.

f521304

Issue #24 - added icon upload

1cb0588

Issue #24. Add note about uploads config error.

c3c9ec0

Issues #24 - added style and icon for upload button

d39f21d

Issue #24: Add pos: relative to wc-Form-upload to prevent pos: abs in…

47ffe9c

…put from being too big.

Issue #24: Add missing url form error class.

d9e7e59

Issue #24: Use Flask-WTF for crsf + File upload validation.

8457d9e

Issue #24: Add unit tests for uploads.

d70f002

Issue #24: Do clientside validation for image type.

fa895e1

Issue #24: Split up form validation tests and add one for valid images.

c1c70eb

Issue #24. Remove unused method import.

7a26480

miketaylr added the status: in progress label Aug 18, 2015

karlcow reviewed Aug 19, 2015
View reviewed changes

Mike Taylor added 2 commits August 19, 2015 13:36

Issue #24. Re-format how we print ImportErrors to the console.

b46af3d

Issue #24. Reformat some for loops.

2162300

karlcow reviewed Aug 20, 2015
View reviewed changes

webcompat/api/uploads.py

# limit image uploads to 4MB

patch_request_class(app, 4 * 1024 * 1024)

This comment was marked as abuse.

Sign in to view

karlcow reviewed Aug 20, 2015
View reviewed changes

webcompat/api/uploads.py

from flask import abort

from flask import Blueprint

from flask import render_template

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

karlcow reviewed Aug 20, 2015
View reviewed changes

tests/test_uploads.py

self.assertEqual(rv.status_code, status_code)

class TestingFileStorage(FileStorage):

This comment was marked as abuse.

Sign in to view

This comment was marked as abuse.

Sign in to view

Mike Taylor added 2 commits August 20, 2015 10:28

Issue #24. Remove unused import.

abbbd9a

Issue #24: Import TestingFileStorage (oops)

c8dc773

Mike Taylor added 3 commits August 20, 2015 11:30

Issue #24: Remove extra uploads/ dir for localhost.

ef65a53

[ci skip]

Issue #24. Obfuscate the uploaded filename.

bb9c875

Issue #24. Save uploads to a year/month/ namespaced location.

bd14e70

Issue #24. Update privacy policy to mention image uploads.

3790df1

karlcow added a commit that referenced this pull request Aug 25, 2015

Merge pull request #679 from /issues/24/1

2936db3

Fixes #24: Allow images to be uploaded when creating a new bug report.

karlcow merged commit 2936db3 into master Aug 25, 2015

karlcow added status: add to changelog and removed status: in progress labels Aug 25, 2015

magsout deleted the issues/24/1 branch August 25, 2015 07:45

miketaylr removed the status: add to changelog label Aug 25, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #24: Allow images to be uploaded when creating a new bug report. #679

Fixes #24: Allow images to be uploaded when creating a new bug report. #679

miketaylr commented Aug 18, 2015

This comment was marked as abuse.

This comment was marked as abuse.

karlcow commented Aug 19, 2015

miketaylr commented Aug 19, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

karlcow commented Aug 20, 2015

This comment was marked as abuse.

This comment was marked as abuse.

miketaylr commented Aug 20, 2015

miketaylr commented Aug 20, 2015

karlcow commented Aug 20, 2015

miketaylr commented Aug 20, 2015

magsout commented Aug 25, 2015

karlcow commented Aug 25, 2015

magsout commented Aug 25, 2015

miketaylr commented Aug 25, 2015


		# limit image uploads to 4MB
		patch_request_class(app, 4 * 1024 * 1024)

		self.assertEqual(rv.status_code, status_code)


		class TestingFileStorage(FileStorage):

Fixes #24: Allow images to be uploaded when creating a new bug report. #679

Fixes #24: Allow images to be uploaded when creating a new bug report. #679

Conversation

miketaylr commented Aug 18, 2015

This comment was marked as abuse.

This comment was marked as abuse.

karlcow commented Aug 19, 2015

miketaylr commented Aug 19, 2015

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

karlcow commented Aug 20, 2015

This comment was marked as abuse.

This comment was marked as abuse.

miketaylr commented Aug 20, 2015

miketaylr commented Aug 20, 2015

karlcow commented Aug 20, 2015

miketaylr commented Aug 20, 2015

magsout commented Aug 25, 2015

karlcow commented Aug 25, 2015

magsout commented Aug 25, 2015

miketaylr commented Aug 25, 2015