Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature shapefile upload #445

Merged
merged 13 commits into from
Aug 31, 2021
Merged

Conversation

ikas
Copy link
Contributor

@ikas ikas commented Aug 12, 2021

Feature shapefile upload

Overview

This PR adds a new endpoint POST /api/v1/projects/:id/features/shapefile that allows users to upload feature shapefiles.

To use this endpoint, you'll need to provide a .zip file containing the different format files that compose a shapefile, and send it as multipart/form-data to the endpoint specified above.

The upload happens synchronously, so once you receive the response, the data should already be available in the database. Also, keep in mind there's a limit on the maximum size of the uploaded file, currently set to 100MB (this limit is configurable via env vars).

Designs

Figma

This PR does not contain the implementation of the UI for this feature, just the BE side of things.

Testing instructions

Use the new endpoint and check that data gets correctly imported to the databases (API and geo) - check the JIRA ticket below for more details on where the data should be placed.

Feature relevant tickets

MARXAN-543


Checklist before submitting

  • Meaningful commits and code rebased on develop.
  • If this PR adds feature that should be tested for regressions when
    deploying to staging/production, please add brief testing instructions
    to the deploy checklist (docs/deployment-checklist.md)
  • Update CHANGELOG file

@ikas ikas added the WIP Work in progress label Aug 12, 2021
@vercel
Copy link

vercel bot commented Aug 12, 2021

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployments, click below or on the icon next to each commit.

marxan – ./app

🔍 Inspect: https://vercel.com/vizzuality1/marxan/9FaFDs7aPnBc7AvkLpTiFZyh5JF5
✅ Preview: https://marxan-git-feature-api543-upload-feature-shapefiles-vizzuality1.vercel.app

marxan-storybook – ./app

🔍 Inspect: https://vercel.com/vizzuality1/marxan-storybook/A3SAfAS5aKsVUo2NKqgekoM5Je1t
✅ Preview: https://marxan-storybook-git-feature-api543-upload-f-cd6a5c-vizzuality1.vercel.app

@@ -258,4 +269,91 @@ export class GeoFeaturesService extends AppBaseService<
): Promise<GeoFeature> {
return entity;
}

public async createFeaturesForShapefile(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

transactions here are little bit sketchy, since I'm writing data to 2 separate DBs 🤔 if you think of a better solution let me know 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it occurred to me with a huge 🤦🏼 when I saw your PR while trying to help with the makevalid bit - I think what you do here will be fine in most circumstances. if we do run into issues in a substantive way I'd think about a final check both in case of success or failure, but since I'm thinking about a grim reaper/garbage collection feature that we will need in any case to be able to invalidate stuff when users edit settings earlier on in the workflow, this - if needed at all - could be done when we implement that.

Copy link
Member

@hotzevzl hotzevzl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ikas! Epic PR, thanks so much for your dedication this week 🙇🏼

I'd ask @aagm to please cross check that there are no blind spots (which, if any, would mainly be due to blind spots in my specification, tbh) before merging, though I'm itching to merge RIGHT NOW so that I can use this feature via bots I'm working on 😅

@hotzevzl
Copy link
Member

oh well I hadn't notice the test failures - though they all seem to be identical and only due to an import lookup issue, so this should be easy to fix (if you get stuck, probably Dominik can help fixing this while blindfolded)

@kgajowy
Copy link
Contributor

kgajowy commented Aug 27, 2021

83a512d
6c5b589

changes:

  • fix uploading file (wouldn't work due to filename difference)
  • provide specs to ensure it works
  • extract shapefile module to libraries

Re-requesting review @hotzevzl

@kgajowy kgajowy requested a review from hotzevzl August 27, 2021 08:05
@kgajowy kgajowy marked this pull request as ready for review August 27, 2021 08:05
@kgajowy kgajowy requested a review from Dyostiq August 27, 2021 08:13
@kgajowy kgajowy force-pushed the feature/api/543-upload-feature-shapefiles branch from 6c5b589 to 08b203e Compare August 30, 2021 09:45
@kgajowy kgajowy merged commit 4c87f45 into develop Aug 31, 2021
@kgajowy kgajowy deleted the feature/api/543-upload-feature-shapefiles branch August 31, 2021 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants