Add additional Entities to an Entity List by uploading a spreadsheet

[matthew-white]

See the story in the release criteria.

[srujner]

Hey, we found some strange issue probably with converting csv file.
I'm using OS: Ubuntu 20.04.5 LTS and Libre Office Calc for creating and saving csv file. If we will follow this steps:

1. Navigate to Entities -> Entity name -> Data Tab -> Upload button;
2. Click on the Download a data template (.csv)
3. Fill the generated csv file and save it.
4. Go back to Central, click on the choose one button and pick the file that has been just saved.

We will see this error: "There is a problem on row X of the CSV file: The label is missing."

In the Text editor looks like there is no row number X and that the csv file should be ok.
Then, at a meeting with Kathleen, we exchanged files.
The file that @ktuite generated worked correctly and ours did not, but we were unable to find the cause.

Project URL:
https://test.getodk.cloud/#/projects/516/entity-lists/trees/entities

csv file:
trees.csv

[ktuite]

Another request from the meeting was for a quick recap of what is functional and what is still missing in terms of CSV validation. More details in the release criteria but here's a summary of what is current:

• header:

  • order of label and properties can be in any order
  • extra properties not allowed, missing properties not allowed

• data:

  • label cannot be blank
    • label cannot be a string of spaces
  • every property value is a string
    • not sure what happens if you put something weird in a cell, like an image, and export it as a csv somehow?
      • matt tried uploading an image file instead of a csv and it showed some of the jpeg header data in the error message
  • there aren't a lot of other restrictions on what can be in the label or property data columns, it just has to be a valid CSV. adding quotes and extra commas can make the CSV invalid.
    • there are more restrictions for the name of the entity list or name of the property.
  • even though the column headers have to match the full list of properties, the values below don't have to all be filled in
  • because these CSVs don't include UUIDs, and the UUIDs are assigned on upload, duplicate rows/labels/other values are allowed

• not finished yet (as of march 22):

  • eventually there will be warnings about things like ragged columns

[matthew-white]

Thanks, @ktuite!

In terms of status, everything described in the release criteria for the story should now be implemented. We haven't implemented warnings yet, but that's a separate story (#593).

not sure what happens if you put something weird in a cell, like an image, and export it as a csv somehow?

I can look into this. I think it'd be possible to construct a CSV file with a valid header but binary data in the rows below. I think it'd get rejected somewhere along the pipeline (by Postgres if not before), but I'll check.

[matthew-white]

If we will follow this steps: ... We will see this error: "There is a problem on row X of the CSV file: The label is missing."

This should be fixed now on the QA server.

[srujner]

Hey, we've got a little trouble with that dialog.
Both Dominica and I, without consulting each other beforehand, thought it was a simple 'tooltip information' and by clicking on the 'x' button we assumed that the box itself would disappear, but instead the entire list of uploaded entities was removed.
So we think it may not be intuitive enough for the user. Maybe we should make the dialog box bigger or the "x" should be in a different place?

[matthew-white]

CC @issa-tseng and @alyblenkin for input from the design team. I chatted about this briefly with @lognaturel, and one idea she had was to replace the x button with a trash can icon. I think that'd make the intention of the button clearer.

[alyblenkin]

@matthew-white Yes, I think the bin icon would be more helpful! It's a pattern we use already, so it's nice to be consistent. And when you hover over it, would the text say delete?

[matthew-white]

A tooltip for the icon sounds great to me. Happy to run with "delete" or any other text.

[alyblenkin]

I noticed we use "Delete" in other places so I think that makes sense!

[matthew-white]

I'm thinking I'll make the bin red as in the screenshot here, but I'm also happy to leave it gray.

[alyblenkin]

I think the red helps make it very clear what will happen.

[matthew-white]

This is sort of a tangent, but I don't think the More button is supposed to wrap in your screenshot above. I've filed the issue #620 to fix that.

[srujner]

Tested with Success!

[dbemke]

Tested with Success!

[matthew-white]

not sure what happens if you put something weird in a cell, like an image, and export it as a csv somehow?

I can look into this. I think it'd be possible to construct a CSV file with a valid header but binary data in the rows below.

I took a look at this in getodk/central-frontend#982. See the PR description for details.