Improve text & messaging to avoid support tickets when ingest fails due to file format

[sbarbosadataverse]

Since this isn't a bug, provide better notification messages in the UI to users so they understand they don't need to contact us about failed ingests and they understand the failed to ingest message is not shown to users of their data.

[TaniaSchlatter]

It sounds like users need feedback about the status of their file @sbarbosadataverse, which is different from a warning or error.

[sbarbosadataverse]

@TaniaSchlatter thanks for chiming in. the message as it exists now, does not include reference as to why the failure is or is not important. users don't understand what it means and I think many assume their files are not "uploaded" when they see it.

1. if the file doesn't get ingested, does the user need to see it? it's not like we are making a great effort in the UI to describe formats that work with the ingest feature, so why do we point out when it fails?

2. is the goal of the error message to have users change their files to ingest successfully? @djbrooke Because if that's the goal, the message still doesn't get that across.

3. If the ingest tool doesn't work for every file, I don't see the purpose of showing the error message when it doesn't work for certain files.

[TaniaSchlatter]

Cases/related issues:
Stata ingest failure (the problem that prompted this issue), line breaks in Excel (#6874), Excel string values (#3383), or other (#7854), if the file is added (without features enabled by completed ingest)

Current images for context
1. Dataset page file card showing Excel file with a string break

2. In-app notifications

3. File page notification

[djbrooke]

It's important IMHO that we have on page messaging to show why a file failed, on both the dataset and file page, as you could have multiple people working with a dataset and I'd like to surface it for all parties. The goal is to get people to at least see the error and determine if it's something they can/should fix. The ingest tool will never work for all files and there will always be failures. I do think the messages are overly persistent and not clear, but I'm not in favor of removing them.

We should revisit notifications and emails, definitely! The failed ingest emails were added in the last few years by an external contributor if I recall correctly.

[sbarbosadataverse]

@djbrooke I think @leonid would be great at capturing the useful message for users if we keep the messaging. That's all we want is for users to understand what they are seeing. Leonid has helped me in the past with figuring out how a depositor should correct their file for successful ingest, but we have cases where file formats and versions shouldn't be changed so we don't want to encourage that. So the message has to be very clear. And in the case where ingest won't work for every file, I don't think the message should encourage changes to the files. But it should tell the user in so many words that "ingest failed, and your file won't have this "Additional" support of working with the data explorer tool. files will still be downloadable in original format by users. This message is only available to you. For additional information on tabular file ingests, follow this link."

[qqmyers]

In some sense, this isn't a failure per se. There are perfectly valid Excel /other files that don't conform to the expectation that they represent one table. Could this be an informative msg versus a warning/error (in addition to text changes)?

[sbarbosadataverse]

@qqmyers agree. it's not a failure. that's part 1 of cleaning up the message. part 2, is cleaning up the message well enough that users aren't sending us support tickets and repeatedly deleting their file and retrying the upload, with the same results, which is what is happening.

@djbrooke how about sending it in the notification under "my data" and leaving it there may help. we can certainly add more text to notifications that go under my data than on the user interface? what do you think? and I agree, to removing "failed" and using some other term to capturing what's happening.

[djbrooke]

@sbarbosadataverse I only have strong feelings regarding the removal of the on page messages, which I feel we should not do. I'm supportive of toning the messages down and making it less persistent and/or dismissable. I'm supportive of providing information to users that will allow them to decide whether or not to take the time to get the file in a state that will allow it to be ingested or to get them information that will allow them to make the decision to be OK with the failed ingest because it's a table of contents or has unsupported characters or some other case. I'm very, very supportive of making any changes to the notifications/emails.

[sbarbosadataverse]

@djbrooke Ok. Then sounds like we can get Leonid on board to help us with that messaging? In case there is technical information we have to include? and where the messaging will be visible? Should we include some community feedback on this? I am sure Sebastian, Philipp may want to add some input...

[TaniaSchlatter]

Proposed
Revised, based on discussions in the issue:

1. Show a message to depositor and others with access on the dataset page
The info style message is the UI affordance that is used for this type of situation:

2. Show the same info style message on the file page? Is this useful?

3. Use the current in-app notification and email to the depositor to inform the user of the status of the dataset/file (that the file is available but that ingest was not completed), what ingest means, and requirements for ingest.

4. We need to work out wording that appears in these places. @sbarbosadataverse @djbrooke @scolapasta

[djbrooke]

Thx @TaniaSchlatter -

1. Yes, with "at least one file has failed ingest" - this introduces a potential issue where the person would have to sift through a large number of files to find the one that failed, but with the median # of files per dataset ~10 in Harvard Dataverse I think it'd be OK as it would generally appear on the first page.
2. Yes
3. Yes, we should update. We have more real estate here.
4. Yes
5. I think we should also beef up the part of the user guide that covers ingest, and link to it from these messages.

[sbarbosadataverse]

@TaniaSchlatter I think we should avoid using words that would cause depositors alarm, like "failed, incomplete" etc. It should be completely informative. I am waiting to see if @djbrooke can lend me someone from development to help with the wording.

[djbrooke]

@sbarbosadataverse when this gets prioritized and brought into a sprint we can work on the wording together.

[sbarbosadataverse]

@djbrooke for #5 in your comments, if we add more content in the guide, are you considering adding use cases where data files should not be changed to a supported ingested version when we are behind in what we support? For example software like Stata? In this case, code files at one version won't work well with data files in an older version of the software. We had to communicate this to users at one point. Just trying to understand what we include and leave out of the guide update.

[djbrooke]

@sbarbosadataverse TBD on the specifics, but I'd probably look to add a subbullet here:

https://guides.dataverse.org/en/latest/user/tabulardataingest/ingestprocess.htm

... about common failure cases, basic troubleshooting, and what's lost when the ingest doesn't happen. Good point also about how we'll possibly be behind on some of the software versions. We can iterate on it.

[TaniaSchlatter]

Here are revised recommendations based on further discussion with @sbarbosadataverse:

1. Add more specific text about file formats above the file upload widget:
   

2. Use "alert alert-info" style and informational messaging in place of "warning" style and "failure" language.

3. Be as specific as possible with messaging text and link to guide in a popover.

Example:

4. (3 from previous list) Use the current in-app notification and email to the depositor to inform the user of the status of the dataset/file (that the file is available but that ingest was not completed), what ingest means, and requirements for ingest.

5. (2 from previous list) Show the same message on the file page.

[djbrooke]

• We should be specific about the messaging where possible, but also acknowledging that there are some failures we won't be able to report on
• The goal is to provide friendlier, more informative messaging (see mockups above)
• The discussion about whether or not allow users to opt out of ingest came up again (File Upload: Allow users to skip ingest as tabular data. #2199)
• We could tweak the text a bit if needed (something else instead of "not supported" for example

[pdurbin]

4. (3 from previous list) Use the current in-app notification and email to the depositor to inform the user of the status of the dataset/file (that the file is available but that ingest was not completed), what ingest means, and requirements for ingest.

It turns out we already have in-app and email notifications so I believe the task is a matter of editing the text. Here's what we have right now:

In-app failure

Email failure

---

Subject: Root: Your ingest has finished with errors!

Hello,

Dataset Awesome Sauce ingest process has finished with errors.

Ingested files:

• badColumns.csv (Error)

You may contact us for support at support@localhost.

Thank you,
Root Support

---

In-app success

Email success

---

Subject: Root: Your ingest has successfully finished!

Hello,

Dataset Awesome Sauce ingest process has successfully finished.

Ingested files:

• pets.tab
• songs.tab

You may contact us for support at support@localhost.

Thank you,
Root Support

---

A few more notes:

• The success message shows "pets.tab" for example but I actually uploaded "pet.csv". People might be confused about why the file extension changed. That said, it changes in the file listing as well (screenshot below).
• I believe these notifications are sent for the current batch of files that were just uploaded. That is to say, you could have ingested files in your dataset from last week that won't be included in these notifications.