NativeAPI: Replace endpoint returns "Failed to add file to dataset" when file PIDs are enabled

[adswa]

Hi all! :)

As reported on Zulip, I've ran into unit test failures of datalad-dataverses' tests for replacing files that coincided with the release of Dataverse 6.4. We internally use pyDataverse for API calls, and test against demo.dataverse.org. I can, however, reproduce the issue with CURL calls following the user guide, which is what I will report.

What steps does it take to reproduce the issue?

I am seeing it using the following steps:

export API_TOKEN=<redacted>
export SERVER_URL=https://demo.dataverse.org
export PARENT=demo
# create a dataverse using dataset-finch1.json from the user guide
curl -H "X-Dataverse-key:$API_TOKEN" -X POST "$SERVER_URL/api/dataverses/$PARENT/datasets" --upload-file dataset-finch1.json -H 'Content-type:application/json'
{"status":"OK","data":{"id":2431411,"persistentId":"doi:10.70122/FK2/TV0CV6"}}
export PERSISTENT_ID=doi:10.70122/FK2/TV0CV6
# create a file and upload
echo "hello!" > datafile.txt
export FILENAME=datafile.txt
curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F "file=@$FILENAME" -F 'jsonData={"description":"My description.","directoryLabel":"data/subdir1","categories":["Data"], "restrict":"false", "tabIngest":"false"}' "$SERVER_URL/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"
{"status":"OK","data":{"files":[{"description":"My description.","label":"datafile.txt","restricted":false,"directoryLabel":"data/subdir1","version":1,"datasetVersionId":270906,"categories":["Data"],"dataFile":{"id":2431412,"persistentId":"doi:10.70122/FK2/TV0CV6/HMHVRG","pidURL":"https://doi.org/10.70122/FK2/TV0CV6/HMHVRG","filename":"datafile.txt","contentType":"text/plain","friendlyType":"Plain Text","filesize":7,"description":"My description.","categories":["Data"],"storageIdentifier":"s3://demo-dataverse-org:192d37720b8-12242b8aba79","rootDataFileId":-1,"md5":"8b8db3dfa426f6bdb1798d578f5239ae","checksum":{"type":"MD5","value":"8b8db3dfa426f6bdb1798d578f5239ae"},"tabularData":false,"creationDate":"2024-10-28","fileAccessRequest":false}}]}}
export ID=2431412 
# change file name and content to replace the file
cp datafile.txt datafile_to_replace.txt
echo "some changes" >> datafile_to_replace.txt 
export FILENAME2=datafile_to_replace.txt
curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F "file=@$FILENAME2" -F 'jsonData={"description":"My description.","directoryLabel":"data/subdir1","categories":["Data"], "restrict":"false", "tabIngest":"false"}' "$SERVER_URL/api/files/$ID/replace"
{"status":"ERROR","message":"Failed to add file to dataset."} 

The HTTP error code is 400.

• To whom does it occur (all users, curators, superusers)?

I have only tried this as a regular user on demo.dataverse.org

• What did you expect to happen?

As of Dataverse version prior to 6.4, our tests (and thus the above reproducer) worked fine, and replaced the file in the dataset

Which version of Dataverse are you using?

v. 6.4 build 1609-906f874
demo.dataverse.org as of today

Any related open or closed issues to this bug report?
@pdurbin pointed to this PR #7334 as related.

[donsizemore]

Just to chime in that I've reproduced this on demo.dataverse.org, and that it seems to be limited to the API (not GUI) when FilePIDs are enabled. Under the hood Dataverse logs an SQL error:

Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "unq_dvobject_0"
  Detail: Key (authority, protocol, identifier)=(10.70122, doi, FK2/TV0CV6/HMHVRG) already exists.
Error Code: 0