reduce Upload time for larger datasets

[frsommer]

Is your feature request related to a problem? Please describe.
when committing larger datasets it takes forever, git status seems to check every few minutes for changes, takes forever
(git status or git status -z -u ...)

Describe the solution you'd like
when committing, reduce status lookup occasions or whatever it is to speed up.

Describe alternatives you've considered
with arccommander its one workflow with arcitect i have to wait for commit and then sync

Additional context
Add any other context or screenshots about the feature request here.

[Hannah-Doerpholz]

Perhaps related to this problem, we have tried to upload two files of about 45 GB size with LFS and couldn't manage to do it (we tried to upload either file as the only change in a commit). I see that the ARCitect tried to upload the file multiple times, but eventually it failed with an i/o timeout. I have used ARCitect v0.0.49 on Ubuntu 22.04. This is the log (I have replaced the actual token with in this issue):

git branch
* hannah
main
git push --verbose --atomic --progress origin hannah
Pushing to https://git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git
warning: current Git remote contains credentials
Locking support detected on remote "origin". Consider enabling it with:
$ git config lfs.https://Usadellab:<token>@git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git/info/lfs.locksverify true
Uploading LFS objects: 0% (0/1), 0 B | 0 B/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 41 GB | 46 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 83 GB | 47 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 124 GB | 46 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 165 GB | 47 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 206 GB | 46 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 248 GB | 47 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 289 GB | 47 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 330 GB | 46 MB/s
warning: current Git remote contains credentials
Uploading LFS objects: 0% (0/1), 372 GB | 46 MB/s
LFS: Put "https://git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git/gitlab-lfs/objects/fbaf9c6db32cb2f1e4dd3b50bb3b2c052d1a867d4f133bc2f327a755f6249c91/41279675277": read tcp 134.94.68.91:58176->132.230.102.154:443: i/o timeout
error: failed to push some refs to 'https://git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git'
git status
On branch hannah
nothing to commit, working tree clean
git status -z -u
git remote -v
origin https://Usadellab:<token>@git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git (fetch)
origin https://Usadellab:<token>@git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git (push)
git branch
* hannah
main
git rev-parse HEAD
02984ea02edc5fc372ee360e84d0c403c21198e7
git ls-remote https://Usadellab:<token>@git.nfdi4plants.org/usadellab/ribes_nigrum_genome.git -h refs/heads/hannah
befed118aa8e4109c5c6684f0902238a66ead9f3 refs/heads/hannah

[JonasLukasczyk]

This looks like a DataHub problem. I will get in touch with the administrators.