ru.bozaro.gitlfs.client.exceptions.RequestException: ... - 422 (status code 422)

[dmhursh]

I'm trying to migrate from a gerrit server to github instance. Both are inside the company firewall. When I run the tool it make a lot of progress before throwing an exception and then hanging/sitting until I CTRL-C out of.

I clone the source repository and ran the following...

java                                                                    \
    -Xmx4g                                                              \
    -jar ~/projects/IS_Streams/DevOps/GHE/migrate/git-lfs-migrate.jar   \
    -s ~/projects/IS_Streams/DevOps/GHE/Streams/.git                    \
    -d ~/projects/IS_Streams/DevOps/GHE/Streams-Test/.git               \
    -g git@github.corp.com:org/repo.git                                 \
    "*.zip" "*.tar.gz"

The last part of the output is

[main] INFO git.lfs.migrate.Main -   processed: 57497/57497, uploaded: 418/423
Exception in thread "main" java.util.concurrent.ExecutionException: ru.bozaro.gitlfs.client.exceptions.RequestException: https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5 - 422 (status code 422)
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:368)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1906)
        at git.lfs.migrate.Main$HttpUploader.close(Main.java:360)
        at git.lfs.migrate.Main.processRepository(Main.java:156)
        at git.lfs.migrate.Main.main(Main.java:77)
Caused by: ru.bozaro.gitlfs.client.exceptions.RequestException: https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5 - 422 (status code 422)
        at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:315)
        at ru.bozaro.gitlfs.client.Client.putObject(Client.java:238)
        at ru.bozaro.gitlfs.client.BatchUploader.lambda$objectTask$13(BatchUploader.java:66)
        at ru.bozaro.gitlfs.client.BatchUploader$$Lambda$22.0000000074005140.exec(Unknown Source)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.processObject(BatchWorker.java:262)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.lambda$submitTask$3(BatchWorker.java:223)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$$Lambda$23.0000000074005F30.run(Unknown Source)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.executeInPool(BatchWorker.java:298)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.access$400(BatchWorker.java:34)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$1.run(BatchWorker.java:317)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:785)

Let me know if I should gather more info or if there is anything I can do to help.

[bozaro]

It seems the error 422 you must have files larger than 2 GB.
But in any case, I will add the output data received from the server for diagnosic this error.

[dmhursh]

My largest file is 369M. Could anything else cause that error?

[bozaro]

I added more diagnostic information: http://dist.bozaro.ru/temporary/git-lfs-migrate.zip
Perhaps it gives you more information for consideration.

[dmhursh]

Thanks for the quick response. I forgot to say that before.

I tried the new code and go the following. Is there a way to tell which file it was trying to upload?

[pool-2-thread-2] INFO git.lfs.migrate.Main -   processed: 57345/57497, uploaded: 418/423
[main] INFO git.lfs.migrate.Main -   processed: 57497/57497, uploaded: 418/423
[main] ERROR git.lfs.migrate.Main - HTTP request failure: Request:
  PUT https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5
  Accept: application/vnd.github.smasher+json
  Authorization: RemoteAuth *****
Response: 422 status code 422
  Server: GitHub.com
  Date: Fri, 19 Aug 2016 16:44:00 GMT
  Content-Type: application/json
  Content-Length: 152
  Access-Control-Allow-Origin: *
  Content-Security-Policy: default-src 'none'
  Strict-Transport-Security: max-age=31557600
  X-Content-Type-Options: nosniff
  X-Frame-Options: deny
  X-Github-Request-Id: 1a51fb3f-662c-11e6-8bbb-7d870e4cd923
  X-Xss-Protection: 1; mode=block

Exception in thread "main" java.util.concurrent.ExecutionException: ru.bozaro.gitlfs.client.exceptions.RequestException: https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5 - 422 (status code 422)
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:368)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1906)
        at git.lfs.migrate.Main$HttpUploader.close(Main.java:369)
        at git.lfs.migrate.Main.processRepository(Main.java:165)
        at git.lfs.migrate.Main.main(Main.java:79)
Caused by: ru.bozaro.gitlfs.client.exceptions.RequestException: https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5 - 422 (status code 422)
        at ru.bozaro.gitlfs.client.Client.doRequest(Client.java:315)
        at ru.bozaro.gitlfs.client.Client.putObject(Client.java:238)
        at ru.bozaro.gitlfs.client.BatchUploader.lambda$objectTask$13(BatchUploader.java:66)
        at ru.bozaro.gitlfs.client.BatchUploader$$Lambda$22.00000000A0008330.exec(Unknown Source)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.processObject(BatchWorker.java:262)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.lambda$submitTask$3(BatchWorker.java:223)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$$Lambda$23.00000000A0008910.run(Unknown Source)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.executeInPool(BatchWorker.java:298)
        at ru.bozaro.gitlfs.client.internal.BatchWorker.access$400(BatchWorker.java:34)
        at ru.bozaro.gitlfs.client.internal.BatchWorker$1.run(BatchWorker.java:317)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:785)

[bozaro]

I have no idea yet.
For some reason GitHub returned an error on uploading object request. It seems necessary to either search for the reason in its logs (if available) or in the body of the request.

I add loggin first 1Kb request/response body http://dist.bozaro.ru/temporary/git-lfs-migrate-2.zip

[dmhursh]

That produced the following. I'm not sure if I can get at the log.

[main] ERROR git.lfs.migrate.Main - HTTP request failure: Request:
  PUT https://media.github.corp.com/networks/8503/lfs/objects/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5
  Accept: application/vnd.github.smasher+json
  Authorization: RemoteAuth *****
Response: 422 status code 422
  Server: openresty/1.9.15.1
  Date: Fri, 19 Aug 2016 21:26:08 GMT
  Content-Type: application/json
  Content-Length: 152
  Connection: keep-alive
  Access-Control-Allow-Origin: *
  Content-Security-Policy: default-src 'none'
  Strict-Transport-Security: max-age=31557600
  X-Content-Type-Options: nosniff
  X-Frame-Options: deny
  X-Github-Request-Id: 83b6170d-6653-11e6-8b3d-c65156b2e3a9
  X-Xss-Protection: 1; mode=block
 7B 22 6D 65 73 73 61 67 65 22 3A 22 56 61 6C 69 64 61 74 69 6F 6E 20 46 61 69 6C 65 64 22 2C 22  {"message":"Validation Failed","
 7B 22 6D 65 73 73 61 67 65 22 3A 22 56 61 6C 69 64 61 74 69 6F 6E 20 46 61 69 6C 65 64 22 2C 22  {"message":"Validation Failed","
 7B 22 6D 65 73 73 61 67 65 22 3A 22 56 61 6C 69 64 61 74 69 6F 6E 20 46 61 69 6C 65 64 22 2C 22  {"message":"Validation Failed","
 7B 22 6D 65 73 73 61 67 65 22 3A 22 56 61 6C 69 64 61 74 69 6F 6E 20 46 61 69 6C 65 64 22 2C 22  {"message":"Validation Failed","
 7B 22 6D 65 73 73 61 67 65 22 3A 22 56 61 6C 69 64 61 74 69 6F 6E 20 46                          {"message":"Validation F

[bozaro]

As I understand, "Validation Failed" means checksum or data length mismatch.
I don't know, how can occured this situation without black magic.

In any case:

1. BLOB checksum is cached in file git-lfs-migrate.mapdb, so you can try remove this file and rerun git-lfs-migrate tool;
2. You can convert repository without uploading (without --git and --lfs params). In this case all LFS object will be stored in lfs directory in target path. Problem file will have name lfs/objects/62/bd/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5

[dmhursh]

In the case of one, every time you sent a new version, I ran it from a new directory.

I just moved the old mapdb and tried again without uploading. This looked interesting...

ls -lh .git/lfs/objects/62/*/*
.git/lfs/objects/62/4f/624fe1b1dcd6b0a50c96f929d59fcdabf7e3b4bb661d270f5ce794eecb38dcb0: gzip compressed data, was "unixODBC-2.3.0-el7-x86_64-install-bin.tar", from Unix, last modified: Tue Apr 15 16:10:34 2014
.git/lfs/objects/62/bd/62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
.git/lfs/objects/62/fc/62fc12bee6e4d1b490d3f321e7b458600308623feafce2e0b44bf654d95c4501: gzip compressed data, was "libxml2-2.9.4-el7-ppc64le-install-bin.tar", from Unix, last modified: Mon Jun 13 17:39:47 2016

I don't know if it's as interesting as it looked to me. I recognize the first and third file. The second one is the problem child. I'm still only doing ".zip" & ".tar.gz" so I would not have expected that. The project was around a while before my involvement so I can rule out an old file in the tree.

[dmhursh]

The problem file is eclipse-SDK-4.3.2-linux-gtk-x86_64.tar.gz from http://archive.eclipse.org/eclipse/downloads/drops4/R-4.3.2-201402211700. Guess it's nothing special. I can try creating an repo that only has that file and see if it has trouble or if it take the more of the repo to trip it up.

[bozaro]

I downloaded this file and it really have SHA256 62bd609d9b26461fcd7fcc5efe6dcbac1a1c46ff70a771a544bcf3a7a2ba00a5

I have no idea yet why can not upload this file :(
I do think what could be the problem and how it can be solved or workarounded.

At the moment, I can offer to add to the repository this file in any other way. For example, through a console git lfs client. If the file is somehow uploaded into the LFS storage, then no more problems with it uploading should not occur.

[dmhursh]

I'll also see if I can get a log from the server side. It's probably worth know what's wrong there. It seems other's in our organization have had problems uploading larger files as well. I've heard of a bug with file larger than 150M. Today I saw someone say a 264M file worked but a 485M didn't. I don't believe they are using git-lfs-migrate.

I don't know if you want to close this for now or if you want to let it sit while I try to contact the server team to see if they can help. I'm will to try anything if it will be useful for you.

[dmhursh]

I haven't been able to get logs so far, but I was able to add the problem file to an empty branch in the repository. I just created an empty branch, copied in the file, tracked it, added it, committed it and pushed.

git checkout --orphan test_branch
git reset --hard
git checkout master -- path/to/eclipse-SDK-4.3.2-linux-gtk-x86_64.tar.gz
git lfs track path/to/eclipse-SDK-4.3.2-linux-gtk-x86_64.tar.gz
git add path/to/eclipse-SDK-4.3.2-linux-gtk-x86_64.tar.gz
git commit -m 'add problem eclipse file'
git push -u ghe test_branch

Do you think the problem is only with attempting to migrate? And do you think this populated LFS enough to reliably migrate?

[dmhursh]

To try and make the migration just work,think I can I create a temp branch, add all the files from lfs/objects and try to commit that?

[dmhursh]

Hi. Sorry it's been a while since I've posted anything here. Our github server was recently upgraded to 2.8. It looks like that fixed our problem. The migration works and I get an error on the final push due to me missing a few large file patterns. I'm cleaning that up now. I don't think there is a bug in git-lfs-migrate to be fixed.