signature from <package> is invalid #93
Labels
bug
Something isn't working
Comments
naclander
changed the title
|
Edit: This is either caused by misbehaving mirrors or a bug in flexo, but I think a bug in flexo is more likely. I'm looking into this issue, but I still haven't found a way to reproduce it. If anyone stumbles on this issue, you can help me troubleshoot this issue as follows:
The more output I receive, the better! Original answer:
I guess so. Probably either the package itself, or the signature (the corresponding file ending with I'd like to find out if the package file (the file ending with Once I understand the problem, I can think of a solution. |
|
So it just happened again. This happens on the client machine, during the The I cleared out the pkg cache and restarted the flexo service, ran |
|
Ok, so the output tells us that:
If the mirror itself served an incomplete file, this would explain this issue. Can you please run the following and show me the output? I'd like to investigate further if any of those mirrors have issues that would explain this before ruling out that this is a bug in flexo. If you want to try a different mirror, you can configure flexo to blacklist the previously used mirrors. Edit sudo rm /var/cache/flexo/state/latency_test_results.jsonThen, restart flexo, so it runs the latency tests again and picks up another mirror. Wait a moment until the latency tests have completed before running |
I went ahead and updated EDIT: I just realized the mirrors potentially causing a problem ( Another way would just be to remove these mirrors from my mirrolist file. I am curious why these specific mirrors would be causing a problem with reflector. EDIT2: I now see that even after pruning my |
|
Thanks for the journalctl output.
Keep in mind that flexo ignores |
|
Right, I see this in the I'm curious why not have an option to have flexo check |
One of flexo's goal is to minimize the time users need to invest for mirrorlist maintenance, and not everyone uses reflector. But of course, it would be possible to implement this as an additional option. |
|
Just wanted to update that this is still happening, even after denylisting several mirrors in flexo. Do you know if there's a more consistent way of preventing the signature error from happening? |
|
I'm still trying to figure out why it happens in the first place. Can you please check if this script runs on your machine? https://gist.github.com/nroi/492ccf2d55400746cb8084984e04002f pip3 install requestsOn my machine, the output is:
If you run this script after the signature errors happen again, you should get output like the following:
If you get this output, please post it in this issue, so I can get a better understanding of what's going on. The script also provides a |
|
I'll run this the next time I'm getting errors ( usually to move forward I change the mirror, then change it back ). Looking at the script, it looks like I'm supposed to run it on the machine hosting the flexo service? The way I have it set up is that I have multiple machines, and the flexo service is running on one machine dedicated to be the package server. It is not all running on localhost. I'll also add that sometimes I get this errors after a |
Correct. The script checks the files downloaded by flexo, including the files with the
Thanks! |
|
I've run into this as well. I'm running Flexo in Docker. Any commands run on the Flexo cache were run on the exported container filesystem. My local Pacman cache was cleared with Pacman output:SHA256 checksumsFlexo filesystem:
|
|
Here are the differing .pkg.tar.zst files: |
Interesting, the corrupted file seems to include only the end of the file. The corrupted file is 6338539 bytes large, and the last 6338539 bytes of the good file are identical to the corrupted file: The same thing seems to have happened when naclander had this issue:
This might be caused by a bug related to how flexo handles HTTP range requests, but so far, I wasn't able to reproduce it. I'm going to look into it. Also, I've updated the flexo-verify-package script to consider all repositories (the multilib repo previously wasn't included, which is why no mismatches have been detected in your case). Anyone who stumbles on this issue again could help me by providing the following info:
Thank you both for your input to help me troubleshoot this. The most recent version (1.6.7) includes a few more log statements which might help me to isolate the issue. |
|
This showed up again for me. I still haven't figured out how to reliably reproduce it, but it does seem to be failing with an HTTP 416. Thanks for looking into this! flexo-verify-package outputNote: the modification time here is correct. I dumped the logs a while back and forgot about them until now... docker logs output |
|
Thanks for the output, @avionix-g! Could you please tell me if:
What I can tell so far from your output is this:
But in your case, Flexo seems to have made an erroneous range request. Probably something like requesting the file starting from byte x when the actual file size is less than x. Another problem is that Flexo has stored a file of size 69665584 when the mirror reported a content length of just 61735728. Just like in the previous case where this error occurred, the locally stored file are the last 61735728 bytes of the file wine-7.17-1-x86_64.pkg.tar.zst. |
|
The most recent version (1.6.8) includes a few more log statements that might help troubleshoot this issue. Until I find a way to reproduce this issue, that's all I can do… Inspect the logs and hope to gain some insights. |
|
This issue finally happened on the host running the flexo service ( not on the clients, which more frequently get this error ). The output from the script you requested is here: |
|
And some logs from the flexo service ( version 1.6.8 ) |
|
Thanks for the output, @naclander The journalctl logs you've posted don't include any of the packages where the checksum verification has failed, do you have more logs available? Try something like: journalctl --since='2022-12-28 05:46' --until '2022-12-28 05:55' -u flexo |
|
Unfortunately the next time I went to run the update the error no longer appears. I try the above command if it happens again. |
|
Output of flexo-verify-packages: some flexo docker logs:
|
|
Thanks for providing the logs, @exalented!
actually, 206 is the expected response when the client requests just part of the file. For example, when pacman initiates a download from flexo, then aborts it, and later on downloads the same file again, the expected behavior is that flexo resumes the download, so it sends a range request to the remote mirror and the remote mirror responds with 206. I wasn't able to completely figure out what exactly is happening for what reasons from the logs, but something that definitely looks wrong is this: The header we have received (1st line in the logs) is this one: Notice it actually says "try again" in the log, so the re-try logic is already in place. And yet, in almost the exact same time, we have this in the logs: So, even though we haven't received the required payload from the server, we just serve whatever trash to the client (probably an empty file or an HTML file saying "Not found"), and only after we served this trash to the client, flexo fetches the actual content from the remote mirror. This does not explain everything, but it's a start. Thanks again for providing the logs, I will look into it. |
|
hi, any progress with this bug? it happens to me about once a week. about a year ago i never had this problem. perhaps because i now have 5 computers using nexo as a central mirror. |
Unfortunately, no. I'd like to devote more time to this project, but I don't have much time to spare at the moment, and this is the kind of problem that really requires more than just a bit of focussed time. The fact that this issue has never happened for me makes it difficult. I hope I find more time in the near future. But if anyone notices a pattern, or even better, a way to reproduce this issue, let me know. |
|
Is there anyone impacted by this bug who is not using flexo via docker? This info could be useful for me when trying to reproduce this issue. |
|
I was not using flexo via docker when I noticed this bug. I haven't been running into this bug in a while, the only main difference I can recall between what changed is that the bandwidth of the server running flexo increased. I don't know how/if slow download speeds and invalid signatures are related, but that's the only difference I can recall. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey,
Sometimes when using flexo, I've noticed errors when my machine is validating packages in the form of:
Neither Y nor n fixes the problem. Clearing out the flexo cache does.
Re-downloading the
archlinux-keyringalso does not fix this issue.Could this be a stale package cached in flexo somehow? Is there any mitigation for this instead of clearing out the flexo cache?
The text was updated successfully, but these errors were encountered: