Image upload error + no keogram / timelapse / star trails produced today only #570
Comments
|
I'm thinking that you have a mismatched of versions scripts given that the startrails tool isn't being called properly. I don't think you'll need to delete everything; you should be able to pull the latest changes with git. Also, there may be a path name escaping/quoting issue since you're trying to upload to Some totally gratuitous advice: FTP is a stupid protocol and should be wiped off the face of the earth. You should switch to SMB/CIFS or NFS. My skycam just mounts a CIFS share from my NAS and it works very well. But that's just my opinion. Perhaps @EricClaeys has some better advice for troubleshooting the FTP issues. |
|
Ok, is it a simple thing to do, as the instructions say to delete the allsky folder and clone a new one? What I do is rename the allsky folder, clone a new one and installed as fresh, manually putting my settings back in. Maybe that's not a good thing to do. The NAS is accessible for both Allsky PI and Allsky PI 2 from my PCs, phones etc no issue. I do have mfa set up on all my NASes but that doesn't affect file access, only web admin stuff. There's nothing in the NAS access logs showing a connection refusal. It was since I did this on the 3rd of October (rename and clone / install the new version), the latest image hasn't updated on the NAS since, none of the timelapses etc have been uploaded either but they have been created and are stored on the PI. Only this morning did they fail. I'm thinking that I should clean up everything allsky related and start again, this is why I have a test PI! Unless there's an easy way to overwrite the files from the latest on Github, which I'm not sure how to do? |
|
You may be right...?
|
|
Ok I'm learning... I did a 'git pull' and it's showing me this:
Obviously something's changed but I'm not sure how to interpret these results. I'll do some more reading / testing, but please feel free to close this 'issue' if noone else needs to contribute. I'll probably end up deleting and starting afresh, to avoid wasting anyone's time. Unless I break something! |
Clearly you've changed something, as we all have.
|
Yep lots of changes there. You can do You'll want to rebuild the the binaries - |
|
Actually, now that you're fully up to date with the latest code... you can run the following from the You'll need to transfer your settings from |
|
Oh, and then restart allsky: |
|
@Jonk2 Please make sure your scripts/upload.sh is the latest - it was updated late yesterday. It has quotes around the temporary name like "TL-26567". If that doesn't fix the "Forbidden filename" error, I suspect your FTP server doesn't allow minus signs in filenames. Try to manually FTP a file with a minus sign in the name. If that also fails, try with and underscore ("_"). If that works, I'll change upload.sh to use an underscore. The "Access denied" error is because lftp can't upload the file due to "Forbidden filename", but then tries to rename the non-existent file, which of course fails. I haven't yet figured out how to get lftp to exit after the first error. |
|
@Jonk2 Can you do me a favor? I don't have access to my Pi but want to test something. In scripts/upload.sh, after the echo set net:timeout 20 line, add this: You can then remove those two lines. I appreciate it. This will help me determine how to exit after the first error. |
|
@linuxkidd Thanks for the advice, but I've really broken something as nothing appeared to change the default image / overlay settings no matter what I changed / saved / checked, so I've given up and deleted / reinstalled from fresh! This has not worked - I notice in the installation of the GUI, it cannot find certain things:
Here's a screenshot of what's been put in the freshly cloned allsky folder:
After rebooting and logging into the gui, all of the camera settings and scripts have gone back to default, but I have no image from the camera (as per attached screenshot) - it's showing a broken image link type icon and notice the "Daytime images updated every 0 seconds, nighttime every 0 seconds", even though the settings say otherwise. Is simply deleting the allsky folder and recloning / install.sh / gui install.sh not working as expected?
@EricClaeys moving files between the PIs, PCs, and NASes using Filezilla via sftp works fine, with minus signs in both directions. Sorry I can't try your suggestion at the moment due to the issue above. |
|
Hmmm...update - opening the camera settings in the gui and simply pressing 'save changes' now shows an image - perhaps that's a bug that needs looking at? |
|
Also, sorry but installing the latest (as of 30 minutes ago) my log shows this:
Is this correct? How can there be so many changes over the past few days but the version has remained the same? Can you add a build number or something because to idiots like me, I don't know whether I'm up to date or what version I have, and this makes it very difficult to sort out issues before asking for help. |
|
I've just manually put all of my settings back in, saved the changes and although the image is displayed correctly now, the overlay is still at default settings.
I don't know what else I can do with this, as this should have been a fresh installation? Something tells me deleting the allsky folder alone and cloning a fresh copy to install doesn't work? |
|
@Jonk2 thanks for reporting the bug. Sorry for the hassle. |
|
No need to apologise, it must be frustrating trying to solve everyone's issues, we'll all have different problems and I suspect not many of us are programmers! I'll do as you suggest and report back. Thanks. |
|
Ok here's the terminal result... does it look correct?
|
|
Ok, done and now the settings appear to be having an effect. Thanks for that. However, have you changed or stopped the 'Allsky software is starting' message(s)? It no longer appears before the first image. |
|
@Jonk2, have you rebooted the Pi? After the installation you need to in order for the new host name ("allsky") to take effect. That might be what's giving the error below. @linuxkidd Michael, any ideas on this error at the beginning of gui/install.sh? |
|
@Jonk2 the messages look good. Could you do me a favor and try the attached installation script? It solves the problem you had AND saves the allsky-website files if they were there. If you do NOT have the website installed in /var/www/html/allsky, please create /var/www/html/allsky (may need sudo) then put a couple files in that directory - they can be junk files. Move allsky/gui/install.sh to allsky/gui/install-ORIGINAL.sh If it works you should see the message above PLUS a message about "Saving the prior website" and then "Restoring the prior website". |
|
@EricClaeys yes I reboot when anything major has been changed. I don't have a /var/www/html/allsky folder, but created that with a couple of files. I've tried your attached install.sh and here's the response in the terminal.
You can see filezilla in the background showing the original and new install.sh. I wonder if the /var/www/html/allsky doesn't have the right access permissions actually (created in Filezilla)? Let me check that. |
|
Nope, changing the /var/www/html/allsky folder structure to 777 didn't make a difference. |
I do have 2 PIs don't forget so if I remember correctly, the install script changes the PI's name to allsky doesn't it? Does having 2 PIs with the same name cause an issue? I would imagine yes. Maybe a prompt to the user to give it a custom name when installed is needed. |
|
Sorry, I"m late to this party... ^ this error is likely caused by changing the Pi's hostname ( in Having two PI's with the same hostname on the same network can result in unpredictable things from your other computers perspective. I'd suggest re-running the Example: There's a mechanism built into that |
|
@Jonk2 Sorry about the error when running gui/install.sh. I edited the file on my PC and it saves in PC format rather than UNIX format. |
|
@linuxkidd I thought so, but didn't realise you could set a custom name like that - could you add it to the install script as a prompt to the user? That would be easy for us - right at the end before reboot say "now rename your PI to or continue with the default (allsky)"....is that easy to do? I don't use my PIs just for allsky, so this makes sense to me. @EricClaeys The install.sh you provided on that link did not work either directly, so I copied the text into a new notepad++ file (on windows and my notepad++ is set to unix for new files) and saved - it overwrote and has a slightly different filesize.
|
|
In general, when reporting issues can you please tell us which git revision you're working with ( Try run it by hand... something like this: That will run the freshly compiled keogram with a single thread and verbose output, copying the output to If it does crash... congratulations, you get to learn how to use gdb (eg. https://www.cs.cmu.edu/~gilpin/tutorial/) Then post the contents of the backtrace. |
|
You'll definitely want to grab the latest commits from git. I added some additional tracing to endofnight so that the exact failed command is logged rather than |
|
My errors are as follows was there any expectation that the latest git contains more debug to the root of the errors? |
|
@Jonk2, Jon, grab the latest scripts/endOfNight.sh. It will expand the variables so you can see EXACTLY what arguments were passed to the keogram and startrails programs. We should then be able to reproduce the problem and fix it. Also, do you have the latest src/startrails.cpp and src/keogram.cpp? If not, please grab them and put them in the allsky/src directory. Then run: My guess is that your endOfNight.sh and .cpp files are out of sync. |
|
Ok, can I check something? Please bear with me, I'm not Linux inclined and am trying to keep up! A fresh git clone copies all of the latest into the allsky folder - I get that. Git diff shows the latest files vs what I have, correct? This way I can see what's changed before I download. To do this correctly I use git pull, correct? Does this only 'pull' or download files that are different, i,e, updated? I assume once I do a git pull I need to 'make' allsky and capture again? Anyway, I've followed the instructions above by @ckuethe and here's a screenshot of the process / result.
I've also follwed @EricClaeys instructions (assuming the git pull has dowloaded any changed files?) and run ./install.sh again, and rebooted. The program is running and everything's set to run in the morning and save to a NAS location, as before. I'm not sure I can do much more until I check the result tomorrow after work (which will be about 19:00 BST) so just over 24 hours from now. |
|
OK, so the git pull failed because image.jpg was in conflict. Then you can run make again which will build the latest code. As you can see from the Another thing for you to try is to add a size filter. If
crashes, add
|
|
Ok I did the first bit, here's the result - I'm not entirely sure when I'm looking at, green good red bad normally?
As I've run make again, is that it or do I need to run the ./install.sh again? |
|
That's perfectly normal git output which summarizes lines/bytes added and removed. Note that I said Also, when grepping for endofnight or timelapse in the logs, please use |
|
@Jonk2, I believe the latest keogram file works and will resolve the problem you were having. |
|
Ok, I'm closing this as the keogram is produced and ftp is working ok at the moment. |
|
I have a fresh pull from a few days ago and mine is doing this now too... |
|
@maphilli14 there's likely a corrupted file in /home/pi/allsky/images/20211208, or the image size or number of colors changed. The keogram program should have caught it, but obviously not. |
|
I lost that dataset when I freshened my install again due to some boot issues. I had a keogram and startrail but a short/corrupt, but I don't seem to have the runForDay.sh script in the folder, it's all brand new clone from yestereday.... |
|
@maphilli14 sorry, it's called |
|
Hah! NP, why did this happen to remove bad images?
That was CTRL+C'ed as it seemed just not doing what I wanted. It re ran fine with that proper command and is very handy! Just didn't seem to fix the length issue. I'll try to rerun removebadimages and try again...
|
|
@maphilli14 how many images do you have in the 20211209 directory? I would guess many thousand and the code is hitting a limit of the number of files allowed on a command line. Until I'm able to fix it you can move, for example, half the files to a new directory in allsky/images, e.g., 20211209b, and run removeBadImages.sh on each directory. Then when done, move all the files from 20211209b back into 20211209. I only need a couple corrupt files to debug the startrails and keogram issue you had. Eric |
|
Do I have too sensitive a camera? I'm using the 290 not the 174 we've been testing in BugID = 650 |
|
@maphilli14 I don't think there is a camera that's too sensitive. Sensitivity is good. Going forward you could try increasing your delay between pictures, especially during the day. That will decrease the number of pictures and hopefully get you below the limit. |
|
What's the actual limit? I split in 2 and it still doesn't run :( |
|
I only record at night and it's using the default delay between photos and a 4s sub exp :( |
|
I'm pruning them by hand to test.... |
|
Hi @maphilli14 ... You can run the following command to get the argument list limit: Note that the value returned is the character length of the command and is enforced by the operating system ( not the shell ). Since it is a character count limit, the path for each file is included in the limit. With that in mind, you may be able to get the script to run with the following: This alone will reduce your character count by 14 for -each- image. |
|
I split it into quarters, or closely... but .... Damn, it's really punchy! GOOD BAD |
|
actually those total file counts are reversed, the 2528 was too long and gave errs but the 2505 processed fine and removed 23 additional bad files above and beyond the small file size i pruned by hand. Why am I the only one with so many files? I backed off the default inter 'frame' capture from 10ms to 100ms to abate the bad file issues.... honestly I was hoping that at 4s subs it would be less churn on the disk i/o and better results... keep tweaking I guess... |
|
Changing the delay from 10 to 100 didn't result in a successful timelapse for the recent end of night... this still confuses me...
|
|
Hi @maphilli14 My settings:
My other system only generates up to 804 exposures:
There can be some code changes in the script to accommodate larger image sets ( creating a loop that processes say.. only 500 images at a time ). If you need assistance tweaking your exposure / delay settings, or the image count you're achieving is necessary, please open a new issue so we can work the problem separately since it's not related to the original purpose of this ticket (also, I'm sure Jonk2 isn't thrilled with getting loads of updates on an issue they consider closed ). Thanks! |
I've tried to keep up with the many new releases / files and this could be a case of delete it all and start again!
To avoid confusion, I have 2 x RPI4bs and 2 x ASi178MCs (outdoors is standard, indoors test one is the cooled version).
Outside known as Allsky PI, inside test one Allsky PI 2.
My log file on the inside test Allsky PI 2 says it's v0.8.1 and for the past couple of days, the image upload (i.e. 'live-view' image) to my NAS fails.
Here's the log from the latest attempt (personal details obscured with ******):
This is on my test PI indoors. There doesn't appear to be any 'saveImageNight.sh' entries. The FTP settings are correct and match the PI outdoors which is also running v0.8.1 (from the log), albeit a different folder (Allsky, not Allsky 2).
The PI outdoors seems to be working fine, and I've had no issue with timelapse, startrails and keogram generation on the PI outdoors, albeit cloudy and wet ones!
All of the files are saving fin on the PI in home/pi//allsky/inages, and thre are keograms etc since the last update (3rd of October) but this morning's (5th of October) failed with the following in the log:
Keogram
Startrails
Timelapse
With my limited understanding of this, the common denominator appears to be the NAS. I don't think it's a NAS access issue, but I suppose I should check that too. Nothing else has an issue accessing the location. It'd the 'Access failed: 550 TL-26567: Forbidden filename' that says it's not an access issue, but then also says 'mv: Access failed: 550 TL-26567: No such file or directory' which would suggest it can't see the location or has been refused.
If this is totally weird and I need to delete allsky and start again with a fresh clone, that's fine, but it might be more sinister. As I said, it could be a case of a botched installation i.e. mix / match of versions, but as both versions are shown as 0.8.1 in the logs, I have to assume they're the same and I haven't done something silly.
The text was updated successfully, but these errors were encountered: