Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Old builds not being deleted #211

Closed
lorvent opened this issue Apr 20, 2016 · 27 comments
Closed

Old builds not being deleted #211

lorvent opened this issue Apr 20, 2016 · 27 comments

Comments

@lorvent
Copy link

lorvent commented Apr 20, 2016

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

  • [*] You're running the latest release version of Deployer.
  • [*] Ensure that you're running at least PHP 5.5.9, you can check this by running php -v
  • [*] You've ran composer install --no-dev from the root of your installation.
  • [*] You've ran npm install --production from the root of your installation.
  • [] You've ran rm -rf bootstrap/cache/* from the root of your installation.
  • [] You have restarted the queue listener and node socket server.

To help us better understand your issue, please answer the following.

Expected behaviour

Previous builds should be removed from server

Actual behaviour

it keep on adding new builds so server running out of space.

when checked, it has 44 builds totally and taking up 6.1GB space 😢

Steps to reproduce

If your issue requires any specific steps to reproduce, please outline them here.

Environment info

Operating System:
PHP Version:
Node Version:
Database System:
Database Version:

Logs (see storage/logs/) or other output that would be helpful

(If logs are large, please upload as attachment).

@REBELinBLUE
Copy link
Owner

Sorry, been a bit busy lately having started a new job so things have kind of falling by the wayside. I shall investigate tomorrow evening

Can you go into Admin > Projects and edit the project and see what you have the "Builds to keep" setting set to?

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

Hi,
congrats on the new job 💥
Currently it has set to 10.


I have checked all repos and some have as many as 85 builds

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

On a side note, today we ran our 1000th build after 80days of usage,
so i am highly thankful to you for this wonderful product 💯

wow

@REBELinBLUE
Copy link
Owner

Awesome!

Are there any logs in the "Purge Old Releases" step in any of the recent deployments?

Can you try running

$ (ls -t|head -n 10;ls)|sort|uniq -u)

in the "builds" directory for one of the projects and paste the output?

Also, can you check if the xargs command exists? I've never heard of a system without it but you never know

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

i don't see logs for any of deployment.

that command shows me error

parse error near

@REBELinBLUE
Copy link
Owner

Sorry, extra bracket at the end, should be

$ (ls -t|head -n 10;ls)|sort|uniq -u

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

please check,

20160401191636
20160402094826
20160404193447
20160405075231
20160405100514
20160405101427
20160409093227
20160409195323
20160412120657
20160412215418
20160414220047
20160415094309
20160416104723
20160418112601
20160418112700
20160418210719
20160418233353
20160419112801
20160420000428
20160420000804
20160420001155
20160420001323
20160420001704
20160420002038
20160420002254
20160420003110
20160420003516
20160420004335
20160420004709
20160420070330
20160420070820
20160420071215

@alsotop
Copy link

alsotop commented Apr 22, 2016

As a side, I've noticed on our deploys that if our software creates files with a different permission (our offending tool was our less compiler) then deployer might fail to remove the files, and thus leave some release folders.

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

hmm.... then how to solve that problem?

@REBELinBLUE
Copy link
Owner

Hmm, very interesting that you have no log output

Have the deployments been marked as "Completed with errors". Failure to delete builds won't cause a "Failed" deployment as it would have been activated by that stage, however in that case I would expect there to be some message in the log saying what the problem is

@REBELinBLUE
Copy link
Owner

Can you try

$ (ls -t|head -n 11;ls)|sort|uniq -u||xargs rm -rf

That SHOULD delete the builds, it is what deployer runs. Interested to see if you get any output when doing it manually.

Notice that the number has increased from 10 to 11, that is what deployer does so that it keeps the new build and the 10 previous builds

@REBELinBLUE
Copy link
Owner

Sorry, I realise when I say log you may have thought I meant a log file. I meant the button on the right hand side of this screen

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

20160401191636
20160402094826
20160404193447
20160405075231
20160405100514
20160405101427
20160409093227
20160409195323
20160412120657
20160412215418
20160414220047
20160415094309
20160416104723
20160418112601
20160418112700
20160418210719
20160418233353
20160419112801
20160420000428
20160420000804
20160420001155
20160420001323
20160420001704
20160420002038
20160420002254
20160420003110
20160420003516
20160420004335
20160420004709
20160420070330
20160420070820

thats the output but builds are not deleted ls shows me all those again

and yes, i mean in deployment page only but i don't see any log against "Purge old releases"

@REBELinBLUE
Copy link
Owner

Crap I've done it again, sorry typo in the previous command, should be

$ (ls -t|head -n 11;ls)|sort|uniq -u|xargs rm -rf

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

that seems to delete all builds except last 10

@REBELinBLUE
Copy link
Owner

Hmm, so it works when running manually, that is odd :|

@lorvent
Copy link
Author

lorvent commented Apr 22, 2016

hmm,
deployer is one server, target is another server..... is that causing problem?

@REBELinBLUE
Copy link
Owner

No, connects to the targer server via SSH and runs commands

@REBELinBLUE
Copy link
Owner

REBELinBLUE commented Apr 24, 2016

Do you have a log in storage/logs/ ? It should contain the commands run on a deployment.

Could you maybe supply a copy of your DB?

$ php artisan db:backup --database=mysql  --destination=backup.sql --compression=null --destination=$HOME

Will backup your DB to ~/backup.sql You'll want to edit the file and remove the SSH keys

@lorvent
Copy link
Author

lorvent commented Apr 24, 2016

I just sent you logs, db over mail,
please let me know if you received them.

@REBELinBLUE
Copy link
Owner

Got them thanks, will try and take a look at that this evening, otherwise tomorrow

@REBELinBLUE
Copy link
Owner

Hey, both the zip files are the DB dump

@lorvent
Copy link
Author

lorvent commented Apr 24, 2016

Hey sorry....
i have sent you logs now.

@lorvent
Copy link
Author

lorvent commented Apr 25, 2016

God.... this problem looks bigger than i though of

today one site failed to load then i found all the builds ate up all the space :(

image

thankfully your above command helped me to remove all previous builds

@REBELinBLUE
Copy link
Owner

@dancryer has confirmed he has the problem also, still trying to recreate it. Really weird

@REBELinBLUE
Copy link
Owner

Found the problem, will fix it when I get home from work and release a new version

Needless to say, I'm an idiot

@lorvent
Copy link
Author

lorvent commented Apr 26, 2016

Hey....

you are a great guy producing something great... don't feel bad for one single bug...

I really appreciate your work.....

you may know or not....i visit this page repo so often that its one of 8 thumbnails in my chrome ;) and you are saving literally many hours of many devs for manually logging in and git pulling.

REBELinBLUE added a commit that referenced this issue Apr 26, 2016
* master:
  Updated CHANGELOG
  Fixes #211 Correctly changes to the releases directory before trying to delete folders
  Updated dependencies
  Add cd to profile
  Update dependencies
  Tweak readme formatting
  Syntax tweak
  Remove bash prompt as this is now managed by my sshrc
  Removed vagrant cachier
  Explicitly enable caches
  Add vagrant cachier
  Comments
  Add a few more bash options
  Fixed .env template
  Fixes #209 Added trusted proxies package
  Tweak vagrantfile to use gitconfig from my dotfiles
@REBELinBLUE REBELinBLUE changed the title old builds are not being deleted Old builds not being deleted Apr 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants