Problem to build new projects after upgrading to 0.0.44

[jbecchi]

Before submitting your issue, please make sure that you've checked all of the checkboxes below.

• You're running the latest release version of Deployer.
• Ensure that you're running at least PHP 5.6.4, you can check this by running php -v
• You've ran composer install --no-dev from the root of your installation.
• You've ran npm install --production from the root of your installation.
• You've ran rm -rf bootstrap/cache/* from the root of your installation.
• You have restarted the queue listener and node socket server.

To help us better understand your issue, please answer the following.

Expected behaviour

The deployer shoud be able to clone properly the repository on BitBucket and run all "Active new release" commands.

Actual behaviour

Our deployer was at version 0.0.36 (I think, not sure). We had some problems about projects that got "corrupted" after running builds. So we upgraded it to version 0.0.44. We ran all the commands listed on the upgrade section at the Github.

The corrupted projects got back to normal, and we can create buils without any problem.

However, if we create a new project (with ou without specific commands to run), deployer can't ever download the repository from Bitbucket. It get's stucked on "Create New Release" as "running". All the next steps get the status "Cancelled".

We added multiple times the deployer's autogenerated SSH key, without any success.

Steps to reproduce

• update deployer to version 0.0.44
• create new project without any command (just clone the repo)
• run a build

Environment info

Operating System: Ubuntu 14.04.4 LTS
PHP Version: 5.6.23
Node Version: 0.10.45
Database System: MySQL
Database Version: 14.14

Logs (see storage/logs/) or other output that would be helpful

I didn't found any log by the correct ocurred time

[REBELinBLUE]

Hmm, very odd.

Can you enable debugging (edit .env and set APP_DEBUG to true and APP_LOG_LEVEL to debug; run php artisan config:clear and restart the queue listener; I am just writing a console command now to do this all in one go as this is a needlessly complicated). Then try again, you should then have a log which contains the actual commands so I can see what is happening

[jbecchi]

Hello,

I just discover it's something about that specifc repository, because I was able to create a deploy from another repository. Any ideas?

The generated log is attached.

cli-2017-03-03.txt

Thanks

[REBELinBLUE]

Hmm, that is interesting. Is it a private repository by any chance? If so, have you added the key to bitbucket (think it is called "access keys" in bitbucket)? Looks like I may have broken the warning which should appear when it can't access the repo because it doesn't have permission, I shall investigate

[jbecchi]

Exactly. I was reading about issue #323 (which was fixed), but can it be related?

The strange thing is that looks like deployer downloads part of the repository, It starts extracting but for some reason it just stops, no error is shown.

[REBELinBLUE]

Oh, that is even weirder. Can you open the output for the first step (i.e. the little button at the end of the row) and paste that? I'll set up a bitbucket repo and try it when I get home in an hour or so

[jbecchi]

Yes, of course.

http://pastebin.com/U9Z9LxwK

I was just checking the other builds that failed, it never failed at the same point of the extraction. Always on a diferent file.

[REBELinBLUE]

Oooh, I misunderstood! That is even more odd, so it isn't actually failing cloning the repo, it is failing extracting the archive (what it does is clone the repo to the machine deployer is on, make an archive of the state of the repo, upload it to the remote server and extract it). I will need to play around a bit to figure out how to debug further as it looks like it is just suddenly dying in the middle of extracting

[REBELinBLUE]

I may have an idea of the cause, I will check if it is when I get home and if so I shall patch and release a new version tonight

[jbecchi]

Oh, thanks a lot. I will wait for the patch.

[REBELinBLUE]

Damn, doesn't look like it was what I thought it was

[jbecchi]

Is there anything I can do to help you find out what is going on?

[REBELinBLUE]

Can you change resources/scripts/deploy/steps/CreateNewRelease.sh to have the following

### Create new release - {{ deployment }}
cd {{ project_path }}

# Create the releases directory if it doesn't exist
if [ ! -d {{ releases_path }} ]; then
    mkdir {{ releases_path }}
fi

# Create the shared directory if it doesn't exist
if [ ! -d {{ shared_path }} ]; then
    mkdir {{ shared_path }}
fi

mkdir {{ release_path }}
cd {{ release_path }}

# Extract the archive
echo -e "Extracting {{ remote_archive }}...\n"
tar -m --gunzip --verbose --verbose --extract --file={{ remote_archive }} --directory={{ release_path }}

#rm -f {{ remote_archive }}

Then restart the queue listener and run a deployment again. The output in the log should be more verbose, I don't think it will give anything useful but you never know.

The last line has been commented out so the archive isn't deleted; the first line of the output will contain the path of the archive. Could you try downloading that from the server and try extracting it on your computer and see if it extracts successfully?

[jbecchi]

I changed the file. I got the file from server and was able to successfully extract the source code in my PC. No problem at all.

The weird thing is that it doesn't seem deployer created a new log file (storage/logs).

The output is attached below

http://pastebin.com/bR5fc6fT

[REBELinBLUE]

Bugger, OK let me keep thinking. Good to know at least it isn't the new archive process introduced since 0.0.36. Could you try one more thing, take out the 2 --verbose entries (and then again restart the queue and start again).

Maybe add an echo "done" after the tar line so we can see if it gets to the end of unarchiving and it is something later in the process

[jbecchi]

Ok, I don't know what just happend now.

I made the changes and suddently I was able to make a build. No problem at all.

`Extracting /wwwroot/dynaduo3-homolog/12_20170303181232.tar.gz...

Done.`

Maybe it was something with BitBucket?

[REBELinBLUE]

hmm, OK then it looks like my original guess was right yet I didn't manage to recreate it. I thought it was maybe the log was too long and hitting the length limit for the output column. Can you take a look at your DB and confirm what data type the output column on the server_logs is? It should be LONGTEXT (which means as long as the log is smaller than 4GB it'll be fine) but at one point it was just TEXT(65KB and your log was 81KB) so there was a migration to change it so I want to see if for some reason that didn't run

[jbecchi]

Looks like that column is LONGTEXT.

[REBELinBLUE]

OK that is now REALLY odd, so the field is big enough for any output yet it dies when the output is verbose but works when it isn't 😧

Next thing... can you add --verbose back and leave the echo. That shouldn't make a difference at all though

[jbecchi]

And there comes the error again.

I'm kind of a noob about linux, can't it be something about some kind of cache limit for the output?

[REBELinBLUE]

Still investigating this, really odd, clearly something to do with it being a large repo so I will set up something with loads of files

[jbecchi]

Alright, meanwhile I'll remove both verbose's. If you need me to do some test, let me know.

Thanks for your help so far

[REBELinBLUE]

OK I have managed to recreate the same situation on my development environment

[REBELinBLUE]

Ah ha, it is the same issue as this #262

For now, the only solution I have found is to increase the size of the beanstalk queue http://stackoverflow.com/questions/29199302/job-too-big-pheanstalk-what-can-be-done/32449935#32449935

I will close this issue and leave the discussion in the other one; I am still trying to figure out a nicer way to do this without having to increase the beanstalkd limits. I have a couple of ideas

[jbecchi]

It work perfectally. Thanks for your help.

[REBELinBLUE]

👍 No problem, working on a proper fix right now