Don't seem to be able to change the version

[tomdz]

I'm trying to install version 0.20.6 (for logstash 1.13) with the 0.3.0 cookbook version and I'm trying to set the version via a wrapper cookbook (to allow for multiple different elasticsearch clusters, not just for logstash). However the version doesn't seem to take effect. The wrapper cookbook has:

node.override["elasticsearch"]["version"] = "0.20.6"

and the output is:

[2013-06-17T21:40:32+00:00] INFO: Processing ark[elasticsearch] action install (elasticsearch::default line 71)
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: release_basename is elasticsearch-0.90.1.tar.gz
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: file_extension is tar.gz
[2013-06-17T21:40:32+00:00] DEBUG: path is /usr/local/elasticsearch-0.20.6
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: new_resource.release_file
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: cmd: /bin/tar xzf /tmp/vagrant-chef-1/elasticsearch.tar.gz --strip-components=1
[2013-06-17T21:40:32+00:00] INFO: Processing directory[/usr/local/elasticsearch-0.20.6] action create (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/ark/providers/default.rb line 37)
[2013-06-17T21:40:32+00:00] INFO: Processing remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] action create (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/ark/providers/default.rb line 43)
[2013-06-17T21:40:32+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] checking for changes
[2013-06-17T21:40:32+00:00] DEBUG: Sending HTTP Request via GET to download.elasticsearch.org:80/elasticsearch/elasticsearch/elasticsearch-0.90.1.tar.gz
[2013-06-17T21:40:32+00:00] DEBUG: Streaming download from http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.1.tar.gz to tempfile /tmp/chef-rest20130617-22513-u49c2g
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] checking for file existence of /tmp/vagrant-chef-1/elasticsearch.tar.gz
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] file exists at /tmp/vagrant-chef-1/elasticsearch.tar.gz
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] target checksum: 01e7df982569ff7f13008cbef361c52f3cd8a3b289423483be5251e7ed4f1064
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] source checksum: 2a5d4390a24a7feff12a0f1ac0e75c53b333316c44cf2e783dad9d7d7b03fb5b
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] checksum changed from 01e7df982569ff7f13008cbef361c52f3cd8a3b289423483be5251e7ed4f1064 to 2a5d4390a24a7feff12a0f1ac0e75c53b333316c44cf2e783dad9d7d7b03fb5b
[2013-06-17T21:40:58+00:00] INFO: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] backed up to /var/chef/backup/tmp/vagrant-chef-1/elasticsearch.tar.gz.chef-20130617214058
[2013-06-17T21:40:58+00:00] INFO: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] updated

so it looks like the version attribute is set correctly (as evident by the directory name), but the filename doesn't use the updated attribute.

[karmi]

This is strange. What if you use a specific role for Logstash clusters and override it in the role? I don't think there's a Ruby test for version override, but I did tested it manually for #61, #68 and #69.

[tomdz]

Sorry, I can't use roles as they are not versioned (hence the wrapper cookbook).

[karmi]

Understood -- which Chef version is this? I'll try to replicate it, but it will take some time, I'm afraid, since I'm tied to another project at the moment.

[tomdz]

I run my tests currently in Vagrant 1.2.2 with the Berkshelf plugin. The VM is Ubuntu 12.04.2 and chef-solo 11.4.2 (omnibus). The wrapper cookbook recipe looks like this:

node.override["elasticsearch"]["version"] = "0.20.6"

include_recipe "elasticsearch"
include_recipe "elasticsearch::plugins"
include_recipe "elasticsearch::proxy"

Fwiw, I also tried to set the attribute before including elasticsearch with no effect, e.g. with the attribute file:

override["elasticsearch"]["version"] = "0.20.6"

include_attribute "elasticsearch"

and the recipe

include_recipe "elasticsearch"
include_recipe "elasticsearch::plugins"
include_recipe "elasticsearch::proxy"

I can stay on commit 9b20d84 for now (which is the last one before you changed the default version).

[karmi]

This is frustrating, I understand that... I'll try to debug the issue, and see whether it's with the cookbook or Ark...

[tomdz]

I had the wrapper cookbook log the configuration (with Chef::Log which happens in the compile phase):

[2013-06-17T22:06:22+00:00] INFO: elasticsearch config: {"version"=>"0.20.6",
"host"=>"http://download.elasticsearch.org", "repository"=>"elasticsearch/elasticsearch",
"filename"=>"elasticsearch-0.90.1.tar.gz",
"download_url"=>"http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.1.tar.gz",
"cluster"=>{"name"=>"elasticsearch"}, "node"=>{"name"=>"logstash-server", "max_local_storage_nodes"=>1},
"dir"=>"/usr/local", "user"=>"elasticsearch", "path"=>{"conf"=>"/usr/local/etc/elasticsearch",
"data"=>"/usr/local/var/data/elasticsearch", "logs"=>"/usr/local/var/log/elasticsearch"},
"pid_path"=>"/usr/local/var/run/elasticsearch", "pid_file"=>"/usr/local/var/run/elasticsearch/logstash_server.pid",
"allocated_memory"=>"512m", "bootstrap"=>{"mlockall"=>false}, "limits"=>{"memlock"=>"unlimited",
"nofile"=>"64000"}, "index"=>{"mapper"=>{"dynamic"=>true}}, "action"=>{"auto_create_index"=>true,
"disable_delete_all_indices"=>true}, "discovery"=>{"zen"=>{"ping"=>{"multicast"=>{"enabled"=>false},
"unicast"=>{"hosts"=>["192.168.27.3"]}}, "minimum_master_nodes"=>1}, "type"=>nil, "ec2"=>{"groups"=>nil,
"tag"=>{}}}, "gateway"=>{"type"=>nil, "expected_nodes"=>1}, "thread_stack_size"=>"256k", "custom_config"=>{},
"plugins"=>{"elasticsearch-cloud-aws"=>{"version"=>"1.12.0"}, "karmi/elasticsearch-paramedic"=>{}},
"plugin"=>{"mandatory"=>[]}, "cloud"=>{"aws"=>{"access_key"=>nil, "secret_key"=>nil, "region"=>nil},
"ec2"=>{"endpoint"=>nil}, "node"=>{"auto_attributes"=>true}}, "data"=>{"devices"=>{}},
"deb_url"=>"https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.1.deb",
"deb_sha"=>"3ea1045b037912d9e2ebfbce5b6fc3aaffb77e5d", "logging"=>{"action"=>"DEBUG",
"com.amazonaws"=>"WARN", "index.search.slowlog"=>"TRACE, index_search_slow_log_file",
"index.indexing.slowlog"=>"TRACE, index_indexing_slow_log_file"}, "nginx"=>{"port"=>"8080",
"dir"=>"/etc/nginx", "user"=>"www-data", "log_dir"=>"/var/log/nginx", "users"=>[{"username"=>"admin",
"password"=>"password"}], "passwords_file"=>"/usr/local/etc/elasticsearch/passwords",
"allow_cluster_api"=>true, "allow_status"=>false, "client_max_body_size"=>"50M"},
"rpm_url"=>"https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.1.noarch.rpm",
"rpm_sha"=>"a56eb8a88a15adad9ab8e59168c6cd51a1bd9ba3", "network"=>{"host"=>"192.168.27.3"},
"master"=>true}

[karmi]

Thanks, this seems like the version is set correctly, but is not propagated further in https://github.com/elasticsearch/cookbook-elasticsearch/blob/master/attributes/default.rb#L18-L19

[chrisroberts]

So in most cases where attributes end up being a composite of other attributes, I'll either set them (or reset them) in the recipe where they are in use. The order things are loaded means that attributes composed within the attribute files will never see the overrides provided either from wrapper cookbooks or roles.

One direct workaround for this now is to explicitly set the required url attribute to the correct location for the required version, and the wrapper cookbook could do this by updating the individual attributes and then composing them for the url.

[davekonopka]

It seems like a problematic idea to blanket merge the default scope of elasticsearch attributes with the normal scope into the normal scope:
https://github.com/elasticsearch/cookbook-elasticsearch/blob/master/attributes/default.rb#L10

The behavior I'm seeing with this is that the version number is being ignored because the download_url ends up set as a normal variable. So default setting of download_url no longer takes precedence in that attribute file in future runs and will never be updated unless overridden explicitly in a wrapper cookbook, role, etc.

I put some debug output and tested locally. The default attributes file is evaluating twice for me. So the first time the default values are not present and not merged into normal scope. On the second evaluation though the default values are all present and put into normal scope.

I'm still not sure why the second evaluation is happening. Regardless though since normal scope attributes are persisted it seems like the approach in the line linked above will be problematic.

[davekonopka]

It looks like double evaluation of attributes/default.rb is due to the include_attribute "elasticsearch::default" statement in attributes/aws.rb and attributes/proxy.rb

It looks like Chef will eval each attributes file once per cookbook and once per include_attribute setting. So on the second eval, everything in default scope under elasticsearch key goes into normal scope.

[albertsun]

Have been experiencing this issue as well. It seems like changing the version attribute does not change the deb_url or rpm_url attributes. We're running Chef 11 with 0.3.3 of this cookbook on an Ubuntu server and it ends up downloading the 0.9.1 .deb file and installing it into a folder named with 0.90.3.

Overriding the deb_url attribute to point to the 0.90.3 download link seemed to fix the problem.

[davekonopka]

I noticed the wip tag applied to this ticket.

Has any work been done on this yet? If not I can take a shot at a pull request.

[Pneumatus]

I don't have a test setup right now to try myself, but can you force a re-resolve of the elasticsearch attributes file after setting the version (so the attributes with interpolated values get changed), i.e.:

node.override["elasticsearch"]["version"] = "0.20.6"

node.from_file(run_context.resolve_attribute("elasticsearch", "default.rb"))

include_recipe "elasticsearch"
include_recipe "elasticsearch::plugins"
include_recipe "elasticsearch::proxy"

(Ref: http://lists.opscode.com/sympa/arc/chef/2013-08/msg00210.html)

[dsennerlov]

I have the same problem.

What I see is that /var/chef/cache/elasticsearch.tar.gz is never overwritten even though a new version of the cookbook is deployed.

I solve it temporarily by adding

name  "elasticsearch-#{node.elasticsearch[:version]}"

to https://github.com/elasticsearch/cookbook-elasticsearch/blob/master/recipes/default.rb#L73

[bkw]

@dsennerlov that leaves you with the install unpacked into /usr/local/elasticsearch-x.xx.x and a symlink elasticsearch-x.xx.x-x.xx.x pointing to it. Probably not what you want.

[karmi]

I'd like to move away from Ark entirely, and manage all the download/extract/symlink/etc actions ourselves, so we have some kind of control over it.

@davekonopka Nobody is working on this specifically right now, if you'd have a fix, please submit a pull request.

[bkw]

+1 for getting rid of ark

[weiner]

Run into the same issue.
I create a fix which worked for me and added it as a pull request.
#165

[mdkent]

👍 seeing this here too in a wrapper cookbook. Temporarily solved with:

node.override["elasticsearch"]["download_url"] = "http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.8.tar.gz"

[lyrixx]

Hello.

Is there any ETA to a real fix ?

[martinb3]

See #178 and #180 -- they seem like duplicates of this issue. Until overriding attributes works differently, derived attributes aren't updated when the first attribute is overridden. Until Chef changes something, this is the way chef works -- you must override any calculated values if you override the variables they refer to. We can either (a) remove all calculated values or (b) convert to LWRPs that wrapper cookbooks can use directly or (b) wait for chef to change how this works.

Until then, there's a definite workaround though -- just copy the three lines and override all three values (example).

[lyrixx]

Thanks. I overridden the value, but it's ... chef ... ;)

[karmi]

Hi, I've merged #242, so this bug should be fixed now. I'm sorry for the inconvenience the bad behaviour caused you. Closing this issue for now, please ping me if you would like to discuss it further.

[karmi]

Hi, I've merged #242, so this bug should be fixed now. I'm sorry for the inconvenience the bad behaviour caused you. Closing this issue for now, please ping me if you would like to discuss it further.