Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use namespace in directory structure when cloning repositories #152

Merged
merged 1 commit into from
Apr 2, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 26 additions & 26 deletions features/update.feature
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"

Scenario: Using skip_broken option and adding a new file to repo without write access
Expand Down Expand Up @@ -120,7 +120,7 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"

Scenario: Adding a new file using global values
Expand Down Expand Up @@ -153,7 +153,7 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"

Scenario: Adding a new file overriding global values
Expand Down Expand Up @@ -189,7 +189,7 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"

Scenario: Adding a new file ignoring global values
Expand Down Expand Up @@ -225,7 +225,7 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"

Scenario: Adding a file that ERB can't parse
Expand Down Expand Up @@ -342,7 +342,7 @@ Feature: update
Files changed:
+diff --git a/Gemfile b/Gemfile
"""
Given I run `cat modules/puppet-test/Gemfile`
Given I run `cat modules/maestrodev/puppet-test/Gemfile`
Then the output should contain:
"""
source 'https://somehost.com'
Expand Down Expand Up @@ -405,7 +405,7 @@ Feature: update
Not managing Gemfile in puppet-test
"""
And the exit status should be 0
Given I run `cat modules/puppet-test/Gemfile`
Given I run `cat modules/maestrodev/puppet-test/Gemfile`
Then the output should contain:
"""
source 'https://rubygems.org'
Expand Down Expand Up @@ -488,8 +488,8 @@ Feature: update
"""
some spec_helper fud
"""
And a directory named "modules/puppetlabs-apache/spec"
And a file named "modules/puppetlabs-apache/spec/spec_helper.rb" with:
And a directory named "modules/puppetlabs/puppetlabs-apache/spec"
And a file named "modules/puppetlabs/puppetlabs-apache/spec/spec_helper.rb" with:
"""
This is a fake spec_helper!
"""
Expand All @@ -499,7 +499,7 @@ Feature: update
Not managing spec/spec_helper.rb in puppetlabs-apache
"""
And the exit status should be 0
Given I run `cat modules/puppetlabs-apache/spec/spec_helper.rb`
Given I run `cat modules/puppetlabs/puppetlabs-apache/spec/spec_helper.rb`
Then the output should contain:
"""
This is a fake spec_helper!
Expand Down Expand Up @@ -538,7 +538,7 @@ Feature: update
Files added:
spec/spec_helper.rb
"""
Given I run `cat modules/puppet-test/spec/spec_helper.rb`
Given I run `cat modules/maestrodev/puppet-test/spec/spec_helper.rb`
Then the output should contain:
"""
require 'puppetlabs_spec_helper/module_helper'
Expand Down Expand Up @@ -577,7 +577,7 @@ Feature: update
Given a file named "managed_modules.yml" with:
"""
---
- puppet-test
- maestrodev/puppet-test
"""
And a file named "modulesync.yml" with:
"""
Expand All @@ -597,8 +597,8 @@ Feature: update
require '<%= required %>'
<% end %>
"""
Given I run `git init modules/puppet-test`
Given a file named "modules/puppet-test/.git/config" with:
Given I run `git init modules/maestrodev/puppet-test`
Given a file named "modules/maestrodev/puppet-test/.git/config" with:
"""
[core]
repositoryformatversion = 0
Expand Down Expand Up @@ -667,7 +667,7 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"

Scenario: When specifying configurations in managed_modules.yml and using a filter
Expand Down Expand Up @@ -702,9 +702,9 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"
And a directory named "modules/puppet-blacksmith" should not exist
And a directory named "modules/maestrodev/puppet-blacksmith" should not exist

Scenario: When specifying configurations in managed_modules.yml and using a negative filter
Given a file named "managed_modules.yml" with:
Expand Down Expand Up @@ -738,15 +738,15 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/test`
Given I run `cat modules/maestrodev/puppet-test/test`
Then the output should contain "aruba"
And a directory named "modules/puppet-blacksmith" should not exist
And a directory named "modules/maestrodev/puppet-blacksmith" should not exist

Scenario: Updating a module with a .sync.yml file
Given a file named "managed_modules.yml" with:
"""
---
- puppet-test
- maestrodev/puppet-test
"""
And a file named "modulesync.yml" with:
"""
Expand All @@ -766,8 +766,8 @@ Feature: update
require '<%= required %>'
<% end %>
"""
Given I run `git init modules/puppet-test`
Given a file named "modules/puppet-test/.git/config" with:
Given I run `git init modules/maestrodev/puppet-test`
Given a file named "modules/maestrodev/puppet-test/.git/config" with:
"""
[core]
repositoryformatversion = 0
Expand All @@ -780,7 +780,7 @@ Feature: update
url = https://github.com/maestrodev/puppet-test.git
fetch = +refs/heads/*:refs/remotes/origin/*
"""
Given a file named "modules/puppet-test/.sync.yml" with:
Given a file named "modules/maestrodev/puppet-test/.sync.yml" with:
"""
---
spec/spec_helper.rb:
Expand Down Expand Up @@ -824,9 +824,9 @@ Feature: update
Files added:
test
"""
Given I run `cat modules/puppet-test/.git/config`
Given I run `cat modules/maestrodev/puppet-test/.git/config`
Then the output should contain "url = https://github.com/maestrodev/puppet-test.git"
Given I run `cat modules/puppet-lib-file_concat/.git/config`
Given I run `cat modules/electrical/puppet-lib-file_concat/.git/config`
Then the output should contain "url = https://github.com/electrical/puppet-lib-file_concat.git"

Scenario: Modifying an existing file with values exposed by the module
Expand Down Expand Up @@ -858,7 +858,7 @@ Feature: update
Files changed:
+diff --git a/README.md b/README.md
"""
Given I run `cat modules/puppet-test/README.md`
Given I run `cat modules/maestrodev/puppet-test/README.md`
Then the output should contain:
"""
echo 'https://github.com/maestrodev'
Expand Down
23 changes: 11 additions & 12 deletions lib/modulesync.rb
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ def self.local_file(config_path, file)
"#{config_path}/#{MODULE_FILES_DIR}/#{file}"
end

def self.module_file(project_root, puppet_module, file)
"#{project_root}/#{puppet_module}/#{file}"
def self.module_file(project_root, namespace, puppet_module, file)
"#{project_root}/#{namespace}/#{puppet_module}/#{file}"
end

def self.local_files(path)
Expand Down Expand Up @@ -78,16 +78,17 @@ def self.hook(options)
end

def self.manage_file(filename, settings, options)
namespace = settings.additional_settings[:namespace]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is correct. If a module is overridden then module_name extracts it from the module name rather than the namespace setting. Wouldn't this break if namespace/module is used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it does. Can you imagine a test case that would make the code fail?

module_name = settings.additional_settings[:puppet_module]
configs = settings.build_file_configs(filename)
if configs['delete']
Renderer.remove(module_file(options[:project_root], module_name, filename))
Renderer.remove(module_file(options[:project_root], namespace, module_name, filename))
else
templatename = local_file(options[:configs], filename)
begin
erb = Renderer.build(templatename)
template = Renderer.render(erb, configs)
Renderer.sync(template, module_file(options[:project_root], module_name, filename))
Renderer.sync(template, module_file(options[:project_root], namespace, module_name, filename))
rescue # rubocop:disable Lint/RescueWithoutErrorClass
STDERR.puts "Error while rendering #{filename}"
raise
Expand All @@ -101,20 +102,18 @@ def self.manage_module(puppet_module, module_files, module_options, defaults, op
raise unless options[:skip_broken]
end

puts "Syncing #{puppet_module}"
namespace, module_name = module_name(puppet_module, options[:namespace])
git_repo = "#{namespace}/#{module_name}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is correct. IMHO you should introduce a new variable namespace_root that is the project root + namespace instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just a matter of taste, or a technical issue?

new variable namespace_root that is the project root + namespace

What do you intend with "project root"? Not the repository? Can you make a concrete example?

P.S.: You can write, "Replace this by ...". I promise, I won't be offended.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just a matter of taste, or a technical issue?

I think it should be that way because the code that's called (Git.pull) does a Dir.mkdir(project_root) unless Dir.exist?(project_root). Your version changes to rely on git clone to create parent directories. My git (2.14.4) does this, but I don't know if all git versions do this. If we rely on git to create parent directories, we might as well remove the Dir.mkdir there. This is me trying to figure out what's going on because I don't know the project that well.

What do you intend with "project root"? Not the repository? Can you make a concrete example?

I meant namespace_root = File.join(options[:project_root], namespace) (though I'm not attached to the variable name).

P.S.: You can write, "Replace this by ...". I promise, I won't be offended.

That sounds like I know what I'm doing :P

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alight, I'm really trying -- don't get me wrong!

There is a problem in the code.

  1. First, "project_root" has nothing to do with the "Git project" (of the repo being cloned locally), which would be the obvious thing to understand. In fact, oddly (or obviously when you read the code long enough), it's the root of this project.
  2. Second, the "namespace" does not belong to this project (but to the ones being cloned locally), hence it would in fact be irritating to combine that project root with this (dynamic) namespace. There must be a better way.

And while I understand the issue with relying on Git to create subdirectories "git_repo", on the other hand, nails the target appropriately. Let me see if I can find a solution that is better than both worlds... 🤞

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this problem is artificial. When we look at the Git.pull code we see that the Dir.mkdir is needed. But not for the Git operation, but because we want to do Git things in that directory.

We can still try to prove this is false and will fail. If we manage to come up with a test case that fails. I think that's the route that makes really sense. (I rely a lot on tests, especially for languages that I don't fully master. The existing ones have helped me to get as far as I am now.)

unless options[:offline]
git_base = options[:git_base]
git_uri = "#{git_base}#{namespace}"
Git.pull(git_uri, module_name, options[:branch], options[:project_root], module_options || {})
Git.pull(options[:git_base], git_repo, options[:branch], options[:project_root], module_options || {})
end
module_configs = Util.parse_config("#{options[:project_root]}/#{module_name}/#{MODULE_CONF_FILE}")
module_configs = Util.parse_config("#{options[:project_root]}/#{namespace}/#{module_name}/#{MODULE_CONF_FILE}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use git_repo to stay consistent

settings = Settings.new(defaults[GLOBAL_DEFAULTS_KEY] || {},
defaults,
module_configs[GLOBAL_DEFAULTS_KEY] || {},
module_configs,
:puppet_module => module_name,
:git_base => git_base,
:git_base => options[:git_base],
:namespace => namespace)
settings.unmanaged_files(module_files).each do |filename|
puts "Not managing #{filename} in #{module_name}"
Expand All @@ -124,10 +123,10 @@ def self.manage_module(puppet_module, module_files, module_options, defaults, op
files_to_manage.each { |filename| manage_file(filename, settings, options) }

if options[:noop]
Git.update_noop(module_name, options)
Git.update_noop(git_repo, options)
elsif !options[:offline]
# Git.update() returns a boolean: true if files were pushed, false if not.
pushed = Git.update(module_name, files_to_manage, options)
pushed = Git.update(git_repo, files_to_manage, options)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekohl Interestingly, even though the tests pass this may turn out to be an issue. Because in Git.update the git_repo value is prepended with a :project_root value. Not sure, though, honestly. (Ruby is not my first language.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense now, chiming in with my discovery above: The :project_root has nothing to do with Git. It's just the place where all Git operations are meant to be performed in.

The Git.update only does things in the final Git project root = :project_root + Git repository path (even if this is two levels).

return nil unless pushed && options[:pr]

# We only do GitHub PR work if the GITHUB_TOKEN variable is set in the environment.
Expand Down
3 changes: 2 additions & 1 deletion lib/modulesync/git.rb
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@ def self.switch_branch(repo, branch)
end

def self.pull(git_base, name, branch, project_root, opts)
puts "Syncing #{name}"
Dir.mkdir(project_root) unless Dir.exist?(project_root)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be dropped if you're going to pass in name as dir/subdir because you're going to rely on git to create a directory anyway.


# Repo needs to be cloned in the cwd
if !Dir.exist?("#{project_root}/#{name}") || !Dir.exist?("#{project_root}/#{name}/.git")
puts 'Cloning repository fresh'
remote = opts[:remote] || (git_base.start_with?('file://') ? "#{git_base}/#{name}" : "#{git_base}/#{name}.git")
remote = opts[:remote] || (git_base.start_with?('file://') ? "#{git_base}#{name}" : "#{git_base}#{name}.git")
local = "#{project_root}/#{name}"
puts "Cloning from #{remote}"
repo = ::Git.clone(remote, local)
Expand Down