Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Poetry 1.5 lockfiles #7834

Merged
merged 4 commits into from
Aug 19, 2023
Merged

Add support for Poetry 1.5 lockfiles #7834

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8e82428
Extract helper to run poetry command
deivid-rodriguez Aug 17, 2023
a75e001
Return stdout when poetry command succeeds, raise error when it fails
deivid-rodriguez Aug 17, 2023
37c5a94
Use a better strategy to parse Poetry dependency types
deivid-rodriguez Aug 17, 2023
77fe93b
Fix minor typos
jeffwidman Aug 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 38 additions & 5 deletions python/lib/dependabot/python/file_parser/pyproject_files_parser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
require "dependabot/python/file_parser"
require "dependabot/python/requirement"
require "dependabot/errors"
require "dependabot/python/helpers"
require "dependabot/python/name_normaliser"

module Dependabot
Expand Down Expand Up @@ -44,16 +45,20 @@ def pyproject_dependencies
end

def poetry_dependencies
@poetry_dependencies ||= parse_poetry_dependencies
end

def parse_poetry_dependencies
dependencies = Dependabot::FileParsers::Base::DependencySet.new

POETRY_DEPENDENCY_TYPES.each do |type|
deps_hash = parsed_pyproject.dig("tool", "poetry", type) || {}
dependencies += parse_poetry_dependencies(type, deps_hash)
dependencies += parse_poetry_dependency_group(type, deps_hash)
end

groups = parsed_pyproject.dig("tool", "poetry", "group") || {}
groups.each do |group, group_spec|
dependencies += parse_poetry_dependencies(group, group_spec["dependencies"])
dependencies += parse_poetry_dependency_group(group, group_spec["dependencies"])
end
dependencies
end
Expand Down Expand Up @@ -92,7 +97,7 @@ def pep621_dependencies
dependencies
end

def parse_poetry_dependencies(type, deps_hash)
def parse_poetry_dependency_group(type, deps_hash)
dependencies = Dependabot::FileParsers::Base::DependencySet.new

deps_hash.each do |name, req|
Expand Down Expand Up @@ -154,21 +159,49 @@ def lockfile_dependencies
parsed_lockfile.fetch("package", []).each do |details|
next if source_types.include?(details.dig("source", "type"))

name = normalise(details.fetch("name"))

dependencies <<
Dependency.new(
name: normalise(details.fetch("name")),
name: name,
version: details.fetch("version"),
requirements: [],
package_manager: "pip",
subdependency_metadata: [{
production: details["category"] != "dev"
production: production_dependency_names.include?(name)
}]
)
end

dependencies
end

def production_dependency_names
@production_dependency_names ||= parse_production_dependency_names
end

def parse_production_dependency_names
SharedHelpers.in_a_temporary_directory do
File.write(pyproject.name, pyproject.content)
File.write(lockfile.name, lockfile.content)

begin
output = Helpers.run_poetry_command("pyenv exec poetry show --only main")
Copy link
Member

@jeffwidman jeffwidman Aug 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for using the poetry CLI rather than inspecting the file directly.

However, the docs for show indicate that --only main and --without dev are both workable here...

At first I was thinking that --without dev might be better here, as main is the default group so there may be other groups, until I saw this note here:

Dependency groups, other than the implicit main group, must only contain dependencies you need in your development process. Installing them is only possible by using Poetry.

So I'm guessing docs/tests, etc are other groups that won't be considered part of main, but aren't part of dev either...

The :dependabot: concept of dep categories isn't quite so sophisticated, we tend to simply have "prod" vs "dev", and in this method solely want prod deps, not "non-dev deps", so --only main makes sense to me!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, in general I tend to consider "dev", everything that's not production/runtime, that's why I went with --only main.


output.split("\n").map { |line| line.split.first }
rescue SharedHelpers::HelperSubprocessFailed
# Sometimes, we may be dealing with an old lockfile that our
# poetry version can't show dependency information for. Other
# commands we use like `poetry update` are more resilient and
# automatically heal the lockfile. So we rescue the error and make
# a best effort approach to this.
poetry_dependencies.dependencies.filter_map do |dep|
dep.name if dep.production?
end
end
end
end

def version_from_lockfile(dep_name)
return unless parsed_lockfile

Expand Down
34 changes: 0 additions & 34 deletions python/lib/dependabot/python/file_parser/subdependency_type_parser.rb
View file
Open in desktop

This file was deleted.

30 changes: 3 additions & 27 deletions python/lib/dependabot/python/file_updater/poetry_file_updater.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
require "dependabot/python/version"
require "dependabot/python/requirement"
require "dependabot/python/file_parser/python_requirement_parser"
require "dependabot/python/file_parser/subdependency_type_parser"
require "dependabot/python/file_updater"
require "dependabot/python/helpers"
require "dependabot/python/native_helpers"
require "dependabot/python/name_normaliser"

Expand Down Expand Up @@ -157,7 +157,7 @@ def lock_declaration_to_new_version!(poetry_object, dep)
end

def create_declaration_at_new_version!(poetry_object, dep)
subdep_type = subdependency_type_parser.subdep_type(dep)
subdep_type = dep.production? ? "dependencies" : "dev-dependencies"

poetry_object[subdep_type] ||= {}
poetry_object[subdep_type][dep.name] = dep.version
Expand Down Expand Up @@ -197,24 +197,7 @@ def run_poetry_update_command
end

def run_poetry_command(command, fingerprint: nil)
start = Time.now
command = SharedHelpers.escape_command(command)
stdout, process = Open3.capture2e(command)
time_taken = Time.now - start

# Raise an error with the output from the shell session if Pipenv
# returns a non-zero status
return if process.success?

raise SharedHelpers::HelperSubprocessFailed.new(
message: stdout,
error_context: {
command: command,
fingerprint: fingerprint,
time_taken: time_taken,
process_exit_value: process.to_s
}
)
Helpers.run_poetry_command(command, fingerprint: fingerprint)
end

def write_temporary_dependency_files(pyproject_content)
Expand Down Expand Up @@ -291,13 +274,6 @@ def python_requirement_parser
)
end

def subdependency_type_parser
@subdependency_type_parser ||=
FileParser::PoetrySubdependencyTypeParser.new(
lockfile: lockfile
)
end

def language_version_manager
@language_version_manager ||=
LanguageVersionManager.new(
Expand Down
34 changes: 34 additions & 0 deletions python/lib/dependabot/python/helpers.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# frozen_string_literal: true

require "time"
require "open3"

require "dependabot/errors"
require "dependabot/shared_helpers"

module Dependabot
module Python
module Helpers
def self.run_poetry_command(command, fingerprint: nil)
start = Time.now
command = SharedHelpers.escape_command(command)
stdout, stderr, process = Open3.capture3(command)
time_taken = Time.now - start

# Raise an error with the output from the shell session if Poetry
# returns a non-zero status
return stdout if process.success?

raise SharedHelpers::HelperSubprocessFailed.new(
message: stderr,
error_context: {
command: command,
fingerprint: fingerprint,
time_taken: time_taken,
process_exit_value: process.to_s
}
)
end
end
end
end
34 changes: 6 additions & 28 deletions python/lib/dependabot/python/update_checker/poetry_version_resolver.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@
require "dependabot/shared_helpers"
require "dependabot/python/file_parser"
require "dependabot/python/file_parser/python_requirement_parser"
require "dependabot/python/file_parser/subdependency_type_parser"
require "dependabot/python/file_updater/pyproject_preparer"
require "dependabot/python/update_checker"
require "dependabot/python/version"
require "dependabot/python/requirement"
require "dependabot/python/helpers"
require "dependabot/python/native_helpers"
require "dependabot/python/authed_url_builder"
require "dependabot/python/name_normaliser"
Expand Down Expand Up @@ -260,8 +260,6 @@ def set_target_dependency_req(pyproject_content, updated_requirement)

# If this is a sub-dependency, add the new requirement
unless dependency.requirements.find { |r| r[:file] == pyproject.name }
subdep_type = subdependency_type_parser.subdep_type(dependency)

poetry_object[subdep_type] ||= {}
poetry_object[subdep_type][dependency.name] = updated_requirement
end
Expand All @@ -281,20 +279,17 @@ def update_dependency_requirement(toml_node, requirement)
end
end

def subdep_type
dependency.production? ? "dependencies" : "dev-dependencies"
end

def python_requirement_parser
@python_requirement_parser ||=
FileParser::PythonRequirementParser.new(
dependency_files: dependency_files
)
end

def subdependency_type_parser
@subdependency_type_parser ||=
FileParser::PoetrySubdependencyTypeParser.new(
lockfile: lockfile
)
end

def language_version_manager
@language_version_manager ||=
LanguageVersionManager.new(
Expand All @@ -315,24 +310,7 @@ def lockfile
end

def run_poetry_command(command, fingerprint: nil)
start = Time.now
command = SharedHelpers.escape_command(command)
stdout, process = Open3.capture2e(command)
time_taken = Time.now - start

# Raise an error with the output from the shell session if poetry
# returns a non-zero status
return if process.success?

raise SharedHelpers::HelperSubprocessFailed.new(
message: stdout,
error_context: {
command: command,
fingerprint: fingerprint,
time_taken: time_taken,
process_exit_value: process.to_s
}
)
Helpers.run_poetry_command(command, fingerprint: fingerprint)
end

def normalise(name)
Expand Down