Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Return single package when multiple indexers are used #849

Merged
merged 6 commits into from
Jul 27, 2022
Merged

Return single package when multiple indexers are used #849

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bf0107c
Return single package when multiple indexers are used
jsoriano Jul 18, 2022
13f6a10
Fix format
jsoriano Jul 26, 2022
98f7a74
Avoid duplicate packages
jsoriano Jul 26, 2022
096ba49
Fix changelog
jsoriano Jul 26, 2022
7d5117c
Use index semantics
jsoriano Jul 26, 2022
0116753
Consolidate orders of preference when a package is available in multi…
jsoriano Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Bugfixes

* Return only the latest version of each package when a combined index is used. [#849](https://github.com/elastic/package-registry/pull/849)

### Added

* Add `elastic.subscription` condition to package index metadata, use this value for backwards compatibility with previous `license` field. [#826](https://github.com/elastic/package-registry/pull/826)
Expand Down
41 changes: 41 additions & 0 deletions indexer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ package main

import (
"context"
"sort"
"time"

"github.com/Masterminds/semver/v3"
"github.com/elastic/package-registry/metrics"
"github.com/elastic/package-registry/packages"
)
Expand Down Expand Up @@ -44,5 +46,44 @@ func (c CombinedIndexer) Get(ctx context.Context, opts *packages.GetOptions) (pa
}
packages = packages.Join(p)
}

if !opts.Filter.AllVersions {
return latestPackagesVersion(packages), nil
}

return packages, nil
}

func latestPackagesVersion(source packages.Packages) (result packages.Packages) {
sort.Sort(byNameVersion(source))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you call sort.Sort on the source, it won't alter the original packages slice? I'm double-checking if this is concurrency-safe.

Copy link
Member Author

@jsoriano jsoriano Jul 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will modify the packages slice defined in the CombinerIndexer.Get, that is built on every call to this method and shouldn't be used concurrently in any other place.


current := ""
for _, p := range source {
if p.Name == current {
continue
}

current = p.Name
result = append(result, p)
}

return result
}

type byNameVersion packages.Packages

func (p byNameVersion) Len() int { return len(p) }
func (p byNameVersion) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
func (p byNameVersion) Less(i, j int) bool {
if p[i].Name != p[j].Name {
return p[i].Name < p[j].Name
}

// Newer versions first.
iSemVer, _ := semver.NewVersion(p[i].Version)
jSemVer, _ := semver.NewVersion(p[j].Version)
if iSemVer != nil && jSemVer != nil {
return jSemVer.LessThan(iSemVer)
}
return p[j].Version < p[i].Version
}
82 changes: 82 additions & 0 deletions indexer_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package main

import (
"testing"

"github.com/elastic/package-registry/packages"
"github.com/stretchr/testify/assert"
)

func TestLatestPackagesVersion(t *testing.T) {
newPackage := func(name string, version string) *packages.Package {
p := new(packages.Package)
p.Name = name
p.Version = version
return p
}

cases := []struct {
title string
source packages.Packages
expected packages.Packages
}{
{
title: "single package",
source: packages.Packages{
newPackage("foo", "1.2.3"),
},
expected: packages.Packages{
newPackage("foo", "1.2.3"),
},
},
{
title: "single package sorted versions",
source: packages.Packages{
newPackage("foo", "1.2.3"),
newPackage("foo", "1.2.2"),
newPackage("foo", "1.2.1"),
},
expected: packages.Packages{
newPackage("foo", "1.2.3"),
},
},
{
title: "single package unsorted versions",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about adding a case where both indexers contain the same package version?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the case where both indexers contain the same package version, what would the behavior be for a query with ?all=true?

I would think even in this case the response should only contain one item for that package version. The reason being that the other APIs don't provide a way to differentiate between the two items (e.g. the only specifiers are name and version in the download request /epr/example/example-1.0.1.zip). Assuming the solution is to only return one in the search/?all=true response then it would be good to specify somewhere what the priority order is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that in that case we have to decide that some indexers have precendence over the others, I will take a look to this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a change so indexes defined later have precedence over the ones defined earlier. I have made that because currently we are defining the zip indexer, with support for signatures, after the plain file indexer, so now if a package is in both indexers, the zip with signature will have precedence.

source: packages.Packages{
newPackage("foo", "1.2.2"),
newPackage("foo", "1.2.1"),
newPackage("foo", "1.2.3"),
},
expected: packages.Packages{
newPackage("foo", "1.2.3"),
},
},
{
title: "multiple packages with multiple versions",
source: packages.Packages{
newPackage("foo", "1.2.2"),
newPackage("bar", "1.2.1"),
newPackage("bar", "1.2.2"),
newPackage("foo", "1.2.1"),
newPackage("bar", "1.2.3"),
newPackage("foo", "1.2.3"),
},
expected: packages.Packages{
newPackage("bar", "1.2.3"),
newPackage("foo", "1.2.3"),
},
},
}

for _, c := range cases {
t.Run(c.title, func(t *testing.T) {
result := latestPackagesVersion(c.source)
assert.EqualValues(t, c.expected, result)
})
}

}
5 changes: 4 additions & 1 deletion main_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,10 @@ func init() {

func TestEndpoints(t *testing.T) {
packagesBasePaths := []string{"./testdata/second_package_path", "./testdata/package"}
indexer := packages.NewFileSystemIndexer(packagesBasePaths...)
indexer := NewCombinedIndexer(
packages.NewFileSystemIndexer(packagesBasePaths...),
packages.NewZipFileSystemIndexer("./testdata/local-storage"),
)

err := indexer.Init(context.Background())
require.NoError(t, err)
Expand Down
4 changes: 2 additions & 2 deletions testdata/generated/categories-experimental.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
{
"id": "azure",
"title": "Azure",
"count": 1
"count": 2
},
{
"id": "cloud",
Expand All @@ -22,7 +22,7 @@
{
"id": "crm",
"title": "CRM",
"count": 1
"count": 2
},
{
"id": "custom",
Expand Down
4 changes: 2 additions & 2 deletions testdata/generated/categories-include-policy-templates.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
{
"id": "azure",
"title": "Azure",
"count": 2
"count": 3
},
{
"id": "crm",
"title": "CRM",
"count": 2
"count": 3
},
{
"id": "custom",
Expand Down
4 changes: 2 additions & 2 deletions testdata/generated/categories-prerelease.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
{
"id": "azure",
"title": "Azure",
"count": 1
"count": 2
},
{
"id": "cloud",
Expand All @@ -22,7 +22,7 @@
{
"id": "crm",
"title": "CRM",
"count": 1
"count": 2
},
{
"id": "custom",
Expand Down
4 changes: 2 additions & 2 deletions testdata/generated/categories.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
{
"id": "azure",
"title": "Azure",
"count": 1
"count": 2
},
{
"id": "crm",
"title": "CRM",
"count": 1
"count": 2
},
{
"id": "custom",
Expand Down
59 changes: 59 additions & 0 deletions testdata/generated/search-all.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,35 @@
"custom"
]
},
{
"name": "nodirentries",
"title": "Example Integration",
"version": "1.0.0",
"release": "ga",
"description": "This is a zip package without directory entries.",
"type": "integration",
"download": "/epr/nodirentries/nodirentries-1.0.0.zip",
"path": "/package/nodirentries/1.0.0",
"policy_templates": [
{
"name": "logs",
"title": "Logs datasource",
"description": "Datasource for your log files."
}
],
"conditions": {
"kibana": {
"version": "~7.x.x"
}
},
"owner": {
"github": "ruflin"
},
"categories": [
"crm",
"azure"
]
},
{
"name": "example",
"title": "Example Integration",
Expand Down Expand Up @@ -69,6 +98,36 @@
"azure"
]
},
{
"name": "example",
"title": "Example Integration",
"version": "1.0.1",
"release": "ga",
"description": "This is the example integration",
"type": "integration",
"download": "/epr/example/example-1.0.1.zip",
"path": "/package/example/1.0.1",
"policy_templates": [
{
"name": "logs",
"title": "Logs datasource",
"description": "Datasource for your log files."
}
],
"conditions": {
"kibana": {
"version": "~7.x.x"
}
},
"owner": {
"github": "ruflin"
},
"categories": [
"crm",
"azure"
],
"signature_path": "/epr/example/example-1.0.1.zip.sig"
},
{
"name": "example",
"title": "Example Integration",
Expand Down
43 changes: 35 additions & 8 deletions testdata/generated/search-kibana721.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,22 +20,19 @@
]
},
{
"name": "example",
"name": "nodirentries",
"title": "Example Integration",
"version": "1.0.0",
"release": "ga",
"description": "This is the example integration",
"description": "This is a zip package without directory entries.",
"type": "integration",
"download": "/epr/example/example-1.0.0.zip",
"path": "/package/example/1.0.0",
"download": "/epr/nodirentries/nodirentries-1.0.0.zip",
"path": "/package/nodirentries/1.0.0",
"policy_templates": [
{
"name": "logs",
"title": "Logs datasource",
"description": "Datasource for your log files.",
"categories": [
"datastore"
]
"description": "Datasource for your log files."
}
],
"conditions": {
Expand All @@ -51,6 +48,36 @@
"azure"
]
},
{
"name": "example",
"title": "Example Integration",
"version": "1.0.1",
"release": "ga",
"description": "This is the example integration",
"type": "integration",
"download": "/epr/example/example-1.0.1.zip",
"path": "/package/example/1.0.1",
"policy_templates": [
{
"name": "logs",
"title": "Logs datasource",
"description": "Datasource for your log files."
}
],
"conditions": {
"kibana": {
"version": "~7.x.x"
}
},
"owner": {
"github": "ruflin"
},
"categories": [
"crm",
"azure"
],
"signature_path": "/epr/example/example-1.0.1.zip.sig"
},
{
"name": "foo",
"title": "Foo",
Expand Down
Loading