Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix broken external dependencies #16326

Closed
Pungyeon opened this issue Feb 14, 2020 · 8 comments
Closed

Fix broken external dependencies #16326

Pungyeon opened this issue Feb 14, 2020 · 8 comments

Comments

@Pungyeon
Copy link

Pungyeon commented Feb 14, 2020 •
edited
Loading

Describe the enhancement:
It is currently not possible to import github.com/elastic/beats/cmd.GenRootWithSettings using go modules. This will break (several things):

  1. This will retrieve a really old version of the docker-client, which is incompatible
  2. The same is true for k8s.io client
  3. The github.com/dop251/goja dependencies which are included (vendored) in the github repository, is completely broken. The function RegisterSimpleMaptype() does not exist anymore, and even worse, there are no releases and no branches for this library. The only way to fix this would be to fork the project and revert the changes back to before the deletion of said function.

I therefore propose to fix these dependencies and include a go.mod, to make working with the libbeat library much easier.

I am willing to help with this project and will happily contribute to fixing this.

Describe a specific use case for the enhancement or feature:
Make writing custom beats much easier (when using go modules).
@kvch
Copy link
Contributor

kvch commented Mar 3, 2020

Beats has just adopted go modules as a dependency manager. Does this fix your problems?

@Pungyeon
Copy link
Author

Pungyeon commented Mar 3, 2020

Hi @kvch I though this would fix it, but it seems that there are still some external dependencies which aren't being imported correctly, when using the libbeat library from a new project.

Perhaps I am doing something wrong though.... This is the error message that I receive, when I try to build my project:

# github.com/elastic/beats/libbeat/common/docker
../../go/pkg/mod/github.com/elastic/beats@v7.6.0+incompatible/libbeat/common/docker/client.go:37:12: undefined: client.Opt
../../go/pkg/mod/github.com/elastic/beats@v7.6.0+incompatible/libbeat/common/docker/client.go:46:23: undefined: client.WithVersion
../../go/pkg/mod/github.com/elastic/beats@v7.6.0+incompatible/libbeat/common/docker/client.go:49:23: undefined: client.WithAPIVersionNegotiation
../../go/pkg/mod/github.com/elastic/beats@v7.6.0+incompatible/libbeat/common/docker/client.go:52:9: undefined: client.NewClientWithOpts
# k8s.io/client-go/rest
../../go/pkg/mod/k8s.io/client-go@v11.0.0+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher
	have (*versioned.Decoder)
	want (watch.Decoder, watch.Reporter)
# github.com/elastic/beats/libbeat/processors/script/javascript
../../go/pkg/mod/github.com/elastic/beats@v7.6.0+incompatible/libbeat/processors/script/javascript/session.go:260:14: s.Runtime().RegisterSimpleMapType undefined (type *goja.Runtime has no field or method RegisterSimpleMapType)
# github.com/corticph/benchmark-model/beater
beater/beater.go:84:5: cannot use "github.com/Pungyeon/beats/libbeat/common".MapStr literal (type "github.com/Pungyeon/beats/libbeat/common".MapStr) as type "github.com/elastic/beats/libbeat/common".MapStr in field value

@kvch
Copy link
Contributor

kvch commented Mar 3, 2020

Could you share your go.mod?

We are using a forks of a few dependencies. You need to add those to the replace section of your Beats because the go version we are using is a bit limited in that regard. I think adding those directives will fix your issue.

Also, I am working on moving Beat generators to go modules as well. It is in progress, but I expect it to be done soon. You might be interested in it: #16288

@Pungyeon
Copy link
Author

Pungyeon commented Mar 3, 2020

@kvch , so having tried quite a few things. I started over... in the end I was still receiving the following error:

../../go/pkg/mod/k8s.io/client-go@v11.0.0+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher
        have (*versioned.Decoder)
        want (watch.Decoder, watch.Reporter)

But updating the version to the following:

go get k8s.io/client-go@kubernetes-1.16.0

Seemed to work a charm. So, now everything is indeed working as intended.

Thank you so much for helping with addressing this 🙏 You are awesome !! 👏

@Pungyeon Pungyeon closed this as completed Mar 5, 2020
@richard-mauri
Copy link

Is it true that beats has adopted gomodules?
Was this publicly announced and if so where and are there samples to adopting it?
EllieMae has several golang projects that need to build beats from source and so far have ben struggling with elastic beat go modules.
I want to upgrade our important custom beats to test if it helps several production issues.

@richard-mauri
Copy link

richard-mauri commented Aug 6, 2020 •
edited
Loading

I got it to work using instructions https://github.com/elastic/go-elasticsearch
The key for me was to has this in go.mod: require github.com/elastic/beats/v7 v7.8.0 and to change import statements to refer to /v7 like import "github.com/elastic/beats/v7/libbeat/cmd/instance"

I think v7.8.0 is the latest released beats...

Thanks eats team for getting go modules.

@bjanders
Copy link

I have this issue too. I receive:

# github.com/elastic/beats/v7/libbeat/processors/script/javascript
../../go/pkg/mod/github.com/elastic/beats/v7@v7.17.8/libbeat/processors/script/javascript/session.go:267:15: s.Runtime().RegisterSimpleMapType undefined (type *goja.Runtime has no field or method RegisterSimpleMapType)
My go.mod: 
go 1.18

require (
	github.com/elastic/beats/v7 v7.17.8
	github.com/magefile/mage v1.14.0
)

require (
	github.com/Microsoft/go-winio v0.5.2 // indirect
	github.com/Shopify/sarama v1.38.1 // indirect
	github.com/StackExchange/wmi v0.0.0-20170221213301-9f32b5905fd6 // indirect
	github.com/akavel/rsrc v0.8.0 // indirect
	github.com/armon/go-radix v1.0.0 // indirect
	github.com/cespare/xxhash/v2 v2.1.2 // indirect
	github.com/containerd/containerd v1.5.13 // indirect
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/dlclark/regexp2 v1.1.7-0.20171009020623-7632a260cbaf // indirect
	github.com/docker/distribution v2.8.0+incompatible // indirect
	github.com/docker/docker v1.4.2-0.20190924003213-a8608b5b67c7 // indirect
	github.com/docker/go-connections v0.4.0 // indirect
	github.com/docker/go-units v0.4.0 // indirect
	github.com/dop251/goja v0.0.0-20200831102558-9af81ddcf0e1 // indirect
	github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6 // indirect
	github.com/dustin/go-humanize v1.0.0 // indirect
	github.com/eapache/go-resiliency v1.3.0 // indirect
	github.com/eapache/go-xerial-snappy v0.0.0-20230111030713-bf00bc1b83b6 // indirect
	github.com/eapache/queue v1.1.0 // indirect
	github.com/elastic/ecs v1.12.0 // indirect
	github.com/elastic/elastic-agent-client/v7 v7.0.0-20210727140539-f0905d9377f6 // indirect
	github.com/elastic/go-concert v0.2.0 // indirect
	github.com/elastic/go-lumber v0.1.0 // indirect
	github.com/elastic/go-seccomp-bpf v1.2.0 // indirect
	github.com/elastic/go-structform v0.0.9 // indirect
	github.com/elastic/go-sysinfo v1.8.1 // indirect
	github.com/elastic/go-txfile v0.0.7 // indirect
	github.com/elastic/go-ucfg v0.8.6 // indirect
	github.com/elastic/go-windows v1.0.1 // indirect
	github.com/elastic/gosigar v0.14.2 // indirect
	github.com/fatih/color v1.13.0 // indirect
	github.com/go-logr/logr v0.4.0 // indirect
	github.com/go-ole/go-ole v1.2.6 // indirect
	github.com/go-sourcemap/sourcemap v2.1.2+incompatible // indirect
	github.com/gofrs/flock v0.7.2-0.20190320160742-5135e617513b // indirect
	github.com/gofrs/uuid v4.2.0+incompatible // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang/protobuf v1.5.2 // indirect
	github.com/golang/snappy v0.0.4 // indirect
	github.com/gomodule/redigo v1.8.3 // indirect
	github.com/google/go-cmp v0.5.6 // indirect
	github.com/google/gofuzz v1.1.0 // indirect
	github.com/googleapis/gnostic v0.4.1 // indirect
	github.com/h2non/filetype v1.1.1 // indirect
	github.com/hashicorp/errwrap v1.1.0 // indirect
	github.com/hashicorp/go-multierror v1.1.1 // indirect
	github.com/hashicorp/go-uuid v1.0.3 // indirect
	github.com/hashicorp/golang-lru v0.5.4 // indirect
	github.com/imdario/mergo v0.3.12 // indirect
	github.com/inconshreveable/mousetrap v1.0.0 // indirect
	github.com/jcmturner/aescts/v2 v2.0.0 // indirect
	github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
	github.com/jcmturner/gofork v1.7.6 // indirect
	github.com/jcmturner/gokrb5/v8 v8.4.3 // indirect
	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
	github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 // indirect
	github.com/jonboulle/clockwork v0.2.2 // indirect
	github.com/josephspurrier/goversioninfo v0.0.0-20190209210621-63e6d1acd3dd // indirect
	github.com/json-iterator/go v1.1.12 // indirect
	github.com/klauspost/compress v1.15.14 // indirect
	github.com/mattn/go-colorable v0.1.12 // indirect
	github.com/mattn/go-isatty v0.0.14 // indirect
	github.com/miekg/dns v1.1.41 // indirect
	github.com/mitchellh/hashstructure v0.0.0-20170116052023-ab25296c0f51 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/opencontainers/go-digest v1.0.0 // indirect
	github.com/opencontainers/image-spec v1.0.2 // indirect
	github.com/pierrec/lz4/v4 v4.1.17 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/prometheus/procfs v0.7.3 // indirect
	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
	github.com/rogpeppe/go-internal v1.9.0 // indirect
	github.com/santhosh-tekuri/jsonschema v1.2.4 // indirect
	github.com/shirou/gopsutil v3.20.12+incompatible // indirect
	github.com/sirupsen/logrus v1.8.1 // indirect
	github.com/spf13/cobra v1.3.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/urso/diag v0.0.0-20200210123136-21b3cc8eb797 // indirect
	github.com/urso/go-bin v0.0.0-20180220135811-781c575c9f0e // indirect
	github.com/urso/magetools v0.0.0-20190919040553-290c89e0c230 // indirect
	github.com/urso/sderr v0.0.0-20210525210834-52b04e8f5c71 // indirect
	github.com/xdg/scram v1.0.3 // indirect
	github.com/xdg/stringprep v1.0.3 // indirect
	go.elastic.co/apm v1.11.0 // indirect
	go.elastic.co/apm/module/apmelasticsearch v1.7.2 // indirect
	go.elastic.co/apm/module/apmhttp v1.7.2 // indirect
	go.elastic.co/ecszap v1.0.1 // indirect
	go.elastic.co/fastjson v1.1.0 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	go.uber.org/multierr v1.8.0 // indirect
	go.uber.org/zap v1.21.0 // indirect
	golang.org/x/crypto v0.5.0 // indirect
	golang.org/x/net v0.5.0 // indirect
	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
	golang.org/x/sys v0.4.0 // indirect
	golang.org/x/term v0.4.0 // indirect
	golang.org/x/text v0.6.0 // indirect
	golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect
	golang.org/x/tools v0.1.12 // indirect
	google.golang.org/appengine v1.6.7 // indirect
	google.golang.org/genproto v0.0.0-20220426171045-31bebdecfb46 // indirect
	google.golang.org/grpc v1.46.0 // indirect
	google.golang.org/protobuf v1.28.0 // indirect
	gopkg.in/inf.v0 v0.9.1 // indirect
	gopkg.in/jcmturner/aescts.v1 v1.0.1 // indirect
	gopkg.in/jcmturner/dnsutils.v1 v1.0.1 // indirect
	gopkg.in/jcmturner/goidentity.v3 v3.0.0 // indirect
	gopkg.in/jcmturner/gokrb5.v7 v7.5.0 // indirect
	gopkg.in/jcmturner/rpc.v1 v1.1.0 // indirect
	gopkg.in/yaml.v2 v2.4.0 // indirect
	howett.net/plist v1.0.0 // indirect
	k8s.io/api v0.21.1 // indirect
	k8s.io/apimachinery v0.21.1 // indirect
	k8s.io/client-go v0.21.1 // indirect
	k8s.io/klog/v2 v2.8.0 // indirect
	k8s.io/utils v0.0.0-20201110183641-67b214c5f920 // indirect
	sigs.k8s.io/structured-merge-diff/v4 v4.1.0 // indirect
	sigs.k8s.io/yaml v1.2.0 // indirect
)

@herry-go
Copy link

herry-go commented Mar 19, 2024 •
edited
Loading

I have this issue too. I receive:我也有这个问题。我收到：

# github.com/elastic/beats/v7/libbeat/processors/script/javascript
../../go/pkg/mod/github.com/elastic/beats/v7@v7.17.8/libbeat/processors/script/javascript/session.go:267:15: s.Runtime().RegisterSimpleMapType undefined (type *goja.Runtime has no field or method RegisterSimpleMapType)

My go.mod:

replace (
github.com/dop251/goja => github.com/andrewkroh/goja v0.0.0-20190128172624-dd2ac4456e20
github.com/dop251/goja_nodejs => github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6
)

Adding the above mod file can solve this problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bjanders @kvch @Pungyeon @richard-mauri @herry-go