Running the import script fails

[thoma01]

Attempting to run the import script fails, using both mage ImportBeats and go run ./dev/import-beats/ *.go

Output:

2021/10/06 16:17:48 Populate icon registry
2021/10/06 16:17:48 Scan tutorial file: ..\kibana\src\plugins\home\server\tutorials\activemq_logs\index.ts
panic: runtime error: slice bounds out of range [:-1]

goroutine 1 [running]:
main.retrieveIconPathFromTutorials({0x76fc20, 0x9})
        C:/git/integrations/dev/import-beats/icons.go:102 +0x5f5
main.populateIconRepository({0x76e9e9, 0x6}, {0x76fc20, 0x9})
        C:/git/integrations/dev/import-beats/icons.go:53 +0x8b
main.newIconRepository({0x76e9e9, 0x6941a0}, {0x76fc20, 0xc000042000})
        C:/git/integrations/dev/import-beats/icons.go:43 +0x27
main.build({{0x76f52a, 0x8}, {0x776cd9, 0x15}, {0x76f292, 0x7}, {0x76f84a, 0x8}, {0x76fc20, 0x9}, ...})
        C:/git/integrations/dev/import-beats/main.go:112 +0x4e
main.main()
        C:/git/integrations/dev/import-beats/main.go:102 +0x4b8
exit status 2

How can this be fixed?

[mtojek]

I checked in the code and there might be a problem with windows separator:

integrations/dev/import-beats/icons.go

Line 102 in b5eaf5d

moduleName = moduleName[:strings.Index(moduleName, "/")]

Could you please check with \?

[thoma01]

Using \\ got me further into the script but it failed somewhere new.

panic: Duplicated key 'categories' in struct util.Package [recovered]
        panic: Duplicated key 'categories' in struct util.Package

goroutine 1 [running]:
gopkg.in/yaml%2ev2.handleErr(0xc001748c98)
        C:/Users/Administrator/go/pkg/mod/gopkg.in/yaml.v2@v2.4.0/yaml.go:249 +0x6d
panic({0x159bc60, 0xc0000360f0})
        C:/Program Files/Go/src/runtime/panic.go:1038 +0x215
gopkg.in/yaml%2ev2.(*encoder).structv(0xc0009b0b00, {0x0, 0x0}, {0x15f88a0, 0xc00004e000, 0x0})
        C:/Users/Administrator/go/pkg/mod/gopkg.in/yaml.v2@v2.4.0/encode.go:211 +0xe9
gopkg.in/yaml%2ev2.(*encoder).marshal(0xc0009b0b00, {0x0, 0x0}, {0x15f88a0, 0xc00004e000, 0x34})
        C:/Users/Administrator/go/pkg/mod/gopkg.in/yaml.v2@v2.4.0/encode.go:160 +0x9a5
gopkg.in/yaml%2ev2.(*encoder).marshalDoc(0xc0009b0b00, {0x0, 0x0}, {0x15f88a0, 0xc00004e000, 0xc001748cb8})
        C:/Users/Administrator/go/pkg/mod/gopkg.in/yaml.v2@v2.4.0/encode.go:93 +0x125
gopkg.in/yaml%2ev2.Marshal({0x15f88a0, 0xc00004e000})
        C:/Users/Administrator/go/pkg/mod/gopkg.in/yaml.v2@v2.4.0/yaml.go:203 +0x37e
main.(*packageRepository).save(0xc001749d98, {0x160f982, 0x8})
        C:/git/integrations/dev/import-beats/packages.go:269 +0x235
main.build({{0x160f52a, 0x8}, {0x1616cd9, 0x15}, {0x160f292, 0x7}, {0x160f84a, 0x8}, {0x160fc20, 0x9}, ...})
        C:/git/integrations/dev/import-beats/main.go:139 +0x2df
main.main()
        C:/git/integrations/dev/import-beats/main.go:102 +0x4b8
exit status 2

[mtojek]

Out of curiosity: which Beats module are trying to migrate?

[thoma01]

A custom filebeat module that I developed.

[mtojek]

Nice! Did you consider writing the integration from scratch? I would be more efficient than migrating. This feature is more oriented on shifting the legacy code.

[thoma01]

I had started the module about a year ago when Elastic Agent wasn't GA yet, so I was hoping to use the script now to fast track the transition to integrations.

[thoma01]

Note that the most recent error occurs for beat modules that already exist in GitHub source code.

[thoma01]

@mtojek Any news on the most recent error? Duplicate keys 'categories' ... occurs for more than just my module, so I think there is indeed a bug somewhere.

[mtojek]

We put in on the roadmap to triage it and potentially apply a bugfix. You're also welcome to prepare a fix on your own and open a PR :)

[thoma01]

I have created a PR on elastic/package-registry#747
The bug is in the package.go file of the package-registry repository.