Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate fields files (required fields) #486

Merged
merged 4 commits into from
Jun 2, 2020
Merged

Validate fields files (required fields) #486

merged 4 commits into from
Jun 2, 2020

Conversation

mtojek
Copy link
Contributor

@mtojek mtojek commented Jun 1, 2020 •
edited
Loading

Prerequisites:

#485

Issue:

#465

Changes:

  • temporarily create base-fields.yml if missing
  • validate fields.yml files for required dataset keys

@mtojek mtojek requested a review from ruflin June 1, 2020 12:08
@mtojek mtojek self-assigned this Jun 1, 2020
@elasticmachine
Copy link

elasticmachine commented Jun 1, 2020 •
edited
Loading

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Started by upstream project "Beats/package-storage/master" build number 8]

  • Start Time: 2020-06-02T14:05:34.114+0000

  • Duration: 78 min 8 sec

Steps errors

Expand to view the steps failures

  • Name: Checks formatting / linting

    • Description: mage -debug build

    • Duration: 0 min 6 sec

    • Start Time: 2020-06-02T15:23:33.442+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-06-02T15:23:30.394Z]  ---> Using cache
[2020-06-02T15:23:30.394Z]  ---> 9c838393ec1c
[2020-06-02T15:23:30.394Z] Step 4/4 : ENV GO111MODULE="auto"
[2020-06-02T15:23:30.394Z]  ---> Using cache
[2020-06-02T15:23:30.394Z]  ---> 2944cdeec887
[2020-06-02T15:23:30.394Z] Successfully built 2944cdeec887
[2020-06-02T15:23:30.394Z] Successfully tagged go-agent:latest
[2020-06-02T15:23:30.709Z] + docker inspect -f . go-agent
[2020-06-02T15:23:30.709Z] .
[2020-06-02T15:23:30.836Z] beats-ci-immutable-ubuntu-1604-1591111234337303676 does not seem to be running inside a container
[2020-06-02T15:23:31.018Z] $ docker run -t -d -u 1156:1157 -w /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry -v /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry:/var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry:rw,z -v /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry@tmp:/var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** go-agent cat
[2020-06-02T15:23:32.127Z] $ docker top 1bdc2f1dda069cda1ad72cdc483c71179468d1898e899e20764ca5470eec05aa -eo pid,comm
[2020-06-02T15:23:32.825Z] + go version
[2020-06-02T15:23:32.825Z] go version go1.14.2 linux/amd64
[2020-06-02T15:23:33.431Z] + mage -version
[2020-06-02T15:23:33.431Z] Mage Build Tool <not set>
[2020-06-02T15:23:33.431Z] Build Date: <not set>
[2020-06-02T15:23:33.431Z] Commit: <not set>
[2020-06-02T15:23:33.431Z] built with: go1.14.2
[2020-06-02T15:23:34.038Z] + mage -debug build
[2020-06-02T15:23:34.038Z] DEBUG: 15:23:33.800351 getting all non-mage files in .
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123461 found non-mage files artifacts.go||categories.go||favicon.go||handler.go||index.go||main.go||mime_types.go||package_index.go||search.go
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123498 marked file as non-mage: "artifacts.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123503 marked file as non-mage: "categories.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123506 marked file as non-mage: "favicon.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123509 marked file as non-mage: "handler.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123512 marked file as non-mage: "index.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123515 marked file as non-mage: "main.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123518 marked file as non-mage: "mime_types.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123522 marked file as non-mage: "package_index.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123525 marked file as non-mage: "search.go"
[2020-06-02T15:23:34.300Z] DEBUG: 15:23:34.123536 getting all files plus mage files
[2020-06-02T15:23:34.872Z] DEBUG: 15:23:34.864181 time to scan for Magefiles: 1.063919598s
[2020-06-02T15:23:34.872Z] DEBUG: 15:23:34.864210 found magefiles: magefile.go
[2020-06-02T15:23:34.872Z] DEBUG: 15:23:34.871143 output exe is  /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry/.magefile/2e958ec6285eefe97d7232e3064828a5949dfb38
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.879320 go build cache exists, will ignore any compiled binary
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.879341 parsing files
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880164 found target AddLicenseHeaders
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880190 found target Build
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880195 found target Check
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880199 found target Clean
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880218 skipping function with invalid signature func FindFilesRecursive(match &{4566 0xc00028d0e0 0xc00028d110})(&{4609 <nil> string}, error)
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880222 found target Format
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880241 found target GoImports
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880253 found target PrepareTest
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880257 found target Test
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880261 found target Vendor
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880279 time parse Magefiles: 931.507µs
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880295 Creating mainfile at mage_output_file.go
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.880337 writing new file at mage_output_file.go
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.881421 compiling to /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry/.magefile/2e958ec6285eefe97d7232e3064828a5949dfb38
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.881426 compiling using gocmd: go
[2020-06-02T15:23:35.138Z] DEBUG: 15:23:34.945284 running go build -o /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry/.magefile/2e958ec6285eefe97d7232e3064828a5949dfb38 magefile.go mage_output_file.go
[2020-06-02T15:23:35.711Z] DEBUG: 15:23:35.635857 time to compile Magefile: 690.507332ms
[2020-06-02T15:23:35.711Z] DEBUG: 15:23:35.635929 running binary /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry/.magefile/2e958ec6285eefe97d7232e3064828a5949dfb38
[2020-06-02T15:23:35.711Z] DEBUG: 15:23:35.635957 running magefile with mage vars:
[2020-06-02T15:23:35.711Z] MAGEFILE_DEBUG=1
[2020-06-02T15:23:35.711Z] MAGEFILE_GOCMD=go
[2020-06-02T15:23:35.711Z] Cloning into 'build/package-storage'...
[2020-06-02T15:23:37.097Z] Already on 'master'
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 >> Copy package: build/package-storage/packages/aws/0.0.3/
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 exec: rsync -a build/package-storage/packages/aws/0.0.3/ public/package/aws/0.0.3
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 >> Copy package: build/package-storage/packages/base/0.1.0/
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 exec: rsync -a build/package-storage/packages/base/0.1.0/ public/package/base/0.1.0
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 >> Copy package: build/package-storage/packages/base/0.2.0/
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 exec: rsync -a build/package-storage/packages/base/0.2.0/ public/package/base/0.2.0
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 >> Copy package: build/package-storage/packages/cisco/0.1.0/
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:38 exec: rsync -a build/package-storage/packages/cisco/0.1.0/ public/package/cisco/0.1.0
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/cisco/0.1.1/
[2020-06-02T15:23:39.075Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/cisco/0.1.1/ public/package/cisco/0.1.1
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/endpoint/0.1.0/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/endpoint/0.1.0/ public/package/endpoint/0.1.0
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/endpoint/0.2.0/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/endpoint/0.2.0/ public/package/endpoint/0.2.0
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/kafka/0.1.0/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/kafka/0.1.0/ public/package/kafka/0.1.0
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/log/0.1.0/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/log/0.1.0/ public/package/log/0.1.0
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/mysql/0.1.0/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/mysql/0.1.0/ public/package/mysql/0.1.0
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/mysql/0.1.1/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/mysql/0.1.1/ public/package/mysql/0.1.1
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 >> Copy package: build/package-storage/packages/netflow/0.1.0/
[2020-06-02T15:23:39.337Z] 2020/06/02 15:23:39 exec: rsync -a build/package-storage/packages/netflow/0.1.0/ public/package/netflow/0.1.0
[2020-06-02T15:23:39.598Z] 2020/06/02 15:23:39 validating required fields failed (datasetPath: /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry/public/package/netflow/0.1.0/dataset/log): walking through fields files failed: unmarshaling file failed (path: /var/lib/jenkins/workspace/Beats_package-registry_PR-486/src/github.com/elastic/package-registry/public/package/netflow/0.1.0/dataset/log/fields/package-fields.yml): yaml: unmarshal errors:
[2020-06-02T15:23:39.598Z]   line 1: cannot unmarshal !!map into []main.MapStr
[2020-06-02T15:23:39.598Z] exit status 1
[2020-06-02T15:23:39.598Z] Error: running "go run ./dev/generator/ -sourceDir=build/package-storage/packages -publicDir=./public -tarGz=true" failed with exit code 1
[2020-06-02T15:23:39.610Z] $ docker stop --time=1 1bdc2f1dda069cda1ad72cdc483c71179468d1898e899e20764ca5470eec05aa
[2020-06-02T15:23:41.287Z] $ docker rm -f 1bdc2f1dda069cda1ad72cdc483c71179468d1898e899e20764ca5470eec05aa
[2020-06-02T15:23:41.432Z] Stage "Test" skipped due to earlier failure(s)
[2020-06-02T15:23:41.461Z] Stage "Publish Docker image" skipped due to earlier failure(s)
[2020-06-02T15:23:41.686Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_package-registry_PR-486
[2020-06-02T15:23:41.851Z] [INFO] getVaultSecret: Getting secrets
[2020-06-02T15:23:41.902Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-02T15:23:42.562Z] + chmod 755 generate-build-data.sh
[2020-06-02T15:23:42.562Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-486/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-486/runs/8 FAILURE 4688158
[2020-06-02T15:23:42.562Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-486/runs/8/steps/?limit=10000 -o steps-info.json
[2020-06-02T15:23:42.812Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-486/runs/8/tests/?status=FAILED -o tests-errors.json
[2020-06-02T15:23:43.063Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-486/runs/8/log/ -o pipeline-log.txt

@mtojek mtojek mentioned this pull request Jun 1, 2020
Closed
@mtojek
Copy link
Contributor Author

mtojek commented Jun 2, 2020

@alakahakai it would be great if you fix the issue reported in elastic/integrations#39 . It's blocking this PR a bit.

ruflin
ruflin approved these changes Jun 2, 2020
View reviewed changes
Copy link
Member

@ruflin ruflin left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM. And part of the code is temporary anyways ;-)

@mtojek mtojek merged commit 86927a0 into elastic:master Jun 2, 2020
@mtojek
Copy link
Contributor Author

mtojek commented Jun 2, 2020

I'm merging this. Netflow package will be fixed in a different PR. This one just detected problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruflin ruflin ruflin approved these changes

Assignees

@mtojek mtojek

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mtojek @elasticmachine @ruflin