Add C dependencies to vendor (#16127)

## What does this PR do?

This PR adds a mechanism on top of `go mod vendor` in order to copy C source files required by Beats to the folder `vendor`. From now on it is not sufficient to run `go mod vendor` as it only copies go files to `vendor`. Instead `mage vendor` or `make vendor` must be used.

If a new dependency with non-Golang sources are added, files can be moved to our vendor folder by adding a new entry to the list `copyAll` in `dev-tools/mage/gomod.go`.

```golang
CopyModules{
	Name: "github.com/godror/godror",  // name of the dependency
	FilesToCopy: []string{ // list of the folders or files to copy
		"odpi",
	},
}
```

This PR also passes `-mod=vendor` to make sure the local vendor folder is used during build.

## Why is it important?

We have two dependencies which contain C source files:

- `github.com/godror/godror`
- `github.com/tsg/go-daemon`

NOTICE.txt

@@ -2406,6 +2406,229 @@ License type (autodetected): Apache-2.0
 Apache License 2.0
 
 
+--------------------------------------------------------------------
+Dependency: github.com/godror/godror/odpi
+License type (autodetected): UPL-1.0
+./vendor/github.com/godror/godror/odpi/LICENSE.md:
+--------------------------------------------------------------------
+Copyright (c) 2016, 2018 Oracle and/or its affiliates.  All rights reserved.
+
+This program is free software: you can modify it and/or redistribute it under
+the terms of:
+
+(i)  the Universal Permissive License v 1.0 or at your option, any
+     later version (<http://oss.oracle.com/licenses/upl>); and/or
+
+(ii) the Apache License v 2.0. (<http://www.apache.org/licenses/LICENSE-2.0>)
+
+
+The Universal Permissive License (UPL), Version 1.0
+===================================================
+
+Subject to the condition set forth below, permission is hereby granted to any
+person obtaining a copy of this software, associated documentation and/or data
+(collectively the "Software"), free of charge and under any and all copyright
+rights in the Software, and any and all patent rights owned or freely
+licensable by each licensor hereunder covering either (i) the unmodified
+Software as contributed to or provided by such licensor, or (ii) the Larger
+Works (as defined below), to deal in both
+
+(a) the Software, and
+
+(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+    one is included with the Software (each a "Larger Work" to which the
+    Software is contributed by such licensors),
+
+without restriction, including without limitation the rights to copy, create
+derivative works of, display, perform, and distribute the Software and make,
+use, sell, offer for sale, import, export, have made, and have sold the
+Software and the Larger Work(s), and to sublicense the foregoing rights on
+either these or other terms.
+
+This license is subject to the following condition:
+
+The above copyright notice and either this complete permission notice or at a
+minimum a reference to the UPL must be included in all copies or substantial
+portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+Apache License
+==============
+
+Version 2.0, January 2004
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1.  **Definitions**.
+
+    "License" shall mean the terms and conditions for use, reproduction, and
+    distribution as defined by Sections 1 through 9 of this document.
+
+    "Licensor" shall mean the copyright owner or entity authorized by the
+    copyright owner that is granting the License.
+
+    "Legal Entity" shall mean the union of the acting entity and all other
+    entities that control, are controlled by, or are under common control with
+    that entity.  For the purposes of this definition, "control" means (i) the
+    power, direct or indirect, to cause the direction or management of such
+    entity, whether by contract or otherwise, or (ii) ownership of fifty
+    percent (50%) or more of the outstanding shares, or (iii) beneficial
+    ownership of such entity.
+
+    "You" (or "Your") shall mean an individual or Legal Entity exercising
+    permissions granted by this License.
+
+    "Source" form shall mean the preferred form for making modifications,
+    including but not limited to software source code, documentation source,
+    and configuration files.
+
+    "Object" form shall mean any form resulting from mechanical transformation
+    or translation of a Source form, including but not limited to compiled
+    object code, generated documentation, and conversions to other media types.
+
+    "Work" shall mean the work of authorship, whether in Source or Object form,
+    made available under the License, as indicated by a copyright notice that
+    is included in or attached to the work (an example is provided in the
+    Appendix below).
+
+    "Derivative Works" shall mean any work, whether in Source or Object form,
+    that is based on (or derived from) the Work and for which the editorial
+    revisions, annotations, elaborations, or other modifications represent, as
+    a whole, an original work of authorship. For the purposes of this License,
+    Derivative Works shall not include works that remain separable from, or
+    merely link (or bind by name) to the interfaces of, the Work and Derivative
+    Works thereof.
+
+    "Contribution" shall mean any work of authorship, including the original
+    version of the Work and any modifications or additions to that Work or
+    Derivative Works thereof, that is intentionally submitted to Licensor for
+    inclusion in the Work by the copyright owner or by an individual or Legal
+    Entity authorized to submit on behalf of the copyright owner. For the
+    purposes of this definition, "submitted" means any form of electronic,
+    verbal, or written communication sent to the Licensor or its
+    representatives, including but not limited to communication on electronic
+    mailing lists, source code control systems, and issue tracking systems that
+    are managed by, or on behalf of, the Licensor for the purpose of discussing
+    and improving the Work, but excluding communication that is conspicuously
+    marked or otherwise designated in writing by the copyright owner as "Not a
+    Contribution."
+
+    "Contributor" shall mean Licensor and any individual or Legal Entity on
+    behalf of whom a Contribution has been received by Licensor and
+    subsequently incorporated within the Work.
+
+2.  **Grant of Copyright License.** Subject to the terms and conditions of this
+    License, each Contributor hereby grants to You a perpetual, worldwide,
+    non-exclusive, no-charge, royalty-free, irrevocable copyright license to
+    reproduce, prepare Derivative Works of, publicly display, publicly perform,
+    sublicense, and distribute the Work and such Derivative Works in Source or
+    Object form.
+
+3.  **Grant of Patent License.** Subject to the terms and conditions of this
+    License, each Contributor hereby grants to You a perpetual, worldwide,
+    non-exclusive, no-charge, royalty-free, irrevocable (except as stated in
+    this section) patent license to make, have made, use, offer to sell, sell,
+    import, and otherwise transfer the Work, where such license applies only to
+    those patent claims licensable by such Contributor that are necessarily
+    infringed by their Contribution(s) alone or by combination of their
+    Contribution(s) with the Work to which such Contribution(s) was submitted.
+    If You institute patent litigation against any entity (including a
+    cross-claim or counterclaim in a lawsuit) alleging that the Work or a
+    Contribution incorporated within the Work constitutes direct or
+    contributory patent infringement, then any patent licenses granted to You
+    under this License for that Work shall terminate as of the date such
+    litigation is filed.
+
+4.  **Redistribution.** You may reproduce and distribute copies of the Work or
+    Derivative Works thereof in any medium, with or without modifications, and
+    in Source or Object form, provided that You meet the following conditions:
+
+    1.  You must give any other recipients of the Work or Derivative Works a
+        copy of this License; and
+
+    2.  You must cause any modified files to carry prominent notices stating
+        that You changed the files; and
+
+    3.  You must retain, in the Source form of any Derivative Works that You
+        distribute, all copyright, patent, trademark, and attribution notices
+        from the Source form of the Work, excluding those notices that do not
+        pertain to any part of the Derivative Works; and
+
+    4.  If the Work includes a "NOTICE" text file as part of its distribution,
+        then any Derivative Works that You distribute must include a readable
+        copy of the attribution notices contained within such NOTICE file,
+        excluding those notices that do not pertain to any part of the
+        Derivative Works, in at least one of the following places: within a
+        NOTICE text file distributed as part of the Derivative Works; within
+        the Source form or documentation, if provided along with the Derivative
+        Works; or, within a display generated by the Derivative Works, if and
+        wherever such third-party notices normally appear. The contents of the
+        NOTICE file are for informational purposes only and do not modify the
+        License. You may add Your own attribution notices within Derivative
+        Works that You distribute, alongside or as an addendum to the NOTICE
+        text from the Work, provided that such additional attribution notices
+        cannot be construed as modifying the License.
+
+        You may add Your own copyright statement to Your modifications and may
+        provide additional or different license terms and conditions for use,
+        reproduction, or distribution of Your modifications, or for any such
+        Derivative Works as a whole, provided Your use, reproduction, and
+        distribution of the Work otherwise complies with the conditions stated
+        in this License.
+
+5.  **Submission of Contributions.** Unless You explicitly state otherwise, any
+    Contribution intentionally submitted for inclusion in the Work by You to
+    the Licensor shall be under the terms and conditions of this License,
+    without any additional terms or conditions. Notwithstanding the above,
+    nothing herein shall supersede or modify the terms of any separate license
+    agreement you may have executed with Licensor regarding such Contributions.
+
+6.  **Trademarks.** This License does not grant permission to use the trade
+    names, trademarks, service marks, or product names of the Licensor, except
+    as required for reasonable and customary use in describing the origin of
+    the Work and reproducing the content of the NOTICE file.
+
+7.  **Disclaimer of Warranty.** Unless required by applicable law or agreed to
+    in writing, Licensor provides the Work (and each Contributor provides its
+    Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied, including, without limitation, any
+    warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or
+    FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for
+    determining the appropriateness of using or redistributing the Work and
+    assume any risks associated with Your exercise of permissions under this
+    License.
+
+8.  **Limitation of Liability.** In no event and under no legal theory, whether
+    in tort (including negligence), contract, or otherwise, unless required by
+    applicable law (such as deliberate and grossly negligent acts) or agreed to
+    in writing, shall any Contributor be liable to You for damages, including
+    any direct, indirect, special, incidental, or consequential damages of any
+    character arising as a result of this License or out of the use or
+    inability to use the Work (including but not limited to damages for loss of
+    goodwill, work stoppage, computer failure or malfunction, or any and all
+    other commercial damages or losses), even if such Contributor has been
+    advised of the possibility of such damages.
+
+9.  **Accepting Warranty or Additional Liability.** While redistributing the
+    Work or Derivative Works thereof, You may choose to offer, and charge a fee
+    for, acceptance of support, warranty, indemnity, or other liability
+    obligations and/or rights consistent with this License. However, in
+    accepting such obligations, You may act only on Your own behalf and on Your
+    sole responsibility, not on behalf of any other Contributor, and only if
+    You agree to indemnify, defend, and hold each Contributor harmless for any
+    liability incurred by, or claims asserted against, such Contributor by
+    reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
 --------------------------------------------------------------------
 Dependency: github.com/gofrs/flock
 Version: v0.7.2
@@ -5339,7 +5562,7 @@ SOFTWARE.
 
 --------------------------------------------------------------------
 Dependency: github.com/tsg/go-daemon
-Revision: 4b60efc26d5f
+Revision: e704b93fd89b
 License type (autodetected): BSD-2-Clause
 ./vendor/github.com/tsg/go-daemon/LICENSE:
 --------------------------------------------------------------------

dev-tools/mage/build.go

@@ -135,6 +135,14 @@ func Build(params BuildArgs) error {
 	}
 	env["CGO_ENABLED"] = cgoEnabled
 
+	var goFlags string
+	goFlags, ok := env["GOFLAGS"]
+	if !ok {
+		env["GOFLAGS"] = "-mod=vendor"
+	} else {
+		env["GOFLAGS"] = strings.Join([]string{goFlags, "-mod=vendor"}, " ")
+	}
+
 	// Spec
 	args := []string{
 		"build",

dev-tools/mage/crossbuild.go

@@ -249,6 +249,7 @@ func (b GolangCrossBuilder) Build() error {
 	}
 	args = append(args,
 		"--rm",
+		"--env", "GOFLAGS=-mod=vendor",
 		"--env", "MAGEFILE_VERBOSE="+verbose,
 		"--env", "MAGEFILE_TIMEOUT="+EnvOr("MAGEFILE_TIMEOUT", ""),
 		"-v", repoInfo.RootDir+":"+mountPoint,

dev-tools/mage/dashboard.go

@@ -44,7 +44,7 @@ func ExportDashboard() error {
 		return err
 	}
 
-	dashboardCmd := sh.RunCmd("go", "run", filepath.Join(beatsDir, "dev-tools/cmd/dashboards/export_dashboards.go"))
+	dashboardCmd := sh.RunCmd("go", "run", "-mod", "vendor", filepath.Join(beatsDir, "dev-tools/cmd/dashboards/export_dashboards.go"))
 
 	// TODO: This is currently hardcoded for KB 7, we need to figure out what we do for KB 8 if applicable
 	file := CWD("module", module, "_meta/kibana/7/dashboard", id+".json")

dev-tools/mage/fields.go

@@ -101,7 +101,7 @@ func generateFieldsYAML(baseDir, output string, moduleDirs ...string) error {
 		return err
 	}
 
-	globalFieldsCmd := sh.RunCmd("go", "run",
+	globalFieldsCmd := sh.RunCmd("go", "run", "-mod", "vendor",
 		filepath.Join(beatsDir, globalFieldsCmdPath),
 		"-es_beats_path", beatsDir,
 		"-beat_path", baseDir,
@@ -125,7 +125,7 @@ func GenerateFieldsGo(fieldsYML, out string) error {
 		return err
 	}
 
-	assetCmd := sh.RunCmd("go", "run",
+	assetCmd := sh.RunCmd("go", "run", "-mod", "vendor",
 		filepath.Join(beatsDir, assetCmdPath),
 		"-pkg", "include",
 		"-in", fieldsYML,
@@ -152,7 +152,7 @@ func GenerateModuleFieldsGo(moduleDir string) error {
 		moduleDir = CWD(moduleDir)
 	}
 
-	moduleFieldsCmd := sh.RunCmd("go", "run",
+	moduleFieldsCmd := sh.RunCmd("go", "run", "-mod", "vendor",
 		filepath.Join(beatsDir, moduleFieldsCmdPath),
 		"-beat", BeatName,
 		"-license", toLibbeatLicenseName(BeatLicense),
@@ -179,7 +179,7 @@ func GenerateIncludeListGo(options IncludeListOptions) error {
 		return err
 	}
 
-	includeListCmd := sh.RunCmd("go", "run",
+	includeListCmd := sh.RunCmd("go", "run", "-mod", "vendor",
 		filepath.Join(beatsDir, moduleIncludeListCmdPath),
 		"-license", toLibbeatLicenseName(BeatLicense),
 		"-out", options.Outfile, "-buildTags", options.BuildTags,

dev-tools/mage/godaemon.go

@@ -43,7 +43,7 @@ func BuildGoDaemon() error {
 
 	// Test if binaries are up-to-date.
 	output := MustExpand("build/golang-crossbuild/god-{{.Platform.GOOS}}-{{.Platform.Arch}}")
-	input := MustExpand("{{ elastic_beats_dir }}/vendor/github.com/tsg/go-daemon/god.c")
+	input := MustExpand("{{ elastic_beats_dir }}/vendor/github.com/tsg/go-daemon/src/god.c")
 	if IsUpToDate(output, input) {
 		log.Println(">>> buildGoDaemon is up-to-date for", Platform.Name)
 		return nil