Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Auditbeat] Add XXH64 File Integrity Checks (#6626) #7311

Merged
merged 3 commits into from
Jun 15, 2018
Merged

[Auditbeat] Add XXH64 File Integrity Checks (#6626) #7311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6105857
[Auditbeat] Add XXH64 File Integrity Checks (#6626)
original-brownbear Jun 13, 2018
f00b119
[Auditbeat] Add XXH64 File Integrity Checks (#6626)
original-brownbear Jun 14, 2018
ba414e0
[Auditbeat] Add XXH64 File Integrity Checks (#6626)
original-brownbear Jun 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ https://github.com/elastic/beats/compare/v6.2.3...master[Check the HEAD diff]
- Updated syscall tables for Linux 4.16. {pull}6978[6978]
- Added better error messages for when the auditd module fails due to the
Linux kernel not supporting auditing (CONFIG_AUDIT=n). {pull}7012[7012]
- Added XXH64 hash option for file integrity checks. {pull}7311[7311]

*Filebeat*

Expand Down
10 changes: 10 additions & 0 deletions NOTICE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1575,6 +1575,16 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

--------------------------------------------------------------------
Dependency: github.com/OneOfOne/xxhash
Version: v1.2.2
Revision: 6def279d2ce6c81a79dd1c1be580f03bb216fb8a
License type (autodetected): Apache-2.0
./vendor/github.com/OneOfOne/xxhash/LICENSE:
--------------------------------------------------------------------
Apache License 2.0


--------------------------------------------------------------------
Dependency: github.com/opencontainers/go-digest
Revision: eaa60544f31ccf3b0653b1a118b76d33418ff41b
Expand Down
2 changes: 1 addition & 1 deletion auditbeat/auditbeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ auditbeat.modules:

# Hash types to compute when the file changes. Supported types are
# blake2b_256, blake2b_384, blake2b_512, md5, sha1, sha224, sha256, sha384,
# sha512, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384 and sha3_512.
# sha512, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384, sha3_512, and xxh64.
# Default is sha1.
hash_types: [sha1]

Expand Down
9 changes: 9 additions & 0 deletions auditbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3192,6 +3192,15 @@ SHA512/256 hash of the file.

--

*`hash.xxh64`*::
+
--
type: keyword

XX64 hash of the file.

--

[[exported-fields-host-processor]]
== Host fields

Expand Down
2 changes: 1 addition & 1 deletion auditbeat/docs/modules/file_integrity.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ a suffix to the value. The supported units are `b` (default), `kib`, `kb`, `mib`
*`hash_types`*:: A list of hash types to compute when the file changes.
The supported hash types are `blake2b_256`, `blake2b_384`, `blake2b_512`, `md5`,
`sha1`, `sha224`, `sha256`, `sha384`, `sha512`, `sha512_224`, `sha512_256`,
`sha3_224`, `sha3_256`, `sha3_384`, and `sha3_512`. The default value is `sha1`.
`sha3_224`, `sha3_256`, `sha3_384`, `sha3_512`, and `xxh64`. The default value is `sha1`.

*`recursive`*:: By default, the watches set to the paths specified in
`paths` are not recursive. This means that only changes to the contents
Expand Down
2 changes: 1 addition & 1 deletion auditbeat/include/fields.go
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion auditbeat/module/file_integrity/_meta/config.yml.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@

# Hash types to compute when the file changes. Supported types are
# blake2b_256, blake2b_384, blake2b_512, md5, sha1, sha224, sha256, sha384,
# sha512, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384 and sha3_512.
# sha512, sha512_224, sha512_256, sha3_224, sha3_256, sha3_384, sha3_512, and xxh64.
# Default is sha1.
hash_types: [sha1]

Expand Down
2 changes: 1 addition & 1 deletion auditbeat/module/file_integrity/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ a suffix to the value. The supported units are `b` (default), `kib`, `kb`, `mib`
*`hash_types`*:: A list of hash types to compute when the file changes.
The supported hash types are `blake2b_256`, `blake2b_384`, `blake2b_512`, `md5`,
`sha1`, `sha224`, `sha256`, `sha384`, `sha512`, `sha512_224`, `sha512_256`,
`sha3_224`, `sha3_256`, `sha3_384`, and `sha3_512`. The default value is `sha1`.
`sha3_224`, `sha3_256`, `sha3_384`, `sha3_512`, and `xxh64`. The default value is `sha1`.

*`recursive`*:: By default, the watches set to the paths specified in
`paths` are not recursive. This means that only changes to the contents
Expand Down
4 changes: 4 additions & 0 deletions auditbeat/module/file_integrity/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,7 @@
- name: sha512_256
type: keyword
description: SHA512/256 hash of the file.

- name: xxh64
type: keyword
description: XX64 hash of the file.
2 changes: 2 additions & 0 deletions auditbeat/module/file_integrity/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ var validHashes = []HashType{
SHA1,
SHA224, SHA256, SHA384, SHA512, SHA512_224, SHA512_256,
SHA3_224, SHA3_256, SHA3_384, SHA3_512,
XXH64,
}

// Enum of hash types.
Expand All @@ -46,6 +47,7 @@ const (
SHA512 HashType = "sha512"
SHA512_224 HashType = "sha512_224"
SHA512_256 HashType = "sha512_256"
XXH64 HashType = "xxh64"
)

// Config contains the configuration parameters for the file integrity
Expand Down
3 changes: 3 additions & 0 deletions auditbeat/module/file_integrity/event.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"strconv"
"time"

"github.com/OneOfOne/xxhash"
"github.com/pkg/errors"
"golang.org/x/crypto/blake2b"
"golang.org/x/crypto/sha3"
Expand Down Expand Up @@ -377,6 +378,8 @@ func hashFile(name string, hashType ...HashType) (map[HashType]Digest, error) {
hashes = append(hashes, sha512.New512_224())
case SHA512_256:
hashes = append(hashes, sha512.New512_256())
case XXH64:
hashes = append(hashes, xxhash.New64())
default:
return nil, errors.Errorf("unknown hash type '%v'", name)
}
Expand Down
1 change: 1 addition & 0 deletions auditbeat/module/file_integrity/event_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ func TestHashFile(t *testing.T) {
SHA3_256: mustDecodeHex("3cb5385a2987ca45888d7877fbcf92b4854f7155ae19c96cecc7ea1300c6f5a4"),
SHA3_384: mustDecodeHex("f19539818b4f29fa0ee599db4113fd81b77cd1119682e6d799a052849d2e40ef0dad84bc947ba2dee742d9731f1b9e9b"),
SHA3_512: mustDecodeHex("f0a2c0f9090c1fd6dedf211192e36a6668d2b3c7f57a35419acb1c4fc7dfffc267bbcd90f5f38676caddcab652f6aacd1ed4e0ad0a8e1e4b98f890b62b6c7c5c"),
XXH64: mustDecodeHex("d3e8573b7abf279a"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Checks out for me.

$ pip install xxhash
macbook15:beats akroh$ python
Python 2.7.14 (default, Mar 10 2018, 00:01:04) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
>>> import xxhash
>>> xxhash.xxh64(b'hello world!\n').hexdigest()
'd3e8573b7abf279a'

}

f, err := ioutil.TempFile("", "input.txt")
Expand Down
5 changes: 5 additions & 0 deletions auditbeat/module/file_integrity/flatbuffers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ func fbWriteHash(b *flatbuffers.Builder, hashes map[HashType]Digest) flatbuffers
schema.HashAddSha512224(b, offset)
case SHA512_256:
schema.HashAddSha512256(b, offset)
case XXH64:
schema.HashAddXx64(b, offset)
}
}
return schema.HashEnd(b)
Expand Down Expand Up @@ -309,6 +311,9 @@ func fbDecodeHash(e *schema.Event) map[HashType]Digest {
case SHA512_256:
length = hash.Sha512256Length()
producer = hash.Sha512256
case XXH64:
length = hash.Xx64Length()
producer = hash.Xx64
default:
panic(errors.Errorf("unhandled hash type: %v", hashType))
}
Expand Down
2 changes: 2 additions & 0 deletions auditbeat/module/file_integrity/schema.fbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ table Hash {
blake2b_256: [byte];
blake2b_384: [byte];
blake2b_512: [byte];

xxh64: [byte];
}

table Event {
Expand Down
25 changes: 24 additions & 1 deletion auditbeat/module/file_integrity/schema/Hash.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,8 +281,25 @@ func (rcv *Hash) Blake2b512Length() int {
return 0
}

func (rcv *Hash) Xx64(j int) int8 {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Hash.Xx64 should have comment or be unexported

o := flatbuffers.UOffsetT(rcv._tab.Offset(34))
if o != 0 {
a := rcv._tab.Vector(o)
return rcv._tab.GetInt8(a + flatbuffers.UOffsetT(j*1))
}
return 0
}

func (rcv *Hash) Xx64Length() int {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported method Hash.Xx64Length should have comment or be unexported

o := flatbuffers.UOffsetT(rcv._tab.Offset(34))
if o != 0 {
return rcv._tab.VectorLen(o)
}
return 0
}

func HashStart(builder *flatbuffers.Builder) {
builder.StartObject(15)
builder.StartObject(16)
}
func HashAddMd5(builder *flatbuffers.Builder, md5 flatbuffers.UOffsetT) {
builder.PrependUOffsetTSlot(0, flatbuffers.UOffsetT(md5), 0)
Expand Down Expand Up @@ -374,6 +391,12 @@ func HashAddBlake2b512(builder *flatbuffers.Builder, blake2b512 flatbuffers.UOff
func HashStartBlake2b512Vector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
return builder.StartVector(1, numElems, 1)
}
func HashAddXx64(builder *flatbuffers.Builder, xx64 flatbuffers.UOffsetT) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function HashAddXx64 should have comment or be unexported

builder.PrependUOffsetTSlot(15, flatbuffers.UOffsetT(xx64), 0)
}
func HashStartXx64Vector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function HashStartXx64Vector should have comment or be unexported

return builder.StartVector(1, numElems, 1)
}
func HashEnd(builder *flatbuffers.Builder) flatbuffers.UOffsetT {
return builder.EndObject()
}
187 changes: 187 additions & 0 deletions vendor/github.com/OneOfOne/xxhash/LICENSE
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading