UDP inputs now uses human friendly size to define MaxMessageSize #6886

ph · 2018-04-17T16:10:35Z

Uses go-humanize to parse values for MaxMessageSize and fallback to
raw bytes if no suffix is found. Also create a new type for future
configuration named cfgtype.ByteSize that will take care of correctly
unpacking values.

houndci-bot · 2018-04-17T16:10:45Z

libbeat/common/size.go

+
+import "github.com/dustin/go-humanize"
+
+type Size int64


exported type Size should have comment or be unexported

kvch · 2018-04-17T18:55:46Z

filebeat/input/udp/config.go

@@ -3,6 +3,8 @@ package udp
 import (
 	"time"

+	humanize "github.com/dustin/go-humanize"


I might be wrong, but AFAIK the prefix "go-" is removed automatically. So you don't need to alias this import.

I really need to fix my editor to stop adding theses.

@kvch Your vim doesn't do that?

ph · 2018-04-17T19:39:10Z

@kvch updated

ph · 2018-04-18T02:02:37Z

jenkins test this please

ph · 2018-04-18T02:04:25Z

@kvch don't merge it yet, I have second thoughts about the name, I think It will go with byteSize instead of size it will a bit more explicit.

ruflin · 2018-04-18T07:32:07Z

filebeat/inputsource/tcp/client.go

@@ -18,7 +19,7 @@ type client struct {
 	done           chan struct{}
 	metadata       Metadata
 	splitFunc      bufio.SplitFunc
-	maxReadMessage size
+	maxReadMessage common.Size


Could we rename the variable here to be in line with the config naming? This will make it easier to read the code base.

Will rename it to maxMessageSize

ruflin · 2018-04-18T07:34:11Z

libbeat/common/size.go

+type Size int64
+
+// Unpack convert a size defined from a human readable format into bytes.
+func (s *Size) Unpack(value string) error {


I like this as a temporarely fix to make migration easier. But I would remove it again in 7.0 to only have 1 way to configure the values.

This would mean to when someone uses it without a unit, we should log a deprecation warning.

I am OK to add a deprecation warning.

@ruflin When I tried to use cfgwarn in that file, I hit aimport cycle not allowed, to fix that I have either to move theses new common type out of common OR maybe cfgwarn out of common and have a backward compatible alias? ...

ph · 2018-04-18T19:38:10Z

@kvch @ruflin renamed updated and added the warning. 👍

ruflin

I would keep the current default of 10KB as a mutline event can reach that limit pretty easily I'm thinking. If we change it, we should add it to the changelog.

Code LGTM.

ruflin · 2018-04-19T09:31:16Z

filebeat/input/udp/config.go

@@ -12,7 +14,7 @@ var defaultConfig = config{
 		Type: "udp",
 	},
 	Config: udp.Config{
-		MaxMessageSize: 10240,
+		MaxMessageSize: 1 * humanize.KiByte,


Sorry for the late comment, but I just realised we are changing the default here from 10KB to 1KB?

ph · 2018-04-19T11:10:31Z

facepalm

On Thu, Apr 19, 2018 at 5:33 AM Nicolas Ruflin ***@***.***> wrote: ***@***.**** commented on this pull request. I would keep the current default of 10KB as a mutline event can reach that limit pretty easily I'm thinking. If we change it, we should add it to the changelog. Code LGTM. ------------------------------ In filebeat/input/udp/config.go <#6886 (comment)>: > @@ -12,7 +14,7 @@ var defaultConfig = config{ Type: "udp", }, Config: udp.Config{ - MaxMessageSize: 10240, + MaxMessageSize: 1 * humanize.KiByte, Sorry for the late comment, but I just realised we are changing the default here from 10KB to 1KB? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6886 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAACgEmxtYIoqVDiS9fEqpn-7yK5dIFOks5tqFnOgaJpZM4TYneA> .

-- ph

ph · 2018-04-19T13:16:50Z

@ruflin all update good catch, waiting for green.

ruflin · 2018-04-19T13:24:45Z

filebeat/docs/inputs/input-udp.asciidoc

@@ -29,13 +29,13 @@ The `udp` input supports the following configuration options plus the
 [id="{beatname_lc}-input-{type}-max-message-size"]
 ==== `max_message_size`

-The maximum size of the message received over UDP. The default is `10240`.
+The maximum size of the message received over UDP. The default is `1KiB`.


@ph I think you forgot the docs. Also see above.

Uses go-humanize to parse values for `MaxMessageSize` and fallback to raw bytes if no suffix is found. Also create a new type for future configuration named `cfgtype.ByteSize` that will take care of correctly unpacking values.

ph · 2018-04-19T13:32:24Z

@ruflin I've fixed the docs issue and manually rebased to have a cleaner history.

ruflin

WFG

ph added the Filebeat Filebeat label Apr 17, 2018

houndci-bot reviewed Apr 17, 2018

View reviewed changes

ph force-pushed the fix/humanize-udp branch from cf883ea to 658c7e4 Compare April 17, 2018 16:13

ph mentioned this pull request Apr 17, 2018

Syslog inputs parses RFC3164 events via TCP or UDP #6842

Merged

kvch reviewed Apr 17, 2018

View reviewed changes

kvch approved these changes Apr 17, 2018

View reviewed changes

ph force-pushed the fix/humanize-udp branch from 658c7e4 to 3942125 Compare April 17, 2018 19:38

ph added in progress Pull request is currently in progress. review and removed in progress Pull request is currently in progress. review labels Apr 18, 2018

ruflin reviewed Apr 18, 2018

View reviewed changes

ph force-pushed the fix/humanize-udp branch 2 times, most recently from 790363b to 77d5271 Compare April 18, 2018 19:36

ph added review and removed in progress Pull request is currently in progress. labels Apr 18, 2018

ruflin reviewed Apr 19, 2018

View reviewed changes

ph added 2 commits April 19, 2018 09:31

MaxMessageSize instead of MaxReadMessage

cbdc861

ph force-pushed the fix/humanize-udp branch from 28805e1 to cbdc861 Compare April 19, 2018 13:32

ruflin approved these changes Apr 19, 2018

View reviewed changes

ruflin merged commit 227dc95 into elastic:master Apr 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UDP inputs now uses human friendly size to define MaxMessageSize #6886

UDP inputs now uses human friendly size to define MaxMessageSize #6886

ph commented Apr 17, 2018 •

edited

Loading

houndci-bot Apr 17, 2018

kvch Apr 17, 2018 •

edited

Loading

ph Apr 17, 2018

ph Apr 17, 2018

ph commented Apr 17, 2018

ph commented Apr 18, 2018

ph commented Apr 18, 2018

ruflin Apr 18, 2018

ph Apr 18, 2018

ruflin Apr 18, 2018

ph Apr 18, 2018

ph Apr 18, 2018

ph commented Apr 18, 2018

ruflin left a comment

ruflin Apr 19, 2018

ph commented Apr 19, 2018 via email

ph commented Apr 19, 2018

ruflin Apr 19, 2018

ph commented Apr 19, 2018

ruflin left a comment

UDP inputs now uses human friendly size to define MaxMessageSize #6886

UDP inputs now uses human friendly size to define MaxMessageSize #6886

Conversation

ph commented Apr 17, 2018 • edited Loading

Choose a reason for hiding this comment

kvch Apr 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Apr 17, 2018

ph commented Apr 18, 2018

ph commented Apr 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Apr 18, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Apr 19, 2018 via email

ph commented Apr 19, 2018

Choose a reason for hiding this comment

ph commented Apr 19, 2018

ruflin left a comment

Choose a reason for hiding this comment

ph commented Apr 17, 2018 •

edited

Loading

kvch Apr 17, 2018 •

edited

Loading