Make parsing log filenames future-proof #4518

europaul · 2025-01-10T18:21:14Z

Main commit is a8725c7. Instead of relying on a certain structure of log filenames, we should just use regexp to look for timestamps or UUIDs - this way the filename structure can change in the future, but it will still be backwards compatible with our filename parsing logic.

This PR contains the changes for pillar. In a later PR those changes will also be applied to newlog and edge-view.

europaul · 2025-01-10T18:35:57Z

the yetus warning can probably be ignored since we do the same in other files that import gomega

yetus: pkg/pillar/types/newlogtypes_test.go#L10
revive: should not use dot imports https://revive.run/r#dot-imports

rene · 2025-01-13T10:00:16Z

the yetus warning can probably be ignored since we do the same in other files that import gomega

yetus: pkg/pillar/types/newlogtypes_test.go#L10 revive: should not use dot imports https://revive.run/r#dot-imports

Actually not in all places, like here gomega it's not dot imported: https://github.com/lf-edge/eve/blob/master/pkg/pillar/types/patchenvelopestypes_test.go#L10

I would avoid as much as we can to introduce new Yetus errors and in this case it's really not hard to fix it...

rene · 2025-01-13T10:10:24Z

@europaul , LGTM and I liked the flexibility added by using regex. However, I would fix the Yetus error in order to avoid YAYEE (Yet Another Yetus Error on EVE 😃 )

OhmSpectator

I think we must address the same functions in the EdgeView package; otherwise, the next time we upgrade the Pillar package used by EdgeView, it will be broken.

pkg/pillar/types/newlogtypes.go

OhmSpectator · 2025-01-13T13:59:07Z

pkg/pillar/types/newlogtypes.go

@@ -85,21 +86,43 @@ type NewlogMetrics struct {
 	AppMetrics logfileMetrics // App metrics
 }

-// GetTimestampFromGzipName - get timestamp from gzip file name
-func GetTimestampFromGzipName(fName string) (time.Time, error) {


As far as I see, we still use this func in another package, edgeveiew:

eve/pkg/edgeview/src/log-search.go

Line 152 in 2edd016

timestamp, err := types.GetTimestampFromGzipName(f.Name())

and

eve/pkg/edgeview/src/system.go

Line 163 in 2edd016

timestamp, err := types.GetTimestampFromGzipName(l.Name())

So, we should be careful here...

europaul · 2025-01-13T19:31:09Z

I think we must address the same functions in the EdgeView package; otherwise, the next time we upgrade the Pillar package used by EdgeView, it will be broken.

The new functions are (mostly) backward compatible with the old ones. The only problem that I see is with logs that get created before the devices sets the correct time (probably through NTP), so the initial log files look like dev.log.123.gz instead of dev.log.1731491904032.gz. The new function will not pick up a timestamp from them since it doesn't fit into \b\d{13,16}\b regexp.

europaul · 2025-01-13T19:32:08Z

As far as I see, we still use this func in another package, edgeveiew:

@OhmSpectator yes, I will change them once this is merged and I can update the pillar dependency.

Instead of parsing filenames according to a certain schema, we should use regexp to look for timestamps or UUIDs. This way the file names can change in the future without affecting backward compatibility with the existing code for parsing filenames. Signed-off-by: Paul Gaiduk <paulg@zededa.com>

Since not all logs in persist are to be sent, we rename LogRemainToSendMBytes to MaxGzipLogMBytesInPersist. This parameter represents the maxinum size of all gzipped logs in persist. Signed-off-by: Paul Gaiduk <paulg@zededa.com>

rene · 2025-01-14T08:55:09Z

I think we must address the same functions in the EdgeView package; otherwise, the next time we upgrade the Pillar package used by EdgeView, it will be broken.

The new functions are (mostly) backward compatible with the old ones. The only problem that I see is with logs that get created before the devices sets the correct time (probably through NTP), so the initial log files look like dev.log.123.gz instead of dev.log.1731491904032.gz. The new function will not pick up a timestamp from them since it doesn't fit into \b\d{13,16}\b regexp.

@europaul , can we affirm that the timestamp will always precede the file extension, like this <any file name>.<TIMESTAMP>.<extension (.gz, etc)> ?

If so, the following regex could be used \b(\d{3,16}).\w{2,3}, then the first occurrence will be the timestamp and it will cover the dev.log.123.gz case. If you want to be more picky with the timestamps, another options is to use this regex: \b(\d{13,16}|\d{3}).\w{2,3}, which will cover only valid timestamps and the case for 3 digits....

OhmSpectator · 2025-01-14T12:25:34Z

As far as I see, we still use this func in another package, edgeveiew:

@OhmSpectator yes, I will change them once this is merged and I can update the pillar dependency.

So, we are expecting a new PR very after this one is merged... Have you tested the changes in edge view somehow?

OhmSpectator

Run the tests

europaul · 2025-01-15T05:29:37Z

If so, the following regex could be used \b(\d{3,16}).\w{2,3}, then the first occurrence will be the timestamp and it will cover the dev.log.123.gz case. If you want to be more picky with the timestamps, another options is to use this regex: \b(\d{13,16}|\d{3}).\w{2,3}, which will cover only valid timestamps and the case for 3 digits....

it's actually not just 3 digits - the timestamp can be anything between 0 and now microseconds.

europaul · 2025-01-15T05:31:02Z

As far as I see, we still use this func in another package, edgeveiew:

@OhmSpectator yes, I will change them once this is merged and I can update the pillar dependency.

So, we are expecting a new PR very after this one is merged... Have you tested the changes in edge view somehow?

ok, let's postpone this merge until we can execute it really in sync with the PR for edge-view and newlog. I'll convert it to draft for now

eriknordmark · 2025-01-15T18:19:25Z

pkg/pillar/cmd/loguploader/loguploader.go

-		appUUID = fStr[0]
+	appUUID, err := types.GetUUIDFromFileName(fName)
+	if err != nil {
+		log.Errorf("buildAppUUIDMap: cannot parse app file name %s: %v", fName, err)


Could this generate/log an error when it is run on a device log filename?

eriknordmark · 2025-01-15T18:21:34Z

pkg/pillar/types/newlogtypes.go

+// GetTimestampFromFileName extracts a microsecond timestamp from a filename
+func GetTimestampFromFileName(filename string) (time.Time, error) {
+	// Regular expression to match a microsecond timestamp (sequence of 13-16 digits)
+	timestampRegex := regexp.MustCompile(`\b\d{13,16}\b`)


Isn't it better to declare this as a variable in the file so that it is compiled at init time and not for each Get call?

eriknordmark · 2025-01-15T18:23:00Z

pkg/pillar/types/newlogtypes.go

-		return time.Time{}, fmt.Errorf("getTimestampFromGzipName: invalid log file name %s", fName)
+// GetTimestampFromFileName extracts a microsecond timestamp from a filename
+func GetTimestampFromFileName(filename string) (time.Time, error) {
+	// Regular expression to match a microsecond timestamp (sequence of 13-16 digits)


Comment below says millisecond and here you said microsecond. Which one is it?

eriknordmark · 2025-01-15T18:23:32Z

pkg/pillar/types/newlogtypes.go

+// GetUUIDFromFileName extracts a UUID from a filename with dot-delimited parts
+func GetUUIDFromFileName(filename string) (string, error) {
+	// UUID regex pattern (supports v4 UUIDs like "123e4567-e89b-12d3-a456-426614174000")
+	uuidRegex := regexp.MustCompile(`[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}`)


Define as a local variable in the file?

eriknordmark · 2025-01-15T18:24:05Z

pkg/pillar/types/newlogtypes.go

+
+	// Check each part for a UUID match
+	for _, part := range parts {
+		if uuidRegex.MatchString(part) {


Or call the UUID parser function we use elsewhere?

europaul requested review from OhmSpectator, deitch, rouming and eriknordmark as code owners January 10, 2025 18:21

europaul force-pushed the newlog-filenames branch 2 times, most recently from 02876f6 to 58d6ee9 Compare January 10, 2025 18:31

OhmSpectator requested changes Jan 13, 2025

View reviewed changes

europaul added 2 commits January 13, 2025 20:33

global params: rename LogRemainToSendMBytes

b703eb2

Since not all logs in persist are to be sent, we rename LogRemainToSendMBytes to MaxGzipLogMBytesInPersist. This parameter represents the maxinum size of all gzipped logs in persist. Signed-off-by: Paul Gaiduk <paulg@zededa.com>

europaul force-pushed the newlog-filenames branch from 58d6ee9 to b703eb2 Compare January 13, 2025 19:33

github-actions bot requested a review from OhmSpectator January 13, 2025 19:34

OhmSpectator approved these changes Jan 14, 2025

View reviewed changes

europaul marked this pull request as draft January 15, 2025 05:31

eriknordmark requested a review from naiming-zededa January 15, 2025 18:18

eriknordmark reviewed Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make parsing log filenames future-proof #4518

Make parsing log filenames future-proof #4518

europaul commented Jan 10, 2025

europaul commented Jan 10, 2025

rene commented Jan 13, 2025

rene commented Jan 13, 2025

OhmSpectator left a comment •

edited

Loading

OhmSpectator Jan 13, 2025

europaul commented Jan 13, 2025

europaul commented Jan 13, 2025

rene commented Jan 14, 2025

OhmSpectator commented Jan 14, 2025

OhmSpectator left a comment

europaul commented Jan 15, 2025

europaul commented Jan 15, 2025

eriknordmark Jan 15, 2025

eriknordmark Jan 15, 2025

eriknordmark Jan 15, 2025

eriknordmark Jan 15, 2025

eriknordmark Jan 15, 2025

Make parsing log filenames future-proof #4518

Are you sure you want to change the base?

Make parsing log filenames future-proof #4518

Conversation

europaul commented Jan 10, 2025

europaul commented Jan 10, 2025

rene commented Jan 13, 2025

rene commented Jan 13, 2025

OhmSpectator left a comment • edited Loading

Choose a reason for hiding this comment

OhmSpectator Jan 13, 2025

Choose a reason for hiding this comment

europaul commented Jan 13, 2025

europaul commented Jan 13, 2025

rene commented Jan 14, 2025

OhmSpectator commented Jan 14, 2025

OhmSpectator left a comment

Choose a reason for hiding this comment

europaul commented Jan 15, 2025

europaul commented Jan 15, 2025

eriknordmark Jan 15, 2025

Choose a reason for hiding this comment

eriknordmark Jan 15, 2025

Choose a reason for hiding this comment

eriknordmark Jan 15, 2025

Choose a reason for hiding this comment

eriknordmark Jan 15, 2025

Choose a reason for hiding this comment

eriknordmark Jan 15, 2025

Choose a reason for hiding this comment

OhmSpectator left a comment •

edited

Loading