feat: supporting meta field in WakuMessage

[gabrielmer]

Description

The "meta" field is late addition to the Waku Message specification. It is currently supported in go-waku and not in nwaku. We're implementing its support in this PR.

Changes

• adding meta field to Relay, Filter and Store Waku Messages
• testing for correct returned fields in all 3 types of messages
• fixed validator test that timed out early

Issue

closes #2214

Note

We also wanted to avoid responding with an error when an extra field is sent in a POST request.
There seems to be an issue in the nim-json-serialization dependency that throws an exception when this happens. Will report and track the issue to make sure it gets fixed

[github-actions]

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2384

Built from ebbf3f5

[Ivansete-status]

It looks great so far. Good work!
I just added a minor comment

[SionoiS]

LGTM!

What happens with the messages in DB do they have meta already but nwaku was discarding the field?

[gabrielmer]

LGTM!

What happens with the messages in DB do they have meta already but nwaku was discarding the field?

Great question! So it seems that meta was already defined in the WakuMessage type

nwaku/waku/waku_core/message/message.nim

Lines 20 to 38 in 849d76d

	type WakuMessage* = object
	# Data payload transmitted.
	payload*: seq[byte]
	# String identifier that can be used for content-based filtering.
	contentTopic*: ContentTopic
	# Application specific metadata.
	meta*: seq[byte]
	# Number to discriminate different types of payload encryption.
	# Compatibility with Whisper/WakuV1.
	version*: uint32
	# Sender generated timestamp.
	timestamp*: Timestamp
	# The ephemeral attribute indicates signifies the transient nature of the
	# message (if the message should be stored).
	ephemeral*: bool
	# Part of RFC 17: https://rfc.vac.dev/spec/17/
	# The proof attribute indicates that the message is not spam. This
	# attribute will be used in the rln-relay protocol.
	proof*: seq[byte]

And this is the one being used in the db

nwaku/waku/waku_store/common.nim

Lines 61 to 63 in 849d76d

	HistoryResponse* = object
	messages*: seq[WakuMessage]
	cursor*: Option[HistoryCursor]

But apparently the field wasn't getting filled when sending neither relay, filter nor store messages

[SionoiS]

But apparently the field wasn't getting filled when sending neither relay, filter nor store messages

Good to know! but...

Let me rephrase my question. Are messages stored in Postgres as blobs? If not, are the messages previously created by nwaku show no value in the DB for that field? Lastly, is this a problem or not?

[gabrielmer]

Good to know! but...

Let me rephrase my question. Are messages stored in Postgres as blobs? If not, are the messages previously created by nwaku show no value in the DB for that field? Lastly, is this a problem or not?

Ooh, I think there's actually an issue with that. I think it's not getting saved in the DB at all 😶
It doesn't seem to be in the schema.

nwaku/waku/waku_archive/driver/postgres_driver/postgres_driver.nim

Lines 30 to 41 in 849d76d

	proc createTableQuery(): string =
	"CREATE TABLE IF NOT EXISTS messages (" &
	" pubsubTopic VARCHAR NOT NULL," &
	" contentTopic VARCHAR NOT NULL," &
	" payload VARCHAR," &
	" version INTEGER NOT NULL," &
	" timestamp BIGINT NOT NULL," &
	" id VARCHAR NOT NULL," &
	" messageHash VARCHAR NOT NULL," &
	" storedAt BIGINT NOT NULL," &
	" CONSTRAINT messageIndex PRIMARY KEY (messageHash)" &
	");"

What I don't understand is it working on the Store test then. Don't we use any DB in those tests? @Ivansete-status

[Ivansete-status]

What I don't understand is it working on the Store test then. Don't we use any DB in those tests? @Ivansete-status

Yup! We use a database in Postgres tests.

[Ivansete-status]

LGTM! Thanks!

I think we can merge this PR, stating clearly that is missing the proper storage of this field, and submit another PR in the future where we start storing the meta field in the database.

[gabrielmer]

Yup! We use a database in Postgres tests.

So in https://github.com/waku-org/nwaku/blob/0dac9f9dd49e2047f3988f4cf0332712028efd0e/tests/wakunode_rest/test_rest_store.nim, are store messages saved in memory?

If so, is it done different in a way only for testing? There's something I don't quite understand - I thought store messages were always retrieved from a db

[Ivansete-status]

So in https://github.com/waku-org/nwaku/blob/0dac9f9dd49e2047f3988f4cf0332712028efd0e/tests/wakunode_rest/test_rest_store.nim, are store messages saved in memory?

Yes, this particular test module stores messages in memory.

---

If so, is it done different in a way only for testing? There's something I don't quite understand - I thought store messages were always retrieved from a db

That represents a test gap that should be filled with a test_rest_store_postgres.nim module.

These are the test modules that interact with a local Postgres database:
https://github.com/waku-org/nwaku/blob/master/tests/waku_archive/test_driver_postgres_query.nim
https://github.com/waku-org/nwaku/blob/master/tests/waku_archive/test_driver_postgres.nim
... and notice that these are compiled in the following location:

nwaku/tests/all_tests_waku.nim

Lines 26 to 28 in 0dac9f9

	defined(postgres):
	import
	./waku_archive/test_driver_postgres_query, ./waku_archive/test_driver_postgres

[gabrielmer]

I think we can merge this PR, stating clearly that is missing the proper storage of this field, and submit another PR in the future where we start storing the meta field in the database.

Super, sounds good! Will merge it and implement in a different PR support for db storage of the meta field :)