pss: add trojan chunks #2148

mortelli · 2020-04-02T23:37:14Z

EXPERIMENTAL
(will not merge to master)

This PR adds implements trojan chunks.

Related issue: #2153

Implementation overview

new package

trojan

new types

Topic
- alias for 32 byte array
Message, contains:
- length (2 byte array)
- topic (of the Topic type)
- payload and padding (both of them byte slices)

main new functions

NewTopic helper function constructs a Topic by hashing an input string
newMessage creates a Message based on the given Topic plus a payload (byte slice)
- this function also determines the length and padding fields based on the given params
Wrap function takes targets (slice of byte slices) plus a Message type, and returns a chunk.Chunk. this function is in charge of:
- calling a func to check that targets are valid
- calling a func to create a new span
- calling a func to iterate a nonce so that when it is hashed along with the span and serialized Message, the resulting hash has a prefix which matches one of the targets
  - this iterator func will return a new Chunk with the matching hash set as its address, plus the aforementioned serialized fields set as its payload

Open questions

~~is it OK to call crypto.Keccack256Hash to hash a string to create a topic, or must we use crypto.Keccack256?~~
please use the exact same hash as used everywhere else. no new variant to be introduced here.
is it OK to have a Messageg var as a param for the Wrap func, rather than a byte slice (payload)? if a payload were received, this would mean that param is the serialization of the trojan message, which would then have to be de-serialized to find a nonce
perfectly. ok imho
~~should the span field be encoded in little-endian or big-endian?~~
~~let's try to be consistent with other parts of the codebase. so i guess little?~~
done
~~why do we need to call Reset on a storage.SwarmHash var?~~
you do not need to call it if it is freshly created hasher
since we don't have a trojan chunk struct, we are relying on the helper func serializeTrojanChunk to serialize the trojan chunk fields (rather than a binary marshaller). don't we need a deserializeTrojanChunk func too? this should be used on the receiving node to figure out whether the received chunk payload contains a serialized trojanMsg serializeTrojanChunk function now removed as it was unnecessary
should Message's UnmarshalBinary function also:
6.1 check if the received byte slice is longer than 4064 bytes? (and return an error if so)
6.2 ignore any bytes in the slice past byte #4064?
6.3 check that for decoded length l, it follows that 0 ≤ l ≤ 4030?
none of them needed, whenever deserialise, you get the data from a chunk which is itself validated for payload length, in fact all restrictions enforced here just enforce the restrictions on content addressed chunks

TODOs

~~make this work with storage.BMTHash rather than storage.SHA3Hash (currently, BMT hash does not change with each iteration loop; not sure why)~~
encrypt the trojan message when forming the trojan chunk (currently just serializes it) — maybe not for the current implementation
prevent an infinite loop in the trojan chunk iteration (how?). the given addresses could be too long to find a nonce in a reasonable time
~~test that if the random nonce overflows, the iteration still works~~
~~add failure tests~~

…ield, add marshalling and unmarshalling functions for trojanMessage, add TestTrojanMessageSerialization func

…ed on new design, add TODOs

…rojanData

…ing-trojan-chunks

santicomp2014

There a couple of minor things I left, I think this I really good implementation.
Awesome job.

santicomp2014 · 2020-04-08T19:53:28Z

pss/trojan/message.go

+// Wrap creates a new trojan chunk for the given targets and trojan message
+// a trojan chunk is a content-addressed chunk made up of span, a nonce, and a payload
+// TODO: discuss if instead of receiving a trojan message, we should receive a byte slice as payload
+func (m *Message) Wrap(targets [][]byte) (chunk.Chunk, error) {


targets [][]byte could be converted into a type ex. type Targets [][]byte
The same argument should be used for Send so exported type would be necessary.

i honestly don't feel strongly either way.

if it's beneficial to us, then yeah, let's add a Targets type. i will keep this in mind when we merge this with the API code

pss/trojan/message.go

pss/trojan/message_test.go

santicomp2014 · 2020-04-08T19:58:50Z

pss/trojan/message_test.go

+
+	payload := c.Data()
+	payloadSize := len(payload)
+	expectedSize := chunk.DefaultSize + 8 // payload + span


Span could be a constant so it's parametrized in all references

you're starting to convince me...

open to more opinions

santicomp2014 · 2020-04-08T19:59:07Z

pss/trojan/message_test.go

+		t.Fatalf("trojan chunk payload has an unexpected size of %d rather than %d", payloadSize, expectedSize)
+	}
+
+	span := binary.BigEndian.Uint64(payload[:8])


same here with span

addressed in previous messages

pss/trojan/message_test.go

santicomp2014

The changes I described are not critical, they can be address if needed later.
LGTM

zelig

really just a few minor things
already approved
also https://travis-ci.org/github/ethersphere/swarm/jobs/672786246#L489
linter indicates another solution

pss/trojan/message.go

zelig · 2020-04-09T05:04:23Z

pss/trojan/message.go

+}
+
+// toChunk finds a nonce so that when the given trojan chunk fields are hashed, the result will fall in the neighbourhood of one of the given targets
+// this is done by iterating the BMT hash of the serialization of the trojan chunk fields until the desired nonce is found


this is done by iteratively enumerating different nonces until the BMT hash of the serialization of the trojan chunk fields results in chunk address that has one of the targets as its prefix

pss/trojan/message.go

mortelli · 2020-04-09T13:00:09Z

really just a few minor things
already approved
also https://travis-ci.org/github/ethersphere/swarm/jobs/672786246#L489
linter indicates another solution

thank you @zelig for the relentless reviewing ❤️

fixed the linter issues as well.

just have a couple of very minor issues that i'd like to resolve before merging and we're good to go

zelig

<3

mortelli added 19 commits April 1, 2020 14:25

chunk: add trojan chunk file

2175c64

chunk: add pssEnvelope, trojan structs

7b61554

chunk, pss: move trojan_chunk file to pss package

122d7fc

pss: add newTrojanChunk func

190e049

pss: add newTrojanMessage func

09f999b

pss: iterate newTrojanMessage

0c3fb7d

pss: add trojanHeaders type, newTrojanHeaders func

2e638ae

pss: add findMessageNonce func

1a1fea2

pss: update copyright notice

5db5ff7

pss: add iterateNonce, TestFindMessageNonce funcs

f9b250d

pss: iterate findMessageNonce, iterateNonce funcs

c604c8b

pss: add TestMarshaling func for trojan chunks

4389272

pss: remove pssEnvelope struct, remove trojanHeaders decryptionHint f…

d849b37

…ield, add marshalling and unmarshalling functions for trojanMessage, add TestTrojanMessageSerialization func

pss: add trojanData and trojanChunk structs, iterate trojan funcs bas…

1f51485

…ed on new design, add TODOs

pss: iterate trojan chunk tests

19ca96c

pss: iterate marshaling and unmarshasling funcs for trojanChunk and t…

57921b5

…rojanData

pss: add equals func for trojanData struct

866a96c

pss: minor refactor to equals receiver variable for trojandata struct

b094d96

Merge remote-tracking branch 'origin/global-pinning' into global-pinn…

9667d56

…ing-trojan-chunks

mortelli added in progress global pinning experimental implementation of global pinning labels Apr 2, 2020

mortelli requested review from zelig and santicomp2014 April 2, 2020 23:37

mortelli self-assigned this Apr 2, 2020

mortelli added 6 commits April 2, 2020 20:39

pss: iterate trojan chunks comments

dab774e

pss: iterate trojan chunks comments

112516f

pss: iterate iterateNonce func

01cd6bd

pss: iterate iterateNonce func

6f9866d

pss: minor change to trojan chunks test addr var

7a77037

pss: iterate trojan chunk comments

b7bd71d

mortelli added 3 commits April 8, 2020 16:30

trojan: rename newTrojanChunk func to Wrap

b1705b1

trojan: remove newSpan func

4d17358

trojan: rename iterTrojanChunk func to toChunk

15aaf8f

santicomp2014 reviewed Apr 8, 2020

View reviewed changes

mortelli added 6 commits April 8, 2020 17:50

trojan: add TestPadBytes func

aa23015

trojan: change wrap signature not to use a receiver

66b29f7

trojan: change span interpretation to little-endian

7111cd1

trojan: remove the word "trojan" from tests

29c59b4

trojan: update comments in tests

9d8a616

trojan: iterate comments in message code

c01ae94

santicomp2014 approved these changes Apr 9, 2020

View reviewed changes

trojan: minor renamings

ebd0b09

zelig approved these changes Apr 9, 2020

View reviewed changes

zelig marked this pull request as ready for review April 9, 2020 05:15

mortelli added 5 commits April 9, 2020 09:26

pss: fix linter error in testTargets var

a2dc971

pss: export errors and add comments to them

10f65b3

pss: export NewMessage func

31ecc0b

trojan: update comment for Wrap func

6225a80

trojan: add receivers to Wrap and toChunk funcs

230453c

mortelli mentioned this pull request Apr 9, 2020

pss: add barebone new pss #2150

Merged

zelig approved these changes Apr 9, 2020

View reviewed changes

mortelli merged commit 683571e into global-pinning Apr 10, 2020

This was referenced Apr 12, 2020

[User-Story] Global Pinning ethersphere/user-stories#74

Open

global pinning: phase 0 #2153

Closed

make trojan chunk mining work on bit vectors #2156

Open

optimize trojan chunk mining #2157

Open

mortelli deleted the global-pinning-trojan-chunks branch April 13, 2020 17:43

This was referenced Jul 14, 2020

make trojan chunk mining work on bit vectors ethersphere/bee#443

Closed

optimize trojan chunk mining ethersphere/bee#444

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pss: add trojan chunks #2148

pss: add trojan chunks #2148

mortelli commented Apr 2, 2020 •

edited

Loading

santicomp2014 left a comment

santicomp2014 Apr 8, 2020 •

edited

Loading

mortelli Apr 8, 2020

santicomp2014 Apr 8, 2020

mortelli Apr 8, 2020

mortelli Apr 8, 2020

santicomp2014 Apr 8, 2020

mortelli Apr 8, 2020

santicomp2014 left a comment

zelig left a comment •

edited

Loading

zelig Apr 9, 2020

mortelli Apr 9, 2020

mortelli commented Apr 9, 2020

zelig left a comment

pss: add trojan chunks #2148

pss: add trojan chunks #2148

Conversation

mortelli commented Apr 2, 2020 • edited Loading

Implementation overview

new package

new types

main new functions

Open questions

TODOs

santicomp2014 left a comment

Choose a reason for hiding this comment

santicomp2014 Apr 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

santicomp2014 left a comment

Choose a reason for hiding this comment

zelig left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mortelli commented Apr 9, 2020

zelig left a comment

Choose a reason for hiding this comment

mortelli commented Apr 2, 2020 •

edited

Loading

santicomp2014 Apr 8, 2020 •

edited

Loading

zelig left a comment •

edited

Loading