core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders #450

albrow · 2019-10-16T00:04:02Z

Fixes #431.

This PR is still a WIP but I'm opening it now so we can discuss the progress so far. This is an implementation of the idea we've been discussing on Slack and a solution to the problem of finite database storage. To quickly re-summarize the idea:

When storage space is plentiful, we start without any kind of limit on expiration time. As the database fills up, we enforce a limit on the maximum expiration time. Any orders that expire after that time will be removed. If more orders keep coming in we can lower the expiration time and remove more orders.

So far this PR adds an index on ExpirationTime and implements a new method called TrimOrdersByExpirationTime which contains the bulk of the core logic for this feature. There are still some challenges around how to use this function and when to call it, but the approach is looking very feasible as of now. We also still need to determine when and how quickly to increase the max expiration time after storage frees up.

constants/constants.go

core/core.go

zeroex/order.go

zeroex/orderwatch/order_watcher.go

browser/ts/index.ts

albrow · 2019-10-22T21:37:05Z

@fabioberger Removed WIP tag. This PR is ready for final review.

I do eventually want to optimize our SlowCounter implementation to reduce memory allocations and math/big operations. Our current strategy of using a polling loop inside of OrderWatcher to check if the max expiration time can be increased is also inefficient, but should be good enough for now.

albrow · 2019-10-22T22:08:50Z

~~Just reminding myself to update the CHANGELOG after we include everything there for the upcoming 5.1.0-beta launch.~~

Update: I added this.

Co-Authored-By: Fabio B <me@fabioberger.com>

…ently removed

browser/ts/index.ts

meshdb/meshdb_test.go

zeroex/orderwatch/order_watcher.go

fabioberger · 2019-10-24T02:42:25Z

zeroex/orderwatch/order_watcher.go

+	// Final expiration time check before inserting the order. We might have just
+	// changed max expiration time above.
+	if orderInfo.SignedOrder.ExpirationTimeSeconds.Cmp(w.maxExpirationTime) == 1 {
+		return nil


Won't this mean that the node operator won't get the STOPPED_WATCHING event emitted?

@fabioberger Yes but it also means they never received the ADDED event, which seems consistent to me. What do you think is the right behavior here?

Since we ACKd the incoming JSON-RPC request, we should return STOPPED_WATCHING.

zeroex/orderwatch/order_watcher.go

zeroex/orderwatch/slowcounter/slow_counter.go

Co-Authored-By: Fabio B <me@fabioberger.com>

…/0x-mesh into feature/dynamic-max-expiration

fabioberger

Small typo but otherwise LGTM!

zeroex/orderwatch/order_watcher.go

Co-Authored-By: Fabio B <me@fabioberger.com>

albrow assigned albrow and fabioberger Oct 16, 2019

albrow force-pushed the feature/dynamic-max-expiration branch from 48fcd11 to f5896bb Compare October 16, 2019 23:47

fabioberger reviewed Oct 17, 2019

View reviewed changes

fabioberger force-pushed the development branch from 11e0fa4 to cdece2c Compare October 18, 2019 04:12

fabioberger force-pushed the feature/dynamic-max-expiration branch from a11d5aa to 6eb6010 Compare October 18, 2019 04:34

albrow force-pushed the feature/dynamic-max-expiration branch 2 times, most recently from 511dc61 to 3c32584 Compare October 21, 2019 22:52

albrow commented Oct 22, 2019

View reviewed changes

browser/ts/index.ts Outdated Show resolved Hide resolved

albrow force-pushed the feature/dynamic-max-expiration branch 2 times, most recently from 6d02665 to b7d3fa6 Compare October 22, 2019 17:52

albrow changed the title ~~WIP: Implement a dynamically decreasing max expiration time for orders~~ Implement a dynamically decreasing max expiration time for orders Oct 22, 2019

albrow changed the title ~~Implement a dynamically decreasing max expiration time for orders~~ core, orderwatcher: Implement a dynamically decreasing max expiration time for orders Oct 22, 2019

albrow changed the title ~~core, orderwatcher: Implement a dynamically decreasing max expiration time for orders~~ core, orderwatch: Implement a dynamically decreasing max expiration time for orders Oct 22, 2019

albrow changed the title ~~core, orderwatch: Implement a dynamically decreasing max expiration time for orders~~ core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders Oct 22, 2019

albrow force-pushed the feature/dynamic-max-expiration branch from 963aa94 to 4eded3b Compare October 22, 2019 22:56

albrow and others added 12 commits October 23, 2019 16:58

Add expiration time index

3c2e9ea

Implement and test TrimOrdersByExpirationTime

82d8c06

Integrate TrimOrdersByExpirationTime in OrderWatcher

b411e10

Add an env var for max orders in storage

b957707

Reject orders with an expiration time greater than the current max

1dc6401

Change default max orders to 100k

7fc2f88

Update constants/constants.go

a09bc23

Co-Authored-By: Fabio B <me@fabioberger.com>

Update zeroex/orderwatch/order_watcher.go

065cb1f

Co-Authored-By: Fabio B <me@fabioberger.com>

Remove in-memory state for orders removed due to expiration time

10acf12

Clarify that UnlimitedExpirationTime = 2^256-1

6837d34

Implement and test SlowCounter

88874a0

Use SlowCounter inside OrderWatcher to increase max expiration time

b1962dc

albrow added 15 commits October 23, 2019 16:58

Check if we can increase max expiration time after an order is perman…

c218452

…ently removed

Handle some additional edge cases where the database could be empty

ef33b0b

Simplify SlowCounter implementation and remove concept of "ticks"

45a2812

Ensure max expiration time is not in the past

acc74b7

Add MaxExpirationTime to GetStats

b8a767f

Start a separate loop inside OrderWatcher for increasing max exp time

9229616

Add maxOrdersInStorage config option to TypeScript bindings

af7d71d

Add note about default values to TypeScript bindings

a609ccb

Add TODO in OrderWatcher about extending transaction

0969ef7

Rename a function in OrderWatcher

607df23

Add isMax boolean cache to SlowCounter

a9be34d

Add test in order_watcher_test for max expiration time

fbd5531

Rename REMOVED order event to STOPPED_WATCHING

8d920b2

Add new maxExpirationTime field to docs/rpc_api.md

b2d7567

Update CHANGELOG

a5bfe82

albrow force-pushed the feature/dynamic-max-expiration branch from ddae068 to a5bfe82 Compare October 23, 2019 23:58

fabioberger suggested changes Oct 24, 2019

View reviewed changes

albrow and others added 6 commits October 24, 2019 14:32

Update zeroex/orderwatch/order_watcher.go

ddf6777

Co-Authored-By: Fabio B <me@fabioberger.com>

Update zeroex/orderwatch/order_watcher.go

7e63d30

Co-Authored-By: Fabio B <me@fabioberger.com>

Update zeroex/orderwatch/slowcounter/slow_counter.go

34fb304

Co-Authored-By: Fabio B <me@fabioberger.com>

Return an error for missing config fields in orderwatch.New

38e0b87

Merge branch 'feature/dynamic-max-expiration' of github.com:0xProject…

3957058

…/0x-mesh into feature/dynamic-max-expiration

Add hack in Watcher.Add for orders that suddenly expire too late

30c9abb

fabioberger approved these changes Oct 24, 2019

View reviewed changes

zeroex/orderwatch/order_watcher.go Outdated Show resolved Hide resolved

albrow and others added 2 commits October 24, 2019 15:42

Update zeroex/orderwatch/order_watcher.go

e346f4d

Co-Authored-By: Fabio B <me@fabioberger.com>

Add STOPPED_WATCHING event and missing default for TypeScript bindings

437373b

albrow merged commit 97c1127 into development Oct 24, 2019

albrow deleted the feature/dynamic-max-expiration branch October 24, 2019 22:59

This was referenced Oct 25, 2019

Solve the finite database storage problem #431

Closed

core: Delete random orders when DB is full #373

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders #450

core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders #450

albrow commented Oct 16, 2019 •

edited

Loading

albrow commented Oct 22, 2019

albrow commented Oct 22, 2019 •

edited

Loading

fabioberger Oct 24, 2019

albrow Oct 24, 2019 •

edited

Loading

fabioberger Oct 24, 2019

fabioberger left a comment

core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders #450

core, orderwatch, meshdb: Implement a dynamically decreasing max expiration time for orders #450

Conversation

albrow commented Oct 16, 2019 • edited Loading

albrow commented Oct 22, 2019

albrow commented Oct 22, 2019 • edited Loading

fabioberger Oct 24, 2019

Choose a reason for hiding this comment

albrow Oct 24, 2019 • edited Loading

Choose a reason for hiding this comment

fabioberger Oct 24, 2019

Choose a reason for hiding this comment

fabioberger left a comment

Choose a reason for hiding this comment

albrow commented Oct 16, 2019 •

edited

Loading

albrow commented Oct 22, 2019 •

edited

Loading

albrow Oct 24, 2019 •

edited

Loading