introduce low memory flag #2012

whyrusleeping · 2015-11-28T00:05:09Z

I want to try an experiment. I want to have an ipfs 'low memory mode' that when enabled, limits ipfs's memory usage by turning down various numbers, reduces concurrency, and makes in memory caches smaller.

If anyone has any thoughts on where these changes should be made, feel free to post proposals.

License: MIT
Signed-off-by: Jeromy jeromyj@gmail.com

License: MIT Signed-off-by: David Dias <daviddias.p@gmail.com>

This used to lead to large refcount numbers, causing Flush to create a lot of IPFS objects, and merkledag to consume tens of gigabytes of RAM. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

OS X sed is documented as "-i SUFFIX", GNU sed as "-iSUFFIX". The one consistent case seems to be "-iSUFFIX", where suffix cannot empty (or OS X will parse the next argument as the suffix). This used to leave around files named `refsout=` on Linux, and was just confusing. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

These secondary copies were never actually queried, and didn't contain the indirect refcounts so they couldn't become the authoritative source anyway as is. New goal is to move pinning into IPFS objects. A migration will be needed to remove the old data from the datastore. This can happen at any time after this commit. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Pinner had method GetManual that returned a ManualPinner, so every Pinner had to implement ManualPinner anyway. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Platform-dependent behavior is not nice, and negative refcounts are not very useful. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> sharness: Don't assume we know all things that can create garbage License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

WARNING: No migration performed! That needs to come in a separate commit, perhaps amended into this one. This is the minimal rewrite, only changing the storage from JSON(+extra keys) in Datastore to IPFS objects. All of the pinning state is still loaded in memory, and written from scratch on Flush. To do more would require API changes, e.g. adding error returns. Set/Multiset is not cleanly separated into a library, yet, as it's API is expected to change radically. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

…lure License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

There was doublewrapping with an unneeded msgio. given that we use a stream muxer now, msgio is only needed by secureConn -- to signal the boundaries of an encrypted / mac-ed ciphertext. Side note: i think including the varint length in the clear is actually a bad idea that can be exploited by an attacker. it should be encrypted, too. (TODO) License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

* ID service stream * make the relay service use msmux * fix nc tests Note from jbenet: Maybe we should remove the old protocol/muxer and see what breaks. It shouldn't be used by anything now. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> Signed-off-by: Juan Batiz-Benet <juan@benet.ai>

The addition of a locking interface to the blockstore allows us to perform atomic operations on the underlying datastore without having to worry about different operations happening in the background, such as garbage collection. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

This commit improves (fixes) the FetchGraph call for recursively fetching every descendant node of a given merkledag node. This operation should be the simplest way of ensuring that you have replicated a dag locally. This commit also implements a method in the merkledag package called EnumerateChildren, this method is used to get a set of the keys of every descendant node of the given node. All keys found are noted in the passed in KeySet, which may in the future be implemented on disk to avoid excessive memory consumption. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Juan Batiz-Benet <juan@benet.ai>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> dont GC blocks used by pinner License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> comment GC algo License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> add lock to blockstore to prevent GC from eating wanted blocks License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> improve FetchGraph License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> separate interfaces for blockstore and GCBlockstore License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> reintroduce indirect pinning, add enumerateChildren dag method License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

if bucket doesnt have enough peers, grab more elsewhere

add closenotify and large timeout to gateway

License: MIT Signed-off-by: rht <rhtbot@gmail.com>

Add config option for flatfs no-sync

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping · 2015-11-28T00:09:20Z

My idea here is that we find a bunch of places where we can tweak things to make usage lower, and make a 'binary' switch for them now. Later, once we've matured the option a bit, we can make various 'levels' of memory consumption mode (maybe even a high memory mode) that would affect the same areas of code.

rht · 2015-11-28T01:06:12Z

A global runtime memory limit was proposed in #1482, which can't be enabled until at least go1.6 (golang/go#10460).

ghost · 2015-11-28T04:20:42Z

This will be great for embedded devices!

rht · 2015-11-28T04:29:16Z

Never mind embedded devices, even VPS got choked with the ram consumption.

Though there is always ulimit (when in unix).

whyrusleeping · 2015-11-28T07:08:55Z

right, the point of this PR wont be to set a hard number limit on memory, it will be policy changes to attempt to use less memory.

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

rht · 2015-11-28T08:15:53Z

concurrentFdDials?

whyrusleeping · 2015-11-28T23:42:17Z

@rht ooooh, yeah. thats a good one

rht · 2015-11-29T01:13:30Z

p2p/net/swarm/swarm_dial.go

+var concurrentFdDials = 160
+
+func init() {
+	concurrentFdDials = 80


you forgot if u.LowMemMode

facepalm I'm good at this...

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jbenet · 2015-12-01T16:26:27Z

cmd/ipfs/main.go

@@ -182,6 +182,9 @@ func (i *cmdInvocation) Run(ctx context.Context) (output io.Reader, err error) {
 	if u.GetenvBool("DEBUG") {
 		u.Debug = true
 	}
+	if u.GetenvBool("IPFS_LOW_MEM") {
+		u.LowMemMode = true


maybe this should go into the config? ok to start like this i guess

Yeah, my thought was to keep this 'unofficial' until we're more confident about it working. Would suck to have a low memory mode that increased memory for some people, lol

jbenet · 2015-12-01T16:27:45Z

great idea! could also

make it accept a memory bound and try to stick to that (back to resource constraints)
run test suites with low mem on
finally add that test with a max memory, and start hiking that lower

whyrusleeping · 2015-12-03T23:14:40Z

make it accept a memory bound and try to stick to that (back to resource constraints)

I definitely want that, but I think it will be best to get this out there to get a better feel for its effect on various workloads first. Then once we have a better understanding, start being smarter

run test suites with low mem on

Yep! I'll enable that

finally add that test with a max memory, and start hiking that lower

mmmm, that would be nice. definitely on the todo list

jbenet · 2015-12-04T00:50:04Z

Yep! I'll enable that

we should run the test suite over both cases. it may have effects in hiding or showing bugs.

daviddias and others added 30 commits November 11, 2015 10:04

update version

f72b675

License: MIT Signed-off-by: David Dias <daviddias.p@gmail.com>

pin: Guard against callers causing refcount underflow

80a8af7

This used to lead to large refcount numbers, causing Flush to create a lot of IPFS objects, and merkledag to consume tens of gigabytes of RAM. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

sharness: fusermount -u is the documented way to unmount FUSE on Linux

86d781a

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

pin: unexport NewIndirectPin, it's not useful and not used elsewhere

cb46a52

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

blocks/set: Remove now-unused NewDBWrapperSet

34f8231

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Simplify Pinner interface by folding ManualPinner into Pinner

3a05a94

Pinner had method GetManual that returned a ManualPinner, so every Pinner had to implement ManualPinner anyway. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

pin: Remove dead code

832d639

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

pin: Remove double bookkeeping of refcount keys

c65bf3f

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Use uint64 for indirect pin refcounts

f85862b

Platform-dependent behavior is not nice, and negative refcounts are not very useful. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Typo

99448d4

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

sharness: Don't assume we know all things that can create garbage

21e7b05

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> sharness: Don't assume we know all things that can create garbage License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

pin: Future-proof against refcount marshaled size changes

283432f

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

bump fsrepo version to 3

81b16e2

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

pin: Do not accidentally delete indirect pins on Flush

5db177e

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

dagmodifier: Don't lose pin if old and new key happen to be equal

fd28a39

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

dagmodifier test: Add TODO note about how bad luck can cause test fai…

1e0d72f

…lure License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

buffer msgio

8ee7129

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

address concerns from PR

506c46f

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

move locking out of GC branch

06184a6

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

lock blockstore for pin add

588f53e

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

renamed {R,}Lock -> {Pin,GC}Lock

d71bb78

License: MIT Signed-off-by: Juan Batiz-Benet <juan@benet.ai>

break up GC logic

8650e44

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jbenet and others added 5 commits November 20, 2015 00:27

Merge pull request #1967 from ipfs/fix/kbucket-fill

d46c0d4

if bucket doesnt have enough peers, grab more elsewhere

Merge pull request #1980 from ipfs/fix/gateway-close-notif

856e250

add closenotify and large timeout to gateway

Add config option for flatfs no-sync

cab5868

License: MIT Signed-off-by: rht <rhtbot@gmail.com>

Merge pull request #1985 from ipfs/rht-add-no-sync

b4ab684

Add config option for flatfs no-sync

introduce low memory flag

a6a1dbc

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jbenet added the status/in-progress In progress label Nov 28, 2015

low memory setting for bitswap

00dc67c

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

rht reviewed Nov 29, 2015
View reviewed changes

low mem experiemnt on dials

c809589

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping force-pushed the feat/low-mem-experiment branch from baeb949 to c809589 Compare November 30, 2015 03:43

jbenet reviewed Dec 1, 2015
View reviewed changes

ghost added topic/perf Performance need/community-input Needs input from the wider community labels Dec 22, 2015

whyrusleeping force-pushed the dev0.4.0 branch 2 times, most recently from 68b9745 to b0a8591 Compare December 28, 2015 14:36

whyrusleeping force-pushed the dev0.4.0 branch from cf514d8 to 3224ae0 Compare January 12, 2016 16:23

rht mentioned this pull request Jan 14, 2016

Miniature solarnet lab: resource constraints, testings, and discoveries ipfs/infra#145

Closed

whyrusleeping closed this Mar 2, 2016

whyrusleeping deleted the feat/low-mem-experiment branch March 2, 2016 07:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce low memory flag #2012

introduce low memory flag #2012

whyrusleeping commented Nov 28, 2015

whyrusleeping commented Nov 28, 2015

rht commented Nov 28, 2015

ghost commented Nov 28, 2015

rht commented Nov 28, 2015

whyrusleeping commented Nov 28, 2015

rht commented Nov 28, 2015

whyrusleeping commented Nov 28, 2015

rht Nov 29, 2015

whyrusleeping Nov 30, 2015

jbenet Dec 1, 2015

whyrusleeping Dec 3, 2015

jbenet commented Dec 1, 2015

whyrusleeping commented Dec 3, 2015

jbenet commented Dec 4, 2015

introduce low memory flag #2012

introduce low memory flag #2012

Conversation

whyrusleeping commented Nov 28, 2015

whyrusleeping commented Nov 28, 2015

rht commented Nov 28, 2015

ghost commented Nov 28, 2015

rht commented Nov 28, 2015

whyrusleeping commented Nov 28, 2015

rht commented Nov 28, 2015

whyrusleeping commented Nov 28, 2015

rht Nov 29, 2015

Choose a reason for hiding this comment

whyrusleeping Nov 30, 2015

Choose a reason for hiding this comment

jbenet Dec 1, 2015

Choose a reason for hiding this comment

whyrusleeping Dec 3, 2015

Choose a reason for hiding this comment

jbenet commented Dec 1, 2015

whyrusleeping commented Dec 3, 2015

jbenet commented Dec 4, 2015