fix rampant memory leak in providers records storage

[whyrusleeping]

This was broken, its better now. provider records still arent signed, but now our gateways wont inadvertently run themselves out of memory.

[cryptix]

pm.run is the only func that access this map, right? might want to document that in the struct

[whyrusleeping]

in the future, i do want to move back to using arrays here, maps use a lot of memory, and iterating maps into an array for every request is moderately expensive. But doing so would require a limit on the number of providers stored for any given key to avoid the problem i'm fixing here again (checking for existence in a large array is unpleasant, but doable) Maybe I should just implement that now?

[jbenet]

this is making a huge array every time, isn't it? nevermind

[jbenet]

@whyrusleeping what i do for this sort of thing is keep both. something like:

var ProviderLifetime = time.Hour * 24

// record key : providers
var providers = map[u.Key]*ProviderSet

type ProviderSet struct {
  records []Record // to be copied right into a message
  set map[u.Key]time.Time // set for fast checking.
}

func (ps *ProviderSet) add(r Record) {
  if   _, found := ps.set[r.key]; !found {
    ps.records = append(ps.records, r)
  }
  ps.set[r.key] = time.Now()
}

func (ps *ProviderSet) expire() {
  // cleanup both ps.set and ps.records for entries whose time is > ProviderLifetime
}

the memory usage is not prohibitive and yields higher req/s

[jbenet]

constant:

var ProviderLifetime = time.Hour*24

[whyrusleeping]

I would argue that the memory usage on that is prohibitive, we are going to have to store a lot of providers (or set limits, which opens pandoras box). a 2x bump in memory usage is gonna hurt.

[whyrusleeping]

but i guess we can cross that bridge when we get there. I'll go ahead and implement the map/slice combo.