pilosa holder has to sync up data with indexes

[kuba--]

Investigate how pilosa holder propagates Open/Close.

[kuba--]

First of all we don't need a holder, at all!
We can create an index directly and return it. We don't need a global container for all indexes. We have a path so if we need to get an index we can create a new instance and load data from a disk.
With holder we would have to sync up in-memory indexes with storage + extra instance to maintain, which consume memory and is very error prone (panics).

Workflow and cascade of calls:

pilosa holder

Open (panics on closed channel)

• index.Open
  • field.Open
    • view.open
      • fragment.Open
        • openStorage (error on already opened)

//...
// Lock the underlying file.
if err := syscall.Flock(int(f.file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
	return fmt.Errorf("flock: %s", err)
}

			-  openCache

Close (panics on closed channel)

• index.Close
  • field.Close
    • view.close
      • fragment.Close
        • flushCache
        • closeStorage

---

current driver api

NewDriver

- pilosa.NewHolder

Create

- holder.CreateIndexIfNotExists
	- index.Open
- lookup
	- index.Open
	- index.Close

LoadAll

- holder.Index(...)

Save

- index.Field(...)	

Delete

- index.Field(...)
- index.DeleteField(...)
	- field.Close

---

Lookup operations (intersect, union, ...)

- Values
	- index.Open (defer index.Close)
	- values()

// evaluate composition of operations
for _, op := range l.operations {
		// ...

// this lookup (il) contains a different index instance, but it points to the same file!
// it will return an error, because inside values we need to load a different index (related to the operation), but opening already opened index (storage) returns an error because of syscall.Flock
r, e = il.values(p) // <--- error
}

We can avoid an error if we open an index in indexLookup.values(..), Close it before we iterate over operations.