Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TimeFieldIndex #117

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

hackerwins
Copy link

@hackerwins hackerwins commented Dec 25, 2021

My team is using memdb with a time.Time field. In my view, time.Time is a type that can be used in general.

I don't know if this is a good way to index the time.Time field, but this PR adds a TimeFieldIndex to make it easy to index time.Time.

Notes for reviewer:

Adding a Time field to TestObject causes panic from quick.Check. So I added a separate object, TestObjectWithTime.
golang/go#27017

@hashicorp-cla
Copy link

hashicorp-cla commented Dec 25, 2021

CLA assistant check
All committers have signed the CLA.

@hackerwins hackerwins force-pushed the time-field-index branch 2 times, most recently from 6643163 to e87deac Compare December 25, 2021 07:38
@kisunji
Copy link
Contributor

kisunji commented Jan 13, 2022

Hi @hackerwins, thank you for this PR.

I think that indexing time.Time should be implemented by clients and not the library because requirements may differ (e.g. datetime precision or memory constraints for indices).

Right now I don't see a strong reason to add it to our library but will leave this PR open as a reference or discussion point for others.

@hackerwins
Copy link
Author

hackerwins commented Jan 13, 2022

@kisunji Thanks for your review.

In my view, I don't think it will cause memory problems much since time.Time can be expressed as 12 bytes(similar to indexing 12 character strings). And If the library does not support TimeFieldIndex, client has to change time.Time fields to byte slices and index it and this is a bit cumbersome.

But it may be just me for now, so I'm curious what other people think.

@radeksimko
Copy link
Member

I understand the concerns mentioned, I'll just say that we have a use case for this exact implementation (full precision), primarily for sorting. Specifically we track changes to modules in Terraform LS, each with a timestamp and when processing these changes we want to iterate from oldest to newest record. We could also just use incrementing uint counter, but we already use the timestamp to batch changes within a certain timespan, so it seems better to reuse that field than add a new one.

I didn't try to compare the memory usage of time.Time field indexed vs extra uint field indexed. Given the anticipated amount of records at any given time though I doubt it would make a difference for us.

I'd be curious about the other use cases which involve lookups and/or reduced precision for memory savings. Relatedly I wonder if the precision could be somehow configurable, similar to how letter casing is configurable for StringFieldIndex? 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants