Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove watching directory introduced in 0.9 #602

Open
fulmicoton opened this issue Jul 25, 2019 · 5 comments
Open

Remove watching directory introduced in 0.9 #602

fulmicoton opened this issue Jul 25, 2019 · 5 comments
Milestone

Comments

@fulmicoton
Copy link
Collaborator

Tantivy 0.9 introduced a funcationality to watch and autoreload an index when a change in meta.json has been detected.

It has caused errors to different users on various platforms.

Let's just remove it and "poll" meta.json, or only enable the functionality when
the writer and the reader are in the same process.

@halvorboe
Copy link
Contributor

halvorboe commented Jan 30, 2020

Polling is one possibility. It will replace the current system and will work.

I see 3 options if we're not going to do polling:

1. Static MessageBroker (between threads)

Idea: Implementing an MessageBroker struct that implements a many-to-many functionality. Having a static variable hold an instance of this struct allows all publishers and subscribers to connect.

// message_broker.rs

struct MessageBroker<F> {
    callbacks: Vec<F>,
}

impl MessageBroker<F> {

    fn publish(index_name: String) {
         ...
    }

    fn subscribe(index_name: String, callback: F) {
        ...
    }

}

// main.rs
static mut broker: Lazy<MessageBroker> = null_ptr; 

This could be used to implement a flow like:

  1. A IndexReader connects to the broker.
  2. A message is published by an IndexWriter.
  3. The message is distributed to all subscribed callbacks.
  4. The IndexReader receives the message through the callback.

Pros

  • Will not change the outward-facing API.
  • Intuitive - I would expect this behavior.
  • Would make it easy to throttle updates (so that reader does not try to update 10k times a second).

Cons

  • Hard to test and debug.
  • Not a simple solution.

2. Non-static MessageBroker (between threads)

This is similar to the approach described above, but the reader is not static. The API would look something like this:

let broker = Broker::new()
reader.attach(&broker)
writer.attach(&broker) 

Pros

  • Easy to implement.
  • Would give great results for an experienced user.

Cons

  • Hard to debug for user. Let's say there is a bug in the code that causes a reader to not be attached. It might not be intuitive what the problem is.
  • Would make the library behave in a way a less experienced user might not expect.

3. Use ZeroMQ

Not sure if this is viable, but could run a ZeroMQ server on localhost and have all the processes connect to it.

https://github.com/erickt/rust-zmq

I'm not the best at "technical communication". Hope it makes sense 😄

@fulmicoton
Copy link
Collaborator Author

no zeromq. That's too heavy.

1 and 2 do not solve the problem of cross-process.
3. is too heavy handed.

I'm ok with a solution that replaces polling when we have an IndexWriter on the process for the given directory. It is tricky but possible to implement

@halvorboe
Copy link
Contributor

halvorboe commented Feb 4, 2020

Notify library has support for PollWatcher.

https://github.com/notify-rs/notify/blob/main/src/lib.rs

Could have a flag to toggle whether to use the recommended or the poll watcher.

@fulmicoton
Copy link
Collaborator Author

@halvorboe I'd rather remove the dependency to the notify library to be honest.

@halvorboe
Copy link
Contributor

@fulmicoton Ok, will look into implementing the same functionality as the library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants