Support for adding external document URLs to the index

[levnikolaevich]

Description:
Currently, there’s no way to add external document URLs to the index. Before starting a task with the agent, I often build a contextual set of best practices based on project-specific approaches and store these files in the project folder. It would be very convenient to be able to save them in external storage instead.

I’ve started using Ref to search for up-to-date manuals, and it would be great if the agent could automatically send the results of its searches as documents into Ref, so they could be referenced later.

Question:
Is such functionality planned or on the roadmap?

Thanks in advance!

[MatthewDailey]

Hi @levnikolaevich thanks for the note! It sounds like you have a Plan step where you manually build of resources (or maybe with the help of an agent?) and you'd like a way save and reference that. Did I get that right?

That seems really interesting. I could imagine a ref_save_notes tool that your agent could use after writing a plan to save notes to reference later. Perhaps they could be exposed as MCP resources. Is that the sort of thing you're talking about?

[levnikolaevich]

Good afternoon! @MatthewDailey, thanks for the reply. Yes, you understood everything exactly right.

The agent first compiles best practices into a single file, since it collects them from different sources—by making queries to Ref or searching the internet. Then it adapts the information to the specifics of the project and prepares a unique document that I refer to periodically afterward. It’s not exactly a “note,” since the document can be quite large, but overall you described it correctly. And the solution you proposed sounds great.

In theory, I can upload it to the index manually, but that’s inconvenient when the entire development pipeline is automated (right now, for example, the document is saved to a folder). However, searching the index offers more advantages than just a folder, and it’s better to have fewer places to search… “just use Ref and get all the answers” instead of “search the folder, search Ref,” etc.

One question: will searches over these documents have priority over other Ref documents? Or would there be a separate endpoint?

It would also be helpful to attach tags to documents so I could ask the agent to use them as a filter—simply to distinguish documents created within a specific project from others, for example.

Thanks a lot in advance!

[MatthewDailey]

I see so if your planning document is very large such that you'd want to search it rather than load the entire thing in context, you could upload it as a resource to your private index on the PDF & Markdown page . Then you'd be able to say "check my private docs with ref" or use the built in my_docs prompt to guide the model to search it.

It's more like a separate endpoint. Ref searches in 2 modes, either public or private docs. I found in practice this is more accurate than trying to mix and prioritize results from public and private indexes.

What you're talking about seems adjacent to Claude Skills and so as I'm exploring this feature space, I'm thinking about how I can have Ref play well with emerging things like Skills. It sounds like your plan docs are bigger than a skill but the idea of using Ref to derive a skill that's used repeatedly on a project is kind of similar.

The timeline for rolling any of this out is basically once I can get an MVP and play with it enough to feel useful. Happy to add you to a feature flag for an mvp whenever I get there if you're interested?

[levnikolaevich]

I’d really appreciate it if you could add me — I’m very interested.

By the way, you’re absolutely right about Skills. I’m actively transitioning to using Skills within my project. The idea is also to delegate to Claude the ongoing search for best practices (and Ref is very helpful here, since it contains a lot of up-to-date information). But again, once the information for a specific task is compiled from different sources and adapted to the specifics of the project, it would be great to have the ability to store it in the index.

There are other solutions and MCPs that can do this (for example, indexing your entire codebase and context). But from what I’ve seen, they usually require setting up your own local vector database or connecting through a separate API. Plus, the context of their MCP tools is often excessively large.

Your product, though, seems just a small step away from this functionality — and it would be amazing if it became available.

Thanks!

[MatthewDailey]

Wonderful, I'll get in the lab and let you know when something's ready :)

[davidmucci]

@MatthewDailey @levnikolaevich - A lot of what y'all are discussing has also come up within my team, so wanted to chime in..

What I'm Currently Using

Running HumanLayer thoughts (git-based) for this exact workflow:

• /create_plan command compiles docs from multiple sources
• Could hook into Ref MCP for pulling up-to-date library docs
• Saves everything to thoughts/shared/plans/ with embedded snippets
• Later sessions and other devs just read from the saved plans

It works, but yeah... the friction is real:

Search is kinda limited:

• Just grep-based (thoughts-locator agent uses Grep/Glob/LS)
• No semantic understanding - you gotta know what terms to search
• Need exact file paths to grab stuff

Git ceremony gets annoying:

• Every save = commit/push/sync dance
• Merge conflicts when multiple people are compiling docs at once
• Not instant - teammates have to git pull to see your research

Cross-project stuff is clunky:

• There's a thoughts/global/ for cross-repo docs but search is still manual
• Can't just ask "show me all webhook patterns across our 5 projects"
• Knowledge ends up siloed per-repo

Why ref_save_notes Would Be Way Better

What would make me ditch git for this:

1. Semantic search - "what's our auth pattern?" just finds it, no filename guessing
2. Cross-project knowledge - search ALL team docs at once
3. Instant - save once, whole team can search it immediately
4. Unified - public docs + team knowledge in one place
5. No git nonsense - just save and done

@MatthewDailey down to beta test whenever you nail down a POC

[MatthewDailey]

@davidmucci I hear the desire for less git ceremony but if you're okay with some you might have some options with Ref as is. Are the repos you're working in indexed in Ref?

The workflow I'd imagine is you create your plan or note doc, commit and push it. From there (after a few minutes) Ref will have it indexed and searchable. You have some git process (need to commit and push) but wouldn't need to sync and would have a single cross-repo index with semantic+full-text search. Could that work?

The other options to enable a save-note I'm considering are either only adding the feature to the local stdio server that can upload files or a ref-cli save <files> CLI that you'd need have a separate system prompt or slash command for. Does either of those sound preferrable to you?