Repo Map Accuracy

[dwash96]

The way the repository map works is that:

• Treesitter extracts a lists of definitions and reference identifiers per file
• These identifiers are chunked into larger maps of identifier -> definer files and identifier -> reference files
• A graph is computed from these lists (which introduced duplication as seen here
• The weights for these multipliers have fairly naive static values (number of references, number of definers)

The gist is that turning the determination of the multipliers to equations should allow for a more dynamic proxy for "relevancy" in the repo map construction, which should give the LLM at least marginally better context to operate off of. My hypothesis is that logarithmic equations based on the number of overall definers (to detect rote/boiler plate code) and on the number of references (in terms of unique files, and absolute references) will serve as a better checker and balancer for how relevant/central identifiers and files are inside of codebases

[dwash96]

To note, none of this replaces having an actual AST/LSP for popular languages to further refine this idea by actually computing the centrality of classes/functions/etc. within a code base. I think a two-pass structure for generating the repo map is probably the best foot forward of using treesitter to quickly extract and heuristically rank identifiers, and then running an AST parser in the relevant languages on top for the top-n files and identifiers from the heuristic method for increased accuracy

[Netzvamp]

Sounds interesting and useful for big repositories. Maybe there should be some kind of threshold where the normal repomap is enough and if the repomap is hitting the limit we get an optimized repomap? Sometimes you have very specific classes like onetime used exceptions in a separate file with an expressive docstring, it would be a bummer to loose these in the repomap unnecessary cause the usage is just 1.
But yes, there are repos where the repomap is 100k+ tokens where it's hard to fit full files + chat into a 250k token limit and there is currently no solution with aider. +1 from me.

[dwash96]

v0.88.27 contains a fairly large milestone on this front with --use-enhanced-map. By modifying the tree-sitter query files to extract import-like statements in the relevant languages, this can be used to create a "dependency estimator".

To more specifically re-iterate the above, the way the repo map is built is that aider extracts the identifiers (function names and class names) from the code base as well as if a given file declares or references that identifier. Then, for each identifier, it iterates through the list of definers for the identifiers and builds a graph that connects referencers to definers. It decides the weight of the edges based on the modified logarithmic heuristics above based on identifier frequency. It then does PageRank on these weights and edges based on the identifiers in the user prompt to surface the most relevant snippets

By adding in import estimation, we can better prune which referencers get joined to which definers. Dependency estimation is keyed off of the assumption that in most programming languages "dependency imports use a filesystem-like syntax" e.g., dot and slash separated identifiers are used to specify the identifier that corresponds to at least a sub part of the path syntax of the filesystem of the code base. Therefore, two files can be said to be linked if they:

a) share an identifier in common
and
b) the file path of the definer contains a sequentially consistent part of one of the referencer's import identifiers

The latter means a file system path like a/b/c/d.language can be mapped to an import source specifier like any of:

a.b.c.d
a/b/c/d
c/d
..d
d.language

By checking whether this condition holds before adding edges to the graph, you can roughly prune/resolve which files actually import which other ones, which should reduce symbol collisions, and build a truer page ranking of which identifiers are relevant to which files

Examples for aider-ce below:

aiders/coder/base_coder.py - import extraction

aider/coders/base_coder.py': { 
    'aider',
    'aider.analytics',
    'aider.coders',
    'aider.commands',
    'aider.exceptions',
    'aider.helpers',
    'aider.history',
    'aider.io',
    'aider.linter',
    'aider.llm',
    'aider.mcp.server',
    'aider.models',
    'aider.reasoning_tags',
    'aider.repo',
    'aider.repomap',
    'aider.run_cmd',
    'aider.sessions',
    'aider.utils',
    'asyncio',
    'babel',
    'base64',
    'collections',
    'datetime',
    'hashlib',
    'httpx',
    'json',
    'json.decoder',
    'jsonschema',
    'litellm',
    'litellm.types.utils',
    'locale',
    'math',
    'mimetypes',
    'os',
    'pathlib',
    'platform',
    'prompt_toolkit.patch_stdout',
    're',
    'rich.console',
    'sys',
    'threading',
    'time',
    'traceback',
    'typing',
    'weakref'
}

aiders/coder/base_coder.py - estimation examples

...

'aider/coders/base_coder.py - get_ex_info'
'---Matches---'
['aider/exceptions.py']

...

'aider/coders/base_coder.py - is_command'
'---Matches---'
['aider/commands.py']

...

'aider/coders/base_coder.py - commit'
'---Matches---'
['aider/repo.py']

...

'aider/coders/base_coder.py - tool_output'
'---Matches---'
['aider/io.py']

...

# Sub Classes Tend to Produce Constructs Like This, Will Certainly Be Updated Further Still
'aider/coders/base_coder.py - get_edits'
'---Matches---'
[ 'aider/coders/single_wholefile_func_coder.py',
  'aider/coders/wholefile_coder.py',
  'aider/coders/editblock_coder.py',
  'aider/coders/patch_coder.py',
  'aider/coders/help_coder.py',
  'aider/coders/udiff_coder.py',
  'aider/coders/base_coder.py']