Regarding Crossfile Completion Prompt Format for Seed-Coder-8B Series Models on CrossCodeEval Benchmark

[YDJSIR-NJU]

Referencing the technical report, could you please provide the specific prompt format used to evaluateSeed-Coder-8B-Base on the CrossCodeEval benchmark?

This information is needed to accurately reproduce the evaluation setup and for comparative analysis.

Specifically:

CrossCodeEval Prompt Format:

• What was the exact prompt structure/template used for Seed-Coder-8B-Base during the CrossCodeEval evaluation?

Cross-File Context Handling:

• How was cross-file context incorporated into the prompts for the CrossCodeEval evaluation (if applicable)?
• More generally, does the Seed-Coder model series utilize special tokens or a specific prompt format for handling multi-file context in repository-level code completion tasks (potentially similar to mechanisms in models like the Qwen2.5-Coder series)?

Details on the format(s), including any special tokens, or providing an example prompt would be very helpful.

Thanks!

[kotlyar-shapirov]

+1 Here

[Nes-tle]

Thanks for raising this issue. We are actively finalizing the fim scripts for easy reproduction. In the mean time, you may fork Qwen 2.5 fim eval for a quick start. For cceval, two simple modifications are required:

In utils.py, add a model name to support Seed-Coder, and a model type to support SPM format as follows:

import json
import os
from typing import List, Dict
from transformers import AutoTokenizer

def prepare_prompt(tokenizer, task, model_type, model_name, left_cxt, right_cxt=None, crossfile_cxt=None, args=None):
    if task == "function_completion":
        args.gen_length = 256

    
    # Set fim indicators for specific models
    if "deepseek" in model_name.lower():
        prefix_token = '<｜fim▁begin｜>'
        middle_token = '<｜fim▁end｜>'
        suffix_token = '<｜fim▁hole｜>'
    elif "qwen1.5" in model_name.lower():
        prefix_token = '<fim_prefix>'
        middle_token = '<fim_middle>'
        suffix_token = '<fim_suffix>'
    elif "qwen" in model_name.lower():
        prefix_token = '<|fim_prefix|>'
        middle_token = '<|fim_middle|>'
        suffix_token = '<|fim_suffix|>'
    elif "seed-coder" in model_name.lower():
        prefix_token = '<[fim-prefix]>'
        middle_token = '<[fim-middle]>'
        suffix_token = '<[fim-suffix]>'
    else:
        prefix_token = '<fim_prefix>'
        middle_token = '<fim_middle>'
        suffix_token = '<fim_suffix>'
        
    if model_type == "codelm_leftright_context":
        left_cxt_truncated = tokenizer.decode(tokenizer.encode(left_cxt)[-(args.max_seq_length - args.gen_length - args.right_context_length):])
        right_cxt_truncated = tokenizer.decode(tokenizer.encode(right_cxt)[:args.right_context_length])
        prompt = f'{prefix_token}{left_cxt_truncated}{suffix_token}{right_cxt_truncated}{middle_token}'
    elif model_type == "codelm_leftright_context_spm":
        left_cxt_truncated = tokenizer.decode(tokenizer.encode(left_cxt)[-(args.max_seq_length - args.gen_length - args.right_context_length):])
        right_cxt_truncated = tokenizer.decode(tokenizer.encode(right_cxt)[:args.right_context_length])
        prompt = f'{suffix_token}{right_cxt_truncated}{prefix_token}{left_cxt_truncated}{middle_token}'
    elif model_type == "codelm_right_cfc_left":
        assert crossfile_cxt is not None
        left_cxt_truncated = tokenizer.decode(tokenizer.encode(left_cxt)[-(args.max_seq_length - args.gen_length - args.right_context_length - args.cfc_seq_length):])
        right_cxt_truncated = tokenizer.decode(tokenizer.encode(right_cxt)[:args.right_context_length])
        crossfile_cxt_truncated = tokenizer.decode(tokenizer.encode('\n\n' + crossfile_cxt)[:args.cfc_seq_length])
        prompt = f'{prefix_token}{left_cxt_truncated}{suffix_token}{right_cxt_truncated}{crossfile_cxt_truncated}{middle_token}'
    elif model_type == "codelm_right_cfc_left_spm":
        assert crossfile_cxt is not None
        left_cxt_truncated = tokenizer.decode(tokenizer.encode(left_cxt)[-(args.max_seq_length - args.gen_length - args.right_context_length - args.cfc_seq_length):])
        right_cxt_truncated = tokenizer.decode(tokenizer.encode(right_cxt)[:args.right_context_length])
        crossfile_cxt_truncated = tokenizer.decode(tokenizer.encode('\n\n' + crossfile_cxt)[:args.cfc_seq_length])
        prompt = f'{suffix_token}{right_cxt_truncated}{crossfile_cxt_truncated}{prefix_token}{left_cxt_truncated}{middle_token}'
    else:
        raise NotImplementedError
    return prompt

Then you can run the original script run_cceval.sh by setting

--model_type codelm_right_cfc_left_spm

and other parameters according to our paper.

[kotlyar-shapirov]

Thanks for your response!

According to the starcoder2 paper (https://arxiv.org/html/2402.19173v1) they format the cross-file context prompt the following way:
<repo_name>reponame<file_sep>filepath0\ncode0<file_sep><fim_prefix>filepath1\ncode1_pre<fim_suffix>code1_suf<fim_middle>code1_mid<file_sep> ...<|endoftext|>

As I understood from the provided example - you prepend the cross-file context to prefix for all models - this might provide lower completion quality for starcoder2

[Nes-tle]

According to the starcoder2 paper, the format contains (possibly randomly ordered) files in a repo, with some of them being in the PSM format. I would admit that the current arrangement of FIM tests might not favor some of the baselines, but it would be beyond the scope of this paper to compare the performance under all possible formats.

As FIM tests are highly flexible, we choose one of the format for each baseline model, which we believe "agrees with" part of the current research works and won't intentionally harm the fairness. The result is used to demostrating the potential of practical code completion, (e.g., the capability when used as an IDE plugin), rather than benchmark optimization.

[kotlyar-shapirov]

Is it possible to share how the crossfile_cxt was formatted in the training phase? - couldn't find it in the technical report.
(meaning - do you include the path to the file in the crossfile_ctx, is there any special separators between code snippets etc.)

In the mentioned link repoeval/cceval format is following:

# Here are some relevant code fragments from other files of the repo:

# the below code fragment can be found in:
# path/to/the/code/snippet1
#         some-code1
#         some-code1
#         some-code1

# the below code fragment can be found in:
# path/to/the/code/snippet2
#         some-code2
#         some-code2
#         some-code2

Did you too format the crossfile_cxt via commenting and adding tabs to all code snippets?

If not - can you please share the formatting applied? It would be reasonable to expect better completions when formatting the input as it was done during the training.

[Nes-tle]

The format mentioned in the FIM section of Data Ingredient is what we applied during the entire training. We neither applied extra indicators nor put a specific "crossfile_cxt" in the training stages.

[kotlyar-shapirov]

Just to clarify - the FIM training step was done exclusively on file-level code examples, never including repo-level?

I'm asking because the section 2.2.1 of technical report mentiones both and I assumed same dataset was used for FIM step:

This two-tier strategy yielded two variants of the code corpus: the file-level variant offered flexibility for training with short context windows, while the repository-level variant preserved the project structure, enabling more coherent long-context learning.

[Nes-tle]

Oh, I see your point. For repo-level data, the FIM format was restricted to at most one file in each repo. To distinguish the files in a repo, the file path was added as a comment line in the begining of the corresponding code file.

[YDJSIR-NJU]

We are seeking clarification on whether it is necessary to convert the code in crossfile content to be commented. To be clear, which is the corrent way when adopting the LLM in IDE completion, A or B?
A:

# the below code fragment can be found in:
# path/to/the/code/snippet1
#         some-code1-line1
#         some-code1-line2
#         some-code1-line3

B:

# the below code fragment can be found in:
# path/to/the/code/snippet1
         some-code1
         some-code1
         some-code1

Any insights into the pros and cons of each would be valuable. Your reply will be appreciated.

[Nes-tle]

The most suggested way is to SFT the model with customized dataset such that it will closely follow your idea. For a quick start, format B is preferred.