Skip to content

Commit

Permalink
feat(config): Add support for project instruction file
Browse files Browse the repository at this point in the history
  • Loading branch information
yamadashy committed Aug 11, 2024
1 parent 0f81046 commit 99d9f1c
Show file tree
Hide file tree
Showing 8 changed files with 148 additions and 19 deletions.
40 changes: 40 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,7 @@ Here's an explanation of the configuration options:
|`output.filePath`| The name of the output file | `"repopack-output.txt"` |
|`output.style`| The style of the output (`plain`, `xml`) |`"plain"`|
|`output.headerText`| Custom text to include in the file header |`null`|
|`output.instructionFilePath`| Path to a file containing detailed project instructions |`null`|
|`output.removeComments`| Whether to remove comments from supported file types | `false` |
|`output.removeEmptyLines`| Whether to remove empty lines from the output | `false` |
|`output.showLineNumbers`| Whether to add line numbers to each line in the output |`false`|
Expand Down Expand Up @@ -284,6 +285,45 @@ Example configuration:
}
```

### Using `instructionFilePath` Option
The `instructionFilePath` option allows you to specify a separate file containing detailed instructions or context about your project. This can be useful for providing AI systems with more comprehensive information about your codebase, coding standards, or specific areas to focus on during analysis.

Here's an example of how you might use this feature:

1. Create a file named `repopack-instruction.md` in your project root:

```markdown
# Project Analysis Instructions

## Code Structure
- The `src/core` directory contains the main business logic
- Unit tests are located in `tests/unit`
- Integration tests are in `tests/integration`

## Coding Standards
- We follow the Airbnb JavaScript Style Guide
- All new features should have corresponding unit tests
- Aim for 80% code coverage on new code

## Focus Areas
- Performance optimization in the data processing module
- Improving error handling and logging
- Refactoring the authentication system for better scalability
```

2. In your `repopack.config.json`, add the `instructionFilePath` option:

```json5
{
"output": {
"instructionFilePath": "repopack-instruction.md",
// other options...
}
}
```

When Repopack generates the output, it will include the contents of `repopack-instruction.md` in a dedicated section. This allows AI systems to understand the specific context and requirements of your project, potentially leading to more relevant and tailored analysis or suggestions.

### Include and Ignore
#### Include Patterns
Repopack now supports specifying files to include using glob patterns. This allows for more flexible and powerful file selection:
Expand Down
49 changes: 49 additions & 0 deletions repopack-instruction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Repopack Project Instructions

## Project Overview
Repopack is a tool designed to pack an entire repository into a single, AI-friendly file. It's primarily used to prepare codebases for analysis by Large Language Models (LLMs) such as Claude, ChatGPT, and Gemini.

## Key Design Principles
1. **Simplicity**: The tool should be easy to use with minimal configuration.
2. **Flexibility**: Users should be able to customize what's included or excluded in the output.
3. **Security**: The tool should help prevent sensitive information from being accidentally included.
4. **AI-Optimized**: The output should be formatted in a way that's easy for AI models to process and understand.

## Project Structure
- `src/`: Contains the source code
- `cli/`: Command-line interface related code
- `config/`: Configuration handling
- `core/`: Core functionality (file processing, output generation, etc.)
- `shared/`: Shared utilities
- `tests/`: Contains all test files
- `bin/`: Contains the executable scripts

## Key Components
1. **CLI (Command Line Interface)**: Handles user input and initiates the packing process.
2. **Config Loader**: Manages loading and merging of configuration from various sources.
3. **File Searcher**: Responsible for finding and filtering files based on include/ignore patterns.
4. **File Sanitizer**: Processes individual files, handling encoding and optionally removing comments.
5. **Output Generator**: Creates the final packed output in either plain text or XML format.
6. **Security Checker**: Uses Secretlint to detect potentially sensitive information in files.

## Important Rules and Conventions
1. Use TypeScript for all new code.
2. Follow the existing code style and use Prettier for formatting.
3. Write unit tests for all new functionality.
4. Use async/await for asynchronous operations.
5. Use meaningful variable and function names that describe their purpose.
6. Keep functions small and focused on a single task.
7. Use dependency injection to improve testability.
8. Handle errors gracefully and provide meaningful error messages.

## Performance Considerations
- Be mindful of memory usage when processing large repositories.
- Use streams or chunked processing for large files when possible.
- Avoid unnecessary file system operations.

## Security Considerations
- Never include sensitive information (like API keys) in the code.
- Use Secretlint rules to detect potential security issues.
- Be cautious when processing user-provided file paths to prevent directory traversal attacks.

When analyzing or suggesting changes to the codebase, please keep these instructions in mind. Focus on maintaining the project's simplicity and flexibility while improving its functionality and performance.
1 change: 1 addition & 0 deletions repopack.config.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"filePath": "repopack-output.xml",
"style": "xml",
"headerText": "This repository contains the source code for the Repopack tool.\nRepopack is designed to pack repository contents into a single file,\nmaking it easier for AI systems to analyze and process the codebase.\n\nKey Features:\n- Configurable ignore patterns\n- Custom header text support\n- Efficient file processing and packing\n\nPlease refer to the README.md file for more detailed information on usage and configuration.\n",
"instructionFilePath": "repopack-instruction.md",
"removeComments": false,
"removeEmptyLines": false,
"topFilesLength": 5,
Expand Down
2 changes: 2 additions & 0 deletions src/config/configTypes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ interface RepopackConfigBase {
filePath?: string;
style?: RepopackOutputStyle;
headerText?: string;
instructionFilePath?: string;
removeComments?: boolean;
removeEmptyLines?: boolean;
topFilesLength?: number;
Expand All @@ -23,6 +24,7 @@ export type RepopackConfigDefault = RepopackConfigBase & {
filePath: string;
style: RepopackOutputStyle;
headerText?: string;
instructionFilePath?: string;
removeComments: boolean;
removeEmptyLines: boolean;
topFilesLength: number;
Expand Down
32 changes: 24 additions & 8 deletions src/core/output/outputGenerator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import path from 'node:path';
import { RepopackConfigMerged } from '../../config/configTypes.js';
import { SanitizedFile } from '../file/fileSanitizer.js';
import { generateTreeString } from '../file/fileTreeGenerator.js';
import { RepopackError } from '../../shared/errorHandler.js';
import { generateXmlStyle } from './xmlStyleGenerator.js';
import { generatePlainStyle } from './plainStyleGenerator.js';
import { OutputGeneratorContext } from './outputGeneratorTypes.js';
Expand All @@ -14,7 +15,7 @@ export const generateOutput = async (
allFilePaths: string[],
fsModule = fs,
): Promise<void> => {
const outputGeneratorContext = buildOutputGeneratorContext(config, allFilePaths, sanitizedFiles);
const outputGeneratorContext = await buildOutputGeneratorContext(rootDir, config, allFilePaths, sanitizedFiles);

let output: string;
switch (config.output.style) {
Expand All @@ -29,13 +30,28 @@ export const generateOutput = async (
await fsModule.writeFile(outputPath, output);
};

export const buildOutputGeneratorContext = (
export const buildOutputGeneratorContext = async (
rootDir: string,
config: RepopackConfigMerged,
allFilePaths: string[],
sanitizedFiles: SanitizedFile[],
): OutputGeneratorContext => ({
generationDate: new Date().toISOString(),
treeString: generateTreeString(allFilePaths),
sanitizedFiles,
config,
});
): Promise<OutputGeneratorContext> => {
let additionalInstruction = '';

if (config.output.instructionFilePath) {
const instructionPath = path.resolve(rootDir, config.output.instructionFilePath);
try {
additionalInstruction = await fs.readFile(instructionPath, 'utf-8');
} catch {
throw new RepopackError(`Instruction file not found at ${instructionPath}`);
}
}

return {
generationDate: new Date().toISOString(),
treeString: generateTreeString(allFilePaths),
sanitizedFiles,
config,
projectInstruction: additionalInstruction,
};
};
1 change: 1 addition & 0 deletions src/core/output/outputGeneratorTypes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ export interface OutputGeneratorContext {
treeString: string;
sanitizedFiles: SanitizedFile[];
config: RepopackConfigMerged;
projectInstruction: string;
}
20 changes: 15 additions & 5 deletions src/core/output/plainStyleGenerator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ const PLAIN_SEPARATOR = '='.repeat(16);
const PLAIN_LONG_SEPARATOR = '='.repeat(64);

export const generatePlainStyle = (data: OutputGeneratorContext): string => {
const { generationDate, treeString, sanitizedFiles, config } = data;
const { generationDate, treeString, sanitizedFiles, config, projectInstruction } = data;

let output = `${PLAIN_LONG_SEPARATOR}
Repopack Output File
Expand Down Expand Up @@ -32,12 +32,14 @@ The content is organized as follows:
Usage Guidelines:
-----------------
1. This file should be treated as read-only. Any changes should be made to the
- This file should be treated as read-only. Any changes should be made to the
original repository files, not this packed version.
2. When processing this file, use the separators and "File:" markers to
- When processing this file, use the separators and "File:" markers to
distinguish between different files in the repository.
3. Be aware that this file may contain sensitive information. Handle it with
- Be aware that this file may contain sensitive information. Handle it with
the same level of security as you would the original repository.
${config.output.headerText ? '- Pay special attention to the Project Summary. These contain important context and guidelines specific to this project.' : ''}
${projectInstruction ? '- Pay special attention to the Project Instruction. These contain important context and guidelines specific to this project.' : ''}
Notes:
------
Expand All @@ -54,10 +56,18 @@ For more information about Repopack, visit: https://github.com/yamadashy/repopac
`;

if (config.output.headerText) {
output += `Additional User-Provided Header:
output += `Project Summary:
--------------------------------
${config.output.headerText}
`;
}

if (projectInstruction) {
output += `Project Instruction:
---------------------------------------
${projectInstruction}
`;
}

Expand Down
22 changes: 16 additions & 6 deletions src/core/output/xmlStyleGenerator.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { OutputGeneratorContext } from './outputGeneratorTypes.js';

export const generateXmlStyle = (data: OutputGeneratorContext): string => {
const { generationDate, treeString, sanitizedFiles, config } = data;
const { generationDate, treeString, sanitizedFiles, config, projectInstruction } = data;

let xml = `<summary>
Expand All @@ -26,12 +26,14 @@ The content is organized as follows:
</file_format>
<usage_guidelines>
1. This file should be treated as read-only. Any changes should be made to the
- This file should be treated as read-only. Any changes should be made to the
original repository files, not this packed version.
2. When processing this file, use the file path attributes to distinguish
- When processing this file, use the file path attributes to distinguish
between different files in the repository.
3. Be aware that this file may contain sensitive information. Handle it with
- Be aware that this file may contain sensitive information. Handle it with
the same level of security as you would the original repository.
${config.output.headerText ? '- Pay special attention to the project_summary. These contain important context and guidelines specific to this project.' : ''}
${projectInstruction ? '- Pay special attention to the project_instruction. These contain important context and guidelines specific to this project.' : ''}
</usage_guidelines>
<notes>
Expand All @@ -49,9 +51,17 @@ For more information about Repopack, visit: https://github.com/yamadashy/repopac

if (config.output.headerText) {
xml += `
<user_provided_header>
<project_summary>
${config.output.headerText}
</user_provided_header>
</project_summary>
`;
}

if (projectInstruction) {
xml += `
<project_instruction>
${projectInstruction}
</project_instruction>
`;
}

Expand Down

0 comments on commit 99d9f1c

Please sign in to comment.