File Concatenation Tool

A powerful tool for merging multiple source files into a single output file with configurable options. Available as both PowerShell script (_concat.ps1) and Python script (concat.py).

📋 Table of Contents

• Features
• Installation
• Usage
  • PowerShell Version
  • Python Version
• Configuration
• Examples
• PowerShell vs. Python Comparison
• License

✨ Features

Common Features (both versions)

• Flexible Source Definition: Process multiple directories and files simultaneously
• Pattern-Based Filtering: Filter files by patterns (e.g., *.ts, *.py)
• Recursive Search: Optionally search recursively through subdirectories
• Exclusion Patterns: Exclude specific files/directories (e.g., node_modules, __pycache__)
• File Headers: Optionally insert headers with filename before each file content
• Duplicate Prevention: Prevents processing the same file twice
• Path Options: Choose between relative and absolute paths
• Whitespace Management: Optionally insert blank lines between files
• UTF-8 Encoding: Consistent character encoding
• Progress Feedback: Real-time console output showing processed files

Additional Features (Python Version)

• JSON Configuration: External configuration files for reusable setups
• Command-Line Arguments: Flexible parameter passing without editing the script
• Enhanced Error Handling: Improved error handling and logging
• Cross-Platform: Runs on Windows, Linux, and macOS
• Return Codes: Exit codes for CI/CD integration
• Type Hints: Fully typed Python code for better IDE support

🚀 Installation

PowerShell Version

No installation required - simply use the _concat.ps1 script.

Requirements:

• Windows PowerShell 5.1 or higher
• PowerShell Core 7+ (optional, for cross-platform usage)

Python Version

Requirements:

• Python 3.7 or higher

Installation:

# Make script executable (Linux/macOS)
chmod +x concat.py

# Or run directly with Python
python concat.py --help

📖 Usage

PowerShell Version

Basic Usage:

# Run the script
.\_concat.ps1

# With specific path
cd C:\MyProject
C:\DEV\pyConcat\_concat.ps1

Configuration: Edit the configuration variables directly in the script (lines 5-31):

# Output file
$outfile = "_concat_src.txt"

# Enable headers
$use_headers = $true

# Use absolute paths
$use_absolute_paths = $false

# Add blank lines between files
$add_blank_lines = $true

# Source definitions
$sources = @(
    @{ Path = "src"; 
       Recursive = $true; 
       Patterns = "*.js *.jsx"; 
       Exclusions = "node_modules dist" }
)

Python Version

Basic Usage:

# With default configuration
python concat.py

# Show help
python concat.py --help

Command-Line Options:

# Custom output file
python concat.py -o my_output.txt

# External configuration
python concat.py -c config.json

# Without headers
python concat.py --no-headers

# Use absolute paths
python concat.py --absolute-paths

# Without blank lines
python concat.py --no-blank-lines

# Custom encoding
python concat.py --encoding utf-16

Combined Options:

python concat.py -c myconfig.json -o output.txt --absolute-paths

⚙️ Configuration

PowerShell Configuration (in script)

Configure variables at the beginning of _concat.ps1:

Variable	Type	Default	Description
$outfile	String	"_concat_src.txt"	Output file name
$use_headers	Boolean	$true	Insert header before each file
$header_start	String	"/*-----...*/	Header start line
$header_file	String	"File: {0}"	Format for filename in header
$header_end	String	"-----...*/	Header end line
$use_absolute_paths	Boolean	$false	Use absolute instead of relative paths
$add_blank_lines	Boolean	$true	Insert blank lines between files
$sources	Array	See script	Array of source definitions

Python Configuration (JSON file)

Create a config.json:

{
  "outfile": "_concat_src.txt",
  "use_headers": true,
  "header_start": "/*------------------------------",
  "header_file": "File: {0}",
  "header_end": "------------------------------*/",
  "use_absolute_paths": false,
  "add_blank_lines": true,
  "encoding": "utf-8",
  "sources": [
    {
      "path": "src",
      "recursive": true,
      "patterns": ["*.js", "*.jsx", "*.ts", "*.tsx"],
      "exclusions": ["node_modules", "dist", "*.test.js"]
    },
    {
      "path": "config",
      "recursive": false,
      "patterns": ["*.json", "*.yaml"],
      "exclusions": ["package-lock.json"]
    }
  ]
}

Then use it:

python concat.py -c config.json

📚 Examples

Example 1: Merge TypeScript Project

PowerShell:

$sources = @(
    @{ Path = "src"; 
       Recursive = $true; 
       Patterns = "*.ts *.tsx"; 
       Exclusions = "node_modules dist *.test.ts *.test.tsx" }
)

Python:

python concat.py -c typescript-config.json

{
  "outfile": "typescript_all.txt",
  "sources": [
    {
      "path": "src",
      "recursive": true,
      "patterns": ["*.ts", "*.tsx"],
      "exclusions": ["node_modules", "dist", "*.test.ts", "*.test.tsx"]
    }
  ]
}

Example 2: Merge Python Backend

PowerShell:

$sources = @(
    @{ Path = "backend"; 
       Recursive = $true; 
       Patterns = "*.py"; 
       Exclusions = "__pycache__ venv *.pyc tests" }
)

Python:

python concat.py -c python-backend.json -o backend_all.py

Example 3: Multiple Sources

Python:

{
  "outfile": "full_project.txt",
  "sources": [
    {
      "path": "config",
      "recursive": false,
      "patterns": ["*.ini", "*.yaml", "*.json"],
      "exclusions": ["node_modules"]
    },
    {
      "path": "frontend/src",
      "recursive": true,
      "patterns": ["*.ts", "*.tsx"],
      "exclusions": ["*.test.ts", "dist"]
    },
    {
      "path": "backend",
      "recursive": true,
      "patterns": ["*.py"],
      "exclusions": ["__pycache__", "venv"]
    },
    {
      "path": "docs",
      "recursive": true,
      "patterns": ["*.md"],
      "exclusions": ["node_modules"]
    }
  ]
}

Example 4: Single Files

PowerShell:

$sources = @(
    @{ Path = "package.json"; Recursive = $false; Patterns = ""; Exclusions = "" },
    @{ Path = "tsconfig.json"; Recursive = $false; Patterns = ""; Exclusions = "" },
    @{ Path = "README.md"; Recursive = $false; Patterns = ""; Exclusions = "" }
)

Python:

{
  "sources": [
    {"path": "package.json"},
    {"path": "tsconfig.json"},
    {"path": "README.md"}
  ]
}

Example 5: AI Context Preparation

Perfect for preparing code context for AI tools like Claude or ChatGPT:

Configuration for Full Stack App:

{
  "outfile": "ai_context.txt",
  "use_headers": true,
  "sources": [
    {
      "path": ".",
      "recursive": false,
      "patterns": ["package.json", "tsconfig.json", "README.md"],
      "exclusions": []
    },
    {
      "path": "frontend/src",
      "recursive": true,
      "patterns": ["*.ts", "*.tsx"],
      "exclusions": ["*.test.ts", "node_modules", "dist"]
    },
    {
      "path": "backend",
      "recursive": true,
      "patterns": ["*.py"],
      "exclusions": ["__pycache__", "venv", "*.pyc", "tests"]
    }
  ]
}

🔄 PowerShell vs. Python Comparison

Feature	PowerShell	Python
Platform	Windows (primary)	Cross-Platform
Configuration	In script	JSON file + CLI
Command-Line Options	❌	✅
External Configuration	❌	✅
Return Codes	Limited	✅
Error Handling	Basic	Enhanced
Speed	Fast	Fast
Dependencies	None	Python 3.7+
Easy Modifications	Edit script	CLI parameters
Type Safety	No	Yes (with hints)

When to Use PowerShell:

• Windows-only environment
• Simple use cases
• No external dependencies desired
• Part of larger PowerShell toolchain
• Quick ad-hoc concatenations

When to Use Python:

• Cross-platform required
• CI/CD integration
• Complex configurations
• Reusable configuration files
• Command-line flexibility
• Type safety preferred

📝 Output Format

Both scripts produce the same output structure:

/*------------------------------
File: src/app.ts
------------------------------*/
// Content of app.ts
console.log("Hello World");


/*------------------------------
File: src/utils.ts
------------------------------*/
// Content of utils.ts
export const util = () => {};

🐛 Troubleshooting

PowerShell

Problem: "Execution of scripts is disabled"

# Solution: Adjust execution policy
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Problem: No files found

• Check if paths are correct
• Check if patterns are space-separated
• Check exclusion patterns
• Verify working directory

Problem: Special characters in output

• Script uses UTF-8 encoding
• Check console encoding: [Console]::OutputEncoding

Python

Problem: "No files were processed"

# Debug mode (verbose output)
python -v concat.py

# Check current directory
python -c "import os; print(os.getcwd())"

Problem: Encoding errors

# Try different encoding
python concat.py --encoding latin-1

# Or in config
{
  "encoding": "latin-1"
}

Problem: Permission denied

# Linux/macOS: Make script executable
chmod +x concat.py
./concat.py

# Or run with Python
python concat.py

Problem: Module not found

# Check Python version
python --version

# Ensure Python 3.7+
python3 concat.py

🎯 Use Cases

1. AI Context Preparation

Prepare comprehensive code context for AI assistants (Claude, ChatGPT, etc.)

2. Code Review

Create a single file for easier code review of multiple files

3. Documentation

Generate combined documentation from multiple markdown files

4. Backup

Create consolidated backups of configuration files

5. Analysis

Combine logs or data files for analysis

6. Archiving

Create single-file archives of project snapshots

🔒 Security Considerations

• ⚠️ Be careful with sensitive data: The output file will contain all source code/data
• ⚠️ Exclusion patterns: Always exclude sensitive files (.env, secrets.json, etc.)
• ⚠️ File permissions: Output file has same permissions as the script
• ✅ Safe defaults: Default exclusions help prevent common mistakes

🚀 Performance

Typical Performance

• PowerShell: ~5000 files/minute (varies by file size)
• Python: ~8000 files/minute (varies by file size)

Optimization Tips

1. Use specific patterns: *.ts instead of *
2. Add proper exclusions: Exclude large directories early
3. Avoid nested recursion: Be specific with paths
4. Use SSD storage: Faster disk I/O improves performance

📊 Statistics

Both scripts provide statistics after execution:

Processing source 1: "frontend" (recursive|*.ts *.tsx|Exclusions: node_modules dist)
  src/app.ts
  src/components/Header.tsx
  src/utils/helpers.ts
(Files: 3)

Processing source 2: "backend" (recursive|*.py|Exclusions: __pycache__ venv)
  main.py
  models/user.py
(Files: 2)

Concatenation complete. 5 files written to _concat_src.txt

Press Enter to continue...

📄 License

MIT License - See LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request or open an Issue.

Development

# Clone repository
git clone <repo-url>

# For Python development
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows

# Run tests (if implemented)
python -m pytest tests/

📧 Support

For questions or issues, please open an issue in the repository.

---

Note: Both scripts are functionally equivalent. Choose the version that best fits your workflow!

Pro Tip: For AI context preparation, combine with token counting tools to stay within model limits.