PyC's architecture is designed to be modular and extensible, following a classic compiler structure while incorporating experimental features for AI and GPU acceleration. The system is built primarily in C, with components in C++ and CUDA, leveraging LLVM for code generation and optimization.
Component | Purpose |
---|---|
Frontend | Tokenizes and parses Python-like source code into Abstract Syntax Trees (ASTs). |
IR System | Generates optimized Intermediate Representation (IR) from the AST. |
Backend | Generates machine code or accelerated bytecode from IR, with support for CPU and GPU. |
AI Graph Compiler | Optimizes computational graphs for tensor workflows. |
Memory Planner | Dynamically allocates and minimizes tensor memory footprints. |
Custom Kernel Loader | Integrates user-written .cu or .cl files into compiled pipelines. |
CLI Driver | Exposes compilation commands to users (pyc build , pyc optimize , etc.). |
- Lexer (
lexer.c
): Tokenizes input, handling Python-like syntax and indentation. - Parser (
parser.c
): Builds AST from tokens, supporting expressions, assignments, and if statements. - Frontend Utilities (
frontend.c
): Manages source code loading and preprocessing.
- Symbol Table Manager (
symbol_table.c
): Tracks variables and scopes, preparing for IR generation.
- IR Generator (
ir_generator.c
,codegen.c
,IR.c
): Converts AST to LLVM IR, supporting basic operations and conditionals.
- Optimizer (
backend.c
,codegen.c
): Applies LLVM optimization passes for efficient code generation.
- Backend Manager (
backend.c
): Handles JIT compilation and object file generation with multithreading. - CUDA Integration (
kernel.cu
,matrix_mult.cu
): Experimental kernels for tokenization and matrix operations.
- Graph Compiler (
AI/graph_compiler.c
): Optimizes tensor operations and computational graphs. - Memory Planner (
AI/memory_planner.c
): Manages dynamic memory allocation for tensors. - Optimizer (
AI/optimizer.c
): Applies runtime strategies for AI models. - Visualizer (
AI/visualizer.c
): Generates visual representations of computational graphs.
- Main Entry Point (
main.c
): Orchestrates the compiler pipeline and handles CLI commands.
- Test Suite (
test_parser.c
): Validates parser functionality with basic expressions and statements.
- Modularity: Independent components allow for easy extension and experimentation.
- LLVM Integration Ensures portability and optimized code generation.
- Multithreading: Accelerates backend compilation using multiple CPU cores.
- GPU Acceleration: Experimental use of CUDA for compiler tasks like tokenization.
- AI Focus: Specialized modules for optimizing AI workflows, including graph compilation and memory planning.
- CLI-First Design: Fully operable from the terminal with simple, intuitive commands.
- Input Handling: Source code is loaded and preprocessed by the frontend.
- Tokenization and Parsing: Lexer and parser transform code into an AST.
- Symbol Resolution: Symbol table manages variable scopes for IR generation.
- IR Generation: AST is converted to LLVM IR, incorporating optimizations.
- Optimization: LLVM passes enhance IR efficiency.
- Code Generation: Backend compiles IR to machine code or faster bytecode.
- AI Workflow: AI modules optimize computational graphs and manage tensor memory.
- Custom Kernels: Users can integrate custom CUDA/OpenCL kernels for acceleration.
- CLI Commands: Users interact with the compiler through a simple CLI interface.
This architecture ensures PyC is functional, extensible, and aligned with its mission to speed up AI workflows through optimized compilation and GPU integration.