v.0.8.1 Expanded data models and housekeeping

Update models.data to include more supported models and Improved the Model Registry Loading

v0.8.0 Hotfix model loading

Added DTO for model loading and ebedded json data for costing

0.7.8 HotFix

HotFix

v0.7.0 — CLI Alignment & Ecosystem Integration

🧠 TokenFlow.AI v0.7.0 — CLI Alignment & Ecosystem Integration

🚀 Highlights

TokenFlow.AI v0.7.0 completes the engine layer of the Flow.AI ecosystem, delivering a refined CLI, smarter cost estimation, and seamless Flow.AI.Core integration.
This release introduces full ecosystem alignment, improved build efficiency, and preserves 100% test coverage across all modules.

---

✨ What’s New

🧩 CLI v3.0 Alignment

• Unified CLI commands for token counting and cost analysis
• Dynamic model resolution through Flow.AI.Core registry
• Improved human-readable output with fallback model indicators (approx, shared)

💰 Enhanced Cost Estimator

• Integrates with Flow.AI.Core model registry for dynamic pricing
• Provides detailed cost breakdowns for prompt + completion usage
• Adds smart fallback resolution for missing models

⚙️ CI/CD Optimizations

• Added RUN_BENCHMARKS toggle to switch between Quick Mode (~4 min) and Full Mode (~15 min)
• Benchmark results now auto-compare against baseline with artifact upload
• Pipeline performance and stability significantly improved

📊 Developer Experience

• Maintains 100% line + branch coverage across all projects
• Expanded test suite with null-guard and fallback path coverage
• Updated documentation and examples through v0.7.0

---

🧾 Changelog Summary

• v0.6.1 → v0.6.2: Added Flow.AI.Core registry + enhanced CostEstimator
• v0.7.0: CLI alignment, optional benchmarks, and streamlined CI/CD
• All prior roadmap items for the TokenFlow engine layer are now complete ✅

---

🪪 Package Versions

Package	Version	Notes
TokenFlow.Core	0.7.0	Shared models + interfaces
TokenFlow.AI	0.7.0	Tokenization + cost engine
TokenFlow.Tokenizers	0.7.0	OpenAI / Claude / Approx providers
TokenFlow.Tools	0.7.0	CLI and benchmark utilities

---

⭐ Next Steps

• Begin development of PromptStream.AI (cockpit layer)
• Extend CLI interoperability across the Flow.AI suite
• Introduce Flow.AI shared configuration system

---

🧠 TokenFlow.AI v0.7.0 — a stable, production-ready release marking the completion of the core engine layer for the Flow.AI ecosystem.

If you find TokenFlow.AI useful, please ⭐ the repo to support continued open-source development!

TokenFlow.AI v0.4.0 — CLI v2.1 Complete

🚀 TokenFlow.AI v0.4.0 — CLI v2.1 Complete + 100 % Coverage

This release finalizes the TokenFlow CLI v2.1 milestone and brings full parity between CLI, registry, and documentation.

---

✨ Highlights

• CLI v2.1 finalized — adds --format, --input, --output, and --quiet options
• Unified structured output via OutputFormatter (Table / CSV / JSON)
• Polished quiet mode for automation and CI environments
• Full ModelRegistry integration with JSON configuration loading
• Developer documentation live on GitHub Pages

---

🧪 Quality & Coverage

• Achieved 100 % line and branch coverage across all projects
• Added [ExcludeFromCodeCoverage] helpers for JIT-optimized fallback methods
• Hardened defensive logic in all CLI commands

---

📘 Documentation

• Updated /docs/cli-usage.md and /docs/model-registry.md
• Rebuilt GitHub Pages docs automatically via docs.yml
• Refreshed README.md and CHANGELOG.md with new CLI v2.1 sections

---

🧱 Architecture

• Compatible with .NET Standard 2.0 and .NET 8.0
• Modular structure:

  src/
    TokenFlow.Core/
    TokenFlow.AI/
    TokenFlow.Tools/
    TokenFlow.Tools.Benchmarks/
  tests/
    TokenFlow.Core.Tests/
    TokenFlow.AI.Tests/
    TokenFlow.Tools.Tests/
  

• Unified build, test, and coverage pipeline via GitHub Actions

---

📦 NuGet Packages

Package	Description	Link
TokenFlow.AI	Core tokenization, cost estimation, chunking, and CLI utilities	NuGet.org
TokenFlow.Core	Shared interfaces, contracts, and models used by TokenFlow.AI	NuGet.org

---

⭐ If you find TokenFlow.AI useful, please star the repo on GitHub!
It helps others discover the project and supports ongoing development.

TokenFlow.AI v0.3.0 — Developer Preview

The first public release of TokenFlow.AI, the lightweight .NET library for tokenization, chunking, and cost estimation across modern LLMs like OpenAI GPT-4o, Anthropic Claude, and Azure OpenAI.

This release forms the foundation of the Flow.AI ecosystem, providing a tested, extensible, and developer-friendly toolkit for AI token analysis and cost tracking.

---

✨ Highlights

• 🧠 TokenFlowClient — unified entry point for token counting, chunking, and cost estimation
• 🧱 TokenChunker — smart text segmentation with token-based limits and optional overlap
• 💰 CostEstimator — model-aware cost calculations for input/output tokens
• 🧩 ModelRegistry — pre-configured model definitions (OpenAI & Anthropic families)
• 🔢 ApproxTokenizer — fast built-in GPT-style tokenizer for local testing and validation
• 🧪 Full test coverage — 100% line and branch coverage across all components
• ⚙️ CI/CD ready — GitHub Actions pipeline with Codecov integration
• 🌐 Dual targeting: .NET Standard 2.0 + .NET 8.0
• 🪶 Zero dependencies — small, fast, and portable

---

🧠 Quick Example

using TokenFlow.AI.Client;

var client = new TokenFlowClient("gpt-4o");
var result = client.AnalyzeText("TokenFlow.AI brings structure to prompt engineering.");

Console.WriteLine($"Model: {result.ModelId}");
Console.WriteLine($"Tokens: {result.TokenCount}");
Console.WriteLine($"Estimated cost: £{result.EstimatedCost:F4}");

---

🧭 Roadmap

🚀 Next Up

• Add TokenUsageTracker for cumulative cost tracking
• Implement ITokenizerFactory for dynamic tokenizer resolution
• Extend ModelRegistry to support JSON model configuration
• CLI utilities via TokenFlow.Tools
• Benchmark suite using BenchmarkDotNet

🌟 Future Goals

• Integration with PromptStream.AI and DataFlow.AI
• Support for tiktoken and Claude tokenizer backends
• Developer documentation and sample projects
• Public release under the Flow.AI brand umbrella

---

🧩 Part of the Flow.AI Ecosystem

Package	Purpose
🧠 TokenFlow.AI	Core tokenization, chunking & cost estimation
💬 PromptStream.AI	Prompt composition & validation
📊 DataFlow.AI	Data ingestion & structured streaming pipelines
🧩 ReasonFlow.AI	Logical reasoning & multi-step orchestration
🧬 ModelFlow.AI	Unified model abstraction & configuration registry
💭 ChatFlow.AI	Conversational orchestration & dialogue management

---

🪪 License

Distributed under the MIT License.
See LICENSE for details.

---

⭐ If you find TokenFlow.AI useful, please give the repository a star on GitHub!
It helps others discover the Flow.AI ecosystem and supports ongoing development.