[TEST PR] Script for CI cosine similarity comparisons#277
Open
leesharkey wants to merge 44 commits intomainfrom
Open
[TEST PR] Script for CI cosine similarity comparisons#277leesharkey wants to merge 44 commits intomainfrom
leesharkey wants to merge 44 commits intomainfrom
Conversation
Added two documentation files to help AI assistants work effectively with the SPD codebase: - CLAUDE_COMPREHENSIVE.md: Complete reference guide covering development philosophy, coding standards, architecture patterns, workflows, and collaboration practices - CLAUDE_CHECKLIST.md: Pre-submission checklist for verifying code changes meet SPD standards before committing These documents ensure consistent code quality and help future AI assistants understand project conventions, reducing onboarding time and maintaining codebase consistency. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added two checklist items to prevent future AI assistants from forgetting important steps: - "Checked existing patterns" item to ensure new files follow existing conventions - "Restarted checklist after any changes" with explicit STOP instruction to prevent incomplete verification Also fixed references from "dev branch" to "main branch" throughout both documentation files, as the repository uses main as the primary development branch. These changes address feedback from PR review process where these steps were accidentally omitted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Per CLAUDE_CHECKLIST.md, removed redundant comment that was obvious from the code itself. The line `alive_mask = mean_component_cis[layer_name] > self.mean_ci_threshold` is self-explanatory. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enhances the model comparison script with CI-based cosine similarity metrics for
more meaningful analysis of learned component alignment between SPD model runs.
Key Changes:
Metric Improvements:
filtering criterion
range)
Code Quality:
error handling
Configuration Updates:
Related Issue
N/A - Enhancement to post-hoc analysis tooling
Motivation and Context
The CI cosine similarity metrics provide better insight into how learned components
align between different model runs. Mean CI is a more meaningful measure of
component importance than activation density, as it directly quantifies each
component's causal contribution to model outputs.
This complements the existing geometric similarity metrics (which compare component
subspace geometry) with a functional similarity metric (which compares component
usage patterns on actual data).
How Has This Been Tested?
Does this PR introduce a breaking change?
Minor breaking change in compare_models.py:
parameter name
SPD training pipeline