md-04 - intelligent index creation with automatic value inference

[azzazzel]

Problem

The index creation process was complex and required users to manually specify many configuration parameters. It also required users to know the interdependencies between different flags, making it difficult for new users to get started quickly. There was no way to discover available embedding models, and users had to know the exact model names and their requirements. Additionally, there was no global option to skip confirmation prompts, making automation and scripting cumbersome.

Solution

This PR introduces intelligent index creation with automatic value inference and comprehensive model management capabilities. The changes include:

• Smart Value Inference: Automatically infers missing index creation parameters based on provided values and available models from the API
• Model Discovery: Added pc models command to list available embedding models with their specifications and requirements
• Intelligent Defaults: Provides sensible defaults for common configurations (AWS us-east-1, cosine metric, etc.) while allowing full customization
• Model-Based Configuration: Automatically configures index parameters based on selected embedding models, including dimensions, vector types, and field mappings
• Global --assume-yes Flag: Added -y/--assume-yes global flag to skip all confirmation prompts for automation and scripting
• Enhanced Validation: Improved validation with better error messages and support for inferred values
• Caching System: Implemented local caching for model data to improve performance and reduce API calls
• Simplified Command Structure: Streamlined index creation with clearer flag organization and better examples

The implementation maintains full backward compatibility while significantly improving the user experience for both beginners and advanced users.

Type of Change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infrastructure change (CI configs, etc)
• Non-code change (docs, etc)
• None of the above: (explain here)

Test Plan

1. Model Discovery Testing: Test the new models command:

   # List available models
   pc models
   
   # List models with JSON output
   pc models --json
   
   # Force refresh from API
   pc models --no-cache

2. Intelligent Index Creation Testing: Test automatic value inference:

   # Minimal serverless index (should infer defaults)
   pc index create my-index
   
   # With model specification (should infer model-specific settings)
   pc index create my-index --model llama-text-embed-v2
   
   # With default model aliases
   pc index create my-index --model dense
   pc index create my-index --model sparse
   
   # Pod index with minimal config (should infer defaults)
   pc index create my-pod --pod

3. Global Assume-Yes Flag Testing: Test automation-friendly behavior:

   # Create index without confirmation prompts
   pc index create my-index -y
   
   # Delete index without confirmation
   pc index delete my-index -y
   
   # Test with other commands that have confirmations
   pc project delete <project> -y

4. Validation Testing: Test improved validation and error handling:

   # Test invalid model names
   pc index create test --model invalid-model
   
   # Test conflicting configurations
   pc index create test --pod --serverless

5. Backward Compatibility Testing: Ensure existing commands still work:

   # Test explicit configuration (should work as before)
   pc index create my-index --dimension 1536 --metric cosine --cloud aws --region us-east-1

This PR builds upon the previous #36 style improvements and contains all changes in that branch as well

[jhamon]

There are some interesting ideas in here, but I'm sorry to say this is not ready to be reviewed in its current state.

Each of your 8 bullet points is a different feature and in order to give them a proper review they should be in small, focused PRs that deal with each feature in isolation. It's not feasible for anyone to test a change and make sure all edge cases are properly handled when you have edits touching so many files at once in one diff. If we gave you feedback and you made changes, it would also be extremely cumbersome to re-review because of the massive volume of change.

More important than these logistics of large diffs, it's simply not productive to argue back and forth in github comments about areas of disagreement that are mixed in with less complex and less controversial work that could otherwise be reviewed and merged much more quickly. Bundling everything together creates unnecessary coupling when many of these ideas can stand on their own.

Things that seem totally non-controversial to me that you could break out:

• Adding a models command

Probably also a good idea:

• Global --assume-yes flag. I'd like to see some specific examples of how this should be used, but flags that work everywhere are nice. Kind of like the idea behind a --json flag for getting machine-readable output.
• Presentational improvements are welcome, but we should take these changes in smaller pieces so we can review them carefully. Ideally these would only touch a few commands at a time so we can test them very thoroughly.

Things I fundamentally disagree with and don't think are a good fit:

• Too much cleverness with defaults. Defaults should be applied on the server side. If something is a good idea as a default, we should have consistency across all customer services by implementing it server-side. And if you bake something into a CLI client, that may clobber the server-side implementation of defaults.
• File caching. I haven't fully grappled with the context around why you introduced caching-related changes in this PR but the CLI should be as near to stateless as we can make it. Bad state stored in caches can be a huge footgun later on and is best avoided unless there's a really compelling reason why it's helpful and necessary.
• "Simplified Command Structure" We've already been having this argument on slack and so I won't rehash it here. We anticipate implementing service account access to various endpoints that will not be able to assume uniqueness by name. So it doesn't make sense to commit to that as a positional arg right now.

Product ideas that need more discussion, refinement, and prioritization before implementation:

• Basically everything else