Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLM analysis and combined Kicad > Netlist > SKIDL > LLM analysis logic #247

Draft
wants to merge 77 commits into
base: development
Choose a base branch
from

Conversation

shanemmattner
Copy link
Contributor

This PR includes:

  • SKIDL logic to analyze each subcircuit with calls to a LLM (either OpenRouter or local Ollama models)
  • kicad_skidl_llm.py script that combines all the logic of generating a netlist from a Kicad schematic, generating SKIDL code from a Kicad Netlist, and analyzing a SKIDL project with LLM calls. All of these functions are independent.
    • Includes features such as the ability to pass in a model as well as a path for your custom Kicad symbol libraries that are needed to parse

Here's a sample command where I point the script towards a sample project which has a part from a custom library. I have the script generate a netlist, then generate SKIDL files, then analyze each subcircuit with LLM calls

python3 kicad_skidl_llm.py --schematic ~/Desktop/example_kicad_project/example_kicad_project.kicad_sch --generate-netlist --generate-skidl --analyze --api-key $OPENROUTER_API_KEY --kicad-lib-paths /Users/shanemattner/Desktop/skip/electronics/PCB/pcb_libraries/kicad/ 

@devbisme the kicad_skidl_llm.py script has a lot of logic in it I wasn't sure how to integrate with the main SKIDL logic.

And one other question I had for you is that it seems like debug statements are being printed twice. Any idea what I might be doing wrong with the logging logic?

venvShanes-MacBook-Pro:skidl shanemattner$ python3 kicad_skidl_llm.py --schematic ~/Desktop/example_kicad_project/example_kicad_project.kicad_sch --generate-netlist --generate-skidl --analyze --api-key $OPENROUTER_API_KEY --kicad-lib-paths /Users/shanemattner/Desktop/skip/electronics/PCB/pcb_libraries/kicad/ 
[INFO] Step 1: Generating netlist from schematic...
[INFO] ✓ Generated netlist: example_kicad_project.net
[INFO] Added KiCad 8 library paths:
[INFO]   ✓ /Users/shanemattner/Desktop/skip/electronics/PCB/pcb_libraries/kicad
[INFO]     - skip_kicad_symbols.kicad_sym
[INFO] Step 2: Generating SKiDL project from netlist...
[INFO] ✓ Generated SKiDL project: example_kicad_project_SKIDL
[INFO] Step 3: Analyzing circuits...
[INFO] Found 5 circuits to analyze:
[INFO]   - resistor_divider1 (example_kicad_project_SKIDL/resistor_divider1.py:5)
[INFO]   - _3v3_regulator (example_kicad_project_SKIDL/_3v3_regulator.py:5)
[INFO]   - USB (example_kicad_project_SKIDL/USB.py:5)
[INFO]   - test_examples (example_kicad_project_SKIDL/test_examples.py:5)
[INFO]   - esp32s3mini1 (example_kicad_project_SKIDL/esp32s3mini1.py:6)
INFO: 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
INFO: Approximate cost for this query: $0.0060 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Approximate cost for this query: $0.0060 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Analysis completed in 10.45 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Analysis completed in 10.45 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
INFO: Approximate cost for this query: $0.0067 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Approximate cost for this query: $0.0067 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Analysis completed in 10.46 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Analysis completed in 10.46 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
INFO: Approximate cost for this query: $0.0088 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Approximate cost for this query: $0.0088 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Analysis completed in 10.32 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Analysis completed in 10.32 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
INFO: Approximate cost for this query: $0.0050 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Approximate cost for this query: $0.0050 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Analysis completed in 8.34 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Analysis completed in 8.34 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
INFO: Approximate cost for this query: $0.0055 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Approximate cost for this query: $0.0055 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Analysis completed in 10.51 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Analysis completed in 10.51 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Starting Circuit Analysis with google/gemini-flash-1.5 === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
Generating analysis... @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] HTTP Request: POST https://openrouter.ai/api/v1/chat/completions "HTTP/1.1 200 OK"
INFO: Approximate cost for this query: $0.0043 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Approximate cost for this query: $0.0043 @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
INFO: 
=== Analysis completed in 7.87 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] 
=== Analysis completed in 7.87 seconds === @ [/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:724=>/Users/shanemattner/Desktop/skidl/kicad_skidl_llm.py:385]
[INFO] Analysis Results:
[INFO] 
Subcircuit: top.USB0
[INFO] ✓ Analysis completed in 10.45 seconds
[INFO]   Tokens used: 1547
venvShanes-MacBook-Pro:skidl shanemattner$ 

@devbisme
Copy link
Owner

devbisme commented Feb 5, 2025

Hi, Shane. I'll try to go through this as time permits. Hope to have some ideas by tonight. Lots of stuff here! Thanks for all the comments!

@devbisme
Copy link
Owner

devbisme commented Feb 5, 2025

Some messages were appearing twice because the SKiDL logger objects were propagating their messages up to the simple logger you were using. I turned off propagation for the SKiDL loggers so now the messages appear only once.

@devbisme
Copy link
Owner

devbisme commented Feb 6, 2025

@shanemmattner , it looks like you're analyzing all the SKiDL code, extracting subcircuits, placing all the subcircuit code into a file, adding wires to the subcircuit I/O, and finally executing the whole thing. What advantage does this give compared to just executing the original SKiDL code?

One concern I have (which may be because I've misunderstood what's going on) is that the call_subcircuit helper function assigns a Net to every subcircuit parameter. But a subcircuit is just a Python function and can accept any type of parameter. A parameter may be a SKiDL Bus, or an integer or flag that selects how the subcircuit instantiates itself. Determining the correct type would require some substantial analysis if it can be done at all given Python's dynamic typing.

@shanemmattner
Copy link
Contributor Author

@shanemmattner , it looks like you're analyzing all the SKiDL code, extracting subcircuits, placing all the subcircuit code into a file, adding wires to the subcircuit I/O, and finally executing the whole thing. What advantage does this give compared to just executing the original SKiDL code?

The netlist_to_skidl logic generates a full SKIDL project including a main() function. So to insert the LLM logic from kicad_skidl_llm.py I copy the circuits 1 by 1 since they are being analyzed independently anyways. The kicad_skidl_llm.py script is meant to be an end-to-end script to tie all the logic together and target an existing Kicad project. SKIDL users can call analyze_circuit() on their normal SKIDL projects as you would expect (I think, see my comment below...)

One concern I have (which may be because I've misunderstood what's going on) is that the call_subcircuit helper function assigns a Net to every subcircuit parameter. But a subcircuit is just a Python function and can accept any type of parameter. A parameter may be a SKiDL Bus, or an integer or flag that selects how the subcircuit instantiates itself. Determining the correct type would require some substantial analysis if it can be done at all given Python's dynamic typing.

That's a good catch on the non-Net arguments. I designed kicad_skidl_llm.py with the idea of targeting a user's existing Kicad project and generate a SKIDL project + LLM analysis. The generated SKIDL projects will only have nets passed in as arguments. But I see your point. I'm not sure if analyze_circuit would properly handle a Bus or other arguments to circuits besides nets. I'll test using a Bus works as expected and see what happens if I pass a non-net, non-bus argument into a @subcircuit.

So maybe the script I wrote is only useful for converting existing Kicad designs to SKIDL and analyzing the subcircuits.

@devbisme
Copy link
Owner

devbisme commented Feb 6, 2025

The netlist_to_skidl logic generates a full SKIDL project including a main() function. So to insert the LLM logic from kicad_skidl_llm.py I copy the circuits 1 by 1 since they are being analyzed independently anyways. The kicad_skidl_llm.py script is meant to be an end-to-end script to tie all the logic together and target an existing Kicad project. SKIDL users can call analyze_circuit() on their normal SKIDL projects as you would expect (I think, see my comment below...)

So you're re-building the SKiDL code so you can execute it within kicad_skidl_llm.py which gives you access to the default_circuit from which you can trigger the analyze_llm method. Is that correct? If so, it seems all you need is a way to execute the original SKiDL code and then get access to the default_circuit. The Python debugger lets you run code and then examine the objects in memory. Is there some way we can make use of that (or something similar) to get access to default_circuit and then trigger analyze_llm?

@devbisme
Copy link
Owner

devbisme commented Feb 6, 2025

I think we can do what I suggested as follows. Suppose we have a SKiDL file called my_circuit.py with this inside it:

from skidl import *

# Create a new circuit.
r = Part('Device', 'R', value='1K')
c = Part('Device', 'C', value='1u')
r & c

Then we have another file called run_circuit.py with this inside it:

# This will run my_circuit.py and the circuit will be in the default_circuit object.
from my_circuit import *

# Now take the default_circuit object and do whatever: generate_netlist, analyze_circuit, etc.
default_circuit.generate_netlist()

Then just run run_circuit.py to perform whatever operations you need on the unmodified my_circuit.py.

I think we can use this same technique in kicad_skidl_llm.py.

@devbisme
Copy link
Owner

devbisme commented Feb 7, 2025

I looked deeper into your code and saw that you already do something like what I suggested (around line 320), only you import the circuit_analysis module you synthesized from the original SKiDL code. So there must be some reason you couldn't just use the original?

@shanemmattner
Copy link
Contributor Author

Thanks for your thorough review. I'll try to refactor the code to use the existing SKIDL circuit since that would be more robust. I got busy today but I'll try to get this done tomorrow or over the weekend.

@devbisme
Copy link
Owner

devbisme commented Feb 7, 2025

No rush! Nobody is holding a stopwatch on us..

A couple of random thoughts:

  • A potential home for kicad_skidl_llm.py might be in src/skidl/scripts.
  • Each subcircuit, part, and net (maybe) could be given a purpose attribute when it is instantiated that describes what its particular function is. For example, a capacitor might have a purpose of "power supply bypass" for one of its instantiations, but a different instantiation of the same type of part might have a purpose of "set low-pass filter cutoff". The same thing could be done with subcircuits since a subcircuit could have different purposes for each instantiation. This would give the LLM more information to drive the design review.

@shanemmattner
Copy link
Contributor Author

Ok, I made a bunch of changes:

  • kicad_skidl_llm is now a command like netlist_to_skidl
  • broke up the large script into smaller files
  • I made the LLM calls logic use worker threads to run them in parallel. I'm finding that each circuit takes ~30-60 seconds for the LLM to analyze so this is a nice upgrade
  • Added a README.md file in src/skidl/scripts/llm_analysis/ directory with a description of the commands available and code structure
  • Added logic to look for component keyword purpose. This is a good idea and goes along with a key advantage of SKIDL over CAD which is easy documentation of thinking around parts/circuits.
  • Remove duplicated logging prints

Here's an example of running the script now:

(venv) shane@shane-ThinkPad-X1-Carbon-Gen-10:~/Desktop/skidl$ kicad_skidl_llm --schematic ~/Desktop/skip_repos/electronics/PCB/BMS/bms_v_0_3/bms_v_0_3.kicad_sch --generate-netlist --generate-skidl --analyze --api-key $OPENROUTER_API_KEY --kicad-lib-paths ~/Desktop/skip_repos/electronics/PCB/pcb_libraries/kicad/
Starting KiCad-SKiDL-LLM pipeline
[2025-02-07 17:29:15] INFO: ✓ Generated netlist: bms_v_0_3.net
[2025-02-07 17:29:24] INFO: ✓ Generated SKiDL project: bms_v_0_3_SKIDL
[2025-02-07 17:29:24] INFO: Importing main module...
[2025-02-07 17:29:24] INFO: Executing circuit main()...
[2025-02-07 17:29:26] INFO: Starting parallel analysis of 1 circuits...
INFO: 
=== Starting Circuit Analysis with google/gemini-2.0-flash-001 === @ [/usr/lib/python3.12/concurrent/futures/thread.py:58=>/home/shane/Desktop/skidl/src/skidl/scripts/llm_analysis/analyzer.py:41]
INFO: 
Generating analysis... @ [/usr/lib/python3.12/concurrent/futures/thread.py:58=>/home/shane/Desktop/skidl/src/skidl/scripts/llm_analysis/analyzer.py:41]
INFO: Approximate cost for this query: $0.0294 @ [/usr/lib/python3.12/concurrent/futures/thread.py:58=>/home/shane/Desktop/skidl/src/skidl/scripts/llm_analysis/analyzer.py:41]
INFO: 
=== Analysis completed in 35.80 seconds === @ [/usr/lib/python3.12/concurrent/futures/thread.py:58=>/home/shane/Desktop/skidl/src/skidl/scripts/llm_analysis/analyzer.py:41]
[2025-02-07 17:30:02] INFO: ✓ Completed analysis of top.bms0
[2025-02-07 17:30:02] INFO: 
Analysis Results:
[2025-02-07 17:30:02] INFO:   ✓ Completed Circuits: 1
[2025-02-07 17:30:02] INFO:   ✓ Total Analysis Time: 35.81 seconds
[2025-02-07 17:30:02] INFO:   ✓ Total Pipeline Time: 38.40 seconds
[2025-02-07 17:30:02] INFO:   ✓ Total Tokens Used: 14706

Analysis results saved to: circuit_analysis.txt

Pipeline completed in 47.81 seconds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants