Calyx Opt Spring 2024 Lab Notebook

[parthsarkar17]

Lab notebook for questions/updates on Calyx-Opt

[parthsarkar17]

Hello! Just a brief overview of what I've been focusing on since I started my mini-pass a couple weeks ago: I got started by learning a bit of Rust syntax and about the whole ownership model, which was pretty new and definitely a bit of a learning curve. I still run into borrow-checker issues, but they are dramatically fewer than when I started. I had also heard that Rust has a lot of functional features, and coming in with a little bit of an OCaml perspective, I was looking forward to using them -- I wasn't disappointed!

I started working on my unused-port-removal pass, which follows this (#1200) issue. I had a bit of trouble learning the IR for Calyx, since there were certain ideas that didn't directly follow from Calyx syntax. In particular, I spoke at length with @calebmkim about the Visitor trait and how it represents and traverses components, and the Component struct, which was a bit confusing with all of its nested fields. However, after working step-by-step and getting information from only the necessary parts of the IR, I started also getting familiar with that, and saw how closely it resembled Calyx programs.

So far, I built a HashMap that maps each component to its used ports. Then, I took the set difference between all of a components ports (defined in its signature) and its used ports, and got a set of unused ports. I then iterated through these unused ports, and added an attribute @internal to them. The next step is to actually remove the ports. The code lives here: https://github.com/calyxir/calyx/blob/unused-port-removal/calyx-opt/src/passes/unused_port_removal.rs

I made a couple assumptions that I'll definitely have to verify. Is it true that:

1. By "unused port", we mean that no external component B uses a port of A? (as in, A could assign to an "outward-facing" port P, but if no other component uses/assigns to P, then P should be marked as @internal?)
2. If component B instantiates component A but never assigns to it, then component A would never be traversed.

Thanks!

[rachitnigam]

By "unused port", we mean that no external component B uses a port of A?

Yup! In other words, the "outside world" does not interact with port p of A transitively through B

If component B instantiates component A but never assigns to it, then component A would never be traversed.

The dead-cell-removal pass should remove A from B if B instantiates A but does not use it. Currently, there is no way to remove components from the context which means your pass will probably end up deleting all of the ports defined in the signature of A. We should define some mechanism to remove entire components from a Calyx program.

I then iterated through these unused ports, and added an attribute https://github.com/internal to them

I think you can just go ahead and remove these ports in this pass. For debugging purposes, you can keep this code path around so that users have a way to see which ports the compiler thinks are unused.

---

Overall, awesome progress! The next step of removing the ports is going to be a little challenging because you need to transitively remove ports and assignments to them. For example, if we have: A instantiates B instantiates C and removing a port from A means that some port in B becomes unused, then we need to remove that port from B and so on. Making this work efficiently requires some careful thought! A HashMap of used ports might not be enough here.

[calebmkim]

For example, if we have: A instantiates B instantiates C and removing a port from A means that some port in B becomes unused, then we need to remove that port from B and so on. Making this work efficiently requires some careful thought! A HashMap of used ports might not be enough here.

This may be the wrong idea, but it seems like for this, it might be beneficial to store not only if A's port P is used by the outside world, but also in which component(s) P is used.

2. If component B instantiates component A but never assigns to it, then component A would never be traversed.

From the method we discussed, I would have thought that A would be traversed by the pass. In general, the visitor will visit all components.

[rachitnigam]

That might still not be enough information; you need to know that, if port p is removed, which other ports q1...qn will be also be removed because p is the only port that writes to them. Does that make sense?

[parthsarkar17]

Hello! Apologies for the delay on updates.

In the last week, Caleb and I were discussing exactly that question: which ports do we feel safe enough to remove (and, by association, how aggressive should we be in removing them)? In addition to those ports (as in your example, Rachit) where removing an unused port of A would make one of B's port unused, we were also considering how guards might play into this.

I initially ran into quite a bit of confusion here, since I thought that if a guard within a component A uses one of A's ports p then, even though the outside world may not use p, we should still consider it to be "used" -- that snowballed into a bit of a crisis about what it means to "use" a port, but Caleb reassured me in our most recent meeting on Thursday that, if port p is used in a guard and is not used by the outside world, then p's presence in the guard is the same as a "low" signal. So, if I'd like to, I could extend the pass to remove these unused ports, and replace all of the port's uses in the guard with a n'b0 constant. Honestly, I think that'd be really cool!

But, as cool as it would be to be super aggressive and also keep the intended "meaning" of the modules by using n'b0's, we were both wondering whether it was "worth it" in terms of overhead on a design. Is there a substantial resource wastage with keeping these unused ports around? Is it not possible to say, since a port could be passed through any number of gates before becoming a part of a guard? What's the "average case"? I think it'd still be very cool to do it (especially if there's no "upper bound" on wasted resources), and I've already gotten started. To be honest, I think the case where a port is unused by the outside world and is also not used in a guard internally is pretty rare, so if we don't do this, then is it possible that we'd be optimizing a kind of trivial case?

If I get the green light, I'm hoping to have things done and tested by next weekend. Then, if possible, I'd love to discuss further plans as they come up! Thanks! Apologies again, I will be sure to be more timely with updates

[rachitnigam]

Thanks for the update @parthsarkar17! You've hit the right high-level questions here, namely, we need to balance the complexity of a pass with its applicability; if it only affects a small number of programs but is quite hard to implement efficiently, is it even worth doing? I think broadly, this is a question that needs to be answered using benchmarks (which is why I keep brining us back to the performance dashboard #1416).

I think you're probably right: this kind of a pass has limited applicability but I think it is still worth implementing it because it captures a lot of different kinds of problems that show up when writing Calyx passes like inter-component analysis, querying and mutating the IR, etc.

if port p is used in a guard and is not used by the outside world, then p's presence in the guard is the same as a "low" signal.

This is correct but it is worth understanding why. Calyx distinguishes between "control" and "data" ports (#1169). When a port is used in a guard context, it becomes a control port and the compiler will assign a default 0 value to it. I had not considered this consequence of the pass you're implementing but it could be potentially cool. Consider this program:

comp add_or_mult(op: 1, ...) -> (...) {
  control {
    if op { <logic for addition> } else { <logic for mul> }
  }
}

If you can prove that either add_or_mul is never given an input to op (which means its always equal to 0) or it is always given a particular input (0 or 1), then we can turn if op into if 0 or if 1 and remove one of the branches! This is a standard kind of analysis and optimization used in software languages: if we can prove that a function is only called with a particular value for an argument, we can specialize it w.r.t. to that argument.

FYI: you might enjoy reading Frances Allen's "A Catalogue of Compiler Optimizations". It's a good paper to read in general but specifically, it'll shed some light on the classic optimizations implemented in a compiler and how to think about them.

[parthsarkar17]

Sounds great! I'll definitely check out that paper.

I also think it'd be cool to implement, especially given the case you've laid out. It would be really awesome to be able to delete entire branches based on the knowledge of an unused port (Although on second thought, would inserting a low signal n'b0 in place of an unused port in a guard be considered deleting the branch? Is there another pass that gets rid of unused branches, or would I have to do that in this pass -- or could I conveniently schedule this pass before the pass that deletes provably unused branches?)

Additionally, I know Caleb mentioned the performance dashboard a few times, but I'm still not quite sure what it is exactly. Does it provide a set of general Calyx designs and see how much each pass "improves" a design versus how long it takes to run the pass?

Finally, I saw this paper on FDO stuff and was wondering if you had heard of it / recommend reading it for background info:
dynamo00.pdf
Thanks!!

[rachitnigam]

Is there another pass that gets rid of unused branches, or would I have to do that in this pass

No, we don't have a pass like that yet but it would be useful to add! Passes like this only become useful when upstream passes change the structure of the program and provide enough information to prove that certain branches are dead or certain loops are bounded.

I know Caleb mentioned the performance dashboard a few times, but I'm still not quite sure what it is exactly

It is meant to be a combination of representative benchmarks in calyx, stored in this the benchmarks repo (https://github.com/calyxir/benchmarks) and a server-based system for automatically running all the benchmarks with a particular compiler configuration and displaying results on the web. This tool does not exist yet but maybe you can see why something like this could be super useful!

[parthsarkar17]

Hello! This week was pretty cool, along with some gnarly head-butting against Rust's type system. Based on feedback, I indeed began by mapping out what it concretely means to get rid of unused ports. I took care of the trivial case pretty easily, which is: at each component, if there exist any unused ports in the signature of the component, then remove those from the signature. As we've mentioned, the tricky part came when I started trying to "collapse" unused ports in the guard tree to ports out a constant cell outputting an n-bit 0 signal.

I remember trying to figure out how I should modify the guard of the existing component; I ended up trying (from what I can tell) all three ways, with minimal initial success. From within guard.rs, I created three functions, each called some variant of collapse_unused, where I tried using a simple reference, a mutable reference, and eventually complete ownership of the guard in order to figure out how to change it. I was really stuck because each of these posed completely distinct memory challenges from one another, and I'd never faced any of them before haha. The "reference" way felt most natural to me because it felt kind of functional, but I remember having major issues with the return type of the function. Taking ownership worked fine for a while, but that eventually scared me because I didn't know how my pass would like it, so I tried the mutable reference path. After working out kinks for a couple hours, I think I have something! I realized just now that I made at least one tiny error, but I generally think I have a function that can take a mutable reference to a guard and squash all unused ports. Now, I'm having issues with the fact that there can be only one mutable reference to a component in my pass, which makes sense. I'll try working around it.

Unfortunately, it looks like I couldn't stick by the personal deadline I made last week to try and have PR done by this weekend. But I will keep at it! I feel like I'm learning a lot; even if this pass turns out to be not super helpful or is too cost inefficient, it's still very cool to reason about, especially given Rust's type system :)

[rachitnigam]

Awesome stuff Parth! Yes, this stuff is tricky when it comes to the ownership model of Rust; it forces you to carefully think about your modifications and how they might affect the validity of other parts of the code that might hold reference to it. In general, passes can provide you with mutable access to ir::Assignment which own ir::Guard structs. For example, you can look at the uses of Guard::update which allows passes to update the guard in-place using a function.

[rachitnigam]

One possible implementation choice is using Guard::update and providing it a method that essentially replaces Guard::Port that have been removed by the pass.

[calebmkim]

Here's a question (relevant to the unused-port-removal optimization) that I don't quite know the answer to.

In a Calyx program, for which ports are we allowed to assume that undriven => 0? (And the flip side of this, when should we assume undriven => ’x?). I've read #1169 but It's still not quite clear to me.

More concretely, suppose we have a component:

component Q(in1: 32, in2: 32) -> (out: 32) {
  // cells ommitted 
  wires {
    group g1 {
      reg1.in = in1; 
      reg2.in = in2; 
      // done condition ommitted 
    }
    group g2 {
      reg3.in = (reg1.out == reg2.out) ? 32'd1;
      reg3.write_en = 1'b1;
      simulate_guard[done] = reg3.done;
    }
    out = reg3.out 
  }
  // control ommitted 
}

Suppose that in our main, we have a group:

group g {
  // Q_inst is an instance of Q 
  Q_inst.go = 1'd1; 
  g[done] = Q_inst.done; 
  // Q_inst.in1 and Q_instin2 are not driven
} 

When we are executing the group g1 inside of Q :

group g1 {
      reg1.in = in1; 
      reg2.in = in2; 
     // done condition ommitted 
 }

are we allowed to assume in1 and in2 are 0? From reading #1169, it seems like Q should be marked as an @control instance, and therefore we should assume in1 and in2 are 0.

In general, it seems that almost any use of Q will be marked as a @control instance. My question is: are there cases where you can instantiate Q as a data instance? And in this case, are we still allowed to replace in1 with 0? Replacing a value that was previously undefined with 0 seems like it should be fine? (obviously the reverse would not be fine).

One final question: suppose we had the following group inside of Q :

group hypothetical_group {
      unused1.in = in1 < in2 ? 32'd9;
      // in1 and in2 used in guards
}

Does that affect whether or not we are allowed to assume in1 and in2 will have “0-by-default” behavior?

[parthsarkar17]

I see, so is it worth taking a look at infer-data-path? Specifically, the part of the pass that deals with the second condition laid out in #1169 (that a component used in a done condition must be marked as @control)? @calebmkim, I know you mentioned filing a bug report. If others also say this is unexpected behavior, how do we do that?

[rachitnigam]

That's right! Specifically, it tries to infer which parts of your circuit are data path components and therefore safe to more aggressively optimize. It uses the @data annotations on ports of primitives for this. Note that for a port to get an 'x assignment, both the port and the corresponding cell need to be marked @data.

This prevents things like the add.left getting assigned 'x when add.out is used in a guard.

[parthsarkar17]

I think my concern is also that, even if the component comp_Q was not supposed to be a @control-marked component, shouldn't any unused ports still be guaranteed to have the value 0 (unless the user defaults to some other 'x)?

[calebmkim]

@calebmkim, I know you mentioned filing a bug report. If others also say this is unexpected behavior, how do we do that?

Just make an issue in the github repo. (it's a good idea to include a minimum buggy example as well).

I think my concern is also that, even if the component comp_Q was not supposed to be a @control-marked component, shouldn't any unused ports still be guaranteed to have the value 0 (unless the user defaults to some other 'x)?

I think that depends on what we want out of our language. There's cases for and agains, and there's a pretty long thread in #922, but this comment outlines a few reasons why "always default to 0" might be a bad idea (see the "Strawperson: zero-by-default" section").

[sampsyo]

Coincidentally, this kind of thing recently came up also in #1919 (comment). My understanding of the current intended state of things is:

• All ports are @control or @data. @control is the default.
• @control ports get default-0 semantics.

(The aforelinked comment vaguely alludes to the idea of flipping the default, but that is not relevant to the present conundrum and it is not even clear if it's a good idea.) The upshot is that unannotated ports get default-0-ed, unless we can opportunistically prove this is not necessary. As @calebmkim says:

my understanding after thinking about it some more, is that people writing Calyx programs are allowed to assume "0-by-default" behavior for ports as part of the guarantee of Calyx programs. The infer-data-path pass is just an optimization we can apply so that we won't have to generate extra assignments to make "0-by-default" behavior when we know we won't read from that port.

In other words, because @control is the default, it is always sound to fail to infer something is @data. Failing to infer @data only results in worse performance, never incorrectness.

Sorry #922 is such an incredibly long thread. We learned something broader there, but I think the specific consequences for this issue are simpler: we may someday want to get rid of it, but currently default-0 semantics is the default, and @data lets you opt out of that and into the world of possibly-undefined signals.

[calebmkim]

We have a question about the following Calyx program:

Example Program

component Q(in1: 32, in2: 32) -> (out: 32) {
  cells {
    guard_output = std_reg(32);
  }
  wires {
    group simulate_guard_output {
      // write 1 if inputs are the same, 0 otherwise
      guard_output.in = !(in1 == in2) ? 32'd0;
      guard_output.in = in1 == in2 ? 32'd1;
      guard_output.write_en = 1'b1;
      simulate_guard_output[done] = guard_output.done;
    }
    out = guard_output.out;
  }
  control {
    seq {simulate_guard_output;}
  }
}

component main() -> () {
  cells {
    @external(1) mem = comb_mem_d1(32, 3, 2);
    reg1 = std_reg(32);
    reg2 = std_reg(32);
    comp_Q = Q();
    output_Q1 = std_reg(32);
  }
  wires {
    group through_Q {
      // doesnt trigger either input of Q
      comp_Q.go = 1'b1;
      through_Q[done] = comp_Q.done;
    }

    group store_output_Q {
      // store output to register
      output_Q1.write_en = 1'b1;
      output_Q1.in = comp_Q.out;
      store_output_Q[done] = output_Q1.done;
    }

    group output_Q_to_mem {
      // write output to memory 
      mem.addr0 = 2'd2;
      mem.write_data = output_Q1.out;
      mem.write_en = 1'b1;
      output_Q_to_mem[done] = mem.done;
    }

  }
  control {
    seq{
      through_Q;
      store_output_Q;
      output_Q_to_mem;
    }
  }
}

Question

For this group:

group through_Q {
  // doesnt trigger either input of Q
  comp_Q.go = 1'b1;
  through_Q[done] = comp_Q.done;
}

During execution of this group, are the values for comp_Q.in1 and comp_Q.in2 0, or are they undefined? The way things are working now, we think it is being lowered to undefined values (even though based on our understanding of Calyx's semantics, they should be 0). Because when we run:

fud e --to dat --through icarus-verilog --from calyx <calyx_file> -s verilog.data <data_file> -s calyx.flags "-d infer-data-path"

we get an error (ValueError: invalid literal for int() with base 16: '0x0000000X'). (As another side note, it seems this error only occurs in icarus not for verilator...this probably means there are undefined values that icarus/verilator are treating differently?)

However, if we change it to:

group through_Q {
  comp_Q.go = 1'b1;
   // Explicit instantiations of in1 and in2 to 0 
  comp_Q.in1 = 32'd0; 
  comp_Q.in2 = 32'd0; 
  through_Q[done] = comp_Q.done;
}

then we don't get an error, and we get the expected results.

Note also that we -d data-path-infer in order to eliminate control vs. data confusion. (in other words, we tried to make the compiler as aggresively as possible insert 0's).

Data File for reproduction purposes

(we expect 1 to be in the final memory in place of 89)

{
  "mem": {
    "data": [
      45,
      36,
      89
    ],
    "format": {
      "numeric_type": "bitnum",
      "is_signed": false,
      "width": 32
    }
  }
}

[rachitnigam]

The input ports of Q should both be marked as control ports (and therefore have a default 0) assignment.

And yes, verilator does not model x values so error will only occur on Icarus. Maybe the problem is #1930. Can you take a look at the output verilog and see if there are any assignments to the ports at all? If there are no assignments, then it is treated as an x value as well which means the default-assigns pass is not doing its job.

[parthsarkar17]

Sure, sounds good. Quick question: if we are not running data-path-infer, then would the ports of Q be marked as control? Or is that independent of data-path-infer?

[parthsarkar17]

Ok, so it's looking like in1 of Q is not being driven according to the Verilog compilation. There's a variable comp_Q_in1 declared through the line logic [31:0] comp_Q_in1;, and comp_Q_in1 is used in component Q as an input, but comp_Q_in1 is never actually assigned any value. I could also send the compiled output as a file for further investigation, if needed.

[rachitnigam]

There's a variable comp_Q_in1 declared through the line logic [31:0] comp_Q_in1;, and comp_Q_in1 is used in component Q as an input, but comp_Q_in1 is never actually assigned any value

Yeah, then that's a bug in default-assigns.

if we are not running data-path-infer, then would the ports of Q be marked as control?

All ports are assumed to be @control by default. I don't think data-path-infer does any cross component reasoning so it should mark leave them as is. Regardless, we should make it work in a cross component manner eventually.

[sampsyo]

Thanks for the super detailed example! FWIW, I agree with @rachitnigam's assessment above: namely, that (under current semantics) this looks like a bug. That is, all unannotated ports are @control, and @control ports should have default-0 semantics. So in1 and in2 should appear to be zero when unassigned.

There is also a separate question about what we should do (i.e., if we should change the semantics someday). In a hypothetical world where @data-like behavior is the default, we would classify this program as erroneous and want some way to reliably catch the problem.

[parthsarkar17]

Hey! Thought I'd present an update even though we've been having pretty consistent discussions. So from my understanding), I think I've gotten down the basic functionality of unused-port-removal! As I mentioned in the calyx-opt channel, the pass defines some unused ports at each component (which don't include go, done, clk, and reset ports. Based on this, the pass then:

• gets rid of all assignments from unused ports
• simplifies any guard expressions that use unused ports, based on the assumption that unused ports emit n bit 0 signals

I wrote a couple of designs, and I'm pretty sure they do exactly as intended! But, as we saw above, one specific design seems to have brought certain issues to light. There was a big thread about it in the Calyx Slack channel, and Caleb and I discussed it at length on Wednesday's Calyx-opt meeting. We were fairly certain something was up because one of the components we designed was marked as @data, even though it definitely needed to be @control.

After looking into it, it seemed to be something to do with the fact that some input ports were not driven (since we needed to test unused-port-removal)-- once we gave them explicit values, the design functioned as expected. So, that reignited a discussion on default values, which is where we're at now. It seems like the pass is safe as it is. If it acts on an @control component, then the ports default to value 0. If it acts on an @data component, the undefined values on ports make it so that (I think?) we can replace any use of that port's value with anything we want (hence, 0 also works here).

Based on the emails I'm receiving after I commit, I'm still failing out on formatting and some tests. But we can bang those issues out at Calyx-opt. Thanks!

[sampsyo]

This all sounds great!! Even if y'all have been chatting regularly, this kind of update is really helpful for me, even if I'm the only one. 😃

IMO, if you feel like you're narrowing in on something that is almost working (but not quite), that is a good time to open the first PR! You can mark it as a draft to make it clear that it's not quite ready to go yet, but having the PR open will make it really easy to point to specific failing tests/whatever to indicate the work you still have to do.

[parthsarkar17]

Gotcha, will do. Thanks for the direction!

[parthsarkar17]

Hello! Sorry for the delayed update, it has been a packed couple of weeks haha. This discussion will mainly serve as a summary of the Calyx meeting around #1960 (planning the performance dashboard). In that meeting, we discussed:

• Types of benchmark designs from which we could gain metrics
  • in addition to the ones on the issue above, we learned about the existing TCAM implementation in Calyx
• More in-depth about the kinds of inputs that the dashboard might want to take in
  • For now, since the main use is gauging improvements to the Calyx compiler, we want to focus on Calyx files directly, meaning users of the dashboard would have to explicitly generate Calyx designs and then get metrics off of different Calyx compiler versions
  • We learned, however, that various frontends basically generate a single Calyx file if they indeed use Calyx as an intermediate language. From my understanding, then, it shouldn't be terribly hard to take this single Calyx file and add it to our list of benchmarks?
    • An issue with this might arise if we make updates to the syntax of Calyx, or something like that. Then, the frontend might generate outdated Calyx.
• Important metrics and how to find them
  • In addition to the previously-discussed metrics, which were critical path / frequency, Calyx compile time, and FSM size, we got some more details on resource usage. Specifically, in order to get the number of memory (units?) / registers / adders, we don't have to synthesize the design. Instead, we can make use of lowered calyx to get the primitives count.
  • While more complex things that may require synthesis might also be helpful, the general message was get the "easy stuff first". This means we can start benchmarking on primitives first, and get to the potentially tedious bits later.
• Extended goals / future use
  • It would be really cool to get resource usage estimates on real hardware, like FPGAs if people decide to use those. However, there was a note about how that might be tricky because of licensing issues (I'm guessing this has something to do with how their synthesis tool is proprietary, and they won't necessarily let us get metrics from their synthesis output?)
  • So, we could also target ASICs by its open-sourceness
  • Didn't get much time to chat about FDO stuff; we said we'd save that for a later date. But, I'm assuming these preliminary metrics will also be super valuable for FDO efforts

That's all! I plan to make a PR on unused-port-removal this week. Thanks!!

[parthsarkar17]

Quick updates! This week, I worked on compiling Calyx designs to be used as benchmarks on the performance dashboard, and I worked on a script to take these benchmarks and get a summary primitive usage (see this commit, apologies for the strange message formatting). A couple open comments / questions on this first attempt at benchmark summary:

• This script gets an average (for each primitive, compute total # primitive uses and divide it of each primitive type) over all the designs. While that's useful, it may not be super representative of what people who work on the compiler might want to see. Should these summary statistics also include things like variance, median, etc. ?
• An extension of the question above: at what point do we start working on metrics we'd need to synthesize designs for? I know primitives were the first step, but do we just go straight for ASIC / FPGA outputs once primitives stuff is banged out?
• What is the purpose of the existing server on perf-dashboard? I'd like to integrate my work with existing work, but wasn't 100% sure of where to start, or even whether the purpose of the dashboard has shifted beyond the existing server. Thoughts?
• How do we organize benchmarks? Right now, I've stuck them in some folder outside of the Calyx compiler directory (see the benchmarks directory in the commit above).
• How do I document the script? I think it's pretty readable now, but as it gets bigger, we might want a solid place to store commands to run and how to edit the script.

That's all! Thanks!

[calebmkim]

Thanks for the update! I'll give my two cents here:

This script gets an average (for each primitive, compute total # primitive uses and divide it of each primitive type) over all the designs. While that's useful, it may not be super representative of what people who work on the compiler might want to see. Should these summary statistics also include things like variance, median, etc. ?

In general, I wouldn't worry too much about summary statistics like variance, median, etc. I think the bigger thing that is useful is how the performance of different versions of the compiler perform on the same benchmark (by "benchmark" I just mean a .futil Calyx file). (Hopefully more recent versions of the compiler will beat older versions of the compiler). This means that engineering-wise, I would focus more on the following questions: is there a way to automatically record which commit hash (and the date of that commit) we are using to profile a benchmark? How should we store this data so that we can easily compare the same benchmark across different commit hashes? Ideally (as an end goal), it would be cool for us to be able to compare, each time we make a new commit to main, how the performance of each benchmark changes. Lmk if this makes sense.

An extension of the question above: at what point do we start working on metrics we'd need to synthesize designs for? I know primitives were the first step, but do we just go straight for ASIC / FPGA outputs once primitives stuff is banged out?

This is a good question, and I'm honestly not sure... I've said this before, but I think we want something that works "end-to-end" before moving on. I recognize this is a somewhat vague bar though...

What is the purpose of the existing server on perf-dashboard? I'd like to integrate my work with existing work, but wasn't 100% sure of where to start, or even whether the purpose of the dashboard has shifted beyond the existing server. Thoughts?

My understanding is that it's very preliminary, but I think Rachit might have more knowledge about this.

How do we organize benchmarks? Right now, I've stuck them in some folder outside of the Calyx compiler directory (see the benchmarks directory in the commit above).

That seems fine for now. If we get more benchmarks, then we might want to organize them by type (for example, if we get the polybench benchmarks, we might want to put them in a polybench folder).

How do I document the script? I think it's pretty readable now, but as it gets bigger, we might want a solid place to store commands to run and how to edit the script.

You can also update the ReadMe once you push to main. We can have a synchronous discussion on what an appropriate level of detail is to include.

As always, open to comments and questions on this.

[parthsarkar17]

Calyx-opt meeting -- Idea of what end-to-end means:

• run a single script that takes in a calyx commit hash, already configured + checked out by user (later on will automate)
• get and save resource estimates for each benchmark for each commit hash
• repeat this for any hash
• command line argument that takes in two commit hashes and a benchmark, and if it has the data for the two hashes for this benchmark, it spits out some comparison
• graphs and things later
  • for graph: x-axis is commit hashes, ordered w.r.t time
  • y-axis is displaying specific resource usage
  • so graph is resource usage over time, effectively

[rachitnigam]

Thanks for the update @parthsarkar17! @calebmkim got basically everything right I think. One note about the server stuff:

I wanted there to be a web interface to look at the results and compare the resource usage across various runs/commits. The code is just a sketch for that kind of a web interface but completely throwable if needed.

I think we want to focus on getting synthesis results instead of resource counts from the Calyx backend; the latter is a very coarse grained estimate and not particularly actionable because it does not say anything about the timing behavior of the design.

[parthsarkar17]

Okay, that sounds good, I'll make sure to focus on synthesis stuff. I made a good chunk of progress on the Calyx primitives front, so if we'd want to include that, that's always there as well! Thanks very much for your input, Rachit!!

[parthsarkar17]

Hello! Some updates and questions:

Following the AMC/Opt meeting a couple weeks ago, @calebmkim and I started working on some FSM optimizations in Calyx after hearing about how an equivalent Vitis design generated significantly lower LUT usage. Right now, I'm using the work that Caleb did to separate the Static FSM object and implementing one hot encoding (OHE) to see if that helps. Here are the steps I've taken so far and things left to do:

• Instantiate a StaticFSM object with OHE. What I'm doing now is, for each query, I'm creating a continuous assignment from the FSM register to a bit slicer. I want to store the output of this slicer continuously, to later create guards that have the condition wire != 0. I haven't thought of a way to do this yet; Caleb mentioned creating an RRC<Cell> wrapping, but those are usually reserved for registers and other primitives, right? Is it possible to just directly read from the output of the slicer without instantiating a primitive?
• TODO: Haven't gotten here yet due to the issue of me not being able to access the continuous assignments I created in the instantiation. But this step requires taking the current FSM register state and adding one to it.
• TODO: Finally, we have to implement the query_between function, which creates a guard based on the upper and lower bounds of the query.

Unfortunately, it's looking like it won't be done in time for the next AMC/Opt meeting because of the blocker mentioned above. But, I'm looking to get it fixed by meeting in-person tomorrow and to keep going after!

Some open, brain-dump comments/questions:

• I'd like to implement OHE for dynamic groups next. Which would impact AMC more, static, dynamic, or both?
• I'd also like to try out the FSM-split optimization. These things, namely seeing their impact from users like AMC, would be really helpful for thinking about what kinds of tweaks an FDO re-compilation might make.
• As for perf-dashboard, that's still on my mind. I got the Havarti Xilinx tools set up a while ago, but haven't made more progress than that; we're still at the script that collects primitives resource usage that we had a while ago. Hoping to make more progress on that as well this week.

Thanks!!