Toy example of using instrumentation

It monitors and bins signal values

Update node widths with Firrtl Transform
Working transform that will adjust widths of
registers, ports and wires through the annotation system.
Second piece of the augmented tool chain that will ultimately
take advantage of firrt interpreters instrumentation output.
Adjusting widths according to data gathered thereby.
Part of Issue #114

Update node widths with Firrtl Transform
Working transform that will adjust widths of
registers, ports and wires through the annotation system.
Second piece of the augmented tool chain that will ultimately
take advantage of firrt interpreters instrumentation output.
Adjusting widths according to data gathered thereby.
Part of Issue #114

A little bit of cleanup

Working on bit reduction calculator
BitReducer does simple min/max to determine new bit size
Can be made more advanced.

Added writer to write bit changing annotations to a file
With this we now have most of the machinery for Issue #114

remove tools.jar dependency

fix context vs. not inconsistencies in type classes. add support for conversion to interval.

add preliminary support for interval type class

forgot to fix some context ops in intervals

Minor changes

fix syntax errors for compilation

few minor syntax errors in intervaltype. add clip op to type classes.

change 1 << n to BigInt(1) << n

fixedprecisionchangerspec checks out

migrate dsp tools/interval tests from fft project

symlink local tests

dsptools.math was conflicting with scala.math

macros need to be enabled for chiselName. also, another scala.math fix

scala strassen winograd matrix multiplication

remove commented out typeclass stuff (to be revisited)

lots of prep work to get matrix ops working. still debugging...

minor update

save minor changes for debug

save temporary progress. too many wires...

pipelining works -- hardcoded

This creates a toolchain that unites the machinery for Issue #114

See InstrumentingSpec:"run with with bits reduced" for example of use

New executeWithBitReduction acts like ordinary dsptools.Driver.execute

Adds in the following

- Run with interpreter bit instrumentation
- Analyzes report, creating annotations to reduce size
- Runs transform to reduce bits in low Firrtl
- Re-runs the tests re-instrumenting

Produces files

- <dut-name>.signal-bitsizes.csv
- <dut-name>.signal-bitsizes-2.csv
- <dut-name>.bit-reduced.fir

debugged matrix lit not working -- breeze orders things differently

Some fixes for Issue #114

- Fixed changes to sizes in sub-module being lost
- Added warnings if annotation to change wire not used or used more than once
- Added a executeWithBitReduction to IAArithSpec, it works
- Fix spelling in createAnnotationIfAppropritate

adam fixed firrtl bug -- interval 4x4 and 8x8 work now

minor changes. trying to debug dspreal

fixed dspreal bug with matrix ops. dspreal direction wasn't getting propagated properly to TestModule, and poking a non-input didn't result in compile-time failure (only observed in sims)

separate out matmul tests

changes for dct

Added bit reduction by standard deviation.
Now you can specify a multiplier of the standard deviation.
The bit reduction will apply to the min and the max
being determined by the mean ± (multiplier * σ)
if these numbers exceed the min and max seen, then the more
restrictive limit will apply.
More testing required

Undo accidental changes

added systolic array matmul. dct constraint bug...

updates for getting random inputs working; bit reduction errors out

In process refactoring of change widths.

In process refactoring of change widths.

filtering should work now. waiting on bit reduction

Added ToWorkingIr and InferWidths to ChangeWidthTransform
Changed to not change width's of IO's, could not get verilator to
run when I did that.
Seems to fix Angie's problems

ChangeWidthTransform broke because
DefInstance became WDefInstance because of ToWorkingIR

Fix Bit operation errors
Add in a few more passes to try and fix up
Bit prim ops whose args are out of syncs with bit reduced signals

A bit of style and dead code cleanup for previous commit

more benchmarks

bump numtests

change tolerance for 8-bit

changed bitwidths. interpreter stuff not saving.

start working on fir filter example. tested real/fixed conversions.

fir working; needs cleanup

working fir example

start working on clicking when limiting to n*sigma

Added some BitWidth convenience tools
Added HTML BitWitdh report to BitReducer
Changed default binning to 16
Added UNFILTERED to two test names
to make it easier to run just that one.

minor code cleanup for fir demo

prep for interpreter tests

New strategy for reducing wires, does not actually reduce them
but creates shadow reduced wire with name <old-name>__reduced
and assigns to original wire with sign or zero extension
 <old-name>__msb joined by recursive cat operators.

Added NODedupAnnotator to SystolicMatMul, this fixes blow ups
due to a given wire getting two different bit reductions for different
instances it appears in

Right now I am not reducing IO's in submodules, doing so breaks IO
naming because of the __reduced.

Added UNFILTERED to two test names so I could select them easier in
sbt

Added some incomplete tests of things as I was tracking down bugs

Don't reduce anything to less than 4 bits.
Hack fix for now for verilog complaining about something
that was reduced to 1 bit

made some minor changes; looks like bit reduction still doesn't work. see help files in top-level dir

minor changes

use different multiply alg in matrixops

Auto pad mixed radix representation

build.sbt

@@ -66,6 +66,8 @@ libraryDependencies ++= Seq(
   "org.scalacheck" %% "scalacheck" % "1.13.4"
 )
 
+addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.0" cross CrossVersion.full)
+
 javacOptions ++= javacOptionsVersion(scalaVersion.value)
 
 publishMavenStyle := true

src/main/scala/dsptools/Driver.scala

@@ -2,19 +2,26 @@
 
 package dsptools
 
+import java.io.{File, PrintWriter}
+
 import chisel3._
 import chisel3.iotesters._
+import dsptools.numbers.resizer.{BitReducer, ChangeWidthTransform}
+import firrtl._
+import numbers.DspRealFactory
 import firrtl.{ExecutionOptionsManager, HasFirrtlOptions}
 import numbers.{DspRealFactory, TreadleDspRealFactory}
 import firrtl_interpreter._
 import treadle.HasTreadleOptions
 
 import scala.util.DynamicVariable
 
+//scalastyle:off regex
+
 object Driver {
 
   private val optionsManagerVar = new DynamicVariable[Option[DspTesterOptionsManager]](None)
-  def optionsManager = optionsManagerVar.value.getOrElse(new DspTesterOptionsManager)
+  def optionsManager: DspTesterOptionsManager = optionsManagerVar.value.getOrElse(new DspTesterOptionsManager)
 
   def execute[T <: Module](dutGenerator: () => T,
       optionsManager: TesterOptionsManager)(testerGen: T => PeekPokeTester[T]): Boolean = {
@@ -55,6 +62,176 @@ object Driver {
     }
   }
 
+  //scalastyle:off method.length
+  def executeWithBitReduction[T <: Module](
+    dutGenerator: () => T,
+    optionsManager: TesterOptionsManager
+  )(
+      testerGen: T => PeekPokeTester[T]
+  ): Boolean = {
+
+    val om = optionsManager match {
+      case d: DspTesterOptionsManager => Some(d)
+      case _ => None
+    }
+
+    optionsManagerVar.withValue(om) {
+      optionsManager.interpreterOptions = optionsManager.interpreterOptions.copy(
+        blackBoxFactories = optionsManager.interpreterOptions.blackBoxFactories :+ new DspRealFactory)
+
+      val maxPassNumber = om.getOrElse(new DspTesterOptionsManager()).dspTesterOptions.bitReduceMaxPasses
+      val fudgeConstant = om.getOrElse(new DspTesterOptionsManager()).dspTesterOptions.bitReduceFudgeConstant
+
+      val requestedName = optionsManager.interpreterOptions.monitorReportFileName
+      if(requestedName.nonEmpty) {
+        println(s"Warning: ignoring monitorReportFileName=$requestedName")
+      }
+
+      //TODO (chick) no improvement over earlier passes should also terminate the loop
+      def bitReductionPass(firrtlSourceOption: Option[String] = None, passNumber: Int = 0): Option[String] = {
+
+        println(s"Running bit reduction pass $passNumber")
+
+        def makeReportFileName = s"signal-bitsizes-$passNumber.csv"
+
+        //
+        // Run the test
+        //
+        def runTheDUT(): Boolean = {
+          optionsManager.interpreterOptions = optionsManager.interpreterOptions.copy(
+            monitorBitUsage = true,
+            monitorReportFileName = makeReportFileName,
+            prettyPrintReport = false
+          )
+
+          iotesters.Driver.execute(dutGenerator, optionsManager, firrtlSourceOption)(testerGen)
+        }
+
+        //
+        // Read in the signal instrumentation report generated by the interpreter
+        //
+        def runBitReducer(): BitReducer = {
+          val reportFileName = optionsManager.interpreterOptions.getMonitorReportFile(optionsManager)
+          val data = io.Source.fromFile(reportFileName).getLines().toList.drop(1)
+          val htmlOutput = optionsManager.getBuildFileName(s".bit-usage-$passNumber.html")
+
+          //
+          // Construct a bitReducer and use it to create annotations to change appropriate signal sizes
+          //
+          val bitReducer = new BitReducer(
+            data, om.getOrElse(new DspTesterOptionsManager()).dspTesterOptions.bitReduceBySigma, fudgeConstant,
+            htmlOutput
+          )
+          bitReducer.run()
+          val report = bitReducer.getReportString
+          println(report)
+
+          val writer = new PrintWriter(new File(optionsManager.getBuildFileName(s"bitreport$passNumber.txt")))
+          writer.println(report)
+          writer.close()
+
+          bitReducer
+        }
+
+
+        //
+        // Apply the annotations to the previous firrtl to generate a more optimized firrtl
+        //
+        def runBitReduction(bitReducer: BitReducer, firrtlString: String): String = {
+          val annotationMap = bitReducer.getAnnotationMap
+//          annotationMap.annotations.foreach { anno =>
+//            println(anno.serialize)
+//          }
+
+          val annoWriter = new PrintWriter(new File(optionsManager.getBuildFileName("anno.width")))
+          annotationMap.annotations.foreach { anno =>
+            annoWriter.println(anno.serialize)
+          }
+          annoWriter.close()
+
+          val circuitState = firrtl.CircuitState(Parser.parse(firrtlString), LowForm, Some(annotationMap))
+
+          val transform = new ChangeWidthTransform
+
+          val newCircuitState = transform.execute(circuitState)
+
+          val newFirrtlString = newCircuitState.circuit.serialize
+
+          // println("Bit-reduced Firrtl\n" + newFirrtlString)
+
+
+          val newFirrtlFileName = {
+            optionsManager.firrtlOptions
+              .getTargetFile(optionsManager)
+              .replaceFirst(""".lo.fir$""", s".bit-reduced-$passNumber.fir")
+          }
+          val writer = new PrintWriter(newFirrtlFileName)
+          writer.write(newFirrtlString)
+          writer.close()
+
+          newFirrtlString
+        }
+
+        if(! runTheDUT()) {
+          println(s"Error: executeWithBitReduction failed in pass $passNumber")
+          None
+        }
+        else {
+          val bitReducer = runBitReducer()
+          val firrtlFilename = optionsManager.firrtlOptions.getTargetFile(optionsManager)
+          val firrtlString = io.Source.fromFile(firrtlFilename).getLines().mkString("\n")
+
+          if(bitReducer.bitsRemoved > 0 && passNumber < maxPassNumber) {
+
+            val newFirrtlSource = runBitReduction(bitReducer, firrtlString)
+
+            bitReductionPass(Some(newFirrtlSource), passNumber + 1)
+          }
+          else {
+            // return what we just processed if nothing changed or maxed out on looping
+            firrtlSourceOption match {
+              case Some(_) => firrtlSourceOption
+              case _ => Some(firrtlString)
+            }
+          }
+        }
+      }
+
+      bitReductionPass() match {
+        case Some(finalFirrtlSource) =>
+          optionsManager.testerOptions = optionsManager.testerOptions.copy(backendName = "verilator")
+
+          iotesters.Driver.execute(dutGenerator, optionsManager, Some(finalFirrtlSource))(testerGen)
+        case _ =>
+          println(s"Error: executeWithBitReduction failed to produce final bit-reduced Firrtl source")
+          false
+      }
+    }
+  }
+
+  def executeWithBitReduction[T <: Module](
+                                            dutGenerator: () => T,
+                                            args: Array[String] = Array.empty
+                                          )
+                                          (
+                                            testerGen: T => PeekPokeTester[T]
+                                          ): Boolean = {
+
+    val optionsManager = new DspTesterOptionsManager {
+      interpreterOptions = interpreterOptions.copy(
+        blackBoxFactories = interpreterOptions.blackBoxFactories :+ new DspRealFactory)
+    }
+
+    if (optionsManager.parse(args)) {
+      executeWithBitReduction(dutGenerator, optionsManager)(testerGen)
+    }
+    else {
+      optionsManager.parser.showUsageAsError()
+      false
+    }
+  }
+
+
   def executeFirrtlRepl[T <: Module](dutGenerator: () => T,
       optionsManager: ReplOptionsManager = new ReplOptionsManager): Boolean = {
 
@@ -75,7 +252,7 @@ object Driver {
           FirrtlRepl.execute(optionsManager)
           true
         case ChiselExecutionFailure(message) =>
-          println("Failed to compile circuit")
+          println(s"Failed to compile circuit: $message")
           false
       }
     }

src/main/scala/dsptools/intervals/IAUtility.scala

@@ -0,0 +1,52 @@
+package dsptools.intervals
+import chisel3.internal.firrtl.IntervalRange
+import firrtl.ir.Closed
+
+object IAUtility {
+
+  /** Expand range by n (twice of `halfn`). If n is negative, shrink. */
+  def expandBy(range: IntervalRange, halfn: Double) = {
+    val newMinT = getMin(range) - halfn
+    val newMax = getMax(range) + halfn
+    val newMin = if (newMinT > newMax) newMax else newMinT
+    if (newMinT > newMax) println("Attempting to shrink range too much!")
+    /*
+    println(
+      (if (halfn < 0) s"[shrink $halfn]: " else s"[expand $halfn]: ") +
+        s"old min: ${getMin(range)} old max: ${getMax(range)}; " +
+        s"new min: $newMin new max: $newMax"
+    )
+    */
+    IntervalRange(Closed(newMin), Closed(newMax), range.binaryPoint)
+  }
+
+  /** Shift range to the right by n. If n is negative, shift left.  */
+  def shiftRightBy(range: IntervalRange, n: Double) = {
+    val newMin = getMin(range) + n
+    val newMax = getMax(range) + n
+    IntervalRange(Closed(newMin), Closed(newMax), range.binaryPoint)
+  }
+
+  /** Check if range contains negative numbers. */
+  def containsNegative(range: IntervalRange) = getMin(range) < 0
+
+  /** Get the # of bits required to represent the integer portion of the
+    * rounded bounds (including sign bit if necessary).
+    */
+  def getIntWidth(range: IntervalRange) = {
+    // TODO: require(range.binaryPoint.get == 0, "getIntWidth only works for bp = 0")
+    val min = BigInt(math.round(getMin(range)).toInt)
+    val max = BigInt(math.round(getMax(range)).toInt)
+    val minWidth = if (min < 0) min.bitLength + 1 else min.bitLength
+    val maxWidth = if (max < 0) max.bitLength + 1 else max.bitLength
+    math.max(minWidth, maxWidth)
+  }
+
+  /** Gets min */
+  def getMin(range: IntervalRange) = range.getPossibleValues.min
+  /** Gets max */
+  def getMax(range: IntervalRange) = range.getPossibleValues.max
+  /** Gets width of range */
+  def getRange(range: IntervalRange) = getMax(range) - getMin(range)
+
+}

src/main/scala/dsptools/math/ExtendedEuclid.scala

@@ -0,0 +1,16 @@
+package dsptools.math
+
+object ExtendedEuclid {
+  /** Extended Euclidean Algorithm
+    * ax + by = gcd(a, b)
+    * Inputs: a, b
+    * Outputs: gcd, x, y
+    */
+  def egcd(a: Int, b: Int): (Int, Int, Int) = {
+    if (a == 0) (b, 0, 1)
+    else {
+      val (gcd, y, x) = egcd(b % a, a)
+      (gcd, x - (b / a) * y, y)
+    }
+  }
+}