Make TLBusWrappers' topology a configurable Field

[hcook]

Overview

This PR changes the selection of the topology of TLBusWrappers that are included in the BaseSubsystem layer of a Rocket-Chip-based design from being a static trait that is mixed in at compile time to a child class of BaseSubystem to being a dynamic choice driven at runtime from a configurable Field. This change means that the choice of internal network-on-chip topology no longer affects the Scala class hierarchy of Subsystems. Instead, a Field is supplied by which users can create a topology on a per-Config basis. The description of the topology is intended to be readable from a serializable format, although the current implementation maintains some programmatic escape hatches. Any number of TLBusWrappers in any connectivity network shape is possible, though the requirements of diplomatic TileLink must still be satisfied (no cycles, single apparent path to each address from each agent, etc).

A topological Config alteration is named WithCoherentBusTopology is supplied to recreate the extant, default, 5-hierarchical-crossbars topology from DefaultConfig+BaseSubsystem with HasHierarchicalBusTopology. The HasHierarchicalBusTopology trait itself is removed, and any child class that extends BaseSubsystem only to supply a different topology from that one should now be redundant (can just extend BaseSubsystem and supply a different value for the Field TLNetworkTopologyLocated(where: HierarchicalLocation). Three included example sub-topologies are included, which build on each other additively in order to recreate the original 5-xbar design. The three sub-topologies are JustOneBusTopologyParams (sbus), HierarchicalBusTopologyParams (fbus, cbus, pbus) and CoherentBusTopologyParams (adds a coherence manager and mbus).

API Additions and Changes

What is changed:

• Use recently-added classes Location and LocationMap: These dynamic maps can store arbitrary things indexable by "location" (just a string name for now). For example, TLBusWrapper references are stored in named locations in HasTileLinkLocations .tlBusWrapperLocationMap.
• Added trait HasTileLinkLocations: intended to be an even more generic version of the extant Attachable trait, which now extends it.
• HasTileLinkLocations provides def locateTLBusWrapper function for child devices to locate a TLBusWrapper instance inside a parent that extends this trait based on a location label.
• TLBusWrapperTopology is the new intended serializable entry point for describing bus topologies, see below.
• CoherenceManagerWrapper is a new wrapper for all collateral related to attaching coherence managers within a topology, allowing use of the TLBusWrapper connectivity and clock crossing APIs. This will change the instance hierarchy of the coherence manager in existing designs by adding another module hierarchy layer wrapping it and its associated adapters together.
• HasHierarchicalBusTopology trait is removed, but configuration equivalents WithCoherentBusTopology and WithIncoherentBusTopology are provided to be applied as alterations to DefaultConfig instead.
• A new class of runtime error becomes possible, where a device or wrapper with a configurable attachment or connection location tries to use a location for an instance that actually does not exist in a given configured topology. So it goes with dynamic configurability...

What is not changed:

• Accessor val members of BaseSubsystem named sbus, fbus, cbus etc. still exist; I would like to deprecate them in the near future in favor of using only the locateTLBusWrapper method.
• @rmac-sifive's recent changes to sifive-blocks are compatible with this PR
• Users can still make BaseSubsystem subclasses with statically defined subnetwork graphs and connect them to the dynamically-created topology. The dynamic topology can have nothing in it (though such a configuration will fail to build e.g. DefaultConfig in this repo). Set TLNetworkTopologyLocated("InSubsystem") => up(TLNetworkTopologyLocated("InSubsystem")).copy(connections = Nil) in your Config class to wire the standard bus locations together by hand.

Intended Usage

TLBusWrapperTopology is a serializable representation of a TileLink-based interconnect. A topology comprises instances of TLBusWrapper, a (previously existing) class which is an abstraction around a piece of interconnect fabric that provides some kind of connectivity between TileLink endpoints. There are currently several subtypes of TLBusWrapper, which each contain a particular crossbar, set of crossbars, chain of TL adapters, and/or internal devices. (In the future, some of these subtypes may just become parameterizations of a single base type.) TLBusWrappers have helper functions for attaching themselves to endpoint devices or to each other with configurable clock crossings.

The new TLBusWrapperTopology representation has two components, instances and connections.

Each instance is tuple that contains a Location and a TLBusWrapperInstantiationLike, where the former is a named reference to the latter, which can later be used when making connections between instances (or when attaching other things to the instance). A TLBusWrapperInstantiationLike is any class that knows how to instantiate a TLBusWrapper. The four previously-existing [Name]BusParams case classes all inherit from it and instantiate some particular TLBusWrapper subtype; the parameters are things like bus data width and parameters for internal devices and adapters' control interfaces.

Each connection is a triple that contains a pair of Locations and a TLBusWrapperConnectionLike, where the former pair identifies two previously instantiated TLBusWrappers, and the latter provides a method for actually programmatically connecting the instances. There is currently a single case class for describing parameterized connection between TLBusWrappers, named TLBusWrapperConnection.

A TLBusWrapperConnection contains the following serializable parameters:

• xType: ClockCrossingType: what type of TL clock crossing adapter to insert between the buses. The appropriate half of the crossing adapter ends up inside each bus.
• driveClockFromMaster: Option[Boolean]: if None, don't bind the bus's diplomatic clockGroupNode, otherwise have either the master or the slave bus bind the other one's clockGroupNode, assuming the inserted crossing type is not asynchronous.
• nodeBinding: NodeBinding: fine-grained control of multi-edge cardinality resolution for diplomatic bindings within the connection.
• flipRendering: Boolean: fine-grained control of the graphML rendering of the connection.
  There are also three programmatic escape hatches allowing users to inject custom node chains into the connection, or control the specific point within the TLBusWrapper that is connected to or from on either side.

Three example sub-topologies are defined, JustOneBusTopologyParams (sbus), HierarchicalBusTopologyParams (fbus, cbus, pbus) and CoherentBusTopologyParams (adds a coherence manager and mbus). These can be combined additively.

At the moment, the BaseSubsystem class looks up the TLNetworkTopologyLocated("InSubsystem") Field to discover its configurable topology. This call into the Parameters object could be replaced with reading in the topology from some serialized format. Given a topology, the subsystem first instantiates all instances, then connects all connections, passing in itself as the instantiating or connecting scope:

topology.foreach(_.instantiate(this))
topology.foreach(_.connect(this))

Related issues: #2034 #2025 #2078

Impact: API modification (but largely backward compatible)

Development Phase: implementation

[mwachs5]

Haven't fully wrapped my head around this. a little unclear which changes are things that are there to preserve backwards compatability and which are "the way to use now". Can these be marked a bit more clearly?

[rmac-sifive]

I reviewed for usability and backwards compatibility and spent less time on the internal dynamic implementation. LGTM!

[rmac-sifive]

🎉👍

[mwachs5]

@hcook can you take a look at #2410 and see if that approach is advised given that this change is coming? Would like to get those OM changes in before this change if there is huge incompatibility. Since that PR is basically assuming BaseSubsystem has 4 bus variables wondering how it will interact with this

[mwachs5]

is there a reason there is no sbus in this list?

[mwachs5]

oh NVM i see it is above

[DecodeTheEncoded]

is there some more detailed infomation I can get about the TLBusWrapper class?