Simulation of Bufferless Networks using Ruby / Garnet

[d-kessel]

Hi everyone,
I’m currently working on simulating an IC developed in-house that incorporates a bufferless network-on-chip. We’d of course like to use Garnet/Ruby to simulate this chip, but as far as I understand, it (so far) only supports buffered networks.

So I wanted to start by asking in general whether my understanding is even correct, whether anyone else has already simulated bufferless networks, and whether there are any other developments in this direction that you might be aware of.

If not, I’d probably try to implement this type of network in Ruby myself.
I have atleast not found any recent issues or discussions regarding this.

In the following, I’ll briefly go over the points that, in my opinion, make simulation impossible in the current version of Garnet, as well as what I’ve found in the source code so far. If you have any comments or ideas on this, I’d be happy to take them into account.

---

Potential Issues in Garnet for Bufferless NoCs

Specifically, our chip is based on a modified version of OpenNoC (https://github.com/kuladeepsaireddy/OpenNoc and https://doi.org/10.1109/LES.2019.2905019).

Router Latency

In Garnet (specifically in src/mem/ruby/network/garnet/Router.cc:85), it is described that the router latency is at least 3 cycles:

credit traversal (1 cycle) + SA (1 cycle) + link traversal (1 cycle)

However, this is not realistic in the context of unbiffered networks, since they cannot buffer messages or flits. Therefore, they must forward their flits to the next router every cycle, which is not possible here (unless maybe if the clock is artificially run at triple speed in Garnet).

But this also leads to the problem that there are actually no separate buffers for receiving and transmitting data. These are present, at most, in the network interfaces, but not in the routers.

Routing and Deflection

As described in the paper, deflection is used in the bufferless NoC. This means that if two packets arrive at the router simultaneously and are destined for the same destination, one of the packets is sent to the destination, while the other is routed to the “wrong” output. In this case the data must then travel an additional path (e.g., an extra loop in a Taurus network) since it can't be buffered for the next cycle.

This seems very complicated to implement. In InputUnit.cc, the wakeup function is implemented such that the path is calculated immediately upon receipt of the first flit, using the RoutingUnit for this purpose.

if ((t_flit->get_type() == HEAD_) || (t_flit->get_type() == HEAD_TAIL_)) {
...
int outport = m_router->route_compute(t_flit->get_route(), m_id, m_direction);
grant_outport(vc, outport);
}

This means that the path:

1. is the same for all flits, which is not the case in the bufferless NoC, and
2. the prioritization of the outputs would have to be random.

Suppose the InputUnit at the router is first called from the Processing Element (PE); in that case, the route from the PE would already be calculated and the ideal Route would be taken. If the InputUnit is then processed from one of the other incoming paths, it might expect the same output—and would actually receive it—since only the PE can be prevented from sending the data. In that case, an invalid state or data loss would occur, which, however, would not actually happen in real-world routing.

In short: Routing can only occur once the state of all incoming data lines is determined, which is not yet the case. Furthermore, the routing may change during the course of the message, which could also lead to out-of-order flits received at the destination PE / NetworkInterface.

---

Implementation Considerations

With a potential implementation in mind, it would be interesting to know whether others would be interested in this network architecture.

One of the main questions would then be whether it would be better to integrate these options into Garnet. An implementation without breaking changes would likely introduce a large number of configurations that would be invalid.
For example, trying to initialize a Network with more than one buffer, one channel, etc., would mostlikely result in invalid configurations, which would need to be handled via the initialization in Ruby (and in the CPP side).

Alternatively, of course, this could be viewed as a standalone version of network.
In that case, it might be interesting to have a kind of adapter that can combine buffered and bufferless networks.

I’d be interested in hearing your opinions and/or use cases, if any exist.