signorm: directed RTLIL, incremental optimization

[widlarizer]

Signorm is an extended redesign of bufnorm (#3967) by @jix. It's a foundational change to the lack of well-defined directionality in general RTLIL. The idea behind signorm is described on an abstract level here. Its potential is hard to overstate since it allows for

• faster optimization
  • timestamps track when cells have been modified
  • work is only done on cells that have been modified and traversed from their fanout/fanin
• reduced setup work done by compatible passes to construct fanout/fanin traversal indices
  • also speeds up optimization passes
• simpler invariants for simpler code
• future opportunities
  • declarative rewrite patterns
  • smart preserving of source locations and other metadata through local rewrites

at no reduction in the support of tristate and conflicting drivers generally - the caveats that made it difficult in the past for Yosys to have a concept of direction in RTLIL.

What this means for you

The impact on synthesis and formal flow users will be purely performance improvements. The impact on synthesis and formal flow maintainers may be significant, meaning your custom Yosys pass sequencing may break, but we'll be ready to help. The impact on plugins is that they may break in silent ways.

Things that can break

• passes manipulating hierarchy (moving objects across modules or designs)
  • unlikely to harm downstream users
  • there may still be latent bugs or usability issues in the design pass for example
• passes assuming exact connectivity due to pass sequencing assumptions or calls to setPort
  • when setPort(sig) is called in signorm, and sig points to a specific wire, subsequent calls to getPort will return a privately named alias of that wire
  • this directly follows from how signorm provides a bufnorm-like invariant without introducing buffers
• passes that hold a SigMap while they violently modify the design
  • the auto connections to auto wires introduced in setPort calls have to be inserted into the SigMap either by calling getPort subsequently or by implementing an RTLIL::Monitor
• flows that play it fast and loose with module port info
  • currently, full blackboxes must be provided before designs get read in. Otherwise directions are unknown and $connect cells will be created to stitch together connected signals to instances of missing modules/blackboxes
  • before landing this, I'd like to add features to the hierarchy command to resolve $connect into appropriately ordered Module::connections to support cases where blackboxes/implementations are added after verilog is elaborated etc
  • internally, a lot of tests using equiv_opt and techmap had to be fixed this way
• techmap files that use positional arguments
  • this has been forbidden by docs/source/yosys_internals/techmap.rst already, but we had some like that anyway
  • also can be alternatively avoided with a hierarchy mode that matches what techmap needs

All passes that match the above have to be fixed - if they may ever run after any signorm-entering pass, which is almost always true. Even if signorm is exited, extra wire aliasing may break passes that silently rely on for example a conventional implicit "normalization" by opt_clean.

Passes that have to exit signorm

Some sketchy parts of the RTLIL interface that are incompatible with signorm (and shouldn't be made compatible with it)

• swap_names, rename, rewrite_sigspecs
  • theoretically they could be made compatible but probably at great costs to efficiency
• mutating the type of a cell, causing an intermediate use of ports that aren't compatible with the type at that instant (e.g. changing $dffe to $dff before removing the EN port)

Corner cases with special handling

• timing cells like $specify2 are excluded from the invariants. They look like hidden conflicting drivers
• the newly constructed $input_port cells have to be ignored in several places

What's next

This working is pending an intense, gradual rebase. Parallel opt_merge will probably be added have alongside the new incremental opt_merge_inc. There's also a couple things fixed in the wrong way as an initial step to passing tests. It's possible this will stay an independent branch for some time and that accumulates unique features.

The actual index

Signorm-specific RTLIL methods are implemented in kernel/rtlil_bufnorm.cc, so is the index:

struct RTLIL::SigNormIndex
{
	SigNormIndex(RTLIL::Module *module) : module(module) {}
	RTLIL::Module *module;

  // Deferred module level connections, not an actual persistent sigmap.
  // Module::connections() triggers Module::connections_
  // to be restored from Module::sig_norm_index.sigmap
	SigMap sigmap;
	size_t restored_connections = 0;

  // Fanout traversal
	dict<SigBit, pool<PortBit>> fanout;
  // Fanin traversal is implemented the same as in bufnorm,
  // as Wire::driverCell_, Wire::driverPort_

	pool<SigBit> newly_driven; // Deferred updates

  // Tracking for incremental optimization
	dict<Cell *, int> cell_timestamps;
	dict<int, pool<Cell *>> timestamp_cells;
	pool<Cell *> dirty;

Performance considerations

It seems that the signorm index doesn't differ from the existing indexing going on all over the place and has a lot of persistence potential. To make it persist sanely, it's necessary to replace swap_names and rename with more expensive object copying that doesn't violate some implicit object identity invariant (loosely "a thing at a pointers won't change what thing in the design it represents", like in #5692). That's still probably worth it and if not it's easier to tackle the overhead of that if traversal is abstracted which is what we're already doing.

Early incremental opt_merge experiments were promising but I haven't reproduced those yet or measured anything on this fixed patch.

[widlarizer]

Currently crashes in parallelized opt_clean (cells_all.cc:209), will investigate

[widlarizer]

opt_merge and opt_expr now have better performance than on main but the repeated construction of the signorm index adds a lot of overhead. This is because opt_dff and opt_clean leave signorm mode. Not sure yet if making opt_clean fully signorm-compatible is possible or a good idea, but at least, parts of it could be improved