README.md

USPAS LLRF System Design

[[TOC]]

Architecture

System Architecture

DSP Architecture

The core tasks of any LLRF controls are:

• RX: Measure RF signal:
  • Frequency conversion to base-band;
  • Diagnostics and record waveforms;
• Control:
  • Feedback control
  • Feed-forward control
  • System identification
  • Diagnostics and record waveforms;
• TX: Drive RF signal:
  • Frequency conversion from base-band;
  • Diagnostics and record waveforms;
• Trigger and interlocks

These tasks are abstracted in the following diagram, which highlights the core elements of each building blocks. The same concept and architecture applies to other platforms such as the [RFSoC based LLRF System design at ALS][https://arxiv.org/abs/2510.13192].

Development tools

As part of the full-stack open source firmware development in Berkeley Lab, we leverage a collection of open source tools including:

• Icarus Verilog: tested version v12.0.
• Verilator: tested version 5.038.
• cocotb: tested version 2.0.1.
• yosys

and Jupyter notebooks for demonstration and documentation.

• For conda users, a python virtual environment can be created as:

  conda create -n uspas_llrf python=3.13
  conda activate uspas_llrf
  pip install -e .

• All tools are native to Debian Trixie, where we used it in the gitlab Continuous Integration, as defined .gitlab-ci.yml, including steps of:

  • Coding style sanity checking using flake8;
  • cocotb LLRF DSP behavioral verification;
  • Clock domain crossing validation;
  • Bit-stream synthesize for all supported variants of applications;
  • Hardware in-the-loop testing;

Firmware and Gateware

Applications and settings

The following of pre-defined LLRF applications are supported, and their frequency settings are described in README.md.

• USPAS: For USPAS LLRF class taught in 2023.
• ALSU: For LBNL ALS-U AR LLRF system.
• LEMP: SLAC Linac Electronics Modernization Project
• AWA: Argonne AWA facility

Detailed description of settings:

• LLRF DSP: All configurations are contained in uspas_llrf/settings.json.
• Board Support: For each application, customized hardware settings and LLRF configurations can be found in soc/<design>/ (for example soc/uspas), where each application's configuration files for FPGA carrier and digitizer (init_marble.c, init_zest.c) are located.

System-On-Chip architecture

A RISC-V soft core PicoRV32 is used for peripheral control, booting and diagnostics. We leverage the cross-compiler tool and common modules described in Berkeley Lab's Bedrock repository.

The design can be found in soc/common, where:

• Open source RTL simulation: See soc/common/sim, for the booting process, using purely icarus verilog.
• Full RTL simulation: See soc/common/top_sim. This simulation process requires Xilinx vivado command tools like xvlog and xelab, for building and execution respectively. This allows the use of UNISIM models provided by vivado's installation package, for a complete behavioral verification of Xilinx primitives including XADC, ISERDES, etc, which is needed for the development of board support package with LVDS digitizer interface.
• Hardware test: See soc/common/synth, where two functions are implemented:
  • Synthesis: A minimal structured bitstream file containing only the soft core and essential peripherals.
  • Boot-loading: The CPU program memory in the deployed production bitstream can be updated using a boot-loading process thanks to a built-in bootloader. This is a well known technique in embedded system designs, and provides flexible and quick iterations for development / troubleshooting. Details and examples can be found in soc/common/synth/README.md.

Top level synthesize

We use LBNL Bedrock's Makefile based building system to find dependencies and synthesize the bitstream file in top/<design>/ (for example top/uspas), with shared rules and sources in top/common. The top level RTL marble_zest_top.v assembles the soft core, board support package and LLRF DSP together.

Board Support Package (BSP)

FPGA carrier (Marble)

See marble_bsp/marble_bsp.v, which features:

• LBNL local bus control interface;
• LBNL Gigabit Ethernet UDP engine (Packet Badger);
• LBNL 8b10b MRF timing event receiver (EVR);
• LBNL Marble micro-controller (MMC) mail-box interface;
• External trigger logic;

Digitizer (Zest)

See bedrock/board_support/zest_soc.

Clocking

As shown in the following diagram of the digitizer (Zest) board support, the clock distribution chip LMK01801 receives an external reference clock, and the outputs of two divider groups drives ADC AD9563 and DAC AD9781 respectively, where the DAC sampling clock is double of the ADC sampling clock, which is the same as the DSP clock.

$$ f_\text{dac_clk} = 2 f_\text{adc_clk} = 2 f_\text{dsp_clk} $$

Digital Signal Processing (DSP)

Numerical Models for simulation

A collection of LLRF DSP numerical models can be found in uspas_llrf/model/llrf_dsp.py, which is shared among all simulations.

The complete feedback controller is modeled, and it can be used for cocotb simulation with cavity emulators, whose parameters are defined in uspas_llrf/model/cavity.json. The detailed cavity model co-simulation with discrete signal process is explained in doc/lti.ipynb.

IQ representation conventions

There are two conventions for IQ decomposition of a RF signal $y$ at carrier frequency $\omega$:

1. Positive carrier frequency:

   As described in Wikipedia,

   $$ \begin{align*} y(t) &= (I + jQ) \cdot e^{j\omega t} \ \Re(y(t)) &= I\cos(\omega t) - Q\sin(\omega t) \end{align*} $$

   This convention is used in this repository.

2. Negative carrier frequency:

   $$ \begin{align*} y(t) &= (I + jQ) \cdot e^{j-\omega t} \ \Re(y(t)) &= I\cos(\omega t) + Q\sin(\omega t) \end{align*} $$

   This convention is used in bedrock/dsp RTL modules.

Digital Direct Synthesis (DDS)

Also known as NCO, it is used to generate a pair of sinusoidal signals at a single frequency, with a known starting phase. The DSP implementation is shown in the following figure. It is consisted of a phase accumulator and a CORDIC for conversion from Polar to Rectangular coordinate.

• RTL implementation

  See llrf_dsp/dds.v.

  

• Simulation

  See llrf_dsp/tests/dds.

Digital Down-Conversion (DDC)

For high precision digitization, Non-IQ direct digital down-conversion is used to avoid aliasing.

With the normalized ADC IF frequency $\omega_d = 2\pi\frac{f_\text{IF_ADC}}{f_\text{S}}$, the DDC NCO provides LO data streams of $\cos(n\omega_d)$ and $\sin(n\omega_d)$. Two measured successive samples are:

$$ \begin{pmatrix} x_{n-1} \\ x \end{pmatrix} = \begin{pmatrix} \cos((n-1)\omega_d) & -\sin((n-1)\omega_d) \\ \cos(n\omega_d) & -\sin(n\omega_d) \end{pmatrix} \begin{pmatrix} I \\ Q \end{pmatrix} $$

Solve $I$ and $Q$ using inverse matrix:

$$ \sin(\omega_d) \begin{pmatrix} I \\ Q \end{pmatrix} =\begin{pmatrix} \sin(n\omega_d) & -\sin((n-1)\omega_d) \\ \cos(n\omega_d) & -\cos((n-1)\omega_d) \end{pmatrix} \begin{pmatrix} x_{n-1} \\ x \end{pmatrix} $$

• RTL implementation

  RTL implementation is in noniq_ddc.v, where a serialized stream of IQ data is generated, with a gain of $\sin(\omega_d)$.

  An interpolation module fiq_interp.v is used to convert to parallel I and Q sample streams. A DC-blocking module fwashout.v is inserted before the noniq_ddc.v. The full DDC is packaged in ddc.v.

  

• Simulation

  See llrf_dsp/tests/ddc.

Digital Up-Conversion (DUC)

A generic digital up-conversion scheme is implemented, in the dac_clk domain.

With the normalized DAC IF frequency $\omega = 2\pi\frac{f_\text{IF_DAC}}{f_\text{S}}$ and phase offset $\theta$, given a complex base band signal $e^{j\alpha}= I + jQ$, the up-converted signal $y$ is:

$$ \begin{align*} y &= e^{j\alpha} \cdot e^{j(\omega n + \theta)} \\ &= (I + jQ) \cdot e^{j(\omega n + \theta)} \\ &= I\cos(\omega n + \theta) - Q\sin(\omega n + \theta) + j\left( Q\cos(\omega n + \theta) + I\sin(\omega n + \theta) \right) \\ \Re(y) &= I\cos(\omega n + \theta) - Q\sin(\omega n + \theta) \\ \Im(y) &= Q\cos(\omega n + \theta) + I\sin(\omega n + \theta) \end{align*} $$

Equation in matrix form:

$$ \begin{pmatrix} I_y \\ Q_y \end{pmatrix} =\begin{pmatrix} I & -Q \\ Q & I \end{pmatrix} \begin{pmatrix} \cos(\omega n + \theta) \\ \sin(\omega n + \theta) \end{pmatrix} $$

To avoid aliasing, condition $\omega \in (-\pi, \pi)$ is due to the Nyquist–Shannon sampling theorem.

When operating in the under-sampling scheme, the signal location at the first Nyquist zone must be calculated to derive the effective NCO frequency value. The following figure illustrates two cases of modulation at carrier frequency $f_c$ with sampling frequency $f_s$:

• First Nyquist zone: $0 < f_c < \frac{f_s}{2}$, see case (b).

• Second Nyquist zone: $\frac{f_s}{2} < f_c < f_s$, see case (c). The NCO frequency is set to be $-(f_s - f_c)$ or $-f_c$. This flip of sign is known as the spectral inversion due to frequency folding around the Nyquist frequency.

• RTL implementation

  In practice, this spectral inversion is implemented by simply flipping the sign of the $\sin(\omega n + \theta)$ for the LO to rotate in a counter clock wise direction.

  Many applications like LEMP and PIP-II LLRF use single-side-band modulation for analog up conversion from IF to RF. When this modulation is configured as the lower-side band modulation, it will result in a phase sign flip between IF and RF. This can be compensated by an additional register tx_afe_spectral_flip. To gurantee the correct phase sign including both digital and analog upconversion, the combined spectral flipping is applied by an XOR between the reigister duc_spectral_flip (digital) and tx_afe_spectral_flip (analog).

  

• Simulation

  See llrf_dsp/tests/duc.

This approach is consistent with the NCO Modulator in many RF-DACs such as the AD9174 (Figure 79), and the AMD RFSoC with its NCO Setting, where a standalone NCO with configurable frequency and phase is instantiated, allowing 1st or 2nd Nyquist zone modulation. For 2nd Nyquist zone operation, most DACs has a Mix-Mode available to increase the amplitude response.

Feedback controller

As a classical PI controller, it is designed to operate in base band with a single, complex input and output signal. There are two parallel PI controllers for amplitude and phase control, respectively. The phase wrapping is taken care of in the difference calculation. A pair of CORDIC are used to convert the complex signal to between rectangular and polar representation, where phase offsets can be added optionally.

• RTL implementation

  See llrf_dsp/dsp_core.v, as shown in the following diagram:

  

• Simulation

  See llrf_dsp/tests/llrf_dsp.

  cocotb tests will go through all test cases for various frequency settings, each has a test of RX, open loop and close loop responses. The results are integrated as part of the gitlab Continuous Integration, where the configuration can be found at .gitlab-ci.cml.

Waveform recorder

Three functions are integrated in the llrf_dsp/cic_waves.v:

• Dynamic waveform with a run-time configurable CIC filter and channel selector. The decimation factor is the production of cic_base_period and wave_samp_per registers. The CIC filter response and calibration factors are calculated at the CICWaveRecorder class in llrf_dsp.py.
• A circular buffer that captures the CIC waveform, associated with diagnostic data including minimum and maximum value of each channel, 64 bit timestamp and fault status. Trigger logic, pre and post trigger buffer, fault freezing features are included.
• Sharing the same integrator, a fixed decimation filtered datastream of all channels are generated for the use of fast interlock purpose.

LLRF Shell

• RTL implementation

  Putting things together, we can form a complete chain of digital frequency conversion with two independent feedback control loops with flexible ADC / DAC channel mapping, with independent ADC / DAC IF frequencies.

  

• Simulation

  See llrf_dsp/tests/llrf_shell.

  A complete instantiation of llrf_shell.v and its pre-processed application settings is tested under cocotb verification through the LBNL Local Bus control interface, which include:

  • A test signal driving an ADC channel with known frequency, amplitude and phase, for testing RX path including down-conversion;
  • A looped-back signal from one DAC channel to an ADC channel, for testing TX path including up-conversion;
  • A looped-back signal from the other DAC channel to an ADC channel, for testing feedback loop settings and closed-loop response;
  • CIC filtered waveform acquisition for multi-channel signals including 2 DACs and 10 ADCs, with configurable decimation factors;
  • Wide bandwidth IQ waveform acquisition for each DAC and ADC base-band signal with sample-to-sample resolution;
  • Fast interlock protection logic and RF permit latching;
  • Trigger logic;

• Calibration

  The numerical DSP transfer function of each building element in llrf_shell.v are modeled in uspas_llrf/model/llrf_dsp.py.

  The DSP models are used in both cocotb simulaiton, and the Python driver in uspas_llrf//app, where the calibration factors are illustrated in the following diagram:

  

Software

Python IO

An example python class for packaging the LLRF application can be found in uspas_llrf//app, where a few Jupyter notebook examples are provided as reference use cases.

EPICS IOC

See LBNL FEED.