Blog

Timing distribution for the High-Luminosity LHC

Written from my talk at the ATLAS Upgrade Week (May 2023), on behalf of the LAr calorimeter group.

Este post está disponível apenas em inglês.Cet article n'existe qu'en anglais.

Everything in a collider experiment runs on one clock: the LHC bunch clock at about 40 MHz, distributed from the accelerator to every board of the experiment over kilometres of fibre. When the clock arrives a little early or a little late, or a little differently after every reset, the effect ranges from nothing at all to a degraded energy measurement. In May 2023 I presented, for the ATLAS Liquid Argon calorimeter group, an analysis of what that means for the upgrade of the experiment for the High-Luminosity LHC. This post follows that talk.

Three ways for a clock to be wrong

It helps to separate three effects, because they have different causes and different cures.

Jitter is the continuous, random wobble of the clock edges around their average position. Measured over many cycles, it gives a distribution with a standard deviation, usually quoted in picoseconds rms.

Phase uncertainty is a jump. The average phase of the clock is stable, but after a reset of the serialisers and deserialisers along the path, it settles at a different value, in discrete steps. Each reset can give a different offset.

Temperature variation is a slow drift. Fibres and electronics change their delay with temperature, so the phase follows the temperature of the rooms and cables it passes through.

Three sketches of phase distributions: a single Gaussian for jitter, several Gaussians at discrete offsets for phase uncertainty, and a drifting distribution for temperature variation
Jitter, phase uncertainty and temperature variation (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

The clock today, and tomorrow

In the current system, the clock comes from the LHC control room, is received in a crate of timing modules, passes through the Central Trigger Processor and the ALTI modules that replaced older timing hardware for Run 3, and reaches the front-end and back-end crates of the calorimeter over optical links. For the analogue readout and trigger there are no FPGA transceivers in the chain, so phase uncertainty is neither measured nor noticed. The jitter of the recovered clock is similar to what it was in Runs 1 and 2, and within the budget of the front-end.

Diagram of the Phase-I clock distribution for the LAr calorimeter, from the LHC control room through timing crates, ALTI modules and optical links to front-end and back-end crates
The clock distribution of the current LAr system (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

For the High-Luminosity LHC the chain changes. New timing modules (LTI) feed the readout system (FELIX) and new LAr timing boards (LATS), and the links that carry the clock are FPGA transceivers everywhere. Part of the chain uses TCLink, an IP core developed at CERN that measures and compensates the phase of the clock in real time.

Diagram of the Phase-II clock distribution, with the LTI, FELIX, LATS and TCLink links from the LHC control room to the LAr front-end and back-end
The clock distribution for the High-Luminosity LHC, with FPGA transceivers and TCLink in the chain (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

For each of the three effects the talk asked two questions: do we care, and what can we do about it.

Jitter

Jitter matters for the transceivers themselves: the faster the links, the cleaner the reference clock they need. Commercial jitter cleaners now meet those requirements with margin.

For the calorimeter it matters more fundamentally. The clock that samples the analogue pulse in the ADC sets the time at which the pulse is measured, and jitter there limits the timing precision and the energy reconstruction. It cannot be compensated later. With 12-bit ADCs, the calorimeter needs 5 to 10 ps rms. The components are good enough: the radiation-tolerant lpGBT link chip has less than 5 ps rms, commercial phase-locked loops less than 100 fs rms, and the LATS recovered clock about 5 ps rms after its jitter cleaner. The total jitter of the chain still has to be measured once prototypes exist, but jitter is the one we care about most and also the one we can address.

Slide on jitter for digitisation and energy reconstruction, with the required and measured jitter values of the components
Jitter requirements and component performance for the calorimeter (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

Phase uncertainty

For digital data transfer, phase uncertainty is a solved problem. The receiving logic uses the LHC clock as a reference to find the frame that belongs to each bunch crossing, and as long as the combined clock and data uncertainty is smaller than one bunch-crossing period, it can be absorbed completely. That is what the synchronisation block of my PhD does.

Slide on phase uncertainty in digital data transfer, with a timing diagram of the system clock and the word data
For digital data, phase uncertainty below one bunch crossing can be absorbed completely (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

For digitisation it is different. If the sampling clock comes up at a different phase after a reset, the pulse is sampled at a different point. The online energy reconstruction uses filter coefficients chosen for a given phase, and the offline calibration combines data from many runs, because not every cell sees enough energy in a single run. If the phase changes from run to run, both become harder.

The good news is how far the technology has come. In the default mode of an FPGA transceiver, the one made for networking, the latency can change by a few nanoseconds after a reset. With the buffer-bypass configuration used in Phase I, which I described in the post on transceiver latency, that drops to tens of picoseconds for transmitter and receiver together. With the techniques used in TCLink it comes down to 10–15 ps, but only on AMD Xilinx UltraScale and Versal devices. The LATS uses an Intel FPGA, where about 85 ps peak to peak is expected for the transmitter alone.

Slide on phase uncertainty for digitisation, listing the reduction from nanoseconds to tens of picoseconds to 10-15 ps over three generations of transceiver configuration, and the expected values for AMD and Intel FPGAs
Phase uncertainty over three generations of transceiver configuration (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

Two ways to deal with it

There are two approaches. The first is to make each reset give the same phase, or as close as possible, by improving the circuits. That is what TCLink and related work do, and it is limited to the FPGAs that support it.

The second is to stop resetting. In what the talk called a non-disruptive clock mode, the timing distribution tree is configured once, after a power cycle or an intervention, and then left alone. Between data-taking runs, the parts that need reconfiguring, such as the readout partitions or the timing of the bunch counter reset, are reconfigured without touching the transceivers and clock cleaners that carry the clock. The phase then stays the same across many runs, and the same calibration constants can be used across runs. It works with any FPGA vendor. What it needs is a clear separation between the reset domains that may be touched every run and those that may only be touched after a power cycle.

Table comparing the disruptive clock mode and the non-disruptive clock mode in terms of FPGA feasibility, approach, current state and dependence between subsystems
The two approaches compared (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).
Two-column list of what is normally reconfigured every run and what is normally only reconfigured after a power cycle
Which parts of the system would be reconfigured every run, and which only after a power cycle (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

One complication is the clock itself. The LHC provides several clocks, and the experiment switches from an internal reference to the beam-synchronous clock when beams are injected. A switch of the input clock could make downstream links lose lock. Modern jitter cleaners can switch between two inputs that differ by up to ±500 ppm without a jump in output phase, so one possible solution is to let the Central Trigger Processor switch clocks locally in that way, instead of upstream. Whether that is enough to keep every link locked has to be tested.

Temperature

A change of 5 °C along 200 m of fibre moves the phase by about 60 ps. TCLink compensates such drifts up to the LATS: over a temperature cycle the phase stays within about 10 ps peak to peak, where without compensation it moves by 400 ps. The path from the LATS to the front-end cannot be compensated that way. In the LAr system so far we had not observed a noticeable change of clock phase with temperature.

Slide on temperature variation challenges with the measured phase sensitivity of fibres and the situation in the LAr system
Temperature effects on the clock phase (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).
Plot of phase versus time during a temperature cycle, with and without TCLink compensation
TCLink: about 10 ps peak to peak over a temperature cycle, instead of 400 ps (Timing distribution for the HL-LHC slides, ATLAS Upgrade Week, May 2023).

Where it landed

For the calorimeter, jitter is the ultimate requirement, and the chosen components meet it on paper. Phase uncertainty is a real concern, well characterised up to the central trigger and less so towards the calorimeter, and the non-disruptive clock mode is a way to live with it that does not depend on any one FPGA vendor. Temperature effects are compensated where TCLink can be used and are not yet visible where it cannot.

The broader lesson I took from preparing this talk is that clock quality is not one number. A link can have excellent jitter and still come up at a different phase every morning. Naming the three effects separately made it much easier to agree on which ones mattered and who could do something about them.