[RFC] New EEM: Stamper (TDC)

[gkasprow]

From @dhslichter on September 25, 2017 22:40

NIST has some interest in an EEM TDC module (currently code-named Stamper), and I'd be interested in getting some draft specs based on potential use cases. Some questions:

• Who else is interested in a TDC module currently?
• What sort of jitter/channel count/overall event rate is currently targeted?
• Are there preferences for an FPGA-based TDC vs. a commercial TDC ASIC with a Kasli as readout? The latter would probably be considerably simpler/cheaper to implement.

@hartytp is this an Oxford push currently?

Copied from original issue: sinara-hw/sinara#326

[gkasprow]

From @cjbe on September 26, 2017 9:55

We at Oxford are interested, but not pushing this yet.

For our use case, we need at least 2 channels. We don't have any strong event rate requirements, but we do want a low latency (<<100ns) way of determining that there was an event. For example, sending a copy of the buffered input signal down an LVDS pair to the master FPGA.

Our desired jitter is ideally ~<40ps FWHM, but we are happy to compromise on this.

We don't have any preference between commercial ASICs or FPGA based systems.

[gkasprow]

From @hartytp on September 26, 2017 9:57

ACK. My guess that we'd go for a similar front panel to Urukul, giving 4 SMA inputs and 1 SMA clock input.

@dhslichter Are you considering funding this?

[gkasprow]

As a reference, here is successful board I designed for CERN a few years ago. Here are some measurements.

[gkasprow]

From @dhslichter on September 27, 2017 22:23

@gkasprow the board you have sent looks like the kind of thing I am imagining, at least for now. In our case, it would be something simpler in various ways, since we'd just look for timestamping, and I think having buffered copies of the inputs sent via LVDS to the FPGA for the low-latency event trigger mentioned by @cjbe would be sensible. There has since been a second generation of ACAM TDC chips with better performance as well, readout over SPI at ~1 Mevent per second or so.

[gkasprow]

From @dhslichter on September 27, 2017 22:28

@hartytp this is still very much in the brainstorm phase out here, but funding might be forthcoming from NIST in the medium term, depending on how things prioritize here.

[gkasprow]

My group is participating in the NICA Experiment in Dubna. We will build the whole processing chain starting from SiPM detectors, frontend, HV supply, temperature compensation, transmission, processing.
It will be based on AFCK boards and probably Metlino in the further stage of the project.
The signal from SiPM will be transferred using SAS cables and enter FMC modules.
Each FMC module will have 16 channels.
Each channel is equipped with:

• TDC-GPX2 channel
• Constant fraction discriminator
• 80MS/s ADC with low pass filter and driver.
  FMC has two SAS input connectors. Such a solution provides sufficient bandwidth needed to achieve tens of ps timing resolution.
  ADC is needed to estimate particle energy.
  The FMC will be designed in such a way to be compatible with both LPC and HPC FMC sockets. In LPC mode only TDCs will work, when plugged to HPC, ADCs will be available.
  And the questions are:

1. Would such TDC module be useful with Sayma AMC?
2. Would the community prefer MCX connectors than SAS?
3. What about building simple EEM FMC carrier with 4 or 5 EEM connectors? With 4 EEMs TDC would work, fifth is needed to fully support all LVDS lanes and M2C clocks
4. Or maybe put small Artix chip on it and make the FMC carrier in "proper" way ?
5. Are SiPM detectors used in quantum labs?

[gkasprow]

Another option is to build simple EEM with just TDC chip and dual EEM.
@dhslichter @hartytp @cjbe any thoughts?

[hartytp]

A simple EEM TDC would be great! Ideally, I'd like to see something that can take an analog input, put it through a fast, low-noise comparator and then feed the signal both to the TDC chip and to the EEM connector. That way one can use the LVDS signal as a trigger as well as getting the time stamp from the TDC.

Edit: i.e. what was described at the beginning of this issue...

[gkasprow]

What about EEM FMC carrier? Would it make any use case for you?

[hartytp]

Might be useful as a separate project, although I'd have to give that some thought. For the TDC I'd definitely prefer a simple TDC EEM to an FMC.

[sbourdeauducq]

The signal from SiPM will be transferred using SAS cables and enter FMC modules.

Serial Attached SCSI?

[gkasprow]

Serial Attached SCSI?

Exactly. Each cable has 8 pairs rated for 6 or 12Gbs, they are individually shielded.
There are 3M twinaxial versions that work up to 20GHz
Cables are damn cheap. A few days ago I purchased new ones with a 1Y guarantee for 10eur/pc.
They are available in various lengths, starting from 30cm up to 10m.
I plan to use them to transmit analog signals with >1GHz bandwidth.

[gkasprow]

We plan to use them in Shuttler as well. We will start with crosstalk measurement first ;)

[dhslichter]

My preference is definitely for an EEM, especially one where the interface could be made (with reasonable adapters) ARTIQ-agnostic. Most of our applications require a few (1-4) channels, thus one card, and users would be price-conscious and complexity-sensitive, both of which are better with EEM vs FMC + FPGA carrier card. Obviously for particle physics or something like that people want hundreds of channels and then FMC probably makes more sense.

One word of caution: some other users at NIST have worked with the TDC-GPX2 and they have complained of dead times and dropped counts -- in other words, the claim is that one can't endlessly stream timestamps out from the chip, which is a problem for many applications. I have an eval board and might set someone up to put it through its paces a bit. It would be unfortunate to build a major design based on this and then discover an important flaw.

[gkasprow]

@dhslichter can you point me to some more details about the TDC-GPX2 chip issues?
I've ordered a devkit so I will b able to check its limitations. You need to periodically reset its Index counter, maybe that was an issue? There is another similar but 2-channel chip from AMS that maybe has some issues fixed.

[gkasprow]

The FMC board is nearly finished. We got funding so plan to produce it soon. I will make a simple code based on FORTH CPU to characterize it briefly.

[dtcallcock]

I just found out Brian Smith's Quantum Optics Group here at Oregon is working on something similar.

He's basing it off a couple of papers:
https://arxiv.org/abs/1303.6840
https://dl.acm.org/citation.cfm?id=3322482

The former paper is by @sbourdeauducq amusingly.

I asked him to weigh in if Sinara is something he's interested in.

Apparently there are quite a few commercial solutions out there (list below), but the per-channel price is comedy. I think if Sinara could cover these use cases there could be lot of new users.

PicoQuant - Hydraharp ~$50k - https://www.picoquant.com/products/category/tcspc-and-time-tagging-modules/hydraharp-400-multichannel-picosecond-event-timer-tcspc-module

idQuantique - ID900 ~$15k - https://www.idquantique.com/single-photon-systems/products/id900-time-controller/

quTools - quTau ~$15k - https://www.qutools.com/time-tagger-time-to-digital-converter/ (not sure what their new product the “quTAG” goes for probably more on the order of $50k)

Swabian Instruments - https://www.swabianinstruments.com/time-tagger/ (not sure how much these cost)

[gkasprow]

@dtcallcock The 16-channel FMC with TDC and ADC for NICA MCORD experiment is leaving the production phase and we will test it soon. We plan to use MiSOC and AFCK board for testing purposes. Ethernet would be the interface on the other side. If the timing resolution of the TDC is fine and we do not observe issues, we could consider this solution for Stamper.
What are the typical conditions we should test the TDC?

[gkasprow]

Swabian Instruments do a great job. I visited their lab a year ago. The only issue is latency. They do all post-processing in software, on PC. So it takes hundreds of ms to get the results.

[dhslichter]

https://arxiv.org/abs/1303.6840 is one of the reasons @sbourdeauducq was hired to do the initial ARTIQ development, FYI...

[gkasprow]

We can take let's say 4 smallest Spartan6 devices, pack them to EEM and implement TDC in gateware. In this way, we get 4 identical channels. AFAIK placing multiple TDC cores in the same FPGA is not trivial, as well as TDC together with complex logic. And for this reason, implementing TDC in Kasli would be a nightmare. Such a solution would be more expensive than TDC-GPX2 which is 40EUR/4 channels. Four XC6SLX9-2TQG144C would cost 60EUR but would give shorter latency.
Maybe it's worth playing with ICE40? @sbourdeauducq what do you think?

[dhslichter]

@gkasprow as mentioned above one of the questions with the TDC-GPX2 is throughput for constantly streamed timestamps -- thus there could be good use cases potentially like discussed above. IIRC from talking to @sbourdeauducq the design he made doesn't scale well to multiple TDCs on a single Spartan, as you were saying. I don't think that people will be too worried about 20 EUR/channel price differences here....

[gkasprow]

I'm waiting for PCBs with TDC-GPX2. Within a few weeks, we should get some measurements.
@kaolpr is preparing a gateware that will read data from this TDC. We can add some logic that will make statistics and detect anomalies.

[sbourdeauducq]

And for this reason, implementing TDC in Kasli would be a nightmare.

I don't know if it's a "nightmare", but it's high-risk and needs to be carefully prototyped and studied along with some attempts to reduce noise by floorplanning. The Spartan-6 TDC demo had a small LM32 SoC along with the TDC, so you can definitely put some extra logic there. Having external FPGAs like ice40 with clean power supplies gives us another tool to fight noise and avoid problems related to latency, throughput, and frustration with proprietary ASIC interfaces.

[gkasprow]

Update: The student who designed Mirny for his engineering thesis will build Stamper for his master thesis. There must be some research part so that he will compare two TDC methods. One is traditional, the second one is based on GTP/GTX in free-running mode (no CDR).
He will do it on a devkit, then EEM board will be designed.

[gkasprow]

Moreover, I recently got funding for that, so w have enough motivation:)

[jbqubit]

Awesome! Congratulations.

[dtcallcock]

This got made by @gkasprow crew: https://github.com/elhep/FMC_ADC100M_10B_TDC_16cha/wiki

[gkasprow]

In the meantime, we continue the TDC approach using MGTs. I have reports in Polish :)