A Simple Timestamping Data Acquisition System for ToF-ERDA

A new data acquisition system, ToF-DAQ, has been developed for a ToF-ERDA telescope and other ToF-E and ToF-ToF measurement systems. ToF-DAQ combines an analogue electronics front-end to asynchronous time stamped data acquisition by means of a FPGA device. Coincidences are sought solely in software based on the timestamps. Timestamping oﬀers more options for data analysis as coincidence events can be built also in oﬄine analysis. The system utilises a National Instruments R-series FPGA device and a Windows PC as a host computer. Both the FPGA code and the host software were developed using the National Instruments LabVIEW graphical programming environment. Up to eight NIM ADCs can be handled by a single FPGA. The host computer and the FPGA can process total continuous count rates of over 750000 counts/s with a timestamping resolution of 8.33 ns.


Introduction
Time-of-Flight Elastic Recoil Detection Analysis (ToF-ERDA) [1] and other similar Time-of-Flight (ToF) measurement set-ups, such as ToF-ToF [2] or ToF-E systems require a multi parameter data acquisition system capable of recording coincident ToF-and energy information. In the most simplest form a coincidence unit is used for generating a common gate signal for the pulse height ADCs associated with ToF and energy channels. The coincident data is then collected directly into a 2-dimensional histogram. More advanced systems save the coincidence data in a list-mode where coincident ToF and energy information pairs are written into a file sequentially [3]. Commercial multi parameter data acquisition systems that can directly recognize coincident data from several detector channels are also used [4].
Total Data Readout (TDR) is a triggerless data acquisition method developed for recoil decay tagging spectrometer located at the Accelerator Laboratory of the University of Jyväskylä in Jyväskylä, Finland. With the TDR method all the detector channels are read continuously and independently, and each data word is saved with a corresponding timestamp. Coincidence events are built solely in software based on the timestamps. [5] Therefore, even very complicated trigger conditions can easily be set by the software during data acquisition or post-processing. Modern computers provide sufficient computing power and disk space not only for saving the timestamped data from numerous independent detector channels but also for online multi parameter histogramming.
In this paper we describe a TDR-derived data acquisition system, ToF-DAQ, that has been developed for ToF measurement systems, especially for the recently built ToF-ERDA telescope accompanying the Pelletron accelerator in Jyväskylä [6][7][8]. ToF-DAQ combines an analogue electronics front-end to the TDR method by using a commercial Field Programmable Gate Array (FPGA) device that is placed between the pulse height ADCs and the data acquisition computer. The FPGA reads conversion values from the ADC of each detector channel asynchronously, timestamps the data and transfers the data words to the host computer as a single data stream.
By comparing a traditional hardware triggered multi parameter data acquisition to the TDR method, several improvements can be recognized. For example, with multichannel hard wired systems, setting up the data acquisition and adjusting the gate conditions must be done carefully to avoid data losses. Readjustment of detector electronics will probably result on a need to readjust the gating also. Moreover, with hardwired systems the adjustment process must be done solely before measurement. If the data is recorded without any associated timing information, possibilities to re-sort the data after the acquisition are limited. The TDR method overcomes these issues since no data is rejected during the acquisition. Also, the TDR method simplifies the data acquisition set-up, because the amount of analogue electronics is reduced. For the ToF-ERDA measurements this approach also enables, like with list-mode data acquisition, investigation of dynamic processes [4] -even without a prior knowledge of the existence of such. In addition to that, one can during the offline analysis, for example, fine-tune the coincidence search parameters to minimize the unwanted background (see chapter 4.3).

Hardware
A field-programmable gate array (FPGA) is an integrated circuit in which the interconnections between the logic blocks are reprogrammable by the end user. Unlike microprocessors that follow Von Neumann's architecture to execute tasks sequentially, FPGAs provide true parallel processing as separate groups of logic blocks can be reserved for each task. Therefore, FPGAs are ideal for timestamped data acquisition as one can easily implement the asynchronous and independent handling of multiple detector channels.
ToF-DAQ uses a National Instruments R-series device as a FPGA module and a standard PC with the Windows 7 operating system as a host computer. The FPGA card is housed in a National Instruments' 5-slot PXI-1033 PXI chassis. A built-in 110 MB/s MXI-Express interface links the chassis to the host computer. Currently models PXI-7811R, PXI-7813R, and PXI-7852R have been used as the FPGA-device, but any of the R-series FPGA modules can be applied to ToF-DAQ, including the PCI versions that directly connects to the PCI slot of the PC. NIM module pulse height ADCs are interfaced via TTL-level digital input/output (DIO) lines of the FPGA card. Depending on the amount of logical resources and DIO lines available on the FPGA, a single FPGA can handle 4-8 detector channels. Since all National Instruments FPGA modules share the same basic resources and connector pin-outs for DIO lines, porting the FPGA code to a different module requires very little effort, even if the actual FPGA chip on the card is a different species.
As the DIO lines of National Instruments R-series FPGA devices are directly connected to the FPGA chip, the lines can be read and written during a single clock cycle. Hence, the timestamping resolution that can be achieved equals the length of the FPGA clock cycle. The nominal clock rate of the devices is 40 MHz, but the rate can be increased by multiples of that. The maximum achievable clock rate depends on the actual device to be used and the complexity of the FPGA code to be compiled. The current version of ToF-DAQ's FPGA code can be compiled to run at frequencies of 40, 80 or 120 MHz. As a result, nominal timestamping resolution of the system is 25, 12.5 or 8.33 ns, respectively. For the ToF-ERDA a sub-nanosecond time resolution is needed for the ToF measurement [6]. Therefore, the timestamps can only be used for searching the coincident data from different detector channels. One must either use a time-to-amplitude converter (TAC) accompanying an ADC or a time-to-digital converter (TDC) to obtain the ToF.
A simple three-piece adapter block was designed for parallel bus connection between ADCs and the FPGAcard. The adapter is the only purpose-made component of the system. The main board accepts two ADC-specific daughter boards. A photograph of the adapter block configuration is shown in Fig. 1.
Several different NIM module pulse height ADCs have been used with the system. Basically any ADC with a parallel bus connection is appropriate. Only minor changes in data readout sequence in the FPGA code and an ADCspecific daughter board for a proper signal routing may be required. Currently the FPGA code supports 13-bit Fast-ComTec ADCs and Silena-compatible ADCs. User can select the ADC type in use from the host interface, i.e. no recompile of the FPGA code is needed.

Software
LabVIEW (Laboratory Virtual Engineering Workbench) is a programming environment developed by National Instruments for visual dataflow programming language G. LabVIEW programs and subroutines are called virtual instruments (VIs). Each VI consists of three components: a front panel for the user interface, a block diagram for the actual graphical code and a connector pane that represents the VI in the block diagram of a calling VI. As the execution order of functions in LabVIEW is based on the dataflow instead of their order of appearance, LabVIEW is well-suited for parallel programming. Not only desktop computers can nowadays be programmed with LabVIEW, but also other platforms like FPGAs and PLCs. Both the host interface of ToF-DAQ and the FPGA code were developed using LabVIEW.   of the FPGA code is also essential. In addition, speed optimization of the FPGA-code is also needed for exceeding the nominal 40 MHz clock rate of the device.
ToF-DAQ provides two options for the timestamping of the data. For applications with only moderate requirements for timestamping resolution, the ADC's DataReady signal indicating the ADC has finished the conversion and the data is available to be read can be used as a trigger signal for timestamping. In this mode variations on the time-to-peak of detector signals and other variables that can affect ADC processing time may induce inaccuracy to the timestamping. For applications with higher precision requirements for timestamping, one can use for example a CFD (Constant Fraction Discriminator) to generate a logical signal from a detector signal and use that as a timing signal for timestamping. With this option, the timestamping resolution is independent of ADC processing time variations; the resolution depends only on the clock rate of the FPGA chip, if the CFD is properly adjusted. Use of the external timing signal is essential if Wilkinson-type ADCs with variable conversion times are used.
In the FPGA code each detector channel is handled by two subVIs (i.e. subroutines) running in parallel. Dead time recording and enabling/disabling of the ADC are handled by sub DT RunStop.vi and sub Read ADC.vi is responsible for more time-critical actions: data readout and transfer to host computer. Fig. 2 shows a schematic overview of the data acquisition system.
In sub Read ADC.vi the data readout procedure starts when sub Wait Trigger.vi acquires a 64-bit timestamp based on the leading edge of the trigger signal (either ADC's DataReady-signal or a signal from a CFD). After that sub Read Data.vi reads the conversion value from the ADC and sub Write FIFO.vi sends the data to the host com-puter via the DMA-FIFO (Direct Memory Access-First In-First Out) channel. The block diagram of sub Read ADC.vi is shown in Fig. 3.
The data word transferred to the host computer consists of an ADC ID number, a conversion value and a timestamp. Each of these values are handled as 64-bit unsigned integers. As a result, the total size of the data word is 192 bits.
Since all detector channels share the same DMA FIFOchannel, arbitration for access to DMA FIFO is needed to prevent data corruption. This is done by determining sub Write FIFO.vi as non-reentrant, i.e. simultaneous calls by different instances are not allowed. The LabVIEW FPGA compiler generates by itself the actual arbitration code to enable the non-reentrancy. The DMA-FIFO channel not only transfers the data bits from the FPGA to the host application but also acts as a data buffer against data bursts and latencies of the host computer's operation system.
As the DMA-FIFO is a shared resource among all the detector channels it is therefore possible that sub Write FIFO.vi is not always immediately available after the data is read from an ADC. To ensure that waiting of sub Write FIFO.vi to be available will not block acquiring new data from the ADC, the data is not written to the DMA-FIFO directly after successful data readout, but during next iteration of sub Read ADC.vi. Thus, sub Read ADC.vi can simultaneously accept new data and write the previously acquired data to the DMA-FIFO. Uninitialised shift registers are used for storing the acquired data between iterations. In Fig. 3 shift registers are shown as boxes with arrowheads inside at the borders of the while-loop. Compared to architecture in which the data is read and sent to DMA-FIFO sequentially during the same iteration, this approach also uses less resources on the FPGA. The unini- Figure 3: Block diagram of sub Read ADC.vi that reads the conversion value from the ADC and transfers the data to the host computer via DMA-FIFO. Uninitialized shift registers are used for uncoupling the data readout from the DMA-FIFO transfer. Two adjacent iterations are needed to acquire the data and to transfer the data word to the host computer. During the first iteration the vi reads data from the ADC (after the trigger is received and the ADC is ready). During the second iteration the data-if considered valid-is sent to the host computer via the DMA-FIFO channel and simultaneously new data can be acquired. tialised shift register "breaks" the data path into shorter pieces that are easier for the FPGA compiler to be optimized. Fig. 4 clarifies the differences of these architectures and the function of the shift register.

Host interface
The host interface of ToF-DAQ at the host computer consists of two major top-level VIs: ToF-DAQ Main.vi and ToF-DAQ Data Handler.vi (See Fig. 2). The first provides the user interface and handles dynamic loading of dialogues and data representers, etc. and the latter, ToF-DAQ Data Handler.vi, as per its name handles the data, i.e. reads the data words from the DMA-FIFO, builds coincidence events and saves the data on the hard disk.
ToF-DAQ Data Handler.vi is constructed using a producer/consumer design pattern. The producer, sub TD-Main Read FIFO.vi, reads data words available in the DMA-FIFO channel as an array and enqueues the data to three separate data transfer queues (i.e. FIFO-type memory structures) for saving, building the coincidence events and for singles histogram builder. Due to the separate queues for each of these main data handling tasks, they run in parallel. The coincidence event builder uses two additional queues to transfer coincidence events to coincidence histogramming and for the optional saving of the coincidence events in list-mode. An overview of the internal dataflow of the system is represented in Fig. 5. In addition to decoupling the reading of the DMA-FIFO from data processing, the queues also act as secondary data buffers.
The data words received from the DMA-FIFO channel are written continuously to the hard disk in binary format.  After data acquisition is stopped the data recorded are read back and rewritten to the hard disk in ASCII format. With this arrangement the data can be written on-line with minimum CPU load but for offline processing the data is also available in an easily readable format. The coincidence event builder de-queues the data from the data transfer queue and builds the coincidence events based on the timestamps of data words and a user-defined event pattern. The builder is based on a ring buffer. New elements are inserted one by one in to the buffer while oldest data elements are discarded at the opposite end of the buffer. The user can define the coincidence pattern via the user interface. One can set, for example, the active detector channels, width of the coincidence window, etc.
The host interface provides several tools to monitor and analyse the collected data on-line. A screenshot of the user interface is shown in Fig. 6. The user can view singles and coincidence spectra, two-parameter histograms, ADC countrates and dead times. The user can also perform simple data analysis, like Gaussian peak fitting and integration over the region of interest. Also the detection efficiency can be examined by comparing a singles spectrum to a coincidence spectrum channel-by-channel. For a quick analysis and comparison without using an actual data analysis program, data recorded earlier can be loaded from disk.
The host interface as a whole is developed using objectoriented programming techniques. Data monitoring and analysis tools are developed as plug-ins that are dynami-  cally loaded in to the subpanel of ToF-DAQ Main.vi. This approach makes tool development easy, as each tool is basically an independent top-level VI that needs only appropriate references to data as input parameters. Adding new tools requires no changes to ToF-DAQ Main.vi.

Accuracy of timestamping
The accuracy of timestamping was demonstrated by a test with a Berkeley Nucleonics Corporation PB-5 pulse generator. The clock rate of the FPGA was 120 MHz, i.e. the resolution of timestamping was 8.33 ns. Coincident signals in different conditions were fed to two Silena Mod. 7423 8k ADCs. To investigate timing properties of ToF-DAQ, histograms of time difference between the detector channels were produced. Three tests were performed: A) A Fixed-height pulser signal was fed to both ADCs while the timestamping was based on the ADC's DataReady line. B) The first ADC was fed with the fixed height signal, but the signal to the second ADC was swept from 0 V to 10 V. C) Similar configuration as in B) was used, but in addition, a TTL signal directly from the pulser was used as a timing signal for both detector channels. The histograms are presented in Fig. 7. By comparing Fig. 7 (A) and (C) one can see that the timestamping resolution is very close to the clock cycle length of the FPGA even if the timing is based on the ADC's DataReady signal, but the use of separate timing signal still provides an improvement. In Fig. 7 (B) the rise-time of the signal has varied as a function of pulse height. As the rise time has an effect on the ADC's processing time, the timestamping precision is also worsened.
In comparison of the timestamp resolution data shown in Fig. 7, a similar plot was made from ToF-E measurement data of a 67 MeV C-12 beam. The time-of-flight was measured using two carbon-foil pickup detectors and an Ortec 566 TAC. The range of the TAC was set to 200 ns. The energy was measured using a Si detector accompanying a preamplifier and an Ortec 575A linear amplifier. Silena Mod. 7423 8k ADCs were used for both channels. Fig. 8 shows the time difference between the coincident ToF and energy signals. Timestamping was based on the ADC's DataReady signals and the FPGA clock rate was 120 MHz. Count rates on both channels were roughly 400 cts/s. Based on the Gaussian fit, a timestamping resolution of 11.3 ns was obtained, even though no efforts were made to obtain a good resolution while adjusting the analogue electronics front end. One would expect to see an improvement if instead of ADC's DataReady signal, an external timing signal generated from the timing signal output of the Si-detector's preamplifier was used.
As a conclusion from Figs. 7 and 8, one can say that the FPGA of ToF-DAQ is capable of timestamping the data at the resolution of its clock frequency. However, the actual timestamping resolution achieved depends on how carefully the analogue electronics front-end is set.

Performance
Throughput of ToF-DAQ was demonstrated with a pulse generator. A Dell laptop with an Intel Core™ i5 processor was used as the host computer. 4 ADCs were connected to the system. The total continuous throughput of the system with full online histogramming was limited by ADC dead times to 770 000 counts/s. Average CPU load was 50 %. Maximum theoretical throughput of the The listbox on the left is for selecting the data monitoring/analysis tool displayed. The settings dialogue can be seen on the right. system is set by the bandwidth of the MXI-Express bus to 4 890 000 counts/s. Taken into account the count rates that a pulse height ADC can reasonably handle, one can assume that in a real experiment neither the FPGA nor the host computer are the bottlenecks of the ToF-DAQ's performance even if the maximum number (8) of ADCs are present.
Since building the coincidence events and updating the 2-parameter histogram are the most CPU intensive tasks, increasing the size of the coincidence event builder's ring buffer or binning of the 2-parameter histogram may lead to the result that the CPU power limits the throughput of ToF-DAQ for very high count rate systems.

Background reduction
To demonstrate the background reduction using a very narrow coincidence window during an offline analysis, a sample was measured using the ToF-E-telescope located in the Pelletron laboratory at the Jyväskylä Accelerator Laboratory. Since the typical background of the ToF-E telescope is very low, the effect of constant random background from the energy detector was simulated by feeding a signal from pulse generator in ramp-mode to the secondary input of the linear amplifier of the energy channel. As the ToF signals are randomly spaced in time, but the artificial energy signals generated by the pulser are produced with a constant rate, the time difference between the signals was random. As a result, a random background covering almost the entire coincident ToF-E spectrum was achieved. This can be seen in Fig. 9 (A) where the time difference between ToF and E signals for coincidence is set to be less than 10 µs. Because the time difference between a time-of-flight signal and a real energy signal induced by a same ion is well defined, the artificial background can be subtracted almost entirely by fine-tuning the coincidence time window. In Fig. 9 (B) the time window was reduced to 250 ns to reject the artificial background almost entirely and leave only the valid ToF-E data. The background rejected from Fig. 9 (B) is shown in Fig. 9 (C).

Flexibility
One advantage of ToF-DAQ is that the system can be very easily modified for different demands. One can implement, for example, counters, device controls or analogue-in and analogue-out functionality on the FPGA. Due to parallel execution of the FPGA, additional tasks do not alter the data acquisition. Additional data can also be timestamped. As the additional data can be at much higher frequencies than the pulse height ADC data, it is essential that the system has the high throughput demonstrated at the chapter 4.2.
With some modifications, ToF-DAQ was used for highprecision energy loss measurements of heavy ions on thin foils with the ToF-E method using a calorimetric lowtemperature detector (CLTD) array as an energy detector [9]. Time-of-flight was determined by using two carbon-foil pick-up detectors connected to a TAC. The TAC signals were digitized using a NIM ADC that was connected to the FPGA. Signals from the CLTD array were digitized using analogue input channels of the FPGA-card. The signals were constantly sampled to buffers on the FPGA and after a peak detection the data from the buffers were transferred to the host computer as waveforms. For quick online analysis peak heights of each CLTD waveform were determined already from the buffered data in the FPGA. Data words with peak height values and timestamps were then formed. Thereafter the peak height data from the CLTD were treated similarly as any ADC data. During offline analysis, based on the timestamps, one could pair the time-of-flight information to the waveform data of the CLTD signals.

Discussion
ToF-DAQ has been successfully used for ToF-ERDA measurements for over four years. During that period, the system has proven to be easy to use, customizable and user-friendly. The timestamping offers new possibilities to the data analysis compared to data acquisition systems traditionally used with ToF-ERDA set-ups.
Even though ToF-DAQ was originally developed for ToF-ERDA, it can be used as a general purpose data acquisition system for measurement set-ups having up to 8 detector channels. The system has been applied for energy loss measurement of protons in liquid water [10], energyloss straggling measurements of Kr ions in gases [2] and, as described in chapter 4, for high-precision energy loss measurements using a calorimetric low-temperature detector.
The TDR method has also some drawbacks compared to more traditional data acquisition systems. The biggest disadvantage is the increased need of disk space. However, taken into account the sizes of modern hard drives, this is mostly a problem of high count rate systems with a large number of detector channels, as a typical ToF-ERDA run measured with ToF-DAQ values around 100 MB. If the amount of data needs to be reduced, pre-sorting of data before writing it to disk is possible. For example, one could save only the coincident events using a wide coincidence time window and narrow the window during the data analysis.
With ToF-DAQ, the ADC dead time can also be a problem. In high count-rate nuclear spectroscopy, gate signals have been used to enable the ADC conversion only when meaningful events are registered to minimize the ADC dead-time losses. Lack of gating can lead to high dead times and thus limit the maximum usable count rates of the measurement set-up. Even though using gating is against the idea of ToF-DAQ, there is no reason that prevents using it if needed. In addition to analogue gating, one could take advantage of extra DIO lines of the FPGA and use it as a digital coincidence unit for generating the gate signals. With ToF applications dead time is usually not a problem because one can partially compensate for it by increasing the measurement time.
While planning a new data acquisition system, an appealing approach would be to discard the analogue frontend by using a fast digitizer to digitize preamplifier signals and determine the ToF based on the digitized waveforms. Digital pulse shaping for peak height analysis using trapezoidal filters has been researched already for decades, and use of digitizers is well established [11]. Unfortunately using fast but reasonably priced digitizers with sampling rate of 2 GS/s for determining the time-of-flight from fast and possibly noisy carbon-foil pick-up detector signals is still non-trivial [12]. Therefore, an analogue front-end still has its place.
Nowadays, VME based systems are widely used in nuclear physics. Typically VME ADCs offer high channel densities and fast conversion times. Timestamping ADCs and TDCs [13] are available. Typically, VME-based sys- The fixed-height signal from the pulse generator is split in to both detector channels. In (B) the same timing mode is also used, but the pulse height in the second ADC is ramped from 0 V to 10 V while a coincident fixed-height signal is fed to the first ACD. Similar pulser configuration as in (B) is used for (C), but TTL-signals from the pulser are used as timing signals for both detector channels. D a t a G a u s s i a n f i t G a u s s i a n f i t R -S q u a r e : 0 . 9 9 8 9 9 P e a k = 7 0 5 . 9 1 n s F W H M = 1 1 . 3 1 n s Figure 8: The time difference between ToF and energy signals of a 67 MeV C-12 beam. The ToF (T 1 ) was measured using two carbonfoil pickup detectors and a silicon detector was used for obtaining the energy signal (T 2 ). Timestamping was based on the DataReady signals of the ADCs and the FPGA clock rate was set to 120 MHz. According to the Gaussian fit, a timestamping resolution of 11.31 ns was achieved.
tems are designed for larger scale detector systems; one would consider them being an overkill for a simple ToF-E setup. Also, in comparison the price tag of the hardware of a ToF-DAQ system can be relatively low. Assuming that one uses the already existing analogue electronics frontend and ADCs, the DAQ hardware requirements of a new ToF-DAQ system are a single National Instruments' PCI type FPGA device housed in a standard PC. In addition to the hardware costs one must consider the price of the LabVIEW license and the FPGA toolkit, but since Lab-VIEW is very widely spread it is likely that the potential end users would be already in possession of at least the standard license.
The closest commercially available alternative to ToF-DAQ is the FAST ComTech MPA4T multi parameter ToF system that has a 5-channel multi stop ToF digitizer with 100 ps timing resolution and a timestamping multi parameter system with connections for 4 or 8 NIM ADC's [14]. The timestamping resolution of MPA4T is stated to be 6.4 ns. For some aspects MPA4T is superior over ToF-DAQ, but it lacks the flexibility and customizability. Also, MPA4T is not capable of reading the external ADC's in parallel, thus the detector channels are not handled truly independently from each other.

Summary
A new data acquisition system derived from the TDR concept was developed using the LabVIEW graphical programming environment and a National Instruments FPGA device. The system combines an analogue electronics frontend to a timestamped data acquisition. A resolution of 8.33 ns for timestamping has been achieved. The new system has been in use for several years with the ToF-ERDA-telescope at Accelerator laboratory of University of Jyväskylä and it has proven to be powerful and reliable tool, easily modifiable for different demands. Timestamping offers new options for data analysis for ToF-ERDA as coincidence events can be built offline.