The Denoise Benchmark (DNB) for task-based fMRI
Questions, comments? E-mail Kendrick Kay


History of major changes


The Denoise Benchmark (DNB) is an architecture for testing and comparing denoising methods for task-based fMRI. The performance metric is cross-validation accuracy, whereby a denoising method is evaluated according to how accurately its estimate of task-related responses predict held-out data. The underlying idea is that if a denoising method genuinely removes (or accounts for) noise in the data, then the estimate of the signal (i.e. the task-related component of the data that we are interested in) should be more accurate. DNB is written in MATLAB and consists of three main components:

  1. fMRI data (21 datasets available)
  2. Code framework for automatic evaluation of denoising methods
  3. Implementations of several denoising methods

The DNB is described in the following paper:

Kay, K.N., Rokem, A., Winawer, J., Dougherty, R.F., & Wandell, B.A. GLMdenoise: a fast, automated technique for denoising task-based fMRI data. Frontiers in Neuroscience (2013).

In this paper, we present a denoising method, GLMdenoise, and use DNB to evaluate GLMdenoise and other denoising methods. If you use DNB in your research, please cite the above paper.

If you have data that you would like to add to the DNB, please contact me. Also, if you have a denoising method that you would like to evaluate using the DNB, you can either try it yourself using the materials available here, or I would be happy to take the method and run it through my existing setup.

Getting started

Download code (latest tagged version 1.01)
Download code (latest development version)
Clone code from github
After downloading and unzipping the files, launch MATLAB and change to the directory containing the files. You can then run the example scripts by typing "example1" or "example2". More information on the example scripts is provided below.

Example scripts

The following are links to HTML pages produced by the MATLAB 'publish' command.

Denoise Benchmark API

The DNB requires that different denoising methods conform to a common API (Application Programming Interface). Any denoising method that conforms to this API can be automatically evaluated in the DNB.

A denoising method is defined to be a function that accepts three inputs:

  1. A dataset struct containing the fMRI data and associated information (see Format of the data).
  2. A directory location to which the function is allowed to write figures and/or output. Omit the trailing slash.
  3. A column vector of dimensions time x 1 with an HRF estimate. It is up to the denoising method to decide whether to use this HRF estimate.

The function should return a model, which is defined to be a function that accepts a single input:

  1. A cell vector of design matrices that are each time x conditions. The number of conditions should be the same across design matrices, but the number of time points may differ across design matrices.

This function should return a cell vector of matrices with the predicted time-series. Each matrix should be X x Y x Z x time (with the number of time points matching the corresponding design matrix).

All of the denoising methods provided in the DNB (i.e. DNBmethod_*.m) conform to the API described above.

Format of the data (datasetNN.mat)

The fMRI data provided in the DNB comprise 21 independent datasets. For convenience, the data have already undergone pre-processing, which consists of the following steps: removal of the first five volumes of each run (to avoid initial magnetization effects), slice time correction (sinc interpolation), motion correction (rigid-body alignment as estimated by SPM), and spatial undistortion based on fieldmap measurements (custom code). The raw fMRI data are available upon request.

The 'datasetNN.mat' file contains the data corresponding to dataset NN (where NN ranges from 01 to 21). The following variables are contained in each dataset file:

In some datasets, physiological data were collected (cardiac and respiratory monitoring). For these datasets (namely, 14–21), the following additional variables are available in the dataset file:

Note that in the regression procedure, the physiological regressors are orthogonalized with respect to constant, linear, and quadratic terms before they are projected out from each run.

Format of the physiological data (datasetNN_physio.tar)

The RETROICOR and RVHRCOR denoising methods have already been applied to the fMRI data provided in the DNB (see description above). However, we provide the raw physiological data and the raw fMRI data for those wishing to try their own methods based on physiological data.

The 'datasetNN_physio.tar' file, after untarring, contains the following files:

Results of the Denoise Benchmark

We used the Denoise Benchmark (DNB) to evaluate a number of different denoising methods. The implementation of these methods is provided in the DNB repository (i.e., DNBmethod_*.m).

The results of the evaluation are provided by the DNBresults/results.mat file, and figures that illustrate these results are provided by the DNBresults/figures.tar file. Both of these files can be downloaded using DNBdownloadresults.m.

The following variables are contained in the 'results.mat' file:

The 'figures.tar' file, after untarring, contains the following files:

Code to generate the figures is provided in the example2.m script (see Example scripts). The figures generated by example2.m are written to the DNB directory and will not interfere with the figures contained in figures.tar since that resides in the DNBresults directory.

Implementation notes

The DNB includes implementation of several denoising methods. Here are some notes regarding this implementation: