Automatic spike sorting evaluation: A website based community approach

The goal of spike sorting is to separate the spike trains of individual neurons from extracellular recordings. This step is crucial in many neuroscientific experiments since one extracellular electrode usually records the activity of several neurons. Despite the large effort to develop automatic algorithms (Lewicki, 1998) to solve the problem, spike sorting can be still considered as well an art as an exact science with a large manual component. For the quantitative evaluation of spike sorting algorithms the ground truth of the data analyzed, i.e. the number of neurons and their firing times, has to be known (Einevoll et al., 2011). Real extracellular recordings provide no suitable benchmark data because of the inherent absence (or at least very limited presence) of ground truth information. Thus, using simulated surrogate data is the traditional way to evaluate spike sorting algorithms. However, most scientists use their own simulated data, making comparisons between different publications very difficult.
Here, we develop a framework for automated spike sorting evaluation based on several different benchmark datasets used in recent publications. The framework is implemented on a website that allows the user to download benchmark files, upload their sorting results, and compare the performance of their sorting algorithm to those of other users. Furthermore, users can also upload their own benchmark datasets and make them available to the community. We hope that the website will help in comparing the performance of different spike sorting algorithms and foster the development of new ones.
The underlying framework, i.e., website frontend and evaluation backend, can be generalized to other, similar, algorithm evaluation problems, such as encountered in EEG data analysis.

The website is available at http://www.g-node.org/spike.

Acknowledgements
Supported by the Research Council of Norway (NevroNor, eScience, Notur), by INCF through its German Node (BMBF grant 01GQ0801), and by the Deutsche Forschungs Gemeinschaft (DFG) with grant GRK 1589/1.

References
Einevoll, G. T., Franke, F., Hagen, E., Pouzat, C., and Harris, K. D. (2011). Towards reliable spike-train recordings from thousands of neurons with multielectrodes. Current Opinion in Neurobiology 27, 1–7.
Lewicki, M. S. (1998). A review of methods for spike sorting: the detection and classification of neural action potentials. Network (Bristol, England) 9, R53–R78