Detecting spikes in extracellular data: Tutorial 1

This tutorial applies to the Phased Array Spike Sorting extension block. It is not an introductory MIEN tutorial. It assumes that you know how to run MIEN, install blocks, and use the Data Viewer. It also assumes that you have installed the "gicspikesort" block. If you haven't, then the menu options that are listed to launch the sorter will not exist in your version of Data Viewer.

CONVENTIONS

This tutorial was written primarily for use inside the CCB, by people using Mac computers. Some aspects may not be generally applicable. Also, I assume less knowledge of computers in general, and unix shells in particular than is usual in the rest of my documentation.

In this tutorial, steps that are listed like this 1) are required. Steps listed like this [2] are optional.

I refer to menu options using the common -> notation. Click the "File" menu and select the "load" option is written File->load. Mien windows each have their own menu bar (unlike many mac apps), so if you have several windows open, you will need to know which window to choose menu items from. In this case I will use something like DV:File->load, where DV identifies the window. I this tutorial, windows include DV (the main Dataviewer), SS (the spike sorter), and AT (the spike sorter analysis tool). The nesting of these windows is such that SS is a child of DV and AT is a child of SS. As with most GUI apps, closing a window closes all its children (and grandchildren, etc), but not its parent.

Generally, I use emphasis for strings that have special meaning to your computer, or that appear in output from the software, and strong for strings or dialog choices that you need to put into the computer.

1) TERMINAL

Open a terminal (also called a command shell). On Mac OS X, you should use Terminal.app, which is found in /Applications/Utilities.

IMPORTANT NOTE: The terminal is the most important part of mien. It's a good idea to check your terminal periodically throughout a mien session, even if you don't think you are using it. Important information gets printed to the terminal. Some GUI functions have no effect other than to print information to the terminal. Also, mien has error handling routines that prevent the GUI from crashing if a particular function or piece of data has errors. Pretty often, the only way to tell that a function failed is to look at the terminal. Here is an example of a mien error message:

This message was generated on loading mien after I inserted some bad code (the string "aoeu") into one of the mien extension modules. These messages may seem cryptic, but they are informative to python programmers. If you get behavior you don't expect (or don't like) when using mien, check for a traceback. If you see one, there may be a bug in mien or the extension modules, or you may have called a function using the wrong sort of data or arguments. Mien, especially the extension blocks like the spike sorter, contains a lot of rapidly changing and limitedly tested code, so there are plenty of bugs.

Also, the extension modules are intended for use in research, so there is no attempt to"idiot proof" them as though they were commercial single-purpose software. Instead, the mien philosophy is to provide maximal flexibility, and expect the user to deploy features in an appropriate context. One thing this means is that not all menu choices will work in all circumstances. If you try to calculate a residual signal that would remain after subtracting a spike template, for example, but you haven't yet calculated a template, you will probably generate an error traceback. The important thing to keep in mind is that in these cases, the GUI will generate no feedback. The only way you can know what is going on is to pay attention to the terminal. Also, if you want to report a bug, please include the Traceback in the bug report. Otherwise, there is no way for me to track or fix your bug (or even convince myself that it actually is a bug rather than user error :)

If you are a python hacker, rather than reporting a bug you can read the traceback and fix the bug. Some of the time this is hard, but many bugs are trivial and anyone who can read tracebacks can fix them. Ask me about how to submit your bug fixes to the svn server.

[2] UP05/29/09

If you haven't run mien in a while, you will want to update it. This is done using the svn command. If you have mien installed at the top of your home directory (which is normal) all you need to do is type:

The part you have to type is after the > prompt, so this is: svn up mien followed by <enter>, etc. The computer may respond with more messages than I show here. My copy of mien is up to date. If you update an old copy you will see lots of lines begining with letters like "U" or "A". As long as none of the lines begin with "C" this is OK.

NOTE: The terminal opens with the current working directory set to your home directory. If you are using a mac, and your user name is bob, this directory is named /Users/bob. If you use the pwd command, you can see where you are:

My user name is gic, not bob. Usually, mien is installed directly under your home directory, in the two directories /Users/bob/mien and /Users/bob/mienblocks. Path names that don't start with a "/" are relative to the current working directory of the terminal. Thus, if I'm in /Users/gic, and I type svn up mien, the machine understands me to mean svn up /Users/gic/mien. If I change directory, the meaning of the relative path will change. Consequently, if you type cd Desktop, now svn up mien probably won't work. You either need to use the cd command to get back to the place mien is installed, or you need to pass full path names to svn (those are ones beginning with a "/", like /Users/gic/mien).

3) RUN MIEN

Usually this is done by typing

How you do this depends on how your PATH environment variable is set up. If just typing mien gets an error like command not found, and you don't know what a PATH is or how to set it, talk to your sys admin.

If things are set up right, you can type mien. This will run the top level gui, which is used for looking at data hierarchies. Typing mien -a data runs the Dataviewer component of mien, which is used for looking at numerical data. The spike sorter is an extension to the Dataviewer, so you probably want to start by launching the Dataviewer.

On most machines I set up myself, I have set a unix alias so that typing dv is equivalent to typing mien -a data. (Just put the line alias dv='mien -a data' in your ~/.bashrc file to set this up.)

No matter how you do this, eventually the Dataviewer GUI should appear. You may get a lot of tracebacks proceeded by the line error loading module .... This happens if your system isn't configured to use particular extension blocks. When you svn up mienblocks, all the blocks I use get installed. This includes, for example, a robot simulation environment. You probably don't have the binary libraries that this depends on installed on your computer, so loading that module causes an error. That's really OK, since you also don't want to simulate robots. The Dataviewer will still load, it just won't include extensions from a particular block.

IMPORTANT NOTE: Do remember, though, that the mien GUI menus are dynamic. Some of the menu options depend on extension modules. If, for example, the gicspikesort module doesn't load, then the menu option to launch the spike sorter won't appear. You will need to figure out why the spikesort module failed and restart mien to get the menu option to come back.

4) LOAD DATA

Use DV:File->load, or drag and drop data files on the Dataviewer. You can also pass the file name of the file you want to open to the mien -a data command, and then the data will already be loaded when the gui opens.

Loading data from the network is slow, and loading DataMAX files is a lot slower than loading mien native files, so you will want to work with a local copy of the data. You can open a remote DataMAX file initially, but then use DV:File->Save As to make a local copy of the data. If you use format guess from file name in the save as dialog, and have your file name end in .mien or .mdat mien will make the local file in its native format. This is highly recommended.

If you are "just playing around", and you loaded a big data file, you may want to chop out an interesting segment to experiment with. Spike sorter operations will be much faster if you are using a smaller data file. Use the zooming commands to select a region of data you like, and use DV:Edit->Crop. Select Crop Mode: Crop to view. Leave Set Start Time to 0? set to Yes. Doing this will usually cause all your data to disappear. This happens because the viewer is looking at some window, for example 20 seconds to 23 seconds, and the three second segment you cropped out is now assigned to the interval 0 seconds to 3 seconds. Press the q key to re-center the view on the data. Now save your data snippet in a local file.

IMPORTANT NOTE: The spike sorter makes extensive use of mien's hierarchal data file structure. You will be much more comfortable with it if you understand this structure. For now what you need to know is that saving a spike sorter file in, e.g., DataMAX or Datastreamer format will destroy most of your work. The only tested safe formats for spike sorter data are mien, mdat, and mat, and the mat file filter is quite slow.

5) LAUNCH SPIKE SORTER

Use DV:Extensions->Launch Phased Array Spike Sorter

6) CONDITION DATA

If you have run the spike sorter on this data file before (and then saved in a mien format), you can skip this step (in fact you have to. The dialogs I mention below won't show up, and the Spike Sorter GUI will appear immediately). If you are opening a DataMAX file, however, you will have to answer some questions in dialog windows in order to allow the spike sorter to do initial setup on the data.

The first question is Would you like to import existing templates?. Say No (use of this option will be described in later tutorials).

Now you will see the data conditioning dialog. It has several options for initial treatment of the data. For now, leave all the Yes/No options set to Yes. Below these are two text entry fields labeled Stimulus Channels and Delete Channels. If your file has all good recordings, and the last two channels are the stimulus, you can leave these as they are. Otherwise, you need to edit them.

The content of these fields are python lists. These look like this: [0,3,4]. Python counts from 0 (not 1), so the list I show here specifies the first, fourth, and fifth channels. Fill out the lists so that any bad recordings you don't want to analyze are listed in Delete. If you want to save the stimulus for later analysis (for example calculating a spike triggered average stimulus for a class of spikes after you sort them), make sure the stimulus channels are indicated in Stimulus. If you don't care about the stimulus, you could put the stimulus channels in Delete instead (don't put them in both!).

If you don't want to construct python lists by hand, you can press the Browse buttons next to these fields. That gets you a list of all the channel names. You can click, shift click, or command click to select a set of names from this list in a typical way. Holding down the command key and clicking on each item you want to add is probably easiest. When you select OK, mien will construct the equivalent python list for you, and enter it in the text entry field. Note that the channel names are strings entered by the human operator of the data collection system. Consequently, they probably don't count from 0. On much of John's data, python channel 0 is named Electrode 1 (No Comment). In any case, the channels are listed in the Browse gui in the same vertical order that they are displayed in by the Dataviewer. I recommend using the Browse guis just to remind yourself of how the python lists are constructed. I find that, in day to day use, it is much faster to type [8,9] than to open a gui and click on two names in a long list and the OK button.

Once you are done setting up your Stimulus and Delete lists, click OKh in the conditioning dialog. It will close, and the spike sorter will open. This might take up to 30 seconds. The data preconditioning tasks you specified will be applied, so the image in the Dataviewer will change. Your stimulus channels will also disappear. They aren't deleted, though, just nested in a hidden data path (this terminology relates to the mien data hierarchy mentioned above).

Also, the spike sorter will resize the dataviewer and itself. This should make everything look nice, but if you have less than 1200 pixels of vertical resolution on your monitor it may not look so good. I highly recommend using a large screen for spike sorting.

[7] MAKE BACKUPS

From here on in, you want to be working with a locally stored mien or mdat file if you want to save your work.

Also, the Undo/Redo feature in Dataviewer is disabled when the spike sorter is open (otherwise performance on big files would be too slow). Many spike sorter function are inherently reversible using other spike sorter functions (e.g. apply/remove shift templates, and subtract/add waveform templates). If you want to be able to undo any possible operation, however, you need to use Dataviewer checkpoints. Use DV:Edit->Save Checkpoint to create one, and DV:edit->Load checkpoint to restore to one you have already saved.

If you save the whole file, you can reload it later, open the spike sorter, and almost all aspects of your workflow should be maintained.

NOTE: A major goal of the spike sorter is to allow subsequent batch processing, which reproduces the actions chosen in an interactive session on a large data file. This requires that the spike sorter keep careful track of every action you take and the order you take it in. Actions like Undo/Redo make this difficult, and interactively "pruning" a template (by selecting and deleting spikes) makes this impossible.

As far as the spike sorter is concerned, a "good" work session is a simple, linear session. Obviously, though, experimenters will need to experiment. Often the best compromise is to experiment with the data first, with no intention to save the results. Once you have a protocol for a given data set in mind, reload the data and construct your templates using as few UI actions as possible.

8) MAKE A TEMPLATE

The spike sorter creates a template when it first opens. One of the first things you will see in the status report line (at the bottom of the SS gui) is something like Active template is spikesort_unit0. This is the newly generated template that was created. It's now ready for you to calculate an associated set of shifts.

After you get done with your first template, though, you will need to make a new one for the second unit you want to sort. To make templates, use SS:Templates->New Template, and choose a name for the new unit (using the automatically generated names "unit0", "unit1", etc is fine for now).

If you have more than one template, you can switch the active one by using SS:Templates->Choose template.

[9] SET PREFERANCES

You can use SS:Data->Preferences to determine how the spike sorter constructs templates. The most important preferences are Template Lead and Template Length. These are numbers in milliseconds. The default is 1.5 lead and 3.0 length. This means that when spikes are detected, each event is represented as a waveform 3 ms long that starts 1.5 ms before a threshold crossing. The template window also determines the window width used to calculate the OptMin statistic.

You might also want to change Threshold Level. This value is used if you automatically set thresholds. If it is .05, ad the global minimum of the data set is M, then the minimum threshold is set at .95*M (aka (1-.05)*M). I recommend ignoring this value and setting thresholds by hand, however.

You can also input your screen resolution here, which may make things prettier on startup.

Once you set preferences, they are persistent. Even if you exit the spike sorter and restart it, the preferences will stay as they are until you set them to something new.

If the OptMin function is acting in a strange way, check your Template Length preference.

10) SELECT A DISCRIMINANT

In the upper left of the SS GUI is a pulldown menu named Discriminant. This allows you to choose the function that calculates the result shown in the spike sorting window. This result is what is used to detect events, so you want it to attain minima where the unit you are trying to sort out is active. Some choices of function work better for some data than others. See the spikesorter overview for ingformation about discriminant functions. For now, try Mean or Mean/OptMin + Threshold. Mean is the default, and is the basic phased array (shift and sum) function.

11) FIND A SPIKE AND ALIGN

You can use zoom controls in the DV or SS windows to look through the data. Find a spike waveform that is well isolated, and choose a zoom window such that only that spike is on the screen. If you are using the Mean discriminant, you should choose the biggest spike you see in the data set. With the OptMin method that isn't as important.

If you can't get the spike entirely on the screen without seeing disturbances from other spikes, look for a more isolated occurrence of the spike. If you can't find an isolated one, you can make it easier to isolate the spike by first applying a linear shift template. In the DV window, left click to put a marker (light blue vertical line) on the occurrence of your spike on the first channel, and another one at the same place in the spike waveform on the last channel. Now use SS:Templates->Align From Marks to calculate a preliminary shift. This should make the spikes more vertically aligned (and easier to isolate) in the DV window.

Once you have a spike on the screen, use SS:Templates->Align to construct a shift template.

NOTE: If your spike is really big, you may not need to isolate it to align it. The alignment function line up minima, so if the lowest value of each channel shown on the screen if from your spike, it will align properly. Be warned, though, that aligning when two spikes are on the screen can result in unreasonable shift templates that can confuse the rest of your analysis. If you think you aligned to the wrong thing, use SS:Templates->Remove All Shifts to get back to the raw data and try again.

12) DETECT SPIKES

Now that the data are aligned to a particular spike propagation rate (this alignment is called a shift template). You should have some nice peaks (negative going ones) in your discriminant. All you need to detect some spikes is some thresholds.

Spike Sorter uses a schmidt trigger, so you need two thresholds. The higher one should be about halfway between the mean and your lower threshold. The lower threshold should be set so that only events you want to detect cross below it.

You can use SS:Detect->Set Thresholds (Auto) to set some thresholds using the threshold parameter in preferences. A better choice is to use Shift-LeftClick to set two horizontal markers in the SS window. Put them where you want the thresholds, and use SS:Detect->Set Thresholds (Markers) to turn them into thresholds. Thresholds are displayed in pink, where markers are blue, so you can see when your markers are set to be thresholds.

You can put in different markers and set thresholds again if you want to move them.

Once you are happy with them, use SS:Detect->Detect to find spikes. Detected spikes will appear as red vertical lines in the SS gui. They will also appear as vertical dashes in the Dataviewer GUI if SS:Data->Show Subdata in Viewer is selected.

You can use the Previous and Next buttons on the bottom of the SS gui to "page through" the spikes you have detected.

[13] RATE YOUR TEMPLATE

Now you have a complete template and some detected spikes. It would be nice to know if it is a good template, in the sense that the spikes might correspond to events in a particular neuron.

About 90% of the features in the spike sorter come into play at this point. You have many options, and there is as much art as science to the process.

Methods include looking at residuals, clustering waveform summary statistics, refining the shift template, or switching to a discriminant function that requires a preliminary waveform template (which you just constructed) in order to work.

I'll provide several entire tutorials on this step. For now I'll just show you the easiest possible thing. Click the Analysis Window button on the right of the SS GUI. A third gui will appear (AT). Use AT:Displays->Template to see a picture of the waveform template you just calculated (this is the mean and standard deviation of the ensemble of detected events). The bottom half of this picture shows the standard deviation. If you see a lot of structure here, you may have a bad template, especially if the structure occurs near the middle of the template (where the spike is), and the template is based on many individual spikes.

Press Previous or Next on the SS window to make sure you have a spike selected, and then you can use AT:Displays->Current Spike Residual to so the difference between the current spike and the template. If you leave that open and press Next on the SS window repeatedly, you can see which detected spikes look a lot like the template, and which ones look a lot different.

[14] SUBTRACT TEMPLATE

If you just detected a bunch of big spikes, you may want to get them out of the data set so that you don't detect them again when looking for smaller spikes. With some discriminant functions you won't need to do this, but with Mean it is pretty much required.

Use SS:Refine->Subtract Templates to subtract the waveform template from the data everywhere that a spike has been detected.

15) NEXT

Go back to step 8, create a new template, and work on the next unit.