DataViewer Tutorial
While reading this tutorial, you probably want to refer to the Dataviewer Manual, which describes how to navigate in Dataviewer (DV), and how to use the menus and preferences. If you do not have a working mien install yet, you will need to look at the install instructions, and if you are running an older version of mien, update it. If you have trouble opening a terminal, starting the apps, or loading files, see the quickstart.
Many of the most useful data analysis functions of DV are implemented in extension blocks. Consequently, what you can do with DV depends on which extensions you have installed. If you have extensions, check for documentation and tutorials particular to those extensions. This document only shows operations that are available using the core MIEN package with no extensions.
The tutorial is just a set of examples of several tasks you can do in DV. The tasks are somewhat independent, but the later tutorial sections assume you have information presented in the earlier, and tutorial 3 uses a data file you generate in tutorial 2. The sections are:
Extract a region of interest from a long data stream.
Generate a signal with several components.
Extracting a Region of Interest
Open the test data
Open a terminal, and type:
This will open DV, and also download and open a test data set from the web. It may take some time, depending on the speed of your net connection. The terminal will print a number (the time in seconds used to open the data), and maybe some messages (such as "no config information"), which you can ignore. A DV window should open and display some lines.
Optionally, you can save a local copy of the data. Use DV->File->Save As. Leave the Format option on "guess from filename", and enter the file name dvtut1.mien. You can save in any directory you like. To restart the tutorial, you can now open the local file, using any of the methods described in the manual, so you won't have to download it twice.
This data sample is recorded from a 6-electrode extracellular array. Channels 0-5 are electrode recordings. Channels 6 and 7 are two parameters of a band-limited noise stimulus. In this case this is an acoustic stimulus generated by two speaker groups. The speaker command voltages are recorded here. (If you are confused that there seem to be 16 line plots rather than 8 read this).
Zoom In
Right now, the image is zoomed out too far to see much detail. It looks like there is a burst of activity near the middle of the dataset, though. Spikes can be seen as widenings of the envelope plots in the first 6 channels. Let's get a closer look at this region (There are actually ~14 areas of large response amplitudes. You can pick any one you want, but I'm going to use the big one near the middle, between 4.9 and 5.0 seconds).
First, left-click on the title bar of the DV window to focus the window. Now, left click again inside the black graphing region, just to the left of the spikes. DV will draw a light blue vertical line, and label the line with its X coordinate (in this case a time in seconds). Left click a third time just to the right of the spikes. Just click - if you drag, DV will zoom in automatically. If this happens, you can return to the initial view by pressing "q".
If you don't like where your markers are, or if you draw more than 2 of them, you can press "x" to delete the last one, or "k" to delete them all and start over.
Once you have markers on either side of the spikes, press "z" to zoom in. Your view will zoom in on the region between your last two markers.
If you want to adjust the view, there are lots of ways to do it. Review the navigation keys in the manual if you want. For now, if you miss, press "q" to get back to the initial view and try again. Your last two markers will have been automatically removed when you zoomed in. If you had more than two, the other ones will still be on the graph. Later on, we'll look at some more efficient ways to move the view.
Probably, you can still barely see spike waveforms, so pick a spike and repeat the "zoom in" procedure to get really close. Once you are zoomed in close enough to see individual data samples, DV automatically switches from the two-line "envelope" plots to more typical single line plots.
You should see a pretty typical-looking extracellular spike waveform on channels 0-4, and you will notice that this waveform occurs at later times on higher numbered channels. The array that recorded these spikes is a linear array laid out along a nerve, so there is a propagation delay between the appearance of a spike on the first electrode and the appearance of the same spike on the next one.
Inspect and Remove a Bad Recording
Channel 5 (the sixth channel - remember that Python counts from 0) looks pretty bad, but the trace is so small that it's hard to tell if this is really a bad recording, or just a low gain setting. Select DV->Display->Global Equalize to display all the channels with the same peak-to-peak amplitude.
If you are using the default view setting of quick sep, and you were looking at a big spike, you may see traces from different channels overlapping each other. If that happened, grab the pull-down menu in the lower right of the DV window and select separate. This chooses a slightly slower, but much prettier, algorithm for laying out the channel traces.
You should now be able to easily see that channel 5 is very noisy, and the waveforms look odd, and maybe even inverted. There might be some ways to clean this up, but for now let's assume that channel 5 is just a bad recording. You can't win them all. Let's get rid of it. Select DV->Edit->Delete Channels. In the dialog that pops up, select "Electrode 6 (No Comment)", and click OK. Note that the names of the channels listed in that dialog are the names assigned by the data collection system. In this case that was a DataMAX ADC, and its software counts from 1, rather than zero. That's rather confusing, but it's a common problem in data analysis. Matlab and Fortran also count from 1, and C, Python, and Ruby count from 0, so it's an issue you have to be aware of when moving data between tools. When possible, MIEN provides interface help to tell you which channels are which.
Now that the bad channel is removed, let's go back to viewing the actual amplitudes. Select DV->Display->Raw Amplitudes to turn off the equalization.
Extract and Save a Region of Interest
Let's look at a slightly larger section of the response. Press "a" a few times to zoom out a bit, so that you can see a region with at least one large spike and several smaller spikes. Obviously, this data set contains spikes from more than one type of cell. It might benefit from signal separation (spike sorting). MIEN can do that, but only using extension blocks. For now, I'll assume you want to have a closer look at some of these spike waveforms using some other tool, such as Matlab.
First, let's select out only the small region of interest we are currently looking at. Select DV->Edit->Crop. In the dialog window, choose Crop Mode = Crop to view. If you want to know when this segment occurred in the data set, set Set Start Time To 0 to No. Now press OK.
Your display may look the same, but the non-displayed data have been deleted. Press "a" or "q" to verify this. Note that if you did Set Start Time To 0, your displayed data probably vanished. They weren't deleted, but they were moved rather far to the left. DV usually doesn't adjust the view when you do a processing operation, since sometimes you need to compare a particular region before and after the operation. In this case, you moved the data you were looking at over to zero time (usually far to the left), deleted all the rest, and left the view port looking at nothing. To find the data again just press "q".
Now let's save this snippet in a format Matlab can read. Select DV->File->Save As. In the dialog, leave Format set to guess from filename, and use a filename of dvtut1.mat.
MIEN saves in Matlab Version 7 format, and stores data elements in a particular sort of Matlab Structure Array. If you open dvtut1.mat in Matlab, you will find a structure named DataFile, with several attributes specifying metadata, such as DataFile.fs (50000, the sampling rate in Hz), and DataFile.StartTime (the time in seconds associated to the first sample). The numerical data is stored in DataFile.data, in this case an Nx7 single precision array.
Generate a Signal
Open DV and Generate a Blank Timeseries
Open a terminal and enter mien -a data to open a DV. For this tutorial, you do not need to open a data file.
Select DV->File->Create Blank (all zeros) Data. In the dialog, select a length of 10.0 (seconds). Leave the sampling rate (Fs) and the number of Channels at their default values of 10000.0 and 1. Select OK. Press "q" to center the view on the new data. NOTE: Some versions of DV have trouble auto-sizing the view for data that is constant-valued. You may see a green line at the very bottom edge of the screen (this can be hard to see). Upgrading to the latest DV will resolve this. If you don't feel like upgrading, just press "down arrow" after pressing "q" in order to center the new data trace on your screen.
(Optional) Edit the Metadata of the New Signal
Metadata are pieces of information that describe other data. They are useful for figuring out where data came from, how it should be used, and how accurate it is. The most important piece of metadata for a DV data element is the "SampleType". This indicates how the numbers in the data array are related to actual measurements. Data with different sample types are interpreted quite differently by DV (and MIEN in general). More information about sample types is provided in the Data element API documentation.
The data element you just generated is a "timeseries" type data element. In these elements, each row in the data array is a sample of one or more variables (called channels). The samples are equally spaced in time, with a particular sampling rate, and the value of a sample gives the value of the measurement at the corresponding time. These measurements usually can have many values. Most often they are approximately continuous quantities, represented by floating point numbers or large integers (such as those generated by an A to D converter).
Timeseries data are a "sampled type", meaning that they have a fixed sampling rate. The next most important piece of metadata for any sampled type is "SamplesPerSecond". This parameter specifies the sampling rate, and is required to convert between an actual value of the independent variable (treated as time in DV, although DV will usually work on data with other X variables) and a row index in the data array.
"Create Blank Data" automatically set the most important metadata for the new data element, using some default values, and the sampling rate you specified. You can go ahead and use this new element as it is (just skip to the next section of the tutorial). If you want to add or modify metadata, you can do so using the xml editor.
First, let's set some marker at the edges of the data trace in DV. This will provide a reference that makes it easier to see the effect of editing metadata. Select the DV window, and Press "a" to zoom out so that ends of the new data trace are visible. Left-click once on the left end, and once on the right end of the line to set two markers (near 0, and near 10 seconds, respectively).
Use DV->Edit->Data Editor to launch the XML Editor. You should see a new MIEN gui, with a tree browser showing some text in the left-hand panel. (If you are on some GTK Linux distributions, the left panel may start out with a very small width. Resize it by hand. This is a known bug, but isn't fixed for all distros yet). This tree contains all the elements in your MIEN document. In the current documents, there will be two: "Nmpml:0", the top-level document element, and "new (timeseries data)", the data element you just created.
Double-click on "new" to open the attribute browser in the right-hand panel. Double-click on the "StartTime" line in the right-hand panel to bring up a dialog for editing the value. StartTime indicates the value of the independent variable that is associated to the first sample in the data array. By default this is 0, but in general it can be any value. Use the editing window you just opened to change the value to 5.0, and click OK. Notice that the left edge of the data trace in the DV window moves to the right when you click OK. DV and the XML Editor are linked, so when you change an attribute that changes the meaning of the data in XML Editor, DV reflects the update in its display.
Now double-click on the SamplesPerSecond line in the XML Editor. In the dialog, change the value to 20000.0, and click OK. Notice that the length of the data line in DV shortens. The right edge of the line should now be near your right-hand marker again. The data array still has the same number of samples (100,000), but it is now interpreted as having half as much time between samples, so the duration of the trace is now 5.0s, and it starts at 5.0, so it ends at 10.0.
For consistency with the rest of the tutorials, change these two elements of metadata back to their default values (StartTime=0.0, and SamplesPerSecond=10000.0) before continuing.
You can also add metadata attributes that aren't defined yet. Click Add Attrib in the right-hand pane of the XML Editor, enter Attribute=Author, and Value=Your Name, and click OK. You will see that the Author attribute is added, but the trace in DV doesn't change. MIEN doesn't know of any meaning for "Author" that effects how to interpret or display data. Having extra attributes like this won't hurt anything, though, so you can use them to mark up data. Many file formats, including "mien", "xml", "nmpml", and "mat" will retain the new attributes when you save the file.
You can also add detailed comments to your data. Since comments may be long, they are stored in their own elements, not in attributes. To add a comment, select new (timeseries data) in the left-hand pane of the XML Editor, and right-click on it. This brings up a context-sensitive menu that provides functions appropriate for use on the currently selected element or elements. Select the first option on this menu, Add Child Object. In the dialog that appears, select Comments and click OK. You will see another dialog asking for the name of your comment object. You can use any simple string you want for a name, but keeping the default name ("Comments") is fine too, so just click OK again.
On most platforms, the only change you will see is that a "disclosure triangle", or a plus sign, appears next to new (timeseries data) in the left pane of the editor. This indicates that this element now has children (other elements which are nested inside it). Click on the disclosure icon to open the new branch of the document tree and display the children. Now you will see your new element, "Comments:". To add text to the comments, double-click on the "Comments" line. The right-hand pane changes to an attribute editor for the comments. There aren't any attributes yet, nor any data or cdata. The text of your comments will be stored as cdata (character data). Click in the text window in the lower right of the XML editor, and type in whatever comment you like (such as "data created for DV tutorial 2"). When you are done, click the Set cData button above the text entry box. The line describing your comment (in the will change to something like "Commments: Data created for DV ...".
The existence of the comment won't affect the function of the data element at all. The comment will be included when saving the file in "complex" file formats (such as mien and mat), and any cData stored in the comment can be recovered later on, both by user interfaces and MIEN scripts.
Add a Sine Wave Signal
Now we are going to generate some signals and add them to the blank data we just created. Older versions of MIEN used a separate Synthesizer GUI to generate signals, but in the current versions, signal generation functions are being moved to the DV and DSP GUIs, using some DSP extensions. The signal generator extensions used here are built in to the MIEN core, so you won't need any extension blocks installed, but you will need to use functions from the DSP menu of DV.
Before continuing, if you have any markers displayed, select the DV window and press "k" to remove them.
Select DV->DSP->generators->Sine. The first parameter in the resulting dialog is a MIEN data selection tuple. This structure allows you to select any subset of the data for modification. We will use it in a later step. For now, leave it at the default value (None, None, None). The remaining values specify the frequency, amplitude, and phase of a sine wave. Set the frequency to 5.0 Hz. Leave the other values at the defaults, and click OK. Press q to re-center the view. Now you should see a sine wave.
Add Some Noise
Select DV->DSP->generators->GWN. In the dialog, set bandMax to 500, and std to .1. This specifies a band-limited gaussian white noise signal with a pass band of 5 to 500 Hz, and a standard deviation of .1 (since we just generated this signal, the meaning of the Y axis units is not defined yet. If we are planning to play this signal to DAC board later on, these units will probably end up being Volts. In any case, the std dev for this noise is now 1/10 of the amplitude of the sine wave we added in the last step). The other parameters can stay at their default values. In addition to a selection tuple, there are parameters rseed and invert, which are useful for generating "frozen" noise. By specifying rseed, we could control the state of the random number generator, to reproducibly generate the same noise sample several times, and we could also set "invert" to True, to generate the exact inverse of a previous noise sample. For now, leave rseed=None, to generate a new pseudo-random noise sample. Click OK.
You will see slight changes to the sign wave signal, but the noise won't be very visible at full scale. If you want, zoom in using the method from the last tutorial to get a close look at the noise. When you are done, press q to return to the default overview.
Add a Second Sine Wave to Part of the Waveform
Now it's time to do something useful with a data selection tuple. What we are going to do is add a higher frequency signal, but only to the middle third of the waveform.
Select DV->DSP->generators->Sine again. This time, set select to (None, None, [33000, 66000]). Leave the other values at their defaults and click OK. Notice that the signal has been changed, but only in the middle, not on either end. Zoom in to get a close look at the changes if you want. In addition to zooming by setting two markers and pressing z, you can zoom by using a left-click-and-drag over the area you would like to see.
Selection tuples allow you to specify what data element, channels, and samples to work on when running a function. They are described in the data api docs. Most of the time, you don't need to type to build them, because DV provides a browser for selecting data. In the example above, we told DV to work with the top level data set, all channels, and the range of samples starting with 33000 and ending with 65999. We could also have written that as ("/", [0], [33000, 66000]); DV replaces None entries with values that mean "the top level element", "all the channels", and "all the samples".
We can do this again using the selection browser, but first, save your file, since it is used in tutorial 3. Select DV->File->Save As, leave Format set to guess from file name, and use the file name dvtut2.mien.
Select DV->File->Preferences. Set Processing Tools Act On to Marked Range, and click OK. Now, set two markers, one near the start of the data, and one near the 1/3 point (setting markers is described in tutorial 1). Once they are there, select DV->DSP->generators->Sine again. In the sine wave dialog, the select parameter will already be set to something like (None, None, (1558, 33636)) (the exact values depend on where you clicked to set your markers. The preference you set tells DV to guess the default value for data selection by using the positions of your markers (it could also use the whole data set, or the current view). You could accept the new default value, but instead, click the Browse button on the select line.
If there were several data elements in this data set, you would first see a dialog asking you to choose an element. This would set the first parameter of the selection tuple. Our data set only has one element, so this value is automatically set to None (or, equivalently, "/"), and you are moved to a second dialog, where you can choose the second and third parameters, Channels, and Range. This dialog has three entries. The first entry, named "Channels", is a list. You can select the channels you want to use here. This list supports multiple selection, using Shift-click and CTRL-click (CMD-click on Mac), so if there were several channels, you could choose any set. Our data set has only one channel, so there isn't much to do here either. Selecting no channels is equivalent to selecting them all (if you really want to operate on no channels, cancel the operation!). The remaining two entries allow you to set the range of samples to use. You can type numbers into these entries by hand, or you can select values from the pull-down lists at the right. These list contain indexes DV thought you might be interested in, including the locations of any markers that you have set.
When you are done setting values, click OK in the browser dialog. You will be returned to the sine wave generator dialog, and the selection tuple value you specified in the browser will now be entered in the select parameter field. All MIEN parameter browsers work the same way, including color, file name, selection, and element browsers. Each one simply helps you to generate a Python data type to specify the thing you want. There is no difference between generating this code with a browser and typing it with the keyboard. Also, you can generate an entry with a browser to get an idea of what the parameter needs to look like, and then edit it by hand later.
We don't really want to add anymore sine waves, so either click Cancel, or click OK to see how the function works, but don't save the change to a file.
Filter a Signal
This tutorial uses the file "dvtut2.mien" that you generated in the last tutorial.
Open the Data
Open a terminal, and use the command mien -a data to open DV. Select File->Load. Browse to the directory where you saved "dvtut2.mien", select that file, and say OK to load it. (You could also have moved to your terminal to this directory with cd, and used mien -a data dvtut2.mien to open the file in DV. That would usually be faster, but some folks like GUI file browsers).
Apply a Filter
This file contains sine waves at 5 and 100 Hz, and some noise in the 5-500 Hz band. Suppose that you are really interested in the 5 Hz component, and the high frequency signals are some sort of noise. The built-in DSP blocks provide some simple conditioning functions, including a bandpass filter, which can get rid of most of the high frequency signals.
Select DV->DSP->signal->bandpass. In the resulting dialog, set minFreq to 1.0 and maxFreq to 40.0. Leave select at the default of (None, None, None). Click OK. The signal should look much simpler now.