DataViewer

Throughout the MIEN documentation, the Data Viewer will often be referred to as DV, and the syntax DV->Menu->Option will be used to mean "Select the choice named 'option' from the menu named 'Menu' in the menubar on the Data Viewer window". This may be used interchangeably with Menu->option if only DV is open, but often MIEN sessions will have several open GUIs, so DV->File, X->File, and CV->File refer to the file menus on different windows.

Starting Data Viewer

Start the GUI using the command mien -a data. Optionally, you can append the name of one or more data files to open at the end of the command.

Loading data

If you didn't pass in the names of any data files, there are two other ways to load data into Data Viewer; Drag and Drop, and the File menu.

To load data by drag and drop, drag a local file from your Desktop or from a graphical file manager, and drop the file onto the black graphing area in the center of the Data Viewer. Dropping files here will cause DV to open them in append mode. This means that the data in each file will be concatenated with any data that is currently open in DV.

To load from the File menu, use File->Load, File->Append, and File-Append Subset. Each of these commands will open a file dialog where you can select a file to open, and will import data from that file. In the case of "Load", these new data will replace any existing data. In the case of "Append" they will be concatenated with the existing data. "Append Subset" allows you to choose parts of a complex document for import, rather than importing the entire contents. "Load" is exactly equivalent to "New Document" followed by "Append", so if you would like to "Load Subset", just call "New Document" before calling "Append Subset".

If you attempt to load data and nothing displays, check your terminal for error messages. If the file loaded correctly, but DV reported "No numerical elements to display" then the problem is that MIEN can't figure out that your file contains numerical data. Here are some suggestions for fixing that.

Types of data to open

• If you use a DataMAX data acquisition system, you can import recorded data directly into MIEN's Data Viewer.

• If you use Matlab for data analysis, you can send data to the Data Viewer in a .mat file. Getting the data to load in MIEN is easy (just don't save sparse arrays, and you should be fine), but getting complex data to be correctly identified as the right data type can be harder. For now, try saving a single 2D array of evenly sampled data, with sample number in the rows of the array, and different channels or variables that are sampled in the columns.

• You can store numerical data in text files. Use new lines in the text file to separate samples, and within a line, use spaces to separate different measurements. If you like you can construct a header at the beginning using Key:Value pairs. The most useful thing to put in a header is the sampling rate. Use something like SamplesPerSecond:10000 to set this value. You don't need to use a header at all, though, if you just want Data Viewer to load and plot the data.

Why Are There Two Lines for Each Plot?

This is the all time most frequently asked question about DV. If you just loaded up a long data set containing two channels of rapidly varying measurements, and all went well you are probably looking at 4, not 2 wiggly lines; 2 green ones and 2 blue ones. Don't worry, this is normal behavior.

What is happening here is that you have many more data points than DV has pixels on the screen to draw them with. When this happens, DV doesn't waste time drawing many sub-pixel lines that you can't see. Instead it draws the "envelope" of the data: one line showing the upper bound, and one showing the lower bound. If you zoom in on a region enough that MIEN has at least one pixel per data point to draw with, you will see only one line. The envelope line style can be changed, but it is much faster than the full line style, and most users have found that it is clear and informative once they get used to it.

Navigating in DV

The core of DV is a MIEN 2D graph widget. This is the black area that takes up most of the DV window. The graph provides a number of navigation features itself. In addition, the DV frame surrounding the graph adds some navigation features of its own.

Graph navigation

Mouse behaviors

Left-clicking the mouse in a DV graph will draw a vertical marker. This is a vertical line with a label showing it's X coordinate. Markers can be used to measure things, to specify parameters (such as thresholds) for analysis functions, and to control zoom commands.

If you Left-click-and-drag rather than just clicking, you can zoom in on a region of the data. This zoom is a horizontal-only zoom. The range of X values displayed is changed but the range of Y values is not.

If you hold down shift while clicking, the behavior is changed from X-axis to Y-axis. A Shift-left-click-and-drag will perform a vertical-only zoom, and a Shift-left-click will draw a horizontal marker. The behavior of Shift is very common in 2-D graphs. You will find that other commands use Shift similarly; for example the keystroke "a" will zoom out horizontally, and "A" (Shift-a) will zoom out vertically.

Right-clicking brings up a contextual menu which gives you access to configuration details for the graph. You can save images, change colors and line styles, and show/hide plots. Note that if you make a custom configuration in the graph menu, you will need to select "Save Configuration" in order to inform DV of the changes. Otherwise, your custom configuration will only last until Dataviewer redraws the graph (for example because you opened new data or ran an analysis function).

Key Bindings

Most navigation on the graph is done using key bindings. You can get a list of the current set of bindings by using the right-click menu and choosing "Show Key Bindings". The Bindings will print in your controlling terminal. As of the time of this writing, the graph key bindings are as follows:

• a => xZoomOut: Double the range of x coordinates in the viewable area.

• A => yZoomOut: Double the range of y coordinates in the viewable area.

• q => FullScale: Set the limits of the viewable area automatically so as to display all plots

  This command is very useful if you get "lost" or do any operation that makes the data display disappear.

• z => Zoom the viewable region to the region between the last two markers.

  Operates in both x and y, but will not zoom an axis if there are fewer than two markers present on that axis.

• up, down, left, right => Pan the viewable region.

• x => delete the most recent vertical marker

• X => delete the most recent horizontal marker

• k => Remove all markers from both axes

• r => redraw the screen

• f => Store the current zoom level, zoom out to full scale view, and set a mouse binding so the the next left click will center the view on the clicked location and zoom back in to the saved size

• m => Set the x zoom level to the last stored zoom level, if there is one

Choosing Display Offsets.

At the bottom of the DV window there is a row of widgets including two pull-down menus and three text boxes. The most important of these is the right-hand pull-down which gives you access to several useful view presets.

The view presets determine how DV displays multiple channels of data. You can separate the channels in various ways, show them just as they are, overlap them, or show them in image mode. The default mode, quick sep separates each channel as though you had given each one different Y offsets on an oscilloscope. The difference between quick sep, local sep and separate is the algorithm used to determine the offsets. The most reliable and best looking is separate, but it can be slow on very large data sets (which is why it isn't the default, since then you would have to wait for it to run before you got a chance to turn it off). Initially, you should probably use separate mode, but feel free to play with the other modes and see if you like them.

Note that with the exception of image mode, all the view modes are simply preset algorithms for setting the 3 values associated to the three text boxes; X off(set), Y off(set), and Y sca(le). You can also set these manually. Use the left-hand pull down menu to select a channel. The text boxes will now reflect the offset and scale values for that channel. You can edit the value in a box and hit enter in order to display that channel with a different horizontal shift, vertical shift, or scaling factor.

If you get DV in a state where the offsets are not doing what you want, you can always select the raw view preset to remove all offsets and show the data just as they are. After that you can go back to a different preset, or set manual values, and things should be working as expected again.

Changing the display offsets doesn't change the actual data, only the view. Also, DV will compensate for the display offsets when sending parameters to analysis functions, so if you want to set a threshold on channel 4 (using a marker), you just put the marker at the level that looks right. You don't need to worry that channel 4 might be displayed with a vertical offset.

The offset controls in DV are designed to be somewhat similar to the offset controls on a multi-channel oscilloscope.

Navigation Commands in the DataViewer Menus

Some functions in the DV menus also have navigational effects, particularly functions in the Edit and Display menus.

Use of all the menu commands is described in the next section.

DV Menu Commands

If you are using Mac OS, read note about Mac menu behavior.

DV has 6 menus: File, Edit, Display, Extensions, DSP, and Python. Each is described in its own section below.

File Menu

Many of the functions in the DV file menu are standard to MIEN GUIs, and are described here.

The following functions are DV specific:

• Open Part of a File

  This differs from "Load Subset" in that it selects only certain channels or certain samples from the primary data element in a file. Load Subset allows you to load only certain elements, but all the data in a selected element will load. This function is useful if you have, for example, a data acquisition system, which may record gigabytes of data and store these in a single data element. Loading the whole data element at once may be very slow, or impossible (depending on your computer).

• Create Blank (all zeros) Data

  This function creates a top-level data element of "timeseries" type, with user specified size and sampling rate, and fills the set with zeros. When DV creates a new document, that document contains no data elements. This means that most data handling functions, and even display functions will fail. If you want to do something such as create a new signal using the various signal synthesis functions, you will need to create a data element first. You could do that in a more flexible way with the XML editor, but this function is faster for many common uses.

• Load SubData

  This function is obsolete. It is effectively equivalent to "Load Subset", although the element browser it uses is slightly different. "Load Subset" is preferred.

• Switch Viewed Data

  DV only displays one data element at a time. If there are many data elements in a document, this function can select which element to show in DV. Also, DV separates data into "datasets", which are nested groups of data elements. Other DV functions that move data around, or that operate on "subdata" will only move within one dataset. It is possible to construct a MIEN document with multiple distinct datasets (by grouping several data elements within an element of a non-data type). This Switch function is one of the few functions that can access data elements outside the displayed dataset, so it can be used to change the active dataset as well as the displayed element.

• Combine Data

  This function can be used to merge data from different elements. The effect of merging data depends on the DV preferences for data merge. Options include combining the data into a single element (provided the types are compatible), and nesting the data into a single dataset. Like Switch Viewed Data, this function can act on any data element in the document, so it can be used to add data from a different data set into the current dataset.

Edit Menu

• Undo

  Attempt to restore the state of the active data set to the state before the last editing action. Undo functionality depends on the preference for Undo memory depth. Setting a high Undo depth can make DV use too much memory. WARNING: Undo in MIEN is NOT RELIABLE. Providing robust Undo functions is on the MIEN feature development list, but there are technical difficulties with balancing performance, flexibility and simplicity of extension writing, and undo. Currently many functions break undo, and some disable it entirely. To be on the safe side, set checkpoints (as described below), as these are more reliable.

• Redo

  Reverse an "Undo".

• Save Checkpoint

  Stores the current state of the data to disk, allowing a later restore. This is more reliable than Undo and doesn't waste memory, but the checkpoints must be set manually, and setting a check point can take a few seconds. Checkpoints are persistent, meaning that they remain after closing and reopening MIEN, but if you edit a different file and set checkpoints, they will overwrite previous checkpoints, so it is still wise to save a file explicitly before exiting MIEN.

• Load Checkpoint

  Restore the data state to a saved checkpoint.

• Crop

  Delete some samples of the data. The crop function opens a menu offering several ways to crop, including to keep only the current view, delete data before or after vertical markers, or specify a range of samples to keep using numbers. Crop also asks if it should reset the start time of the data. Timeseries data usually start at time 0, but if you want to align a small chuck of data with a larger series later, you may want to store a time offset for a cropped chunk of data. Some extension functions won't support start times correctly, so if you don't need to use the start time, you might as well set it to 0. NOTE: if you view a small region of a timeseries, crop to view, and set start time to 0, all the data will disappear! Don't worry, it isn't really gone, it has just been shifted left, and you are left looking at a region of time after your data samples have ended. Use the "q" key to recenter the view on the data.

• Data Editor

  Launch the MIEN XML editor on the current document. You can use it to make changes to the document that will be reflected in DV.

• Delete Channels

  Remove some of the channels from the current data element. This function brings up a menu for selecting which channels to remove. This function compliments Crop. Crop changes the horizontal extent of all traces in a data set, and this function changes the number of traces.

Display Menu

• Marker locations

  Print the locations of all markers (horizontal and vertical) using full precision. The markers are labeled with their locations on screen, but these labels use limited precision, so to really measure things, this function is better. In addition to printing the values of the markers, the sample index of X markers is also printed. Output goes to the controlling terminal (or the status window, if there is no terminal).

• Marker difference

  If there are two markers of a given type, this function print the separation between them. Useful for measuring things.

• Redraw

  This plots the data again. It can be needed if a function changed the data and the DV display wasn't updated (usually the update is automatic, but some changes made by extensions or other GUIs might require manual refresh). This also removes any display customizations that were made using the Graph element's configuration menu (unless those changes were saved as a display style).

• Global Equalize

  Sets the Y Scale of every data channel so that the peak-to-peak value of each channel will take up the same size on screen. This adjusts only the display style, not the numerical values of the data.

• Local Equalize

  Sets the Y Scale so that the currently displayed regions of each channel have the same peak-to-peak display size.

• Raw Amplitudes

  Sets the Y Scale of every channel to 1.0.

Extensions

Note that this menu provides functions for managing extensions, as well as listing DV type extension block functions. The content of this menu may change if you add blocks. Only the built-in functions are documented here. Extension block functions for signal processing, that don't define custom GUIs, are listed in the DSP menu, rather than here.

• CalcStats

  Print the mean, max, min, and std dev for each time series data channel displayed.

• Create Events From Marks

  Generate an "events" type subdata element containing one event for each vertical marker. This is useful if you would like to event-condition or wrap data manually, or save your markers in the data file.

• DisplaySubData

  Launch a menu to select a sub-element of the current data set, and then launch a new viewer showing that element.

• Graphing Server

  Launch a server that listens for incoming display requests, and plots them using the current DV window. This allows text-based Python scripts or interactive sessions to send data to this DV for display. See the API documentation to learn how to send graphing requests to the server.

• Help for DSP Functions

  Print a list of the documentation for all installed and documented DSP extension functions.

• Hide Data

  Move some data into a hidden group, so that it remains in the dataset, but will not be displayed

• Reload DSP Functions

  If the files specifying any DSP or DV extensions have been changed since DV started, load the new definitions. This may change the behavior of existing functions, or register new functions, but it will not register new extension packages. If you have added a whole new package, you will need to restart DV. Note that if any of the reloaded extensions define GUIs which are currently open, DV will attempt to reload and relaunch them. This may not always work, depending on the extension. If it doesn't, you should always be able to close and relaunch the extension GUI by hand in order to access the new version.

• ShowData

  Move data out of the hidden group, allowing it to be displayed. This is intended to re-expose data hidden with "HideData", but it will usually also work on data hidden with the XML editor or the preferences menu.

DSP

This menu provides signal processing functions, including some defined by MIEN, and those defined by any loaded extensions. See the MIEN DSP documentation, or the documentation for the relevant block for usage of each function.

The functions are organized using Python package notation. If you install an extension block named "myext", which provides a module named "mydsp" with a function in it named "mytest", then you should see a sub-menu in the DV DSP menu named "myext.mydsp", an within that submenu, you will see a function "mytest".

When you activate a DSP function from this menu, MIEN auto-generates a GUI interface for you to enter the parameters for the function. How polished this interface is depends on the design of the function. Some functions provide polished step-by-step GUIs. The worst functions provide text entry fields in which you will need to enter correctly constructed Python code. Most extensions are somewhere in between. In particular, if a complex entry of Python code is needed, MIEN will often provide a "browser" helper function to construct this code for you. If you see an intimidating text field, don't know what to type, but see a "browse" button on the right, try pressing that. It should help.

The most common of the complex function parameters is the data selection tuple. This is a Python tuple with three elements: the path to the data to use, a list of channels to use, and a range of samples to use. For example: ('/filter', [0,1], [1000,10000]) specifies 9000 samples on two channels of the subdata element named "fiter". False arguments mean "all of it", or "the top level data", so (None, None, None) refers to all samples of all channels of the currently displayed main data element.

Python

The menu lets you launch Python interactive sessions that interact with the DV gui. See the general-purpose gui documentation.

Preferences

Choosing DV->File->Preferences launches the preference editor for DV. DV has the most extensive preferences menu of the MIEN GUIs. The majority of preferences handle how data are displayed. Some others handle what to do when loading or saving complex data.

• Use image mode for more than X channels

  This value is an integer, typically between 30 and 100. Displaying line plots very many channels of data can be slow. Consequently, opening a new file with several hundred channels may lock up DV for some time. This preference sets a maximum number of channels that will be displayed as line plots. Image display mode scales to larger numbers of channels better, so by forcing the display mode to image multi-channel files will open quicker. Forcing image view limits features, though, so on a powerful computer, you may want to set this number higher

• Image Contrast

  There are three choices for this value, and if "Manual" is chosen, it also takes two parameters, which are floating point numbers. This preference determines how data viewer sets the color scale when showing data in Image mode. In Image mode the display scaling and offset values used in other modes are ignored. Instead, the Y value of each data sample is mapped to a greyscale color, with smaller (more negative) values in black and larger values in white. There are only 256 levels of grey, so to see the details you want, you may need to adjust the map between numerical values and color using this preference.

• local: Black is set to the minimum currently displayed value, and white to the maximum. This offers maximal contrast, but it means that the value of a given color will change if you change the displayed data (by zooming, panning, or running DSP functions). Consequently, different views of the data can't be quantitatively compared using this mode.

• global: Black is set to the minimum value in the whole data array, and white to the maximum. This is the default value, and it offers good contrast and consistency between views for data that have consistent ranges. It will reduce contrast if there are a few substantial outliers in the data, and views can't be compared if the numerical values of the data change (for example through running a DSP function)

• Manual: Set the minimum (value to map to black) and maximum (value to map to white) by hand. Useful if you know what you are looking for, and it ensures comparability between all views of any data.

• Display Range

  Choose global, local, or manual. This controls how DV calculates automatic offsets for display modes like "separate". You usually want "global". "Manual" will turn offset calculation off, causing the traces to overlay in separated views (but saving some CPU cycles for really big data, and allowing you to see the exact numerical effects of certain DSP functions). "Local" is mostly obsolete. Use the "local sep" display mode instead.

• Processing Tools Act On

  Choose "Marked Range", "Current View" or "All Data". This parameter is used to determine the default values for DSP functions that have arguments that select data. If you edit these parameters by hand, then this preference has no effect. "Marked Range" refers to the space between the first two vertical markers (if there are at least two).

• Save Referenced Data Files When Saving XML

  Usually say "Yes" (and it usually doesn't matter). This preference is used if you save a local copy of a file that has pointers to remote data files. If this is "Yes" the data files will be copied locally as well. If you save in formats such as "mien" or "mdat" this parameter is ignored since those formats always save data locally. Also, if the referenced data files are local, or use the "auto" reference url type, then this preference has no effect.

• Ensemble Display Mode

  Data of "ensemble" type are generated by folding or event conditioning timeseries data. Ensembles consist of many repetitions of a data segment. DV can display these in several ways, as determined by this parameter. "Stat" shows three lines for each ensemble channel: the mean the mean plus one std dev, and the mean minus one std dev. Stats are calculated across repetitions in the ensemble, and each channel has separately calculated stats. "Overlay" shows every repetition plotted on top of each other. This can be visually confusing for big ensembles, but it can provide more information about higher order moments than stats. "Sequential" plots each repetition one after another, allowing you a close look at every instance. "Hide" instructs DV not to show ensemble data at all (useful because they can be expensive to display, so if you generate a big ensemble as an intermediate step in a sequence of DSP functions, you may not want to waste time plotting it).

• On Multiple Open

  Tells DV what to do with data elements after the first when opening multiple files, or a single complex file. "Nest" causes DV to add all additional elements to the current data set as subdata of the current data element. Any types of data may be nested. Depending on other preferences, the nested data may or may not be part of the default display image. "Merge" causes DV to attempt to add channels to the data array of the current top level data element. Only data of the same type, sampling rate, and length will merge. Other data types will nest. "Ignore" leaves additional elements where they were, in separate datasets. Data in different datasets can't be simultaneously accessed by DV or by one DSP function.

• Default View Mode

• This sets the view mode to use when opening DV. The options are the same as the modes selected from the view mode pulldown in a running DV. Modes like "Quick Sep" and "Image" and "Raw" will allow you to open big files faster. "Separate" will produce a prettier display.

• Always Reload Extension

  If you are not a developer, say "False". If this is True, then DV will reload all extension blocks every time you run a DSP or Extension function. This is convenient if you are trying to debug and extension or perfect a new algorithm, since if prevents you from having to use the "Reload DSP Functions" menu function before every test. If you aren't changing the extension source code a lot, this just wastes a bunch of CPU time.

• Show Only Same Length Data

  When the primary data element has subdata, DV uses several heuristics to decide which of these to plot in the main display. In general, subdata with sampled types (events, labeledevents, ensembles, and timeseries) and the same sampling rate are shown (unless explicitly hidden). If you set this value to "True", then ensembles and timeseries will not be displayed unless they also contain the same number of samples as the top level data. This is especially useful if you are running analysis functions that create many snippets from the top level data and save them in subdata. Displaying all those segments gets slow, and visually confusing.

• Simple Subsample

  Unless DV is running really slow, say "No". This determines the algorithm used to display data when there are many more data points to plot then pixels to plot them in. The simple subsample is faster than DVs default envelope algorithm, but it can lead to some nasty aliasing and misleading artifacts. If you are waiting 2 minutes for the plot to refresh, though, setting this to "Yes" may make things usable again.

• Default Plot Style

• This sets how DV shows timeseries data. The default, "envelope" is usually best. "Points" shows only the actual samples, without connecting lines, and is useful if you don't want to assume point-to-point correlation in the data. "Line" plots only a single line, rather than a 2-line envelope, even when there are very many samples in view. Both line and points are MUCH slower than envelope.

• Checkpoint Directory

  Pick a directory for saving checkpoint files in.

• Save Multiple Data as

  This preference is used if a dataset with many data elements is saved in a format that doesn't support nested elements. This has no effect when saving "mien", "mdat", or "mat". If you save a dataset with 10 elements in "txt", for example, 10 files are saved. "Automatic File Names" constructs unique names for each file. "Directory" constructs a directory tree that mirrors the nested element structure of the data set, and saves the files in the appropriate subdirectories.

• Display Nested Data To Depth

• This is an integer value, and it specifies how far down into a nested dataset DV should look for sub-data elements to add to the main display. If this is 0, only the primary data element is ever displayed (which speeds things up). If it is 1, only direct children of the primary element are searched, etc. Older versions of DV had the preference "Display Nested Data (yes/no)" which operated like switching this preference between 0 and infinity.

• Hide Data In

  This is a Python list of strings. The default value is "['hidden']", and there is usually no reason to change it. Sub-data elements that contain any of the strings in this list in their path are not displayed. The "Hide Data" menu option operates by creating a data group named "hidden" (if it doesn't already exist), and moving selected data elements into this group. Changing this preference to a value that doesn't contain the string "hidden" may break the "Hide Data" and "Show Data" functions.

• Number of Undo Steps

  This is an integer that tells the Undo function how many steps to remember. Smaller values save memory. 0 disables Undo. Note that some extensions (such as the spike sorter) will set this to 0 for you, and you will need to set it back manually in order to re-enable Undo.