Mien Extension Overview

Mien attempts to provide a core set of functionality that is useful in a variety of modeling and data analysis contexts. These functions are provided by the mien "core" (the svn package "mien"). In addition, Mien provides a framework for quickly extending functionality with task specific capabilities. Some extensions are already provided with mien (by the mienblocks module), and users should be able to quickly write there own new extensions as well. Mien extensions are called "blocks" and are implemented as Python packages, although in some cases these packages may be very thin wrappers around C, Fortran, Matlab, or other tools. This document is intended for readers who are familiar with the basic functions of the mien core, have some experience with programming (although this experience doesn't need to be extensive), and would like to write their own mien extensions.

Extension terminology

Mien extensions are called "blocks". Extensions are implemented as Python packages, which contain Python modules, functions, and classes. In this document, the following terminology will be used:

A Package is any directory on disk that contains a file named __init__.py. Probably, it also contains other ".py" files, and maybe other directories, that may themselves be packages. An "extension package" is any package that is located as (depth 1) child of the MIEN_PACKAGE_DIR directory (see "How Mien finds extensions" below). "All files" in a package refers to every file with a ".py" extension under the directory - this search is recursive only if subdirectories are also packages. Every Mien block is stored in an extension package.

A Module is any file on disk with the '.py' extension, containing python code. Extension modules are modules that are located in an extension package.

A Module Attribute is any name defined in a module. This may be a module level variable, function, or class. It can also be a name imported into the module by an "import" or "from" statement.

A Class or Function is a python object defined within a module. Any number of classes or functions may be defined in a single module.

A Mien Block is a chunk of functionality. In simple cases, a block will correspond to a function, but it might be a set of functions and classes with a single launcher function that acts as an entry point for the block (footnote 1). I will also refer to block arguments. In a complex block, these are the arguments expected by the entry point. If the entry point is a class, these are then the arguments expected by that class's "__init__" method.

Should I write a block

Blocks can be used to provide additional functions to the primary mien guis, extend the capabilities of the nmpml xml dialect (and, by extension, of mien models), add support for new file types, and provide additional functions to the Dsp and Spatial toolkits. If you constantly wish there was a menu command "DoMyCoolProcedure" in CellViewer, you should write a block. If you are making a variety of models that reuse a particular function (e.g. a noise reduction function, or an algorithm for distributing ion channels), or you want to build an optimization routine that uses this function, you should write a block.

If you are doing a one-time operation, especially if it is a complex multi-step operation, you might be better off writing a script using the mien libraries, or doing your task from the mien command line interface.

If you already have a function that operates on numerical data, it isn't written in python, but you want to use it with mien (for example in Dataviewer or in a DSP tool chain) you probably don't need to write a block. You can write a wrapper around your function, in any language that you like, that reads data from some file type mien supports, calls the function, and writes the data back to the same file it read it from. You can then use the fileSystemCall DSP block in the built-in mien.dsp.nmpml package to call your external function. This is a common method for calling Matlab functions. Performance will be slightly better if you write a block. If you have a performance-critical algorithm in C, you may be better off wrapping that algorithm in a python C extension or SWIG library, and using the wrapped function call inside a mien block.

How Mien finds extensions

Mien finds extensions by looking for packages in extension directories. Usually, there are at most two these, a system extension directory and a user extension directory. Both are optional, so some systems may have only one, only the other, or neither. The system extension directory is PYSITE/mienblocks, where PYSITE is the site-packages directory of the Python interpreter that is running MIEN (for example /usr/local/lib/python2.5/site-packages on many Linux machines). The user extension directory is defined by the value of the environment variable MIEN_EXTENSION_DIR. If the variable is not set, it defaults to $HOME/mienblocks. (MIEN requires a value for HOME. On systems that don't set HOME, such as Windows, MIEN will set HOME to the current working directory at runtime. This can sometimes cause problems. If you run MIEN on Windows, it is recommended to set the value of the environment variable HOME to refer to some directory.)

Any Python package in the extension directories is evaluated as a block extension. Since blocks publish their provided features in the __init__.py file, packages that are not blocks, and thus don't name any MIEN features in __init__, will be registered as empty blocks. This shouldn't hurt anything.

Only packages at depth 1 in the extension directory are scanned. Block packages can contain sub-packages, but a block can't be included in a sub-directory of a non-package directory.

Mien automatically adds MIEN_EXTENSION_DIR to the python import path. You don't need to do that yourself. If your extensions import python modules that aren't part of an extension package, they may need to add the path for those modules to sys.path befor importing them.

Mien expects extension packages to provide some indexing information. In general, the file __init__.py that specifies a package may be empty. For a mien extension, however, this module must provide some attributes indication that it contains Mien Blocks of particular types. See "Writing the package index" below for more information.

What types of extensions are there

Mien blocks have different types determined by what the block arguments are. This will also determine what contexts the blocks can be called in, and what UI menus they appear in.

Block types include DSP, SPATIAL, DV, CV, ME, MECM, PARSERS, and NMPML.

DSP

These blocks expect an argument list in which the first argument is an instance of the nmpml Data class (a carrier object for numerical data and associated metadata), and the remaining argument list can be of any length, and contain arguments of any python built-in data type.

These are probably the most common blocks. They were the first motivation for adding block support, and mien provides a large number of DSP blocks built in to the core code. They are called DSP (digital signal processing) blocks since a common use is to transform time-series sampled data using some data processing algorithm (e.g. a digital filter). They can also be used for other tasks related to numerical data, however - writing it to files, calculating statistics from it, printing information about it, etc. The determining aspect is simply that the first argument to the block is a Data instance and the remain arguments (if they exist) are basic data types.

DSP blocks appear in the DSP menu of the Dataviewer, and in the function selection browser of the DSP GUI. Any DSP block can be assigned as the "Function" attribute of a MienBlock instance inside an AbstractModel. Class AbstractModel provides the foundation for abstract mathematical models in Mien, and AbstractModel instances operate primarily by stringing together DSP block functions (that are wrapped in MienBlock instances). Batch data processing and numerical optimization both depend on building an Abstract model. The DSP GUI panel is devoted exclusively to assembling DSP blocks into an Abstract model.

DSP functions can't reliably acquire references to the active GUI windows, so they shouldn't be used to provide display or user interaction functions. Stick to transforming data. Data transformations that are made by DSP blocks will be automatically reflected by the displays of any open GUIs that are viewing that Data instance, however.

Although I'm calling them functions (and will continue to do so), DSP blocks act like procedures. They should modify their Data argument in place, and return None

SPATIAL

Spatial blocks expect at least two arguments. The first is an nmpml Document instance (containing an entire Mien model), and the second is a list of upaths to elements in the document. (Upaths specify individual elements in the model. If you use XML tools, you will recognize upath as being similar to xpath, with the difference that upaths are guaranteed unique, and can thus be used to implement reliable references to a particular instance of a tag in the document.) There may be additional arguments, which must be basic data types.

Spatial blocks operate much like DSP blocks. They are called "spatial" because they were originally developed to handle modifications to 3D morphological data structures (e.g. Cells, Fiducials, Sections), including scaling and distribution of channels and electrical properties. Like DSP functions, however, they are quite flexible, and can be used for any operation that can be specified with the allowed arguments. If you want a function that operates on a Document instance, but doesn't care about a list of selected elements, you can still use a Spatial block. Just give the second argument a default value of on empty list, and ignore it inside your function.

Spatial blocks appear in the "Spatial" menu of the Cellviewer GUI. They can also be used in the context of optimization or Abstract models, but not directly. You will need to call them via the mien.dsp.nmpml.spatialBlock built in DSP block.

Like DSP blocks, spatial blocks shouldn't expect to manipulate UI state, and should modify their Document argument in place and return None.

DV, CV, and ME

These blocks are all quite similar. They expect one argument only, which is an instance of one of the principal Mien GUIs - the Dataviewer, Cellviewer, or Model Editor, respectively.

These blocks will be added to the "Extensions" menu of the appropriate Mien GUI. They can be used in two main ways. The first is to add a new menu callback to the GUI by using a block which is a simple function. Since the function receives a pointer to the GUI it can access GUI state and use the GUIs user interaction methods.

A second approach is to have the block be a class that itself implements a whole new GUI panel. This new gui can be linked to the launching GUI reference its document and state. Note that this class will instantiated by a single call to its __init__ method with one argument (the calling gui), so the __init__ method needs to handle that case, and do everything needed to display the new gui (for example, call self.Show(True)). If this isn't desirable, you can also write a launcher function for your new gui class, and supply this launcher as the entry point for your block.

The mien block handler doesn't discriminate between simple function that do something and exit, and complex classes that launch a whole new gui. Consequently the parent gui won't be "aware" of your extension gui. If you need correspondence between them, you will have to handle this within the extension block. Note, however, that if your extension GUI is a subclass of mien.wx.base.BaseGui, it will inherit the update_all/update_self API common to all mien apps, and can expect to receive update_self calls whenever any other Mien GUI modifies the document. The BaseGui class is quite flexible, and I recommend subclassing it for any complex extension guis that need to care about maintaining state.

MECM

These blocks add functions to the Model Editor's contextual (right click) menu. The entry point for one of these blocks isn't a function or class. Instead, it is a tuple with two elements. The first element is a function (or class) that implements the block functionality. The second is a string, tuple, or function that is used to determine what contexts the block can be used in (and thus when it should be displayed in the contextual menu).

The first element should expect to receive two arguments. First, a reference to the calling GUI (the model editor), and second a list of instances, containing all of the elements that are currently selected in the model editor.

The second element can have three forms:

• If it is a string (or unicode object), then the block will be displayed on the context menu only when the current selection is a single element with an nmpml tag type (e.g.g "Data", "Cell", or whatever) that is the same as the string. The block's function can thus assume that it will only ever be called with the instance list containing a single instance of the specified tag type.

• If it is a list (of strings), the block will only be displayed if the current selection is a single element and its tag type is one of the strings in the list. The block function can assume that the instance list it gets will have one element with one of the indicated types.

• If it is callable, then it should take a single argument of type List of Strings, and return Bool. When the contextual menu is pulled up, Mien will pass a list of the tag types of all the currently selected elements to this function. If it returns true, the block will be included in the context menu. Since you wrote this function, only you know what constraints can be placed on the list of instances that will be passed to your block function.

MECM type extensions are extremely useful for adding introspection features (for example, functions that can calculate and display some metadata or statistics about elements of a particular type). They can also be used to add model building features. For example, if your lab has a cool algorithm for distributing sodium channels, and you use it to add sodium channels to all your new cell models, you might want to provide an MECM block that looks like (NaDistributionAlg, 'Cell'), so that you can quickly dump sodium channels into a Cell model by right clicking on it in the model editor.

Parsers

Parser blocks provide support for additional file formats. Each parser block describes a file format, and provides a mapping between that type of file and some structure of Mien nmpml objects. If you provide a parser block with read/write support, you should gain full use of all Mien's function on files of your format, including the ability to save as you file format from any mien UI or DSP/Spatial block, and to convert from that format to any other format Mien supports.

These blocks are probably the most unusual and tricky to write, but they can be vary useful. Unlike all other block types, the entry point is _not_ a callable object. Instead it is a dictionary (hash table) describing the file type.

Writing parser modules is somewhat complex compared to other extensions. Read the whole howto for writing a parser module before you start. Failure to observe certain subtleties can result in circular import errors and cause the whole Mien package (including code in the core) to fail on startup.

If you have a Mien bug that you think may be caused by extension code, all you need to do is move the suspect extension package out of MIEN_EXTENSION_DIR and restart mien. The extension functionality will be lost, but all the core functions should still be working normally.

Note that in order to write a parser, you need to have nmpml tag types that provide appropriate ways to represent the data that is in your file type. For common neuroscience data, these should already exist in core Mien, but if, for example, you would like to add extensive support for spreadsheet files, you will likely need to include an NMPML extension block (probably a subclass of Data or Table) as well as a parser.

NMPML

These blocks have to be classes. More specifically, they have to be sub-Classes of mien.nmpml.basic_tools.NmpmlObject. The __init__ method of this class expects a dictionary argument representing a node in an XML file. You shouldn't modify this behavior.

Nmpml blocks add allowed tags to the nmpml dialect of XML. Additionally, they add python classes to the set of possible element types that can be represented as instances in the tree of objects stored in a a Mien Document. These classes can have special properties and functions. For example, Mien's built-in class "Cell" provides many special methods that are only appropriate to compartmental models of cells (for example, methods that can measure dendrite length). If you want to add a model type with new functions that are particular to that type of model, you may want to add an NMPML block for your new type.

Like parsers, nmpml extensions are trickier to write. They require some knowledge of the internals of the NmpmlObject class, and they have a reasonable chance of fatally fubarring mien if they don't work. Removing a corrupt extension from MIEN_EXTENSION_DIR (as suggested above) should get mien working again, but any files that were written while the bogus nmpml extension was loaded may be corrupted.

footnote 1: Parser and MECM type blocks have unusual entry points. See the sections parsers and nmpml for details