Development of a workflow system for the CARMEN Neuroscience Portal

We discuss the development of a workflow system for the CARMEN neuroscience portal. CARMEN is a virtual (browser- based) collaboration environment for the analysis and management of electrophysiology data. It has been designed to facilitate community based sharing of tools, services and data relating to neurophysiology research. To date, CARMEN has provided mechanisms to allow legacy software code and applications from the lab to be deployed, via a ligtweight wrapping process, as interactive services that run on the CARMEN cloud resource. This is a software as a service (SaaS) delivery model. This approach offers many benefits to the neuroscience community, including the reuse of software applications, the ability to compare processing algorithms and access to considerable computing CARMEN resource for high intensity computing tasks. However, many researchers would derive extra benefit from being able to string combine analysis services together in an orchestrated processing pipeline, or workflow. The CARMEN project has now addressed this requirement and developed a workflow generation and execution system within the platform.
CARMEN has relatively specific workflow requirements due to its cloud execution model and the use of the it's NDF data format (www.carmen.org.uk/standards/CarmenDataSpecs.pdf). Hence, although we evaluated current workflow tools, such as Taverna (www.taverna.org.uk/) and E-Science Central (www.esciencecentral.co.uk/), it was found that neither met the functional requirements and a bespoke workflow service was developed. The CARMEN Workflow Tool is Java-based and designed to make use of CARMEN Services and NDF. The workflow tool supports both data and control flow, and allows parallel execution of services. Using a Service Invocation API to invoke CARMEN services simplifies the workflow infrastructure substantially. This API allows the workflow engine to make use of our dynamic service deployment and execution system, achieving scalable heterogeneous distributed processing. The complete workflow tool consists of a graphical design tool, a workflow engine, and access to a library of CARMEN services and common workflow tasks.
The poster will provide an overview of the main design considerations for the workflow system and provide an overview of its use within CARMEN. We provide details of the workflow execution engine and describe the XML scripting approach that has been developed to control the workflow orchestration. The workflow description XML schema follows a similar form to myGrid’s SCUFL (Simple Conceptual Unified Flow Language - www.mygrid.org.uk/dev/wiki/display/developer/SCUFL2) script but modified to suit our service and data description formats. Example workflows for neurophysiology will be presented, demonstrating the flexibility of the workflow tool to support complex analysis pipelines.