LunchBytes: Workflow Managers

21 July 2022 - 13:30-14:30
Google Meet
Will Furnass, Magda Dabrowska, Ian Sudbery

All of our events may be recorded and shared via the University of Sheffield Kaltura platform so those who cannot attend may still benefit. We will consider your attendance implict consent to this.

Scientific workflows (for example in bioinformatics) often require several different scripts or pieces of software to be run and their outputs fed into each other. Ideally, we’d like to be able to understand the provenance of the output, without having to re-run the entire workflow every time a minor change is made to code or data. This is often difficult to achieve with bash or Python scripts (for example), but workflow managers can help.

In this session we will learn more about the problems of controlling workflows / pipelines using scripts, and about what workflow managers are, and how they can help.

Save the date! More information to follow.

The trouble with scripts (Will Furnass)

Details TBA.

Nextflow (Magda Dabrowska)

Details TBA.

Ruffus (Ian Sudbery)

Ruffus is one of the oldest of the modern style workflow management systems. Ruffus pipelines consist of a series of python functions that are linked together using a python feature known as decorators. Ruffus is a lighter weight alternative to systems such as Nextflow, while still offering the ability to create rich dependency graphs of tasks and orchestrate their submission to an HPC. It embodies the philosophy that for users to choose to use a reproducible pipeline in the real world, when under pressure, using a pipeline must be as easy, or easier, than the manual alternative. In my talk I will demonstrate the creation of a pipeline for a bioinformatics task consisting of multiple non-trival steps within 15 minutes.

Joining Instructions

This session will take place on Google Meet and participants can join 15 minutes before the start of the session.

We also have a Google Jam Board where you can note down any questions or comments before or during the event.

Contact Us

For queries relating to collaborating with the RSE team on projects:

Information and access to JADE II and Bede.

Join our mailing list so as to be notified when we advertise talks and workshops by subscribing to this Google Group.

Queries regarding free research computing support/guidance should be raised via our Code clinic or directed to the University IT helpdesk.