wr: the workflow runner
wr is a workflow runner. You use it to run the commands in your workflow easily, automatically, reliably, with repeatability, all while making optimal use of your available computing resources.
wr is implemented as a polling-free in-memory job queue with an on-disk acid transactional embedded database, written in go.
- High performance
Low latency and overhead, plus high performance at scale, lets you confidently run any number of any kind of command.
- Real-time
Real-time status updates with a view on all your workflows on one screen.
- History
Permanent searchable history of all the commands you ever ran, with details and summary statistics on how long they took to run and how much memory they used.
- Dependencies
“Live” dependencies allow for easy automation of on-going projects. Read more about Dependencies.
- OpenStack
Best-in-class OpenStack support, with increadibly easy deployment and auto-scaling up and down. All without you having to know anything about OpenStack. Read more about OpenStack.
- S3
Mount S3-like object stores, for an easy way to run commands against remote files whilst enjoying high performance. Read more about S3.
- Containers
Built-in support for working with docker and singularity containers.
wr is being used in production by multiple groups at the Sanger Institute and has processed hundreds of TB of data and run millions of commands.
First steps
Tutorial: Basics (Why use wr? What does it do? How do you use it?)
Getting started: Install & configure | Start the manager (resolve problems with that) | Add jobs | Check status
Schedulers
After you’ve added jobs to wr’s queue, wr schedules the jobs so they will be executed on your compute resources efficiently. You can learn more about how it schedules in general, but you probably just need to pick the right scheduler for your circumstances:
Local for executing on just your local machine.
LSF for executing on an LSF cluster.
OpenStack for executing on an OpenStack cluser.
Advanced topics
Learn about some of the more advanced features of wr.
Integrations
While wr can be used by itself to execute workflows, you need to manually figure out a way to specify all the commands you want to run and their dependencies.
Other workflow management systems offer ways of easily specifying your workflows in general and shareable ways, but might not be as efficient or capable as wr in actually scheduling, executing and tracking the commands.
For these systems, wr can integrate with them for use as an execution “backend”, giving you the best of both worlds. Learn how to use wr as a backend for:
How-to Guides
These guides will help walk you through specific use cases.
Get in touch
The best way to report bugs or make feature requests is to create an issue on github.
To chat with the developers and get help or discuss feature requests, join us on gitter.
If you’d like to contribute to wr’s development, follow this guide.