| CARVIEW |
Cylc
Cylc is a general purpose workflow engine with a particular gift for cycling.
What is Cylc?
Cylc is a decentralised, distributed, DAG/DCG scheduler.
Scheduler
Cylc can schedule inter-dependent collections of tasks (aka workflows) to run on platforms of your choice.
Decentralised
Cylc employs a decentralised architecture where each workflow execution is fully independent. This means that workflows are isolated for robustness and Cylc deployments scale horizontally.
Distributed
Cylc workflows can be distributed across a pool of servers to facilitate large multi-user setups. Workflows can distribute their workload across multiple platforms.
DAG / DCG
Cylc can schedule Directed Acyclic Graphs of tasks (i.e. workflows which do not cycle) similar to other tools.
Cylc's unique cycling model allows it to run cyclic workflows too. This makes Cylc especially powerful for cases where later repetitions of tasks depend on the output of earlier instances. Tasks can repeat on different intervals within the same workflow and complex inter-cycle dependencies can be defined between them.
Features
Scalable
Thanks to Cylc's horizontal architecture, deployments can scale to any number of workflows. Cylc's novel scheduling algorithm allows it to schedule complex and even infinite workflows.
Flexible
Start with a simple linear pipeline, grow it into a complex branched cycling workflow.
Production Ready
Used in production since 2010, trusted by institutions worldwide.
Cycling
Cylc's cycling can do much more than just run a DAG to a schedule.
Different parts of your workflow can cycle to different schedules, tasks can have inter-cycle dependencies and multiple cycles can run simultaneously.
Advanced Scheduling
Cylc supports branched graphs where runtime events direct the direction the workflow takes.
Cycling workflows can run in real-time or simulated time with multiple calendars supported. And much more
Powerful User Interfaces
The Cylc dashboard (the GUI) provides a multi-workflow multi-pane interface for monitoring and controlling workflows.
View job logs and outputs, re-run subgraphs, modify the workflow configuration on the fly and more.
Alongside the GUI is a feature complete command line with an interactive in-terminal user interface (the Tui).
Jupyter Integration
Deploy the Cylc dashboard via Jupyter Hub to unlock institution-scale multi-user deployments with fine-grained authorisation allowing users to be granted permissions to view and control workflows running under other user accounts.
Deploy the Cylc dashboard alongside Jupyter Lab for an all-in-one virtual workstation in the browser.
Publications
If you use Cylc in your work, please consider citing us.
Workflow Automation for Cycling Systems: The
Cylc Workflow Engine
H. Oliver et al
Computing in Science & Engineering Vol
21, Issue 4, July/Aug 2019. DOI: 10.1109/MCSE.2019.2906593
Cylc: A Workflow Engine for Cycling Systems
Oliver et al., (2018).
Journal of Open Source Software, 3(27), 737.
DOI: 10.21105/joss.00737
Citable DOI for Cylc code releases.
Support
Community support is available on the Discourse forum .
Commercial support for Cylc is available from Altair, a leading provider of workload management and scheduling resources for the weather industry. Learn more about the Altair Weather Solution .
Testimony
Cylc has been used to orchestrate all climate, weather, and
weather-driven forecasting workflows at NIWA, for research and
production, since 2010.
On deployment to production at NIWA, Cylc's automatic cycle-interleaving
cut catch-up time from serious delays down from ~24 hours to ~30
minutes.
- Hillary Oliver, NIWA
Cylc has been used at the Met Office since 2012 and operationally since 2014.
“Every week we have 300+ users who rely on Cylc. At any one time we
typically have 700+ active workflows. Of the 11.7 million jobs run on
our HPC systems in February 2020, 99.6% of them were submitted by Cylc.”
- David Matthews, Met Office