Skip to content

Run Workflows Across Nodes

This page shows how to run a Snakemake workflow on the TES services of several Nodes, with the demonstrator demo-tes-hybrid-cloud. The workflow splits into steps that a GA4GH Task Execution Service (TES) runs as tasks. The demonstrator runs it on each TES service in turn, and then on all of them together through the proTES gateway, which forwards each task to one of the TES services.

The demonstrator runs with the current products: its CI runs it end to end against proTES 0.6.0 and two Funnel instances. Try it on your computer first, then run it through the deployment's proTES or on your own services. The services of the deployment have no guaranteed uptime; if one doesn't answer, write to cloud-service@elixir-europe.org. See Demos for the state of all demos.

Cited version

Publications cite the release 1.0.0 (10.5281/zenodo.17038656), which was built for the services of 2023. The steps below follow the default branch, which runs with the current products; check out the tag 1.0.0 only to reproduce the cited version.

Try it out locally

The directory local of the repository runs all services on one machine with Docker Compose: two Funnel instances, proTES in front of them with a stand-in identity provider, and an S3 service. You need Docker with Compose and access to the internet. From a clone of the repository:

docker compose -f local/compose.yaml run --build --rm demo

The run ends with PASSED: the demo ran end to end. The section "Try it out locally" of the README has the details and the commands to remove the setup afterwards. All credentials of the setup are dummy values: use it for tests only.

Run it on your services

You need:

  • Git, and Conda or Mamba (the README lists all requirements);
  • the address of a proTES gateway, and an access token for it: e.g., the deployment's proTES at https://protes.rahtiapp.fi, with an LS Login access token from Krini (see Which TES services you can use);
  • the addresses of the TES services that the notebook runs the workflow on one by one, and their credentials: the TES services behind your gateway, or others that you may use;
  • an S3 bucket that the TES services can write to, and its credentials. The workflow writes its results there.

Then follow the README of the repository: install the Conda environment, create the files .tes_instances and .env as its section "Create configuration files" shows, and run the notebook demo.ipynb. The notebook runs the workflow on each TES service in turn, and then through the gateway, and sends each service only its own credentials. The file .env holds passwords and a token: never commit or share it.

proTES sends each task to one of its TES services at random by default; the operator of the gateway can enable routing by distance to the data.

Use your own workflow

Snakemake, Nextflow and cwl-tes can send the tasks of any workflow to a TES service. See Run workflows through TES.