JSON API Documentation

Disclaimer: The API is still under heavy development and breaking changes to endpoints or parameters may be made

If you make anything utilizing our API, please let us know about it - we would love to see!

All queries require the following header to receive a JSON response:

accept: application/json

Workflow

Add a new Workflow

Note: Due to cwltool run times, the creation of workflows is done asynchronously and an intermediate queue resource is used
POST
/workflows

Parameters

Field Type Description
url String The URL of the repository. Either a Gitlab or Github URL, or any Git repository URL. If a Gitlab or Github repository URL is used, the following fields will take priority over those parsed from the URL if provided.
branch String The branch of the repository. Required if the URL is not from Gitlab or Github.
path String The path within the repository to the workflow file. Required if the URL is not from Gitlab or Github.
packedId String The ID of the workflow, to be provided if the workflow file is packed (contains multiple descriptions with $graph).

Input

{
    "url": "https://github.com/common-workflow-language/workflows/tree/master/workflows/compile/compile1.cwl"
}
{
    "url": "https://bitbucket.org/markrobinson96/workflows.git",
    "branch": "master",
    "path": "/workflows/make-to-cwl/dna.cwl"
}

Success

HTTP/1.1 202 Accepted
Location: /queue/:queueid

Already Exists

HTTP/1.1 303 See Other
Location: /workflows/:URL

Packed Workflow with Multiple Choices

HTTP/1.1 422 Unprocessable Entity
{
    "message": "This workflow file is packed and contains multiple workflow descriptions. Please provide a packedId parameter with one of the following values"
    "packedId": [
        "workflow.cwl",
        "main",
        "workflow_exome.cwl",
        "align.cwl",
        "workflow.cwl_2"
    ]
}

Error

HTTP/1.1 400 Bad Request

Get Workflow Details

GET
/workflows/:url

Parameters

Field Type Description
url String Partial URL to the workflow eg "github.com/owner/repo/tree/branch/workflow.cwl" or "bitbucket.org/owner/repo.git/master/workflow.cwl

Success 200

Field Type Description
retrievedFrom.repoUrl String URL of the Github repository
retrievedFrom.branch String Github branch name or commit ID
retrievedFrom.path String Path to the workflow within the repository
retrievedFrom.packedId String The ID of the workflow within the file, if this is a packed CWL file (contains a document graph).
retrievedFrom.url URL The URL to the page on the site if type is not GENERIC, otherwise the same as repoUrl
retrievedFrom.rawUrl URL The URL directly to the file if type is not GENERIC, otherwise the same as repoUrl
retrievedFrom.type Enum One of GENERIC, GITHUB, GITLAB, BITBUCKET
retrievedOn Date The last time this workflow was updated from the branch
lastCommit String The last commit ID on the branch
label String The label string for the workflow from the CWL description
doc String The doc string for the workflow from the CWL description
inputs
outputs
steps
dockerLink String The URL to dockerhub or "true" if an unrecognised image is used in a docker hint or requirement
cwltoolVersion String The version of cwltool used to parse the workflow
visualisationDot String The DOT source used to generate the visualisation image
visualisationXdot URL A link to the xdot source code for the visualisation image
visualisationPng URL A link to the png format visualisation image
visualisationSvg URL A link to the svg format visualisation image
robundle URL A link to the Research Object Bundle download
Example URL
/workflows/github.com/common-workflow-language/workflows/tree/master/workflows/lobSTR/lobSTR-workflow.cwl
Example Response
{
    "retrievedFrom": {
        "repoUrl": "https://github.com/common-workflow-language/workflows.git",
        "branch": "master",
        "path": "workflows/lobSTR/lobSTR-workflow.cwl",
        "url": "https://github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
        "rawUrl": "https://raw.githubusercontent.com/common-workflow-language/workflows/master/workflows/lobSTR/lobSTR-workflow.cwl",
        "type": "GITHUB"
    },
    "retrievedOn": 1502714218616,
    "lastCommit": "17b65ea19d81527090fded62ffa0e1ba3b25d561",
    "label": "lobSTR-workflow.cwl",
    "inputs": {
        "reference": {
            "type": "File"
        },
        "rg-sample": {
            "type": "String"
        },
        "p1": {
            "type": "ee54066e73d068edfbf9ca5059e14acd"
        },
        "p2": {
            "type": "79ab371b6d0fdbadbf9a4a89ef393353"
        },
        "output_prefix": {
            "type": "String"
        },
        "rg-lib": {
            "type": "String"
        },
        "strinfo": {
            "type": "File"
        },
        "noise_model": {
            "type": "File"
        }
    },
    "outputs": {
        "vcf": {
            "type": "File"
        },
        "vcf_stats": {
            "type": "File"
        },
        "bam_stats": {
            "type": "File"
        },
        "bam": {
            "type": "File"
        }
    },
    "steps": {
        "allelotype": {
            "run": "allelotype.cwl",
            "runType": "COMMANDLINETOOL",
            "sources": {
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/bam": {
                    "sourceIDs": [
                        "samindex"
                    ]
                },
                "allelotype": {
                    "sourceIDs": [
                        "strinfo"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/noise_model": {
                    "sourceIDs": [
                        "noise_model"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/reference": {
                    "sourceIDs": [
                        "reference"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/output_prefix": {
                    "sourceIDs": [
                        "output_prefix"
                    ]
                }
            }
        },
        "samsort": {
            "run": "samtools-sort.cwl",
            "runType": "COMMANDLINETOOL",
            "sources": {
                "samsort": {
                    "sourceIDs": [
                        "lobSTR"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#samsort/output_name": {
                    "defaultVal": "\\\"aligned.sorted.bam\\\""
                }
            }
        },
        "lobSTR": {
            "label": "lobSTR",
            "run": "lobSTR-tool.cwl",
            "runType": "COMMANDLINETOOL",
            "sources": {
                "lobSTR": {
                    "sourceIDs": [
                        "rg-lib"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/p1": {
                    "sourceIDs": [
                        "p1"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/p2": {
                    "sourceIDs": [
                        "p2"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/output_prefix": {
                    "sourceIDs": [
                        "output_prefix"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/rg-sample": {
                    "sourceIDs": [
                        "rg-sample"
                    ]
                },
                "file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/reference": {
                    "sourceIDs": [
                        "reference"
                    ]
                }
            }
        },
        "samindex": {
            "run": "samtools-index.cwl",
            "runType": "COMMANDLINETOOL",
            "sources": {
                "samindex": {
                    "sourceIDs": [
                        "samsort"
                    ]
                }
            }
        }
    },
    "dockerLink": "https://hub.docker.com/r/rabix/lobstr",
    "cwltoolVersion": "1.0.20170810192106",
    "visualisationDot": "digraph workflow {\n  graph [\n    bgcolor = \"#eeeeee\"\n    color = \"black\"\n    fontsize = \"10\"\n    labeljust = \"left\"\n    clusterrank = \"local\"\n    ranksep = \"0.22\"\n    nodesep = \"0.05\"\n  ]\n  node [\n    fontname = \"Helvetica\"\n    fontsize = \"10\"\n    fontcolor = \"black\"\n    shape = \"record\"\n    height = \"0\"\n    width = \"0\"\n    color = \"black\"\n    fillcolor = \"lightgoldenrodyellow\"\n    style = \"filled\"\n  ];\n  edge [\n    fontname=\"Helvetica\"\n    fontsize=\"8\"\n    fontcolor=\"black\"\n    color=\"black\"\n    arrowsize=\"0.7\"\n  ];\n  subgraph cluster_inputs {\n    rank = \"same\";\n    style = \"dashed\";\n    label = \"Workflow Inputs\";\n    \"p2\" [fillcolor=\"#94DDF4\",label=\"p2\"];\n    \"p2\" [fillcolor=\"#94DDF4\",label=\"p2\"];\n    \"output_prefix\" [fillcolor=\"#94DDF4\",label=\"output_prefix\"];\n    \"reference\" [fillcolor=\"#94DDF4\",label=\"reference\"];\n    \"noise_model\" [fillcolor=\"#94DDF4\",label=\"noise_model\"];\n    \"strinfo\" [fillcolor=\"#94DDF4\",label=\"strinfo\"];\n    \"p1\" [fillcolor=\"#94DDF4\",label=\"p1\"];\n    \"p1\" [fillcolor=\"#94DDF4\",label=\"p1\"];\n    \"rg-lib\" [fillcolor=\"#94DDF4\",label=\"rg-lib\"];\n    \"rg-sample\" [fillcolor=\"#94DDF4\",label=\"rg-sample\"];\n  }\n  subgraph cluster_outputs {\n    rank = \"same\";\n    style = \"dashed\";\n    labelloc = \"b\";\n    label = \"Workflow Outputs\";\n    \"bam\" [fillcolor=\"#94DDF4\",label=\"bam\"];\n    \"vcf\" [fillcolor=\"#94DDF4\",label=\"vcf\"];\n    \"bam_stats\" [fillcolor=\"#94DDF4\",label=\"bam_stats\"];\n    \"vcf_stats\" [fillcolor=\"#94DDF4\",label=\"vcf_stats\"];\n  }\n  \"allelotype\" [label=\"allelotype\"];\n  \"samsort\" [label=\"samsort\"];\n  \"lobSTR\" [label=\"lobSTR\"];\n  \"samindex\" [label=\"samindex\"];\n  \"strinfo\" -> \"allelotype\" [label=\"strinfo\"];\n  \"reference\" -> \"allelotype\" [label=\"reference\"];\n  \"output_prefix\" -> \"allelotype\" [label=\"output_prefix\"];\n  \"noise_model\" -> \"allelotype\" [label=\"noise_model\"];\n  \"samindex\" -> \"allelotype\" [label=\"bam\"];\n  \"lobSTR\" -> \"samsort\" [label=\"input\"];\n  \"default1\" -> \"samsort\" [label=\"output_name\"];\n  \"default1\" [label=\"\\\"aligned.sorted.bam\\\"\", fillcolor=\"#D5AEFC\"];\n  \"rg-lib\" -> \"lobSTR\" [label=\"rg-lib\"];\n  \"rg-sample\" -> \"lobSTR\" [label=\"rg-sample\"];\n  \"p2\" -> \"lobSTR\" [label=\"p2\"];\n  \"p1\" -> \"lobSTR\" [label=\"p1\"];\n  \"reference\" -> \"lobSTR\" [label=\"reference\"];\n  \"output_prefix\" -> \"lobSTR\" [label=\"output_prefix\"];\n  \"samsort\" -> \"samindex\" [label=\"input\"];\n  \"samindex\" -> \"bam\";\n  \"allelotype\" -> \"vcf\";\n  \"lobSTR\" -> \"bam_stats\";\n  \"allelotype\" -> \"vcf_stats\";\n}\n",
    "visualisationPng": "/graph/png/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
    "visualisationSvg": "/graph/svg/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
    "roBundle": "/robundle/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
    "visualisationXdot": "/graph/xdot/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl"
}

View Workflows

GET
/workflows

Parameters

Field Type Description
search String Optional: Limits the workflows received to those containing this string in the label or doc
page Integer The page number to access (0 indexed, defaults to 0)
size Integer The page size requested (defaults to 10)

Success 200

Field Type Description
content List<Workflow> The workflows returned as per Get Workflow Details
last Boolean Whether this page is the last one
totalElements Integer The total number of workflows
totalPages Integer The total number of pages of this size
first Boolean Whether this page is the first one
numberOfElements Integer The number of workflows on this page
size Integer The page size requested
number Integer The current page number
Example Response
{
    "content": [
        {

        },
        {

        }
    ],
    "last": true,
    "totalElements": 2,
    "totalPages": 1,
    "sort": null,
    "first": true,
    "numberOfElements": 2,
    "size": 10,
    "number": 0
}

Queue

Check Queue Element

GET
/queue/:id

Parameters

Field Type Description
id String The ID of the queue element from the Location header of Add a new Workflow

Pending 200

Field Type Description
cwltoolStatus enum One of RUNNING, ERROR
cwltoolVersion String The version of cwltool being used to parse the workflow
message String Error message if cwltoolStatus is ERROR
Example Responses
{
    "cwltoolStatus": "RUNNING",
    "cwltoolVersion": "1.0.20170622090721",
}
{
    "cwltoolStatus": "ERROR",
    "cwltoolVersion": "1.0.20170622090721",
    "message": "Tool definition failed initialization:\n('https://sparql-test.commonwl.org/schema.rdf', SSLError(SSLError(CertificateError(\"hostname 'sparql-test.commonwl.org' doesn't match either of 'biowardrobe.com', 'demo.biowardrobe.com', 'www.biowardrobe.com'\",),),))\n"
}

Finished 303

HTTP/1.1 303 See Other
Location: /workflows/:url

Examples

Python - Build script for Travis

The API can be utilised as a build step in continuous integration such as in the following example

travis.yml

The python script can be included in the 'script' or 'after_success' areas of travis.yml depending on your usage

language: python

python:
  - "2.7"

install:
  - pip install requests

script:
  - ...

after_success:
  - python cwlviewer.py

cwlviewer.py

# Must list 'pip install requests' in travis.yml
import requests
import time
import os

# You can alternatively define these in travis.yml as env vars or arguments
BASE_URL = 'https://view.commonwl.org'
WORKFLOW_PATH = '/workflows/workflow.cwl'

# Travis environment variables
TRAVIS_COMMIT = os.getenv('TRAVIS_COMMIT')
TRAVIS_REPO_SLUG = os.getenv('TRAVIS_REPO_SLUG')

# Whole workflow URL on github
workflowURL = 'https://github.com/' + TRAVIS_REPO_SLUG + '/blob/' + TRAVIS_COMMIT + WORKFLOW_PATH

# Headers
HEADERS = {
    'user-agent': 'my-app/0.0.1',
    'accept': 'application/json'
}

# Add new workflow with the specific commit ID of this build
addResponse = requests.post(BASE_URL + '/workflows',
                            data={'url': workflowURL},
                            headers=HEADERS)

if addResponse.status_code == requests.codes.accepted:
    qLocation = addResponse.headers['location']

    # Get the queue item until success
    qResponse = requests.get(BASE_URL + qLocation,
                             headers=HEADERS,
                             allow_redirects=False)
    maxAttempts = 5
    while qResponse.status_code == requests.codes.ok and qResponse.json()['cwltoolStatus'] == 'RUNNING' and maxAttempts > 0:
        time.sleep(5)
        qResponse = requests.get(BASE_URL + qLocation,
                                 headers=HEADERS,
                                 allow_redirects=False)
        maxAttempts -= 1

    if qResponse.headers['location']:
        # Success, get the workflow
        workflowResponse = requests.get(BASE_URL + qResponse.headers['location'], headers=HEADERS)
        if (workflowResponse.status_code == requests.codes.ok):
            workflowJson = workflowResponse.json()
            # Do what you want with the workflow JSON
            # Include details in documentation files etc
            print(BASE_URL + workflowJson['visualisationSvg'])
            print('Verified with cwltool version ' + workflowJson['cwltoolVersion'])
            # etc...
        else:
            print('Could not get returned workflow')
    elif qResponse.json()['cwltoolStatus'] == 'ERROR':
        # Cwltool failed to run here
        print(qResponse.json()['message'])
    elif maxAttempts == 0:
        print('Timeout: Cwltool did not finish')

else:
    print('Error adding workflow')