JSON API Documentation
Disclaimer: The API is still under heavy development and breaking changes to endpoints or parameters may be made
If you make anything utilizing our API, please let us know about it - we would love to see!
Except where noted otherwise, all queries require the following header to receive a JSON response:
accept: application/json
Workflow
Add a new Workflow
Note: Due to cwltool run times, the creation of workflows is done asynchronously and an intermediate queue resource is used
POST
/workflows
Parameters
| Field | Type | Description |
|---|---|---|
| url | String | The URL of the repository. Either a gitlab.com or github.com URL, or any Git repository URL. If a gitlab.com or github.com repository URL is used, the following fields will take priority over those parsed from the URL if provided. |
| branch | String | The branch of the repository. Required if the URL is not from gitlab.com or github.com. |
| path | String | The path within the repository to the workflow file. Required if the URL is not from gitlab.com or github.com. |
| packedId | String | The ID of the workflow, to be provided if the workflow file is packed (contains multiple descriptions with $graph). |
Input
{
"url": "https://github.com/common-workflow-language/workflows/tree/master/workflows/compile/compile1.cwl"
}
{
"url": "https://bitbucket.org/markrobinson96/workflows.git",
"branch": "master",
"path": "/workflows/make-to-cwl/dna.cwl"
}
Success
HTTP/1.1 202 Accepted Location: /queue/:queueid
Already Exists
HTTP/1.1 303 See Other Location: /workflows/:URL
Packed Workflow with Multiple Choices
HTTP/1.1 422 Unprocessable Entity
{
"message": "This workflow file is packed and contains multiple workflow descriptions. Please provide a packedId parameter with one of the following values"
"packedId": [
"workflow.cwl",
"main",
"workflow_exome.cwl",
"align.cwl",
"workflow.cwl_2"
]
}
Error
HTTP/1.1 400 Bad Request
Get Workflow Details
GET/workflows/:url
Parameters
| Field | Type | Description |
|---|---|---|
| url | String | Partial URL to the workflow eg "github.com/owner/repo/tree/branch/workflow.cwl" or "bitbucket.org/owner/repo.git/master/workflow.cwl |
Success 200
| Field | Type | Description |
|---|---|---|
| retrievedFrom.repoUrl | String | URL of the Github repository |
| retrievedFrom.branch | String | Github branch name or commit ID |
| retrievedFrom.path | String | Path to the workflow within the repository |
| retrievedFrom.packedId | String | The ID of the workflow within the file, if this is a packed CWL file (contains a document graph). |
| retrievedFrom.url | URL | The URL to the page on the site if type is not GENERIC, otherwise the same as repoUrl |
| retrievedFrom.rawUrl | URL | The URL directly to the file if type is not GENERIC, otherwise the same as repoUrl |
| retrievedFrom.type | Enum | One of GENERIC, GITHUB, GITLAB, BITBUCKET |
| retrievedOn | Date | The last time this workflow was updated from the branch |
| lastCommit | String | The last commit ID on the branch |
| label | String | The label string for the workflow from the CWL description |
| doc | String | The doc string for the workflow from the CWL description |
| inputs | ||
| outputs | ||
| steps | ||
| dockerLink | String | The URL to dockerhub or "true" if an unrecognised image is used in a docker hint or requirement |
| cwltoolVersion | String | The version of cwltool used to parse the workflow |
| visualisationDot | String | The DOT source used to generate the visualisation image |
| visualisationXdot | URL | A link to the xdot source code for the visualisation image |
| visualisationPng | URL | A link to the png format visualisation image |
| visualisationSvg | URL | A link to the svg format visualisation image |
| robundle | URL | A link to the Research Object Bundle download |
Example URL
/workflows/github.com/common-workflow-language/workflows/tree/master/workflows/lobSTR/lobSTR-workflow.cwl
Example Response
{
"retrievedFrom": {
"repoUrl": "https://github.com/common-workflow-language/workflows.git",
"branch": "master",
"path": "workflows/lobSTR/lobSTR-workflow.cwl",
"url": "https://github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
"rawUrl": "https://raw.githubusercontent.com/common-workflow-language/workflows/master/workflows/lobSTR/lobSTR-workflow.cwl",
"type": "GITHUB"
},
"retrievedOn": 1502714218616,
"lastCommit": "17b65ea19d81527090fded62ffa0e1ba3b25d561",
"label": "lobSTR-workflow.cwl",
"inputs": {
"reference": {
"type": "File"
},
"rg-sample": {
"type": "String"
},
"p1": {
"type": "ee54066e73d068edfbf9ca5059e14acd"
},
"p2": {
"type": "79ab371b6d0fdbadbf9a4a89ef393353"
},
"output_prefix": {
"type": "String"
},
"rg-lib": {
"type": "String"
},
"strinfo": {
"type": "File"
},
"noise_model": {
"type": "File"
}
},
"outputs": {
"vcf": {
"type": "File"
},
"vcf_stats": {
"type": "File"
},
"bam_stats": {
"type": "File"
},
"bam": {
"type": "File"
}
},
"steps": {
"allelotype": {
"run": "allelotype.cwl",
"runType": "COMMANDLINETOOL",
"sources": {
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/bam": {
"sourceIDs": [
"samindex"
]
},
"allelotype": {
"sourceIDs": [
"strinfo"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/noise_model": {
"sourceIDs": [
"noise_model"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/reference": {
"sourceIDs": [
"reference"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#allelotype/output_prefix": {
"sourceIDs": [
"output_prefix"
]
}
}
},
"samsort": {
"run": "samtools-sort.cwl",
"runType": "COMMANDLINETOOL",
"sources": {
"samsort": {
"sourceIDs": [
"lobSTR"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#samsort/output_name": {
"defaultVal": "\\\"aligned.sorted.bam\\\""
}
}
},
"lobSTR": {
"label": "lobSTR",
"run": "lobSTR-tool.cwl",
"runType": "COMMANDLINETOOL",
"sources": {
"lobSTR": {
"sourceIDs": [
"rg-lib"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/p1": {
"sourceIDs": [
"p1"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/p2": {
"sourceIDs": [
"p2"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/output_prefix": {
"sourceIDs": [
"output_prefix"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/rg-sample": {
"sourceIDs": [
"rg-sample"
]
},
"file:///data/git/3ec0d19076841a1140e71cc3a1b330a56099aac2/workflows/lobSTR/lobSTR-workflow.cwl#lobSTR/reference": {
"sourceIDs": [
"reference"
]
}
}
},
"samindex": {
"run": "samtools-index.cwl",
"runType": "COMMANDLINETOOL",
"sources": {
"samindex": {
"sourceIDs": [
"samsort"
]
}
}
}
},
"dockerLink": "https://hub.docker.com/r/rabix/lobstr",
"cwltoolVersion": "1.0.20170810192106",
"visualisationDot": "digraph workflow {\n graph [\n bgcolor = \"#eeeeee\"\n color = \"black\"\n fontsize = \"10\"\n labeljust = \"left\"\n clusterrank = \"local\"\n ranksep = \"0.22\"\n nodesep = \"0.05\"\n ]\n node [\n fontname = \"Helvetica\"\n fontsize = \"10\"\n fontcolor = \"black\"\n shape = \"record\"\n height = \"0\"\n width = \"0\"\n color = \"black\"\n fillcolor = \"lightgoldenrodyellow\"\n style = \"filled\"\n ];\n edge [\n fontname=\"Helvetica\"\n fontsize=\"8\"\n fontcolor=\"black\"\n color=\"black\"\n arrowsize=\"0.7\"\n ];\n subgraph cluster_inputs {\n rank = \"same\";\n style = \"dashed\";\n label = \"Workflow Inputs\";\n \"p2\" [fillcolor=\"#94DDF4\",label=\"p2\"];\n \"p2\" [fillcolor=\"#94DDF4\",label=\"p2\"];\n \"output_prefix\" [fillcolor=\"#94DDF4\",label=\"output_prefix\"];\n \"reference\" [fillcolor=\"#94DDF4\",label=\"reference\"];\n \"noise_model\" [fillcolor=\"#94DDF4\",label=\"noise_model\"];\n \"strinfo\" [fillcolor=\"#94DDF4\",label=\"strinfo\"];\n \"p1\" [fillcolor=\"#94DDF4\",label=\"p1\"];\n \"p1\" [fillcolor=\"#94DDF4\",label=\"p1\"];\n \"rg-lib\" [fillcolor=\"#94DDF4\",label=\"rg-lib\"];\n \"rg-sample\" [fillcolor=\"#94DDF4\",label=\"rg-sample\"];\n }\n subgraph cluster_outputs {\n rank = \"same\";\n style = \"dashed\";\n labelloc = \"b\";\n label = \"Workflow Outputs\";\n \"bam\" [fillcolor=\"#94DDF4\",label=\"bam\"];\n \"vcf\" [fillcolor=\"#94DDF4\",label=\"vcf\"];\n \"bam_stats\" [fillcolor=\"#94DDF4\",label=\"bam_stats\"];\n \"vcf_stats\" [fillcolor=\"#94DDF4\",label=\"vcf_stats\"];\n }\n \"allelotype\" [label=\"allelotype\"];\n \"samsort\" [label=\"samsort\"];\n \"lobSTR\" [label=\"lobSTR\"];\n \"samindex\" [label=\"samindex\"];\n \"strinfo\" -> \"allelotype\" [label=\"strinfo\"];\n \"reference\" -> \"allelotype\" [label=\"reference\"];\n \"output_prefix\" -> \"allelotype\" [label=\"output_prefix\"];\n \"noise_model\" -> \"allelotype\" [label=\"noise_model\"];\n \"samindex\" -> \"allelotype\" [label=\"bam\"];\n \"lobSTR\" -> \"samsort\" [label=\"input\"];\n \"default1\" -> \"samsort\" [label=\"output_name\"];\n \"default1\" [label=\"\\\"aligned.sorted.bam\\\"\", fillcolor=\"#D5AEFC\"];\n \"rg-lib\" -> \"lobSTR\" [label=\"rg-lib\"];\n \"rg-sample\" -> \"lobSTR\" [label=\"rg-sample\"];\n \"p2\" -> \"lobSTR\" [label=\"p2\"];\n \"p1\" -> \"lobSTR\" [label=\"p1\"];\n \"reference\" -> \"lobSTR\" [label=\"reference\"];\n \"output_prefix\" -> \"lobSTR\" [label=\"output_prefix\"];\n \"samsort\" -> \"samindex\" [label=\"input\"];\n \"samindex\" -> \"bam\";\n \"allelotype\" -> \"vcf\";\n \"lobSTR\" -> \"bam_stats\";\n \"allelotype\" -> \"vcf_stats\";\n}\n",
"visualisationPng": "/graph/png/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
"visualisationSvg": "/graph/svg/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
"roBundle": "/robundle/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl",
"visualisationXdot": "/graph/xdot/github.com/common-workflow-language/workflows/blob/master/workflows/lobSTR/lobSTR-workflow.cwl"
}
View Workflows
GET/workflows
Parameters
| Field | Type | Description |
|---|---|---|
| search | String | Optional: Limits the workflows received to those containing this string in the label or doc |
| page | Integer | The page number to access (0 indexed, defaults to 0) |
| size | Integer | The page size requested (defaults to 10) |
Success 200
| Field | Type | Description |
|---|---|---|
| content | List<Workflow> | The workflows returned as per Get Workflow Details |
| last | Boolean | Whether this page is the last one |
| totalElements | Integer | The total number of workflows |
| totalPages | Integer | The total number of pages of this size |
| first | Boolean | Whether this page is the first one |
| numberOfElements | Integer | The number of workflows on this page |
| size | Integer | The page size requested |
| number | Integer | The current page number |
Example Response
{
"content": [
{
},
{
}
],
"last": true,
"totalElements": 2,
"totalPages": 1,
"sort": null,
"first": true,
"numberOfElements": 2,
"size": 10,
"number": 0
}
Queue
Check Queue Element
GET/queue/:id
Parameters
| Field | Type | Description |
|---|---|---|
| id | String | The ID of the queue element from the Location header of Add a new Workflow |
Pending 200
| Field | Type | Description |
|---|---|---|
| cwltoolStatus | enum | One of RUNNING, ERROR |
| cwltoolVersion | String | The version of cwltool being used to parse the workflow |
| message | String | Error message if cwltoolStatus is ERROR |
Example Responses
{
"cwltoolStatus": "RUNNING",
"cwltoolVersion": "1.0.20170622090721",
}
{
"cwltoolStatus": "ERROR",
"cwltoolVersion": "1.0.20170622090721",
"message": "Tool definition failed initialization:\n('https://sparql-test.commonwl.org/schema.rdf', SSLError(SSLError(CertificateError(\"hostname 'sparql-test.commonwl.org' doesn't match either of 'biowardrobe.com', 'demo.biowardrobe.com', 'www.biowardrobe.com'\",),),))\n"
}
Finished 303
HTTP/1.1 303 See Other Location: /workflows/:url
Examples
Python - Build script for Travis
The API can be utilised as a build step in continuous integration such as in the following example
travis.yml
The python script can be included in the 'script' or 'after_success' areas of travis.yml depending on your usage
language: python python: - "2.7" install: - pip install requests script: - ... after_success: - python cwlviewer.py
cwlviewer.py
# Must list 'pip install requests' in travis.yml
import requests
import time
import os
# You can alternatively define these in travis.yml as env vars or arguments
BASE_URL = 'https://view.commonwl.org'
WORKFLOW_PATH = '/workflows/workflow.cwl'
# Travis environment variables
TRAVIS_COMMIT = os.getenv('TRAVIS_COMMIT')
TRAVIS_REPO_SLUG = os.getenv('TRAVIS_REPO_SLUG')
# Whole workflow URL on github
workflowURL = 'https://github.com/' + TRAVIS_REPO_SLUG + '/blob/' + TRAVIS_COMMIT + WORKFLOW_PATH
# Headers
HEADERS = {
'user-agent': 'my-app/0.0.1',
'accept': 'application/json'
}
# Add new workflow with the specific commit ID of this build
addResponse = requests.post(BASE_URL + '/workflows',
data={'url': workflowURL},
headers=HEADERS)
if addResponse.status_code == requests.codes.accepted:
qLocation = addResponse.headers['location']
# Get the queue item until success
qResponse = requests.get(BASE_URL + qLocation,
headers=HEADERS,
allow_redirects=False)
maxAttempts = 5
while qResponse.status_code == requests.codes.ok and qResponse.json()['cwltoolStatus'] == 'RUNNING' and maxAttempts > 0:
time.sleep(5)
qResponse = requests.get(BASE_URL + qLocation,
headers=HEADERS,
allow_redirects=False)
maxAttempts -= 1
if qResponse.headers['location']:
# Success, get the workflow
workflowResponse = requests.get(BASE_URL + qResponse.headers['location'], headers=HEADERS)
if (workflowResponse.status_code == requests.codes.ok):
workflowJson = workflowResponse.json()
# Do what you want with the workflow JSON
# Include details in documentation files etc
print(BASE_URL + workflowJson['visualisationSvg'])
print('Verified with cwltool version ' + workflowJson['cwltoolVersion'])
# etc...
else:
print('Could not get returned workflow')
elif qResponse.json()['cwltoolStatus'] == 'ERROR':
# Cwltool failed to run here
print(qResponse.json()['message'])
elif maxAttempts == 0:
print('Timeout: Cwltool did not finish')
else:
print('Error adding workflow')
SVG sketch of CWL workflow
Note: This method uses a brief YAML
parsing of the standalone CWL file that does not support annotations
like label or doc, and that does not check
for validity or completeness.
POST
/graph/svg
Input
Content-Type: text/yaml
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
usermessage: string
outputs:
response:
outputSource: step0/response
type: File
steps:
step0:
run:
class: CommandLineTool
inputs:
message:
type: string
inputBinding:
position: 1
baseCommand: echo
outputs:
response:
type: stdout
in:
message: usermessage
out: [response]
Output
Content-Type: image/svg+xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg width="450pt" height="270pt"
viewBox="0.00 0.00 463.00 278.00" xmlns="http://www.w3.org/2000/svg">
<!-- ... -->
</svg>
Success
HTTP/1.1 200 OK Content-Type: image/svg+xml
Error
HTTP/1.1 400 Bad Request
PNG sketch of CWL workflow
Note: This method uses a brief YAML
parsing of the standalone CWL file that does not support annotations
like label or doc, and that does not check
for validity or completeness.
POST
/graph/png
Input
Content-Type: text/yaml
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
usermessage: string
outputs:
response:
outputSource: step0/response
type: File
steps:
step0:
run:
class: CommandLineTool
inputs:
message:
type: string
inputBinding:
position: 1
baseCommand: echo
outputs:
response:
type: stdout
in:
message: usermessage
out: [response]
Output
\x89PNG\n\x1a\n\x00\x00 (binary)
Success
HTTP/1.1 200 OK Content-Type: image/png
Error
HTTP/1.1 400 Bad Request
