Workflow: CWL Example - Text Corpus Search Pipeline

Fetched 2026-09-25 09:10:27 GMT

End-to-end example workflow used to teach CWL concepts. It verifies and extracts a tar.gz text corpus, searches every extracted file for a pattern (as a scattered sub-workflow), lists a reference directory, writes and conditionally compresses a summary report, and (only if explicitly requested) would send a webhook notification. See diagrams/cwl-conceptual-overview.md and presentation/CWL-Example-Walkthrough.pptx for a guided explanation.

children parents
Workflow as SVG
  • Selected
  • Default Values
  • Nested Workflows
  • Tools
  • Inputs/Outputs

Inputs

ID Type Title Doc
output_format https://w3id.org/cwl/view/git/82e29a8c71a6dcbf3b0993bf26d2e42817bb39e5/example/types/output-format.yml#OutputFormat Summary rendering mode
reference_dir Directory Reference materials directory

Directory whose contents are listed (not searched) for context.

corpus_archive File [format] Text corpus archive

A tar.gz archive of .txt reports, with a .sha256 sidecar as a secondaryFile.

search_parameters https://w3id.org/cwl/view/git/82e29a8c71a6dcbf3b0993bf26d2e42817bb39e5/example/types/search-parameters.yml#SearchParameters Search pattern + case sensitivity

Custom record type (SchemaDefRequirement) grouping dependent parameters.

send_notification Boolean Whether to attempt the (demo-only) webhook notification
compress_threshold Integer Minimum total matches required to compress the summary
notification_message String (Optional) Optional custom notification message

If omitted, notify-webhook.cwl falls back to its own default message via the `self || \"...\"` pattern documented in the CWL User Guide \"Inputs\" topic (Inclusive and Exclusive Inputs).

Steps

ID Runs Label Doc
notify
steps/notify-webhook.cwl (CommandLineTool)
Send a demo-only webhook notification

Illustrates NetworkAccess. Gated by `send_notification` (default false) at the workflow-step level via `when` (see main-workflow.cwl), so with the bundled sample job this step is always skipped and no network call is ever attempted. `send_notification` is declared here with no `inputBinding` purely so the workflow's `when` expression can read it.

extract
steps/extract-archive.cwl (CommandLineTool)
Extract corpus archive

Extracts all `.txt` reports from the checksum-verified tar.gz corpus archive. Demonstrates: baseCommand as an array (fixed leading arguments), ResourceRequirement, array-of-File outputs via glob, and capturing the tool's own stdout as a log file.

compress
steps/compress-report.cwl (CommandLineTool)
Compress the summary report

Gzips the persisted summary file. Only runs when the workflow step's `when` condition is true (see main-workflow.cwl). `total_matches` and `compress_threshold` are declared here with no `inputBinding` purely so the workflow's `when` expression can read them - they never reach the actual command line. Demonstrates: ToolTimeLimit, WorkReuse, and \"control-only\" step inputs.

write_summary
steps/write-summary.cwl (CommandLineTool)
Persist the summary text as a file

Materializes a runtime string (`summary_text`) into a real file using InitialWorkDirRequirement, then copies it to a stable output name. Demonstrates the \"Creating Files at Runtime\" pattern from the CWL User Guide: `entry: $(inputs.summary_text)` writes the parameter reference's value verbatim into `entryname`.

list_reference
steps/list-reference-dir.cwl (ExpressionTool)
List reference directory contents

Recursively lists the contents of the `reference_dir` Directory input and returns the file names as a string array, without running any external command. Demonstrates: ExpressionTool, Directory input, LoadListingRequirement (deep_listing), and a multi-line `${...}` JS expression with a helper function.

verify_checksum
steps/verify-checksum.cwl (CommandLineTool)
Verify archive checksum

Confirms the SHA-256 checksum of the corpus archive matches its accompanying `.sha256` sidecar file (declared via `secondaryFiles`) before extraction proceeds. On success it passes the *same* File object through as `verified_archive`, which the workflow uses to force a real data dependency (extraction only starts once verification has succeeded).

Demonstrates: secondaryFiles, successCodes default (mismatch = real failure), simple parameter references with array indexing (`secondaryFiles[0].path`, no InlineJavascriptRequirement needed), and an `outputEval` that passes an input straight through as an output.

search_subworkflow Search sub-workflow (scatter + aggregate)

Nested Workflow used as a single step of main-workflow.cwl (requires the parent to declare SubworkflowFeatureRequirement). Scatters search-tool.cwl over every extracted text file, then aggregates the per-file counts. Demonstrates: ScatterFeatureRequirement + scatterMethod, and a Workflow whose own inputs/outputs use SchemaDefRequirement custom types.

Outputs

ID Type Label Doc
summary String Rendered summary text
final_report File Final report (compressed if the threshold was met, plain otherwise)
match_reports File[] One match report per corpus file
total_matches Integer Total matches across the whole corpus
extraction_log File Archive extraction log
verification_log File Checksum verification log
reference_file_names String[] Names of files found under reference_dir
Permalink: https://w3id.org/cwl/view/git/82e29a8c71a6dcbf3b0993bf26d2e42817bb39e5/example/main-workflow.cwl