Workflow: CWL Example - Text Corpus Search Pipeline
End-to-end example workflow used to teach CWL concepts. It verifies and extracts a tar.gz text corpus, searches every extracted file for a pattern (as a scattered sub-workflow), lists a reference directory, writes and conditionally compresses a summary report, and (only if explicitly requested) would send a webhook notification. See diagrams/cwl-conceptual-overview.md and presentation/CWL-Example-Walkthrough.pptx for a guided explanation.
- Selected
- |
- Default Values
- Nested Workflows
- Tools
- Inputs/Outputs
Inputs
| ID | Type | Title | Doc |
|---|---|---|---|
| output_format | https://w3id.org/cwl/view/git/82e29a8c71a6dcbf3b0993bf26d2e42817bb39e5/example/types/output-format.yml#OutputFormat | Summary rendering mode | |
| reference_dir | Directory | Reference materials directory |
Directory whose contents are listed (not searched) for context. |
| corpus_archive | File [format] | Text corpus archive |
A tar.gz archive of .txt reports, with a .sha256 sidecar as a secondaryFile. |
| search_parameters | https://w3id.org/cwl/view/git/82e29a8c71a6dcbf3b0993bf26d2e42817bb39e5/example/types/search-parameters.yml#SearchParameters | Search pattern + case sensitivity |
Custom record type (SchemaDefRequirement) grouping dependent parameters. |
| send_notification | Boolean | Whether to attempt the (demo-only) webhook notification | |
| compress_threshold | Integer | Minimum total matches required to compress the summary | |
| notification_message | String (Optional) | Optional custom notification message |
If omitted, notify-webhook.cwl falls back to its own default message via the `self || \"...\"` pattern documented in the CWL User Guide \"Inputs\" topic (Inclusive and Exclusive Inputs). |
Steps
| ID | Runs | Label | Doc |
|---|---|---|---|
| notify |
steps/notify-webhook.cwl
(CommandLineTool)
|
Send a demo-only webhook notification |
Illustrates NetworkAccess. Gated by `send_notification` (default false) at the workflow-step level via `when` (see main-workflow.cwl), so with the bundled sample job this step is always skipped and no network call is ever attempted. `send_notification` is declared here with no `inputBinding` purely so the workflow's `when` expression can read it. |
| extract |
steps/extract-archive.cwl
(CommandLineTool)
|
Extract corpus archive |
Extracts all `.txt` reports from the checksum-verified tar.gz corpus archive. Demonstrates: baseCommand as an array (fixed leading arguments), ResourceRequirement, array-of-File outputs via glob, and capturing the tool's own stdout as a log file. |
| compress |
steps/compress-report.cwl
(CommandLineTool)
|
Compress the summary report |
Gzips the persisted summary file. Only runs when the workflow step's `when` condition is true (see main-workflow.cwl). `total_matches` and `compress_threshold` are declared here with no `inputBinding` purely so the workflow's `when` expression can read them - they never reach the actual command line. Demonstrates: ToolTimeLimit, WorkReuse, and \"control-only\" step inputs. |
| write_summary |
steps/write-summary.cwl
(CommandLineTool)
|
Persist the summary text as a file |
Materializes a runtime string (`summary_text`) into a real file using InitialWorkDirRequirement, then copies it to a stable output name. Demonstrates the \"Creating Files at Runtime\" pattern from the CWL User Guide: `entry: $(inputs.summary_text)` writes the parameter reference's value verbatim into `entryname`. |
| list_reference |
steps/list-reference-dir.cwl
(ExpressionTool)
|
List reference directory contents |
Recursively lists the contents of the `reference_dir` Directory input and returns the file names as a string array, without running any external command. Demonstrates: ExpressionTool, Directory input, LoadListingRequirement (deep_listing), and a multi-line `${...}` JS expression with a helper function. |
| verify_checksum |
steps/verify-checksum.cwl
(CommandLineTool)
|
Verify archive checksum |
Confirms the SHA-256 checksum of the corpus archive matches its
accompanying `.sha256` sidecar file (declared via `secondaryFiles`) before
extraction proceeds. On success it passes the *same* File object through as
`verified_archive`, which the workflow uses to force a real data
dependency (extraction only starts once verification has succeeded). |
| search_subworkflow |
steps/search-subworkflow.cwl
(Workflow)
|
Search sub-workflow (scatter + aggregate) |
Nested Workflow used as a single step of main-workflow.cwl (requires the parent to declare SubworkflowFeatureRequirement). Scatters search-tool.cwl over every extracted text file, then aggregates the per-file counts. Demonstrates: ScatterFeatureRequirement + scatterMethod, and a Workflow whose own inputs/outputs use SchemaDefRequirement custom types. |
Outputs
| ID | Type | Label | Doc |
|---|---|---|---|
| summary | String | Rendered summary text | |
| final_report | File | Final report (compressed if the threshold was met, plain otherwise) | |
| match_reports | File[] | One match report per corpus file | |
| total_matches | Integer | Total matches across the whole corpus | |
| extraction_log | File | Archive extraction log | |
| verification_log | File | Checksum verification log | |
| reference_file_names | String[] | Names of files found under reference_dir |
https://w3id.org/cwl/view/git/82e29a8c71a6dcbf3b0993bf26d2e42817bb39e5/example/main-workflow.cwl
