Explore Workflows

This ChIP-Seq pipeline is based on the original [BioWardrobe's](https://biowardrobe.com) [PubMed ID:26248465](https://www.ncbi.nlm.nih.gov/pubmed/26248465) **ChIP-Seq** basic analysis workflow for a **paired-end** experiment with Trim Galore. ### Data Analysis SciDAP starts from the .fastq files which most DNA cores and commercial NGS companies return. Starting from raw data allows us to ensure that all experiments have been processed in the same way and simplifies the deposition of data to GEO upon publication. The data can be uploaded from users computer, downloaded directly from an ftp server of the core facility by providing a URL or from GEO by providing SRA accession number. Our current pipelines include the following steps: 1. Trimming the adapters with TrimGalore. This step is particularly important when the reads are long and the fragments are short-resulting in sequencing adapters at the end of read. If adapter is not removed the read will not map. TrimGalore can recognize standard adapters, such as Illumina or Nexterra/Tn5 adapters. 2. QC 3. (Optional) trimming adapters on 5' or 3' end by the specified number of bases. 4. Mapping reads with BowTie. Only uniquely mapped reads with less than 3 mismatches are used in the downstream analysis. Results are saved as a .bam file. 5. (Optional) Removal of duplicates (reads/pairs of reads mapping to exactly same location). This step is used to remove reads overamplified in PCR. Unfortunately, it may also remove \"good\" reads. We usually do not remove duplicates unless the library is heavily duplicated. Please note that MACS2 will remove 'excessive' duplicates during peak calling ina smart way (those not supported by other nearby reads). 6. Peakcalling by MACS2. (Optionally), it is possible to specify read extension length for MACS2 to use if the length determined automatically is wrong. 7. Generation of BigWig coverage files for display on the browser. The coverage shows the number of fragments at each base in the genome normalized to the number of millions of mapped reads. In the case of PE sequencing the fragments are real, but in the case of single reads the fragments are estimated by extending reads to the average fragment length found by MACS2 or specified by the user in 6. ### Details _Trim Galore_ is a wrapper around [Cutadapt](https://github.com/marcelm/cutadapt) and [FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) to consistently apply adapter and quality trimming to FastQ files, with extra functionality for RRBS data. A [FASTQ](http://maq.sourceforge.net/fastq.shtml) input file has to be provided. In outputs it returns coordinate sorted BAM file alongside with index BAI file, quality statistics for both the input FASTQ files, reads coverage in a form of BigWig file, peaks calling data in a form of narrowPeak or broadPeak files, islands with the assigned nearest genes and region type, data for average tag density plot (on the base of BAM file). Workflow starts with running fastx_quality_stats (steps fastx_quality_stats_upstream and fastx_quality_stats_downstream) from FASTX-Toolkit to calculate quality statistics for both upstream and downstream input FASTQ files. At the same time Bowtie is used to align reads from input FASTQ files to reference genome (Step bowtie_aligner). The output of this step is unsorted SAM file which is being sorted and indexed by samtools sort and samtools index (Step samtools_sort_index). Depending on workflow’s input parameters indexed and sorted BAM file could be processed by `samtools markdup` *samtools\_remove\_duplicates* to remove all possible read duplicates. Right after that macs2 callpeak performs peak calling (Step macs2_callpeak). On the base of returned outputs the next step (Step macs2_island_count) calculates the number of islands and estimated fragment size. If the last one is less that 80 (hardcoded in a workflow) macs2 callpeak is rerun again with forced fixed fragment size value (Step macs2_callpeak_forced). If at the very beginning it was set in workflow input parameters to force run peak calling with fixed fragment size, this step is skipped and the original peak calling results are saved. In the next step workflow again calculates the number of islands and estimated fragment size (Step macs2_island_count_forced) for the data obtained from macs2_callpeak_forced step. If the last one was skipped the results from macs2_island_count_forced step are equal to the ones obtained from macs2_island_count step. Next step (Step macs2_stat) is used to define which of the islands and estimated fragment size should be used in workflow output: either from macs2_island_count step or from macs2_island_count_forced step. If input trigger of this step is set to True it means that macs2_callpeak_forced step was run and it returned different from macs2_callpeak step results, so macs2_stat step should return [fragments_new, fragments_old, islands_new], if trigger is False the step returns [fragments_old, fragments_old, islands_old], where sufix \"old\" defines results obtained from macs2_island_count step and sufix \"new\" - from macs2_island_count_forced step. The following two steps (Step bamtools_stats and bam_to_bigwig) are used to calculate coverage on the base of input BAM file and save it in BigWig format. For that purpose bamtools stats returns the number of mapped reads number which is then used as scaling factor by bedtools genomecov when it performs coverage calculation and saves it in BED format. The last one is then being sorted and converted to BigWig format by bedGraphToBigWig tool from UCSC utilities. Step get_stat is used to return a text file with statistics in a form of [TOTAL, ALIGNED, SUPRESSED, USED] reads count. Step island_intersect assigns genes and regions to the islands obtained from macs2_callpeak_forced. Step average_tag_density is used to calculate data for average tag density plot on the base of BAM file.

https://github.com/datirium/workflows.git

Path: workflows/trim-chipseq-pe.cwl

Branch/Commit ID: 93b844a80f4008cc973ea9b5efedaff32a343895

A workflow for processing a single 16S sample via a QIIME2 pipeline. ## __Outputs__ #### Output files: - overview.md, list of inputs - demux.qzv, summary visualizations of imported data - alpha-rarefaction.qzv, plot of OTU rarefaction - taxa-bar-plots.qzv, relative frequency of taxomonies barplot ## __Inputs__ #### General Info - Sample short name/Alias: Used for samplename in downstream analyses. Ensure this is the same name used in the metadata samplesheet. - Environment: where the sample was collected - Catalog No.: catalog number if available (optional) - Read 1 FASTQ file: Read 1 FASTQ file from a paired-end sequencing run. - Read 2 FASTQ file: Read 2 FASTQ file that pairs with the input R1 file. - Trim 5' of R1: Recommended if adapters are still on the input sequences. Trims the first J bases from the 5' end of each forward read. - Trim 5' of R2: Recommended if adapters are still on the input sequences. Trims the first K bases from the 5' end of each reverse read. - Truncate 3' of R1: Recommended if quality drops off along the length of the read. Clips the forward read starting M bases from the 5' end (before trimming). - Truncate 3' of R2: Recommended if quality drops off along the length of the read. Clips the reverse read starting N bases from the 5' end (before trimming). - Threads: Number of threads to use for steps that support multithreading. ### __Data Analysis Steps__ 1. Generate FASTX quality statistics for visualization of unmapped, raw FASTQ reads. 2. Import the data, make a qiime artifact (demux.qza), and summary visualization 3. Denoising will detect and correct (where possible) Illumina amplicon sequence data. This process will additionally filter any phiX reads (commonly present in marker gene Illumina sequence data) that are identified in the sequencing data, and will filter chimeric sequences. 4. Generate a phylogenetic tree for diversity analyses and rarefaction processing and plotting. 5. Taxonomy classification of amplicons. Performed using a Naive Bayes classifier trained on the Greengenes2 database \"gg_2022_10_backbone_full_length.nb.qza\". ### __References__ 1. Bolyen E, Rideout JR, Dillon MR, Bokulich NA, Abnet CC, Al-Ghalith GA, Alexander H, Alm EJ, Arumugam M, Asnicar F, Bai Y, Bisanz JE, Bittinger K, Brejnrod A, Brislawn CJ, Brown CT, Callahan BJ, Caraballo-Rodríguez AM, Chase J, Cope EK, Da Silva R, Diener C, Dorrestein PC, Douglas GM, Durall DM, Duvallet C, Edwardson CF, Ernst M, Estaki M, Fouquier J, Gauglitz JM, Gibbons SM, Gibson DL, Gonzalez A, Gorlick K, Guo J, Hillmann B, Holmes S, Holste H, Huttenhower C, Huttley GA, Janssen S, Jarmusch AK, Jiang L, Kaehler BD, Kang KB, Keefe CR, Keim P, Kelley ST, Knights D, Koester I, Kosciolek T, Kreps J, Langille MGI, Lee J, Ley R, Liu YX, Loftfield E, Lozupone C, Maher M, Marotz C, Martin BD, McDonald D, McIver LJ, Melnik AV, Metcalf JL, Morgan SC, Morton JT, Naimey AT, Navas-Molina JA, Nothias LF, Orchanian SB, Pearson T, Peoples SL, Petras D, Preuss ML, Pruesse E, Rasmussen LB, Rivers A, Robeson MS, Rosenthal P, Segata N, Shaffer M, Shiffer A, Sinha R, Song SJ, Spear JR, Swafford AD, Thompson LR, Torres PJ, Trinh P, Tripathi A, Turnbaugh PJ, Ul-Hasan S, van der Hooft JJJ, Vargas F, Vázquez-Baeza Y, Vogtmann E, von Hippel M, Walters W, Wan Y, Wang M, Warren J, Weber KC, Williamson CHD, Willis AD, Xu ZZ, Zaneveld JR, Zhang Y, Zhu Q, Knight R, and Caporaso JG. 2019. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology 37: 852–857. https://doi.org/10.1038/s41587-019-0209-9

https://github.com/datirium/workflows.git

Path: workflows/qiime2-sample-pe.cwl

Branch/Commit ID: 93b844a80f4008cc973ea9b5efedaff32a343895

https://github.com/common-workflow-language/cwl-utils.git

Path: testdata/count-lines7-wf_v1_0.cwl

Branch/Commit ID: b76b039edb62dea76c43f173848cdc57e4b4aab7

https://github.com/common-workflow-language/cwl-utils.git

Path: testdata/record-output-wf_v1_2.cwl

Branch/Commit ID: b76b039edb62dea76c43f173848cdc57e4b4aab7

https://github.com/common-workflow-language/cwl-utils.git

Path: testdata/record-output-wf_v1_0.cwl

Branch/Commit ID: b76b039edb62dea76c43f173848cdc57e4b4aab7

https://github.com/genome/analysis-workflows.git

Path: definitions/subworkflows/gatk_haplotypecaller_iterator.cwl

Branch/Commit ID: 39ac49f5d080bbb6bfa97246f46a5b621254f622

https://github.com/common-workflow-language/cwl-utils.git

Path: testdata/md5sum.cwl

Branch/Commit ID: b76b039edb62dea76c43f173848cdc57e4b4aab7

Interval overlapping alignments counts ====================================== Reports the count of alignments from multiple samples that overlap specific intervals.

https://github.com/datirium/workflows.git

Path: workflows/bedtools-multicov.cwl

Branch/Commit ID: 93b844a80f4008cc973ea9b5efedaff32a343895

A CWL workflow for discovering known or novel miRNAs from deep sequencing data using the miRDeep2 tool. The ExoCarta exosome database is also used for identifying exosome-related miRNAs, and TargetScan's organism-specific databases are used for identifying miRNA gene targets. ## __Outputs__ #### Primary Output files: - mirs_known.tsv, detected known mature miRNAs, \"Known miRNAs\" tab - mirs_novel.tsv, detected novel mature miRNAs, \"Novel miRNAs\" tab #### Secondary Output files: - mirs_known_exocarta_deepmirs.tsv, list of detected miRNA also in ExoCarta's exosome database, \"Detected Exosome miRNAs\" tab - mirs_known_gene_targets.tsv, pre-computed gene targets of known mature mirs, downloadable - known_mirs_mature.fa, known mature mir sequences, downloadable - known_mirs_precursor.fa, known precursor mir sequences, downloadable - novel_mirs_mature.fa, novel mature mir sequences, downloadable - novel_mirs_precursor.fa, novel precursor mir sequences, downloadable #### Reports: - overview.md (input list, alignment & mir metrics), \"Overview\" tab - mirdeep2_result.html, summary of mirdeep2 results, \"miRDeep2 Results\" tab ## __Inputs__ #### General Info - Sample short name/Alias: unique name for sample - Experimental condition: condition, variable, etc name (e.g. \"control\" or \"20C 60min\") - Cells: name of cells used for the sample - Catalog No.: vender catalog number if available - Bowtie2 index: Bowtie2 index directory of the reference genome. - Reference Genome FASTA: Reference genome FASTA file to be used for alignment. - Genome short name: Name used for setting organism name, genus, species, and tax ID. - Input FASTQ file: FASTQ file from a single-end miRNA sequencing run. #### Advanced - Adapter: Adapter sequence to be trimmed from miRNA sequence reads. (Default: TCGTAT) - Threads: Number of threads to use for steps that support multithreading (Default: 4). ## Hints & Tips: #### For the identification of novel miRNA candidates, the following may be used as a filtering guideline: 1. miRDeep score > 4 (some authors use 1) 2. not present a match with rfam 3. should present a significant RNAfold (\"yes\") 4. a number of mature reads > 10 5. if applicable, novel mir must be expressed in multiple samples #### For filtering mirbase by organism. | genome | organism | division | name | tree | NCBI-taxid | | ---- | --- | --- | ----------- | ----------- | ----------- | | hg19 | hsa | HSA | Homo sapiens | Metazoa;Bilateria;Deuterostoma;Chordata;Vertebrata;Mammalia;Primates;Hominidae | 9606 | | hg38 | hsa | HSA | Homo sapiens | Metazoa;Bilateria;Deuterostoma;Chordata;Vertebrata;Mammalia;Primates;Hominidae | 9606 | | mm10 | mmu | MMU | Mus musculus | Metazoa;Bilateria;Deuterostoma;Chordata;Vertebrata;Mammalia;Rodentia | 10090 | | rn7 | rno | RNO | Rattus norvegicus | Metazoa;Bilateria;Deuterostoma;Chordata;Vertebrata;Mammalia;Rodentia | 10116 | | dm3 | dme | DME | Drosophila melanogaster | Metazoa;Bilateria;Ecdysozoa;Arthropoda;Hexapoda | 7227 | ## __Data Analysis Steps__ 1. The miRDeep2 Mapper module processes Illumina FASTQ output and maps it to the reference genome. 2. The miRDeep2 miRDeep2 module identifies known and novel (mature and precursor) miRNAs. 3. The ExoCarta database of miRNA found in exosomes is then used to find overlap between mirs_known.tsv and exosome associated miRNAs. 4. Finally, TargetScan organism-specific miRNA gene target database is used to find overlap between mirs_known.tsv and gene targets. ## __References__ 1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3245920 2. https://github.com/rajewsky-lab/mirdeep2 3. https://biocontainers.pro/tools/mirdeep2 4. https://www.mirbase.org/ 5. http://exocarta.org/index.html 6. https://www.targetscan.org/vert_80/

https://github.com/datirium/workflows.git

Path: workflows/mirna-mirdeep2-se.cwl

Branch/Commit ID: 93b844a80f4008cc973ea9b5efedaff32a343895

DESeq2 Multi-factor Analysis Runs DeSeq2 multi-factor analysis with manual control over major parameters

https://github.com/datirium/workflows.git

Path: workflows/deseq-multi-factor.cwl

Branch/Commit ID: 93b844a80f4008cc973ea9b5efedaff32a343895