nf-core/methylseq v4.0.0 Release Notes

This release introduces major new features, significant refactoring, and extensive test suite improvements. Key highlights include the addition of a targeted sequencing analysis workflow, substantial refactoring of input handling and subworkflows for better modularity and robustness, and a complete overhaul of the testing framework.

Bug Fixes & Refactoring

• Core Refactoring (#539):
  • The FASTA index subworkflow was refactored to remove implicit parameter passing.
  • Fixed workflow versions collection and improved conditional logic for FastQC and MultiQC execution.
• Parabricks Fixes:
  • The minimum memory for PARABRICKS_FQ2BAMMETH has been increased to 100GB as suggested by the documentation (#528).
  • Fixed a parabricks/fq2bammeth failure on AWS Batch-like executors where symlinks don’t exist (#536).
• General:
  • Reverted PR #523 which involved direct changes to subworkflows in the pipeline.
  • The pipeline template has been updated to nf-core/tools v3.2.0 (#504).
  • The pipeline template has been updated to nf-core/tools v3.3.1 (#504).

Pipeline Updates

• Targeted Sequencing (#516, #539):
  • A new --run_targeted_sequencing flag has been added to enable further analysis of targeted methylation experiments.
  • A new Targeted Sequencing subworkflow has been introduced with improved parameter handling and region-specific analysis capabilities.
  • The bedtools/intersect and picard/collecthsmetrics modules from nf-core/modules have been installed.
  • A new --collecthsmetrics parameter has been added to allow the collection of Picard HS metrics.
• Workflow & Parameter Changes:
  • Bwameth:
    • Added --all_contexts and --merge_context parameters (#516).
    • Removed the --comprehensive parameter (#516).
  • Bismark:
    • Decoupled --merge_non_CpG from --comprehensive in the Bismark workflow (#516).
  • Trimming:
    • Added a new skip_trimming_presets flag to overwrite any preset trimming options (#506).
    • Updated --pbat trimming options to 8|8|8|8 (#506).
    • Removed --cegx and --epignome preset trimming options as the kits are discontinued (#506).
  • GPU Usage:
    • Removed the --use_gpu parameter. Activating the gpu profile with --profile gpu is now the standard way to enable GPU-based pathways (#520).
  • Quality Control:
    • Added --skip_fastqc and --skip_multiqc parameters to allow skipping quality control steps (#539).

Testing Updates (#539)

• Test Suite Consolidation:
  • Removed 19 individual test files, replacing them with comprehensive test suites.
• New Test Suites:
  • bismark_variants.nf.test for comprehensive Bismark testing.
  • bismark_hisat_variants.nf.test for Bismark with HISAT2 aligner.
  • bwameth_variants.nf.test for comprehensive Bwameth testing.
  • targeted_sequencing_variants.nf.test for the new targeted sequencing workflow.
  • index_downloads.nf.test for testing index download scenarios.

What’s Changed

• post release bump version by @sateeshperi in https://github.com/nf-core/methylseq/pull/497
• Update test command in usage.md by @fbnrst in https://github.com/nf-core/methylseq/pull/502
• Important! Template update for nf-core/tools v3.2.0 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/504
• Update snapshot 3.2.0 template by @sateeshperi in https://github.com/nf-core/methylseq/pull/505
• create separate test_gpu profile by @sateeshperi in https://github.com/nf-core/methylseq/pull/517
• update ci runners to runson by @sateeshperi in https://github.com/nf-core/methylseq/pull/522
• Fix faix notprovided error by @lpantano in https://github.com/nf-core/methylseq/pull/523
• Added CUDA test for runsOn by @GallVp in https://github.com/nf-core/methylseq/pull/534
• Removed parameter use_gpu by @GallVp in https://github.com/nf-core/methylseq/pull/533
• Bumped PARABRICKS_FQ2BAMMETH memory to 100.GB by @GallVp in https://github.com/nf-core/methylseq/pull/535
• Revert fix faix error by @sateeshperi in https://github.com/nf-core/methylseq/pull/530
• Updated parabricks/fq2bammeth by @GallVp in https://github.com/nf-core/methylseq/pull/537
• skip trimming presets by @sateeshperi in https://github.com/nf-core/methylseq/pull/518
• Important! Template update for nf-core/tools v3.2.1 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/525
• Targeted analysis by @dcarrillox in https://github.com/nf-core/methylseq/pull/516
• Important! Template update for nf-core/tools v3.3.1 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/538
• refactor-workflow-tests by @sateeshperi in https://github.com/nf-core/methylseq/pull/539
• bump-version 4.0.0 by @sateeshperi in https://github.com/nf-core/methylseq/pull/540
• update metro map 4.0.0 by @sateeshperi in https://github.com/nf-core/methylseq/pull/542
• apply suggestions from code review by @sateeshperi in https://github.com/nf-core/methylseq/pull/543
• Dev -> Main v4.0.0 by @sateeshperi in https://github.com/nf-core/methylseq/pull/541

New Contributors

• @lpantano made their first contribution in https://github.com/nf-core/methylseq/pull/523
• @dcarrillox made their first contribution in https://github.com/nf-core/methylseq/pull/516

Full Changelog: https://github.com/nf-core/methylseq/compare/3.0.0…4.0.0

What’s Changed

• post-release version bump by @sateeshperi in https://github.com/nf-core/methylseq/pull/450
• Fix/fasta gunzip & bismark align cache by @sateeshperi in https://github.com/nf-core/methylseq/pull/457
• Feat/fastq align dedup bismark by @sateeshperi in https://github.com/nf-core/methylseq/pull/465
• Fasta index bismark bwameth by @sateeshperi in https://github.com/nf-core/methylseq/pull/468
• Conf modules configs by @sateeshperi in https://github.com/nf-core/methylseq/pull/469
• run_preseq param + skip preseq/lcextrap by default by @sateeshperi in https://github.com/nf-core/methylseq/pull/470
• run_qualimap param and skip qualimap/bamqc by default by @sateeshperi in https://github.com/nf-core/methylseq/pull/471
• Removed ambiguity in the multilane sample information by @bounlu in https://github.com/nf-core/methylseq/pull/452
• Disable gunzip publish by @sateeshperi in https://github.com/nf-core/methylseq/pull/474
• CI: add gpu tag test job by @sateeshperi in https://github.com/nf-core/methylseq/pull/478
• Fasta align dedup bwameth by @sateeshperi in https://github.com/nf-core/methylseq/pull/475
• Checking if shards > 0 are present before firing tests by @GallVp in https://github.com/nf-core/methylseq/pull/484
• Updated metro maps. by @ewels in https://github.com/nf-core/methylseq/pull/479
• Test replicates by @sateeshperi in https://github.com/nf-core/methylseq/pull/486
• Doc updates by @sateeshperi in https://github.com/nf-core/methylseq/pull/489
• Template update for nf-core/tools v3.1.0 by @sateeshperi in https://github.com/nf-core/methylseq/pull/490
• Important! Template update for nf-core/tools v3.1.0 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/488
• 2.8.0 bump-version by @sateeshperi in https://github.com/nf-core/methylseq/pull/491
• Cytosine report test by @sateeshperi in https://github.com/nf-core/methylseq/pull/493
• First attempt at fact-checking chatGPT by @FelixKrueger in https://github.com/nf-core/methylseq/pull/495
• add bs-seq-primer doc by @sateeshperi in https://github.com/nf-core/methylseq/pull/494
• bump ver to 3.0.0 semver by @sateeshperi in https://github.com/nf-core/methylseq/pull/496
• Dev -> Master 3.0.0 by @sateeshperi in https://github.com/nf-core/methylseq/pull/492

New Contributors

• @bounlu made their first contribution in https://github.com/nf-core/methylseq/pull/452
• @GallVp made their first contribution in https://github.com/nf-core/methylseq/pull/484

Full Changelog: https://github.com/nf-core/methylseq/compare/2.7.1…v3.0.0

What’s Changed

• post-release version bump by @sateeshperi in https://github.com/nf-core/methylseq/pull/446
• Patch release 2.7.1 by @sateeshperi in https://github.com/nf-core/methylseq/pull/448
• Dev -> Master 2.7.1 by @sateeshperi in https://github.com/nf-core/methylseq/pull/449

Full Changelog: https://github.com/nf-core/methylseq/compare/2.7.0…2.7.1

What’s Changed

• post-release dev bump version and snapshots by @sateeshperi in https://github.com/nf-core/methylseq/pull/371
• add missing comma to awstestfull yml by @sateeshperi in https://github.com/nf-core/methylseq/pull/375
• Important! Template update for nf-core/tools v2.12 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/377
• Fix/samplename join by @sateeshperi in https://github.com/nf-core/methylseq/pull/381
• Important! Template update for nf-core/tools v2.13 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/380
• Important! Template update for nf-core/tools v2.13.1 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/384
• remove duplicate section from usage.md by @fbnrst in https://github.com/nf-core/methylseq/pull/386
• Important! Template update for nf-core/tools v2.14.1 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/399
• Fix/sample merge by @sateeshperi in https://github.com/nf-core/methylseq/pull/401
• test(#392): Add boilerplate for tests by @edmundmiller in https://github.com/nf-core/methylseq/pull/393
• fix usage of Fasta references for bwameth / samtools faidx by @maxulysse in https://github.com/nf-core/methylseq/pull/403
• Bump preseq by @ewels in https://github.com/nf-core/methylseq/pull/390
• Add length_trim parameter to nextflow.config file by @geethavenk in https://github.com/nf-core/methylseq/pull/416
• Add missing type field to entry in nextflow_schema.json by @scoughlan2 in https://github.com/nf-core/methylseq/pull/417
• Feat/workflow updates by @sateeshperi in https://github.com/nf-core/methylseq/pull/414
• Fix/readme 391 by @sateeshperi in https://github.com/nf-core/methylseq/pull/418
• bump-version and update CHANGELOG by @sateeshperi in https://github.com/nf-core/methylseq/pull/419
• update date for 2.7.0 release by @sateeshperi in https://github.com/nf-core/methylseq/pull/421
• Template update for nf-core/tools v3.0.2 by @sateeshperi in https://github.com/nf-core/methylseq/pull/426
• Important! Template update for nf-core/tools v3.0.2 by @nf-core-bot in https://github.com/nf-core/methylseq/pull/427
• update bismark+bwameth modules; sub-wf; wf by @sateeshperi in https://github.com/nf-core/methylseq/pull/428
• udpate tests to use nft-bam for bam files by @sateeshperi in https://github.com/nf-core/methylseq/pull/429
• update SAMTOOLS_FAIDX fasta input after reorg by @sateeshperi in https://github.com/nf-core/methylseq/pull/430
• apply code review suggestions and activate igenomes bwameth index use by @sateeshperi in https://github.com/nf-core/methylseq/pull/431
• @sateeshperi fix bismark, bwameth align modules * other module updates + update snaps by @sateeshperi in https://github.com/nf-core/methylseq/pull/432
• update CI and Changelog by @sateeshperi in https://github.com/nf-core/methylseq/pull/433
• update spacing in nextflow.config to test nf-test trigger by @sateeshperi in https://github.com/nf-core/methylseq/pull/434
• rm conda from CI tests until the file lock resolution by @sateeshperi in https://github.com/nf-core/methylseq/pull/435
• Adding Metromap (with Sections) by @Shaun-Regenbaum in https://github.com/nf-core/methylseq/pull/438
• rm resourceLimits for test_full profile by @sateeshperi in https://github.com/nf-core/methylseq/pull/439
• update changelog and readme by @sateeshperi in https://github.com/nf-core/methylseq/pull/442
• Update metromap.svg by @ewels in https://github.com/nf-core/methylseq/pull/443
• add hisat2 bismark index to igenomes by @sateeshperi in https://github.com/nf-core/methylseq/pull/444
• apply suggestions from nextflow vscode formatter + update date in cha… by @sateeshperi in https://github.com/nf-core/methylseq/pull/445
• Dev -> Master 2.7.0 by @sateeshperi in https://github.com/nf-core/methylseq/pull/420

New Contributors

• @fbnrst made their first contribution in https://github.com/nf-core/methylseq/pull/386
• @geethavenk made their first contribution in https://github.com/nf-core/methylseq/pull/416
• @scoughlan2 made their first contribution in https://github.com/nf-core/methylseq/pull/417
• @Shaun-Regenbaum made their first contribution in https://github.com/nf-core/methylseq/pull/438

Full Changelog: https://github.com/nf-core/methylseq/compare/2.6.0…2.7.0

Pipeline Updates

• 🔄 Updated template to nf-core/tools v2.11.1 #368 @sateeshperi @edmundmiller

Bug fixes & refactoring

• 🛠 Copy methylKit-compatible files to publishDir #357 @ltosti-tagomics
• 🐛 fix ignore_r1 and ignore_3prime_r1 variable expansion #359 @noecochetel
• 🐛 No groups #362 @edmundmiller

Pipeline Updates

• 🔄 Updated template to nf-core/tools v2.9
• 🔄 Updated template to nf-core/tools v2.10
• 🔧 Updated nf-core modules for FastQC, samtools sort, samtools flagstat @ewels
  • ❌ Removes problematic -m memory assignment for samtools sort #81
• 🧾 Use fromSamplesheet from nf-validation #341 @Emiller88
• 🚀 Update Maintainers and add CODEOWNERS #345 @Emiller88
• ⚙️ Update schema to utilize exists and add more patterns #342 @Emiller88
• 📁 Support pipeline-specific configs #343 @maxulysse

Bug fixes & refactoring

• 🛠️ Added publishing of coverage (*cov.gz) files for NOMe-seq filtered reads for coverage2cytosine @FelixKrueger
• 🛠️ Wrong display values for “zymo” and “em_seq” presets on help page #335 @wassimsalam01
• 📚 Use new Citation tools functions #336 @Emiller88

Full Changelog: https://github.com/nf-core/methylseq/compare/2.4.0…2.5.0

Pipeline Updates

• Updated template to nf-core/tools v2.8
• Add --bamqc_regions_file parameter for targeted methylation sequencing data #302
• ✨ Add NF-TEST tests and snapshots for the pipeline test profile #310

Bug fixes & refactoring

• 🛠️ update index file channels to explicit value channels #310
• 🐛 fix params.test_data_base in test and test_full configs #310
• 🤖 GitHub Actions CI - pull_reqest to dev tests with NXF_VER latest-everything #310
• 🤖 GitHub Actions CI - pull_reqest to master tests with NXF_VER 22.10.1 & latest-everything #310
• 🤖 GitHub Actions CI - fail-fast set to false #310
• 🐛 get to the bottom of index tests #278
• ✨ Support for Bismark methylation extraction ignore and ignore_3prime parameters when ignore_r1 or ignore_3prime_r1 are greater than 0. #322
• 🛠️ rename ignore -> ignore_r1 and ignore_3prime -> ignore_3prime_r1 params #322
• 🐛 removed unused directory #297

Pipeline Updates

• ⚙️ Dramatically increase the default process time config requests for Bismark and bwa meth alignment
• ✨ Add a tower.yml file to enable Reports in Nextflow Tower
• 🤖 GitHub Actions CI - download the test data prior to running tests

Bug fixes & refactoring

• 🧹 Refactor genome indices preparation into a separate workflow
• 🧹 Refactor subworkflow logic out of alignment subworkflows, for later sharing
• 🐛 Fix a bug with using a local genome reference FASTA file
• 🐛 Fix a bunch of problems in the CI tests using nf-test (#279)

Pipeline Updates

• ✨ Updated the bismark2summary step so that it no longer stages the aligned BAM files into the working directory (#268)
  • Should be much faster / cheaper for running on the cloud
• ✨ Added ability to merge FastQ files based on shared IDs in sample sheet (#272)

Bug fixes & refactoring

• 🐛 Fixed typo in parameter handling for input reference indices (#263)
• 🧹 Removed orphaned --bismark_align_cpu_per_multicore and --bismark_align_cpu_per_multicore parameters.
  • Multi-core usage for Bismark alignment is now automatically set. If you would like to overwrite this, you can do so by setting ext.args for the process in a custom config.
• 🧹 Removed duplicate option --coverage2cytosine (#273)
  • Use the existing option --cytosine_report to launch the new COVERAGE2CYTOSINE process.
  • Removed option --cytosine_report genome_index from the Bismark methylation extractor.

Pipeline Updates

• ✨ Added option to run the Bismark coverage2cytosine script using the --coverage2cytosine and --nomeseq parameters.
• 🐛 Fixed bad bug where trimming presets were not being applied (#261)

Software Updates

• Update Bismark v0.23.0 to v0.24.0

Pipeline Updates

Major pipeline rewrite to use DSL2 with shared nf-core/modules.

Warning: Breaking change! ⚠️

The pipeline now requires a sample sheet to be passed to the pipeline with --input:

See an example here

Note: The genome column is not yet used but will give the ability to map to multiple genomes in a single run in a future release. See #181.

Supplying the reference geneome with --genome as before works as usual.

Software Updates

Major updates in commands and software versions for nearly every tool.

Please treat this new version with a little more care than usual and let us know if you find any problems!

Minor release to fix a bug in the command for the Bismark alignment. This change should have no effect on the pipeline output, though single-end alignments with Bismark should now be 2x faster. See issue #210 for full investigation.

Pipeline Updates

• Added new config profile to run minimal test paired-end dataset, with -profile test_paired. Added to the CI tests.

Bug fixes

• Fixed silent bug in Bismark alignment command that had no effect on the output (#210)

Software updates

• Picard 2.25.1 > 2.25.4
• MultiQC 1.10 > 1.10.1

:warning: Breaking change!

In line with a standardisation change across all of nf-core, we have changed the main parameter name for supplying files to the pipeline. In this release, please use --input instead of --reads. The parameter still works in the same way as before.

Pipeline Updates

• Increased resources for fastqc process (#143)
• Raised Nextflow version requirement to 20.07.1
• Updated template to nf-core/tools 1.13.3
• Renamed --reads to --input
• Added new --maxins and --minins parameters to pass on to Bismark
• New --em_seq preset
  • Sets bismark_maxins = 1000, clip_r1 = 8, clip_r2 = 8, three_prime_clip_r1 = 8, three_prime_clip_r2 = 8
• New --publish_dir_mode parameter to customise results folder behaviour
• Fix bug on AWS for bismark_hisat known splice file (#177)
• Moved parameter documentation into new nextflow_schema.json file
  • This improves web documentation and enables nf-core launch functionality. See https://nf-co.re/launch?pipeline=methylseq
• Added a -profile test_full config for running the pipeline with a full-size test dataset
  • See the config file for details
  • This will be used for automated release tests on AWS, results browsable on the website

Software updates

• Python base 3.7.3 > 3.8.8
• markdown 3.1.1 > 3.3.4
• pymdown-extensions 6.0 > 8.1.1
• pygments 2.6.1 > 2.8.1
• pigz 2.3.4 > 2.6
• samtools 1.9 > 1.11
• TrimGalore! 0.6.5 > 0.6.6
• Bowtie2 2.3.5 > 2.4.2
• Hisat2 2.2.0 > 2.2.1
• Bismark 0.22.3 > 0.23.0
• Picard 2.22.2 > 2.25.1
• MethylDackel 0.5.0 > 0.5.2
• MultiQC 1.8 > 1.10

New features

• Added multicore support for TrimGalore!
• Improved the multicore support for Bismark methXtract for more parallelisation (#121)
• Added --cytosine_report option to tell Bismark to give reports for all cytosines in the genome.
• Added options --bismark_align_cpu_per_multicore and --bismark_align_cpu_per_multicore to customise how Bismark align --multicore is decided (#124)

Software updates

• New: pigz 2.3.4
• Python base 2.7 > 3.7.3
• FastQC 0.11.8 > 0.11.9
• TrimGalore! 0.6.4 > 0.6.5
• HiSAT2 2.1.0 > 2.2.0
• Bismark 0.22.2 > 0.22.3
• Qualimap 2.2.2c > 2.2.2d
• Picard 2.21.3 > 2.22.2
• MethylDackel 0.4.0 > 0.5.0
• MultiQC 1.7 > 1.8

Pipeline Updates

• Fixed bug where the iGenomes config was loaded after the references were set (#121)
• Removed awsbatch config profile because it is now served by nf-core/configs
• Tidied up the summary log messages when starting the pipeline
  • Fewer messages saying what you’re not doing, sanitised the order of some logs and removed a few things
• Slightly refactored the code for trimming parameters
• Updated template to tools 1.9

Bug fixes

• Fixed error where MethylDackel would consume the Nextflow channels and not work with more than one sample #140

New features

• Changed all parameter names to snake_case
• Added --local_alignment option to run Bismark with the --local flag to allow soft-clipping of reads.
• Added support for bismark’s SLAM-seq mode
• Added support for running bismark with HISAT2 as an aligner option #85
• Added support for centralized configuration profiles nf-core/configs
• Add --meth_cutoff parameter to change default for bismark_methylation_extractor
  • eg. use --meth_cutoff 5 on the command line or params.meth_cutoff = 5 to require 5 overlapping reads to call a methylation site.
• Added --methyl_kit option to run MethylDackel with the --methylKit flag, producing output suitable for the methylKit R package.

Software updates

• new dependency: hisat2 2.1.0
• new dependency: r-markdown 1.1
• TrimGalore! 0.5.0 > 0.6.4
• Bismark 0.20.0 > 0.22.2
• Bowtie2 2.3.4.3 > 2.3.5
• Picard 2.18.21 > 2.21.3
• Qualimap 2.2.2b > 2.2.2c
• MethylDackel 0.3.0 > 0.4.0

Pipeline updates

• Keep memory in GB for samtools, to avoid problems with unit conversion (#99)
• Changed params.container for process.container
• Synchronised with version 1.7 of the nf-core/tools template

Bug fixes

• Fixed a bug that caused conda dependencies to be resolved very slowly
• Allowed some spare memory in the samtools sort steps, avoiding crashes for some users (#81)

A fairly quick release on the heels of 1.2 to fix a nasty bug introduced that prevented the pipeline from running on more than one file. Whilst we’re here we added in a new tool: preseq.

New features

• Added preseq analysis to calculate sample complexity.
  • This new step can help decide sufficient sequencing depth has been reached.

Bug fixes

• Fixed new bug that meant pipeline only worked with one sample at a time #66
  • Introduced in previous release. TrimGalore onwards would only process one sample.

New features

• Trim 9bp from both ends of both reads for PBAT mode.
• Save where_are_my_files.txt to results directory to inform the user about missing intermediate files (#42)

Software updates

• Fastqc 0.11.7 > 0.11.8
• Bowtie2 2.3.4.2 > 2.3.4.3
• Bismark 0.19.1 > 0.20.0
• Qualimap 2.2.2a > 2.2.2b
• Picard 2.18.11 > 2.18.21
• MultiQC 1.6 > 1.7

Bug fixes

• Fixed error when running the pipeline with --unmapped
  • Previously, could result in error Error ~ No such variable: bismark_unmapped
• Fixed error where single-sample reports could mix up log files (#48)
• Fixed bug in MultiQC process that skipped results from some tools
• Supply available memory as argument to Picard MarkDuplicates

• Tests simplified - now work by simply using the test config profile
  • eg: nextflow run nf-core/methylseq -profile test,docker
  • Removed previous run_test.sh script and data
• New Singularity build script for direct compatibility with singularity-hub
• Minor improvements to the docs
• A number of boilerplate nf-core code updates
• Updated process$name nextflow syntax to avoid warnings in new versions of nextflow
• Updated software tools
  • trim-galore v0.4.5 update to 0.5.0
  • samtools v1.8 update to 1.9
  • bowtie2 v2.3.4.1 update to 2.3.4.2
  • multiqc v1.5 update to 1.6
  • picard v2.18.2 update to 2.18.11
  • bwameth v0.2.0 update to 0.2.2

First release of this pipeline under the nf-core flag. It also marks a significant step up in the maturity of the workflow, with everything now in a single script and both aligner workflows fully supported.

• Renamed and moved SciLifeLab/NGI-MethylSeq to nf-core/methylseq
• Merged bwa-meth and bismark pipeline scripts, now chosen with --aligner flag
• Refactored multi-core parameters for Bismark alignment and methylation extraction
• Rewrote most of the documentation
• Changed the Docker container to use Bioconda installations

For earlier releases and development notes, please see the (now archived) fork at SciLifeLab/NGI-MethylSeq.