data QC

Description

1. QC of raw data

~~FastQC~~

output files: /projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/fastqc_results

~~visualization (multiqc ?)~~

output files: /projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/multiqc_report

Is data ok to proceed?

according to the report there are adapters so we must do trimming
there are some duplicates (so maybe we could do also deduplication)
maybe it is contaminated by human dna (we need to check)

2. Trimming

~~Trimming with fastp (there is a lot of G’s so that might be better)~~

output files: /projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/qc_after_fastp

Trimming with cutadapt

Good

output files: /projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/QC/03cutadapt

3. Deduplication

Deduplication with clumpify :

‣

shell script

output files: /projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/QC/03cutadapt/deduplicated_clumpify

Another tool that I would like to check: Embed GitHub

Iternal thoughts + questions:

Question: are there human samples in the mix?

Metaphlan does human genes removal i think.

#!/bin/bash
#SBATCH --account=torch_pr_169_general
#SBATCH --job-name=dedup
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=12:00:00
#SBATCH --output=dedup_%A_%a.log
#SBATCH --error=dedup_%A_%a.err
#SBATCH --array=0-16

module load bioinformatics/20260224

TRIMDIR=/projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/QC/03cutadapt/03_cutadapt_trimmed
OUT=/projects/rps/emh9/henafflab/2025-Gowanus_Biofilm_BioBAT/data/QC/03cutadapt/deduplicated_clumpify

SAMPLES=($(ls $TRIMDIR/*_R1.trimmed.fastq.cutadapt0_14bp.gz | xargs -n1 basename | sed 's/_R1.trimmed.fastq.cutadapt0_14bp.gz//'))
SAMPLE=${SAMPLES[$SLURM_ARRAY_TASK_ID]}

R1=$TRIMDIR/${SAMPLE}_R1.trimmed.fastq.cutadapt0_14bp.gz
R2=$TRIMDIR/${SAMPLE}_R2.trimmed.fastq.cutadapt0_14bp.gz

if [[ ! -f "$R1" ]] || [[ ! -f "$R2" ]]; then
  echo "Error: files not found for $SAMPLE" >&2
  exit 1
fi

echo "Deduplicating: $SAMPLE"

clumpify.sh \
  in=$R1 \
  in2=$R2 \
  out=$OUT/${SAMPLE}_R1.dedup.fastq.gz \
  out2=$OUT/${SAMPLE}_R2.dedup.fastq.gz \
  dedupe=t \
  optical=t \
  dupedist=12000 \
  threads=$SLURM_CPUS_PER_TASK \
  -Xmx28g

echo "Done: $SAMPLE"