I have fasta files with multiple sequences. They are reads that mapped to the genome outside of probe-targeted regions. From a quick perusal, they appear to be repetitive and have low complexity. Is there a way I can quantify their complexity and use that as an explanation as to why those regions were mapped to?
"Complexity" is also known as entropy. A quick and dirty way is to use a file compression utility - zip will get good compression for a repetitive sequence.
You can also look at the k-mer distribution for some suitable k: in a repetitive sequence it will be biased. Here is a tool that give a local plot: https://rdrr.io/github/vsbuffalo/qrqc/man/kmerEntropyPlot.html
Use Picard EstimateLibraryComplexity.
From the docs:
Estimates the numbers of unique molecules in a sequencing library.
java -jar picard.jar EstimateLibraryComplexity \ I=input.bam \ O=est_lib_complex_metrics.txt
How long are your reads, and how many? Some colleagues of mine wrote https://github.com/eclarke/komplexity to quickly score complexity in large numbers of short reads. The score is derived from the number of unique k-mers normalized by the sequence length, so it's simple and fast, though the score inherently becomes less informative for longer and longer sequences (so, best for short reads). We've used it for filtering repetitive or other low-complexity sequences in metagenomics projects.