# Removing duplicate FASTA sequences based on headers with Bash

I used the following command to remove duplicate FASTA sequences based on the header sequence:

paste -d $$'\t' - - '\t' -uk1,1 | awk 'BEGIN{FS="\t";OFS="\n"}{print $$1,$$2}'


After this command, the output result is shown below:

>_10|_10
AAAGCAAAACACAGAATTGACGGAAAGACTTACGTTATTAAACGTGTTAAATATAATAACGAGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTC
>_10|_11
ATTATGATCCTGAGACCAGTGATGATTCTCTTGAGAGCAGTGATTATGATCCTGAGAACAGCAAAAATAGTTCAAGGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACACA
>_10|_12
TTTGGAACTCTTTGAACAAATAACAAAAGGGGTGGATTATATACATTCAAAAAAATTAATTCATAGAGATCTTAAGGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACACA
>_10|_13
ACTTGTAACATCTCTGAAAAATGATGGAAAGCGAACAAGGAGTAAGGGAACTTTGCGATACATGAGCCCAGAACAGGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACACA
>_10|_14
AACGAATTTCTTCGCAAGACTATGGAAAGGAAGTGGACCTCTACGCTTTGGGGCTAATTCTTGCTGAACTTCTTCATGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACAC


This is my actual FASTA file: I don't want to lose header information from my FASTA sequence after removing duplicate sequences based on header information.

>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGCATGAAC
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGAAGTTG
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGGTGGGG
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGTAATTA
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGTCATGG
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGTTCCTTG
>ENST00000642616_8|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTATTTGGTGCCTATCCTAAAACTTCCAGTGGGTTTTCTTGCCTTCCAGTGGCCATATAAAGGGCGTCCTGAAAAC
>ENST00000642616_7|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTCATGAACAAGGCTCAGTTATGATGTTTGAGCACCTTATTAATTCTAATGGAATTAAGCCTGTCATGGAACAATA
>ENST00000642616_1|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGCGCCTGCGCGCGGGTCCGGCGCCGAGGTTCTTGACTGCTGTGCCGGACGCCAGGTGTAGCCATGCAGCGAGCC
>ENST00000642616_9|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGGACTGCCATCATCTTGGAATCCAAAATAATTTTGATTACAAGCGCTTTATTAAGTTTGCCCGTGTCTGTGAAG
>ENST00000642616_5|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGGTGGGGTATCTAGCAGGATGTCTAGTTCACGCACTGGGTGAAAAACAACCAGAGCTGCAGATAAGTGAACGAG
>ENST00000642616_4|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTAATTAATGATCCTATCCATGGCCACATTGAGCTCCACCCTCTCCTCGTCCGAATCATTGATACACCTCAATT
>ENST00000642616_6|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTCATGGGCCATTTTCTCACATGTTTGATGGACGATTTATTCCACTTGCTCGCCCGGAGGTGAAATGGACGATT
>ENST00000642616_6|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTCATGGGCCATTTTCTCACATGTTTGATGGACGATTTATTCCACTTGCTCGCCCGGAGGTGAAATGGACGCAT
>ENST00000642616_6|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTCATGGGCCATTTTCTCACATGTTTGATGGACGATTTATTCCACTTGCTCGCCCGGAGGTGAAATGGACGGAA
>ENST00000642616_3|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTTTCCTTGGGGGAGAGGAAGAAGCTGCTTAGTTATATCCAGCGATTGGTTCAAATCCACGTTGATACAATGAAGA
>ENST00000642616_3|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTTTCCTTGGGGGAGAGGAAGAAGCTGCTTAGTTATATCCAGCGATTGGTTCAAATCCACGTTGATACAATGAAGC
>ENST00000642616_3|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTTTCCTTGGGGGAGAGGAAGAAGCTGCTTAGTTATATCCAGCGATTGGTTCAAATCCACGTTGATACAATGAAGG
>ENST00000462861_2|ENST00000462861_3
AAAAAACTCTTCTACAGAAATTACTCTCAAAGAAACCTGAGGATCGACCTAATTTTTCACAGACCTACGGGATGGCATCATCTCAGATATATTTGATAAAAAAG

• Can you please clarify in what form/language you need the solution? I see that the solutions provided so far are not in the same form/language as your example. Mar 30, 2021 at 10:02
• I don't understand what you actually have and what you need. What output are you expecting from this example? Did the command you used work or not? I tried running your command and it seems to work fine, I don't get the output you show in the first example. What operating system are you using? Mar 30, 2021 at 10:13

## 3 Answers

If you want to avoid using extra libraries for any reason, you can just use a simple Python script (version 3.6 and above) to do this:

fr = open("dup_test.fasta", "r")
fw = open("dup_edited.fasta", "w")
seq_dict = {}
curr_header = ''

for line in fr:
line = line.strip()
if line[0] == '>':
if line not in seq_dict:
seq_dict[line] = ''
curr_header = line
else:
if seq_dict[curr_header] == '':
seq_dict[curr_header] = line
fw.write(f"{curr_header}\n{line}\n")

fr.close()
fw.close()



You can use seqkit for this purpose.

seqkit rmdup -n seqs.fa -o seqs_without_duplicate.fa

• Very cool kit, thanks for sharing! Mar 30, 2021 at 9:42
• @Bioinfomrat you're welcome :) Your Script is also a quite nice solution.
– Mr_Z
Mar 30, 2021 at 10:25

One way using :

awk '/^>/ { f = !a[$0]++ } f' seqs.fa  Explanation: The above is basically the same as: awk '/^>/ { f = !($0 in a); a[$0]++ } f' seqs.fa  That is, for each header line (lines that start with a '>' character), set a flag, 'f' to true if the current line is not in an array, 'a'. Then add the (header) line to the array. Note that in this form, it's not necessary to increment the value for each element - the following would be sufficient: awk '/^>/ { f = !($0 in a); a[$0] } f' seqs.fa  The default action is to print, and we print lines only when 'f' is true. This is the same as: awk '/^>/ { f = !($0 in a); a[$0] } f==1 { print$0 }' seqs.fa

• Thanks for the explanation! This is a nifty solution. Any idea on its limitations? I can see that it will work as long as the array can be managed in memory, which is quite a small limitation. Apr 6, 2021 at 16:13
• No worries @RamRS. Nothing comes to mind. It should work on most platforms. All bets are off if the input is badly formatted like in the example. It's worth mentioning also that AWK arrays are associative, so only unique keys will be stored in memory. This might be an issue if the input contains a large number of sequences and the number of duplicates is low. Apr 6, 2021 at 23:56
• Also, note that only the header lines are kept in memory, not the sequence, meaning that it is vanishingly unlikely that you will ever come across a file with enough sequences not to fit in RAM on any modern machine. Apr 12, 2021 at 18:18