5
$\begingroup$

I used the following command to remove duplicate FASTA sequences based on the header sequence:

paste -d $'\t' - - <fastaFileWithNoLinebreaksInSeq | sort -t $'\t' -uk1,1 | awk 'BEGIN{FS="\t";OFS="\n"}{print $1,$2}'

After this command, the output result is shown below:

>_10|_10
AAAGCAAAACACAGAATTGACGGAAAGACTTACGTTATTAAACGTGTTAAATATAATAACGAGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTC
>_10|_11
ATTATGATCCTGAGACCAGTGATGATTCTCTTGAGAGCAGTGATTATGATCCTGAGAACAGCAAAAATAGTTCAAGGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACACA
>_10|_12
TTTGGAACTCTTTGAACAAATAACAAAAGGGGTGGATTATATACATTCAAAAAAATTAATTCATAGAGATCTTAAGGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACACA
>_10|_13
ACTTGTAACATCTCTGAAAAATGATGGAAAGCGAACAAGGAGTAAGGGAACTTTGCGATACATGAGCCCAGAACAGGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACACA
>_10|_14
AACGAATTTCTTCGCAAGACTATGGAAAGGAAGTGGACCTCTACGCTTTGGGGCTAATTCTTGCTGAACTTCTTCATGTTTGGCATGGATTTTAAAGAAATAGAATTAATTGGCTCAGGTGGATTTGGCCAAGTTTTCAAAGCAAAACAC

This is my actual FASTA file: I don't want to lose header information from my FASTA sequence after removing duplicate sequences based on header information.

>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGCATGAAC
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGAAGTTG
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGGTGGGG
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGTAATTA
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGGTCATGG
>ENST00000642616_2|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTAAAATGAAATCACAGGCGCATTACTGCCTTGTCTTGATGAGTCTCGTTTTGAAAATCTTGGAGTAAGTTCCTTG
>ENST00000642616_8|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTATTTGGTGCCTATCCTAAAACTTCCAGTGGGTTTTCTTGCCTTCCAGTGGCCATATAAAGGGCGTCCTGAAAAC
>ENST00000642616_7|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTCATGAACAAGGCTCAGTTATGATGTTTGAGCACCTTATTAATTCTAATGGAATTAAGCCTGTCATGGAACAATA
>ENST00000642616_1|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGCGCCTGCGCGCGGGTCCGGCGCCGAGGTTCTTGACTGCTGTGCCGGACGCCAGGTGTAGCCATGCAGCGAGCC
>ENST00000642616_9|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGGACTGCCATCATCTTGGAATCCAAAATAATTTTGATTACAAGCGCTTTATTAAGTTTGCCCGTGTCTGTGAAG
>ENST00000642616_5|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGGTGGGGTATCTAGCAGGATGTCTAGTTCACGCACTGGGTGAAAAACAACCAGAGCTGCAGATAAGTGAACGAG
>ENST00000642616_4|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTAATTAATGATCCTATCCATGGCCACATTGAGCTCCACCCTCTCCTCGTCCGAATCATTGATACACCTCAATT
>ENST00000642616_6|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTCATGGGCCATTTTCTCACATGTTTGATGGACGATTTATTCCACTTGCTCGCCCGGAGGTGAAATGGACGATT
>ENST00000642616_6|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTCATGGGCCATTTTCTCACATGTTTGATGGACGATTTATTCCACTTGCTCGCCCGGAGGTGAAATGGACGCAT
>ENST00000642616_6|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTGTCATGGGCCATTTTCTCACATGTTTGATGGACGATTTATTCCACTTGCTCGCCCGGAGGTGAAATGGACGGAA
>ENST00000642616_3|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTTTCCTTGGGGGAGAGGAAGAAGCTGCTTAGTTATATCCAGCGATTGGTTCAAATCCACGTTGATACAATGAAGA
>ENST00000642616_3|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTTTCCTTGGGGGAGAGGAAGAAGCTGCTTAGTTATATCCAGCGATTGGTTCAAATCCACGTTGATACAATGAAGC
>ENST00000642616_3|ENST00000642616_10
AAAAAAAATCTAGCATAAAGTTAATTTAGTTGGAGGAAAAGGAGGGAAGTGGATAAAGGGGCAGATGGAGGGAAGTTTCCTTGGGGGAGAGGAAGAAGCTGCTTAGTTATATCCAGCGATTGGTTCAAATCCACGTTGATACAATGAAGG
>ENST00000462861_2|ENST00000462861_3
AAAAAACTCTTCTACAGAAATTACTCTCAAAGAAACCTGAGGATCGACCTAATTTTTCACAGACCTACGGGATGGCATCATCTCAGATATATTTGATAAAAAAG
Improve this question
$\endgroup$
2
  • $\begingroup$ Can you please clarify in what form/language you need the solution? I see that the solutions provided so far are not in the same form/language as your example. $\endgroup$
    – Noah Sprent
    Commented Mar 30, 2021 at 10:02
  • 1
    $\begingroup$ I don't understand what you actually have and what you need. What output are you expecting from this example? Did the command you used work or not? I tried running your command and it seems to work fine, I don't get the output you show in the first example. What operating system are you using? $\endgroup$
    – terdon ♦
    Commented Mar 30, 2021 at 10:13

3 Answers 3

Reset to default
10
$\begingroup$

If you want to avoid using extra libraries for any reason, you can just use a simple Python script (version 3.6 and above) to do this:

fr = open("dup_test.fasta", "r")
fw = open("dup_edited.fasta", "w")
seq_dict = {}
curr_header = ''

for line in fr:
   line = line.strip()
   if line[0] == '>':
        if line not in seq_dict:
          seq_dict[line] = ''
          curr_header = line
   else:
        if seq_dict[curr_header] == '':
          seq_dict[curr_header] = line
          fw.write(f"{curr_header}\n{line}\n")

fr.close()
fw.close()
Improve this answer
$\endgroup$
0
9
$\begingroup$

You can use seqkit for this purpose.

seqkit rmdup -n seqs.fa -o seqs_without_duplicate.fa

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Very cool kit, thanks for sharing! $\endgroup$
    – Bioinfomrat
    Commented Mar 30, 2021 at 9:42
  • 1
    $\begingroup$ @Bioinfomrat you're welcome :) Your Script is also a quite nice solution. $\endgroup$
    – Mr_Z
    Commented Mar 30, 2021 at 10:25
4
$\begingroup$

One way using :

awk '/^>/ { f = !a[$0]++ } f' seqs.fa

Explanation:

The above is basically the same as:

awk '/^>/ { f = !($0 in a); a[$0]++ } f' seqs.fa

That is, for each header line (lines that start with a '>' character), set a flag, 'f' to true if the current line is not in an array, 'a'. Then add the (header) line to the array. Note that in this form, it's not necessary to increment the value for each element - the following would be sufficient:

awk '/^>/ { f = !($0 in a); a[$0] } f' seqs.fa

The default action is to print, and we print lines only when 'f' is true. This is the same as:

awk '/^>/ { f = !($0 in a); a[$0] } f==1 { print $0 }' seqs.fa
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the explanation! This is a nifty solution. Any idea on its limitations? I can see that it will work as long as the array can be managed in memory, which is quite a small limitation. $\endgroup$
    – Ram RS
    Commented Apr 6, 2021 at 16:13
  • $\begingroup$ No worries @RamRS. Nothing comes to mind. It should work on most platforms. All bets are off if the input is badly formatted like in the example. It's worth mentioning also that AWK arrays are associative, so only unique keys will be stored in memory. This might be an issue if the input contains a large number of sequences and the number of duplicates is low. $\endgroup$
    – Steve
    Commented Apr 6, 2021 at 23:56
  • $\begingroup$ Also, note that only the header lines are kept in memory, not the sequence, meaning that it is vanishingly unlikely that you will ever come across a file with enough sequences not to fit in RAM on any modern machine. $\endgroup$
    – terdon ♦
    Commented Apr 12, 2021 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.