Getting rid of duplicates in a dictionary

Question

I have a tsv file that lists the reads and read lengths from a FASTA file but some reads are duplicated - that's just from the analysis I did previously - but I want to only take one instance of the read and have it be the SHORTER length than the other duplicate.

For instance, here is the beginning of the tsv file:

read                                    length
91290b97-5ca1-489d-b488-17afc1d217e7    3584
dad08436-3cd6-40a1-9af1-b846bd24a815    3787
f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
f4a7bbdf-de62-48a8-a8b0-06076e251820    4717
21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520

You can see that f4a7bbdf-de62-48a8-a8b0-06076e251820 is found twice with different lengths 3637 and 4717. I want to go through and take the shorter length of that read and include it in the final tsv file, resulting in this structure:

read                                    length
91290b97-5ca1-489d-b488-17afc1d217e7    3584
dad08436-3cd6-40a1-9af1-b846bd24a815    3787
f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520

I'm trying to do this in python and have started with making the tsv file a list:

newlist = []
duplist = []

with open(tsv_path) as file:
    tsv_file = csv.reader(file, delimiter="\t")
    
    for line in tsv_file:
        if line[0] not in newlist:
            newlist.append(line[0])
        else:
            duplist.append(line)
            
    print("All the duplicate reads: ", duplist)
    print("The normal reads: ", newlist)

Which works to contain the first instance of a read in one list and the second/third instance of the read to go in a separate list with their lengths. However, the shorter length of a read is not always the first instance of a read. Do I need to include another loop that looks at the lengths and compares each instance of a read? Is there an easier way to do this?

Answer 1

Not sure how you'd do this in python but you need a group-by operation here. Group by the read name and pick the min length per group.

EDIT: With pandas, you can do:

df.groupby(['read'])['length'].agg('min')

Answer 2

You can do this with the sort command-line tool. This command pipe sequence sorts numerically by the second field, then chooses unique elements based on the first field

sort -r -k 2n,2 read_lengths.txt | sort -r -u -k 1,1 > unique_lengths.txt

[the additional -r bits are not strictly necessary; they help make sure that the headings stay at the top]