I'm no expert with VCF (few can say they are!) but I have worked a lot with VCF data in the last few years, both tools to consume and produce VCF. I've never seen variants encoded in this fashion, and it seems to be non-canonical. Typically:

- Single nucleotide variants (SNVs) are encoded with a single base as the REF allele and a single base as the ALT allele.
- In the case of insertions or deletions, the shorter of the REF and ALT alleles will be a single base, the base preceding the inserted/deleted sequence. Thus the first base of the REF and ALT alleles is always the same.
- In the rarer case of two or more consecutive substitutions forming a multinucleotide variant (MNV) the REF and ALT alleles will have the same length.

Using multi-bp strings of the same length to encode SNVs is unnecessary and, as you've pointed out, problematic. This makes me think its a bug or a "feature" of the variant predictor that produced the VCF.

In this case, I'd write a small script that would check for variants where the REF and ALT alleles have the same length. If the base is the same for REF and ALT in any position, drop it, and adjust the position accordingly.

The script below will convert these funky SNVs to the canonical representation, and will also work on MNVs. Standard tools should then work to remove the duplicates.

<!-- language: python -->

    #!/usr/bin/env python3
    
    
    def canonicalize(instream):
        for line in instream:
            if not line.startswith('#'):
                values = line.split('\t')
                pos = int(values[1])
                ref, alt = values[3:5]
                if len(ref) > 1 and len(ref) == len(alt):
                    # How many bp to trim off the end
                    for n, (r, a) in enumerate(zip(ref[::-1], alt[::-1])):
                        if r != a:
                            revoffset = -1 * n
                            break

                    # How many bp to trim off the front
                    for n, (r, a) in enumerate(zip(ref, alt)):
                        if r != a:
                            offset = n
                            values[1] = str(pos + offset)
                            values[3] = ref[offset:revoffset]
                            values[4] = alt[offset:revoffset]
                            break
                    line = '\t'.join(values)
            yield line
    
    
    if __name__ == '__main__':
        import sys
        for line in canonicalize(sys.stdin):
            print(line, end='')