For pandas you simply load the fasta names and fasta sequences into separate arrays. The "fasta names" array becomes the index in a pandas Series, whilst the sequence are the values. The dataframe will correctly re-associated the names with each sequence. You load all the files into a single dataframe and use the duplicated() function then write out.
The easiest way to do this is concatenate all your files into one via shell ...
cat file*.fasta > concat_file.fa
Python script ...
from Bio import SeqIO
import pandas as pd
path = "/Users/username/location/concat_file.fa"
outpath = "/Users/username/Desktop/duplicates.fasta"
fasta_name = []
fasta_seq = []
for record in SeqIO.parse(path, "fasta"):
fasta_name.append('>' + record.id)
fasta_seq.append(record.seq)
df = pd.Series(fasta_seq, index = fasta_name)
df_dups = df.index.duplicated()
mydups = df[df_dups]
mydups.to_csv(outpath, sep="\n", index=True, header=False)
...
I checked it on a ~4000 sequence fasta file of unique sequence ids (dengue), duplicated the last sequence and the only sequence output was the very last sequence.