I am performing a haplotype and nucleotide diversity analysis of my Sanger sequences (from Watermelon mosaic virus coat protein region) using the Arlequin program. My aim is to see the probability that two randomly sampled alleles are different, and the expansion and genetic variation of the virus respectively.

Some of my sequences have gaps (-) because the PCR reaction could not cover all my region of interest. After making the alignment in Mega and export in a Nexus file to DnaSp6 I saw that there are many options to export to the Arlequin program after DnaSP6. I do not know what is the best option to generate the output from DNAsp6 to Arlequin because I must choose one option for the sites with the alignment gaps option and another option for the invariable sites option.

These are the options:

Sites with Alignment Gaps options:

Not considered: These sites are ignored (complete deletion).
Considered: Gaps are considered just like another nucleotide variant (fifth state).
Only gaps are considered: Only gaps information is considered to build haplotypes.

Invariable Sites options:

Removed: Invariable (monomorphic) sites will not be included in the output file.
Included: Invariable (monomorphic) sites will be included in the output file.

From my point of view I think that it is more certain for my analysis (above explained) to use the Not consider gaps option (because they can buffer the information of the other sequences) and include invariable sites option because I want to analyse these parameters of my dataset.

Is it sensible? I would like to know what is the best option for my dataset.


1 Answer 1


Yes, your question makes alot of sense and its a good question because hits at the heart of differences between population genetic analysis and phylogenetic analysis. However, when you speak about gaps you are referring to 'indels' (insertion/deletion).

Indels The conservative approach is to simply delete sites with gaps. Arlequin uses a lot of maximum likelihood solutions and gaps would automatically be deleted in an Arlequin max. likelihood analysis, because it is the output per alignment per matrix. If the matrix doesn't have a 'gap'.

Indels and pop gen Thats tricky, it is better to remove them. HOWEVER the indel may contain underlying population signal, if its a maximum likelihood analysis (pop gen does this too) I suspect they will be ignored. If its pairwise, such Fst including indels might give you a better answer because there is more polymorphisms to assess population structure. So if you're doing a pairwise analysis, I'd do it both ways and then report the methodology correctly if there are differences in results and simply explain the theory.

Invariable sites This is different from an indel. An invariable site is a site which carries no phylogenetic signal, it carries information but a phylogenetic analysis could draw any conclusions because all the nucleotides/amino acids are identical at a give alignment position.

The output you obtain in allele frequency will be different and here's the rub ...

Phylogenetics analysis Arlequin is really pop gen, but it will contain some phylogenetics programs. If you see anything with 'Jukes Cantor correction', 'Kimura 2 (or 3) parameter model', 'HGW model', '"REV" model' (eek I can't remember its name any its a reverible 6-parameter model of nucleotides), you are dealing with phylogenetics. Keeping invariable sites is normal here because the rate matrix is calculation "per alignment site".

Population genetics analysis Fst, Fis, linkage disequilibrium statistics, REMOVE THEM. Why? Because you want SNPs (single nucleotide polymorphisms). The invariable sites will get in the way of the statistics. For example is the locus recombines and you want to assess a linkage disequilibrium statistic (recombination), invariable sites will recombine, but the algorithm can't see this because the nucleotides are all the same. The only thing the algorithm can see and analyse are SNPs, which 'representative' of all the surrounding invariant sites. Thus removing invariants is essential. Arelquin might do this automatically (but I don't know the preparatory algorithms), but if you submit multiple loci Arlquin will want to know which are different loci. It will then recode the loci automatically if a single locus comprises more than one SNP. We are going into the underlying theory of linkage disequilibrium here and I'm not sure if say Fst/Fis is particularly bothered with this or is just concerned with the populations of genetic diversity.

Summary So gaps ... if they are genuine indels and not just 'N', ie indeterminant nucleotide calls, remove them for phylogenetics. If its pop gen do the calculation both ways and assess the output, if there is no difference report the 'gap/indel removal' result.

Invariant sites .. for pop gen you remove them but Arlequin might do this automatically to prepare a data set for analysis (I don't know). For phylogenetics you never remove invariant sites.


  • $\begingroup$ Wow, thanks you Michael for your answer. It looks great but I have some doubts. Perhaps my lack of knowledge in this field does not allow to me unterstand well your answer. I am analyzing the Fst fixation index too. I though that conservative regions may introduce information about my sequences and if all my sequences consider this regions in the FST analysis the results will be sense, aren't them? Thank you for your answer. $\endgroup$ Commented Apr 29, 2020 at 11:48
  • 1
    $\begingroup$ Hi @AdrianPL you are welcome. For Fst I would remove invariants (no change, if there is one SNP+ keep it) because it give stronger population structure, but indels/gaps are more complicated (because they might contain information and honest I would just leave them in). Arlequin was particularly written in the days of microsatellites where every position was an allele. The Fst I know is the classic pairwise calculation, there are maximum likelihood versions, but I don't know how they work. Fixation, if you mean homozygosity, also consider Fis (which is what is designed for-separate question). $\endgroup$
    – M__
    Commented Apr 29, 2020 at 12:04
  • $\begingroup$ Thanks you Michael. I removed the indels of my dataset and I think that your explanation is very sense because the Fst values gave to me a stronger population structure than those produced by Rstudio or Dnasp6. I prefer make all analysis with the same program, Arlequin because It gives to me Standard error and permutation values. Is there any article that remove the invariant sites to calculate the Fst value? Thanks you!! $\endgroup$ Commented Apr 29, 2020 at 12:27
  • 1
    $\begingroup$ In NGS era there are lots, anything on NGS Plasmodium falciparum population genetics, but it would standard practice and Dnasp6 isn't regarded in the same calibre as Arlequin, which is an 'old school standard'. R however has a lot of pop gen approaches, so its difficult to generalise here. You could ask this as a separate question if you thought it was critical, but from my point of view Fst is a measure of population migration, not genetic diversity, so I would see the removal of invariants as a positive step from an emperical mathematical perspective. $\endgroup$
    – M__
    Commented Apr 29, 2020 at 14:44
  • 1
    $\begingroup$ Thanks you a lot Michael! $\endgroup$ Commented Apr 29, 2020 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.