The problem is that the enrichment score is modified based on whether the sets are positively or negatively associated, which means that an unbalanced set via either associative statistic or gene count (as is almost always the case) will lead to an incorrect total score.
Put in other words, the calculated score from the positive genes alters the calculated score from the negative genes.
Given this, it's a better idea to either separately calculate scores for the positive set and the negative set (this is the easiest approach, and works with existing methods), or (equivalently) recalculate enrichment score modifiers for positive and negative genes separately (i.e. adjust the graph) so that the zero point for the enrichment score crosses the zero point for association.
A common argument against this (when I have mentioned it to others) is that GSEA has always been done on the total gene set, so it makes sense to do it on the total gene set because it has always been done that way. I'm not a huge fan the status-quo argument.
I alluded to this issue in this previous answer.