A site is simply an individual discrete position, normally a single nucleotide (or amino acid). For example, consider a toy alignment:
sequence1 ATGC
sequence2 ATCC
# ^ arrow points to the third site
Here there is 1 mutation in 4 sites, so 0.25 substitutions per nucleotide site.
I'm not sure where the term originated, but probably one of the earlier definitions is from renowned population geneticist Motoo Kimura, who simply stated in 1968:
In this paper, “site” refers to a single nucleotide pair
A pair in this context, means either a homozygous or heterozygous site.
Synonymous sites are positions where nucleotide changes lead to no amino acid change, these are typically at the 3rd position of the codon where the nucleotide is often degenerate, for example, TAT, TAC TAG, and TAA all encode the Serine amino acid. By contrast non-synonymous sites are where a single nucleotide change does lead to an amino acid change, and therefore possibly a more important functional change to the resulting protein.
Thus, the ratio of Ka/Ks (or Dn/Ds) is thus often used as a crude proxy of selection, with Ka/Ks < 0, Ka/Ks = 0 and Ka/Ks > 0 implying purifying (negative) selection, neutral evolution and positive selection respectively. Note that this assumes the neutrality of synonymous sites, which is increasingly questioned due codon usage bias and regulatory elements overlapping coding regions (and also ignores mutational biases and biased gene conversion, although these can be accounted for in an evolutionary model, and also assumes no sequencing errors and perfect alignments, which can impact a lot on the results too).
