I need to lightly verify a set of docking simulations from the reference RMSD of the outputs, and as we all know reference RMSD depends on the initial coordinates of the ligand set. What value of reference RMSD after docking from any starting coordinate would be considered outrageous to have?
Let's get some definitions out of the way:
RMSD, root mean square deviation, is a metric of distance of molecule A and molecule B. Think of it as average euclidean distance. It us casually called just 'deviation', but has nothing to do with standard deviation (the root of variance, 2nd moment). The units are Ångström.
Reference RMSD is a term that appears in some manuals and is a the RMSD between the calculated pose and the starting pose. When you set up your docking you as a user define the starting position and how much it can deviate from that position. If you have a crystal structure with a ligand and you are docking the same or similar compound, then yes, it makes total sense to worry about this metric. In which case, RMSD greater than 2 are worrisome in a 2 Å resolution structure. You especially see RMSD as a metric of accuracy when PDB structures are redocked, wherein the same ligand is docked and reference is the file from the PDB (or its PDB Redo derivative), so RMSD greater than 2 Å is bad as it would not match the density, while under 1 is really good —note that in tests about 50% and 75% of re-docked ligands are docked less than 2 with AutoDock (free) and ICM-Dock (pay) respectively. But if you are doing it for an experiment and not for a test, you would probably have set the docking to be highly constrained in the first place (the grid size in AutoDock or position randomisation factor in most others, eg. Rosetta Ligand dock). If you actually care about reference RMSD the thing to manually look at is the top 3-4 poses in the electron density of your native structure. If you don't really know how your ligand docks please ignore this value as it is just a product of your starting pose. It is not a measure of how "happy" the complex is.
ΔGibbs is a negative value in kcal/mol that is basically a metric of how "happy" the atoms are. It is actually an energy potential, but confer Wikipedia pages for the proper definition. The more negative it is the happier the atoms are. Force-field calculations solve this value and the difference between undocked and docked (ΔΔG) is a good metric of how well your ligand docks. Please use this.
Affinity ΔΔG is not a k_off or a k_on. That is a result of the ligand binding to nearby positions and how well it can get into the binding site (e.g. in P450 enzymes the ligand orientation is dictated by how it squeezes into the entrance way). And how insoluble the ligand is, which is a major problem*. For a better metric, there is dynamic undocking where a constrained MD simulation is done with the ligand getting pulled out of the pocket. But this is not trivial.
∗ As any comp-chemist can tell you it is mighty hard to calculate solubility. Some force-field calculations do not even take account of π-π or S-π interactions. Drug-like compounds are rigid aromatic and rigid and highly hydrophobic. Many programs use a hack called implicit waters, where the water is not simulated as individual molecules but as a flowing thing. This means that ΔΔG benefits of constrained waters are ignored and alternative protontation states of water too. This is fine for protein (mostly), but has major errors for drugs. But in terms of thing to look out for in a docking algorithm, explicit waters is not the first as conformer sampling and side chain perturbation are more important.