Oh, that's what KVM means
I'm not entirely sure that the system is not being confused by this. Even though the switches are external, the data flows two ways. When the system is booted up, it senses what type type of monitor is attached to what video card, and fills in a database of capabilities. I don't know if DirectX is smart enough to sense a change in configuration.
Just grasping at straws here, but that's what happens when I'm looking right at something which would seem impossible. The code seems very clear - if the object shows up at all on a monitor, it should be moving. It's like counting your fingers, and coming up with 11. Logic says it can't be right, but there it is...