The pan pots on a conventional mixer just control the level of the signal going to the left and right channels. However, if you mic up a room with a stereo pair and move a "mono" sound source around in it, the effect is actually quite a lot more complicated -- there are timing and phase differences between the channels. If you're making a binaural recording, the differences are even more pronounced.
I think someone ought to make a mixer where the pan pots are an accurate simulation of moving a source around in the room. In combination with some sort of 3D placement device, this could work very well for surround mixes. The technology must exist already, since there are plenty of bits of software that produce simulated 3D sound output.