Subjective quality perception studies with human observers are essential for multimedia system design. Such studies are known to be expensive and difficult to administer. They re- quire time, a detailed knowledge of experimental designs and a level of control which can often only be achieved in a lab- oratory setting. Hence, only very few researchers consider running subjective studies at all. In this paper we present Randomised Pair Comparison (r/pc), an easy-to-use, flexible, economic and robust exten- sion to conventional pair comparison methods. r/pc uses random sampling to select a unique and small subset of pairs for each assessor, thus separating session duration from the experimental design. With r/pc an experimenter can freely define the duration of sessions and balance between costs and accuracy of an experiment. On a realistic example study we show that r/pc is able to create stable results with an accuracy close to full factorial designs, yet much lower costs. We also provide initial evi- dence that r/pc can avoid unpleasant fatigue and learning effects which are common in long experiment sessions.