Measurement & evidence
The measure is part of the experiment.
Instructions, scales, timing and response formats can change what is observed.
Martha Bajec, PhD · 2 min read · References included
A change in the task can change what the result represents.
Measurement does not simply reveal a response
It helps create the conditions under which the response is produced.
Instructions direct attention. Scale labels define a frame of reference. Anchors constrain interpretation. Question order activates context. Timing changes memory and adaptation. A forced choice requires a distinction even when the participant experiences none.
Self-report research has shown that wording, format and question context systematically shape answers because respondents must interpret what is being asked and construct a response from available information (Schwarz, 1999; Tourangeau et al., 2000). In sensory and consumer work, the same principle applies to ballots, scales, sample order, serving conditions and the language used to define attributes.
Change the task, change the construct
“Which do you prefer?” is comparative. “How much do you like it?” is evaluative. “Would you buy it?” introduces price, identity, access and imagined future behaviour. “Which is stronger?” directs attention to intensity rather than quality or pleasure.
Even familiar scale terms can be unstable across people. Labels such as “very strong” do not denote the same absolute sensation for everyone, which is why cross-group sensory comparisons require careful scaling logic (Bartoshuk et al., 2004).
The researcher can also become part of the stimulus. Brand information, consent language, study purpose, interviewer behaviour and repeated probing can alter expectation, demand and response. Common-method bias is not a single nuisance factor; it can arise from the source, item context, measurement context and response process (Podsakoff et al., 2003).
Design the measurement as carefully as the product test
Before fieldwork, ask:
• What construct does this task require the participant to produce? • What does the wording make salient? • What reference frame do the anchors create? • What response is forced, permitted or impossible? • Could order, repetition or timing change the experience? • Would another measure support a different inference?
Pilot the instructions, not just the technology. Randomize or counterbalance when order can matter. Separate measurement from interpretation. Report enough operational detail for another scientist to understand what participants actually did.
The measure is not a transparent window onto experience. It is part of the experiment.
References
- Bartoshuk, L. M., Duffy, V. B., Green, B. G., Hoffman, H. J., Ko, C. W., Lucchina, L. A., Marks, L. E., Snyder, D. J., & Weiffenbach, J. M. (2004). Valid across-group comparisons with labeled scales: The gLMS versus magnitude matching. Physiology & Behavior, 82(1), 109–114. https://doi.org/10.1016/j.physbeh.2004.02.033 ↗
- Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879 ↗
- Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105. https://doi.org/10.1037/0003-066X.54.2.93 ↗
- Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.
A useful next conversation
How does this connect with your question?
We can look at the result, brief, or product problem together and work out what to investigate next.