Evaluate validity, reliability and replicability
Judge whether a measure captures its intended behaviour, whether it is consistent and whether another researcher could repeat the procedure.
Original StudyForA teaching mapped to the Cambridge 2028–2030 syllabus. These lessons are independent study support, not Cambridge material, official mark schemes or awarded grades. No lesson reproduces a research article or Cambridge question.
Before you begin
- This pathway supports the 2028–2030 syllabus only. AS candidates take Papers 1 and 2; A Level candidates also study two of the four specialist options and take Papers 3 and 4. The separate 2027 syllabus is not mapped here.
Work on paper. Try each question before revealing help, and explain why your method works.
Quick readiness check
Two observers agree on 95 of 100 coded events. Which reliability type is being assessed?
What you will learn
- Explain validity and ecological validity
- Evaluate subjectivity, objectivity, demand characteristics and generalisability
- Distinguish reliability types and replicability
- Apply evaluation to new research scenarios
Validity asks whether the evidence supports the intended interpretation
A measure has stronger validity when it captures the intended concept rather than a different one. Ecological validity concerns how far a task or setting reflects the real-life behaviour of interest. Subjective measures depend on judgment or report; objective rules can reduce some judgment, but a consistently scored measure can still measure the wrong thing.
Demand characteristics occur when participants infer the study purpose and change their behaviour. Generalisability concerns applying findings beyond the studied people, setting or time. A narrow sample or artificial task may limit generalisation, but one limitation does not automatically make all evidence useless.
Reliability is consistency; replicability is whether the procedure can be repeated
Inter-rater or inter-observer reliability concerns agreement between different people recording or scoring the same behaviour. Test-retest reliability concerns whether a measure gives consistent results when repeated under comparable conditions. Low reliability can add noise and weaken comparisons; high reliability alone does not prove validity.
A replicable procedure is described clearly enough for another researcher to repeat it. Standardised instructions, materials, timing, sampling and scoring support replication. A replication may produce different findings because of sampling variation or changes in context; the procedure should report what was held constant and what changed.
Sources for this lesson
Pause and explain
A clear procedure can be repeated, but the repeated study finds a different result. What can be concluded?
Worked example
Two observers count helping behaviour using the same video but disagree about what counts as a ‘help’. Which quality is most directly in question?
Show the worked solution
- The same behaviour is being coded by different observers.
- The issue is agreement between observers, which is inter-observer reliability.
- The definition may also need operational clarification; agreement alone would not prove the measure captures all helping behaviour.
Answer Inter-observer reliability is weak; clearer operational categories may improve agreement, though validity still needs separate evaluation.
Try a topic test
Choose Easy, Normal or Difficult. Save your answers and open worked review after finishing.
Test this topic →From guided practice to a new situation
Write your answer and reasoning first. Use a hint only if you are stuck. The model solution is for self-checking; your written work is not automatically marked.
A ‘confidence’ measure records how many times a participant taps a desk. Give one validity concern and one way to improve the measure.
Give me a hint
Ask whether tapping directly captures the intended construct.
Compare my reasoning
- Desk taps may reflect restlessness, habit or the room rather than confidence.
- The operational measure may lack construct validity.
- Use a justified self-report or a validated behaviour measure and compare more than one indicator if appropriate.
Desk-tap frequency may not validly measure confidence; justify a direct measure and check whether it corresponds to the intended construct.
Look for these in your work
- Mismatch identified
- Validity term applied
- Specific improvement proposed
A test gives similar scores each time but measures reaction to bright light rather than the intended fear of crowds. Evaluate reliability and validity.
Give me a hint
Separate consistency from whether the right idea is measured.
Compare my reasoning
- Similar scores suggest test-retest consistency under comparable conditions.
- The measure appears to capture light sensitivity, not fear of crowds.
- It may be reliable but invalid for the stated construct.
The measure appears reliable but lacks validity for fear of crowds.
Look for these in your work
- Reliability evaluated
- Construct mismatch explained
- The two qualities kept distinct
A researcher wants another school to repeat an observation of classroom interruptions. Name information needed to make the procedure replicable and one generalisability limit.
Give me a hint
Another observer needs the operational rules and sample context.
Compare my reasoning
- Specify the interruption categories, observation interval, observer position, lesson type, sampling schedule and scoring rule.
- Train observers and report how disagreements are handled.
- One school or subject may not represent classrooms elsewhere, limiting generalisability.
Provide a standardised category system, setting and sampling protocol, and observer training; findings from one school may not generalise to different schools or lessons.
Look for these in your work
- Replicable procedure detailed
- Scoring consistency addressed
- Generalisability limit tied to sample
Common mistakes
- Treating a reliable measure as necessarily valid.
- Using ecological validity as a synonym for all forms of validity.
- Saying a study is not replicable just because a later result differs.
- Claiming that one unusual participant always makes a study invalid.
What can you explain now?
Can a measure be reliable but invalid?
Compare with the explanation
Yes. It can produce consistent scores while measuring something other than the intended concept.
Consistency is reliability; correspondence to the intended construct is a validity question.
After trying it yourself, choose your next review. This is your self-assessment.
Your review choice appears on Today. Sign in to sync it across devices.
Make it stick
How do validity, reliability, replicability and generalisability answer different evaluation questions?
If you needed the solution, close it and solve the problem again from a blank page. A correct answer is stronger when you can explain the reason for each step.