Can you trust your own data? The three checks that decide

Someone looks at your graph and asks the only question that matters: how do you know this is true?

Not “did the child improve”. Something earlier and harder: how do you know the line on that graph is about the child, and not about who was holding the clipboard, or what got typed in afterwards?

There are three separate ways a record can be wrong, and they need three separate answers.

1. Would a second person have written the same thing?

This is interobserver agreement, and it is the one that decides whether a dependent variable can be reported at all. If only one person ever scored a trial, there is no evidence that the scoring was consistent — only that it was confident.

The procedure is unglamorous: a second adult watches the same child and scores the same trials, independently, and afterwards the two records are compared. The word doing the work is independently. Two observers who can see each other’s answers are one observer with a witness.

Two numbers come out of it:

• Percentage agreement. Ledford and Gast (2018) put the bar at 90% or above for applied research, and treat anything below 80% as unacceptable. Those are their thresholds, and they do not become friendlier because a session went badly.
• Cohen’s kappa. Raw agreement inflates when one category dominates. If the child gets almost everything right, two observers agree constantly by accident. Kappa subtracts the agreement you would expect from chance alone and reports what is left.

There is a third number people forget, and it changes the meaning of the other two: how many trials were scored twice. Agreement of 100% over three trials and agreement of 100% over eighty are not the same claim. Only the denominator tells you which one you are reading.

In Interlaza, you arm the next session for a second observer before it starts, from the child’s Results tab. Agreement cannot be arranged afterwards — the trials have to be observed while they happen. The second adult scores on their own device, and the screen does not show them how the first observer scored a trial until they have committed their own answer. Afterwards the report gives you agreement and kappa, broken down by session and by concept, with the number of trials it was computed over.

The breakdown is not decoration. Ledford and Gast warn specifically against reporting a single averaged agreement figure, because averaging hides the condition where the two observers were pulling apart. So the overall number never appears without its tables.

2. Did the session actually run the way it was designed?

This is procedural fidelity, and it is treated as a risk-of-bias criterion, not as an administrative nicety: a study that measures and reports it has low risk of bias, and one that does not, does not.

On paper this is expensive. It means a second adult with a checklist, ticking boxes: was the prompt delivered at the planned level, was the interval respected, was the reinforcer delivered on the right schedule, was the correction procedure run when it should have been.

Software knows most of that without asking anyone. The engine that ran the session already wrote down, on every single trial, what it did — so the checklist is a reading of the record, not a second person in the room. Interlaza checks four things:

• Prompt fading stayed inside its configured range, never rose after an error, and never jumped more than one step.
• Unprompted opportunities — probes and error-correction trials — arrived with no help on screen. A probe carrying a prompt measures the prompt.
• Reinforcement matched the response, and thinning never started before the concept had earned it.
• Error correction ran the configured procedure and stayed inside its ceiling.

Only invariants that survive the app’s own adaptivity are counted. If the intervention monitor holds a fading phase, or adaptive fading changes how many correct responses a phase needs, or a correction procedure stops early because the child is showing distress — none of those are deviations, and none of them are counted as one.

And the panel says out loud what it cannot check. The trial record stores when a response arrived, not when the trial began, so the inter-trial interval is not verifiable from it; the number of comparisons on screen is not stored either. Both are listed on screen, with the reason, and neither is folded into the percentage. A fidelity figure that quietly drops the items it could not measure is worse than no figure at all.

3. Could the record have been changed afterwards?

The first two checks are about how the data was produced. This one is about what happened to it since.

Every trial in Interlaza carries a hash computed over the trial before it, so the trials form a chain. Editing a trial after the fact — a timestamp, a response, a latency — breaks the chain from that point on, and the break is visible. When a session finishes it is locked, and the server refuses to insert new trials into a session that is already closed.

The honest wording matters here. This does not make the data impossible to alter. It makes alteration evident, which is the property you actually need: nobody has to trust that the file was left alone, because the file can be asked.

What none of this proves

None of the three tells you the child learned anything. Agreement says two people saw the same thing. Fidelity says the procedure ran as designed. The chain says the record is the one that was written at the time.

Together they buy you something narrower and more useful than confidence: they make the graph a claim about the child, rather than a claim about the recording. Whether the child is learning is the next question — and it is worth asking only once these three are answered.

Agreement thresholds and the procedural-fidelity criterion are from Ledford, J. R., & Gast, D. L. (2018). Single Case Research Methodology: Applications in Special Education and Behavioral Sciences (3rd ed.), chapter 5.