Estimated reading time: 10 minutes
Doing effective therapy is hard enough on its own. Integrating measures of outcome into the process adds additional complexity. Do it well, and the purpose, meaning and value of using measures is integrated into conversations. Do it poorly, and we hear the crunch of gears and a trail of potential alliance fractures. As this recent study neatly illustrates.
Table of contents
- What goes on behind that door?
- Study outline
- How not to use outcome measures in therapy
- Theme 1: Lack of consistency in the administration of outcome measures
- Theme 2 Outcome measures administered as a stand-alone task lacking in flexibility
- Theme 3: Outcome measures as impersonal numbers
- Theme 4: Missed opportunities to use outcome measures therapeutically
- Meanwhile, in the client’s shoes…
- Using measures – Back to basics
- Final thoughts
What goes on behind that door?
It’s not often we get an insight into what happens behind the therapy room door, and how those conversations affect client engagement. A study I came across recently offers a glimpse into the dangers that using outcome measures, when poorly integrated into practice, pose for the working alliance and wider engagement of clients.
From measures being administered as a stand-alone task to a lack of consistency in their administration, and measure data being treated as impersonal numbers rather than an opportunity to explore the client experience, this study is studded with examples of poor practice. More of this below, but first a brief outline of the study and its aims.
Study outline
This study looked at the administration and therapeutic use of sessional outcome measures within and between sessions delivered by phone in Step 2 care in five IAPT services across the North and East of England. Participants were 11 patients and 11 practitioners recruited from those services. Measures were the GAD-7, PHQ-9, IAPT phobia scale and the Work and Social Adjustment Scale (WSAS).
Thematic analysis was used to analyse transcripts of session recordings between September 2018 and July 2019. Sixteen transcripts were selected for analysis, nine from session one and seven from session two.
How not to use outcome measures in therapy
Many of the study’s findings are illustrated using extracts from session transcripts. For anyone versed in integrating measures into their practice, many make painful reading. They show, in example after example, practice in administration and use of measures that falls far short of good practice. So much so that at times I found myself wincing and wondering how those transactions might have felt for the client.
Four main themes emerge from the study (each with two or three sub-themes)
Theme 1: Lack of consistency in the administration of outcome measures
Theme 2: Outcome measures administered as a stand-alone task lacking flexibility
Theme 3: Outcome measures as impersonal numbers
Theme 4: Missed opportunities to use outcome measures therapeutically
Theme 1: Lack of consistency in the administration of outcome measures
Analysis revealed inconsistent wording to present questionnaire items and rating options, presentation of conflicting information and inconsistencies in the rationale presented for using measures.
Often, items were shortened or rephrased. For example, item 6 on the PHQ-9 reads:

This was rephrased by one practitioner to simply ‘feeling bad about yourself.’
Item 3 of the IAPT Phobia Scale asks ‘How much do you avoid certain situations because of a fear of particular objects or activities (such as animals, heights, seeing blood, being in confined spaces, driving or flying)’
Contrast this with the question asked by the Psychological Wellbeing Practitioner (PWP) and the resulting dialogue:

Data from session one transcripts often indicated that measures were presented to clients as something that needs to be done at every session, but without providing a rationale.

Theme 2 Outcome measures administered as a stand-alone task lacking in flexibility
Measures tended to be administered as a stand-alone administrative task, often disconnected from the rest of the session. They were commonly framed as an encumbrance rather than an opportunity to understand clients’ experiences, as the following extracts illustrate:


Clients sometimes provided a narrative description of their experiences in response to an item question and this information was omitted, forgotten or disregarded by practitioners. In the following illustration the practitioner jumps from question three to question five of the PHQ-9 without actually getting a client rating of the first question.

Theme 3: Outcome measures as impersonal numbers
Typically, clients’ responses to the measure items were treated as abstract, impersonal numbers, generally devoid of the background significance that would have given scores, or shifts in scores, personal meaning.

There were only four examples of practitioners looking for explanations behind the total scores. Here, practitioners attempted to crosscheck clients’ experiences with total scores using closed questions such as ‘Does that seem accurate?’, ‘Does that feel right?’, ‘Does that kind of fit with how things have been?’, ‘Is that reflective of how you are feeling at the moment?’.
In only one case was there an example of the practitioner using an open question in Session 2 inviting the client to reflect on what they had been doing between sessions that might explain the reduction in total scores: ‘But what do you think has kind of happened over the last week to kind of have such a big change on your scores?’
Initially the client did not know why their scores were reduced but went on to talk about making changes to their sleep routine, eating healthier, relaxing more and making a conscious effort to feel better, and talking about their problems. Subsequently, they indicated that reflecting together on the measures with the practitioner was helpful.
Theme 4: Missed opportunities to use outcome measures therapeutically
If you have experience of using outcome measures in your practice you’ve probably spotted a range of missed opportunities to extract personal meaning and relevance from clients’ responses to individual measure items and their overall scores.
The study highlights that further opportunities were routinely missed. This extract illustrates the chance missed by the practitioner to explore information provided by the client about medication and their perception of its impact on their sleeping patterns.

These two final extracts demonstrate the opportunities missed by practitioners to explore client responses to the Work and Social Adjustment Scale (WSAS) and IAPT phobia scale.

Meanwhile, in the client’s shoes…
How do you imagine some of these exchanges were for the client? How would it have been to be asked to reveal highly personal information with no discernible rationale? To try to follow the practitioner’s rambling responses? To offer contextual information and have that ignored?
I imagine any of those exchanges might have felt somewhere between mildly to moderately discounting, and you won’t need me tell you how toxic that can be for the therapeutic relationship.
Much as my sympathies lie mainly with the clients, I also feel for the PWP’s involved. Despite the fact that they had an average of three years’ practice experience and were under supervision, they have the air of people thrown in the deep end with little or no preparation, and receiving little by way of ongoing support. That’s pretty unforgivable.
Using measures – Back to basics
If the examples above highlight some basic pitfalls in using measures, what should we be doing instead? Here are my top five principles.
1. Do a little homework
There’s plenty of research showing what works in therapy, as well as what doesn´t. This includes evidence showing the contribution that routine outcome measurement makes to therapy outcomes. A simple grasp of the basics (plus a bit of practice) is all that´s required to develop a degree of confidence in using measures.
2. Stick to the script
Like them or loathe them, a great deal of care and science goes into the construction of a robust measure. If you’re reading the items to the client, read them in full. If you don’t, then, however the client may respond, they won’t be responding to the item on the measure. You’ll also be undermining the validity of the client’s responses.
3. Offer a coherent rationale
Don’t, ever, ask the client to do anything without providing a good reason. If, during our conversation, you ask me to jump through a hoop, I’d like to know how that’s going to help me. Convince me, then pass the hoop. At its most basic, it’s a respect thing. We’ve even written some guidance to help.
4. Value the client’s responses
Value the client’s responses and make them part of the therapeutic conversation. Hopefully few of us would be as crass as to say “Thanks for completing that form. Now you’ve done that we can get on with the real work.” Might we, however, convey a similar message more subtly?
5. Use the client’s responses to enhance your understanding of their experience
Above, under Theme 3, are a few of the limited examples where the practitioner has taken the opportunity to crosscheck the client’s score with their felt sense. Better still, there’s an example of the practitioner inviting the client to connect the reduction in their scores with their own actions.
Here are a couple of further ways we might open up the conversation. Both examples use the measure I know best and use routinely, the CORE-OM. First, a response I might make in trying to understand more about a client´s experience at assessment.
[In response to the CORE-OM item I have had trouble getting to sleep or staying asleep] “I notice that you’ve checked Most or all of the time in response. Can I get you to say a bit more? Would you say the past week is more or less typical?”
The second example might be typical of a session later in therapy reflecting on a shift in a client’s predominant pattern of self-blame.
[In response to the item I have thought I am to blame for my problems and difficulties] “I notice that your score has reduced from 4 at the start of our work together to 1 now. That means you’ve gone from Most or all of the time to Only occasionally. That’s a big change. Does that feel about right and if so what sense do you make of it?
Final thoughts
This was a small-scale study of interventions conducted by phone in a particular time period. To what extent its findings are generalisable, either across IAPT, or wider, we just don’t know. If they are, then, for reasons discussed above, that’s a real concern. Clients can be remarkably forgiving of our mistakes, but when we keep on making the same mistake over and over, we’re on borrowed time.
If you use measures in your practice, what do you do to make clients´ responses part of a therapeutic conversation? If you don´t, what do you think of what you´ve read here? Either way, we´d love to hear from you.

Just like you we thrive on feedback.
Please leave your thoughts on what you’ve read in the comments section below.
In the above examples, it’s pretty clear to me that the Outcome Measures are being used as a “tick box” exercise with clients, and the therapists don’t really buy into the reliability, validity, and feasibility of the measures.
This isn’t a clinical issue – it’s a performance management issue.
Many thanks for your comment Stephen, and I agree completely with you that it appears that measures are being used as tick box exercise and little more. If that’s true it exposes the paradox at work in the services behind this study – which is that when you use measures competently you enhance your outcomes. I’d give my right arm to know the recovery and attrition rates for those services!
Cheers, Barry
When introducing outcome measures, I always say to clients that “It’s another way for you to tell us how you’ve been doing”. I find them a very helpful session “opener”, if things have deteriorated (or improved !) since we last met , I’ll remark on it (showing it to the client the graph) and we will usually then discuss what’s been happening over the previous week. I also find them enormously valuable in showing clients how much things have improved over time (since the Initial Meeting & during the course of our work). With around 70% of progress down to clients own actions, it offers a great opportunity for us to explore the changes they’ve made to their lives / thinking etc that led to the improvements. They rightly feel a sense of pride at the achievement. I even find some clients like outcome measures to show their partners, the results from CORE will often facilitate deeper conversations. So I think if used as a way to facilitate conversation, Outcome Measures are extremely useful & should not be seen as a “tick box” exercise
Thanks Graeme – I really can’t add anything to your excellent list of opportunities to use measures in a therapeutic way. Except maybe framing the progress chart for the client at the end? 🙂
Cheers
Barry
William M Epstein is his several books the illusion of psychotherapy, psychotherapy a religion and psychotherapy and the social clinic, soothing fictions has done an excellent job at looking at the alleged evidence for psychotherapy and he’s also taken apart the various psychometric measures commonly used in therapy.
I would love to see these books and the evidence they present fully reviewed and commented on this blog.
The PHQ9 and GAD7 in IAPT might offer as any measure might, an opportunity to ask further questions for exploration – as for recovery they are more often than not tools of people pleasing or tools of manipulation and tools of secondary gains.
They are also tools used to micromanage and bully staff and the use of them as the be all and end all of ‘recovery’ pits therapist against client, causing huge stress and adding to burn out. These two measures in particular also help to frame suffering in a medicalised framework of ‘depression and or ‘GAD’ The glaring issues with the medicalisation of distress is well outlined here
https://www.madinamerica.com/insane-medicine/
I have had people tell me that past therapists have essentially cajoled and bullied them into revising the scores down. In fact in some services they ran ‘recovery workshops’ that were all about manipulating the scores eg – I see you’ve scored a 3 here but didn’t you tell me you went out last week…so would this be closer to a 2? Stuff like that, do this a few times and people either get the message or double down.
Consider the evidence here
https://thefutureoftherapy.org/iaptsurvey
Clients fought against the monolith of psychiatry to get ‘recovery’ back on the agenda from the message that you are mentally ill and will need drugs for life – I am sure they would be horrified to know that recovery has come to mean so little – recovery is a changeable, nuanced and personal thing not a score on a drug company funded questionnaire that has helped to over medicalise and drug millions of people.