{
.. or the Dunning-Kruger effect is alive & well and living in a therapy service near you!
}
You would like to think, wouldn’t you, that given the nature of what we do as therapists, including promoting fairness and balance in our clients’ self-appraisal of themselves, that we would be able to apply this to our own professional self-appraisal? You might like to believe that, but sadly it appears that belief would be mistaken.
The Dunning-Kruger effect refers to a form of cognitive bias whereby we may over-estimate our ability as being higher than it actually is.
The phenomenon was observed in a series of experiments by David Dunning and Justin Kruger at Cornell University, inspired by the case of McArthur Wheeler. Wheeler had robbed two banks after covering his face with lemon juice in the mistaken belief that, because lemon juice is usable as invisible ink, it would prevent his face from being recorded on surveillance cameras. [i]
Dunning and Kruger were not the first to observe this type of phenomena, however. In the late 1970’s the industrial psychologist Herbert Meyer conducted a now classic study of engineers based at the General Electric Company. Meyer asked the engineers to self-assess their own performance relative to other engineers with similar jobs and salary levels.
Question
Rate your performance compared to others you know with similar job description & pay grade
The average engineer rated their performance to be at the 78th percentile (In others words, better than 77% of their peers). Only 2 of the 92 engineers placed themselves below the 50th percentile
Since that study was conducted, other studies have found similar results in a diverse range of performance areas from driving skills to medical practice. These findings led Walfish et al. [ii] to remark “it is common to think of ourselves as somewhat remarkable compared to others.”
You would like to think, wouldn’t you, that given the nature of what we do as therapists, including promoting fairness and balance in our clients’ self-appraisal of themselves, that we would be immune to this self-assessment bias in our appraisal of our own professional selves?
You might like to think that, but sadly, it appears that belief would be mistaken. It would seem that we are as prone to an over-inflated sense of our ability as anyone else. In fact, it may just be that the engineers edge us in the modesty stakes. Walfish and colleagues conducted a similar study of 129 mental health professionals that comprised psychiatrists (9.3%), psychologists (26.4%), professional counsellors (21.7%), clinical social workers (28.7%), and marriage and family therapists (14%). They asked two questions of them:
Question 1
Compared to other mental health professionals within your field (with similar credentials), how would you rate your overall clinical skills and performance in terms of a percentile (out of 0–100% : e.g., 25% = below average, 50% = average, 75% = above average)?
In response to question 1, on average respondents rated themselves at the 80th percentile. In other words, better than 79% of their peers. Twenty-five percent rated themselves at or above the 90th percentile. The most common rating was the 75th percentile. Just 8.4% percent rated themselves below the 75th percentile. None rated themselves below the 50th percentile, or, in other words, below average.
Question 2
What percentage (0–100%) of your clients get better (i.e., experience significant symptom reduction during treatment? What percentage stays the same? What percentage gets worse?)
On average, these practitioners believed that 77% of their clients improved significantly as a result of therapy. Fifty-eight percent believed that 80% or more of their clients improved, and just over one in five (21%) that 90% or more of their clients showed improvement. Almost half of practitioners (47.7%) believed that none of their clients deteriorated.
Bucking this general trend, one practitioner indicated that only 25% of his clients improved, with 50% remaining the same and the remainder deteriorating. I’ve always been a little statistically challenged, but even I can work out that squeezing 25% of practitioners into the top 10% of performance is going to make things feel a little crowded.
As for their estimated improvement rates, I suspect that the practitioner who indicated that more of his clients showed no change than improved was likely to be the only one who had read the relevant outcome research.
In forming our beliefs about our impact as practitioners, is it possible that what we believe is determined more by what we would like to believe about ourselves than by supporting evidence? The study above would seem to point in that direction.
Whether or not we over-inflate our impact with clients, I would argue that neither we nor our clients can lose if we attempt some objective measure of that impact. If we do over-estimate our impact, then we have something to strive for. And if the evidence points in the other direction, then I will not begrudge you a degree of evidence based self-congratulation.
Here’s a possible starting point for you:
1. Compared to other mental health professionals within your field (with similar credentials), how would you rate your overall clinical skills and performance in terms of a percentile (0–100%) e.g:
25% = below average
50% = average
75% = above average
2. What percentage (0–100%) of your clients get better (i.e., experience significant symptom reduction) during treatment? What percentage stays the same? What percentage gets worse?
References
[i] https://en.m.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect accessed 11 October 2016 [ii] Walfish S, McAlister B, O’Donnell P, Lambert MJ. 2012. An investigation of self-assessment bias in mental health providers. Psychological Reports · April 2012
I am starting to feel a bit depressed. LOL:> Can we have a positive blog next(!)
Oh dear, some of this does feel a bit depressing doesn’t it? Worrying also that this is probably news for many of our colleagues, however, unwelcome though it may feel.
I’ll do my best to bring some more positive news shortly. Feel free to suggest what you’d like to hear!
Hi Barry, I would just love to have some method of proving how good I am to myself! I have used CORE for many years, just the outcome measures, but really want a method of analysing the information to show me how effective I am. So, if a client show up with 1.24, and six sessions later they are 0.95 I can see some improvement (and so say they) but how do I compare this with your band of supershrinks? It’s been frustrating that there doesn’t seem to be any standardised way of actually comparing clients.
Next, OK, I would have put myself in the top 10% of therapists too, but I’m including the therapists I trained with, and those who I know work with others and have had no training at all. I DO hope many of the ones I trained with never actually practised! Once again this goes back to my experience of other’s giving therapy as only being skills work when training, and Gloria.
Hi David and thanks for your post
Well, if you’ve collected the outcome measure data from CORE then I’d say you’ve probably done the hardest part! Using the pre and post scores that you use to illustrate your point, I can see that this client would have achieved both clinical and reliable change. Nice work!
There are probably a couple of main ways of benchmarking this kind of impact. The first is to work our what proportions of your clients fall into the various categories of change e.g. clinical only, reliable only, clinical and reliable, no change. Unless we’re familiar and comfortable with those definitions it can all feel a bit overwhelming, though it needn’t be.
The second way is to calculate your effect size, based on the overall change for a group of clients. Again, not rocket science, but you need to know how. This only gives you a global figure, however, and disguises a range across clients, some of whom we might have done better with.
In the coming weeks I’m going to be starting a blog series of measurement, from starting to use measures with clients, through working out what the numbers mean, to simple benchmarking of your figures to get a sense of relative performance. Hopefully then you’ll be able to place yourself in the top 10% with confidence and the data to back it up!
Watch this space won’t you, there will be more coming. Thanks again – Barry