Anyone using Assessment or Development Centres (ADCs) for promotions, selection into new roles or benchmarking employee capability, will be well aware of the challenges of creating a realistic, meaningful and valid (i.e. accurate) process. Providing the right level of challenge for participants, while giving regular opportunities for feedback and learning, is a difficult balance to strike.
With this challenge in mind, we have been working with our clients to design progressive, innovative, fair, flexible and accurate assessment and development centres to support talent management. We have designed centres with our clients that deliberately vary the context, the situation and the timing of meetings in order to achieve more realistic and varied measures of capability. For example, we have used short meetings, standing meetings, meetings that are frequently interrupted and difficult teleconference calls. And by doing so, we have been achieving stronger outcomes, increasing the realism of situations and improving the quality of observations and the usefulness of the feedback. None of these alternative designs reduced the validity of our observations – they only enhanced the flexibility and improved the experience for participants.
So it wasn’t a great surprise when, at this year’s BPS Occupational Psychology conference, a number of speakers questioned whether traditionally structured assessment centres were likely to be inconsistent or even inaccurate in what they claim to be measuring. These questions emerged based on recent studies into the effectiveness of assessment centres. Using a new and well regarded approach to analysing all of the possible effects that may contribute to variance in assessor ratings in a standard assessment centre, the studies indicate that less than 2% of the variance in ratings from that assessment centre could attributable to the actual measurement of competencies (see Dewberry, 2017 (in press)). In other words, the final ratings that came from the assessment centre bore little, if any, relationship to the behaviours that they claim to measure.
Why is this? It’s difficult to say exactly, but it would appear that assessment centres are actually better at measuring behaviour within a specific context (i.e. how somebody handles a particular one-to-one meeting or group situation) than trying to do so across a range of different contexts. This perhaps reflects the view that behaviours – and leadership behaviours in particular – are far more ‘situational’ than previously thought by most practitioners.
Our approach has been further backed up by the research of Filip Lievens (2016), which questioned the traditional format and construct of assessment centres. This research proposed that an assessment centre could break with the traditional format of two role plays, each lasting for 30 minutes, and instead run eighteen interactive role plays, each lasting three minutes. For an experimental approach, the results demonstrated good validity and identified, in particular, those most extravert and agreeable, which is ideal if the assessment centre is for a people or sales role. Other methods tested open-ended video responses, rather than written responses, and webinar facilities – in the same way that many employees work on a day-today basis.
All of this raises some critical questions for any organisations using standardised, off the shelf exercises in their assessment and development centre processes. It is impossible to ignore these findings. Instead, we need to embrace it as an opportunity to further question the approach and look at ways to reflect the constant changes in work environments.
Most importantly perhaps, as we make changes to assessment and development methods, we have to continue to improve fairness, reduce the bias that is so inherent in so many processes, and give everyone an equitable starting point, whether they are being assessed for a role or developed for a future position. We have a very interesting time ahead.
To read more of Stuart Duff's blogs on leadership and and other related business psychology topics, click here.