{"id":34109,"date":"2018-01-16T08:07:00","date_gmt":"2018-01-16T08:07:00","guid":{"rendered":"https:\/\/hoozoocomm.co.za\/cognadev\/?p=34109"},"modified":"2026-06-29T09:54:17","modified_gmt":"2026-06-29T09:54:17","slug":"the-achilles-heel-of-psychometrics","status":"publish","type":"post","link":"https:\/\/hoozoocomm.co.za\/cognadev\/2018\/01\/16\/the-achilles-heel-of-psychometrics\/","title":{"rendered":"The Achilles\u2019 Heel of Psychometrics"},"content":{"rendered":"

[vc_row][vc_column][vc_empty_space height=”20px”][vc_custom_heading text=”The Achilles\u2019 Heel of Psychometrics” font_container=”tag:h1|font_size:35px|text_align:left|color:%23049EB3″ google_fonts=”font_family:Arvo%3Aregular%2Citalic%2C700%2C700italic|font_style:700%20bold%20regular%3A700%3Anormal” css=””][vc_empty_space height=”10px”][vc_custom_heading text=”By Paul Barrett” font_container=”tag:p|font_size:20px|text_align:left|color:%23000000″ google_fonts=”font_family:Lato%3A100%2C100italic%2C300%2C300italic%2Cregular%2Citalic%2C700%2C700italic%2C900%2C900italic|font_style:700%20bold%20regular%3A700%3Anormal” css=””][\/vc_column][\/vc_row][vc_row][vc_column][vc_empty_space height=”40px”][vc_column_text css=””]<\/p>\n

\u201cAn Achilles\u2019 heel is a weakness in spite of overall strength, which can lead to downfall.\u201d\u00a0Wikipedia<\/a>.<\/h4>\n

Paradoxically, the Achilles\u2019 Heel of Psychometrics is that modern and classical test theory assume that an attribute such as Conscientiousness varies as a quantity. That is, it varies in the same way the temperature gauge on your car varies, the clock on your mobile phone, the digits on the airport baggage weight machine.<\/p>\n

But there never has been any empirical evidence that a personality, ability, motivation, value, or indeed any psychological attribute varies as a quantity.<\/p>\n

Psychologists, test publisher \u2018psychometricians, those who train others in BPS Level A and B test use, professional bodies like the ITC, EFPA, Veritas, and APA who issue \u2018best practice guidelines\u2019 or offer assessment accreditation, never mention this one cold fact to those they address, and what inevitably follows from that fact.<\/p>\n

But, you ask, does it matter to HR or indeed any psychometric test-user since psychometric test theory and its products have proven to be so useful in practice that such a concern is really only of interest to a few navel-gazing academics?<\/p>\n

Well, as I have explained in a detailed paper published recently (open-access),\u00a0The EFPA Test-Review Model: When Good Intentions Meet a Methodological Thought Disorder<\/a>, the purposeful ignorance of test publisher psychometricians, professional society accreditation agencies have left HR and other users exposed to legal challenge where the\u00a0precision<\/em>\u00a0of a test score is of substantive judicial interest.<\/p>\n

But, I hear you say, we use confidence intervals, standard errors of measurement, alpha reliability, factor analysis, IRT, and our test publisher R&D experts all follow best-practice psychometric guidelines.<\/p>\n

And what do all these techniques rely upon, including the statistical methods used to calculate all those parameters? Yes, you guessed it; that the attribute in question (e.g. \u201cAbstract Reasoning\u201d) varies as a quantity like length, mass, or electrical current. As a famous beer advert in NZ says: \u201cYeah Right\u201d!<\/p>\n

In practice, nobody except psychometricians interpret test scores as though they were quantitative\u00a0measures<\/em>\u00a0of length or volume. And therein lies the stinger, if a test score is used as a cut-score, or in some other way acts as a screening device, then the legal status of its precision might now be of potential legal interest.<\/p>\n

Of course, there are many ways of providing a robust empirical evidence base for the reliability and validation of the use of particular scores, but, to sustain legal challenge of the kind I am setting out in my article, these will not use psychometric methodology or indeed anything that invokes hypothetical true-score theory.<\/p>\n

For those who rely upon their test publisher R&D experts to defend the use of their particular test scores ask them this simple question, and listen very carefully to what comes back:<\/p>\n

\u201cShow me the empirical evidence that this measure of attribute X\u00a0(say \u201ctrait\u201d emotional intelligence)<\/em>\u00a0varies as a quantity?\u201d<\/p>\n

When they have finished replying, think how that kind of response will now look in a court which requires\u00a0empirical-evidence-based<\/em>\u00a0statements from its expert-witnesses rather than personal opinions, hand-waving, and untested assumptions.<\/p>\n

The reader of this blog might wonder why I\u2019m so disparaging of those who have portrayed psychometric methodology for so long as \u2018best practice\u2019. The reason is that for 20 or more years, psychometricians and those professional organizations issuing guidelines for others to follow, have known about the substantive issues published by experts in measurement, but have studiously kept that information from users.<\/p>\n

Just take a skim of my article at what has been published\/said over the years by many measurement experts, but carefully hidden from you by your local \u2018expert\u2019 psychometrician and test publisher.<\/p>\n

You can also acquaint yourself with the precedence already set within another area of psychological \u2018assessment\u2019 that will now form the basis of the new legal challenge that may now await those needing to defend their use of psychometric test scores from aggrieved individuals\/groups. The phrase \u201cHouse of Cards\u201d springs to mind.<\/p>\n

 <\/p>\n

The Next Generation of Psychological Assessments<\/strong><\/h2>\n

But, on a more positive note, the world thankfully is moving on from 20th<\/sup>\u00a0Century psychometrics. The \u201cNext Generation\u201d of assessments no longer conform to any of these outdated guidelines or test-theory invocations\/mantras.<\/p>\n

Cognadev has been at the forefront of these new innovations for two decades, joined now by a host of other companies now creating and selling truly innovative assessments. Finally, innovation is taking hold big-time.<\/p>\n

My article describes some of these new innovations, the other organisations producing some of them, and lays out a new framework for constructing legally-sound evidence-bases for reliability and validation of any Next Generation and even earlier assessments. And yes, it is referred to as a framework for good reason, and not a\u00a0\u2018do this by the numbers\u2019\u00a0<\/em>cookbook of assumption-laden statistical test theory.<\/p>\n

When working within a non-quantitative science, robust evidence-base construction requires careful thought, innovation, the use of methodologies suited to the properties of data at hand, and an honest realism about the status of test-scores.[\/vc_column_text][vc_column_text css=””]<\/p>\n

<\/div>\n

!function(e,n,i,s){var d=”InfogramEmbeds”;var o=e.getElementsByTagName(n)[0];if(window[d]&&window[d].initialized)window[d].process&&window[d].process();else if(!e.getElementById(i)){var r=e.createElement(n);r.async=1,r.id=i,r.src=s,o.parentNode.insertBefore(r,o)}}(document,”script”,”infogram-async”,”https:\/\/e.infogram.com\/js\/dist\/embed-loader-min.js”);<\/p>\n

Meta-Analytic Validity Coefficients for Predictors of Job Performance<\/a>
\nInfogram<\/a><\/div>\n

[\/vc_column_text][vc_column_text css=””]<\/p>\n

\n

From these data, it is no longer clear whether Big Five personality attributes are even worth assessing via self-report, contextualised or otherwise, such is their explanatory inaccuracy. However, as shown in: Oh, I-S., Wang, G., & Mount, M.K. (2011).\u00a0Validity of observer ratings of the five-factor model of personality traits: A meta-analysis\u00a0<\/a>.\u00a0Journal of Applied Psychology<\/em>, 96, 4, 762-773, Big Five validities computed using single-rater observed ratings do exceed self-report questionnaire validities.<\/p>\n

As to GMA, its 1998 validity has almost halved given the more recent article from Sackett et al (2022). I have said more on this issue in a recent\u00a0Cognadev blog\u00a0<\/a>. And, an earlier article: Richardson, K., & Norgate, S.H. (2015).\u00a0Does IQ Really Predict Job Performance?\u00a0<\/a>Applied Developmental Science,<\/em>\u00a019, 3, 153-169.\u00a0(open-access)\u00a0had already questioned the accuracy of the earlier meta-analytic validity estimates in this area.<\/p>\n

Anyway, I’m sure controversy will remain as to the wisdom or otherwise of correcting correlations for attenuation due to a variety of factors, but the graph above does at least present the competing evidence to date.<\/p>\n<\/div>\n

[\/vc_column_text][\/vc_column][\/vc_row]<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"

[vc_row][vc_column][vc_empty_space height=”20px”][vc_custom_heading text=”The Achilles\u2019 Heel of Psychometrics” font_container=”tag:h1|font_size:35px|text_align:left|color:%23049EB3″ google_fonts=”font_family:Arvo%3Aregular%2Citalic%2C700%2C700italic|font_style:700%20bold%20regular%3A700%3Anormal” css=””][vc_empty_space height=”10px”][vc_custom_heading text=”By Paul Barrett” font_container=”tag:p|font_size:20px|text_align:left|color:%23000000″ google_fonts=”font_family:Lato%3A100%2C100italic%2C300%2C300italic%2Cregular%2Citalic%2C700%2C700italic%2C900%2C900italic|font_style:700%20bold%20regular%3A700%3Anormal” css=””][\/vc_column][\/vc_row][vc_row][vc_column][vc_empty_space height=”40px”][vc_column_text css=””] \u201cAn<\/p>\n","protected":false},"author":1,"featured_media":34096,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[],"class_list":["post-34109","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-assessment-issues"],"_links":{"self":[{"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/posts\/34109","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/comments?post=34109"}],"version-history":[{"count":1,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/posts\/34109\/revisions"}],"predecessor-version":[{"id":34110,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/posts\/34109\/revisions\/34110"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/media\/34096"}],"wp:attachment":[{"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/media?parent=34109"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/categories?post=34109"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hoozoocomm.co.za\/cognadev\/wp-json\/wp\/v2\/tags?post=34109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}