1241: Understanding Psychometric Limitations
Author(s): Scott Douglas Jacobsen
Publication (Outlet/Website): Medium (Personal)
Publication Date (yyyy/mm/dd): 2025/05/27
I stand by the mainstream standards of psychometrics: APA, BPS, and CPA standards. The Standards for Educational and Psychological Testing(co‑published by APA, AERA, and NCME) outlines the stringent test construction and validation requirements. Tests not adhering to these standards are not considered psychometrically credible instruments. First, individuals must not have conflicts of interest in creating, administering, or scoring psychometric tests. Otherwise, the results become invalid by professional standards, whether friends with the scorer, the hirer of the scorer or working for the company in which the score has been claimed validated. Any such score cannot be considered valid when a clear conflict of interest is present.
High-range tests began in the late 20th century. They attempt to measure intelligence beyond the ceilings of professionally accepted instruments. Their development occurred largely outside institutional psychology, which has resulted in widespread methodological flaws and a lack of academic legitimacy. The high-range testing community (e.g., Prometheus Society, Mega Society, etc.) features self-created tests often designed by individuals without formal training in psychometrics.
Many of these people work with one another, for one another, found and join societies, then invite one another’s participation, and then take their friends’ tests and garner a listing on the rankings or directories. While some independent efforts reflect intellectual seriousness and commitment, they remain outside the scope of recognized professional standards. Their creators are intellectually serious. However, most lack doctorates in psychometrics or clinical psychology, licenses from relevant bodies, and their tests are not peer-reviewed or professionally validated. The overlapping roles of test creators, scorers, and participants within small, insular communities raise valid concerns about objectivity and the independence of results.
According to Mensa International’s former international supervisory psychometrician, Dr. Kristóf Kovács, even scores above ~145 on SD 15 are unreliable after that point. Not just him, according to Dr. Abbie Salny and other psychometricians, most mainstream IQ tests (e.g., WAIS-IV, SB5) have diminishing reliability past 130–145 SD15, even under proctored conditions. Beyond 145 on SD 15, scores become increasingly speculative due to standard error of measurement and ceiling effects.
Fundamentally, a claim of an IQ score out of these would be illegitimate, particularly as the higher rarities of IQ are claimed. This would mean above 145 on SD 15 if these were mainstream and proctored by certified professionals. Even there, that is based on the best professional tools. This is to say, self-administered, untimed, non-proctored IQ tests developed without psychometric oversight are not considered valid, reliable psychological instruments and are often non-standardized. That is, they violate standardization, reliability, and validity criteria. They are often unnormed and lack clinical controls.
To be considered a valid psychological instrument, a test must meet three core criteria: validity (it measures what it claims to measure), reliability (it produces consistent results over time and contexts), and standardization (it has been normed on a sufficiently large and representative sample under controlled conditions). Tests that do not meet these criteria — especially those created outside professional oversight — cannot produce scores that meaningfully reflect cognitive ability in the same manner.
Some high-range tests employ inflated standard deviations to generate higher numerical scores. While this may appear impressive to lay readers, it is mathematically misleading and lacks empirical justification within a scientifically valid norming structure. Disseminating inflated or unsupported IQ scores can mislead individuals about their cognitive abilities, reinforce elitist attitudes, and erode public trust in psychological science. Ethical responsibility demands accuracy, humility, and transparency in any discussion of intelligence testing. Such claims are not recognized as valid by contemporary professional standards and should not be equated with WAIS‑V, SB5, or Raven’s APM scores — each of which is psychometrically credible only when administered under professionally controlled conditions. Therefore, any claim to this or that IQ score falls into the same illegitimate category by those mainstream, well-established standards.
Last updated May 3, 2025. These terms govern all In Sight Publishing content—past, present, and future—and supersede any prior notices. In Sight Publishing by Scott Douglas Jacobsen is licensed under a Creative Commons BY‑NC‑ND 4.0; © In Sight Publishing by Scott Douglas Jacobsen 2012–Present. All trademarks, performances, databases & branding are owned by their rights holders; no use without permission. Unauthorized copying, modification, framing or public communication is prohibited. External links are not endorsed. Cookies & tracking require consent, and data processing complies with PIPEDA & GDPR; no data from children < 13 (COPPA). Content meets WCAG 2.1 AA under the Accessible Canada Act & is preserved in open archival formats with backups. Excerpts & links require full credit & hyperlink; limited quoting under fair-dealing & fair-use. All content is informational; no liability for errors or omissions: Feedback welcome, and verified errors corrected promptly. For permissions or DMCA notices, email: scott.jacobsen2025@gmail.com. Site use is governed by BC laws; content is “as‑is,” liability limited, users indemnify us; moral, performers’ & database sui generis rights reserved.
In-Sight Publishing 