On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry
Publisher: In-Sight Publishing
Publisher Founding: March 1, 2014
Web Domain: http://www.in-sightpublishing.com
Location: Fort Langley, Township of Langley, British Columbia, Canada
Journal: In-Sight: Independent Interview-Based Journal
Journal Founding: August 2, 2012
Frequency: Three (3) Times Per Year
Review Status: Non-Peer-Reviewed
Access: Electronic/Digital & Open Access
Fees: None (Free)
Volume Numbering: 12
Issue Numbering: 3
Section: E
Theme Type: Idea
Theme Premise: “Outliers and Outsiders”
Theme Part: 31
Formal Sub-Theme: High-Range Test Construction
Individual Publication Date: August 1, 2024
Issue Publication Date: September 1, 2024
Author(s): Scott Douglas Jacobsen
Word Count: 1,052
Image Credits: None.
International Standard Serial Number (ISSN): 2369-6885
*Please see the footnotes, bibliography, and citations, after the publication.*
Abstract
Matthew Scillitani, member of the Glia Society and Giga Society, is a software engineer living in Cary, North Carolina. He is of Italian and British lineage, and is fluent in English and Dutch (reading and writing). He holds a B.S. in Computer Science and a B.A. in Psychology. You may contact him via e-mail at mattscil@gmail.com. Scillitani discusses: Divine Psychometry; “a journey to Paradise”; the origin of the title of the test; number of test items; verbal at 20 questions; language a barrier with English as a basis; three easiest test items; three hardest test items ; roadblocks test-takers tend to make; a good numerical test; a good verbal test; a good spatial test; help with this test; sample size of this test; tests and test constructors; and learned from making these tests.
Keywords: based on Dante’s Divine Comedy, critical feedback on test items, enjoyable or boring problems, feedback from non-English speakers, good numerical test items, good spatial test items, good verbal test items, impact of pacing on test performance, interest in verbal problems, learning from making tests.
On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry
Scott Douglas Jacobsen: You did make a test: Divine Psychometry. What was the general feedback on it?
Matthew Scillitani: Mixed. A few people who tried it told me that it was interesting, enjoyable, or even beautiful. I’ve also had people tell me that it was a little boring or that the problems weren’t as satisfying as those found in Paul Cooijmans’ tests. Since Paul’s the greatest I.Q. test and puzzle constructor of all time, that’s understandable, though.
Jacobsen: It warns, “While this test constitutes a journey to Paradise, it does include an ever so short stay in Purgatory, sorry.” Who wrote that?
Scillitani: Paul did. There’s a good reason why, but you’d have to see the test yourself to know.
Jacobsen: What is the origin of the title of the test?
Scillitani: I had just finished reading Dante’s Divine Comedy, and thought it would be fun to make an I.Q. test loosely based around that. Hence, Divine Psychometry.
Jacobsen: The number of test items is 32 with 20 verbal, 8 numerical-symbolic, and 4 visuo-spatial. Why this ratio?
Scillitani: At the time, I enjoyed verbal problems the most, both creating and solving them. So, I heavily focused on verbal items only because it was more interesting for me as the test constructor. Nowadays, I like numerical and spatial problems about the same as verbal.
Jacobsen: Does verbal at 20 questions bias the structure of the cognitive profile tested?
Scillitani: Probably. Someone who does better at solving verbal problems than other types may perform better. Sometimes that doesn’t happen, though. For example, Psychometric Qrosswords is a purely verbal-looking test but its numerical g loading is slightly higher than its verbal g loading. This seems counter-intuitive, but it shows that what a test appears to measure isn’t necessarily what it’s exactly measuring.
Jacobsen: Is language a barrier with English as a basis? If so, how so?
Scillitani: Non-English speakers should still be able to do pretty well on my test. Maybe they’ll lose a single raw point at worst, but having the test in English shouldn’t act as a barrier. I’ve taken several verbal tests in foreign languages and my scores on them are always around what I’d score on an English test.
Jacobsen: What do you consider your three easiest test items in it?
Scillitani: To keep the test secure, I can’t say. However, what I’d think were the easiest items may not actually be the easiest items. What we intuitively think is easy or hard isn’t always so.
Jacobsen: What do you consider your three hardest test items in it?
Scillitani: Same as above. There are definitely some items that I made extremely hard but I can’t say which, and as far as I know they ended up being much easier than I expected.
Jacobsen: What are roadblocks test-takers tend to make in terms of thought processes and assumptions around time commitments on these tests?
Scillitani: Most test-takers tend to perform much better than they think they can if only they pace themselves. That is to say, work on the test a little at a time, take breaks, and put the test aside for a while before they submit answers. Cramming a test is going to exhaust the mind just like sprinting exhausts the body. If we try to solve an extremely hard test in a matter of a few hours, it rarely ends well, and we’ll probably be left with a headache and a low score.
Aside from pacing, waiting a while before submitting answers is good practice. Almost every Glian I’ve spoken to has said that they submitted a test only to find more answers right after submitting to the scorer. I’ve experienced that myself a few times too. It’s better to finish a test, wait at least a few days (or even a few weeks or months!), re-examine your answers, and then finally submit. You may find that a few of your answers change for the better after a break and review.
Jacobsen: What goes into making a good numerical test question?
Scillitani: Aside from the use of numbers, I’d say that good numerical test items (1) don’t rely on any advanced mathematics knowledge to solve and (2) can’t be brute-forced. If any problem, numerical or not, can be solved by guessing over and over or by writing a script, it’s a bad problem.
Jacobsen: What goes into making a good verbal test question?
Scillitani: Ensuring the problem requires little to no knowledge to solve. Good verbal analogies and word association problems should follow strict logic, leading the smart candidate to an “aha!” moment, and not need the candidate to spend hours hunting down obscure knowledge.
Jacobsen: What goes into making a good spatial test question?
Scillitani: Good spatial test questions can involve (1) rotation, like showing different sides of an object and having the candidate draw the missing view, (2) showing the candidate images and asking for the common association between them, or (3) having the candidate select or even draw the next shape in a series or analogy.
Jacobsen: Did anyone help with this test? If so, how?
Scillitani: Paul Cooijmans helped a lot by giving critical feedback on every item to improve their quality and formatting the test to reduce file size and make it cleaner overall.
Jacobsen: What is the current sample size of this test? What is the highest score so far?
Scillitani: Fewer than 16 people have taken it, though I don’t know the exact number. Probably 12-15 as of this interview. The highest score achieved so far is 32/32 for an I.Q. of 190.
Jacobsen: What tests and test constructors have you considered good?
Scillitani: Paul Cooijmans, Mahir Wu, Ron Hoeflin, and Kevin Langdon are the best test constructors in my opinion. My favorite tests are Dicing with death, Narcissus’ Last Stand, and The Smell Test by Paul Cooijmans, as well as N-World by Mahir Wu.
Jacobsen: What have you learned from making these tests?
Scillitani: I learned that making good tests takes time, problems shouldn’t rely on knowledge to solve (or if they do, the relevant info should be easy to find), and to make a truly good puzzle is more art than science.
Jacobsen: Thank you for the opportunity and your time, Matthew.
Scillitani: Always a pleasure!
Bibliography
None
Footnotes
None
Citations
American Medical Association (AMA 11th Edition): Jacobsen S. On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry. August 2024; 12(3). http://www.in-sightpublishing.com/high-range-3
American Psychological Association (APA 7th Edition): Jacobsen, S. (2024, August 1). On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry. In-Sight Publishing. 12(3).
Brazilian National Standards (ABNT): JACOBSEN, S. On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry. In-Sight: Independent Interview-Based Journal, Fort Langley, v. 12, n. 3, 2024.
Chicago/Turabian, Author-Date (17th Edition): Jacobsen, Scott. 2024. “On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry.” In-Sight: Independent Interview-Based Journal 12, no. 3 (Summer). http://www.in-sightpublishing.com/high-range-3.
Chicago/Turabian, Notes & Bibliography (17th Edition): Jacobsen, S “On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry.” In-Sight: Independent Interview-Based Journal 12, no. 3 (August 2024).http://www.in-sightpublishing.com/high-range-3.
Harvard: Jacobsen, S. (2024) ‘On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry’, In-Sight: Independent Interview-Based Journal, 12(3). <http://www.in-sightpublishing.com/high-range-3>.
Harvard (Australian): Jacobsen, S 2024, ‘On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry’, In-Sight: Independent Interview-Based Journal, vol. 12, no. 3, <http://www.in-sightpublishing.com/high-range-3>.
Modern Language Association (MLA, 9th Edition): Jacobsen, Scott. “On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry.” In-Sight: Independent Interview-Based Journal, vo.12, no. 3, 2024, http://www.in-sightpublishing.com/high-range-3.
Vancouver/ICMJE: Scott J. On High-Range Test Construction 3: Matthew Scillitani on Divine Psychometry [Internet]. 2024 Aug; 12(3). Available from: http://www.in-sightpublishing.com/high-range-3.
License & Copyright
In-Sight Publishing by Scott Douglas Jacobsen is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. ©Scott Douglas Jacobsen and In-Sight Publishing 2012-Present. Unauthorized use or duplication of material without express permission from Scott Douglas Jacobsen strictly prohibited, excerpts and links must use full credit to Scott Douglas Jacobsen and In-Sight Publishing with direction to the original content.
In-Sight Publishing 