On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ

 

 

 

 

 

 

 

Publisher: In-Sight Publishing

Publisher Founding: March 1, 2014

Web Domain: http://www.in-sightpublishing.com

Location: Fort Langley, Township of Langley, British Columbia, Canada

Journal: In-Sight: Independent Interview-Based Journal

Journal Founding: August 2, 2012

Frequency: Three (3) Times Per Year

Review Status: Non-Peer-Reviewed

Access: Electronic/Digital & Open Access

Fees: None (Free)

Volume Numbering: 13

Issue Numbering: 1

Section: E

Theme Type: Idea

Theme Premise: “Outliers and Outsiders”

Theme Part: 32

Formal Sub-Theme: High-Range Test Construction

Individual Publication Date: November 22, 2024

Issue Publication Date: January 1, 2025

Author(s): Scott Douglas Jacobsen

Word Count: 1,648

Image Credits: Photo by Ben Mullins on Unsplash.

International Standard Serial Number (ISSN): 2369-6885

*Updated November 23, 2024.*

*Please see the footnotes, bibliography, and citations, after the publication.*

Abstract

Marco Ripà gives some opening commentary on his involvement in the dynamic style high-range test. Roberto Enea talks about his work on DynamIQ, a dynamically generated spatial IQ test. Enea, inspired by Marco Ripà, aimed to create a test that minimizes the “training effect” and resists cheating, making it suitable for repeated use. Developing DynamIQ involved balancing design, coding, security, and fairness. The test generates questions dynamically with consistent difficulty across tests. Enea acknowledged challenges in norming tests due to low and self-selected high-IQ samples and emphasized the need for diverse populations. DynamIQ avoids linguistic bias and ensures privacy by anonymizing user data. Its credit system allows economical, flexible use over time.

Keywords: cheating prevention, dynamic IQ tests, high-IQ samples, linguistic bias, norming challenges, privacy protection, spatial reasoning.

On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ

Scott Douglas Jacobsen: As an opening question, what is your involvement in this dynamic style test?

Marco Ripà: Since I deleted all my data around early 2017, when I left the project, my memory of the test construction and normalization process is limited. Additionally, my memory isn’t solid. I do recall following Paul Cooijmans’s guidelines at the time, and if I’m not mistaken (we should verify this to be specific), I also communicated with him about the test norming to ensure everything was done correctly. The info I used to norm the test at the time also included the testee previous scores on reputable HRTs and supervised tests. Cooijmans and I talked about the z-scores method between September 3rd and September 4th 2016.I used the z-scores instead of the rank equations since our norming sample was too small for the latter method.

I remember choosing circles, triangles, and squares to create a culture-fair test with geometric properties. This decision was based on my initial idea to develop a spatial version of the ENNDT, which I had previously developed with Gaetano Morelli (see this paper). Regarding the colour selection—yellow, green, and white/blank—we opted for these colours to avoid underestimating the performance of colour-blind individuals.

Here is a summary of the entire story (please feel free to ask Roberto for more details):

In 2011, I envisioned developing a dynamic high-range test featuring thousands of unique software-generated items. Thus, this would ensure that any collection would share the same norm. Initially, we planned to use OEIS sequences characterized by unique properties. A few years later, with Gaetano, we developed the ENNDT tests, as described in the paper above. However, this test proved extremely challenging. Only highly skilled individuals achieved positive scores, making it impractical for screening purposes—even among the gifted populations.

Our next goal was to create a spatial, culture-fair IQ test for the high range that wouldn’t require participants to be exceptionally gifted. This was a more difficult challenge than the previous one. I enlisted another Italian Mensa member to help us. Roberto Enea joined the project, handling the programming and technical aspects, which I couldn’t manage myself. This collaboration involved over six months of work.

After internal testing, the tool was ready. We began beta testing. Then I normalized it using the standard techniques described by Paul Cooijmans. However, that was the last time I engaged in such work. I can’t recall the details. There were a lot of linear regression attempts and the corresponding R² values. I left the project in early 2017. 

Subsequently, I deleted all documentation related to the test, the item projects, and the data on testees’ performances, which only included initials, raw scores, and attempt numbers. So, I can’t provide more technical information than Roberto, who has been the sole owner of the spatial dynamic test platform since 2017.

Nonetheless, I’m pleased to receive credit for the concept I envisioned over a dozen years ago—a true dynamic IQ test generator developed by software that is resilient against cheating (at least in the pre-ChatGPT era).

Anyway, I created the items following my initial idea of a mathematical method to combine those three geometrical shapes and Roberto wrote the program and created the tool to make the whole thing happen. I remember that the idea was to divide every shape in corners and/or sides and merge different shapes together.

Now I have to go.

Jacobsen: When did this interest in test construction truly come forward for you? 

Roberto Enea: Hi Scott, thanks for this interview. Actually, I was mostly interested in solving tests rather than constructing them but I have found Marco’s idea about implementing a system to automatically generate IQ tests very interesting and challenging.

Jacobsen: What were the realizations about the tests, at the time, and the need to develop yours?

Enea: At the time we started working on DynamIQ there was nothing similar meaning no systems that were able to generate dynamically spatial IQ tests. We were conscious of being creating something completely new. 

Jacobsen: What was the origin and inspiration for the creation of the DynamIQ – the facts and the feelings? You have made some comments about it. Marco Ripà first mentioned this to me about 8 years ago with particular excitement about the ‘ambition’ behind this project.

Enea: This is more a question for Marco since he had the initial idea. The main idea was creating an IQ test that cannot be cheated and where the “training effect” has a lower impact so that you can take it several times without loosing accuracy. That is a great idea because it could be used to monitor the actual efficacy of brain training systems. The initial idea was making DynamIQ also something that could be used by professionals and academics but we never accomplished this final step.

Jacobsen: Any word of credit to others who helped in the development of this test?

Enea: Unfortunately, not. DynamIQ has been developed only by Marco and me

Jacobsen: How does design and coding play into the construction of DynamIQ?

Enea: I would say 50% designing and 50% coding. Design is not just about the test design but it includes other aspects like security, anti-cheating, data privacy, etc.

Jacobsen: What skills and considerations, in an overview, seem important for both the construction of test questions and making an effective schema for them?

Enea: The main challenge of creating a dynamic iq test is defining a sort of generation rule for test questions rather than defining the single test question, because you have to automatically generate tests whose difficulty should increase during the test (let’s say from the 1st question to the 25th) but it should also be almost the same for the same question across different tests (the question n. 13 should have almost the same difficulty across all the tests generated). This requires a deep understanding about how spatial tests work “behind the scenes”, meaning that you have to know what makes a spatial test more difficult than another one beyond the intuition of it.  

Jacobsen: How do we know with confidence listed norms are, in fact, reasonably accurate on many of these tests? 

Enea: In my opinion the answer is that we cannot be 100% sure. Most of the time the norms are computed by the authors and they are not verified by other peers. Unfortunately, until there isn’t a rigorous scientific validation of the test you cannot be sure about the accuracy declared.

Jacobsen: What are the most appropriate means by which to norm and re-norm a test when, in the high-range environment so far, the sample sizes tend to be low and self-selected, so attracting a limited supply and a tendency in a type of personality? Pragmatically speaking, for really good statistics, what is your ideal number of test-takers? You can’t say, “8,126,000,000.”

Enea: Rather than being the number of test takers the problem is the sample composition. For example, selecting people in the high range is not difficult. There are a lot of test takers coming from High IQ societies like Mensa who have an official assessment of their IQ. It is much more difficult selecting people in the low and middle range because in a lot of countries like Italy IQ assessment is not a common practice so most of the people are not aware of their IQ. For this reason, it is difficult to collect a sample that is homogeneous enough to actually represent the whole population.

Jacobsen: Is English-based bias a prominent problem throughout tests? Could this be limiting the global spread of possible test-takers of these tests rather than limiting them to particular language spheres?

Enea: About DynamIQ that is definitely not a limit since it is a spatial test. No language knowledge is required.

Jacobsen: How do you ensure protection of the “privacy and personal information” of test-takers? Why “share information about your use of our site with our social media, advertising and analytics partners”?

Enea: About this I would like to clarify that sharing information about website usage does not mean sharing results. The website collects anonymous information about the provenance of the visitors and other information that are usually useful for marketing but these are not related to the test itself. The test is anonymized also because we don’t collect any personal information of the user. We store the email used for the registration but there is no information stored that can connect the email to the real person.

Jacobsen: What is the purpose of the account and credit system of the DynamIQ test setup?

Enea: The credit system allows you to buy several tests executions in one shot so that the user can save some money. The credit never expires so you can take the test even several years after the purchase.

Jacobsen: With the advent of the internet, cheating on individual questions and on whole tests is a possibility and a reality on these high-range tests. How do we prevent such occurrences? Also, things like the law and ethics.

Enea: That is actually the main purpose of DynamIQ: avoiding cheating in IQ tests. The number of combinations you can have avoids that the user can somehow “memorize” the answers. Of course, in the age of AI, it might be possible to cheat using systems like ChatGPT but at least for now it seems that they are not smart enough to solve this kind of tests. There is a challenge in place called ARC prize designed by researchers at Google that is focused on creating AI model to solve spatial tests. The tests designed in the ARC prize are quite simple, not comparable to DynamIQ. Nevertheless, the best result so far is 42% accuracy meaning that the best model fails almost 60% of the times.

Jacobsen: Thank you for the opportunity and your time, Roberto. 

Enea: My pleasure.

Footnotes

None

Citations

American Medical Association (AMA 11th Edition): Jacobsen S. On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ. November 2024; 13(1). http://www.in-sightpublishing.com/high-range-26

American Psychological Association (APA 7th Edition): Jacobsen, S. (2024, November 22). ‘On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ’. In-Sight Publishing. 13(1).

Brazilian National Standards (ABNT): JACOBSEN, S. ‘On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ’. In-Sight: Independent Interview-Based Journal, Fort Langley, v. 13, n. 1, 2024.

Chicago/Turabian, Author-Date (17th Edition): Jacobsen, S. 2024. “On High-Range Test Construction 26: Roberto Enea, DynamIQ’.” In-Sight: Independent Interview-Based Journal 13, no. 1 (Winter). http://www.in-sightpublishing.com/high-range-26.

Chicago/Turabian, Notes & Bibliography (17th Edition): Jacobsen, S. “On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ.” In-Sight: Independent Interview-Based Journal 13, no. 1 (November 2024). http://www.in-sightpublishing.com/high-range-26.

Harvard: Jacobsen, S. (2024) ‘On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ’, In-Sight: Independent Interview-Based Journal, 13(1). http://www.in-sightpublishing.com/high-range-26.

Harvard (Australian): Jacobsen, S 2024, ‘On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ’, In-Sight: Independent Interview-Based Journal, vol. 13, no. 1, http://www.in-sightpublishing.com/high-range-26.

Modern Language Association (MLA, 9th Edition): Jacobsen, Scott. “On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ.” In-Sight: Independent Interview-Based Journal, vo.13, no. 1, 2024, http://www.in-sightpublishing.com/high-range-26.

Vancouver/ICMJE: Jacobsen S. On High-Range Test Construction 26: Marco Ripà and Roberto Enea, DynamIQ [Internet]. 2024 Nov; 13(1). Available from: http://www.in-sightpublishing.com/high-range-26.

License & Copyright

In-Sight Publishing by Scott Douglas Jacobsen is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. ©Scott Douglas Jacobsen and In-Sight Publishing 2012-Present. Unauthorized use or duplication of material without express permission from Scott Douglas Jacobsen strictly prohibited, excerpts and links must use full credit to Scott Douglas Jacobsen and In-Sight Publishing with direction to the original content.