Ask A Genius 1020: Multimodality and Many Senses
Author(s): Rick Rosner and Scott Douglas Jacobsen
Publication (Outlet/Website): Ask A Genius
Publication Date (yyyy/mm/dd): 2024/07/25
55Rick Rosner, American Comedy Writer, www.rickrosner.org
Scott Douglas Jacobsen, Independent Journalist, www.in-sightpublishing.com
Rick Rosner: I think Coursera and AI Trends mentioned modalities, and they were using a term we’ve been using for longer, multimodality. They were referencing text, image, YouTube, and other coherent forms of media, but I do not mean that. I am referring to dragging sensory information from the world. Whatever sense it is, it interacts with the world and creates an impression.
It is the raw percept, not the concept. We need to clarify what we are discussing regarding AI and what AI would need to do to become conscious. We have determined that one of the significant steps toward achieving conscious AI is multimodality. Other people are discussing multimodality, and you are annoyed by this report on the big developments in AI this year and how they used multimodality.
I think it is incorrect when they refer to modality as a sensory modality. By modality, I mean raw sense, and there are different senses. If it is a term of art in the field of AI, it seems like it. If they are talking like that, I feel like we saw it somewhere and then started using it.
Did we see someone else use it and decide to adopt it for ourselves? I remember using it and then you saying, as you say, Scott, you’re saying stuff like that.So, either we came up with the term ourselves, and someone else did too, or you or I saw it somewhere and understood it in a way that if AI is using it in the same way in that field.
Scott Douglas Jacobsen: They are taking their meaning but with different representational modes. I am referring to different modes of sensory input. You could say different modes of communication, but that is more about output. In terms of input from the world, it is the multimodality as sensory.
Rosner: This article suggests that multimodality will become more common over the next year. You will have some kind of AI engine, or what do you call it, an AI? Just an AI. I think the term artificial intelligence is a misnomer to begin with, but you have a graphic AI that draws pictures for you, and then you have an LLM AI that writes words for you. I suppose you just call it AI. A multimodal AI, according to this article, can absorb information.
Pre-processed information in more than one way can be fed chunks of text, videos, or still images. That is the example the article used. I think consciousness is basically like an information processor or an awareness engine. I believe the sensory input helps breathe life into it, but it is not the awareness itself. When I think of multimodality, I think of different channels of input that then get fed into and create awareness.
We have sensory channels. We have the five senses plus some partial senses like proprioception, which is your internal feelings from your limbs that let you determine how your body is oriented in space, and probably a few other partial senses. But in addition to the senses, we have the analytics that, all along the line from when the images hit your optic nerve to when they enter consciousness, process the signals as they pass from your eye into your brain.
Advertisements
blob:https://insightjournalonline.wordpress.com/50b023d2-8aa0-44f5-975f-24d93429f675
REPORT THIS AD
Cleaning up the signals and interpreting them for you involves looking for straight lines, shading that helps you determine position in space, and perspective. You could argue that all those analytic subroutines or mini-engines are modalities themselves.
But there are a significant number of them attached to each sense and also modalities that operate off of your combined senses. They operate in whatever way that they can be productive and feasible. You could say recall is a modality except it’s likely reasonably a bunch of modalities that come together to help you remember things.
Sometimes it’s easy to remember stuff; it just pops into your head. Then sometimes you have to work at it, such as when you are trying to remember the name of an actor. Sometimes it helps to remember what they have been in, and sometimes it helps to go through letters of the alphabet when you are desperate. For example, I have trouble recalling names that start with B and M. For some reason, the B’s and M’s are linked weirdly and not productively in my mind. I used to have trouble remembering Julianne Moore, who has an M name. The work you have to do to recall stuff is probably a set of modalities. It’s probably like that for all of our modalities.
I’m talking about an AI that can receive inputs that are either visual or verbal isn’t doing justice to the idea of modality, right?
Jacobsen: So, you have each sense. Call those a modality. Those modalities integrate together efficiently, and they feed into what becomes awareness. So it’s almost like when we talk about awareness and consciousness, those terms aren’t well-defined. I agree with Cooijmans. Awareness is just easier to spell than consciousness. There is no other distinction.
Rosner: That’s funny. People don’t have well-defined ideas of what consciousness and awareness are. That certainly extends to modality. We’ll probably have a sloppy definition into the future. AI itself annoys you because the term denotes many different things that are all wildly different. When you’re using voice commands on your car radio, you could call it AI, but that’s not the same thing as a large language model. You could argue that we don’t even have AI yet. We don’t have, what’s the term for AI with actual general intelligence? Deepmind, AGI.
Jacobsen: At the same time, I don’t like the term artificial intelligence because we call it artificial when we mean engineered because we make it. But I don’t see any distinction because if you’re evolving a system, it’s in the natural world. If you’re consciously engineering a system in the natural world, it’s just a distinction between engineered and evolved, but both are natural intelligence. So I would say that modality and every other term associated with AI is going to suffer this problem for a long time. People will use these terms sloppily and with wildly different meanings.
Rosner: And maybe when there’s more mathematics involved with AI, it’ll get cleaned up a little bit. But do you think the terminology and the definitions will ever get cleaned up? Like when you talk about an electron or gravitation, p eople are pretty much talking about the same things, and those can be described mathematically.
And I don’t see the terms in AI being able to be described with that precision in the near future. Though consciousness has just as much of a mathematical underpinning, I think, as the elements of physics. It’s just that we’re early on, and consciousness is unwieldy. You can describe an electron with just a few equations. Consciousness is different.
We’ll eventually be able to talk about its mathematical characteristics and how different consciousnesses differ mathematically. But that’s not soon. The mathematics is more complicated, right? I’ll just conclude that I think the most important thing is, I don’t care if no one else thinks about this because I believe they’re just using the wrong definitions. They’re not thinking about it. They’re just taking what’s been given to them in their training. There are more people trying to sell AI than who are doing deep thinking about AI.
Jacobsen: Yes, I just want to make this point quickly: artificial intelligence and human intelligence are both natural. Artificial intelligence is a misnomer because it’s engineered intelligence, not evolved intelligence. This conceptually removes the barriers to thinking about creating human-style intelligence in a different substrate. It removes the magic because you’re engineering it, not evolving it.
Another thing, modality is a more precise way of saying sense, and that would have to do purely with input. I agree with Paul Cooijmans’s point that consciousness and awareness are only different insofar as awareness is easier to spell. You can probably think of the central hub of conscious life and its surrounding substructure of actual processes apart from modalities and the multimodality coming in as something like an awareness engine. Because there is a generative capacity to the mind and the term awareness engine is appropriate. So that’s all I wanted to say. What I was going to say is that the abuse of the ideas of AI and the term AI, and every other term associated with AI, is problematic because AI is big business now and will be bigger business later. There are more people involved in selling AI than in conceptualizing AI. There are more people bullshitting about AI than there are people trying to remove the BS from AI and put it on a firm theoretical footing.
Advertisements
https://c0.pubmine.com/sf/0.0.7/html/safeframe.html
REPORT THIS AD
I’m sure there are people who work in AI who are as annoyed as you are by the loose use of terminology and misleading terminology. But you don’t hear as much from those people; they’re busy doing technology rather than selling AI systems. I see a bunch of come-ons on Twitter from people who want to teach you how to talk to AI to get it to deliver more of what you want. So yes, it’s a big booming, bullshitty field.
Jacobsen: OK, let’s end on that fun note.
License & Copyright
In-Sight Publishing by Scott Douglas Jacobsen is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. ©Scott Douglas Jacobsen and In-Sight Publishing 2012-Present. Unauthorized use or duplication of material without express permission from Scott Douglas Jacobsen strictly prohibited, excerpts and links must use full credit to Scott Douglas Jacobsen and In-Sight Publishing with direction to the original content.
In-Sight Publishing 