By skyforbes I am essentially a complete layman in terms of machine learning. However, it is essentially impossible to exist today without constantly being bombared by news and discussions regarding AI. As a result, I have developed some questions which I do not know the answer to and am hoping you could ame with. Specifically, it's regarding the concept of AGI (which is not the best term due to its ubiquity) and ASI or an artificial intelligence that goes beyond the human understanding.
Here are my questions and my thoughts surrounding them:
Large Language Models ability to generalize past their training data:
My understanding has always been that LLMs are incapable of generalizing beyond their training data. However, I have recieved pushback for this in the past, with people claiming they absolutely can. To me, this seems impossible unless I have misunderstood something. My understanding is that:
LLMs can not generalize beyond their training data. You will not find an LLM can come up with novel ideas beyond the training data.
LLMs can make connections from the training data that were not previously known. For example, if it knows datapoint A, B and C, and these datapoints had no previously know connection between them, the model can make a connection between them, making it appear as it can generalize beyond its dataset. However, this connection already existed in the dataset, it just had not been made (or at least not documented) before.
Is this a correct interpretation of how LLMs work or is there more nuance here that I am missing?
Automated AI research:
This is seemingly the highest priority of every single major AI lab out there; if you can automate AI research then you can theoretically build more advanced AI models and systems much faster, outcompeting your competitors. However, I struggle with understanding how this would occur in practice? What is even the theoretical framework for how this would occur? I can think of feasible approaches: automated hypothesis creation and validation AN/OR automatic validation of a provided hypothesis. However, I struggle with seeing this as possible using current approaches for a few reasons:
To generate a hypothesis, you would likely need to use LLMs.
If my understanding from question 1 holds, then it will be impossible for the model to generate true novel hypothesis. It could make new connections and come up with some hypothesis that borrows from other research (which is arguably what you do in any research; understand the domain and then expand on current knowledge), but to what extent these hypothesis would be truly novel I doubt.
The obstacle in my view is the fact that (1) the model would not be able to theorize something truly new, therefor limiting how useful it could actually be in coming up with new hypothesis. What I'm imagining is its inability of coming up with something truly revolutionary or novel. For example, say LLMs had no prior knowledge about the transformer architecture; would it be able to come up with the idea on its own? I'm definitely not an expert here but I am doubtful of that.
To validate a hypothesis, LLMs would likely be involved.
This one seems more plausible. Say you provide an AI system with a hypothesis and ask it to validate the hypothesis, an LLM would likely be used to essentially scaffold the experiment. However, assuming you provide the model with an explanation for how to test this novel hypothesis; if the data you provide is entirely foreign to it, would it not be unable to understand what it is validating? Even if you provided it a very detailed description?
The toy example I have in my head to sort of illustrate what I mean is imagining if you had a model that was trained exclusively on pancake recipes. One day, you ask the model for a meatball recipe, and the model responds "Ah, I understand. You want a pancake recipe!". And you say, "No I want a meatball recipe. It has X, Y, Z ingredients and is made by doing A, B, C in that order". The model would still likely respond, insisting that you are after a pancake recipe. All this to say, is this what would happen if we tried to create a system that could automate hypothesis validation (assuming the hypothesis is novel)?
The seeming impossibility of superintelligence:
I'll make this more brief. The concept of superintelligence seems to me rooted almost entierly in SciFi-fantasy. However, I now see actual respected scientists talking about the risks of it, and as if it were a guarantee it will happen, so I suppose I would be a fool not to try and understand it.
My question is fairly straight forward: how could a system improve on itself, using its own data, when it is fundamentally limited to the data it knows? This is why it seems impossible for the current LLM approaches to ever lead to "ASI". Maybe "AGI", but even then I'm not sure (but the industry leaders and researchers seem sure of it so I guess I am wrong).
The only way I can see superintelligence would be continual learning on an enormous scale, which is currently not possible using the transformer NN architecture. This would imply we need considerable advances in AI, and likely a completely new and different paradigm, for us to reach superintelligence in an AI system. Even then, how good could such a system actually become?
The arguments I have seen from people who think/know superintelligence is possible and imminent can be classified as either "There is no reason why its not possible", "Look at the current advances and say we wont have superintelligence soon" or "An AGI system will be able to improve upon itself". The first two "arguments" are basically self-explanitory in how irrelevant they are as actual explenations. However the second one also seems impossible. Assuming we achieve AGI via scaling LLMs, how would a system which (assuming question 1 is true) improve upon itself, as it would require it generalizing beyond its dataset? I see people saying vauge things like "it will improve its own code!". Okay, put a coding agent at task with making a function better, loop it a million times, come back and find its more or less the same but maybe slightly more efficient and considerably more refactored.
This is where I am the most out of my depth, so if someone could actually explain this in a scientific manner that would be great. Even the researchers whom you hear talking about this never actually bother talking about how superintelligence will be achieved, or why it is/is not possible.
TL;R
Can LLMs truly generalize beyond their training data or only "remix" what’s already there?
How would automated AI research could actually work if models can’t generate or validate genuinely novel hypotheses?
Why do some experts believe superintelligence is possible when current systems seem limited by their data and architecture? I’m asking for a clear, scientific explanation of these points rather than vague claims about inevitable AGI/ASI.
Thank you! 😄