GPT-6.2 Astra Tested on Automating AI Research
- Source
- x.com
- Date
We conducted pre-release testing of GPT-6.2 Astra to assess its ability to automate AI research. GPT-6.2 Astra is the highest scoring OpenAI model we have tested. It scored 29.01% on the Vals RSI-Index*, an incremental gain over 27.07% for GPT-6 Astra. The RSI-Index is calibrated such that 50% represents human-level performance on each task.

Context
Vals AI tested GPT-6.2 Astra before OpenAI released it and reports a score of 29.01% on its RSI Index, the highest of any OpenAI model Vals has tested and what Vals calls an incremental gain over GPT-6 Astra's 27.07%. RSI stands for recursive self-improvement, meaning AI systems developing their own successors. The index asks whether a model can do the research work that goes into training a new model.
The percentage is not a share of tasks passed. Vals sets the model four open-ended research problems under fixed compute and time budgets: build a lossless compressor for a 100 MB slice of Wikipedia text, improve a small GPT training program, improve a judging setup that grades coding submissions, and further train an open model to do better on finance tasks. Each task is scored on a log scale where 0 is the task's starting point, 0.5 matches a chosen reference result and 1 is the theoretical best, and the index is the average of the four. Vals's post describes 50% as human-level, but its methodology page is more careful: the references mix published research, agreement between human annotators and a frontier model, and are "not a uniform measure" of the strongest human performance.
Astra's average hides an uneven profile. On the two tasks it nearly matched the references: its compressor reached 1.2582 bits per byte against 1.2435 for the reference, from a 2.92 starting point, and its trained model reached 0.8744 against 0.85, for task scores of 0.49 and 0.46. The judge it built agreed with human graders 79.67% of the time, against a 50% starting point and 93.6% agreement between human annotators, scoring 0.21. On post-training it finished at 33.46%, below the 35.38% it started from, which scores zero; Vals's chart notes that this run had internet access limited to approved sites, while the task's published setup allows unrestricted access, so this zero is not a like-for-like comparison with other models. In October, Vals reported that most models had not got past the starting point on that task. Vals describes all of these as single runs on narrowly scoped tasks, so the result says Astra can come close to published results on two well-defined training problems but is still far from doing a model's full research cycle unaided.
- pretraining — The first, biggest phase of building a model: training it on enormous amounts of text so it learns language, facts, and reasoning in general.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- mixture-of-experts — A model built from many specialist sub-networks where only a few activate per token, giving big-model capability at small-model running cost.
Checking sign-in…
Loading comments…






