From Generating Plans to Deciding What Comes Next: How AutoResearch Closes the Research Loop Without Hallucinating Success

AutoResearch connects problem discovery, plan generation, swarm experimentation, and independent validation into a research process that can keep moving forward. In a Django repair task, it raised the official new-feature test result from 2/7 to 7/7 while keeping all 203/203 regression tests passing. Across three Kaggle experiments, it also distinguished success, a near miss, and a plateau, then used that evidence to continue, change direction, or stop.

AutoResearch is an automated research system. It continuously observes papers, open-source projects, community discussions, and technical anomalies, identifies questions worth investigating, organizes multiple models to turn ideas into experiments, and lets external results determine what happens next.

The process addresses a failure mode more subtle than inventing facts: the code is real and the local tests really pass, yet the system mistakes “this part is correct” for “the entire problem is solved.”

Figure 1 | A green local test suite proves only that the checks it covers passed. Independent evaluation exposes critical gaps that those checks did not cover.

In a real Django repair task, GPT-5.6 passed 716 migration tests and 39 optimizer tests, but scored 0/7 on the official new-feature tests. AutoResearch also failed to solve the whole problem in its first round: its local tests were green, while the official result was only 2/7. The difference appeared after that failure. The system kept reformulating the problem from external feedback and progressed through 2/7, 4/7, and 7/7 until both the new-feature tests and regression tests passed.

1. Four actions form the core research loop

Figure 2 | Find a problem, forge a testable idea, run the experiment, and let evidence decide. These four actions form the AutoResearch loop.

The AutoResearch loop contains four consecutive actions:

1. Discover a problem: extract traceable new phenomena, results, or unresolved contradictions from open information.
2. Form a testable idea: rewrite the signal as a research hypothesis with success criteria, resource limits, and failure conditions.
3. Organize swarm experiments: assign planning, implementation, execution, and review to collaborating roles.
4. Let evidence decide: external results determine whether to iterate, change direction, or stop.

Failure does not close the workflow automatically. It becomes input to the next round and pushes the system to narrow the problem boundary or change the experimental path.

2. Find a question worth studying before trying to answer it

AutoResearch monitors arXiv, Hugging Face Daily Papers, OpenReview, and GitHub Trending, along with Reddit, Hacker News, research blogs, Chinese technology media, and Twitter/X. These channels differ substantially in data format, popularity signals, and information density. Collected items must first be normalized, then filtered for quality and deduplicated within a batch and across days.

A first-stage model quickly decides whether an item contains an insight that can become a research question. The retained candidates go to stronger models for further screening on novelty, technical feasibility, resource needs, and evaluation design. When enough candidates accumulate, the system ranks them again and selects the research seeds that should enter Idea Forge first.

This funnel compresses an open information stream into a small set of candidates worth experimental resources. Each research question retains an explicit source, so later plans can be checked against the original observation.

3. Idea Forge turns signals into experimental plans

A research seed is only a phenomenon or question from the outside world. Idea Forge also selects an established research direction from the knowledge base and combines it with the seed to form candidate plans. Current knowledge directions include multimodal fusion, visual token management, large-model reasoning and test-time compute, and agent memory.

Gemini, GPT, and Claude independently propose and review candidate ideas. Their training backgrounds and judgment preferences differ: some react more strongly to novelty, some focus on technical feasibility, and some identify resource or evaluation gaps first. Before a candidate can enter execution, it must pass four checks:

• Cross-review: three models judge it independently, and an idea needs at least two votes to continue.
• Currency check: outdated models, old dates, and obsolete technical claims are excluded.
• Knowledge-consensus check: the plan must genuinely use the selected knowledge direction instead of attaching fashionable terms mechanically.
• Multi-stage plan: the first stage uses a low-cost pilot that can quickly decide the direction; later stages expand the experiment, add ablations, and complete evaluation.

The majority vote exists to expose blind spots early. The final plan must still reduce to a measurable hypothesis, an explicit budget, and experiments that can determine success or failure.

4. From plan to experiment: how the swarm collaborates

Figure 3 | After cross-review and planning, a research seed enters a collaborative flow of planning, implementation, execution, review, and external evaluation.

During execution, AutoResearch divides the work into consecutive task units. A coordinating role maintains global state; a planning role rewrites the objective as a falsifiable experimental design; an implementation role writes the code; an execution role runs it and organizes the results; reviewers look for omissions from different perspectives.

Simple tasks can use fewer roles, while complex tasks can add a subcoder, monitor, or independent validator. Role names do not determine the result. The important property is that planning, implementation, execution, and review no longer occupy a single model response.

Plans, code, experiment records, and review findings remain in one persistent workspace. The loop continues processing unfinished units until external metrics and stopping conditions permit closure. Experimental results can overturn the plan, and reviewers can make mistakes. A role's recommendation to finish cannot close the workflow. Reproducible results and independent success criteria determine whether to continue and when to stop.

5. System validation I: advance a research objective to a verifiable result

AutoResearch used a similar experimental protocol for three Kaggle problems: start with a low-cost pilot to test the direction, then design the main experiment from its result. Every stage has a success criterion, code review, execution record, and stopping decision. The three problems use accuracy, root mean squared logarithmic error (RMSLE), and F1 respectively, following the metric direction defined by each Kaggle competition [1][2][3].

Figure 4 | One protocol produces three different outcomes: Titanic reaches its target, House Prices approaches its target, and Disaster Tweets reaches a plateau. Each chart has an independent vertical scale and should not be compared numerically across tasks.

5.1 Titanic: a strict control isolates the value of feature engineering

The first stage used nine basic features to establish a baseline. Five-fold cross-validation accuracy for Gradient Boosting was 0.8272. The second stage added features including Title, FamilySize, IsAlone, age bins, and fare bins.

Control and Treatment used the same random seed, data folds, and model. Control retained the basic features; Treatment added the extended feature set. Control reproduced the first-stage result exactly, while Treatment reached 0.8428 and crossed the 0.8300 target. The feature set was the main difference between the two conditions, so the result supports feature engineering as the cause of this round's improvement.

5.2 House Prices: identify the changes responsible for most of the gain

The target was a five-fold cross-validation RMSLE below 0.1200. The system first established a Ridge baseline on a small sample, then changed only the prediction target by applying log1p to sale prices. RMSLE fell from 0.2008 to 0.1566. Adding a small number of derived features produced 0.1563, showing that further feature accumulation had little value.

For the full experiment, the model family changed from a linear model to LightGBM and reduced the result to 0.1258. Extended features and interactions eventually reached 0.1251. The target was not met, but the experiments located the dominant sources of gain: target transformation and the model-family change accounted for nearly all improvement, while additional features had only marginal effect. The system therefore stopped mechanically widening the tuning range.

5.3 Disaster Tweets: a plateau triggers a stop

The target was F1 of 0.8350. The system began with TF-IDF and logistic regression, establishing a lexical baseline of 0.7680, then tried SBERT, DistilBERT fine-tuning, and soft-voting ensembles. The best result was 0.8049.

It then increased the parameter count from 66M to 125M and switched to RoBERTa-base, whose best score was 0.8005 and did not beat DistilBERT. After 21 experiments, the best result remained 0.8049. A larger model did not improve the metric, so the system stopped adding experiments along the same path and reduced the next step to three choices: obtain useful data, add training resources, or change the modeling approach.

6. System validation II: advance a local repair to a complete solution

The second evaluation uses django__django-16820 from SWE-bench Lite. SWE-bench applies a model-generated patch to a real repository and runs repository tests to determine whether the issue is solved [4]. This task originated in Django's deprecation of Meta.index_together. It requires migration optimizer and autodetector behavior to work together across RenameIndex, AddIndex, RemoveIndex, and cyclic foreign keys. The upstream Django discussion also records the relationship between these index operations and complex migration optimization [5].

Figure 5 | GPT-5.6 stopped after local tests passed and the official new-feature tests scored 0/7. AutoResearch reformulated the problem after the 2/7 and 4/7 results, ultimately reaching 7/7 while keeping 203/203 regression tests passing. The two conditions had different time and feedback access; the comparison concerns what happened after failure.

GPT-5.6 found a real gap: CreateModel.reduce() lacked folding logic for RenameIndex(old_fields=...). Its patch passed 39 optimizer tests, 716 migration tests, and all 203 official regression tests, yet scored 0/7 on the official new-feature tests. It fixed the RenameIndex subproblem but did not cover AddIndex, RemoveIndex, the autodetector, or cyclic foreign keys.

AutoResearch also focused on RenameIndex in its first round. Code review found a missing no-match guard. After that correction, 41 optimizer tests, 11 squashmigrations integration tests, and 719 migration tests passed, while the first official result was 2/7.

External results expanded the research view from one reduce() branch to coordination between the optimizer and autodetector. The second version reached 4/7. The next round continued tracing how index state enters the migration flow, gets folded, and converges. The final result reached 7/7 on the official new-feature tests while preserving 203/203 regression tests.

7. AutoResearch automates research decisions

The two evaluations show that the core capability of AutoResearch lies in five parts:

1. Turn a research question into an executable object: an idea must produce a pilot, success criteria, a resource budget, an ablation plan, and deliverables.
2. Put different models in suitable positions: lightweight models screen many candidates, stronger models perform deeper judgment, and multiple model families propose and review ideas independently.
3. Make stage results determine the next action: success, a near miss, and a negative result all leave evidence and trigger expansion, adjustment, or stopping.
4. Give completion authority to independent metrics: internal agreement and green local tests cannot replace a complete success criterion defined in advance.
5. Allow the system to stop: when a direction reaches a plateau or its return on effort falls, the system closes the current path while preserving the state and conclusions required to restart it.

Models propose, implement, and challenge. Evidence decides what happens next.

8. Success is an unbroken chain of evidence

Large models are already good at generating candidate answers quickly. AutoResearch designs the complete loop after an answer: how a problem becomes an experiment, who reviews the result, how failure rewrites the next step, what evidence justifies continuation, and which conditions stop the system.

This method can be called loop engineering. The design covers more than roles and prompts: it governs how state persists, how evidence moves, and how judgments are checked. Roles may make mistakes. The loop must expose those mistakes and prevent an unsupported claim of success from closing the workflow.

9. References

1. Kaggle. (2012). Titanic - Machine Learning from Disaster. Kaggle Getting Started Competition.
2. Kaggle. House Prices - Advanced Regression Techniques. Kaggle Getting Started Competition.
3. Kaggle. Natural Language Processing with Disaster Tweets. Kaggle Getting Started Competition.
4. Jimenez, C. E., Yang, J., Wettig, A., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. The Twelfth International Conference on Learning Representations.
5. Django Software Foundation. (2023). Fixed #34529, Refs #34525: Reduced index operations with Meta.indexes / index_together when optimizing migrations. django/django Pull Request #16820.