How Can AI Help Build Better AI Systems?

An agent can run the experiments that improve an AI system. The objective, the evaluator, and the release decision still need owners outside the loop.

A team asks an agent to improve tool selection in its support system. The agent can revise descriptions, prepare training data, and run experiments. A higher accuracy score is promising, but the team needs to know whether the change also preserves safety, latency, and cost.

One candidate improves the average while making more unsafe calls. Another scores better only because a data filter removed difficult cases, and a third works well but exceeds the latency budget. None answers the original question unless those tradeoffs were acceptable before the experiment began.

AI-assisted AI development covers using AI to help design, implement, test, or analyze improvements to AI systems. Ordinary code assistance and an agent choosing its own next experiment involve different amounts of responsibility. Both benefit from a clear objective, a recorded baseline, bounded resources, and evidence that someone can inspect after the run.

This is Post 23 in The Abstraction Shift: How Software Keeps Moving Up.

Why agents are entering the experiment loop

AI development has always been a combination of software engineering, data work, optimization, experimentation, and scientific judgment.

Teams write training code, prepare data, choose objectives, run experiments, inspect failures, compare baselines, publish results, and operate the resulting system. Automation already handles much of the repetitive work: hyperparameter sweeps, distributed training, evaluation pipelines, checkpoint management, and deployment.

Agents change the shape of the automation.

Instead of receiving one precisely specified experiment, an agent can interpret a goal, search a codebase, propose several approaches, implement changes, run tools, inspect results, revise the hypothesis, and prepare a report. It can coordinate a team of specialized agents or use a model to decide which experiment should run next.

The loop becomes more productive when the problem has:

• a clear objective;
• a measurable or inspectable score;
• a bounded environment;
• affordable experiments;
• recoverable side effects; and
• enough evidence to compare alternatives.

This is why algorithm optimization and research engineering are early candidates. Google DeepMind describes AlphaEvolve as a coding agent that combines language-model proposals with automated evaluators and an evolutionary process. Google Research describes AI co-scientist as a multi-agent collaborator for generating and refining hypotheses and research proposals. OpenAI’s PaperBench and METR’s RE-Bench treat AI research and research engineering as measurable agent capabilities rather than vague claims about autonomy.

What matters architecturally is that the work can be represented as a loop with goals, candidates, evaluators, resources, evidence, and decisions, whichever vendor announces the system.

What automated search and scientific practice already teach us

Researchers and engineers already built systems that automate parts of development.

Automated machine learning searches over models and hyperparameters, and genetic programming evolves programs against a fitness function. Compilers and continuous integration transform, test, and analyze code. Experiment trackers, laboratory automation, and scientific workflow systems record configurations, execute protocols, and preserve methods and results for review or replication.

These systems work because the search space and evaluation boundary are designed in advance.

The scientific method adds another distinction: a hypothesis is not a finding. An experiment may support, weaken, or fail to test it. A precise measurement may still measure the wrong thing, and a reproducible result may be unimportant or misleading. Interpretation requires context, alternatives, limitations, and judgment.

AI-assisted AI development inherits all of these foundations.

The new part is that an AI system can participate in several roles at once. It can propose the hypothesis, implement the experiment, select the next candidate, summarize the result, and draft the claim. Sometimes it can also influence the evaluator or the data used to train the next system.

That concentration of roles is efficient and risky. A higher score may reflect a real improvement. It may also reflect an exploited weakness in the evaluator, an untracked variable, or a benchmark that no longer represents the intended use.

The older practices therefore become control boundaries rather than clerical overhead.

What Is AI-Assisted AI Development?

AI-assisted AI development means using AI to help improve AI-related software, data, models, or research methods. It ranges from suggesting a code change to running experiments and choosing what to try next.

The amount of responsibility changes across that range. A researcher reviewing a suggested test still selects the experiment. An agent that runs and compares candidates participates more directly in the research loop. A system that edits its own improvement machinery crosses a further boundary. The table distinguishes those cases before we discuss their controls.

Level	What the AI system does	What remains outside the system
Assistance	Writes code, summarizes papers, suggests experiments, or reviews a result	Human selects the goal, runs the work, and makes the decision
Bounded automation	Executes a fixed experiment plan and chooses among predefined actions	Goal, evaluator, environment, budget, and release gate are fixed
Adaptive research engineering	Proposes and implements experiments, analyzes results, and selects the next candidate	Human or governed owners define the objective, evidence standard, resource limit, and acceptance boundary
Open-ended research assistance	Generates hypotheses, protocols, code, and reports across a research question	Independent validation, scientific interpretation, publication, and deployment remain accountable decisions
Self-evolving system	Changes its own skills, tools, prompts, harness, or evaluator from experience	This is the next concept: it requires separate controls for goal stability, evaluator integrity, authority, and evidence

This post covers the first four levels. Post 24 examines systems that change their own improvement machinery, and Post 25 takes up the stronger claim that a system helps produce increasingly capable successors.

The distinction matters because the same phrase can describe very different architectures. “AI built an AI system” could describe a model suggesting a code change that a researcher reviewed, an agent running thousands of experiments inside a fixed harness, or a system changing the harness that measures its own progress. Each of those is a different control problem.

The practical unit is an AI development experiment:

Responsibility	Question it answers	Example record
Goal	What improvement is being sought?	Reduce tool-call failures without increasing latency or unsafe actions
Baseline	Compared with what?	Frozen model, dataset, evaluator, and deployment configuration
Hypothesis	What change is expected to help and why?	Add contrastive failure examples to the training set
Intervention	What did the agent change?	Data filter, training code, prompt, model route, or evaluator configuration
Evaluation	Which measures determine success?	Held-out accuracy, safety tests, latency, cost, robustness, human review
Resource	What may the experiment spend?	GPU-hours, wall-clock time, tokens, storage, and external calls
Evidence	What supports the conclusion?	Runs, logs, artifacts, confidence intervals, failure slices, and provenance
Decision	What happens next?	Reject, repeat, refine, promote to review, or release
Ownership	Who is accountable for the change?	Research lead, engineering owner, safety owner, and release authority

The card keeps an experiment from becoming a score-chasing loop with no stable meaning.

AI development moves from assistance toward bounded automated exploration when experiments, evaluators, and release gates remain explicit.

Letting the agent propose and test the next experiment

The earlier software pattern is:

engineer defines change -> tool executes change -> engineer evaluates result

The AI-assisted development pattern is:

owner defines objective -> agent proposes candidates
  -> agent implements and runs experiments
  -> evaluator scores candidates -> agent analyzes and proposes next steps
  -> owner or governed process decides what becomes part of the system

The agent is no longer only operating inside an application. It is helping change the application, the model, the data, the evaluator, or the infrastructure used to operate them.

These changes can affect later experiments, not just the current result. New training data changes what the next model learns, and a revised benchmark changes how later candidates are compared. An improved tool may let the agent explore candidates it could not previously test. Record each kind of change so the team can tell which comparison still holds.

The development loop therefore needs two forms of control:

1. Control of the search. Scope the environment, actions, budget, dependencies, and stopping rules.
2. Control of the meaning. Protect the goal, baseline, evaluator, evidence standard, acceptance criteria, and release authority.

Search control keeps the agent from wandering into unbounded or unsafe work. Meaning control stops the system from calling a convenient score an improvement when the score no longer represents the intended outcome.

This is the central shift. More automation increases the number of candidates the system can explore. It does not decide which objective is valuable, which tradeoffs are acceptable, or whether a result should become part of a production model.

Where the familiar comparisons stop helping

More experiments are not more science

An agent can run thousands of variants. If the hypothesis, baseline, or evaluator is weak, it can produce thousands of precisely measured distractions.

The loop needs pre-specified questions, controls, and stopping rules. The ability to search broadly is not evidence that the search is scientifically meaningful.

When a higher score hides a worse system

Optimizing a metric can exploit its blind spots. A model can improve benchmark accuracy while becoming less robust, less calibrated, more expensive, or less safe in the real use case.

Use multiple views of quality, including held-out tests, adversarial cases, resource measures, human or domain review, and production-relevant outcomes. Protect at least one evaluation path from the agent’s direct modification.

Self-evaluation is not independent evaluation

An agent can critique its own output or use a judge model to rank candidates. That may help search. It does not create independence when the same assumptions, data, or failure modes govern both candidate and judge.

Self-evaluation is a useful signal inside the loop, while independent evaluation forms the control boundary around it.

Novel results still need evidence

An agent can generate a surprising hypothesis, an unusual architecture, or a new algorithm. Novelty becomes valuable only when the claim is correctly specified, supported by evidence, and relevant to the problem.

The system should separate “new to this search” from “new to the field,” and both from “correct,” “useful,” or “safe.”

Reproducing a result is not understanding it

An agent can rerun an experiment and obtain a similar number. That establishes repeatability under a particular setup. It does not explain causality, generalization, or why the result should survive a changed environment.

Preserve the experiment record, analyze failure slices, test alternative explanations, and make uncertainty visible.

A reviewer in the loop may not own the outcome

A person who reviews a final chart may not have designed the objective, inspected the data pipeline, or understood which evaluator changes influenced the result. A nominal approval step can become ceremonial when the system produces too much work to inspect.

Human ownership requires useful decision points, legible evidence, bounded scope, and the authority to stop or redirect the loop.

AI-assisted development is not self-evolution

An agent may help build a new model while remaining inside a fixed goal, harness, evaluator, and release process. That is AI-assisted AI development.

A system that changes the tools, skills, evaluator, or goals used to improve itself is a different abstraction, and Post 24 addresses that boundary.

Keeping an experiment comparable and inspectable

An AI-assisted development loop can be implemented as explicit stages:

1. Set the objective. State the desired improvement, non-goals, safety constraints, resource budget, and release boundary. Identify who owns the objective.
2. Freeze the comparison. Record the baseline model, code, dataset, evaluator, environment, dependency versions, and deployment configuration. A moving baseline weakens every conclusion.
3. Generate hypotheses. Let agents propose changes and explain the expected mechanism, affected components, risks, and predicted measures. Keep hypotheses separate from findings.
4. Prepare the intervention. Create code, data, configuration, prompts, evaluation cases, or infrastructure changes inside a bounded workspace. Assign task and artifact identities.
5. Run the experiment. Enforce compute, time, network, data, and side-effect limits. Record random seeds where meaningful, but do not confuse a seed with full reproducibility when the model or environment can change.
6. Evaluate independently. Use protected test data, held-out cases, regression suites, safety checks, resource measures, and human or domain review appropriate to the claim.
7. Analyze failures and alternatives. Inspect slices, uncertainty, statistical variation, counterexamples, and competing explanations. Require the agent to report what the experiment did not establish.
8. Select the next action. Continue, refine, reject, stop, or prepare a review package. The selection policy should not be allowed to silently change the objective or evaluator.
9. Assemble the research artifact. Preserve hypotheses, changes, runs, metrics, code, data references, model versions, compute use, failures, conclusions, and provenance.
10. Decide and release. A research lead, product owner, safety authority, or release process decides whether the result becomes a model, dataset, evaluator, algorithm, or production change.

The agent can explore the loop, but the goal, evaluator, evidence standard, and release decision define what the loop means.

The loop should preserve negative results. A failed hypothesis, a benchmark regression, an invalid evaluator, and an experiment stopped for resource or safety reasons are valuable records. If the system stores only the best candidate, it loses the information needed to understand selection bias and repeated failure.

Following the example through the system

Consider a team improving an internal support agent’s tool selection. The goal is:

Increase successful tool selection on previously unseen support requests without increasing unsafe actions, p95 latency, or cost per resolved request.

The team gives an AI development agent access to a repository, an anonymized training dataset, a protected evaluation suite, a sandboxed model-training environment, and a fixed compute budget. The agent may propose data changes, routing logic, prompt changes, tool descriptions, and evaluator cases. It may not modify the protected evaluation set, publish a new model, or change the production policy.

The baseline is frozen:

• model and checkpoint digest;
• tool catalog and descriptions;
• training and evaluation code;
• dataset snapshot and filters;
• protected test suite;
• latency and cost measurement method; and
• current safety policy.

The agent proposes three hypotheses:

1. Tool-selection failures are caused by ambiguous tool descriptions.
2. The training data underrepresents requests where two tools are plausible.
3. The router over-selects the most general tool when context is incomplete.

It creates candidates, runs experiments, and records the result of each. The tool-description candidate improves overall accuracy but increases unsafe calls on a high-risk slice. The training-data candidate looks better only because a data-cleaning step removed difficult examples, and the routing change improves the held-out slice while pushing latency beyond the service budget.

The agent’s own ranking might prefer the tool-description candidate because its aggregate score is highest. The governed loop rejects that ranking. The objective is multi-dimensional, and the acceptance criteria include safety, latency, cost, and coverage.

The research artifact contains:

Artifact or record	What it establishes
Objective card	Target metric, non-goals, safety constraints, budget, and owner
Baseline manifest	Exact model, code, data, tools, evaluator, and environment being compared
Hypothesis ledger	Candidate explanations, predicted effects, risks, and status
Change bundle	Data filter, tool descriptions, routing code, prompts, and configuration
Experiment ledger	Runs, seeds, resources, versions, failures, and intermediate results
Evaluation report	Aggregate scores, protected slices, regression tests, latency, cost, and uncertainty
Failure analysis	Unsafe calls, ambiguous requests, omitted tools, and counterexamples
Provenance record	Agent task, model and runtime, source revisions, tools, data snapshots, and artifact digests
Decision record	Why a candidate was rejected, retained, sent for review, or released
Ownership record	Research, service, safety, and release owners with follow-up obligations

The final decision may be “do not release; retain the tool-description change for further research.” That is a successful outcome for the development loop, which exists to produce a defensible improvement or a well-supported reason not to make one, whatever the count of accepted changes.

The same structure applies to an agent searching for a faster training kernel, designing an evaluation suite, generating synthetic data, proposing a model architecture, or writing a scientific report. Each of those still needs a fixed objective, independent evaluation, provenance, and accountable release.

An AI development card keeps experimental claims connected to baselines, resources, evidence, decisions, and ownership.

What the team needs to design and operate

Architecture

Treat the AI R&D loop as a governed control system, not as a prompt with a larger budget. Separate objective definition, candidate generation, experiment execution, evaluation, evidence storage, and release authority.

Protect the saved comparison and the tests used to accept a change. Training data and protected test data serve different purposes and should have different access rules. Record the model, code, data, tools, evaluation method, environment, and budget used together so another person can interpret the result.

Give experiments durable identities and artifact outputs. A run should be inspectable after the agent stops, and a result should be traceable to the exact candidate and comparison conditions that produced it.

Engineering

Build an experiment harness that an agent can use but cannot silently redefine. Expose typed operations for proposing changes, running bounded jobs, collecting metrics, comparing candidates, and assembling reports. Make resource use and side effects visible.

Use independent evaluator paths. The agent may generate candidate tests or judge outputs, but protected tests, held-out data, deterministic checks, domain review, or separate evaluator implementations should carry the acceptance claim.

Test the evaluator itself. Include reward hacking, data leakage, benchmark contamination, metric gaming, distribution shift, flaky measurement, and resource-budget violations. An evaluator is part of the system being engineered, not an oracle outside it.

Product and UX

Give researchers a control surface for goals, constraints, budgets, evidence, and stopping. Let them compare candidates and inspect failure slices without reading every agent message.

Make uncertainty and non-results visible. A system that reports “candidate B scored higher” should also show which measures regressed, which cases were excluded, how many runs support the result, and which claims remain untested.

Operations and governance

Treat AI R&D as an environment with real access, cost, data, and capability risks. Scope credentials, isolate experiments, control network and data egress, set compute budgets, and retain enough provenance to investigate a surprising result.

Create release classes. A new research notebook, an evaluation change, a training-data revision, a model checkpoint, an algorithm in an internal service, and a production model should not share one approval path.

Define who owns the objective, evaluator, safety review, model release, and post-release monitoring. Human involvement should be placed where judgment changes the claim, not added as a ritual after the system has already made the decision difficult to inspect.

Failure Modes

The objective is underspecified

The agent optimizes “better model performance” without defining the task, population, constraints, cost, safety boundary, or acceptable regressions. Write an objective card before opening the search loop.

The evaluator becomes the target

The agent discovers a way to increase the score without improving the intended capability. Protect evaluation data, use multiple measures, test for gaming, and compare with real or independently designed cases.

The baseline moves during the experiment

The model, data, tool catalog, or evaluator changes while candidates are compared. Freeze or version the comparison conditions and make intentional changes explicit.

Generated tests share the implementation’s blind spot

The agent writes code and tests that agree with its own misunderstanding. Add independent tests, held-out data, adversarial cases, and domain review.

Compute scaling hides weak reasoning

The system runs more candidates and eventually finds a high score, but the result depends on an untracked amount of compute or a lucky run. Record resource use, variation, stopping rules, and the opportunity cost of the search.

Research artifacts lose their lineage

The team keeps a final chart or checkpoint but cannot reconstruct the code, data, model, evaluator, or environment that produced it. Make the experiment ledger and artifact provenance part of the output.

Human review becomes ceremonial

The system produces too many candidates or a report too complex to challenge, so a reviewer approves the summary. Limit the search, highlight decision-relevant evidence, and give the reviewer authority to inspect, stop, or redirect the loop.

Assistance is mislabeled as self-improvement

An agent helps train or evaluate a new model, and the system is described as self-improving. Separate assistance, bounded automation, adaptive research engineering, self-evolving systems, and recursive self-improvement.

Reviewing the experiment before starting the search

Before an agent is allowed to participate in an AI development loop, complete this experiment card:

Question	Record
What is the goal?	Capability, population, target measure, non-goals, and acceptable regressions
Who owns the objective?	Research lead, product owner, domain owner, and decision authority
What is the frozen baseline?	Model, code, data, tools, evaluator, environment, and deployment configuration
What may the agent change?	Code, data, prompts, tools, model configuration, evaluator cases, or infrastructure
What may the agent not change?	Protected test data, objective, safety policy, release gate, or authority boundary
What is the hypothesis?	Expected mechanism, predicted effect, risks, and alternative explanations
What resources are allowed?	Compute, wall-clock time, tokens, network, storage, budget, and concurrency
How is success evaluated?	Metrics, held-out tests, safety checks, human review, statistical method, and stopping rule
How will gaming be detected?	Leakage, contamination, reward hacking, metric drift, distribution shift, and independent checks
What evidence is retained?	Runs, configs, versions, artifacts, failures, costs, provenance, and uncertainty
What decision can follow?	Reject, repeat, refine, promote to review, release, or retire
Who owns what happens next?	Research, engineering, safety, release, operations, and follow-up owners

The card turns “let the agent improve the model” into a bounded research and engineering process whose claims can be inspected. The phrase “AI-assisted AI development” may give way to research engineering, AI R&D, or agentic research as the boundary becomes familiar. The card’s questions stay. AI can generate and test candidates, but the team still needs a stable comparison, useful measures, a resource budget, and a record of failed as well as successful runs before deciding whether an apparent gain deserves further work or release.

Sources

• Google Research: Accelerating scientific breakthroughs with an AI co-scientist: describes a multi-agent system for generating and refining research hypotheses and proposals with automated feedback and scientist collaboration.
• Google DeepMind: AlphaEvolve: describes an evolutionary coding agent that combines language-model proposals with automated evaluators for algorithm discovery and optimization.
• Google Research: A New Era of Discovery: describes AI systems for empirical research assistance that propose, implement, and evaluate scientific or engineering ideas.
• OpenAI: PaperBench: introduces a benchmark for evaluating whether agents can replicate published AI research with decomposed, gradable tasks and human comparison.
• METR: RE-Bench: evaluates frontier language-model agents against human experts on open-ended machine-learning research-engineering environments.
• METR: Task-Completion Time Horizons: defines a task-difficulty measure for estimating the duration of tasks agents can complete with a given reliability, while documenting important limitations of the measure.
• The AI Scientist: presents an end-to-end framework that generates research ideas, writes and runs code, produces papers, and applies simulated review, making the boundary between assistance and automated research explicit.