![\"Photonic](\"https://engineering.flexcompute.com/images/autoresearch/pic_chip.png\")
\nI wanted to learn auto-routing for photonic chips. The story below is how I got there: a failed brute-force attempt, then using AI as a learning partner, then building interactive HTML routing arenas, then translating that into a PhotonForge router, and finally wrapping an auto-design agent around the whole thing.
\nMy first instinct was lazy and honest. I opened Claude Code, dropped in screenshots of a half-routed projector chip, and asked the model to fix the layout by editing PhotonForge's route_manhattan call. No algorithm in mind, no plan, just \"make this work.\"
It sort of worked. And then it didn't.
\nEach round of \"please fix this\" traded one failure for another. Waypoints helped two nets and broke three. A tweak that untangled the left half pushed the mess to the right half. The AI was writing plausible code, and I was nodding along, but neither of us actually understood routing. We were patching symptoms.
\nThis is the trap that's easy to miss: AI is so good at producing code that reads right that you can spend days iterating without stepping back to ask the real question.
\nSo I stopped asking for code and started asking for a map of the territory. What does this problem look like when people have thought about it carefully?
\nThe AI pointed me at a 60-year-old field: VLSI and PCB autorouting. Lee's algorithm from 1961, A* with Manhattan heuristics, Dijkstra over weighted cost fields, PathFinder-style negotiation routing, rip-up and reroute from commercial EDA tools. And then it pointed me at the best single resource it knew of: Rob Rutenbar's VLSI CAD: Logic to Layout lecture series from the University of Illinois, weeks 7 and 8 specifically, free on the Internet Archive.
\nI watched them. Not skimmed, watched. And this is where the story changes. From this point on, the AI is no longer guessing with me. It is helping me build and test ideas I actually understand.
\nRouting is spatial. Reading about BFS wavefront expansion does not click until you see the wavefront propagate across a grid. So I asked the AI to build me interactive visualizers. Not in Python, not in a Jupyter notebook, in pure HTML with Canvas, so I could open them on my phone, on the couch, with zero setup.
\nEach lecture concept turned into an interactive prototype within minutes. Click to place a source, a target, and obstacles. Step through BFS one layer at a time. Watch the wavefront wrap around walls. Trace the shortest path back. Then: add a second net, then four, then a cost-field heatmap for bundling.
\nOnce the building blocks worked, I stacked them into a comparison arena that runs twelve algorithms on the same obstacle layout and the same nets, side by side.
\nThe arena made trade-offs visceral. I could drag obstacle walls around with a slider and watch every algorithm respond in real time. New hybrid strategies (rip-up plus bundle, smooth bundle with turn-cost penalties) came out of playing with the arena, not from staring at pseudocode.
\nI want to single this out because it felt counterintuitive at first. The algorithms themselves are Python-native, and the production target is PhotonForge. Why prototype in a browser?
\nBecause the point of the prototype was to build my intuition, not to ship code. HTML means zero setup, works on any device, and Canvas makes it trivial to animate the wavefront at a rate a human can follow. Every time I wanted to understand something — why does net ordering matter? what does the congestion history look like after iteration 5? what turn-penalty kills the sawtooth pattern? — I could have an interactive answer in minutes. That loop is what learning looked like.
\nOnce the algorithms were intuitive on an abstract grid, translating them to PhotonForge was mechanical. The architecture has two layers:
\nPath on the routing layer.from pf_routing_arena import PFRoutingArena\nfrom routing_algorithms import HybridRipUpBundle\n\narena = PFRoutingArena(\n bbox_min=(-200, -200), bbox_max=(2200, 1400),\n grid_pitch=25, router_class=HybridRipUpBundle,\n)\narena.add_obstacles_from_layer(component, M2_layer, margin=1)\narena.add_obstacles_from_layer(component, M1_heater, margin=1)\n\nfor i in range(n_nets):\n arena.add_terminal_pair(f'D{i}', left[i]['T0'], right[i]['T0'],\n source_dir='E', target_dir='W')\n\nresults = arena.route_all(trace_width=20, layer=M2_layer)\nThe integration work was almost entirely about the boundary between the grid world and the layout world: reserving a net's own pad exit zone while keeping other pads blocked, rasterizing heater pads on a different metal layer as routing obstacles, and enforcing a 1-cell margin between committed routes so they do not physically overlap at real trace widths.
\nI prototyped all of this in a Jupyter notebook first, then moved it to the library. The notebook let me flip between algorithms on the same layout and see the difference, which is the same feedback loop the HTML arena gave me, just now in the domain that ships.
\nThis is where I thought I was done. Twelve algorithms, a clean bridge to PhotonForge, a notebook that can route a small test layout. Ship it.
\nThen I pointed it at a real quantum photonic chip with 32 active elements, each needing a metal trace to its own bondpad, with heaters and other obstacles scattered through the middle. Picking an algorithm was no longer the hard part. The hard parts were all the human decisions an engineer makes before the router ever runs: how to pair pins with bondpads, where to place the bondpad row, what directions to let routes approach from, how wide the trace should be.
\nNo single algorithm solves those. They are design decisions, and they interact. Change the bondpad row position and the net-to-pad assignment flips. Change the assignment and the approach directions become wrong. Change the trace width and obstacle inflation changes with it.
\nThis is the regime where an auto-design agent earns its keep. You give it a goal, the tools to build a layout, a design-rule checker that counts violations, and let it propose, try, measure, and revise.
\nThe starting script was the laziest possible: call route_manhattan independently for each of 32 pin-to-pad connections and hope.
That first attempt produced 192 violations: 30 heater crossings and 162 pairs of routes crossing each other.
\nOver 27 iterations the agent:
\nTotal wall-clock for all 27 iterations: 2 minutes 25 seconds. A human engineer laying this out by hand — pairing pins to pads, sketching the bundles, chasing the last DRC violations — typically needs two to three hours.
\nThe shortest version of this story: the AI did not replace the thinking, it compressed the distance between question and interactive answer.
\nThe brute-force attempt failed because I was outsourcing the understanding. The learning partner approach worked because I stopped. The lectures still had to be watched. The wavefront still had to be visualized with my own eyes. The pin-reservation bug still had to be diagnosed by me, not narrated by the model.
\nBut everything between \"I want to explore this idea\" and \"I can see this idea running\" collapsed. Interactive HTML prototype in minutes. PhotonForge bridge in an afternoon. Twelve algorithms ready to compare by the end of a weekend. And finally, an auto-design agent that picks up those algorithms and runs them 27 times in the time it takes to make coffee.
\nYou can outsource implementation friction, algorithm discovery, and visualization scaffolding. You cannot outsource understanding. The surprise is that this style of working actually forces deeper understanding than the old way, because you are constantly building and testing your mental model instead of reading about someone else's.
\nInterested in testing these algorithms on your own layout? Reach out to prash@flexcompute.com.
", "attachments": [ { "url": "https://engineering.flexcompute.com/articles/electrical-routing-agents.md", "mime_type": "text/markdown", "title": "Learning Auto-Routing by Building: From Brute Force to an Auto-Design Agent markdown" }, { "url": "https://engineering.flexcompute.com/images/og/electrical-routing.png", "mime_type": "image/png", "title": "Learning Auto-Routing by Building: From Brute Force to an Auto-Design Agent social image" } ], "_flexcompute": { "kind": "Essay", "tags": [ "ai-engineering", "agents", "photonics" ], "markdown_url": "https://engineering.flexcompute.com/articles/electrical-routing-agents.md" } }, { "id": "https://engineering.flexcompute.com/articles/mode-solver-memory-calibration/", "url": "https://engineering.flexcompute.com/articles/mode-solver-memory-calibration/", "title": "Predicting Peak Memory for an Electromagnetic Mode Solver", "summary": "How we replaced a heuristic memory estimate with a calibrated model for Tidy3D mode solver workloads, eliminating under-predictions across the calibration set.", "image": "https://engineering.flexcompute.com/images/og/mode-solver-memory-calibration.png", "banner_image": "https://engineering.flexcompute.com/images/og/mode-solver-memory-calibration.png", "date_published": "2026-04-15T00:00:00.000Z", "date_modified": "2026-04-15T00:00:00.000Z", "authors": [ { "name": "Momchil Minkov", "path": "/authors/momchil-minkov/", "url": "https://engineering.flexcompute.com/authors/momchil-minkov/" } ], "tags": [ "Photonics", "Tidy3D", "Verification" ], "content_html": "import MemoryScatterFigure from '../../components/MemoryScatterFigure.astro';
\nBefore launching a mode solve, the platform has to choose a machine and a parallelism strategy without yet knowing the solve's true peak memory. That estimate depends less on the sparse matrix size than on what happens around it: fill-in during factorization, whether the problem is real or complex, how much state forked workers can share, and how MPI overhead grows with rank count.
\nOur original estimate was a heuristic, and it missed in both directions. Some jobs OOMed on the first worker and had to be retried on a larger one; others landed on larger machines than they needed from the start. We replaced it with a calibrated model built from measured runs. On the calibration dataset, it removed under-predictions and pulled small-job estimates closer to reality.
\nAt a glance
\nPredict peak mode-solver memory before launch well enough to choose machines safely.
\n\n Fit an affine model, detect matrix regime from metadata, and model copy-on-write and MPI\n overhead explicitly.\n
\nZero under-predictions on the calibration set, with less excess padding on small jobs.
\nA mode solver finds the guided electromagnetic modes of a waveguide by assembling a sparse eigenvalue problem and extracting a small number of eigenpairs with shift-invert iteration. The expensive step is LU factorization of the shifted matrix A−σI. Most of the peak memory comes from fill-in during that factorization, not from the original sparse matrix itself.
\nThe scheduler has to estimate that peak before the solve starts, because it needs to choose both a machine and a parallelism plan. Two execution modes make that harder:
\nmultiprocessing.Pool of N workers, each solving one frequency independently. Total memory is not N times a single solve because forked workers share the parent process image via copy-on-write.K MPI ranks for the LU factorization. Overhead scales sublinearly with rank count.The old estimate knew about these backends in a coarse way, but it missed several effects that turned out to matter. It did not distinguish lossless from lossy materials, or PML from non-PML boundaries, even though either can double or quadruple the effective matrix size. It also ignored mode count, sequential frequency accumulation, and copy-on-write sharing. In practice, it was optimistic in some regions and overly padded in others.
\nWhy the baseline failed
\n\n Many runs sit above the diagonal. Each such point is a job the scheduler would\n have started on too little memory, forcing a retry on a larger worker.\n
\nThe shift-invert eigensolver builds a sparse matrix and factors it. Memory is dominated by fill-in: the additional non-zero entries created during LU factorization. For a 2D modal cross-section with N grid points, the matrix structure depends on the physics:
\n| Regime | \nTypical trigger | \nEffective matrix form | \nRelative weight | \n
|---|---|---|---|
| Scalar real | \nLossless materials without PML | \n2N x 2N real | \nBaseline | \n
| Scalar complex | \nLossy materials or PML boundaries | \n2N x 2N complex, often expanded to 4N x 4N real | \nRoughly 4x the scalar-real non-zeros | \n
| Tensorial | \nFully anisotropic media or off-axis propagation | \n4N x 4N complex | \nAbout 5x the scalar-real memory at the same grid size | \n
N can range from a few thousand to nearly a million, and the matrix dimension determines fill-in volume. A flat per-point heuristic will be right for one regime and wrong for the others.
\nChoosing the right memory model before launch means inferring two things from the simulation definition alone: whether the problem is tensorial, and whether the matrix will be complex. We have to get both right before assembling anything.
\nTensorial detection
\n\n Treat the problem as tensorial for fully anisotropic media, or for off-axis propagation when\n angle_rotation is not active.\n
\n But angle_rotation can rotate the geometry back into an effectively scalar case,\n so angle alone can over-predict.\n
Complex detection
\n\n Treat the matrix as complex if any material has imaginary permittivity, or if any boundary\n uses PML.\n
\n\n PML introduces complex coordinate stretching even when every material is real, so missing it\n under-predicts.\n
\nThis logic has to err on the side of safety. A conservative call wastes some capacity; an optimistic one can trigger an OOM and a retry. We checked the metadata-time prediction against the matrix structure the solver actually built at runtime across the full test suite, and treated any under-prediction as a bug.
\nWe built a harness to run hundreds of controlled solves while sweeping:
\n7k to 924k
\nGrid points
\n1 to 10
\nModes
\n1 to 5
\nFrequencies
\n1 to 5
\nPool workers
\n\n scipy and MUMPS/PETSc\n
Solver backends
\nLossless, lossy, PML, tensorial
\nMaterial regimes
\nEach experiment runs in a fresh subprocess so we get a clean resident set size (RSS) measurement for that solve, rather than allocator leftovers from earlier work. We used a separate MPI scaling dataset for larger problems, spanning 366k to 8.9M grid points and 1 to 28 MPI ranks.
\nThe final estimator has four parts:
\n01
\n\n Start from scalar real cases and fit intercept, per-point slope, and mode-count slope\n separately for each solver backend.\n
\n02
\n\n Apply the regime multiplier to the slopes, not the intercept, so small jobs do not inherit\n exaggerated startup overhead.\n
\n03
\n\n Model the first worker separately, then add only the incremental per-worker overhead that\n survives copy-on-write sharing.\n
\n04
\n\n For distributed solves, model extra rank cost separately and use different coefficients for\n real and complex matrices.\n
\nThe base model is affine, fitted on scalar real cases:
\nBase affine fit
\nsingle_solve_gb = intercept + pts_coeff * (pts / 1000) + modes_coeff * num_modes\nWe fit separate coefficients for each solver backend. The intercept, roughly 1 GB, captures Python and library startup overhead.
\nTensorial and complex problems need a regime scale factor on the memory estimate, but it should apply only to the size-dependent terms. The intercept is startup overhead; it does not change with matrix structure.
\nScaling the whole estimate inflates predictions by an order of magnitude at small sizes, where the intercept dominates, without buying anything at large sizes, where the slope dominates. Scaling only the growing terms kept small solves tighter while preserving headroom where it mattered.
\nRegime-aware scaling
\nscaled_solve = intercept + (pts_coeff * pts / 1000 + modes_coeff * num_modes) * regime_scale\nFor frequency-parallel execution with a pool of N workers, total memory accounts for copy-on-write sharing:
Frequency-parallel pool model
\nfirst_worker = scaled_solve + accumulation * (freqs_per_worker - 1)\nincremental = min(per_worker_overhead, first_worker)\ntotal = first_worker + (N - 1) * incremental + safety_buffer\nThe accumulation term captures memory growth when one worker handles multiple frequencies sequentially.
For MPI-parallel execution with K ranks, where frequencies are solved sequentially, the overhead follows a power law:
MPI overhead model
\nmpi_overhead = (alpha * pts_millions + beta) * (K - 1)^gamma\ntotal = scaled_solve + mpi_overhead + accumulation * (nf - 1) + safety_buffer\nThere are separate (alpha, beta, gamma) coefficients for real versus complex matrices because the distributed solver expands complex matrices to 4N×4N real, quadrupling fill-in.
When Python forks pool workers, the parent's memory is shared copy-on-write, so the cost of adding a worker is much smaller than duplicating a full solve. But the calibration data exposed a second effect: glibc malloc arena accumulation was inflating RSS across sequential solves handled by the same worker. Each frequency solve allocates and frees large sparse arrays, yet the allocator often keeps that memory mapped instead of returning it to the OS. After a few solves, the worker can look materially larger than a fresh process doing the same job. That polluted the calibration data, especially when one worker processed multiple frequencies, and led to scale factors that were too pessimistic.
\nSetting maxtasksperchild=1 in the multiprocessing pool fixed it: each worker gets recycled after a single task. Re-running calibration produced cleaner data and tighter coefficients. Single-frequency or single-worker cases skip the pool entirely and run in-process.
Operational change
\n\n `maxtasksperchild=1` became part of the runtime policy. Recycle each worker\n after one task so allocator arenas do not accumulate across sequential frequencies.\n
\nCalibration also surfaced a failure that looked like a memory problem but wasn't. At large grid counts in the tensorial regime, the solver crashed with a cryptic error even though the machine still had plenty of RAM left.
\nThe cause was a 32-bit integer overflow in the sparse factorization's internal indexing. For a tensorial 4N×4N matrix at large N, fill-in scales roughly as O(n3/2) and can exceed the 2.15 billion non-zero limit of 32-bit indices. More RAM doesn't help.
\nWe now validate the grid point count before submission and return a clear error, rather than letting it crash deep inside the factorization.
\nHard limit
\n\n This is an indexing ceiling, not a RAM ceiling. When a solver fails at a\n suspiciously round problem size, check the index types before adding more RAM.\n
\nCalibration result
\n\n Every point sits below the diagonal. On this dataset, the estimator is\n conservative everywhere.\n
\nThe calibrated model removed the cases where the first attempt would have OOMed. A few lessons from this work carry beyond mode solvers:
\nmaxtasksperchild=1 where it matters.\n Get the memory estimate wrong, and you pay for it in retries, latency, or wasted RAM.
", "attachments": [ { "url": "https://engineering.flexcompute.com/articles/mode-solver-memory-calibration.md", "mime_type": "text/markdown", "title": "Predicting Peak Memory for an Electromagnetic Mode Solver markdown" }, { "url": "https://engineering.flexcompute.com/images/og/mode-solver-memory-calibration.png", "mime_type": "image/png", "title": "Predicting Peak Memory for an Electromagnetic Mode Solver social image" } ], "_flexcompute": { "kind": "Case Study", "tags": [ "photonics", "tidy3d", "verification" ], "markdown_url": "https://engineering.flexcompute.com/articles/mode-solver-memory-calibration.md" } }, { "id": "https://engineering.flexcompute.com/articles/autoresearch-photonic-design/", "url": "https://engineering.flexcompute.com/articles/autoresearch-photonic-design/", "title": "Can AI Agents Autonomously Design Components on Photonic Chips?", "summary": "We gave AI agents a photonic simulator, a DRC engine, and four design challenges. They autonomously designed waveguide bends, crossings, splitters, and demultiplexers — some reaching near-perfect performance.", "image": "https://engineering.flexcompute.com/images/og/autoresearch-photonic-design.png", "banner_image": "https://engineering.flexcompute.com/images/og/autoresearch-photonic-design.png", "date_published": "2026-04-13T00:00:00.000Z", "date_modified": "2026-04-13T00:00:00.000Z", "authors": [ { "name": "Tyler Hughes", "path": "/authors/tyler-hughes/", "url": "https://engineering.flexcompute.com/authors/tyler-hughes/" } ], "tags": [ "Photonics", "Inverse Design", "Optimization", "AI Agents" ], "content_html": "import DesignLoopFigure from '../../components/DesignLoopFigure.astro';
\nInspired by the recent hype around autoresearch loops with AI agents, I decided to explore a simple question:
\n\n\nIf you give agents access to a well-defined photonic component design problem, a local electromagnetic simulator, and a fabrication constraint checker, could they reliably design photonic devices on their own, just by iterating on geometries?
\n
To my surprise, the answer was a resounding yes! The agents were able to create a handful of common photonic components, with designs that met our performance criteria and were simple and intuitive. There were some caveats and surprising results though, which we'll get into later. For now, let's dive into the problem setup and show what the agents came up with.
\nPhotonic integrated circuits route light through structures etched on a silicon chip, much like electronic circuits route current through wires. Light can carry far more information than electronic signals and with very low power loss. As a result, photonic technologies are playing an increasingly important role in improving performance of computing platforms for AI and other applications. These photonic devices are composed of many components, such as waveguides, splitters, filters, etc., which help guide the light around the chip. Engineering these components is hard because, unlike with electrons, the wave nature of light needs to be accounted for in the design. Even small geometry changes can scatter light, create unwanted reflections, or couple power into the wrong mode.
\n\nA standard approach is to use a simulator for Maxwell's equations to model and optimize the device. One such algorithm is FDFD (finite-difference frequency-domain), which discretizes Maxwell's equations on a grid and solves for the steady-state field pattern at a given frequency. You define the geometry and materials, pick a wavelength, launch light into the simulation, and measure what comes out the other end. The figure of merit (FoM) is typically a mode overlap, which measures how much of the output light matches the desired profile on the other side of the device.
\nOne can use simulation combined with optimization techniques like gradient descent or parameter sweeps to come up with a candidate device. But a device that looks great in simulation may be impossible to fabricate since these structures are so small (typically features are measured on the tens of nanometers scale). Foundries impose design rule checks (DRC): minimum feature width, minimum gap between features, minimum area, no tiny holes. A design that scores well in simulation but violates DRC is useless. In our case, we chose a minimum feature size of 300 nm — more relaxed than typical foundry PDKs, which often enforce 150–200 nm, but strict enough to impose meaningful constraints.
\n\n\nTherefore, the real problem for any photonic device designer is: find a geometry that maximizes the photonic objective while passing foundry design rule checks!
\n
We wanted to explore whether we could automate this design process, with a simulator, a DRC engine, and a few well-defined design problems.
\nFor a given design problem, each agent got the same interface:
\nGiven the figure of merit and acceptance criteria, we evaluate the submission and instruct the agent to continue iterating until a suitable design is found. The challenge is defined in a repository that makes it reusable: a challenge wrapper, evaluator, autoresearch harness, and leaderboard. The whole setup was designed so that any agent can enter this directory, read the prompt, and begin submitting solutions. As the agents work, they create visual artifacts: geometry visualizations, electromagnetic field profiles. The agents can see each other's previous submissions, in fact they often used them!
\nLet's walk through what happened. We ran two agents, Claude Opus 4.6 and OpenAI GPT 5.4, on four challenges of increasing difficulty — from a simple waveguide bend up to a three-channel wavelength demultiplexer. Both agents could see each other's submissions.
\nWe often need to steer light around on photonic circuits, so having the ability to bend light on chip is a critical functionality. However, if you make such a bend too sharp, the light will tend to leak out.
\nProblem: Route light from a horizontal waveguide to a vertical waveguide through a 90-degree turn in a 3.0×3.0μm design region.
\nFoM: What fraction of light makes it to the vertical output waveguide? Perfect score = 1.0.
\nStarting point: A naive, circular arc bend with matching sections and a corner pad, FoM = 0.904.
\nThe agents first performed a general test of several different candidate geometries. Claude tested 14 diverse geometries (Euler bends, corner mirrors, MMI regions (\"multi-mode interference\" regions), chamfered L-bends) and noticed that simply widening the arc from 0.50 to 0.70 μm beat everything else. GPT arrived at the same conclusion through a more incremental path, starting by tuning the size of rectangular pads at the bend junctions and progressively stripping away decorative geometry until we were left with a simple bend again with a variable width.
\nWith the basic strategy in place, both agents naturally progressed to a second stage: fine tuning the geometry through parameter sweep. Both agents independently swept the width of the bend and found that 0.75–0.80 μm was the sweet spot for a uniform-width arc with a corner pad. But things got interesting as they tried pushing further and ran into constraints.
\nDRC Violation: Wider arcs without the pad gave raw FoM up to 0.977, but failed DRC because the gap between the wider arc and the 0.50 μm arm waveguides at the junction dropped below 300 nm. Both agents hit this wall and both found the same way around it.
\nSolution - Variable-width arcs: Instead of a uniform-width arc, the agents decided to use one that starts at the arm width (0.50 μm) at the ports and smoothly widens at the bend apex. This solves two problems simultaneously: no DRC gap at the ports (width matches the arms exactly) and reduced radiation at the apex (wider waveguide confines the mode better).
\nClaude used a sinusoidal width profile and found the optimal apex width at 0.65 μm. GPT used a power-law profile with a tunable exponent and found a similar optimum at 0.648 μm with exponent 0.79. Both reached FoM ≈ 1.0 — effectively perfect transmission in the 2D single-wavelength simulation.
\nBoth agents achieved FoM ≈ 1.0 with a single bare polygon. The winning design family is a variable-width quarter-circle arc: w(θ)=wport+Δw⋅f(θ) where f peaks at the 45-degree apex and returns to zero at the ports.
\n\n\nAnd just to reiterate: this was all done 100% autonomously! The agents came up with the candidate geometries, swept parameters, and corrected the DRC violations all without intervention.
\n
This was about as simple as it gets for a design problem, so the result is interesting but not groundbreaking: the agents found a good parameterization through experimentation and then swept parameters to optimize. Next we tried some harder problems.
\nOften when designing complex circuits, we need waveguides to cross so we can connect components together. Since this is all happening on a single plane, we need crossing components that allow the light to pass through without any power leaking into the perpendicular section.
\nProblem: Pass light straight through a 3.0×3.0μm intersection where two orthogonal waveguides cross, minimizing crosstalk into the perpendicular waveguide. The design should have 90-degree rotational symmetry.
\nFoM: 0.5(Th+Tv)−0.25(XTh+XTv). Basically balancing through transmission and adding an explicit extra penalty for crosstalk. Perfect score = 1.0.
\nStarting point: A simple plus-shaped cross (two perpendicular waveguides), FoM = 0.770.
\nClaude tested circles, diamonds, ellipses, MMI regions, tapered transitions, and variable-width crosses. Every expansion at the intersection made things worse: circle (0.375), diamond (0.553), ellipse (0.517), square MMI (0.623).
\nGPT started by adding a wider central cross and found that this solution worked well off the bat. After it tuned parameters, the optimal uniform width was found to be right around 0.75 μm (FoM = 0.962).
\nThe sweep curve shows that the device is quite sensitive: 0.74 or 0.76 drops the FoM by 0.003. From looking at the field plots, we can see that this is because of a self-imaging condition in the center intersection region: the width is chosen so that the light in the junction interferes constructively at the output after crossing the perpendicular waveguide. This interference is inherently sensitive to any changes in the geometry, so it's not a great design.
\nAfter some more prodding, GPT then found an improvement: adding another ~0.95 μm square dielectric pad at the center of a 0.74 μm cross (FoM = 0.968). Claude independently converged to the same design family, achieving 0.968 with a Gaussian-flared cross and a 0.96 μm pad. The two agents tied within solver noise.
\nFoM = 0.968 (both agents). The optimal crossing is a 0.74 μm wide cross with a square center pad. Unlike the bend, variable-width (polygon) approaches didn't help — the crossing relies on a self-imaging condition that requires a well-defined multimode section. The center pad is the only refinement that provides a measurable improvement.
\n{/* Winning crossing report image not included in source zip */}
\n\n\nThe takeaway from this: the crossing problem is almost entirely solved by only three parameters: the sizes of the rectangles in the design region. This shows that devices can work well with just a small number of good knobs and disciplined iteration — and the agents figured that out on their own.
\n
It is important to be able to split the power in a waveguide into two or more arms so we can do routing or interfere the light with itself later on. A 1×2 splitter is a very common photonic component that seeks to do a 50-50 split with very low loss.
\nProblem: Split light from one input waveguide equally into two output waveguides separated by 2.0 μm, in a 4.0×4.5μm design region.
\nFoM: 2Ttop⋅Tbottom, rewarding balanced 50/50 splitting via the geometric mean. Perfect score = 1.0.
\nStarting point: A big rectangle filling the design region, FoM = 0.507.
\nThis was the hardest challenge of the three (so far):
\nBoth agents immediately tried Y-branches: simply taper the input section into two curved output arms. Claude's smooth Y-branch with cubic Hermite S-curves got FoM = 0.966, nearly perfect. GPT's initial Y-branches were similarly strong. But they all failed DRC. The two arms at the split point have zero gap, violating the 300 nm minimum.
\nThe agents then struggled to fix the DRC violations; they both pre-separated the arms by 300+ nm at the split point and added tapers to feed both arms. The forced gap scattered light because the single-lobed taper mode didn't match the two-lobed arm mode. Claude's best separated Y reached FoM = 0.812 after sweeping the split point position.
\nGPT continued iterating on the separated Y family and discovered a parameterization that Claude missed. After the split, it kept the arms wide for a short section before tapering them back to the output width. This let the wider arms better capture the splitting mode before narrowing down. GPT also tuned the arm width independently (0.57 μm instead of 0.50), brought the gap size down to the minimum of 300 nm, and added a width-exponent parameter controlling how aggressively the arms taper.
\nThe result was FoM = 0.906, which was a significant jump past Claude's 0.812 and into territory that Claude couldn't reach with the simpler parameterization.
\n{/* GPT hold-and-taper report image not included in source zip */}
\nClaude also tried:
\nBut none of these matched GPT's result.
\nFoM = 0.906 (GPT's hold-and-taper Y-branch) was the best DRC-clean result. Claude's best was 0.812 with a simpler parameterization. The raw (DRC-violating) Y-branch reached 0.966 for both agents. The 300 nm DRC gap constraint at the split junction was a major challenge.
\nThe splitter was fundamentally different from the bend and crossing. Both the bend and crossing were single-input single-output problems where light stays in one continuous structure. The splitter requires light to transition from one waveguide to two separate waveguides, and DRC forces a minimum gap at that transition.
\nThe gap between DRC-clean and DRC-free performance (0.906 vs. 0.966) was where the real engineering challenge lives. GPT's hold-and-taper design showed that there's still room for better parameterizations: the splitter rewarded more knobs than the bend or crossing did. This is the device where the agents' search strategies diverged the most, and where GPT's more incremental, parameter-by-parameter approach paid off.
\nThe bend, crossing, and splitter were all single-wavelength, single-mode problems where the agent proposes a shape, gets a score, and iterates. They could be solved by parametric sweeps in low-dimensional design spaces where one or two dominant parameters (bend width, crossing width, split position) determine most of the performance.
\nNext, we tried a more challenging problem involving wavelength selectivity, where the conventional geometric approaches failed miserably.
\nMany photonic devices end up splitting information into different wavelength channels (wavelength division multiplexing), which allows them to process more information in parallel. Being able to split and recombine these wavelengths is a major challenge. In this design challenge, we had a single input containing light with a mix of three different wavelengths (1.45, 1.55, 1.65 μm) and tried to design a single device to direct each of these wavelengths to one of three separate output ports (top, center, bottom) through a 10.0×10.0μm design region.
\nFoM: Average normalized transmission into the correct output port across all three channels. Perfect score = 1.0 (each wavelength fully routed to its target).
\nEvery parametric approach, whether proposed by the agents or designed by hand, scored below 0.5:
\nThe approach that (somewhat) worked was to break the design region into a grid of pixels, each with a material density between 0 and 1. Think of this as similar to a \"material density\" image. Starting from uniform grey (0.5), gradient-based optimization iterated on the pixels for 60 steps of Adam optimization and reached FoM = 0.935, an approach similar to training ML models. The optimizer found a complex density pattern that creates different interference pathways for each wavelength, routing each to its correct output port.
\n\n\nBut this design did not pass DRC. The pixel-based structure contains sub-wavelength features, tiny gaps, and isolated islands that no foundry would accept. And it was not binarized properly.
\n
We'll explore this approach in a follow-up blog. But the takeaway was that the demultiplexer seemed to be a problem too challenging for the agents to tackle while passing DRC at this stage.
\n| Challenge | \nBaseline | \nGeometric Parameters | \nPixel-Based Density | \nDRC | \n
|---|---|---|---|---|
| Bend | \n0.904 | \n≈1.0 | \nN/A | \nPASS | \n
| Crossing | \n0.770 | \n0.968 | \nN/A | \nPASS | \n
| Splitter | \n0.507 | \n0.906 | \nN/A | \nPASS | \n
| 3-ch WDM demux | \n0.016 | \n0.315 | \n0.935 | \nFAIL | \n
For the bend and crossing, Claude and GPT arrived at the same optimal design families. The variable-width arc for the bend and the widened cross for the crossing exploit properties of the physics that a photonic designer could understand.
\nHowever, an important caveat: the agents could see each other's previous submissions throughout the challenge, and they used this. I found Claude regularly taking inspiration from GPT's designs and proposing variations on them. So the convergence is not purely independent. The shared leaderboard acted as a forum for them to communicate. The fact that both agents ended up at the same designs is encouraging, but it's weaker evidence than if they had converged in complete isolation. A cleaner experiment would run each agent in a separate sandbox with no visibility into the other's submissions.
\nThe splitter was the exception. GPT's hold-and-taper parameterization (six tunable parameters: split position, gap, arm width, center exponent, hold fraction, width exponent) explored a richer design space than Claude's three-parameter separated Y. The extra degrees of freedom turned out to matter: they let the optimizer find designs that better balance the physics of mode splitting against the DRC gap constraint.
\nIn all three challenges, the best raw designs (ignoring DRC) significantly outperformed the best DRC-clean designs. The fabrication constraint is what limits performance. Both agents spent more iterations working around DRC failures than optimizing the actual photonics. A lot of the failed designs were not bad ideas, they were ideas that did not survive the interaction between wave physics and fabrication constraints. It is important therefore to consider a systematic way to incorporate DRC validity into the geometry generation process.
\nThe bend champion is only one polygon. The crossing champion is two rectangles. The splitter champion is three polygons. Every attempt at complex multi-structure designs (corner mirrors, MMI regions, directional couplers, resonant cavities) performed worse than simple, well-tuned geometries. For simple problems, the best solutions were not inventive shapes but rather simple parameterizations.
\nOn the simpler problems (bend, crossing), hand design with parametric sweeps found near-optimal solutions. But on the harder problem, it was insufficient. Exploring the full space of pixel-based density optimization seemed to be the more promising path forward here. To be fair, the size of the design region is one variable where we could play with to increase the possible design degrees of freedom for simple parameterizations.
\nThe clearest pattern from these experiments: agents excel at low-dimensional parametric search with fast, scalar feedback. Give them a figure of merit, a few geometric knobs, and DRC guardrails, and they will systematically explore the space and do a pretty good job.
\nBut the demux shows the boundary. As problems get harder, agents need exposure to more advanced parameterizations and techniques based on pixel-based optimization.
\n\n\nThe interesting frontier for agent-based exploration may be between these two extremes. For example, allowing agents to systematically explore different parameterizations for problems that have perhaps a handful of design degrees of freedom. Too hard for a human designer to solve but also not too hard that it requires full pixel-based optimization. Suggestion: use autoresearch loops to automate the sweep and exploration process for photonic device design.
\n
Can agents learn to guide topology optimization and post-process the results into manufacturable designs? That's the next question we want to answer.
", "attachments": [ { "url": "https://engineering.flexcompute.com/articles/autoresearch-photonic-design.md", "mime_type": "text/markdown", "title": "Can AI Agents Autonomously Design Components on Photonic Chips? markdown" }, { "url": "https://engineering.flexcompute.com/images/og/autoresearch-photonic-design.png", "mime_type": "image/png", "title": "Can AI Agents Autonomously Design Components on Photonic Chips? social image" } ], "_flexcompute": { "kind": "Essay", "tags": [ "photonics", "inverse-design", "optimization", "agents" ], "markdown_url": "https://engineering.flexcompute.com/articles/autoresearch-photonic-design.md" } }, { "id": "https://engineering.flexcompute.com/articles/photonic-inverse-design-45-lines/", "url": "https://engineering.flexcompute.com/articles/photonic-inverse-design-45-lines/", "title": "Designing a Photonic Chip Component with ~45 Lines of Python", "summary": "A compact introduction to photonic inverse design with Tidy3D, using a pre-built simulation and a ~45-line optimization loop.", "image": "https://engineering.flexcompute.com/images/og/photonic-inverse-design-45-lines.png", "banner_image": "https://engineering.flexcompute.com/images/og/photonic-inverse-design-45-lines.png", "date_published": "2026-03-05T00:00:00.000Z", "date_modified": "2026-03-05T00:00:00.000Z", "authors": [ { "name": "Tyler Hughes", "path": "/authors/tyler-hughes/", "url": "https://engineering.flexcompute.com/authors/tyler-hughes/" } ], "tags": [ "Photonics", "Inverse Design", "Optimization", "Tidy3D" ], "content_html": "Photonic chips guide light through tiny waveguides etched into silicon, much like electrical wires carry current on a circuit board. These chips are increasingly important for high-speed data links, sensing, and quantum computing. Routing light around corners is surprisingly hard: a smooth 90-degree waveguide bend often needs a radius of several microns to keep loss low, and that takes up valuable chip real estate. What if a computer could design a structure that makes the bend in a fraction of the space?
\nThat's the idea behind inverse design: instead of designing a device and checking if it works, you specify what you want and let an algorithm figure out the geometry, pixel by pixel, using the same gradient-based methods that train neural networks.
\nThe idea isn't new. Structural engineers have used topology optimization to design bridges and aircraft parts since the 1980s, and Sigmund's \"99 line topology optimization code\" showed the core algorithm fits in a single MATLAB script. This post does the same for photonic inverse design: once the base simulation is given, the core optimization loop fits in ~45 lines of Python.
\nLet's build it.
\nAt a glance
\nRoute 1.0 μm light around a 90-degree bend inside a 3x3 μm design region.
\nUse Tidy3D's adjoint gradients to optimize a pixelized material layout with Adam.
\n\n In ten iterations, the design routes roughly 89% of the power into the desired output mode.\n
\nLight travels through a waveguide, a thin strip of high-refractive-index material (like silicon) surrounded by a lower-index material (like air). The light is confined to the strip by total internal reflection, similar to how fiber optics work.
\nWe want to route light at wavelength 1.0 μm around a 90-degree corner. It enters horizontally from the left and must exit vertically downward. Between input and output sits a design region, a 3x3 μm square where the optimizer can freely place or remove material. The question is: what pattern maximally routes the light from input to output?
\nTo simulate how light propagates through a given geometry, we use Tidy3D, a cloud-based electromagnetic solver. Given a device geometry and material properties, Tidy3D solves Maxwell's equations and tells us where the light goes. Crucially, Tidy3D exposes an autograd-based inverse-design workflow, which lets us compute gradients through the simulation (more on this in Step 3).
\nThe base simulation (waveguides, light source, output monitor, and absorbing boundary conditions) is pre-built and stored in sim_base.yaml. We load it and focus entirely on the optimization algorithm.
import autograd\nimport autograd.numpy as np\nimport tidy3d as td\nimport tidy3d.web as web\nfrom tidy3d.plugins.autograd import make_filter_and_project\n\nsim_base = td.Simulation.from_file(\"sim_base.yaml\")\nWe need a function that maps a set of design variables to a complete electromagnetic simulation. Each pixel in the design region gets a variable ρ, a number between 0 and 1. We then convert ρ to a permittivity value. Permittivity is the square of the refractive index and controls how light interacts with the material. Our material has refractive index n=2, so its permittivity is n2=4. Air has permittivity 1.
\nBut we don't map ρ to permittivity directly. Two transformations happen first:
\nA convolutional filter with radius R blurs each pixel's value with its neighbors. This is important because these devices will eventually be fabricated, and real manufacturing processes have a minimum feature size they can reliably produce. The filter ensures no feature in our design is smaller than R, acting as a simple proxy for more sophisticated fabrication-aware design checks. We use R=150nm.
\nAfter filtering, a tanh function pushes the smoothed values toward 0 or 1, controlled by a sharpness parameter β. At low β, the mapping is nearly linear, so the optimizer can explore intermediate values freely. At high β, it becomes a hard threshold that forces every pixel to be pure material or pure air.
\nThe figure below shows the effect of these two transformations applied to random noise. Filtering smooths out fine features; projection pushes values toward binary. When applied during optimization, they guide the optimizer toward clean, fabricable geometries.
\nn_mat = 2.0 # material refractive index\neps_mat = n_mat ** 2 # permittivity = n^2 = 4.0\ndesign_size = 3.0 # design region side length (um)\npixel_size = 1.0 / 50 # pixel resolution (um)\nradius = 0.150 # filter radius R (um), sets minimum feature size\nnx = ny = int(design_size / pixel_size)\n\ndesign_region_geo = td.Box(center=(0, 0, 0), size=(design_size, design_size, td.inf))\n\nfilter_project = make_filter_and_project(radius=radius, dl=pixel_size)\nNow we construct the function that takes our design parameters ρ, applies the filter and projection to get a permittivity map, builds a structure from it, and adds it to the base simulation we loaded from file.
\ndef make_sim(params, beta):\n \"\"\"Map design variables through filter, projection, and into a simulation.\"\"\"\n density = filter_project(params, beta=beta)\n eps_data = 1.0 + (eps_mat - 1.0) * density\n structure = td.Structure.from_permittivity_array(\n eps_data=eps_data, geometry=design_region_geo,\n )\n return sim_base.updated_copy(\n structures=list(sim_base.structures) + [structure],\n )\nWe need a single number that tells us how well the device works. At the output waveguide, Tidy3D measures the mode amplitude a, a complex number describing how much light couples into the waveguide's guided mode. The power carried by that mode is ∣a∣2, so our figure of merit is simply the output mode power:
\nFOM=∣a∣2\nA perfect device would have FOM=1 (all input power reaching the output). In code: we build a simulation from our design variables, run it on the cloud, extract the mode amplitude, and return the power.
\ndef objective(params, beta):\n \"\"\"Run electromagnetic simulation and return output mode power.\"\"\"\n sim = make_sim(params, beta)\n data = web.run(sim, task_name=\"invdes\", verbose=False)\n amps = data[\"mode\"].amps.sel(direction=\"-\", mode_index=0).values\n return np.sum(np.abs(amps) ** 2)\nTo optimize, we need the gradient dFOM/dρ for every pixel: how does tweaking the ρ value of each pixel affect the output power? The brute-force approach would perturb each pixel one at a time and re-simulate. For our 150x150 grid, that's 22,500 simulations per optimization step. Completely impractical.
\nThe adjoint method computes the exact same gradient using just two simulations, regardless of how many pixels there are:
\nAfter both simulations, the gradient at each pixel is simply the overlap of the forward and adjoint electric fields:
\n∂εi∂F∝Re(Ei⋅Eiadj)\nTwo simulations instead of 22,500. Intuitively, the forward field tells you \"how strongly does this pixel interact with the input light?\" and the adjoint field tells you \"how strongly does this pixel influence the output?\" Their product gives the sensitivity of the objective to each pixel.
\nThis is the same principle behind backpropagation in neural networks. The adjoint simulation is the vector-Jacobian product (VJP) of the forward electromagnetic solve, and both exploit the chain rule to avoid redundant computation (see Minkov et al., 2020 for a detailed treatment connecting adjoint methods and automatic differentiation in photonics).
\nTidy3D implements the adjoint math as the VJP of its electromagnetic solver, so this second simulation happens automatically behind the scenes. When we wrap our objective in autograd.value_and_grad, Tidy3D runs both the forward and adjoint simulations and backpropagates the gradient through the entire computational pipeline (simulation, mode decomposition, filter, projection, and all).
val_and_grad = autograd.value_and_grad(objective)\n# val_and_grad is a function: given (params, beta), it returns (fom, gradient)\n# e.g. fom, grad = val_and_grad(params, beta=10)\nWith cheap gradients in hand, we can use any gradient-based optimizer. The autograd library provides Adam out of the box. Adam is the same optimizer that trains most neural networks: it maintains running averages of the gradient (momentum) and its square (adaptive learning rate), giving more stable convergence than plain gradient ascent.
Adam's grad function takes (params, iteration). We use the iteration number to gradually increase β: early on, a low β keeps the design continuous, giving the optimizer freedom to explore many possible solutions; as the design matures, we ramp β up to push toward a binary (material or air) structure that can actually be fabricated. We also negate the gradient so that Adam maximizes our objective instead of minimizing.
from autograd.misc.optimizers import adam\n\nn_steps = 10\nparams0 = 0.5 * np.ones((nx, ny, 1))\nhistory, param_history = [], [np.array(params0)]\n\ndef neg_grad(params, i):\n \"\"\"Negative gradient with projection schedule (Adam minimizes, we negate to maximize).\"\"\"\n params = np.clip(params, 0, 1)\n beta = 5 + 45 * i / max(n_steps - 1, 1)\n fom, g = val_and_grad(params, beta)\n history.append(float(fom))\n param_history.append(np.array(params))\n print(f\" step {i:2d} | FOM = {fom:.4f} | beta = {beta:.1f}\")\n return -g\nStarting from a uniform design (ρ=0.5 everywhere, halfway between air and material), the optimizer discovers the structure from scratch. Each step runs two simulations on the cloud (forward + adjoint), computes the gradient across all 22,500 pixels, and updates the design.
\nparams_opt = np.clip(adam(neg_grad, params0, num_iters=n_steps, step_size=0.3), 0, 1)\n step 0 | FOM = 0.0022 | beta = 5.0\n step 1 | FOM = 0.0531 | beta = 10.0\n step 2 | FOM = 0.3975 | beta = 15.0\n step 3 | FOM = 0.3932 | beta = 20.0\n step 4 | FOM = 0.6367 | beta = 25.0\n step 5 | FOM = 0.7139 | beta = 30.0\n step 6 | FOM = 0.7795 | beta = 35.0\n step 7 | FOM = 0.8487 | beta = 40.0\n step 8 | FOM = 0.8861 | beta = 45.0\n step 9 | FOM = 0.8856 | beta = 50.0\nThe final design looks nothing like what a human engineer would draw. There's no smooth curve, no gradual taper. Instead, the optimizer found a pattern of material and air that manipulates the electromagnetic field through interference to route the light around the corner.
\nWatch the design emerge from a uniform gray starting point. Early steps explore broadly (low β, soft features); later steps sharpen into a clean binary design (high β).
\nThe animation shows how the geometry sharpens visually. The convergence trace shows the same story numerically: rapid early gains, then diminishing returns as the design binarizes.
\nZooming out, each optimization step does the same five things in order:
\n01
\nMap the raw design variables into a smooth, increasingly binary material distribution.
\n02
\nInsert that material layout into the pre-built Tidy3D model of the waveguide bend.
\n03
\nCompute the output-mode power, which becomes the figure of merit.
\n04
\nBackpropagate sensitivity information through the electromagnetic simulation.
\n05
\nUse Adam to take a gradient step, then repeat with a slightly sharper projection.
\nEach iteration costs two electromagnetic simulations. The adjoint method makes this feasible. Without it, we'd need 22,500 simulations per step instead of 2.
\nThis post is a basic introduction to the technique, but a functional one. There are many more advanced variations for real problems in photonic device design. Production systems add:
\nIf you're interested in going deeper, check out these resources:
\nWant the exact working files for this post? Download the companion bundle. It includes the notebook, the Jupytext script export, sim_base.yaml, and the helper script that rebuilds the base simulation. Re-running the optimization requires a Tidy3D account.
import HarnessWorkflowFigure from '../../components/HarnessWorkflowFigure.astro';\nimport MutationPipelineFigure from '../../components/MutationPipelineFigure.astro';
\nI have been building with AI coding agents for over a year.1 Most of that time was unstructured: I used whatever worked, fixed problems as they came up, and did not pay much attention to the patterns. Then, starting around November 2025, I spent three months building a mostly personal internal autodiff library: graph-based, eagerly traced, NumPy-integrated, and almost entirely agent-written. By the end, it had reached roughly 80,000 lines of Python.
\nI wanted a graph-first, NumPy-native library where the graph itself was explicit, inspectable, and available for transformations beyond autodiff. JAX is excellent, but it optimizes for a different set of tradeoffs: staged execution, JIT compilation, and the accelerator stack.
\nOver those three months, my role shifted from reviewing every line of agent output to opening many sessions with a single question: \"what should we work on next?\" That shift did not happen only because the agents improved. It happened because the verification infrastructure improved enough that I could trust the floor: a minimum quality bar would hold regardless of what the agent produced.
\nThis article is about that infrastructure — what it looks like, how it grew, and why it matters more than you might think. But the infrastructure did not build itself. Every check exists because a human saw a failure and decided to formalize the fix. The code is not open source, so the point of this article is not the artifact itself, but the verification patterns and harness design that emerged while building it.
\nThis library did not start with agents. I had been sketching it on the side for a while — a design document, a basic tracing engine, some scaffolding. By the time I opened the first AI session, the repo had about a thousand lines of Python and a clear picture of what the library should be. What it did not have was momentum. A side project I picked up between other work, never for long enough to get past the foundation.
\nThe first session was a comparison: how did the library stack up against autograd and MyGrad?2 I was using the agent as a reviewer, asking for architectural opinions. When the agent got the target API wrong — proposing explicit tracing contexts when the library was designed for implicit tracing — I corrected it. The agent was a consultant. I was the authority.
\nThe second session set up CI: ruff with all rules enabled, mypy strict, Hypothesis property-based testing, conventional commits, and AGENTS.md.3 None of this was about agent harnesses; I did not even know the term yet. Engineering discipline and personal curiosity, which turned out to benefit agents enormously.
\nWhat followed was a burst of rapid development. Over a handful of intense sessions, the codebase grew from a small prototype into something much larger: graph optimization passes, workflow orchestration, xarray integration, debugging and visualization tools. But the derivative engine remained the most verification-intensive part, and the one that drove most of the harness. My workflow settled into a ritual:
\n\n\n\"review the current state of the repo, what do you think would be highest leverage to work on next?\"
\n
Let the agent propose priorities. Select or redirect. Ask for a detailed implementation plan with user stories, test strategy, and acceptance criteria. Review it. Paste it back with: \"PLEASE IMPLEMENT THIS PLAN.\"4
\nThe workflow was productive but not self-correcting. Pushing back mattered — and I had to push back constantly.
\nOn VJP coverage, the agent claimed near-complete parity with autograd. I was skeptical; autograd had far more VJPs than that.5 I was right — the comparison was incomplete. Compatibility shims revealed a similar pattern: the agent kept proposing backward-compatibility layers for a greenfield project with no users.
\nThe shims deserve their own mention because they are so universal. Old behavior kept via fallbacks and re-exports, new features built on top, instead of clean breaks. Even with explicit instructions, the agents reached for shims.6 If you are building anything with coding agents, you will hit this.
\nFiles bloated too. Core modules grew past 1,800 lines. Test files reached similar sizes. A vicious cycle: longer files fill agent context faster, the agent gets worse at navigating the codebase, the code gets worse, the files grow further. The 500-line file limit I eventually imposed came directly from that pain.
\nOne moment captures why human judgment still matters in this workflow.
\nAutodiff frameworks need derivative rules for every operation, typically written as either VJPs (reverse-mode pullbacks) or JVPs (forward-mode pushforwards).7 This library had accumulated reverse-mode rules first, like most frameworks. In the middle of a session about expanding derivative coverage, the agent proposed deriving JVP rules from the existing VJP rules. That sounded reasonable.
\nThen I asked the question that changed the architecture: \"Doesn't it scale better to make JVPs the default source of truth and derive VJPs from them where possible, rather than vice versa?\"
\nFor this codebase, yes. JVPs were the cleaner authoring primitive for much of the covered NumPy surface, and reverse-mode pullbacks could often be synthesized through the transpose machinery. Some operations still needed explicit reverse-mode exceptions for correctness or performance, but the default direction was backwards from what the agent proposed. Roughly twenty commits in a single day shifted the covered NumPy surface to a JVP-first policy.
\nLater that evening, I checked the agent's work again: if JVPs were now the default source of truth, why did the rule layout still look overwhelmingly reverse-mode? The file structure told a different story than the runtime claims. The agent had wired the runtime correctly, but too much of the old reverse-mode formula structure was still in place. It took another full session — running overnight, largely autonomously — to make the migration real in the authored rule layout, not just the plumbing.
\nA well-defined mathematical problem, clear correctness criteria, and a human catching both the architectural direction and the incomplete execution. If I hadn't asked, I probably wouldn't have caught it until the derivative layer was deeply entrenched — and unwinding it would have been painful.
\nThe JVP migration went (reasonably) well because the problem was mathematically well-defined. Most problems aren't. Every piece of verification infrastructure that followed was a response to something that went wrong.
\nThe first tool was pre_pr.sh — a small shell script born from watching the agent skip steps with each PR. It was the \"unenforced verification\" failure mode from the first article, playing out in real time. The agent would forget to run mypy, or skip the VJP coverage check, or not rebuild docs. The script was my first attempt at \"one command to verify everything\": check for a clean working tree, rebase on main, run the linter, type checker, tests, coverage checks, grad contracts, and docs build, in sequence. If any step failed, the whole thing failed.
The shell script could not keep up, so I replaced it with quality.py — a Python CLI consolidating the scripts into a single framework.
But the commands to run verification differed across AGENTS.md, CI workflow files, and docs.8 And nothing was scoped: every check ran against the full codebase regardless of what changed.
\nEach tool was a response to pain.
\nThe architecture split into layers: a backend-agnostic core, a numpy integration layer, and a grad package for autodiff. The intent was clean separation — core should have zero knowledge of NumPy, so adding a CuPy or JAX backend later would be more tractable. It also gave agents clear lanes to work in.
On the day this layered architecture was declared, the boundary checker caught its first violation within hours. Then another. The architecture was established, violated, fixed, violated again, and fixed again. The original checker only caught cross-layer internal-package imports; it missed bare import numpy entirely, so I had to extend it, again and again.
The boundary checker ran continuously, but it only caught what it knew to look for. Ten weeks later, I manually scanned what the agent had built in the gradient package and found dozens of files with import numpy as np, well over a thousand np.* callsites, and a package that would not run without numpy installed even though its pyproject.toml declared no numpy dependency. After a big-bang refactor, the agent introduced a backend-agnostic proxy module — but exported the proxy object as np, not xp. Those files then did from ..._backend_runtime import np. Syntactically different from import numpy as np. Visually identical.
Each round taught me to probe deeper. Was there any code in core that imported numpy, even lazily? Any mention of numpy in core, even as a variable name or string literal? There were — hardcoded \"numpy\" string literals, prefix-coupled registration, multiple compatibility wrapper modules. Every question I learned to ask was a check I should have automated earlier.
Boundary enforcement was about rules within sessions. The next problem was continuity between them.
\nI tried to scale development with sub-agents. In Claude Code, I built seven specialized agents: dispatch, test-gen, quality, architect, numpy-protocol, docs, debug. An agent team, each specialized on a domain. It did not stick. I burned a lot of tokens, but the output quality was arguably worse than if I had stuck with a single agent and manual steering. I think the problem was context: each agent starts from only a prompt and has to derive all its context from there. Handoff documents either included too little — and the receiving agent made wrong assumptions — or too much — and the agent could not distinguish signal from background.
\nWhat did work was something simpler. Near the end of a long session, I was about to ask the agent to continue with the next phase. Instead, I asked it to write the prompt I should use for that next phase.
\nThe agent generated a detailed handoff prompt — project state, remaining work, constraints, validation commands — that I would then curate and paste into a fresh session. It was a \"relay\" pattern, where the prompt is a compressed representation of what matters: what was just done, what remains, what constraints apply.9
\nWithin days, I was running multiple agents in parallel. Each session got its own git worktree.10 I broke tasks down into work streams and dispatched them: \"Give me a prompt for each of these work streams, and tell me which ones I can kick off in parallel.\" Without quality.py running in each worktree, parallel agents would have been utter chaos. But now, each agent could independently verify its own changes, enabling a workflow that would not have been sustainable otherwise.
Still, the quality gate turned red.11 The jvp_grad_runtime_ratio had drifted above the 10.0x threshold because of measurement noise on higher-order workloads. It was effectively blocking merges unrelated to performance.
Around the same time, \"harness engineering\" was becoming a more explicit frame for what many teams were converging on. OpenAI had published their article on the topic, and others were describing similar patterns.12 The general idea is simple: agent reliability comes from the environment, not just the model, and verification infrastructure deserves primary investment. Seeing the pattern named made the next step clear.
\nI took a day off — no sessions, no commits. Then I decided to do it properly.
\nThe next session started with a single instruction: review the repo in light of OpenAI's harness engineering article and suggest how it should be restructured. That kicked off a hard cutover. pre_pr.sh and quality.py gave way to a dedicated harness, diff-scoped mutation testing, and JSON output with bounded content for agent-friendly context windows. That was the harness turning point.
The harness is a CLI that wraps the repo's verification into three progressively broader commands:
\nuv run python scripts/harness.py loop # fast scoped, 180s budget\nuv run python scripts/harness.py mutate # diff-scoped mutation\nuv run python scripts/harness.py gate # full blocking merge gate\nThe agent no longer needs to know what verification steps exist or where they live. It runs a command and gets a go or no-go result.
\nloop is the tight inner cycle. It figures out what changed, expands through a dependency graph to find affected packages, and runs only the relevant checks — lint, type-checking, tests, quality gates — under a 180-second budget. If it runs out of time, it defers remaining checks rather than failing. In practice, this changed the agent's behavior: instead of running the full test suite after every edit — or worse, running nothing until the end — the agent started running loop after each logical change, catching issues while the context was still fresh.
mutate answers a different question: do the tests actually verify the changed behavior, or do they just execute it?
gate is the full merge requirement. All checks, no scoping, no budget. Everything must pass.
At the time of writing, the harness runs dozens of checks across those three commands. Some of the most useful exist because of specific agent behaviors. The suppression guard blocks new # type: ignore and # noqa annotations — without it, the agent's first instinct when a type check fails is to suppress the error rather than fix it. The import boundary checker enforces architectural layering with AST-level analysis.
Every command returns a JSON envelope with bounded content — at most 20 checks, 8 details per check, 240 characters per detail — so the agent's context window is not flooded with log output. A typical response looked like this (simplified):
\n{\n \"ok\": false,\n \"command\": \"harness loop\",\n \"result\": {\n \"scope\": { \"expanded_scopes\": [\"core\", \"grad\", \"numpy\", \"...\"] },\n \"checks\": [\n {\n \"id\": \"pytest_scoped\",\n \"status\": \"fail\",\n \"details\": [\"FAILED test_tracer.py::test_record - AssertionError\"]\n }\n ],\n \"summary\": { \"total\": 5, \"passed\": 4, \"failed\": 1 }\n },\n \"next_actions\": [{ \"command\": \"harness loop\", \"description\": \"Re-run after fix.\" }]\n}\nThe next_actions field is context-aware: after a successful loop, the harness suggests mutate; after a failure, it suggests the specific post-fix command.
None of these techniques are new individually. Scope-aware test selection exists in Bazel, Nx, and plenty of CI tools. Mutation testing has been around for decades.13 The pieces needed to be wired together in a specific way: JSON output bounded for agent context windows, a progression from fast-and-scoped to slow-and-complete, and every check degrading toward strictness when anything is uncertain.14
\nThe core mechanic that makes the harness practical is scope resolution — turning \"what files changed\" into \"what checks to run.\" It starts with git diff origin/main to get changed paths, matches each against a path map that routes file paths to one of nine package scopes, then expands through a dependency graph using BFS. In simplified form, the config looked like this:
[scope.path_map]\n\"packages/core/\" = \"core\"\n\"packages/numpy/\" = \"numpy\"\n\"packages/grad/\" = \"grad\"\n\"packages/grad_numpy/\" = \"grad_numpy\"\n\"packages/xarray/\" = \"xarray\"\n\n[scope.dependencies]\ncore = []\ngrad = [\"core\"]\nnumpy = [\"core\"]\ngrad_numpy = [\"core\", \"grad\", \"numpy\"]\nxarray = [\"core\", \"numpy\"]\nChange a file in packages/core and BFS expands to all dependent packages. Change only packages/xarray and only xarray checks run. Anything unrecognized — unmapped paths, missing merge-base, diff failures — falls back to the full gate.
This is what enables the 180-second loop budget. Without scoped checks, every change triggers every check — too slow for a tight agent loop. With scoping, a change to one leaf package triggers seconds of verification, not minutes.
\nCoverage measures execution. Mutation testing measures verification.
\nAn agent can (and will) write a test like this:
\ndef test_gradient_scaling():\n result = scale_gradient(x, factor=0.5)\n assert result is not None # 100% coverage, 0% verification\nThat test covers every line of scale_gradient. But flip factor > 0 to factor >= 0 inside the function, and the test still passes. It is not testing the behavior it claims to cover.
The idea came from a Slack conversation with Frederik: mutation testing is expensive on the whole codebase, but what if you scope it to the diff? Mutating only the five to twenty lines you just changed makes the cost manageable. The pipeline: find changed source lines via git diff, filter to lines actually executed by tests via coverage, generate AST-level mutations (comparison flips, boolean inversions, arithmetic swaps, constant flips), select at most twelve mutants with breadth across files, and run the relevant tests against each. If the tests still pass after a mutation, the mutant survived — the tests do not verify the behavior they claim to cover.
The policy is strict: 100% kill rate on changed lines. If any mutant survives, the PR is blocked. And require_changed_tests = true adds another constraint: if you change runtime code, you must also change or add tests. No silent runtime changes. The mutation check then verifies that those tests actually catch real behavioral differences, not just execute code paths.
Pure refactors and equivalent mutants occasionally cause friction. The tradeoff is worth it when the entity writing your tests is an AI that optimizes for making them pass rather than making them meaningful. Diff-scoping is what makes it practical. Full-codebase mutation testing is a research project. Diff-scoped mutation testing is a CI check.
\nWhat changed after the harness is not that the agent writes better code. What changed is that bad code gets caught — missing tests, broken boundaries, skipped steps, silent regressions — before it compounds.
\nAcross 287 commits from November 27, 2025 to February 24, 2026, with February 16 as the harness cutover, the share of commits touching test files rose from about 41% to 76%, consistent with require_changed_tests.15 The fix ratio held roughly flat at about 11%. Commits also got larger — averaging about 1,187 insertions versus 668 pre-harness — suggesting more confidence in landing bigger changes when the harness catches mistakes.
But the more interesting change was behavioral. I had always asked the agent to propose priorities — that ritual started early. But before the harness, I treated the proposals as suggestions and drove each session with specific goals: \"implement this JVP,\" \"fix the module layout,\" \"split these files.\" After, I could actually follow the agent's lead. Most sessions started the same way — \"what should we work on next?\" — and this time I meant it. The default shifted from \"I tell you what to build\" to \"you tell me what needs building, and I decide whether to approve.\"
\nThat shift felt strange the first few times. Asking an AI \"what should we work on?\" and actually trusting the answer requires believing the floor will catch whatever goes wrong. The harness was that floor. It could not prevent the agent from writing mediocre abstractions or introducing unnecessary complexity — architectural judgment still requires a human. But it could enforce that tests existed, that they verified behavior, that imports respected boundaries, that type annotations were real16, that suppressions were not growing. The minimum quality was no longer me. It was the tooling.
\nThe relay pattern from the earlier sprints would not have scaled without it. Parallel agents exacerbate every problem a single agent has — more drift, more skipped steps, more context confusion — and the harness was what kept them honest.
\nWhen something went wrong, the response changed too. Finding a bug in np.pad tracing, I did not just ask for a fix. I also asked why it slipped through, what would have prevented it, and how the instructions or harness should change so the same class of bug would not recur. Every bug became a harness improvement opportunity. After feature sessions, I started asking: was the harness useful? what did it catch? did mutation tests surface anything? They almost always had — mutation testing reliably catches weak tests that the agent wrote to pass, not to verify. The system learned from its failures, not through any kind of machine learning, but through a human treating each failure as evidence that a check was missing.17
This implementation is purely Python, and that shows in the choice of technologies. AST-based mutation testing, pytest-driven coverage, ruff and mypy as lint and type gates — these are ecosystem-specific. The implementation is shaped by its context. But the exercise transfers: figure out what your agent keeps getting wrong, build a check for it, and wire that check into a loop the agent cannot skip. The specific checks will differ. The discipline should not.
\nStart with the check that would have caught the last thing that went wrong. The harness will grow from there.
\nI built this library intentionally as a learning experience for myself — paying attention to the patterns, documenting what went wrong, formalizing each fix. I suspect much of it will be familiar to anyone who has spent real time building with agents. The stacks and checks will vary. The habit should not: when the agent fails, turn the failure into a constraint, a check, or a better handoff, then keep going.
\nThis article was written with AI assistance. I dictated raw thoughts using dictation software, then worked with Claude to turn them into prose — iterating paragraph by paragraph and pushing back whenever something did not sound like me. The research was AI-assisted too: I used a local transcript indexing and search tool, and had agents crawl through three months of git history to surface the timelines and stats behind the claims here. The result is more thoroughly researched than what I would have produced on my own, which is part of the point. ↩
\nAutograd by Maclaurin, Duvenaud, and Johnson is the original NumPy autodiff library — elegant, influential, and the reason most of us know reverse-mode AD can feel native to Python. I had used it extensively, and it was the main reference point in my head. MyGrad by Ryan Soklaski takes a different approach — a Tensor object with NumPy ufunc/function overrides rather than autograd's tracing tape — and its Hypothesis-heavy testing style directly influenced this library's. ↩
\nAutodiff is the perfect application for property-based testing — you can express real mathematical invariants (gradient correctness, chain rule composition, forward/reverse agreement) and let the framework try to break them. ↩
\nThe plans were remarkably specific. A typical one would include exact file paths, function signatures, test case names, and threshold constants — for example, AUTO_MIN_NODES_FOR_ANY_OPT = 128, AUTO_MIN_NODES_FOR_CSE = 512. The agent produced these from its analysis of the codebase; I reviewed and approved. ↩
Autograd was the reference point I had in mind when I was sanity-checking the agent's parity claim. ↩
\nMy best guess at the cause: post-training fine-tuning rewards \"safe\" code — strong compatibility guarantees, no breaking changes, defensive patterns. The training data is overwhelmingly production code with real users, where preserving backward compatibility is the right default. The model has no way to distinguish that context from a greenfield repo with zero consumers. Sensible instinct, wrong situation. ↩
\nFor readers who want to dive deeper: JAX's Autodiff Cookbook is an excellent introduction to forward- and reverse-mode differentiation, and their Custom derivative rules page explains exactly the JVP-first approach adopted here — define a JVP rule, and the framework derives VJPs automatically by transposing the linear computation. ↩
\nThe naive question is: why not just keep them in sync? In practice, a growing codebase accumulates multiple places that encode the same information — agent instructions, CI configs, contributor docs, READMEs — and no matter how explicit the instructions are, they drift. Each source gets updated in its own context, by a different session or a different agent, and nobody notices the divergence until something breaks. This is hard to stay on top of even with human contributors; with agents that read whatever file they find first, it compounds fast. The harness solved this by making the single CLI the only source of truth — AGENTS.md says \"run harness loop,\" CI runs \"harness gate,\" and neither needs to enumerate individual steps. ↩
\nBoth Claude Code and Codex CLI had automatic context compaction by this point — summarizing conversation history when the context window fills up. The relay pattern solves a different problem. Auto-compaction tries to preserve everything; the relay deliberately discards, starting a fresh session with only what the human judges relevant. The compression is lossy by design — that is the point. ↩
\nAround this time, I shifted most implementation work from Claude Code to Codex. Different tools for different strengths — Codex sessions averaged five hours for heavy, instruction-driven implementation; Claude Code sessions averaged under two hours for analysis, architecture review, and focused tasks. ↩
\nThis library's graph-based tracing is inherently heavier than autograd's flat tape — you pay for node allocation, edge management, and scope tracking on every operation. Early benchmarks showed very large overhead. A dedicated performance sprint across several parallel workstreams brought this down significantly, and a 10x ratio was set as the acceptable threshold: slow enough to reflect the architectural cost, fast enough to be usable. The threshold was wired into quality.py as a blocking gate — which worked until measurement noise on higher-order workloads pushed it past 10x on runs where nothing performance-related had changed. ↩
The term gained traction in early 2026. Mitchell Hashimoto's \"My AI Adoption Journey\" described the practice of engineering a solution for every agent mistake so it never recurs — each line in his AGENTS.md traced to a specific past failure. OpenAI's \"Harness engineering\" made the case at scale using Codex. Anthropic demonstrated it by having sixteen parallel Claude agents build a C compiler in Rust. Cursor showed what happens at the extreme — agents building a browser from scratch, running unattended for a week. Martin Fowler's analysis provided conceptual framing. Can Duruk's \"The Harness Problem\" argued the harness is the bottleneck, not the model. ↩
\nGoogle runs diff-based mutation testing on every code change to its monorepo, using the same core idea: generate mutants only in changed lines, use coverage data to select relevant tests, suppress unproductive mutations. Their system serves tens of thousands of developers. See Petrovic and Ivankovic, \"State of Mutation Testing at Google\" (ICSE 2018). ↩
\nThis is the \"fail closed\" principle from the first article: when the system cannot determine whether something is safe, it should assume it is not. Every ambiguity resolves toward more checking, not less. ↩
\n\"Touched test files\" means the commit's diffstat includes at least one file under a tests/ directory. \"Fix\" commits are classified by conventional commit prefix (fix:). Insertions are raw git log --stat totals. ↩
In a typed codebase, agents do not have to guess what a function expects or returns. Types are how agents navigate a large codebase without reading every implementation. In Python that discipline is optional rather than enforced by the language, so tools like mypy strict have to carry the load. ↩
\nThis extended to the agent instructions themselves. Early on, I established a rule that AGENTS.md should self-update: if an agent followed a rule and it still led to the wrong outcome, the rule needed to be refined. The instructions file became a living document maintained by the agents who consumed it. ↩
Consider two recent experiments with coding agents. Similar ambition. Opposite outcomes.
\nIn the first, a team pointed hundreds of agents at a browser project. In a week they produced roughly a million lines of code. By coordination metrics it was a success: parallel work, lots of merged PRs, visible throughput. But when the project went public, outside observers pointed to failing CI and questioned how much of that visible throughput translated into a clean, working system.
\nIn the second, as described in Claude is not a senior engineer (yet), a single engineer connected Claude to an automated browser testing suite (Playwright) and an error monitoring tool (Sentry). The agent wrote code, ran the tests, read the error traces, and fixed its own bugs. Ninety minutes later, it worked.
\nI am not trying to offer a definitive postmortem on either case. I am using them as contrasting examples of a broader engineering pattern.
\nThe pattern is a standard engineering concept: tolerance.
\nMechanical engineering abandoned binary \"works/doesn't work\" thinking decades ago. A bridge doesn't just \"work\". It tolerates a specific load variance under specific conditions. We ask about allowable margin of error, i.e., the acceptable error band around the ideal.
\nSoftware engineering has tolerances, too.
\nTolerance isn't one number. In software it decomposes into dimensions like correctness, security, latency, cost, reversibility, and blast radius (think error budgets). UI copy may be flexible on exact wording but not on brand tone. A refactor may tolerate new implementation details but not behavior changes.
\nSome tasks are high tolerance: exploratory prototyping, quick internal tools, one-off scripts, early drafts. Drift is acceptable because the goal is discovery and speed.
\nOther tasks are low tolerance: production infrastructure, security boundaries, customer-facing behavior, billing and permissions. Here \"close enough\" isn't a solution. It's a failure that may not show up immediately, but will surface later as incidents and churn.
\nThe difference between those two agent experiments was that one treated a low-tolerance problem with a high-tolerance process.
\nThat mismatch shows up as a control problem.
\nIn an open loop, the agent writes code and opens pull requests, but verification bottlenecks at human review minutes, hours, or days later. The delay between action and verification lets error accumulate. Drift becomes visible only after it's expensive.
\nIn a closed loop, the agent makes a change and immediately runs verification against hard constraints. The loop itself damps error. Closed loops require fast, reliable feedback; slow or flaky verification re-opens the loop.
\n| \n | Browser Project (Open Loop) | \nLLM + Tests + Traces (Closed Loop) | \n
|---|---|---|
| Feedback signal | \nInformational (throughput, mergeability, activity) | \nStructural (tests pass, errors resolved) | \n
| Verification timing | \nAfter the fact, by humans | \nEvery turn, by the agent | \n
| Termination condition | \nPR merged | \nConstraints satisfied | \n
| Outcome | \nImplied success; required human fixes | \nWorking fix in 90 minutes | \n
Read this way, the browser project optimized for coordination metrics, while the Claude setup optimized for correctness under feedback. The point is not that many agents are inherently bad. The point is that open loops amplify drift when verification is weak. Agent reliability isn't magic model behavior, it's an environment where correctness is continuously verified.
\nBut verification has a precondition: success must be expressible as constraints the agent can actually verify.
\n\n\nFor a picture of what the browser experiment could have looked like with human coordination and a tight loop, see One Human + One Agent = One Browser From Scratch.
\n
The fundamental challenge with agents is that intent is latent (in your head), while evidence is explicit (text documents in the repo).
\nAmbiguity is the distance between intent and evidence.
\nWhen you delegate to a human engineer, they bridge that gap with judgment: they ask clarifying questions, infer missing context, notice anomalies, and sanity-check against domain knowledge.
\nWhen you delegate to an AI agent, it can't feel that gap. It needs measurable constraints to know whether it has actually crossed from \"plausible output\" to \"correct result.\"
\nWithout constraints, the agent will still do as instructed. It will just optimize for the easiest proxy it can satisfy: producing output, closing tickets, merging PRs. Not correctness and integration.
\nA practical predictor of these outcomes is:
\nCan you describe success in terms of constraints the agent can verify?
\nIf you can, agents compound your effort. If you can't, you're doing exploration, and you should treat the output as exploration.
\nWhen agents appear unreliable, it's usually a failure of the surrounding system design rather than the model itself. We see four common patterns.
\nUndefined Specs
\n\n You have intent, but no mechanism to verify it. The requirements are unsettled, or the\n definition of done is just a feeling, like \"make onboarding feel simpler.\"\n
\n\n Fix: Don't delegate the decision-making. Use the agent to prototype and\n explore, but treat the output as raw material that helps you write the spec, not as\n the final product. If you don't know what done looks like, the agent won't either.\n
\nUnenforced Verification
\n\n The specs exist and are accessible, but the agent isn't forced to check them. Tests are nice\n to have. CI failures don't block merges. The system rewards speed or volume over correctness.\n The result is a workflow where \"it probably works\" is treated as progress.\n
\n\n Fix: Verification must be a termination condition. CI gates must fail closed.\n Pre-commit hooks tighten the loop. If the tests don't pass, the agent hasn't finished.\n
\nInadequate Constraints
\n\n The agent is verifying, but the constraints are too weak or too game-able. Tests pass, yet the\n system is still wrong: coverage is thin, assertions encode the wrong intent, or non-functional\n requirements (performance, security, UX) aren't represented. For example, unit tests stay\n green while latency quietly doubles.\n
\n\n Fix: Widen the constraint surface. Add invariants and golden tests for\n critical flows, static analysis (types, linters), and where it matters, property\n tests/fuzzing. For production-adjacent changes, pair verification with observability and\n rollback criteria.\n
\nAs models get smarter and faster, and context windows expand, the temptation is to throw them at larger, fuzzier problems. But a smarter, faster agent in a fuzzy environment mostly produces the wrong thing faster. It cannot know what it cannot read.
\nTo decide whether to delegate, ask:
\nThat gives four cases:
\nHigh-level work (architecture, strategy, trade-offs) often starts in the hardest case: low verifiability, low tolerance. Assumptions hide best there. But high-level work is usually decomposable. Break it into constrained subtasks, then delegate those.
\nFor example, \"defining multi-tenant permissions\" is low tolerance and low verifiability at the start. Don't delegate the decision; delegate the work of making it verifiable: draft a permission matrix and invariants, write tests for escalation paths, prototype a policy-as-code layer. Once those constraints exist, implementation becomes a low-tolerance but verifiable task.
\nNone of this is new engineering wisdom. What changes with agents is the rate at which small omissions compound.
\nHumans bridge gaps socially: they ask questions, notice contradictions, remember \"that one incident from last year,\" and hesitate when something feels off. Agents don't get those dampeners. They will happily produce plausible work until the system forces contact with reality.
\nThat's why what used to be technical debt becomes context failure. If a constraint (decision records, schemas, invariants, style guides, runbooks) isn't captured in the repo, it effectively doesn't exist for an agent. Treat context as code, and treat verification as the termination condition, not a suggestion.
\nThe future workflow isn't exotic. It's the old best practices, made load-bearing by speed. Start with the last thing your agent got wrong. Turn it into a constraint or a check. Wire it into the loop, then repeat.
", "attachments": [ { "url": "https://engineering.flexcompute.com/articles/agent-control-loop.md", "mime_type": "text/markdown", "title": "The Agent Control Loop — Engineering for Tolerance markdown" }, { "url": "https://engineering.flexcompute.com/images/og/agent-control-loop.png", "mime_type": "image/png", "title": "The Agent Control Loop — Engineering for Tolerance social image" } ], "_flexcompute": { "kind": "Essay", "tags": [ "ai-engineering", "agents", "verification" ], "series": "AI Engineering", "series_order": 1, "markdown_url": "https://engineering.flexcompute.com/articles/agent-control-loop.md" } } ] }