Where the work runs: the NPU, graphics card and processor

The local AI#

The staff use a small AI model to label, name, summarise and describe: what a screenshot shows, what a log says, what changed in an update. The model runs on this PC, never online, on one of its accelerators:

• The NPU (neural processing unit), on PCs that have one, such as Snapdragon and recent Intel and AMD laptops. It does AI work using little power, without the processor or the graphics card.
• A graphics card: a separate card, or the graphics built into the processor.
• The processor, as a last resort for short requests.

The model only describes. Code still decides everything that matters: whether a test passed, whether a file is a duplicate, whether a program is signed. Where an agent shows you something the model wrote, it says so.

Castellan's welcome sets the local AI up on this PC (see Your first day with Castellan), and the Smith looks after the model servers from then on.

Taking turns#

Every agent that uses a model takes turns on an accelerator, in line, instead of fighting over it:

• The NPU comes first. A request the NPU can do waits for it, even while a graphics card is free.
• Someone waiting comes first. A request you're waiting on (a search you typed, say) goes ahead of background work. Background work that has waited two minutes takes its turn in order.
• A turn lasts seconds. An agent that holds an accelerator too long is moved aside for the next in line.
• A failing accelerator is skipped. If one stops answering, the work moves to the next for ten minutes, then it's tried again.

That's why the NPU, the graphics card or the processor is busy for a moment, then quiet again: it's one agent's turn, then nobody's. Windows' Task Manager shows the NPU's use on its Performance tab, on PCs that have one.

Games come first#

A gaming PC is for games first. While a game, or any full-screen 3D program, is using a graphics card, background work keeps off that card: it goes to another accelerator, or waits until the game is done.

Seeing who's using what#

Castellan's page shows it as it happens:

• The NPU pill in the title bar: NPU free, or which agent is using it.
• Accelerators, beside This PC: a card for each, with who holds it, for how long, when it should let go, and who's waiting, in the order they'll be served. A card marked Failed is being skipped for now; one marked as in use by a game is being kept clear.
• About has a table of every agent you've hired: what it does on the processor, on a graphics card and with the model, and when.

Your choices#

Under Settings > The manor:

• Background work: Go easy on this PC (the default) has agents new to the manor do their heavy first work one at a time (a first scan of every drive, a first look at every repository), and runs their scheduled work at low priority, so it gives way to whatever you're doing. Full speed runs everything at once, as fast as it can.
• Use the graphics card for models when there's an NPU, shown only on a PC with both. On (the default), the NPU still comes first, and the graphics card takes only what the NPU can't do: longer requests, search indexing it doesn't serve, and a fallback when it fails. Off, the NPU does all the model work and the graphics card stays free, even when the NPU fails.

The Smith's own page lists the accelerators it has set up, and sets up more. See The Smith, which keeps your local AI ready.