toolbox

A tool is a directory holding two Agency files. impl.agency is the part the coding agent writes: export type Request and export def run(request: Request): Json. tool.agency is generated from a fixed template and wraps run in a guard with time and cost limits. It exports tool and node main, both taking a Request.

Useful exports

• listTools reads a toolbox directory into a catalog. It raises a std::toolbox::scan interrupt, then std::ls for the listing.
• designTool has the coding agent draft a new tool. designTool then checks and tests the draft, shows it to the user through a std::toolbox::review interrupt that can ask for a revision, and saves it through the same save gate writeTool uses.
• writeTool saves a tool whose run function is already written. It wraps and typechecks the source, shows it through a std::toolbox::save interrupt, and saves it. No model is called.
• runTool runs a saved tool and records the use.

import { designTool, listTools, runTool } from "std::toolbox"

node main() {
  handle {
    const written = designTool(
      name: "getNews",
      purpose: "Summarize today's news for a list of topics as Markdown.",
      request: "{ topics: string[]; maxItems: number }",
    )
    if (written is failure(err)) {
      print("not written: ${err}")
      return
    }
    printJSON(listTools())
    const news = runTool("getNews", { topics: ["tech"], maxItems: 5 })
    print(news)
  } with (intr) {
    return match (intr.effect) {
      "std::toolbox::review" => {
        print(intr.data.source)
        print("effects: ${intr.data.effects.join(", ")}")
        const answer = input("accept, or feedback for the next draft (empty rejects): ")
        if (answer == "") {
          return reject("cancelled by user")
        }
        return approve(answer)
      }
      "std::toolbox::save" => approve()
      _ => pass()
    }
  }
}

A program can also import a saved tool directly: import { tool as getNews } from "~/.agency-agent/tools/getNews/tool.agency".

Types

Unresolved

A review point the drafting agent could not fix, in the reviewer's words, and the agent's reason.

/** A review point the drafting agent could not fix, in the reviewer's
  words, and the agent's reason. */
export type Unresolved = {
  point: string;
  reason: string
}

(source)

ModuleFacts

What describe says about a tool's run function.

/** What `describe` says about a tool's `run` function. */
export type ModuleFacts = {
  signature: string;
  docstring?: string;
  effects: string[]
}

(source)

ToolMeta

What meta.json holds: how the tool was made, and how often it ran. requestSchema is the JSON Schema of the Request type, as the tool's own requestSchema node reported it at save time; null for a tool saved before schemas were recorded.

/** What meta.json holds: how the tool was made, and how often it ran.
  `requestSchema` is the JSON Schema of the Request type, as the tool's
  own `requestSchema` node reported it at save time; null for a tool
  saved before schemas were recorded. */
export type ToolMeta = {
  purpose: string;
  request: string;
  requestSchema?: Json;
  createdAt: string;
  maxTime: number;
  version: number;
  uses: number;
  lastUsedAt?: string
}

(source)

ToolEntry

One tool in a toolbox. This is what listTools returns. If the tool could not be read correctly, broken holds the reason.

/** One tool in a toolbox. This is what `listTools` returns. If the tool
  could not be read correctly, `broken` holds the reason. */
export type ToolEntry = {
  name: string;
  dir: string;
  module: ModuleFacts;
  meta: ToolMeta;
  broken?: string
}

(source)

Effects

std::toolbox::scan

@alwaysUnder(dir)
effect std::toolbox::scan {
  dir: string
}

(source)

std::toolbox::recordUse

@alwaysUnder(dir)
effect std::toolbox::recordUse {
  dir: string;
  name: string
}

(source)

std::toolbox::writeFile

@alwaysUnder(root)
effect std::toolbox::writeFile {
  root: string;
  dir: string;
  filename: string;
  content: string
}

(source)

std::toolbox::createStaging

@alwaysUnder(root)
effect std::toolbox::createStaging {
  root: string;
  dir: string;
  name: string
}

(source)

std::toolbox::removeStaging

@alwaysUnder(root)
effect std::toolbox::removeStaging {
  root: string;
  dir: string;
  name: string
}

(source)

std::toolbox::removeStagedFile

@alwaysUnder(root)
effect std::toolbox::removeStagedFile {
  root: string;
  dir: string;
  name: string;
  filename: string
}

(source)

std::toolbox::review

effect std::toolbox::review {
  name: string;
  stagingDir: string;
  source: string;
  /** The draft the previous round showed, or "" on the first round. The
    approval prompt diffs `source` against it, so a redraft shows what
    changed rather than the whole tool again. */
  previous: string;
  effects: string[];
  tested: boolean;
  /** Reviewer findings that do not block the draft: advice to the user
    who asked for the tool. */
  notes: string[];
  /** Reviewer findings the author was asked to fix and the reviewer
    raised again. Empty unless the draft is here because of them. */
  blocking: string[];
  /** Review points the author reported it could not fix, with its reason. */
  unresolved: Unresolved[];
  /** True when this is a draft from earlier in the session that nobody
    accepted, shown again instead of writing a new one. */
  resumed: boolean
}

(source)

std::toolbox::save

@alwaysUnder(dir)
effect std::toolbox::save {
  dir: string;
  name: string;
  source: string;
  effects: string[]
}

(source)

Functions

listTools

listTools(dir: string = "~/.agency-agent/tools"): Result<ToolEntry[]>

List the tools in a toolbox directory. Raises a std::toolbox::scan interrupt before reading anything, then std::ls for the listing. Each entry has: the tool's name and directory; what describe says about its run function (signature, docstring, effects); and its meta.json record (purpose, request type, creation time, time limit, version, run count, last-run time). An entry runTool would refuse carries the reason in broken.

@param dir - The toolbox directory to scan

Parameters:

Name	Type	Default
dir	string	"~/.agency-agent/tools"

Returns: Result<ToolEntry[]>

Throws: std::toolbox::scan, std::ls

(source)

designTool

designTool(
  name: string,
  purpose: string,
  request: string,
  dir: string = "~/.agency-agent/tools",
  maxRounds: number = 3,
  maxTime: number = 3m,
  maxCost: number = $1.00,
  model: string = "",
  provider: string = "",
  review: boolean = false,
): Result<ToolEntry>

Design a reusable tool with the user and save it into a toolbox directory. The coding agent drafts the tool's run function against the request type. The draft is typechecked, tested when it is pure computation, and shown to the user, who accepts it or gives feedback for another draft.

@param name - The tool's name; also its directory under dir @param purpose - What the tool should do, in plain language @param request - The tool's input type as Agency type text, such as { topics: string[]; maxItems: number } @param dir - The toolbox directory to write into @param maxRounds - Draft-review rounds before giving up @param maxTime - Time limit baked into the tool's guard, under one hour @param maxCost - Cost limit baked into the tool's guard @param model - Model override for the coding and review agents, or "" @param provider - Provider for the model override @param review - True has a review agent read each draft before its checks, and sends its blocking findings back to the author once. Off by default: on the evals/design-tool suite it doubled the time and tripled the cost of a tool and did not change the score

The design loop: the coding agent drafts the tool, the review agent and the typecheck vet the draft, a pure tool gets generated tests, and the result goes to the user in a std::toolbox::review interrupt that can accept it or send feedback for another round. An accepted draft is published through the same std::toolbox::save gate writeTool uses.

Parameters:

Name	Type	Default
name	string
purpose	string
request	string
dir	string	"~/.agency-agent/tools"
maxRounds	number	3
maxTime	number	3m
maxCost	number	$1.00
model	string	""
provider	string	""
review	boolean	false

Returns: Result<ToolEntry>

Throws: std::toolbox::removeStaging, std::toolbox::createStaging, std::toolbox::review, std::toolbox::save, std::toolbox::scan, std::toolbox::writeFile, std::run, std::guard, std::toolbox::removeStagedFile

(source)

writeTool

writeTool(
  name: string,
  purpose: string,
  request: string,
  source: string,
  dir: string = "~/.agency-agent/tools",
  maxTime: number = 3m,
  maxCost: number = $1.00,
): Result<ToolEntry>

Save an already-written tool into a toolbox directory, after approval. The source must export type Request (matching the request text) and def run(request: Request): Json. It is wrapped in the guarded tool module and typechecked before the user is asked. A rejection saves nothing and fails the call.

@param name - The tool's name, also its directory under dir @param purpose - What the tool does, in plain language @param request - The tool's input type as Agency type text, such as { topics: string[]; maxItems: number } @param source - The complete impl.agency source @param dir - The toolbox directory to write into @param maxTime - Time limit baked into the tool's guard, under one hour @param maxCost - Cost limit baked into the tool's guard

The plain primitive for saving a tool whose run function is already written: wrap the source in the guarded tool module, typecheck the pair, show the source and its effects in a std::toolbox::save interrupt, and publish. No model is called and no tests are generated. The design loop in designTool ends by publishing through this same gate.

Parameters:

Name	Type	Default
name	string
purpose	string
request	string
source	string
dir	string	"~/.agency-agent/tools"
maxTime	number	3m
maxCost	number	$1.00

Returns: Result<ToolEntry>

Throws: std::toolbox::removeStaging, std::toolbox::createStaging, std::toolbox::save, std::toolbox::scan, std::toolbox::writeFile, std::run, std::guard

(source)

runTool

runTool(
  name: string,
  request: Json,
  dir: string = "~/.agency-agent/tools",
): Result<Json>

Run a saved tool's main node in a subprocess and return what it returned. The tool's meta.json counts one more use and the time. A declined or failed count does not fail the run. The result is not recorded.

@param name - The tool's name under dir @param request - The tool's input, a value of its Request type @param dir - The toolbox directory holding the tool

Parameters:

Name	Type	Default
name	string
request	Json
dir	string	"~/.agency-agent/tools"

Returns: Result<Json>

Throws: std::toolbox::scan, std::run, std::guard, std::toolbox::recordUse

(source)