Tune tracker workers and bound agent budgets

The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.

Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.

Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-05-29 16:37:32 +02:00
parent 50c5454681
commit 55302d18f0
17 changed files with 818 additions and 339 deletions

View File

@@ -29,8 +29,22 @@ import (
)
const (
agentTimeout = 60 * time.Second
agentMaxTurns = 5
// defaultAgentTimeout caps a single mapping or enrichment agent run
// when the worker config does not supply one. It guards against a
// hung LLM provider or a slow web search.
defaultAgentTimeout = 45 * time.Second
// defaultMappingMaxTurns and defaultEnrichmentMaxTurns bound the
// agent reasoning loop (LLM call + tool round-trips) when the worker
// config does not supply a value.
defaultMappingMaxTurns = 4
defaultEnrichmentMaxTurns = 3
// defaultAgentMaxTokens caps the structured output of the mapping
// and enrichment agents when the agent config carries no max-tokens
// budget. Both outputs are tiny structured JSON.
defaultAgentMaxTokens = 1024
agentThirdPartyConfidenceThreshold = 0.6
// agentSourceConfidence is the fixed confidence stored on catalog
// rows the agent attributes to a third party. The agent's own
@@ -70,16 +84,37 @@ func buildTrackerMappingAgent(
panic(fmt.Sprintf("cookiebanner: cannot build tracker identification output type: %s", err))
}
return agent.New(
"tracker-mapping",
cfg.LLMClient,
maxTurns := cfg.MappingMaxTurns
if maxTurns < 1 {
maxTurns = defaultMappingMaxTurns
}
opts := []agent.Option{
agent.WithInstructionsFunc(trackerMappingInstructions),
agent.WithModel(cfg.Model),
agent.WithTools(tools...),
agent.WithOutputType(outputType),
agent.WithMaxTurns(agentMaxTurns),
agent.WithMaxTurns(maxTurns),
agent.WithMaxTokens(resolveAgentMaxTokens(cfg.MaxTokens)),
agent.WithLogger(logger),
)
}
if cfg.Temperature != nil {
opts = append(opts, agent.WithTemperature(*cfg.Temperature))
}
return agent.New("tracker-mapping", cfg.LLMClient, opts...)
}
// resolveAgentMaxTokens returns the configured max-tokens budget for the
// mapping and enrichment agents, falling back to defaultAgentMaxTokens
// when none is set.
func resolveAgentMaxTokens(configured *int) int {
if configured != nil && *configured > 0 {
return *configured
}
return defaultAgentMaxTokens
}
func trackerMappingInstructions(_ context.Context, _ *agent.Agent) string {