Tune tracker workers and bound agent budgets

The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.

Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.

Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-05-29 16:37:32 +02:00
parent 50c5454681
commit 55302d18f0
17 changed files with 818 additions and 339 deletions

View File

@@ -14,15 +14,29 @@
package cookiebanner
import "go.probo.inc/probo/pkg/llm"
import (
"time"
"go.probo.inc/probo/pkg/llm"
)
// TrackerAgentsConfig configures the tracker agents that share one LLM
// client, model, and tool surface: the tracker-mapping agent (catalog
// identification) and the common-pattern enrichment agent (description
// research). Both use DB-backed search tools and may also use Firecrawl
// for web search when an API key is supplied.
//
// MaxTokens and Temperature bound and steer each LLM call (both
// outputs are tiny structured JSON). AgentTimeout caps a single agent
// run, and the per-worker max-turns bound the agent reasoning loop.
// Zero-valued tuning fields fall back to package defaults.
type TrackerAgentsConfig struct {
LLMClient *llm.Client
Model string
FirecrawlAPIKey string
LLMClient *llm.Client
Model string
FirecrawlAPIKey string
MaxTokens *int
Temperature *float64
AgentTimeout time.Duration
MappingMaxTurns int
EnrichmentMaxTurns int
}