Tune tracker workers and bound agent budgets

The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.

Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.

Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-05-29 16:37:32 +02:00
parent 50c5454681
commit 55302d18f0
17 changed files with 818 additions and 339 deletions

View File

@@ -29,13 +29,14 @@ import (
"go.probo.inc/probo/pkg/llm"
)
const enrichmentStaleAfter = 10 * time.Minute
const defaultEnrichmentStaleAfter = 10 * time.Minute
type commonPatternEnrichmentHandler struct {
pg *pg.Client
logger *log.Logger
enrichmentAgent *agent.Agent
staleAfter time.Duration
agentTimeout time.Duration
}
// NewCommonPatternEnrichmentWorker builds the worker that fills
@@ -49,12 +50,23 @@ func NewCommonPatternEnrichmentWorker(
pgClient *pg.Client,
logger *log.Logger,
cfg TrackerAgentsConfig,
staleAfter time.Duration,
opts ...worker.Option,
) *worker.Worker[coredata.CommonTrackerPattern] {
if staleAfter <= 0 {
staleAfter = defaultEnrichmentStaleAfter
}
agentTimeout := cfg.AgentTimeout
if agentTimeout <= 0 {
agentTimeout = defaultAgentTimeout
}
h := &commonPatternEnrichmentHandler{
pg: pgClient,
logger: logger,
staleAfter: enrichmentStaleAfter,
pg: pgClient,
logger: logger,
staleAfter: staleAfter,
agentTimeout: agentTimeout,
}
if cfg.LLMClient != nil {
@@ -187,7 +199,7 @@ func (h *commonPatternEnrichmentHandler) research(
) (string, error) {
prompt := buildEnrichmentPrompt(cp, thirdPartyName)
agentCtx, cancel := context.WithTimeout(ctx, agentTimeout)
agentCtx, cancel := context.WithTimeout(ctx, h.agentTimeout)
defer cancel()
result, err := agent.RunTyped[CommonPatternEnrichmentResult](