Tune tracker workers and bound agent budgets
The tracker-mapping and common-pattern-enrichment workers ran with the kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped the resolved per-agent max-tokens/temperature, so up to ten LLM pipelines could run unbounded on one OpenAI client. The mapping worker also held a FOR UPDATE transaction across the LLM and Firecrawl calls while its DB search tools acquired a second pooled connection, risking pool exhaustion under concurrency. Plumb max-tokens, temperature, agent timeout, and per-worker max-turns through TrackerAgentsConfig and DisambiguationConfig into all three agent builders, replacing the hard-coded constants with config-fed fields and package fallbacks. Expose worker interval, concurrency, stale-after, agent timeout, and max-turns as config (env, Helm values, deployment template) mirroring the evidence-describer pattern, and apply them at registration. Refactor Process into deterministic-read, agent (no transaction), and persist phases so neither the mapping agent nor disambiguation runs inside an open transaction, removing the row locks held across network latency and the nested-connection pressure. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
@@ -29,13 +29,14 @@ import (
|
||||
"go.probo.inc/probo/pkg/llm"
|
||||
)
|
||||
|
||||
const enrichmentStaleAfter = 10 * time.Minute
|
||||
const defaultEnrichmentStaleAfter = 10 * time.Minute
|
||||
|
||||
type commonPatternEnrichmentHandler struct {
|
||||
pg *pg.Client
|
||||
logger *log.Logger
|
||||
enrichmentAgent *agent.Agent
|
||||
staleAfter time.Duration
|
||||
agentTimeout time.Duration
|
||||
}
|
||||
|
||||
// NewCommonPatternEnrichmentWorker builds the worker that fills
|
||||
@@ -49,12 +50,23 @@ func NewCommonPatternEnrichmentWorker(
|
||||
pgClient *pg.Client,
|
||||
logger *log.Logger,
|
||||
cfg TrackerAgentsConfig,
|
||||
staleAfter time.Duration,
|
||||
opts ...worker.Option,
|
||||
) *worker.Worker[coredata.CommonTrackerPattern] {
|
||||
if staleAfter <= 0 {
|
||||
staleAfter = defaultEnrichmentStaleAfter
|
||||
}
|
||||
|
||||
agentTimeout := cfg.AgentTimeout
|
||||
if agentTimeout <= 0 {
|
||||
agentTimeout = defaultAgentTimeout
|
||||
}
|
||||
|
||||
h := &commonPatternEnrichmentHandler{
|
||||
pg: pgClient,
|
||||
logger: logger,
|
||||
staleAfter: enrichmentStaleAfter,
|
||||
pg: pgClient,
|
||||
logger: logger,
|
||||
staleAfter: staleAfter,
|
||||
agentTimeout: agentTimeout,
|
||||
}
|
||||
|
||||
if cfg.LLMClient != nil {
|
||||
@@ -187,7 +199,7 @@ func (h *commonPatternEnrichmentHandler) research(
|
||||
) (string, error) {
|
||||
prompt := buildEnrichmentPrompt(cp, thirdPartyName)
|
||||
|
||||
agentCtx, cancel := context.WithTimeout(ctx, agentTimeout)
|
||||
agentCtx, cancel := context.WithTimeout(ctx, h.agentTimeout)
|
||||
defer cancel()
|
||||
|
||||
result, err := agent.RunTyped[CommonPatternEnrichmentResult](
|
||||
|
||||
Reference in New Issue
Block a user