Tune tracker workers and bound agent budgets
The tracker-mapping and common-pattern-enrichment workers ran with the kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped the resolved per-agent max-tokens/temperature, so up to ten LLM pipelines could run unbounded on one OpenAI client. The mapping worker also held a FOR UPDATE transaction across the LLM and Firecrawl calls while its DB search tools acquired a second pooled connection, risking pool exhaustion under concurrency. Plumb max-tokens, temperature, agent timeout, and per-worker max-turns through TrackerAgentsConfig and DisambiguationConfig into all three agent builders, replacing the hard-coded constants with config-fed fields and package fallbacks. Expose worker interval, concurrency, stale-after, agent timeout, and max-turns as config (env, Helm values, deployment template) mirroring the evidence-describer pattern, and apply them at registration. Refactor Process into deterministic-read, agent (no transaction), and persist phases so neither the mapping agent nor disambiguation runs inside an open transaction, removing the row locks held across network latency and the nested-connection pressure. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
@@ -14,15 +14,29 @@
|
||||
|
||||
package cookiebanner
|
||||
|
||||
import "go.probo.inc/probo/pkg/llm"
|
||||
import (
|
||||
"time"
|
||||
|
||||
"go.probo.inc/probo/pkg/llm"
|
||||
)
|
||||
|
||||
// TrackerAgentsConfig configures the tracker agents that share one LLM
|
||||
// client, model, and tool surface: the tracker-mapping agent (catalog
|
||||
// identification) and the common-pattern enrichment agent (description
|
||||
// research). Both use DB-backed search tools and may also use Firecrawl
|
||||
// for web search when an API key is supplied.
|
||||
//
|
||||
// MaxTokens and Temperature bound and steer each LLM call (both
|
||||
// outputs are tiny structured JSON). AgentTimeout caps a single agent
|
||||
// run, and the per-worker max-turns bound the agent reasoning loop.
|
||||
// Zero-valued tuning fields fall back to package defaults.
|
||||
type TrackerAgentsConfig struct {
|
||||
LLMClient *llm.Client
|
||||
Model string
|
||||
FirecrawlAPIKey string
|
||||
LLMClient *llm.Client
|
||||
Model string
|
||||
FirecrawlAPIKey string
|
||||
MaxTokens *int
|
||||
Temperature *float64
|
||||
AgentTimeout time.Duration
|
||||
MappingMaxTurns int
|
||||
EnrichmentMaxTurns int
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user