rohit/probo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tune tracker workers and bound agent budgets

Browse Source 
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.

Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.

Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-05-29 16:37:32 +02:00
parent 50c5454681
commit 55302d18f0
17 changed files with 818 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
pkg/probodconfig/llm_config.go
View File

@@ -40,6 +40,30 @@ type (
MaxConcurrency int `json:"max-concurrency"`
}
// TrackerMappingWorkerConfig holds worker-side tuning for the
// tracker-mapping background worker. LLM parameters for the agents
// it runs live under AgentsConfig.TrackerMapping. AgentTimeout and
// AgentMaxTurns bound a single agent run (the identification and
// disambiguation agents).
TrackerMappingWorkerConfig struct {
Interval int `json:"interval"` // seconds between polls
MaxConcurrency int `json:"max-concurrency"`
AgentTimeout int `json:"agent-timeout"` // seconds, single agent run
AgentMaxTurns int `json:"agent-max-turns"`
}
// CommonPatternEnrichmentWorkerConfig holds worker-side tuning for
// the common-pattern enrichment background worker. LLM parameters
// for the enrichment agent live under AgentsConfig.TrackerMapping
// (the agents share one config slot).
CommonPatternEnrichmentWorkerConfig struct {
Interval int `json:"interval"` // seconds between polls
MaxConcurrency int `json:"max-concurrency"`
StaleAfter int `json:"stale-after"` // seconds before a claim is recycled
AgentTimeout int `json:"agent-timeout"` // seconds, single agent run
AgentMaxTurns int `json:"agent-max-turns"`
}
// AgentToolsConfig holds API keys and settings for external tools
// that agents can use (web search, scraping, etc.).
AgentToolsConfig struct {