rohit/probo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tune tracker workers and bound agent budgets

Browse Source 
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.

Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.

Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-05-29 16:37:32 +02:00
parent 50c5454681
commit 55302d18f0
17 changed files with 818 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
pkg/probod/probod.go
View File

@@ -725,7 +725,14 @@ func (impl *Implm) Run(
},
)
trackerMappingWorker := cookiebanner.NewTrackerMappingWorker(pgClient, l, trackerAgentsCfg, thirdPartyDisambiguationCfg)
trackerMappingWorker := cookiebanner.NewTrackerMappingWorker(
pgClient,
l,
trackerAgentsCfg,
thirdPartyDisambiguationCfg,
worker.WithInterval(time.Duration(impl.cfg.TrackerMappingWorker.Interval)*time.Second),
worker.WithMaxConcurrency(impl.cfg.TrackerMappingWorker.MaxConcurrency),
)
trackerMappingWorkerCtx, stopTrackerMappingWorker := context.WithCancel(context.Background())
wg.Go(
@@ -742,7 +749,17 @@ func (impl *Implm) Run(
stopCommonPatternEnrichmentWorker := func() {}
if trackerAgentsCfg.LLMClient != nil {
commonPatternEnrichmentWorker := cookiebanner.NewCommonPatternEnrichmentWorker(pgClient, l, trackerAgentsCfg)
enrichmentCfg := trackerAgentsCfg
enrichmentCfg.AgentTimeout = time.Duration(impl.cfg.CommonPatternEnrichmentWorker.AgentTimeout) * time.Second
commonPatternEnrichmentWorker := cookiebanner.NewCommonPatternEnrichmentWorker(
pgClient,
l,
enrichmentCfg,
time.Duration(impl.cfg.CommonPatternEnrichmentWorker.StaleAfter)*time.Second,
worker.WithInterval(time.Duration(impl.cfg.CommonPatternEnrichmentWorker.Interval)*time.Second),
worker.WithMaxConcurrency(impl.cfg.CommonPatternEnrichmentWorker.MaxConcurrency),
)
var commonPatternEnrichmentWorkerCtx context.Context