rohit/probo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add common third party enricher worker

Browse Source 
Introduce a poll-based worker that fills the global common_third_parties
catalog (URLs, headquarter address, legal name, certifications, logo)
so each tenant no longer starts from sparse, name-only rows. Enrichment
is requested at row creation by ResolveOrCreateCommonThirdParty; curated
seed rows are not enqueued, to avoid a re-seed storm.

The pipeline uses two specialized agents plus a deterministic logo step.
Agent A (company profile) resolves legal name, headquarter address, and
the canonical website over web search; its website and legal name feed
Agent B and the logo step. Agent B (compliance docs) resolves the legal
document URLs, trust/security/status pages, and certifications using the
browser read-only toolset (gated on ChromeDPAddr) plus web search. The
logo step restores pkg/webinspect as a pure deterministic package and
stores the discovered icon in S3, linked via logo_file_id.

Each agent returns per-field value/confidence/source_url. The worker
writes a column only when confidence clears a configurable threshold and
the field is not externally owned (seed or human), and always records
full per-field provenance in a new enrichment JSONB column so re-runs
fill only gaps and human edits are never clobbered. New bookkeeping
columns (enrichment_requested_at, enrichment, enrichment_attempts) back
the claim queue and stale recovery; agents run outside transactions and
results persist in one final transaction.

The worker is opt-in: it no-ops unless its agent provider is configured.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-06-10 13:49:56 +02:00
parent 99d568d07d
commit 229c6b99c6
23 changed files with 2325 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
pkg/bootstrap/builder.go
View File

@@ -242,6 +242,15 @@ func (b *Builder) Build() (*probodconfig.FullConfig, error) {
Temperature: b.getEnvFloatPtr("AGENT_TRACKER_ENRICHMENT_TEMPERATURE"),
MaxTokens: new(b.getEnvIntOrDefault("AGENT_TRACKER_ENRICHMENT_MAX_TOKENS", 4096)),
},
CommonThirdPartyEnrichment: probodconfig.LLMAgentConfig{
Provider: b.getEnvOrDefault("AGENT_COMMON_THIRD_PARTY_ENRICHMENT_PROVIDER", ""),
ModelName: b.getEnvOrDefault("AGENT_COMMON_THIRD_PARTY_ENRICHMENT_MODEL_NAME", ""),
// Agent B browses pages and emits a moderate structured
// output; the budget must leave headroom for reasoning
// models whose reasoning tokens count against max_tokens.
Temperature: b.getEnvFloatPtr("AGENT_COMMON_THIRD_PARTY_ENRICHMENT_TEMPERATURE"),
MaxTokens: new(b.getEnvIntOrDefault("AGENT_COMMON_THIRD_PARTY_ENRICHMENT_MAX_TOKENS", 8192)),
},
Tools: probodconfig.AgentToolsConfig{
FirecrawlAPIKey: b.getEnv("FIRECRAWL_API_KEY"),
},
@@ -292,6 +301,15 @@ func (b *Builder) Build() (*probodconfig.FullConfig, error) {
AgentTimeout: b.getEnvIntOrDefault("COMMON_PATTERN_ENRICHMENT_AGENT_TIMEOUT", 45),
AgentMaxTurns: b.getEnvIntOrDefault("COMMON_PATTERN_ENRICHMENT_AGENT_MAX_TURNS", 10),
},
CommonThirdPartyEnrichmentWorker: probodconfig.CommonThirdPartyEnrichmentWorkerConfig{
Interval: b.getEnvIntOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_INTERVAL", 10),
MaxConcurrency: b.getEnvIntOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_MAX_CONCURRENCY", 1),
StaleAfter: b.getEnvIntOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_STALE_AFTER", 900),
AgentTimeout: b.getEnvIntOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_AGENT_TIMEOUT", 90),
AgentMaxTurns: b.getEnvIntOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_AGENT_MAX_TURNS", 12),
ConfidenceThreshold: b.getEnvFloatOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_CONFIDENCE_THRESHOLD", 0.7),
MaxAttempts: b.getEnvIntOrDefault("COMMON_THIRD_PARTY_ENRICHMENT_MAX_ATTEMPTS", 3),
},
Branding: b.getEnvBoolOrDefault("BRANDING", true),
},
}

29
pkg/bootstrap/builder_test.go
View File

@@ -245,6 +245,13 @@ func TestBuilder_Build_Defaults(t *testing.T) {
assert.Equal(t, 600, cfg.Probod.CommonPatternEnrichmentWorker.StaleAfter)
assert.Equal(t, 45, cfg.Probod.CommonPatternEnrichmentWorker.AgentTimeout)
assert.Equal(t, 10, cfg.Probod.CommonPatternEnrichmentWorker.AgentMaxTurns)
assert.Equal(t, 10, cfg.Probod.CommonThirdPartyEnrichmentWorker.Interval)
assert.Equal(t, 1, cfg.Probod.CommonThirdPartyEnrichmentWorker.MaxConcurrency)
assert.Equal(t, 900, cfg.Probod.CommonThirdPartyEnrichmentWorker.StaleAfter)
assert.Equal(t, 90, cfg.Probod.CommonThirdPartyEnrichmentWorker.AgentTimeout)
assert.Equal(t, 12, cfg.Probod.CommonThirdPartyEnrichmentWorker.AgentMaxTurns)
assert.Equal(t, 0.7, cfg.Probod.CommonThirdPartyEnrichmentWorker.ConfidenceThreshold)
assert.Equal(t, 3, cfg.Probod.CommonThirdPartyEnrichmentWorker.MaxAttempts)
assert.Equal(t, 10, cfg.Probod.ThirdPartyVetting.Interval)
assert.Equal(t, 1500, cfg.Probod.ThirdPartyVetting.StaleAfter)
assert.Equal(t, 1, cfg.Probod.ThirdPartyVetting.MaxConcurrency)
@@ -369,6 +376,17 @@ func TestBuilder_Build_CustomValues(t *testing.T) {
env["COMMON_PATTERN_ENRICHMENT_STALE_AFTER"] = "900"
env["COMMON_PATTERN_ENRICHMENT_AGENT_TIMEOUT"] = "50"
env["COMMON_PATTERN_ENRICHMENT_AGENT_MAX_TURNS"] = "5"
// Common third party enrichment agent + worker tuning override
env["AGENT_COMMON_THIRD_PARTY_ENRICHMENT_PROVIDER"] = "openai"
env["AGENT_COMMON_THIRD_PARTY_ENRICHMENT_MODEL_NAME"] = "gpt-4o"
env["AGENT_COMMON_THIRD_PARTY_ENRICHMENT_MAX_TOKENS"] = "16384"
env["COMMON_THIRD_PARTY_ENRICHMENT_INTERVAL"] = "25"
env["COMMON_THIRD_PARTY_ENRICHMENT_MAX_CONCURRENCY"] = "2"
env["COMMON_THIRD_PARTY_ENRICHMENT_STALE_AFTER"] = "1200"
env["COMMON_THIRD_PARTY_ENRICHMENT_AGENT_TIMEOUT"] = "120"
env["COMMON_THIRD_PARTY_ENRICHMENT_AGENT_MAX_TURNS"] = "8"
env["COMMON_THIRD_PARTY_ENRICHMENT_CONFIDENCE_THRESHOLD"] = "0.85"
env["COMMON_THIRD_PARTY_ENRICHMENT_MAX_ATTEMPTS"] = "5"
env["THIRD_PARTY_VETTING_INTERVAL"] = "15"
env["THIRD_PARTY_VETTING_STALE_AFTER"] = "1800"
env["THIRD_PARTY_VETTING_MAX_CONCURRENCY"] = "2"
@@ -490,6 +508,17 @@ func TestBuilder_Build_CustomValues(t *testing.T) {
assert.Equal(t, 900, cfg.Probod.CommonPatternEnrichmentWorker.StaleAfter)
assert.Equal(t, 50, cfg.Probod.CommonPatternEnrichmentWorker.AgentTimeout)
assert.Equal(t, 5, cfg.Probod.CommonPatternEnrichmentWorker.AgentMaxTurns)
assert.Equal(t, "openai", cfg.Probod.Agents.CommonThirdPartyEnrichment.Provider)
assert.Equal(t, "gpt-4o", cfg.Probod.Agents.CommonThirdPartyEnrichment.ModelName)
require.NotNil(t, cfg.Probod.Agents.CommonThirdPartyEnrichment.MaxTokens)
assert.Equal(t, 16384, *cfg.Probod.Agents.CommonThirdPartyEnrichment.MaxTokens)
assert.Equal(t, 25, cfg.Probod.CommonThirdPartyEnrichmentWorker.Interval)
assert.Equal(t, 2, cfg.Probod.CommonThirdPartyEnrichmentWorker.MaxConcurrency)
assert.Equal(t, 1200, cfg.Probod.CommonThirdPartyEnrichmentWorker.StaleAfter)
assert.Equal(t, 120, cfg.Probod.CommonThirdPartyEnrichmentWorker.AgentTimeout)
assert.Equal(t, 8, cfg.Probod.CommonThirdPartyEnrichmentWorker.AgentMaxTurns)
assert.Equal(t, 0.85, cfg.Probod.CommonThirdPartyEnrichmentWorker.ConfidenceThreshold)
assert.Equal(t, 5, cfg.Probod.CommonThirdPartyEnrichmentWorker.MaxAttempts)
assert.Equal(t, 15, cfg.Probod.ThirdPartyVetting.Interval)
assert.Equal(t, 1800, cfg.Probod.ThirdPartyVetting.StaleAfter)
assert.Equal(t, 2, cfg.Probod.ThirdPartyVetting.MaxConcurrency)