rohit/probo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add common third party enricher worker

Browse Source 
Introduce a poll-based worker that fills the global common_third_parties
catalog (URLs, headquarter address, legal name, certifications, logo)
so each tenant no longer starts from sparse, name-only rows. Enrichment
is requested at row creation by ResolveOrCreateCommonThirdParty; curated
seed rows are not enqueued, to avoid a re-seed storm.

The pipeline uses two specialized agents plus a deterministic logo step.
Agent A (company profile) resolves legal name, headquarter address, and
the canonical website over web search; its website and legal name feed
Agent B and the logo step. Agent B (compliance docs) resolves the legal
document URLs, trust/security/status pages, and certifications using the
browser read-only toolset (gated on ChromeDPAddr) plus web search. The
logo step restores pkg/webinspect as a pure deterministic package and
stores the discovered icon in S3, linked via logo_file_id.

Each agent returns per-field value/confidence/source_url. The worker
writes a column only when confidence clears a configurable threshold and
the field is not externally owned (seed or human), and always records
full per-field provenance in a new enrichment JSONB column so re-runs
fill only gaps and human edits are never clobbered. New bookkeeping
columns (enrichment_requested_at, enrichment, enrichment_attempts) back
the claim queue and stale recovery; agents run outside transactions and
results persist in one final transaction.

The worker is opt-in: it no-ops unless its agent provider is configured.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-06-10 13:49:56 +02:00
parent 99d568d07d
commit 229c6b99c6
23 changed files with 2325 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
pkg/probodconfig/llm_config.go
View File

@@ -76,6 +76,22 @@ type (
AgentMaxTurns int `json:"agent-max-turns"`
}
// CommonThirdPartyEnrichmentWorkerConfig holds worker-side tuning for
// the common-third-party enrichment background worker. LLM parameters
// for its agents live under AgentsConfig.CommonThirdPartyEnrichment.
// ConfidenceThreshold is the floor a resolved value must clear before
// it is written to its column; MaxAttempts caps stale-recovery
// retries.
CommonThirdPartyEnrichmentWorkerConfig struct {
Interval int `json:"interval"` // seconds between polls
MaxConcurrency int `json:"max-concurrency"`
StaleAfter int `json:"stale-after"` // seconds before a claim is recycled
AgentTimeout int `json:"agent-timeout"` // seconds, single agent run
AgentMaxTurns int `json:"agent-max-turns"`
ConfidenceThreshold float64 `json:"confidence-threshold"`
MaxAttempts int `json:"max-attempts"`
}
// AgentToolsConfig holds API keys and settings for external tools
// that agents can use (web search, scraping, etc.).
AgentToolsConfig struct {
@@ -86,15 +102,16 @@ type (
// settings. Default is used as a fallback when an agent-specific field
// is zero-valued.
AgentsConfig struct {
Providers map[string]LLMProviderConfig `json:"providers"`
Default LLMAgentConfig `json:"defaults"`
Probo LLMAgentConfig `json:"probo"`
EvidenceDescriber LLMAgentConfig `json:"evidence-describer"`
ThirdPartyVetter LLMAgentConfig `json:"third-party-vetter"`
ThirdPartyDisambiguation LLMAgentConfig `json:"third-party-disambiguation"`
TrackerMapping LLMAgentConfig `json:"tracker-mapping"`
TrackerEnrichment LLMAgentConfig `json:"tracker-enrichment"`
Tools AgentToolsConfig `json:"tools"`
Providers map[string]LLMProviderConfig `json:"providers"`
Default LLMAgentConfig `json:"defaults"`
Probo LLMAgentConfig `json:"probo"`
EvidenceDescriber LLMAgentConfig `json:"evidence-describer"`
ThirdPartyVetter LLMAgentConfig `json:"third-party-vetter"`
ThirdPartyDisambiguation LLMAgentConfig `json:"third-party-disambiguation"`
TrackerMapping LLMAgentConfig `json:"tracker-mapping"`
TrackerEnrichment LLMAgentConfig `json:"tracker-enrichment"`
CommonThirdPartyEnrichment LLMAgentConfig `json:"common-third-party-enrichment"`
Tools AgentToolsConfig `json:"tools"`
}
)