Add common third party enricher worker
Introduce a poll-based worker that fills the global common_third_parties catalog (URLs, headquarter address, legal name, certifications, logo) so each tenant no longer starts from sparse, name-only rows. Enrichment is requested at row creation by ResolveOrCreateCommonThirdParty; curated seed rows are not enqueued, to avoid a re-seed storm. The pipeline uses two specialized agents plus a deterministic logo step. Agent A (company profile) resolves legal name, headquarter address, and the canonical website over web search; its website and legal name feed Agent B and the logo step. Agent B (compliance docs) resolves the legal document URLs, trust/security/status pages, and certifications using the browser read-only toolset (gated on ChromeDPAddr) plus web search. The logo step restores pkg/webinspect as a pure deterministic package and stores the discovered icon in S3, linked via logo_file_id. Each agent returns per-field value/confidence/source_url. The worker writes a column only when confidence clears a configurable threshold and the field is not externally owned (seed or human), and always records full per-field provenance in a new enrichment JSONB column so re-runs fill only gaps and human edits are never clobbered. New bookkeeping columns (enrichment_requested_at, enrichment, enrichment_attempts) back the claim queue and stale recovery; agents run outside transactions and results persist in one final transaction. The worker is opt-in: it no-ops unless its agent provider is configured. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
@@ -62,8 +62,9 @@ type (
|
||||
EvidenceDescriber EvidenceDescriberConfig `json:"evidence-describer"`
|
||||
ThirdPartyVetting ThirdPartyVettingWorkerConfig `json:"third-party-vetting-worker"`
|
||||
|
||||
TrackerMappingWorker TrackerMappingWorkerConfig `json:"tracker-mapping-worker"`
|
||||
CommonPatternEnrichmentWorker CommonPatternEnrichmentWorkerConfig `json:"common-pattern-enrichment-worker"`
|
||||
TrackerMappingWorker TrackerMappingWorkerConfig `json:"tracker-mapping-worker"`
|
||||
CommonPatternEnrichmentWorker CommonPatternEnrichmentWorkerConfig `json:"common-pattern-enrichment-worker"`
|
||||
CommonThirdPartyEnrichmentWorker CommonThirdPartyEnrichmentWorkerConfig `json:"common-third-party-enrichment-worker"`
|
||||
|
||||
ChromeDPAddr string `json:"chrome-dp-addr"`
|
||||
CustomDomains CustomDomainsConfig `json:"custom-domains"`
|
||||
|
||||
@@ -76,6 +76,22 @@ type (
|
||||
AgentMaxTurns int `json:"agent-max-turns"`
|
||||
}
|
||||
|
||||
// CommonThirdPartyEnrichmentWorkerConfig holds worker-side tuning for
|
||||
// the common-third-party enrichment background worker. LLM parameters
|
||||
// for its agents live under AgentsConfig.CommonThirdPartyEnrichment.
|
||||
// ConfidenceThreshold is the floor a resolved value must clear before
|
||||
// it is written to its column; MaxAttempts caps stale-recovery
|
||||
// retries.
|
||||
CommonThirdPartyEnrichmentWorkerConfig struct {
|
||||
Interval int `json:"interval"` // seconds between polls
|
||||
MaxConcurrency int `json:"max-concurrency"`
|
||||
StaleAfter int `json:"stale-after"` // seconds before a claim is recycled
|
||||
AgentTimeout int `json:"agent-timeout"` // seconds, single agent run
|
||||
AgentMaxTurns int `json:"agent-max-turns"`
|
||||
ConfidenceThreshold float64 `json:"confidence-threshold"`
|
||||
MaxAttempts int `json:"max-attempts"`
|
||||
}
|
||||
|
||||
// AgentToolsConfig holds API keys and settings for external tools
|
||||
// that agents can use (web search, scraping, etc.).
|
||||
AgentToolsConfig struct {
|
||||
@@ -86,15 +102,16 @@ type (
|
||||
// settings. Default is used as a fallback when an agent-specific field
|
||||
// is zero-valued.
|
||||
AgentsConfig struct {
|
||||
Providers map[string]LLMProviderConfig `json:"providers"`
|
||||
Default LLMAgentConfig `json:"defaults"`
|
||||
Probo LLMAgentConfig `json:"probo"`
|
||||
EvidenceDescriber LLMAgentConfig `json:"evidence-describer"`
|
||||
ThirdPartyVetter LLMAgentConfig `json:"third-party-vetter"`
|
||||
ThirdPartyDisambiguation LLMAgentConfig `json:"third-party-disambiguation"`
|
||||
TrackerMapping LLMAgentConfig `json:"tracker-mapping"`
|
||||
TrackerEnrichment LLMAgentConfig `json:"tracker-enrichment"`
|
||||
Tools AgentToolsConfig `json:"tools"`
|
||||
Providers map[string]LLMProviderConfig `json:"providers"`
|
||||
Default LLMAgentConfig `json:"defaults"`
|
||||
Probo LLMAgentConfig `json:"probo"`
|
||||
EvidenceDescriber LLMAgentConfig `json:"evidence-describer"`
|
||||
ThirdPartyVetter LLMAgentConfig `json:"third-party-vetter"`
|
||||
ThirdPartyDisambiguation LLMAgentConfig `json:"third-party-disambiguation"`
|
||||
TrackerMapping LLMAgentConfig `json:"tracker-mapping"`
|
||||
TrackerEnrichment LLMAgentConfig `json:"tracker-enrichment"`
|
||||
CommonThirdPartyEnrichment LLMAgentConfig `json:"common-third-party-enrichment"`
|
||||
Tools AgentToolsConfig `json:"tools"`
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user