Add common third party enricher worker
Introduce a poll-based worker that fills the global common_third_parties catalog (URLs, headquarter address, legal name, certifications, logo) so each tenant no longer starts from sparse, name-only rows. Enrichment is requested at row creation by ResolveOrCreateCommonThirdParty; curated seed rows are not enqueued, to avoid a re-seed storm. The pipeline uses two specialized agents plus a deterministic logo step. Agent A (company profile) resolves legal name, headquarter address, and the canonical website over web search; its website and legal name feed Agent B and the logo step. Agent B (compliance docs) resolves the legal document URLs, trust/security/status pages, and certifications using the browser read-only toolset (gated on ChromeDPAddr) plus web search. The logo step restores pkg/webinspect as a pure deterministic package and stores the discovered icon in S3, linked via logo_file_id. Each agent returns per-field value/confidence/source_url. The worker writes a column only when confidence clears a configurable threshold and the field is not externally owned (seed or human), and always records full per-field provenance in a new enrichment JSONB column so re-runs fill only gaps and human edits are never clobbered. New bookkeeping columns (enrichment_requested_at, enrichment, enrichment_attempts) back the claim queue and stale recovery; agents run outside transactions and results persist in one final transaction. The worker is opt-in: it no-ops unless its agent provider is configured. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
@@ -378,6 +378,52 @@ spec:
|
||||
- name: COMMON_PATTERN_ENRICHMENT_AGENT_MAX_TURNS
|
||||
value: {{ .Values.probo.commonPatternEnrichmentWorker.agentMaxTurns | quote }}
|
||||
{{- end }}
|
||||
# Common Third Party Enrichment Agent
|
||||
{{- if .Values.probo.commonThirdPartyEnrichment.provider }}
|
||||
- name: AGENT_COMMON_THIRD_PARTY_ENRICHMENT_PROVIDER
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichment.provider | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichment.modelName }}
|
||||
- name: AGENT_COMMON_THIRD_PARTY_ENRICHMENT_MODEL_NAME
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichment.modelName | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichment.temperature }}
|
||||
- name: AGENT_COMMON_THIRD_PARTY_ENRICHMENT_TEMPERATURE
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichment.temperature | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichment.maxTokens }}
|
||||
- name: AGENT_COMMON_THIRD_PARTY_ENRICHMENT_MAX_TOKENS
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichment.maxTokens | quote }}
|
||||
{{- end }}
|
||||
# Common Third Party Enrichment Worker
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.interval }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_INTERVAL
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.interval | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.maxConcurrency }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_MAX_CONCURRENCY
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.maxConcurrency | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.staleAfter }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_STALE_AFTER
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.staleAfter | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.agentTimeout }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_AGENT_TIMEOUT
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.agentTimeout | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.agentMaxTurns }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_AGENT_MAX_TURNS
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.agentMaxTurns | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.confidenceThreshold }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_CONFIDENCE_THRESHOLD
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.confidenceThreshold | quote }}
|
||||
{{- end }}
|
||||
{{- if .Values.probo.commonThirdPartyEnrichmentWorker.maxAttempts }}
|
||||
- name: COMMON_THIRD_PARTY_ENRICHMENT_MAX_ATTEMPTS
|
||||
value: {{ .Values.probo.commonThirdPartyEnrichmentWorker.maxAttempts | quote }}
|
||||
{{- end }}
|
||||
# Custom Domains
|
||||
{{- if .Values.probo.customDomains.enabled }}
|
||||
- name: CUSTOM_DOMAINS_RENEWAL_INTERVAL
|
||||
|
||||
@@ -229,6 +229,26 @@ probo:
|
||||
# agentTimeout: 45
|
||||
# agentMaxTurns: 10
|
||||
|
||||
# Common third party enrichment agent (optional, fills common_third_parties
|
||||
# metadata: URLs, address, certifications, logo).
|
||||
# commonThirdPartyEnrichment:
|
||||
# provider: "openai"
|
||||
# modelName: "gpt-4o"
|
||||
# temperature: "0.1"
|
||||
# maxTokens: "8192"
|
||||
|
||||
# Common third party enrichment worker tuning (optional; seconds for
|
||||
# interval/staleAfter/agentTimeout). confidenceThreshold is the 0-1 floor
|
||||
# a value must clear before it is written.
|
||||
# commonThirdPartyEnrichmentWorker:
|
||||
# interval: 10
|
||||
# maxConcurrency: 1
|
||||
# staleAfter: 900
|
||||
# agentTimeout: 90
|
||||
# agentMaxTurns: 12
|
||||
# confidenceThreshold: 0.7
|
||||
# maxAttempts: 3
|
||||
|
||||
# OpenTelemetry tracing (optional)
|
||||
tracing:
|
||||
enabled: true
|
||||
|
||||
@@ -331,6 +331,30 @@ probo:
|
||||
agentTimeout: 45
|
||||
agentMaxTurns: 10
|
||||
|
||||
# Common third party enrichment agent (optional, requires a provider
|
||||
# key). Powers the company-profile and compliance-docs agents that fill
|
||||
# common_third_parties metadata. Browser tools for the compliance-docs
|
||||
# agent use the chromeDpAddr endpoint when set; web search uses
|
||||
# firecrawl.apiKey when set.
|
||||
commonThirdPartyEnrichment:
|
||||
provider: ""
|
||||
modelName: ""
|
||||
temperature: ""
|
||||
maxTokens: ""
|
||||
|
||||
# Common third party enrichment background worker tuning (optional).
|
||||
# interval, staleAfter, and agentTimeout are in seconds.
|
||||
# confidenceThreshold is the floor (0-1) a resolved value must clear
|
||||
# before it is written to its column.
|
||||
commonThirdPartyEnrichmentWorker:
|
||||
interval: 10
|
||||
maxConcurrency: 1
|
||||
staleAfter: 900
|
||||
agentTimeout: 90
|
||||
agentMaxTurns: 12
|
||||
confidenceThreshold: 0.7
|
||||
maxAttempts: 3
|
||||
|
||||
# Custom domains configuration (optional)
|
||||
customDomains:
|
||||
enabled: false
|
||||
|
||||
Reference in New Issue
Block a user