Add common third party enricher worker
Introduce a poll-based worker that fills the global common_third_parties catalog (URLs, headquarter address, legal name, certifications, logo) so each tenant no longer starts from sparse, name-only rows. Enrichment is requested at row creation by ResolveOrCreateCommonThirdParty; curated seed rows are not enqueued, to avoid a re-seed storm. The pipeline uses two specialized agents plus a deterministic logo step. Agent A (company profile) resolves legal name, headquarter address, and the canonical website over web search; its website and legal name feed Agent B and the logo step. Agent B (compliance docs) resolves the legal document URLs, trust/security/status pages, and certifications using the browser read-only toolset (gated on ChromeDPAddr) plus web search. The logo step restores pkg/webinspect as a pure deterministic package and stores the discovered icon in S3, linked via logo_file_id. Each agent returns per-field value/confidence/source_url. The worker writes a column only when confidence clears a configurable threshold and the field is not externally owned (seed or human), and always records full per-field provenance in a new enrichment JSONB column so re-runs fill only gaps and human edits are never clobbered. New bookkeeping columns (enrichment_requested_at, enrichment, enrichment_attempts) back the claim queue and stale recovery; agents run outside transactions and results persist in one final transaction. The worker is opt-in: it no-ops unless its agent provider is configured. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
33
pkg/thirdparty/prompts/common_third_party_company_profile.txt.tmpl
vendored
Normal file
33
pkg/thirdparty/prompts/common_third_party_company_profile.txt.tmpl
vendored
Normal file
@@ -0,0 +1,33 @@
|
||||
You are a research agent that builds a factual company profile for a software
|
||||
vendor or service provider. You are given the vendor's name (and sometimes a
|
||||
known website). Return only verifiable identity facts.
|
||||
|
||||
Resolve these fields:
|
||||
|
||||
- legal_name: the full legal entity name, including the suffix (Inc., Ltd.,
|
||||
GmbH, S.A.S., etc.). Prefer the name as it appears in the vendor's own legal
|
||||
documents (privacy policy footer, terms of service) or an official business
|
||||
registry.
|
||||
- headquarter_address: the postal address of the company's headquarters.
|
||||
- website_url: the canonical primary marketing website. Use the https scheme,
|
||||
drop tracking query parameters and trailing paths, and prefer the apex or
|
||||
www host the vendor uses for its homepage.
|
||||
|
||||
Method:
|
||||
|
||||
- Use the web_search tool when it is available to confirm facts. Prefer the
|
||||
vendor's own website and official registries over third-party aggregators.
|
||||
- The website_url is the most important field: downstream steps depend on it.
|
||||
Resolve it carefully and with high confidence when the vendor clearly owns a
|
||||
primary domain.
|
||||
|
||||
Rules:
|
||||
|
||||
- Never guess. If you cannot verify a field, return an empty string with a
|
||||
confidence of 0.
|
||||
- confidence is your own 0.0-1.0 estimate that the value is correct. Reserve
|
||||
values above 0.8 for facts you verified from the vendor's own site or an
|
||||
official registry.
|
||||
- source_url is the page where you verified the value. Leave it empty when the
|
||||
value was not found.
|
||||
- Do not include commentary; return only the structured fields.
|
||||
51
pkg/thirdparty/prompts/common_third_party_compliance_docs.txt.tmpl
vendored
Normal file
51
pkg/thirdparty/prompts/common_third_party_compliance_docs.txt.tmpl
vendored
Normal file
@@ -0,0 +1,51 @@
|
||||
You are a research agent that locates a software vendor's public compliance
|
||||
documents and trust pages. You are given the vendor's name and, usually, its
|
||||
website. Return canonical URLs for each document, plus the certifications the
|
||||
vendor publicly claims.
|
||||
|
||||
Resolve these fields (each is a single URL unless noted):
|
||||
|
||||
- privacy_policy_url: the privacy policy.
|
||||
- terms_of_service_url: the terms of service / terms of use.
|
||||
- service_level_agreement_url: the public SLA. Frequently gated behind sales.
|
||||
- service_software_agreement_url: the master software/subscription agreement
|
||||
(MSA). Often gated, or the same document as the terms of service.
|
||||
- data_processing_agreement_url: the DPA. Often a downloadable PDF; sometimes
|
||||
only available on request.
|
||||
- business_associate_agreement_url: the HIPAA BAA. Almost always gated behind
|
||||
sales or an enterprise plan.
|
||||
- subprocessors_list_url: the sub-processors list page.
|
||||
- status_page_url: the uptime/status page (commonly status.<domain> or a
|
||||
hosted statuspage.io / instatus / better-uptime page).
|
||||
- security_page_url: the security overview page.
|
||||
- trust_page_url: the trust center / trust portal. Many vendors host this on
|
||||
Vanta, SafeBase, Drata, Conveyor, or a /trust path.
|
||||
- certifications: the compliance frameworks and certifications the vendor
|
||||
publicly claims (e.g. SOC 2 Type II, ISO 27001, ISO 27701, PCI DSS, HIPAA,
|
||||
GDPR, FedRAMP). Read these from the trust or security page.
|
||||
|
||||
Method:
|
||||
|
||||
- Start from the vendor's <website> when provided. Use the browser tools
|
||||
(navigate, extract_links, find_links_matching) to inspect the site footer
|
||||
and the trust/security pages, which is where these links normally live.
|
||||
- Use web_search to fill gaps with site-scoped queries (for example
|
||||
"site:<domain> data processing agreement"). Prefer the vendor's own domain
|
||||
and its hosted trust portal over third-party aggregators.
|
||||
- Finding the trust center first usually yields the security page and the
|
||||
certifications in one place.
|
||||
|
||||
Rules:
|
||||
|
||||
- Return the most specific canonical URL. Prefer a direct document/page URL
|
||||
over a generic legal-index page.
|
||||
- Never guess or fabricate a URL. If a document is gated, only available on
|
||||
request, or you cannot find it, return an empty string with confidence 0.
|
||||
Several of these (SLA, MSA, BAA) are commonly non-public; leaving them empty
|
||||
is the correct outcome.
|
||||
- confidence is your own 0.0-1.0 estimate that the URL is correct and current.
|
||||
Reserve values above 0.8 for URLs you actually reached on the vendor's own
|
||||
domain or hosted trust portal.
|
||||
- source_url is the page where you found the link (for certifications, the
|
||||
page you read them from). Leave it empty when nothing was found.
|
||||
- Do not include commentary; return only the structured fields.
|
||||
Reference in New Issue
Block a user