The common-third-party enrichment pipeline ran Agent B (compliance
docs), Agent C (owned domains), and the deterministic logo step
sequentially even though, once Agent A resolves the website, the
three depend only on that website and not on each other. Fan them
out across goroutines under a WaitGroup so wall time is the slowest
of the three rather than their sum. Each step builds its own per-run
browser and writes only into its own locals; the shared LLM, HTTP,
and FileManager clients are safe for concurrent use and the database
is untouched until persist. Results merge in a fixed order so
runErrors and log output stay deterministic.
Also replace the single-sentence confidence guidance in the three
agent prompts with an explicit, calibrated rubric tied to evidence
strength, and remind the model that a downstream threshold gates
persistence so it should neither inflate nor deflate its estimates.
Signed-off-by: Émile Ré <emile@probo.com>
Add a domain-discovery step to the enrichment pipeline so the catalog's
domain set, previously written only by the curated seed, grows
automatically. A focused agent enumerates the registrable domains a
vendor owns and operates - marketing, product and sub-brand, app, API,
and CDN/asset domains - from links seen while browsing and from web
search, anchored on the website resolved earlier in the run.
A deterministic ownership gate reduces the candidates to eTLD+1 and
keeps only those that clear a strict confidence floor and match the
vendor by domain label. Shared tracker-delivery and CDN infrastructure
is dropped unless the vendor itself is that provider, in which case its
own brand-matching domain passes a stricter exact-label check. The
survivors are upserted into common_third_party_domains in the run's
final transaction and recorded in the enrichment payload, feeding the
tracker-mapping domain step and disambiguation.
Signed-off-by: Émile Ré <emile@probo.com>
Give the company-profile agent (Agent A) the read-only browser toolset
and build it per-run, so it can read footer, imprint, about, and legal
pages and follow a product domain to the corporate one to resolve the
legal name and headquarters address rather than failing cold.
Make the website the hard precondition: when Agent A cannot resolve a
canonical website, skip the compliance-docs agent and logo step instead
of running them blind, which previously produced inconsistent
cross-domain document URLs. Fall back to the catalog display name for
the legal name when nothing better is found, recorded with a distinct
provenance status so a later real find overwrites it.
Rewrite both enrichment prompts in the project's role/task/instructions
XML style, add a domain-consistency rule for document URLs and a
tool-budget directive, and document the prompt style as a rule and guide.
Fix the find_links_matching browser tool, which double-encoded its
pattern and made JSON.parse fail on every keyword, starving any agent
that used it until it hit the turn cap. Salvage output when an agent
exhausts its turn budget while still exploring with a pending structured
output by forcing one final synthesis turn instead of failing outright.
Signed-off-by: Émile Ré <emile@probo.com>
Introduce a poll-based worker that fills the global common_third_parties
catalog (URLs, headquarter address, legal name, certifications, logo)
so each tenant no longer starts from sparse, name-only rows. Enrichment
is requested at row creation by ResolveOrCreateCommonThirdParty; curated
seed rows are not enqueued, to avoid a re-seed storm.
The pipeline uses two specialized agents plus a deterministic logo step.
Agent A (company profile) resolves legal name, headquarter address, and
the canonical website over web search; its website and legal name feed
Agent B and the logo step. Agent B (compliance docs) resolves the legal
document URLs, trust/security/status pages, and certifications using the
browser read-only toolset (gated on ChromeDPAddr) plus web search. The
logo step restores pkg/webinspect as a pure deterministic package and
stores the discovered icon in S3, linked via logo_file_id.
Each agent returns per-field value/confidence/source_url. The worker
writes a column only when confidence clears a configurable threshold and
the field is not externally owned (seed or human), and always records
full per-field provenance in a new enrichment JSONB column so re-runs
fill only gaps and human edits are never clobbered. New bookkeeping
columns (enrichment_requested_at, enrichment, enrichment_attempts) back
the claim queue and stale recovery; agents run outside transactions and
results persist in one final transaction.
The worker is opt-in: it no-ops unless its agent provider is configured.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
