Discover and persist common third-party domains

Add a domain-discovery step to the enrichment pipeline so the catalog's
domain set, previously written only by the curated seed, grows
automatically. A focused agent enumerates the registrable domains a
vendor owns and operates - marketing, product and sub-brand, app, API,
and CDN/asset domains - from links seen while browsing and from web
search, anchored on the website resolved earlier in the run.

A deterministic ownership gate reduces the candidates to eTLD+1 and
keeps only those that clear a strict confidence floor and match the
vendor by domain label. Shared tracker-delivery and CDN infrastructure
is dropped unless the vendor itself is that provider, in which case its
own brand-matching domain passes a stricter exact-label check. The
survivors are upserted into common_third_party_domains in the run's
final transaction and recorded in the enrichment payload, feeding the
tracker-mapping domain step and disambiguation.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-06-11 17:37:37 +02:00
parent 8182c61fa8
commit 7649f19e33
6 changed files with 637 additions and 5 deletions

View File

@@ -71,6 +71,16 @@ type (
UpdatedAt time.Time `json:"updated_at"`
}
// EnrichmentDomainMeta is the provenance of one domain written to
// common_third_party_domains during a run, recorded in the enrichment
// payload so the discovered domain set is auditable.
EnrichmentDomainMeta struct {
Domain string `json:"domain"`
Confidence float64 `json:"confidence"`
SourceURL string `json:"source_url,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
}
// EnrichmentMetadata is the full payload stored in the enrichment
// JSON column: run-level bookkeeping plus per-field provenance keyed
// by the column name.
@@ -80,6 +90,7 @@ type (
Status string `json:"status"`
Error string `json:"error,omitempty"`
Fields map[string]EnrichmentFieldMeta `json:"fields"`
Domains []EnrichmentDomainMeta `json:"domains,omitempty"`
}
)