Discover and persist common third-party domains
Add a domain-discovery step to the enrichment pipeline so the catalog's domain set, previously written only by the curated seed, grows automatically. A focused agent enumerates the registrable domains a vendor owns and operates - marketing, product and sub-brand, app, API, and CDN/asset domains - from links seen while browsing and from web search, anchored on the website resolved earlier in the run. A deterministic ownership gate reduces the candidates to eTLD+1 and keeps only those that clear a strict confidence floor and match the vendor by domain label. Shared tracker-delivery and CDN infrastructure is dropped unless the vendor itself is that provider, in which case its own brand-matching domain passes a stricter exact-label check. The survivors are upserted into common_third_party_domains in the run's final transaction and recorded in the enrichment payload, feeding the tracker-mapping domain step and disambiguation. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
11
pkg/thirdparty/common_third_party_enrichment.go
vendored
11
pkg/thirdparty/common_third_party_enrichment.go
vendored
@@ -71,6 +71,16 @@ type (
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// EnrichmentDomainMeta is the provenance of one domain written to
|
||||
// common_third_party_domains during a run, recorded in the enrichment
|
||||
// payload so the discovered domain set is auditable.
|
||||
EnrichmentDomainMeta struct {
|
||||
Domain string `json:"domain"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
SourceURL string `json:"source_url,omitempty"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// EnrichmentMetadata is the full payload stored in the enrichment
|
||||
// JSON column: run-level bookkeeping plus per-field provenance keyed
|
||||
// by the column name.
|
||||
@@ -80,6 +90,7 @@ type (
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Fields map[string]EnrichmentFieldMeta `json:"fields"`
|
||||
Domains []EnrichmentDomainMeta `json:"domains,omitempty"`
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user