Filter first-party domains from tracker mapping
Tracker scripts loaded through a first-party reverse proxy (e.g. t.probo.com proxying PostHog) share the scanned site's eTLD+1 and were incorrectly matched against the site owner's own CommonThirdParty entry in matchByDomain. This caused trackers like ph_phc_* to be attributed to the site owner instead of PostHog. Load the CookieBanner origin in Process and pass it to both matchByDomain and identifyWithAgent. Both now filter out initiator domains whose eTLD+1 matches the site before querying the catalog or feeding domains to the LLM agent. The prompt is also updated to warn about proxy domains. Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
@@ -23,6 +23,7 @@ Return a structured JSON response with:
|
||||
- For localStorage keys, include the type: "[name] localStorage tracking script".
|
||||
- If the first query returns nothing useful, broaden: "[name] web tracker" or "site:[domain] cookie documentation".
|
||||
- Stop searching once you get a confident match; do not exhaust all query slots if the first one succeeds.
|
||||
- When evaluating web search results, verify that the tracker name discussed in the result shares a meaningful prefix with the pattern you are identifying. Trackers with different prefixes are distinct — for example, _hjCookieTest (Hotjar's _hj prefix) must not be confused with a pattern named cookietest (no _hj prefix). If the search result discusses a tracker whose prefix does not match, discard it and continue searching or lower your confidence.
|
||||
|
||||
4. Common cookie naming conventions to recognize:
|
||||
- _ga*, _gid, _gat*: Google Analytics
|
||||
@@ -35,7 +36,7 @@ Return a structured JSON response with:
|
||||
- hubspot*: HubSpot
|
||||
- _cls_*: Clarity (Microsoft)
|
||||
|
||||
5. The observed domains are strong signals. If the cookie comes from a well-known tracking domain (e.g. doubleclick.net, facebook.com, analytics.google.com), that is strong evidence of the third party.
|
||||
5. The observed domains are useful signals but not conclusive on their own. Many sites load third-party tracker scripts through a first-party reverse proxy (e.g. t.example.com proxying PostHog). When a domain matches the scanned site, it reveals nothing about which third party set the tracker — rely on the naming convention or a database/web search instead. First-party proxy domains are filtered before they reach you, but if you still see the scanned site's own domain, ignore it as evidence. Well-known third-party tracking domains (e.g. doubleclick.net, facebook.com, analytics.google.com) remain strong evidence.
|
||||
|
||||
6. Be conservative with confidence:
|
||||
- 0.9-1.0: exact pattern match found in database or unmistakable naming convention + matching domain
|
||||
@@ -45,6 +46,8 @@ Return a structured JSON response with:
|
||||
|
||||
7. If you truly cannot identify the tracker, set third_party_name to an empty string and confidence below 0.3.
|
||||
|
||||
8. For the category field, use one of: {{.Categories}}.
|
||||
8. Only attribute a tracker to a company or service when you have concrete evidence: a database match, an unmistakable naming convention, or a clear web search result. Never guess or invent attributions based on vague similarity or general knowledge. If no evidence supports a match, return an empty third_party_name with confidence below 0.3.
|
||||
|
||||
9. For the category field, use one of: {{.Categories}}.
|
||||
Most cookies fall under ANALYTICS or MARKETING.
|
||||
</instructions>
|
||||
|
||||
@@ -29,6 +29,7 @@ import (
|
||||
"go.probo.inc/probo/pkg/llm"
|
||||
"go.probo.inc/probo/pkg/slug"
|
||||
"go.probo.inc/probo/pkg/thirdparty"
|
||||
"go.probo.inc/probo/pkg/uri"
|
||||
)
|
||||
|
||||
type trackerMappingHandler struct {
|
||||
@@ -109,20 +110,27 @@ func (h *trackerMappingHandler) Process(ctx context.Context, tp coredata.Tracker
|
||||
if tp.CommonTrackerPatternID != nil {
|
||||
commonPatternID = tp.CommonTrackerPatternID
|
||||
} else {
|
||||
scope := coredata.NewScopeFromObjectID(tp.ID)
|
||||
|
||||
var banner coredata.CookieBanner
|
||||
if err := banner.LoadByID(ctx, tx, scope, tp.CookieBannerID); err != nil {
|
||||
return fmt.Errorf("cannot load cookie banner for domain filtering: %w", err)
|
||||
}
|
||||
|
||||
commonPatternID, err = h.matchByPattern(ctx, tx, tp)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot match by pattern: %w", err)
|
||||
}
|
||||
|
||||
if commonPatternID == nil {
|
||||
commonPatternID, err = h.matchByDomain(ctx, tx, tp)
|
||||
commonPatternID, err = h.matchByDomain(ctx, tx, tp, banner.Origin)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot match by domain: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
if commonPatternID == nil && h.mappingAgent != nil {
|
||||
commonPatternID, err = h.identifyWithAgent(ctx, tx, tp)
|
||||
commonPatternID, err = h.identifyWithAgent(ctx, tx, tp, banner.Origin)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot identify with agent: %w", err)
|
||||
}
|
||||
@@ -212,10 +220,16 @@ func (h *trackerMappingHandler) matchByPattern(
|
||||
// overlap the pattern's observed initiator domains, and upserts a
|
||||
// CommonTrackerPattern linking the two. As with matchByPattern,
|
||||
// third-party resolution is deferred to promoteThirdParty.
|
||||
//
|
||||
// Domains that share the scanned site's eTLD+1 are filtered out before
|
||||
// querying. Tracker scripts loaded through a first-party proxy (e.g.
|
||||
// t.probo.com proxying PostHog on a probo.com site) would otherwise
|
||||
// match the site owner's own CommonThirdParty entry.
|
||||
func (h *trackerMappingHandler) matchByDomain(
|
||||
ctx context.Context,
|
||||
tx pg.Tx,
|
||||
tp coredata.TrackerPattern,
|
||||
siteOrigin string,
|
||||
) (*gid.GID, error) {
|
||||
var trackers coredata.DetectedTrackers
|
||||
|
||||
@@ -224,6 +238,8 @@ func (h *trackerMappingHandler) matchByDomain(
|
||||
return nil, fmt.Errorf("cannot load initiator domains: %w", err)
|
||||
}
|
||||
|
||||
domains = uri.FilterFirstPartyDomains(domains, siteOrigin)
|
||||
|
||||
if len(domains) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
@@ -266,6 +282,7 @@ func (h *trackerMappingHandler) identifyWithAgent(
|
||||
ctx context.Context,
|
||||
tx pg.Tx,
|
||||
tp coredata.TrackerPattern,
|
||||
siteOrigin string,
|
||||
) (*gid.GID, error) {
|
||||
var trackers coredata.DetectedTrackers
|
||||
|
||||
@@ -274,6 +291,8 @@ func (h *trackerMappingHandler) identifyWithAgent(
|
||||
h.logger.WarnCtx(ctx, "cannot load initiator domains for agent", log.Error(err))
|
||||
}
|
||||
|
||||
domains = uri.FilterFirstPartyDomains(domains, siteOrigin)
|
||||
|
||||
prompt := buildAgentPrompt(tp, domains)
|
||||
|
||||
agentCtx, cancel := context.WithTimeout(ctx, agentTimeout)
|
||||
|
||||
Reference in New Issue
Block a user