The source headers, LICENSE files, and license metadata had drifted
apart. Align the entire project to MIT:
- Convert every source-file header to the MIT text across all comment
styles (Go, TS, TSX, JS, MJS, SQL, CSS, GraphQL, shell), including
SPDX-License-Identifier tags
- Set the root and cookie-banner LICENSE files to the MIT text with a
"MIT License" title line
- Switch the package.json license fields, Docker image label, and
cookie-banner README to MIT
- Update docs and the genmodels header generator accordingly
- Normalize copyright lines to a single format
(Copyright (c) <year(s)> Probo Inc <hello@probo.com>.): unify the
hello@getprobo.com and hello@probo.inc emails to hello@probo.com and
the comma-separated years to a hyphenated range
Genuine third-party references are intentionally left untouched: the
Lucide icon attributions (Lucide is ISC) and the trivy dependency
license allowlist.
Signed-off-by: Sacha Al Himdani <sacha@probo.com>
Subprocessor filtering for the compliance portal happens in the backend
rather than the client. Add a SubprocessorFilter (query, category,
country) to the trust API's subprocessors connection, thread it through
the resolver and service, and extend the coredata ThirdParty filter with
category equality and country array membership. The connection stores the
filter so totalCount reflects the filtered set. Add e2e coverage for the
new filtering.
On the frontend, convert the page to a refetchable fragment whose filter
arguments are driven by URL-persisted, debounced toolbar state (category
and region selects plus a search field), populate the dropdowns from an
unfiltered facet selection, and offer to clear filters from the empty
state.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker re-arms same-banner siblings after a pattern
resolves a vendor. Its predicate only excluded promoted siblings
(third_party_id IS NULL), but since org-party auto-creation was dropped a
pattern can resolve a common third party yet never gain an org
third_party_id. Those siblings, and terminal first-party ones, stayed
eligible forever, so every cascade step re-enqueued and reprocessed them,
amplifying Process runs to O(N^2) per banner. The deadlock fix in the
last release removed the rollbacks that had accidentally throttled the
cascade, so the latent amplification surfaced as an INFO-log flood.
Tighten the re-enqueue to skip siblings already linked to a catalog row
that carries a common third party or marked FIRST_PARTY, dropping
per-banner reprocessing back to O(N). Also demote the two per-run handler
logs ("mapped tracker pattern", "re-enqueued unmapped sibling tracker
patterns") to Debug so routine processing no longer logs at INFO.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs many Process calls in parallel. In
Phase 4 a single transaction locked the worker's own claimed pattern
row via UpdateMapping and then locked sibling rows on the same banner
via the re-enqueue. Two workers mapping sibling patterns on one banner
each held their own row and waited on the other's, forming a lock cycle
that Postgres aborted with deadlock detected (40P01).
Split the sibling re-enqueue into its own short transaction that runs
after the mapping commits, so the claimed-row lock is released before
any sibling row is locked. Also take the sibling UPDATE row locks in a
deterministic id order through an ORDER BY id ... FOR UPDATE subquery,
so overlapping re-enqueues can no longer invert lock order between
themselves. The re-enqueue only flags siblings, so deferring it past
the commit is safe and lets reprocessed siblings observe committed data.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-pattern source ranking collapsed HTTP into the PRE_EXISTING
tier, so a cookie first enumerated as pre-existing and later re-observed
only via a Set-Cookie response header stayed pre-existing. That left it
on the agent-skipped tier (isPreExistingSource), even though an HTTP
server-set cookie is real page evidence, not the extension-state
catch-all the skip was built to suppress.
Give HTTP its own rank between SCRIPT and EXTENSION
(SCRIPT > HTTP > EXTENSION > PRE_EXISTING) in both sourceRank and the
bestSource merge rollup, so an HTTP re-detection now promotes the
pattern and re-arms mapping, unblocking the identification agent.
Signed-off-by: Émile Ré <emile@probo.com>
Three paths could leave a catalog row's attribution out of step with its
common_third_party_id. A FIRST_PARTY reclassification in the mapping
worker kept a stale org ThirdParty link instead of clearing it. The
upsert requeued terminal FIRST_PARTY rows for enrichment on a vendor
they never adopt, since the vendor-preservation clause nulls it. And the
proboctl upsert command did not normalize the verdict when an operator
linked or unlinked a vendor without passing --attribution.
Clear the org link on a first-party verdict, exclude FIRST_PARTY rows
from the enrichment requeue, and have the CLI downgrade THIRD_PARTY to
UNDETERMINED on unlink and promote UNDETERMINED to THIRD_PARTY on link.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-pattern catalog was binary (linked to a vendor or not), so
generic and first-party artifacts (loglevel keys, wallet-extension keys,
an org's own trackers) were retried forever and, once one row was wrongly
attributed, re-propagated to every organization with no re-check.
Give catalog rows a terminal attribution verdict (UNDETERMINED,
THIRD_PARTY, FIRST_PARTY): FIRST_PARTY short-circuits the whole mapping
pipeline so the artifact is never attributed again. Gate deterministic
vendor adoption behind a trust bar so only curated/operator rows
auto-propagate; lower-confidence agent/heuristic rows are reused as hints
and re-resolved, and an independent agent re-confirmation corroborates and
promotes them. Make the mapping agent emit an evidence source and reject
any attribution that lacks concrete evidence, and let it declare a
first-party verdict. Skip the speculative agent for PRE_EXISTING-source
patterns, whose low signal invites invented vendors.
Add proboctl "ctp mark-first-party" and an --attribution list filter to
audit and remediate existing wrong links, and a cursor rule documenting
migration naming so the timestamp is taken from date -u, not invented.
Signed-off-by: Émile Ré <emile@probo.com>
The migration dropped the DEFAULT on
common_tracker_patterns.enrichment_attempts, so the column is now
NOT NULL with no default and inserts must set it explicitly. The e2e
seedCommonTrackerPattern helper still did a raw insert without it,
which broke TestThirdParty_ImportFromCommon and
TestTrackerPattern_CommonTrackerPatternID with a 23502 violation. Pass
0 for the seeded rows.
Signed-off-by: Émile Ré <emile@probo.com>
Make common_tracker_patterns and common_third_parties share one
enrichment-tracking model and fix the misleading proboctl status.
Both tables now carry the enrichment JSONB provenance payload, an
enrichment_attempts counter, and a last_enrichment_attempt_at clock.
On common_tracker_patterns the enriched_at done-flag is renamed to
last_enrichment_attempt_at and stamped at claim time, so it is truthful
to "attempt" rather than "success". A row is considered to have been
through the workflow when it carries an enrichment payload, not when a
timestamp is set, which lets stale recovery key off the payload being
absent with budget remaining, exactly like common_third_parties.
The claim path reads the attempt counter and timestamp back via
RETURNING so the in-memory receiver matches the database clock instead
of a separate app-side time.Now.
The enricher builds a per-field provenance payload (description and
third-party outcomes plus the mapping attribution) and persists it via
UpdateEnrichment, named to mirror the common-third-party sibling. The
common pattern enrichment worker gains a max-attempts ceiling so a
permanently failing row stops looping.
proboctl now shows "enriched" only when every field the last run
recorded an outcome for resolved a value, otherwise "partial (X/Y)",
replacing the misleading "enriched (no description)" label.
Signed-off-by: Émile Ré <emile@probo.com>
Unbounded LoadAll* loaders materialised an entire result set in one
query with no ceiling. A table that is small in development can grow
without bound in production, so these loaders were a latent memory
and query-time hazard.
Remove the LoadAll* methods from pkg/coredata and walk the cursor-
paginated LoadBy* siblings instead through a shared page.LoadAll
helper. The helper advances a MaxCursorSize forward cursor until the
result set is exhausted and concatenates the pages. It caps a single
call at MaxLoadAllPages (20) batches of 500 rows and errors past that
rather than materialising an unbounded set, so a runaway caller fails
loudly instead of exhausting memory.
Callers that genuinely need every row now express that explicitly,
and the coredata load-naming rule and docs are updated to discourage
new unbounded loaders.
Signed-off-by: Sacha Al Himdani <sacha@probo.com>
The enrichment agent now reads full cookie-database and vendor pages
through the browser. The facts it gathers (setter, purpose, data
stored) are not protectable, but a source page's written description
is original expression. Instruct the agent to state the facts in its
own words and never copy a sentence verbatim, so stored descriptions
stay clear of third-party prose.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern enrichment agents only had
web search, which returns title/url/snippet, so they could never
open a cookie-database or cookie-policy page to read which vendor
actually sets a tracker. This mis-attributed setters whose snippet
is misleading (e.g. _li_* read as LinkedIn rather than LiveIntent).
Wire the read-only headless-browser toolset into both agents, gated
on a configured Chrome endpoint, mirroring the common-third-party
enrichment worker: agent construction moves into the run path so each
run can carry a per-run browser that is closed when the run returns.
The prompts now direct the agent to open a promising result and read
the named setter from the full page text. Both agents stay unchanged
when no Chrome endpoint is configured.
Signed-off-by: Émile Ré <emile@probo.com>
Address PR review feedback: rename the shared string-helper package
from strutil to stringsx to avoid the discouraged util suffix and the
collision with the standard strings package, updating all import paths
and call sites.
Replace the manual wg.Add/wg.Done bookkeeping in the enrichment worker
with wg.Go, which is less error-prone.
Signed-off-by: Émile Ré <emile@probo.com>
The alphanumeric-normalisation helper was duplicated verbatim in the
common-third-party owned-domains resolver and the cookiebanner tracker
mapping worker. Hoist it into a new dependency-free strutil package so
both call sites share one implementation and one test.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the bare `getprobo.com` domain and the `www.getprobo.com`
marketing host with `probo.com` / `www.probo.com` across the codebase.
Functional subdomains (app, console, notification, custom, test,
cookie-banner, compliance) keep their existing `getprobo.com` hosts,
and changelog entries are left untouched.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping worker had been reduced to catalog resolution only,
which removed not just the auto-creation of an org ThirdParty but also
the auto-linking of an existing one. Only the creation needed to go: it
raced the load-then-create check and produced duplicate vendors.
Restore the full org ThirdParty resolution (exact common-id link,
sibling direct-link, high-confidence heuristic, and the disambiguation
agent) and remove only the CreateFromCommon branch and its
categorisation gate. When nothing matches, the worker now leaves
third_party_id unset rather than creating a vendor; creation happens
exclusively through the explicit ImportFromCommon action. Drop the
now-dead CreateFromCommon helper and rename match.go to common_match.go.
Fix a latent test bug surfaced by actually running the DB-backed suite
(skipped in CI without Postgres): the heuristic-match candidate lacked
Level 1, so the level-filtered candidate loader excluded it and the old
fallback create masked the miss.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker materialized a per-org ThirdParty for every
categorized tracker, linking or creating one through heuristic and
disambiguation-agent matching. Concurrent mapping of two patterns for
the same common third party raced the load-then-create check and left
duplicate org third parties with the same name.
Reduce the worker to catalog resolution only: it resolves the shared
common_tracker_pattern_id / common_third_party_id link and leaves
third_party_id untouched, preserving any link set elsewhere. Org third
parties will instead be created through an explicit per-vendor import
action added in a later commit.
Remove resolveOrgThirdParty, prepareOrgThirdParty, the sibling
direct-link signal, and the disambiguation-agent wiring (including its
constructor parameter and buildTrackerAgents return), and update the
worker tests to assert the catalog link is resolved while third_party_id
is preserved.
Signed-off-by: Émile Ré <emile@probo.com>
The ResetProgressFunc callback streamed per-phase messages from inside
the reset transaction, so a rollback could leave already-printed
progress describing work that never committed. Remove the callback
entirely; the final result summary printed by proboctl still conveys
the outcome without the misleading intermediate output.
Signed-off-by: Émile Ré <emile@probo.com>
When IP geolocation does not resolve a country, or resolves one with no
known cookie-consent regulation (common on localhost and unmapped
regions), the banner previously fell back to OPT_OUT with no recorded
regulation. Apply GDPR (OPT_IN) as the safe default in that case so the
strictest consent model wins when origin is unknown.
To keep consent records auditable, stamp each one with a regulation
source of DETECTED (resolved from geolocation) or DEFAULT (fell back to
GDPR). The shared cookiebanner.ResolveRegulation helper centralizes the
decision for both the config and consent endpoints, and the new value is
exposed through GraphQL, MCP, the CLI, the n8n node, and the console
consent-records views.
Signed-off-by: Émile Ré <emile@probo.com>
The cookie-database aggregator backstop normalised the agent's
third-party name and looked it up against bare brand keys, but
normalizeAlnum folds the eTLD into the key (cookiedatabase.org ->
cookiedatabaseorg). Domain- and URL-form attributions therefore
slipped past the exact lookup, letting noisy aggregator names be
accepted instead of discarded.
Add uri.DomainLabel to reduce a host-like string to its primary
registrable label and check it alongside the normalised name, so
both brand ("Cookiepedia") and domain forms ("cookiedatabase.org",
"https://www.cookiepedia.co.uk/list") resolve to the same key.
Signed-off-by: Émile Ré <emile@probo.com>
Cookie-database and consent-directory sites (Cookifi, Cookiepedia,
cookiedatabase.org, CookieServe, ...) rank highly in web search only
because they catalog cookies, not because they set them. The mapping
agent could attribute a tracker to the directory operator itself
instead of the vendor the page names.
Instruct the agent to treat such results as reference directories and
extract the named vendor, never the operator, while keeping a CMP's own
product cookie attributable (OptanonConsent -> OneTrust, CookieConsent
-> Cookiebot). Add a conservative code backstop that discards
attributions to pure aggregators, scoped to exclude CMP vendors so
legitimate own-cookie attributions survive.
Signed-off-by: Émile Ré <emile@probo.com>
Patterns like "ethereum-https://letaido.com" embed the scanned site's
own origin, appended either by a browser extension (e.g. a wallet
injecting window.ethereum) or by an owner-set tracker. The mapping
agent read that embedded domain and attributed the site to itself as a
third party ("Letaido"), but a site owner is never a third party of its
own site.
Pass the scanned site's eTLD+1 to the agent as a <scanned_site> signal
and instruct it to ignore a domain embedded in the pattern that matches
the site, returning low confidence when that is the only cue. Add a
deterministic backstop that discards any agent attribution whose vendor
name resolves to the site's own domain, so the pattern falls through to
the unmatched fallback regardless of whether the model honored the
prompt. Deterministic catalog, naming, and sibling signals run before
the agent and are untouched, so an independent strong vendor signal
still attributes normally.
Signed-off-by: Émile Ré <emile@probo.com>
The reset-trackers operator command reset every uncategorised,
non-excluded pattern of a banner and printed only a single summary
line once the transaction committed, giving no feedback during long
rebuilds.
Add a --keyword flag that scopes both the glob decomposition and the
mapping reset to patterns whose pattern or display name contains the
substring. The match lives in a new TrackerPatternFilter.WithPatternKeyword
field so it runs in SQL and is shared by the glob load and the
ResetAndRequestMappingByCookieCategoryID update, keeping the two in
lockstep. The banner-wide pattern-analysis re-arm is left unscoped.
Thread an optional progress callback through ResetBannerTrackers so the
command streams per-phase updates (category load, per-glob decomposition,
mapping reset, analysis re-arm) as the work runs.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the many-to-many junction table with a direct
parent_third_party_id foreign key on third_parties. Each
sub-third-party now belongs to exactly one parent, making
duplicates across parents independent entities.
Replace the firstLevel boolean with an integer level field
(1 = direct, 2+ = parent level + 1) to support arbitrary
nesting depth.
Remove the createThirdPartyThirdPartyMapping and
deleteThirdPartyThirdPartyMapping mutations, the CLI
link/unlink commands, and the corresponding MCP tools.
Creating a child third party now just requires passing
parentThirdPartyId on the existing createThirdParty mutation.
The frontend walks the parentThirdParty chain to build
display names like "Name (Ancestor1/Ancestor2)" and shows
clickable ancestor links on the detail page.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping, common-pattern enrichment, and third-party
disambiguation agents were all built from one shared
TrackerAgentsConfig fed by a single tracker-mapping config slot. That
forced a single AgentTimeout to be reused and patched per worker, and
two unrelated max-turns fields to share one struct.
Split the in-code config into TrackerMappingAgentConfig,
TrackerEnrichmentAgentConfig, and DisambiguationAgentConfig, each with
its own timeout and max-turns, and add dedicated tracker-enrichment and
third-party-disambiguation provider slots (the latter resolving next to
third-party-vetter). Enrichment and disambiguation fall back to the
tracker-mapping slot when their own provider is unset, preserving
single-config deployments.
Drop the shared pkg/agentsbuild package and duplicate its small wiring
into probod and proboctl so the two executables stay decoupled. Wire
the new env vars, builder test coverage, and Helm values.
Signed-off-by: Émile Ré <emile@probo.com>
Expose a non-destructive re-trigger that re-arms tracker policy
generation for a banner that already has a published version, so the
tracker-policy worker regenerates the document after iterating on the
generator. RegenerateTrackerPolicy returns a conflict when nothing has
been published yet.
Wire it across all API surfaces per the api-surface rule: the console
GraphQL mutation and resolver, the MCP tool, the prb cookie-banner
regenerate-policy command, and the n8n operation, with console e2e
coverage for the success and no-published-version paths.
Signed-off-by: Émile Ré <emile@probo.com>
Add operator commands to proboctl for iterating on the cookie-banner
agents.
The global catalog groups (common-tracker-pattern, common-third-party)
list/filter/sort/show the catalogs using the shared coredata cursor
layer, and common-tracker-pattern reenrich re-describes selected rows by
running the enricher in-process (so it completes synchronously rather
than racing the async queue); a --cfg-file flag reuses probod's config
to wire the agent. --linked-banner/--linked-org target exactly the
catalog rows a banner or org depends on.
The cookie-banner reset-trackers command is tenant-scoped (it derives a
coredata.Scope from the banner/org GID) and rebuilds a banner's
uncategorised, non-excluded patterns from detected_trackers, decomposing
derived globs back into exacts, then re-arms the analysis and mapping
workers. --mapping-only skips the rebuild. A DB-backed test covers the
rebuild, link clearing, and preservation of categorised/excluded
patterns.
Signed-off-by: Émile Ré <emile@probo.com>
Introduce an API-style data layer for the global common tracker pattern
and common third party catalogs: typed filters, order fields, CursorKey,
cursor-paginated Load and CountAll, plus by-id enrichment re-queue and a
scoped reset/remap helper for a banner's tracker patterns. These reuse
the same page.Cursor/filter/order types the GraphQL API consumes, so a
future proboctl API can back them unchanged.
Extract the common-pattern enrichment logic out of the worker into a
CommonPatternEnricher service so it can run either from the background
queue or synchronously over a known set of ids; the worker becomes a
thin poller that delegates to it.
Extract the LLM client and tracker-agents config wiring into
pkg/agentsbuild so probod and other binaries build agents identically;
probod now delegates to it.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis tokenizer split names only on underscore first,
then on dash. A key like "name_done:<uuid>" left the colon glued to
the leading hex group, so the trailing UUID was never recognised as a
single token and instead shredded on dash into short hex anchors. The
derived template kept those anchors fixed, producing a distinct glob
per UUID and preventing any merge.
Treat colon and dot as primary separators alongside underscore so the
embedded UUID is isolated, matched by isUUIDShape, and collapsed to a
wildcard. Extend templateHasFixedAnchor to ignore the new separators
so a separator-only template stays rejected by the anti-overmerge
guard.
Signed-off-by: Émile Ré <emile@probo.com>
Each package that exercises the database against a real Postgres
carried its own copy of the connection bootstrap and schema setup.
Those copies had already drifted: some keyed off PROBO_TEST_PG_ADDR
with hardcoded defaults, others off PROBO_TEST_PG_URL, and the
agentrun/coredata suites hand-applied individual agent_runs
migrations to ensure the table existed.
Introduce a single test.PGClient helper that parses PROBO_TEST_PG_URL
(falling back to the local compose database), runs the full coredata
migration set once per process, and skips when no database is
reachable so make test stays a pure unit-test run. Migrate the
agentrun, coredata, cookiebanner, iam, and thirdparty suites onto it
and delete the duplicated helpers so the bootstrap can no longer
diverge.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Trackers sharing a display name can differ in type, so the generated
cookie and tracking technologies policy was ambiguous without it. Carry
the tracker type through the banner version snapshot and surface it as a
dedicated column in the policy table.
Stop the snapshot from dropping non-cookie trackers so storage, IndexedDB
and cache technologies appear in the policy and served banner config with
their real type. Duration now reflects the type when no max-age applies:
session storage clears with the tab, the remaining storage technologies
persist. Legacy snapshots predate the field and only ever held cookies,
so GetSnapshot backfills an empty type as COOKIE, keeping the non-null
GraphQL enum and policy output valid without a migration.
Signed-off-by: Émile Ré <emile@probo.com>
Add a markdown "Cookie and Tracking Technologies Policy" template in
the policy-writer tone that covers cookies and other tracking
technologies, with GDPR, UK GDPR, FADP, and CCPA/CPRA privacy-rights
framing. Convert it to ProseMirror JSON and add PublishTrackerPolicy,
which builds the document from a banner's latest published version
snapshot and its tracker third parties. The document is GENERATED and
private in the trust center, linked to the banner via
policy_document_id.
Add CookieItem.HumanizedDuration so server-rendered lifetimes match the
banner's humanizeDuration helper, and reword the publish trigger and
backfill comment to reflect the broader tracker scope.
Signed-off-by: Émile Ré <emile@probo.com>
Add policy_document_id and policy_generation_requested_at columns to
cookie_banners and backfill banners that already have a published
version, so existing live banners get a policy on the worker's first
pass.
Flag the banner for policy generation inside the publish transaction so
generation is requested only when a cookie banner version is published,
not on draft edits. Fold policy_document_id into the existing Update so
the upcoming worker can persist the generated document id with the
scope it already holds.
Signed-off-by: Émile Ré <emile@probo.com>
Queue vetting on third_parties with PENDING, PROCESSING,
COMPLETED, and FAILED states. Expose enqueue and status through
GraphQL, MCP, CLI, and n8n, validate vet requests, tune the
worker via config, and poll the detail page while vetting runs.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping and common-pattern enrichment agents ran with
4 and 3 max turns. Those budgets are smaller than the tool-call
rounds their prompts authorize (two/one DB searches plus up to
three web searches) plus the forced structured-output synthesis
turn, so the run loop tripped MaxTurnsExceededError before it
could emit JSON.
Raise the budgets to 10 across every layer that drives them: the
bootstrap env-var defaults, the Helm chart values and production
example, and the in-code fallback. Update the bootstrap test
defaults to match.
Revert the prompt instructions that told the model to reserve a
turn for its final answer; the model cannot reliably count its
remaining turns, so the fix belongs in the turn budget rather
than the prompt text.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker clears mapping_requested_at at claim time, so
a crash or hard failure between Process phases left the pattern dequeued,
unmapped, and with nothing to re-trigger it. Only an incidental sibling
remap could rescue it, so a lone pattern could stay stranded forever.
Implement the worker.StaleRecoverer interface, mirroring the enrichment
worker. ResetStaleMappings re-arms rows that were claimed but never
assigned a catalog row (common_tracker_pattern_id IS NULL) once idle past
a configurable window; a successful Process always assigns one via the
unmatched fallback, so the predicate cleanly detects interrupted runs and
self-heals after a single pass. ClearMappingRequestedAt now bumps
updated_at so the stale clock starts at claim time and the sweep never
recycles an in-flight claim.
Plumb a StaleAfter knob (default 600s) through the config struct, builder
env var, probod wiring, and Helm templates.
Signed-off-by: Émile Ré <emile@probo.com>
The identification agent attributed probo_distinct_id to Mixpanel
purely on the shared distinct_id token, and the enrichment agent
returned no description for the glob ph_phc_*_posthog because it
searched the literal "*" string and found nothing.
Tighten the identification prompt so attribution requires a perfect
pattern match or a meaningful prefix that belongs to the vendor; a
generic token behind a different prefix is not a match. Teach the
enrichment prompt to strip wildcard and variable parts before
searching, and to treat a vendor name embedded in the key as
corroboration so clearly-named trackers still get a description.
Signed-off-by: Émile Ré <emile@probo.com>
The deterministic tracker-mapping heuristics group patterns by shared
initiator domain, but tag managers, customer-data platforms, and
generic CDNs (Google Tag Manager, Segment, cloudfront.net, ...)
initiate trackers for many unrelated vendors. Grouping on such a
domain mis-attributes one vendor's tracker to another.
Add uri.FilterSharedInfrastructureDomains backed by a curated eTLD+1
denylist and apply it once in resolveDeterministic, so sibling
grouping, catalog domain matching, and the sibling re-enqueue cascade
all ignore shared-infrastructure hosts. Vendor-specific domains such
as google-analytics.com are intentionally kept as a same-vendor
signal. The agent path is unchanged: it still sees observed domains,
now with a prompt caveat about shared infrastructure.
Update the two sibling tests that used googletagmanager.com as the
initiator domain to a vendor domain, since that host is now stripped
before grouping.
Signed-off-by: Émile Ré <emile@probo.com>
Address review feedback on the agent-driven tracker catalog path:
- Return initiator-domain load failures instead of swallowing them,
so the worker retries rather than running the agent on partial
context.
- In the resolver, treat only ErrResourceNotFound as a catalog miss
and propagate genuine name/slug lookup errors.
- Insert the new vendor inside a savepoint and, on the slug
unique-violation race, reload and return the winning row instead of
aborting the caller's transaction.
- Stop seeding common_third_party_domains from observed initiator
domains. They are a co-occurrence signal, not verified ownership,
and writing them into the global cross-tenant catalog pollutes the
domain-based matcher. The curated seed owns that data.
- Warn the mapping agent that observed domains may belong to shared
CDNs, tag managers, or hosting infrastructure rather than the
vendor, so it does not attribute on that basis alone.
- Extract a shared tracker-identification prompt helper and move the
common-pattern identification prompt next to the enrichment agent.
Signed-off-by: Émile Ré <emile@probo.com>
resolveOrCreateCommonThirdParty lived as a package-level helper in the
tracker mapping worker, but the common pattern enrichment worker now
reuses it. Homing shared catalog logic in a mapping-named file made the
enrichment worker quietly depend on the mapping worker's file, and it is
not a mapping concern.
Move it to pkg/thirdparty as exported ResolveOrCreateCommonThirdParty,
decoupled from cookiebanner's TrackerMappingAgentResult (it now takes a
name and category) to avoid an import cycle. It stays a transaction-
scoped free function so both workers compose it into their own tx for
atomicity rather than receiving a service that owns its own connection.
Relocate the catalog dedup DB test alongside it.
Signed-off-by: Émile Ré <emile@probo.com>
The enrichment worker no longer invents a description when a tracker's
purpose cannot be substantiated; it records an empty description and
marks the row enriched so the stale-recovery loop does not retry it.
Vendor identification is the mapping pipeline's job, so the enricher
reuses the existing tracker-mapping agent to attribute a third party
for an unlinked common pattern before describing it. A confident
catalog match seeds the enrichment prompt and links the pattern, but
the enricher never creates or overrides an attribution.
When a blank, unlinked catalog row later gains a third party through
the mapping pipeline's upsert, enrichment is re-armed so the now-known
vendor gets a second, better-informed description attempt.
Signed-off-by: Émile Ré <emile@probo.com>
