The tracker-mapping, common-pattern enrichment, and third-party
disambiguation agents were all built from one shared
TrackerAgentsConfig fed by a single tracker-mapping config slot. That
forced a single AgentTimeout to be reused and patched per worker, and
two unrelated max-turns fields to share one struct.
Split the in-code config into TrackerMappingAgentConfig,
TrackerEnrichmentAgentConfig, and DisambiguationAgentConfig, each with
its own timeout and max-turns, and add dedicated tracker-enrichment and
third-party-disambiguation provider slots (the latter resolving next to
third-party-vetter). Enrichment and disambiguation fall back to the
tracker-mapping slot when their own provider is unset, preserving
single-config deployments.
Drop the shared pkg/agentsbuild package and duplicate its small wiring
into probod and proboctl so the two executables stay decoupled. Wire
the new env vars, builder test coverage, and Helm values.
Signed-off-by: Émile Ré <emile@probo.com>
Expose a non-destructive re-trigger that re-arms tracker policy
generation for a banner that already has a published version, so the
tracker-policy worker regenerates the document after iterating on the
generator. RegenerateTrackerPolicy returns a conflict when nothing has
been published yet.
Wire it across all API surfaces per the api-surface rule: the console
GraphQL mutation and resolver, the MCP tool, the prb cookie-banner
regenerate-policy command, and the n8n operation, with console e2e
coverage for the success and no-published-version paths.
Signed-off-by: Émile Ré <emile@probo.com>
Add operator commands to proboctl for iterating on the cookie-banner
agents.
The global catalog groups (common-tracker-pattern, common-third-party)
list/filter/sort/show the catalogs using the shared coredata cursor
layer, and common-tracker-pattern reenrich re-describes selected rows by
running the enricher in-process (so it completes synchronously rather
than racing the async queue); a --cfg-file flag reuses probod's config
to wire the agent. --linked-banner/--linked-org target exactly the
catalog rows a banner or org depends on.
The cookie-banner reset-trackers command is tenant-scoped (it derives a
coredata.Scope from the banner/org GID) and rebuilds a banner's
uncategorised, non-excluded patterns from detected_trackers, decomposing
derived globs back into exacts, then re-arms the analysis and mapping
workers. --mapping-only skips the rebuild. A DB-backed test covers the
rebuild, link clearing, and preservation of categorised/excluded
patterns.
Signed-off-by: Émile Ré <emile@probo.com>
Introduce an API-style data layer for the global common tracker pattern
and common third party catalogs: typed filters, order fields, CursorKey,
cursor-paginated Load and CountAll, plus by-id enrichment re-queue and a
scoped reset/remap helper for a banner's tracker patterns. These reuse
the same page.Cursor/filter/order types the GraphQL API consumes, so a
future proboctl API can back them unchanged.
Extract the common-pattern enrichment logic out of the worker into a
CommonPatternEnricher service so it can run either from the background
queue or synchronously over a known set of ids; the worker becomes a
thin poller that delegates to it.
Extract the LLM client and tracker-agents config wiring into
pkg/agentsbuild so probod and other binaries build agents identically;
probod now delegates to it.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis tokenizer split names only on underscore first,
then on dash. A key like "name_done:<uuid>" left the colon glued to
the leading hex group, so the trailing UUID was never recognised as a
single token and instead shredded on dash into short hex anchors. The
derived template kept those anchors fixed, producing a distinct glob
per UUID and preventing any merge.
Treat colon and dot as primary separators alongside underscore so the
embedded UUID is isolated, matched by isUUIDShape, and collapsed to a
wildcard. Extend templateHasFixedAnchor to ignore the new separators
so a separator-only template stays rejected by the anti-overmerge
guard.
Signed-off-by: Émile Ré <emile@probo.com>
Each package that exercises the database against a real Postgres
carried its own copy of the connection bootstrap and schema setup.
Those copies had already drifted: some keyed off PROBO_TEST_PG_ADDR
with hardcoded defaults, others off PROBO_TEST_PG_URL, and the
agentrun/coredata suites hand-applied individual agent_runs
migrations to ensure the table existed.
Introduce a single test.PGClient helper that parses PROBO_TEST_PG_URL
(falling back to the local compose database), runs the full coredata
migration set once per process, and skips when no database is
reachable so make test stays a pure unit-test run. Migrate the
agentrun, coredata, cookiebanner, iam, and thirdparty suites onto it
and delete the duplicated helpers so the bootstrap can no longer
diverge.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Trackers sharing a display name can differ in type, so the generated
cookie and tracking technologies policy was ambiguous without it. Carry
the tracker type through the banner version snapshot and surface it as a
dedicated column in the policy table.
Stop the snapshot from dropping non-cookie trackers so storage, IndexedDB
and cache technologies appear in the policy and served banner config with
their real type. Duration now reflects the type when no max-age applies:
session storage clears with the tab, the remaining storage technologies
persist. Legacy snapshots predate the field and only ever held cookies,
so GetSnapshot backfills an empty type as COOKIE, keeping the non-null
GraphQL enum and policy output valid without a migration.
Signed-off-by: Émile Ré <emile@probo.com>
Add a markdown "Cookie and Tracking Technologies Policy" template in
the policy-writer tone that covers cookies and other tracking
technologies, with GDPR, UK GDPR, FADP, and CCPA/CPRA privacy-rights
framing. Convert it to ProseMirror JSON and add PublishTrackerPolicy,
which builds the document from a banner's latest published version
snapshot and its tracker third parties. The document is GENERATED and
private in the trust center, linked to the banner via
policy_document_id.
Add CookieItem.HumanizedDuration so server-rendered lifetimes match the
banner's humanizeDuration helper, and reword the publish trigger and
backfill comment to reflect the broader tracker scope.
Signed-off-by: Émile Ré <emile@probo.com>
Add policy_document_id and policy_generation_requested_at columns to
cookie_banners and backfill banners that already have a published
version, so existing live banners get a policy on the worker's first
pass.
Flag the banner for policy generation inside the publish transaction so
generation is requested only when a cookie banner version is published,
not on draft edits. Fold policy_document_id into the existing Update so
the upcoming worker can persist the generated document id with the
scope it already holds.
Signed-off-by: Émile Ré <emile@probo.com>
Queue vetting on third_parties with PENDING, PROCESSING,
COMPLETED, and FAILED states. Expose enqueue and status through
GraphQL, MCP, CLI, and n8n, validate vet requests, tune the
worker via config, and poll the detail page while vetting runs.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping and common-pattern enrichment agents ran with
4 and 3 max turns. Those budgets are smaller than the tool-call
rounds their prompts authorize (two/one DB searches plus up to
three web searches) plus the forced structured-output synthesis
turn, so the run loop tripped MaxTurnsExceededError before it
could emit JSON.
Raise the budgets to 10 across every layer that drives them: the
bootstrap env-var defaults, the Helm chart values and production
example, and the in-code fallback. Update the bootstrap test
defaults to match.
Revert the prompt instructions that told the model to reserve a
turn for its final answer; the model cannot reliably count its
remaining turns, so the fix belongs in the turn budget rather
than the prompt text.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker clears mapping_requested_at at claim time, so
a crash or hard failure between Process phases left the pattern dequeued,
unmapped, and with nothing to re-trigger it. Only an incidental sibling
remap could rescue it, so a lone pattern could stay stranded forever.
Implement the worker.StaleRecoverer interface, mirroring the enrichment
worker. ResetStaleMappings re-arms rows that were claimed but never
assigned a catalog row (common_tracker_pattern_id IS NULL) once idle past
a configurable window; a successful Process always assigns one via the
unmatched fallback, so the predicate cleanly detects interrupted runs and
self-heals after a single pass. ClearMappingRequestedAt now bumps
updated_at so the stale clock starts at claim time and the sweep never
recycles an in-flight claim.
Plumb a StaleAfter knob (default 600s) through the config struct, builder
env var, probod wiring, and Helm templates.
Signed-off-by: Émile Ré <emile@probo.com>
The identification agent attributed probo_distinct_id to Mixpanel
purely on the shared distinct_id token, and the enrichment agent
returned no description for the glob ph_phc_*_posthog because it
searched the literal "*" string and found nothing.
Tighten the identification prompt so attribution requires a perfect
pattern match or a meaningful prefix that belongs to the vendor; a
generic token behind a different prefix is not a match. Teach the
enrichment prompt to strip wildcard and variable parts before
searching, and to treat a vendor name embedded in the key as
corroboration so clearly-named trackers still get a description.
Signed-off-by: Émile Ré <emile@probo.com>
The deterministic tracker-mapping heuristics group patterns by shared
initiator domain, but tag managers, customer-data platforms, and
generic CDNs (Google Tag Manager, Segment, cloudfront.net, ...)
initiate trackers for many unrelated vendors. Grouping on such a
domain mis-attributes one vendor's tracker to another.
Add uri.FilterSharedInfrastructureDomains backed by a curated eTLD+1
denylist and apply it once in resolveDeterministic, so sibling
grouping, catalog domain matching, and the sibling re-enqueue cascade
all ignore shared-infrastructure hosts. Vendor-specific domains such
as google-analytics.com are intentionally kept as a same-vendor
signal. The agent path is unchanged: it still sees observed domains,
now with a prompt caveat about shared infrastructure.
Update the two sibling tests that used googletagmanager.com as the
initiator domain to a vendor domain, since that host is now stripped
before grouping.
Signed-off-by: Émile Ré <emile@probo.com>
Address review feedback on the agent-driven tracker catalog path:
- Return initiator-domain load failures instead of swallowing them,
so the worker retries rather than running the agent on partial
context.
- In the resolver, treat only ErrResourceNotFound as a catalog miss
and propagate genuine name/slug lookup errors.
- Insert the new vendor inside a savepoint and, on the slug
unique-violation race, reload and return the winning row instead of
aborting the caller's transaction.
- Stop seeding common_third_party_domains from observed initiator
domains. They are a co-occurrence signal, not verified ownership,
and writing them into the global cross-tenant catalog pollutes the
domain-based matcher. The curated seed owns that data.
- Warn the mapping agent that observed domains may belong to shared
CDNs, tag managers, or hosting infrastructure rather than the
vendor, so it does not attribute on that basis alone.
- Extract a shared tracker-identification prompt helper and move the
common-pattern identification prompt next to the enrichment agent.
Signed-off-by: Émile Ré <emile@probo.com>
resolveOrCreateCommonThirdParty lived as a package-level helper in the
tracker mapping worker, but the common pattern enrichment worker now
reuses it. Homing shared catalog logic in a mapping-named file made the
enrichment worker quietly depend on the mapping worker's file, and it is
not a mapping concern.
Move it to pkg/thirdparty as exported ResolveOrCreateCommonThirdParty,
decoupled from cookiebanner's TrackerMappingAgentResult (it now takes a
name and category) to avoid an import cycle. It stays a transaction-
scoped free function so both workers compose it into their own tx for
atomicity rather than receiving a service that owns its own connection.
Relocate the catalog dedup DB test alongside it.
Signed-off-by: Émile Ré <emile@probo.com>
The enrichment worker no longer invents a description when a tracker's
purpose cannot be substantiated; it records an empty description and
marks the row enriched so the stale-recovery loop does not retry it.
Vendor identification is the mapping pipeline's job, so the enricher
reuses the existing tracker-mapping agent to attribute a third party
for an unlinked common pattern before describing it. A confident
catalog match seeds the enrichment prompt and links the pattern, but
the enricher never creates or overrides an attribution.
When a blank, unlinked catalog row later gains a third party through
the mapping pipeline's upsert, enrichment is re-armed so the now-known
vendor gets a second, better-informed description attempt.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker mapping, common-pattern enrichment, and third-party
disambiguation agents default to a small max-tokens budget on the
premise that their final output is tiny structured JSON. On
reasoning models such as the GPT-5 family, reasoning tokens count
against max_tokens, so a small budget is consumed by reasoning and
the JSON is truncated, surfacing as "unexpected end of JSON input".
Raise the defaults to 4096 (1024 -> 4096 for tracker mapping, 512
-> 4096 for disambiguation) to leave headroom for the reasoning
phase. Update the bootstrap builder default, its test, and the
production values example to match.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs its LLM and web-search phases
between short transactions and holds no row lock across them. The
pattern-analysis worker can merge a pattern into a glob and delete
it in that window, so the final UpdateMapping then fails with
ErrResourceNotFound and the task errors out spuriously.
A vanished pattern has nothing left to map, so treat the concurrent
delete as a no-op: log it and return nil instead of failing.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis worker created the merged glob blank and re-armed
mapping, discarding the org ThirdParty and description already resolved
on the exacts it absorbed. That forced a full re-map (LLM/web-search)
and opened a window where an in-flight exact could vanish mid-mapping.
Seed the glob from the merged exacts when they unanimously agree on a
single third party, carrying its description too, while still re-arming
mapping so the glob derives its own catalog row. With the third party
pre-set, the mapping worker skips the expensive org/disambiguation
resolution. Conflicting or unresolved groups stay blank as before.
The catalog link is deliberately not inherited: it is keyed on the
exact pattern string, not the glob template, so the mapping worker
resolves the right row itself.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.
Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.
Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.
Signed-off-by: Émile Ré <emile@probo.com>
The agent returned a single confidence that conflated two unrelated
judgments: whether an artifact is a meaningful web tracker and which
vendor set it. The prompt's tracker-worthiness skepticism drove the
number down for extension state like __darkreader__wasEnabledForHost,
pushing it below the gate and dropping the attribution entirely, so a
clearly-named vendor never reached the catalog.
Rename the agent field to ThirdPartyConfidence and scope it to the
attribution alone. The identify gate now checks that a vendor is named
with sufficient confidence; on success the catalog row is stored at a
fixed agent confidence like the other heuristic signals, and on failure
the unmatched fallback still records the pattern with no third party.
The stored pattern confidence was only used for ordering and as agent
context, never as a gate, so a separate LLM-provided number is dropped
rather than split out.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker descriptions were only filled on the agent-identification path,
so patterns resolved by domain, sibling, or fallback stayed without one,
and empty mapping upserts could clobber a researched description on the
shared catalog row.
Move description ownership to a dedicated, global common-pattern
enrichment worker. New catalog rows are queued on insert; the worker
researches a compliance-grade description with web search, records it on
the common pattern, and fans it out to every linked tracker pattern. The
mapping worker no longer generates descriptions and only propagates an
already-enriched one at link time.
Rename TrackerMappingConfig to TrackerAgentsConfig since the mapping and
enrichment agents now share it.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker loaded a pattern in its claim transaction and
committed the resolution in a separate, later transaction. A full-row
Update would write back stale values and clobber any user edit made in
between. Add UpdateMapping, which writes only the worker-resolved
columns (common_tracker_pattern_id, third_party_id, and a description
filled only when still empty), leaving user-editable fields untouched.
Also add ORDER BY tracker_pattern_id to the sibling pattern lookup: the
query used LIMIT without an ORDER BY, so an over-limit match set
returned an arbitrary subset and could resolve the third party
differently across runs.
Signed-off-by: Émile Ré <emile@probo.com>
A tracker pattern's source ratchets PRE_EXISTING -> EXTENSION -> SCRIPT
as stronger detections arrive, but that promotion was never reflected
back to the mapping pipeline. The detection that promotes the source
also brings a fresh initiator domain that matchByDomain and
matchBySiblingOrigin can use, and an EXTENSION -> SCRIPT promotion lifts
the creationAllowed gate that blocks org third-party creation. Yet the
pattern's mapping_requested_at was already cleared after its first pass,
so the worker never revisited it.
Re-arm mapping_requested_at via SetMappingRequested at each
source-promotion site (reportDetectedTracker plus the glob-merge and
adoption paths in the pattern-analysis worker). Update's SET clause does
not cover mapping_requested_at, so assigning the field before Update
would be a silent no-op; SetMappingRequested only writes when the column
is NULL, keeping already-queued patterns from being double-enqueued.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker processes one pattern at a time and
matchBySiblingOrigin only reads already-resolved siblings, so vendor
propagation across a banner was forward-only. A sibling processed
before its peer resolved a vendor (for example, one that failed the
agent and fell back to an unmatched catalog row) was never revisited,
even once a later sibling clearly identified the same third party.
When a Process run newly establishes a common third party, re-arm
mapping_requested_at on same-banner siblings that share an initiator
domain and are still unpromoted and non-extension-sourced. The worker
re-claims them and matchBySiblingOrigin now finds the freshly mapped
pattern. Guarding on third_party_id IS NULL, mapping_requested_at IS
NULL, and a not-pre-existing common third party keeps cascades finite.
Signed-off-by: Émile Ré <emile@probo.com>
Sibling matching is an org-local co-occurrence signal: two patterns
served from the same origin on one banner are likely the same vendor,
even when that origin is the site's own (first-party) host. First-party
filtering only protects the global catalog (domain) match, where a
proxied tracker would otherwise hit the site owner's own entry. It now
runs solely before matchByDomain, so matchBySiblingOrigin sees the
unfiltered domains and promotion happens for patterns detected on the
banner's own origin.
Resolve the sibling's direct org third party and its catalog third party
as independent signals, so a single shared org third party no longer
short-circuits the common-pattern backfill.
Make the shared test fixtures unique per tenant: common_third_parties
and common_tracker_patterns are global with unique indexes, so parallel
tests previously collided on name, slug, and pattern. Also align the
sibling tests' stored initiator domains with production, which records
the eTLD+1.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker patterns detected on the same banner that share initiator
domains are a strong indicator of the same third party. Previously the
mapping worker only checked the global third-party domain catalog, so a
tracker whose domain was not registered there fell through to the
expensive LLM identification step even when a co-located pattern was
already mapped.
Add a matchBySiblingOrigin step that finds other patterns on the same
banner sharing the same initiator domains and reuses their resolved
common third party. It prefers siblings already promoted to an org
third party (the strongest signal) and falls back to siblings carrying
only a catalog link, skipping when the siblings disagree. The step runs
before the catalog domain lookup since an already-qualified sibling is
at least as reliable as a raw domain match.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker scripts loaded through a first-party reverse proxy (e.g.
t.probo.com proxying PostHog) share the scanned site's eTLD+1 and
were incorrectly matched against the site owner's own
CommonThirdParty entry in matchByDomain. This caused trackers like
ph_phc_* to be attributed to the site owner instead of PostHog.
Load the CookieBanner origin in Process and pass it to both
matchByDomain and identifyWithAgent. Both now filter out initiator
domains whose eTLD+1 matches the site before querying the catalog
or feeding domains to the LLM agent. The prompt is also updated to
warn about proxy domains.
Signed-off-by: Émile Ré <emile@probo.com>
When the mapping worker resolves a CommonTrackerPattern, propagate
its description back to the org TrackerPattern if the latter is
still empty. This ensures agent-produced descriptions reach the
user-facing tracker instead of staying only in the catalog.
The Update method now covers all mutable TrackerPattern columns
including common_tracker_pattern_id and third_party_id, replacing
the removed UpdateMapping method.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the two-step ID-materializing pattern (fetch IDs in Go, pass
as ANY(@ids)) with an IN-subquery that keeps the filtering entirely
in the database and eliminates an extra round trip. Remove the now
unused LoadIDsByCommonThirdPartyID and its service wrapper. Update
the coredata rule to clarify that subqueries for filtering are OK.
Signed-off-by: Émile Ré <emile@probo.com>
Third parties created by the tracker mapping worker are confirmed
active on the organization's cookie banner, making them first-level
by definition. Also remove unused ptr test helpers.
Signed-off-by: Émile Ré <emile@probo.com>
Catalog resolution (common_tracker_pattern_id) still runs for every
pattern, but promoteThirdParty is now gated on the tracker's cookie
category: patterns still sitting in the uncategorised bucket are not
promoted to an org ThirdParty until the user moves them to a real
category, which re-triggers the worker via SetMappingRequested.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
Surface the third-party links carried by tracker patterns (org-scoped
ThirdParty via third_party_id, or global CommonThirdParty via
common_tracker_pattern_id) requires three new batch loaders and two
filter dimensions, all kept inside their owning entity tables to honour
the no-cross-entity-JOIN rule.
* CommonTrackerPatterns gains LoadByIDs and the ID-only
LoadIDsByCommonThirdPartyID helper, which lets callers translate a
common third party into a set of common_tracker_pattern_id values
without ever JOINing against tracker_patterns.
* CommonThirdParties gains LoadByIDs.
* TrackerPatterns gains LoadDistinctThirdPartyIDsByCookieBannerID and
LoadDistinctCommonTrackerPatternIDsByCookieBannerID, used by the
upcoming CookieBanner.linkedThirdParties resolver to enumerate the
third parties referenced in a banner.
* TrackerPatternFilter gains thirdPartyID and commonTrackerPatternIDs
filter dimensions; the GraphQL layer will dispatch a single
thirdPartyId argument to the right one based on the GID entity-type
prefix.
Service-layer wrappers (cookiebanner.GetCommonTrackerPatternsByIDs,
cookiebanner.LoadCommonTrackerPatternIDsByCommonThirdPartyID,
cookiebanner.LoadDistinctThirdPartyIDsByCookieBannerID,
cookiebanner.LoadDistinctCommonTrackerPatternIDsByCookieBannerID, and
thirdparty.GetCommonThirdPartiesByIDs) expose the new loaders to the
console resolvers and dataloaders that follow.
Signed-off-by: Émile Ré <emile@probo.com>
Every PromoteSource caller already loaded the tracker pattern under
the same transaction, so a dedicated single-column UPDATE only
duplicated machinery and forced callers to learn a second mutation
verb. Add `source = @source` to Update's SET clause, mutate
Source/UpdatedAt on the receiver, and call Update at the three
promotion sites (worker merge loop, worker adoption loop, and
reportDetectedTracker). The shouldPromoteSource gate still ranks the
candidate against the loaded value; Update is now the single write
path that can advance source, with a doc comment spelling out the
load-first contract.
Re-cast the coredata tests around Update: WritesSource pins the
round-trip from receiver to DB, NotFoundForMissingRow preserves the
ErrResourceNotFound contract callers rely on. The old
OnlyTouchesSourceAndUpdatedAt test was a property of the narrow
PromoteSource UPDATE and no longer applies — Update intentionally
rewrites the full editable column set from the receiver.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis worker dropped two signals on every run. When
InsertIfNotExists hit a pre-existing glob, the computed bestSource
was discarded by the LoadByBannerIDTypeAndPattern fallback, so the
SCRIPT > EXTENSION > PRE_EXISTING precedence advertised on
bestSource was only ever enforced at first insert. Subsequent
batches with stronger sources could not promote the glob, even
though the page-script-wins rule already lives in detected_trackers
at the row level.
Separately, adoptUncategorisedPatterns returned an adopted bool
that the worker discarded; the function moves detected trackers
from uncategorised exact patterns into categorised globs, which is
a real consent transition, but no draft banner version was created
on adoption-only runs.
Add a focused TrackerPattern.PromoteSource that only updates the
source and updated_at columns. Express the precedence as a pure-Go
shouldPromoteSource helper alongside bestSource so the rule is
unit-testable without a database. The worker now calls
InsertIfNotExists, then on conflict loads, skips when the slot is
held by an exact pattern or a user-recategorised glob, and only
calls PromoteSource when the candidate source ranks above the
existing one.
The skip branch is now documented: adoptUncategorisedPatterns is
the safety net that re-homes uncategorised exacts into the existing
glob via globMatch. Capture its adopted return value and use it
(instead of the previous over-eager consentChanged flag) to gate
ensureDraftVersionForBanner. Merging exacts into a glob in their
own category never changes visitor consent, so the prior flag
produced redundant draft versions on every non-uncategorised merge.
Cover the new pieces with three test layers: pure-unit cases for
shouldPromoteSource (precedence matrix including HTTP/nil collapse
and equal-rank no-write), DB-backed tests for PromoteSource (touch
only source + updated_at, ErrResourceNotFound for missing rows),
and end-to-end worker tests for source promotion on an existing
glob, draft-on-adoption, and the merge-only no-draft case.
Signed-off-by: Émile Ré <emile@probo.com>
The previous cleanup deleted every isExtensionCaller() site, including
the one in cookie/storage detectors that did fire reliably for the
residual case: page-world extensions (MV3 main world, userscripts with
@grant none) whose stack contains a chrome-/moz-/safari-web-extension
frame at the synchronous write. Recover that signal for free by
returning fromExtension from getInitiatorURL (it already walks the
stack and discards extension frames via continue), and have the cookie
and storage detectors report source: "extension" instead of "script"
when the flag is set.
End-to-end plumbing reuses the existing source column: extend the
cookie_source Postgres enum with EXTENSION, add the CookieSourceExtension
constant with a doc block describing each bucket's actual semantics,
add the handler.go switch cases, expose EXTENSION on the GraphQL and
MCP CookieSource enums, and add the Extension option to the console
source filter.
Update bestSource in the pattern analysis worker so a glob merging
only extension-attributed exact patterns is no longer silently rolled
up to PRE_EXISTING. New precedence is SCRIPT > EXTENSION > PRE_EXISTING,
matching the upsert SQL's "page-script wins" rule and the asymmetric
signal strength of each bucket.
Out of scope: any behavioural use of EXTENSION (auto-exclusion,
denylist classification, dashboard surfacing) -- that belongs in the
follow-up backend denylist plan.
Signed-off-by: Émile Ré <emile@probo.com>
Unrelated third-party trackers that happened to share a leading
separator run (e.g. __support__, __darkreader__wasEnabledForHost,
__EXT_APP_REFRESH_BLACK_SUB_DOMAINS__) were being merged under
overly broad globs such as __* because templateCandidates emitted
every prefix at each '_' or '-' position without requiring any
fixed anchor.
Add a templateHasFixedAnchor helper and apply it to both
templateCandidates loops and the heuristicTemplate result so
candidates consisting solely of '_', '-', and '*' are rejected.
Signed-off-by: Émile Ré <emile@probo.com>
The cookie detector tags every detection with a source (script,
pre-existing, http) but the storage detector did not, so storage
rows always landed in detected_trackers with source NULL even though
the SDK already distinguishes wrapper writes from pre-load scans.
Plumb a "script"/"pre-existing" source from the storage detector
through the report endpoint into DetectedStorageItem, so the column
gets populated for localStorage, sessionStorage, indexedDB and
cacheStorage entries. No schema change is needed: detected_trackers
already accepts CookieSource values regardless of tracker_type, and
the existing row renders the badge as soon as it is non-null.
Signed-off-by: Émile Ré <emile@probo.com>
