The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.
Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.
Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.
Signed-off-by: Émile Ré <emile@probo.com>
The agent returned a single confidence that conflated two unrelated
judgments: whether an artifact is a meaningful web tracker and which
vendor set it. The prompt's tracker-worthiness skepticism drove the
number down for extension state like __darkreader__wasEnabledForHost,
pushing it below the gate and dropping the attribution entirely, so a
clearly-named vendor never reached the catalog.
Rename the agent field to ThirdPartyConfidence and scope it to the
attribution alone. The identify gate now checks that a vendor is named
with sufficient confidence; on success the catalog row is stored at a
fixed agent confidence like the other heuristic signals, and on failure
the unmatched fallback still records the pattern with no third party.
The stored pattern confidence was only used for ordering and as agent
context, never as a gate, so a separate LLM-provided number is dropped
rather than split out.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker descriptions were only filled on the agent-identification path,
so patterns resolved by domain, sibling, or fallback stayed without one,
and empty mapping upserts could clobber a researched description on the
shared catalog row.
Move description ownership to a dedicated, global common-pattern
enrichment worker. New catalog rows are queued on insert; the worker
researches a compliance-grade description with web search, records it on
the common pattern, and fans it out to every linked tracker pattern. The
mapping worker no longer generates descriptions and only propagates an
already-enriched one at link time.
Rename TrackerMappingConfig to TrackerAgentsConfig since the mapping and
enrichment agents now share it.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker loaded a pattern in its claim transaction and
committed the resolution in a separate, later transaction. A full-row
Update would write back stale values and clobber any user edit made in
between. Add UpdateMapping, which writes only the worker-resolved
columns (common_tracker_pattern_id, third_party_id, and a description
filled only when still empty), leaving user-editable fields untouched.
Also add ORDER BY tracker_pattern_id to the sibling pattern lookup: the
query used LIMIT without an ORDER BY, so an over-limit match set
returned an arbitrary subset and could resolve the third party
differently across runs.
Signed-off-by: Émile Ré <emile@probo.com>
A tracker pattern's source ratchets PRE_EXISTING -> EXTENSION -> SCRIPT
as stronger detections arrive, but that promotion was never reflected
back to the mapping pipeline. The detection that promotes the source
also brings a fresh initiator domain that matchByDomain and
matchBySiblingOrigin can use, and an EXTENSION -> SCRIPT promotion lifts
the creationAllowed gate that blocks org third-party creation. Yet the
pattern's mapping_requested_at was already cleared after its first pass,
so the worker never revisited it.
Re-arm mapping_requested_at via SetMappingRequested at each
source-promotion site (reportDetectedTracker plus the glob-merge and
adoption paths in the pattern-analysis worker). Update's SET clause does
not cover mapping_requested_at, so assigning the field before Update
would be a silent no-op; SetMappingRequested only writes when the column
is NULL, keeping already-queued patterns from being double-enqueued.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker processes one pattern at a time and
matchBySiblingOrigin only reads already-resolved siblings, so vendor
propagation across a banner was forward-only. A sibling processed
before its peer resolved a vendor (for example, one that failed the
agent and fell back to an unmatched catalog row) was never revisited,
even once a later sibling clearly identified the same third party.
When a Process run newly establishes a common third party, re-arm
mapping_requested_at on same-banner siblings that share an initiator
domain and are still unpromoted and non-extension-sourced. The worker
re-claims them and matchBySiblingOrigin now finds the freshly mapped
pattern. Guarding on third_party_id IS NULL, mapping_requested_at IS
NULL, and a not-pre-existing common third party keeps cascades finite.
Signed-off-by: Émile Ré <emile@probo.com>
Sibling matching is an org-local co-occurrence signal: two patterns
served from the same origin on one banner are likely the same vendor,
even when that origin is the site's own (first-party) host. First-party
filtering only protects the global catalog (domain) match, where a
proxied tracker would otherwise hit the site owner's own entry. It now
runs solely before matchByDomain, so matchBySiblingOrigin sees the
unfiltered domains and promotion happens for patterns detected on the
banner's own origin.
Resolve the sibling's direct org third party and its catalog third party
as independent signals, so a single shared org third party no longer
short-circuits the common-pattern backfill.
Make the shared test fixtures unique per tenant: common_third_parties
and common_tracker_patterns are global with unique indexes, so parallel
tests previously collided on name, slug, and pattern. Also align the
sibling tests' stored initiator domains with production, which records
the eTLD+1.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker patterns detected on the same banner that share initiator
domains are a strong indicator of the same third party. Previously the
mapping worker only checked the global third-party domain catalog, so a
tracker whose domain was not registered there fell through to the
expensive LLM identification step even when a co-located pattern was
already mapped.
Add a matchBySiblingOrigin step that finds other patterns on the same
banner sharing the same initiator domains and reuses their resolved
common third party. It prefers siblings already promoted to an org
third party (the strongest signal) and falls back to siblings carrying
only a catalog link, skipping when the siblings disagree. The step runs
before the catalog domain lookup since an already-qualified sibling is
at least as reliable as a raw domain match.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker scripts loaded through a first-party reverse proxy (e.g.
t.probo.com proxying PostHog) share the scanned site's eTLD+1 and
were incorrectly matched against the site owner's own
CommonThirdParty entry in matchByDomain. This caused trackers like
ph_phc_* to be attributed to the site owner instead of PostHog.
Load the CookieBanner origin in Process and pass it to both
matchByDomain and identifyWithAgent. Both now filter out initiator
domains whose eTLD+1 matches the site before querying the catalog
or feeding domains to the LLM agent. The prompt is also updated to
warn about proxy domains.
Signed-off-by: Émile Ré <emile@probo.com>
When the mapping worker resolves a CommonTrackerPattern, propagate
its description back to the org TrackerPattern if the latter is
still empty. This ensures agent-produced descriptions reach the
user-facing tracker instead of staying only in the catalog.
The Update method now covers all mutable TrackerPattern columns
including common_tracker_pattern_id and third_party_id, replacing
the removed UpdateMapping method.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the two-step ID-materializing pattern (fetch IDs in Go, pass
as ANY(@ids)) with an IN-subquery that keeps the filtering entirely
in the database and eliminates an extra round trip. Remove the now
unused LoadIDsByCommonThirdPartyID and its service wrapper. Update
the coredata rule to clarify that subqueries for filtering are OK.
Signed-off-by: Émile Ré <emile@probo.com>
Third parties created by the tracker mapping worker are confirmed
active on the organization's cookie banner, making them first-level
by definition. Also remove unused ptr test helpers.
Signed-off-by: Émile Ré <emile@probo.com>
Catalog resolution (common_tracker_pattern_id) still runs for every
pattern, but promoteThirdParty is now gated on the tracker's cookie
category: patterns still sitting in the uncategorised bucket are not
promoted to an org ThirdParty until the user moves them to a real
category, which re-triggers the worker via SetMappingRequested.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
Surface the third-party links carried by tracker patterns (org-scoped
ThirdParty via third_party_id, or global CommonThirdParty via
common_tracker_pattern_id) requires three new batch loaders and two
filter dimensions, all kept inside their owning entity tables to honour
the no-cross-entity-JOIN rule.
* CommonTrackerPatterns gains LoadByIDs and the ID-only
LoadIDsByCommonThirdPartyID helper, which lets callers translate a
common third party into a set of common_tracker_pattern_id values
without ever JOINing against tracker_patterns.
* CommonThirdParties gains LoadByIDs.
* TrackerPatterns gains LoadDistinctThirdPartyIDsByCookieBannerID and
LoadDistinctCommonTrackerPatternIDsByCookieBannerID, used by the
upcoming CookieBanner.linkedThirdParties resolver to enumerate the
third parties referenced in a banner.
* TrackerPatternFilter gains thirdPartyID and commonTrackerPatternIDs
filter dimensions; the GraphQL layer will dispatch a single
thirdPartyId argument to the right one based on the GID entity-type
prefix.
Service-layer wrappers (cookiebanner.GetCommonTrackerPatternsByIDs,
cookiebanner.LoadCommonTrackerPatternIDsByCommonThirdPartyID,
cookiebanner.LoadDistinctThirdPartyIDsByCookieBannerID,
cookiebanner.LoadDistinctCommonTrackerPatternIDsByCookieBannerID, and
thirdparty.GetCommonThirdPartiesByIDs) expose the new loaders to the
console resolvers and dataloaders that follow.
Signed-off-by: Émile Ré <emile@probo.com>
Every PromoteSource caller already loaded the tracker pattern under
the same transaction, so a dedicated single-column UPDATE only
duplicated machinery and forced callers to learn a second mutation
verb. Add `source = @source` to Update's SET clause, mutate
Source/UpdatedAt on the receiver, and call Update at the three
promotion sites (worker merge loop, worker adoption loop, and
reportDetectedTracker). The shouldPromoteSource gate still ranks the
candidate against the loaded value; Update is now the single write
path that can advance source, with a doc comment spelling out the
load-first contract.
Re-cast the coredata tests around Update: WritesSource pins the
round-trip from receiver to DB, NotFoundForMissingRow preserves the
ErrResourceNotFound contract callers rely on. The old
OnlyTouchesSourceAndUpdatedAt test was a property of the narrow
PromoteSource UPDATE and no longer applies — Update intentionally
rewrites the full editable column set from the receiver.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis worker dropped two signals on every run. When
InsertIfNotExists hit a pre-existing glob, the computed bestSource
was discarded by the LoadByBannerIDTypeAndPattern fallback, so the
SCRIPT > EXTENSION > PRE_EXISTING precedence advertised on
bestSource was only ever enforced at first insert. Subsequent
batches with stronger sources could not promote the glob, even
though the page-script-wins rule already lives in detected_trackers
at the row level.
Separately, adoptUncategorisedPatterns returned an adopted bool
that the worker discarded; the function moves detected trackers
from uncategorised exact patterns into categorised globs, which is
a real consent transition, but no draft banner version was created
on adoption-only runs.
Add a focused TrackerPattern.PromoteSource that only updates the
source and updated_at columns. Express the precedence as a pure-Go
shouldPromoteSource helper alongside bestSource so the rule is
unit-testable without a database. The worker now calls
InsertIfNotExists, then on conflict loads, skips when the slot is
held by an exact pattern or a user-recategorised glob, and only
calls PromoteSource when the candidate source ranks above the
existing one.
The skip branch is now documented: adoptUncategorisedPatterns is
the safety net that re-homes uncategorised exacts into the existing
glob via globMatch. Capture its adopted return value and use it
(instead of the previous over-eager consentChanged flag) to gate
ensureDraftVersionForBanner. Merging exacts into a glob in their
own category never changes visitor consent, so the prior flag
produced redundant draft versions on every non-uncategorised merge.
Cover the new pieces with three test layers: pure-unit cases for
shouldPromoteSource (precedence matrix including HTTP/nil collapse
and equal-rank no-write), DB-backed tests for PromoteSource (touch
only source + updated_at, ErrResourceNotFound for missing rows),
and end-to-end worker tests for source promotion on an existing
glob, draft-on-adoption, and the merge-only no-draft case.
Signed-off-by: Émile Ré <emile@probo.com>
The previous cleanup deleted every isExtensionCaller() site, including
the one in cookie/storage detectors that did fire reliably for the
residual case: page-world extensions (MV3 main world, userscripts with
@grant none) whose stack contains a chrome-/moz-/safari-web-extension
frame at the synchronous write. Recover that signal for free by
returning fromExtension from getInitiatorURL (it already walks the
stack and discards extension frames via continue), and have the cookie
and storage detectors report source: "extension" instead of "script"
when the flag is set.
End-to-end plumbing reuses the existing source column: extend the
cookie_source Postgres enum with EXTENSION, add the CookieSourceExtension
constant with a doc block describing each bucket's actual semantics,
add the handler.go switch cases, expose EXTENSION on the GraphQL and
MCP CookieSource enums, and add the Extension option to the console
source filter.
Update bestSource in the pattern analysis worker so a glob merging
only extension-attributed exact patterns is no longer silently rolled
up to PRE_EXISTING. New precedence is SCRIPT > EXTENSION > PRE_EXISTING,
matching the upsert SQL's "page-script wins" rule and the asymmetric
signal strength of each bucket.
Out of scope: any behavioural use of EXTENSION (auto-exclusion,
denylist classification, dashboard surfacing) -- that belongs in the
follow-up backend denylist plan.
Signed-off-by: Émile Ré <emile@probo.com>
Unrelated third-party trackers that happened to share a leading
separator run (e.g. __support__, __darkreader__wasEnabledForHost,
__EXT_APP_REFRESH_BLACK_SUB_DOMAINS__) were being merged under
overly broad globs such as __* because templateCandidates emitted
every prefix at each '_' or '-' position without requiring any
fixed anchor.
Add a templateHasFixedAnchor helper and apply it to both
templateCandidates loops and the heuristicTemplate result so
candidates consisting solely of '_', '-', and '*' are rejected.
Signed-off-by: Émile Ré <emile@probo.com>
The cookie detector tags every detection with a source (script,
pre-existing, http) but the storage detector did not, so storage
rows always landed in detected_trackers with source NULL even though
the SDK already distinguishes wrapper writes from pre-load scans.
Plumb a "script"/"pre-existing" source from the storage detector
through the report endpoint into DetectedStorageItem, so the column
gets populated for localStorage, sessionStorage, indexedDB and
cacheStorage entries. No schema change is needed: detected_trackers
already accepts CookieSource values regardless of tracker_type, and
the existing row renders the badge as soon as it is non-null.
Signed-off-by: Émile Ré <emile@probo.com>
Replace duplicated LoadConsentCategoriesByCookieBannerID,
CountConsentCategoriesByCookieBannerID, and
LoadAllConsentCategoriesByCookieBannerID with a single
CookieCategoryFilter in pkg/coredata. The filter uses the
standard CASE WHEN idiom to optionally exclude a kind,
eliminating branching in the service layer.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the cross-entity JOIN in
DetectedTrackers.LoadCommonThirdPartyIDByDomainMatch with two
idiomatic coredata calls: LoadInitiatorDomainsByTrackerPatternID
on DetectedTrackers, then a new CommonThirdPartyDomains.Load with
a CommonThirdPartyDomainFilter. Each entity now queries only its
own table, and the caller orchestrates the lookup.
Document the Load vs LoadAll naming convention and the no
cross-entity JOINs rule in contrib/claude/coredata.md.
Signed-off-by: Émile Ré <emile@probo.com>
Upsert methods now RETURNING all struct columns and scan the result
back into the pointer receiver, keeping the caller in sync with the
actual DB state (id, created_at, etc. from the existing row on
conflict). Insert detection compares the saved original ID with the
returned ID instead of relying on the PostgreSQL-internal xmax column.
Signed-off-by: Émile Ré <emile@probo.com>
Generate the category list from coredata.ThirdPartyCategories() at
runtime instead of hardcoding it in the prompt text. Type the
TrackerIdentification.Category field as coredata.ThirdPartyCategory so
JSON unmarshaling validates values automatically.
Also documents the .txt.tmpl template file naming convention.
Signed-off-by: Émile Ré <emile@probo.com>
Write errors from Insert/Upsert were logged but swallowed,
letting the transaction commit with incomplete data. All
helper methods now return errors so the transaction rolls
back on failure.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl has a single public API at https://api.firecrawl.dev/v2.
The endpoint was configurable but never varied across environments,
so hardcode it as a package-level const and remove the Endpoint
field from FirecrawlConfig and all downstream wiring (bootstrap,
Helm chart, probod, vetting, cookiebanner).
Signed-off-by: Émile Ré <emile@probo.com>
SearXNG was a fallback search backend that added complexity without
being used in practice. All search-dependent features (web search,
government DB checks, vetting orchestrator, tracker mapping) now use
Firecrawl exclusively. Removes the SEARCH_ENDPOINT config plumbing
from probodconfig, bootstrap, Helm charts, and all callers.
Signed-off-by: Émile Ré <emile@probo.com>
Update go-style guide and cursor rule to clarify that even a single
argument spanning multiple lines must break after the opening
parenthesis. Fix six violations across the branch.
Signed-off-by: Émile Ré <emile@probo.com>
splitTokens previously picked a single separator (_ winning over -).
Cookies like __Secure-1PSID were split on _ only, leaving
"Secure-1PSID" as one token that falsely passed looksVariable,
producing the bogus ___* heuristic pattern.
The function now splits by _ first, then sub-splits each non-UUID
part by -, tracking per-gap separators. heuristicTemplate trims
leading/trailing empty tokens before analysis so that prefix
underscores are preserved in the output but do not pollute the
variable detection.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl provides higher quality search results than SearXNG.
When configured (firecrawl-endpoint + firecrawl-api-key), the
tracker-mapping agent and search toolset prefer it over the
SearXNG backend. Also improves the tracker identification prompt
with multi-strategy search queries that leverage domain signals
and adapt to tracker type.
Signed-off-by: Émile Ré <emile@probo.com>
When both pattern matching and domain matching fail to identify a
tracker, an opt-in LLM agent can now attempt identification using
internal database searches and optional web search. The agent returns
structured output (third party name, category, description, confidence)
and the worker auto-creates CommonThirdParty records when needed.
The feature is gated behind the `llm.tracker-mapping.provider` config
field; when unset the worker behaves exactly as before.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern analysis worker creates glob patterns (e.g. prism_*) by
merging exact patterns but never set MappingRequestedAt, so the tracker
mapping worker never picked them up to match against common tracker
patterns.
Signed-off-by: Émile Ré <emile@probo.com>
When neither pattern nor domain matching finds an existing common
tracker pattern, upsert one with no third party association and a
lower confidence (0.5) so every seen pattern is cataloged.
Also fix table name in LoadByOrganizationIDAndCommonThirdPartyID
(vendors -> third_parties).
Signed-off-by: Émile Ré <emile@probo.com>
Move tracker_mapping_worker.go from pkg/probo to pkg/cookiebanner and
rename worker.go to pattern_analysis_worker.go to reflect the worker
name.
Signed-off-by: Émile Ré <emile@probo.com>
Poll-based worker that maps org-scoped tracker patterns to the
common knowledge base via pattern matching and domain-based
attribution. Populates initiator_domain on detected trackers
at report time. Resolves org-scoped vendors through the common
third party link.
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
The consent mode is now determined dynamically by the visitor's
country and its applicable regulation. The configured consent_mode
column is dropped from cookie_banners and added to
cookie_consent_records to persist the geo-derived mode at
consent-recording time. When no regulation matches, the default
is OPT_OUT.
Signed-off-by: Émile Ré <emile@getprobo.com>
Replace per-row UPDATE inside the detected-tracker loop with a single
bulk UPDATE ... WHERE id = ANY(...) after the loop. The old approach
locked pattern rows in request-dependent order, causing deadlocks
under concurrent ReportDetectedTrackers calls.
Signed-off-by: Émile Ré <emile@getprobo.com>
When a cookie name has no separator and looks variable (e.g. a long
hex string), heuristicTemplate returned "*" which created a catch-all
glob pattern matching every tracker of the same type and duration
bucket. Skip these separator-less names entirely so they remain as
individual exact-match patterns for human triage.
Signed-off-by: Émile Ré <emile@getprobo.com>
The pattern analysis worker now recognises UUID-like, hash-like,
and long numeric tokens as variable parts and replaces them with
wildcards heuristically, even from a single observation. This
prevents site-specific identifiers from being treated as static
text while meaningful suffixes (window_id, posthog, …) get
incorrectly wildcarded.
Also upgrades globMatch and the FindMatchingPattern SQL query
to support multiple wildcards in a single pattern.
Signed-off-by: Émile Ré <emile@getprobo.com>
The method uses ON CONFLICT ... DO UPDATE, so the name now
matches the actual behaviour. TrackerPattern.InsertIfNotExists
keeps its name because it uses ON CONFLICT ... DO NOTHING.
Signed-off-by: Émile Ré <emile@getprobo.com>
- Detectors: keep batched entries in `pending` until the POST succeeds
and guard against concurrent flushes, so transient network errors no
longer silently drop detection reports.
- Worker: add stable tie-breakers to the merge-candidate sort so the
greedy assignment produces deterministic groups across runs.
- Handler: skip resource entries with an empty URL (zero-value `uri.URI`
when the `url` field is missing) before persisting them.
- Third-party detector: allow same-origin service worker scripts through
`processResource` -- service workers are always same-origin by spec,
so the previous filter made `wrapServiceWorker` unreachable.
- Resource row edit: bump the description cell `colSpan` to 3 so the
edit row spans all five table columns.
- Resolver: handle `ErrSameResourceCategoryMove` explicitly so the no-op
move returns a validation error instead of an internal one.
Signed-off-by: Émile Ré <emile@getprobo.com>
When third-party JS sets a cookie or writes to local/sessionStorage
inside a customer page, the SDK now walks the synchronous call stack
to find the first non-extension, non-Probo, non-first-party http(s)
URL. That origin+path is sent as initiator_url on the report payload,
persisted in a new nullable column on detected_trackers, and preserved
across upserts via COALESCE.
This unlocks per-vendor attribution for cookies and storage writes
without needing pattern name matching, so future categorisation logic
can simply look up the initiator URL in the existing tracker_resources
table and inherit that vendor's category.
GraphQL/MCP exposure is intentionally deferred -- the column is captured
now, surfaced later.
Signed-off-by: Émile Ré <emile@getprobo.com>
