The source headers, LICENSE files, and license metadata had drifted
apart. Align the entire project to MIT:
- Convert every source-file header to the MIT text across all comment
styles (Go, TS, TSX, JS, MJS, SQL, CSS, GraphQL, shell), including
SPDX-License-Identifier tags
- Set the root and cookie-banner LICENSE files to the MIT text with a
"MIT License" title line
- Switch the package.json license fields, Docker image label, and
cookie-banner README to MIT
- Update docs and the genmodels header generator accordingly
- Normalize copyright lines to a single format
(Copyright (c) <year(s)> Probo Inc <hello@probo.com>.): unify the
hello@getprobo.com and hello@probo.inc emails to hello@probo.com and
the comma-separated years to a hyphenated range
Genuine third-party references are intentionally left untouched: the
Lucide icon attributions (Lucide is ISC) and the trivy dependency
license allowlist.
Signed-off-by: Sacha Al Himdani <sacha@probo.com>
Subprocessor filtering for the compliance portal happens in the backend
rather than the client. Add a SubprocessorFilter (query, category,
country) to the trust API's subprocessors connection, thread it through
the resolver and service, and extend the coredata ThirdParty filter with
category equality and country array membership. The connection stores the
filter so totalCount reflects the filtered set. Add e2e coverage for the
new filtering.
On the frontend, convert the page to a refetchable fragment whose filter
arguments are driven by URL-persisted, debounced toolbar state (category
and region selects plus a search field), populate the dropdowns from an
unfiltered facet selection, and offer to clear filters from the empty
state.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker re-arms same-banner siblings after a pattern
resolves a vendor. Its predicate only excluded promoted siblings
(third_party_id IS NULL), but since org-party auto-creation was dropped a
pattern can resolve a common third party yet never gain an org
third_party_id. Those siblings, and terminal first-party ones, stayed
eligible forever, so every cascade step re-enqueued and reprocessed them,
amplifying Process runs to O(N^2) per banner. The deadlock fix in the
last release removed the rollbacks that had accidentally throttled the
cascade, so the latent amplification surfaced as an INFO-log flood.
Tighten the re-enqueue to skip siblings already linked to a catalog row
that carries a common third party or marked FIRST_PARTY, dropping
per-banner reprocessing back to O(N). Also demote the two per-run handler
logs ("mapped tracker pattern", "re-enqueued unmapped sibling tracker
patterns") to Debug so routine processing no longer logs at INFO.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs many Process calls in parallel. In
Phase 4 a single transaction locked the worker's own claimed pattern
row via UpdateMapping and then locked sibling rows on the same banner
via the re-enqueue. Two workers mapping sibling patterns on one banner
each held their own row and waited on the other's, forming a lock cycle
that Postgres aborted with deadlock detected (40P01).
Split the sibling re-enqueue into its own short transaction that runs
after the mapping commits, so the claimed-row lock is released before
any sibling row is locked. Also take the sibling UPDATE row locks in a
deterministic id order through an ORDER BY id ... FOR UPDATE subquery,
so overlapping re-enqueues can no longer invert lock order between
themselves. The re-enqueue only flags siblings, so deferring it past
the commit is safe and lets reprocessed siblings observe committed data.
Signed-off-by: Émile Ré <emile@probo.com>
Three paths could leave a catalog row's attribution out of step with its
common_third_party_id. A FIRST_PARTY reclassification in the mapping
worker kept a stale org ThirdParty link instead of clearing it. The
upsert requeued terminal FIRST_PARTY rows for enrichment on a vendor
they never adopt, since the vendor-preservation clause nulls it. And the
proboctl upsert command did not normalize the verdict when an operator
linked or unlinked a vendor without passing --attribution.
Clear the org link on a first-party verdict, exclude FIRST_PARTY rows
from the enrichment requeue, and have the CLI downgrade THIRD_PARTY to
UNDETERMINED on unlink and promote UNDETERMINED to THIRD_PARTY on link.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-pattern catalog was binary (linked to a vendor or not), so
generic and first-party artifacts (loglevel keys, wallet-extension keys,
an org's own trackers) were retried forever and, once one row was wrongly
attributed, re-propagated to every organization with no re-check.
Give catalog rows a terminal attribution verdict (UNDETERMINED,
THIRD_PARTY, FIRST_PARTY): FIRST_PARTY short-circuits the whole mapping
pipeline so the artifact is never attributed again. Gate deterministic
vendor adoption behind a trust bar so only curated/operator rows
auto-propagate; lower-confidence agent/heuristic rows are reused as hints
and re-resolved, and an independent agent re-confirmation corroborates and
promotes them. Make the mapping agent emit an evidence source and reject
any attribution that lacks concrete evidence, and let it declare a
first-party verdict. Skip the speculative agent for PRE_EXISTING-source
patterns, whose low signal invites invented vendors.
Add proboctl "ctp mark-first-party" and an --attribution list filter to
audit and remediate existing wrong links, and a cursor rule documenting
migration naming so the timestamp is taken from date -u, not invented.
Signed-off-by: Émile Ré <emile@probo.com>
Unbounded LoadAll* loaders materialised an entire result set in one
query with no ceiling. A table that is small in development can grow
without bound in production, so these loaders were a latent memory
and query-time hazard.
Remove the LoadAll* methods from pkg/coredata and walk the cursor-
paginated LoadBy* siblings instead through a shared page.LoadAll
helper. The helper advances a MaxCursorSize forward cursor until the
result set is exhausted and concatenates the pages. It caps a single
call at MaxLoadAllPages (20) batches of 500 rows and errors past that
rather than materialising an unbounded set, so a runaway caller fails
loudly instead of exhausting memory.
Callers that genuinely need every row now express that explicitly,
and the coredata load-naming rule and docs are updated to discourage
new unbounded loaders.
Signed-off-by: Sacha Al Himdani <sacha@probo.com>
The tracker-mapping and common-pattern enrichment agents only had
web search, which returns title/url/snippet, so they could never
open a cookie-database or cookie-policy page to read which vendor
actually sets a tracker. This mis-attributed setters whose snippet
is misleading (e.g. _li_* read as LinkedIn rather than LiveIntent).
Wire the read-only headless-browser toolset into both agents, gated
on a configured Chrome endpoint, mirroring the common-third-party
enrichment worker: agent construction moves into the run path so each
run can carry a per-run browser that is closed when the run returns.
The prompts now direct the agent to open a promising result and read
the named setter from the full page text. Both agents stay unchanged
when no Chrome endpoint is configured.
Signed-off-by: Émile Ré <emile@probo.com>
Address PR review feedback: rename the shared string-helper package
from strutil to stringsx to avoid the discouraged util suffix and the
collision with the standard strings package, updating all import paths
and call sites.
Replace the manual wg.Add/wg.Done bookkeeping in the enrichment worker
with wg.Go, which is less error-prone.
Signed-off-by: Émile Ré <emile@probo.com>
The alphanumeric-normalisation helper was duplicated verbatim in the
common-third-party owned-domains resolver and the cookiebanner tracker
mapping worker. Hoist it into a new dependency-free strutil package so
both call sites share one implementation and one test.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker had been reduced to catalog resolution only,
which removed not just the auto-creation of an org ThirdParty but also
the auto-linking of an existing one. Only the creation needed to go: it
raced the load-then-create check and produced duplicate vendors.
Restore the full org ThirdParty resolution (exact common-id link,
sibling direct-link, high-confidence heuristic, and the disambiguation
agent) and remove only the CreateFromCommon branch and its
categorisation gate. When nothing matches, the worker now leaves
third_party_id unset rather than creating a vendor; creation happens
exclusively through the explicit ImportFromCommon action. Drop the
now-dead CreateFromCommon helper and rename match.go to common_match.go.
Fix a latent test bug surfaced by actually running the DB-backed suite
(skipped in CI without Postgres): the heuristic-match candidate lacked
Level 1, so the level-filtered candidate loader excluded it and the old
fallback create masked the miss.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker materialized a per-org ThirdParty for every
categorized tracker, linking or creating one through heuristic and
disambiguation-agent matching. Concurrent mapping of two patterns for
the same common third party raced the load-then-create check and left
duplicate org third parties with the same name.
Reduce the worker to catalog resolution only: it resolves the shared
common_tracker_pattern_id / common_third_party_id link and leaves
third_party_id untouched, preserving any link set elsewhere. Org third
parties will instead be created through an explicit per-vendor import
action added in a later commit.
Remove resolveOrgThirdParty, prepareOrgThirdParty, the sibling
direct-link signal, and the disambiguation-agent wiring (including its
constructor parameter and buildTrackerAgents return), and update the
worker tests to assert the catalog link is resolved while third_party_id
is preserved.
Signed-off-by: Émile Ré <emile@probo.com>
The cookie-database aggregator backstop normalised the agent's
third-party name and looked it up against bare brand keys, but
normalizeAlnum folds the eTLD into the key (cookiedatabase.org ->
cookiedatabaseorg). Domain- and URL-form attributions therefore
slipped past the exact lookup, letting noisy aggregator names be
accepted instead of discarded.
Add uri.DomainLabel to reduce a host-like string to its primary
registrable label and check it alongside the normalised name, so
both brand ("Cookiepedia") and domain forms ("cookiedatabase.org",
"https://www.cookiepedia.co.uk/list") resolve to the same key.
Signed-off-by: Émile Ré <emile@probo.com>
Cookie-database and consent-directory sites (Cookifi, Cookiepedia,
cookiedatabase.org, CookieServe, ...) rank highly in web search only
because they catalog cookies, not because they set them. The mapping
agent could attribute a tracker to the directory operator itself
instead of the vendor the page names.
Instruct the agent to treat such results as reference directories and
extract the named vendor, never the operator, while keeping a CMP's own
product cookie attributable (OptanonConsent -> OneTrust, CookieConsent
-> Cookiebot). Add a conservative code backstop that discards
attributions to pure aggregators, scoped to exclude CMP vendors so
legitimate own-cookie attributions survive.
Signed-off-by: Émile Ré <emile@probo.com>
Patterns like "ethereum-https://letaido.com" embed the scanned site's
own origin, appended either by a browser extension (e.g. a wallet
injecting window.ethereum) or by an owner-set tracker. The mapping
agent read that embedded domain and attributed the site to itself as a
third party ("Letaido"), but a site owner is never a third party of its
own site.
Pass the scanned site's eTLD+1 to the agent as a <scanned_site> signal
and instruct it to ignore a domain embedded in the pattern that matches
the site, returning low confidence when that is the only cue. Add a
deterministic backstop that discards any agent attribution whose vendor
name resolves to the site's own domain, so the pattern falls through to
the unmatched fallback regardless of whether the model honored the
prompt. Deterministic catalog, naming, and sibling signals run before
the agent and are untouched, so an independent strong vendor signal
still attributes normally.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the many-to-many junction table with a direct
parent_third_party_id foreign key on third_parties. Each
sub-third-party now belongs to exactly one parent, making
duplicates across parents independent entities.
Replace the firstLevel boolean with an integer level field
(1 = direct, 2+ = parent level + 1) to support arbitrary
nesting depth.
Remove the createThirdPartyThirdPartyMapping and
deleteThirdPartyThirdPartyMapping mutations, the CLI
link/unlink commands, and the corresponding MCP tools.
Creating a child third party now just requires passing
parentThirdPartyId on the existing createThirdParty mutation.
The frontend walks the parentThirdParty chain to build
display names like "Name (Ancestor1/Ancestor2)" and shows
clickable ancestor links on the detail page.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping, common-pattern enrichment, and third-party
disambiguation agents were all built from one shared
TrackerAgentsConfig fed by a single tracker-mapping config slot. That
forced a single AgentTimeout to be reused and patched per worker, and
two unrelated max-turns fields to share one struct.
Split the in-code config into TrackerMappingAgentConfig,
TrackerEnrichmentAgentConfig, and DisambiguationAgentConfig, each with
its own timeout and max-turns, and add dedicated tracker-enrichment and
third-party-disambiguation provider slots (the latter resolving next to
third-party-vetter). Enrichment and disambiguation fall back to the
tracker-mapping slot when their own provider is unset, preserving
single-config deployments.
Drop the shared pkg/agentsbuild package and duplicate its small wiring
into probod and proboctl so the two executables stay decoupled. Wire
the new env vars, builder test coverage, and Helm values.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker clears mapping_requested_at at claim time, so
a crash or hard failure between Process phases left the pattern dequeued,
unmapped, and with nothing to re-trigger it. Only an incidental sibling
remap could rescue it, so a lone pattern could stay stranded forever.
Implement the worker.StaleRecoverer interface, mirroring the enrichment
worker. ResetStaleMappings re-arms rows that were claimed but never
assigned a catalog row (common_tracker_pattern_id IS NULL) once idle past
a configurable window; a successful Process always assigns one via the
unmatched fallback, so the predicate cleanly detects interrupted runs and
self-heals after a single pass. ClearMappingRequestedAt now bumps
updated_at so the stale clock starts at claim time and the sweep never
recycles an in-flight claim.
Plumb a StaleAfter knob (default 600s) through the config struct, builder
env var, probod wiring, and Helm templates.
Signed-off-by: Émile Ré <emile@probo.com>
The deterministic tracker-mapping heuristics group patterns by shared
initiator domain, but tag managers, customer-data platforms, and
generic CDNs (Google Tag Manager, Segment, cloudfront.net, ...)
initiate trackers for many unrelated vendors. Grouping on such a
domain mis-attributes one vendor's tracker to another.
Add uri.FilterSharedInfrastructureDomains backed by a curated eTLD+1
denylist and apply it once in resolveDeterministic, so sibling
grouping, catalog domain matching, and the sibling re-enqueue cascade
all ignore shared-infrastructure hosts. Vendor-specific domains such
as google-analytics.com are intentionally kept as a same-vendor
signal. The agent path is unchanged: it still sees observed domains,
now with a prompt caveat about shared infrastructure.
Update the two sibling tests that used googletagmanager.com as the
initiator domain to a vendor domain, since that host is now stripped
before grouping.
Signed-off-by: Émile Ré <emile@probo.com>
Address review feedback on the agent-driven tracker catalog path:
- Return initiator-domain load failures instead of swallowing them,
so the worker retries rather than running the agent on partial
context.
- In the resolver, treat only ErrResourceNotFound as a catalog miss
and propagate genuine name/slug lookup errors.
- Insert the new vendor inside a savepoint and, on the slug
unique-violation race, reload and return the winning row instead of
aborting the caller's transaction.
- Stop seeding common_third_party_domains from observed initiator
domains. They are a co-occurrence signal, not verified ownership,
and writing them into the global cross-tenant catalog pollutes the
domain-based matcher. The curated seed owns that data.
- Warn the mapping agent that observed domains may belong to shared
CDNs, tag managers, or hosting infrastructure rather than the
vendor, so it does not attribute on that basis alone.
- Extract a shared tracker-identification prompt helper and move the
common-pattern identification prompt next to the enrichment agent.
Signed-off-by: Émile Ré <emile@probo.com>
resolveOrCreateCommonThirdParty lived as a package-level helper in the
tracker mapping worker, but the common pattern enrichment worker now
reuses it. Homing shared catalog logic in a mapping-named file made the
enrichment worker quietly depend on the mapping worker's file, and it is
not a mapping concern.
Move it to pkg/thirdparty as exported ResolveOrCreateCommonThirdParty,
decoupled from cookiebanner's TrackerMappingAgentResult (it now takes a
name and category) to avoid an import cycle. It stays a transaction-
scoped free function so both workers compose it into their own tx for
atomicity rather than receiving a service that owns its own connection.
Relocate the catalog dedup DB test alongside it.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs its LLM and web-search phases
between short transactions and holds no row lock across them. The
pattern-analysis worker can merge a pattern into a glob and delete
it in that window, so the final UpdateMapping then fails with
ErrResourceNotFound and the task errors out spuriously.
A vanished pattern has nothing left to map, so treat the concurrent
delete as a no-op: log it and return nil instead of failing.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.
Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.
Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.
Signed-off-by: Émile Ré <emile@probo.com>
The agent returned a single confidence that conflated two unrelated
judgments: whether an artifact is a meaningful web tracker and which
vendor set it. The prompt's tracker-worthiness skepticism drove the
number down for extension state like __darkreader__wasEnabledForHost,
pushing it below the gate and dropping the attribution entirely, so a
clearly-named vendor never reached the catalog.
Rename the agent field to ThirdPartyConfidence and scope it to the
attribution alone. The identify gate now checks that a vendor is named
with sufficient confidence; on success the catalog row is stored at a
fixed agent confidence like the other heuristic signals, and on failure
the unmatched fallback still records the pattern with no third party.
The stored pattern confidence was only used for ordering and as agent
context, never as a gate, so a separate LLM-provided number is dropped
rather than split out.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker descriptions were only filled on the agent-identification path,
so patterns resolved by domain, sibling, or fallback stayed without one,
and empty mapping upserts could clobber a researched description on the
shared catalog row.
Move description ownership to a dedicated, global common-pattern
enrichment worker. New catalog rows are queued on insert; the worker
researches a compliance-grade description with web search, records it on
the common pattern, and fans it out to every linked tracker pattern. The
mapping worker no longer generates descriptions and only propagates an
already-enriched one at link time.
Rename TrackerMappingConfig to TrackerAgentsConfig since the mapping and
enrichment agents now share it.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker loaded a pattern in its claim transaction and
committed the resolution in a separate, later transaction. A full-row
Update would write back stale values and clobber any user edit made in
between. Add UpdateMapping, which writes only the worker-resolved
columns (common_tracker_pattern_id, third_party_id, and a description
filled only when still empty), leaving user-editable fields untouched.
Also add ORDER BY tracker_pattern_id to the sibling pattern lookup: the
query used LIMIT without an ORDER BY, so an over-limit match set
returned an arbitrary subset and could resolve the third party
differently across runs.
Signed-off-by: Émile Ré <emile@probo.com>
A tracker pattern's source ratchets PRE_EXISTING -> EXTENSION -> SCRIPT
as stronger detections arrive, but that promotion was never reflected
back to the mapping pipeline. The detection that promotes the source
also brings a fresh initiator domain that matchByDomain and
matchBySiblingOrigin can use, and an EXTENSION -> SCRIPT promotion lifts
the creationAllowed gate that blocks org third-party creation. Yet the
pattern's mapping_requested_at was already cleared after its first pass,
so the worker never revisited it.
Re-arm mapping_requested_at via SetMappingRequested at each
source-promotion site (reportDetectedTracker plus the glob-merge and
adoption paths in the pattern-analysis worker). Update's SET clause does
not cover mapping_requested_at, so assigning the field before Update
would be a silent no-op; SetMappingRequested only writes when the column
is NULL, keeping already-queued patterns from being double-enqueued.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker processes one pattern at a time and
matchBySiblingOrigin only reads already-resolved siblings, so vendor
propagation across a banner was forward-only. A sibling processed
before its peer resolved a vendor (for example, one that failed the
agent and fell back to an unmatched catalog row) was never revisited,
even once a later sibling clearly identified the same third party.
When a Process run newly establishes a common third party, re-arm
mapping_requested_at on same-banner siblings that share an initiator
domain and are still unpromoted and non-extension-sourced. The worker
re-claims them and matchBySiblingOrigin now finds the freshly mapped
pattern. Guarding on third_party_id IS NULL, mapping_requested_at IS
NULL, and a not-pre-existing common third party keeps cascades finite.
Signed-off-by: Émile Ré <emile@probo.com>
Sibling matching is an org-local co-occurrence signal: two patterns
served from the same origin on one banner are likely the same vendor,
even when that origin is the site's own (first-party) host. First-party
filtering only protects the global catalog (domain) match, where a
proxied tracker would otherwise hit the site owner's own entry. It now
runs solely before matchByDomain, so matchBySiblingOrigin sees the
unfiltered domains and promotion happens for patterns detected on the
banner's own origin.
Resolve the sibling's direct org third party and its catalog third party
as independent signals, so a single shared org third party no longer
short-circuits the common-pattern backfill.
Make the shared test fixtures unique per tenant: common_third_parties
and common_tracker_patterns are global with unique indexes, so parallel
tests previously collided on name, slug, and pattern. Also align the
sibling tests' stored initiator domains with production, which records
the eTLD+1.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker patterns detected on the same banner that share initiator
domains are a strong indicator of the same third party. Previously the
mapping worker only checked the global third-party domain catalog, so a
tracker whose domain was not registered there fell through to the
expensive LLM identification step even when a co-located pattern was
already mapped.
Add a matchBySiblingOrigin step that finds other patterns on the same
banner sharing the same initiator domains and reuses their resolved
common third party. It prefers siblings already promoted to an org
third party (the strongest signal) and falls back to siblings carrying
only a catalog link, skipping when the siblings disagree. The step runs
before the catalog domain lookup since an already-qualified sibling is
at least as reliable as a raw domain match.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker scripts loaded through a first-party reverse proxy (e.g.
t.probo.com proxying PostHog) share the scanned site's eTLD+1 and
were incorrectly matched against the site owner's own
CommonThirdParty entry in matchByDomain. This caused trackers like
ph_phc_* to be attributed to the site owner instead of PostHog.
Load the CookieBanner origin in Process and pass it to both
matchByDomain and identifyWithAgent. Both now filter out initiator
domains whose eTLD+1 matches the site before querying the catalog
or feeding domains to the LLM agent. The prompt is also updated to
warn about proxy domains.
Signed-off-by: Émile Ré <emile@probo.com>
When the mapping worker resolves a CommonTrackerPattern, propagate
its description back to the org TrackerPattern if the latter is
still empty. This ensures agent-produced descriptions reach the
user-facing tracker instead of staying only in the catalog.
The Update method now covers all mutable TrackerPattern columns
including common_tracker_pattern_id and third_party_id, replacing
the removed UpdateMapping method.
Signed-off-by: Émile Ré <emile@probo.com>
Catalog resolution (common_tracker_pattern_id) still runs for every
pattern, but promoteThirdParty is now gated on the tracker's cookie
category: patterns still sitting in the uncategorised bucket are not
promoted to an org ThirdParty until the user moves them to a real
category, which re-triggers the worker via SetMappingRequested.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the cross-entity JOIN in
DetectedTrackers.LoadCommonThirdPartyIDByDomainMatch with two
idiomatic coredata calls: LoadInitiatorDomainsByTrackerPatternID
on DetectedTrackers, then a new CommonThirdPartyDomains.Load with
a CommonThirdPartyDomainFilter. Each entity now queries only its
own table, and the caller orchestrates the lookup.
Document the Load vs LoadAll naming convention and the no
cross-entity JOINs rule in contrib/claude/coredata.md.
Signed-off-by: Émile Ré <emile@probo.com>
Upsert methods now RETURNING all struct columns and scan the result
back into the pointer receiver, keeping the caller in sync with the
actual DB state (id, created_at, etc. from the existing row on
conflict). Insert detection compares the saved original ID with the
returned ID instead of relying on the PostgreSQL-internal xmax column.
Signed-off-by: Émile Ré <emile@probo.com>
Generate the category list from coredata.ThirdPartyCategories() at
runtime instead of hardcoding it in the prompt text. Type the
TrackerIdentification.Category field as coredata.ThirdPartyCategory so
JSON unmarshaling validates values automatically.
Also documents the .txt.tmpl template file naming convention.
Signed-off-by: Émile Ré <emile@probo.com>
Write errors from Insert/Upsert were logged but swallowed,
letting the transaction commit with incomplete data. All
helper methods now return errors so the transaction rolls
back on failure.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl has a single public API at https://api.firecrawl.dev/v2.
The endpoint was configurable but never varied across environments,
so hardcode it as a package-level const and remove the Endpoint
field from FirecrawlConfig and all downstream wiring (bootstrap,
Helm chart, probod, vetting, cookiebanner).
Signed-off-by: Émile Ré <emile@probo.com>
SearXNG was a fallback search backend that added complexity without
being used in practice. All search-dependent features (web search,
government DB checks, vetting orchestrator, tracker mapping) now use
Firecrawl exclusively. Removes the SEARCH_ENDPOINT config plumbing
from probodconfig, bootstrap, Helm charts, and all callers.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl provides higher quality search results than SearXNG.
When configured (firecrawl-endpoint + firecrawl-api-key), the
tracker-mapping agent and search toolset prefer it over the
SearXNG backend. Also improves the tracker identification prompt
with multi-strategy search queries that leverage domain signals
and adapt to tracker type.
Signed-off-by: Émile Ré <emile@probo.com>
When both pattern matching and domain matching fail to identify a
tracker, an opt-in LLM agent can now attempt identification using
internal database searches and optional web search. The agent returns
structured output (third party name, category, description, confidence)
and the worker auto-creates CommonThirdParty records when needed.
The feature is gated behind the `llm.tracker-mapping.provider` config
field; when unset the worker behaves exactly as before.
Signed-off-by: Émile Ré <emile@probo.com>
When neither pattern nor domain matching finds an existing common
tracker pattern, upsert one with no third party association and a
lower confidence (0.5) so every seen pattern is cataloged.
Also fix table name in LoadByOrganizationIDAndCommonThirdPartyID
(vendors -> third_parties).
Signed-off-by: Émile Ré <emile@probo.com>
Move tracker_mapping_worker.go from pkg/probo to pkg/cookiebanner and
rename worker.go to pattern_analysis_worker.go to reflect the worker
name.
Signed-off-by: Émile Ré <emile@probo.com>
