Provider 4xx on an instance-name lookup is rarely actionable -- it
usually reflects a configuration state (deleted org, revoked token)
rather than a system error worth paging on. Downgrade Probo's own
log entry to Warn. The kit/worker entry for the same failure is
still Error pending the retry-limit work, which will mark sources
synced after N attempts and stop the duplicate log line.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
When a GitHub access-source connector has no organization configured
yet (user finished OAuth but abandoned the picker), the source-name
worker called https://api.github.com/orgs/ and got a 404 every 10
seconds, flooding logs. All other picker resolvers (sentry, gitlab,
bitbucket, heroku, asana, netlify, clickup, vercel) short-circuit to
("", nil) for empty settings -- this aligns github with them so the
worker falls into its existing "empty instance name -> mark synced
with generic name" branch instead of retrying forever.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Queue vetting on third_parties with PENDING, PROCESSING,
COMPLETED, and FAILED states. Expose enqueue and status through
GraphQL, MCP, CLI, and n8n, validate vet requests, tune the
worker via config, and poll the detail page while vetting runs.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping and common-pattern enrichment agents ran with
4 and 3 max turns. Those budgets are smaller than the tool-call
rounds their prompts authorize (two/one DB searches plus up to
three web searches) plus the forced structured-output synthesis
turn, so the run loop tripped MaxTurnsExceededError before it
could emit JSON.
Raise the budgets to 10 across every layer that drives them: the
bootstrap env-var defaults, the Helm chart values and production
example, and the in-code fallback. Update the bootstrap test
defaults to match.
Revert the prompt instructions that told the model to reserve a
turn for its final answer; the model cannot reliably count its
remaining turns, so the fix belongs in the turn budget rather
than the prompt text.
Signed-off-by: Émile Ré <emile@probo.com>
The client no longer strips sampling parameters per model, so the
sanitization test and its capturing provider helper assert behavior
that was intentionally removed. Delete them to restore a green build.
Signed-off-by: Émile Ré <emile@probo.com>
ResolveAgent aliased the default config's Temperature and MaxTokens
pointers into every agent that left them unset, so all resolved
agents shared one backing value. A mutation through any of those
pointers would corrupt the default and every other agent. It also
dereferenced the default unconditionally even though it can be nil.
Allocate a fresh pointer holding a copy of the default value, and
guard against a nil default so each resolved agent owns independent
state.
Signed-off-by: Émile Ré <emile@probo.com>
Remove sanitizeRequest and its call sites so the client no longer
strips sampling parameters based on the model registry. Unsupported
knobs are now avoided by setting the config carefully per model
rather than silently mutating outgoing requests.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker clears mapping_requested_at at claim time, so
a crash or hard failure between Process phases left the pattern dequeued,
unmapped, and with nothing to re-trigger it. Only an incidental sibling
remap could rescue it, so a lone pattern could stay stranded forever.
Implement the worker.StaleRecoverer interface, mirroring the enrichment
worker. ResetStaleMappings re-arms rows that were claimed but never
assigned a catalog row (common_tracker_pattern_id IS NULL) once idle past
a configurable window; a successful Process always assigns one via the
unmatched fallback, so the predicate cleanly detects interrupted runs and
self-heals after a single pass. ClearMappingRequestedAt now bumps
updated_at so the stale clock starts at claim time and the sweep never
recycles an in-flight claim.
Plumb a StaleAfter knob (default 600s) through the config struct, builder
env var, probod wiring, and Helm templates.
Signed-off-by: Émile Ré <emile@probo.com>
The identification agent attributed probo_distinct_id to Mixpanel
purely on the shared distinct_id token, and the enrichment agent
returned no description for the glob ph_phc_*_posthog because it
searched the literal "*" string and found nothing.
Tighten the identification prompt so attribution requires a perfect
pattern match or a meaningful prefix that belongs to the vendor; a
generic token behind a different prefix is not a match. Teach the
enrichment prompt to strip wildcard and variable parts before
searching, and to treat a vendor name embedded in the key as
corroboration so clearly-named trackers still get a description.
Signed-off-by: Émile Ré <emile@probo.com>
The deterministic tracker-mapping heuristics group patterns by shared
initiator domain, but tag managers, customer-data platforms, and
generic CDNs (Google Tag Manager, Segment, cloudfront.net, ...)
initiate trackers for many unrelated vendors. Grouping on such a
domain mis-attributes one vendor's tracker to another.
Add uri.FilterSharedInfrastructureDomains backed by a curated eTLD+1
denylist and apply it once in resolveDeterministic, so sibling
grouping, catalog domain matching, and the sibling re-enqueue cascade
all ignore shared-infrastructure hosts. Vendor-specific domains such
as google-analytics.com are intentionally kept as a same-vendor
signal. The agent path is unchanged: it still sees observed domains,
now with a prompt caveat about shared infrastructure.
Update the two sibling tests that used googletagmanager.com as the
initiator domain to a vendor domain, since that host is now stripped
before grouping.
Signed-off-by: Émile Ré <emile@probo.com>
Address review feedback on the agent-driven tracker catalog path:
- Return initiator-domain load failures instead of swallowing them,
so the worker retries rather than running the agent on partial
context.
- In the resolver, treat only ErrResourceNotFound as a catalog miss
and propagate genuine name/slug lookup errors.
- Insert the new vendor inside a savepoint and, on the slug
unique-violation race, reload and return the winning row instead of
aborting the caller's transaction.
- Stop seeding common_third_party_domains from observed initiator
domains. They are a co-occurrence signal, not verified ownership,
and writing them into the global cross-tenant catalog pollutes the
domain-based matcher. The curated seed owns that data.
- Warn the mapping agent that observed domains may belong to shared
CDNs, tag managers, or hosting infrastructure rather than the
vendor, so it does not attribute on that basis alone.
- Extract a shared tracker-identification prompt helper and move the
common-pattern identification prompt next to the enrichment agent.
Signed-off-by: Émile Ré <emile@probo.com>
resolveOrCreateCommonThirdParty lived as a package-level helper in the
tracker mapping worker, but the common pattern enrichment worker now
reuses it. Homing shared catalog logic in a mapping-named file made the
enrichment worker quietly depend on the mapping worker's file, and it is
not a mapping concern.
Move it to pkg/thirdparty as exported ResolveOrCreateCommonThirdParty,
decoupled from cookiebanner's TrackerMappingAgentResult (it now takes a
name and category) to avoid an import cycle. It stays a transaction-
scoped free function so both workers compose it into their own tx for
atomicity rather than receiving a service that owns its own connection.
Relocate the catalog dedup DB test alongside it.
Signed-off-by: Émile Ré <emile@probo.com>
The enrichment worker no longer invents a description when a tracker's
purpose cannot be substantiated; it records an empty description and
marks the row enriched so the stale-recovery loop does not retry it.
Vendor identification is the mapping pipeline's job, so the enricher
reuses the existing tracker-mapping agent to attribute a third party
for an unlinked common pattern before describing it. A confident
catalog match seeds the enrichment prompt and links the pattern, but
the enricher never creates or overrides an attribution.
When a blank, unlinked catalog row later gains a third party through
the mapping pipeline's upsert, enrichment is re-armed so the now-known
vendor gets a second, better-informed description attempt.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker mapping, common-pattern enrichment, and third-party
disambiguation agents default to a small max-tokens budget on the
premise that their final output is tiny structured JSON. On
reasoning models such as the GPT-5 family, reasoning tokens count
against max_tokens, so a small budget is consumed by reasoning and
the JSON is truncated, surfacing as "unexpected end of JSON input".
Raise the defaults to 4096 (1024 -> 4096 for tracker mapping, 512
-> 4096 for disambiguation) to leave headroom for the reasoning
phase. Update the bootstrap builder default, its test, and the
production values example to match.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs its LLM and web-search phases
between short transactions and holds no row lock across them. The
pattern-analysis worker can merge a pattern into a glob and delete
it in that window, so the final UpdateMapping then fails with
ErrResourceNotFound and the task errors out spuriously.
A vanished pattern has nothing left to map, so treat the concurrent
delete as a no-op: log it and return nil instead of failing.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis worker created the merged glob blank and re-armed
mapping, discarding the org ThirdParty and description already resolved
on the exacts it absorbed. That forced a full re-map (LLM/web-search)
and opened a window where an in-flight exact could vanish mid-mapping.
Seed the glob from the merged exacts when they unanimously agree on a
single third party, carrying its description too, while still re-arming
mapping so the glob derives its own catalog row. With the third party
pre-set, the mapping worker skips the expensive org/disambiguation
resolution. Conflicting or unresolved groups stay blank as before.
The catalog link is deliberately not inherited: it is keyed on the
exact pattern string, not the glob template, so the mapping worker
resolves the right row itself.
Signed-off-by: Émile Ré <emile@probo.com>
The common-pattern enrichment and tracker-mapping agents run on
reasoning models such as gpt-5-nano, which reject an explicit
temperature and fail the whole request with a 400 ("Unsupported
value: 'temperature' does not support 0.1 with this model"). The
model registry already records this capability, but nothing
consulted it before dispatch, and dated provider snapshots like
gpt-5-nano-2025-08-07 did not resolve in the registry.
Resolve dated snapshots to their undated base model in registry
Lookup, and sanitize each chat completion request in the LLM
client by omitting the sampling knobs the target model does not
accept (temperature, top_p, frequency/presence penalties, stop).
Unknown models are left untouched, so models absent from the
registry keep their current behavior.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.
Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.
Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.
Signed-off-by: Émile Ré <emile@probo.com>
SCIM DELETE returned 500 when a profile was still referenced
elsewhere in the org, which disabled the identity-provider bridge
after repeated sync failures. Fall back to deactivation when delete
is blocked, log the conflict without failing sync, and still
attempt delete for excluded users even when inactive.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Allow documents to be archived or unarchived directly from the
list row actions, matching the detail-page behavior. Remove the
row from the active or archived connection after the status change
so filtered lists update immediately.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Address the highest-impact Kubernetes chart issues from #1245.
Align PG_ADDR with postgresql.host/port, wire
OAUTH2_SERVER_SIGNING_KEY, add early validation for required
base64 and PEM secrets, and isolate the main service/deployment
with component labels so Chrome pods are not selected by server
traffic.
Update chart values and documentation to clarify required secret
formats, managed PostgreSQL prerequisites, ACME account key
persistence, and current Azure Blob compatibility caveats for S3
proxy deployments.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Allow campaigns to continue when a source fetch fails by keeping
that failure on the source fetch record only.
The source fetch worker now logs the failure after persisting it and
returns success so campaign execution is not interrupted by source-level
fetch errors.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Treat a single source fetch failure as tolerated so campaigns can
continue fetching and transition normally.
The worker now records failed fetches and only propagates a process
error once the failed source count exceeds one. This keeps the first
failed source visible on the source fetch while preventing the
campaign-level run from being marked failed too early.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Turn the category column into an inline Select so a tracker pattern or
resource can be recategorised in place. On the pattern row the move is
gated by a confirm that surfaces the existing behaviour of promoting or
linking a third party. The remaining row actions (include/exclude and
delete) move into a single action dropdown, leaving only Edit inline.
Add a Max Age column to the pattern table and a Category column to the
resource table, and lay the edit forms out with the name above the
inputs so the description can use the full row width. Shrink the action
column to its content so the freed space goes to the data columns.
Signed-off-by: Émile Ré <emile@probo.com>
The agent returned a single confidence that conflated two unrelated
judgments: whether an artifact is a meaningful web tracker and which
vendor set it. The prompt's tracker-worthiness skepticism drove the
number down for extension state like __darkreader__wasEnabledForHost,
pushing it below the gate and dropping the attribution entirely, so a
clearly-named vendor never reached the catalog.
Rename the agent field to ThirdPartyConfidence and scope it to the
attribution alone. The identify gate now checks that a vendor is named
with sufficient confidence; on success the catalog row is stored at a
fixed agent confidence like the other heuristic signals, and on failure
the unmatched fallback still records the pattern with no third party.
The stored pattern confidence was only used for ordering and as agent
context, never as a gate, so a separate LLM-provided number is dropped
rather than split out.
Signed-off-by: Émile Ré <emile@probo.com>
The coredata CookieSource enum and the ingestion path both support an
HTTP source, but the GraphQL CookieSource enum never declared it. The
generated marshaler is a plain map lookup with no fallback, so an HTTP
value missed the map and serialized to an empty string. The console UI
treats that empty string as falsy and rendered no source badge at all,
making HTTP-sourced trackers look sourceless.
Add the HTTP member to the GraphQL enum so the value round-trips, and
fold the duplicated tracker-type and tracker-source badge helpers from
three components into a shared @probo/helpers module, adding an explicit
HTTP label while consolidating.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker descriptions were only filled on the agent-identification path,
so patterns resolved by domain, sibling, or fallback stayed without one,
and empty mapping upserts could clobber a researched description on the
shared catalog row.
Move description ownership to a dedicated, global common-pattern
enrichment worker. New catalog rows are queued on insert; the worker
researches a compliance-grade description with web search, records it on
the common pattern, and fans it out to every linked tracker pattern. The
mapping worker no longer generates descriptions and only propagates an
already-enriched one at link time.
Rename TrackerMappingConfig to TrackerAgentsConfig since the mapping and
enrichment agents now share it.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker loaded a pattern in its claim transaction and
committed the resolution in a separate, later transaction. A full-row
Update would write back stale values and clobber any user edit made in
between. Add UpdateMapping, which writes only the worker-resolved
columns (common_tracker_pattern_id, third_party_id, and a description
filled only when still empty), leaving user-editable fields untouched.
Also add ORDER BY tracker_pattern_id to the sibling pattern lookup: the
query used LIMIT without an ORDER BY, so an over-limit match set
returned an arbitrary subset and could resolve the third party
differently across runs.
Signed-off-by: Émile Ré <emile@probo.com>
A tracker pattern's source ratchets PRE_EXISTING -> EXTENSION -> SCRIPT
as stronger detections arrive, but that promotion was never reflected
back to the mapping pipeline. The detection that promotes the source
also brings a fresh initiator domain that matchByDomain and
matchBySiblingOrigin can use, and an EXTENSION -> SCRIPT promotion lifts
the creationAllowed gate that blocks org third-party creation. Yet the
pattern's mapping_requested_at was already cleared after its first pass,
so the worker never revisited it.
Re-arm mapping_requested_at via SetMappingRequested at each
source-promotion site (reportDetectedTracker plus the glob-merge and
adoption paths in the pattern-analysis worker). Update's SET clause does
not cover mapping_requested_at, so assigning the field before Update
would be a silent no-op; SetMappingRequested only writes when the column
is NULL, keeping already-queued patterns from being double-enqueued.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker processes one pattern at a time and
matchBySiblingOrigin only reads already-resolved siblings, so vendor
propagation across a banner was forward-only. A sibling processed
before its peer resolved a vendor (for example, one that failed the
agent and fell back to an unmatched catalog row) was never revisited,
even once a later sibling clearly identified the same third party.
When a Process run newly establishes a common third party, re-arm
mapping_requested_at on same-banner siblings that share an initiator
domain and are still unpromoted and non-extension-sourced. The worker
re-claims them and matchBySiblingOrigin now finds the freshly mapped
pattern. Guarding on third_party_id IS NULL, mapping_requested_at IS
NULL, and a not-pre-existing common third party keeps cascades finite.
Signed-off-by: Émile Ré <emile@probo.com>
Sibling matching is an org-local co-occurrence signal: two patterns
served from the same origin on one banner are likely the same vendor,
even when that origin is the site's own (first-party) host. First-party
filtering only protects the global catalog (domain) match, where a
proxied tracker would otherwise hit the site owner's own entry. It now
runs solely before matchByDomain, so matchBySiblingOrigin sees the
unfiltered domains and promotion happens for patterns detected on the
banner's own origin.
Resolve the sibling's direct org third party and its catalog third party
as independent signals, so a single shared org third party no longer
short-circuits the common-pattern backfill.
Make the shared test fixtures unique per tenant: common_third_parties
and common_tracker_patterns are global with unique indexes, so parallel
tests previously collided on name, slug, and pattern. Also align the
sibling tests' stored initiator domains with production, which records
the eTLD+1.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker patterns detected on the same banner that share initiator
domains are a strong indicator of the same third party. Previously the
mapping worker only checked the global third-party domain catalog, so a
tracker whose domain was not registered there fell through to the
expensive LLM identification step even when a co-located pattern was
already mapped.
Add a matchBySiblingOrigin step that finds other patterns on the same
banner sharing the same initiator domains and reuses their resolved
common third party. It prefers siblings already promoted to an org
third party (the strongest signal) and falls back to siblings carrying
only a catalog link, skipping when the siblings disagree. The step runs
before the catalog domain lookup since an already-qualified sibling is
at least as reliable as a raw domain match.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker scripts loaded through a first-party reverse proxy (e.g.
t.probo.com proxying PostHog) share the scanned site's eTLD+1 and
were incorrectly matched against the site owner's own
CommonThirdParty entry in matchByDomain. This caused trackers like
ph_phc_* to be attributed to the site owner instead of PostHog.
Load the CookieBanner origin in Process and pass it to both
matchByDomain and identifyWithAgent. Both now filter out initiator
domains whose eTLD+1 matches the site before querying the catalog
or feeding domains to the LLM agent. The prompt is also updated to
warn about proxy domains.
Signed-off-by: Émile Ré <emile@probo.com>
When the mapping worker resolves a CommonTrackerPattern, propagate
its description back to the org TrackerPattern if the latter is
still empty. This ensures agent-produced descriptions reach the
user-facing tracker instead of staying only in the catalog.
The Update method now covers all mutable TrackerPattern columns
including common_tracker_pattern_id and third_party_id, replacing
the removed UpdateMapping method.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the two-step ID-materializing pattern (fetch IDs in Go, pass
as ANY(@ids)) with an IN-subquery that keeps the filtering entirely
in the database and eliminates an extra round trip. Remove the now
unused LoadIDsByCommonThirdPartyID and its service wrapper. Update
the coredata rule to clarify that subqueries for filtering are OK.
Signed-off-by: Émile Ré <emile@probo.com>
When an org third party already has a common_third_party_id set,
LinkToCommon now skips the write instead of overwriting it with a
different catalog ID. This prevents heuristic or agent false
positives from corrupting a previous, more accurate association.
Signed-off-by: Émile Ré <emile@probo.com>
Third parties created by the tracker mapping worker are confirmed
active on the organization's cookie banner, making them first-level
by definition. Also remove unused ptr test helpers.
Signed-off-by: Émile Ré <emile@probo.com>
Catalog resolution (common_tracker_pattern_id) still runs for every
pattern, but promoteThirdParty is now gated on the tracker's cookie
category: patterns still sitting in the uncategorised bucket are not
promoted to an org ThirdParty until the user moves them to a real
category, which re-triggers the worker via SetMappingRequested.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
