The tracker-mapping worker re-arms same-banner siblings after a pattern
resolves a vendor. Its predicate only excluded promoted siblings
(third_party_id IS NULL), but since org-party auto-creation was dropped a
pattern can resolve a common third party yet never gain an org
third_party_id. Those siblings, and terminal first-party ones, stayed
eligible forever, so every cascade step re-enqueued and reprocessed them,
amplifying Process runs to O(N^2) per banner. The deadlock fix in the
last release removed the rollbacks that had accidentally throttled the
cascade, so the latent amplification surfaced as an INFO-log flood.
Tighten the re-enqueue to skip siblings already linked to a catalog row
that carries a common third party or marked FIRST_PARTY, dropping
per-banner reprocessing back to O(N). Also demote the two per-run handler
logs ("mapped tracker pattern", "re-enqueued unmapped sibling tracker
patterns") to Debug so routine processing no longer logs at INFO.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs many Process calls in parallel. In
Phase 4 a single transaction locked the worker's own claimed pattern
row via UpdateMapping and then locked sibling rows on the same banner
via the re-enqueue. Two workers mapping sibling patterns on one banner
each held their own row and waited on the other's, forming a lock cycle
that Postgres aborted with deadlock detected (40P01).
Split the sibling re-enqueue into its own short transaction that runs
after the mapping commits, so the claimed-row lock is released before
any sibling row is locked. Also take the sibling UPDATE row locks in a
deterministic id order through an ORDER BY id ... FOR UPDATE subquery,
so overlapping re-enqueues can no longer invert lock order between
themselves. The re-enqueue only flags siblings, so deferring it past
the commit is safe and lets reprocessed siblings observe committed data.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-pattern source ranking collapsed HTTP into the PRE_EXISTING
tier, so a cookie first enumerated as pre-existing and later re-observed
only via a Set-Cookie response header stayed pre-existing. That left it
on the agent-skipped tier (isPreExistingSource), even though an HTTP
server-set cookie is real page evidence, not the extension-state
catch-all the skip was built to suppress.
Give HTTP its own rank between SCRIPT and EXTENSION
(SCRIPT > HTTP > EXTENSION > PRE_EXISTING) in both sourceRank and the
bestSource merge rollup, so an HTTP re-detection now promotes the
pattern and re-arms mapping, unblocking the identification agent.
Signed-off-by: Émile Ré <emile@probo.com>
Unlinking previously cleared only the catalog vendor link, leaving the
stale description that still named the removed vendor and leaving the
linked org tracker patterns pointing at it. Because the verdict becomes
UNDETERMINED the pipeline is meant to re-probe the row, so the stale
state was misleading until that happened.
Clear the description on both the catalog row and its uncategorised org
tracker patterns, and remap those org patterns so they drop the stale
vendor and re-resolve. A re-resolved vendor re-arms catalog enrichment
via the blank-and-unlinked upsert path, re-deriving the description.
This mirrors the cleanup mark-first-party already performs, minus the
terminal verdict.
Signed-off-by: Émile Ré <emile@probo.com>
Surface every CookieSource value in the console: the trackers page
filter was missing the HTTP option and the source badge helper had no
EXTENSION case, so HTTP-sourced rows could not be filtered and
extension-sourced rows rendered the raw enum string.
On the backend, the mark-first-party verdict now blanks the stale
description on both the catalog row and its uncategorised org tracker
patterns. A terminal non-third-party row keeps no vendor link, so a
description naming the (now-cleared) vendor would be misleading; the
mapping worker only copies descriptions into empty rows and never
clears them, so clearing is done explicitly here.
Signed-off-by: Émile Ré <emile@probo.com>
Manual OAuth bearer tokens worked on Console and Connect but
were rejected by MCP, which only ran the personal API key
middleware. Align MCP with the shared bearer chain used
elsewhere: API key, OAuth access token, then identity
presence. Drop the local RequireAPIKeyHandler.
Add e2e coverage for MCP calls authenticated with a manual
OAuth token, including scope enforcement.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Device logins only requested OIDC scopes while the authorizer now
gates API calls on v1:* scopes. Register the full scope set on the
well-known prb client, request it at login via CLIClientScopes, and
cover the device flow in e2e.
Collapse API scopes under an accordion on the consent screen and
document scope sync for future namespace additions.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Let users create, list, and revoke manual bearer tokens from
/me/oauth-tokens, scoped to their identity rather than an
organization. Manual tokens store a null client_id and are
authorized with a self-manage IAM policy.
Wire Connect GraphQL on Identity (list, create, revoke), add
console UI with scoped create flow and credentials dialog, and
cover the flow in e2e tests. Fix list pagination ordering and
keep the Relay connection in sync after create.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
OAuth grant tokens issued before API scope enforcement only stored
OIDC scopes. Set every existing iam_oauth2_access_tokens row to the
full supported scope set so bearer tokens keep working under the
OAuth2 scope gate.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Register v1 API scopes in coredata, advertise them in OIDC discovery
and protected-resource metadata, show them on the consent screen, and
enforce scope-to-action mapping in the IAM Authorizer before policy
evaluation.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
The Upsert behavior is documented in the Go comment above the
method; inline SQL comments duplicated that and cluttered the
query.
Signed-off-by: Émile Ré <emile@probo.com>
Three paths could leave a catalog row's attribution out of step with its
common_third_party_id. A FIRST_PARTY reclassification in the mapping
worker kept a stale org ThirdParty link instead of clearing it. The
upsert requeued terminal FIRST_PARTY rows for enrichment on a vendor
they never adopt, since the vendor-preservation clause nulls it. And the
proboctl upsert command did not normalize the verdict when an operator
linked or unlinked a vendor without passing --attribution.
Clear the org link on a first-party verdict, exclude FIRST_PARTY rows
from the enrichment requeue, and have the CLI downgrade THIRD_PARTY to
UNDETERMINED on unlink and promote UNDETERMINED to THIRD_PARTY on link.
Signed-off-by: Émile Ré <emile@probo.com>
Mirror the common third party upsert for the tracker pattern catalog so
an operator can curate rows by hand. The command is keyed on the natural
key (tracker type, pattern, max age) rather than a slug, applies each
optional field only when its flag is passed, and reuses the existing
Upsert invariants that protect a terminal FIRST_PARTY verdict and never
clobber a researched description with an empty one. Pass --enrich to
re-arm the row for the async enrichment worker.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-pattern catalog was binary (linked to a vendor or not), so
generic and first-party artifacts (loglevel keys, wallet-extension keys,
an org's own trackers) were retried forever and, once one row was wrongly
attributed, re-propagated to every organization with no re-check.
Give catalog rows a terminal attribution verdict (UNDETERMINED,
THIRD_PARTY, FIRST_PARTY): FIRST_PARTY short-circuits the whole mapping
pipeline so the artifact is never attributed again. Gate deterministic
vendor adoption behind a trust bar so only curated/operator rows
auto-propagate; lower-confidence agent/heuristic rows are reused as hints
and re-resolved, and an independent agent re-confirmation corroborates and
promotes them. Make the mapping agent emit an evidence source and reject
any attribution that lacks concrete evidence, and let it declare a
first-party verdict. Skip the speculative agent for PRE_EXISTING-source
patterns, whose low signal invites invented vendors.
Add proboctl "ctp mark-first-party" and an --attribution list filter to
audit and remediate existing wrong links, and a cursor rule documenting
migration naming so the timestamp is taken from date -u, not invented.
Signed-off-by: Émile Ré <emile@probo.com>
Replace the immediate per-document approval email and the manual
"send signing notifications" action with a single debounced worker that
batches pending requests per recipient and organization.
The worker (go.gearno.de/kit/worker) polls on an interval (default 5m)
and claims one (organization, recipient) group at a time, sending one
consolidated signing email and/or one approval email per recipient/org
that lists every document awaiting their signature or approval. The
claim is a conditional UPDATE that doubles as concurrency-safe dedup, so
several workers never email the same group twice.
Each request is notified once it has been pending past the debounce
delay (default 15m), then reminded at 1x, 2x and 3x the reminder
interval (default 1 day) after the previous email, after which it stops.
New last_notified_at and notification_count columns on signatures and
approval decisions drive the debounce, the widening reminder cadence and
the four-email cap.
Email copy lists each document with its title, type and a deep link to
the employee page. Removed the inline approval-on-publish email, the
SendSigningNotifications service method/mutation/MCP tool, its IAM action,
and the related console UI and n8n operation.
Signed-off-by: Sacha Al Himdani <sacha@probo.com>
Map membership, profile, and inactive-user failures from
OpenOIDCChildSessionForOrganization to a generic 404 instead
of 500 so org-scoped OIDC callbacks do not reveal tenant
access details.
Signed-off-by: Bryan Frimin <bryan@probo.com>
OIDC login dropped organization_id before the provider redirect, so
callbacks with an existing matching root session never created an org
child session. Persist organization_id in OIDC state, open the child
session on callback, and forward the parameter from the sign-in UI.
Signed-off-by: Bryan Frimin <bryan@probo.com>
The common tracker pattern "show" only printed attempt counters, so the
enrichment payload — run status, model, agent attribution, and per-field
outcomes — was invisible, and a partial run was indistinguishable from a
complete one. Render that payload the same way the common third party
"show" already does.
Surface the last enrichment attempt timestamp in both catalog list views
so an operator can spot stale or never-attempted rows at a glance.
Signed-off-by: Émile Ré <emile@probo.com>
The migration dropped the DEFAULT on
common_tracker_patterns.enrichment_attempts, so the column is now
NOT NULL with no default and inserts must set it explicitly. The e2e
seedCommonTrackerPattern helper still did a raw insert without it,
which broke TestThirdParty_ImportFromCommon and
TestTrackerPattern_CommonTrackerPatternID with a 23502 violation. Pass
0 for the seeded rows.
Signed-off-by: Émile Ré <emile@probo.com>
Three crash- and migration-recovery gaps in the unified enrichment
model left rows stuck or misclassified:
- Upsert re-armed a blank, newly-linked tracker pattern without
clearing its prior enrichment payload. A crash between the worker's
claim and persist then left the row with a stale payload, so the
stale-recovery sweep (which only catches rows with a null payload)
skipped it forever. Clear enrichment on re-arm so the row reads as
not-yet-completed again, and pin the behavior with a test.
- The migration added last_enrichment_attempt_at to
common_third_parties without seeding it. Rows with prior attempts
kept a NULL clock and could never satisfy the stale-reset predicate.
Backfill from updated_at, the historical claim-time proxy.
- The migration switched the tracker-pattern enriched-state source to
the enrichment payload without backfilling rows previously marked by
enriched_at, making already-enriched rows read as unenriched.
Seed a provenance sentinel for rows that carried the old done-flag.
Signed-off-by: Émile Ré <emile@probo.com>
Let an operator create or update a catalog third party and queue it for
the async enrichment worker in one step, so a minimal name/category row
gets its profile, compliance documents, owned domains, and logo filled
in without a separate reenrich call.
Enrichment is armed explicitly via RequestEnrichmentByIDs rather than
through the receiver: the coredata Upsert sets enrichment_requested_at
from the receiver only on insert and leaves it untouched on conflict,
so this is the only path that re-arms an existing row uniformly while
also resetting the attempt budget for a fresh run.
Signed-off-by: Émile Ré <emile@probo.com>
Make common_tracker_patterns and common_third_parties share one
enrichment-tracking model and fix the misleading proboctl status.
Both tables now carry the enrichment JSONB provenance payload, an
enrichment_attempts counter, and a last_enrichment_attempt_at clock.
On common_tracker_patterns the enriched_at done-flag is renamed to
last_enrichment_attempt_at and stamped at claim time, so it is truthful
to "attempt" rather than "success". A row is considered to have been
through the workflow when it carries an enrichment payload, not when a
timestamp is set, which lets stale recovery key off the payload being
absent with budget remaining, exactly like common_third_parties.
The claim path reads the attempt counter and timestamp back via
RETURNING so the in-memory receiver matches the database clock instead
of a separate app-side time.Now.
The enricher builds a per-field provenance payload (description and
third-party outcomes plus the mapping attribution) and persists it via
UpdateEnrichment, named to mirror the common-third-party sibling. The
common pattern enrichment worker gains a max-attempts ceiling so a
permanently failing row stops looping.
proboctl now shows "enriched" only when every field the last run
recorded an outcome for resolved a value, otherwise "partial (X/Y)",
replacing the misleading "enriched (no description)" label.
Signed-off-by: Émile Ré <emile@probo.com>
A token without admin.directory.customer.readonly makes
Customers.Get return 403 forever. Treat that as terminal so
the source-name worker keeps the generic name and stops
retrying.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Unbounded LoadAll* loaders materialised an entire result set in one
query with no ceiling. A table that is small in development can grow
without bound in production, so these loaders were a latent memory
and query-time hazard.
Remove the LoadAll* methods from pkg/coredata and walk the cursor-
paginated LoadBy* siblings instead through a shared page.LoadAll
helper. The helper advances a MaxCursorSize forward cursor until the
result set is exhausted and concatenates the pages. It caps a single
call at MaxLoadAllPages (20) batches of 500 rows and errors past that
rather than materialising an unbounded set, so a runaway caller fails
loudly instead of exhausting memory.
Callers that genuinely need every row now express that explicitly,
and the coredata load-naming rule and docs are updated to discourage
new unbounded loaders.
Signed-off-by: Sacha Al Himdani <sacha@probo.com>
The MCP Profile schema declares additional_email_addresses as a
required, non-nullable array, but NewProfile passed the mail.Addrs
value through unchanged. A profile with no extra emails has a nil
slice, which marshals to JSON null and fails tool output validation
with "type: null, want array".
Default the nil slice to an empty mail.Addrs so the marshalled output
always honors the strict array schema.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Long role strings in the access review table broke row layout when
drivers joined many roles into one comma-separated value. Expose
roles as a string array in GraphQL by splitting the stored role at
the API layer, and render the first three roles as badges with a
"+X more" popover for the rest.
Closes ENG-459.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Employee document signatures recorded an acknowledgment with no
cryptographic proof, unlike document approvals which already create
and accept an electronic signature on every decision.
Mirror the approval flow on the sign path: generate the signed
document PDF, create-and-accept an esign record, and persist its id
on the document_version_signatures row through a new
electronic_signature_id column. Capture the signer IP and user agent
in the resolver, and re-check the published/archived preconditions
inside the transaction so the seal cannot race document state.
Make the consent wording a single backend source of truth shared by
the text that is sealed and the text shown in the UI. Define
DocumentSignatureConsentText and DocumentApprovalConsentText in the
probo service package and the NDA copy in the trust service, each
owned by the flow that uses it, and stop esign from appending the
generic clause to caller-provided consent text so approvals no
longer seal a duplicated sentence.
Expose the resolved consent text through GraphQL on
EmployeeDocumentVersion and DocumentVersionApprovalDecision, and have
the signing, approval, and NDA pages render it from the API instead
of hard-coded strings, mirroring how the NDA page already worked.
Align the wording with the actual interaction: the buttons read
"Review and sign" and "Review and approve", the clauses reference
those actions, and the inaccurate "typing my full name" phrasing is
removed everywhere.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Denials were invisible in the audit trail and evaluator explainability
(policy_id, reason) was discarded before reaching logs. Emit a structured
authz decision line on every evaluation in evaluateMultiInTx — allow,
deny, no_match, and assumption errors — using the existing authorizer
logger with opaque IDs only.
Add decision_log.go with DecisionRecord and logDecision. Surface
PolicyID and Reason on EvaluationResult for logging. Audit log
behavior is unchanged (allow-only). Document the convention in
authorization.md.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Expose campaign sources as first-class nodes, paginate fetch attempts
instead of denormalized status fields, and bind entries to their
campaign snapshot. Update GraphQL, MCP, CLI, console, and e2e coverage
to match.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Source category was never surfaced in the product and added noise to
snapshots and APIs. Remove the enum, columns, and service fields so
campaign sources track only the identity fields reviewers need.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Fresh installs already had csv_data on campaign sources from the
initial migration; add a follow-up migration to backfill existing
rows from the live source snapshot.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Bad API keys and expired OAuth tokens showed Connected because
probes ran only for OAuth2 and many providers had no ProbeURL.
Add a registry ProbeConnection dispatcher with static, dynamic,
and custom probes so all 41 providers are checked on demand.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Cursor's Admin API exposes two independent removal signals: the isRemoved
boolean and a role value of "removed". They are not always consistent —
a member can carry role "removed" while isRemoved is still false, a known
gap documented on the Cursor community forum.
Previously Active was derived from isRemoved alone, so a member with
role "removed" but isRemoved=false was incorrectly reported as active.
Now either signal is sufficient to mark the account inactive.
Add a cassette entry and test case covering the inconsistent state
(role "removed", isRemoved false) to prevent regression.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Decouple each campaign from the live access-review sources it was started
with by introducing a per-campaign source snapshot table
(access_review_campaign_sources). The snapshot captures the source name,
category, and connector at start time, so a review remains coherent even
after the underlying source is edited or deleted. Fetch tracking becomes
an append-only log (access_review_campaign_source_fetch_attempts) that
preserves every attempt with its own status and error rather than
overwriting a single row.
Rename the shared access-review tables and enums to use a consistent
access_review_ prefix throughout:
access_entries → access_review_entries
access_sources → access_review_sources
access_source_category → access_review_source_category
access_entry_* → access_review_entry_*
The same rename propagates to every coredata type, service, GraphQL
schema, MCP specification, CLI command, frontend component, and e2e test.
The accessreview package gains dedicated actions.go and policies.go files
for its own IAM policy set, mirroring the agentrun package pattern.
Signed-off-by: Bryan Frimin <bryan@probo.com>
The enrichment agent now reads full cookie-database and vendor pages
through the browser. The facts it gathers (setter, purpose, data
stored) are not protectable, but a source page's written description
is original expression. Instruct the agent to state the facts in its
own words and never copy a sentence verbatim, so stored descriptions
stay clear of third-party prose.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern enrichment agents only had
web search, which returns title/url/snippet, so they could never
open a cookie-database or cookie-policy page to read which vendor
actually sets a tracker. This mis-attributed setters whose snippet
is misleading (e.g. _li_* read as LinkedIn rather than LiveIntent).
Wire the read-only headless-browser toolset into both agents, gated
on a configured Chrome endpoint, mirroring the common-third-party
enrichment worker: agent construction moves into the run path so each
run can carry a per-run browser that is closed when the run returns.
The prompts now direct the agent to open a promising result and read
the named setter from the full page text. Both agents stay unchanged
when no Chrome endpoint is configured.
Signed-off-by: Émile Ré <emile@probo.com>
Brand assets served at /api/files/v1/static had no cache headers.
Introduce brand.Assets to own the embedded filesystem, content-hash
ETags, and HTTP serving. Responses now carry Cache-Control and ETag
so clients can cache and revalidate; stable email URLs stay
revalidatable (max-age=3600, no immutable).
Replace hardcoded Default*Path constants with StaticPathPrefix,
logical filename constants, and StaticPath(). NewAssets validates
required assets at startup so a rename fails fast instead of 404ing
in sent emails. The files handler keeps routing and 404 rendering;
ServeAssets sets cache headers and serves the file.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Address PR review feedback: rename the shared string-helper package
from strutil to stringsx to avoid the discouraged util suffix and the
collision with the standard strings package, updating all import paths
and call sites.
Replace the manual wg.Add/wg.Done bookkeeping in the enrichment worker
with wg.Go, which is less error-prone.
Signed-off-by: Émile Ré <emile@probo.com>
The alphanumeric-normalisation helper was duplicated verbatim in the
common-third-party owned-domains resolver and the cookiebanner tracker
mapping worker. Hoist it into a new dependency-free strutil package so
both call sites share one implementation and one test.
Signed-off-by: Émile Ré <emile@probo.com>
The common-third-party enrichment pipeline ran Agent B (compliance
docs), Agent C (owned domains), and the deterministic logo step
sequentially even though, once Agent A resolves the website, the
three depend only on that website and not on each other. Fan them
out across goroutines under a WaitGroup so wall time is the slowest
of the three rather than their sum. Each step builds its own per-run
browser and writes only into its own locals; the shared LLM, HTTP,
and FileManager clients are safe for concurrent use and the database
is untouched until persist. Results merge in a fixed order so
runErrors and log output stay deterministic.
Also replace the single-sentence confidence guidance in the three
agent prompts with an explicit, calibrated rubric tied to evidence
strength, and remind the model that a downstream threshold gates
persistence so it should neither inflate nor deflate its estimates.
Signed-off-by: Émile Ré <emile@probo.com>
Reject oversized logo responses instead of silently truncating them,
which could persist corrupt image bytes as a valid logo.
Tighten ownership substring matching with a length-ratio guard so a
short label root no longer attributes unrelated domains to a vendor.
Render the worker confidence threshold when set to zero by testing
against nil, so an explicit "accept all" value is not dropped by Helm's
falsy-numeric truthiness.
Sanitize and bound per-agent error text before persisting it to the
enrichment metadata column to avoid leaking unbounded internal detail.
Signed-off-by: Émile Ré <emile@probo.com>
