The description field was never surfaced in the UI and added no value.
Drop it from the database, Go structs, GraphQL schema, import tool,
frontend fragment, and vendor seed data.
Signed-off-by: Émile Ré <emile@getprobo.com>
Fetch favicons at import time instead of calling Google's favicon
service per page load. Logos are stored as public files in S3 and
served through the existing /api/files/v1/{id} endpoint.
Signed-off-by: Émile Ré <emile@getprobo.com>
The CreateVendorDialog previously loaded the entire @probo/vendors
JSON bundle client-side and used MiniSearch for fuzzy search. This
replaces it with a GraphQL query against the common_third_parties
database table, searched server-side via ILIKE filtering.
Backend: adds CommonThirdParty GraphQL type, a pkg/thirdparty
service, and a commonThirdParties(name) root query. Frontend:
splits into CommonThirdPartyCombobox (display) and an @inline
fragment read on selection via readInlineData.
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduce a globally-shared, non-tenant-scoped common_third_parties
table that mirrors the public subset of vendor metadata, plus a
one-shot cmd/common-third-parties-import CLI that seeds it from
packages/vendors/data.json. The catalog will back future flows (e.g.
vendor autocomplete) so each tenant no longer needs to duplicate the
same baseline data.
The importer is idempotent via ON CONFLICT (lower(name)) DO UPDATE
and prints inserted/updated counts. GIDs use gid.NilTenant since the
table is not tenant-scoped; uniqueness still comes from the entity
type plus 14 bytes of timestamp/random suffix.
Signed-off-by: Émile Ré <emile@getprobo.com>
tracker_patterns rows were created with entity type 88 (removed
CookiePatternEntityType) instead of 89 (TrackerPatternEntityType), and
detected_trackers rows migrated from the cookies table carried entity
type 85 (removed CookieEntityType) instead of 90 (DetectedTrackerEntityType).
Signed-off-by: Émile Ré <emile@getprobo.com>
When IP geolocation returned no matching CIDR block, LookupCountryByIP
returned an empty string with nil error. The handler took the address of
that empty string, producing a non-nil pointer to "", which was inserted
into the database. Guard against this by returning nil when the resolved
country code is empty, and backfill existing rows with a migration.
Signed-off-by: Émile Ré <emile@getprobo.com>
fmt.Sprintf interprets the literal % characters in the LIKE escape
clause as format verbs, corrupting the query and causing a 500 on
the /report endpoint. Reorder tracker type / source filters in the
trackers page.
Signed-off-by: Émile Ré <emile@getprobo.com>
The pattern analysis worker now recognises UUID-like, hash-like,
and long numeric tokens as variable parts and replaces them with
wildcards heuristically, even from a single observation. This
prevents site-specific identifiers from being treated as static
text while meaningful suffixes (window_id, posthog, …) get
incorrectly wildcarded.
Also upgrades globMatch and the FindMatchingPattern SQL query
to support multiple wildcards in a single pattern.
Signed-off-by: Émile Ré <emile@getprobo.com>
The trackers page now lets users filter by tracker type
(Cookie, localStorage, sessionStorage, IndexedDB, Cache
Storage) in addition to the existing source filter. Each
tracker type and cookie source badge uses a distinct color
for quick visual scanning.
Signed-off-by: Émile Ré <emile@getprobo.com>
The method uses ON CONFLICT ... DO UPDATE, so the name now
matches the actual behaviour. TrackerPattern.InsertIfNotExists
keeps its name because it uses ON CONFLICT ... DO NOTHING.
Signed-off-by: Émile Ré <emile@getprobo.com>
A registered service worker is a URL-shaped artifact (origin+path of
the worker script), so it goes in tracker_resources as a new
SERVICE_WORKER resource type. A Cache Storage bucket is an opaque
named string with no URL, so it goes in detected_trackers as a new
CACHE_STORAGE tracker type.
Frontend:
- StorageDetector wraps caches.open() and enumerates caches.keys()
on start to surface pre-existing buckets that pre-date the SDK
load (service workers commonly populate caches eagerly on
install).
- ThirdPartyDetector wraps navigator.serviceWorker.register() and
enumerates getRegistrations() on start.
Both wrappers degrade silently on insecure contexts where these APIs
are unavailable.
Signed-off-by: Émile Ré <emile@getprobo.com>
ThirdPartyDetector previously only saw <script src> and <iframe src>
because it scanned the DOM and watched mutations. Add a single
PerformanceObserver({type:'resource', buffered:true}) that picks up
everything the browser actually loaded:
- tracking pixels (<img>, <picture>, srcset)
- cross-origin stylesheets and web fonts
- fetch / XHR / sendBeacon / ping calls (SDK call-homes)
- video, audio, embed, object media
initiatorType is mapped to six new tracker_resource_type enum values
(IMAGE, STYLESHEET, FONT, BEACON, FETCH, MEDIA) and the existing
upsert path in tracker_resources picks them up unchanged.
Closes a real gap with headless cookie scanners: most SDKs phone home
via beacons after their script is gone, and the DOM scan never saw it.
Signed-off-by: Émile Ré <emile@getprobo.com>
When third-party JS sets a cookie or writes to local/sessionStorage
inside a customer page, the SDK now walks the synchronous call stack
to find the first non-extension, non-Probo, non-first-party http(s)
URL. That origin+path is sent as initiator_url on the report payload,
persisted in a new nullable column on detected_trackers, and preserved
across upserts via COALESCE.
This unlocks per-vendor attribution for cookies and storage writes
without needing pattern name matching, so future categorisation logic
can simply look up the initiator URL in the existing tracker_resources
table and inherit that vendor's category.
GraphQL/MCP exposure is intentionally deferred -- the column is captured
now, surfaced later.
Signed-off-by: Émile Ré <emile@getprobo.com>
Move resource tracking (scripts, iframes) out of the pattern-based
tracker_patterns/detected_trackers machinery into its own
tracker_resources table keyed by (banner, type, origin, path).
- Add migration that creates the tracker_resource_type enum, the
tracker_resources table with a unique index, drops existing
SCRIPT/IFRAME rows (not yet in production), and recreates the
tracker_type enum without those values.
- Add TrackerResource coredata model with full CRUD, Upsert (bumps
last_detected_at on conflict), list/count/move operations, filter,
and order field support.
- Register TrackerResourceEntityType (91) in the entity type registry.
- Drop TrackerTypeScript/TrackerTypeIframe from TrackerType enum.
- Update handler to use TrackerResourceType for resource detection.
- Temporarily stub out resource ingestion in ReportDetectedTrackers
pending the service-layer wiring in the next commit.
- Drop SCRIPT/IFRAME from the GraphQL TrackerType enum.
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduces a wildcard-based match type that supports prefix, suffix,
and sandwich patterns (e.g. ph_phc_*_posthog). The SQL matching uses
starts_with/ends_with on the parts split at '*', avoiding LIKE and
its underscore escaping issues. Existing PREFIX rows are migrated to
GLOB with a trailing '*'.
Signed-off-by: Émile Ré <emile@getprobo.com>
Progressive enhancement for Chromium browsers: listen on the
CookieStore change event to catch cookies set by Set-Cookie HTTP
response headers, which the document.cookie setter hook cannot see.
Adds a new "http" cookie source through the full stack.
Signed-off-by: Émile Ré <emile@getprobo.com>
Trackers sharing a prefix but with materially different lifetimes
(e.g. session vs 1-year) were incorrectly merged into a single
prefix pattern. Port the snap table from cookie-utils.ts into Go
and use it to bucket durations so only trackers that display the
same human-readable lifetime can merge. Update the unique index
to include COALESCE(max_age_seconds, -1) so prefix patterns with
different durations can coexist.
Signed-off-by: Émile Ré <emile@getprobo.com>
Add the missing MICROSOFT_365 value to the connector_provider enum
so the connector loader stops failing with SQLSTATE 22P02.
Scope each Identity Provider card to its own SCIMBridge type so
connecting Microsoft 365 no longer marks Google Workspace as
connected (and vice versa).
Filter Microsoft Graph /users to userType eq 'Member' so the bridge
only syncs home-tenant members and skips B2B guest accounts that
were polluting the synced People list.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Store the visitor's resolved country code alongside regulation in
cookie_consent_records so downstream surfaces can expose it.
Signed-off-by: Émile Ré <emile@getprobo.com>
Replace TRUNCATE+COPY inside a single transaction with a staging table
approach so that SELECT queries from the cookie banner handler are never
blocked during import. The ACCESS EXCLUSIVE lock is now only held for
the sub-millisecond DROP+RENAME swap.
Signed-off-by: Émile Ré <emile@getprobo.com>
Move cookie banner types (CookieItem, Category, Regulation, BannerConfig,
etc.) into a dedicated types.ts file. Add a coredata.Regulation type with
parsing, JSON marshaling, and database scanning methods. Hardcode the
geoloc-import data directory since the submodule path is fixed.
Signed-off-by: Émile Ré <emile@getprobo.com>
Add a regulation column to cookie_consent_records so each consent
captures which privacy law was in effect. Thread the value from the
handler's geoloc resolution through the service into the DB insert.
On the SDK side, add a Regulation union type to BannerConfig and expose
it via a getter on CookieBannerClient and in the probo-ready event
detail so themed-banner consumers can adapt their UI per regulation.
Signed-off-by: Émile Ré <emile@getprobo.com>
Extract struct and query functions into
pkg/coredata/ip_country_block.go following the convention that
all raw SQL lives in coredata. The geoloc service now delegates
to these functions.
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduce a geoloc package that stores CIDR-to-country mappings in
PostgreSQL using the native cidr type with a GiST index for fast
containment lookups. Data comes from the ipverse/country-ip-blocks
dataset added as a git submodule.
A standalone geoloc-import command reads the TXT files from disk
and bulk-loads them via COPY. probod wires the service and logs a
warning when the table is empty.
Signed-off-by: Émile Ré <emile@getprobo.com>
The application always sets status explicitly when creating a campaign,
so the database does not need to provide a default.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Source fetch failures stay surfaced on the source fetch (status and
last error) instead of failing the entire campaign, so reviewers can
proceed on the sources that succeeded.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Microsoft 365's native SCIM endpoint is unreliable, so mirror the
Google Workspace bridge over Microsoft Graph: a new MICROSOFT_365
OAuth2 connector, a SCIM bridge provider listing /v1.0/users with
$select pagination, and an access review driver that derives admin
status from /directoryRoles members. Refactor the bridge runner to
share OAuth2 plumbing across providers and surface the new bridge
type, scopes, UI card, and bootstrap env wiring.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
- Fix TotalCount resolver to dispatch by parent type instead
of always using the uncategorised banner counter
- Sync MCP tracker_type enum with canonical TrackerType values
- Add validation for UpdateTrackerPatternRequest
- Validate tracker_type on CreateTrackerPatternRequest
- Set LastMatchedAt when creating pattern from detection
- Use COALESCE for SOURCE cursor pagination with NULLs
- Make source nullable in CLI tracker-pattern list
Signed-off-by: Émile Ré <emile@getprobo.com>
Delete coredata.CookiePattern and all associated CRUD methods,
rename shared types (CookiePatternOrderField, CookiePatternFilter,
CookiePatternMatchType) to TrackerPattern equivalents, and migrate
all API surfaces (GraphQL, MCP, CLI, n8n) to tracker_pattern naming.
The worker was already migrated in the base branch; this commit
completes the removal by dropping the old GraphQL schema/resolvers,
service methods, CLI commands, and n8n operations that operated on
the legacy cookie_patterns table.
Signed-off-by: Émile Ré <emile@getprobo.com>
The worker now operates on TrackerPattern/DetectedTrackers
instead of CookiePattern/Cookies, with TrackerType included
in merge group keys to prevent cross-type merging.
Signed-off-by: Émile Ré <emile@getprobo.com>
- TrackerType enum (cookie, local_storage, session_storage, indexed_db, script, iframe)
- TrackerPattern model with EXACT + PREFIX matching for all types
- DetectedTracker model with upsert on conflict
- ReportDetectedTrackers service method handling cookies, storage, and resources
- POST /detected-trackers endpoint on cookie-banner v1 API
- buildSnapshot() now reads from tracker_patterns (cookie type only)
- Entity types registered (89, 90)
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduce a unified detection schema that supports cookies,
localStorage, sessionStorage, IndexedDB, scripts, and iframes
as tracker types. Existing cookie_patterns and cookies data is
backfilled into the new tables to prepare for the organic
privacy scanner feature.
Signed-off-by: Émile Ré <emile@getprobo.com>
Exclude the UNCATEGORISED category at the SQL level so
the admin cookie/display/translations pages only see
consent-relevant categories. Removes dead client-side
UNCATEGORISED filters that are no longer needed.
Signed-off-by: Émile Ré <emile@getprobo.com>
Backend for the cookie banner detection page: a new
uncategorisedPatterns connection on CookieBanner with
sortable (NAME, LAST_MATCHED_AT, UPDATED_AT, SOURCE)
and filterable (text ILIKE on name/description, source
enum) paginated results. COALESCE handles NULL-first
ordering for last_matched_at.
Signed-off-by: Émile Ré <emile@getprobo.com>
Track when cookies are last detected (last_detected_at on cookies)
and when patterns last matched a detected cookie
(last_matched_at on cookie_patterns). The cookie timestamp is
refreshed on every detection report; the pattern timestamp is
computed as MAX(last_detected_at) during pattern analysis.
Signed-off-by: Émile Ré <emile@getprobo.com>
Instead of loading all categories and filtering out
UNCATEGORISED in Go, add LoadConsentCategoriesByCookieBannerID
which excludes it in the query. This avoids fetching data we
immediately discard and makes the intent explicit at each call
site.
Signed-off-by: Émile Ré <emile@getprobo.com>
Replace the old snapshot-based system for risks with the publish
document system, mirroring the prior vendor / processing activity / DPIA
/ TIA migration. Includes the GraphQL mutation, MCP tool, CLI command,
n8n operation, frontend publish dialog, e2e tests, and a prosemirror
register template covering name, description, category, treatment,
owner, inherent and residual scoring, and notes.
The risk register lives as a generated DocumentTypeRegister document on
the organization, reused across publishes (the major version bumps on
every republish). Approvers can be passed in to create a draft pending
approval; otherwise the version is published immediately. The frontend
Risks page exposes a Publish button and a Document link button when the
document exists, and pre-fills the previous default approvers.
Risks was the last remaining snapshot type, so this commit also removes
the entire snapshot system: drop snapshotId from the Risk GraphQL type
and RiskFilter; remove RiskSnapshotter, Risks.Snapshot,
InsertRiskSnapshots, and the SnapshotID/SourceID fields on Risk; delete
Snapshot, ControlSnapshot, SnapshotsType, SnapshotOrderField,
Snapshottable, the SnapshotService, the Snapshot console resolvers and
GraphQL schema, the Snapshot MCP types and operations
(list/get/take/listControlSnapshots), the snapshot CLI (prb snapshot),
the snapshot frontend pages, routes, banner, LinkedSnapshotsCard,
SnapshotGraph, snapshot helpers, and the snapshot n8n resource and
control link/unlink snapshot operations. The snapshot_id columns remain
in the database but are now filtered out with snapshot_id IS NULL.
Add Get/Upsert/Clear GeneratedDocumentID methods on Risk backed by a new
risks_document_id column on generated_documents, matching the
ProcessingActivity/Finding/Vendor pattern. The migration command
migrate-risk-snapshots-to-documents uses raw SQL queries instead of the
Go snapshot types, since those are gone.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Translation changes are cosmetic, not consent-contract changes,
so they should not trigger a version bump. Translations are now
loaded live from the database at serve time instead of being
frozen in the snapshot.
Signed-off-by: Émile Ré <emile@getprobo.com>
Adds an `excluded` boolean column to the cookie_patterns table
so operators can mark patterns to be omitted from the consent
banner without deleting them. Includes the migration, struct
field, updated SQL queries, and filter support.
Signed-off-by: Émile Ré <emile@getprobo.com>
