When neither pattern nor domain matching finds an existing common
tracker pattern, upsert one with no third party association and a
lower confidence (0.5) so every seen pattern is cataloged.
Also fix table name in LoadByOrganizationIDAndCommonThirdPartyID
(vendors -> third_parties).
Signed-off-by: Émile Ré <emile@probo.com>
Remove redundant resolveThirdParty fallback that re-attempted
resolution with the same inputs, and rename
LoadCommonThirdPartyIDByTrackerPatternID to
LoadCommonThirdPartyIDByDomainMatch to reflect that it joins on
initiator_domain.
Signed-off-by: Émile Ré <emile@probo.com>
Change the unique constraint on common_third_party_domains from (domain)
to (common_third_party_id, domain) so a domain like google.com can be
associated with Google, Google Analytics, Google AdSense, etc.
Also fix Beamer domain (was hotjar.com, now getbeamer.com) and preserve
the original url.Parse error in pgconn DSN parsing.
Signed-off-by: Émile Ré <emile@probo.com>
Add a slug column to common_third_parties for stable cross-source
deduplication. The OCD tracker patterns importer now resolves third
parties via a three-step cascade (slug lookup, domain fallback,
auto-create) instead of silently dropping unmatched links.
Signed-off-by: Émile Ré <emile@probo.com>
- Replace copy-pasted New Relic URLs in Resend third-party entry
with correct resend.com URLs and drop inapplicable fields
- Escape single '%' instead of '%%' in LIKE pattern conversion so
literal percent signs are not treated as wildcards
- Return actual row ID from CommonTrackerPattern.Upsert via
RETURNING id so conflict-path callers get the existing ID
- Add ORDER BY id ASC to vendor-by-common-third-party query for
deterministic LIMIT 1 selection
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
LoadByID and LoadByIDs used 4-space indentation and were missed
by the bulk column addition. The struct field existed but the
SELECT did not return it, causing pgx scan failures.
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
Poll-based worker that maps org-scoped tracker patterns to the
common knowledge base via pattern matching and domain-based
attribution. Populates initiator_domain on detected trackers
at report time. Resolves org-scoped vendors through the common
third party link.
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
CommonThirdPartyDomain maps eTLD+1 domains to third parties for
fast domain-based attribution. CommonTrackerPattern stores the
global tracker knowledge base with glob/exact matching support.
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
Introduces common_third_party_domains (CITEXT domain aliases per
third party), common_tracker_patterns (global tracker knowledge
base), and new FK columns on tracker_patterns (third_party_id,
common_tracker_pattern_id), vendors (common_third_party_id), and
detected_trackers (initiator_domain) to support automated
tracker-to-vendor attribution.
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
- Require CLIENT_SECRET for new access-review connectors
- Use Heroku account UUID as ExternalID
- Bump GitHub orgs picker to per_page=100
- Drop Snyk, Ramp, Lever, Deel access-review providers
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
- Rename access-review migration with random time
- Move PagerDuty token-response handling to its own file
- Strip OAuth error_description from log and redirect
- Validate PagerDuty subdomain on OAuth callback
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
The register/document model has fully replaced the snapshot system.
Delete every snapshot-scoped row and strip the application code that
referenced them: SnapshotID/SourceID struct fields, snapshot_id IS NULL
filters, snapshot columns from SELECT/INSERT statements and named args,
and the eight migrate-*-snapshots-to-documents one-shot tools.
The remaining snapshot_id / source_id columns, the snapshots and
controls_snapshots tables, the snapshots_type enum, and the
snapshot-scoped indexes are now unused; they are dropped in a follow-up
schema migration so this change can roll back cleanly without leaving
orphaned data.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Renames the user-facing 'vendor' concept to 'third party' across the
entire codebase. The shared common_third_parties reference table is
unchanged.
Migration. Renames the vendor_category enum, the vendors and
vendor_<entity> tables (contacts, services, compliance_reports,
business_associate_agreements, data_privacy_agreements,
risk_assessments) and their vendor_id columns, the asset_vendors /
data_vendors / processing_activity_vendors junction tables,
generated_documents.vendors_document_id, the webhook_event_type
'vendor:<verb>' values, and the snapshots_type 'VENDORS' value.
Backend. Renames coredata models and SQL queries, probo services,
GraphQL / MCP API surface, console / trust / webhook resolvers and
types, the CLI (prb vendor* -> prb third-party*; pkg/cmd/vendormgmt
-> pkg/cmd/thirdpartymgmt), the document generator, vetting agent
prompts, and the common-third-parties-import command.
Frontend, packages, n8n, e2e. Renames apps/console pages, components,
hooks, routes, dialogs, and tabs; the shared @probo/vendors package
(now @probo/third-parties); the @probo/ui Vendors atoms (now
ThirdParties, VendorLogo -> ThirdPartyLogo); the n8n community node
actions/vendor folder (now actions/thirdParty); and the e2e Go test
suite (console and MCP). Filesystem and URL paths use kebab-case
(third-parties), GraphQL fields and TypeScript identifiers use
camelCase (thirdParty / thirdParties), Go types use PascalCase
(ThirdParty), and human-facing text uses 'third party' with a space.
Co-authored-by: Bryan Frimin <bryan@getprobo.com>
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
These tables were superseded by tracker_patterns and
detected_trackers in migration 20260505T090000Z, which
backfilled all data. The old tables are no longer referenced
by any Go code, GraphQL schema, MCP specification, or CLI
command.
The cookie_source and cookie_pattern_match_type enum types
are retained because tracker_patterns still uses them.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Émile Ré <nemile.re@gmail.com>
The consent mode is now determined dynamically by the visitor's
country and its applicable regulation. The configured consent_mode
column is dropped from cookie_banners and added to
cookie_consent_records to persist the geo-derived mode at
consent-recording time. When no regulation matches, the default
is OPT_OUT.
Signed-off-by: Émile Ré <emile@getprobo.com>
Replace per-row UPDATE inside the detected-tracker loop with a single
bulk UPDATE ... WHERE id = ANY(...) after the loop. The old approach
locked pattern rows in request-dependent order, causing deadlocks
under concurrent ReportDetectedTrackers calls.
Signed-off-by: Émile Ré <emile@getprobo.com>
The description field was never surfaced in the UI and added no value.
Drop it from the database, Go structs, GraphQL schema, import tool,
frontend fragment, and vendor seed data.
Signed-off-by: Émile Ré <emile@getprobo.com>
Fetch favicons at import time instead of calling Google's favicon
service per page load. Logos are stored as public files in S3 and
served through the existing /api/files/v1/{id} endpoint.
Signed-off-by: Émile Ré <emile@getprobo.com>
The CreateVendorDialog previously loaded the entire @probo/vendors
JSON bundle client-side and used MiniSearch for fuzzy search. This
replaces it with a GraphQL query against the common_third_parties
database table, searched server-side via ILIKE filtering.
Backend: adds CommonThirdParty GraphQL type, a pkg/thirdparty
service, and a commonThirdParties(name) root query. Frontend:
splits into CommonThirdPartyCombobox (display) and an @inline
fragment read on selection via readInlineData.
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduce a globally-shared, non-tenant-scoped common_third_parties
table that mirrors the public subset of vendor metadata, plus a
one-shot cmd/common-third-parties-import CLI that seeds it from
packages/vendors/data.json. The catalog will back future flows (e.g.
vendor autocomplete) so each tenant no longer needs to duplicate the
same baseline data.
The importer is idempotent via ON CONFLICT (lower(name)) DO UPDATE
and prints inserted/updated counts. GIDs use gid.NilTenant since the
table is not tenant-scoped; uniqueness still comes from the entity
type plus 14 bytes of timestamp/random suffix.
Signed-off-by: Émile Ré <emile@getprobo.com>
tracker_patterns rows were created with entity type 88 (removed
CookiePatternEntityType) instead of 89 (TrackerPatternEntityType), and
detected_trackers rows migrated from the cookies table carried entity
type 85 (removed CookieEntityType) instead of 90 (DetectedTrackerEntityType).
Signed-off-by: Émile Ré <emile@getprobo.com>
When IP geolocation returned no matching CIDR block, LookupCountryByIP
returned an empty string with nil error. The handler took the address of
that empty string, producing a non-nil pointer to "", which was inserted
into the database. Guard against this by returning nil when the resolved
country code is empty, and backfill existing rows with a migration.
Signed-off-by: Émile Ré <emile@getprobo.com>
fmt.Sprintf interprets the literal % characters in the LIKE escape
clause as format verbs, corrupting the query and causing a 500 on
the /report endpoint. Reorder tracker type / source filters in the
trackers page.
Signed-off-by: Émile Ré <emile@getprobo.com>
The pattern analysis worker now recognises UUID-like, hash-like,
and long numeric tokens as variable parts and replaces them with
wildcards heuristically, even from a single observation. This
prevents site-specific identifiers from being treated as static
text while meaningful suffixes (window_id, posthog, …) get
incorrectly wildcarded.
Also upgrades globMatch and the FindMatchingPattern SQL query
to support multiple wildcards in a single pattern.
Signed-off-by: Émile Ré <emile@getprobo.com>
The trackers page now lets users filter by tracker type
(Cookie, localStorage, sessionStorage, IndexedDB, Cache
Storage) in addition to the existing source filter. Each
tracker type and cookie source badge uses a distinct color
for quick visual scanning.
Signed-off-by: Émile Ré <emile@getprobo.com>
The method uses ON CONFLICT ... DO UPDATE, so the name now
matches the actual behaviour. TrackerPattern.InsertIfNotExists
keeps its name because it uses ON CONFLICT ... DO NOTHING.
Signed-off-by: Émile Ré <emile@getprobo.com>
A registered service worker is a URL-shaped artifact (origin+path of
the worker script), so it goes in tracker_resources as a new
SERVICE_WORKER resource type. A Cache Storage bucket is an opaque
named string with no URL, so it goes in detected_trackers as a new
CACHE_STORAGE tracker type.
Frontend:
- StorageDetector wraps caches.open() and enumerates caches.keys()
on start to surface pre-existing buckets that pre-date the SDK
load (service workers commonly populate caches eagerly on
install).
- ThirdPartyDetector wraps navigator.serviceWorker.register() and
enumerates getRegistrations() on start.
Both wrappers degrade silently on insecure contexts where these APIs
are unavailable.
Signed-off-by: Émile Ré <emile@getprobo.com>
ThirdPartyDetector previously only saw <script src> and <iframe src>
because it scanned the DOM and watched mutations. Add a single
PerformanceObserver({type:'resource', buffered:true}) that picks up
everything the browser actually loaded:
- tracking pixels (<img>, <picture>, srcset)
- cross-origin stylesheets and web fonts
- fetch / XHR / sendBeacon / ping calls (SDK call-homes)
- video, audio, embed, object media
initiatorType is mapped to six new tracker_resource_type enum values
(IMAGE, STYLESHEET, FONT, BEACON, FETCH, MEDIA) and the existing
upsert path in tracker_resources picks them up unchanged.
Closes a real gap with headless cookie scanners: most SDKs phone home
via beacons after their script is gone, and the DOM scan never saw it.
Signed-off-by: Émile Ré <emile@getprobo.com>
When third-party JS sets a cookie or writes to local/sessionStorage
inside a customer page, the SDK now walks the synchronous call stack
to find the first non-extension, non-Probo, non-first-party http(s)
URL. That origin+path is sent as initiator_url on the report payload,
persisted in a new nullable column on detected_trackers, and preserved
across upserts via COALESCE.
This unlocks per-vendor attribution for cookies and storage writes
without needing pattern name matching, so future categorisation logic
can simply look up the initiator URL in the existing tracker_resources
table and inherit that vendor's category.
GraphQL/MCP exposure is intentionally deferred -- the column is captured
now, surfaced later.
Signed-off-by: Émile Ré <emile@getprobo.com>
Move resource tracking (scripts, iframes) out of the pattern-based
tracker_patterns/detected_trackers machinery into its own
tracker_resources table keyed by (banner, type, origin, path).
- Add migration that creates the tracker_resource_type enum, the
tracker_resources table with a unique index, drops existing
SCRIPT/IFRAME rows (not yet in production), and recreates the
tracker_type enum without those values.
- Add TrackerResource coredata model with full CRUD, Upsert (bumps
last_detected_at on conflict), list/count/move operations, filter,
and order field support.
- Register TrackerResourceEntityType (91) in the entity type registry.
- Drop TrackerTypeScript/TrackerTypeIframe from TrackerType enum.
- Update handler to use TrackerResourceType for resource detection.
- Temporarily stub out resource ingestion in ReportDetectedTrackers
pending the service-layer wiring in the next commit.
- Drop SCRIPT/IFRAME from the GraphQL TrackerType enum.
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduces a wildcard-based match type that supports prefix, suffix,
and sandwich patterns (e.g. ph_phc_*_posthog). The SQL matching uses
starts_with/ends_with on the parts split at '*', avoiding LIKE and
its underscore escaping issues. Existing PREFIX rows are migrated to
GLOB with a trailing '*'.
Signed-off-by: Émile Ré <emile@getprobo.com>
Progressive enhancement for Chromium browsers: listen on the
CookieStore change event to catch cookies set by Set-Cookie HTTP
response headers, which the document.cookie setter hook cannot see.
Adds a new "http" cookie source through the full stack.
Signed-off-by: Émile Ré <emile@getprobo.com>
Trackers sharing a prefix but with materially different lifetimes
(e.g. session vs 1-year) were incorrectly merged into a single
prefix pattern. Port the snap table from cookie-utils.ts into Go
and use it to bucket durations so only trackers that display the
same human-readable lifetime can merge. Update the unique index
to include COALESCE(max_age_seconds, -1) so prefix patterns with
different durations can coexist.
Signed-off-by: Émile Ré <emile@getprobo.com>
Add the missing MICROSOFT_365 value to the connector_provider enum
so the connector loader stops failing with SQLSTATE 22P02.
Scope each Identity Provider card to its own SCIMBridge type so
connecting Microsoft 365 no longer marks Google Workspace as
connected (and vice versa).
Filter Microsoft Graph /users to userType eq 'Member' so the bridge
only syncs home-tenant members and skips B2B guest accounts that
were polluting the synced People list.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Store the visitor's resolved country code alongside regulation in
cookie_consent_records so downstream surfaces can expose it.
Signed-off-by: Émile Ré <emile@getprobo.com>
Replace TRUNCATE+COPY inside a single transaction with a staging table
approach so that SELECT queries from the cookie banner handler are never
blocked during import. The ACCESS EXCLUSIVE lock is now only held for
the sub-millisecond DROP+RENAME swap.
Signed-off-by: Émile Ré <emile@getprobo.com>
Move cookie banner types (CookieItem, Category, Regulation, BannerConfig,
etc.) into a dedicated types.ts file. Add a coredata.Regulation type with
parsing, JSON marshaling, and database scanning methods. Hardcode the
geoloc-import data directory since the submodule path is fixed.
Signed-off-by: Émile Ré <emile@getprobo.com>
