Fold POSTHOG_SELF_HOSTED into POSTHOG: one provider now covers Cloud (OAuth + region-pinned API key) and self-hosted (API key + instance URL), since both already share the driver, name resolver, and PostHogConnectorSettings{BaseURL}. The API-key form picks a deployment (Cloud US/EU or self-hosted URL); the resolver requires exactly one of region/instanceUrl.
Drop the POSTHOG_SELF_HOSTED enum value, registration, migration, and logo mapping. Extract the deployment selector into a dedicated PostHogDeploymentField component. Point the driver tests at us.posthog.com instead of the legacy app.posthog.com host.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
The us/eu host strings lived in the driver and were duplicated in the
connector-settings resolver. Expose drivers.PostHogRegionBaseURL as the
single source and call it from the resolver.
Also parse the base URL only inside the absolute-next branch of
resolveNextURL, where it is actually used for the host check.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
buildTokenRequest repeated the form-encoded POST construction and the
shared headers across the basic-form, none and post-form branches. Extract
newFormTokenRequest, and fold the public-client none case into the default
branch — it differs only in omitting client_secret.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Marshal the region (us/eu) for PostHog Cloud and the instance URL
for PostHog Self-Hosted from the create-connector input, validating
each, and add the matching GraphQL inputs and the POSTHOG_SELF_HOSTED
enum value.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
PostHog Cloud authenticates via CIMD OAuth (public client, PKCE)
through the region-agnostic oauth.posthog.com gateway, with an API-key
fallback. PostHog Self-Hosted is a separate provider using an API key
and an instance URL.
The shared driver discovers the data region by probing us/eu for OAuth
connections, since the gateway does not serve the data API, and pins
pagination to the resolved host.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Public-client (CIMD) providers need no operator credentials: derive
their client_id from the deployment base URL and their state-signing key
from the active OAuth2 server signing key, registering them at startup
unless the operator configured them explicitly.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
A public client identifies itself to the provider with a hosted
Client ID Metadata Document. Serve it unauthenticated and outside the
auth group, since the provider fetches it server-to-server, exposing the
deployment-derived client_id and redirect_uri alongside the Probo brand
name, homepage and logo.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Public clients authenticate with PKCE and no client secret, using a
hosted Client ID Metadata Document (CIMD) as the client_id.
Add a no-secret token-endpoint mode, derive the state-token salt and the
PKCE verifier from a server-side key so the verifier never appears in
the signed-but-unencrypted state, and expose Registration.PublicClient,
Registry.PublicClients and the CIMD metadata path for provider wiring.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
CommonThirdParty.logoUrl and TrustCenterReference.logoUrl were
returning expiring S3 presigned URLs, which break if cached or
shared past their TTL.
Replace with stable /api/files/v1/{id} application URLs.
file.Service now generates these via baseurl; a new filesign
package owns presigning for the files/v1 HTTP handler that
does the internal redirect.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Expose the generated cookie policy Document on the CookieBanner
GraphQL type through a nullable policyDocument field and resolver,
and surface a link to it from the banner configuration header next
to the origin and ID. The link is hidden until a banner version is
published and the document exists.
Rework the tracker table rows: drop the Source column in favour of
a tracker Type badge, and move each tracker's description inline
beneath its name (and into the add/edit row inputs) instead of a
separate Description column.
Signed-off-by: Émile Ré <emile@probo.com>
Other generated policy documents number their top-level sections.
Apply the same convention to the cookie and tracking technologies
policy by prefixing each "##" heading with a running counter. A
counter is used rather than literal numbers so the sequence stays
contiguous when the optional "Third parties that set trackers"
section is omitted.
Signed-off-by: Émile Ré <emile@probo.com>
PublishTrackerPolicy ran template execution, markdown parsing and JSON
marshaling inside the WithTx callback, holding a write transaction open
across CPU-bound work. Split it into a read phase (WithConn) that
gathers data and renders the document, followed by a write phase that
persists the document and version, mirroring PublishThirdPartyList.
Also sort the generated third-party rows before returning. LoadByIDs
has no ORDER BY, so the policy document could otherwise be emitted in a
different row order on each regeneration.
Signed-off-by: Émile Ré <emile@probo.com>
Trackers sharing a display name can differ in type, so the generated
cookie and tracking technologies policy was ambiguous without it. Carry
the tracker type through the banner version snapshot and surface it as a
dedicated column in the policy table.
Stop the snapshot from dropping non-cookie trackers so storage, IndexedDB
and cache technologies appear in the policy and served banner config with
their real type. Duration now reflects the type when no max-age applies:
session storage clears with the tab, the remaining storage technologies
persist. Legacy snapshots predate the field and only ever held cookies,
so GetSnapshot backfills an empty type as COOKIE, keeping the non-null
GraphQL enum and policy output valid without a migration.
Signed-off-by: Émile Ré <emile@probo.com>
An organization can run several websites, each with its own banner and
generated policy. Add the banner origin to the document title and the
rendered heading so the policies are distinguishable in document lists
and the trust center.
Signed-off-by: Émile Ré <emile@probo.com>
Add a poll-based tracker policy worker that claims banners flagged for
policy generation, clears the flag, and rebuilds the cookie and
tracking technologies policy from the latest published snapshot. A
banner that lost its published version is skipped rather than failed.
Wire the worker into probod with crash propagation and graceful
shutdown alongside the other tracker workers.
Signed-off-by: Émile Ré <emile@probo.com>
Add a markdown "Cookie and Tracking Technologies Policy" template in
the policy-writer tone that covers cookies and other tracking
technologies, with GDPR, UK GDPR, FADP, and CCPA/CPRA privacy-rights
framing. Convert it to ProseMirror JSON and add PublishTrackerPolicy,
which builds the document from a banner's latest published version
snapshot and its tracker third parties. The document is GENERATED and
private in the trust center, linked to the banner via
policy_document_id.
Add CookieItem.HumanizedDuration so server-rendered lifetimes match the
banner's humanizeDuration helper, and reword the publish trigger and
backfill comment to reflect the broader tracker scope.
Signed-off-by: Émile Ré <emile@probo.com>
Add policy_document_id and policy_generation_requested_at columns to
cookie_banners and backfill banners that already have a published
version, so existing live banners get a policy on the worker's first
pass.
Flag the banner for policy generation inside the publish transaction so
generation is requested only when a cookie banner version is published,
not on draft edits. Fold policy_document_id into the existing Update so
the upcoming worker can persist the generated document id with the
scope it already holds.
Signed-off-by: Émile Ré <emile@probo.com>
Provider 4xx on an instance-name lookup is rarely actionable -- it
usually reflects a configuration state (deleted org, revoked token)
rather than a system error worth paging on. Downgrade Probo's own
log entry to Warn. The kit/worker entry for the same failure is
still Error pending the retry-limit work, which will mark sources
synced after N attempts and stop the duplicate log line.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
When a GitHub access-source connector has no organization configured
yet (user finished OAuth but abandoned the picker), the source-name
worker called https://api.github.com/orgs/ and got a 404 every 10
seconds, flooding logs. All other picker resolvers (sentry, gitlab,
bitbucket, heroku, asana, netlify, clickup, vercel) short-circuit to
("", nil) for empty settings -- this aligns github with them so the
worker falls into its existing "empty instance name -> mark synced
with generic name" branch instead of retrying forever.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Queue vetting on third_parties with PENDING, PROCESSING,
COMPLETED, and FAILED states. Expose enqueue and status through
GraphQL, MCP, CLI, and n8n, validate vet requests, tune the
worker via config, and poll the detail page while vetting runs.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
The tracker-mapping and common-pattern enrichment agents ran with
4 and 3 max turns. Those budgets are smaller than the tool-call
rounds their prompts authorize (two/one DB searches plus up to
three web searches) plus the forced structured-output synthesis
turn, so the run loop tripped MaxTurnsExceededError before it
could emit JSON.
Raise the budgets to 10 across every layer that drives them: the
bootstrap env-var defaults, the Helm chart values and production
example, and the in-code fallback. Update the bootstrap test
defaults to match.
Revert the prompt instructions that told the model to reserve a
turn for its final answer; the model cannot reliably count its
remaining turns, so the fix belongs in the turn budget rather
than the prompt text.
Signed-off-by: Émile Ré <emile@probo.com>
The client no longer strips sampling parameters per model, so the
sanitization test and its capturing provider helper assert behavior
that was intentionally removed. Delete them to restore a green build.
Signed-off-by: Émile Ré <emile@probo.com>
ResolveAgent aliased the default config's Temperature and MaxTokens
pointers into every agent that left them unset, so all resolved
agents shared one backing value. A mutation through any of those
pointers would corrupt the default and every other agent. It also
dereferenced the default unconditionally even though it can be nil.
Allocate a fresh pointer holding a copy of the default value, and
guard against a nil default so each resolved agent owns independent
state.
Signed-off-by: Émile Ré <emile@probo.com>
Remove sanitizeRequest and its call sites so the client no longer
strips sampling parameters based on the model registry. Unsupported
knobs are now avoided by setting the config carefully per model
rather than silently mutating outgoing requests.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker clears mapping_requested_at at claim time, so
a crash or hard failure between Process phases left the pattern dequeued,
unmapped, and with nothing to re-trigger it. Only an incidental sibling
remap could rescue it, so a lone pattern could stay stranded forever.
Implement the worker.StaleRecoverer interface, mirroring the enrichment
worker. ResetStaleMappings re-arms rows that were claimed but never
assigned a catalog row (common_tracker_pattern_id IS NULL) once idle past
a configurable window; a successful Process always assigns one via the
unmatched fallback, so the predicate cleanly detects interrupted runs and
self-heals after a single pass. ClearMappingRequestedAt now bumps
updated_at so the stale clock starts at claim time and the sweep never
recycles an in-flight claim.
Plumb a StaleAfter knob (default 600s) through the config struct, builder
env var, probod wiring, and Helm templates.
Signed-off-by: Émile Ré <emile@probo.com>
The identification agent attributed probo_distinct_id to Mixpanel
purely on the shared distinct_id token, and the enrichment agent
returned no description for the glob ph_phc_*_posthog because it
searched the literal "*" string and found nothing.
Tighten the identification prompt so attribution requires a perfect
pattern match or a meaningful prefix that belongs to the vendor; a
generic token behind a different prefix is not a match. Teach the
enrichment prompt to strip wildcard and variable parts before
searching, and to treat a vendor name embedded in the key as
corroboration so clearly-named trackers still get a description.
Signed-off-by: Émile Ré <emile@probo.com>
The deterministic tracker-mapping heuristics group patterns by shared
initiator domain, but tag managers, customer-data platforms, and
generic CDNs (Google Tag Manager, Segment, cloudfront.net, ...)
initiate trackers for many unrelated vendors. Grouping on such a
domain mis-attributes one vendor's tracker to another.
Add uri.FilterSharedInfrastructureDomains backed by a curated eTLD+1
denylist and apply it once in resolveDeterministic, so sibling
grouping, catalog domain matching, and the sibling re-enqueue cascade
all ignore shared-infrastructure hosts. Vendor-specific domains such
as google-analytics.com are intentionally kept as a same-vendor
signal. The agent path is unchanged: it still sees observed domains,
now with a prompt caveat about shared infrastructure.
Update the two sibling tests that used googletagmanager.com as the
initiator domain to a vendor domain, since that host is now stripped
before grouping.
Signed-off-by: Émile Ré <emile@probo.com>
Address review feedback on the agent-driven tracker catalog path:
- Return initiator-domain load failures instead of swallowing them,
so the worker retries rather than running the agent on partial
context.
- In the resolver, treat only ErrResourceNotFound as a catalog miss
and propagate genuine name/slug lookup errors.
- Insert the new vendor inside a savepoint and, on the slug
unique-violation race, reload and return the winning row instead of
aborting the caller's transaction.
- Stop seeding common_third_party_domains from observed initiator
domains. They are a co-occurrence signal, not verified ownership,
and writing them into the global cross-tenant catalog pollutes the
domain-based matcher. The curated seed owns that data.
- Warn the mapping agent that observed domains may belong to shared
CDNs, tag managers, or hosting infrastructure rather than the
vendor, so it does not attribute on that basis alone.
- Extract a shared tracker-identification prompt helper and move the
common-pattern identification prompt next to the enrichment agent.
Signed-off-by: Émile Ré <emile@probo.com>
resolveOrCreateCommonThirdParty lived as a package-level helper in the
tracker mapping worker, but the common pattern enrichment worker now
reuses it. Homing shared catalog logic in a mapping-named file made the
enrichment worker quietly depend on the mapping worker's file, and it is
not a mapping concern.
Move it to pkg/thirdparty as exported ResolveOrCreateCommonThirdParty,
decoupled from cookiebanner's TrackerMappingAgentResult (it now takes a
name and category) to avoid an import cycle. It stays a transaction-
scoped free function so both workers compose it into their own tx for
atomicity rather than receiving a service that owns its own connection.
Relocate the catalog dedup DB test alongside it.
Signed-off-by: Émile Ré <emile@probo.com>
The enrichment worker no longer invents a description when a tracker's
purpose cannot be substantiated; it records an empty description and
marks the row enriched so the stale-recovery loop does not retry it.
Vendor identification is the mapping pipeline's job, so the enricher
reuses the existing tracker-mapping agent to attribute a third party
for an unlinked common pattern before describing it. A confident
catalog match seeds the enrichment prompt and links the pattern, but
the enricher never creates or overrides an attribution.
When a blank, unlinked catalog row later gains a third party through
the mapping pipeline's upsert, enrichment is re-armed so the now-known
vendor gets a second, better-informed description attempt.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker mapping, common-pattern enrichment, and third-party
disambiguation agents default to a small max-tokens budget on the
premise that their final output is tiny structured JSON. On
reasoning models such as the GPT-5 family, reasoning tokens count
against max_tokens, so a small budget is consumed by reasoning and
the JSON is truncated, surfacing as "unexpected end of JSON input".
Raise the defaults to 4096 (1024 -> 4096 for tracker mapping, 512
-> 4096 for disambiguation) to leave headroom for the reasoning
phase. Update the bootstrap builder default, its test, and the
production values example to match.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker runs its LLM and web-search phases
between short transactions and holds no row lock across them. The
pattern-analysis worker can merge a pattern into a glob and delete
it in that window, so the final UpdateMapping then fails with
ErrResourceNotFound and the task errors out spuriously.
A vanished pattern has nothing left to map, so treat the concurrent
delete as a no-op: log it and return nil instead of failing.
Signed-off-by: Émile Ré <emile@probo.com>
The pattern-analysis worker created the merged glob blank and re-armed
mapping, discarding the org ThirdParty and description already resolved
on the exacts it absorbed. That forced a full re-map (LLM/web-search)
and opened a window where an in-flight exact could vanish mid-mapping.
Seed the glob from the merged exacts when they unanimously agree on a
single third party, carrying its description too, while still re-arming
mapping so the glob derives its own catalog row. With the third party
pre-set, the mapping worker skips the expensive org/disambiguation
resolution. Conflicting or unresolved groups stay blank as before.
The catalog link is deliberately not inherited: it is keyed on the
exact pattern string, not the glob template, so the mapping worker
resolves the right row itself.
Signed-off-by: Émile Ré <emile@probo.com>
The common-pattern enrichment and tracker-mapping agents run on
reasoning models such as gpt-5-nano, which reject an explicit
temperature and fail the whole request with a 400 ("Unsupported
value: 'temperature' does not support 0.1 with this model"). The
model registry already records this capability, but nothing
consulted it before dispatch, and dated provider snapshots like
gpt-5-nano-2025-08-07 did not resolve in the registry.
Resolve dated snapshots to their undated base model in registry
Lookup, and sanitize each chat completion request in the LLM
client by omitting the sampling knobs the target model does not
accept (temperature, top_p, frequency/presence penalties, stop).
Unknown models are left untouched, so models absent from the
registry keep their current behavior.
Signed-off-by: Émile Ré <emile@probo.com>
