Replace the single-case switch in the agent-run handler with an if,
and add the blank lines wsl_v5 requires around statements in the
approval checkpoint, worker tests, and policy-set registration.
Signed-off-by: Bryan Frimin <bryan@probo.com>
The agent-run actions and policies lived in the core probo policy set,
which forced every authorization change for the agent-run domain to
touch unrelated core files. Move the actions and the OWNER/ADMIN and
VIEWER/AUDITOR policies into the agentrun package and have it expose a
PolicySet that probod registers into the authorizer at composition
time, so the rules live alongside the domain logic they govern.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Move agent-run orchestration from the legacy supervisor path into the new
agentrun worker/service package and wire it through coredata, server,
policies, and GraphQL resolvers.
This consolidates run lifecycle handling around lease-aware workers and
aligns API surface with the new agent-run domain model so reviewers can
follow one coherent execution path.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Replace the hand-rolled credential chain (static, ECS endpoint, EC2
role) with config.LoadDefaultConfig, which handles the full standard
AWS credential resolution chain automatically.
Co-authored-by: Neil McGibbon <code@neilmcgibbon.com>
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Google Workspace SMTP relay (smtp-relay.gmail.com) rejects generic
EHLO identifiers such as "localhost", which is Go net/smtp's default.
Operators can now set SMTP_HELLO_NAME to a valid hostname so that
c.Hello() is called before StartTLS, satisfying strict relay policies.
Fixes https://github.com/getprobo/probo/issues/1284
Signed-off-by: Bryan Frimin <bryan@probo.com>
Public-client (CIMD) providers need no operator credentials: derive
their client_id from the deployment base URL and their state-signing key
from the active OAuth2 server signing key, registering them at startup
unless the operator configured them explicitly.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
CommonThirdParty.logoUrl and TrustCenterReference.logoUrl were
returning expiring S3 presigned URLs, which break if cached or
shared past their TTL.
Replace with stable /api/files/v1/{id} application URLs.
file.Service now generates these via baseurl; a new filesign
package owns presigning for the files/v1 HTTP handler that
does the internal redirect.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Add a poll-based tracker policy worker that claims banners flagged for
policy generation, clears the flag, and rebuilds the cookie and
tracking technologies policy from the latest published snapshot. A
banner that lost its published version is skipped rather than failed.
Wire the worker into probod with crash propagation and graceful
shutdown alongside the other tracker workers.
Signed-off-by: Émile Ré <emile@probo.com>
Queue vetting on third_parties with PENDING, PROCESSING,
COMPLETED, and FAILED states. Expose enqueue and status through
GraphQL, MCP, CLI, and n8n, validate vet requests, tune the
worker via config, and poll the detail page while vetting runs.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
ResolveAgent aliased the default config's Temperature and MaxTokens
pointers into every agent that left them unset, so all resolved
agents shared one backing value. A mutation through any of those
pointers would corrupt the default and every other agent. It also
dereferenced the default unconditionally even though it can be nil.
Allocate a fresh pointer holding a copy of the default value, and
guard against a nil default so each resolved agent owns independent
state.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping worker clears mapping_requested_at at claim time, so
a crash or hard failure between Process phases left the pattern dequeued,
unmapped, and with nothing to re-trigger it. Only an incidental sibling
remap could rescue it, so a lone pattern could stay stranded forever.
Implement the worker.StaleRecoverer interface, mirroring the enrichment
worker. ResetStaleMappings re-arms rows that were claimed but never
assigned a catalog row (common_tracker_pattern_id IS NULL) once idle past
a configurable window; a successful Process always assigns one via the
unmatched fallback, so the predicate cleanly detects interrupted runs and
self-heals after a single pass. ClearMappingRequestedAt now bumps
updated_at so the stale clock starts at claim time and the sweep never
recycles an in-flight claim.
Plumb a StaleAfter knob (default 600s) through the config struct, builder
env var, probod wiring, and Helm templates.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.
Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.
Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.
Signed-off-by: Émile Ré <emile@probo.com>
Tracker descriptions were only filled on the agent-identification path,
so patterns resolved by domain, sibling, or fallback stayed without one,
and empty mapping upserts could clobber a researched description on the
shared catalog row.
Move description ownership to a dedicated, global common-pattern
enrichment worker. New catalog rows are queued on insert; the worker
researches a compliance-grade description with web search, records it on
the common pattern, and fans it out to every linked tracker pattern. The
mapping worker no longer generates descriptions and only propagates an
already-enriched one at link time.
Rename TrackerMappingConfig to TrackerAgentsConfig since the mapping and
enrichment agents now share it.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
Three follow-ups from review of the registry consolidation:
- Build Vercel's authorization URL with net/url instead of a
hand-rolled "{integration_slug}" placeholder resolved by
strings.ReplaceAll. The slug is escaped via url.PathEscape in a
per-provider Registration.BuildAuthURL closure, and the unused
AuthURLParams plumbing on Registration and OAuth2Connector is
removed (OAuth2Connector now carries a typed IntegrationSlug).
- Drop the SettingsInput union type and the per-provider
MarshalSettings closures. The create resolvers now build the typed
coredata.*ConnectorSettings directly from the gqlgen input, the
same way the OAuth callback path already does, so there is no
shared catch-all DTO and no stringly-typed boundary.
- Restore ConnectorProviders() to a plain ordered slice literal; the
intermediate map + slices.Sort added nondeterminism and a sort for
no benefit.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
The console previously dispatched per-provider logic through a fan of
init()-side-effect maps (driver names, OAuth2 metadata, probe URLs,
display names, settings switches), spread across pkg/connector,
pkg/accessreview/drivers and the console v1 resolvers. Adding a new
provider required edits in every one of those places and a corresponding
switch arm in CreateConnectorRequest. The same per-provider knowledge
also leaked into Helm templates as hand-rolled environment-variable
blocks per connector.
This commit collapses the dispatch surface into a single typed
*provider.Registry. The registry is constructed once by
NewBuiltinRegistry at probod startup and threaded as an explicit
dependency into every consumer (accessreview service, console v1
resolver, OAuth2 wiring). There is no package-level state. Each
provider lives in one file under pkg/connector/provider/ that exposes
a private xxxRegistration() *Registration constructor; NewBuiltinRegistry
enumerates them.
CreateConnectorRequest loses its per-provider settings fields and
takes a single RawSettings json.RawMessage produced by the
per-provider MarshalSettings closure. The 1Password SCIM bridge URL
is validated at create time (http(s) scheme + non-empty host) so a
malformed value fails fast at the resolver boundary. The Helm chart
gains probo.connectorEnv and probo.connectorSecretEntries templates
so adding a connector requires zero Helm changes. Access-review name
resolution moves into the same Registration value to keep one
authoritative dispatch table.
Tests cover every Registration (DisplayName, NewDriver wired),
Register error paths (nil, empty Provider, empty DisplayName,
duplicate), All / ProviderDisplayName / ProviderOAuth2Scopes /
ProbeURL hit and miss paths, the ApplyOAuth2Defaults templating and
PKCE branches, and ConnectorSettings[T] round-trip plus malformed-JSON
error path. The pre-refactor ApplyProviderDefaults test in
pkg/connector is replaced by the equivalent in
pkg/connector/provider.
Signed-off-by: Aurélien Sibiril <81782+aureliensibiril@users.noreply.github.com>
Apply five style rules: convert iota string enums to typed
string constants, replace errors.As with errors.AsType,
merge three-group imports into two groups, fix multiline
parameter/argument formatting, and replace fmt.Sprintf URL
construction with net/url.
Signed-off-by: Émile Ré <emile@probo.com>
Introduce a hierarchical risk assessment model with six entity types:
- Risk Assessment: top-level container scoped to an organization
- Risk Assessment Scope: sub-container for scoping threat modeling
exercises within an assessment
- Risk Assessment Node: DFD elements typed as ENTITY, BOUNDARY,
ASSET, or DATA within a scope
- Risk Assessment Process: directed data flows between two nodes
- Risk Assessment Threat: descriptive threats attached to a process
with a free-text category (e.g. Confidentiality, Integrity)
- Risk Scenario: thin join linking a threat to a risk from the
register, carrying only a name and description
Risk scoring (likelihood, impact, treatment) remains on the existing
Risk entity. Threats are purely descriptive. Risk Scenarios connect
the threat model to the risk register without duplicating scores.
Backend: migration with PG enum for node types, coredata structs,
service layer with full CRUD and validation, GraphQL schema with
18 mutations and paginated connections, authorization actions and
policies, and base_resolvers.go Node dispatch for all entity types.
Frontend: Risk Assessments list page with create dialog, detail page
showing scopes as cards with nodes/processes/threats tables, inline
create/edit/delete actions on all entities, and a Scenarios tab on
the Risk detail page linking threats to risks. Existing RiskGraph.ts
hook file removed in favor of colocated queries in page files.
E2E tests cover CRUD for all entity types, RBAC, and tenant
isolation.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Firecrawl is a tool used by agents (tracker mapping, third-party
assessor), so its configuration belongs under AgentsConfig rather than
as a standalone Config field. Adds AgentToolsConfig to hold agent tool
credentials and updates all config propagation consumers.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl has a single public API at https://api.firecrawl.dev/v2.
The endpoint was configurable but never varied across environments,
so hardcode it as a package-level const and remove the Endpoint
field from FirecrawlConfig and all downstream wiring (bootstrap,
Helm chart, probod, vetting, cookiebanner).
Signed-off-by: Émile Ré <emile@probo.com>
SearXNG was a fallback search backend that added complexity without
being used in practice. All search-dependent features (web search,
government DB checks, vetting orchestrator, tracker mapping) now use
Firecrawl exclusively. Removes the SEARCH_ENDPOINT config plumbing
from probodconfig, bootstrap, Helm charts, and all callers.
Signed-off-by: Émile Ré <emile@probo.com>
Group firecrawl-endpoint and firecrawl-api-key under a nested firecrawl
config key. Add env var mappings (FIRECRAWL_ENDPOINT, FIRECRAWL_API_KEY,
SEARCH_ENDPOINT, AGENT_TRACKER_MAPPING_*) to the bootstrap builder with
test coverage. Wire the new values through the Helm chart (values,
deployment, secret, production example).
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl provides higher quality search results than SearXNG.
When configured (firecrawl-endpoint + firecrawl-api-key), the
tracker-mapping agent and search toolset prefer it over the
SearXNG backend. Also improves the tracker identification prompt
with multi-strategy search queries that leverage domain signals
and adapt to tracker type.
Signed-off-by: Émile Ré <emile@probo.com>
When both pattern matching and domain matching fail to identify a
tracker, an opt-in LLM agent can now attempt identification using
internal database searches and optional web search. The agent returns
structured output (third party name, category, description, confidence)
and the worker auto-creates CommonThirdParty records when needed.
The feature is gated behind the `llm.tracker-mapping.provider` config
field; when unset the worker behaves exactly as before.
Signed-off-by: Émile Ré <emile@probo.com>
Move tracker_mapping_worker.go from pkg/probo to pkg/cookiebanner and
rename worker.go to pattern_analysis_worker.go to reflect the worker
name.
Signed-off-by: Émile Ré <emile@probo.com>
Poll-based worker that maps org-scoped tracker patterns to the
common knowledge base via pattern matching and domain-based
attribution. Populates initiator_domain on detected trackers
at report time. Resolves org-scoped vendors through the common
third party link.
Signed-off-by: Émile Ré <emile@getprobo.com>
Signed-off-by: Émile Ré <emile@probo.com>
Renames the user-facing 'vendor' concept to 'third party' across the
entire codebase. The shared common_third_parties reference table is
unchanged.
Migration. Renames the vendor_category enum, the vendors and
vendor_<entity> tables (contacts, services, compliance_reports,
business_associate_agreements, data_privacy_agreements,
risk_assessments) and their vendor_id columns, the asset_vendors /
data_vendors / processing_activity_vendors junction tables,
generated_documents.vendors_document_id, the webhook_event_type
'vendor:<verb>' values, and the snapshots_type 'VENDORS' value.
Backend. Renames coredata models and SQL queries, probo services,
GraphQL / MCP API surface, console / trust / webhook resolvers and
types, the CLI (prb vendor* -> prb third-party*; pkg/cmd/vendormgmt
-> pkg/cmd/thirdpartymgmt), the document generator, vetting agent
prompts, and the common-third-parties-import command.
Frontend, packages, n8n, e2e. Renames apps/console pages, components,
hooks, routes, dialogs, and tabs; the shared @probo/vendors package
(now @probo/third-parties); the @probo/ui Vendors atoms (now
ThirdParties, VendorLogo -> ThirdPartyLogo); the n8n community node
actions/vendor folder (now actions/thirdParty); and the e2e Go test
suite (console and MCP). Filesystem and URL paths use kebab-case
(third-parties), GraphQL fields and TypeScript identifiers use
camelCase (thirdParty / thirdParties), Go types use PascalCase
(ThirdParty), and human-facing text uses 'third party' with a space.
Co-authored-by: Bryan Frimin <bryan@getprobo.com>
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Fetch favicons at import time instead of calling Google's favicon
service per page load. Logos are stored as public files in S3 and
served through the existing /api/files/v1/{id} endpoint.
Signed-off-by: Émile Ré <emile@getprobo.com>
The CreateVendorDialog previously loaded the entire @probo/vendors
JSON bundle client-side and used MiniSearch for fuzzy search. This
replaces it with a GraphQL query against the common_third_parties
database table, searched server-side via ILIKE filtering.
Backend: adds CommonThirdParty GraphQL type, a pkg/thirdparty
service, and a commonThirdParties(name) root query. Frontend:
splits into CommonThirdPartyCombobox (display) and an @inline
fragment read on selection via readInlineData.
Signed-off-by: Émile Ré <emile@getprobo.com>
Resolve the visitor's IP to a country code via the geoloc service and
map it to the applicable privacy regulation (GDPR, UK GDPR, FADP, CCPA,
PIPEDA, LGPD, LFPDPPP, POPIA, PDPA, PIPL, PIPA, APPI, DPDP, PDPL).
The regulation and its implied consent mode (OPT_IN / OPT_OUT) are
injected into the GET /config response so the SDK can adapt its behavior.
Also makes geoloc.Service self-contained: LookupCountry and IsPopulated
now manage their own DB connections instead of requiring callers to pass
a pg.Querier.
Signed-off-by: Émile Ré <emile@getprobo.com>
Introduce a geoloc package that stores CIDR-to-country mappings in
PostgreSQL using the native cidr type with a GiST index for fast
containment lookups. Data comes from the ipverse/country-ip-blocks
dataset added as a git submodule.
A standalone geoloc-import command reads the TXT files from disk
and bulk-loads them via COPY. probod wires the service and logs a
warning when the table is empty.
Signed-off-by: Émile Ré <emile@getprobo.com>
The worker now operates on TrackerPattern/DetectedTrackers
instead of CookiePattern/Cookies, with TrackerType included
in merge group keys to prevent cross-type merging.
Signed-off-by: Émile Ré <emile@getprobo.com>
The HTTP middleware and proxy-protocol listeners both pinned trust
to exact IPs, which forced re-applying terraform every time AWS
rotated an ALB or NLB ENI. Trusted-proxies entries now accept CIDR
ranges in addition to plain IPs, so callers can trust whole subnets
(where the load balancer ENIs always live) and stop chasing
rotating IPs.
The HTTP middleware splits parsed entries into IPs and IPNets and
checks both. The proxy-protocol listeners switch from
TrustProxyHeaderFrom (IP-only, REJECT) to ConnStrictWhiteListPolicy
(IP or CIDR, REJECT) which preserves the existing reject-on-unknown
semantics.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Background worker polls cookie_banners with pattern_analysis_requested_at
set, groups EXACT patterns sharing a common prefix, and merges groups of
3+ into a PREFIX pattern. Detection sets the flag when new EXACT patterns
are created. The worker relinks cookies, removes orphaned patterns, and
updates the draft version via ensureDraftVersionForBanner.
Signed-off-by: Émile Ré <emile@getprobo.com>
probod-bootstrap only needs the config struct definitions for
YAML marshaling but transitively pulled in ~40 heavy runtime
dependencies via pkg/probod. Move all config types and their
methods to a new pkg/probodconfig package and re-export them
from pkg/probod via type aliases for backward compatibility.
Signed-off-by: Émile Ré <emile@getprobo.com>
Move PDF generation from synchronous publish flow to a background polling
job. Published versions with file_id IS NULL are picked up by the job,
which generates the PDF, uploads to S3, and links the file. Export PDF
now serves stored files for published versions (with optional signature
page and watermark) and generates on the fly for drafts.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Introduces a BRANDING boolean config (default true) propagated through
the standard config pipeline. Cookie banners now initialize their
show_branding column from this config instead of hardcoding true.
Signed-off-by: Émile Ré <emile@getprobo.com>
The OAuth2/OIDC server accepted its signing key via a file path
(key-file), while every other PEM key in the probod config (SAML
private key, ACME account key) is embedded inline. Switch the
field to a private-key string so the convention is uniform.
The signing key is operator-supplied material that must outlive
any process restart, so the bootstrap builder now treats
OAUTH2_SERVER_SIGNING_KEY as required and refuses to start
without one; silently minting a fresh key per boot would break
token validation across rollouts. The OAUTH2_SERVER_* env vars
otherwise flow through builder.Build like the existing SAML
block so the new OAuth2Server section is populated end-to-end.
Rework the e2e harness to render its config via bootstrap at
test setup, which removes the static
e2e/console/testdata/config.yaml and the previously generated
test-only PEM file. A per-run RSA key is minted via
bootstrap.GenerateOAuth2SigningKey (kept public for test
tooling) and injected through the builder env map. CI now
passes ACME_ROOT_CA inline instead of mutating a YAML on disk.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Implement a full OAuth2 2.0 and OpenID Connect 1.0 authorization
server with support for authorization code flow (with PKCE),
refresh token rotation, device authorization grant, dynamic
client registration, token introspection, and token revocation.
Includes database schema, coredata layer, service logic, HTTP
handlers, OIDC discovery endpoint, and JWKS publishing.
Signed-off-by: Bryan Frimin <bryan@getprobo.com>
Strip forwarded headers (Forwarded, X-Forwarded-For, X-Real-Ip)
from requests originating from untrusted proxies at the HTTP
server level, reusing the existing proxy-protocol trusted-proxies
config. The clientip package is now a pure extraction helper;
context plumbing and middleware wrappers are removed.
Signed-off-by: Émile Ré <emile@getprobo.com>
