Each package that exercises the database against a real Postgres
carried its own copy of the connection bootstrap and schema setup.
Those copies had already drifted: some keyed off PROBO_TEST_PG_ADDR
with hardcoded defaults, others off PROBO_TEST_PG_URL, and the
agentrun/coredata suites hand-applied individual agent_runs
migrations to ensure the table existed.
Introduce a single test.PGClient helper that parses PROBO_TEST_PG_URL
(falling back to the local compose database), runs the full coredata
migration set once per process, and skips when no database is
reachable so make test stays a pure unit-test run. Migrate the
agentrun, coredata, cookiebanner, iam, and thirdparty suites onto it
and delete the duplicated helpers so the bootstrap can no longer
diverge.
Signed-off-by: Bryan Frimin <bryan@probo.com>
CommonThirdParty.logoUrl and TrustCenterReference.logoUrl were
returning expiring S3 presigned URLs, which break if cached or
shared past their TTL.
Replace with stable /api/files/v1/{id} application URLs.
file.Service now generates these via baseurl; a new filesign
package owns presigning for the files/v1 HTTP handler that
does the internal redirect.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Queue vetting on third_parties with PENDING, PROCESSING,
COMPLETED, and FAILED states. Expose enqueue and status through
GraphQL, MCP, CLI, and n8n, validate vet requests, tune the
worker via config, and poll the detail page while vetting runs.
Signed-off-by: Sacha Al Himdani <sacha@getprobo.com>
Address review feedback on the agent-driven tracker catalog path:
- Return initiator-domain load failures instead of swallowing them,
so the worker retries rather than running the agent on partial
context.
- In the resolver, treat only ErrResourceNotFound as a catalog miss
and propagate genuine name/slug lookup errors.
- Insert the new vendor inside a savepoint and, on the slug
unique-violation race, reload and return the winning row instead of
aborting the caller's transaction.
- Stop seeding common_third_party_domains from observed initiator
domains. They are a co-occurrence signal, not verified ownership,
and writing them into the global cross-tenant catalog pollutes the
domain-based matcher. The curated seed owns that data.
- Warn the mapping agent that observed domains may belong to shared
CDNs, tag managers, or hosting infrastructure rather than the
vendor, so it does not attribute on that basis alone.
- Extract a shared tracker-identification prompt helper and move the
common-pattern identification prompt next to the enrichment agent.
Signed-off-by: Émile Ré <emile@probo.com>
resolveOrCreateCommonThirdParty lived as a package-level helper in the
tracker mapping worker, but the common pattern enrichment worker now
reuses it. Homing shared catalog logic in a mapping-named file made the
enrichment worker quietly depend on the mapping worker's file, and it is
not a mapping concern.
Move it to pkg/thirdparty as exported ResolveOrCreateCommonThirdParty,
decoupled from cookiebanner's TrackerMappingAgentResult (it now takes a
name and category) to avoid an import cycle. It stays a transaction-
scoped free function so both workers compose it into their own tx for
atomicity rather than receiving a service that owns its own connection.
Relocate the catalog dedup DB test alongside it.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker mapping, common-pattern enrichment, and third-party
disambiguation agents default to a small max-tokens budget on the
premise that their final output is tiny structured JSON. On
reasoning models such as the GPT-5 family, reasoning tokens count
against max_tokens, so a small budget is consumed by reasoning and
the JSON is truncated, surfacing as "unexpected end of JSON input".
Raise the defaults to 4096 (1024 -> 4096 for tracker mapping, 512
-> 4096 for disambiguation) to leave headroom for the reasoning
phase. Update the bootstrap builder default, its test, and the
production values example to match.
Signed-off-by: Émile Ré <emile@probo.com>
The tracker-mapping and common-pattern-enrichment workers ran with the
kit/worker defaults (interval 10s, max-concurrency 5 each) and dropped
the resolved per-agent max-tokens/temperature, so up to ten LLM
pipelines could run unbounded on one OpenAI client. The mapping worker
also held a FOR UPDATE transaction across the LLM and Firecrawl calls
while its DB search tools acquired a second pooled connection, risking
pool exhaustion under concurrency.
Plumb max-tokens, temperature, agent timeout, and per-worker max-turns
through TrackerAgentsConfig and DisambiguationConfig into all three
agent builders, replacing the hard-coded constants with config-fed
fields and package fallbacks. Expose worker interval, concurrency,
stale-after, agent timeout, and max-turns as config (env, Helm values,
deployment template) mirroring the evidence-describer pattern, and
apply them at registration.
Refactor Process into deterministic-read, agent (no transaction), and
persist phases so neither the mapping agent nor disambiguation runs
inside an open transaction, removing the row locks held across network
latency and the nested-connection pressure.
Signed-off-by: Émile Ré <emile@probo.com>
When an org third party already has a common_third_party_id set,
LinkToCommon now skips the write instead of overwriting it with a
different catalog ID. This prevents heuristic or agent false
positives from corrupting a previous, more accurate association.
Signed-off-by: Émile Ré <emile@probo.com>
Third parties created by the tracker mapping worker are confirmed
active on the organization's cookie banner, making them first-level
by definition. Also remove unused ptr test helpers.
Signed-off-by: Émile Ré <emile@probo.com>
Manual moves of a non-extension TrackerPattern lacking a ThirdPartyID
now request mapping, which the tracker-mapping worker resolves with a
four-stage pipeline: exact common_third_party_id link, heuristic
ranking, agent disambiguation, and finally CreateFromCommon. Existing
fuzzy-matched org rows are tagged with common_third_party_id so the
next promotion takes the O(1) exact-link path.
The matching primitives live in pkg/thirdparty (RankCandidates,
LinkToCommon, CreateFromCommon, ScoredCandidate, threshold constants)
so the disambiguation agent and the heuristic share one candidate
type. Cookiebanner orchestrates them; cookie-banner-specific concerns
(pattern -> common-pattern -> common-party navigation, the EXTENSION
gate, and structured logs) stay in the worker.
Signed-off-by: Émile Ré <emile@probo.com>
Surface the third-party links carried by tracker patterns (org-scoped
ThirdParty via third_party_id, or global CommonThirdParty via
common_tracker_pattern_id) requires three new batch loaders and two
filter dimensions, all kept inside their owning entity tables to honour
the no-cross-entity-JOIN rule.
* CommonTrackerPatterns gains LoadByIDs and the ID-only
LoadIDsByCommonThirdPartyID helper, which lets callers translate a
common third party into a set of common_tracker_pattern_id values
without ever JOINing against tracker_patterns.
* CommonThirdParties gains LoadByIDs.
* TrackerPatterns gains LoadDistinctThirdPartyIDsByCookieBannerID and
LoadDistinctCommonTrackerPatternIDsByCookieBannerID, used by the
upcoming CookieBanner.linkedThirdParties resolver to enumerate the
third parties referenced in a banner.
* TrackerPatternFilter gains thirdPartyID and commonTrackerPatternIDs
filter dimensions; the GraphQL layer will dispatch a single
thirdPartyId argument to the right one based on the GID entity-type
prefix.
Service-layer wrappers (cookiebanner.GetCommonTrackerPatternsByIDs,
cookiebanner.LoadCommonTrackerPatternIDsByCommonThirdPartyID,
cookiebanner.LoadDistinctThirdPartyIDsByCookieBannerID,
cookiebanner.LoadDistinctCommonTrackerPatternIDsByCookieBannerID, and
thirdparty.GetCommonThirdPartiesByIDs) expose the new loaders to the
console resolvers and dataloaders that follow.
Signed-off-by: Émile Ré <emile@probo.com>
Fetch favicons at import time instead of calling Google's favicon
service per page load. Logos are stored as public files in S3 and
served through the existing /api/files/v1/{id} endpoint.
Signed-off-by: Émile Ré <emile@getprobo.com>
The CreateVendorDialog previously loaded the entire @probo/vendors
JSON bundle client-side and used MiniSearch for fuzzy search. This
replaces it with a GraphQL query against the common_third_parties
database table, searched server-side via ILIKE filtering.
Backend: adds CommonThirdParty GraphQL type, a pkg/thirdparty
service, and a commonThirdParties(name) root query. Frontend:
splits into CommonThirdPartyCombobox (display) and an @inline
fragment read on selection via readInlineData.
Signed-off-by: Émile Ré <emile@getprobo.com>
