Replace the cross-entity JOIN in
DetectedTrackers.LoadCommonThirdPartyIDByDomainMatch with two
idiomatic coredata calls: LoadInitiatorDomainsByTrackerPatternID
on DetectedTrackers, then a new CommonThirdPartyDomains.Load with
a CommonThirdPartyDomainFilter. Each entity now queries only its
own table, and the caller orchestrates the lookup.
Document the Load vs LoadAll naming convention and the no
cross-entity JOINs rule in contrib/claude/coredata.md.
Signed-off-by: Émile Ré <emile@probo.com>
Upsert methods now RETURNING all struct columns and scan the result
back into the pointer receiver, keeping the caller in sync with the
actual DB state (id, created_at, etc. from the existing row on
conflict). Insert detection compares the saved original ID with the
returned ID instead of relying on the PostgreSQL-internal xmax column.
Signed-off-by: Émile Ré <emile@probo.com>
Generate the category list from coredata.ThirdPartyCategories() at
runtime instead of hardcoding it in the prompt text. Type the
TrackerIdentification.Category field as coredata.ThirdPartyCategory so
JSON unmarshaling validates values automatically.
Also documents the .txt.tmpl template file naming convention.
Signed-off-by: Émile Ré <emile@probo.com>
Write errors from Insert/Upsert were logged but swallowed,
letting the transaction commit with incomplete data. All
helper methods now return errors so the transaction rolls
back on failure.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl has a single public API at https://api.firecrawl.dev/v2.
The endpoint was configurable but never varied across environments,
so hardcode it as a package-level const and remove the Endpoint
field from FirecrawlConfig and all downstream wiring (bootstrap,
Helm chart, probod, vetting, cookiebanner).
Signed-off-by: Émile Ré <emile@probo.com>
SearXNG was a fallback search backend that added complexity without
being used in practice. All search-dependent features (web search,
government DB checks, vetting orchestrator, tracker mapping) now use
Firecrawl exclusively. Removes the SEARCH_ENDPOINT config plumbing
from probodconfig, bootstrap, Helm charts, and all callers.
Signed-off-by: Émile Ré <emile@probo.com>
Firecrawl provides higher quality search results than SearXNG.
When configured (firecrawl-endpoint + firecrawl-api-key), the
tracker-mapping agent and search toolset prefer it over the
SearXNG backend. Also improves the tracker identification prompt
with multi-strategy search queries that leverage domain signals
and adapt to tracker type.
Signed-off-by: Émile Ré <emile@probo.com>
When both pattern matching and domain matching fail to identify a
tracker, an opt-in LLM agent can now attempt identification using
internal database searches and optional web search. The agent returns
structured output (third party name, category, description, confidence)
and the worker auto-creates CommonThirdParty records when needed.
The feature is gated behind the `llm.tracker-mapping.provider` config
field; when unset the worker behaves exactly as before.
Signed-off-by: Émile Ré <emile@probo.com>
When neither pattern nor domain matching finds an existing common
tracker pattern, upsert one with no third party association and a
lower confidence (0.5) so every seen pattern is cataloged.
Also fix table name in LoadByOrganizationIDAndCommonThirdPartyID
(vendors -> third_parties).
Signed-off-by: Émile Ré <emile@probo.com>
Move tracker_mapping_worker.go from pkg/probo to pkg/cookiebanner and
rename worker.go to pattern_analysis_worker.go to reflect the worker
name.
Signed-off-by: Émile Ré <emile@probo.com>
