Parallelize enrichment agents, calibrate prompts

The common-third-party enrichment pipeline ran Agent B (compliance
docs), Agent C (owned domains), and the deterministic logo step
sequentially even though, once Agent A resolves the website, the
three depend only on that website and not on each other. Fan them
out across goroutines under a WaitGroup so wall time is the slowest
of the three rather than their sum. Each step builds its own per-run
browser and writes only into its own locals; the shared LLM, HTTP,
and FileManager clients are safe for concurrent use and the database
is untouched until persist. Results merge in a fixed order so
runErrors and log output stay deterministic.

Also replace the single-sentence confidence guidance in the three
agent prompts with an explicit, calibrated rubric tied to evidence
strength, and remind the model that a downstream threshold gates
persistence so it should neither inflate nor deflate its estimates.

Signed-off-by: Émile Ré <emile@probo.com>

pkg/thirdparty/common_third_party_enrichment_worker.go

@@ -21,6 +21,7 @@ import (
 	"fmt"
 	"net/http"
 	"strings"
+	"sync"
 	"time"
 
 	"go.gearno.de/kit/httpclient"
@@ -215,11 +216,13 @@ func (h *enrichmentHandler) Claim(ctx context.Context) (coredata.CommonThirdPart
 }
 
 // Process runs the enrichment pipeline for one catalog row: Agent A
-// (company profile) first, then Agent B (compliance docs) and the
-// deterministic logo step, all outside any transaction. The merged
-// result and per-field provenance are persisted in a single final
-// transaction. Process always writes an enrichment payload, even on a
-// no-result run, so stale recovery does not re-queue the row.
+// (company profile) first, then Agent B (compliance docs), Agent C
+// (owned domains), and the deterministic logo step concurrently, all
+// outside any transaction. Agent A runs first because the others depend
+// on the website it resolves. The merged result and per-field provenance
+// are persisted in a single final transaction. Process always writes an
+// enrichment payload, even on a no-result run, so stale recovery does not
+// re-queue the row.
 func (h *enrichmentHandler) Process(ctx context.Context, party coredata.CommonThirdParty) error {
 	if h.cfg.LLMClient == nil {
 		return nil
@@ -275,11 +278,50 @@ func (h *enrichmentHandler) Process(ctx context.Context, party coredata.CommonTh
 
 	legalName := effectiveLegalName(party, company, h.cfg.ConfidenceThreshold)
 
+	// Agent B (compliance docs), Agent C (owned domains), and the
+	// deterministic logo step all depend only on the resolved website
+	// (Agent B also on the legal name) and are independent of each other.
+	// Run them concurrently so wall time is the slowest of the three
+	// rather than their sum. Each builds its own per-run browser and
+	// writes only into its own locals; the shared LLM/HTTP/FileManager
+	// clients are safe for concurrent use and the database is not touched
+	// until persist below. Results are merged after Wait in a fixed order
+	// to keep runErrors and log output deterministic.
+	var (
+		compliance    ComplianceDocsResult
+		complianceErr error
+
+		domainsResult DomainsResult
+		domainsErr    error
+
+		logoFile *coredata.File
+	)
+
+	var wg sync.WaitGroup
+
+	wg.Add(3)
+
+	go func() {
+		defer wg.Done()
+		compliance, complianceErr = h.runComplianceDocs(ctx, party.Name, website, legalName)
+	}()
+
+	go func() {
+		defer wg.Done()
+		domainsResult, domainsErr = h.runDomains(ctx, party.Name, website)
+	}()
+
+	go func() {
+		defer wg.Done()
+		logoFile = h.prepareLogo(ctx, party, website)
+	}()
+
+	wg.Wait()
+
 	// Agent B: compliance documents and trust pages.
-	compliance, err := h.runComplianceDocs(ctx, party.Name, website, legalName)
-	if err != nil {
-		h.logger.WarnCtx(ctx, "compliance docs agent failed", log.Error(err), log.String("common_third_party_id", party.ID.String()))
-		runErrors = append(runErrors, "compliance_docs: "+sanitizeAgentError(err))
+	if complianceErr != nil {
+		h.logger.WarnCtx(ctx, "compliance docs agent failed", log.Error(complianceErr), log.String("common_third_party_id", party.ID.String()))
+		runErrors = append(runErrors, "compliance_docs: "+sanitizeAgentError(complianceErr))
 	} else {
 		anySuccess = true
 	}
@@ -288,19 +330,14 @@ func (h *enrichmentHandler) Process(ctx context.Context, party coredata.CommonTh
 	// resolved website, so it runs only on this website-resolved path.
 	var owned []ownedDomain
 
-	domainsResult, err := h.runDomains(ctx, party.Name, website)
-	if err != nil {
-		h.logger.WarnCtx(ctx, "domains agent failed", log.Error(err), log.String("common_third_party_id", party.ID.String()))
-		runErrors = append(runErrors, "domains: "+sanitizeAgentError(err))
+	if domainsErr != nil {
+		h.logger.WarnCtx(ctx, "domains agent failed", log.Error(domainsErr), log.String("common_third_party_id", party.ID.String()))
+		runErrors = append(runErrors, "domains: "+sanitizeAgentError(domainsErr))
 	} else {
 		anySuccess = true
 		owned = resolveOwnedDomains(party.Name, website, domainsResult, defaultEnrichmentDomainConfidenceThreshold)
 	}
 
-	// Deterministic logo step (no LLM). Uploads to S3 outside the final
-	// transaction; the File row is inserted below.
-	logoFile := h.prepareLogo(ctx, party, website)
-
 	for _, field := range scalarFields(company, compliance) {
 		applyScalarField(&party, meta, prior, field, h.cfg.ConfidenceThreshold, now)
 	}

pkg/thirdparty/prompts/common_third_party_company_profile.txt.tmpl

@@ -22,7 +22,13 @@ Each field carries a value, a 0.0-1.0 confidence, and the source_url where you v
 
 4. Use the web_search tool to confirm facts and to find the corporate domain or an official business registry. Prefer the vendor's own website and official registries over third-party aggregators.
 
-5. Never guess. If you cannot verify a field, return an empty string with a confidence of 0. confidence is your own 0.0-1.0 estimate that the value is correct; reserve values above 0.8 for facts you verified on the vendor's own site or an official registry.
+5. Never guess. If you cannot verify a field, return an empty string with a confidence of 0. confidence is your own calibrated 0.0-1.0 estimate that the value is correct — it measures how sure you are, not how hard you looked. Aim for it to be well-calibrated: across many vendors, the facts you tag around 0.9 should turn out correct roughly nine times in ten. Anchor your estimate on the strength of the evidence:
+   - 0.9-1.0: verified on the vendor's own site (footer, imprint/impressum, about, legal) or an official business registry, with no conflicting evidence.
+   - 0.7-0.9: found on the vendor's own site but with minor ambiguity (for example a parent or brand name you could not fully disambiguate), or the same value corroborated across two independent reputable sources.
+   - 0.4-0.7: drawn from a single third-party aggregator, or an inference you could not confirm against an authoritative source.
+   - 0.1-0.4: a weak or partial signal you are mostly guessing from.
+   - 0: not found, or you cannot verify it at all.
+   A downstream step only persists a field when its confidence clears a threshold, so calibrate honestly: do not inflate a value to push it over the bar, and do not deflate a fact you genuinely verified.
 
 6. source_url is the page where you verified the value. Leave it empty when the value was not found.
 

pkg/thirdparty/prompts/common_third_party_compliance_docs.txt.tmpl

@@ -32,7 +32,13 @@ Each field carries a value, a 0.0-1.0 confidence, and the source_url where you f
 
 5. Never guess or fabricate a URL. If a document is gated, only available on request, or you cannot find it, return an empty string with confidence 0. Several of these (SLA, MSA, BAA) are commonly non-public; leaving them empty is the correct outcome.
 
-6. confidence is your own 0.0-1.0 estimate that the URL is correct and current. Reserve values above 0.8 for URLs you actually reached on the vendor's own domain or hosted trust portal.
+6. confidence is your own calibrated 0.0-1.0 estimate that the URL is correct and current — it measures how sure you are, not how hard you looked. Aim for it to be well-calibrated: across many vendors, the URLs you tag around 0.9 should turn out correct roughly nine times in ten. Anchor your estimate on the strength of the evidence:
+   - 0.9-1.0: a URL you actually reached and that loaded the expected document on the vendor's own domain or hosted trust portal.
+   - 0.7-0.9: a URL you found linked from the vendor's own site or trust portal but did not fully open or verify, or one with minor ambiguity about whether it is the current canonical version.
+   - 0.4-0.7: a URL from a single third-party source or search result you could not confirm on the vendor's own domain.
+   - 0.1-0.4: a weak guess, for example a path you assume exists by convention but never reached.
+   - 0: not found, gated, or you cannot verify it at all.
+   A downstream step only persists a field when its confidence clears a threshold, so calibrate honestly: do not inflate a value to push it over the bar, and do not deflate a URL you genuinely reached.
 
 7. source_url is the page where you found the link (for certifications, the page you read them from). Leave it empty when nothing was found.
 

pkg/thirdparty/prompts/common_third_party_domains.txt.tmpl

@@ -29,7 +29,13 @@ Each entry carries the domain, a 0.0-1.0 confidence that the vendor owns it, and
 
 4. Return registrable domains or full hosts; do not invent subdomains you have not seen. Never guess ownership: if you cannot confirm the vendor owns a domain, leave it out or give it a confidence of 0.
 
-5. confidence is your own 0.0-1.0 estimate that the vendor owns the domain. Reserve values above 0.85 for domains whose ownership you confirmed from the vendor's own site, its documentation, or an authoritative source.
+5. confidence is your own calibrated 0.0-1.0 estimate that the vendor owns the domain — it measures how sure you are about ownership, not how hard you looked. Aim for it to be well-calibrated: across many domains, the ones you tag around 0.9 should genuinely be vendor-owned roughly nine times in ten. Anchor your estimate on the strength of the ownership evidence:
+   - 0.9-1.0: ownership confirmed from the vendor's own site, its documentation, sub-processors/trust pages, or another authoritative source that names the domain as theirs.
+   - 0.7-0.9: strong but indirect evidence, for example a domain consistently serving the vendor's assets or linked as a product/brand from its own site, without an explicit ownership statement.
+   - 0.4-0.7: a plausible association (similar branding or naming) you could not confirm the vendor controls.
+   - 0.1-0.4: a weak guess.
+   - 0: ownership not confirmed, or the domain is shared/third-party infrastructure.
+   A downstream step only keeps a domain when its confidence clears a threshold, so calibrate honestly: do not inflate a value to push it over the bar, and do not deflate a domain whose ownership you genuinely confirmed.
 
 6. source_url is the page where you confirmed ownership. Leave it empty when nothing was found.