Unify enrichment tracking and outcome-based status

Make common_tracker_patterns and common_third_parties share one
enrichment-tracking model and fix the misleading proboctl status.

Both tables now carry the enrichment JSONB provenance payload, an
enrichment_attempts counter, and a last_enrichment_attempt_at clock.
On common_tracker_patterns the enriched_at done-flag is renamed to
last_enrichment_attempt_at and stamped at claim time, so it is truthful
to "attempt" rather than "success". A row is considered to have been
through the workflow when it carries an enrichment payload, not when a
timestamp is set, which lets stale recovery key off the payload being
absent with budget remaining, exactly like common_third_parties.

The claim path reads the attempt counter and timestamp back via
RETURNING so the in-memory receiver matches the database clock instead
of a separate app-side time.Now.

The enricher builds a per-field provenance payload (description and
third-party outcomes plus the mapping attribution) and persists it via
UpdateEnrichment, named to mirror the common-third-party sibling. The
common pattern enrichment worker gains a max-attempts ceiling so a
permanently failing row stops looping.

proboctl now shows "enriched" only when every field the last run
recorded an outcome for resolved a value, otherwise "partial (X/Y)",
replacing the misleading "enriched (no description)" label.

Signed-off-by: Émile Ré <emile@probo.com>
This commit is contained in:
Émile Ré
2026-06-15 16:25:53 +02:00
parent a6ed64caaf
commit ef6cead482
17 changed files with 705 additions and 155 deletions

View File

@@ -16,6 +16,7 @@ package commontrackerpattern
import (
"context"
"encoding/json"
"errors"
"fmt"
@@ -47,20 +48,62 @@ func NewCmdCommonTrackerPattern(f *cmdutil.Factory) *cobra.Command {
}
// enrichmentState classifies a pattern's position in the enrichment
// lifecycle for display.
// lifecycle for display. A row that has been through the workflow (it
// carries an enrichment payload) reads "enriched" only when every field
// the last run recorded an outcome for resolved a value; otherwise it
// reads "partial (X/Y)".
func enrichmentState(p *coredata.CommonTrackerPattern) string {
switch {
case p.EnrichmentRequestedAt != nil:
return "queued"
case p.EnrichedAt != nil && p.Description == "":
return "enriched (no description)"
case p.EnrichedAt != nil:
return "enriched"
case len(p.Enrichment) > 0:
resolved, total := enrichmentCompleteness(p)
if total == 0 || resolved == total {
return "enriched"
}
return fmt.Sprintf("partial (%d/%d)", resolved, total)
default:
return "unenriched"
}
}
// resolvedFieldStatuses are the per-field enrichment statuses that carry a
// value, as opposed to not_found.
var resolvedFieldStatuses = map[string]struct{}{
"found": {},
"exists_external": {},
}
// enrichmentCompleteness counts how many of the fields the last enrichment
// run recorded an outcome for resolved a value (X) versus the total it
// recorded (Y), parsed from the enrichment payload's per-field provenance.
func enrichmentCompleteness(p *coredata.CommonTrackerPattern) (resolved, total int) {
if len(p.Enrichment) == 0 {
return 0, 0
}
var meta struct {
Fields map[string]struct {
Status string `json:"status"`
} `json:"fields"`
}
if err := json.Unmarshal(p.Enrichment, &meta); err != nil {
return 0, 0
}
for _, f := range meta.Fields {
total++
if _, ok := resolvedFieldStatuses[f.Status]; ok {
resolved++
}
}
return resolved, total
}
// resolveCommonThirdPartyID accepts either a common third party GID or a
// slug and returns the corresponding id.
func resolveCommonThirdPartyID(ctx context.Context, conn pg.Querier, value string) (gid.GID, error) {