Fix certmanager provisioning retry and metrics

Address several provisioning defects that either stalled the retry
budget or crashed the process:

- Classify CAA resolver/transport failures apart from a real CAA policy
  denial. Both shared the "caa records" wording, so a transient resolver
  error was persisted as customer misconfiguration and retried forever
  without consuming the retry budget. A new ErrCAANotPermitted sentinel
  now marks the genuine misconfiguration; other CAA errors are treated
  as ordinary transient failures.

- Honor an explicit Retry-After: 0 (or a past date) as permission for an
  immediate retry instead of promoting it to the one-hour default
  cooldown. acme.RateLimit collapses zero, invalid, and absent headers
  to a zero duration, so the header is now parsed directly to tell an
  explicit zero apart from a missing one.

- Reuse already-registered Prometheus collectors when a second
  ACMEService shares a registerer. The fixed-name collectors were
  MustRegistered, so a duplicate registration panicked the process.

- Persist provisioning failures on a context detached from the process
  tick deadline. A timed-out attempt reached persistFailure with an
  expired context, so the write-back failed and the retry budget never
  advanced, leaving the certificate indefinitely retriable.

- Use pgx.StrictNamedArgs in the certificate FOR UPDATE loaders to match
  the coredata SQL contract.

Signed-off-by: Bryan Frimin <bryan@probo.com>
This commit is contained in:
Bryan Frimin
2026-07-22 12:43:22 +02:00
parent 9724a2ce50
commit 121d4dcf93
7 changed files with 200 additions and 52 deletions

View File

@@ -34,6 +34,13 @@ const (
ProvisioningErrorACMEFailed = "ACME_FAILED"
)
// ErrCAANotPermitted marks a CAA policy that actively forbids issuance by our
// CA. This is a customer-side misconfiguration and is intentionally
// non-terminal. It must stay distinct from a CAA resolver/transport failure,
// which shares the "caa records" wording but is a transient error that has to
// consume the normal retry budget instead of retrying forever.
var ErrCAANotPermitted = errors.New("caa records do not permit issuance")
func classifyProvisioningError(err error) string {
if err == nil {
return ""
@@ -47,12 +54,14 @@ func classifyProvisioningError(err error) string {
return ProvisioningErrorACMEInvalidOrder
}
if errors.Is(err, ErrCAANotPermitted) {
return ProvisioningErrorDNSCAA
}
msg := strings.ToLower(err.Error())
switch {
case strings.Contains(msg, "cname"):
return ProvisioningErrorDNSCNAME
case strings.Contains(msg, "caa record"):
return ProvisioningErrorDNSCAA
case strings.Contains(msg, "status: invalid"), strings.Contains(msg, "order is in unexpected status \"invalid\""):
return ProvisioningErrorACMEInvalidOrder
default: