Operators could see that a rate-limit cooldown was active, but not
when it ends, and failure logs omitted most of the CA problem
document. Add a until-timestamp gauge and log the full acme.Error
surface so cooldowns and ACME responses are diagnosable.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Several race and validity gaps could leave certificate provisioning
stuck, unusable, or noisy:
- Accept the HTTP-01 challenge only after the key authorization is
committed, so the CA cannot hit the token before this instance can
serve it and invalidate the order.
- Persist challenge metadata under a blocking write-back lock; a row
merely locked by a competing transaction no longer silently drops the
accepted order.
- Abandon a recovered VALID order and restart instead of issuing it
with a freshly generated key that cannot match the existing cert.
- Exclude rate-limited rows from the ten-minute stale reset so the
resumable order survives the ACME cooldown.
- Size the provisioning poll lease to exceed the max processing window
so a released claim lock cannot let another worker process the same
row concurrently.
- Parse Retry-After as unsigned seconds and clamp overflow so malformed
values fall back to the default cooldown instead of disabling it.
- Normalize the acme_errors problem_type label to the RFC 8555 set to
bound Prometheus cardinality.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Address several provisioning defects that either stalled the retry
budget or crashed the process:
- Classify CAA resolver/transport failures apart from a real CAA policy
denial. Both shared the "caa records" wording, so a transient resolver
error was persisted as customer misconfiguration and retried forever
without consuming the retry budget. A new ErrCAANotPermitted sentinel
now marks the genuine misconfiguration; other CAA errors are treated
as ordinary transient failures.
- Honor an explicit Retry-After: 0 (or a past date) as permission for an
immediate retry instead of promoting it to the one-hour default
cooldown. acme.RateLimit collapses zero, invalid, and absent headers
to a zero duration, so the header is now parsed directly to tell an
explicit zero apart from a missing one.
- Reuse already-registered Prometheus collectors when a second
ACMEService shares a registerer. The fixed-name collectors were
MustRegistered, so a duplicate registration panicked the process.
- Persist provisioning failures on a context detached from the process
tick deadline. A timed-out attempt reached persistFailure with an
expired context, so the write-back failed and the retry budget never
advanced, leaving the certificate indefinitely retriable.
- Use pgx.StrictNamedArgs in the certificate FOR UPDATE loaders to match
the coredata SQL contract.
Signed-off-by: Bryan Frimin <bryan@probo.com>
Rate limits no longer inflate ssl_retry_count into an instant FAILED
path. Centralize outcomes in decideProvisioningOutcome, keep ACME
order state on transient and rate-limit errors, bound each Process
tick with a timeout, and block on FOR UPDATE when persisting a
freshly issued certificate.
Signed-off-by: Ludovic Vielle <ludovic@probo.com>
Signed-off-by: Bryan Frimin <bryan@probo.com>
