rohit/probo
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refacto load all functions

Browse Source 
Unbounded LoadAll* loaders materialised an entire result set in one
query with no ceiling. A table that is small in development can grow
without bound in production, so these loaders were a latent memory
and query-time hazard.

Remove the LoadAll* methods from pkg/coredata and walk the cursor-
paginated LoadBy* siblings instead through a shared page.LoadAll
helper. The helper advances a MaxCursorSize forward cursor until the
result set is exhausted and concatenates the pages. It caps a single
call at MaxLoadAllPages (20) batches of 500 rows and errors past that
rather than materialising an unbounded set, so a runaway caller fails
loudly instead of exhausting memory.

Callers that genuinely need every row now express that explicitly,
and the coredata load-naming rule and docs are updated to discourage
new unbounded loaders.

Signed-off-by: Sacha Al Himdani <sacha@probo.com>
This commit is contained in:
Sacha Al Himdani
2026-06-09 19:33:36 +02:00
committed by Sacha Al Himdani
parent 853f2404a6
commit 9ab8ea2085
46 changed files with 1218 additions and 1674 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
pkg/iam/scim/service.go
View File

@@ -38,6 +38,7 @@ import (
"go.probo.inc/probo/pkg/crypto/rand"
"go.probo.inc/probo/pkg/gid"
"go.probo.inc/probo/pkg/mail"
"go.probo.inc/probo/pkg/page"
"go.probo.inc/probo/pkg/webhook"
webhooktypes "go.probo.inc/probo/pkg/webhook/types"
)
@@ -407,10 +408,27 @@ func (s *Service) ListUsers(
err = s.pg.WithConn(
ctx,
func(ctx context.Context, conn pg.Querier) error {
if err := profiles.LoadAllByOrganizationID(ctx, conn, scope, config.OrganizationID, filter); err != nil {
return fmt.Errorf("cannot load profiles: %w", err)
loaded, err := page.LoadAll(
ctx,
page.OrderBy[coredata.MembershipProfileOrderField]{
Field: coredata.MembershipProfileOrderFieldCreatedAt,
Direction: page.OrderDirectionAsc,
},
func(ctx context.Context, cursor *page.Cursor[coredata.MembershipProfileOrderField]) ([]*coredata.MembershipProfile, error) {
var batch coredata.MembershipProfiles
if err := batch.LoadByOrganizationID(ctx, conn, scope, config.OrganizationID, cursor, filter); err != nil {
return nil, fmt.Errorf("cannot load profiles: %w", err)
}
return batch, nil
},
)
if err != nil {
return err
}
profiles = loaded
return nil
},
)