Add AI-powered evidence description generation

Introduce a background worker that automatically generates
compliance-focused descriptions for uploaded evidence files
using configurable LLM providers. Descriptions are surfaced
across all interfaces: GraphQL API, MCP API, CLI, and the
console UI.

Key changes:
- Multi-provider LLM config with per-agent settings (pointer
  types for Temperature/MaxTokens to preserve zero values)
- Evidence description worker with bounded concurrency
- EvidenceDescriptionStatus typed enum with PostgreSQL enum type
- New `prb evidence` CLI commands (list, view, delete)
- Evidence description displayed in console table and preview
- Migration only marks evidences without files as completed

Signed-off-by: Bryan Frimin <bryan@getprobo.com>
This commit is contained in:
Bryan Frimin
2026-03-26 14:30:04 +01:00
parent 8bbed534a9
commit 619ec7b882
24 changed files with 1390 additions and 125 deletions

View File

@@ -0,0 +1,81 @@
// Copyright (c) 2025-2026 Probo Inc <hello@getprobo.com>.
//
// Permission to use, copy, modify, and/or distribute this software for any
// purpose with or without fee is hereby granted, provided that the above
// copyright notice and this permission notice appear in all copies.
//
// THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
// AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
// OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
// PERFORMANCE OF THIS SOFTWARE.
package evidencedescriber
import (
"context"
_ "embed"
"fmt"
"go.probo.inc/probo/pkg/agent"
"go.probo.inc/probo/pkg/llm"
)
//go:embed prompt.txt
var systemPrompt string
type (
Config struct {
Model string
Temp float64
MaxTokens int
}
Describer struct {
client *llm.Client
config Config
}
)
func New(client *llm.Client, cfg Config) *Describer {
return &Describer{
client: client,
config: cfg,
}
}
func (d *Describer) Describe(ctx context.Context, filename string, mimeType string, fileBase64 string) (*string, error) {
ag := agent.New(
"evidence_describer",
d.client,
agent.WithInstructions(systemPrompt),
agent.WithModel(d.config.Model),
agent.WithTemperature(d.config.Temp),
agent.WithMaxTokens(d.config.MaxTokens),
)
result, err := ag.Run(
ctx,
[]llm.Message{
{
Role: llm.RoleUser,
Parts: []llm.Part{
llm.TextPart{Text: fmt.Sprintf("Filename: %s", filename)},
llm.FilePart{
Data: fileBase64,
MimeType: mimeType,
Filename: filename,
},
},
},
},
)
if err != nil {
return nil, fmt.Errorf("cannot describe evidence: %w", err)
}
text := result.FinalMessage().Text()
return &text, nil
}

View File

@@ -0,0 +1,17 @@
You are an ISO/SOC auditor writing evidence descriptions for a compliance review.
Input: a single image of an evidence file (screenshot or document export).
Output: plain text, 1–2 sentences. No markdown, no line breaks, no labels, no preamble.
Include these elements if clearly present; omit any that are not:
— System: the tool or platform shown (e.g. GitHub, Google Workspace, AWS).
— Setting: the specific configuration, feature, or state demonstrated.
— Scope: who or what it applies to (e.g. organization-wide, all users, a specific repository).
Use the language of the document. Do not include greetings, caveats, file names, image quality comments, or phrases like "This screenshot shows." Never guess — if something is not clearly visible, leave it out.
If the image is unreadable or is not compliance evidence, respond with exactly: "Unable to describe: unreadable or not recognized as compliance evidence."
Example — good: "Google Workspace admin console showing enforced 2-step verification for all users in the organization, with no exceptions permitted."
Example — bad: "This shows Google security settings."