229c6b99c6
Introduce a poll-based worker that fills the global common_third_parties catalog (URLs, headquarter address, legal name, certifications, logo) so each tenant no longer starts from sparse, name-only rows. Enrichment is requested at row creation by ResolveOrCreateCommonThirdParty; curated seed rows are not enqueued, to avoid a re-seed storm. The pipeline uses two specialized agents plus a deterministic logo step. Agent A (company profile) resolves legal name, headquarter address, and the canonical website over web search; its website and legal name feed Agent B and the logo step. Agent B (compliance docs) resolves the legal document URLs, trust/security/status pages, and certifications using the browser read-only toolset (gated on ChromeDPAddr) plus web search. The logo step restores pkg/webinspect as a pure deterministic package and stores the discovered icon in S3, linked via logo_file_id. Each agent returns per-field value/confidence/source_url. The worker writes a column only when confidence clears a configurable threshold and the field is not externally owned (seed or human), and always records full per-field provenance in a new enrichment JSONB column so re-runs fill only gaps and human edits are never clobbered. New bookkeeping columns (enrichment_requested_at, enrichment, enrichment_attempts) back the claim queue and stale recovery; agents run outside transactions and results persist in one final transaction. The worker is opt-in: it no-ops unless its agent provider is configured. Signed-off-by: Émile Ré <emile@probo.com>
83 lines
3.5 KiB
Go
83 lines
3.5 KiB
Go
// Copyright (c) 2026 Probo Inc <hello@probo.com>.
|
|
//
|
|
// Permission to use, copy, modify, and/or distribute this software for any
|
|
// purpose with or without fee is hereby granted, provided that the above
|
|
// copyright notice and this permission notice appear in all copies.
|
|
//
|
|
// THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
|
|
// REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
|
|
// AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
|
|
// INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
|
|
// LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
|
|
// OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
|
|
// PERFORMANCE OF THIS SOFTWARE.
|
|
|
|
package probodconfig
|
|
|
|
type (
|
|
// FullConfig represents the complete configuration file structure.
|
|
// This is used by bootstrap to generate the YAML config file.
|
|
FullConfig struct {
|
|
Unit UnitConfig `json:"unit"`
|
|
Probod Config `json:"probod"`
|
|
}
|
|
|
|
// UnitConfig contains unit framework configuration.
|
|
UnitConfig struct {
|
|
Metrics MetricsConfig `json:"metrics"`
|
|
Tracing TracingConfig `json:"tracing"`
|
|
}
|
|
|
|
// MetricsConfig contains metrics server configuration.
|
|
MetricsConfig struct {
|
|
Addr string `json:"addr"`
|
|
}
|
|
|
|
// TracingConfig contains tracing configuration.
|
|
TracingConfig struct {
|
|
Addr string `json:"addr"`
|
|
MaxBatchSize int `json:"max-batch-size"`
|
|
BatchTimeout int `json:"batch-timeout"`
|
|
ExportTimeout int `json:"export-timeout"`
|
|
MaxQueueSize int `json:"max-queue-size"`
|
|
}
|
|
|
|
// ESignConfig contains electronic signature configuration.
|
|
ESignConfig struct {
|
|
TSAURL string `json:"tsa-url"`
|
|
}
|
|
|
|
// Config represents the probod application configuration.
|
|
Config struct {
|
|
BaseURL string `json:"base-url"`
|
|
EncryptionKey string `json:"encryption-key"`
|
|
Pg PgConfig `json:"pg"`
|
|
Api APIConfig `json:"api"`
|
|
Auth AuthConfig `json:"auth"`
|
|
TrustCenter TrustCenterConfig `json:"trust-center"`
|
|
AWS AWSConfig `json:"aws"`
|
|
Notifications NotificationsConfig `json:"notifications"`
|
|
Connectors []ConnectorConfig `json:"connectors"`
|
|
Agents AgentsConfig `json:"llm"`
|
|
EvidenceDescriber EvidenceDescriberConfig `json:"evidence-describer"`
|
|
ThirdPartyVetting ThirdPartyVettingWorkerConfig `json:"third-party-vetting-worker"`
|
|
|
|
TrackerMappingWorker TrackerMappingWorkerConfig `json:"tracker-mapping-worker"`
|
|
CommonPatternEnrichmentWorker CommonPatternEnrichmentWorkerConfig `json:"common-pattern-enrichment-worker"`
|
|
CommonThirdPartyEnrichmentWorker CommonThirdPartyEnrichmentWorkerConfig `json:"common-third-party-enrichment-worker"`
|
|
|
|
ChromeDPAddr string `json:"chrome-dp-addr"`
|
|
CustomDomains CustomDomainsConfig `json:"custom-domains"`
|
|
SCIMBridge SCIMBridgeConfig `json:"scim-bridge"`
|
|
ESign ESignConfig `json:"esign"`
|
|
Branding bool `json:"branding"`
|
|
}
|
|
|
|
// TrustCenterConfig contains trust center server configuration.
|
|
TrustCenterConfig struct {
|
|
HTTPAddr string `json:"http-addr"`
|
|
HTTPSAddr string `json:"https-addr"`
|
|
ProxyProtocol ProxyProtocolConfig `json:"proxy-protocol"`
|
|
}
|
|
)
|