Guard vetting agent HTTP tools against SSRF

The third-party vetting agent runs a suite of HTTP "security" tools on
the internal worker network against a caller-supplied URL that is only
validated for length and charset, not host. Several tools reached
internal, loopback, and link-local addresses:

  - analyze_csp used a bare http.Client with no host validation, no
    redirect control, and no rebinding-safe transport, reflecting the
    target's CSP header back to the caller.
  - check_security_headers, fetch_robots_txt, and fetch_sitemap
    validated only the initial host, then followed 3xx redirects with an
    ordinary client, yielding full-read SSRF via a redirect to an
    internal address.
  - check_cors validated the URL but still dialed through an ordinary
    transport, leaving it exposed to DNS-rebinding TOCTOU.

Route every one of these clients through the house-standard
httpclient.DefaultPooledClient(WithSSRFProtection()), which rejects
dials to loopback, private, CGNAT, link-local, ULA, IPv4-mapped, and
reserved ranges on the resolved peer IP at connect time (defeating DNS
rebinding on every redirect hop) and refuses cross-origin redirects.
download_pdf moves onto the same client, and the now-unused local
netcheck.NewPinnedTransport is removed. analyze_csp also gains an
up-front ValidatePublicURL check for a clean early error and scheme
enforcement.

Signed-off-by: Sacha Al Himdani <sacha@probo.com>

pkg/agent/tools/internal/netcheck/netcheck.go

@@ -17,10 +17,8 @@
 package netcheck
 
 import (
-	"context"
 	"fmt"
 	"net"
-	"net/http"
 	"net/url"
 )
 
@@ -88,41 +86,3 @@ func ValidatePublicDomain(domain string) error {
 
 	return nil
 }
-
-// NewPinnedTransport returns an *http.Transport with a custom DialContext that
-// resolves the target host once, validates all resolved IPs with IsPublicIP,
-// and dials the validated IP directly. This prevents DNS rebinding attacks
-// where the first lookup returns a public IP but a subsequent lookup (at
-// connection time) returns a private IP.
-func NewPinnedTransport() *http.Transport {
-	return &http.Transport{
-		DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
-			host, port, err := net.SplitHostPort(addr)
-			if err != nil {
-				return nil, fmt.Errorf("cannot parse address: %w", err)
-			}
-
-			ips, err := net.DefaultResolver.LookupIPAddr(ctx, host)
-			if err != nil {
-				return nil, fmt.Errorf("cannot resolve host: %w", err)
-			}
-
-			if len(ips) == 0 {
-				return nil, fmt.Errorf("cannot resolve host: no addresses found")
-			}
-
-			for _, ip := range ips {
-				if !IsPublicIP(ip.IP) {
-					return nil, fmt.Errorf("cannot connect to non-public IP %s", ip.IP)
-				}
-			}
-
-			// Dial the first validated IP directly to prevent DNS rebinding.
-			pinnedAddr := net.JoinHostPort(ips[0].IP.String(), port)
-
-			var d net.Dialer
-
-			return d.DialContext(ctx, network, pinnedAddr)
-		},
-	}
-}