Help witth Caddy as reverse proxy of LXD API

1. The problem I’m having:

I’m trying to use caddy as a reverse proxy for LXD API.

The problem is the TLS authentication is not working when connecting behind caddy. My guess is that caddy is not sending/passing the client certificate to the backend.

I’m really lost on how this should work or if it is possible at all. Please advise if I’m doing something wrong.

2. Error messages and/or full log output:

To test LXD authentication using curl:

When connecting directly to LXD (port 8443):

$ curl -k -s --cert ~/client.crt --key ~/client.key https://vfm1-ub20-sdev-11.int.fmorales.net:8443/1.0 | jq .metadata.auth
"trusted"

When connecting using caddy as reverse proxy (port 443)

$ curl -s --cert ~/client.crt --key ~/client.key https://vfm1-ub20-sdev-11.int.fmorales.net/1.0 | jq .metadata.auth
"untrusted"

3. Caddy version:

$ caddy version
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 20.04 amd64 server installed on a KVM virtual machine.

b. Command:

Caddy is started by systemd. See next.

c. Service/unit/compose file:

Systemd service file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

vfm1-ub20-sdev-11.int.fmorales.net {
	reverse_proxy https://127.0.0.1:8443 {
		transport http {
			tls_trusted_ca_certs /var/snap/lxd/common/lxd/server.crt
		}
	}
}

5. Links to relevant resources:

LXD client certificate authentication

Note:

For the ones who know something about LXD, this is the output of lxc config show:

config:
  core.https_address: :8443

Correct. Caddy terminates TLS, so when proxying it has to make a new TLS handshake. It cannot reuse the client cert for the new handshake because it doesn’t have the private key attached to the client cert to perform encryption with it.

You could pass the client cert through using an HTTP header if your upstream knows how to grab the cert from headers. You could do something like:

header_up X-Client-Cert {tls_client_certificate_der_base64}

Is that the certificate that is passed to curl with --cert?

curl --cert ~/client.crt

I still can’t authenticate. This is the new Caddyfile:

vfm1-ub20-sdev-11.int.fmorales.net {
	reverse_proxy https://127.0.0.1:8443 {
		transport http {
			tls_trusted_ca_certs /var/snap/lxd/common/lxd/server.crt
		}
		header_up X-Client-Cert {tls_client_certificate_der_base64}
	}
}

I have tried with X-Client-Cert and Client-Cert. Is there another header I can try? I have tried to find out how LXD works with this with no luck.

This is the curl commend with full log (using jq to better display):

* TCP_NODELAY set
* Connected to vfm1-ub20-sdev-11.int.fmorales.net (192.168.1.229) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3806 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=vfm1-ub20-sdev-11.int.fmorales.net
*  start date: Jun 26 18:00:55 2023 GMT
*  expire date: Sep 24 18:00:54 2023 GMT
*  subjectAltName: host "vfm1-ub20-sdev-11.int.fmorales.net" matched cert's "vfm1-ub20-sdev-11.int.fmorales.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55d4588a2610)
} [5 bytes data]
> GET /1.0 HTTP/2
> Host: vfm1-ub20-sdev-11.int.fmorales.net
> user-agent: curl/7.68.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [130 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
} [5 bytes data]
< HTTP/2 200 
< alt-svc: h3=":443"; ma=2592000
< content-security-policy: frame-ancestors 'self';
< content-type: application/json
< date: Wed, 28 Jun 2023 14:50:49 GMT
< permissions-policy: microphone=(), geolocation=(), camera=()
< referrer-policy: no-referrer-when-downgrade
< server: Caddy
< strict-transport-security: max-age=31536000;
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< 
{ [5 bytes data]
100  7548    0  7548    0     0   273k      0 --:--:-- --:--:-- --:--:--  273k
* Connection #0 to host vfm1-ub20-sdev-11.int.fmorales.net left intact
{
  "type": "sync",
  "status": "Success",
  "status_code": 200,
  "operation": "",
  "error_code": 0,
  "error": "",
  "metadata": {
    "api_extensions": [
      "storage_zfs_remove_snapshots",
      "container_host_shutdown_timeout",
      "container_stop_priority",
      "container_syscall_filtering",
      "auth_pki",
      "container_last_used_at",
      "etag",
      "patch",
      "usb_devices",
      "https_allowed_credentials",
      "image_compression_algorithm",
      "directory_manipulation",
      "container_cpu_time",
      "storage_zfs_use_refquota",
      "storage_lvm_mount_options",
      "network",
      "profile_usedby",
      "container_push",
      "container_exec_recording",
      "certificate_update",
      "container_exec_signal_handling",
      "gpu_devices",
      "container_image_properties",
      "migration_progress",
      "id_map",
      "network_firewall_filtering",
      "network_routes",
      "storage",
      "file_delete",
      "file_append",
      "network_dhcp_expiry",
      "storage_lvm_vg_rename",
      "storage_lvm_thinpool_rename",
      "network_vlan",
      "image_create_aliases",
      "container_stateless_copy",
      "container_only_migration",
      "storage_zfs_clone_copy",
      "unix_device_rename",
      "storage_lvm_use_thinpool",
      "storage_rsync_bwlimit",
      "network_vxlan_interface",
      "storage_btrfs_mount_options",
      "entity_description",
      "image_force_refresh",
      "storage_lvm_lv_resizing",
      "id_map_base",
      "file_symlinks",
      "container_push_target",
      "network_vlan_physical",
      "storage_images_delete",
      "container_edit_metadata",
      "container_snapshot_stateful_migration",
      "storage_driver_ceph",
      "storage_ceph_user_name",
      "resource_limits",
      "storage_volatile_initial_source",
      "storage_ceph_force_osd_reuse",
      "storage_block_filesystem_btrfs",
      "resources",
      "kernel_limits",
      "storage_api_volume_rename",
      "macaroon_authentication",
      "network_sriov",
      "console",
      "restrict_devlxd",
      "migration_pre_copy",
      "infiniband",
      "maas_network",
      "devlxd_events",
      "proxy",
      "network_dhcp_gateway",
      "file_get_symlink",
      "network_leases",
      "unix_device_hotplug",
      "storage_api_local_volume_handling",
      "operation_description",
      "clustering",
      "event_lifecycle",
      "storage_api_remote_volume_handling",
      "nvidia_runtime",
      "container_mount_propagation",
      "container_backup",
      "devlxd_images",
      "container_local_cross_pool_handling",
      "proxy_unix",
      "proxy_udp",
      "clustering_join",
      "proxy_tcp_udp_multi_port_handling",
      "network_state",
      "proxy_unix_dac_properties",
      "container_protection_delete",
      "unix_priv_drop",
      "pprof_http",
      "proxy_haproxy_protocol",
      "network_hwaddr",
      "proxy_nat",
      "network_nat_order",
      "container_full",
      "candid_authentication",
      "backup_compression",
      "candid_config",
      "nvidia_runtime_config",
      "storage_api_volume_snapshots",
      "storage_unmapped",
      "projects",
      "candid_config_key",
      "network_vxlan_ttl",
      "container_incremental_copy",
      "usb_optional_vendorid",
      "snapshot_scheduling",
      "snapshot_schedule_aliases",
      "container_copy_project",
      "clustering_server_address",
      "clustering_image_replication",
      "container_protection_shift",
      "snapshot_expiry",
      "container_backup_override_pool",
      "snapshot_expiry_creation",
      "network_leases_location",
      "resources_cpu_socket",
      "resources_gpu",
      "resources_numa",
      "kernel_features",
      "id_map_current",
      "event_location",
      "storage_api_remote_volume_snapshots",
      "network_nat_address",
      "container_nic_routes",
      "rbac",
      "cluster_internal_copy",
      "seccomp_notify",
      "lxc_features",
      "container_nic_ipvlan",
      "network_vlan_sriov",
      "storage_cephfs",
      "container_nic_ipfilter",
      "resources_v2",
      "container_exec_user_group_cwd",
      "container_syscall_intercept",
      "container_disk_shift",
      "storage_shifted",
      "resources_infiniband",
      "daemon_storage",
      "instances",
      "image_types",
      "resources_disk_sata",
      "clustering_roles",
      "images_expiry",
      "resources_network_firmware",
      "backup_compression_algorithm",
      "ceph_data_pool_name",
      "container_syscall_intercept_mount",
      "compression_squashfs",
      "container_raw_mount",
      "container_nic_routed",
      "container_syscall_intercept_mount_fuse",
      "container_disk_ceph",
      "virtual-machines",
      "image_profiles",
      "clustering_architecture",
      "resources_disk_id",
      "storage_lvm_stripes",
      "vm_boot_priority",
      "unix_hotplug_devices",
      "api_filtering",
      "instance_nic_network",
      "clustering_sizing",
      "firewall_driver",
      "projects_limits",
      "container_syscall_intercept_hugetlbfs",
      "limits_hugepages",
      "container_nic_routed_gateway",
      "projects_restrictions",
      "custom_volume_snapshot_expiry",
      "volume_snapshot_scheduling",
      "trust_ca_certificates",
      "snapshot_disk_usage",
      "clustering_edit_roles",
      "container_nic_routed_host_address",
      "container_nic_ipvlan_gateway",
      "resources_usb_pci",
      "resources_cpu_threads_numa",
      "resources_cpu_core_die",
      "api_os",
      "container_nic_routed_host_table",
      "container_nic_ipvlan_host_table",
      "container_nic_ipvlan_mode",
      "resources_system",
      "images_push_relay",
      "network_dns_search",
      "container_nic_routed_limits",
      "instance_nic_bridged_vlan",
      "network_state_bond_bridge",
      "usedby_consistency",
      "custom_block_volumes",
      "clustering_failure_domains",
      "resources_gpu_mdev",
      "console_vga_type",
      "projects_limits_disk",
      "network_type_macvlan",
      "network_type_sriov",
      "container_syscall_intercept_bpf_devices",
      "network_type_ovn",
      "projects_networks",
      "projects_networks_restricted_uplinks",
      "custom_volume_backup",
      "backup_override_name",
      "storage_rsync_compression",
      "network_type_physical",
      "network_ovn_external_subnets",
      "network_ovn_nat",
      "network_ovn_external_routes_remove",
      "tpm_device_type",
      "storage_zfs_clone_copy_rebase",
      "gpu_mdev",
      "resources_pci_iommu",
      "resources_network_usb",
      "resources_disk_address",
      "network_physical_ovn_ingress_mode",
      "network_ovn_dhcp",
      "network_physical_routes_anycast",
      "projects_limits_instances",
      "network_state_vlan",
      "instance_nic_bridged_port_isolation",
      "instance_bulk_state_change",
      "network_gvrp",
      "instance_pool_move",
      "gpu_sriov",
      "pci_device_type",
      "storage_volume_state",
      "network_acl",
      "migration_stateful",
      "disk_state_quota",
      "storage_ceph_features",
      "projects_compression",
      "projects_images_remote_cache_expiry",
      "certificate_project",
      "network_ovn_acl",
      "projects_images_auto_update",
      "projects_restricted_cluster_target",
      "images_default_architecture",
      "network_ovn_acl_defaults",
      "gpu_mig",
      "project_usage",
      "network_bridge_acl",
      "warnings",
      "projects_restricted_backups_and_snapshots",
      "clustering_join_token",
      "clustering_description",
      "server_trusted_proxy",
      "clustering_update_cert",
      "storage_api_project",
      "server_instance_driver_operational",
      "server_supported_storage_drivers",
      "event_lifecycle_requestor_address",
      "resources_gpu_usb",
      "clustering_evacuation",
      "network_ovn_nat_address",
      "network_bgp",
      "network_forward",
      "custom_volume_refresh",
      "network_counters_errors_dropped",
      "metrics",
      "image_source_project",
      "clustering_config",
      "network_peer",
      "linux_sysctl",
      "network_dns",
      "ovn_nic_acceleration",
      "certificate_self_renewal",
      "instance_project_move",
      "storage_volume_project_move",
      "cloud_init",
      "network_dns_nat",
      "database_leader",
      "instance_all_projects",
      "clustering_groups",
      "ceph_rbd_du",
      "instance_get_full",
      "qemu_metrics",
      "gpu_mig_uuid",
      "event_project",
      "clustering_evacuation_live",
      "instance_allow_inconsistent_copy",
      "network_state_ovn",
      "storage_volume_api_filtering",
      "image_restrictions",
      "storage_zfs_export",
      "network_dns_records",
      "storage_zfs_reserve_space",
      "network_acl_log",
      "storage_zfs_blocksize",
      "metrics_cpu_seconds",
      "instance_snapshot_never",
      "certificate_token",
      "instance_nic_routed_neighbor_probe",
      "event_hub",
      "agent_nic_config",
      "projects_restricted_intercept",
      "metrics_authentication",
      "images_target_project",
      "cluster_migration_inconsistent_copy",
      "cluster_ovn_chassis",
      "container_syscall_intercept_sched_setscheduler",
      "storage_lvm_thinpool_metadata_size",
      "storage_volume_state_total",
      "instance_file_head",
      "resources_pci_vpd",
      "qemu_raw_conf",
      "storage_cephfs_fscache",
      "vsock_api",
      "storage_volumes_all_projects",
      "projects_networks_restricted_access",
      "cluster_join_token_expiry",
      "remote_token_expiry",
      "init_preseed",
      "cpu_hotplug"
    ],
    "api_status": "stable",
    "api_version": "1.0",
    "auth": "untrusted",
    "public": false,
    "auth_methods": [
      "tls"
    ]
  }
}

Thank you.

I can’t help with that part. I don’t know what your upstream application is or does. You’ll need to figure out whether it can accept the client cert as a header.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.