Vault PKI Intermediate ca etcd Roles and permissions for real cluster Kubernetes the hard way v2
Вводная часть
Тут описывается создание сертификатов для etcd
Это продолжение Статьи про создание промежуточного СА
Упрощение работы
Путь к PKI (k8s_pki_intermediate_ca_for_service_etcd) встречается многократно, и он вынесен в переменную
cat 00_env
export PKI_NAME="k8s_pki_intermediate_ca_for_service_etcd"
Роли и пользователи для peer-to-peer сертификатов
В кластере участвуют три сервера, соответственно для каждого из них требуется отдельные настройки - роль, пользователь ...
Роли для получения peer-to-peer сертификатов
Три зоны - три роли, по одной для каждого сервера
client_flag=true server_flag=true enforce_hostnames=true key_usage="DigitalSignature,KeyEncipherment" ext_key_usage="ClientAuth,ServerAuth"
#!/bin/bash source ./00_env for AZ in $(seq 1 3); do #!/bin/bash source ./00_env for AZ in $(seq 1 3); do DOMAIN="etcd.master.az${AZ}.k8s.cluster.home" NAME="${DOMAIN}-client-and-server" vault \ write \ ${PKI_NAME}/roles/${NAME}-client-and-server-crt \ country="Ukraine" \ locality="Kharkov" \ street_address="Lui Pastera st 322 app. 311"\ postal_code="61172" \ organization="Home Network" \ ou="IT" \ allowed_domains="etcd.master.az1.k8s.cluster.home" \ allow_subdomains=false \ max_ttl="87600h" \ key_bits="2048" \ key_type="rsa" \ allow_any_name=false \ allow_bare_domains=false \ allow_glob_domain=false \ allow_ip_sans=true \ allow_localhost=false \ client_flag=true \ server_flag=true \ enforce_hostnames=true \ key_usage="DigitalSignature,KeyEncipherment" \ ext_key_usage="ClientAuth,ServerAuth" \ require_cn=true done
Success! Data written to: k8s_pki_intermediate_ca_for_service_etcd/roles/etcd.master.az1.k8s.cluster.home-client-and-server-client-and-server-crt Success! Data written to: k8s_pki_intermediate_ca_for_service_etcd/roles/etcd.master.az2.k8s.cluster.home-client-and-server-client-and-server-crt Success! Data written to: k8s_pki_intermediate_ca_for_service_etcd/roles/etcd.master.az3.k8s.cluster.home-client-and-server-client-and-server-crt
Создание политики, пользователя и привязка политики к пользователю
#!/bin/bash source ./00_env for AZ in $(seq 1 3); do DOMAIN="etcd.master.az${AZ}.k8s.cluster.home" NAME="${DOMAIN}-client-and-server" cat << EOF > ${NAME}-policy.hlc path "${PKI_NAME}/roles/${NAME}-crt" { capabilities = ["read", "create", "list", "update"] } EOF vault \ policy \ write \ ${NAME}-policy \ ${NAME}-policy.hlc vault \ write \ auth/userpass/users/${NAME}-user \ password=${NAME}-password \ policies=" ${NAME}-policy,default" done
Success! Uploaded policy: etcd.master.az1.k8s.cluster.home-client-and-server-policy Success! Data written to: auth/userpass/users/etcd.master.az1.k8s.cluster.home-client-and-server-user Success! Uploaded policy: etcd.master.az2.k8s.cluster.home-client-and-server-policy Success! Data written to: auth/userpass/users/etcd.master.az2.k8s.cluster.home-client-and-server-user Success! Uploaded policy: etcd.master.az3.k8s.cluster.home-client-and-server-policy Success! Data written to: auth/userpass/users/etcd.master.az3.k8s.cluster.home-client-and-server-user
Просмотр созданных политик
#!/bin/bash source ./00_env echo "---------ROLES---------------------" vault \ list \ ${PKI_NAME}/roles echo "---------USERS---------------------" vault \ list \ auth/userpass/users echo "------------------------------" for AZ in $(seq 1 3); do DOMAIN="etcd.master.az${AZ}.k8s.cluster.home" NAME="${DOMAIN}-client-and-server" vault \ policy \ read \ ${NAME}-policy vault \ read \ auth/userpass/users/${NAME}-user done
Получения сертефикатов
- Получать непосредственно на нодах
Получение серверного сертефиката
vault \ login \ -method=userpass \ username=example-dot-home-server-crt-user \ password=server
Key Value --- ----- token s.Wiy7YVCrte88i0QIHc4jmvQP token_accessor ptB7o9d6yOq5w4Ra5Q44W1FF token_duration 768h token_renewable true token_policies ["default" "example-dot-home-server-crt-policy"] identity_policies [] policies ["default" "example-dot-home-server-crt-policy"] token_meta_username example-dot-home-server-crt-user
Получение сертефиката
- Получаем сертификат для тестового домена - vault.example.home и Alt Name pki.example.home
vault \ write \ -format=json \ ${PKI_NAME}/issue/example-dot-home-server-crt \ common_name="vault.example.home" \ alt_names="pki.example.home" \ ttl="43800h" > vault.example.home.crt
Проверка полученного сертефиката
Результат в файле:
cat vault.example.home.crt
Сокращенный вывод:
{ "request_id": "d2ff6e16-1bf6-730a-f4d1-2aefa2fcbb3e", "lease_id": "", "lease_duration": 0, "renewable": false, "data": { "ca_chain": [ "-----BEGIN CERTIFICATE-----rCdewvVGkgifREEGI2GltrDrH6rMugtVjuNAWzW40pVLKl/+0/Jv87wAv\nfITqQgiqa0FjitvfYQO2qIxSCJkt+kD3Vg==\n-----END CERTIFICATE-----" ], "certificate": "-----BEGIN CERTIFICATE-----m35waa9ld+hkNIcf/1qR4Gvwae9w0\n-----END CERTIFICATE-----", "expiration": 1822736540, "issuing_ca": "-----BEGIN CERTIFICATE-----rCdewvVGkgifREEGI2GltrDrH6rMugtVjuNAWzW40pVLKl/+0/Jv87wAv\nfITqQgiqa0FjitvfYQO2qIxSCJkt+kD3Vg==\n-----END CERTIFICATE-----", "private_key": "-----BEGIN RSA PRIVATE KEY-----vNhpK6RKn2b4EExuuZTRAcPEV3ddhOOoZpyy48WVEF5Iq3s+7/NZOq66poZUz17z\nwhRJIuic/EzYBnmKy0T4wdCyhkqLVGHIvH+412cJ5eqyHeMQbzG+oA==\n-----END RSA PRIVATE KEY-----", "private_key_type": "rsa", "serial_number": "28:ac:20:32:fe:46:b2:78:61:11:f0:46:da:e7:d2:cf:02:fc:4f:f1" }, "warnings": null }
Из JSON можно выделить отдельные поля
cat vault.example.home.crt | jq -r .data.ca_chain[] > vault.example.home.ca_chain.pem cat vault.example.home.crt | jq -r .data.certificate > vault.example.home.certificate.pem cat vault.example.home.crt | jq -r .data.issuing_ca > vault.example.home.issuing_ca.pem
Сертификат сервера
Наиболее интересные поля - ожидаемые значения:
- CN = vault.example.home
- X509v3 Subject Alternative Name: DNS:pki.example.home, DNS:vault.example.home
- IP адреса в AltNames не присутвуют
openssl x509 -in vault.example.home.certificate.pem -text -noout
Certificate: Data: Version: 3 (0x2) Serial Number: 28:ac:20:32:fe:46:b2:78:61:11:f0:46:da:e7:d2:cf:02:fc:4f:f1 Signature Algorithm: sha256WithRSAEncryption Issuer: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = K8s The Hardest Way Labs, OU = IT, CN = Intermediate CA for service ETCd Validity Not Before: Oct 6 11:41:54 2022 GMT Not After : Oct 5 11:42:20 2027 GMT Subject: C = Ukraine, L = Kharkov, street = Lui Pastera st 322 app. 311, postalCode = 61172, O = Home Network, OU = IT, CN = vault.example.home Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:ad:4e:38:25:33:11:b0:aa:ab:1a:17:7f:8e:df: ... e5:80:12:b9:d2:68:29:5e:85:d3:7a:3e:06:e8:e7: e6:27 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Subject Key Identifier: 4C:4C:22:C1:44:68:25:D0:6B:AA:7F:AB:5C:8E:16:F5:B7:BC:DF:AF X509v3 Authority Key Identifier: keyid:60:41:31:79:58:90:A9:63:62:C2:26:FD:8F:02:B6:07:1A:1D:5C:50 Authority Information Access: CA Issuers - URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/ca X509v3 Subject Alternative Name: DNS:pki.example.home, DNS:vault.example.home X509v3 CRL Distribution Points: Full Name: URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/crl Signature Algorithm: sha256WithRSAEncryption 7a:ee:f9:5a:88:dc:ed:0d:b1:f7:26:97:d9:fa:98:e7:b3:43: ...
Промежуточный CA
openssl x509 -in vault.example.home.issuing_ca.pem -text -noout
- Видно CN = Intermediate CA for service ETCd, другими словами это промежуточный сертификат удостоверяющего центра (СА) для ETCd
Certificate: Data: Version: 3 (0x2) Serial Number: 35:0e:d5:7b:85:55:57:d3:99:ed:2c:35:c7:db:d3:99:ac:9b:a3:01 Signature Algorithm: sha256WithRSAEncryption Issuer: C = Ukraine, L = Kharkov, street = app. 131 + street = Lui Pastera St. 322, postalCode = 61172, O = Home Network, OU = IT, CN = Root Certificate Authority for Home Network v2 Validity Not Before: Oct 3 15:53:05 2022 GMT Not After : Sep 28 15:53:35 2042 GMT Subject: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = K8s The Hardest Way Labs, OU = IT, CN = Intermediate CA for service ETCd Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:c3:04:35:f0:29:59:33:db:a3:8a:88:f6:44:b0: ... 58:4a:d6:3d:3c:5d:3f:42:a0:d8:5b:65:5f:71:ba: 3e:57 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Subject Key Identifier: 60:41:31:79:58:90:A9:63:62:C2:26:FD:8F:02:B6:07:1A:1D:5C:50 X509v3 Authority Key Identifier: keyid:02:F8:85:2B:75:F8:E1:1C:69:28:30:32:21:2D:86:71:AF:AB:EC:3C Authority Information Access: CA Issuers - URI:http://vault.home:8200/v1/k8s_pki_root_ca/ca X509v3 CRL Distribution Points: Full Name: URI:http://vault.home:8200/v1/k8s_pki_root_ca/crl Signature Algorithm: sha256WithRSAEncryption 65:f8:9d:6c:6d:6a:14:d3:37:32:ae:c1:de:26:f3:c8:9f:e6: ... 42:08:aa:6b:41:63:8a:db:df:61:03:b6:a8:8c:52:08:99:2d: fa:40:f7:56
Верефикация
Окончательная проверка цепочки доверия
Корневой Сертификат → Промежуточный для СА Etcd → Тестовый Сертификат
- Корневой сертификат не является секретным (в отличие от ключа!) и его всегда можно скачать с Vault:
echo "-----BEGIN CERTIFICATE-----" > k8s_pki_root_ca.pem && \ curl "http://vault.home:8200/v1/k8s_pki_root_ca/ca" | base64 >> k8s_pki_root_ca.pem && \ echo "-----END CERTIFICATE-----" >> k8s_pki_root_ca.pem
- Указать промежуточный как untrusted (или добавить в доверенные)
openssl \ verify \ -verbose \ -CAfile k8s_pki_root_ca.pem \ -untrusted vault.example.home.issuing_ca.pem \ vault.example.home.certificate.pem
vault.example.home.certificate.pem: OK
Однако в этой команде меня смущает слово -untrusted
Второй способ - это создание "бандла" (те поместить в один файл промежуточный CA а за ним сертификат)
cat vault.example.home.issuing_ca.pem > vault.example.home.ca_bundle cat vault.example.home.certificate.pem >> vault.example.home.ca_bundle
Теперь верификация проходит без беспокоящего -untrusted
openssl \ verify \ -verbose \ -CAfile k8s_pki_root_ca.pem \ vault.example.home.ca_bundle
vault.example.home.ca_bundle: OK
Верефикация c nginx
Для полноты картины создадим тестовый web-сервер и используем подписанный сертификат
- Пример виртуалхоста
- Путь к ключу и сертификату (ключ можно получить из оригинального вывода Vault при создании сертификата,
ssl_certificate /etc/nginx/certs/vault.example.home.ca_bundle; ssl_certificate_key /etc/nginx/certs/vault.example.home.key;
Формат в nginx отличается - в файле /etc/nginx/certs/vault.example.home.ca_bundle СНАЧАЛА идет сертификат сервера, а потом - промежуточный сертификат (https://nginx.org/en/docs/http/configuring_https_servers.html)
* Полный конфиг server { listen 8203 default_server ssl; root /var/www/html; server_name vault.example.home; access_log /var/log/nginx/vault.example.home-access.log postdata; error_log /var/log/nginx/vault.example.home-error.log; ssl_certificate /etc/nginx/certs/vault.example.home.ca_bundle; ssl_certificate_key /etc/nginx/certs/vault.example.home.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers HIGH:!aNULL:!MD5; location / { client_body_buffer_size 64k; client_body_in_single_buffer on; proxy_pass http://127.0.0.1:8200; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_http_version 1.1; proxy_request_buffering off; } }
Для того что бы проверить, нужно на том хосте откуда делается запрос добавить сертификат СА в доверенные
- в /usr/local/share/ca-certificates/extra положить СА (его всегда можно скачать)
- запустить update-ca-certificates
# update-ca-certificates
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... Adding debian:k8s_pki_root_ca.pem done. done.
После этого curl может подключиться без ошибок верефикации
curl -v https://vault.example.home:8203
* successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=Ukraine; L=Kharkov; street=Lui Pastera st 322 app. 311; postalCode=61172; O=Home Network; OU=IT; CN=vault.example.home * start date: Oct 6 11:41:54 2022 GMT * expire date: Oct 5 11:42:20 2027 GMT * subjectAltName: host "vault.example.home" matched cert's "vault.example.home" * issuer: C=Ukraine; L=Kharkov; street=Lui Pastera st. 322 app. 131; postalCode=61172; O=K8s The Hardest Way Labs; OU=IT; CN=Intermediate CA for service ETCd * SSL certificate verify ok. > GET / HTTP/1.1 > Host: vault.example.home:8203 > User-Agent: curl/7.64.0 > Accept: */* > < HTTP/1.1 404 Not Found < Server: nginx/1.14.2 < Date: Fri, 07 Oct 2022 17:22:22 GMT < Content-Type: text/plain; charset=utf-8 < Content-Length: 19 < Connection: keep-alive < Cache-Control: no-store < X-Content-Type-Options: nosniff
Код 404 тут ожидаем так как нет задачи получить данные, а только проверить правильность настройки SSL
Получение клиентского сертефиката
Проверка прав (негативный сценарий)
С "серверным" пользователем - нет прав
source ./00_env vault \ login \ -method=userpass \ username=example-dot-home-server-crt-user \ password=server
Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token s.P9nYzZ3Pev2IeNKaUYvDIDdt token_accessor AbXtLIzNhYJRvv6paZr3U6cn token_duration 768h token_renewable true token_policies ["default" "example-dot-home-server-crt-policy"] identity_policies [] policies ["default" "example-dot-home-server-crt-policy"] token_meta_username example-dot-home-server-crt-user
vault \ write \ -format=json \ ${PKI_NAME}/issue/example-dot-home-client-crt \ common_name="vault.example.home" \ alt_names="pki.example.home" \ ttl="43800h" > vault.example.home-client.crt
Error writing data to k8s_pki_intermediate_ca_for_service_etcd/issue/example-dot-home-client-crt: Error making API request. URL: PUT http://127.0.0.1:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/issue/example-dot-home-client-crt Code: 403. Errors: * 1 error occurred: * permission denied
Результат соответствует ожидаемому.
Получение клиентского сертификата c правильным пользователем
vault \ login \ -method=userpass \ username=example-dot-home-any-crt-user \ password=any vault \ write \ -format=json \ ${PKI_NAME}/issue/example-dot-home-client-crt \ common_name="vault.example.home" \ alt_names="pki.example.home" \ ttl="43800h" > vault.example.home-client.crt.json Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token s.vAKcP6hGLv9OHkm8ToBHFq5M token_accessor OoNShUinoMhne5s8scFcZ0Hm token_duration 768h token_renewable true token_policies ["default" "example-dot-home-client-crt-policy" "example-dot-home-server-crt-policy"] identity_policies [] policies ["default" "example-dot-home-client-crt-policy" "example-dot-home-server-crt-policy"] token_meta_username example-dot-home-any-crt-user
Проверка полученного результата
cat vault.example.home-client.crt.json | jq -r '.data.certificate' > vault.example.home-client.crt cat vault.example.home-client.crt.json | jq -r '.data.private_key' > vault.example.home-client.key
X509v3 Extended Key Usage: TLS Web Client Authentication
openssl x509 -noout -text -in certificate_client.pem <PRE> Certificate: Data: Version: 3 (0x2) Serial Number: 51:de:46:e0:72:1e:2b:28:30:3b:9e:94:f4:da:71:f8:19:5e:26:6d Signature Algorithm: sha256WithRSAEncryption Issuer: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = K8s The Hardest Way Labs, OU = IT, CN = Intermediate CA for service ETCd Validity Not Before: Oct 8 16:53:43 2022 GMT Not After : Oct 7 16:54:10 2027 GMT Subject: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = Home Network, OU = IT, CN = vault.example.home Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:c3:21:f6:55:f9:f0:a7:19:52:b9:22:a6:9a:99: ... 05:e5 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Key Usage: critical Digital Signature X509v3 Extended Key Usage: TLS Web Client Authentication X509v3 Subject Key Identifier: C9:F1:9F:83:D7:0B:B8:F2:7E:BB:84:D1:FE:6F:7E:75:7B:40:F4:86 X509v3 Authority Key Identifier: keyid:60:41:31:79:58:90:A9:63:62:C2:26:FD:8F:02:B6:07:1A:1D:5C:50 Authority Information Access: CA Issuers - URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/ca X509v3 Subject Alternative Name: DNS:pki.example.home, DNS:vault.example.home X509v3 CRL Distribution Points: Full Name: URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/crl Signature Algorithm: sha256WithRSAEncryption 94:bb:a8:54:41:86:16:75:06:e7:fb:5a:5f:e0:56:61:5d:ff: ... 3e:56:1a:6f
Проверка полученного результата
Расширенная проверка - проверить что сертификат нельзя использовать в качестве серверного Конфиг nginx не отличается ничем кроме собственно содержимого сертификата
- Ответ собственно и говорит об этом - unsupported certificate purpose, клиентский сертификат не оч
curl: (60) SSL certificate problem: unsupported certificate purpose More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
Проверка прав для пользователя any
- работает каки ожидалось - можно получить сертификаты как для сервера так и для клиента
#!/bin/bash source ./00_env unset VAULT_TOKEN vault \ login \ -method=userpass \ username=example-dot-home-any-crt-user \ password=any vault \ write \ -format=json \ ${PKI_NAME}/issue/example-dot-home-client-crt \ common_name="vault.example.home" \ alt_names="pki.example.home" \ ttl="43800h" > vault.example.home.CLIENT_by_any_user.json vault \ write \ -format=json \ ${PKI_NAME}/issue/example-dot-home-server-crt \ common_name="vault.example.home" \ alt_names="pki.example.home" \ ttl="43800h" > vault.example.home.SERVER_by_any_user.json