Vault PKI Intermediate ca etcd Roles and permissions for real cluster Kubernetes the hard way v2

Материал из noname.com.ua
Перейти к навигацииПерейти к поиску

Вводная часть

Тут описывается создание сертификатов для etcd

Это продолжение Статьи про создание промежуточного СА

Упрощение работы

Путь к PKI (k8s_pki_intermediate_ca_for_service_etcd) встречается многократно, и он вынесен в переменную 

cat 00_env

export PKI_NAME="k8s_pki_intermediate_ca_for_service_etcd"

Роли и пользователи для peer-to-peer сертификатов

В кластере участвуют три сервера, соответственно для каждого из них требуется отдельные настройки - роль, пользователь ...


Роли для получения peer-to-peer сертификатов

Три зоны - три роли, по одной для каждого сервера 

client_flag=true
server_flag=true
enforce_hostnames=true
key_usage="DigitalSignature,KeyEncipherment"
ext_key_usage="ClientAuth,ServerAuth"

#!/bin/bash

source ./00_env


for AZ in $(seq 1 3);
do
#!/bin/bash

source ./00_env


for AZ in $(seq 1 3);
do
    DOMAIN="etcd.master.az${AZ}.k8s.cluster.home"
    NAME="${DOMAIN}-client-and-server"


    vault \
        write \
            ${PKI_NAME}/roles/${NAME}-client-and-server-crt \
            country="Ukraine" \
            locality="Kharkov" \
            street_address="Lui Pastera st 322 app. 311"\
            postal_code="61172" \
            organization="Home Network" \
            ou="IT" \
            allowed_domains="etcd.master.az1.k8s.cluster.home" \
            allow_subdomains=false \
            max_ttl="87600h" \
            key_bits="2048" \
            key_type="rsa" \
            allow_any_name=false \
            allow_bare_domains=false \
            allow_glob_domain=false \
            allow_ip_sans=true \
            allow_localhost=false \
            client_flag=true \
            server_flag=true \
            enforce_hostnames=true \
            key_usage="DigitalSignature,KeyEncipherment" \
            ext_key_usage="ClientAuth,ServerAuth" \
            require_cn=true
done

Success! Data written to: k8s_pki_intermediate_ca_for_service_etcd/roles/etcd.master.az1.k8s.cluster.home-client-and-server-client-and-server-crt
Success! Data written to: k8s_pki_intermediate_ca_for_service_etcd/roles/etcd.master.az2.k8s.cluster.home-client-and-server-client-and-server-crt
Success! Data written to: k8s_pki_intermediate_ca_for_service_etcd/roles/etcd.master.az3.k8s.cluster.home-client-and-server-client-and-server-crt

Создание политики, пользователя и привязка политики к пользователю

#!/bin/bash

source ./00_env


for AZ in $(seq 1 3);
do
    DOMAIN="etcd.master.az${AZ}.k8s.cluster.home"
    NAME="${DOMAIN}-client-and-server"

    cat << EOF > ${NAME}-policy.hlc
path "${PKI_NAME}/roles/${NAME}-crt"
{
  capabilities = ["read", "create", "list", "update"]
}
EOF

    vault \
        policy \
            write \
                ${NAME}-policy \
                 ${NAME}-policy.hlc
    vault \
        write \
            auth/userpass/users/${NAME}-user \
            password=${NAME}-password \
            policies=" ${NAME}-policy,default"
done


Success! Uploaded policy: etcd.master.az1.k8s.cluster.home-client-and-server-policy
Success! Data written to: auth/userpass/users/etcd.master.az1.k8s.cluster.home-client-and-server-user
Success! Uploaded policy: etcd.master.az2.k8s.cluster.home-client-and-server-policy
Success! Data written to: auth/userpass/users/etcd.master.az2.k8s.cluster.home-client-and-server-user
Success! Uploaded policy: etcd.master.az3.k8s.cluster.home-client-and-server-policy
Success! Data written to: auth/userpass/users/etcd.master.az3.k8s.cluster.home-client-and-server-user

Просмотр созданных политик

#!/bin/bash

source ./00_env

echo "------------------------------"
vault \
    list \
        ${PKI_NAME}/roles
echo "------------------------------"
vault \
    list \
        auth/userpass/users
echo "------------------------------"

for AZ in $(seq 1 3);
do

    NAME="etcd.master-az${AZ}-k8s-cluster-home-client-and-server"
    vault \
        policy \
            read \
                ${NAME}-policy

    vault \
        read \
            auth/userpass/users/${NAME}-user

done

Получения сертефикатов

  • Получать непосредственно на нодах

Получение серверного сертефиката

vault \
    login \
    -method=userpass \
    username=example-dot-home-server-crt-user \
    password=server

Key                    Value
---                    -----
token                  s.Wiy7YVCrte88i0QIHc4jmvQP
token_accessor         ptB7o9d6yOq5w4Ra5Q44W1FF
token_duration         768h
token_renewable        true
token_policies         ["default" "example-dot-home-server-crt-policy"]
identity_policies      []
policies               ["default" "example-dot-home-server-crt-policy"]
token_meta_username    example-dot-home-server-crt-user

Получение сертефиката

  • Получаем сертификат для тестового домена - vault.example.home и Alt Name pki.example.home
vault \
    write \
        -format=json \
        ${PKI_NAME}/issue/example-dot-home-server-crt \
        common_name="vault.example.home" \
        alt_names="pki.example.home" \
        ttl="43800h" > vault.example.home.crt

Проверка полученного сертефиката

Результат в файле: 

cat vault.example.home.crt

Сокращенный вывод: 

{
  "request_id": "d2ff6e16-1bf6-730a-f4d1-2aefa2fcbb3e",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "ca_chain": [
      "-----BEGIN CERTIFICATE-----rCdewvVGkgifREEGI2GltrDrH6rMugtVjuNAWzW40pVLKl/+0/Jv87wAv\nfITqQgiqa0FjitvfYQO2qIxSCJkt+kD3Vg==\n-----END CERTIFICATE-----"
    ],
    "certificate": "-----BEGIN CERTIFICATE-----m35waa9ld+hkNIcf/1qR4Gvwae9w0\n-----END CERTIFICATE-----",
    "expiration": 1822736540,
    "issuing_ca": "-----BEGIN CERTIFICATE-----rCdewvVGkgifREEGI2GltrDrH6rMugtVjuNAWzW40pVLKl/+0/Jv87wAv\nfITqQgiqa0FjitvfYQO2qIxSCJkt+kD3Vg==\n-----END CERTIFICATE-----",
    "private_key": "-----BEGIN RSA PRIVATE KEY-----vNhpK6RKn2b4EExuuZTRAcPEV3ddhOOoZpyy48WVEF5Iq3s+7/NZOq66poZUz17z\nwhRJIuic/EzYBnmKy0T4wdCyhkqLVGHIvH+412cJ5eqyHeMQbzG+oA==\n-----END RSA PRIVATE KEY-----",
    "private_key_type": "rsa",
    "serial_number": "28:ac:20:32:fe:46:b2:78:61:11:f0:46:da:e7:d2:cf:02:fc:4f:f1"
  },
  "warnings": null
}

Из JSON можно выделить отдельные поля 

cat vault.example.home.crt | jq -r .data.ca_chain[]  > vault.example.home.ca_chain.pem
cat vault.example.home.crt | jq -r .data.certificate > vault.example.home.certificate.pem
cat vault.example.home.crt | jq -r .data.issuing_ca  > vault.example.home.issuing_ca.pem

Сертификат сервера

Наиболее интересные поля - ожидаемые значения:

  • CN = vault.example.home
  • X509v3 Subject Alternative Name: DNS:pki.example.home, DNS:vault.example.home
  • IP адреса в AltNames не присутвуют
openssl  x509 -in vault.example.home.certificate.pem -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            28:ac:20:32:fe:46:b2:78:61:11:f0:46:da:e7:d2:cf:02:fc:4f:f1
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = K8s The Hardest Way Labs, OU = IT, CN = Intermediate CA for service ETCd
        Validity
            Not Before: Oct  6 11:41:54 2022 GMT
            Not After : Oct  5 11:42:20 2027 GMT
        Subject: C = Ukraine, L = Kharkov, street = Lui Pastera st 322 app. 311, postalCode = 61172, O = Home Network, OU = IT, CN = vault.example.home
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:ad:4e:38:25:33:11:b0:aa:ab:1a:17:7f:8e:df:
                    ...
                    e5:80:12:b9:d2:68:29:5e:85:d3:7a:3e:06:e8:e7:
                    e6:27
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Key Identifier:
                4C:4C:22:C1:44:68:25:D0:6B:AA:7F:AB:5C:8E:16:F5:B7:BC:DF:AF
            X509v3 Authority Key Identifier:
                keyid:60:41:31:79:58:90:A9:63:62:C2:26:FD:8F:02:B6:07:1A:1D:5C:50

            Authority Information Access:
                CA Issuers - URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/ca

            X509v3 Subject Alternative Name:
                DNS:pki.example.home, DNS:vault.example.home
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/crl

    Signature Algorithm: sha256WithRSAEncryption
         7a:ee:f9:5a:88:dc:ed:0d:b1:f7:26:97:d9:fa:98:e7:b3:43:
         ...

Промежуточный CA

openssl  x509 -in vault.example.home.issuing_ca.pem -text -noout
  • Видно CN = Intermediate CA for service ETCd, другими словами это промежуточный сертификат удостоверяющего центра (СА) для ETCd
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            35:0e:d5:7b:85:55:57:d3:99:ed:2c:35:c7:db:d3:99:ac:9b:a3:01
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = Ukraine, L = Kharkov, street = app. 131 + street = Lui Pastera St. 322, postalCode = 61172, O = Home Network, OU = IT, CN = Root Certificate Authority for Home Network v2
        Validity
            Not Before: Oct  3 15:53:05 2022 GMT
            Not After : Sep 28 15:53:35 2042 GMT
        Subject: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = K8s The Hardest Way Labs, OU = IT, CN = Intermediate CA for service ETCd
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c3:04:35:f0:29:59:33:db:a3:8a:88:f6:44:b0:
                    ...
                    58:4a:d6:3d:3c:5d:3f:42:a0:d8:5b:65:5f:71:ba:
                    3e:57
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                60:41:31:79:58:90:A9:63:62:C2:26:FD:8F:02:B6:07:1A:1D:5C:50
            X509v3 Authority Key Identifier:
                keyid:02:F8:85:2B:75:F8:E1:1C:69:28:30:32:21:2D:86:71:AF:AB:EC:3C

            Authority Information Access:
                CA Issuers - URI:http://vault.home:8200/v1/k8s_pki_root_ca/ca

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://vault.home:8200/v1/k8s_pki_root_ca/crl

    Signature Algorithm: sha256WithRSAEncryption
         65:f8:9d:6c:6d:6a:14:d3:37:32:ae:c1:de:26:f3:c8:9f:e6:
         ...
         42:08:aa:6b:41:63:8a:db:df:61:03:b6:a8:8c:52:08:99:2d:
         fa:40:f7:56

Верефикация

Окончательная проверка цепочки доверия
Корневой Сертификат → Промежуточный для СА Etcd → Тестовый Сертификат


  • Корневой сертификат не является секретным (в отличие от ключа!) и его всегда можно скачать с Vault:
echo "-----BEGIN CERTIFICATE-----" > k8s_pki_root_ca.pem && \
curl "http://vault.home:8200/v1/k8s_pki_root_ca/ca" | base64  >> k8s_pki_root_ca.pem && \
echo "-----END CERTIFICATE-----" >> k8s_pki_root_ca.pem
  • Указать промежуточный как untrusted (или добавить в доверенные)
openssl \
  verify \
  -verbose \
  -CAfile k8s_pki_root_ca.pem \
  -untrusted vault.example.home.issuing_ca.pem \
  vault.example.home.certificate.pem

vault.example.home.certificate.pem: OK

Однако в этой команде меня смущает слово -untrusted

Второй способ - это создание "бандла" (те поместить в один файл промежуточный CA а за ним сертификат) 

cat vault.example.home.issuing_ca.pem  >  vault.example.home.ca_bundle
cat vault.example.home.certificate.pem >> vault.example.home.ca_bundle

Теперь верификация проходит без беспокоящего -untrusted 

openssl \
  verify \
  -verbose \
  -CAfile k8s_pki_root_ca.pem \
  vault.example.home.ca_bundle

vault.example.home.ca_bundle: OK

Верефикация c nginx

Для полноты картины создадим тестовый web-сервер и используем подписанный сертификат

  • Пример виртуалхоста
  • Путь к ключу и сертификату (ключ можно получить из оригинального вывода Vault при создании сертификата,
    ssl_certificate     /etc/nginx/certs/vault.example.home.ca_bundle;
    ssl_certificate_key /etc/nginx/certs/vault.example.home.key;

Формат в nginx отличается - в файле /etc/nginx/certs/vault.example.home.ca_bundle СНАЧАЛА идет сертификат сервера, а потом - промежуточный сертификат (https://nginx.org/en/docs/http/configuring_https_servers.html) 

* Полный конфиг
server {
    listen 8203 default_server ssl;
    root /var/www/html;
    server_name vault.example.home;
    access_log /var/log/nginx/vault.example.home-access.log postdata;
    error_log /var/log/nginx/vault.example.home-error.log;

    ssl_certificate     /etc/nginx/certs/vault.example.home.ca_bundle;
    ssl_certificate_key /etc/nginx/certs/vault.example.home.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    location / {
        client_body_buffer_size      64k;
        client_body_in_single_buffer on;
        proxy_pass                   http://127.0.0.1:8200;
        proxy_set_header             Host $host:$server_port;
        proxy_set_header             X-Real-IP $remote_addr;
        proxy_set_header             X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header             X-Forwarded-Proto $scheme;
        proxy_http_version           1.1;
        proxy_request_buffering      off;
    }
}

Для того что бы проверить, нужно на том хосте откуда делается запрос добавить сертификат СА в доверенные

  • в /usr/local/share/ca-certificates/extra положить СА (его всегда можно скачать)
  • запустить update-ca-certificates
# update-ca-certificates

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Adding debian:k8s_pki_root_ca.pem
done.
done.

После этого curl может подключиться без ошибок верефикации 

 curl -v  https://vault.example.home:8203

* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=Ukraine; L=Kharkov; street=Lui Pastera st 322 app. 311; postalCode=61172; O=Home Network; OU=IT; CN=vault.example.home
*  start date: Oct  6 11:41:54 2022 GMT
*  expire date: Oct  5 11:42:20 2027 GMT
*  subjectAltName: host "vault.example.home" matched cert's "vault.example.home"
*  issuer: C=Ukraine; L=Kharkov; street=Lui Pastera st. 322 app. 131; postalCode=61172; O=K8s The Hardest Way Labs; OU=IT; CN=Intermediate CA for service ETCd
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: vault.example.home:8203
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx/1.14.2
< Date: Fri, 07 Oct 2022 17:22:22 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 19
< Connection: keep-alive
< Cache-Control: no-store
< X-Content-Type-Options: nosniff

Код 404 тут ожидаем так как нет задачи получить данные, а только проверить правильность настройки SSL

Получение клиентского сертефиката

Проверка прав (негативный сценарий)

С "серверным" пользователем - нет прав 

source ./00_env


vault \
    login \
    -method=userpass \
    username=example-dot-home-server-crt-user \
    password=server

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.P9nYzZ3Pev2IeNKaUYvDIDdt
token_accessor         AbXtLIzNhYJRvv6paZr3U6cn
token_duration         768h
token_renewable        true
token_policies         ["default" "example-dot-home-server-crt-policy"]
identity_policies      []
policies               ["default" "example-dot-home-server-crt-policy"]
token_meta_username    example-dot-home-server-crt-user

vault \
    write \
        -format=json \
        ${PKI_NAME}/issue/example-dot-home-client-crt \
        common_name="vault.example.home" \
        alt_names="pki.example.home" \
        ttl="43800h" > vault.example.home-client.crt

Error writing data to k8s_pki_intermediate_ca_for_service_etcd/issue/example-dot-home-client-crt: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/issue/example-dot-home-client-crt
Code: 403. Errors:

* 1 error occurred:
	* permission denied

Результат соответствует ожидаемому.

Получение клиентского сертификата c правильным пользователем



vault \
    login \
    -method=userpass \
    username=example-dot-home-any-crt-user \
    password=any

vault \
    write \
        -format=json \
        ${PKI_NAME}/issue/example-dot-home-client-crt \
        common_name="vault.example.home" \
        alt_names="pki.example.home" \
        ttl="43800h" > vault.example.home-client.crt.json

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.vAKcP6hGLv9OHkm8ToBHFq5M
token_accessor         OoNShUinoMhne5s8scFcZ0Hm
token_duration         768h
token_renewable        true
token_policies         ["default" "example-dot-home-client-crt-policy" "example-dot-home-server-crt-policy"]
identity_policies      []
policies               ["default" "example-dot-home-client-crt-policy" "example-dot-home-server-crt-policy"]
token_meta_username    example-dot-home-any-crt-user
Проверка полученного результата
cat vault.example.home-client.crt.json  | jq -r '.data.certificate' > vault.example.home-client.crt
cat vault.example.home-client.crt.json  | jq -r '.data.private_key' > vault.example.home-client.key

X509v3 Extended Key Usage:
  TLS Web Client Authentication

openssl x509 -noout -text -in certificate_client.pem
<PRE>
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            51:de:46:e0:72:1e:2b:28:30:3b:9e:94:f4:da:71:f8:19:5e:26:6d
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = K8s The Hardest Way Labs, OU = IT, CN = Intermediate CA for service ETCd
        Validity
            Not Before: Oct  8 16:53:43 2022 GMT
            Not After : Oct  7 16:54:10 2027 GMT
        Subject: C = Ukraine, L = Kharkov, street = Lui Pastera st. 322 app. 131, postalCode = 61172, O = Home Network, OU = IT, CN = vault.example.home
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c3:21:f6:55:f9:f0:a7:19:52:b9:22:a6:9a:99:
                    ...
                    05:e5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Subject Key Identifier:
                C9:F1:9F:83:D7:0B:B8:F2:7E:BB:84:D1:FE:6F:7E:75:7B:40:F4:86
            X509v3 Authority Key Identifier:
                keyid:60:41:31:79:58:90:A9:63:62:C2:26:FD:8F:02:B6:07:1A:1D:5C:50

            Authority Information Access:
                CA Issuers - URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/ca

            X509v3 Subject Alternative Name:
                DNS:pki.example.home, DNS:vault.example.home
            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://vault.home:8200/v1/k8s_pki_intermediate_ca_for_service_etcd/crl

    Signature Algorithm: sha256WithRSAEncryption
         94:bb:a8:54:41:86:16:75:06:e7:fb:5a:5f:e0:56:61:5d:ff:
         ...
         3e:56:1a:6f
Проверка полученного результата

Расширенная проверка - проверить что сертификат нельзя использовать в качестве серверного Конфиг nginx не отличается ничем кроме собственно содержимого сертификата

  • Ответ собственно и говорит об этом - unsupported certificate purpose, клиентский сертификат не оч
curl: (60) SSL certificate problem: unsupported certificate purpose
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Проверка прав для пользователя any

  • работает каки ожидалось - можно получить сертификаты как для сервера так и для клиента
#!/bin/bash


source ./00_env
unset VAULT_TOKEN


vault \
    login \
    -method=userpass \
    username=example-dot-home-any-crt-user \
    password=any


vault \
    write \
    -format=json \
    ${PKI_NAME}/issue/example-dot-home-client-crt \
    common_name="vault.example.home" \
    alt_names="pki.example.home" \
    ttl="43800h" > vault.example.home.CLIENT_by_any_user.json


vault \
    write \
    -format=json \
    ${PKI_NAME}/issue/example-dot-home-server-crt \
    common_name="vault.example.home" \
    alt_names="pki.example.home" \
    ttl="43800h" > vault.example.home.SERVER_by_any_user.json
Источник — https://noname.com.ua/mediawiki/index.php?title=Vault_PKI_Intermediate_ca_etcd_Roles_and_permissions_for_real_cluster_Kubernetes_the_hard_way_v2&oldid=10236