Харденинг кубов

1. Мы запускаем kube-apiserver со следующим набором аргументов:

--kubelet-client-certificate=<path>
--kubelet-client-key=<path>
--authorization-mode=Node,RBAC
--enable-admission-plugins=EventRateLimit
--secure-port =6443
--audit-log-path=/var/log/audit/kube-apiserver-audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
--service-account-lookup=true
--service-account-key-file=<path>
--etcd-certfile=<path>
--etcd-keyfile=<path>
--tls-cert-file=<path>
--tls-private-key-file=<path>
--client-ca-file=<path>
--etcd-cafile=<path>
--audit-policy-file=/etc/kubernetes/audit-policy/apiserver-audit-policy.yaml
--bind-address=0.0.0.0
--advertise-address=<IP>

Конфигурация TLS:

--tls-cert-file=<file>
--tls-private-key-file=<file>
--secure-port=6443
--cert-dir=<path>
--tls-cipher-suites=<TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305>
--tls-min-version =VersionTLS12
--requestheader-client-ca-file=<path>
--proxy-client-cert-file=<path>
--proxy-client-key-file=<path>

2. Controller manager:

--terminate-pod-gc-threshold=50
--profiling=false
--use-service-account-credentials=true
--service-account-private-key-file=<filename>
--root-ca-file=<path>

3. kube-scheduler:

--profiling=false
--bind-address=127.0.0.1

4. etcd:

 --cert-file=<path>
 --key-file=<path>
 --client-cert-auth=true
 --auto-tls=false
 --peer-cert-file=<path>
 --peer-key-file=<path>
 --peer-client-cert-auth=true
 --peer-auto-tls=false
 --trusted-ca-file=<path>
--etcd-cafile=<path>
--etcd-certfile=<path>
--etcd-keyfile=<path>

5. Kubelet:

 --anonymous-auth=false
 --client-ca-file=<path>
 --streaming-connection-idle-timeout=5m
 --make-iptables-util-chains=true
 --tls-cert-file=<path>
 --tls-private-key-file=<path>
 --rotate-certificates=true
 --tls-cipher-suites=<TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305>

Чтобы достичь этого, используя kubespray как ansible коллекцию, мне достаточно было добавить следующее в group_vars:

1. Файл k8s-cluster.yml

kube_network_plugin: <plugin>
kube_service_addresses: <service network cidr>
kube_pods_subnet: <pod network cidr>
kube_proxy_mode: ipvs
kube_proxy_strict_arp: true
dns_mode: coredns
enable_nodelocaldns: true
enable_nodelocaldns_secondary: false
nodelocaldns_ip: <nodelocaldns ip>
nodelocaldns_health_port: <port>
nodelocaldns_second_health_port: <port>
nodelocaldns_bind_metrics_host_ip: false
nodelocaldns_secondary_skew_seconds: 5
cluster_name: <cluster name>
container_manager: containerd

tls_cipher_suites:
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

# kubelet extra args

kubelet_make_iptables_util_chains: true
kubelet_streaming_connection_idle_timeout: "5m"
kubelet_rotate_certificates: true
kube_read_only_port: 0

kubelet_config_extra_args:
  tlsCertFile: "/var/lib/kubelet/pki/kubelet.crt"
  tlsPrivateKeyFile: "/var/lib/kubelet/pki/kubelet.key"
  authentication:
    x509:
      clientCAFile: "{{ kube_cert_dir }}/ca.crt"
    anonymous:
      enabled: false

2. kube_control_plane.yml

kube_scheduler_bind_address: 127.0.0.1
tls_min_version: VersionTLS12
kube_apiserver_admission_control_config_file: true
kube_apiserver_enable_admission_plugins:
  - EventRateLimit
kube_apiserver_admission_event_rate_limits:
  limit_namespace:
    type: Namespace
    qps: 50
    burst: 100
    cache_size: 2000
  limit_user:
    type: User
    qps: 50
    burst: 100

Многие параметры для других компонентов уже "из коробки" kubespray идут с правильными параметрами, достаточными для харденинга.

#k8s_sec