kubeadm快速部署kubernetes(1.26.4,HA)

2023/04/18 docker Cloud

当前版本的kubeadm已经原生支持部署HA模式集群,非常方便即可实现HA模式的kubernetes集群。本次部署基于Ubuntu20.04,并使用最新的docker版本:20.10.24,kubernetes适用1.26.x版本,本文采用1.26.4。

1 环境准备

准备了六台机器作安装测试工作,机器信息如下:

IP Name Role OS
172.16.2.1 Master01 Controller,etcd Ubuntu20.04
172.16.2.2 Master02 Controller,etcd Ubuntu20.04
172.16.2.3 Master03 Controller,etcd Ubuntu20.04
172.16.2.11 Node01 Compute Ubuntu20.04
172.16.2.12 Node02 Compute Ubuntu20.04
172.16.2.13 Node03 Compute Ubuntu20.04
172.16.2.251 Dns01 DNS Ubuntu20.04
172.16.2.252 Dns01 DNS Ubuntu20.04

注意:需要在/etc/hosts中配置本机的主机名解析,如Master01,添加:172.16.2.1 master01 。

关闭Ubuntu20.04自带的dns解析编辑/etc/systemd/resolved.conf,修改配置如下:

[Resolve]
DNS=114.114.114.114 8.8.8.8
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
DNSStubListener=no
#ReadEtcHosts=yes

创建软链接并删除现有的resolved.conf,重启服务:

ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

systemctl restart systemd-resolved.service

2 安装docker

apt update

apt install ca-certificates curl

install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

apt update

VERSION_STRING=5:20.10.24~3-0~ubuntu-focal
apt install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

配置docker使用systemd驱动,相比默认的cgroups更稳定。

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload && systemctl restart docker

Kubernetes自v1.24移除了对docker-shim的支持,而Docker默认又不支持CRI规范,因而二者将无法直接完成整合。为此,Mirantis和Docker联合创建了cri-dockerd项目,用于为Docker Engine提供一个能够支持到CRI规范的垫片,从而能够让Kubernetes基于CRI控制Docker。

curl -LO https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd_0.3.1.3-0.ubuntu-focal_amd64.deb 

apt install ./cri-dockerd_0.3.1.3-0.ubuntu-focal_amd64.deb

##确认服务启动情况
systemctl status cri-docker.service

配置cri-dockerd,编辑文件/usr/lib/systemd/system/cri-docker.service,修改配置参数:

ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

重启服务

systemctl daemon-reload && systemctl restart cri-docker.service

3 安装etcd集群

使用了docker-compose安装,当然,如果觉得麻烦,也可以直接docker run。

Master01节点的ETCD的docker-compose.yml:

version: "3.7"
services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.6
    command: etcd --name etcd-srv1 --data-dir=/var/etcd/calico-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://172.16.2.1:2379,http://172.16.2.1:2380 --initial-advertise-peer-urls http://172.16.2.1:2380 --listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster -initial-cluster "etcd-srv1=http://172.16.2.1:2380,etcd-srv2=http://172.16.2.2:2380,etcd-srv3=http://172.16.2.3:2380" -initial-cluster-state new
    network_mode: "host"
    restart: always
    stdin_open: true
    tty: true
    environment:
    - ETCDCTL_API=3
    volumes:
    - /store/etcd:/var/etcd

Master02节点的ETCD的docker-compose.yml:

version: "3.7"
services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.6
    command: etcd --name etcd-srv2 --data-dir=/var/etcd/calico-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://172.16.2.2:2379,http://172.16.2.2:2380 --initial-advertise-peer-urls http://172.16.2.2:2380 --listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster -initial-cluster "etcd-srv1=http://172.16.2.1:2380,etcd-srv2=http://172.16.2.2:2380,etcd-srv3=http://172.16.2.3:2380" -initial-cluster-state new
    network_mode: "host"
    restart: always
    stdin_open: true
    tty: true
    environment:
    - ETCDCTL_API=3
    volumes:
    - /store/etcd:/var/etcd

Master03节点的ETCD的docker-compose.yml:

version: "3.7"
services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.6
    command: etcd --name etcd-srv3 --data-dir=/var/etcd/calico-data --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://172.16.2.3:2379,http://172.16.2.3:2380 --initial-advertise-peer-urls http://172.16.2.3:2380 --listen-peer-urls http://0.0.0.0:2380 -initial-cluster-token etcd-cluster -initial-cluster "etcd-srv1=http://172.16.2.1:2380,etcd-srv2=http://172.16.2.2:2380,etcd-srv3=http://172.16.2.3:2380" -initial-cluster-state new
    network_mode: "host"
    restart: always
    stdin_open: true
    tty: true
    environment:
    - ETCDCTL_API=3
    volumes:
    - /store/etcd:/var/etcd

创建好docker-compose.yml文件和数据存储目录/store/etcd,使用命令docker compose up -d部署。

关于docker-compose的使用,可以参考:docker compose安装文档

4 安装k8s工具包

阿里源安装

curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

apt update
apt install -y kubelet=1.26.4-00 kubeadm=1.26.4-00 kubectl=1.26.4-00 ipvsadm ipset
apt-mark hold kubelet kubeadm kubectl ipvsadm docker-ce

5 启用ipvs模块

本方案中采用ipvs作为kube-proxy的转发机制,效率比iptables高很多,开启ipvs模块支持。

modprobe ip_vs && modprobe ip_vs_rr && modprobe ip_vs_wrr && modprobe ip_vs_sh

启用的ipvs相关模块重启机器后需要重启加载,为了避免麻烦,可以将加载模块配置在为开机启动(所有节点上都需要配置):

root@master01:~# vi /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
ip_vs_rr
ip_vs_wrr
ip_vs_sh
ip_vs

6 Api-Server负载均衡

配置负载均衡器对kube-apiserver进行负载均衡,可采用DNS轮询解析或者Haproxy(Nginx)反向代理实现负载均衡。

本文采用DNS轮询解析实现简单的负载均衡,在Dns01,Dns02节点上部署DNS。

1、添加/etc/dnsmasq/hosts.dnsmasq文件,添加域名解析

172.16.2.1 api.k8s.com
172.16.2.2 api.k8s.com
172.16.2.3 api.k8s.com

2、docker-compose部署dnsmasq服务:

version: "3"
services:
  dnsmasq:
    image: cloudnil/dnsmasq:2.76
    command: -q --log-facility=- --addn-hosts=/etc/hosts.dnsmasq
    network_mode: "host"
    volumes:
    - /root/dnsmasq/hosts.dnsmasq:/etc/hosts.dnsmasq
    cap_add:
    - NET_ADMIN
    restart: always
    stdin_open: true
    tty: true

3、在除了部署dnsmasq服务的其他所有节点上(包括Master和Node),配置DNS/etc/systemd/resolved.conf(同时也要先停用本机默认启动的DNS服务)

[Resolve]
DNS=172.16.2.251 172.16.2.252
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
DNSStubListener=no
#ReadEtcHosts=yes

重启使配置生效。

systemctl restart systemd-resolved.service

7 安装master节点

kubeadm配置文件kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///var/run/cri-dockerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
imageRepository: registry.aliyuncs.com/google_containers
etcd:
  external:
    endpoints:
    - http://172.16.2.1:2379
    - http://172.16.2.2:2379
    - http://172.16.2.3:2379
networking:
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.68.0.0/16
kubernetesVersion: v1.26.4
controlPlaneEndpoint: api.k8s.com:6443
apiServer:
  certSANs:
  - api.k8s.com
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved: 
  cpu: "0.25"
  memory: 128Mi
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
ipvs:
  minSyncPeriod: 1s
  #rr-轮询  wrr-加权轮询  sh-地址哈希
  scheduler: rr
  syncPeriod: 10s
mode: ipvs

说明:因为k8s.io在墙外,导致镜像无法获取,感谢阿里云提供了镜像仓库:registry.aliyuncs.com/google_containers,镜像下载需要点时间,也可以提前下载镜像:kubeadm config images pull --config kubeadm-config.yaml

相关镜像:

registry.aliyuncs.com/google_containers/kube-apiserver:v1.26.4
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.26.4
registry.aliyuncs.com/google_containers/kube-scheduler:v1.26.4
registry.aliyuncs.com/google_containers/kube-proxy:v1.26.4
registry.aliyuncs.com/google_containers/pause:3.9
registry.aliyuncs.com/google_containers/coredns:v1.9.3

master01初始化指令:

kubeadm init --config=kubeadm-config.yaml --upload-certs

如果镜像已经提前下载,安装过程大概30秒,输出结果如下:

[init] Using Kubernetes version: v1.26.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local yny2023141] and IPs [10.96.0.1 172.16.2.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.507426 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node yny2023141 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node yny2023141 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: xn39cs.mjlg0er3cqr8t5jd
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join api.k8s.com:6443 --token obrxsw.2wzhv0z6nfnc4nyr \
    --discovery-token-ca-cert-hash sha256:312a5bab2f699a4e701fb08ce65218b01aff7521fc962f35ad56e5d3214d3e6f \
    --control-plane --certificate-key fbf4fd608065cb4478a6269de8ca7402ceec7640209df73dbdef254e3d02efbd

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join api.k8s.com:6443 --token obrxsw.2wzhv0z6nfnc4nyr \
    --discovery-token-ca-cert-hash sha256:312a5bab2f699a4e701fb08ce65218b01aff7521fc962f35ad56e5d3214d3e6f

说明:certificate-key用于其他master节点获取证书文件时验证,有小时间为2小时,超过2小时候需要重新生成:

kubeadm init phase upload-certs --upload-certs

token是使用指令kubeadm token generate生成的,执行过程如有异常,用命令kubeadm reset初始化后重试,生成的token有效时间为24小时,超过24小时后需要重新使用命令kubeadm token create创建新的tokendiscovery-token-ca-cert-hash的值可以使用命令生成:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

加载管理员配置文件

方式一:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

方式二:

export KUBECONFIG=/etc/kubernetes/admin.conf

8 安装calico网络

网络组件选择很多,可以根据自己的需要选择calico、weave、flannel,calico性能最好,flannel的vxlan也不错,默认的UDP性能较差,weave的性能比较差,测试环境用下可以,生产环境不建议使用。calico的安装配置可以参考官方部署:点击查看

calico.yml:

---
# Source: calico/templates/calico-kube-controllers.yaml
# This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
---
# Source: calico/templates/calico-kube-controllers.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
---
# Source: calico/templates/calico-node.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system
---
# Source: calico/templates/calico-etcd-secrets.yaml
# The following contains k8s Secrets for use with a TLS enabled etcd cluster.
# For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # The keys below should be uncommented and the values populated with the base64
  # encoded contents of each file that would be associated with the TLS data.
  # Example command for encoding a file contents: cat <file> | base64 -w 0
  # etcd-key: null
  # etcd-cert: null
  # etcd-ca: null
---
# Source: calico/templates/calico-config.yaml
# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: "http://172.16.7.141:2379"
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: ""   # "/calico-secrets/etcd-ca"
  etcd_cert: "" # "/calico-secrets/etcd-cert"
  etcd_key: ""  # "/calico-secrets/etcd-key"
  # Typha is disabled.
  typha_service_name: "none"
  # Configure the backend to use.
  calico_backend: "bird"

  # Configure the MTU to use for workload interfaces and tunnels.
  # By default, MTU is auto-detected, and explicitly setting this field should not be required.
  # You can override auto-detection by providing a non-zero value.
  veth_mtu: "0"

  # The CNI network configuration to install on each node. The special
  # values in this config will be automatically populated.
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "log_file_path": "/var/log/calico/cni/cni.log",
          "etcd_endpoints": "__ETCD_ENDPOINTS__",
          "etcd_key_file": "__ETCD_KEY_FILE__",
          "etcd_cert_file": "__ETCD_CERT_FILE__",
          "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }
---
# Source: calico/templates/calico-kube-controllers-rbac.yaml
# Include a clusterrole for the kube-controllers component,
# and bind it to the calico-kube-controllers serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  # Pods are monitored for changing labels.
  # The node controller monitors Kubernetes nodes.
  # Namespace and serviceaccount labels are used for policy.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
      - serviceaccounts
    verbs:
      - watch
      - list
      - get
  # Watch for changes to Kubernetes NetworkPolicies.
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
---
# Source: calico/templates/calico-node-rbac.yaml
# Include a clusterrole for the calico-node DaemonSet,
# and bind it to the calico-node serviceaccount.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  # Used for creating service account tokens to be used by the CNI plugin
  - apiGroups: [""]
    resources:
      - serviceaccounts/token
    resourceNames:
      - calico-node
    verbs:
      - create
  # The CNI plugin needs to get pods, nodes, and namespaces.
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  # EndpointSlices are used for Service-based network policy rule
  # enforcement.
  - apiGroups: ["discovery.k8s.io"]
    resources:
      - endpointslices
    verbs:
      - watch 
      - list
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      # Used to discover service IPs for advertisement.
      - watch
      - list
  # Pod CIDR auto-detection on kubeadm needs access to config maps.
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      # Needed for clearing NodeNetworkUnavailable flag.
      - patch
---
# Source: calico/templates/calico-kube-controllers-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system
---
# Source: calico/templates/calico-node-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system
---
# Source: calico/templates/calico-node.yaml
# This manifest installs the calico-node container, as well
# as the CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        # Make sure calico-node gets scheduled on all nodes.
        - effect: NoSchedule
          operator: Exists
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
      # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
      terminationGracePeriodSeconds: 0
      priorityClassName: system-node-critical
      initContainers:
        # This container installs the CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: docker.io/calico/cni:v3.25.1
          imagePullPolicy: IfNotPresent
          command: ["/opt/cni/bin/install"]
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            # Name of the CNI config file to create.
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # CNI MTU Config variable
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Prevents the container from sleeping forever.
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
            - mountPath: /calico-secrets
              name: etcd-certs
          securityContext:
            privileged: true
        # This init container mounts the necessary filesystems needed by the BPF data plane
        # i.e. bpf at /sys/fs/bpf and cgroup2 at /run/calico/cgroup. Calico-node initialisation is executed
        # in best effort fashion, i.e. no failure for errors, to not disrupt pod creation in iptable mode.
        - name: "mount-bpffs"
          image: docker.io/calico/node:v3.25.1
          imagePullPolicy: IfNotPresent
          command: ["calico-node", "-init", "-best-effort"]
          volumeMounts:
            - mountPath: /sys/fs
              name: sys-fs
              # Bidirectional is required to ensure that the new mount we make at /sys/fs/bpf propagates to the host
              # so that it outlives the init container.
              mountPropagation: Bidirectional
            - mountPath: /var/run/calico
              name: var-run-calico
              # Bidirectional is required to ensure that the new mount we make at /run/calico/cgroup propagates to the host
              # so that it outlives the init container.
              mountPropagation: Bidirectional
            # Mount /proc/ from host which usually is an init program at /nodeproc. It's needed by mountns binary,
            # executed by calico-node, to mount root cgroup2 fs at /run/calico/cgroup to attach CTLB programs correctly.
            - mountPath: /nodeproc
              name: nodeproc
              readOnly: true
          securityContext:
            privileged: true
      containers:
        # Runs calico-node container on each Kubernetes node. This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: docker.io/calico/node:v3.25.1
          imagePullPolicy: IfNotPresent
          envFrom:
          - configMapRef:
              # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
              name: kubernetes-services-endpoint
              optional: true
          env:
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Set noderef for node controller.
            - name: CALICO_K8S_NODE_REF
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            # Enable or Disable VXLAN on the default IP pool.
            - name: CALICO_IPV4POOL_VXLAN
              value: "Never"
            # Enable or Disable VXLAN on the default IPv6 IP pool.
            - name: CALICO_IPV6POOL_VXLAN
              value: "Never"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the VXLAN tunnel device.
            - name: FELIX_VXLANMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # Set MTU for the Wireguard tunnel device.
            - name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.68.0.0/16"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/calico-node
                - -shutdown
          livenessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-live
              - -bird-live
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-ready
              - -bird-ready
            periodSeconds: 10
            timeoutSeconds: 10
          volumeMounts:
            # For maintaining CNI plugin API credentials.
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
              readOnly: false
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - mountPath: /calico-secrets
              name: etcd-certs
            - name: policysync
              mountPath: /var/run/nodeagent
            # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
            # parent directory.
            - name: bpffs
              mountPath: /sys/fs/bpf
            - name: cni-log-dir
              mountPath: /var/log/calico/cni
              readOnly: true
      volumes:
        # Used by calico-node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: sys-fs
          hostPath:
            path: /sys/fs/
            type: DirectoryOrCreate
        - name: bpffs
          hostPath:
            path: /sys/fs/bpf
            type: Directory
        # mount /proc at /nodeproc to be used by mount-bpffs initContainer to mount root cgroup2 fs.
        - name: nodeproc
          hostPath:
            path: /proc
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Used to access CNI logs.
        - name: cni-log-dir
          hostPath:
            path: /var/log/calico/cni
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0400
        # Used to create per-pod Unix Domain Sockets
        - name: policysync
          hostPath:
            type: DirectoryOrCreate
            path: /var/run/nodeagent
---
# Source: calico/templates/calico-kube-controllers.yaml
# See https://github.com/projectcalico/kube-controllers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        # Mark the pod as a critical add-on for rescheduling.
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      priorityClassName: system-cluster-critical
      # The controllers must run in the host network namespace so that
      # it isn't governed by policy that would prevent it from working.
      hostNetwork: true
      containers:
        - name: calico-kube-controllers
          image: docker.io/calico/kube-controllers:v3.25.1
          imagePullPolicy: IfNotPresent
          env:
            # The location of the etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Location of the CA certificate for etcd.
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key
            # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            # Choose which controllers to run.
            - name: ENABLED_CONTROLLERS
              value: policy,namespace,serviceaccount,workloadendpoint,node
          volumeMounts:
            # Mount in the etcd TLS secrets.
            - mountPath: /calico-secrets
              name: etcd-certs
          livenessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -l
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -r
            periodSeconds: 10
      volumes:
        # Mount in the etcd TLS secrets with mode 400.
        # See https://kubernetes.io/docs/concepts/configuration/secret/
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
            defaultMode: 0440

执行命令:

kubectl apply -f calico.yml

检查各节点组件运行状态:

root@master01:~# kubectl get nodes
NAME       STATUS   ROLES    AGE   VERSION
master01   Ready    master   13m   v1.26.4

root@master01:~# kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-697d964cc4-p8jcn   1/1     Running   0          32s
calico-node-wg9l4                          1/1     Running   0          32s
coredns-89cc84847-l48q8                    1/1     Running   0          13m
coredns-89cc84847-mf5nr                    1/1     Running   0          13m
kube-apiserver-master01                    1/1     Running   0          12m
kube-controller-manager-master01           1/1     Running   0          12m
kube-proxy-9l287                           1/1     Running   0          13m
kube-scheduler-master01                    1/1     Running   0          12m

9 安装Master02和Master03节点

在master02,master03上执行加入集群命令:

kubeadm join api.k8s.com:6443 --token obrxsw.2wzhv0z6nfnc4nyr \
    --discovery-token-ca-cert-hash sha256:312a5bab2f699a4e701fb08ce65218b01aff7521fc962f35ad56e5d3214d3e6f \
    --control-plane --certificate-key fbf4fd608065cb4478a6269de8ca7402ceec7640209df73dbdef254e3d02efbd \
    --cri-socket unix:///var/run/cri-dockerd.sock

可以查看下各节点及组件运行状态:

root@master01:~# kubectl get nodes
NAME       STATUS   ROLES    AGE     VERSION
master01   Ready    master   25m     v1.26.4
master02   Ready    master   8m6s    v1.26.4
master03   Ready    master   7m33s   v1.26.4
root@master01:~# kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-697d964cc4-p8jcn   1/1     Running   0          12m
calico-node-lvnl8                          1/1     Running   0          7m35s
calico-node-p8h5z                          1/1     Running   0          8m9s
calico-node-wg9l4                          1/1     Running   0          12m
coredns-89cc84847-l48q8                    1/1     Running   0          25m
coredns-89cc84847-mf5nr                    1/1     Running   0          25m
kube-apiserver-master01                    1/1     Running   0          24m
kube-apiserver-master02                    1/1     Running   0          8m9s
kube-apiserver-master03                    1/1     Running   0          7m35s
kube-controller-manager-master01           1/1     Running   0          24m
kube-controller-manager-master02           1/1     Running   0          8m9s
kube-controller-manager-master03           1/1     Running   0          7m35s
kube-proxy-9l287                           1/1     Running   0          25m
kube-proxy-jmsfb                           1/1     Running   0          8m9s
kube-proxy-wzh62                           1/1     Running   0          7m35s
kube-scheduler-master01                    1/1     Running   0          24m
kube-scheduler-master02                    1/1     Running   0          8m9s
kube-scheduler-master03                    1/1     Running   0          7m35s

10 安装Node节点

Master节点安装好了Node节点就简单了,在各个Node节点上执行。

kubeadm join api.k8s.com:6443 --token obrxsw.2wzhv0z6nfnc4nyr \
    --discovery-token-ca-cert-hash sha256:312a5bab2f699a4e701fb08ce65218b01aff7521fc962f35ad56e5d3214d3e6f \
    --cri-socket unix:///run/cri-dockerd.sock

11 DNS集群部署

删除原单点coredns

kubectl delete deploy coredns -n kube-system

部署多实例的coredns集群,参考配置coredns.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: k8s-app
                    operator: In
                    values:
                    - kube-dns
                topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: registry.aliyuncs.com/google_containers/coredns:1.9.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/coredns
          name: config-volume
          readOnly: true
      dnsPolicy: Default
      nodeSelector:
        beta.kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: coredns
      serviceAccountName: coredns
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: Corefile
            path: Corefile
          name: coredns
        name: config-volume

12 部署Metrics-Server

kubernetesv1.11以后不再支持通过heaspter采集监控数据,支持新的监控数据采集组件metrics-server,比heaspter轻量很多,也不做数据的持久化存储,提供实时的监控数据查询还是很好用的。

获取部署文档,点击这里

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        image: registry.aliyuncs.com/google_containers/metrics-server:v0.7.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 10250
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

执行部署命令:

kubectl apply -f components.yaml

查看监控数据:

root@master01:~# kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master01   153m         8%     1748Mi          46%       
master02   108m         6%     1250Mi          33%       
master03   91m          5%     1499Mi          40%       
node01     256m         7%     1047Mi          13%       
node02     196m         5%     976Mi           10%       
node03     206m         5%     907Mi           12%       

13 部署Dashboard

推荐skooner,比官方的dashboard顺眼多了,项目地址:https://github.com/skooner-k8s/skooner

kind: Deployment
apiVersion: apps/v1
metadata:
  name: skooner
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: skooner
  template:
    metadata:
      labels:
        k8s-app: skooner
    spec:
      containers:
      - name: skooner
        image: ghcr.io/skooner-k8s/skooner:stable
        ports:
        - containerPort: 4654
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /
            port: 4654
          initialDelaySeconds: 30
          timeoutSeconds: 30
      nodeSelector:
        'kubernetes.io/os': linux
---
kind: Service
apiVersion: v1
metadata:
  name: skooner-svc
  namespace: kube-system
spec:
  ports:
    - port: 80
      targetPort: 4654
  selector:
    k8s-app: skooner
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: skooner-ing
  labels:
    k8s-app: skooner
  namespace: kube-system
spec:
  rules:
  - host: console.cloudnil.com
    http:
      paths:
      - path: /
        pathType: "Prefix"
        backend:
          service:
            name: skooner-svc
            port: 
              number: 80

采用Token方式进行登陆:

kubectl create serviceaccount skooner-sa -n kube-system

kubectl create clusterrolebinding skooner-sa --clusterrole=cluster-admin --serviceaccount=default:skooner-sa -n kube-system

# K8S 1.22+版本需单独创建token
kubectl create token skooner-sa -n kube-system

# K8S 1.21-版本直接查看Token
kubectl get secrets -n kube-system |grep skooner-sa|awk '{print $1}'| xargs kubectl describe secret -n kube-system

暴露本地端口到Master01访问测试:

#直接暴露Pod端口到本地
kubectl port-forward  pod/k8dash-fc78cd558-thdrv --address 0.0.0.0 12345:4654 -n kube-system

#直接暴露Service端口到本地
kubectl port-forward  svc/k8dash-svc --address 0.0.0.0 12345:80 -n kube-system

访问地址:http://172.16.2.1:12345

14 服务暴露到公网

kubernetes中的Service暴露到外部有三种方式,分别是:

  • LoadBlancer Service
  • NodePort Service
  • Ingress

LoadBlancer Service是kubernetes深度结合云平台的一个组件;当使用LoadBlancer Service暴露服务时,实际上是通过向底层云平台申请创建一个负载均衡器来向外暴露服务;目前LoadBlancer Service支持的云平台已经相对完善,比如国外的GCE、DigitalOcean,国内的 阿里云,私有云 Openstack 等等,由于LoadBlancer Service深度结合了云平台,所以只能在一些云平台上来使用。

NodePort Service顾名思义,实质上就是通过在集群的每个node上暴露一个端口,然后将这个端口映射到某个具体的service来实现的,虽然每个node的端口有很多(0~65535),但是由于安全性和易用性(服务多了就乱了,还有端口冲突问题)实际使用可能并不多。

Ingress可以实现使用nginx等开源的反向代理负载均衡器实现对外暴露服务,可以理解Ingress就是用于配置域名转发的一个东西,在nginx中就类似upstream,它与ingress-controller结合使用,通过ingress-controller监控到pod及service的变化,动态地将ingress中的转发信息写到诸如nginx、apache、haproxy等组件中实现方向代理和负载均衡。

15 部署Nginx-ingress-controller

说明:Nginx-controller和Traefik-controller二选一。

Nginx-ingress-controller是kubernetes官方提供的集成了Ingress-controller和Nginx的一个docker镜像。

配置可参考Nginx-ingress-controller官网:https://kubernetes.github.io/ingress-nginx

16 部署Traefik-ingress-controller

说明:Nginx-ingress-controller和Traefik-ingress-controller二选一。

Traefik-ingress-controller是一个为了让部署微服务更加便捷而诞生的现代HTTP反向代理、负载均衡工具。 它支持多种后台 (Docker, Swarm, Kubernetes, Marathon, Mesos, Consul, Etcd, Zookeeper, BoltDB, Rest API, file…) 来自动化、动态的应用它的配置文件设置。

相比起NginxTraefik更轻量,速度更快,配置更简单,不过功能及拓展性不如Nginx丰富多样,各位可根据实际情况选择。

本次部署中,将Traefik-ingress部署到master01、master02、master03上,监听宿主机的80端口:

Namespace 和 RBAC角色配置:

apiVersion: v1
kind: Namespace
metadata:
  name: traefik
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-deploy
  namespace: traefik
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: traefik-ingress-deploy
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - middlewaretcps
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
      - serverstransports
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: traefik-ingress-deploy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-deploy
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-deploy
    namespace: traefik

CRD配置:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: ingressroutes.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: IngressRoute
    listKind: IngressRouteList
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: IngressRoute is the CRD implementation of a Traefik HTTP Router.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IngressRouteSpec defines the desired state of IngressRoute.
            properties:
              entryPoints:
                description: 'EntryPoints defines the list of entry point names to
                  bind to. Entry points have to be configured in the static configuration.
                  More info: https://doc.traefik.io/traefik/v2.9/routing/entrypoints/
                  Default: all.'
                items:
                  type: string
                type: array
              routes:
                description: Routes defines the list of routes.
                items:
                  description: Route holds the HTTP route configuration.
                  properties:
                    kind:
                      description: Kind defines the kind of the route. Rule is the
                        only supported kind.
                      enum:
                      - Rule
                      type: string
                    match:
                      description: 'Match defines the router''s rule. More info: https://doc.traefik.io/traefik/v2.9/routing/routers/#rule'
                      type: string
                    middlewares:
                      description: 'Middlewares defines the list of references to
                        Middleware resources. More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#kind-middleware'
                      items:
                        description: MiddlewareRef is a reference to a Middleware
                          resource.
                        properties:
                          name:
                            description: Name defines the name of the referenced Middleware
                              resource.
                            type: string
                          namespace:
                            description: Namespace defines the namespace of the referenced
                              Middleware resource.
                            type: string
                        required:
                        - name
                        type: object
                      type: array
                    priority:
                      description: 'Priority defines the router''s priority. More
                        info: https://doc.traefik.io/traefik/v2.9/routing/routers/#priority'
                      type: integer
                    services:
                      description: Services defines the list of Service. It can contain
                        any combination of TraefikService and/or reference to a Kubernetes
                        Service.
                      items:
                        description: Service defines an upstream HTTP service to proxy
                          traffic to.
                        properties:
                          kind:
                            description: Kind defines the kind of the Service.
                            enum:
                            - Service
                            - TraefikService
                            type: string
                          name:
                            description: Name defines the name of the referenced Kubernetes
                              Service or TraefikService. The differentiation between
                              the two is specified in the Kind field.
                            type: string
                          namespace:
                            description: Namespace defines the namespace of the referenced
                              Kubernetes Service or TraefikService.
                            type: string
                          passHostHeader:
                            description: PassHostHeader defines whether the client
                              Host header is forwarded to the upstream Kubernetes
                              Service. By default, passHostHeader is true.
                            type: boolean
                          port:
                            anyOf:
                            - type: integer
                            - type: string
                            description: Port defines the port of a Kubernetes Service.
                              This can be a reference to a named port.
                            x-kubernetes-int-or-string: true
                          responseForwarding:
                            description: ResponseForwarding defines how Traefik forwards
                              the response from the upstream Kubernetes Service to
                              the client.
                            properties:
                              flushInterval:
                                description: 'FlushInterval defines the interval,
                                  in milliseconds, in between flushes to the client
                                  while copying the response body. A negative value
                                  means to flush immediately after each write to the
                                  client. This configuration is ignored when ReverseProxy
                                  recognizes a response as a streaming response; for
                                  such responses, writes are flushed to the client
                                  immediately. Default: 100ms'
                                type: string
                            type: object
                          scheme:
                            description: Scheme defines the scheme to use for the
                              request to the upstream Kubernetes Service. It defaults
                              to https when Kubernetes Service port is 443, http otherwise.
                            type: string
                          serversTransport:
                            description: ServersTransport defines the name of ServersTransport
                              resource to use. It allows to configure the transport
                              between Traefik and your servers. Can only be used on
                              a Kubernetes Service.
                            type: string
                          sticky:
                            description: 'Sticky defines the sticky sessions configuration.
                              More info: https://doc.traefik.io/traefik/v2.9/routing/services/#sticky-sessions'
                            properties:
                              cookie:
                                description: Cookie defines the sticky cookie configuration.
                                properties:
                                  httpOnly:
                                    description: HTTPOnly defines whether the cookie
                                      can be accessed by client-side APIs, such as
                                      JavaScript.
                                    type: boolean
                                  name:
                                    description: Name defines the Cookie name.
                                    type: string
                                  sameSite:
                                    description: 'SameSite defines the same site policy.
                                      More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite'
                                    type: string
                                  secure:
                                    description: Secure defines whether the cookie
                                      can only be transmitted over an encrypted connection
                                      (i.e. HTTPS).
                                    type: boolean
                                type: object
                            type: object
                          strategy:
                            description: Strategy defines the load balancing strategy
                              between the servers. RoundRobin is the only supported
                              value at the moment.
                            type: string
                          weight:
                            description: Weight defines the weight and should only
                              be specified when Name references a TraefikService object
                              (and to be precise, one that embeds a Weighted Round
                              Robin).
                            type: integer
                        required:
                        - name
                        type: object
                      type: array
                  required:
                  - kind
                  - match
                  type: object
                type: array
              tls:
                description: 'TLS defines the TLS configuration. More info: https://doc.traefik.io/traefik/v2.9/routing/routers/#tls'
                properties:
                  certResolver:
                    description: 'CertResolver defines the name of the certificate
                      resolver to use. Cert resolvers have to be configured in the
                      static configuration. More info: https://doc.traefik.io/traefik/v2.9/https/acme/#certificate-resolvers'
                    type: string
                  domains:
                    description: 'Domains defines the list of domains that will be
                      used to issue certificates. More info: https://doc.traefik.io/traefik/v2.9/routing/routers/#domains'
                    items:
                      description: Domain holds a domain name with SANs.
                      properties:
                        main:
                          description: Main defines the main domain name.
                          type: string
                        sans:
                          description: SANs defines the subject alternative domain
                            names.
                          items:
                            type: string
                          type: array
                      type: object
                    type: array
                  options:
                    description: 'Options defines the reference to a TLSOption, that
                      specifies the parameters of the TLS connection. If not defined,
                      the `default` TLSOption is used. More info: https://doc.traefik.io/traefik/v2.9/https/tls/#tls-options'
                    properties:
                      name:
                        description: 'Name defines the name of the referenced TLSOption.
                          More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#kind-tlsoption'
                        type: string
                      namespace:
                        description: 'Namespace defines the namespace of the referenced
                          TLSOption. More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#kind-tlsoption'
                        type: string
                    required:
                    - name
                    type: object
                  secretName:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    type: string
                  store:
                    description: Store defines the reference to the TLSStore, that
                      will be used to store certificates. Please note that only `default`
                      TLSStore can be used.
                    properties:
                      name:
                        description: 'Name defines the name of the referenced TLSStore.
                          More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#kind-tlsstore'
                        type: string
                      namespace:
                        description: 'Namespace defines the namespace of the referenced
                          TLSStore. More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#kind-tlsstore'
                        type: string
                    required:
                    - name
                    type: object
                type: object
            required:
            - routes
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: ingressroutetcps.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: IngressRouteTCP
    listKind: IngressRouteTCPList
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: IngressRouteTCP is the CRD implementation of a Traefik TCP Router.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IngressRouteTCPSpec defines the desired state of IngressRouteTCP.
            properties:
              entryPoints:
                description: 'EntryPoints defines the list of entry point names to
                  bind to. Entry points have to be configured in the static configuration.
                  More info: https://doc.traefik.io/traefik/v2.9/routing/entrypoints/
                  Default: all.'
                items:
                  type: string
                type: array
              routes:
                description: Routes defines the list of routes.
                items:
                  description: RouteTCP holds the TCP route configuration.
                  properties:
                    match:
                      description: 'Match defines the router''s rule. More info: https://doc.traefik.io/traefik/v2.9/routing/routers/#rule_1'
                      type: string
                    middlewares:
                      description: Middlewares defines the list of references to MiddlewareTCP
                        resources.
                      items:
                        description: ObjectReference is a generic reference to a Traefik
                          resource.
                        properties:
                          name:
                            description: Name defines the name of the referenced Traefik
                              resource.
                            type: string
                          namespace:
                            description: Namespace defines the namespace of the referenced
                              Traefik resource.
                            type: string
                        required:
                        - name
                        type: object
                      type: array
                    priority:
                      description: 'Priority defines the router''s priority. More
                        info: https://doc.traefik.io/traefik/v2.9/routing/routers/#priority_1'
                      type: integer
                    services:
                      description: Services defines the list of TCP services.
                      items:
                        description: ServiceTCP defines an upstream TCP service to
                          proxy traffic to.
                        properties:
                          name:
                            description: Name defines the name of the referenced Kubernetes
                              Service.
                            type: string
                          namespace:
                            description: Namespace defines the namespace of the referenced
                              Kubernetes Service.
                            type: string
                          port:
                            anyOf:
                            - type: integer
                            - type: string
                            description: Port defines the port of a Kubernetes Service.
                              This can be a reference to a named port.
                            x-kubernetes-int-or-string: true
                          proxyProtocol:
                            description: 'ProxyProtocol defines the PROXY protocol
                              configuration. More info: https://doc.traefik.io/traefik/v2.9/routing/services/#proxy-protocol'
                            properties:
                              version:
                                description: Version defines the PROXY Protocol version
                                  to use.
                                type: integer
                            type: object
                          terminationDelay:
                            description: TerminationDelay defines the deadline that
                              the proxy sets, after one of its connected peers indicates
                              it has closed the writing capability of its connection,
                              to close the reading capability as well, hence fully
                              terminating the connection. It is a duration in milliseconds,
                              defaulting to 100. A negative value means an infinite
                              deadline (i.e. the reading capability is never closed).
                            type: integer
                          weight:
                            description: Weight defines the weight used when balancing
                              requests between multiple Kubernetes Service.
                            type: integer
                        required:
                        - name
                        - port
                        type: object
                      type: array
                  required:
                  - match
                  type: object
                type: array
              tls:
                description: 'TLS defines the TLS configuration on a layer 4 / TCP
                  Route. More info: https://doc.traefik.io/traefik/v2.9/routing/routers/#tls_1'
                properties:
                  certResolver:
                    description: 'CertResolver defines the name of the certificate
                      resolver to use. Cert resolvers have to be configured in the
                      static configuration. More info: https://doc.traefik.io/traefik/v2.9/https/acme/#certificate-resolvers'
                    type: string
                  domains:
                    description: 'Domains defines the list of domains that will be
                      used to issue certificates. More info: https://doc.traefik.io/traefik/v2.9/routing/routers/#domains'
                    items:
                      description: Domain holds a domain name with SANs.
                      properties:
                        main:
                          description: Main defines the main domain name.
                          type: string
                        sans:
                          description: SANs defines the subject alternative domain
                            names.
                          items:
                            type: string
                          type: array
                      type: object
                    type: array
                  options:
                    description: 'Options defines the reference to a TLSOption, that
                      specifies the parameters of the TLS connection. If not defined,
                      the `default` TLSOption is used. More info: https://doc.traefik.io/traefik/v2.9/https/tls/#tls-options'
                    properties:
                      name:
                        description: Name defines the name of the referenced Traefik
                          resource.
                        type: string
                      namespace:
                        description: Namespace defines the namespace of the referenced
                          Traefik resource.
                        type: string
                    required:
                    - name
                    type: object
                  passthrough:
                    description: Passthrough defines whether a TLS router will terminate
                      the TLS connection.
                    type: boolean
                  secretName:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    type: string
                  store:
                    description: Store defines the reference to the TLSStore, that
                      will be used to store certificates. Please note that only `default`
                      TLSStore can be used.
                    properties:
                      name:
                        description: Name defines the name of the referenced Traefik
                          resource.
                        type: string
                      namespace:
                        description: Namespace defines the namespace of the referenced
                          Traefik resource.
                        type: string
                    required:
                    - name
                    type: object
                type: object
            required:
            - routes
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: ingressrouteudps.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: IngressRouteUDP
    listKind: IngressRouteUDPList
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: IngressRouteUDP is a CRD implementation of a Traefik UDP Router.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: IngressRouteUDPSpec defines the desired state of a IngressRouteUDP.
            properties:
              entryPoints:
                description: 'EntryPoints defines the list of entry point names to
                  bind to. Entry points have to be configured in the static configuration.
                  More info: https://doc.traefik.io/traefik/v2.9/routing/entrypoints/
                  Default: all.'
                items:
                  type: string
                type: array
              routes:
                description: Routes defines the list of routes.
                items:
                  description: RouteUDP holds the UDP route configuration.
                  properties:
                    services:
                      description: Services defines the list of UDP services.
                      items:
                        description: ServiceUDP defines an upstream UDP service to
                          proxy traffic to.
                        properties:
                          name:
                            description: Name defines the name of the referenced Kubernetes
                              Service.
                            type: string
                          namespace:
                            description: Namespace defines the namespace of the referenced
                              Kubernetes Service.
                            type: string
                          port:
                            anyOf:
                            - type: integer
                            - type: string
                            description: Port defines the port of a Kubernetes Service.
                              This can be a reference to a named port.
                            x-kubernetes-int-or-string: true
                          weight:
                            description: Weight defines the weight used when balancing
                              requests between multiple Kubernetes Service.
                            type: integer
                        required:
                        - name
                        - port
                        type: object
                      type: array
                  type: object
                type: array
            required:
            - routes
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: Middleware
    listKind: MiddlewareList
    plural: middlewares
    singular: middleware
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: 'Middleware is the CRD implementation of a Traefik Middleware.
          More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/overview/'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: MiddlewareSpec defines the desired state of a Middleware.
            properties:
              addPrefix:
                description: 'AddPrefix holds the add prefix middleware configuration.
                  This middleware updates the path of a request before forwarding
                  it. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/addprefix/'
                properties:
                  prefix:
                    description: Prefix is the string to add before the current path
                      in the requested URL. It should include a leading slash (/).
                    type: string
                type: object
              basicAuth:
                description: 'BasicAuth holds the basic auth middleware configuration.
                  This middleware restricts access to your services to known users.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/basicauth/'
                properties:
                  headerField:
                    description: 'HeaderField defines a header field to store the
                      authenticated user. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/basicauth/#headerfield'
                    type: string
                  realm:
                    description: 'Realm allows the protected resources on a server
                      to be partitioned into a set of protection spaces, each with
                      its own authentication scheme. Default: traefik.'
                    type: string
                  removeHeader:
                    description: 'RemoveHeader sets the removeHeader option to true
                      to remove the authorization header before forwarding the request
                      to your service. Default: false.'
                    type: boolean
                  secret:
                    description: Secret is the name of the referenced Kubernetes Secret
                      containing user credentials.
                    type: string
                type: object
              buffering:
                description: 'Buffering holds the buffering middleware configuration.
                  This middleware retries or limits the size of requests that can
                  be forwarded to backends. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/buffering/#maxrequestbodybytes'
                properties:
                  maxRequestBodyBytes:
                    description: 'MaxRequestBodyBytes defines the maximum allowed
                      body size for the request (in bytes). If the request exceeds
                      the allowed size, it is not forwarded to the service, and the
                      client gets a 413 (Request Entity Too Large) response. Default:
                      0 (no maximum).'
                    format: int64
                    type: integer
                  maxResponseBodyBytes:
                    description: 'MaxResponseBodyBytes defines the maximum allowed
                      response size from the service (in bytes). If the response exceeds
                      the allowed size, it is not forwarded to the client. The client
                      gets a 500 (Internal Server Error) response instead. Default:
                      0 (no maximum).'
                    format: int64
                    type: integer
                  memRequestBodyBytes:
                    description: 'MemRequestBodyBytes defines the threshold (in bytes)
                      from which the request will be buffered on disk instead of in
                      memory. Default: 1048576 (1Mi).'
                    format: int64
                    type: integer
                  memResponseBodyBytes:
                    description: 'MemResponseBodyBytes defines the threshold (in bytes)
                      from which the response will be buffered on disk instead of
                      in memory. Default: 1048576 (1Mi).'
                    format: int64
                    type: integer
                  retryExpression:
                    description: 'RetryExpression defines the retry conditions. It
                      is a logical combination of functions with operators AND (&&)
                      and OR (||). More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/buffering/#retryexpression'
                    type: string
                type: object
              chain:
                description: 'Chain holds the configuration of the chain middleware.
                  This middleware enables to define reusable combinations of other
                  pieces of middleware. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/chain/'
                properties:
                  middlewares:
                    description: Middlewares is the list of MiddlewareRef which composes
                      the chain.
                    items:
                      description: MiddlewareRef is a reference to a Middleware resource.
                      properties:
                        name:
                          description: Name defines the name of the referenced Middleware
                            resource.
                          type: string
                        namespace:
                          description: Namespace defines the namespace of the referenced
                            Middleware resource.
                          type: string
                      required:
                      - name
                      type: object
                    type: array
                type: object
              circuitBreaker:
                description: CircuitBreaker holds the circuit breaker configuration.
                properties:
                  checkPeriod:
                    anyOf:
                    - type: integer
                    - type: string
                    description: CheckPeriod is the interval between successive checks
                      of the circuit breaker condition (when in standby state).
                    x-kubernetes-int-or-string: true
                  expression:
                    description: Expression is the condition that triggers the tripped
                      state.
                    type: string
                  fallbackDuration:
                    anyOf:
                    - type: integer
                    - type: string
                    description: FallbackDuration is the duration for which the circuit
                      breaker will wait before trying to recover (from a tripped state).
                    x-kubernetes-int-or-string: true
                  recoveryDuration:
                    anyOf:
                    - type: integer
                    - type: string
                    description: RecoveryDuration is the duration for which the circuit
                      breaker will try to recover (as soon as it is in recovering
                      state).
                    x-kubernetes-int-or-string: true
                type: object
              compress:
                description: 'Compress holds the compress middleware configuration.
                  This middleware compresses responses before sending them to the
                  client, using gzip compression. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/compress/'
                properties:
                  excludedContentTypes:
                    description: ExcludedContentTypes defines the list of content
                      types to compare the Content-Type header of the incoming requests
                      and responses before compressing.
                    items:
                      type: string
                    type: array
                  minResponseBodyBytes:
                    description: 'MinResponseBodyBytes defines the minimum amount
                      of bytes a response body must have to be compressed. Default:
                      1024.'
                    type: integer
                type: object
              contentType:
                description: ContentType holds the content-type middleware configuration.
                  This middleware exists to enable the correct behavior until at least
                  the default one can be changed in a future version.
                properties:
                  autoDetect:
                    description: AutoDetect specifies whether to let the `Content-Type`
                      header, if it has not been set by the backend, be automatically
                      set to a value derived from the contents of the response. As
                      a proxy, the default behavior should be to leave the header
                      alone, regardless of what the backend did with it. However,
                      the historic default was to always auto-detect and set the header
                      if it was nil, and it is going to be kept that way in order
                      to support users currently relying on it.
                    type: boolean
                type: object
              digestAuth:
                description: 'DigestAuth holds the digest auth middleware configuration.
                  This middleware restricts access to your services to known users.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/digestauth/'
                properties:
                  headerField:
                    description: 'HeaderField defines a header field to store the
                      authenticated user. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/basicauth/#headerfield'
                    type: string
                  realm:
                    description: 'Realm allows the protected resources on a server
                      to be partitioned into a set of protection spaces, each with
                      its own authentication scheme. Default: traefik.'
                    type: string
                  removeHeader:
                    description: RemoveHeader defines whether to remove the authorization
                      header before forwarding the request to the backend.
                    type: boolean
                  secret:
                    description: Secret is the name of the referenced Kubernetes Secret
                      containing user credentials.
                    type: string
                type: object
              errors:
                description: 'ErrorPage holds the custom error middleware configuration.
                  This middleware returns a custom page in lieu of the default, according
                  to configured ranges of HTTP Status codes. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/errorpages/'
                properties:
                  query:
                    description: Query defines the URL for the error page (hosted
                      by service). The {status} variable can be used in order to insert
                      the status code in the URL.
                    type: string
                  service:
                    description: 'Service defines the reference to a Kubernetes Service
                      that will serve the error page. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/errorpages/#service'
                    properties:
                      kind:
                        description: Kind defines the kind of the Service.
                        enum:
                        - Service
                        - TraefikService
                        type: string
                      name:
                        description: Name defines the name of the referenced Kubernetes
                          Service or TraefikService. The differentiation between the
                          two is specified in the Kind field.
                        type: string
                      namespace:
                        description: Namespace defines the namespace of the referenced
                          Kubernetes Service or TraefikService.
                        type: string
                      passHostHeader:
                        description: PassHostHeader defines whether the client Host
                          header is forwarded to the upstream Kubernetes Service.
                          By default, passHostHeader is true.
                        type: boolean
                      port:
                        anyOf:
                        - type: integer
                        - type: string
                        description: Port defines the port of a Kubernetes Service.
                          This can be a reference to a named port.
                        x-kubernetes-int-or-string: true
                      responseForwarding:
                        description: ResponseForwarding defines how Traefik forwards
                          the response from the upstream Kubernetes Service to the
                          client.
                        properties:
                          flushInterval:
                            description: 'FlushInterval defines the interval, in milliseconds,
                              in between flushes to the client while copying the response
                              body. A negative value means to flush immediately after
                              each write to the client. This configuration is ignored
                              when ReverseProxy recognizes a response as a streaming
                              response; for such responses, writes are flushed to
                              the client immediately. Default: 100ms'
                            type: string
                        type: object
                      scheme:
                        description: Scheme defines the scheme to use for the request
                          to the upstream Kubernetes Service. It defaults to https
                          when Kubernetes Service port is 443, http otherwise.
                        type: string
                      serversTransport:
                        description: ServersTransport defines the name of ServersTransport
                          resource to use. It allows to configure the transport between
                          Traefik and your servers. Can only be used on a Kubernetes
                          Service.
                        type: string
                      sticky:
                        description: 'Sticky defines the sticky sessions configuration.
                          More info: https://doc.traefik.io/traefik/v2.9/routing/services/#sticky-sessions'
                        properties:
                          cookie:
                            description: Cookie defines the sticky cookie configuration.
                            properties:
                              httpOnly:
                                description: HTTPOnly defines whether the cookie can
                                  be accessed by client-side APIs, such as JavaScript.
                                type: boolean
                              name:
                                description: Name defines the Cookie name.
                                type: string
                              sameSite:
                                description: 'SameSite defines the same site policy.
                                  More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite'
                                type: string
                              secure:
                                description: Secure defines whether the cookie can
                                  only be transmitted over an encrypted connection
                                  (i.e. HTTPS).
                                type: boolean
                            type: object
                        type: object
                      strategy:
                        description: Strategy defines the load balancing strategy
                          between the servers. RoundRobin is the only supported value
                          at the moment.
                        type: string
                      weight:
                        description: Weight defines the weight and should only be
                          specified when Name references a TraefikService object (and
                          to be precise, one that embeds a Weighted Round Robin).
                        type: integer
                    required:
                    - name
                    type: object
                  status:
                    description: Status defines which status or range of statuses
                      should result in an error page. It can be either a status code
                      as a number (500), as multiple comma-separated numbers (500,502),
                      as ranges by separating two codes with a dash (500-599), or
                      a combination of the two (404,418,500-599).
                    items:
                      type: string
                    type: array
                type: object
              forwardAuth:
                description: 'ForwardAuth holds the forward auth middleware configuration.
                  This middleware delegates the request authentication to a Service.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/forwardauth/'
                properties:
                  address:
                    description: Address defines the authentication server address.
                    type: string
                  authRequestHeaders:
                    description: AuthRequestHeaders defines the list of the headers
                      to copy from the request to the authentication server. If not
                      set or empty then all request headers are passed.
                    items:
                      type: string
                    type: array
                  authResponseHeaders:
                    description: AuthResponseHeaders defines the list of headers to
                      copy from the authentication server response and set on forwarded
                      request, replacing any existing conflicting headers.
                    items:
                      type: string
                    type: array
                  authResponseHeadersRegex:
                    description: 'AuthResponseHeadersRegex defines the regex to match
                      headers to copy from the authentication server response and
                      set on forwarded request, after stripping all headers that match
                      the regex. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/forwardauth/#authresponseheadersregex'
                    type: string
                  tls:
                    description: TLS defines the configuration used to secure the
                      connection to the authentication server.
                    properties:
                      caOptional:
                        type: boolean
                      caSecret:
                        description: CASecret is the name of the referenced Kubernetes
                          Secret containing the CA to validate the server certificate.
                          The CA certificate is extracted from key `tls.ca` or `ca.crt`.
                        type: string
                      certSecret:
                        description: CertSecret is the name of the referenced Kubernetes
                          Secret containing the client certificate. The client certificate
                          is extracted from the keys `tls.crt` and `tls.key`.
                        type: string
                      insecureSkipVerify:
                        description: InsecureSkipVerify defines whether the server
                          certificates should be validated.
                        type: boolean
                    type: object
                  trustForwardHeader:
                    description: 'TrustForwardHeader defines whether to trust (ie:
                      forward) all X-Forwarded-* headers.'
                    type: boolean
                type: object
              headers:
                description: 'Headers holds the headers middleware configuration.
                  This middleware manages the requests and responses headers. More
                  info: https://doc.traefik.io/traefik/v2.9/middlewares/http/headers/#customrequestheaders'
                properties:
                  accessControlAllowCredentials:
                    description: AccessControlAllowCredentials defines whether the
                      request can include user credentials.
                    type: boolean
                  accessControlAllowHeaders:
                    description: AccessControlAllowHeaders defines the Access-Control-Request-Headers
                      values sent in preflight response.
                    items:
                      type: string
                    type: array
                  accessControlAllowMethods:
                    description: AccessControlAllowMethods defines the Access-Control-Request-Method
                      values sent in preflight response.
                    items:
                      type: string
                    type: array
                  accessControlAllowOriginList:
                    description: AccessControlAllowOriginList is a list of allowable
                      origins. Can also be a wildcard origin "*".
                    items:
                      type: string
                    type: array
                  accessControlAllowOriginListRegex:
                    description: AccessControlAllowOriginListRegex is a list of allowable
                      origins written following the Regular Expression syntax (https://golang.org/pkg/regexp/).
                    items:
                      type: string
                    type: array
                  accessControlExposeHeaders:
                    description: AccessControlExposeHeaders defines the Access-Control-Expose-Headers
                      values sent in preflight response.
                    items:
                      type: string
                    type: array
                  accessControlMaxAge:
                    description: AccessControlMaxAge defines the time that a preflight
                      request may be cached.
                    format: int64
                    type: integer
                  addVaryHeader:
                    description: AddVaryHeader defines whether the Vary header is
                      automatically added/updated when the AccessControlAllowOriginList
                      is set.
                    type: boolean
                  allowedHosts:
                    description: AllowedHosts defines the fully qualified list of
                      allowed domain names.
                    items:
                      type: string
                    type: array
                  browserXssFilter:
                    description: BrowserXSSFilter defines whether to add the X-XSS-Protection
                      header with the value 1; mode=block.
                    type: boolean
                  contentSecurityPolicy:
                    description: ContentSecurityPolicy defines the Content-Security-Policy
                      header value.
                    type: string
                  contentTypeNosniff:
                    description: ContentTypeNosniff defines whether to add the X-Content-Type-Options
                      header with the nosniff value.
                    type: boolean
                  customBrowserXSSValue:
                    description: CustomBrowserXSSValue defines the X-XSS-Protection
                      header value. This overrides the BrowserXssFilter option.
                    type: string
                  customFrameOptionsValue:
                    description: CustomFrameOptionsValue defines the X-Frame-Options
                      header value. This overrides the FrameDeny option.
                    type: string
                  customRequestHeaders:
                    additionalProperties:
                      type: string
                    description: CustomRequestHeaders defines the header names and
                      values to apply to the request.
                    type: object
                  customResponseHeaders:
                    additionalProperties:
                      type: string
                    description: CustomResponseHeaders defines the header names and
                      values to apply to the response.
                    type: object
                  featurePolicy:
                    description: 'Deprecated: use PermissionsPolicy instead.'
                    type: string
                  forceSTSHeader:
                    description: ForceSTSHeader defines whether to add the STS header
                      even when the connection is HTTP.
                    type: boolean
                  frameDeny:
                    description: FrameDeny defines whether to add the X-Frame-Options
                      header with the DENY value.
                    type: boolean
                  hostsProxyHeaders:
                    description: HostsProxyHeaders defines the header keys that may
                      hold a proxied hostname value for the request.
                    items:
                      type: string
                    type: array
                  isDevelopment:
                    description: IsDevelopment defines whether to mitigate the unwanted
                      effects of the AllowedHosts, SSL, and STS options when developing.
                      Usually testing takes place using HTTP, not HTTPS, and on localhost,
                      not your production domain. If you would like your development
                      environment to mimic production with complete Host blocking,
                      SSL redirects, and STS headers, leave this as false.
                    type: boolean
                  permissionsPolicy:
                    description: PermissionsPolicy defines the Permissions-Policy
                      header value. This allows sites to control browser features.
                    type: string
                  publicKey:
                    description: PublicKey is the public key that implements HPKP
                      to prevent MITM attacks with forged certificates.
                    type: string
                  referrerPolicy:
                    description: ReferrerPolicy defines the Referrer-Policy header
                      value. This allows sites to control whether browsers forward
                      the Referer header to other sites.
                    type: string
                  sslForceHost:
                    description: 'Deprecated: use RedirectRegex instead.'
                    type: boolean
                  sslHost:
                    description: 'Deprecated: use RedirectRegex instead.'
                    type: string
                  sslProxyHeaders:
                    additionalProperties:
                      type: string
                    description: 'SSLProxyHeaders defines the header keys with associated
                      values that would indicate a valid HTTPS request. It can be
                      useful when using other proxies (example: "X-Forwarded-Proto":
                      "https").'
                    type: object
                  sslRedirect:
                    description: 'Deprecated: use EntryPoint redirection or RedirectScheme
                      instead.'
                    type: boolean
                  sslTemporaryRedirect:
                    description: 'Deprecated: use EntryPoint redirection or RedirectScheme
                      instead.'
                    type: boolean
                  stsIncludeSubdomains:
                    description: STSIncludeSubdomains defines whether the includeSubDomains
                      directive is appended to the Strict-Transport-Security header.
                    type: boolean
                  stsPreload:
                    description: STSPreload defines whether the preload flag is appended
                      to the Strict-Transport-Security header.
                    type: boolean
                  stsSeconds:
                    description: STSSeconds defines the max-age of the Strict-Transport-Security
                      header. If set to 0, the header is not set.
                    format: int64
                    type: integer
                type: object
              inFlightReq:
                description: 'InFlightReq holds the in-flight request middleware configuration.
                  This middleware limits the number of requests being processed and
                  served concurrently. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/inflightreq/'
                properties:
                  amount:
                    description: Amount defines the maximum amount of allowed simultaneous
                      in-flight request. The middleware responds with HTTP 429 Too
                      Many Requests if there are already amount requests in progress
                      (based on the same sourceCriterion strategy).
                    format: int64
                    type: integer
                  sourceCriterion:
                    description: 'SourceCriterion defines what criterion is used to
                      group requests as originating from a common source. If several
                      strategies are defined at the same time, an error will be raised.
                      If none are set, the default is to use the requestHost. More
                      info: https://doc.traefik.io/traefik/v2.9/middlewares/http/inflightreq/#sourcecriterion'
                    properties:
                      ipStrategy:
                        description: 'IPStrategy holds the IP strategy configuration
                          used by Traefik to determine the client IP. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/ipwhitelist/#ipstrategy'
                        properties:
                          depth:
                            description: Depth tells Traefik to use the X-Forwarded-For
                              header and take the IP located at the depth position
                              (starting from the right).
                            type: integer
                          excludedIPs:
                            description: ExcludedIPs configures Traefik to scan the
                              X-Forwarded-For header and select the first IP not in
                              the list.
                            items:
                              type: string
                            type: array
                        type: object
                      requestHeaderName:
                        description: RequestHeaderName defines the name of the header
                          used to group incoming requests.
                        type: string
                      requestHost:
                        description: RequestHost defines whether to consider the request
                          Host as the source.
                        type: boolean
                    type: object
                type: object
              ipWhiteList:
                description: 'IPWhiteList holds the IP whitelist middleware configuration.
                  This middleware accepts / refuses requests based on the client IP.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/ipwhitelist/'
                properties:
                  ipStrategy:
                    description: 'IPStrategy holds the IP strategy configuration used
                      by Traefik to determine the client IP. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/ipwhitelist/#ipstrategy'
                    properties:
                      depth:
                        description: Depth tells Traefik to use the X-Forwarded-For
                          header and take the IP located at the depth position (starting
                          from the right).
                        type: integer
                      excludedIPs:
                        description: ExcludedIPs configures Traefik to scan the X-Forwarded-For
                          header and select the first IP not in the list.
                        items:
                          type: string
                        type: array
                    type: object
                  sourceRange:
                    description: SourceRange defines the set of allowed IPs (or ranges
                      of allowed IPs by using CIDR notation).
                    items:
                      type: string
                    type: array
                type: object
              passTLSClientCert:
                description: 'PassTLSClientCert holds the pass TLS client cert middleware
                  configuration. This middleware adds the selected data from the passed
                  client TLS certificate to a header. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/passtlsclientcert/'
                properties:
                  info:
                    description: Info selects the specific client certificate details
                      you want to add to the X-Forwarded-Tls-Client-Cert-Info header.
                    properties:
                      issuer:
                        description: Issuer defines the client certificate issuer
                          details to add to the X-Forwarded-Tls-Client-Cert-Info header.
                        properties:
                          commonName:
                            description: CommonName defines whether to add the organizationalUnit
                              information into the issuer.
                            type: boolean
                          country:
                            description: Country defines whether to add the country
                              information into the issuer.
                            type: boolean
                          domainComponent:
                            description: DomainComponent defines whether to add the
                              domainComponent information into the issuer.
                            type: boolean
                          locality:
                            description: Locality defines whether to add the locality
                              information into the issuer.
                            type: boolean
                          organization:
                            description: Organization defines whether to add the organization
                              information into the issuer.
                            type: boolean
                          province:
                            description: Province defines whether to add the province
                              information into the issuer.
                            type: boolean
                          serialNumber:
                            description: SerialNumber defines whether to add the serialNumber
                              information into the issuer.
                            type: boolean
                        type: object
                      notAfter:
                        description: NotAfter defines whether to add the Not After
                          information from the Validity part.
                        type: boolean
                      notBefore:
                        description: NotBefore defines whether to add the Not Before
                          information from the Validity part.
                        type: boolean
                      sans:
                        description: Sans defines whether to add the Subject Alternative
                          Name information from the Subject Alternative Name part.
                        type: boolean
                      serialNumber:
                        description: SerialNumber defines whether to add the client
                          serialNumber information.
                        type: boolean
                      subject:
                        description: Subject defines the client certificate subject
                          details to add to the X-Forwarded-Tls-Client-Cert-Info header.
                        properties:
                          commonName:
                            description: CommonName defines whether to add the organizationalUnit
                              information into the subject.
                            type: boolean
                          country:
                            description: Country defines whether to add the country
                              information into the subject.
                            type: boolean
                          domainComponent:
                            description: DomainComponent defines whether to add the
                              domainComponent information into the subject.
                            type: boolean
                          locality:
                            description: Locality defines whether to add the locality
                              information into the subject.
                            type: boolean
                          organization:
                            description: Organization defines whether to add the organization
                              information into the subject.
                            type: boolean
                          organizationalUnit:
                            description: OrganizationalUnit defines whether to add
                              the organizationalUnit information into the subject.
                            type: boolean
                          province:
                            description: Province defines whether to add the province
                              information into the subject.
                            type: boolean
                          serialNumber:
                            description: SerialNumber defines whether to add the serialNumber
                              information into the subject.
                            type: boolean
                        type: object
                    type: object
                  pem:
                    description: PEM sets the X-Forwarded-Tls-Client-Cert header with
                      the certificate.
                    type: boolean
                type: object
              plugin:
                additionalProperties:
                  x-kubernetes-preserve-unknown-fields: true
                description: 'Plugin defines the middleware plugin configuration.
                  More info: https://doc.traefik.io/traefik/plugins/'
                type: object
              rateLimit:
                description: 'RateLimit holds the rate limit configuration. This middleware
                  ensures that services will receive a fair amount of requests, and
                  allows one to define what fair is. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/ratelimit/'
                properties:
                  average:
                    description: Average is the maximum rate, by default in requests/s,
                      allowed for the given source. It defaults to 0, which means
                      no rate limiting. The rate is actually defined by dividing Average
                      by Period. So for a rate below 1req/s, one needs to define a
                      Period larger than a second.
                    format: int64
                    type: integer
                  burst:
                    description: Burst is the maximum number of requests allowed to
                      arrive in the same arbitrarily small period of time. It defaults
                      to 1.
                    format: int64
                    type: integer
                  period:
                    anyOf:
                    - type: integer
                    - type: string
                    description: 'Period, in combination with Average, defines the
                      actual maximum rate, such as: r = Average / Period. It defaults
                      to a second.'
                    x-kubernetes-int-or-string: true
                  sourceCriterion:
                    description: SourceCriterion defines what criterion is used to
                      group requests as originating from a common source. If several
                      strategies are defined at the same time, an error will be raised.
                      If none are set, the default is to use the request's remote
                      address field (as an ipStrategy).
                    properties:
                      ipStrategy:
                        description: 'IPStrategy holds the IP strategy configuration
                          used by Traefik to determine the client IP. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/ipwhitelist/#ipstrategy'
                        properties:
                          depth:
                            description: Depth tells Traefik to use the X-Forwarded-For
                              header and take the IP located at the depth position
                              (starting from the right).
                            type: integer
                          excludedIPs:
                            description: ExcludedIPs configures Traefik to scan the
                              X-Forwarded-For header and select the first IP not in
                              the list.
                            items:
                              type: string
                            type: array
                        type: object
                      requestHeaderName:
                        description: RequestHeaderName defines the name of the header
                          used to group incoming requests.
                        type: string
                      requestHost:
                        description: RequestHost defines whether to consider the request
                          Host as the source.
                        type: boolean
                    type: object
                type: object
              redirectRegex:
                description: 'RedirectRegex holds the redirect regex middleware configuration.
                  This middleware redirects a request using regex matching and replacement.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/redirectregex/#regex'
                properties:
                  permanent:
                    description: Permanent defines whether the redirection is permanent
                      (301).
                    type: boolean
                  regex:
                    description: Regex defines the regex used to match and capture
                      elements from the request URL.
                    type: string
                  replacement:
                    description: Replacement defines how to modify the URL to have
                      the new target URL.
                    type: string
                type: object
              redirectScheme:
                description: 'RedirectScheme holds the redirect scheme middleware
                  configuration. This middleware redirects requests from a scheme/port
                  to another. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/redirectscheme/'
                properties:
                  permanent:
                    description: Permanent defines whether the redirection is permanent
                      (301).
                    type: boolean
                  port:
                    description: Port defines the port of the new URL.
                    type: string
                  scheme:
                    description: Scheme defines the scheme of the new URL.
                    type: string
                type: object
              replacePath:
                description: 'ReplacePath holds the replace path middleware configuration.
                  This middleware replaces the path of the request URL and store the
                  original path in an X-Replaced-Path header. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/replacepath/'
                properties:
                  path:
                    description: Path defines the path to use as replacement in the
                      request URL.
                    type: string
                type: object
              replacePathRegex:
                description: 'ReplacePathRegex holds the replace path regex middleware
                  configuration. This middleware replaces the path of a URL using
                  regex matching and replacement. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/replacepathregex/'
                properties:
                  regex:
                    description: Regex defines the regular expression used to match
                      and capture the path from the request URL.
                    type: string
                  replacement:
                    description: Replacement defines the replacement path format,
                      which can include captured variables.
                    type: string
                type: object
              retry:
                description: 'Retry holds the retry middleware configuration. This
                  middleware reissues requests a given number of times to a backend
                  server if that server does not reply. As soon as the server answers,
                  the middleware stops retrying, regardless of the response status.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/retry/'
                properties:
                  attempts:
                    description: Attempts defines how many times the request should
                      be retried.
                    type: integer
                  initialInterval:
                    anyOf:
                    - type: integer
                    - type: string
                    description: InitialInterval defines the first wait time in the
                      exponential backoff series. The maximum interval is calculated
                      as twice the initialInterval. If unspecified, requests will
                      be retried immediately. The value of initialInterval should
                      be provided in seconds or as a valid duration format, see https://pkg.go.dev/time#ParseDuration.
                    x-kubernetes-int-or-string: true
                type: object
              stripPrefix:
                description: 'StripPrefix holds the strip prefix middleware configuration.
                  This middleware removes the specified prefixes from the URL path.
                  More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/stripprefix/'
                properties:
                  forceSlash:
                    description: 'ForceSlash ensures that the resulting stripped path
                      is not the empty string, by replacing it with / when necessary.
                      Default: true.'
                    type: boolean
                  prefixes:
                    description: Prefixes defines the prefixes to strip from the request
                      URL.
                    items:
                      type: string
                    type: array
                type: object
              stripPrefixRegex:
                description: 'StripPrefixRegex holds the strip prefix regex middleware
                  configuration. This middleware removes the matching prefixes from
                  the URL path. More info: https://doc.traefik.io/traefik/v2.9/middlewares/http/stripprefixregex/'
                properties:
                  regex:
                    description: Regex defines the regular expression to match the
                      path prefix from the request URL.
                    items:
                      type: string
                    type: array
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: middlewaretcps.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: MiddlewareTCP
    listKind: MiddlewareTCPList
    plural: middlewaretcps
    singular: middlewaretcp
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: 'MiddlewareTCP is the CRD implementation of a Traefik TCP middleware.
          More info: https://doc.traefik.io/traefik/v2.9/middlewares/overview/'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: MiddlewareTCPSpec defines the desired state of a MiddlewareTCP.
            properties:
              inFlightConn:
                description: InFlightConn defines the InFlightConn middleware configuration.
                properties:
                  amount:
                    description: Amount defines the maximum amount of allowed simultaneous
                      connections. The middleware closes the connection if there are
                      already amount connections opened.
                    format: int64
                    type: integer
                type: object
              ipWhiteList:
                description: IPWhiteList defines the IPWhiteList middleware configuration.
                properties:
                  sourceRange:
                    description: SourceRange defines the allowed IPs (or ranges of
                      allowed IPs by using CIDR notation).
                    items:
                      type: string
                    type: array
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: serverstransports.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: ServersTransport
    listKind: ServersTransportList
    plural: serverstransports
    singular: serverstransport
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: 'ServersTransport is the CRD implementation of a ServersTransport.
          If no serversTransport is specified, the default@internal will be used.
          The default@internal serversTransport is created from the static configuration.
          More info: https://doc.traefik.io/traefik/v2.9/routing/services/#serverstransport_1'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ServersTransportSpec defines the desired state of a ServersTransport.
            properties:
              certificatesSecrets:
                description: CertificatesSecrets defines a list of secret storing
                  client certificates for mTLS.
                items:
                  type: string
                type: array
              disableHTTP2:
                description: DisableHTTP2 disables HTTP/2 for connections with backend
                  servers.
                type: boolean
              forwardingTimeouts:
                description: ForwardingTimeouts defines the timeouts for requests
                  forwarded to the backend servers.
                properties:
                  dialTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: DialTimeout is the amount of time to wait until a
                      connection to a backend server can be established.
                    x-kubernetes-int-or-string: true
                  idleConnTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: IdleConnTimeout is the maximum period for which an
                      idle HTTP keep-alive connection will remain open before closing
                      itself.
                    x-kubernetes-int-or-string: true
                  pingTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: PingTimeout is the timeout after which the HTTP/2
                      connection will be closed if a response to ping is not received.
                    x-kubernetes-int-or-string: true
                  readIdleTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: ReadIdleTimeout is the timeout after which a health
                      check using ping frame will be carried out if no frame is received
                      on the HTTP/2 connection.
                    x-kubernetes-int-or-string: true
                  responseHeaderTimeout:
                    anyOf:
                    - type: integer
                    - type: string
                    description: ResponseHeaderTimeout is the amount of time to wait
                      for a server's response headers after fully writing the request
                      (including its body, if any).
                    x-kubernetes-int-or-string: true
                type: object
              insecureSkipVerify:
                description: InsecureSkipVerify disables SSL certificate verification.
                type: boolean
              maxIdleConnsPerHost:
                description: MaxIdleConnsPerHost controls the maximum idle (keep-alive)
                  to keep per-host.
                type: integer
              peerCertURI:
                description: PeerCertURI defines the peer cert URI used to match against
                  SAN URI during the peer certificate verification.
                type: string
              rootCAsSecrets:
                description: RootCAsSecrets defines a list of CA secret used to validate
                  self-signed certificate.
                items:
                  type: string
                type: array
              serverName:
                description: ServerName defines the server name used to contact the
                  server.
                type: string
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: tlsoptions.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: TLSOption
    listKind: TLSOptionList
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: 'TLSOption is the CRD implementation of a Traefik TLS Option,
          allowing to configure some parameters of the TLS connection. More info:
          https://doc.traefik.io/traefik/v2.9/https/tls/#tls-options'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TLSOptionSpec defines the desired state of a TLSOption.
            properties:
              alpnProtocols:
                description: 'ALPNProtocols defines the list of supported application
                  level protocols for the TLS handshake, in order of preference. More
                  info: https://doc.traefik.io/traefik/v2.9/https/tls/#alpn-protocols'
                items:
                  type: string
                type: array
              cipherSuites:
                description: 'CipherSuites defines the list of supported cipher suites
                  for TLS versions up to TLS 1.2. More info: https://doc.traefik.io/traefik/v2.9/https/tls/#cipher-suites'
                items:
                  type: string
                type: array
              clientAuth:
                description: ClientAuth defines the server's policy for TLS Client
                  Authentication.
                properties:
                  clientAuthType:
                    description: ClientAuthType defines the client authentication
                      type to apply.
                    enum:
                    - NoClientCert
                    - RequestClientCert
                    - RequireAnyClientCert
                    - VerifyClientCertIfGiven
                    - RequireAndVerifyClientCert
                    type: string
                  secretNames:
                    description: SecretNames defines the names of the referenced Kubernetes
                      Secret storing certificate details.
                    items:
                      type: string
                    type: array
                type: object
              curvePreferences:
                description: 'CurvePreferences defines the preferred elliptic curves
                  in a specific order. More info: https://doc.traefik.io/traefik/v2.9/https/tls/#curve-preferences'
                items:
                  type: string
                type: array
              maxVersion:
                description: 'MaxVersion defines the maximum TLS version that Traefik
                  will accept. Possible values: VersionTLS10, VersionTLS11, VersionTLS12,
                  VersionTLS13. Default: None.'
                type: string
              minVersion:
                description: 'MinVersion defines the minimum TLS version that Traefik
                  will accept. Possible values: VersionTLS10, VersionTLS11, VersionTLS12,
                  VersionTLS13. Default: VersionTLS10.'
                type: string
              preferServerCipherSuites:
                description: 'PreferServerCipherSuites defines whether the server
                  chooses a cipher suite among his own instead of among the client''s.
                  It is enabled automatically when minVersion or maxVersion is set.
                  Deprecated: https://github.com/golang/go/issues/45430'
                type: boolean
              sniStrict:
                description: SniStrict defines whether Traefik allows connections
                  from clients connections that do not specify a server_name extension.
                type: boolean
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: tlsstores.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: TLSStore
    listKind: TLSStoreList
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: 'TLSStore is the CRD implementation of a Traefik TLS Store. For
          the time being, only the TLSStore named default is supported. This means
          that you cannot have two stores that are named default in different Kubernetes
          namespaces. More info: https://doc.traefik.io/traefik/v2.9/https/tls/#certificates-stores'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TLSStoreSpec defines the desired state of a TLSStore.
            properties:
              certificates:
                description: Certificates is a list of secret names, each secret holding
                  a key/certificate pair to add to the store.
                items:
                  description: Certificate holds a secret name for the TLSStore resource.
                  properties:
                    secretName:
                      description: SecretName is the name of the referenced Kubernetes
                        Secret to specify the certificate details.
                      type: string
                  required:
                  - secretName
                  type: object
                type: array
              defaultCertificate:
                description: DefaultCertificate defines the default certificate configuration.
                properties:
                  secretName:
                    description: SecretName is the name of the referenced Kubernetes
                      Secret to specify the certificate details.
                    type: string
                required:
                - secretName
                type: object
              defaultGeneratedCert:
                description: DefaultGeneratedCert defines the default generated certificate
                  configuration.
                properties:
                  domain:
                    description: Domain is the domain definition for the DefaultCertificate.
                    properties:
                      main:
                        description: Main defines the main domain name.
                        type: string
                      sans:
                        description: SANs defines the subject alternative domain names.
                        items:
                          type: string
                        type: array
                    type: object
                  resolver:
                    description: Resolver is the name of the resolver that will be
                      used to issue the DefaultCertificate.
                    type: string
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: traefikservices.traefik.containo.us
spec:
  group: traefik.containo.us
  names:
    kind: TraefikService
    listKind: TraefikServiceList
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: 'TraefikService is the CRD implementation of a Traefik Service.
          TraefikService object allows to: - Apply weight to Services on load-balancing
          - Mirror traffic on services More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#kind-traefikservice'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TraefikServiceSpec defines the desired state of a TraefikService.
            properties:
              mirroring:
                description: Mirroring defines the Mirroring service configuration.
                properties:
                  kind:
                    description: Kind defines the kind of the Service.
                    enum:
                    - Service
                    - TraefikService
                    type: string
                  maxBodySize:
                    description: MaxBodySize defines the maximum size allowed for
                      the body of the request. If the body is larger, the request
                      is not mirrored. Default value is -1, which means unlimited
                      size.
                    format: int64
                    type: integer
                  mirrors:
                    description: Mirrors defines the list of mirrors where Traefik
                      will duplicate the traffic.
                    items:
                      description: MirrorService holds the mirror configuration.
                      properties:
                        kind:
                          description: Kind defines the kind of the Service.
                          enum:
                          - Service
                          - TraefikService
                          type: string
                        name:
                          description: Name defines the name of the referenced Kubernetes
                            Service or TraefikService. The differentiation between
                            the two is specified in the Kind field.
                          type: string
                        namespace:
                          description: Namespace defines the namespace of the referenced
                            Kubernetes Service or TraefikService.
                          type: string
                        passHostHeader:
                          description: PassHostHeader defines whether the client Host
                            header is forwarded to the upstream Kubernetes Service.
                            By default, passHostHeader is true.
                          type: boolean
                        percent:
                          description: 'Percent defines the part of the traffic to
                            mirror. Supported values: 0 to 100.'
                          type: integer
                        port:
                          anyOf:
                          - type: integer
                          - type: string
                          description: Port defines the port of a Kubernetes Service.
                            This can be a reference to a named port.
                          x-kubernetes-int-or-string: true
                        responseForwarding:
                          description: ResponseForwarding defines how Traefik forwards
                            the response from the upstream Kubernetes Service to the
                            client.
                          properties:
                            flushInterval:
                              description: 'FlushInterval defines the interval, in
                                milliseconds, in between flushes to the client while
                                copying the response body. A negative value means
                                to flush immediately after each write to the client.
                                This configuration is ignored when ReverseProxy recognizes
                                a response as a streaming response; for such responses,
                                writes are flushed to the client immediately. Default:
                                100ms'
                              type: string
                          type: object
                        scheme:
                          description: Scheme defines the scheme to use for the request
                            to the upstream Kubernetes Service. It defaults to https
                            when Kubernetes Service port is 443, http otherwise.
                          type: string
                        serversTransport:
                          description: ServersTransport defines the name of ServersTransport
                            resource to use. It allows to configure the transport
                            between Traefik and your servers. Can only be used on
                            a Kubernetes Service.
                          type: string
                        sticky:
                          description: 'Sticky defines the sticky sessions configuration.
                            More info: https://doc.traefik.io/traefik/v2.9/routing/services/#sticky-sessions'
                          properties:
                            cookie:
                              description: Cookie defines the sticky cookie configuration.
                              properties:
                                httpOnly:
                                  description: HTTPOnly defines whether the cookie
                                    can be accessed by client-side APIs, such as JavaScript.
                                  type: boolean
                                name:
                                  description: Name defines the Cookie name.
                                  type: string
                                sameSite:
                                  description: 'SameSite defines the same site policy.
                                    More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite'
                                  type: string
                                secure:
                                  description: Secure defines whether the cookie can
                                    only be transmitted over an encrypted connection
                                    (i.e. HTTPS).
                                  type: boolean
                              type: object
                          type: object
                        strategy:
                          description: Strategy defines the load balancing strategy
                            between the servers. RoundRobin is the only supported
                            value at the moment.
                          type: string
                        weight:
                          description: Weight defines the weight and should only be
                            specified when Name references a TraefikService object
                            (and to be precise, one that embeds a Weighted Round Robin).
                          type: integer
                      required:
                      - name
                      type: object
                    type: array
                  name:
                    description: Name defines the name of the referenced Kubernetes
                      Service or TraefikService. The differentiation between the two
                      is specified in the Kind field.
                    type: string
                  namespace:
                    description: Namespace defines the namespace of the referenced
                      Kubernetes Service or TraefikService.
                    type: string
                  passHostHeader:
                    description: PassHostHeader defines whether the client Host header
                      is forwarded to the upstream Kubernetes Service. By default,
                      passHostHeader is true.
                    type: boolean
                  port:
                    anyOf:
                    - type: integer
                    - type: string
                    description: Port defines the port of a Kubernetes Service. This
                      can be a reference to a named port.
                    x-kubernetes-int-or-string: true
                  responseForwarding:
                    description: ResponseForwarding defines how Traefik forwards the
                      response from the upstream Kubernetes Service to the client.
                    properties:
                      flushInterval:
                        description: 'FlushInterval defines the interval, in milliseconds,
                          in between flushes to the client while copying the response
                          body. A negative value means to flush immediately after
                          each write to the client. This configuration is ignored
                          when ReverseProxy recognizes a response as a streaming response;
                          for such responses, writes are flushed to the client immediately.
                          Default: 100ms'
                        type: string
                    type: object
                  scheme:
                    description: Scheme defines the scheme to use for the request
                      to the upstream Kubernetes Service. It defaults to https when
                      Kubernetes Service port is 443, http otherwise.
                    type: string
                  serversTransport:
                    description: ServersTransport defines the name of ServersTransport
                      resource to use. It allows to configure the transport between
                      Traefik and your servers. Can only be used on a Kubernetes Service.
                    type: string
                  sticky:
                    description: 'Sticky defines the sticky sessions configuration.
                      More info: https://doc.traefik.io/traefik/v2.9/routing/services/#sticky-sessions'
                    properties:
                      cookie:
                        description: Cookie defines the sticky cookie configuration.
                        properties:
                          httpOnly:
                            description: HTTPOnly defines whether the cookie can be
                              accessed by client-side APIs, such as JavaScript.
                            type: boolean
                          name:
                            description: Name defines the Cookie name.
                            type: string
                          sameSite:
                            description: 'SameSite defines the same site policy. More
                              info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite'
                            type: string
                          secure:
                            description: Secure defines whether the cookie can only
                              be transmitted over an encrypted connection (i.e. HTTPS).
                            type: boolean
                        type: object
                    type: object
                  strategy:
                    description: Strategy defines the load balancing strategy between
                      the servers. RoundRobin is the only supported value at the moment.
                    type: string
                  weight:
                    description: Weight defines the weight and should only be specified
                      when Name references a TraefikService object (and to be precise,
                      one that embeds a Weighted Round Robin).
                    type: integer
                required:
                - name
                type: object
              weighted:
                description: Weighted defines the Weighted Round Robin configuration.
                properties:
                  services:
                    description: Services defines the list of Kubernetes Service and/or
                      TraefikService to load-balance, with weight.
                    items:
                      description: Service defines an upstream HTTP service to proxy
                        traffic to.
                      properties:
                        kind:
                          description: Kind defines the kind of the Service.
                          enum:
                          - Service
                          - TraefikService
                          type: string
                        name:
                          description: Name defines the name of the referenced Kubernetes
                            Service or TraefikService. The differentiation between
                            the two is specified in the Kind field.
                          type: string
                        namespace:
                          description: Namespace defines the namespace of the referenced
                            Kubernetes Service or TraefikService.
                          type: string
                        passHostHeader:
                          description: PassHostHeader defines whether the client Host
                            header is forwarded to the upstream Kubernetes Service.
                            By default, passHostHeader is true.
                          type: boolean
                        port:
                          anyOf:
                          - type: integer
                          - type: string
                          description: Port defines the port of a Kubernetes Service.
                            This can be a reference to a named port.
                          x-kubernetes-int-or-string: true
                        responseForwarding:
                          description: ResponseForwarding defines how Traefik forwards
                            the response from the upstream Kubernetes Service to the
                            client.
                          properties:
                            flushInterval:
                              description: 'FlushInterval defines the interval, in
                                milliseconds, in between flushes to the client while
                                copying the response body. A negative value means
                                to flush immediately after each write to the client.
                                This configuration is ignored when ReverseProxy recognizes
                                a response as a streaming response; for such responses,
                                writes are flushed to the client immediately. Default:
                                100ms'
                              type: string
                          type: object
                        scheme:
                          description: Scheme defines the scheme to use for the request
                            to the upstream Kubernetes Service. It defaults to https
                            when Kubernetes Service port is 443, http otherwise.
                          type: string
                        serversTransport:
                          description: ServersTransport defines the name of ServersTransport
                            resource to use. It allows to configure the transport
                            between Traefik and your servers. Can only be used on
                            a Kubernetes Service.
                          type: string
                        sticky:
                          description: 'Sticky defines the sticky sessions configuration.
                            More info: https://doc.traefik.io/traefik/v2.9/routing/services/#sticky-sessions'
                          properties:
                            cookie:
                              description: Cookie defines the sticky cookie configuration.
                              properties:
                                httpOnly:
                                  description: HTTPOnly defines whether the cookie
                                    can be accessed by client-side APIs, such as JavaScript.
                                  type: boolean
                                name:
                                  description: Name defines the Cookie name.
                                  type: string
                                sameSite:
                                  description: 'SameSite defines the same site policy.
                                    More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite'
                                  type: string
                                secure:
                                  description: Secure defines whether the cookie can
                                    only be transmitted over an encrypted connection
                                    (i.e. HTTPS).
                                  type: boolean
                              type: object
                          type: object
                        strategy:
                          description: Strategy defines the load balancing strategy
                            between the servers. RoundRobin is the only supported
                            value at the moment.
                          type: string
                        weight:
                          description: Weight defines the weight and should only be
                            specified when Name references a TraefikService object
                            (and to be precise, one that embeds a Weighted Round Robin).
                          type: integer
                      required:
                      - name
                      type: object
                    type: array
                  sticky:
                    description: 'Sticky defines whether sticky sessions are enabled.
                      More info: https://doc.traefik.io/traefik/v2.9/routing/providers/kubernetes-crd/#stickiness-and-load-balancing'
                    properties:
                      cookie:
                        description: Cookie defines the sticky cookie configuration.
                        properties:
                          httpOnly:
                            description: HTTPOnly defines whether the cookie can be
                              accessed by client-side APIs, such as JavaScript.
                            type: boolean
                          name:
                            description: Name defines the Cookie name.
                            type: string
                          sameSite:
                            description: 'SameSite defines the same site policy. More
                              info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite'
                            type: string
                          secure:
                            description: Secure defines whether the cookie can only
                              be transmitted over an encrypted connection (i.e. HTTPS).
                            type: boolean
                        type: object
                    type: object
                type: object
            type: object
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

ConfigMap配置:

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
  namespace: traefik
  labels:
    app: traefik
data:
  traefik.yaml: |-
    ping: ""                    ## 启用 Ping
    serversTransport:
      insecureSkipVerify: true  ## Traefik 忽略验证代理服务的 TLS 证书
    api:
      insecure: true            ## 允许 HTTP 方式访问 API
      dashboard: true           ## 启用 Dashboard
      debug: false              ## 启用 Debug 调试模式
    # metrics:
    #   prometheus: ""            ## 配置 Prometheus 监控指标数据,并使用默认配置
    entryPoints:
      http:
        address: ":80"          ## 配置 80 端口,并设置入口名称为 web
        forwardedHeaders:       ## 获取客户端真实IP
          insecure: true
          trustedIPs:
          - "10.68.0.0/16"
          - "172.16.0.0/12"
      # https:
      #   address: ":443"         ## 配置 443 端口,并设置入口名称为 https
      #   forwardedHeaders:       ## 获取客户端真实IP
      #     insecure: true
      #     trustedIPs:
      #     - "10.68.0.0/16"
      #     - "172.16.0.0/12"
    providers:
      kubernetesCRD: ""         ## 启用 Kubernetes CRD 方式来配置路由规则
      kubernetesIngress: ""     ## 启动 Kubernetes Ingress 方式来配置路由规则
    log:
      filePath: ""              ## 设置调试日志文件存储路径,如果为空则输出到控制台
      level: INFO               ## 设置调试日志级别
      format: common            ## 设置调试日志格式
    accessLog:
      filePath: ""              ## 设置访问日志文件存储路径,如果为空则输出到控制台
      format: common            ## 设置访问调试日志格式
      bufferingSize: 100        ## 设置访问日志缓存行数
      filters:
        statusCodes: ["403","413","500","502","503","504"]    ## 设置只保留指定状态码范围内的访问日志
        retryAttempts: true     ## 设置代理访问重试失败时,保留访问日志
        minDuration: 20         ## 设置保留请求时间超过指定持续时间的访问日志
      fields:                   ## 设置访问日志中的字段是否保留(keep 保留、drop 不保留)
        defaultMode: keep       ## 设置默认保留访问日志字段
        names:                  ## 针对访问日志特别字段特别配置保留模式
          ClientUsername: drop  
        headers:                ## 设置 Header 中字段是否保留
          defaultMode: keep     ## 设置默认保留 Header 中字段
          names:                ## 针对 Header 中特别字段特别配置保留模式
            User-Agent: redact
            Authorization: drop
            Content-Type: keep

应用部署文件配置:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik-ingress-deploy
  namespace: traefik
  labels:
    app: traefik
spec:
  replicas: 3
  selector:
    matchLabels:
      app: traefik
  minReadySeconds: 10
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      name: traefik
      labels:
        app: traefik
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cluster.role
                operator: In
                values:
                - proxy
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values: 
                    - traefik-ingress-deploy
              topologyKey: "kubernetes.io/hostname"
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      serviceAccountName: traefik-ingress-deploy
      terminationGracePeriodSeconds: 1
      hostNetwork: true
      containers:
      - image: traefik:2.9
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
        resources:
          limits:
            cpu: 2
            memory: 2048Mi
          requests:
            cpu: 0.5
            memory: 1024Mi
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --configfile=/config/traefik.yaml
        volumeMounts:
        - mountPath: /config
          name: config
        readinessProbe:
          httpGet:
            path: /ping
            port: 8080
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 5
        livenessProbe:
          httpGet:
            path: /ping
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5    
      volumes:
      - name: config
        configMap:
          name: traefik-config

部署完成之后,可以访问Traefik的控制台:MasterIP:8080,更多配置可参考Traefik官网:https://docs.traefik.io

17 Dashboard开启TLS访问

针对单个应用开启TLS安全访问的步骤:

  • 1.Traefik配置443端口监听
  • 2.创建证书Secret
  • 3.Ingress上配置证书及使用的Traefik Entrypoint

修改Traefik Configmap配置文件中的entryPoints,修改更新Configmap后需要重建Traefik容器以便快速应用新的配置。

    entryPoints:
      http:
        address: ":80"          ## 配置 80 端口,并设置入口名称为 http
      https:
        address: ":443"         ## 配置 443 端口,并设置入口名称为 https

创建证书Secret可以使用自建证书或购买证书,自建证书为不受信任证书,地址栏访问时会提示不安全。

#生成证书
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/cloudnil.com.key -x509 -days 365 -out certs/cloudnil.com.crt

#创建证书Secret
kubectl create secret tls tls-cert --cert=cloudnil.com.crt --key=cloudnil.com.key -n kube-system

修改k8dash的Ingress配置,可以使用原生Ingress或者Traefik拓展的IngressRoute(二选一)。

#原生Ingress配置,推荐
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: k8dash-ing
  labels:
    k8s-app: k8dash
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.entrypoints: https
spec:
  tls:
  - hosts:
    - console.cloudnil.com
    secretName: tls-cert
  rules:
  - host: console.cloudnil.com
    http:
      paths:
      - path: /
        pathType: "Prefix"
        backend:
          service:
            name: k8dash-svc
            port: 
              number: 80
---
#Traefik IngressRoute配置
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: k8dash-ingroute
  labels:
    k8s-app: k8dash
  namespace: kube-system
spec:
  entryPoints:
  - https
  routes:
  - match: Host(`console.cloudnil.com`) && PathPrefix(`/`)
    kind: Rule
    services:
    - name: earth-web-svc
      port: 80
  tls:
    certResolver: tls-cert

18 结语

Nginx-ingress-controllerTraefik-ingress-controller部署完成之后,解析相关域名如console.cloudnil.com到master01、master02、master03的外网IP,就可以使用console.cloudnil.com访问dashboard,其他应用类似。

版权声明:允许转载,请注明原文出处:http://cloudnil.com/2023/04/18/Deploy-kubernetes(1.26.4)-HA-with-kubeadm/。

Search

    Post Directory