반응형
구축 환경
Proxmox VM으로 node 환경을 구축했다.
- Node
- test-master 172.16.100.91
- test-worker1 172.16.100.92
- test-worker2 172.16.100.93
- 설치 정보
- Rocky Linux 9.5
- kubernetes 1.32
- runtime : cri-o 1.32
- CNI : calico 3.25
runtime 성능 지표 참조
CNI 성능 지표 참조
- https://itnext.io/benchmark-results-of-kubernetes-network-plugins-cni-over-10gbit-s-network-36475925a560
- https://ykarma1996.tistory.com/179
Swap Off (Master/Worker)
swapoff -a
free -m
total used free shared buff/cache available
Mem: 15735 379 15452 8 143 15356
Swap: 0 0 0fstab 수정
# swap 부분 주석
vim /etc/fstab
...
UUID=7db19900-b434-49c5-a613-ea1ef3ec3e35 / xfs defaults 0 0
#UUID=31e825fe-a419-469c-82bb-22ee4b8e2da3 none swap defaults 0 0
Selinux Disable (Master/Worker)
vim /etc/selinux/config
...
SELINUX=disabled
...
Hosts 등록 (Master/Worker)
vim /etc/hosts
...
172.16.100.91 test-master
172.16.100.92 test-worker1
172.16.100.93 test-worker2
Firewalld 비활성화 (Master/Worker)
systemctl stop firewalld; systemctl disable firewalld
Package Repository 추가 (Master/Worker)
kubernetes
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
EOF
cri-o
cat <<EOF | tee /etc/yum.repos.d/cri-o.repo
[cri-o]
name=CRI-O
baseurl=https://pkgs.k8s.io/addons:/cri-o:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/addons:/cri-o:/stable:/v1.32/rpm/repodata/repomd.xml.key
EOF
의존성 패키지 설치 (Master/Worker)
dnf install -y conntrack container-selinux ebtables ethtool iptables socat vim bind-utils net-tools tcpdump wget git tar
kubernetes, runtime 설치 (Master/Worker)
dnf install -y --repo cri-o --repo kubernetes cri-o kubeadm kubectl kubelet
systemd 시작 설정 (Master/Worker)
systemctl start crio
systemctl enable crio
systemctl enable kubelet
커널 파라미터 및 모듈 설정 (Master/Worker)
모듈
vim /etc/modules-load.d/crio.conf
overlay
br_netfilter
modprobe br_netfilter
커널파라미터
cat <<EOF | sudo tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
kubeadm init (Master)
kubeadm init --pod-network-cidr=<파드에 할당할 IP 대역> --apiserver-advertise-address=<Api server 가 있는 Master IP>
참고로 service ip(cluster ip)로 사용할 IP 대역을 지정할 수 있는데 지정하지 않으면 default로 10.96.0.0/12 대역이 할당 된다.
--service-cidr <IP 대역>
참고 : https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
kubeadm init --pod-network-cidr=10.255.0.0/16 --apiserver-advertise-address=172.16.100.91아래 결과물의 마지막 부분에 kubeadm join(token) 부분은 worker node가 클러스터에 조인시키기위해 필요하므로 기억하고 있는다.
...
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.100.91:6443 --token kopbd2.4ao8vajy... \
--discovery-token-ca-cert-hash sha256:cda18bbc1a9f8d74f2de735...
아래 명령어 실행
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Token 확인은 아래와 같이 하며 만료시간이 있기때문에 만료되면 재생성 해주어야 한다.
kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
qmyqds.hes7mio... 23h 2025-02-05T08:47:04Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
t6xggy.vbkztv... 23h 2025-02-05T08:48:24Z authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
usca1j.b0b6w1... 23h 2025-02-05T08:48:22Z authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
Token 재생성
kubeadm token create --print-join-command
해시값 확인은
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
Woker node Join (Worker)
kubeadm join 172.16.100.91:6443 --token fwf6hp.vj4p9g232kf82h7t --discovery-token-ca-cert-hash sha256:a536e08e3fdbc2ea0d89006098c1340...
Calico CNI 설치 (Master)
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
calicoctl 설치 (Master)
curl -L https://github.com/projectcalico/calico/releases/download/v3.25.0/calicoctl-linux-amd64 -o calicoctl
chmod +x calicoctl && mv calicoctl /usr/bin
calico version 확인
calicoctl version
Client Version: v3.25.0
Git commit: 3f7fe4d29
Cluster Version: v3.25.0
Cluster Type: k8s,bgp,kubeadm,kdd
Node 상태 확인
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
test-master Ready control-plane 7m3s v1.32.1 172.16.100.91 <none> Rocky Linux 9.5 (Blue Onyx) 5.14.0-503.14.1.el9_5.x86_64 cri-o://1.32.1
test-worker1 Ready <none> 2m4s v1.32.1 172.16.100.92 <none> Rocky Linux 9.5 (Blue Onyx) 5.14.0-503.14.1.el9_5.x86_64 cri-o://1.32.1
test-worker2 Ready <none> 5m28s v1.32.1 172.16.100.93 <none> Rocky Linux 9.5 (Blue Onyx) 5.14.0-503.14.1.el9_5.x86_64 cri-o://1.32.1
CNI가 설치되어 있지 않으면 worker status가 NotReady로 표기 된다.
Kubectl 명령어 자동완성
dnf install bash-completionecho 'source <(kubectl completion bash)' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -o default -F __start_kubectl k' >> ~/.bashrc반응형