[k8s] 쿠버네티스 설치

총 3개의 가상머신을 설치해서

노드로 쓸 가상머신 2개, 이 노드들을 다룰 어드민서버(matser) 1개 로 구성하였다.

 

 

root@k8sadmin:~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
OK
root@k8sadmin:~# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
> deb http://apt.kubernetes.io/ kubernetes-xenial main
> EOF

root@k8sadmin:~# wget -qO- get.docker.com | sh
# Executing docker install script, commit: 4f282167c425347a931ccfd95cc91fab041d414f
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c mkdir -p /etc/apt/keyrings && chmod -R 0755 /etc/apt/keyrings
...
...

 

우선 도커부터 설치

 

root@k8sadmin:~# apt-get install -y kubelet kubeadm kubectl kubernetes-cni
Reading package lists... Done
Building dependency tree       
...
...

k8s 관련 패키지 설치

 

root@k8sadmin:~# kubeadm init --apiserver-advertise-address 192.168.10.50 --pod-network-cidr=192.168.0.0/16

...
...

The connection to the server localhost:8080 was refused - did you specify the right host or port?

kubeadm init 명령으로 설치하려는데
The connection to the server localhost:8080 was refused - did you specify the right host or port? 에러가 추력됨

 

root@k8sadmin:~# mkdir -p $HOME/.kube
root@k8sadmin:~# cd .kube
root@k8sadmin:~/.kube# cp -i /etc/kubernetes/admin.conf ./config
root@k8sadmin:~/.kube# chown $(id -u):$(id -g) ./config 
root@k8sadmin:~/.kube# cd ..
root@k8sadmin:~# kubectl version
\WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff"}
Kustomize Version: v4.5.7
The connection to the server 192.168.10.50:6443 was refused - did you specify the right host or port?

홈에 .kube 폴더를 만든 후 conf 파일을 복사함

이번엔 6443 포트 맞냐고 묻는 에러 출력

 

root@k8sadmin:~# sudo -i
root@k8sadmin:~# swapoff -a
root@k8sadmin:~# strace -eopenat kubectl version
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=250257, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=250257, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=250257, si_uid=0} ---
...
...
openat(AT_FDCWD, "/root/.kube/config", O_RDONLY|O_CLOEXEC) = 3
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use 
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff"}
Kustomize Version: v4.5.7
The connection to the server 192.168.10.50:6443 was refused - did you specify the right host or port?
+++ exited with 1 +++
root@k8sadmin:~# kubectl get nodes
The connection to the server 192.168.10.50:6443 was refused - did you specify the right host or port?
root@k8sadmin:~# exit
logout
root@k8sadmin:~# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use 
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff"}
root@k8sadmin:~# kubectl get nodes
NAME       STATUS     ROLES    AGE   VERSION
k8sadmin   NotReady   <none>   3m    v1.25.4

 

포트 확인해달라는 에러가 이제 나오지 않으니 다시 설치 진행

 

root@k8sadmin:~# kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1201 15:52:05.996529  250674 reset.go:103] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to
W1201 15:52:05.996723  250674 preflight.go:55] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm 
[reset] Are you sure you want to proceed? [y/N]: y
W1201 15:52:06.949615  250674 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directo
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf 
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.


root@k8sadmin:~# kubeadm init --apiserver-advertise-address 192.168.10.50 --pod-network-cidr=192.168.0.0/16
[init] Using Kubernetes version: v1.25.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
...
...
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.50:6443 --token 2bgdgr.wu4a2xtmhy0iw6qd \
	--discovery-token-ca-cert-hash sha256:6dcbaab9766f933bafff13d3509dca1ca8905c5a9260a63d66909367e748b63b 
	
	
root@k8sadmin:~#   mkdir -p $HOME/.kube
root@k8sadmin:~#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? 
root@k8sadmin:~# 
root@k8sadmin:~#   export KUBECONFIG=/etc/kubernetes/admin.conf

kubeadm reset 으로 혹시모르니 다시 초기화 시키고

위에서 입력했던 init 명령어 다시 입력

정상적으로 잘 설치되었다.

 

중간에 표시되는 mkdir ~ 부터 3줄의 명령어는 현재 어드민으로 사용할 이 서버에 입력해준다.

가장 아래 표시되는

kubeadm join ~ 명령어는 노드서버로 사용할 곳에 가서 복사해서 입력해주면 된다.

 

 

node 서버 설치

 

root@k8snode1:~# kubeadm join 192.168.10.50:6443 --token 2bgdgr.wu4a2xtmhy0iw6qd \
> --discovery-token-ca-cert-hash sha256:6dcbaab9766f933bafff13d3509dca1ca8905c5a9260a63d66909367e748b63b 

[preflight] Running pre-flight checks
	[WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature 
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR CRI]: container runtime is not running: output: E1201 16:04:02.371658  178251 remote_runtime.go:948] "Stace"
time="2022-12-01T16:04:02+09:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unkn
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher



root@k8snode1:~# cd /etc/containerd/
root@k8snode1:/etc/containerd# ll
total 20
drwxr-xr-x   2 root root  4096 11월 28 10:26 ./
drwxr-xr-x 133 root root 12288 11월 29 06:20 ../
-rw-r--r--   1 root root   886 11월 17 23:09 config.toml
root@k8snode1:/etc/containerd# vi config.toml 

#disabled_plugins = ["cri"]
위 문구 주석처리 하면 됨
 
root@k8snode1:/etc/containerd# systemctl restart containerd

node로 사용할 서버에서 아까 복사해둔 kubdadm join 명령어 입력

 

ERROR CRI 에러가 출력되어 아래 config.toml 파일을 수정해주고 containerd 를 재기동시켜줬다.

 

 

다시

root@k8snode1:/etc/containerd# kubeadm join 192.168.10.50:6443 --token 2bgdgr.wu4a2xtmhy0iw6qd --discovery-token-ca-cert
[preflight] Running pre-flight checks
	[WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature 
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localho
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localho
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localho
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localho
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localho

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'
error execution phase kubelet-start: timed out waiting for the condition
To see the stack trace of this error execute with --v=5 or higher



root@k8snode1:/etc/containerd# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2022-12-01 17:22:45 KST; 5s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 181855 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBE
   Main PID: 181855 (code=exited, status=1/FAILURE)

12월 01 17:22:45 k8snode1 systemd[1]: kubelet.service: Failed with result 'exit-code'.



root@k8snode1:/etc/containerd# mkdir /etc/docker
mkdir: cannot create directory ‘/etc/docker’: File exists
root@k8snode1:/etc/containerd# cat <<EOF | sudo tee /etc/docker/daemon.json
> {
>   "exec-opts": ["native.cgroupdriver=systemd"],
>   "log-driver": "json-file",
>   "log-opts": {
>     "max-size": "100m"
>   },
>   "storage-driver": "overlay2"
> }
> EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

root@k8snode1:/etc/containerd# kubeadm reset
W1201 17:23:04.914076  181877 preflight.go:55] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm 
[reset] Are you sure you want to proceed? [y/N]: y
W1201 17:23:05.638268  181877 removeetcdmember.go:85] [reset] No kubeadm config, using etcd pod spec to get data directo
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf 
[reset] Deleting contents of stateful directories: [/var/lib/kubelet]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

root@k8snode1:/etc/containerd# reboot

 

다시 시도해보니 

[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' 

에러 뜨면서 진행이 안됨

그래서 kubelet 작동상태를 살펴보니 kubelet 도 loaded 상태

 

cgroup을 systemd 로 맞춰주어야 한대서 daemon.json 파일 수정 후

 

kubeadm reset

그이후 혹시 모르니 reboot 시켜주었다.

 

root@k8snode1:/etc/kubernetes# kubeadm join 192.168.10.50:6443 --token 2bgdgr.wu4a2xtmhy0iw6qd \
> --discovery-token-ca-cert-hash sha256:6dcbaab9766f933bafff13d3509dca1ca8905c5a9260a63d66909367e748b63b 

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

이후 접속해서 다시 설치하니 확인됨

 

 

 

다시 admin 서버에서 네트워크 등록

root@k8sadmin:~# kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
configmap/calico-config created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
...
...
ensure CRDs are installed first

 

이후 설치 완료 후 잘 되나 확인

root@k8sadmin:~# kubectl get nodes
NAME       STATUS     ROLES           AGE    VERSION
k8sadmin   NotReady   control-plane   102m   v1.25.4
k8snode1   Ready      <none>          17m    v1.25.4
k8snode2   Ready      <none>          6m2s   v1.25.4
root@k8sadmin:~# kubectl get pods --namespace kube-system
NAME                                       READY   STATUS              RESTARTS        AGE
calico-kube-controllers-7549b9dcd8-4tng2   0/1     ContainerCreating   0               6m6s
calico-node-qwpbt                          0/1     CrashLoopBackOff    4 (25s ago)     6m7s
calico-node-x4v44                          0/1     CrashLoopBackOff    4 (55s ago)     6m7s
calico-node-xb5gw                          0/1     CrashLoopBackOff    4 (67s ago)     6m7s
coredns-565d847f94-5hcrn                   0/1     ContainerCreating   0               105m
coredns-565d847f94-bk5d5                   0/1     ContainerCreating   0               105m
etcd-k8sadmin                              1/1     Running             1               106m
kube-apiserver-k8sadmin                    1/1     Running             1               106m
kube-controller-manager-k8sadmin           1/1     Running             3 (3m20s ago)   106m
kube-proxy-c55lp                           1/1     Running             0               10m
kube-proxy-hgmbz                           1/1     Running             0               105m
kube-proxy-rqc5s                           1/1     Running             0               21m
kube-scheduler-k8sadmin                    1/1     Running             3 (3m45s ago)   106m
root@k8sadmin:~#

 

 

 

** 근데 다음날 접속해보니 export 가 풀렸는지 에러뜸..

root@k8sadmin:~# kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
root@k8sadmin:~#

 

root@k8sadmin:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@k8sadmin:~# kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
k8sadmin   Ready    control-plane   18h   v1.25.4
k8snode1   Ready    <none>          17h   v1.25.4
k8snode2   Ready    <none>          17h   v1.25.4
root@k8sadmin:~#

어제 export 했던게 풀려서 그랬다.

bashrc 에 등록해둬야할듯