techtsubame’s blog

備忘録であり、何が起きても責任は取りません

ServiceのClusterIPとkube-proxyについて(iptablesモード)

CentOS kubectl service kubernetes pods Service

実施すること

  • kube-proxyの動作モードの概要を理解する
  • 実際にiptablesを見て設定を確認する

引用元

qiita.com

kube-proxy

  • ノードのネットワーク設定を実施に行うコンポーネント
  • ServiceのIPやロードバランサの実態はなく、iptablesまたはipvs上の設定
  • デフォルトはiptablesモード

試してみる

iptables確認 (任意のワーカーノード)

[tsubame@worker01 ~]$ sudo iptables -n -t nat -L > before_iptables.txt

deployment(pods)作成

[tsubame@control-plane01 ~]$kubectl create deployment --image nginx nginx
[tsubame@control-plane01 ~]$

service作成

[tsubame@control-plane01 ~]$ kubectl expose deployment --port 80 --target-port 80 nginx
service/nginx exposed
[tsubame@control-plane01 ~]$ kubectl get service nginx
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.97.152.214   <none>        80/TCP    5s
[tsubame@control-plane01 ~]$

iptables比較

[tsubame@worker01 ~]$ sudo iptables -n -t nat -L > after_iptables.txt
[tsubame@worker01 ~]$
[tsubame@worker01 ~]$ diff -U0 before_iptables.txt  after_iptables.txt  | cat -n
     1 --- before_iptables.txt 2023-08-17 14:37:44.626615800 +0900
     2 +++ after_iptables.txt  2023-08-17 14:40:17.896888507 +0900
     3 @@ -19 +19 @@
     4 -Chain KUBE-MARK-MASQ (17 references)
     5 +Chain KUBE-MARK-MASQ (19 references)
     6 @@ -69,0 +70,5 @@
     7 +Chain KUBE-SEP-POSKT3RSSQHQXPFY (1 references)
     8 +target     prot opt source               destination
     9 +KUBE-MARK-MASQ  all  --  192.168.133.2        0.0.0.0/0            /* default/nginx */
    10 +DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.133.2:80
    11 +
    12 @@ -91,0 +97,3 @@
    15 +KUBE-SVC-2CMXP7HKUVJN7L6M  tcp  --  0.0.0.0/0            10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
    19 @@ -98,0 +105,5 @@
    20 +
    21 +Chain KUBE-SVC-2CMXP7HKUVJN7L6M (1 references)
    22 +target     prot opt source               destination
    23 +KUBE-MARK-MASQ  tcp  -- !192.168.128.0/18     10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
    24 +KUBE-SEP-POSKT3RSSQHQXPFY  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.133.2:80 */
[tsubame@worker01 ~]$

iptablesの追加の流れ

全体のサービスに定義を追加

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
+ KUBE-SVC-2CMXP7HKUVJN7L6M  tcp  --  0.0.0.0/0            10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
KUBE-SVC-I24EZXP75AX5E7TU  tcp  --  0.0.0.0/0            10.107.73.67         /* calico-apiserver/calico-api:apiserver cluster IP */ tcp dpt:443
KUBE-SVC-RK657RLKDNVNU64O  tcp  --  0.0.0.0/0            10.98.16.233         /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:5473
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Serviceに対応するChainを追加

Chain KUBE-SVC-2CMXP7HKUVJN7L6M (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  tcp  -- !192.168.128.0/18     10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
KUBE-SEP-POSKT3RSSQHQXPFY  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.133.2:80 */

以下のChainを作成

Chain KUBE-SEP-POSKT3RSSQHQXPFY (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  192.168.133.2        0.0.0.0/0            /* default/nginx */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.133.2:80

podの数を３個に増やしてみる

kubectlにてreplicasを変更

[tsubame@control-plane01 ~]$ kubectl edit deployments.apps nginx

--- snip ---
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
--- snip --- 

deployment.apps/nginx edited
[tsubame@control-plane01 ~]$


Every 1.0s: kubectl get pods -o wide                                                                                                              control-plane01: Thu Aug 17 15:40:05 2023

NAME                     READY   STATUS    RESTARTS   AGE   IP                NODE       NOMINATED NODE   READINESS GATES
nginx-7f5bdb5fd4-98q8c   1/1     Running   0          18s   192.168.133.5     worker01   <none>           <none>
nginx-7f5bdb5fd4-9dpnl   1/1     Running   0          13s   192.168.133.152   worker03   <none>           <none>
nginx-7f5bdb5fd4-kqfhl   1/1     Running   0          16s   192.168.158.98    worker02   <none>           <none>

iptablesの差分

[tsubame@worker01 ~]$ diff -U0 before_iptables.txt after_3_iptables.txt
--- before_iptables.txt 2023-08-17 14:37:44.626615800 +0900
+++ after_3_iptables.txt    2023-08-17 15:40:43.179526545 +0900
@@ -19 +19 @@
-Chain KUBE-MARK-MASQ (17 references)
+Chain KUBE-MARK-MASQ (21 references)
@@ -35 +35 @@
-Chain KUBE-SEP-23EG545C6FRHA2U2 (1 references)
+Chain KUBE-SEP-257TP7TB2MGU3OCP (1 references)
@@ -37,2 +37,7 @@
-KUBE-MARK-MASQ  all  --  192.168.184.194      0.0.0.0/0            /* kube-system/kube-dns:dns */
-DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.184.194:53
+KUBE-MARK-MASQ  all  --  192.168.184.203      0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */ tcp to:192.168.184.203:5443
+
+Chain KUBE-SEP-2JUOIR6IA5SUWG3I (1 references)
+target     prot opt source               destination
+KUBE-MARK-MASQ  all  --  192.168.184.196      0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */ tcp to:192.168.184.196:5443
@@ -44,0 +50,15 @@
+Chain KUBE-SEP-4HNY76IF6CK63TCP (1 references)
+target     prot opt source               destination
+KUBE-MARK-MASQ  all  --  192.168.133.152      0.0.0.0/0            /* default/nginx */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.133.152:80
+
+Chain KUBE-SEP-6GTBKAZV3WD7EDL6 (1 references)
+target     prot opt source               destination
+KUBE-MARK-MASQ  all  --  192.168.184.193      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.184.193:53
+
+Chain KUBE-SEP-6V2ZLF2VEGR4VBTT (1 references)
+target     prot opt source               destination
+KUBE-MARK-MASQ  all  --  192.168.184.195      0.0.0.0/0            /* kube-system/kube-dns:dns */
+DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.184.195:53
+
@@ -50,6 +70 @@
-Chain KUBE-SEP-I4QJKZN36IZCLXAY (1 references)
-target     prot opt source               destination
-KUBE-MARK-MASQ  all  --  192.168.50.88        0.0.0.0/0            /* calico-system/calico-typha:calico-typha */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* calico-system/calico-typha:calico-typha */ tcp to:192.168.50.88:5473
-
-Chain KUBE-SEP-JJQPEIJ7BP3AIODN (1 references)
+Chain KUBE-SEP-EKBOV47Q3GE6GMSN (1 references)
@@ -57,2 +72,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.255      0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */ tcp to:192.168.184.255:5443
+KUBE-MARK-MASQ  all  --  192.168.158.98       0.0.0.0/0            /* default/nginx */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.158.98:80
@@ -60 +75 @@
-Chain KUBE-SEP-KJFZIJFS5TZL53K3 (1 references)
+Chain KUBE-SEP-HKRVYPYLPFHSI25O (1 references)
@@ -62,2 +77,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.198      0.0.0.0/0            /* kube-system/kube-dns:metrics */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.184.198:9153
+KUBE-MARK-MASQ  all  --  192.168.184.195      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.184.195:53
@@ -65 +80 @@
-Chain KUBE-SEP-NLJGA2YTZBYPIVCY (1 references)
+Chain KUBE-SEP-I4QJKZN36IZCLXAY (1 references)
@@ -67,2 +82,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.254      0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver */ tcp to:192.168.184.254:5443
+KUBE-MARK-MASQ  all  --  192.168.50.88        0.0.0.0/0            /* calico-system/calico-typha:calico-typha */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* calico-system/calico-typha:calico-typha */ tcp to:192.168.50.88:5473
@@ -70 +85 @@
-Chain KUBE-SEP-PUYUIQ3REKJQ2RI4 (1 references)
+Chain KUBE-SEP-ICL5G2MSQTI3NGGI (1 references)
@@ -72,2 +87,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.198      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.184.198:53
+KUBE-MARK-MASQ  all  --  192.168.184.193      0.0.0.0/0            /* kube-system/kube-dns:dns */
+DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.184.193:53
@@ -75 +90 @@
-Chain KUBE-SEP-RYZQZPQA2LNWFBSP (1 references)
+Chain KUBE-SEP-IYM6AZ63GISRWKBE (1 references)
@@ -77,2 +92,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.194      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.184.194:53
+KUBE-MARK-MASQ  all  --  192.168.133.5        0.0.0.0/0            /* default/nginx */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.133.5:80
@@ -80 +95 @@
-Chain KUBE-SEP-TY3DRXQSLWZJ5MOP (1 references)
+Chain KUBE-SEP-JP2LKO4NRHUMHLPU (1 references)
@@ -82,2 +97,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.194      0.0.0.0/0            /* kube-system/kube-dns:metrics */
-DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.184.194:9153
+KUBE-MARK-MASQ  all  --  192.168.184.195      0.0.0.0/0            /* kube-system/kube-dns:metrics */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.184.195:9153
@@ -85 +100 @@
-Chain KUBE-SEP-UEC2AOGO4GZPWFQG (1 references)
+Chain KUBE-SEP-TQ6RLKMB4GESVDUB (1 references)
@@ -87,2 +102,2 @@
-KUBE-MARK-MASQ  all  --  192.168.184.198      0.0.0.0/0            /* kube-system/kube-dns:dns */
-DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.184.198:53
+KUBE-MARK-MASQ  all  --  192.168.184.193      0.0.0.0/0            /* kube-system/kube-dns:metrics */
+DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.184.193:9153
@@ -92,2 +106,0 @@
-KUBE-SVC-I24EZXP75AX5E7TU  tcp  --  0.0.0.0/0            10.107.73.67         /* calico-apiserver/calico-api:apiserver cluster IP */ tcp dpt:443
-KUBE-SVC-RK657RLKDNVNU64O  tcp  --  0.0.0.0/0            10.98.16.233         /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:5473
@@ -94,0 +108 @@
+KUBE-SVC-2CMXP7HKUVJN7L6M  tcp  --  0.0.0.0/0            10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
@@ -97,0 +112,2 @@
+KUBE-SVC-I24EZXP75AX5E7TU  tcp  --  0.0.0.0/0            10.107.73.67         /* calico-apiserver/calico-api:apiserver cluster IP */ tcp dpt:443
+KUBE-SVC-RK657RLKDNVNU64O  tcp  --  0.0.0.0/0            10.98.16.233         /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:5473
@@ -99,0 +116,7 @@
+Chain KUBE-SVC-2CMXP7HKUVJN7L6M (1 references)
+target     prot opt source               destination
+KUBE-MARK-MASQ  tcp  -- !192.168.128.0/18     10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
+KUBE-SEP-4HNY76IF6CK63TCP  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.133.152:80 */ statistic mode random probability 0.33333333349
+KUBE-SEP-IYM6AZ63GISRWKBE  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.133.5:80 */ statistic mode random probability 0.50000000000
+KUBE-SEP-EKBOV47Q3GE6GMSN  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.158.98:80 */
+
@@ -103,2 +126,2 @@
-KUBE-SEP-RYZQZPQA2LNWFBSP  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp -> 192.168.184.194:53 */ statistic mode random probability 0.50000000000
-KUBE-SEP-PUYUIQ3REKJQ2RI4  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp -> 192.168.184.198:53 */
+KUBE-SEP-6GTBKAZV3WD7EDL6  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp -> 192.168.184.193:53 */ statistic mode random probability 0.50000000000
+KUBE-SEP-HKRVYPYLPFHSI25O  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp -> 192.168.184.195:53 */
@@ -109,2 +132,2 @@
-KUBE-SEP-NLJGA2YTZBYPIVCY  all  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver -> 192.168.184.254:5443 */ statistic mode random probability 0.50000000000
-KUBE-SEP-JJQPEIJ7BP3AIODN  all  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver -> 192.168.184.255:5443 */
+KUBE-SEP-2JUOIR6IA5SUWG3I  all  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver -> 192.168.184.196:5443 */ statistic mode random probability 0.50000000000
+KUBE-SEP-257TP7TB2MGU3OCP  all  --  0.0.0.0/0            0.0.0.0/0            /* calico-apiserver/calico-api:apiserver -> 192.168.184.203:5443 */
@@ -115,2 +138,2 @@
-KUBE-SEP-TY3DRXQSLWZJ5MOP  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics -> 192.168.184.194:9153 */ statistic mode random probability 0.50000000000
-KUBE-SEP-KJFZIJFS5TZL53K3  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics -> 192.168.184.198:9153 */
+KUBE-SEP-TQ6RLKMB4GESVDUB  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics -> 192.168.184.193:9153 */ statistic mode random probability 0.50000000000
+KUBE-SEP-JP2LKO4NRHUMHLPU  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics -> 192.168.184.195:9153 */
@@ -132,2 +155,2 @@
-KUBE-SEP-23EG545C6FRHA2U2  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns -> 192.168.184.194:53 */ statistic mode random probability 0.50000000000
-KUBE-SEP-UEC2AOGO4GZPWFQG  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns -> 192.168.184.198:53 */
+KUBE-SEP-ICL5G2MSQTI3NGGI  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns -> 192.168.184.193:53 */ statistic mode random probability 0.50000000000
+KUBE-SEP-6V2ZLF2VEGR4VBTT  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns -> 192.168.184.195:53 */
[tsubame@worker01 ~]$

見てみる

KUBE-ERVICESを見る

  • 変わらず
Chain KUBE-SERVICES (2 references)
target     prot opt source               destination
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-2CMXP7HKUVJN7L6M  tcp  --  0.0.0.0/0            10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
KUBE-SVC-I24EZXP75AX5E7TU  tcp  --  0.0.0.0/0            10.107.73.67         /* calico-apiserver/calico-api:apiserver cluster IP */ tcp dpt:443
KUBE-SVC-RK657RLKDNVNU64O  tcp  --  0.0.0.0/0            10.98.16.233         /* calico-system/calico-typha:calico-typha cluster IP */ tcp dpt:5473
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

サービスのClusterIPがDesitinatioinにあるKUBE-SVC-2CMXP7HKUVJN7L6Mを見る

  • podsに設定されているIPアドレス分定義が追加されている
  • podは1個のときにあったKUBE-SEP-POSKT3RSSQHQXPFY のchainは削除されている
Chain KUBE-SVC-2CMXP7HKUVJN7L6M (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  tcp  -- !192.168.128.0/18     10.97.152.214        /* default/nginx cluster IP */ tcp dpt:80
KUBE-SEP-4HNY76IF6CK63TCP  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.133.152:80 */ statistic mode random probability 0.33333333349
KUBE-SEP-IYM6AZ63GISRWKBE  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.133.5:80 */ statistic mode random probability 0.50000000000
KUBE-SEP-EKBOV47Q3GE6GMSN  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx -> 192.168.158.98:80 */

KUBE-SEP-4HNY76IF6CK63TCP

Chain KUBE-SEP-4HNY76IF6CK63TCP (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  192.168.133.152      0.0.0.0/0            /* default/nginx */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.133.152:80

KUBE-SEP-IYM6AZ63GISRWKBE

Chain KUBE-SEP-IYM6AZ63GISRWKBE (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  192.168.133.5        0.0.0.0/0            /* default/nginx */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.133.5:80

KUBE-SEP-EKBOV47Q3GE6GMSN

Chain KUBE-SEP-EKBOV47Q3GE6GMSN (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  192.168.158.98       0.0.0.0/0            /* default/nginx */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx */ tcp to:192.168.158.98:80

図としては以下