V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
ice2016
V2EX  ›  Kubernetes

kube-prometheus-0.7 部署出错问题求助

  •  
  •   ice2016 · 55 天前 · 600 次点击
    这是一个创建于 55 天前的主题,其中的信息可能已经有所发展或是发生改变。

    hUdgY9.png 系统环境是 CentOS7.9 + K8s v1.19.14 ,根据官方兼容性说明下载安装的是 kube-prometheus-release-0.7 版本,老是报错,安装包是 https://github.com/prometheus-operator/kube-prometheus/archive/refs/heads/release-0.7.zip

    cd /root/kube-prometheus-release-0.7/manifests
    kubectl create -f setup
    
    [[email protected] manifests]# kubectl get pod -o wide -n monitoring
    NAME                                   READY   STATUS             RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
    prometheus-operator-7649c7454f-q9m29   1/2     CrashLoopBackOff   6          10m   10.201.1.58   k8s-node1   <none>           <none>
    

    查看日志:

    [[email protected] manifests]# kubectl logs prometheus-operator-7649c7454f-q9m29 --all-containers -n monitoring
    level=info ts=2021-08-31T03:16:07.410585761Z caller=main.go:235 msg="Starting Prometheus Operator" version="(version=0.44.1, branch=refs/tags/pkg/apis/monitoring/v0.44.1, revision=d8b7d3766225908d0239fd0d78258892cd0fc384)"
    level=info ts=2021-08-31T03:16:07.410684044Z caller=main.go:236 build_context="(go=go1.14.12, user=simonpasquier, date=20201209-17:17:30)"
    ts=2021-08-31T03:16:07.415831751Z caller=main.go:107 msg="Starting insecure server on [::]:8080"
    ts=2021-08-31T03:16:10.422020132Z caller=main.go:422 msg="Unhandled error received. Exiting..." err="communicating with server failed: Get \"https://10.96.0.1:443/version?timeout=32s\": dial tcp 10.96.0.1:443: connect: connection timed out"
    I0831 03:10:09.496650       1 main.go:190] Valid token audiences:
    I0831 03:10:09.496764       1 main.go:262] Generating self signed cert as no cert is provided
    I0831 03:10:10.447004       1 main.go:311] Starting TCP socket on :8443
    I0831 03:10:10.447347       1 main.go:318] Listening securely on :8443
    

    根据报错测试几个节点上 10.96.0.1:443 这个地址都可以访问

    [[email protected] manifests]# curl -k https://10.96.0.1:443/version?timeout=32s
    {
      "major": "1",
      "minor": "19",
      "gitVersion": "v1.19.14",
      "gitCommit": "0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c",
      "gitTreeState": "clean",
      "buildDate": "2021-08-11T18:02:17Z",
      "goVersion": "go1.15.15",
      "compiler": "gc",
      "platform": "linux/amd64"
    
    

    iptables 信息:

    [[email protected] manifests]# iptables -L -n
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
    KUBE-EXTERNAL-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    KUBE-FORWARD  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
    DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0
    DOCKER-ISOLATION-STAGE-1  all  --  0.0.0.0/0            0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
    ACCEPT     all  --  10.244.0.0/16        0.0.0.0/0
    ACCEPT     all  --  0.0.0.0/0            10.244.0.0/16
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0
    
    Chain DOCKER (1 references)
    target     prot opt source               destination
    
    Chain DOCKER-ISOLATION-STAGE-1 (1 references)
    target     prot opt source               destination
    DOCKER-ISOLATION-STAGE-2  all  --  0.0.0.0/0            0.0.0.0/0
    RETURN     all  --  0.0.0.0/0            0.0.0.0/0
    
    Chain DOCKER-ISOLATION-STAGE-2 (1 references)
    target     prot opt source               destination
    DROP       all  --  0.0.0.0/0            0.0.0.0/0
    RETURN     all  --  0.0.0.0/0            0.0.0.0/0
    
    Chain DOCKER-USER (1 references)
    target     prot opt source               destination
    RETURN     all  --  0.0.0.0/0            0.0.0.0/0
    
    Chain KUBE-EXTERNAL-SERVICES (1 references)
    target     prot opt source               destination
    
    Chain KUBE-FIREWALL (2 references)
    target     prot opt source               destination
    DROP       all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
    DROP       all  -- !127.0.0.0/8          127.0.0.0/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
    
    Chain KUBE-FORWARD (1 references)
    target     prot opt source               destination
    DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */ mark match 0x4000/0x4000
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
    
    Chain KUBE-KUBELET-CANARY (0 references)
    target     prot opt source               destination
    
    Chain KUBE-PROXY-CANARY (0 references)
    target     prot opt source               destination
    
    Chain KUBE-SERVICES (3 references)
    target     prot opt source               destination
    

    环境信息如下:

    [[email protected] ~]# kubectl version
    Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:07:41Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:02:17Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
    
    [[email protected] ~]# cat /etc/redhat-release
    CentOS Linux release 7.9.2009 (Core)
    
    [[email protected] ]# kubectl get node
    NAME         STATUS   ROLES    AGE   VERSION
    k8s-master   Ready    master   10d   v1.19.14
    k8s-node1    Ready    <none>   10d   v1.19.14
    k8s-node2    Ready    <none>   10d   v1.19.14
    k8s-node3    Ready    <none>   10d   v1.19.14
    
    6 条回复    2021-09-03 08:57:29 +08:00
    defunct9
        1
    defunct9   55 天前
    开 ssh,让我上去看看
    vemier
        2
    vemier   55 天前
    @defunct9 老哥又来了
    nino789pzw
        3
    nino789pzw   54 天前
    10.96.0.1 是什么的 ip ?
    你的 curl 是 insecure 的 command,ssh 到 Promethus 的 pod 上 ssh 到 10.96.0.1 试试?
    ice2016
        4
    ice2016   54 天前
    @nino789pzw 把网络插件由 flannel 换成 calico 以后,正常部署了,也不报错,但是打开 svc
    对外暴露端口以后,一会正常一会就打不开了,过会又正常,,
    感觉哪里不对。。
    SIGEV13
        5
    SIGEV13   54 天前
    老哥去开个 issue, 看看社区有没有懂得。
    ethangogogo
        6
    ethangogogo   52 天前
    暴露端口之后有报错吗?
    关于   ·   帮助文档   ·   FAQ   ·   API   ·   我们的愿景   ·   广告投放   ·   感谢   ·   实用小工具   ·   3252 人在线   最高记录 5497   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 05:20 · PVG 13:20 · LAX 22:20 · JFK 01:20
    ♥ Do have faith in what you're doing.