V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Mrkon
V2EX  ›  Elasticsearch

求助: 没有到主机的路由(no route to host)

  •  
  •   Mrkon · 43 天前 · 590 次点击
    这是一个创建于 43 天前的主题,其中的信息可能已经有所发展或是发生改变。

    系统:ubuntu 16.04 (已经关闭防火墙,sudo ufw disable, telnet 可以联通) ES 版本 : 7.13.1

    目的: 使用两个节点通过外网组成集群

    问题描述: 已经开启一台服务器作为 master 节点,将第二个节点加入时报错: 没有到主机的路由

    关键配置:

    network.host: 0.0.0.0
    network.publish_host: 192.168.3.134
    discovery.zen.minimum_master_nodes: 2
    discovery.seed_hosts: ["61.15.284.58:10091", "1.65.184.8:10081"]
    cluster.initial_master_nodes: ["node-1", "node-2"]
    
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    

    我的猜想: DNS 原因,我通过路由器将内网端口映射到固定 IP 导致的问题,报错是找不到内网 192.168.3.86:19300

    映射关系: node-1: 61.15.284.58:10091 -> 192.168.3.86:19300

    node-2: 1.65.184.8:10081 -> 192.168.3.134:29300

    错误 log:

    [2021-09-24T09:29:32,786][WARN ][o.e.d.HandshakingTransportAddressConnector] [
    node-2] [connectToRemoteMasterNode[61.15.284.58:10091]] completed handshake with [{node-1}{oapoq8khQ9GtiWj63kfbCQ}{VwXYChZEQdKceJYNrCRPmw}{192.168.3.86}{192.168.3.86:19300}{cdfhilmrstw}{ml.machine_memory=33599610880, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=12884901888, transform.node=true}] but followup connection failed
    org.elasticsearch.transport.ConnectTransportException: [node-1][192.168.3.86:19300] connect_exception
            at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:971) ~[elasticsearch-7.13.1.jar:7.13.1]
            at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[elasticsearch-7.13.1.jar:7.13.1]
            at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.13.1.jar:7.13.1]
            at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
            at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
            at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
            at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
            at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.13.1.jar:7.13.1]
            at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:57) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
            at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
            at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
            at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
            at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
            at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
            at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
            at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
            at java.lang.Thread.run(Thread.java:831) [?:?]
    Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: 没有到主机的路由: 192.168.3.86/192.168.3.86:19300
    Caused by: java.net.NoRouteToHostException: 没有到主机的路由
            at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
            at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
            at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
            at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
            at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
            ... 7 more
    
    
    8 条回复    2021-11-12 16:00:58 +08:00
    wujiezero
        1
    wujiezero   43 天前
    我上次遇到这个问题改了下 /etc/hosts 好了
    Mrkon
        2
    Mrkon   43 天前
    @wujiezero 感谢回复,该如何修改呢?
    julyclyde
        3
    julyclyde   43 天前
    iptables 的问题。对方使用了 ICMP host unreachable 来拒绝连接
    Mrkon
        4
    Mrkon   43 天前
    @julyclyde 感觉不太像我的 iptables 表分别是

    node1

    ```
    Chain INPUT (policy ACCEPT)
    target prot opt source destination
    ACCEPT tcp -- anywhere anywhere tcp dpt:9600

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination
    DOCKER-USER all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

    Chain DOCKER (11 references)
    target prot opt source destination
    ACCEPT tcp -- anywhere 172.21.0.2 tcp dpt:tproxy
    ACCEPT tcp -- anywhere 172.21.0.2 tcp dpt:http-alt
    ACCEPT tcp -- anywhere 192.168.32.2 tcp dpt:3000
    ACCEPT tcp -- anywhere 172.20.0.2 tcp dpt:6379
    ACCEPT tcp -- anywhere 172.24.0.2 tcp dpt:7687
    ACCEPT tcp -- anywhere 172.24.0.2 tcp dpt:7474
    ACCEPT tcp -- anywhere 172.24.0.3 tcp dpt:27017
    ACCEPT tcp -- anywhere 172.24.0.4 tcp dpt:6379
    ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:5000
    ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:3000
    ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:ssh
    ACCEPT tcp -- anywhere 172.20.0.4 tcp dpt:3000
    ACCEPT tcp -- anywhere 172.20.0.3 tcp dpt:5000

    Chain DOCKER-ISOLATION-STAGE-1 (1 references)
    target prot opt source destination
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    RETURN all -- anywhere anywhere

    Chain DOCKER-ISOLATION-STAGE-2 (11 references)
    target prot opt source destination
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    RETURN all -- anywhere anywhere

    Chain DOCKER-USER (1 references)
    target prot opt source destination
    RETURN all -- anywhere anywhere
    ```

    node2

    ```
    Chain INPUT (policy ACCEPT)
    target prot opt source destination

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination
    ```
    julyclyde
        5
    julyclyde   43 天前
    看 192.168.3.86 那边的 iptables
    @Mrkon
    Mrkon
        6
    Mrkon   41 天前
    @julyclyde 192.168.3.86 的 iptables 是:上面 node1 的,就是 ACCEPT 。我实在是找不到问题点了,能帮我看下吗?

    还有如果安装了 docker,当 es network.host 设置为:0.0.0.0 时,好像默认地址就会绑定为 docker 的 IP 段时怎么回事呢?
    julyclyde
        7
    julyclyde   40 天前
    @Mrkon 那你抓包看看吧。看看是不是 ICMP host unreacable 、是谁发过来的
    Mrkon
        8
    Mrkon   15 天前
    看了下官方文档,问题解决了,需要将所有节点 network.publish_host 设置映射前的 ip 地址 例如:61.15.284.58 ,transport.publish_port 同样需要设置 例如:10091
    关于   ·   帮助文档   ·   API   ·   FAQ   ·   我们的愿景   ·   广告投放   ·   感谢   ·   实用小工具   ·   2414 人在线   最高记录 5497   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 08:29 · PVG 16:29 · LAX 00:29 · JFK 03:29
    ♥ Do have faith in what you're doing.