Kubernetes Pod – DNS does not resolve after establishing OpenVPN client connection

by

I have a Kubernetes deployment that when deployed into Kubernetes in docker-desktop for Mac works fine, but the exact same configuration (config files, Docker images) in Azure Kubernetes does not.

Requirements: The Pod must connect to a VPN connection, all outbound web traffic must route through the VPN connection, while maintaining connectivity to the “local” Kubernetes resources.

All networking works fine prior to establishing the VPN connection.

Route tables before the VPN connection is established:

/app # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.244.1.1      0.0.0.0         UG    0      0        0 eth0
10.244.1.0      *               255.255.255.0   U     0      0        0 eth0

Route tables after the VPN connection is established:

/app # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.7.1.1        128.0.0.0       UG    0      0        0 tun0
default         10.7.1.1        0.0.0.0         UG    0      0        0 tun0
10.7.1.0        *               255.255.255.0   U     0      0        0 tun0
10.244.1.0      *               255.255.255.0   U     0      0        0 eth0
107.174.17.243  10.244.1.1      255.255.255.255 UGH   0      0        0 eth0
128.0.0.0       10.7.1.1        128.0.0.0       UG    0      0        0 tun0

Basically, the “up” script removes the default gateway for the original network, replaces it with the VPN gateway, and the “down” script restores the original default gateway.

The primary issue is that once the VPN connection is established, I am no longer able to get any domain name resolution. kube-dns is running in both places, and the pod spec has explicit DNS configuration:

      dnsConfig:
        nameservers:
          - 8.8.8.8
          - 8.8.4.4

Again, I will reiterate all networking works fine prior to establishing the VPN connection.

When I run nslookup google.com with the VPN connection up, it works

/app # nslookup google.com
Server:         8.8.8.8
Address:        8.8.8.8:53

Non-authoritative answer:
Name:   google.com
Address: 172.217.11.238

Non-authoritative answer:
Name:   google.com
Address: 2607:f8b0:400f:800::200e

But when I run ping google.com while the VPN is up, it fails

/app # ping google.com
ping: bad address 'google.com'

However, if I know the exact IP address of the server I want to talk to, I can get it to give me a response. For example, calling CURL against Google’s previously resolved IP address.

/app # curl "http://172.217.11.238" > output.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   219  100   219    0     0    782      0 --:--:-- --:--:-- --:--:--   782
/app # cat output.txt
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
/app #

So the issue appears to be ONLY DNS resolution while the VPN connection is up, but I’m not sure how to go about fixing it.

Answer

So I managed to solve this, but I’m not really a fan.

In my OpenVPN --up script

ip route add {IP_OF_KUBE_DNS} via $network_net_gateway

This adds an explicit route to the DNS server’s IP address on the internal network, telling it to go through the original network Gateway

Attribution
Source : Link , Question Author : Matt Baker , Answer Author : Matt Baker

Leave a Comment