Case study

Kubernetes application uses a public DNS name for an internal API: how to keep traffic inside the cluster

The client contacted us because their website would occasionally become noticeably slow.

Most of the time it worked normally. CPU and memory utilization looked reasonable, application Pods were healthy, and there was no obvious correlation with deployment activity or database load.

Then, without an apparent change inside the cluster, page generation time would increase.

A few minutes later it could return to normal.

That pattern made the problem difficult to reproduce and easy to misattribute to the application itself.

The first useful correlation: external network RTT

During one of the slow periods, we compared application response time with network measurements from the Kubernetes nodes and Pods.

The interesting part was not latency between Kubernetes Services.

It was latency to external network resources.

Periods of poor website performance correlated with increased RTT outside the cluster.

The exact trigger was not always the same. We observed several conditions that could produce it:

slower external DNS resolution;
temporary saturation of the external network channel;
increased packet loss or RTT upstream;
routing changes;
provider maintenance or degradation.

None of these conditions was severe enough to make the site completely unavailable.

They simply added latency.

The question was why external network latency affected requests between components that were running in the same Kubernetes cluster.

Finding the unexpected external dependency

The application used an API URL similar to:

`1`	`https://api.example.com`

That was perfectly reasonable for browsers accessing the application from the Internet.

The problem was that the server-side application used the same URL.

A request originating inside a Pod therefore followed roughly this path:

  flowchart LR
    APP["Application Pod"]
    DNS["CoreDNS"]
    PDNS["Public DNS"]
    EDGE["Public network / CDN / Load Balancer"]
    GW["Kubernetes Gateway"]
    API["API Service"]

    APP --> DNS
    DNS --> PDNS
    PDNS --> EDGE
    EDGE --> GW
    GW --> API

The application and API could be running on neighboring Kubernetes nodes, but communication between them still depended on external infrastructure.

In effect, an internal request was leaving the cluster and entering it again.

This made application latency dependent on systems that should not have been involved in that request at all.

Why the problem was intermittent

This also explained why the website was difficult to troubleshoot.

When external RTT was normal, the extra network path added little enough latency that nobody noticed.

When external DNS, the uplink, upstream routing, or the provider became slower, every server-side API request inherited that delay.

For example, a page render may require several API calls:

frontend
    -> /api/user
    -> /api/menu
    -> /api/products
    -> /api/permissions

If each request gains additional network latency, the effect can accumulate.

Depending on whether the application performs those requests sequentially or in parallel, a relatively modest increase in RTT can become a visible increase in page generation time.

The infrastructure itself can therefore look healthy:

CPU                OK
Memory             OK
Database           OK
Pods               OK
Kubernetes network OK

while users still experience a slow site.

The hidden dependency is:

server-side request
        ↓
public DNS
        ↓
external network path
        ↓
same Kubernetes cluster

The obvious solution is not always possible

If application configuration can distinguish between browser-side and server-side API URLs, the cleanest solution is simple.

Use the public hostname for browsers:

`1`	`https://api.example.com`

and Kubernetes service discovery for server-side communication:

`1`	`http://api.backend.svc.cluster.local:8080`

For example:

1
2

PUBLIC_API_URL=https://api.example.com
INTERNAL_API_URL=http://api.backend.svc.cluster.local:8080

Then the internal request becomes:

  flowchart LR
    APP["Application Pod"]
    API["API Service"]

    APP -->|"cluster-local traffic"| API

No public DNS resolution, external route, CDN, or public load balancer is involved.

But changing the application was not necessarily the best option in this case.

The same API hostname was already part of the application’s configuration, and there was another complication: the hostname did not necessarily map to a single Kubernetes Service.

One hostname can represent several Kubernetes Services

Consider a Gateway API configuration where:

`1`	`https://api.example.com/auth/*`

goes to one Service,

`1`	`https://api.example.com/files/*`

goes to another,

and everything else goes to the main API.

For example:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: public-api
  namespace: application
spec:
  parentRefs:
    - name: public-gateway
      namespace: gateway-system

  hostnames:
    - api.example.com

  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /auth
      backendRefs:
        - name: auth-api
          port: 8080

    - matches:
        - path:
            type: PathPrefix
            value: /files
      backendRefs:
        - name: files-api
          port: 8080

    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: core-api
          port: 8080

In this configuration, changing:

`1`	`api.example.com`

to:

`1`	`core-api.application.svc.cluster.local`

would be wrong.

The request:

`1`	`https://api.example.com/auth/login`

must still pass through the routing layer so that /auth reaches auth-api.

The same applies to routing based on headers, weighted backends, URL rewrites, redirects, or other Gateway policies.

So the requirement became more precise:

Keep api.example.com unchanged from the application’s point of view, but resolve it to a cluster-internal entry point when the request originates inside Kubernetes.

Split-horizon DNS

This can be implemented with internal DNS resolution.

Externally:

1
2
3

api.example.com
    ↓
public address

Inside Kubernetes:

1
2
3

api.example.com
    ↓
internal Gateway

The application continues requesting exactly the same URL:

`1`	`https://api.example.com/auth/login`

but CoreDNS directs the connection to an internal Kubernetes destination.

  flowchart LR
    APP["Application Pod"]
    DNS["CoreDNS"]
    GW["Internal Gateway"]
    ROUTE["HTTPRoute"]
    AUTH["auth-api"]
    FILES["files-api"]
    CORE["core-api"]

    APP -->|"api.example.com"| DNS
    DNS -->|"internal address"| GW
    GW --> ROUTE

    ROUTE -->|"/auth/*"| AUTH
    ROUTE -->|"/files/*"| FILES
    ROUTE -->|"/*"| CORE

This has an important property: only DNS resolution changes.

The URL still contains:

`1`	`api.example.com`

so the HTTP request still carries:

`1`	`Host: api.example.com`

and an HTTPS connection still uses api.example.com as the TLS server name.

Gateway routing based on the original hostname therefore continues to work.

CoreDNS rewrite

One way to implement this inside Kubernetes is with the CoreDNS rewrite plugin.

Assume the internal Gateway is reachable through:

`1`	`internal-api-gateway.gateway-system.svc.cluster.local`

A CoreDNS rule can map the public name to it:

rewrite stop {
    name exact api.example.com internal-api-gateway.gateway-system.svc.cluster.local
    answer auto
}

The lookup then becomes:

Application
    |
    | lookup api.example.com
    v
CoreDNS
    |
    | rewrite
    v
internal-api-gateway.gateway-system.svc.cluster.local
    |
    v
Gateway Service ClusterIP

The application does not need to know that this happened.

A simplified CoreDNS configuration could look like:

.:53 {
    errors
    health
    ready

    rewrite stop {
        name exact api.example.com internal-api-gateway.gateway-system.svc.cluster.local
        answer auto
    }

    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }

    prometheus :9153

    forward . /etc/resolv.conf

    cache 30
    loop
    reload
    loadbalance
}

Now the application still connects to:

`1`	`https://api.example.com`

but the packets remain inside the cluster.

Keep the Gateway when it provides application routing

For a single backend, CoreDNS could theoretically point directly to a Service.

For more complex applications, keeping an internal Gateway is safer.

The resulting architecture looks like this:

  flowchart TD
    INTERNET["Internet clients"]
    PODS["Application Pods"]

    PDNS["Public DNS"]
    CDNS["CoreDNS"]

    PUBLIC["Public Gateway"]
    INTERNAL["Internal Gateway"]

    ROUTE["HTTPRoute<br/>api.example.com"]

    AUTH["auth-api"]
    FILES["files-api"]
    CORE["core-api"]

    INTERNET --> PDNS
    PDNS --> PUBLIC

    PODS --> CDNS
    CDNS --> INTERNAL

    PUBLIC --> ROUTE
    INTERNAL --> ROUTE

    ROUTE --> AUTH
    ROUTE --> FILES
    ROUTE --> CORE

Both external and internal clients use the same logical hostname and routing rules.

The difference is only how they reach the Gateway.

HTTPS requires special attention

DNS rewriting does not change ports or terminate TLS.

If the application requests:

`1`	`https://api.example.com`

the internal endpoint must support HTTPS on the expected port.

This would not work:

1
2
3

https://api.example.com:443
        ↓
api-service:8080 HTTP

DNS cannot convert an HTTPS connection into HTTP.

An internal Gateway is useful here because it can terminate TLS using a certificate valid for:

`1`	`api.example.com`

and then forward requests to internal HTTP Services.

For example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal-api-gateway
  namespace: gateway-system
spec:
  gatewayClassName: example-gateway-class

  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: api.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: api-example-com-tls

The client still verifies the certificate against api.example.com, so the certificate presented by the internal Gateway must be valid for that hostname.

Test the route before changing DNS

DNS does not need to be changed to test the architecture.

Suppose the internal Gateway Service has ClusterIP:

`1`	`10.96.80.25`

From the application Pod:

1
2
3

curl \
  --resolve api.example.com:443:10.96.80.25 \
  https://api.example.com/auth/health

curl --resolve forces the connection to the specified IP while preserving the hostname.

That means the test still uses:

`1`	`Host: api.example.com`

and TLS still sees:

`1`	`api.example.com`

This makes it a useful test for exactly this scenario.

Test all important routes:

1
2
3

curl \
  --resolve api.example.com:443:10.96.80.25 \
  https://api.example.com/auth/health

1
2
3

curl \
  --resolve api.example.com:443:10.96.80.25 \
  https://api.example.com/files/health

1
2
3

curl \
  --resolve api.example.com:443:10.96.80.25 \
  https://api.example.com/v1/health

Only after those requests reach the expected backends should the DNS override be introduced.

Verify from the actual application Pod

After applying the CoreDNS change:

1
2

kubectl exec -n application deploy/frontend -- \
  getent hosts api.example.com

The returned address should now belong to the internal path rather than the public load balancer or CDN.

Then test the real URL:

1
2

kubectl exec -n application deploy/frontend -- \
  curl -v https://api.example.com/auth/health

There should be no need for --resolve anymore.

The Gateway route should also be checked:

`1`	`kubectl get httproute -A`

and:

1
2
3

kubectl describe httproute \
  -n application \
  public-api

For troubleshooting, it is worth checking the complete path rather than stopping after successful DNS resolution:

DNS resolution
      ↓
TCP connection
      ↓
TLS handshake
      ↓
Gateway hostname match
      ↓
HTTPRoute path match
      ↓
backend Service

A successful lookup alone does not prove that the request reaches the intended backend.

`hostAliases` can be useful for proving the hypothesis

Before changing CoreDNS cluster-wide, a temporary Pod-level override can be useful:

spec:
  hostAliases:
    - ip: "10.96.80.25"
      hostnames:
        - "api.example.com"

This is a reasonable diagnostic step.

If application latency stops following external network RTT after this change, the hypothesis is strongly confirmed.

But hostAliases is usually not a good permanent implementation.

The IP address is hardcoded. If the Gateway Service is recreated with another ClusterIP, the configuration becomes stale.

CoreDNS can instead map the public hostname to the Kubernetes Service DNS name:

api.example.com
        ↓
internal-api-gateway.gateway-system.svc.cluster.local
        ↓
current Service ClusterIP

That preserves Kubernetes service discovery.

`ExternalName` solves the opposite problem

An ExternalName Service can make a Kubernetes Service name point to an external hostname:

apiVersion: v1
kind: Service
metadata:
  name: external-api
spec:
  type: ExternalName
  externalName: api.example.com

This gives:

1
2
3

external-api.application.svc.cluster.local
        ↓
api.example.com

Our requirement is the reverse:

1
2
3

api.example.com
        ↓
internal Kubernetes destination

so ExternalName does not solve this case.

Do not accidentally remove functionality provided by the public edge

One more check is necessary before bypassing the public path.

The external endpoint may provide functionality such as:

WAF
authentication
rate limiting
header modification
IP restrictions
URL rewrites
CDN behavior

If internal requests depend on any of those, bypassing the edge can change application behavior.

The right objective is not:

bypass as much infrastructure as possible.

It is:

remove infrastructure that should not be in the internal request path while preserving infrastructure that is part of application behavior.

In this case, keeping the Gateway routing layer while removing public DNS and the external network path gave us that separation.

Before and after

The original request path was:

  flowchart LR
    APP["Application Pod"]
    CDNS["CoreDNS"]
    PDNS["Public DNS"]
    NET["External network"]
    EDGE["CDN / Public LB"]
    GW["Gateway"]
    ROUTE["HTTPRoute"]
    API["API"]

    APP --> CDNS
    CDNS --> PDNS
    PDNS --> NET
    NET --> EDGE
    EDGE --> GW
    GW --> ROUTE
    ROUTE --> API

Website performance therefore depended partly on external network conditions.

After the change:

  flowchart LR
    APP["Application Pod"]
    CDNS["CoreDNS"]
    GW["Internal Gateway"]
    ROUTE["HTTPRoute"]
    API["API"]

    APP -->|"api.example.com"| CDNS
    CDNS -->|"cluster-local"| GW
    GW --> ROUTE
    ROUTE --> API

The application URL remained unchanged:

`1`	`https://api.example.com`

The routing behavior remained unchanged.

But server-side communication no longer depended on external DNS resolution, upstream network RTT, public load-balancer reachability, or provider routing conditions.

The broader troubleshooting lesson

The initial symptom looked like an application performance problem:

The website is usually fast, but occasionally becomes slow for no obvious reason.

CPU, memory, database performance, and Kubernetes health were not enough to explain it.

The useful clue appeared only after comparing application latency with external network RTT.

That exposed an architectural dependency which was easy to miss: two applications running inside the same Kubernetes cluster were communicating through a public endpoint.

Public hostnames inside application configuration are not necessarily a problem. They become a problem when server-side traffic unnecessarily inherits the availability and latency characteristics of an external network path.

When troubleshooting intermittent application latency in Kubernetes, it is therefore worth checking not only which service a request reaches, but also how it gets there.

Get a Quote Back to services

Kubernetes application uses a public DNS name for an internal API: how to keep traffic inside the cluster

The first useful correlation: external network RTT

Finding the unexpected external dependency

Why the problem was intermittent

The obvious solution is not always possible

One hostname can represent several Kubernetes Services

Split-horizon DNS

CoreDNS rewrite

Keep the Gateway when it provides application routing

HTTPS requires special attention

Test the route before changing DNS

Verify from the actual application Pod

hostAliases can be useful for proving the hypothesis

ExternalName solves the opposite problem

Do not accidentally remove functionality provided by the public edge

Before and after

The broader troubleshooting lesson

`hostAliases` can be useful for proving the hypothesis

`ExternalName` solves the opposite problem