A couple of weeks in the past, I spent a while with our help and engineering groups serving to a buyer resolve an issue that occurred after they enabled Group Managed Service Accounts (gMSA) on Azure Kubernetes Service (AKS).

I made a decision to put in writing this weblog so different prospects with the identical problem can keep away from going via it altogether. I’m writing the weblog within the sequence as I skilled it, however in case you’re simply in search of the answer, be happy to skip to the top.

When that buyer enabled gMSA on their cluster, a couple of issues began to occur:

• Any gMSA enabled deployment/container/pod entered a failed state. The occasions from the deployments would present the pods with the next error: Occasion Element: Didn’t setup the exterior credentials for Container ‘<redacted>’: The RPC server is unavailable.
• Any non-gMSA deployment/container/pod utilizing the shopper’s personal photos and operating on Home windows nodes additionally entered a failed state. The deployments had been displaying an occasion of ErrImagePull.
• All different deployments/containers/pods each on Home windows and Linux nodes that weren’t utilizing personal photos saved their wholesome state.

Eradicating the gMSA configuration from the cluster would routinely revert to a wholesome state for the complete cluster.

The error with the gMSA pods took me instantly to different instances by which I’ve seen prospects having comparable points due to community connectivity. The most typical gMSA points I’ve seen thus far are:

• Blocked ports: Having a firewall between your AKS cluster and the Lively Listing (AD) Area Controllers (DCs). AD makes use of a number of protocols for communication between shoppers and DCs. I even created a simple script that validates the ports.
• Incorrect DNS configuration: AD makes use of DNS for service discovery. Area Controllers have a “SRV” entry within the DNS that shoppers question to allow them to discover not solely all DCs, however the closest one. If both the nodes or pods can’t resolve the area fqdn to a DC, gMSA gained’t work.
• Incorrect secret on Azure Key Vault (AKV): A consumer account is utilized by the Window nodes, slightly than a pc account because the nodes will not be domain-joined. The format of the key ought to be <area dns fqdn><consumer account>:<consumer password>.

There are different minor points that I’ve seen, however these are the principle ones. Within the case of this prospects, we reviewed the above and the whole lot gave the impression to be configured correctly.

At that time, I introduced other people and so they caught on one thing that I knew existed, however had not seen utilizing gMSA but: AKS personal clusters.

This buyer has a safety coverage in-place that mandates Azure sources ought to be utilizing personal endpoints at any time when potential. That was true for the AKS cluster and due to this fact, it launched a habits that broke the cluster.

I discussed above that gMSA makes use of DNS for DC discovering. Let me clarify what the default config is and what occurred after enabling gMSA:

By default, Linux and Home windows nodes on AKS will use the Azure vNet DNS server for DNS queries. Home windows and Linux pods will use CoreDNS for DNS queries. Azure DNS can’t resolve AD area FQDNs since these are typically personal to on-premises or personal cloud networks.

For that motive, whenever you allow gMSA and move the parameter of DNS server for use, two issues are modified within the AKS cluster. First, the Home windows nodes will begin utilizing the DNS server offered. Second, the CoreDNS setting is modified so as to add a forwarder. This forwards something associated to the area FQDN to the required DNS server. With these two configs, Home windows nodes and Home windows pods can now “discover” the DCs.

Azure Portal displaying the CoreDNS configuration with a DNS forwarder after gMSA has been configured.

Nonetheless, this introduces one other problem when mixed with a personal AKS cluster. Personal endpoints are behind a personal DNS zone. Azure DNS servers can resolve for these zones, however non-Azure DNS servers can’t. Since now the Home windows nodes and Home windows pods are utilizing a DNS server exterior of Azure, the personal zone of the AKS cluster can’t be resolved so the DCs can’t entry the Home windows nodes and Home windows pods.

Not solely that, however this buyer additionally had their Azure Container Registry (ACR) behind a personal endpoint. The second symptom above was additionally attributable to this configuration, as now the Home windows nodes can’t resolve for the personal zone of the ACR registry and consequently can’t pull their personal photos.

For reference, these are the container associated companies and their personal zones:

For a full record of zones, try the Azure documentation.

The answer right here is straightforward. For the non-Azure DNS servers to resolve Personal Endpoint zones, a DNS forwarder will be created.

This buyer had a really particular implementation, however on the whole what you’ll want to configure is a DNS forwarder to the zones associated to the companies you might be utilizing. For instance:

–          AKS clusters: Create a forwarder of azmk8s.io to 168.63.129.16.

–          For ACR registries: Create a forwarder of azurecr.io to 168.63.129.16.

168.63.129.16. is the digital IP handle of the Azure platform that serves because the communication channel to the platform sources. One in every of its companies is DNS. In reality, that is the unique service that the Home windows nodes and Home windows pods had been utilizing earlier than gMSA was enabled.

It’s all the time DNS!

In case you are utilizing gMSA on AKS, remember that Home windows nodes and Home windows pods will begin utilizing a DNS server exterior of Azure (or that has no visibility into the Azure platform instantly, resembling Personal Endpoint zones). You may have to configure DNS forwarders when you begin utilizing gMSA on AKS, though this will likely be true for any service.

I hope this weblog put up helps you keep away from this problem – or helps you troubleshoot it. Tell us within the feedback!