Kubeconform Explained: Validate Kubernetes Manifests and Prevent API Errors

Kubeconform Explained: Validate Kubernetes Manifests and Prevent API Errors

Kubernetes API changes quite a lot, and we know that in every new version, they are adding new capabilities at the same time that they are deprecating the old ones, so it is a constant evolution, as we already stated in previous articles, as you can see, here regarding Autoscaling v2 and Vertical Autoscaling.

Some of these changes are related to the shift in the apiVersion of some objects, and you have probably already suffered from that v1/alpha going to v1/beta or just moving to a final v1 and deprecating the previous one. So, in the end, it is crucial to ensure that your manifest is in sync with the target version you’re deploying, and some tools can help us with that, including Kubeconform.

What is Kubeconform?

Kubeconform is a powerful utility designed to assist in Kubernetes configuration management and validation. As Kubernetes continues to gain popularity as the go-to container orchestration platform, ensuring the correctness and consistency of configuration files becomes crucial. Kubeconform addresses this need by providing a comprehensive toolset to validate Kubernetes configuration files against predefined standards or custom rules.

Kubeconform supports multiple versions of Kubernetes, allowing you to validate configuration files against different API versions. This flexibility is beneficial when working with clusters running different Kubernetes versions or migrating applications across sets with varying configurations.

Another great feature of Kubeconform is its ability to enforce best practices and standards across Kubernetes configurations. It allows you to define rules, such as enforcing proper labels, resource limits, or security policies, and then validates your configuration files against these rules. This helps catch potential issues early on and ensures that your deployments comply with established guidelines.

How to install Kubeconform?

Kubeconform can be installed from different sources, the most usual ones the standard for your environment using package managers such as brew, apt or similar ones or just getting the binaries from its GitHub page: https://github.com/yannh/kubeconform/releases.

Kubeconform Explained: Validate Kubernetes Manifests and Prevent API Errors

How to launch Kubeconform from the Command Line?

Kubeconform is shipped as a small binary targeted to be executed in the CLI interface and tries to keep its interface minimal to ensure compatibility. Hence, it receives an argument with the file or folder with the manifest files that you want to check, as you can see here:

Kubeconform Explained: Validate Kubernetes Manifests and Prevent API Errors

Then you have several options to do other things, such as the ones shown below:

-ignore-filename-pattern value

regular expression specifying paths to ignore (can be specified multiple times)

-ignore-missing-schemas

skip files with missing schemas instead of failing

-Kubernetes-version string

version of Kubernetes to validate against, e.g.: 1.18.0 (default “master”)

-output string

output format – json, junit, pretty, tap, text (default “text”)

-reject string

comma-separated list of kinds or GVKs to reject

-skip string

comma-separated list of kinds or GVKs to ignore

-strict

disallow additional properties not in schema or duplicated keys

-summary

print a summary at the end (ignored for junit output)

Use-cases of Kuberconform

There are different use cases where Kubeconfrom can play a good role. One is regarding Kubernetes upgrades, sometimes you need to ensure that your current manifest is still going to work in the new release that the cluster will be upgraded to, and with this tool, we can ensure that our YAML is still compatible with the latest version directly getting it from the environment and validate it properly.

Another notable aspect of Kubeconform is its seamless integration into existing CI/CD pipelines. You can easily incorporate kubeconform as a step in your pipeline to automatically validate Kubernetes configuration files before deploying them. By doing so, you can catch configuration errors early in the development process, reduce the risk of deployment failures, and maintain high configuration consistency.

In addition to its validation capabilities, kubeconform provides helpful feedback and suggestions for improving your Kubernetes configuration files. It highlights specific issues or deviations from the defined rules and offers guidance on addressing them. This simplifies the troubleshooting process and helps developers and administrators become more familiar with best practices and Kubernetes configuration standards.

Conclusion

Kubeconform is an invaluable utility for Kubernetes users who strive for reliable and consistent deployments. It empowers teams to maintain a high standard of configuration quality, reduces the likelihood of misconfigurations, and improves the overall stability and security of Kubernetes-based applications.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Istio Security Policies Explained: PeerAuthentication, RequestAuthentication, and AuthorizationPolicy

Istio Security Policies Explained: PeerAuthentication, RequestAuthentication, and AuthorizationPolicy

Istio Security Policies are crucial in securing microservices within a service mesh environment. We have discussed Istio and the capabilities that it can introduce to your Kubernetes workloads. Still, today we’re going to be more detailed regarding the different objects and resources that would help us make our workloads much more secure and enforce the communication between them. These objects include PeerAuthentication, RequestAuthentication, and AuthorizationPolicy objects.

PeerAuthentication: Enforcing security on pod-to-pod communication

PeerAuthentication focuses on securing communication between services by enforcing mutual TLS (Transport Layer Security) authentication and authorization. It enables administrators to define authentication policies for workloads based on the source of the requests, such as specific namespaces or service accounts. Configuring PeerAuthentication ensures that only authenticated and authorized services can communicate, preventing unauthorized access and man-in-the-middle attacks. This can be achieved depending on the value of the mode where defining this object being STRICT for only allowed mTLS communication, PERMISSIVE to allow both kinds of communication, DISABLE to forbid the mTLS connection and keep the traffic insecure, and UNSET to use the inherit option. This is a sample of the definition of the object:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: foo
spec:
  mtls:
    mode: PERMISSIVE

RequestAuthentication: Defining authentication methods for Istio Workloads

RequestAuthentication, on the other hand, provides fine-grained control over the authentication of inbound requests. It allows administrators to specify rules and requirements for validating and authenticating incoming requests based on factors like JWT (JSON Web Tokens) validation, API keys, or custom authentication methods. With RequestAuthentication, service owners can enforce specific authentication mechanisms for different endpoints or routes, ensuring that only authenticated clients can access protected resources. Here you can see a sample of a RequestAuthentication object:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth-policy
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  jwtRules:
    - issuer: "issuer.example.com"
      jwksUri: "https://example.com/.well-known/jwks.json"

As commented, JWT validation is the most used approach as JWT tokens are becoming the de-facto industry standard for incoming validations and the OAuth V2 authorization protocol. Here you can define the rules the JWT needs to meet to be considered a valid request. But the RequestAuthentication only describes the “authentication methods” supported by the workloads but doesn’t enforce it or provide any details regarding the Authorization.

That means that if you define a workload to need to use JWT authentication, sending the request with the token will validate that token and ensure it is not expired. It meets all the rules you have specified in the object definition, but it will also allow bypassing requests with no token at all, as you’re just defining what the workloads support but not enforcing it. To do that, we need to introduce the last object of this set, the AuthorizationPolicy object.

AuthorizationPolicy: Fine-grained Authorization Policy Definition for Istio Policies

AuthorizationPolicy offers powerful access control capabilities to regulate traffic flow within the service mesh. It allows administrators to define rules and conditions based on attributes like source, destination, headers, and even request payload to determine whether a request should be allowed or denied. AuthorizationPolicy helps enforce fine-grained authorization rules, granting or denying access to specific resources or actions based on the defined policies. Only authorized clients with appropriate permissions can access specific endpoints or perform particular operations within the service mesh. Here you can see a sample of an Authorization Policy object:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: rbac-policy
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  rules:
    - from:
        - source:
            principals: ["user:user@example.com"]
      to:
        - operation:
            methods: ["GET"]

Here you can go as detailed as you need; you can apply rules on the source of the request to ensure that only some recommendations can go through (for example, requests that are from a JWT token to use in combination with the RequestAuthenitcation object), but also rules on the target, if this is going to a specific host or path or method or a combination of both. Also, you can apply to ALLOW rules or DENY rules (or even CUSTOM) and define a set of them, and all of them will be enforced as a whole. The evaluation is determined by the following rules as stated in the Istio Official Documentation:

  • If there are any CUSTOM policies that match the request, evaluate and deny the request if the evaluation result is denied.
  • If there are any DENY policies that match the request, deny the request.
  • If there are no ALLOW policies for the workload, allow the request.
  • If any of the ALLOW policies match the request, allow the request. Deny the request.
Authorization Policy validation flow from: https://istio.io/latest/docs/concepts/security/

This will provide all the requirements you could need to be able to do a full definition of all the security policies needed.

 Conclusion

In conclusion, Istio’s Security Policies provide robust mechanisms for enhancing the security of microservices within a service mesh environment. The PeerAuthentication, RequestAuthentication, and AuthorizationPolicy objects offer a comprehensive toolkit to enforce authentication and authorization controls, ensuring secure communication and access control within the service mesh. By leveraging these Istio Security Policies, organizations can strengthen the security posture of their microservices, safeguarding sensitive data and preventing unauthorized access or malicious activities within their service mesh environment.

Kubernetes Vertical Pod Autoscaling Explained: When and Why to Scale Vertically

Kubernetes Vertical Pod Autoscaling Explained: When and Why to Scale Vertically

Kubernetes has introduced as its alpha version in its Kubernetes 1.27 release the Vertical Pod Autoscaling capability to provide the option for the Kubernetes workload to be able to scale using the “vertical” approach by adding more resources to an existing pod. This increases the autoscaling capabilities of your Kubernetes workloads that you have at your disposal such as KEDA or Horizontal Pod Autoscaling.

Vertical Scaling vs Horizontal Scaling

Vertical and horizontal scaling are two approaches used in scaling up the performance and capacity of computer systems, particularly in distributed systems and cloud computing. Vertical scaling, also known as scaling up or scaling vertically, involves adding more resources, such as processing power, memory, or storage, to a single instance or server. This means upgrading the existing compute components or migrating to a more powerful infrastructure. Vertical scaling is often straightforward to implement and requires minimal changes to the software architecture. It is commonly used when the system demands can be met by a single, more powerful infrastructure.

On the other hand, horizontal scaling, also called scaling out or scaling horizontally, involves adding more instances or servers to distribute the workload. Instead of upgrading a single instance, multiple instances are employed, each handling a portion of the workload. Horizontal scaling offers the advantage of increased redundancy and fault tolerance since multiple instances can share the load. Additionally, it provides the ability to handle larger workloads by simply adding more machines to the cluster. However, horizontal scaling often requires more complex software architectures, such as load balancing and distributed file systems, to efficiently distribute and manage the workload across the machines.

In summary, vertical scaling involves enhancing the capabilities of a single object, while horizontal scaling involves distributing the workload across multiple instances. Vertical scaling is easier to implement but may have limitations in terms of the maximum resources available on a single machine. Horizontal scaling provides better scalability and fault tolerance but requires more complex software infrastructure. The choice between vertical and horizontal scaling depends on factors such as the specific requirements of the system, the expected workload, and the available resources.

Why Kubernetes Vertical AutoScaling?

This is an interesting topic because we have been living in a world where the state was that was always better to scale out (using Horizontal Scaling) rather than scaling up (using Vertical Scaling) and especially this was one of the mantras you heard in cloud-native developments. And, that hasn’t changed because horizontal scaling provides much more benefits than vertical scaling and it is well covered with the Autoscaling capabilities or side-projects such as KEDA. So, in that case, why is Kubernetes including this feature and why are we using this site to discuss it?

Because with the transformation of Kubernetes to be the de-facto alternative to any deployment you do nowadays, the characteristic and capabilities of the workloads that you need to handle have extended and that’s why you need to use different techniques to provide the best experience to each of the workloads types

How Kubernetes Vertical Autoscaling?

Here you will find all the documentation about this new feature that as commented is still in the “alpha” stage to is something to try as an experimental mode rather than using it at the production level HPA Documentation

Vertical Scaling works in the way that you will be able to change the resources assigned to the pod, CPU, and memory without needing to restart the pod and change the manifest declaration and that’s a clear benefit of this approach. As you know, until now if you want to change the resources applied to a workload you need to update the manifest document and restart the pod to apply the new changes.

To define this you need to specify the resizePolicy by adding a new section to the manifest pod as you can see here:

apiVersion: v1
kind: Pod
metadata:
  name: qos-demo-5
  namespace: qos-example
spec:
  containers:
  - name: qos-demo-ctr-5
    image: nginx
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      limits:
        memory: "200Mi"
        cpu: "700m"
      requests:
        memory: "200Mi"
        cpu: "700m"

For example in this case we define for the different resource names the policy that we want to apply, if we’re going to change the cpu assigned it won’t require a restart but in case we’re changing the memory it would require a restart.

That implied that if would like to change the CPU assigned you can directly patch the manifest as you can see in the snippet below and that provides an update of the assigned resources:

 kubectl -n qos-example patch pod qos-demo-5 --patch '{"spec":{"containers":[{"name":"qos-demo-ctr-5", "resources":{"requests":{"cpu":"800m"}, "limits":{"cpu":"800m"}}}]}}'

When to use Vertical Scaling are the target scenarios?

It will depend on a lot of different scenarios from the use-case but also from the technology stack that your workload is using to know what of these capabilities can apply. As a normal thing, the CPU change will be easy to adapt to any technology but the memory one would be more difficult depending on the technology used as in most of the technologies the memory assigned is defined at the startup time.

This will help to update components that have changed their requirements as an average scenario or when you’re testing new workloads with live load and you don’t want to disrupt the current processing of the application or simply workloads that don’t support horizontal scaling because are designed on a single-replica mode

 Conclusion

In conclusion, Kubernetes has introduced Vertical Pod Autoscaling, enabling Kubernetes vertical autoscaling of workloads by adding resources to existing pods. Kubernetes Vertical autoscaling allows for resource changes without restarting pods, providing flexibility in managing CPU and memory allocations.

Kubernetes Vertical autoscaling offers a valuable option for adapting to evolving workload needs. It complements horizontal scaling by providing flexibility without the need for complex software architectures. By combining vertical and horizontal scaling approaches, Kubernetes users can optimize their deployments based on specific workload characteristics and available resources.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Enable Sticky Sessions in Kubernetes Using Istio (Session Affinity Explained)

Enable Sticky Sessions in Kubernetes Using Istio (Session Affinity Explained)

Istio allows you to configure Sticky Session, among other network features, for your Kubernetes workloads. As we have commented in several posts regarding Istio, istio deploys a service mesh that provides a central control plane to have all the configuration regarding the network aspects of your Kubernetes workloads. This covers many different aspects of the communication inside the container platform, such as security covering security transport, authentication or authorization, and, at the same time, network features, such as routing and traffic distribution, which is the main topic for today’s article.

These routing capabilities are similar to what a traditional Load Balancer of Level 7 can provide. When we talk about Level 7, we’re referring to the conventional levels that compound the OSI stack, where level 7 is related to the Application Level.

A Sticky Session or Session Affinity configuration is one of the most common features you can need to implement in this scenario. The use-case is the following one:

How To Enable Sticky Session on Your Kubernetes Workloads using Istio?

You have several instances of your workloads, so different pod replicas in a Kubernetes situation. All of these pods behind the same service. By default, it will redirect the requests in a round-robin fashion among the pod replicas in a Ready state, so Kubernetes understand that they’re ready to get the request unless you define it differently.

But in some cases, mainly when you are dealing with a web application or any stateful application that handles the concept of a session, you could want the replica that processes the first request and also handles the rest of the request during the lifetime of the session.

Of course, you could do that easily just by routing all traffic to one request, but in that case, we lose other features such as traffic load balancing and HA. So, this is usually implemented using Session Affinity or Sticky Session policies that provides best of both worlds: same replica handling all the request from an user, but traffic distribution between different users.

How Sticky Session Works?

The behavior behind this is relatively easy. Let’s see how it works.

First, the important thing is that you need “something” as part of your network requests that identify all the requests that belong to the same session, so the routing component (in this case, this role is played by istio) can determine which part needs to handle these requests.

This is “something” that we use to do that, it can be different depending on your configuration, but usually, this is a Cookie or an HTTP Header that we send in each request. Hence, we know that the replica handles all requests of that specific type.

How does Istio implement Sticky Session support?

In the case of using Istio to do this role, we can implement that by using a specific Destination Rule that allows us, among other capabilities, to define the traffic policy to define how we want the traffic to be split and to implement the Sticky Session we need to use the “consistentHash” feature, that allows that all the requests that compute to the same hash will be sent to the replica.

When we define the consistentHash features, we can say how this hash will be created and, in other words, which components will be used to generate this hash, and this can be one of the following options:

  • httpHeaderName: Uses an HTTP Header to do the traffic distribution
  • httpCookie: Uses an HTTP Cookie to do the traffic distribution
  • httpQueryParameterName: Uses a Query String to do the traffic Distribution.
  • maglev: Uses Google’s Maglev Load Balancer to do the determination. You can read more about Maglev in the article from Google.
  • ringHash: Uses a ring-based hashed approach to load balancing between the available pods.

So, as you can see, you will have a lot of different options. Still, just the first three would be the most used to implement a sticky session, and usually, the HTTP Cookie (httpCookie) option will be the preferred one, as it would rely on the HTTP approach to manage the session between clients and servers.

Sticky Session Implementation Sample using TIBCO BW

We will define a very simple TIBCO BW workload to implement a REST service, serving a GET reply with a hardcoded value. To simplify the validation process, the application will log the hostname of the pod so quickly we can see who is handling each of the requests:

How To Enable Sticky Session on Your Kubernetes Workloads using Istio?

We deploy this in our Kubernetes cluster and expose it using a Kubernetes service; in our case, the name of this service will be test2-bwce-srv

On top of that, we apply the istio configuration, which will require three (3) istio objects: gateway, virtual service, and the destination rule. As our focus is on the destination rule, we will try to keep it as simple as possible in the other two objects:

 apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: default-gw
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

Virtual Service:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: test-vs
spec:
  gateways:
  - default-gw
  hosts:
  - test.com
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: test2-bwce-srv
        port:
          number: 8080

And finally, the DestinationRule will use a httpCookie that we will name ISTIOD, as you can see in the snippet below:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
    name: default-sticky-dr
    namespace: default
spec:
    host: test2-bwce-srv.default.svc.cluster.local
    trafficPolicy:
      loadBalancer:
        consistentHash:
          httpCookie: 
            name: ISTIOID
            ttl: 60s

Now, that we have already started our test, and after launching the first request, we get a new Cookie that is generated by istio itself that is shown in the Postman response window:

Enable Sticky Sessions in Kubernetes Using Istio (Session Affinity Explained)

This request has been handled for one of the replicas available of the service, as you can see here:

Enable Sticky Sessions in Kubernetes Using Istio (Session Affinity Explained)

All subsequent request from Postman already includes the cookie, and all of them are handled from the same pod:

Enable Sticky Sessions in Kubernetes Using Istio (Session Affinity Explained)

While the other replica’ log is empty, as all the requests have been routed to that specific pod.

Enable Sticky Sessions in Kubernetes Using Istio (Session Affinity Explained)

Summary

We covered in this article the reason behind the need for a sticky session in Kubernetes workload and how we can achieve that using the capabilities of the Istio Service Mesh. So, I hope this can help implement this configuration on your workloads that you can need today or in the future

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Kubernetes Autoscaling 1.26 Explained: HPA v2 Changes and Impact on KEDA

Kubernetes Autoscaling 1.26 Explained: HPA v2 Changes and Impact on KEDA

Introduction

Kubernetes Autoscaling has suffered a dramatic change. Since the Kubernetes 1.26 release, all components should migrate their HorizontalPodAutoscaler objects from the v1 to the new release v2that has been available since Kubernetes 1.23.

HorizontalPodAutoscaler is a crucial component in any workload deployed on a Kubernetes cluster, as the scalability of this solution is one of the great benefits and key features of this kind of environment.

A little bit of History

Kubernetes has introduced a solution for the autoscaling capability since the version Kubernetes 1.3 a long time ago, in 2016. And the solution was based on a control loop that runs at a specific interval that you can configure with the property --horizontal-pod-autoscaler-sync-period parameters that belong to the kube-controller-manager.

So, once during this period, it will get the metrics and evaluate through the condition defined on the HorizontalPodAutoscaler component. Initially, it was based on the compute resources used by the pod, main memory, and CPU.

Kubernetes Autoscaling 1.26: A Game-Changer for KEDA Users?

This provided an excellent feature, but with the past of time and adoption of the Kubernetes environment, it has been shown as a little narrow to handle all the scenarios that we should have, and here is where other awesome projects we have discussed here, such as KEDA brings into the picture to provide a much more flexible set of features.

Kubernetes AutoScaling Capabilities Introduced v2

With the release of the v2 of the Autoscaling API objects, we have included a range of capabilities to upgrade the flexibility and options available now. There most relevant ones are the following:

  • Scaling on custom metrics: With the new release, you can configure an HorizontalPodAutoscaler object to scale using custom metrics. When we talk about custom metrics, we talk about any metric generated from Kubernetes. You can see a detailed walkthrough about using Custom metrics in the official documentation
  • Scaling on multiple metrics: With the new release, you also have the option to scale based on more than one metric. So now the HorizontalPodAutoscalerwill evaluate each scaling rule condition, propose a new scale value for each of them, and take the maximum value as the final one.
  • Support for Metrics API: With the new release, the controller from the HoriztalPodAutoscaler components retrieves metrics from a series of registered APIs, such as metrics.k8s.io, custom.metrics.k8s.io ,external.metrics.k8s.io. For more information on the different metrics available, you can take a look at the design proposal
  • Configurable Scaling Behavior: With the new release, you have a new field, behavior, that allows configuring how the component will behave in terms of scaling up or scaling down activity. So, you can define different policies for the scaling up and others for the scaling down, limit the max number of replicas that can be added or removed in a specific period, to handle the issues with the spikes of some components as Java workloads, among others. Also, you can define a stabilization window to avoid stress when the metric is still fluctuating.

Kubernetes Autoscaling v2 vs KEDA

We have seen all the new benefits that Autoscaling v2 provides, so I’m sure that most of you are asking the same question: Is Kubernetes Autoscaling v2 killing KEDA?

Since the latest releases of KEDA, KEDA already includes the new objects under the autoscaling/v2 group as part of their development, as KEDA relies on the native objects from Kubernetes, and simplify part of the process you need to do when you want to use custom metric or external ones as they have scalers available for pretty much everything you could need now or even in the future.

But, even with that, there are still features that KEDA provides that are not covered here, such as the scaling “from zero” and “to zero” capabilities that are very relevant for specific kinds of workloads and to get a very optimized use of resources. Still, it’s safe to say that with the new features included in the autoscaling/v2 release, the gap is now smaller. Depending on your needs, you can go with the out-of-the-box capabilities without including a new component in your architecture.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Helm Templates in Files Explained: Customize ConfigMaps and Secrets Content

Helm Templates in Files Explained: Customize ConfigMaps and Secrets Content

If you have ever built a Helm chart that includes configuration files, scripts, or property files inside a ConfigMap or Secret, you have probably hit the same wall: the default templating engine only processes YAML files inside the templates/ directory. Everything else is treated as static content.

This is a problem because real-world applications rarely deploy with hardcoded configuration. You need environment-specific values in your .properties files, tokens in your JSON configs, or dynamic hostnames in your shell scripts. Helm provides three core functions to handle this: .Files.Get, .Files.Glob, and tpl. Each solves a different piece of the puzzle, and combining them is where things get powerful.

This guide covers every practical pattern you will need, from the simplest .Files.Get call to the advanced tpl + .Files.Glob combination that gives you full templating inside external files. For a broader look at Helm packaging, see our definitive Helm package management guide.

Understanding the Helm Chart File Structure

Before diving into the functions, it is important to understand what Helm considers a “file” and where it can access them. A typical chart looks like this:

my-chart/
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── configmap.yaml
│   ├── secret.yaml
│   └── deployment.yaml
├── config/
│   ├── app.properties
│   ├── logging.json
│   └── init.sh
└── files/
    └── zones.json

Files inside templates/ go through the full Helm template engine. Files outside it (like config/app.properties or files/zones.json) are accessible via the .Files object, but they are not templated automatically. This is the key distinction that catches most people off guard.

.Files.Get: Reading a Single File

The .Files.Get function reads the content of a specific file by its path, relative to the chart root. This is the simplest way to include file content in a ConfigMap or Secret.

Basic ConfigMap Example with .Files.Get

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-app-config
  namespace: {{ .Release.Namespace }}
data:
  app.properties: |
{{ .Files.Get "config/app.properties" | indent 4 }}

This reads config/app.properties from the chart root and injects it into the ConfigMap, preserving the original content. The indent 4 ensures correct YAML indentation.

Secret Example with .Files.Get and b64enc

For Secrets, Kubernetes expects base64-encoded values in the data field. Combine .Files.Get with b64enc:

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-tls-config
  namespace: {{ .Release.Namespace }}
type: Opaque
data:
  tls.crt: {{ .Files.Get "certs/tls.crt" | b64enc }}
  tls.key: {{ .Files.Get "certs/tls.key" | b64enc }}

.Files.Get Path Limitations: Why “../” Does Not Work

A very common question is whether you can use .Files.Get to access files outside the chart directory, for example .Files.Get "../shared/config.yaml". The answer is no. Helm restricts file access to the chart root for security reasons. Any path that tries to escape the chart directory with ../ will silently return an empty string.

If you need to share files between charts, the recommended patterns are:

  • Use a library chart or dependency that includes the shared files
  • Copy shared files into each chart during your CI/CD pipeline before packaging
  • Pass the content through values.yaml using a parent chart

Also note that files inside the templates/ directory and Chart.yaml itself are not accessible through .Files. Only files that are packaged with the chart and not in templates/ can be read.

.Files.Glob: Working with Multiple Files

When you have multiple configuration files to include, .Files.Glob lets you match files using glob patterns and iterate over them. This is especially useful when your chart ships with several config files that all need to end up in the same ConfigMap.

.Files.Glob with .AsConfig Example: ConfigMap from Multiple Files

The .AsConfig helper takes a set of matched files and formats them as ConfigMap data entries, where each filename becomes the key and the file content becomes the value:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-all-configs
  namespace: {{ .Release.Namespace }}
data:
{{ (.Files.Glob "config/*").AsConfig | indent 2 }}

If your config/ directory contains app.properties, logging.json, and init.sh, the result would be:

data:
  app.properties: |
    server.port=8080
    db.host=postgres
  logging.json: |
    {"level": "info", "format": "json"}
  init.sh: |
    #!/bin/bash
    echo "Initializing..."

This is the cleanest approach when you want to include all files from a directory without modifying their content.

.Files.Glob with .AsSecrets Example

.AsSecrets works identically to .AsConfig, but automatically base64-encodes each file’s content for use in Secret resources:

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-credentials
  namespace: {{ .Release.Namespace }}
type: Opaque
data:
{{ (.Files.Glob "secrets/*").AsSecrets | indent 2 }}

Iterating with range: The Explicit Approach

For more control over how each file is processed, you can iterate explicitly using range:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-scripts
  namespace: {{ .Release.Namespace }}
data:
  {{- range $path, $_ := .Files.Glob "scripts/*.sh" }}
  {{ base $path }}: |
{{ $.Files.Get $path | indent 4 }}
  {{- end }}

Notice two important details here. First, base $path extracts just the filename (e.g., init.sh) from the full path (scripts/init.sh). Second, inside a range loop the context changes, so you must use $. (dollar-dot) to access the root scope when calling $.Files.Get.

The tpl Function: Full Templating Inside External Files

This is where things get really interesting. The .Files.Get and .Files.Glob functions read file content as-is. If your app.properties file contains {{ .Values.database.host }}, it will be included literally as that string, not replaced with the actual value.

The tpl function solves this by passing a string through the Helm template engine. When you combine tpl with .Files.Get, your external files get the same templating power as files inside templates/.

tpl + .Files.Get: Templated ConfigMap

Consider this config/app.properties file in your chart:

# config/app.properties
server.port={{ .Values.app.port | default 8080 }}
server.host={{ .Values.app.host }}
database.url=jdbc:postgresql://{{ .Values.database.host }}:{{ .Values.database.port }}/{{ .Values.database.name }}
database.pool.size={{ .Values.database.poolSize | default 10 }}
logging.level={{ .Values.logging.level | default "INFO" }}

And the corresponding template that processes it:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-app-config
  namespace: {{ .Release.Namespace }}
data:
  app.properties: |
{{ tpl (.Files.Get "config/app.properties") . | indent 4 }}

The tpl function takes two arguments: the string to process and the context (.). It runs the content through the template engine, replacing all {{ .Values.* }} references with actual values. The result is a fully dynamic configuration file.

tpl + .Files.Glob: Templating Multiple Files

To template every file matched by a glob pattern, combine the range iteration with tpl:

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-dynamic-secrets
  namespace: {{ .Release.Namespace }}
type: Opaque
data:
  {{- range $path, $_ := .Files.Glob "secrets/*.json" }}
  {{ base $path }}: {{ tpl ($.Files.Get $path) $ | b64enc }}
  {{- end }}

This iterates over all .json files in the secrets/ directory, passes each through the template engine (so {{ .Values.* }} references are resolved), base64-encodes the result, and includes it in the Secret. This is the most powerful pattern for managing multiple dynamic configuration files.

Common Pitfalls and Troubleshooting

Working with .Files in Helm can produce confusing errors or silent failures. Here are the most common issues and how to fix them.

.Files.Get Returns Empty String

If .Files.Get returns nothing, check these three things:

  • Wrong path: The path is relative to the chart root, not to the templates directory. Use .Files.Get "config/app.properties", not .Files.Get "../config/app.properties".
  • File excluded by .helmignore: Check your .helmignore file. If it matches the file’s path, the file will not be packaged with the chart and .Files.Get will return empty.
  • File in templates/ directory: Files inside templates/ are not accessible via .Files. Move them to a different directory.

YAML Indentation Errors

The most frequent rendering error is incorrect indentation. Always use indent N (or nindent N) when including file content in YAML. The difference between indent and nindent is that nindent adds a newline before the content, which is cleaner when using {{- to trim whitespace:

# Using indent (requires | on the previous line)
data:
  app.properties: |
{{ .Files.Get "config/app.properties" | indent 4 }}

# Using nindent (cleaner, self-contained)
data:
  app.properties: {{ .Files.Get "config/app.properties" | nindent 4 }}

Chart Size Limit

Helm charts stored in Kubernetes as Secrets or ConfigMaps are subject to the 1 MB limit imposed by etcd. If your chart includes many large files, you may hit this limit during helm install. The error typically reads release: invalid or etcd: request is too large. In that case, consider mounting files via persistent volumes or external config management instead of embedding them in the chart.

Context Issues Inside range Loops

Inside a range block, . refers to the current iteration item, not the root context. This means .Files.Get will fail. Use $. to access the root context:

# Wrong: . is the loop item, not the root context
{{- range $path, $_ := .Files.Glob "config/*" }}
  {{ .Files.Get $path }}  {{/* This will fail */}}
{{- end }}

# Correct: use $ to access root context
{{- range $path, $_ := .Files.Glob "config/*" }}
  {{ $.Files.Get $path }}  {{/* This works */}}
{{- end }}

Real-World Pattern: Multi-Environment Configuration

A practical pattern that combines everything above is organizing environment-specific configuration files and selecting them dynamically:

# Chart structure
my-chart/
├── config/
│   ├── application.yaml      # Shared base config
│   ├── db-pool.properties    # Database pool settings
│   └── logging.xml           # Log4j/Logback config
├── templates/
│   └── configmap.yaml
└── values.yaml

The ConfigMap template processes all config files with templating:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-config
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "my-chart.labels" . | nindent 4 }}
data:
  {{- range $path, $_ := .Files.Glob "config/*" }}
  {{ base $path }}: |
{{ tpl ($.Files.Get $path) $ | indent 4 }}
  {{- end }}

This way, every file in config/ is automatically included, templated, and properly formatted. Adding a new configuration file is as simple as dropping it into the directory. No template changes required.

Quick Reference: .Files Functions Summary

Here is a quick reference of all file-related functions available in Helm:

FunctionWhat it doesExample
.Files.GetReturns the content of a single file as a string.Files.Get "config/app.yaml"
.Files.GlobReturns all files matching a glob pattern.Files.Glob "config/*.json"
.AsConfigFormats matched files as ConfigMap data entries(.Files.Glob "config/*").AsConfig
.AsSecretsFormats matched files as base64-encoded Secret data(.Files.Glob "secrets/*").AsSecrets
tplPasses a string through the Helm template enginetpl (.Files.Get "f.yaml") .
.Files.LinesReturns file content as a list of lines.Files.Lines "config/hosts.txt"
.Files.AsBytesReturns file content as a byte array.Files.Get "bin/tool" \| b64enc

Conclusion

Helm file handling follows a clear escalation path depending on what you need. Use .Files.Get when you need a single file as-is. Use .Files.Glob with .AsConfig or .AsSecrets when you have multiple files that need no modification. And use tpl combined with either function when your files need dynamic values from values.yaml.

The most common mistake is forgetting that .Files only reads files — it does not template them. The moment you need {{ .Values.* }} inside an external file, tpl is the function you are looking for. For more Helm patterns and advanced tips, explore our guides on Helm hooks, Helm loops, and Helm dependencies.

Frequently Asked Questions

Can .Files.Get access files outside the chart directory?

No. Helm restricts .Files.Get to files within the chart root directory. Paths containing ../ will silently return an empty string. This is a security constraint to prevent charts from reading arbitrary files from the filesystem. If you need shared files across charts, use library charts, copy files during CI/CD, or pass content through parent chart values.

What is the difference between .AsConfig and .AsSecrets in Helm?

.AsConfig formats files as plain text ConfigMap data entries (key: filename, value: file content). .AsSecrets does the same but automatically base64-encodes each file’s content, which is required for the data field in Kubernetes Secret resources. Both are called on the result of .Files.Glob.

How do I use Helm values inside a properties file or JSON config?

Use the tpl function. Instead of .Files.Get "config/file.json", use tpl (.Files.Get "config/file.json") .. This passes the file content through the Helm template engine, so any {{ .Values.* }} references in the file will be resolved against your chart’s values.

Why does .Files.Get return an empty string?

Three common causes: the file path is wrong (paths are relative to the chart root, not the templates directory), the file is excluded by .helmignore, or the file is inside the templates/ directory which is not accessible via .Files. Run helm template locally and check the output to debug.

Is there a size limit for files included via .Files.Get?

Helm itself does not impose a file size limit, but the packaged chart is stored as a Kubernetes Secret or ConfigMap which is subject to the etcd 1 MB size limit. If your chart with all its files exceeds this, helm install will fail. For large files, consider external config management or persistent volumes instead of embedding them in the chart.

Nomad vs Kubernetes: Key Differences, Use Cases, and When Nomad Makes Sense

Nomad vs Kubernetes: Key Differences, Use Cases, and When Nomad Makes Sense

Nomad is the Hashicorp alternative to the typical pattern of using a Kubernetes-based platform as the only way to orchestrate your workloads efficiently. Nomad is a project started in 2019, but it is getting much more relevant nowadays after 95 releases, and the current version of this article is 1.4.1, as you can see in their GitHub profile.

Nomad approaches the traditional challenges of isolating the application lifecycle for the infrastructure operation lifecycle where that application resides. Still, instead of going full to a container-based application, it tries to provide a solution differently.

What are the main Nomad Features?

Based on its own definition, as you can read on their GitHub profile, they already highlight some of the points of difference between the de-facto industry standard:

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications

Easy-to-use: This is the first statement they include in their definition because the Nomad approach is much simpler than alternatives such as Kubernetes because it works on a single-binary approach where it has all the capabilities that are needed running as a node agent based on its own “vocabulary” that you can read more of it in their official documentation.

Flexibility: This is the other critical thing they provide a hypervisor, an intermediate layer between the application and the underlying infrastructure. It is not just limited to container applications but also supports this kind of deployment. It also allows the deploy it as part of a traditional virtual machine approach. The primary use cases highlighted are running standard windows applications, which is tricky when talking about Kubernetes deployments; even though Windows containers have been a thing for so long, their adoption is not at the same level, as you can see in the Linux container world.

Hashicorp Integration: As part of the Hashicorp portfolio, it also includes seamless integration with other Hashicorp projects such as Hashicorp Vault, which we have covered in several articles, or Hashicorp Consul, which helps to provide additional capabilities in terms of security, configuration management, and communication between the different workloads.

Nomad vs Kubernetes: How Nomad Works?

As commented above, Nomad covers everything with a single-component approach. A nomad binary is an agent that can work in server mode or client mode, depending on the role of the machine executing it.

So Nomad is based on a Nomad cluster, a set of machines running a nomad agent in server mode. Those servers are split depending on the role of the leader or followers. The leader performs most of the cluster management, and the followers can create scheduling plans and submit them to the leader for approval and execution. This is represented in the picture below from the Hashicorp Nomad official page:

Nomad vs Kubernetes: 1 Contestant Against the Orchestration King
Nomad simple architecture from nomadproject.io

Once we have the cluster ready, we need to create our jobs, and a job is a definition of the task we would like to execute on the Nomad cluster we have previously set up. A task is the smallest unit of work in Nomad. Here is where the flexibility comes to Nomad because the task driver executes each task, allowing different drivers to execute various workloads. This is how following the same approach, we will have a docker driver to run our container deployment or an exec driver to execute it on top of a virtual infrastructure. Still, you can create your task drivers following a plugin mechanism that you can read more about here.

Jobs and Task are defined using a text-based approach but not following the usual YAML or JSON kind of files but a different format, as you can see in the picture below (click here to download the whole file from the GitHub Nomad Samples repo):

 Is Nomad a Replace for Kubernetes?

It is a complex question to answer, and even Hashicorp they have documented different strategies. You can undoubtedly use Nomad to run container-based deployments instead of running them on Kubernetes. But at the same time, they also position the solution alongside Kubernetes to run some workloads on one solution and another on the other.

In the end, both try to solve and address the same challenges in terms of scalability, infrastructure sharing and optimization, agility, flexibility, and security from traditional deployments.

Kubernetes focus on different kind of workloads, but everything follows the same deployment mode (container-based) and adopts recent paradigms (service-based, microservice patterns, API-led, and so on) with a robust architecture that allows excellent scalability, performance, flexibility, and with adoption levels in the industry that has become the current de-facto only alternative for modern workload orchestration platforms.

But, at the same time, it also requires effort in management and transforming existing applications to new paradigms.

On the other hand, Nomad tries to address it differently, minimizing the change of the existing application to take advantage of the platform’s benefits and reducing the overhead of management and complexity that a usual Kubernetes platform provides, depending on the situation.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Helm Dependencies Explained: How Chart Dependencies Work in Helm

Helm Dependencies Explained: How Chart Dependencies Work in Helm

Helm Dependency is a critical part of understanding how Helm works as it is the way to establish relationships between different helm packages. We have talked a lot here about what Helm is, and some topics around that, and we even provided some tricks if you create your charts.

Understanding chart dependencies is crucial for building scalable Helm architectures. Explore more Helm patterns and best practices in our comprehensive Helm guide.

So, as commented, Helm Chart is nothing more than a package that you put around the different Kubernetes objects that need to be deployed for your application to work. The usual comparison is that it is similar to a Software Package. When you install an application that depends on several components, all of those components are packaged together, and here is the same thing.

What is a Helm Dependency?

A Helm Dependency is nothing more than the way you define that your Chart needs another chart to work. For sure, you can create a Helm Chart with everything you need to deploy your application, but something you would like to split that work into several charts just because they are easy to maintain or the most common use case because you want to leverage another Helm Chart that is already available.

One use case can be a web application that requires a database, so you can create on your Helm Chart all the YAML files to deploy your web application and your Database in Kubernetes, or you can have your YAML files for your web application (Deployment, Services, ConfigMaps,…) and then say: And I need a database and to provide it I’m going to use this chart.

This is similar to how it works with the software packages in UNIX systems; you have your package that does the job, like, for example, A, but for that job to be done, it requires the library L, and to ensure that when you are installing A, Library L is already there or if not it will be installed you declare that your application A depends on Library L, so here is the same thing. You declare that your Chart depends on another Chart to work. And that leaves us to the next point.

How do we declare a Helm Dependency?

This is the next point; now that we understand what a Helm Dependency is conceptually and we have a use case, how can we do that in our Helm Chart?

All the work is done in the Chart.yml file. If you remember, the Chart.yml file is the file where you declare all the metadata of your Helm Chart, such as the name, the version of the chart, the application version, location URL, icon, and much more. And usually has a structure like this one:

apiVersion: v2
name: MyChart
description: My Chart Description
type: application
version: 0.2.0
appVersion: "1.16.0"

So here we can add a section dependencies and, in that section is where we are going to define the charts that we depend on. As you can see in the snippet below:

apiVersion: v2
name: MyChart
description: My Chart Description
type: application
version: 0.2.0
appVersion: "1.16.0"
dependencies:
- name: Dependency
  version: 1.0.0
  repository: "file:///location_of_my_chart"

Here we are declaring Dependency as our Helm Dependency. We specify the version that we would like to use (similar to the version we say in our chart), and that will help us to ensure that we will provide the same version that has been tested as part of the resolution of the dependency and also the location using an URL that can be an external URL is this is pointing to a Helm Chart that is available on the internet or outside your computer or using a File Path in case you are pointing to a local resource in your machine.

That will do the job of defining the helm dependency, and this way, when you install your chart using the command helm install, it will also provide the dependence.

How do I declare a Helm Conditional Dependency?

Until now, we learned how to declare a dependency, and each time I provision my application, it will also provide the dependence. But usually, we would like to have a fine-grained approach to that. Imagine the same scenario as above: We have our Web Application that depends on the Database, and we have two options, we can provision the database as part of the installation of the web application, or we can point to an external database and in that case, it makes no sense to provision the Helm Dependency. How can we do that?

So, easy, because one of the optional parameters you can add to your dependency is condition and do exactly that, condition allow you to specify a flag in your values.yml that in the case is equal to true, it will provide the dependency but in the case is equal to false it will skip that part similar to the snippet shown below:

 apiVersion: v2
name: MyChart
description: My Chart Description
type: application
version: 0.2.0
appVersion: "1.16.0"
dependencies:
- name: Dependency
  version: 1.0.0
  repository: "file:///location_of_my_chart"
  condition: database.enabled 

And with that, we will set the enabled parameter under database in our values.yml to true if we would like to provision it.

How do I declare a Helm Dependency With a Different version?

As shown in the snippets above, we offer that when we declare a Helm Dependency, we specify the version; that is a safe way to do it because it ensures that any change done to the helm chart will not affect your package. Still, at the same time, you cannot be aware of security fixes or patches to the chart that you would like to leverage in your deployment.

To simplify that, you have the option to define the version in a more flexible way using the operator ~ in the definition of the version, as you can see in the snippet below:

apiVersion: v2
name: MyChart
description: My Chart Description
type: application
version: 0.2.0
appVersion: "1.16.0"
dependencies:
- name: Dependency
  version: ~1.0.0
  repository: "file:///location_of_my_chart"
  condition: database.enabled 

This means that any patch done to the chart will be accepted, so this is similar that this chart will use the latest version of 1.0.X. Still, it will not use the 1.1.0 version, so that allows to have more flexibility, but at the same time keeping things safe and secured in case of a breaking change on the Chart you depend on. This is just one way to define that, but the flexibility is enormous as the Chart versions use “Semantic Versions,” You can learn and read more about that here: https://github.com/Masterminds/semver.

Multi-Stage Dockerfiles Explained: Reduce Docker Image Size the Right Way

Multi-Stage Dockerfiles Explained: Reduce Docker Image Size the Right Way

Multi-Stage Dockerfile is the pattern you can use to ensure that your docker image is at an optimized size. We already have covered the importance of keeping the size of your docker image at a minimum level and what tools you could use, such as dive, to understand the size of each of your layers. But today, we are going to follow a different approach and that approach is a multi-stage build for our docker containers.

What is a Multi-Stage Dockerfile Pattern?

The multi-Stage Dockerfile is based on the principle that the same Dockerfile can have different FROM sentences and each of the FROM sentences starts a new stage of the build.

Multi-Stage Dockerfile Pattern

Why Multi-Stage Build Pattern Helps Reducing The Size of Container Images?

The main reason the usage of multi-stage build patterns helps reduce the size of the containers is that you can copy any artifact or set of artifacts from one stage to the other. And that is the most important reason. Why? Because that means that everything you do not copy is discarded and you are not carrying all these not required components from layer to layer and generating a bigger unneeded size of the final Docker image.

How do you define a Multi-Stage Dockerfile

First, you need to have a Dockerfile with more than one FROM. As commented, each of the FROM will indicate the start of one stage of the multi-stage dockerfile. To differentiate them or reference them, you can name each of the stages of the Dockerfile by using the clause AS alongside the FROM command, as shown below:

 FROM eclipse-temurin:11-jre-alpine AS builder

As a best practice, you can also add a new label stage with the same name you provided before, but that is not required. So, in a nutshell, a Multi-Stage Dockerfile will be something like this:

FROM eclipse-temurin:11-jre-alpine AS builder
LABEL stage=builder
COPY . /
RUN apk add  --no-cache unzip zip && zip -qq -d /resources/bwce-runtime/bwce-runtime-2.7.2.zip "tibco.home/tibcojre64/*"
RUN unzip -qq /resources/bwce-runtime/bwce*.zip -d /tmp && rm -rf /resources/bwce-runtime/bwce*.zip 2> /dev/null


FROM  eclipse-temurin:11-jre-alpine 
RUN addgroup -S bwcegroup && adduser -S bwce -G bwcegroup

How do you copy resources from one stage to another?

This is the other important part here. Once we have defined all the stages we need, and each is doing its part of the job, we need to move data from one stage to the next. So, how can we do that?

The answer is by using the command COPY. COPY is the same command you use to move data from your local storage to the container image, so you will need a way to differentiate that this time you are not copying it from your local storage but another stage, and here is where we are going to use the argument --from. The value will be the name of the stage we learned in the previous section to declare. So a complete COPYcommand will be something like the snippet shown below:

 COPY --from=builder /resources/ /resources/

What is the Improvement you can get?

That is the essential part and will depend on how your Dockerfiles and images are created, but the primary factor you can consider is the number of layers your current image has. The bigger the number of layers, the more significant that you can probably save on the amount of the final container image in a multi-stage dockerfile.

The main reason is that each layer will duplicate part of the data, and I am sure you will not need all of the layer’s data in the next one. And using the approach comments in this article, you will get a way to optimize it.

 Where can I read more about this?

If you want to read more, you would need to know that the multi-stage dockerfile is documented as one of the best practices on the Docker official web page, and they have a great article about this by Alex Ellis that you can read here.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Inject Secrets into Kubernetes Pods Using HashiCorp Vault (Agent Injector Guide)

Inject Secrets into Kubernetes Pods Using HashiCorp Vault (Agent Injector Guide)

Introduction

This article will cover how to inject secrets in Pods using Hashicorp Vault. In previous articles, we covered how to install Hashicorp Vault in Kubernetes, configure and create secrets in Hashicorp, and how tools such as TIBCO BW can retrieve them. Still, today, we are going to go one step ahead.

The reason why Inject secrets in pods is very important is that it allows the application inside the pod to be transparent around any communication to Hashicorp. After all, for the applications, the secret will be just a regular file located in a specific path inside the container. It doesn’t need to worry if this file came from a Hashicorp Secret or a total of different resources.

This injection approach facilitates the Kubernetes ecosystem’s polyglot approach because it frees any responsibility for the underlying application. The same happens with injector approaches such as Istio or much more.

But, let’s explain how this approach works to inject secrets in Pods using Hashicorp Vault. As part of the installation alongside the vault server we have installed (or several ones if you have done a distributed installation), we have seen another pod under the name of value-agent-injector, as you can see in the picture below:

Inject Secrets In Pods: Vault Injector Pod

This agent will be responsible for listening to the new deployments you do and, based on the annotations this deployment has, will launch a sidecar alongside your application and send the configuration to be able to connect to the vault and download the secrets required and mount it as files inside your pod as shown in the picture below:

To do that, we need to do several steps as part of the configuration as we are going to include in the upcoming sections of the article

Enabling Kubernetes Authentication in Hashicorp

The first thing we need to do at this stage is to enable the Kubernetes Authentication in Hashicorp. This method allows clients to authenticate with a Kubernetes Service Account Token. We do that with the following command:

 Vault auth enable kubernetes

Vault accepts a service token from any client in the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a token review Kubernetes endpoint. Now, we need to configure this authentication method providing the location our the Kubernetes API, and to do that, we need to run the following command:

 vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Defining a Kubernetes Services Account and Defining a Policy

Now, we will create a Kubernetes Service Account that will run our pods, and this service account will be allowed to retrieve the secret we generated in the previous post.

To do that, we will start with the creation of the service account by running this command from outside the pod:

kubectl create sa internal-app

This will create a new service account under the name of internal-app, and now we are going to generate a policy inside the Hashicorp Vault server by using this command inside the vault server pod:

 vault policy write internal-app - <<EOF
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOF

And now, we associated this policy with the service account by running this command also inside the vault server pod:

  vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=24h

And that’s pretty much all the configuration we need to do at the Vault side to be able to inject secrets in pods using Hashicorp Vault. Now, we need to configure our application accordingly by doing the following modifications:

  • Specify the ServiceAccountName to the deployment to be the one we created previously: internal-app
  • Specify the specific annotations to inject the vault secrets and the configuration of those secrets.

Let’s start with the first point. We need to add the serviceAccountName to our Kubernetes Manifest YAML file as shown below:

Inject Secrets In Pods: Service Account Name definition

And regarding the second point, we would solve it by adding several annotations to our deployment, as shown below:

Inject Secrets In Pods: Annotations

The annotations used to inject secrets in pods are the following ones:

  • vault.hashicorp.com/agent-inject: ‘true’: This tells the vault injector that we would like to inject the sidecar agent in this deployment and have the Vault configuration. This is required to do any further configuration
  • vault.hashicorp.com/role: internal-app: This is the vault role we are going to use when we’re asking for secrets and information to the vault to be sure that we only access the secrets we have allowed based on the policy we created in the previous section
  • vault.hashicorp.com/agent-inject-secret-secret-database-config.txt: internal/data/database/config: This will be one annotation per secret we plan to add, and it is composed of three parts:
    • vault.hashicorp.com/agent-inject-secret- this part is fixed
    • secret-database-config.txt this part will be the filename that is created under /vault/secrets inside our pod
    • internal/data/database/config This is the path inside the vault of our secret to being linked to that file.

And that’s it! If we deploy now our deployment we will see the following things:

  • Our deployment application has been launched with three containers instead of one because two of them are Hashicorp Vault related, as you can see in the picture below:
Inject Secrets into Kubernetes Pods Using HashiCorp Vault (Agent Injector Guide)
  • vault-agent-init will be the container that establishes the connection to the vault-server because any container starts and does the first download and injecting secrets in the pod based on the configuration provided.
  • vault-agent will be the container running as a watcher to detect any modification on the related secrets and update them.
Inject Secrets into Kubernetes Pods Using HashiCorp Vault (Agent Injector Guide)

And now, if we go to the main container, we will see in the /vault/secrets path the secret has been finally injected as expected:

Inject Secrets into Kubernetes Pods Using HashiCorp Vault (Agent Injector Guide)

And this is how easily and without any knowledge about the underlying app we can inject secrets in pods using Hashicorp Vault.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.