Hadolint Explained: Dockerfile Best Practices Using CLI, Docker, and VS Code

Hadolint Explained: Dockerfile Best Practices Using CLI, Docker, and VS Code

Introduction

Hadolint is an open-source tool that will help you ensure that all the Dockerfiles you create follow all the Dockerfile best practices available in an automated way. Hadolint, as the number already suggested, is a linter tool and, because of that, can also help you to teach you all these best practices when creating Dockerfiles yourself. We already talked about it the optimization of container image size, but today we are going to try to cover it more in-depth.

Hadolint is a smaller tool written in Haskell that parses the Dockerfile into an AST and performs rules on top of the AST. It stands on the shoulders of ShellCheck to lint the Bash code inside RUN instructions, as shown in the picture below:

Hadolint Explained: Dockerfile Best Practices Using CLI, Docker, and VS Code

There are several ways to run the tool, depending on what you try to achieve, and we will talk a little bit about the different options.

Running it as a standalone tool

This is the first way we can run it as a complete standalone tool that you can download from here , and it will need to do the following command.

 hadolint <Dockerfile path>

It will run against it and show any issue that is found, as you can see in the picture below:

Hadolint execution

For each of the issues found, it will show the line where the problem is detected, the code of the Dockerfile best practice check that is being performed (DL3020), the severity of the check (error, warn, info, and so on), and the description of the issue.

To see all the rules that are being executed, you can check them in the GitHub Wiki , and all of them are based on the Dockerfile best practices published directly from Docker on its official web page here.

For each of them, you will find a specific wiki page with all the information you need about the issue and why this is something that should be changed, and how it should be changed, as you can see in the picture below:

Hadolint GitHub Wiki page

Ignore Rules Capability

You can ignore some rules if you don’t want them to be applied because there are some false-positive or just because the checks are not aligned with the Dockerfile best practices used in your organization. To do that, you can include an —ignore parameter with the rule to be applied:

 hadolint --ignore DL3003 --ignore DL3006 <Dockerfile>

Running it as Docker Container

Also, the tool is available as a Docker container in the following repos:

docker pull hadolint/hadolint
# OR
docker pull ghcr.io/hadolint/hadolint

And this will help you to be introduced to your Continuous Integration and Continuous Deployment or just to be used in your local environment if you prefer not to install software locally.

 Running it inside VS Code

Like many linters, it is essential to have it close to your development environment; this time is nothing different. We would like to have the Dockerfile best practice relative to the editor while we are typing for two main reasons:

  • As soon as you get the issue, you will fix it faster so the code always will have better quality
  • As soon as you know of the issue, you will not make it again in newer developments.

You will have a Hadolint as part of the Extensions: Marketplace, and you can install it:

Hadolint VS Code Extension


Once you have that done, each time you open a Dockerfile, you will validate against all these Dockerfile best practices, and it will show the issues detected in the Problems view, as you can see in the picture below:

Hadolint: VS Code Extension Execution

And those issues will be re-evaluated as soon as you modify and save the Dockerfile again, so you will always see the live version of the problem detected against the Dockerfile best practices.

TIBCO BusinessWorks HashiCorp Vault Integration: Secure Secrets in 3 Steps

TIBCO BusinessWorks HashiCorp Vault Integration: Secure Secrets in 3 Steps

Introduction

This article aims to show the TIBCO BW Hashicorp Vault Configuration to integrate your TIBCO BW application with the secrets stored in Hashicorp Vault, mainly for the externalization and management of password and credentials resources.

This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.

As you probably know, in the TIBCO BW application, the configuration is stored in Properties at different levels (Module or Application properties). You can read more about them here. And the primary purpose of that properties is to provide flexibility to the application configuration.

These properties can be of different types, such as String, Integer, Long, Double, Boolean, and DateTime, among other technical resources inside TIBCO BW, as shown in the picture below:

TIBCO BW Hashicorp Vault Configuration: BW Property Types

The TIBCO BW Hashicorp Vault integration will affect only those properties of Password type (at least up to 2.7.2/6.8.1 BW version). The reason behind that is that those properties are the kind of data relevant to the information that is sensitive and needs to be secure. Other concepts can be managed through standard Kubernetes components such as ConfigMaps.

BW Application Definition

We are going to start with a straightforward application, as you can see in the picture below:

TIBCO BW Hashicorp Vault Configuration: Property sample

Just a simple timer that will be executed once and insert the current time into the PostgreSQL database. We will use Hashicorp Vault to store the password of the database user to be able to connect to it. The username and the connection string will reside on a ConfigMap.

We will skip the part of the configuration regarding the deployment of the TIBCO BW application Containers and link to a ConfigMap you have an article covering that in detail in case you need to follow it, and we will focus just on the topic regarding TIBCO BW Hashicorp Vault integration.

So we will need to tell TIBCO BW that the password of the JDBC Shared Resource will be linked to Hashicorp Vault configuration, and to do that, the first thing is to have tied the Password of the Shared Resources to a Module Property as shown in the picture below:

TIBCO BW Hashicorp Vault Configuration: Password linked to Module Property

Now, we need to tell this Module Property that is Linked to Hashicorp Vault, and we will do that on the Application Property View, selecting that this property is linked to a Credential Management Solution as shown in the picture below:

TIBCO BW Hashicorp Vault Configuration: Credential Management Configuration for Property

And it is now when we establish the TIBCO BW Hashicorp Vault relationship. We need to click directly on the green plus sign, and we will have a modal window asking for the technology of credentials management that we’re going to use and the data needed for each of them, as you can see in the following picture:

TIBCO BW Hashicorp Vault Configuration: Credential Management Configuration for Property

We will select Hashicorp Vault as the provided. Then we will need to provide three attributes that we already commented on in the previous article when we start creating secrets in Hashicorp Vault:

  • Secret Name: this is the secret name path after the root path of the element.
  • Secret Key: This is the key inside the secret itself
  • Mount Path: This is the root path of the secret

To get more details about these three concepts, please look at our article about how to create secrets in Hashicorp Vault.

So with all this, we have pretty much everything we need to connect to Hashicorp Vault and grab the secret, and from the TIBCO BW BusinessStudio side, everything is done; we can generate the EAR file and deploy it into Kubernetes because here it is the last part of our configuration.

 Kubernetes Deployment

Until this moment, we have the following information already provided:

  • BW Process that has the login to connect to the Database and insert information
  • Link between the password property used to connect and the Hashicorp Secret definition

So, pretty much everything is there, but one concept is missing. How will the Kubernetes Pod connect to Hashicorp once the pod is deployed? Until this point, we didn’t provide the Hashicorp Vault server location of the authentication method to connect to it. This is the missing part of the TIBCO BW Hashicorp Vault integration and will be part of the Kubernetes Deployment YAML file.

We will do that using the following environment properties in this sample:

TIBCO BW Hashicorp Vault Configuration: Hashicorp Environment Variables
  • HASHICORP_VAULT_ADDR: This variable will point to where the Hashicorp Vault server is located
  • HASHICORP_VAULT_AUTH: This variable will indicate which authentication options will be used. In our case, we will use the token one as we used in the previous article
  • HASHICORP_VAULT_KV_VERSION: This variable indicates which version of the KV storage solution we are using and will be two by default.
  • HASHICORP_VAULT_TOKEN: This will be just the token value to be able to authentication against the Hashicorp Vault server

If you are using other authentication methods or just want to know more about those properties please take a look at this documentation from TIBCO.

With all that added to the environment properties of our TIBCO BW application, we can run it, and we will get an output similar to this one, and that shows that the TIBCO BW Hashicorp Vault integration has been done and the application was able to start without any issue

TIBCO BW Hashicorp Vault Configuration: Running sample

Create Secrets in HashiCorp Vault: CLI and REST API Explained Step by Step

Create Secrets in HashiCorp Vault: CLI and REST API Explained Step by Step

Introduction

Create secrets in Hashicorp Vault is one of the most important and relevant things you can do once you have installed Hashicorp Vault on your environment, probably by recovering and getting these secrets from the components they need it. But in today’s article, we will focus on the first part so you can learn how easily you can create secrets in Hashicorp Vault.

In previous articles we commented on the importance of Hashicorp Vault and the installation process, as you can read here. Hence, at this point, we already have our vault ready to start working with it wholly initialized and unseal to be able to start serving requests.

Create Secrets in Hashicorp Vault using Hashicorp Vault CLI Commands

All the commands we will do will use a critical component named Hashicorp Vault CLI, and you will notice that because all of our commands will start with vault. To be honest, we already started with that in the previous article; if you remember, we already run some of these commands to initialize or unseal the vault, but now this will be our main component to interact with.

The first thing we need to do is to be able to log into the vault, and to do that; we are going to use the root token that was provided to us when we initialized the vault; we are going to store this vault in an environment variable so it will be easy to work with it. All the commands we are going to run now are going to be inside the vault agent server pod, as shown in the picture below:

Create Secrets in Hashicorp Vault: Detecting Vault Server Pod

Once we are inside of it, we are going to do the log command with the following syntax:

 vault login 

And we will get an output similar to this one:

Create Secrets in Hashicorp Vault: Login in Hashicorp Vault

If we do not provide the token in advance, the console will ask for the token to be typed afterward, and it will be automatically hidden, as you can see in the picture below:

Create Secrets in Hashicorp Vault: Login without Token provided

After this point, we are already logged into the vault, so we can start typing commands to create secrets in Hashicorp Vault. Let’s start with that process.

To start with our process for creating secrets in Hashicorp Vault, we first need to make or be more accurate to the Hashicorp Vault syntax to enable a secret path that you can think about as the root path to which all your secrets will be related to. If we are talking about having secrets for different applications, each path can be each of the applications, but the organization of secrets can be other depending on the context. We will cover that in much more detail in the following articles.

To enable the secret path to start the creation of secrets in Hashicorp Vault, we will type the following command:

 vault secrets enable -path=internal kv-v2

That will enable a secret store of the type kv-v2 (key-value store in its v2), and the path will be “internal,” so everything else that we create after that will be under this “internal” root path.

And now, we’re going to do the creation of the secret in Hashicorp Vault, and as we are using a key-value store, the syntax is also related to that because we are going to “put” a secret using the following command:

 vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"

That will create inside the internal path a child path /database/config where it will store two keys:

  • username that will have the value db-readonly-username
  • password that will have the value db-secret-password

As you can see, it is quite easy to create new secrets on the Vault linked to the path, and if you want to retrieve its content, you can also do it using the Vault CLI, but this time using the get command as shown in the snippet below:

 vault kv get internal/database/config

And the output will be similar to the one shown below:

Create Secrets in Hashicorp Vault: Retrieving Secrets from the Vault

This will help you interact with your store’s content to retrieve, add or update what you already have there. Once you have everything ready there, we can move to the client side to configure it to gather all this data as part of its lifecycle workflow.

Create Secrets in Hashicorp Vault using REST API

The Hashicorp Vault CLI simplifies the interaction with the vault server, but all the interaction between the CLI and the server happens through a REST API that the server exposes and the CLI client consumes. It provides a simplified syntax to the user and translates the parameters provided into REST requests to the server, but you can use also REST requests to go to the server directly. Please look at this article in the official documentation to get more details about the REST API.

HashiCorp Vault Installation on Kubernetes Using Helm (Quick Start Guide)

HashiCorp Vault Installation on Kubernetes Using Helm (Quick Start Guide)

Introduction

In this article, we are going to cover the Hashicorp Vault Installation on Kubernetes. Hashicorp Vault has become one of the industry standards when we talk about managing secrets and sensitive data in production environments, and this covers cloud and non-cloud-native deployments. But especially in Kubernetes, this is a critical component. We have already commented that the Kubernetes Secrets are not very secured by default, so HashiCorp Vault solves that problem.

Installation Methods

Hashicorp Vault provides many different installation methods that you can read about on their official page here; most still focus on a traditional environment. But in summary, these are the ones you have available:

  • Install from Package Manager
  • Install from pre-existing binary
  • Install it from the source
  • Helm for Kubernetes

As you can imagine, the path we will follow here is the Helm way. I guess you are already familiar with help, but if not, I have some articles focus around Helm Charts that you can find all valuable information.

Helm Chart for Hashicorp Vault

For the sake of this article, we are going to what is called a standalone hashicorp vault installation, so we are not going to create in this post an architecture with High-Availability (HA) that is production-ready but something that can help you to start playing with the tool and see how this tool can be integrated with other ones that belong to the same cloud-native environment. To get more information about deploying Hashicorp Vault into a production-ready setup, please look at the following link.

We first need to install the helm chart in our local environment, but we need to be very careful about the helm version we have. When writing this article, Hashicorp Vault Installation requires a 3.7+ Helm Version, so you must first check the version you have installed.

In case you’re running on an older version, you will get the following error:

 Error: parse error at (vault/templates/_helpers.tpl:38): unclosed action

You can get more details on this GitHub issue.

At the time of writing this article, the latest version of Helm is 3.9, but this version generates an issue with AWS EKS with this error:

 Error: Kubernetes cluster unreachable: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1."
Failed installing **** with helm

You can get more details on this GitHub issue.

So, in that case, the best way to ensure there will not be a problem with the Hashicorp Vault Installation is to downgrade to 3.8, and you will be able to deploy the helm chart without any issue.

Hashicorp Vault Installation Process

To proceed with the Hashicorp Vault Installation, we need to run the following commands:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault

This will install two different components a single vault server as part of a StatefulSet and a vault-agent-injector to manage the injection of vault configuration on the various components and deployments on the other namespaces.

To get the pods running, we need to initialize and unseal the vault before being ready to use. To do that, we need to enter inside the vault-server pod and execute the following commands:

 vault operator init

This will generate several essential things:

  • It will generate the keys to be able to unseal the vault to be able to start using it. It will prompt a different number of keys, in our sample 5, and you will need at least 3 of them to be able to unseal the vault as
  • It will also generate a root token to be able to log into the CLI and interactor with the server to be able to read and write secrets

After that, we will need to run the following command at least three times, providing each of them with a different unseal key:

 Vault operator unseal

After that point, all components are Running and Ready and we can conclude our Hashicorp Vault Installation and start interacting with the vault to create your secrets.

Hashicorp Vault Installation: All Components Ready

Helm Loops Explained: A Practical Helm Hack to Avoid Deployment Issues

Helm Loops Explained: A Practical Helm Hack to Avoid Deployment Issues

Introduction



When working with complex Helm deployments, mastering loops is just one piece of the puzzle. For a comprehensive understanding of Helm from fundamentals to advanced patterns, check out our complete Helm Charts & Kubernetes Package Management Guide.

Helm Charts are becoming the default de-factor solution when you want to package your Kubernetes deployment to be able to distribute or quickly install it in your system.

Defined several times as the apt for Kubernetes for its similarity with the ancient package manager from GNU/Linux Debian-like distributions, it seems to continue to grow in popularity each month compared with other similar solutions even more tightly integrated into Kubernetes such as Kustomize, as you can see in the Google Trends picture below:

Helm Loops: Helm Charts vs Kustomize

But creating these helm charts is not as easy as it shows. If you already have been on the work of doing so, you probably get stuck at some point, or you spend a lot of time trying to do some things. If this is the first time you are creating one or trying to do something advanced, I hope all these tricks will help you on your journey. Today we are going to cover one of the most important tricks, and those are Helm Loops.

Helm Loops Introduction

If you see any helm chart for sure, you will have a lot of conditional blocks. Pretty much everything is covered under an if/else structure based on the values.yml files you are creating. But this gets a little bit tricky when we talk about loops. But the great thing is that you will have the option to execute a helm loop inside your helm charts using the rangeprimitive.

How to create a Helm Loop?

The usage of the rangeprimitive is quite simple, as you only need to specify the element you want to iterate across, as shown in the snippet below:

{{- range .Values.pizzaToppings }}
- {{ . | title | quote }}
{{- end }}    

This is a pretty simple sample where the yaml will iterate over the values that you have assigned to the pizzaToppings structure in your values.yml

There are some concepts to keep in mind in this situation:

  • You can easily access everything inside this structure you are looping across. So, if pizza topping has additional fields, you can access them with something similar to this:
{{- range.Values.pizzaToppings }}
- {{ .ingredient.name | title | quote }}
{{- end }}    

And this will access a structure similar to this one in your values.yml:

 pizzaToppings:
	- ingredient:
		name: Pinneaple
		weight: 3

The good thing is that you can access their underlying attribute without replicating all the parent hierarchy until you reach the looping structure because inside the range section, the scope has changed. We will refer to the root of each element we are iterating across.

How to access parent elements inside a Helm Loop?

In the previous section, we covered how easily we can access the inner attribute inside the loop structure because of the change of scope, which also has an issue. In case I want to access some element in the parent of my values.yml file or somewhere outside the structure, how can I access them?

The good thing is that we also have a great answer to that, but you can get there. We need to understand a little bit about the scopes in Helm.

As commented, . refers to the root element in the current scope. If you have never defined a range section or another primitive that switches the context, .always will refer to the root of your values.yml. That is why when you see a helm chart, you see all the structures with the following way of working: .Values.x.y.z, but we already have seen that when we have a range section, this is changing, so this is not a good way.

To solve that, we have the context $ that constantly refers to the root of the values. ymlno matter which one is the current scope. So that means that if I have the following values.yml:

base:
	- type: slim 
pizzaToppings:
	- ingredient:
		name: Pinneaple
		weight: 3
	- ingredient:
		name: Apple
		weight: 3

And I want to refer to the base type inside the range section similar to before I can do it using the following snippet:

{{- range .Values.pizzaToppings }}
- {{ .ingredient.name | title | quote }} {{ $.Values.base.type }}
{{- end }}    

That will generate the following output:

 - Pinneaple slim
 - Apple slim

So I hope this helm chart trick will help you with the creation, modification, or improvement of your upgraded helm charts in the future by using helm loops without any further concern!

BanzaiCloud Logging Operator on Kubernetes: Log Aggregation in 5 Minutes

BanzaiCloud Logging Operator on Kubernetes: Log Aggregation in 5 Minutes

In the previous article, we described what capability BanzaiCloud Logging Operator provides and its main features. So, today we are going to see how we can implement it.

The first thing we need to do is to install the operator itself, and to do that, we have a helm chart at our disposal, so the only thing that we will need to do are the following commands:

 helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm upgrade --install --wait --create-namespace --namespace logging logging-operator banzaicloud-stable/logging-operator

That will create a logging namespace (in case you didn’t have it yet), and it will deploy the operator components itself, as you can see in the picture below:

BanzaiCloud Logging Operator on Kubernetes: Log Aggregation in 5 Minutes
BanzaiCloud Logging Operator installed using HelmChart

So, now we can start creating the resources we need using the CRD that we commented on in the previous article but to do a recap. These are the ones that we have at our disposal:

  • logging – The logging resource defines the logging infrastructure for your cluster that collects and transports your log messages. It also contains configurations for Fluentd and Fluent-bit.
  • output / clusteroutput – Defines an Output for a logging flow, where the log messages are sent. output will be namespaced based, and clusteroutput will be cluster based.
  • flow / clusterflow – Defines a logging flow using filters and outputs. The flow routes the selected log messages to the specified outputs. flow will be namespaced based, and clusterflows will be cluster based.

So, first of all, we are going to define our scenario. I don’t want to make something complex; I wish that all the logs that my workloads generate, no matter what namespace they are in, are sent to a Grafana Loki instance that I have also installed on that Kubernetes Cluster on a specific endpoint using the Simple Scalable configuration for Grafana Loki.

So, let’s start with the components that we need. First, we need a Logging object to define my Logging infrastructure, and I will create it with the following command.

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: default-logging-simple
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: logging
EOF

We will keep the default configuration for fluentd and fluent-bit just for the sake of the sample, and later on in upcoming articles, we can talk about a specific design, but that’s it.

Once the CRD is processed, the components will appear on your logging namespace. In my case that I’m using a 3-node cluster, I will see 3 instances for fluent-bit deployed as a DaemonSet and a single example of fluentd, as you can see in the picture below:

BanzaiCloud Logging Operator on Kubernetes: Log Aggregation in 5 Minutes
BanzaiCloud Logging Operator configuration after applying Logging CRD

So, now we need to define the communication with Loki, and as I would like to use this for any namespace I can have on my cluster, I will use the ClusterOutput option instead of the normal Output one that is namespaced based. And to do that, we will use the following command (please ensure that the endpoint is the right one; in our case, this is loki-gateway. default as it is running inside the Kubernetes Cluster:

kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
 name: loki-output
spec:
 loki:
   url: http://loki-gateway.default
   configure_kubernetes_labels: true
   buffer:
     timekey: 1m
     timekey_wait: 30s
     timekey_use_utc: true
EOF

And pretty much we have everything; we just need one flow to communicate our Logging configuration to the ClusterOutput we just created. And again, we will go with the ClusterFlow because we would like to define this at the Cluster level and not in a by-namespaced fashion. So we will use the following command:

 kubectl -n logging  apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: loki-flow
spec:
  filters:
    - tag_normaliser: {}
  match:
    - select: {}
  globalOutputRefs:
    - loki-output
EOF

And after some time to do the reload of the configuration (1-2 minutes or so), you will start to see in the Loki traces something like this:

BanzaiCloud Logging Operator on Kubernetes: Log Aggregation in 5 Minutes
Grafana showing the logs submitted by the BanzaiCloud Logging Operator

And that indicates that we are already receiving push of logs from the different components, mainly the fluentd element we configured in this case. But I think it is better to see it graphically with Grafana:

BanzaiCloud Logging Operator on Kubernetes: Log Aggregation in 5 Minutes
Grafana showing the logs submitted by the BanzaiCloud Logging Operator

And that’s it! And to change our logging configuration is as simple as changing the CRD component we defined, applying matches and filters, or sending it to a new place. Straightforwardly we have this completely managed.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Log Aggregation in Kubernetes Explained with BanzaiCloud Logging Operator

Log Aggregation in Kubernetes Explained with BanzaiCloud Logging Operator

We already have talked about the importance of Log Aggregation in Kubernetes and why the change in the behavior of the components makes it a mandatory requirement for any new architecture we deployed today.

To solve that part, we have a lot of different stacks that you probably have heard about. For example, if we follow the traditional Elasticsearch path, we will have the pure ELK stack from Elasticsearch, Logstash, and Kibana. Now this stack has been extended with the different “Beats” (FileBeat, NetworkBeat, …) that provides a light log forwarder to be added to the task.

Also, you can change Logstash for a CNCF component such as Fluentd that probably you have heard about, and in that case, we’re talking about an EFK stack following the same principle. And also, have the Grafana Labs view using promtail, Grafana Loki, and Grafana for dashboarding following a different perspective.

Then you can switch and change any component for the one of your preference, but in the end, you will have three different kinds of components:

  • Forwarder: Component that will listen to all the log inputs, mainly the stdout/stderr output from your containers, and push it to a central component.
  • Aggregator: Component that will receive all the traces for the forwarded, and it will have some rules to filter some of the events, format, and enrich the ones received before sending it to central storage.
  • Storage: Component that will receive the final traces to be stored and retrieved for the different clients.

To simplify the management of that in Kubernetes, we have a great Kubernetes Operator named BanzaiCloud Logging Operator that tries to follow that approach in a declarative / policy manner. So let’s see how it works, and to explain it better, I will use its central diagram from its website:

Log Aggregation in Kubernetes Explained with BanzaiCloud Logging Operator
BanzaiCloud Logging Operator Architecture

This operator uses the same technologies we were talking about. It covers mainly the two first steps: Forwarding and Aggregation and the configuration to be sent to a Central Storage of your choice. To do that works with the following technologies, all of them part of the CNCF Landscape:

  • Fluent-bit will act as a forwarded deployed on a DaemonSet mode to collect all the logs you have configured.
  • Fluentd will act as an aggregator defining the flows and rules of your choice to adapt the trace flow you are receiving and sending to the output of your choice.

And as this is a Kubernetes Operator, this works in a declarative way. We will define a set of objects that will define our logging policies. We have the following components:

  • logging – The logging resource defines the logging infrastructure for your cluster that collects and transports your log messages. It also contains configurations for Fluentd and Fluent-bit.
  • output / clusteroutput – Defines an Output for a logging flow, where the log messages are sent. output will be namespaced based, and clusteroutput will be cluster based.
  • flow / clusterflow – Defines a logging flow using filters and outputs. The flow routes the selected log messages to the specified outputs. flow will be namespaced based, and clusterflows will be cluster based.

In the picture below, you will see how these objects are “interacting” to define your desired logging architecture:

Log Aggregation in Kubernetes Explained with BanzaiCloud Logging Operator
BanzaiCloud Logging Operator CRD Relationship

And apart from the policy mode, it also includes a lot of great features such as:

  • Namespace isolation
  • Native Kubernetes label selectors
  • Secure communication (TLS)
  • Configuration validation
  • Multiple flow support (multiply logs for different transformations)
  • Multiple output support (store the same logs in multiple storages: S3, GCS, ES, Loki, and more…)
  • Multiple logging system support (multiple Fluentd, Fluent Bit deployment on the same cluster)

In upcoming articles we were talking about how we can implement this so you can see all the benefits that this CRD-based, policy-based approach can provide to your architecture.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Grafana Loki with MinIO: Scalable Log Storage for Kubernetes without S3

Grafana Loki with MinIO: Scalable Log Storage for Kubernetes without S3

Grafana Loki is becoming one of the de-facto standards for log aggregation in Kubernetes workloads nowadays, and today, we are going to show how we can use together Grafana Loki and MinIO. We already have covered on several occasions the capabilities of Grafana Loki that have emerged as the main alternative to the Elasticsearch leadership in the last 5-10 years for log aggregation.

With a different approach, more lightweight, more cloud-native, more focus on the good things that Prometheus has provided but for logs and with the sponsorship of a great company such as Grafana Labs with the dashboard tools as the leader of each day more enormous stack of tools around the observability world.

And also, we already have covered MinIO as an object store that can be deployed anywhere. It’s like having your S3 service on whatever cloud you like or on-prem. So today, we are going to see how both can work together.

Grafana Loki mainly supports three deployment models: monolith, simple-scalable, and distributed. Pretty much everything but monolith has the requirement to have an Object Storage solution to be able to work on a distributed scalable mode. So, if you have your deployment in AWS, you already have covered with S3. Also, Grafana Loki supports most of the Object Storage solutions for the cloud ecosystem of the leading vendors. Still, the problem comes when you would like to rely on Grafana Loki for a private cloud or on-premises installation.

In that case, is where we can rely on MinIO. To be honest, you can use MinIO also in the cloud world to have a more flexible and transparent solution and avoid any lock-in with a cloud vendor. Still, for on-premises, its uses have become mandatory. One of the great features of MinIO is that it implements the S3 API, so pretty much anything that supports S3 will work with MinIO.

In this case, I just need to adapt some values on the helm chart from Loki in the simple-distributed mode as shown below:

 loki:
  storage:
    s3:
      s3: null
      endpoint: http://minio.minio:9000
      region: null
      secretAccessKey: XXXXXXXXXXX
      accessKeyId: XXXXXXXXXX
      s3ForcePathStyle: true
      insecure: true

We’re just pointing to the endpoint from our MinIO tenant, in our case, also deployed on Kubernetes on port 9000. We’re also providing the credentials to connect and finally just showing that needs s3ForcePathSyle: true is required for the endpoint to be transformed to minio.minio:9000/bucket instead to bucket.minio.minio:9000, so it will work better on a Kubernetes ecosystem.

And that’s pretty much it; as soon as you start it, you will begin to see that the buckets are starting to be populated as they will do in case you were using S3, as you can see in the picture below:

MinIO showing buckets and objects from Loki configuration
MinIO showing buckets and objects from Loki configuration

We already covered the deployment models from MinIO. As shown here, you can use its helm chart or the MinIO operator. But, the integration with Loki it’s even better because the helm charts from Loki already included MinIO as a sub-chart so you can deploy MinIO as part of your Loki deployment based on the configuration you will find on the values.yml as shown below:

 # -------------------------------------
# Configuration for `minio` child chart
# -------------------------------------
minio:
  enabled: false
  accessKey: enterprise-logs
  secretKey: supersecret
  buckets:
    - name: chunks
      policy: none
      purge: false
    - name: ruler
      policy: none
      purge: false
    - name: admin
      policy: none
      purge: false
  persistence:
    size: 5Gi
  resources:
    requests:
      cpu: 100m
      memory: 128Mi

So with a single command, you can have both platforms deployed and configured automatically! I hope this is as useful for you as it was for me when I discovered and did this process.

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

Write Kubernetes YAML Manifests More Efficiently: Tools, Templates, and Best Practices

Write Kubernetes YAML Manifests More Efficiently: Tools, Templates, and Best Practices

When we are all in this new cloud-native environment where Kubernetes is the uncontestable king, you need to learn how to deal with Kubernetes YAML manifest all the time. You will become an expert on indent sections to make sure this can be processed and so on. But we need to admit that it is tedious. All the benefits from the Kubernetes deployment make an effort worth it, but even with that, it is pretty complex to be able to handle it.

It is true that, to simplify this situation, there are a lot of projects that have been launched, such as Helm to manage templates of related Kubernetes YAML manifest or Kustomize different approaches to get to the sample place or even solutions that are specific to a Kubernetes distribution such as the Openshift Templates. But in the end, none of this can solve the problem at the primary level. So you need to write those files manually yourself.

And what is the process now? You are probably following a different one, but I will tell you my approach. Depending on what I’m trying to create, I try to find a template available for the Kubernetes YAML Manifest that I want to make. This template can be some previous resource that I have already created. Hence, I use that as a base, it could be something generated for some workload that is already deployed (so great that Lens has existed to simplify the management of Running Kubernetes workloads! If you don’t know Lens, please take a look at this article) or if you don’t have anything at hand, you search on google about something similar probably in the Kubernetes documentation, stack overflow or the first reasonable resource that Google provides to you.

And after that, the approach is the same. You go to your Text Editor, VS Code in my case. I have a lot of different plugins to make this process less painful. A lot of different linters validate the structure of the Kubernetes YAML Manifest to make sure everything is indented property, that there are no repeated tags or no missing mandatory tags in the latest version of the resource, and so on.

Things got a bit tricky if you are creating a Helm Chart because in that case the linters for YAML don’t work that well and detect some false positives because they don’t truly understand the Helm syntax. You also complete your setup with a few more linters for Helm, and that’s it. You fight error and error and change by change to have your desired Kubernetes YAML Manifest.

But, it should be a better way to do that? Yes, it should, and this is what tools such as Monokle try to provide a better experience of that process. Let’s see how that works. Starting from their contributor words:

Monokle is your friendly desktop UI for managing Kubernetes manifests. Monokle helps you quickly get a high-level view of your manifests and their contained resources, easily edit resources without having to learn yaml syntax, diff resources against your cluster, preview and debug resources generated with kustomize or Helm, and more.

Monokle helps you in the following ways. First of all, present at the beginning of your work with a set of templates to create your Kubernetes YAML Manifests, as you can see in the picture below:

Write Kubernetes YAML Manifests More Efficiently: Tools, Templates, and Best Practices
Monokle Template Selection Dialog

When you select a template, you can populate the required values graphically without needing to type YAML code yourself, as you can see in the picture below:

Write Kubernetes YAML Manifests More Efficiently: Tools, Templates, and Best Practices
Monokle Template Value Population Process

It also supports Helm Chart and Kustomize resource recognition so you will quickly see your charts, and you can edit them in a more fashion mode even graphically for some of the resources as well:

Write Kubernetes YAML Manifests More Efficiently: Tools, Templates, and Best Practices
Helm Chart Modification using Monokle

If allows good integration in several ways, first of all with OPA so it can validate all the rules and best-practice that you have defined and also you can connect to a running cluster to see the resources from there and also see the difference between them if exists to simplify the process and provide more agility on the Kubernetes YAML Manifest creation process

On top of all of that, Monokle is a certified component from the CNCF foundation, so you will be using a project that is backup from the same foundation is that takes care of Kubernetes itself, among other tasks:

Write Kubernetes YAML Manifests More Efficiently: Tools, Templates, and Best Practices
Monokle is part of the CNCF Foundation Landscape

If you want to download Monokle, give it a try and you can do it from its web page: https://monokle.kubeshop.io/ and I’m sure your performance writing Kubernetes YAML Manifest will thank you shortly!

📚 Want to dive deeper into Kubernetes? This article is part of our comprehensive Kubernetes Architecture Patterns guide, where you’ll find all fundamental and advanced concepts explained step by step.

TIBCO BusinessWorks Modules Explained: Types, Limitations, and Best Practices

TIBCO BusinessWorks Modules Explained: Types, Limitations, and Best Practices

TIBCO BW Modules are one of the most relevant contents on your TIBCO BW developments. Learn all the details about the different TIBCO BW Modules available and when to use each of them.

TIBCO BW has evolved in several ways and adapter to the latest changes of architecture. Because of that, since the conception of the latest major version, it has introduced several concepts that is important to master to be able to unleash all the power that this remarkable tool provides to you. Today we are going to talk about the Modules.

This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.

Every TIBCO BW application is composed of different modules that are the components that host all the logic that you can create, and that’s the first thing to write down: All your code and everything you do in your application will belong to one TIBCO BW Module.

If we think about the normal hierarchy of TIBCO BW components it will be something like that picture below:

TIBCO BusinessWorks Modules Explained: Types, Limitations, and Best Practices

At the top level, we will have the Application; at the second level, we will have the modules, and after that, we will have the packages and finally, the technical components such as Process, Resources, Classes, Schemas, Interfaces, and so on. Learn more about this here.

TIBCO BW Module Classification

There are several kind of module and each of them provides a specific use-case and has some characteristics associated with it.

  • Application Module: It is the most important kind of module because without each you cannot have an application. It is the master module and only can be one per application. It is where all your main logic to that application will reside.
  • Shared Module: It is the other only BW native module and it is main purpose as the name shows it is to host all the code and components that can be shared between several applications. If you have experience with previous versions of TIBCO BW you can think on this TIBCO BW Module as a replacement of a Design Time Library (a.k.a DTL) or if you have experience with a programming language a library that is imported to the code. Because of that it doesn’t have a restriction on the number of applications that can use a share module and there is no limitation on the number of share modules that a TIBCO BW Application can have.
  • OSGI Module: This module is the one that is not BW native and it is not going to be include BW objects such as Processes, Resources and so on, and there are mainly concieved to have Java classes. And again it is more like a helper module that also can be shared as needed. Usual scenarios for use this kind of module is to define Custom XPath Functions for example or to have Java Code shared between several applications.

Both Shared Modules and OSGI Modules can be defined as Maven dependencies and use the Maven process to publish them in a Maven repository and also to be retrieved from it based on the declaration.

That provides a very efficient way for distribution and version control of these shared components and, at the same, offers a similar process for other programming languages such as Java so that it will decrease the learning curve for that process.

TIBCO BW Module Limitations

As we already commented, there are some limitations or special characteristics that each module has. We should be very aware of it to help us properly distribute our code using the right kind of modules.

As commented, one application can have only one TIBCO BW Application Module. Even though it is technically possible to have the same BW Application Module in more than one application, that has no sense because both applications will be the same as its main code will be the same.

TIBCO BW Shared Modules at other hand, cannot have Starter components or Activator process as part of its declaration and all of them should reside on the TIBCO BW Application Module

Both TIBCO BW Application Module and TIBCO BW Shared Module can have Java code, but on the other way, the OSGI module can only have Java code and no other TIBCO BW resources.

TIBCO BW Shared Modules can be exported in two different ways, as regular modules (ZIP file with the source code) and in Binary format, to be shared among other developers but not allowing them to change or change their view of the implementation details. This is still supported for legacy reasons, but today’s recommended way to distribute the software is using Maven, as discussed above.

TIBCO BW Module Use-Cases

As commented there are different use cases for each of the module that because of that it will help you decide which component work best for each scenario:

  • TIBCO BW Shared Modules covers all the standard components needed for all the applications. Here, the main use-case is the framework components or main patterns that simplify the development and homogenize. This helps control standard capabilities such as error handling, auditing, logging, or even internal communication, so the developers only need to focus on the business logic for their use case.
  • Another use-case for TIBCO BW Shared Module encapsulates anything shared between applications, such as Resources, to connect to one backend, so all the applications that need to connect to that backend can import and avoid the need to re-work that part.
  • OSGi Module is to have Java code that has a weak relationship with the BW code, such as component to perform an activity such as Sign a PDF Document or Integrate with an element using a Java native API so we can keep it and evolve it separate to the TIBCO BW Code.
  • Another case for OSGI Module is defining the Custom XPath Functions that you will need as part of your Shared Module or your Application Module.
  • TIBCO BW Application Module, on the other hand, only should have code that is specific to the business problem that we are resolving here, moving all code that can be used for more than one application to a Shared Module.