On this article we are going to cover how TIBCO BW EMS Reconnection works and how you can apply it on your application and the pros and const about the different options available.
One of the main issues we all have faced when working on a TIBCO BW and EMS integration is the reconnection part. Even though this is something that we need on minimal occasions because of the TIBCO EMS server’s extreme reliability, it can have severe consequences if we don’t have that well configured.
This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.
This article will cover how to quickly integrate Grafana and LDAP server to increase the security of your application
Grafana is one of the most used dashboard tools, mainly for the observability platforms on cloud-workload environments. But none of these tools is complete until you can configure them to meet your security standards. Security is becoming each time more and more important and this is especially true when we are talking about any cloud-related component. So this topic should always something that any cloud architect has in mind when they are defining or implementing any piece of software today.
And at that point, the LDAP integration is one of the main things you will always need to do. I don’t know any enterprise, big or small, that allows the usage of any tool with a Graphical User Interface if it is not connected to the corporate directory.
So, let’s see how we can implement this with a popular tool such as Grafana. In my case, I’m going to use a containerized version of grafana, but the steps and things to do remains the same no matter the deployment model.
The task that we are going to perform is based on three steps:
Enable the LDAP settings
Basic LDAP settings configuration
Group mapping configuration
The first thing we need to modify is the grafana.ini adding the following snippet:
That means that we are enabling the authentication based on LDAP and setting the location of the LDAP configuration file. This ldap.toml has all the configuration needed for the LDAP:
The first part is regarding the primary connection. We will establish the host location and port, the admin user, and the password. After that, we will need a second section that definesearch_filter and search_base_dns to define the users that can access the system.
Finally, we have another section to define the mapping between the LDAP attributes and the grafana attributes to be able to gather the data from the LDAP.
With all those changes in place, you need to restart the Grafana server to see all this configuration applied. After that point, you can log in using the LDAP credentials as you can see in the picture below and see all the data retrieved from the LDAP Server:
Using the default admin server, you can also use a new feature to test the LDAP configuration using the LDAP option that you can see in the picture below:
And then you can search for a User and you will see how this user will play on the Grafana Server:
Check the attributes will map from the LDAP server to the Grafana server
Also check the status of activity and the role allowed to this user
You can see one sample in the picture below:
I hope that this article helps you in a way to level up the security settings on your Grafana installations to integrate with the LDAP server. If you want to see more information about this and similar topics I encourage you to take a look at the links that you would find below these sentences.
TIBCO BW supports many different integration methods and hundreds of connectors that allow you to connect to any possible source. But truth must be told, EMS is one of the standard connectors you need to enable. That’s why TIBCO BW and EMS usually goes together when it comes to a proper Integration Platform.
This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.
JMS Support for TIBCO BW is out of the box, but like any other JMS implementation, you need to provide the client libraries to establish a real connection.
To do that, since TIBCO BW 6, a simple way is provided to simplify that process, and this is what we are going to cover in this article.
Problem description
The first thing is to know that you need to do something and the most important thing is to learn to understand what kind of error is related to this problem. You could find two different errors depending on where you are testing this: design-time or runtime.
If we are talking about a runtime issue, you can see a trace similar to this one:
2022-06-02T13:27:15,867 ERROR [pool-13-thread-2] c.t.b.thor.runtime.model.Constituent - The following error has occurred for "name: test-app version: 1.0.0.qualifier bundle name: test-app " which needs to be resolved.
2022-06-02T13:27:15,878 ERROR [pool-13-thread-2] c.t.b.thor.runtime.model.Constituent - TIBCO-BW-FRWK-600053: Failed to initialize BW Component [ComponentStarter].
<CausedBy> com.tibco.bw.core.runtime.api.BWEngineException: TIBCO-BW-CORE-500232: Failed to initialize BW Component [ComponentStarter], Application [test-app:1.0] due to activity initialization error.
<CausedBy> com.tibco.bw.core.runtime.ActivityInitException: TIBCO-BW-CORE-500408: Failed to initialize the ProcessStarter activity [JMSReceiveMessage] in process [com.test.Starter], module [test-app] **due to unexpected activity lifecycle error.**
**<CausedBy> java.lang.NullPointerException**
Each time you see a java.lang.NullPointerExceptionrelated to a JMS Receive activity, you can be sure the issue is related to the installation of the drivers.
If we are talking about Design-time, you will see the same error when you are trying to start a Run or Debug session, but additional you will see the following error when you are testing a JMS Connection Resource, as you can see in the picture below:
Installation Process
The installation process is quite simple, but you need access to an EMS installation or at least a disk location with the clients stored. If you already have that, you just need to go to the following location:
TIBCO_HOME/bw/<version>/bin
Where TIBCO_HOME is the installation folder for the BusinessWorks application, and version is the minor version format (such as 6.7, 2.7, 6.8, and so on).
At this location, you will run the following command:
./bwinstall ems-driver
This will start and ask for the location of the client libraries, as you can see in the picture below:
And after that, it will do the process of installing it will end with the BUILD SUCCESSFULL output. And after that point, you will need to restart the Business Studio or the runtime components (such as AppNodes or bwagent) to have the configuration applied.
Kubernetes Operators: 5 Things You Truly Need to Know (Photo by Dan Lohmar on Unsplash)
Kubernetes Operator has been the new normal to deploy big workloads on Kubernetes, but as some of these principles don’t align immediately with the main concepts of Kubernetes usually generates a little bit of confusion and doubts when you need to use them or even create them.
What Are Kubernetes Operators?
Operators are the way to extend Kubernetes capabilities to manage big workloads where different options are related. In components with a distributed architecture such as monitoring system, log aggregation system, or even service mesh, you can find that. Based on the words from the Kubernetes official documentation , operators are defined as below:
Operators are software extensions to Kubernetes that use custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.
Its primary usage is for standard services and not as much as the simple application or user workloads, but it could be used in cases even for that scenario.
How Does Kubernetes Operator Works?
The central concept behind the Kubernetes Operator is the extension concept. It is based on the definition and management of custom Kubernetes objects named Custom Resource Definition (CRDs) that allow a description in a Kubernetes way of new concepts that you could need for your workloads.
Some samples of these CRDs are the ServiceMonitor and PodMonitor that we explained in the previous posts, for example, but many more to add. So, that means that now you have a new YAML file to define your objects, and you can use the main primitives from Kubernetes to create, edit, or delete them as needed.
So, for these components to do any work, you need to code some specific controllers that are translating the changes done to those YAML files to reach primitives to the status of the cluster.
The Kubernetes operator can be installed like any other Kubernetes workload, so depending on the case can be distributed as a YAML file or a Helm Chart. You even can find a shared repository of operators on OperatorsHub.
OperatorHub: Central Repository for Kubernetes Operators
Kubernetes Operator vs. Helm Charts
As already discussed, they are not the same kind of object as Helm Charts because Helm Charts only work at the deployment level doing the packaging and managing of those releases, but operators go a step beyond that because managing and controlling the lifecycle at the runtime level. And as commented, Helm and Operators are compatible; this is, for example, how Prometheus Operator works that have a Helm Chart to deploy itself, as you can find here.
How To Build a Kubernetes Operator
If your goal after reading this is to create a Kubernetes Operator, you need to know that there are already some frameworks that will make your life easier at that task.
Tools like Kopf, kubebuilder, metacontroller , or even the CNCF Operator Framework will provide you the tools and the everyday tasks to start focusing on what your operator needs to do, and they will handle the main daily tasks for you.
More Resources To Learn about Kubernetes Operator
Suppose you want to learn more about Kubernetes Operators or the Operator pattern. In that case, I strongly recommend you look at the CNCF Operator Whitepaper that you can find here.
This will cover all the topics discussed above in more technical detail and introduce other vital issues, such as security lifecycle management or event best practices.
Other interesting resources are the bibliography resource from the Whitepaper itself that I am going to add here just in case you want to jump directly to the source:
Dobies, J., & Wood, J. (2020). Kubernetes Operators. O’Reilly.
Kubectl command can be the most used command you can type when working with the Kubernetes ecosystem. As you know, kubectl is the open the door to all the Kubernetes world as pretty much all of our interactions go through that part, unless you are using a CLI approach.
So, based on the productivity principles, if you can improve just 1% in the task that you perform the most, the global improvement will be massive. So, let’s see how we can do that here.
kubectl is a command with many different options that could help boost your productivity a lot. But at the same time, as it has so many options, it is pretty complex to know all of them or be aware that there is a faster way to do the job, and that’s why I would like to add some options here to try to help you with this set of kubectl tips.
Kubectl Commands Tips
Let’s start with the first kubectl commands that help a lot to improve your productivity:
kubectl explain <resource-object>
This command will show the API reference for any Kubernetes Object, so it would help you know the exact spelling of the option that you always miswrite.
kubectl get <resource-object> —watch-output
The —watch-output option added to any kubectl command will work in the same way as the watch command itself, so it will refresh the same command every 2.0 seconds to see the real-time version of that command and avoid that you need to type it again or rely on an external command such as watch
kubectl get events --sort-by=".lastTimestamp"
This command will help you when you want to see the events in your current context, but the main difference is that it will sort the output by the timestamp from more recent to older, so you will avoid needing to scroll to find the latest events.
kubectl logs --previous
We always talk about one of the needs for a Log Aggregation Architecture because the logs are disposable, but what about if you want to get the logs in a killed container? You can use the --previous flag to access the logs for a recently terminated container. This will not remove the need for a logging aggregation technique, but it will help troubleshoot when Kubernetes start killing things and you need to know what happened.
kubectl create allows us to create an object of our preference by providing the required arguments imperatively, but if we add the -o=yaml --dry-run=client option, we will not get our object created. Instead, we will have a YAML file defining that object. So we can easily modify it to our needs without needing to make it from scratch by searching Google for a sample to start with.
kubectl top pods --all-namespaces --sort-by='memory'
This command will alter the standard top pods order to show the pods and the resources they are consuming, and at the same time, it will sort that output by the memory usage. So, in environments with many pods, it will provide just at the top the ones you should focus on first to optimize the resources for your whole cluster.
Kubectl Alias
One step beyond that is to simplify those commands by adding an alias to this. As you can see, most of these commands are pretty long as they have many options, so writing each of these options will take a while.
So, if you want to go one step further on this optimization, you can always add an alias to that command to simplify it a lot. And if you want to learn more about those aliases, I strongly recommend the GitHub repo from Ahmet Alp Balkan:
All The Power of Object Storage In Your Kubernetes Environment
In this post, I would like to bring to you MinIO, a real cloud object storage solution with all the features you can imagine and even some more. You are probably aware of Object Storage from the AWS S3 service raised some years ago and most of the alternatives in the leading public cloud providers such as Google or Azure.
But what about private clouds? Is it something available that can provide all the benefits of object storage, but you don’t need to rely on a single cloud provider. And even more important than that, in the present and future, that all companies are going to be multi cloud do we have at our disposal a tool that provides all these features but doesn’t force us to have a vendor lock-in. Even some software, such as Loki, encourages you to use an object storage solution
The answer is yes! And this is what MinIO is all about, and I just want to use their own words:
“MinIO offers high-performance, S3 compatible object storage. Native to Kubernetes, MinIO is the only object storage suite available on every public cloud, Kubernetes distribution, the private cloud, and the edge. MinIO is software-defined and is 100% open source under GNU AGPL v3.”
So, as I said, everything you can imagine and even more. Let’s focus on some points:
Native to Kubernetes: You can deploy it in any Kubernetes distribution of choice, whether this is public or private (or even edge).
100% open source under GNU AGPL v3, so no vendor lock-in.
S3 compatible object storage, so it even simplifies the transition for customers with a strong tie with the AWS service.
High-Performance is the essential feature.
Sounds great. Let’s try it in our environment! So I’m going to install MinIO in my rancher-desktop environment, and doing that, I am going to use the operator that they have available here:
To be able to install, the recommended option is to use krew, the plugin manager we already talked about it in another article. The first thing we need to do is run the following command.
kubectl minio init
This command will deploy the operator on the cluster as you can see in the picture below:
Once done and all the components are running we can launch the Graphical interfaces that will help us create the storage tenant. To do so we need to run the following command:
kubectl minio proxy -n minio-operator
This will expose the internal interface that will help us during that process. We will be provided a JWT token to be able to log into the platform as you can see in the picture below:
Now we need to click on the button that says “Create Tenant” which will provide us a Wizard menu to create our MinIO object storage tenant:
On that wizard we can select several properties depending on our needs, as this is for my rancher desktop, I’ll try to keep the settings at the minimum as you can see here:
It would help if you had the namespace created in advance to be retrieved here. Also, you need to be aware that there can be only one tenant per namespace, so you will need additional namespaces to create other tenants.
As soon as you hit create, you will be provided with an API Key and Secret that you need to store (or download) to be able to use later, and after that, the tenant will start its deployment. After a few minutes, you will have all your components running, as you can see in the picture below:
If we go to our console-svc, you will find the following GUI available:
After the credentials are download in the previous step, we will enter the console for our cloud object store and be able to start creating our buckets as you can see in the picture below:
On the screen of creating a bucket, you can see several options, such as Versioning, Quota, and Object Locking, that give a view of the features and capability this solution has
And we can start uploading and downloading objects to this new bucket created:
I hope you can see this as an option for your deployments, especially when you need an Object Storage solution option for private deployments or just as an AWS S3 alternative.
Deployment Models for a Scalable Log Aggregation Architecture using Loki
Deploy a scalable Loki is not an straightforward task. We already have talked about Loki in previous posts on the site, and it is becoming more and more popular, and usage becomes much more regular each day. That is why I think it makes sense to include another post regarding Loki Architecture.
Loki has several advantages that promote it as a default choice to deploy a Log Aggregation Stack. One of them is its scalability because you can see across different deployment models how many components you like to deploy and their responsibilities. So the target of the topic is to show you how to deploy an scalable Loki solution and this is based on two concepts: components available and how you group them.
So we will start with the different components:
ingester: responsible for writing log data to long-term storage backends (DynamoDB, S3, Cassandra, etc.) on the write path and returning log data for in-memory queries on the read path.
distributor: responsible for handling incoming streams by clients. It’s the first step in the write path for log data.
query-frontend: optional service providing the querier’s API endpoints and can be used to accelerate the read path
querier: service handles queries using the LogQL query language, fetching logs from the ingesters and long-term storage.
ruler: responsible for continually evaluating a set of configurable queries and performing an action based on the result.
Then you can join them into different groups, and depending on the size of these groups, you have a different deployment topology, as shown below:
Loki Monolith Deployment Mode
Monolith: As you can imagine, all components are running together in a single instance. This is the simplest option and is recommended as a 100 GB / day starting point. You can even scale this deployment, but it will scale all components simultaneously, and it should have a shared object state.
Loki Simple Scalable Deployment Mode
Simple Scalable Deployment Model: This is the second level, and it can scale up at several TB of logs per day. It consists of splitting the components into two different profiles: read and write.
Loki Microservice Deployment Mode
Microservices: That means that each component will be managed independently, giving you all the power at your hand to scale each of these components alone.
Defining the deployment model of each instance is very easy, and it is based on a single parameter named target. So depending on the value of the target it will follow one of the previous deployment models:
all (default): It will deploy as in monolith mode.
write: It will be the write path on the simple scalable deployment model
read: It will be the reading group on the simple, scalable deployment model
ingester, distributor, query-frontend, query-scheduler, querier, index-gateway, ruler, compactor: Individual values to deploy a single component for the microservice deployment model.
The target argument will help for an on-premises kind of deployment. Still, if you are using Helm for the installation, Loki already provides different helm charts for the other deployment models:
But all those helm charts are based on the same principle commented above on defining the role of each instance using the argument target, as you can see in the picture below:
Put some brain when a route is not working as expected, or your consumers are not able to reach the service
We all know that Openshift is an outstanding Kubernetes Distribution and one of the most used mainly when talking about private-cloud deployments. Based on the solid reputation of Red Hat Enterprise Linux, Openshift was able to create a solid product that is becoming almost a standard for most enterprises.
It provides a lot of extensions from the Vanilla Kubernetes style, including some of the open-source industry standards such as Prometheus, Thanos, and Grafana for Metrics Monitoring or ELK stack for Logging Aggregation but also including its extensions such as the Openshift Routes.
Openshift Routes was the initial solution before the Ingress concept was a reality inside the standard. Now, it also implements following that pattern to keep it compatible. It is backed by HAProxy, one of the most known reverse-proxy available in the open-source community.
One of the tricky parts by default is knowing how to debug when one of your routes is not working as expected. The way you create routes is so easy that anyone can make it in a few clicks, and if everything works as expected, that’s awesome.
But if it doesn’t, the problems start because, by default, you don’t get any logging about what’s happening. But that’s what we are going to solve here.
First, we will talk a little more about how this is configured. Currently (Openshift 4.8 version), this is implemented, as I said, using HAProxy by default so if you are using other technology as ingresses such as Istio or Nginx, this article is not for you (but don’t forget to leave a comment if a similar kind of article would be of your interest so I can also bring it to the back-log 🙂 )
From the implementation perspective, this is implemented using the Operator Framework, so the ingress is deployed as an Operator, and it is available in the openshift-ingress-operator namespace.
ingress-operator pods on Openshift ecosystem
So, as this is an operator, several Custom Resources Definition (CRD) have been installed to work with this, one of the most interesting of this article. This CRD is Ingress Controllers.
Ingress instances on Openshift Ecosystem
By default, you will only see one instance named default. This is the one that includes the configuration of the ingress that is being deployed, so we need to add here an additional configuration to have also the logs.
Ingress controller YAML file
The snippet that we need to that is the one shown below under the spec parameter that starts the definition of the specification of the IngressController itself:
This will make another container deployed on the router pods in the openshift-ingressnamespace following the sidecar pattern named logs.
Router pods on Openshift Installation
This container will print the logs from the requests reaching the ingress component, so next time your consumer is not able to call your service, you will be able to see the incoming requests with all their metadata and know at least what is doing wrong:
Openshift Route Access Logs
As you can see, simple and easy!! If you don’t need it anymore, you can again remove the configuration and save it, and the new version will be rolled out and go back to normal.
We all know how cloud-native workloads and Kubernetes have changed how we do things. There are a lot of benefits that come with the effect of containerization and orchestration platforms such as Kubernetes, and we have discussed a lot about it: scalability, self-healing, auto-discovery, resilience, and so on.
But some challenges have been raised, most of them on the operational aspect that we have a lot of projects focused on tackling, but usually, we forget about what the ambassador has defined as the “inner dev cycle.”
The “inner dev cycle” is the productive workflow that each developer follows when working on a new application, service, or component. This iterative flow is where we code, test what we’ve coded, and fix what is not working or improve what we already have.
This flow has existed since the beginning of time; it doesn’t matter if you were coding in C using STD Library or COBOL in the early 1980 or doing nodejs with the latest frameworks and libraries at your disposal.
We have seen movements towards making this inner cycle more effective, especially in front-end development. We have many options to see the last change we have done in code, just saving the file. But for the first time when the movement to a container-based platform, this flow makes devs less productive.
The main reason is that the number of tasks a dev needs to do has increased. Imagine this set of steps that we need to perform:
Build the app
Build the container image
Deploy the container image in Kubernetes
These actions are not as fast as testing your changes locally, making devs less productive than before, which is what the “telepresence” project is trying to solve.
Telepresence is an incubator project from the CNCF that has recently focused a lot of attention because it has included OOTB in the latest releases of the Docker Desktop component. Based on its own words, this is the definition of the telepresence project:
Telepresence is an open-source tool that lets developers code and test microservices locally against a remote Kubernetes cluster. Telepresence facilitates more efficient development workflows while relieving the need to worry about other service dependencies.
Ok, so let’s see how we can start? Let’s dive in together. The first thing we need to do is to install telepresence in our Kubernetes cluster:
Install client | Telepresence
Install the Telepresence client on your workstation by running the commands below for your OS.
Note: It is also a way to install telepresence using Helm in your cluster following these steps:
Now I will create a simple container that will host a Golang application that exposes a simple REST service and make it more accessible; I will follow the tutorial that is available below; you can do it as well.
Tutorial: Developing a RESTful API with Go and Gin – The Go Programming Language
This tutorial introduces the basics of writing a RESTful web service API with Go and the Gin Web Framework (Gin).
Once we have our golang application ready, we are going to generate the container from it, using the following Dockerfile:
FROM golang:latest
RUN apt-get update
RUN apt-get upgrade -y
ENV GOBIN /go/bin
WORKDIR /app
COPY *.go ./
RUN go env -w GO111MODULE=off
RUN go get .
RUN go build -o /go-rest
EXPOSE 8080
CMD [ "/go-rest" ]
Then once we have the app, we’re going to upload to the Kubernetes server and run it as a deployment, as you can see in the picture below:
Once we have that, it is the moment to start executing the telepresence, and we will start connecting to the cluster using the following command telepresence connect, and it will show an output like this one:
Then we are going to list the endpoints available to intercept with the command telepresence listand we will see our rest-service that we have exposed before:
Now, we will run the specific interceptor, but before that, we’re going to do the trick so we can connect it to our Visual Studio Code. We will generate a launch.json file in Visual Studio Code with the following content:
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Launch with env file",
"type": "go",
"request": "launch",
"mode": "debug",
"program": "1",
"envFile": "NULL/go-debug.env"
}
]
}
The interesting part here is the envFile argument that points to a non-existent file go-debug.env on the same folder, so we need to make sure that we generate that file when we do the interception. So we will use the following command:
Discover the differences between two of the most used CRDs from Prometheus Operator and how to use each of them.
ServiceMonitor and PodMonitor are terms that you will start to see more often when talking about using Prometheus. We have covered a lot about Prometheus in the past articles. It is one of the primary references when we talk about monitoring in a cloud-native environment and is specially focused on the Kubernetes ecosystem.
Prometheus has a new deployment model under the Kubernetes Operator Framework in recent times. That has generated several changes in terms of resources and how we configure several aspects of the monitoring of our workloads. Some of these concepts are now managed as Customer Resource Definition (CRD) that are included to simplify the system’s configuration and be more aligned with the capabilities of the Kubernetes platform itself. This is great but, at the same time, changes how we need to use this excellent monitoring tool for cloud-native workloads.
Today, we will cover two of these new CRDs, one of the most relevant ones: ServiceMonitor and PodMonitor. These are the new objects that specify the resources that will be under monitoring scope to the platform, and each of them covers a different type of object, as you can imagine: Services and Pods.
Each of them has its definition file with its particular fields and metadata, and to highlight them, I will present a sample for each of them below:
As you can see, the definitions of the components are very similar and very intuitive, focusing on the selector to detect which pods or services we should monitor and some data regarding the specific target of the monitoring, so Prometheus knows how to scrape them.
If you want to take a look more in detail at any option you can configure on this CRD, I would recommend you to take a look at this URL which includes a detailed field to field documentation of the most common CRDs:
PodMonitor.monitoring.coreos.com/v1
Automatic documentation for your CustomResourceDefinitions.
These components will belong to the definition of your workloads, which means that the creation and maintenance of these objects will be from the application’s developers.
That is great because several reasons:
It will include the Monitoring aspect of the component itself, so you will never forget the add the configuration from a specific component. That means it can be included in the duplicate YAML files or Helm Chart or a Kustomize resources as another needed resource.
It will de-centralize the monitoring configuration making it more agile, and it will progress as the software components do it.
It will reduce the impact on other monitored components as there is no need to act in any standard file or resource, so any different workloads will continue to work as expected.
Both objects are very similar in their purposes as both of them scrape all the endpoints that match the selector that we added. So, in which cases should I use one or the other?
The answer will be straightforward. By default, you will go with a ServiceMonitor because it will provide the metrics from the service itself and each of the endpoints that the service has, so each of the pods that are implementing the service will be discovered and scraped as part of this action.
So, in which cases should I use PodMonitor? Where the workload you are trying to monitor doesn’t act behind a service, so as there is no service defined, you cannot use ServiceMonitor. Do you want some examples of those? Let’s bring some!
Services that interact using other protocols that are not HTTP-based, such as Kafka, SQS/SNS, JMS, or similar ones.
Components such as CronJobs, DaemonSets, or non exposing any incoming connection model.
So I hope this article will help you understand the main difference between those objects and go a little deeper into how the new Prometheus Operator Framework resources work. We will continue covering other aspects in upcoming posts.