Kubernetes Archives - Page 9 Of 9

EKS Fargate: Hybrid Kubernetes AWS cluster

2022-10-072020-10-18 by Alexandre Vazquez

EKS Fargate AWS Kubernetes Cluster: Learn how to create a Kubernetes cluster that can use also all the power of serverless computing using AWS Fargate

We know that there are several movements and paradigms that are pushing us hard to change our architectures trying to leverage much more managed services and taking care of the operational level so we can focus on what’s really important for our own business: creating applications and deliver value through them.

AWS from Amazon has been a critical partner during that journey, especially in the container world. With the release of EKS some time ago were able to provide a managed Kubernetes service that everyone can use, but also introducing the CaaS solution Fargate also gives us the power to run a container workload in a serverless fashion, without needing to worry about anything else.

But you could be thinking about if those services can work together. And the short answer is yes. But even more important than that we’re seeing that also they can work in a mixed mode:

So you can have an EKS cluster that has some nodes that are Fargate services and some nodes that are normal EC2 machines for workloads that are working in a state-full fashion or fits better in a traditional EC2 approach. And everything works by the same rules and is managed by the same EKS Cluster.

So, that sounds amazing but, How we can do that? Let’s see.

eksctl

To get to that point there is a tool that we need to introduce first, and that tool is named eksctl and it is a command-line utility that helps us to do any action to interact with the EKS service and simplifies a lot the work to do and also be able to automate most of the tasks in a non-human required mode. So, the first thing we need to is to get eksctl in our platforms ready. Let’s see how we can get that.

We have here all the detailed from Amazon itself about how to install eksctl in different platforms, no matter if you’re using Windows, Linux, or MacOS X:

Installing or updating eksctl – Amazon EKS

Learn how to install or update the eksctl command line tool. This tool is used to create and work with an Amazon EKS cluster.

After doing that we can check that we have installed the eksctl software running the command:

eksctl version

And we should get an output similar to this one:

So after doing that we can see that we have access to all the power behind the EKS service just typing these simple commands into our console window.

Creating the EKS Hybrid Cluster

Now, we’re going to create a mixed environment with some EC2 machines and enable the Fargate support for EKS. To do that, we will start with the following command:

eksctl create cluster --version=1.15 --name=cluster-test-hybrid --region=eu-west-1 --max-pods-per-node=1000 --fargate
[ℹ]  eksctl version 0.26.0
[ℹ]  using region eu-west-1
[ℹ]  setting availability zones to [eu-west-1c eu-west-1a eu-west-1b]
[ℹ]  subnets for eu-west-1c - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for eu-west-1a - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for eu-west-1b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  using Kubernetes version 1.15
[ℹ]  creating EKS cluster "cluster-test-hybrid" in "eu-west-1" region with Fargate profile
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=eu-west-1 --cluster=cluster-test-hybrid'
[ℹ]  CloudWatch logging will not be enabled for cluster "cluster-test-hybrid" in "eu-west-1"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=eu-west-1 --cluster=cluster-test-hybrid'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "cluster-test-hybrid" in "eu-west-1"
[ℹ]  2 sequential tasks: { create cluster control plane "cluster-test-hybrid", create fargate profiles }
[ℹ]  building cluster stack "eksctl-cluster-test-hybrid-cluster"
[ℹ]  deploying stack "eksctl-cluster-test-hybrid-cluster"
[ℹ]  creating Fargate profile "fp-default" on EKS cluster "cluster-test-hybrid"
[ℹ]  created Fargate profile "fp-default" on EKS cluster "cluster-test-hybrid"
[ℹ]  "coredns" is now schedulable onto Fargate
[ℹ]  "coredns" is now scheduled onto Fargate
[ℹ]  "coredns" pods are now scheduled onto Fargate
[ℹ]  waiting for the control plane availability...
[✔]  saved kubeconfig as "C:\\Users\\avazquez/.kube/config"
[ℹ]  no tasks
[✔]  all EKS cluster resources for "cluster-test-hybrid" have been created
[ℹ]  kubectl command should work with "C:\\Users\\avazquez/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "cluster-test-hybrid" in "eu-west-1" region is ready

This command will setup the EKS cluster enabling the Fargate support.

NOTE: The first thing that we should notice is that the Fargate support for EKS is not yet available in all the AWS regions. So, depending on the region that you’re using you could get an error. At this moment this is just enabled in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo) based on the information from AWS Announcements: https://aws.amazon.com/about-aws/whats-new/2020/04/eks-adds-fargate-support-in-frankfurt-oregon-singapore-and-sydney-aws-regions/

So, now, we should add to that cluster a Node Group. a Node Group is a set of EC2 instances that are going to be managed as part of it. And to do that we will use the following command:

eksctl create nodegroup --cluster cluster-test-hybrid --managed
[ℹ]  eksctl version 0.26.0
[ℹ]  using region eu-west-1
[ℹ]  will use version 1.15 for new nodegroup(s) based on control plane version
[ℹ]  nodegroup "ng-1262d9c0" present in the given config, but missing in the cluster
[ℹ]  1 nodegroup (ng-1262d9c0) was included (based on the include/exclude rules)
[ℹ]  will create a CloudFormation stack for each of 1 managed nodegroups in cluster "cluster-test-hybrid"
[ℹ]  2 sequential tasks: { fix cluster compatibility, 1 task: { 1 task: { create managed nodegroup "ng-1262d9c0" } } }
[ℹ]  checking cluster stack for missing resources
[ℹ]  cluster stack has all required resources
[ℹ]  building managed nodegroup stack "eksctl-cluster-test-hybrid-nodegroup-ng-1262d9c0"
[ℹ]  deploying stack "eksctl-cluster-test-hybrid-nodegroup-ng-1262d9c0"
[ℹ]  no tasks
[✔]  created 0 nodegroup(s) in cluster "cluster-test-hybrid"
[ℹ]  nodegroup "ng-1262d9c0" has 2 node(s)
[ℹ]  node "ip-192-168-69-215.eu-west-1.compute.internal" is ready
[ℹ]  node "ip-192-168-9-111.eu-west-1.compute.internal" is ready
[ℹ]  waiting for at least 2 node(s) to become ready in "ng-1262d9c0"
[ℹ]  nodegroup "ng-1262d9c0" has 2 node(s)
[ℹ]  node "ip-192-168-69-215.eu-west-1.compute.internal" is ready
[ℹ]  node "ip-192-168-9-111.eu-west-1.compute.internal" is ready
[✔]  created 1 managed nodegroup(s) in cluster "cluster-test-hybrid"
[ℹ]  checking security group configuration for all nodegroups
[ℹ]  all nodegroups have up-to-date configuration

So now we should be able to use kubectl to manage this new cluster. If you don’t have installed kubectl or you haven’t heard about it. This is the command line tool that allow us to manage your Kubernetes Cluster and you can install it based on the documentation shown here:

Installing or updating eksctl – Amazon EKS

Learn how to install or update the eksctl command line tool. This tool is used to create and work with an Amazon EKS cluster.

So, now, we should start taking a look at the infrastructure that we have. So if we type the following command to see the nodes at our disposal:

kubectl get nodes

We see an output similar to this:

NAME                                                    STATUS   ROLES    AGE   VERSION
fargate-ip-192-168-102-22.eu-west-1.compute.internal    Ready    <none>   10m   v1.15.10-eks-094994
fargate-ip-192-168-112-125.eu-west-1.compute.internal   Ready    <none>   10m   v1.15.10-eks-094994
ip-192-168-69-215.eu-west-1.compute.internal            Ready    <none>   85s   v1.15.11-eks-bf8eea
ip-192-168-9-111.eu-west-1.compute.internal             Ready    <none>   87s   v1.15.11-eks-bf8eea

As you can see we have 4 “nodes” two that start with the fargate name that are fargate nodes and two that just start with ip-… and those are the traditional EC2 instances. And after that moment that’s it, we have our mixed environment ready to use.

We can check the same cluster using the AWS EKS page to see that configuration more in detail. If we enter in the EKS page for this cluster we see in the Compute tab the following information:

We see under Node Groups the data around the EC2 machines that are managed as part of this cluster and as you can see we saw 2 as the Desired Capacity and that’s why we have 2 EC2 instances in our cluster. And regarding the Fargate profile, we see the namespaces set to default and kube-system and that means that all the deployments to those namespaces are going to be deployed using Fargate Tasks.

Summary

In the following articles in these series, we will see how to progress on our Hybrid cluster, deploy workloads scale it based on the demand that we’re getting, enabling integration with other services like AWS CloudWatch, and so on. So, stay tuned, and don’t forget to follow my articles to not miss any new updates as soon as it’s available to you!

Managed Container Platform: Top 3 Reasons why you should go it

2022-09-142020-06-15 by Alexandre Vazquez

Managed Container Platform provides advantages to any system inside any company. Take a look at the three critical ones.

Table Of Contents

Managed Container Platform provides advantages to any system inside any company. Take a look at the three critical ones.
1. You will get all components isolated and updated more quickly
2.- You will optimize the use of your infrastructure
3.- You will not need infrastructure for administration
Wrap up and next steps

Managed Container Platform is disrupting everything. We’re living in a time where development and the IT landscape are changing, new paradigms like microservices and containers seem to be out there for the last few years, and if we trust the reality that the blog posts and the articles show today, we’re all of the users already using them all the time.

Did you see any blog posts about how to develop a J2EE application running on your Tomcat server on-prem? Probably not. The most similar article should probably be how to containerize your tomcat-based application.

But do you know what? Most companies still are working that way. So even if all companies have a new digital approach in some departments, they also have other ones being more traditional.

So, that seems that we need to find a different way to translate the main advantages of a container-based platform to a kind of speech they can see and realize the tangible benefits they can get from there and have the “Hey, this can work for me!” kind of spirit.

1. You will get all components isolated and updated more quickly

That’s one of the great things about container-based platforms compared with previous approaches like application server-based platforms. When you have an application server cluster, you still have one cluster with several applications. So you usually do some isolation, keep related applications, provide independent infrastructure for the critical ones, and so on.

But even with that, at some level, the application continues to be coupled, so some issues with some applications could bring down another one that was not expected for business reasons.

With a container-based platform, you’re getting each application in its bubble, so any issue or error will affect that application and nothing more. Platform stability is a priority for all companies and all departments inside them. Just ask yourself: Do you want to end with those “domino’s chains” of failure? How much will your operations improve? How much will your happiness increase?

Additionally, based on the container approach, you will get smaller components. Each of them will do a single task providing a single capability to your business, which means that it will be much easier to update, test, and deploy in production. So that, in the end will generate more deployments into the production environment and reduce the time to market for your business capabilities.

You will be able to deploy faster and have more stable operations simultaneously.

2.- You will optimize the use of your infrastructure

Costs, everything is about costs. There are no single conversations with customers who are not trying to pay less for their infrastructure. So, let’s face it. We should be able to run operations in an optimized way. So, if our infrastructure cost is going higher, that needs to mean that our business increases.

Container-based platforms will allow optimizing infrastructure in two different ways. First, if using two main concepts: Elasticity and Infrastructure Sharing.

Elasticity is related because I’m only going to have the infrastructure I need to support the load I have at this moment. So, if the load increases, my infrastructure will increase to handle it, but after that moment goes away, it will go back to what it needs now after that peak happened.

Infrastructure sharing is about using each server’s part that is free to deploy other applications. Imagine a traditional approach where I have two servers for my set of applications. Probably I don’t have 100% usage of those servers because I need to have some spare computer to be able to act when the load increases. I probably have 60–70% percent. That means 30% free. If we have different departments with different systems, and each has its infrastructure 30% free, how much of our infrastructure are we just throwing away? How many dollars/euros/pounds are you just throwing off the window?

Container-based platforms don’t need specific tools or software installed on the platform to run a different kind of application. It is not required because everything resides inside the container, so I can use any free space to deploy other applications doing a more efficient usage of those.

3.- You will not need infrastructure for administration

Each system that is big enough has some resources dedicated to being able to manage it. However, even most of the recommended architectures recommend placing those components isolated from your runtime components to avoid any issue regarding administrator or maintenance that can affect your runtime workloads, which means specific infrastructure that you’re using for something that isn’t helping your business. Of course, you can explain to any business user that you need a machine to run that provides the capabilities required. But it is more complex than using additional infrastructure (and generating cost) to place other components that are not helping the business.

So, managed container platforms take that problem away because you’re going to provide the infrastructure you need to run your workloads, and you’re going to be given for free or such low fee the administration capabilities. And addition to that, you don’t even need to worry that administration features are always available and working fine because this is leverage to the provider itself.

Wrap up and next steps

As you can see, we describe very tangible benefits that are not industry-based or development focus. Of course, we can have so many more to add to this list, but these are the critical ones that affect any company in any industry worldwide. So, please, take your time to think about how these capabilities can help to improve your business. But not only that, take your time to quantify how that will enhance your business. How much can you save? How much can you get from this approach?

And when you have in front of you a solid business case based on this approach, you will get all the support and courage you need to move forward during that route!! So I wish you a peaceful transition!

Harbor Registry: How to use to increase security on your platform?

2022-09-292020-06-11 by Alexandre Vazquez

Learn how you can include Harbor registry in your DevSecOps toolset to increase the security and management on your container-based platform

With the transition to a more agile development process where the number of deployments has been increased in an exponential way. That situation has made it quite complex to keep pace to make sure we’re not just deploying code more often into production that provides the capabilities that are required by the business. But, also, at the same time, we’re able to do it securely and safely.

That need is leading toward the DevSecOps idea to include security as part of the DevOps culture and practices as a way to ensure safety from the beginning on development and across all the standard steps from the developer machine to the production environment.

Additional to that, because of the container paradigm we have a more polyglot approach with different kinds of components running on our platform using a different base image, packages, libraries, and so on. We need to make sure they’re still secure to use and we need tools to be able to govern that in a natural way. To help us on that duty is where components like Harbor help us to do that.

Harbor is a CNCF project at the incubator stage at the moment of writing this article, and it provides several capabilities regarding how to manage container images from a project perspective. It gives a project approach with its docker registry and also a chart museum if we’d like to use Helm Charts as part of our project development. But it includes security features too, and that’s the one that we’re going to cover in this article:

Vulnerabilities Scan: it allows you to scan all the docker images registered in the different repositories to check if they have vulnerabilities. It also provides automation during that process to make sure that every time we push a new image, this is scanned automatically. Also, it will enable defining policies to avoid pulling any image with vulnerabilities and also set the level of vulnerabilities (low, medium, high, or critical) that we’d like to tolerate it. By default, it comes with Clair as the default scanner, but you can introduce others as well.
Signed images: Harbor registry provides options to deploy notary as part of its components to be able to sign images during the push process to make sure that no modifications are done to that image
Tag Inmuttability and Retention Rules: Harbor registry also provides the option to define tag immutability and retention rules to make sure that we don’t have any attempt to replace images with others using the same tag.

Harbor registry is based on docker so you can run it locally using docker and docker-compose using the procedure that is available on its official web page. But it also supports being installed on top of your Kubernetes platform using the helm chart and operator that is available.

Once the tool is installed, we have access to the UI Web Portal, and we’re able to create a project that has repositories as part of it.

As part of the project configuration, we can define the security policies that we’d like to provide to each project. That means that different projects can have different security profiles.

And once we push a new image to the repository that belongs to that project we’re going to see the following details:

In this case, I’ve pushed a TIBCO BusinessWorks Container Edition application that doesn’t contain any vulnerability and just shows that and also where this was checked.

Also, if we see the details, we can check additional information like if the image has been signed or not, or be able to check it again.

Summary

So, this is just a few features that Harbor provides from the security perspective. But Harbor is much more than only that so probably we cover more of its features in further articles I hope based on what you read today you’d like to give it a chance and start introducing it in your DevSecOps toolset.

Prometheus Metrics: How to change the Metric Name?

2022-09-152020-05-29 by Alexandre Vazquez

Find a way to re-define and re-organize the name of your Prometheus metrics to meet your requirements

Prometheus Metrics: How to change the names of a metric? — Photo by Chris Ried on Unsplash

Prometheus has become the new standard when we’re talking about monitoring our new modern application architecture, and we need to make sure we know all about its options to make sure we can get the best out of it. I’ve been using it for some time until I realized about a feature that I was desperate to know how to do, but I couldn’t find anywhere clearly define. So as I didn’t found it easily, I thought about writing a small article to show you how to do it without needed to spend the same time as I did.

We have plenty of information about how to configure Prometheus and use some of the usual configuration plugins, as we can see on its official webpage [1]. Even I already write about some configuration and using it for several purposes, as you can see also in other posts [2][3][4].

One of these configuration plugins is about relabeling, and this is a great thing. We have that each of the exporters can have its labels and meaning for those, and when you try to manage different technologies or components makes complex that all of them match together even if all of them follow the naming convention that Prometheus has [5].

But I had this situation, and I’m sure you have gone or will go towards that as well, that I have similar metrics for different technologies that for me are the same, and I need to keep them with the same name, but as they belong to other technologies they are not. So I need to find a way to rename the metric, and the great thing is that you can do that.

To do that, you just need to do a metric_relabel configuration. This configuration applies to relabel (as the name already indicates) labels of your prometheus metrics in this case before being ingested but also allow us to use some notable terms to do different things, and one of these notable terms is __name__. __name__ is a particular label that will enable you to rename your prometheus metrics before being ingested in the Prometheus Timeseries Database. And after that point, this will be as it will have that name since the beginning.

How to use that is relatively easy, is as any other relabel process, and I’d like to show you a sample about how to do it.

- source_labels: [__name__]

regex:  'jvm_threads_current'

target_label: __name__

replacement: 'process_thread_count'

Here it is a simple sample to show how we can rename a metric name jvm_threads_current to count the threads inside the JVM machine to do it more generic to be able to include the threads for the process in a process_thread_count prometheus metrics that we can use now as it was the original name.

References

[1] Prometheus: Configuration https://prometheus.io/docs/prometheus/latest/configuration/configuration/

[2] https://medium.com/@alexandrev/prometheus-monitoring-in-tibco-cloud-integration-96a6811416ce

[3] https://medium.com/@alexandrev/prometheus-monitoring-for-microservices-using-tibco-772018d093c4

[4] https://medium.com/@alexandrev/kubernetes-service-discovery-for-prometheus-fcab74237db6

[5] Prometheus: Metric and Label Naming https://prometheus.io/docs/practices/naming/

Kubernetes Batch Processing using TIBCO BW

2022-09-152020-04-19 by Alexandre Vazquez

Kubernetes Batch Processing using TIBCO BusinessWorks — Photo by Lukas Blazek on Unsplash

Table Of Contents

Batch Managed by TIBCO
Cron-Job Kubernetes API
Pros and Cons of each approach

We all know that in the rise of the cloud-native development and architectures, we’ve seen Kubernetes based platforms as the new standard all focusing on new developments following the new paradigms and best practices: Microservices, Event-Driven Architectures new shiny protocols like GraphQL or gRPC, and so on and so forth.

But we should be aware that most of the enterprises that are today adopting these technologies are not green-field opportunities. They have a lot of systems already running that all new developments in the new cloud-native architectures need to interact so the rules become more complex and there are a lot of shades of gray when we’re creating our Kubernetes application.

And based on my experience when we are talking with the customer is the new cloud architecture most of them bring the batch pattern here. They know this is not a best practice anymore, that they cannot continue to build their systems in a batch fashion trying to do by night what were should be doing in real-time. But, most of the time they need to co-exist with the systems they already have and could be needed this kind of procedure. So we need to find a way to provide Kubernetes Batch Processing.

Also for customers that already have their existing applications using that pattern and they like to move to the new world is better for them that they can do it without needing to re-architect their existing application. Something that probably they will end up doing but at their own pace.

Batch processing patterns in TIBCO development is a quite used pattern and we have customers all over the world with this kind of development used in production for many years. You know, this kind of process that is executed at a specific moment in time (weekly on Monday, each day at 3 PM .. or just each hour to do some regular job).

It’s a straightforward pattern here. Simply add a timer that you can configure when it will be launched and add your logic. That’s it. As simple as that:

But, how this can be translated into a Kubernetes approach? How we can create applications that work with this pattern and still be managed by our platform? So, fortunately, this is something that can be done, and you have also different ways of doing it depending on what you want to achieve.

Mainly today we are going to describe two ways to be able to do that and try to explain the main differences between one and the other so you can know which one you should use depending on your use case. Those two methods I’m going to call it this way: Batch managed by TIBCO Kubernetes and Cron Job API approach

Batch Managed by TIBCO

This is the simplest one. It is exactly the same approach you have in your existing on-premises BusinessWorks application just deployed into the Kubernetes cluster.

So, you don’t need to change anything in your logic you only will have an application that is started by a Timer with their configuration regarding the scheduling inside the scope of the BusinessWorks application and that’s it.

You only need to provide the artifacts needed to deploy your application into Kubernetes like any other TIBCO BusinessWorks app and that’s it. That means that you’re creating a Deployment to launch this component and you will always have a running pod evaluating when the condition is true to launch the process as you have in your on-premises approach.

Cron-Job Kubernetes API

The batch processing pattern is something that is already covered out of the box by the Kubernetes API and that’s why we have the Cron-Job concept. You probably remind the Cron Jobs that we have in our Linux machines. If you’re a developer or a system administrator I’m sure you’ve played with these cronjobs to schedule tasks or commands to be executed at a certain time. If you’re a Windows-based person, this is the Linux equivalent of your Windows Task Scheduler Jobs. Same approach.

And this is very simple, you only need to create a new artifact using this Cron Job API that mainly says the job to execute and the schedule of that job similar of what we’ve done in the past with the cronjob in our Linux machine:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

The jobs that we can use here should be compliant with a single rule that also applied in the Unix cron jobs as well: The command should end when the job is done.

And this is something that is critical, which means that our container should exit when the job is done to be able to be used inside this approach. That means that we cannot use a “server approach” as we’re doing in the previous approach because in that case, the pod is never-ending. Does that mean that I cannot use a TIBCO BusinessWorks application as part of a Kubernetes Cron Job? Absolutely Not! Let’s see how you can do it.

So, we should focus on two things, the business logic should be able to run as soon as the process start and the container should end as soon as the job is done. Let’s start with the first one.

The first one is easy. Let’s use our previous sample: Simple batch approach that writes a log trace each minute. We need to make it start as soon as the process started but this is pretty Out of the box (OOTB) with TIBCO BusinessWorks we only need to configure that timer to run only once and that’s going to start as soon as the application is running:

So, we already have the first requirement fixed, let’s see with the other one.

We should be able to end the container completely as the process ends and that’s challenging because BusinessWorks application doesn’t behave that way. They’re supposed to be always running. But this also can be sorted.

The only thing that we need is to make use of command at the end of the flow. It’s like an exit command in our shell script or Java code to end the process and the container. To do that, we should add an “Execute External Command” Activity and simply configure it to send a signal to the BusinessWorks process running inside the container.

The signal that we’re going to send is SIGINT that is the similar one that we send when we press CTRL+C in a Terminal to require the process to stop. We’re are going to do the same. To do that we’re going to use the command kill that is shipped in all Unix machines and most of the docker base images as well. And the command kill requires two arguments:

kill -s <SIGNAL_NAME> <PID>

SIGNAL_NAME: We already cover that part and we’re going to use the Singal named SIGINT.
PID: Regarding the PID we need to provide the PID of the BusinessWorks process running inside container.

The part of finding the PID for that process running BusinessWorks inside the container can be difficult to locate but if you see the running processes inside a BusinessWorks container you can see that this is not so complicated to find:

If we enter inside a running container of a BusinessWorks application we will check that this PID is always the number 1. So we already have everything ready to configure the activity as it is shown below:

And that’s it with these two changes we are able to deploy this using the CronJob API and use it as part of your toolset of application patterns.

Pros and Cons of each approach

As you can imagine there is not a single solution about when to use one of the other because it is going to depend on your use case and your requirements. So I’ll try to enumerate here the main differences between both approaches so you can choose better when it is on your hand:

TIBCO Managed approach is faster because the pod is already running and as soon as the condition is met the logic started. Using the Cron Job API requires a warm-up period because the pod starts when the condition is met so some delay can be applied here.
TIBCO Managed approach requires the pod to be running all the time so it uses more resources when the condition is not reached. So in case you’re running stateless containers like AWS Fargate or similar Cron JOB API is a better fit.
CronJob API is a standard Kubernetes API that means that integrated completely with the ecosystem, rather TIBCO Managed is managed by TIBCO using application configuration settings.
TIBCO Managed approach is not aware of other instances of the same application running so you managed to keep a single instance running to avoid running the same logic many times. In the case of the Cron Job API, this is managed by the Kubernetes platform itself.

Kubernetes Service Discovery for Prometheus

2022-05-092019-08-05 by Alexandre Vazquez

In previous posts, we described how to set up Prometheus to work with your TIBCO BusinessWorks Container Edition apps, and you can read more about it here.

[visual-link-preview encoded=”eyJ0eXBlIjoiaW50ZXJuYWwiLCJwb3N0IjoxMTEsInBvc3RfbGFiZWwiOiJQb3N0IDExMSAtIFByb21ldGhldXMgTW9uaXRvcmluZyBmb3IgTWljcm9zZXJ2aWNlcyB1c2luZyBUSUJDTyIsInVybCI6IiIsImltYWdlX2lkIjoyNjI4LCJpbWFnZV91cmwiOiJodHRwOi8vYWxleGFuZHJlLXZhenF1ZXouY29tL3dwLWNvbnRlbnQvdXBsb2Fkcy8yMDIyLzAxLzEtNmIyS25hQWNtME9TSVpySFJwcFRnLnBuZyIsInRpdGxlIjoiUHJvbWV0aGV1cyBNb25pdG9yaW5nIGZvciBNaWNyb3NlcnZpY2VzIHVzaW5nIFRJQkNPIiwic3VtbWFyeSI6Ildl4oCZcmUgbGl2aW5nIGEgd29ybGQgd2l0aCBjb25zdGFudCBjaGFuZ2VzIGFuZCB0aGlzIGlzIGV2ZW4gbW9yZSB0cnVlIGluIHRoZSBFbnRlcnByaXNlIEFwcGxpY2F0aW9uIHdvcmxkLiBJ4oCZbGwgbm90IHNwZW5kIG11Y2ggdGltZSB0YWxraW5nIGFib3V0IHRoaW5ncyB5b3UgYWxyZWFkeSBrbm93LCBidXQganVzdCBzYXkgdGhhdCB0aGUgbWljcm9zZXJ2aWNlcyBhcmNoaXRlY3R1cmUgYXBwcm9hY2ggYW5kIHRoZSBQYWFTIHNvbHV0aW9ucyBoYXZlIGJlZW4gYSBnYW1lLWNoYW5nZXIgZm9yIGFsbCBlbnRlcnByaXNlIGludGVncmF0aW9uIHRlY2hub2xvZ2llcy4gVGhpcyB0aW1lIEnigJlkIGxpa2UgdG8gWyZoZWxsaXA7XSIsInRlbXBsYXRlIjoidXNlX2RlZmF1bHRfZnJvbV9zZXR0aW5ncyJ9″]

In that post, we described that there were several ways to update Prometheus about the services that ready to monitor. And we choose the most simple at that moment that was the static_config configuration which means:

Don’t worry Prometheus, I’ll let you know the IP you need to monitor and you don’t need to worry about anything else.

And this is useful for a quick test in a local environment when you want to test quickly your Prometheus set up or you want to work in the Grafana part to design the best possible dashboard to handle your need.

But, this is not too useful for a real production environment, even more, when we’re talking about a Kubernetes cluster when services are going up & down continuously over time. So, to solve this situation Prometheus allows us to define a different kind of ways to perform this “service discovery” approach. In the official documentation for Prometheus, we can read a lot about the different service discovery techniques but at a high level these are the main service discovery techniques available:

[visual-link-preview encoded=”eyJ0eXBlIjoiZXh0ZXJuYWwiLCJwb3N0IjowLCJwb3N0X2xhYmVsIjoiIiwidXJsIjoiaHR0cHM6Ly9wcm9tZXRoZXVzLmlvL2RvY3MvcHJvbWV0aGV1cy9sYXRlc3QvY29uZmlndXJhdGlvbi9jb25maWd1cmF0aW9uLyIsImltYWdlX2lkIjotMSwiaW1hZ2VfdXJsIjoiaHR0cHM6Ly9wcm9tZXRoZXVzLmlvL2Fzc2V0cy9mYXZpY29ucy9hbmRyb2lkLWNocm9tZS0xOTJ4MTkyLnBuZyIsInRpdGxlIjoiQ29uZmlndXJhdGlvbiB8IFByb21ldGhldXMiLCJzdW1tYXJ5IjoiQW4gb3Blbi1zb3VyY2UgbW9uaXRvcmluZyBzeXN0ZW0gd2l0aCBhIGRpbWVuc2lvbmFsIGRhdGEgbW9kZWwsIGZsZXhpYmxlIHF1ZXJ5IGxhbmd1YWdlLCBlZmZpY2llbnQgdGltZSBzZXJpZXMgZGF0YWJhc2UgYW5kIG1vZGVybiBhbGVydGluZyBhcHByb2FjaC4iLCJ0ZW1wbGF0ZSI6InVzZV9kZWZhdWx0X2Zyb21fc2V0dGluZ3MifQ==”]

azure_sd_configs: Azure Service Discovery
consul_sd_configs: Consul Service Discovery
dns_sd_configs: DNS Service Discovery
ec2_sd_configs: EC2 Service Discovery
openstack_sd_configs: OpenStack Service Discovery
file_sd_configs: File Service Discovery
gce_sd_configs: GCE Service Discovery
kubernetes_sd_configs: Kubernetes Service Discovery
marathon_sd_configs: Marathon Service Discovery
nerve_sd_configs: AirBnB’s Nerve Service Discovery
serverset_sd_configs: Zookeeper Serverset Service Discovery
triton_sd_configs: Triton Service Discovery
static_config: Static IP/DNS for the configuration. No Service Discovery.

And even, it all these options are not enough for you and need something more specific you have an API available to extend the Prometheus capabilities and create your own Service Discovery technique. You can find more info about it here:

[visual-link-preview encoded=”eyJ0eXBlIjoiZXh0ZXJuYWwiLCJwb3N0IjowLCJwb3N0X2xhYmVsIjoiIiwidXJsIjoiaHR0cHM6Ly9wcm9tZXRoZXVzLmlvL2Jsb2cvMjAxOC8wNy8wNS9pbXBsZW1lbnRpbmctY3VzdG9tLXNkLyIsImltYWdlX2lkIjotMSwiaW1hZ2VfdXJsIjoiaHR0cHM6Ly9wcm9tZXRoZXVzLmlvL2Fzc2V0cy9mYXZpY29ucy9hbmRyb2lkLWNocm9tZS0xOTJ4MTkyLnBuZyIsInRpdGxlIjoiSW1wbGVtZW50aW5nIEN1c3RvbSBTZXJ2aWNlIERpc2NvdmVyeSB8IFByb21ldGhldXMiLCJzdW1tYXJ5IjoiQW4gb3Blbi1zb3VyY2UgbW9uaXRvcmluZyBzeXN0ZW0gd2l0aCBhIGRpbWVuc2lvbmFsIGRhdGEgbW9kZWwsIGZsZXhpYmxlIHF1ZXJ5IGxhbmd1YWdlLCBlZmZpY2llbnQgdGltZSBzZXJpZXMgZGF0YWJhc2UgYW5kIG1vZGVybiBhbGVydGluZyBhcHByb2FjaC4iLCJ0ZW1wbGF0ZSI6InVzZV9kZWZhdWx0X2Zyb21fc2V0dGluZ3MifQ==”]

But this is not our case, for us, the Kubernetes Service Discovery is the right choice for our approach. So, we’re going to change the static configuration we had in the previous post:

- job_name: 'bwdockermonitoring'
  honor_labels: true
  static_configs:
    - targets: ['phenix-test-project-svc.default.svc.cluster.local:9095']
      labels:
        group: 'prod'

For this Kubernetes configuration

- job_name: 'bwce-metrics'
  scrape_interval: 5s
  metrics_path: /metrics/
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: (.*)
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: prom
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: 1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: $1
    action: replace

As you can see this is quite more complex than the previous configuration but it is not as complex as you can think at first glance, let’s review it by different parts.

- role: endpoints
    namespaces:
      names:
      - default

It says that we’re going to use role for endpoints that are created under the default namespace and we’re going to specify the changes we need to do to find the metrics endpoints for Prometheus.

scrape_interval: 5s
 metrics_path: /metrics/
 scheme: http

This says that we’re going to execute the scrape process in a 5 seconds interval, using http on the path /metrics/

And then, we have a relabel_config section:

- source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: (.*)
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: prom
    replacement: $1
    action: keep

That means that we’d like to keep that label for prometheus:

- source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: 1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: $1
    action: replace

That means that we want to do a replace of the label value and we can do several things:

Rename the label name using the target_label to set the name of the final label that we’re going to create based on the source_labels.
Replace the value using the regex parameter to define the regular expression for the original value and the replacement parameter that is going to express the changes that we want to do to this value.

So, now after applying this configuration when we deploy a new application in our Kubernetes cluster, like the project that we can see here:

Automatically we’re going to see an additional target on our job-name configuration “bwce-metrics”

Prometheus TIBCO Monitoring for Containers: Quick and Simple in 5 Minutes!

2022-09-152019-07-22 by Alexandre Vazquez

Prometheus is becoming the new standard for Kubernetes monitoring and today we are going to cover how we can do Prometheus TIBCO monitoring in Kubernetes.

We’re living in a world with constant changes and this is even more true in the Enterprise Application world. I’ll not spend much time talking about things you already know, but just say that the microservices architecture approach and the PaaS solutions have been a game-changer for all enterprise integration technologies.

This time I’d like to talk about monitoring and the integration capabilities we have of using Prometheus to monitor our microservices developed under TIBCO technology. I don’t like to spend too much time either talking about what Prometheus is, as you probably already know, but in a summary, this is an open-source distributed monitoring platform that has been the second project released by the Cloud Native Computing Foundation (after Kubernetes itself) and that has been established as a de-facto industry standard for monitoring K8S clusters (alongside with other options in the market like InfluxDB and so on).

Prometheus has a lot of great features, but one of them is that it has connectors for almost everything and that’s very important today because it is so complicated/unwanted/unusual to define a platform with a single product for the PaaS layer. So today, I want to show you how to monitor your TIBCO BusinessWorks Container Edition applications using Prometheus.

Most of the info I’m going to share is available in the bw-tooling GitHub repo, so you can get to there if you need to validate any specific statement.

bw-tooling/prometheus-integration at master · TIBCOSoftware/bw-tooling

Collection of tools designed to simplify deployment and management of TIBCO BusinessWorks applications – bw-tooling/prometheus-integration at master · TIBCOSoftware/bw-tooling

Ok, are we ready? Let’s start!!

I’m going to assume that we already have a Kubernetes cluster in place and Prometheus installed as well. So, the first step is to enhance the BusinessWorks Container Edition base image to include the Prometheus capabilities integration. To do that we need to go to the GitHub repo page and follow these instructions:

Download & unzip the prometheus-integration.zip folder.

Prometheus TIBCO Monitoring for Containers

Open TIBCO BusinessWorks Studio and point it to a new workspace.
Right-click in Project Explorer → Import… → select Plug-ins and Fragments → select Import from the directory radio button

Browse it to prometheus-integration folder (unzipped in step 1)

Now click Next → Select Prometheus plugin → click Add button → click Finish. This will import the plugin in the studio.

Now, to create JAR of this plugin so first, we need to make sure to update com.tibco.bw.prometheus.monitor with ‘.’ (dot) in Bundle-Classpath field as given below in META-INF/MANIFEST.MF file.

Right-click on Plugin → Export → Export…

Select type as JAR file click Next

Now Click Next → Next → select radio button to use existing MANIFEST.MF file and browse the manifest file

Click Finish. This will generate prometheus-integration.jar

Now, with the JAR already created what we need to do is include it in your own base image. To do that we place the JAR file in the <TIBCO_HOME>/bwce/2.4/docker/resources/addons/jar

And we launch the building image command again from the <TIBCO_HOME>/bwce/2.4/docker folder to update the image using the following command (use the version you’re using at the moment)

docker build -t bwce_base:2.4.4 .

So, now we have an image with Prometheus support! Great! We’re close to the finish, we just create an image for our Container Application, in my case, this is going to be a very simple echo service that you can see here.

And we only need to keep these things in particular when we deploy to our Kubernetes cluster:

We should set an environment variable with the BW_PROMETHEUS_ENABLE to “TRUE”
We should expose the port 9095 from the container to be used by Prometheus to integrate.

Now, we only need to provide this endpoint to the Prometheus scrapper system. There are several ways to do that, but we’re going to focus on the simple one.

We need to change the prometheus.yml to add the following job data:

- job_name: 'bwdockermonitoring'
  honor_labels: true
  static_configs:
    - targets: ['phenix-test-project-svc.default.svc.cluster.local:9095']
      labels:
        group: 'prod'

And after restarting Prometheus we have all the data indexed in the Prometheus database to be used for any dashboard system.

In this case, I’m going to use Grafana to do quick dashboard.

Each of these graph components is configured based on the metrics that are being scraped by Prometheus TIBCO exporter.

Kubernetes Probes for a TIBCO BusinessWorks Container Edition Application

2022-05-072019-05-20 by Alexandre Vazquez

Introduction

Probes are how we’re able to say to Kubernetes that everything inside the pod is working as expected. Kubernetes has no way to know what’s happening inside at the fine-grained and has no way to know for each container if it is healthy or not, that’s why they need help from the container itself.

Imagine that you’re Kubernetes controller and you have like eight different pods , one with Java batch application, another with some Redis instance, other with nodejs application, other with a Flogo microservice (Note: Haven’t you heard about Flogo yet? Take some minutes to know about one of the next new things you can use now to build your cloud-native applications) , another with a Oracle database, other with some jetty web server and finally another with a BusinessWorks Container Edition application. How can you tell that every single component is working fine?

First, you can think that you can do it with the entrypoint component of your Dockerfile as you only specify one command to run inside each container, so check if that process is running, and that means that everything is healthy? Ok… fair enough…

But, is this true always? A running process at the OS/container level means that everything is working fine? Let’s think about the Oracle database for a minute, imagine that you have an issue with the shared memory and it keeps in an initializing status forever, K8S is going to check the command, it is going to find that is running and says to the whole cluster: Ok! Don’t worry! Database is working perfectly, go ahead and send your queries to it!!

This could happen with similar components like a web server or even with an application itself, but it is too common when you have servers that can handle deployments on it, like BusinessWorks Container Edition itself. And that’ why this is very important for us as developers and even more important for us as administrators. So, let’s start!

The first thing we’re going to do is to build a BusinessWorks Container Edition Application, as this is not the main purpose of this article, we’re going to use the same ones I’ve created for the BusinessWorks Container Edition — Istio Integration that you could find here.

So, this is a quite simple application that exposes a SOAP Web Service. All applications in BusinessWorks Container Edition (as well as in BusinessWorks Enterprise Edition) has its own status, so you can ask them if they’re Running or not, that something the BusinessWorks Container internal “engine” (NOTE: We’re going to use the word engine to simplify when we’re talking about the internals of BWCE. In detail, the component that knows the status of the application is the internal AppNode the container starts, but let’s keep it simple for now)

Kubernetes Probes

In Kubernetes, exists the “probe” concept to perform health check to your container. This is performed by configuring liveness probes or readiness probes.

Liveness probe: Kubernetes uses liveness probes to know when to restart a Container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress.
Readiness probe: Kubernetes uses readiness probes to know when a Container is ready to start accepting traffic. A Pod is considered ready when all of its Containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balance

Even when there are two types of probes for BusinessWorks Container Edition both are handling the same way, the idea is the following one: As long as the application is Running, you can start sending traffic and when it is not running we need to restart the container, so that makes it simpler for us.

Implementing Probes

Each BusinessWorks Container Edition application that is started has an out of the box way to know if it is healthy or not. This is done by a special endpoint published by the engine itself:

http://localhost:7777/_ping/

So, if we have a normal BusinessWorks Container Edition application deployed on our Kubernetes cluster as we had for the Istio integration we have logs similar to these ones:

As you can see logs says that the application is started. So, as we can’t launch a curl request from the inside the container (as we haven’t exposed the port 7777 to the outside yet and curl is not installed in the base image), the first thing we’re going to do is to expose it to the rest of the cluster.

To do that we change our Deployment.yml file that we have used to this one:

Now, we can go to any container in the cluster that has “curl” installed or any other way to launch a request like this one with the HTTP 200 code and the message “Application is running”.

NOTE: If you forget the last / and try to invoke _ping instead of _ping/ you’re going to get an HTTP 302 Found code with the final location as you can see here:

Ok, let’s see what happens if now we stop the application. To do that we’re going to go inside the container and use the OSGi console.

To do that once you’re inside the container you execute the following command:

ssh -p 1122 equinox@localhost

It is going to ask for credentials and use the default password ‘equinox’. After that is going to give you the chance to create a new user and you can use whatever credentials work for you. In my example, I’m going to use admin / adminadmin (NOTE: Minimum length for a password is eight (8) characters.

And now, we’re in. And this allows us the option to execute several commands, as this is not the main topic for today I’m going to skip all the explanation but you can take a look at this link with all the info about this console.

If we execute frwk:la is going to show the applications deployed, in our case the only one, as it should be in BusinessWorks Container Edition application:

To stop it, we are going to execute the following command to list all the OSGi bundle we have at the moment running in the system:

frwk:lb

Now, we find the bundles that belong to our application (at least two bundles (1 per BW Module and another for the Application)

And now we can stop it using felix:stop <ID>, so in my case, I need to execute the following commands:

stop “603”

stop “604”

And now the application is stopped

So, if now we try to launch the same curl command as we executed before, we’re getting the following output:

As you can see an HTTP 500 Error which means something is not fine. If now we try to start again the application using the start bundle command (equivalent to the stop bundle command that we used before) for both bundles of the application, you are going to see that the application says is running again:

And the command has the HTTP 200 output as it should have and the message “Application us running”

So, now, after knowing how the _ping/ endpoint works we only need to add it to our deployment.yml file from Kubernetes. So we modified again our deployment file to be something like this:

NOTE: It’s quite important the presence of initialDelaySeconds parameter to make sure the application has the option to start before start executing the probe. In case you don’t put this value you can get a Reboot Loop in your container.

NOTE: Example shows port 7777 as an exported port but this is only needed for the steps we’ve done before and you will not be needed in a real production environment.

So now we deploy again the YML file and once we get the application running we’re going to try the same approach, but now as we have the probes defined as soon as I stop the application containers has going to be restarted. Let’s see!

As you can see in the picture above after the application is Stopped the container has been restarted and because of that, we’ve got expelled from inside the container.

So, that’s all, I hope that helps you to set up your probes and in case you need more details, please take a look at the Kubernetes documentation about httpGet probes to see all the configuration and option that you can apply to them.