Serverless is already here. We know that. Even for companies that are starting their cloud journey moving to container-based platforms, they already know that serverless is in their future view at least for some specific use cases.
Its simplicity without needed to worry about anything related to the platform just focuses on the code and also the economics that comes with them makes this computing model a game-changer.
The mainstream platform for this approach at this moment is AWS Lambda from Amazon, we have read and heard a lot of AWS Lamba, but all the platforms are going to that path. Google with its Google Functions, Microsoft with its Azure. This is not just a thing for public cloud providers, also if you’re running on a private cloud approach you can use this deployment mode using frameworks like KNative or OpenFaaS. If you need more details about it, also we had some articles about this topic:
Deploying Flogo App on OpenFaaS
OpenFaaS is an alternative to enable the serverless approach in your infrastructure when you’re not running in the public cloud and you don’t have available those other options like AWS Lambda Functions or Azure Functions or even in public cloud, you’d like the features and customizations option it provides. OpenFaaS® (Functions as a Service) is […]
All of these platforms are updating and enhancing its option to execute any kind of code that you want on it, and this is what Azure Function just announced some weeks ago, the release of a Custom Handler approach to support any kind of language that you could think about:
Azure Functions custom handlers
Learn to use Azure Functions with any language or runtime version.
And we’re going to test that doing what we’d like best ………..
The first thing we need to do is to create our Flogo Application. I’m going to use Flogo Enterprise to do that, but you can do the same using the Open Source version as well:
We’re going to create a simple application just a Hello World REST service. As you can see here:
The JSON file of the Flogo Application you can find it in the GitHub repository shown below:
GitHub - alexandrev/flogo-azure-function: Sample of Flogo Application running as a Azure Function
Sample of Flogo Application running as a Azure Function - GitHub - alexandrev/flogo-azure-function: Sample of Flogo Application running as a Azure Function
But the main configurations are the following ones:
Server will listen on the port specified by the environment variable FUNCTIONS_HTTPWORKER_PORT
Server will listen for a GET request on the URI /hello-world
Response will be quite simple “Invoking a Flogo App inside Azure Functions”
Now, that we have the JSON file, we only need to start creating the artifacts needed to run an Azure Function.
We will need to install the npm package for the Azure Functions. To do that we need to run the following command:
npm i -g azure-functions-core-tools@3 --unsafe-perm true
It is important to have the “@3” because that way we reference the version that has this new Custom Handler logic to be able to execute it.
We will create a new folder call flogo-func and run the following command inside that folder:
func init --docker
Now we should select node as the runtime to be used and javascriptas the language. That could be something strange because we are going to use neither node, nor dotnet or pythonor powershell. But we will select that to keep it simple as we just try to focus on the Docker approach to do that.
After that we just need to create a new function, and to do that we need to type the following command:
func new
In the CLI-based interface that the Azure Function Core Tools shows us, we just need to select HTTP trigger and provide a name.
In our case, we will use hello-world as the name to keep it similar to what we defined as part of our Flogo Application. We will end up with the following folder structure:
Now we need to open the folder that is has been created and we need to do several things:
First of all, we need to remove the index.js file because we don’t need that file as this will not be a node JS function.
We need to copy the HelloWorld.json (our Flogo application) to the root folder of the function.
We need to change the host.json file to the following content:
Now, we need to generate the engine-windows-amd64.exe and to be able to do that we need to go to the FLOGO_HOME in our machine and go to the folder FLOGO_HOME/flogo/<VERSION>/bin and launch the following command:
./builder-windows-amd64.exe build
And you should get the engine-windows-amd64.exe as the output as you can see in the picture below:
Now, you just need to copy that file inside the folder of your function, and you should have the following folder structure as you can see here:
And once you have that, you just need to run your function to be able to test it locally:
func start
After running that command you should see an output similar to the one shown below:
I’d just like to highlight the startup time for our Flogo application around 15 milliseconds!!! Now, you only need to test it using any browser and just go to the following URL:
http://localhost:7071/api/hello-world
This has been just the first step on our journey but it was the steps needed to be able to run our Flogo application as part of your serverless environment hosted by Microsoft Azure platform!
We’re living in an age where technologies are switching standards are changing all the time. You forget to read Medium/Stackoverflow/Reddit and you found there are at least five (5) new industry standards that are taking the place of the existing ones that you know (those that have been releasing something like a year ago 🙂 ).
Do you still remember the old ages when SOAP was the unbeatable format? How much time did we spend building our SOAP Services in our enterprises? REST replace it as the new standard.. but just a few years and we’re back in a new battle just for synchronous communication: gRPC, GraphQL, are here to conquer everything again. It is crazy, huh?
But the situation is similar to asynchronous communication. Asynchronous communication has been here for a long time. Even, a long time before the terms Event-Driven Architecture or Streaming was really a “cool” term or a thing you be aware of.
We’ve been using these patterns for so long in our companies. Big enterprises have been using this model into their enterprise integrations for so long. Pub/Sub based protocols and technologies like TIBCO Rendezvous has been using since the late 90, and then we also incorporate more standards approaches like JMS using a different kind of servers to have all these event-based communications.
But now with the cloud-native revolution, the need for distributed computing, more agility, more scalability, centralized solutions are not valid anymore, and we’ve seen an explosion in the number of options to communicate based on these patterns.
You could think that this is the same situation as we were discussing at the beginning of this article regarding REST predominance and new cutting-edge technologies trying to replace it, but this is something quite different. Because experience has told us that a single size doesn’t fit all.
You cannot find a single technology or component that can provide all the communication needs that you need for all your use-cases. You can name any technology or protocol that you want: Kafka, Pulsar, JMS, MQTT, AMQP, Thrift, FTL, and so on.
Think about each of them and you probably will find some use-cases that one technology plays better than the others, so it makes no sense to just trying to find a single technology solution to cover all the needs. What it is needed is more a polyglot approach when you have different technologies that play well together and use the one that works best for your use case (the right tool for the right job approach) as we’re doing for the different technologies we’re deploying in our cluster.
Probably we’re not going to use the same technology to do a Machine Learning based Microservice, than a Streaming Application, right? The same principle applies here.
But the problem here when we try to talk about different technologies playing together is about standardization. If we think about REST, gRPC, or GraphQL even that they’re different they play based some common grounds. They rely on the same base HTTP protocol for a standard so it is easy to support all of them in the same architecture.
But this is not true with the technologies about Asynchronous Communication. And I’d like to focus on standardization and specification today. And that’s what AsyncAPI Initiative is trying to solve. And to define what AsyncAPI is I’d like to use their own words from their official website:
AsyncAPI is an open source initiative that seeks to improve the current state of Event-Driven Architectures (EDA). Our long-term goal is to make working with EDA’s as easy as it is to work with REST APIs. That goes from documentation to code generation, from discovery to event management. Most of the processes you apply to your REST APIs nowadays would be applicable to your event-driven/asynchronous APIs too.
So, their goal is to provide a set of tools to have a better world in all those EDA architectures that all companies have or starting to have at this moment and everything pivots around one thing: The OpenAPI Specification.
Similar to the OpenAPI specification it allows us to define a common interface for our EDA Interfaces and the most important part is that this is multi-channel. So the same specification can be used for your MQTT-based API or your Kafka API. Let’s take a look at how it looks like this AsyncAPI Specification:
As you can see it is very similar to the OpenAPI 3.0 and they already done that with the purpose to ease the transition between OpenAPI 3.0 and AsyncAPI and also to try to join both worlds together: It is more about just API, no matter if they’re synchronous or asynchronous and provide the same benefits regarding the ecosystem from one to the other.
Show me the code!!
But let’s stop talking and let’s start coding and to do that I’d like to use one of the tools that in my view has the greater support for AsyncAPI, and that’s it Project Flogo.
Probably you remember some of the different posts I’ve been done regarding Project Flogo and TIBCO Flogo Enterprise as a great technology to use for your microservices development (low-code/all-code approach, Golang based, a lot of connectors and open source extensions as well).
But today we’re going to use it to create our first AsyncAPI compliant microservice. And we’re going to rely on that because it provides a set of extensions to support the AsyncAPI initiative as you can see here:
GitHub – project-flogo/asyncapi: Flogo extensions to support AsyncAPI
Flogo extensions to support AsyncAPI. Contribute to project-flogo/asyncapi development by creating an account on GitHub.
So the first thing that we’re going to do is to create our AsyncAPI definition and to do it simpler, we’re going to use the sample one that we have available in the OpenAsync API with a simple change: We’re going to change from AMQP protocol to Kafka protocol because this is cool these days, isn’t it? 😉
asyncapi: '2.0.0'
info:
title: Hello world application
version: '0.1.0'
servers:
production:
url: broker.mycompany.com
protocol: kafka
description: This is "My Company" broker.
security:
- user-password: []
channels:
hello:
publish:
message:
$ref: '#/components/messages/hello-msg'
goodbye:
publish:
message:
$ref: '#/components/messages/goodbye-msg'
components:
messages:
hello-msg:
payload:
type: object
properties:
name:
type: string
sentAt:
$ref: '#/components/schemas/sent-at'
goodbye-msg:
payload:
type: object
properties:
sentAt:
$ref: '#/components/schemas/sent-at'
schemas:
sent-at:
type: string
description: The date and time a message was sent.
format: datetime
securitySchemes:
user-password:
type: userPassword
As you can see something simple. Two operations “hello” and “goodbye” with easy payload:
name: Name that we’re going to use for the greeting.
sentAt: The date and time a message was sent.
So the first thing we’re going to do is to create a Flogo Application that complies to that AsyncAPI specification:
git clone https://github.com/project-flogo/asyncapi.git
cd asyncapi/
go install
Now we have the generator installer so we only need to execute and provide our YML as the input in the following command:
And we will create a HelloWorld application for us, that we need to tweak a little bit. Only to make you be up & running quickly, I’m just sharing the code in my GitHub repository that you can borrow from them (But I really encourage you to take the time to take a look at the code to see the beauty of the Flogo App Development 🙂 )
Now, that we already have the app, we have just a simple dummy application that allows us to receive the message that complies with the specification, and in our case just log the payload, which can be our starting point to build our new Event-Driven Microservices compliant with AsyncAPI.
So, let’s try it but to do so, we need a few things. First of all, we need a Kafka server running and to do that in a quick way we’re going to leverage on the following docker-compose.yml file:
And to run that we just need to fire the following command from the same folder we have this file named as docker-compose.yml:
docker-compose up -d
And after doing that, we just need a sample application and what better that use Flogo again to create it but this time, let’s use the Graphical Viewer to create it right away:
Simple Flogo Application to send a AsyncAPI complaint-message each minute using Kafka as a protocol
So we need just to configure the Publish Kafka activity to provide the broker (localhost:9092), the topic (hello) and the message :
{
"name": "hello world",
"sentAt": "2020-04-24T00:00:00"
}
And that’s it! Let’s run it!!!:
First we start the AsyncAPI Flogo Microservice:
Async API Flogo Microservices Started!
And then we just launch the tester, that is going to send the same message each minute, as you can see in the picture below:
Sample Tester sending sample messages
And each time we sent that message, is going to be received in our Async API Flogo Microservice:
So, I hope this first introduction to the AsyncAPI world has been of the interest of you, but don’t forget to take a look at more resources in their own website:
GitHub – project-flogo/asyncapi: Flogo extensions to support AsyncAPI
Flogo extensions to support AsyncAPI. Contribute to project-flogo/asyncapi development by creating an account on GitHub.
GitHub – alexandrev/asyncapi-flogo-test: This is the material that supports the post on Medium regarding “Welcome to the AsyncAPI Revoluton” and provide the flogo code needed to run the same sample that is being shown in the article
This is the material that supports the post on Medium regarding "Welcome to the AsyncAPI Revoluton" and provide the flogo code needed to run the same sample that is being shown in the art…
Add a header to begin generating the table of contents
We all know that in the rise of the cloud-native development and architectures, we’ve seen Kubernetes based platforms as the new standard all focusing on new developments following the new paradigms and best practices: Microservices, Event-Driven Architectures new shiny protocols like GraphQL or gRPC, and so on and so forth.
This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.
In previous posts, I’ve explained how to integrate TIBCO BusinessWorks 6.x / BusinessWorks Container Edition (BWCE) applications with Prometheus, one of the most popular monitoring systems for cloud layers. Prometheus is one of the most widely used solutions to monitor your microservices inside a Kubernetes cluster. In this post, I will explain steps to leverage Prometheus for integrating with applications running on TIBCO Cloud Integration (TCI).
TCI is TIBCO’s iPaaS and primarily hides the application management complexity of an app from users. You need your packaged application (a.k.a EAR) and manifest.json — both generated by the product to simply deploy the application.
Isn’t it magical? Yes, it is! As explained in my previous post related to Prometheus integration with BWCE, which allows you to customize your base images, TCI allows integration with Prometheus in a slightly different manner. Let’s walk through the steps.
TCI has its own embedded monitoring tools (shown below) to provide insights into Memory and CPU utilization, plus network throughput, which is very useful.
While the monitoring metrics provided out-of-the-box by TCI are sufficient for most scenarios, there are hybrid connectivity use-cases (application running on-prem and microservices running on your own cluster that could be on a private or public cloud) that might require a unified single-pane view of monitoring.
Import the Prometheus plugin by choosing Import → Plug-ins and Fragments option and specifying the directory downloaded from the above mentioned GitHub location. (shown below)
Step two involves adding the Prometheus module previously imported to the specific application as shown below:
Step three is just to build the EAR file along with manifest.json.
NOTE: If the EAR doesn’t get generated once you add the Prometheus plugin, please follow the below steps:
Export the project with the Prometheus module to a zip file.
Remove the Prometheus project from the workspace.
Import the project from the zip file generated before.
Before you deploy the BW application on TCI, we need to enable an additional port on TCI to scrape the Prometheus metrics.
Step four Updating manifest.json file.
By default, a TCI app using the manifest.json file only exposes one port to be consumed from outside (related to functional services) and the other to be used internally for health checks.
For Prometheus integration with TCI, we need an additional port listening on 9095, so Prometheus server can access the metrics endpoints to scrape the required metrics for our TCI application.
We need to slightly modify the generated manifest.json file (of BW app) to expose an additional port, 9095 (shown below) .
Also, to tell TCI that we want to enable Prometheus endpoint we need to set a property in the manifest.json file. The property is TCI_BW_CONFIG_OVERRIDES and provide the following value: BW_PROMETHEUS_ENABLE=true, as shown below:
We also need to add an additional line (propertyPrefix) in the manifest.json file as shown below.
Now, we are ready to deploy the BW app on TCI and once it is deployed we can see there are two endpoints
If we expand the Endpoints options on the right (shown above), you can see that one of them is named “prometheus” and that’s our Prometheus metrics endpoint:
Just copy the prometheus URL and append it with /metrics (URL in the below snapshot) — this will display the Prometheus metrics for the specific BW app deployed on TCI.
Note: appending with /metrics is not compulsory, the as-is URL for Prometheus endpoint will also work.
In the list you will find the following kind of metrics to be able to create the most incredible dashboards and analysis based on that kind of information:
JVM metrics around memory used, GC performance and thread pools counts
CPU usage by the application
Process and Activity execution counts by Status (Started, Completed, Failed, Scheduled..)
Duration by Activity and Process.
With all this available the information you can create dashboards similar to the one shown below, in this case using Spotfire as the Dashboard tool:
But you can also integrate those metrics with Grafana or any other tool that could read data from Prometheus time-series database.
OpenFaaS is an alternative to enable the serverless approach in your infrastructure when you’re not running in the public cloud and you don’t have available those other options like AWS Lambda Functions or Azure Functions or even in public cloud, you’d like the features and customizations option it provides.
OpenFaaS® (Functions as a Service) is a framework for building serverless functions with Docker and Kubernetes which has first-class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.
GitHub - openfaas/faas: OpenFaaS - Serverless Functions Made Simple
OpenFaaS - Serverless Functions Made Simple. Contribute to openfaas/faas development by creating an account on GitHub.
There is good content on medium about OpenFaaS so I don’t like to spend so much time on this, but I’d like to leave you here some links for reference:
Running serverless functions on premises using OpenFaas with Kubernetes
Earlier most of companies had monolithic architecture for their cloud applications (all the things in a single package). Nowadays companies…
An Introduction to Serverless DevOps with OpenFaaS | HackerNoon
DevOps isn’t about just doing CI/CD. But a CI/CD pipeline has an important role inside DevOps. I’ve been investing my time on recently and as I started creating multiple functions, I wanted an easy to use and accessible development and delivery flow, in other words a CI/CD pipeline. OpenFaaS One day…
We already have a lot of info about how to run a Flogo Application as a Lambda Function as you can see here:
https://www.youtube.com/watch?v=TysuwbXODQI
But.. what about OpenFaaS? Can we run our Flogo application inside OpenFaaS? Sure! Let me explain to you how.
OpenFaaS is a very customize framework to build zero-scaled functions and it could need some time to get familiar with concepts. Everything is based on watchdogs that are the components listening to the requests are responsible for launching the forks to handle the requests:
We’re going to use the new watchdog named as of-watchdog that is the one expected to be the default one in the feature and all the info is here:
GitHub - openfaas/of-watchdog: Reverse proxy for STDIO and HTTP microservices
Reverse proxy for STDIO and HTTP microservices. Contribute to openfaas/of-watchdog development by creating an account on GitHub.
This watchdog provides several modes, one of them is named HTTP and it is the default one, and is based on some HTTP Forward to the internal server running in the container. That fits perfectly with our Flogo application and means that the only thing we need is to deploy an HTTP Receive Request trigger in our Flogo Application and that’s it.
The only things you need to configure is the method (POST) and the Path (/) to be able to handle the requests. In our case we’re going to do a simple Hello World app as you can see here:
To be able to run this application we need to use several things, and let’s explain it here:
First of all, we need to do the installation of the OpenFaaS environment, I’m going to skip all the details about this process and just point to you to the detailed tutorial about it:
Deploy OpenFaaS on Amazon EKS | Amazon Web Services
We’ve talked about FaaS (Functions as a Service) in Running FaaS on a Kubernetes Cluster on AWS Using Kubeless by Sebastien Goasguen. In this post, Alex Ellis, founder of the OpenFaaS project, walks you through how to use OpenFaaS on Amazon EKS. OpenFaaS is one of the most popular tools in the FaaS…
Now we need to create our template and to do that, we are going to use a Dockerfile template. To create it we’re going to execute:
faas-cli new --lang dockerfile
We’re going to name the function flogo-test. And now we’re going to update the Dockerfile to be like this:
Most of this content is common for any other template using the new of-watchdog and the HTTP mode.
I’d like to highlight the following things:
We use several environment variables to define the behavior:
mode = HTTP to define what we’re going to use this method
upstream_url = URL that we are going to forward the request to
fprocess = OS command that we need to execute, in our case means to run the Flogo App.
Other things are the same as you should do in case you want to run flogo apps in Docker:
Add the engine executable for your platform (UNIX in most cases as the image base is almost always a Linux based)
Add the JSON file of the application that you want to use.
We also need to change the yml file to look like this:
Usually, when you’re developing or running your container application you will get to a moment when something goes wrong. But not in a way you can solve with your logging system and with testing.
A moment when there is some bottleneck, something that is not performing as well as you want, and you’d like to take a look inside. And that’s what we’re going to do. We’re going to watch inside.
Because our BusinessWorks Container Edition provides so great features to do it that you need to use it into your favor because you’re going to thank me for the rest of your life. So, I don’t want to spend one more minute about this. I’d like to start telling you right now.
The first thing we need to do, we need to go inside the OSGi console from the container. So, the first thing we do is to expose the 8090 port as you can see in the picture below
Now, we can expose that port to your host, using the port-forward command
And as you can see it says that statistics has been enabled for echo application, so using that application name we’re going to gather the statistics at the level
And you can see the statistics at the process level where you can see the following metrics:
Process metadata (name, parent process and version)
Total instance by status (create, suspended, failed and executed)
Execution time (total, average, min, max, most recent)
Elapsed time (total, average, min, max, most recent)
And we can get the statistics at the activity level:
And with that, you can detect any bottleneck you’re facing into your application and also be sure which activity or which process is responsible for it. So you can solve it in a quick way.
Prometheus Monitoring for Microservices using TIBCO
We’re living a world with constant changes and this is even more true in the Enterprise Application world. I’ll not spend much time talking about things you already know, but just say that the microservices architecture approach and the PaaS solutions have been a game-changer for all enterprise integration technologies. This time I’d like to […]
In that post, we described that there were several ways to update Prometheus about the services that ready to monitor. And we choose the most simple at that moment that was the static_config configuration which means:
Don’t worry Prometheus, I’ll let you know the IP you need to monitor and you don’t need to worry about anything else.
And this is useful for a quick test in a local environment when you want to test quickly your Prometheus set up or you want to work in the Grafana part to design the best possible dashboard to handle your need.
But, this is not too useful for a real production environment, even more, when we’re talking about a Kubernetes cluster when services are going up & down continuously over time. So, to solve this situation Prometheus allows us to define a different kind of ways to perform this “service discovery” approach. In the official documentation for Prometheus, we can read a lot about the different service discovery techniques but at a high level these are the main service discovery techniques available:
Configuration | Prometheus
An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.
azure_sd_configs: Azure Service Discovery
consul_sd_configs: Consul Service Discovery
dns_sd_configs: DNS Service Discovery
ec2_sd_configs: EC2 Service Discovery
openstack_sd_configs: OpenStack Service Discovery
file_sd_configs: File Service Discovery
gce_sd_configs: GCE Service Discovery
kubernetes_sd_configs: Kubernetes Service Discovery
marathon_sd_configs: Marathon Service Discovery
nerve_sd_configs: AirBnB’s Nerve Service Discovery
serverset_sd_configs: Zookeeper Serverset Service Discovery
triton_sd_configs: Triton Service Discovery
static_config: Static IP/DNS for the configuration. No Service Discovery.
And even, it all these options are not enough for you and need something more specific you have an API available to extend the Prometheus capabilities and create your own Service Discovery technique. You can find more info about it here:
Implementing Custom Service Discovery | Prometheus
An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.
But this is not our case, for us, the Kubernetes Service Discovery is the right choice for our approach. So, we’re going to change the static configuration we had in the previous post:
As you can see this is quite more complex than the previous configuration but it is not as complex as you can think at first glance, let’s review it by different parts.
- role: endpoints
namespaces:
names:
- default
It says that we’re going to use role for endpoints that are created under the default namespace and we’re going to specify the changes we need to do to find the metrics endpoints for Prometheus.
That means that we want to do a replace of the label value and we can do several things:
Rename the label name using the target_label to set the name of the final label that we’re going to create based on the source_labels.
Replace the value using the regex parameter to define the regular expression for the original value and the replacement parameter that is going to express the changes that we want to do to this value.
So, now after applying this configuration when we deploy a new application in our Kubernetes cluster, like the project that we can see here:
Automatically we’re going to see an additional target on our job-name configuration “bwce-metrics”
Flogo Enterprise is so great platform to build your microservices and Out of the box, you’re going to reach an incredible performance number.
This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.
!– /wp:paragraph –>
But, even with that, we’re working in a world where each milliseconds count and each memory MB count so it is important to know the tools that we have in our hands to tune at a finer-grained level our Flogo Enterprise application.
As you already know, Flogo is built in top of Go programing language, so we are going to differentiate the parameters that belong to the programing language itself, then other parameters that are Flogo specifics.
All these parameters have to be defined as environment variables, so the way to apply these are going to rely on how you set environment variables in your own target platform (Windows, Linux, OSX, Docker, etc…)
Flogo OSS Specific Parameters
You can check all the parameters and it is default values at the Flogo documentation:
Performance Related Settings
FLOGO_LOG_LEVEL: Allows to set at start-up level the log level you want to use for the application execution. The default value is set to “INFO” and it can be increased to DEBUG to do some additional troubleshooting or analysis and set to “WARN” or “ERROR” for production applications that need most performance avoiding printing additional traces.
FLOGO_RUNNER_TYPE: Allows to set at the start-up level the type of the runner and the default value is POOLED.
FLOGO_RUNNER_WORKERS: Allows to set at the start-up level the number of Flogo workers that are going to be executing logic. The default value is 5 and can be increased when you’re running on powerful hardware that has better parallelism capabilities.
FLOGO_RUNNER_QUEUE: Allows to set up at the start-up level the size of the runner queue that is going to keep in memory the requests that are going to be handled by the workers. The default value is 50 and when the number is high it will be also high the memory consumption but the access form the workers to the task will be faster.
Other Settings
FLOGO_CONFIG_PATH: Sets the path of the config JSON file that is going to be used to run the application in case it is not embedded in the binary itself. The default value is flogo.json
FLOGO_ENGINE_STOP_ON_ERROR: Set the behavior of the engine when an internal error occurs. By default is set to true and means that engine will stop as soon as the error occurs.
FLOGO_APP_PROP_RESOLVERS: Set how application properties are going to be gathered for the application to be used. The value is property resolver to use at runtime. By default is None and additional information is included in application properties documentation section.
FLOGO_LOG_DTFORMAT: Set how the dates are going to be displayed in the log traces. The default value is “2006–01–02 15:04:05.000”.
Flogo Enterprise Specific Parameters
Even when all the Project Flogo properties are supported by Flogo Enterprise, the enterprise version includes additional properties that can be used to set additional behaviors of the engine.
FLOGO_HTTP_SERVICE_ PORT: This property set the port where the internal endpoints will be hosted. This internal endpoint is used for healthcheck endpoint as well as metrics exposition and any other internal access that is provided by the engine.
FLOGO_LOG_FORMAT: This property allows us to define the notation format for our log traces. TEXT is the default value but we can use JSON to make our traces to be generated in JSON, for example, to be included in some kind of logging ingestion platform
Go Programing Language Specific Parameters
GOMAXPROCS: Limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit.
GOTRACEBACK: Controls the amount of output generated when a Go program fails due to an unrecovered panic or an unexpected runtime condition. By default, a failure prints a stack trace for the current goroutine, eliding functions internal to the run-time system and then exits with exit code 2. The failure prints stack traces for all goroutines if there is no current goroutine or the failure is internal to the run-time. GOTRACEBACK=none omits the goroutine stack traces entirely. GOTRACEBACK=single (the default) behaves as described above. GOTRACEBACK=all adds stack traces for all user-created goroutines. GOTRACEBACK=system is like “all” but adds stack frames for run-time functions and shows goroutines created internally by the run-time. GOTRACEBACK=crash is like “system” but crashes in an operating system-specific manner instead of exiting.
GOGC: Sets the initial garbage collection target percentage. A collection is triggered when the ratio of freshly allocated data to live data remaining after the previous collection reaches this percentage. The default is GOGC=100. Setting GOGC=off disables the garbage collector entirely.
Prometheus is becoming the new standard for Kubernetes monitoring and today we are going to cover how we can do Prometheus TIBCO monitoring in Kubernetes.
This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.
We’re living in a world with constant changes and this is even more true in the Enterprise Application world. I’ll not spend much time talking about things you already know, but just say that the microservices architecture approach and the PaaS solutions have been a game-changer for all enterprise integration technologies.
This time I’d like to talk about monitoring and the integration capabilities we have of using Prometheus to monitor our microservices developed under TIBCO technology. I don’t like to spend too much time either talking about what Prometheus is, as you probably already know, but in a summary, this is an open-source distributed monitoring platform that has been the second project released by the Cloud Native Computing Foundation (after Kubernetes itself) and that has been established as a de-facto industry standard for monitoring K8S clusters (alongside with other options in the market like InfluxDB and so on).
Prometheus has a lot of great features, but one of them is that it has connectors for almost everything and that’s very important today because it is so complicated/unwanted/unusual to define a platform with a single product for the PaaS layer. So today, I want to show you how to monitor your TIBCO BusinessWorks Container Edition applications using Prometheus.
Most of the info I’m going to share is available in the bw-tooling GitHub repo, so you can get to there if you need to validate any specific statement.
bw-tooling/prometheus-integration at master · TIBCOSoftware/bw-tooling
Collection of tools designed to simplify deployment and management of TIBCO BusinessWorks applications – bw-tooling/prometheus-integration at master · TIBCOSoftware/bw-tooling
Ok, are we ready? Let’s start!!
I’m going to assume that we already have a Kubernetes cluster in place and Prometheus installed as well. So, the first step is to enhance the BusinessWorks Container Edition base image to include the Prometheus capabilities integration. To do that we need to go to the GitHub repo page and follow these instructions:
Download & unzip the prometheus-integration.zip folder.
Open TIBCO BusinessWorks Studio and point it to a new workspace.
Right-click in Project Explorer → Import… → select Plug-ins and Fragments → select Import from the directory radio button
Browse it to prometheus-integration folder (unzipped in step 1)
Now click Next → Select Prometheus plugin → click Add button → click Finish. This will import the plugin in the studio.
Now, to create JAR of this plugin so first, we need to make sure to update com.tibco.bw.prometheus.monitor with ‘.’ (dot) in Bundle-Classpath field as given below in META-INF/MANIFEST.MF file.
Right-click on Plugin → Export → Export…
Select type as JAR file click Next
Now Click Next → Next → select radio button to use existing MANIFEST.MF file and browse the manifest file
Click Finish. This will generate prometheus-integration.jar
Now, with the JAR already created what we need to do is include it in your own base image. To do that we place the JAR file in the <TIBCO_HOME>/bwce/2.4/docker/resources/addons/jar
And we launch the building image command again from the <TIBCO_HOME>/bwce/2.4/docker folder to update the image using the following command (use the version you’re using at the moment)
docker build -t bwce_base:2.4.4 .
So, now we have an image with Prometheus support! Great! We’re close to the finish, we just create an image for our Container Application, in my case, this is going to be a very simple echo service that you can see here.
And we only need to keep these things in particular when we deploy to our Kubernetes cluster:
We should set an environment variable with the BW_PROMETHEUS_ENABLE to “TRUE”
We should expose the port 9095 from the container to be used by Prometheus to integrate.
Now, we only need to provide this endpoint to the Prometheus scrapper system. There are several ways to do that, but we’re going to focus on the simple one.
We need to change the prometheus.yml to add the following job data:
Flogo Test is one of the main steps in your CI/CD lifecycle if you are using Flogo. You probably have done it previously in all your other developments like Java developments or even using BusinessWorks 6 using the bw6-maven-plugin:
This article is part of my comprehensive TIBCO Integration Platform Guide where you can find more patterns and best practices for TIBCO integration platforms.
GitHub – TIBCOSoftware/bw6-plugin-maven: Plug-in Code for Apache Maven and TIBCO ActiveMatrix BusinessWorks™
Plug-in Code for Apache Maven and TIBCO ActiveMatrix BusinessWorks™ – GitHub – TIBCOSoftware/bw6-plugin-maven: Plug-in Code for Apache Maven and TIBCO ActiveMatrix BusinessWorks™
So, you’re probably wondering… How this is going to be done by Flogo? Ok!! I’ll tell you.
First of all, you need to keep in mind that Flogo Enterprise is a product that was designed with all those aspects in mind, so you don’t need to worry about it.
Regarding testing, when we need to include inside a CI/CD lifecycle approach, these testing capabilities should meet the following requirements:
It should be defined in some artifacts.
It should be executed automatically
It should be able to check the outcome.
Flogo Enterprise includes by default Testing capabilities in the Web UI where you can not only test your flows using the UI from a debug/troubleshooting perspective but also able to generate the artifacts that are going to allow you to perform more sophisticated testing
So, we need to go to our Web UI and when we’re inside a flow we have a “Start Testing” button:
And we can see all our Launch Configuration change and most important part for this topic be able to export it and download it to your local machine:
Once everything is downloaded and we have the binary for our application we can execute the tests in an automatic way from the CLI using the following command
This is going to generate an output file with the output of the execution test:
And if we open the file we will get the exact same output the flow is returned so we can perform any assert on it
That was easy, right? Let’s do some additional tweaks to avoid your need to go to the Web UI. You can generate the launch configuration using only the CLI.
To do that, you only need to execute the following command:
