In a healthy development environment it is a good idea to catch issues early, and Continuous Integration (CI) Continuous integration - Wikipedia is a technique that can catch errors at the point changes are introduced into the code base by developers. We can use popular CI tools such as Jenkins with Apama, to provide CI and ensure that developer commits are tested and checked. This blog post shows step by step how you can do this.
Background
Continuous Integration involves testing each modification a developer introduces to the code base automatically, immediately when they occur. This practice helps software developers to complete tasks in a timely and predictable manner. Continuous Integration involves some integral components. The first is to make a self-testing build. A self-testing build is repeatable, scalable and modifiable, thus highlighting clandestine bugs that, due to human error, may go unnoticed.
Jenkins is a popular open source tool that can be used to perform continuous integration and build automation. Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software. Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with a Java Runtime Environment (JRE) installed.
Here we will be exploring on how to run and test Apama application in nightly basis using Jenkins.
Starting out & Prerequisites
Deployment
To run your Apama EPL application in any CI Tool, it needs to be deployed and the sources has to be committed to your version control system.
Use engine_deploy to generate a correlator deployment directory.
For example:
engine_deploy -d or --outputDeployDir <deploy dir location>
See “Deploying a correlator” in Apama documentation for more information. You can also use Ant to generate a correlator deployment directory or a correlator deployment package. The apama-macros.xml file includes the generate-correlator-deploy-dir for this purpose. See “About deploying Apama applications with an Ant script” in Apama documentation.
On deployment, it copies all required files from the project into a deployment directory. It also generates all required YAML configuration files and properties files using the information that is currently defined in the project’s launch configuration.
Environment
To run the Apama application and the pysys test , we need to set up our Apama environment. To do this your client/slave Jenkins machine should have Apama installation.
You can create a script to set the Apama environment from your installation location and then run tests. The test run will start the correlator with configuration for the application(EPLApp) and verifies the correlator output.
#This will cd to Jenkins work dir
cd $WORKSPACE
#For Linux slave
source <Apama_Home>/bin/apama_env
or
#for a windows slave
<Apama_Home>\bin\apama_env.bat
#this runs all the test under test folder
#for the example project I have provided
cd Test
pysys run
To run your application in docker, your client/slave Jenkins machine should have docker installed and docker credential to run the apama application in docker container. You can provide build script as following to run your application in docker container. We can change the port of our correlator using the -p argument.
cd $WORKSPACE
docker build -t builder-test:latest
docker run -it --rm -p 40000:15903 builder-test:latest
If you need guidance on this, one of the easier ways to create a self contained environment for running an EPL application is to use Docker. For more information on this process you can look at the steps mentioned in Build, test and deploy Apama projects using the Apama Builder Image blog post. It describes how you can integrate the build, deployment and testing into the creation of a Docker container dedicated to running your application. The use of Docker is optional of course and the process could occur on VMs, or physical hardware.
Automation using PySys test framework
We have support for the PySys test framework to test Apama applications developed with EPL. Apama includes extensions to PySys that let you test correlators, IAF components, EPL injection, sending and receiving events, and similar operations.
The PySys framework can start adapters, which enables you to perform end-to-end testing of an entire system. Samples for how to use PySys are in the samples\pysys folder of your Apama installation.
PySys can be used easily to test aspects of your application in your CI job, the only prerequisite is that apama_env needs to be run or sources in your environment. Then write some tests, and use the command ‘pysys run’ to start up the suite of tests you have written. You can find the steps in detail in README located in \samples\pysys
Assumptions
As a convenience we have a project you can play with to try this out with GitHub - sasmitaA/ApamaEPLApp . It includes a deployed sample EPLApp and a test to test it.
This post assumes your application is committed to SCM similar to my example above.
Also I am assuming that:
– your project builds, and you can deploy the project so that it runs in the Apama Correlator.
– you have a pysys test to test your application in the same location where you have your deployed project.
This guide will assume you will create a docker file for your application using a process similar to Build, test and deploy Apama projects using the Apama Builder Image blog post and copy it to your deployed project.
Jenkins Integration
Jenkins can be easily set up and configured via its web interface, which includes on-the-fly error checks and built-in help.
All right! The first look at a project configuration is a bit scary. There are so many options that maybe you will freeze but don’t. You only need a few of them to start. With time you will gain experience and understand the purpose of each option
At the first step configure your client/slave machine where you have the apama installation and docker(if required) in Jenkins server. To do that follow below mentioned the steps.
- Add your slave machine to Jenkins where it will run the SCM project and relevant tests.
Follow the steps mentioned here, Using Jenkins agents - Configure the node for slave machine to connect :
NOTE: Make sure Remote root directory exists before this node configuration done.
Click on the “Advanced” button below the credential Add button and mention the other connection configuration and the work dir, as following.
Once “Save” Button is clicked , it will run the process and will try to connect to the slave machine with credential given. It should be successful.
Once the process completed, you can check if it is connected by selecting the slave machine from Jenkins.Node.
- (Optional) Create a New View using Jenkins>New View.
Next step is create a new job and configure the project. For using the simplest example as it possible, let’s check out our Apama EPL application from GitHub. It could be also a local Git server or a Subversion. To make it happen, there are only two steps to configure.
-
Create a Job:
-
Select “New Item” in your Jenkins
-
Enter your job/project name, and select the kind of project as per your requirement. Here we have selected Freestyle project and click OK.
-
It will open configuration page:
-
Select the option “Restrict where this project can be run” in General tab. You will see Label Expression field, select the slave machine there.
-
-
Configure the Project:
-
Go to the Source Code Management section to configure source code repository.
In this case provide the git project: GitHub - sasmitaA/ApamaEPLApp as Repository URL.
-
Select the options in the Build Triggers section to configure when to trigger the build.
-
Go to Build section to set the Build environment . Click on the Add Build step , and select the option you prefer your build to be executed. Here we are selecting “Execute Shell”
-
Provide a set of commands to execute to build the Apama project. Here we will build the EPL Application and run the pysys test without docker. Provide the build script in your jenkins configuration similar to the one given below.
-
If you are running from docker, the Dockerfile creating your image has the configuration to build and run. So use the docker command (see the Prerequisite section above) to execute in the build script in your jenkins configuration instead. Again you can see more detail here.
Click on Save button.
Success?
That’s it! After clicking on Save to finish the configuration. Jenkins will redirect you to the initial page of your project.
To execute the project, click on Build Now link.
Crack!! Houston we have a problem!
As you may have noticed you got a red ball on your left. Your #1 build crashed! Don’t worry, let’s understand what happened. Knowing how to solve problems is the best skill that you can have.
Console Output
Don’t panic! Every build has a specific page showing everything that happened, just click on the build number link. It will be available the user that started the process, how much time it took and many others great information.
For now, the Console Output link is the best bet to understand what happened. It will show the output of the entire process.
In above image you can see the test failed as as it’s not finding the initialization.yaml file. Fix the issue and trigger the build again, you should be seeing a successful build as in following image.
Now you are all set to trigger a build. And your scheduled build will be triggered based on the option you have selected Build Triggers section of your jenkins job configuration.