Ant build only modified IS packages

, ,
1

What product/components do you use and which version/fix level are you on?

10.5 IS packages

Is your question related to the free trial, or to a production (customer) instance?

Customer instance dev environment

What are you trying to achieve? Please describe it in detail.

I have local service development to create and modify the existing assets (IS packages), when the changes has been added via designer I’m moving the newly created packages into local service development and finally pushing the changes into azure repository.

After publish the changes into remote repository, our azure pipeline will automatically based on the remote repo update.

  1. first step azure pipeline by default checkout the code from git repository to local VM where our azure Agent is running (self hosted Agent)
  2. Under the packages folder we have our all the IS packages
/home/user/_work/1/s/packages
  1. From the above location we are moving our all the packages into some custom location to keep our assets
/home/user/workspace/packages
  1. Then we are calling our ant file from our SAG installation location path
cp -r /home/user/_work/1/s/packages* /home/user/workspace/packages
/home/user/softwareAg/common/AssetBuildEnvironment/bin/build.sh -Dbuild.output.dir="/home/user/build_ouput/" -Denable.build.IS=true -Dbuild.source.dir="/home/user/workspace/packages"

using our build.sh file we can able to build our assets and its creating zip file as well.

please find the below build successful logs

2022-10-19T15:41:36.4550730Z generate-composites:
2022-10-19T15:41:38.0150881Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read packages LSD_6,devopsLSD_POC1,devopsLSD_POC2,devopsLSD_POC3,scm_poc1,testLSD_POC1,testLSD_POC2,testLSD_POC3,testLSD_POC4,scm_poc2,CAR,Drinks. 
2022-10-19T15:41:38.0151837Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [LSD_6]. 
2022-10-19T15:41:38.0152433Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [LSD_6]. 
2022-10-19T15:41:38.0153042Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [devopsLSD_POC1]. 
2022-10-19T15:41:38.0153637Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [devopsLSD_POC1]. 
2022-10-19T15:41:38.0154224Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [devopsLSD_POC2]. 
2022-10-19T15:41:38.0158322Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [devopsLSD_POC2]. 
2022-10-19T15:41:38.0159052Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [devopsLSD_POC3]. 
2022-10-19T15:41:38.0159641Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [devopsLSD_POC3]. 
2022-10-19T15:41:38.0160207Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [scm_poc1]. 
2022-10-19T15:41:38.0161374Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [scm_poc1]. 
2022-10-19T15:41:38.0162508Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [testLSD_POC1]. 
2022-10-19T15:41:38.0163080Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [testLSD_POC1]. 
2022-10-19T15:41:38.0163630Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [testLSD_POC2]. 
2022-10-19T15:41:38.0164313Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [testLSD_POC2]. 
2022-10-19T15:41:38.0164858Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [testLSD_POC3]. 
2022-10-19T15:41:38.0165431Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [testLSD_POC3]. 
2022-10-19T15:41:38.0168608Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [testLSD_POC4]. 
2022-10-19T15:41:38.0169392Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [testLSD_POC4]. 
2022-10-19T15:41:38.0169958Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [scm_poc2]. 
2022-10-19T15:41:38.0170467Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: [scm_poc2] is not an Integration Server package. 
2022-10-19T15:41:38.0170990Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [CAR]. 
2022-10-19T15:41:38.0171532Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [CAR]. 
2022-10-19T15:41:38.0172054Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Starting to read package [Drinks]. 
2022-10-19T15:41:38.0172591Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed creation of ACDL and binary for a package [Drinks]. 
2022-10-19T15:41:38.0173105Z      [java] [10/19/22 3:41 PM][MESSAGE] ispackage.ispackage: Completed all packages. 
2022-10-19T15:41:38.0243523Z 
2022-10-19T15:41:38.0244130Z iterate_single_project_folders:
2022-10-19T15:41:38.0250841Z      [echo] Integration Server Build complete...........

If you see the build output logs i have list of packages inside my azure repository, but i’m not modifying all the IS assets all the time.

Here the list of available packages inside my azure repo.
LSD_6,devopsLSD_POC1,devopsLSD_POC2,devopsLSD_POC3,scm_poc1,testLSD_POC1,testLSD_POC2,testLSD_POC3,testLSD_POC4,scm_poc2,CAR,Drinks

Let say currently I have been modified single existing package from above list if i modify only CAR packages, and i pushed my changes into azure repo then my build pipeline started to pick all the packages by default, but i don’t want to build un-touched packages all the time (instead i want build only latest modified package alone)

Do you get any error messages? Please provide a full error message screenshot and log file.

NA

Have you installed all the latest fixes for the products and systems you are using?

YES

Thanks,

2

I get what you’re trying to do but can I ask: why?

In the past, I have also worked on webMethods CI/CD pipelines that had intelligence to determine what files had changed, and it would then attempt to only include those files in the build and in the Deployer project. However, with time, I’ve learned to appreciate a simpler pipeline where the entire repo is built and deployed each time. They are simpler to maintain, you don’t run into issues of dependencies accidentally not being included/deployed, and the outcome is always predictable, i.e. at the end of a successful deployment, your target servers should match exactly what it is in the repo.

Now, if your repo grows large enough, then you may find that deployment is taking too long and perhaps then you may want to consider a different strategy, but until then, I’d keep it simple.

If after reading this you still want to stick with your approach, then given that you’re working with a git repo, you could use a command like git diff --name-only HEAD HEAD~1 to determine what files have changed. Your script can then identify the parent packages of those files and only move those to the build.source.dir location.

Hope this helps,
Percio

1 Like
3

I’m not sure how much it will helpful for git diff command right now i have test to modify the single package via designer and captured the output into some text file for later use (as you said to pass thee names into source file)

git diff --name-only HEAD HEAD~1 > package.txt

cat package.txt
packages/CAR/ns/Car/Manaul/flow.xml
packages/CAR/ns/Car/Manaul/flow.xml.bak
packages/CAR/ns/Car/Manaul/node.ndf

$ cat package.txt | cut -d'/' -f2 | sort -u
Car

But still this is not a right solution to control the package names what I feel, please suggest if any good approach.

And also git diff command we have specify the branch name always before executing it.

4

I’m not sure I understand what the problem is and why “this is not a right solution.” Please explain. If you’re looking for an out-of-the-box solution to your problem, it doesn’t exist.

By the way, I see flow.xml.bak in your list of files. I recommend adding *.bak, *.class, and java.frag into your .gitignore file. Those files don’t need to be version controlled.

Hope this helps,
Percio

5

Let me create .gitingnore and i will add above files, kindly advise if any other files need to be added inside git ignore file itself.

6

Adding note here, few min back i have added alone .gitingnore file after adding the file into remote repository my pipeline start the build and executed the git diff command and re-directed the output into another file. pipeline getting failed while running build.

cp: cannot stat ‘/home/user/CICD/_work/1/s/packages/.gitignore’: No such file or directory
Buildfile: /home/user/SAG/common/AssetBuildEnvironment/master_build/build.xml

I know the reason because there is no packages has been changes from existing services… since i have modified some general file. suppose if future if any config related file has been updated from the remote repo then pipeline will trigger then it will fail.

7

There are different ways to handle this scenario. Your script could have some intelligence to only continue if the files were modified within a package. Some platforms offer the ability to configure rules so that the pipeline job only runs if the changed files match a given rule. For example, I’m currently using Gitlab and Gitlab offers an only:changes rule that you can configure on your pipeline job so it only runs if certain files are modified:

Percio

1 Like
8

thanks for your valuable time let me check azure side whether they will provide the option to build if only:changes condition.

9

i have added like this in my azure yaml, its working perfectly

trigger:
  paths:
    include:
    - path/to/include/*
1 Like
10

Hi @Percio_Castro1, I have identified one issue using HEAD revision command, let say i have couple of commits in my develop branch it has been not deployed yet into target server. when i am executing below command it will show always head revision only, it will not go backward latest package which is not deployed yet.

$ git diff --name-only HEAD HEAD~1
packages/LSD_6/ns/LSD_6/D/flow.xml
packages/LSD_6/ns/LSD_6/D/flow.xml.bak
packages/LSD_6/ns/LSD_6/D/node.ndf

$ git diff --name-only HEAD HEAD~1
packages/LSD_5/.classpath
packages/LSD_5/.project
packages/LSD_5/manifest.v3
packages/LSD_5/ns/LSD_5/A/flow.xml
packages/LSD_5/ns/LSD_5/A/flow.xml.bak
packages/LSD_5/ns/LSD_5/A/node.ndf
packages/LSD_5/ns/LSD_5/node.idf
packages/LSD_5/pub/index.html
11

There are different ways to get the information you’re looking for using git diff or git diff-tree. Give Google a shot and I think you’ll find what you’re looking for. Having said this, another thing you may want to look into is whether Azure already has this information available that you can retrieve from an environment variable or via a function call. I recall Jenkins had this feature so perhaps Azure does as well.

Good luck,
Percio

12

H @Percio_Castro1 i have tried to implement like below but its not working.

  1. Let say Edited build.yaml file for testing purpose and pushed the changes into remote repo but build is not triggered due to disabled option.
  2. Added new packages and build getting triggered automatically reason is i included if any changes packages trigger the build, but when i’m using git diff command not able to get the right modified, can you please give me some input on this.
13

@palani_samy, when you say you are “not able to get the right modified”, can you be more specific? Is the git diff command returning too many files or not enough files? What exactly are you diff’ing?

Now, before I provide a potential solution, I’d like to go back to my first post and once again ask: why are you doing this? I’m still of the opinion that you should avoid doing this if you can. When it comes to CI/CD pipelines, it is best to keep them simple. You are already busy enough managing the complexities of your webMethods integrations, I assume. You don’t want to develop a pipeline that is so complex that takes time away from actually developing and maintaining those integrations.

With that said, consider the following approach (again, there are multiple ways to skin this cat): upon a successful build/deploy, tag your current branch with a tag like “latest” or “lastSuccess” or whatever you’d like. Then, instead of diffing head with the previous commit, you can diff head with that tag. That should give you the changes that haven’t yet been successfully built/deployed.

Hope this helps,
Percio

14

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.