Source control and release management

, , ,
1

Hi,

What does release and source control management look like at your place?

In my personal experience, webMethods code tends to degenerate to an unmaintainable mess that no person in his right mind would like to modify – mainly thanks to zero change traceability, no diff functionality, no efficient search functions, no refactoring functions, minimal support for unit testing, no data hiding etc.

So given the task of synchronizing several developer teams working on the same code base, what is the best way of implementing release management and source control without having to introduce a giant Excel document saying “Team X is working on package Y”?

2

Developer has (minor) refactoring functions, but I suppose I’m nitpicking. :slight_smile:

An approach that I’ve seen working reasonably well at a couple of places:

  • Artifacts managed at the package level (not service level).

  • A package is focused on a source, a target or a “bridge” between the two. E.g. the simplest integration might have 3 packages. Some places have a single package for an integration, to include source(s) and target(s) and integration control, but I think that’s too much. As with most things, strike a balance between too coarse grained and too fine grained.

  • Packages managed by a VCS. One client deployed to non-dev environments using a custom deployment package that could get from VCS the version of a package as defined in a deployment config file.

  • Package versioning was critical. A change log file was maintained in the package’s pub directory.

  • A package is checked out to one person at a time. If multiple people needed to work on a package at the same time 1) they coordinated their effort; 2) packages sometimes refactored to minimize the need for multiple people to work on it at once.

  • Checking in changes included repeating the material put into the package change log in the VCS check in. Change descriptions could be viewed easily in VCS or in the package itself.

It’s all very reminiscent of old-school change control/tracking, but that’s more or less what’s needed. Once people get used to it, it’s pretty low impact but effective.

3

Well, you basically outlined a tedious manual process – not unlike the way it used to work in most programming languages during early 90:s. The issues we’re having are still:

  • Manual checkout, basically maintaining a huge list of package versions in all environments (dev, test, prod etc.) that must be manually kept up to date.

  • No way of automatically backporting code changes to all environments (normally done manually by creating and executing custom deployer scripts). The development will most certainly be done on outdated code base.

  • No feasable way of doing development in parallell on same packages. Two developers accidentally checking out and working on the same code package will potentially mean weeks of lost work effort.

  • No way of merging changes or tracing committed changes on code level, only on package level. Keeping a manual code change log is the only option, which is considered to be an antipattern in modern development practices.

So once you have more than one development server and development teams, you will get problems with traceability, testing, maintenance and release management, hugely relying on manual routines. It can be manageable but certainly not effective!

4

You bring up some good points. I guess we’ll see how things evolve with the dev tool now being based on Eclipse. As I mentioned, the process is definitely old school.

You asked how the issues you listed might be addressed. I outlined an approach. It would be nice if there was some automated assistance for some of these but there is not.

Regardless of the environment/team size one needs to establish policies and procedures to avoid creating a mess. These might be more manual than you’d like but it is what it is.

Touching on each of your points:

  • List of package versions – A tactic here is to keep versions/branching to a minimum. I’ve been in JEE environments where branching has run amok and a full time person was dedicated to basically merging branches. There is a balance to be maintained. For IS development, the balance needs to tend toward minimal/no branching.

  • Backporting/dev on outdated code – As backporting would need to be manual, one tactic is to avoid it. Try to always work on the latest and move it forward. Not always doable, but will reduce the amount of rework. (same as first bullet)

  • Parallel development – It is feasible. I’ve done it. It isn’t a “VCS helps you merge changes at check-in” approach but rather a coordination between developers working on the same package. Having developers coordinate/collaborate isn’t a horrible approach. As far as 2 or more people checking out the same package, that’s easily addressable–configure the tool to not let them. Only one can “check out for edit”.

  • Code merge – Alas, the only way is manual with zero tool assistance. The tactic here is to avoid/minimize the need to do code merges (back to the “avoid branching” again).

Your post seems to imply that the lack of these facilities means doom. And that manual processes would be an undue burden. In my experience, they are not. They may not be as efficient as you might like (or perceive) but they can be quite effective. Establish the procedures/rules and make sure everyone knows them and follows/enforces them. Basically it comes down to developer discipline. Not necessarily ideal but not the end of the world.

5

Sorry, I was actually overly negative – and I’m definitely not trying to accuse you of anything, I’m really thankful for your response :happy: Just a bit frustrated, that’s all.

I think the issues I’m talking about are manageable if you have a localized team of developers, or a few local teams sitting in the same room.

Problems arise once you have several development teams on different locations, or when you have a high release load and have to test several releases at once. Doing parallel development on the same package, or even parallell releases on the same test server normally means doing one big development job, since all developers will affect each other when testing etc. – with no good way of keeping the code / releases separated.

It’s all or nothing: if you’re not testing one release at a time, then a difficult bug can clog up the whole release flow for a week or more, or require a full rollback/retest of several release candidates.

This is not a unique problem for webMethods specifically, but it’s much more pronounced.

I guess we have rather specific requirements: several highly independent development teams and as a result many environments that must be kept updated, and a tight production release schema on top of that (once a week or more often).

However, I think the points are still valid for “normal” projects as well. Automated tasks that are taken for granted in many languages today are entirely missing or poorly implemented, making the maintenance costs unreasonably high. At the end of the day, maintenance costs are much higher than development costs in almost any development project. The heated debates here on wmusers about whether to use java or flow for development actually doesn’t mean that much compared to the importance of release management, source management, built-in documentation and refactoring tools and other project infrastructure (the boring stuff, I know :slight_smile:

Do you have any experience with some third-party tools that could help ease the pain? iTKO Lisa, Greenhat, Team Vista?

6

I have not had the opportunity to use those tools. Perhaps others have an can chime in.

7

Have you had an opportunity to trial CrossVista Team? It looks like a 30-day trial is available. I’d be interested in hearing feedback about it. Based on the descriptions it seems like a decent tool.

8

Not yet but it certainly looks promising! I just need to find time to do it.

9

Hi Reamon,

I have been working with crossvista for about 2 years now here are some of the FAQ’s

-Is this product a reliable Configuration Management System? -yes its relaible from version 1.5 onwards. It is compatible with any VCS in the market like subversion and CVS. The performance was bad pre 1.5 as there were loads of bugs. Had found a few myself.
-Any objective opinion about it? Pros-Cons?
[COLOR=Red]-Pros

  1. Its is pretty good in how you can deploy your deployment sets. You can set domain paramaters on the fly e.g your configurations in QA and PROD might be different it can help you do that while deploying.
  2. Doesnt make your IS’es talk directly. It used a Subversion repo to deploy sets across different IS’es across environments.
  3. You can revert to any deployment set which has been done on TEAM so you have total control over deployment sets.
  4. Good user accountability, i.e. you can give accesses like Developer/Admin/Operator/Auditor based on your need.
  5. If you have a clustered environment you can do site comparisons to find differences in configurations directly.
  6. Auto deployment feature is cool…i.e you can schedule your deployments for a future period and forget about it. Cross Vista will do it in ease.
    lot more are there actually
  7. You can compare flow services with a previous build and with the ones running live.
  8. Concurrent development is possible where 2 users can use the same package which is not possible in webMethods directly.
    -Cons
  9. The user management is very bad and has much to be desired.
  10. Additional administration overhead is required to Administer Cross Vista TEAM.
  11. Product is relatively new so there are a few bugs that keep propping up.
    [/color] -Any concurrent product?
    none that i know about.
    -What’s about traceability regarding Requirements/Change Requests?
    This is not a Change System. It is a VCS with Release management capabilities revolving around web Methods.
10

FWIW, I have written a “Flow Printer” program that dumps out a flow service as a readable text file. It’s not shrink-wrapped – I run it via ANT. The output isn’t 100% complete, e.g. there isn’t enough detail to regenerate the flow XML should one want to “compile” back the other way, but I’ve found it invaluable for DIFFing, or just understanding a flow without clicking through a bizillion dialogs.

If anyone’s interested I could upload it (I assume there’s an upload facility here somewhere?)

11

hi mlemaire,

I am ready to take a leap of faith…upload what you have and we can see if it really works??

12

Nightfox,

I’ve uploaded FlowPrinter to the shareware area. I tidied it up a bit so it can be run as an executable JAR. It has external dependency on Apache commons logging and log4j – I didn’t include these but they’re easy enough to get hold of.

Have fun!

13

Well, I uploaded the FlowPrinter quite a while ago now and it still hasn’t appeared in the downloads area. My emails to shareware@wmusers.com are being rejected for some technuckle reason. Perhaps someone here has more influence and can help me sort it out?