Putting literal values into source code has long been considered poor practice. For flexibility and code stability, values that can change from time to time should be placed in an external location, such as configuration/properties file or in a database table.
This applies to Integration Server solutions too. URLs, hostnames, IP addresses, directory names, user accounts, and so on are great candidates for “externalizing.” That way, they can be easily changed as the solution migrates from development to test to production, or if partners change their solutions, or as new partners are added.
I wonder, however, if we over do it sometimes. I’ve seen cases where there are properties that externalized, but they never change. Ever. Oh, they could, but they don’t. I’ve seen other cases where the approval/testing/validation process for making a property change is as intense as making a code change. One has to ask, what’s the point then? I’ve seen externalization run amok where there was no real standard approach, just every developer for him/herself and finding where %foo% is actually loaded from can be a real treat.
As with just about everything that provides flexibility, complexity is also introduced. Externalizing variable values most definitely provides flexiblity (and all things being equal, is a good thing to do) but it also adds a level of indirection that those that come behind you have to figure out. I’d propose that before one just blindly externalizes everything “just because it can be” that each integration designer out there take a hard look at the value of doing so.
So, best practice recommendations for managing configurable properties might be:
Never hardcode values into a Flow or java service that might need to change frequently or somewhat frequently. Common examples of these would be URLs, hostnames, IP addresses, directory names or user accounts.
Develop a configurable properties management approach and use it consistently. Try to avoid “over engineering” the approach. Keep it simple.
Use a logical naming structure for property keys (property names)
Provide the ability to change properties at runtime without restarting the IS or performing other manual intervention (except for making the property value change itself)
Avoid making something a property if it is likely to never change
Maintain a history of property values either by placing properties files into source code control or archiving old values in an archive table. (This will help support your request to have a “light weight” change control process for properties changes because you can roll back to the previous value more easily
If the same value is going to be retrieved frequently, use service caching. However this will somewhat violate Mark’s rule number 4 unless you are ok with waiting for the cache to expire.
Actually, my current properties management approach involves providing a separate properties file for each package. The files are always named consistently and stored in the package’s config folder.
The goal is to provide a simple UI that will allow the properties to be edited (similar to the IS Admin Extended Settings UI). When the file is saved the UI will also invoke a web service to refresh the cached properties data (in our case by reloading them into the system properties).
I guess by manual intervention, I was thinking of manually editing a flat file, copying it to the right folder and then manually running a service to refresh the properties cache.
This is obviously a tradeoff between adding too much complexity while avoiding too many manual steps in production. The web page approach also makes it easier to restrict access to only authorized individuals.
I am running into a similar issues with code conversions. I have externalized them into a properties process within SQL Server and save properties to XML and XML to cache which works ok for the most part. I had idea regarding the use of a transformer as a conversion utility. I wanted to use the link to the to: element to determine what conversion to use for my in memory conversion as part of the value pair. Looking through the java API I have not determined how I can determine the namespace of to: linked element. Any ideas? Not trying to over design. Just trying to make it easier to manage code conversion into our canonical documents.
Having recently been bitten by a hard-coded value that changed, I’m trying to bring myself up to speed on property files and how they function. I have some values, hostname of the SMTP server, for example, that I want to be the same for all services in all packages on a given IS. Others, such as mail recipients, could change from package to package. I know that server-wide settings can be placed in the %IS_DIR%\config\server.cnf file and that package-specific properties can be put in a properties.cnf file in the package’s config directory. My questions are:
How do I reference these properties in a flow service? Does each service in the package need to call the service to load that package’s property file?
Can I have properties for a package that have the same name as a system-wide property, essentially overriding it for services in that package only?
When are package properties available to reference? Only when a service in that package is executing?
I saw a post by Mark on Advantage that included a java service for loading a specified properties file into the system properties. When would something like this be invoked?
If there’s some documentation I should be reading instead of bothering folks here, please let me know. I looked here and on Advantage without finding a lot. Thanks,
I use a utility java service called getSystemProperty to retrieve the value of a specific property. This service can be called as a transformer so that you can get multiple properties in a single MAP statement
My approach has been to load the properties for a package into the system properties on package startup (or reload). This allows me to avoid maintaining cached properties variables, but would not support your desire to overload properties.
As mentioned above, I load the properties for a package into the system-wide properties at package startup. That makes all of those properties availble system wide. Other approaches to narrow the scope of properties certainly exist.
I add a small Flow service that loads the properties in each package and then make that service a startup service for the package.
You mentioned that your solution involves each package having its own properties file in the package’s config directory.
How have you addressed the potential problem of overwriting a Production properties file with a QA file, for example, when deploying a Full release of the package? Do you simply not perform Full package releases? Ie. Only patch releases are allowed OR builds are created from invidual components stored in version control?
If you could elaborate on your solution, I’d extermely appreciate it.
Nothing fancy, we just use patch releases to avoid overwriting production properties values. Because property file persistence is isolated in only a couple of services, chaning over to DBMS or LDAP storage of properties would not be too difficult. Doing so would allow us to make “environment” part of the key for retrieving property keys and values on server or package start.
An approach I’ve used is to put the values for all environments into a single XML file grouped by environment. Then, var placed in server.cnf to indicate which environment the server belongs to (dev, test, prod, etc.). Then the service that reads the vars reads the apprpopriate group. Works well, though storing passwords this way isn’t a good idea.
To address the problem of properties changing on code promotion:
You can store properties in the repo using the pub.storage:* services. As the repo contents don’t migrate with package deployments, full package releases won’t change anything on the server.
Another possible approach I’ve explored - though not tried in production - is adding entries to the server’s properties.cnf file. You can add arbitrary properties as long as they’re prefixed with “watt.” (if the prefix isn’t used, they’re simply discarded by IS). For eg:
It has been a very good thread, i usually use to get properties from a file. this method looks better since my service hadle good number of transactions. can you please let me know if there is a new property to be added and after adding new property do we need to reload the package or it automatically gets the updated file into memory.
Does this property file need to be placed in Pub folder.
If we change the existing property how does this going to be handled?
If we add new property how does this going to be handled?
If we need to reload the package for uploading new file everytime, does the ongoing transaction get effected?
