load file-based properties for each packages

,
1

Hi all,
which is the best way to store/load the properties for each new is-package? Each package has other or maybe new properties to load.

Current Solution: config.properties for each package. a Java-Service that read the config.properties and load it to to Property-Object. If i load the properties from file to the property-object all services can access to the properties of other packages or override some properties (last load-methode wins).

To load the properties each time we start an flow/java-Service maybe not an good idea or?

Have somebody an better solution for that? in the the future we will have more than 10-15 IS-Packages with custom properties

thx

1 Like
2

Having different property files for different package is fine. I hope you are maintaining the data as key value pairs. why do you want to use the same key name in each file that overrides one over the other?

I would suggest the following…

Have one property file for each pkg as want to maintain separately
Maintain data as key value pairs
Write loader services that loads data into the cache as key value
Use get cache, put cache services to read/write data

Senthil

3

Hi,

if possible, the propertie files should be located under IntegrationServer/config/ and the be loaded by the packages.

Benefits:

  1. The files are automatically backed up when IntegrationServer starts up succesfully
  2. The values can be environment specific and will not be overwritten (accidentally) during deployments.

File pathes are usually relative to the IntegrationServer directory.

Regards,
Holger

4

Hi Guys,
i solved the problem like this:

  1. Create a basic Flow-Services which can added to each new/custom package as a startup-Service. This service create a Cachemanager(if not exists) and add this Cachemanager a custom Cache for the calling service. Read the properties from the package/config/myconfig.properties and put the Properties as key/values in the custom Cache.

  2. Write a second Flow-Service which can used in other custom packages. This Service try to get the properties from the cache. if the Cache not exits, it load the properties from the file.

Each new Package has only to create a config-File. Add the Flow-Service(from 1.) as Startup-Service. Call each time you need the second Flow-Service and extract the Properties from the Cache.

Benefits of this Solution:

  • You can see and administrate The CacheManager and the Caches in the Intergrations-Server-Admin-Site.
  • Fast access to all Properties.

If somebody have a better solution please share it with us.

best regards.

Eugen BuchmĂĽller

5

Perfect!

Now that you will have your solution which accesses the key/value from cache. Make sure, the solution when it verifies in cache and if the key is not present, it should verify in file system, and if it finds the key/value, it should put the entry into cache and return back to the solution.

This is to make sure, if somebody has deleted the cache (admin or support etc.,), how would you make sure that the solution is not failing the processing of data?

Otherwise, all good!

-Senthil

6

I would suggest the following (which has been used successfully in multiple places):

  • Store such files in one location. Holger suggested a new folder under the /config directory which is one idea. Another is one I’ve seen used /properties. But DO NOT store them in a directory of the package.

  • Use XML files instead of properties. Key=value pairs are okay but using XML helps in many ways. Define a doc type that matches the XML layout you want to use. Then use built-in services to create a common service, accepting the document type name, that reads XML to return an IS document with the config. Using XML one can create nested documents if desired to help structure the configuration.

  • Don’t use caching. It isn’t really necessary, unless you have some really fast response time requirements and high load. The OS will effectively cache the reading of the file for you–it is very, very unlikely that your service speed is meaningfully impacted by reading a local/mounted file.

  • As Holger notes, these files are typically environment specific. NEVER use automated deployment techniques, such as Deployer, to deploy these files. Always create/update them directly in the environment they exist.

  • Keep it simple. Don’t over-engineer it.

7

Hi Rob,
thx for your recommendations.

  • Store such files in one location. Holger suggested a new folder under the /config directory which is one idea. Another is one I’ve seen used /properties. But DO NOT store them in a directory of the package.

Why not store the property-files in the package? At the moment i only see the disadvantage to store the properties in the Package/config/myconfig.properties that you cant edit/see the file directly in the Eclipse Designer. But this i also have with a centralized properties-Folder.
Each package has his custom properties. If i add a new IS-Instances i maybe want only some package active and check them out. How can somebody know which properties have to be loaded or move to IS/properties…? In my opinion the properties also have to be saved in a version control system.
Can you please explain how your suggestion work in practice?

  • Use XML files instead of properties. Key=value pairs are okay but using XML helps in many ways. Define a doc type that matches the XML layout you want to use. Then use built-in services to create a common service, accepting the document type name, that reads XML to return an IS document with the config. Using XML one can create nested documents if desired to help structure the configuration.

[b]I transform the loaded Properties in a Document with key,values. I found xml-Files harder to create and maintain.

In my flow-Services i do following:

  1. Call getProperties-Service which return a Properties-Doc.

If i need a Property i do this:

%PropertyDoc/ftp.hostname% or %PropertyDoc/ftp.targdir% [/b]

  • Don’t use caching. It isn’t really necessary, unless you have some really fast response time requirements and high load. The OS will effectively cache the reading of the file for you–it is very, very unlikely that your service speed is meaningfully impacted by reading a local/mounted file.

OS will cache the reading file? I dont see this. Each time i read the Properties from File, all Property-Data are readed from file and not Cache. You can maybe set the flow-Service to Cache the results, do you mean this? If not please explain when is what been cached from file. I use a Windows-System.

  • As Holger notes, these files are typically environment specific. NEVER use automated deployment techniques, such as Deployer, to deploy these files. Always create/update them directly in the environment they exist.

    But what if i delete a property which is needed by an package or if i install a new IS-Instance.

  • Keep it simple. Don’t over-engineer it.
    [/quote]

Agree with you :wink:

Thx for your tips.

8

I create a flow-service which try to get Values from Cache(Cache.get…). In The Catch-Block i call the getProperties-Service direktly from File. Try it in the admin-site from the is and it work like a charm :slight_smile:

9

Each configuration file typically has environment specific settings in it. For example, a URL. Which will differ in dev, test and prod. If you deploy the configuration file along with the package, then your production environment will have dev/test settings in it. This can be disastrous, even if configured that way for just a short time.

Config files don’t change often. IME, there is no need to automatically deploy them. Just create/update them in place. If someone is deploying a package to another IS, they need to know the conventions and know that there is a config file to replicate too.

Adding new IS instances doesn’t happen that often. In my current environment, we haven’t added a new instance in years. And the instances we have do not share most packages. There are common packages that are on all, but in general we do not put packages on multiple servers. Of course this will vary based upon how you scope what goes in your packages.

That’s okay. I do not like using variable substition. So I map from the source document to the target var. XML is indeed more verbose but it also much more flexible. And you don’t need to create your own Java services to use it.

Yup. It caches reads and writes. You just don’t see it. But that’s really beside the point. The main point is that reading from the disk, even if it is a physical read, is not a significant time concern. You’ll spend far more time figuring out why something didn’t work (“oh, I forgot to clear the cache!”) or debugging. Just read the file every time.

But perhaps I’m misunderstanding how you’re using “Cache” and the “CacheManager” you mentioned?

Log into that server and update the file directly. For a new IS instance (again, this should be rare), replicate the file to that instance. It really isn’t any slower than setting up Deployer to deploy it–and doesn’t have the disadvantage of incorrectly putting non-prod settings in a prod environment. Deployer has ways to manipulate the file for deployment but when someone forgets to set that (when, not if) then you’ve got a problem. I have two rules that I always try to enforce:

  • Never deploy properties/configuration files. Create/update them in place.
  • Never deploy adapter connection pools. Never define them in the package that uses them (use a common “Connections” package for all. Never deploy that “Connections” package. Create/update adapter connection pools in place.
10

Tundra includes a package configuration service tundra.configuration:get which returns the configuration settings associated with the given package.

This service supports the tiering of configuration settings across a package-specific file and a server-specific file, where both files are merged with the server-specific file taking precedence. This approach allows for settings which are not environment-specific or for default settings to be stored within the package itself, and for other environment-specific settings or overridden settings to be stored in a configuration file specific to an Integration Server instance. Note that if this is not required, a single file can still be used.

Package-specific settings are loaded from the following file:

./packages//config/package.

Server-specific settings are loaded from the following file:

./config/packages/.

Where is the name of the package in question, and is one of the following supported file extensions:

  • hjson - Hjson formatted file
  • json - JSON formatted file
  • properties - Java properties formatted file
  • values - IData or Values encoded file
  • xml - Arbitrary XML or IData or Values encoded file
  • yaml - YAML formatted file

Package configurations are cached lazily in memory: read first from disk on the initial invocation, and then returned from an in-memory cache thereafter.

Prior to a configuration being cached in memory, variable substitution %key% substrings found in configuration values are automatically substituted as follows:

  • Keys prefixed with $system are resolved against the $system document returned by tundra.system:reflect.

This approach allows for server-specific global configuration values stored as environment variables, java system properties, or global variables to be included in the returned package configurations. Note that because the variable substitution occurs when the configuration is first read from disk and cached (to optimise performance), changes to environment variables, java system properties, or global variables will not be reflected in the returned package configurations until those configurations are refreshed from disk.

Inputs:

$package is the optional name of the package whose configuration is to be returned. If not specified, the package is automatically inferred from the invoking service.
$refresh? is an optional boolean which when true will force a refresh of the cached configuration from disk. Defaults to false.

Outputs:

$package is the name of the package whose configuration was returned.
$configuration is an IData document containing the configuration settings for the given package.

Refer to ConfigurationManager.java for the underlying Java implementation, if you are interested.

11

Dears,

Use ehCache mechanism from wmRoot packages

OR

Use Databases for defining properties and call and use them when it is required.

OR

Download WmSamples package which has the Java services for getting and loading properties…

WmSamples.zip (471 KB)