We have a project and determining if the validation process should be done in webMethods IS or not. At a high-level, this project will use an ETL tool to load the data into a database. The next process is to validate the content of the data and each partner will have its own unique validation requirements. It can be simple or complex based on the partner. Does anyone have any recommendation on whether this should be in using IS/TN or does wM have a specific product that would greatly make this more “effortless.”.
Are you saying webMethods will be used as ETL tool in your project instead of widely available tools Informatica or SAAS or Hyperion or BO?
What type of data file received from your partners and trying to do validation portion?
Just trying to understand the scope:
HTH,
RMG
RMG,
From an ETL perspective, we have a process for that. I’m thinking maybe that should also be part of the ETL tool is to perform the validation step as well. Before sending the data to the target system, the data has to be validated first. Validation can pertain to field or file validation. Something like as follows:
Field Validation:
- System must remove all punctuation (dashes, commas, periods, etc.)
- System must remove all unnecessary spaces.
- If field is < 9 digits, auto-filled with leading zeros.
- System must validate that either field1, field2, etc. are populated.
File Validation: - If more than 20% of the records receive critial errors, the entire file will fail.
As you can see, the list could be lengthy or not based on the partner requirements. I’m usre you could do this in IS/TN but is there a better method or better tool design specifically for data validation. I’m assuming not considering this was brought up.
I don’t see any other tool design unless ETL itself has some type of validation capabilities may be they do newer versions…Yes wM can perform those mentioned validations depends on what type of file (flat file or XML or EDI??) standard you are receiving from the partners etc…this way you will have some idea and determine where the validation step can be performed error handling in a easier/faster solution. Just my few cents:
I know this can be done at the IS/TN level but that would some knowledge of developing the flow services to perform the validation. I would at least thought the SAG might have a product that could “easily” handle the validation process and not have to write complex flow language.
I recall a product called Item Field does that complex validation/processing which Informatica brought it and that product has some integrated capability can talk with webMethods.
You can do that any field level validation creating via flat file schema (via format service specification/template) part of WmFlatFile parsing services isn’t?
HTH,
RMG
Most of the steps described are not “validation.” They are transformation/normalization. The steps identified are modifying data so that is consumable by an end-point.
If an ETL tool is being used for this then the steps you’ve described should be implemented using the ‘T’ part of the tool. And should likely be done before or as part of the ‘L’ portion of the work.
Since an ETL tool is already in the picture I’d use it for the entire process. The value of using wM IS in this case is suspect.
Rob those steps can come under validation/transformation/data cleansing and yes wM part can be eliminated unless ETL tool deals with it and I think they do have those capabilities handling data cleansing.But how would they received it from partners?