My company recently installed a new webMethods setup, in which we feed a large number of EDI 835 files to be processed/validated and then deliver them to our A/R system. We are experiencing problems with what I can only describe as ‘file handling’. Two symptoms:
We drop several different files at the wM pickup location which have different content and checksums, but the processed files all have the same content/checksums. This does not occur all the time, just sometimes, usually when we have many hundreds of files being processed. When it happens it usually results in two duplicate files, but on occasion as many as 5 duplicates come out of the system.
We have files that get logged as being processed, but later are somehow lost, and we only get an error message that reads something like: [ISS.0086.9256]File[/usr/local/wmedi/edi835/prod/processing/SPLTY_0803200309.cntl]doesnotexist
Again, this seems to happen only when volume is high, not when just a few files are being processed.
These both seem to be the product of some file handling issue. Is this common? Does wM use a queue position to identify files and can that somehow get ‘mixed up’ when many fiels are being processed?
From your description I am assuming that we are talking about file polling port, right?
Did you try to tweak the settings on the file polling port config?
See IS Administrators Guide for further informations.
You might want to consider decoupling the file handling from the processing of the content by using a messaging component (either Broker or UM).
After you have read in the file publish it to the messaging comnponent and then retrieve it with a trigger for further processing.
Thanks Holger! I am not 100% sure of the version, but it was the ‘latest version’ when we started the installation 6 months ago. Also, I should say that I am not an IT support person myself, but a business analyst. Our company (Fortune 50 organization) uses 100% offshore support supplied by a contractor, and getting them to even admit this problem exists, much less work on a fix, is difficult. So I am sort of fishing for hints to give these guys, hoping that if some percentage of the analysis is done it may spur them to action. I will ask if we are using the messaging component as you described…
By decoupling via messaging the processing of the content does not block the file handling part before the file is moved to the processed folder.
The “file does not exists” means that the file was already processed when IS gets triggered to process it once again.
Additionally to checking the file polling port config the IS Resources config (i.e. number of threads) should be checked too.
Furthermore the jvm memory settings of the IS should be checked as well.
I agree with Holger on the configuration checks (tuning IS resource threads) and also it seems to be another opportunity to implement using reliability pattern techniques (utilizing Universal Messaging component) be in-place especially when dealing with heavy volumes of files processing and handling duplicates (Idempotency methods) as you were noticing on few occasions.
I am pretty sure your contractors may have already implemented moving the processed files to complete/archive/error folders check based on the file operation status and send alerts to Ops team if any errors based on the business needs etc…
Please run this brainstorm with in your and team and troubleshooting.
When using messaging component for reliabitlity you can set some parameters on the IS trigger (related to Document History which is part of the IS Internal database component).
See the Properties Panel for the Trigger in Designer (check for the section “Exactly Once”).
See the Working with Designer Guide and the Publish/Subscribe Guide for further informations.
When using messaging component the IS and the Messaging component handle the frequency by which the documents can be processed and the messaging component persists the documents until IS was able to process them successfully.
Only thing which can cause trouble here is that the file polling and publishing might not be fast enough.
An option here might to have two IS connecting the same messageing component, one doing the file handling and publishing to the messaging component and the other one subscribing to these published objects and do the remaining processing.