File integration, how much a thing of the past?

,
1

Good day, everyone

I have started a new company recently and the strategy behind this is to “make things easier”. In my years with SAG I have seen many companies reinvent the wheel and quite a number of other things that should not require reinventing.

As an example, I have taken the topic of file integration. This enterprise integration pattern describes the path from data stored in files to data stored in caches, databases, repositories, events, messages or just other files. And even though we live in the era of real-time processing, I have found that many, or rather most, organizations still have the requirement to do this and still spend a lot of time and effort on it.

I am in the process of publishing a new framework (GA mid September 2016) that runs on top of IS, that allows me to create file integrations, using files of any type, format or size, within half an hour. Tried and tested with 10 GB files at 400,000 records per minute with a 250MB memory footprint. The framework allows developers to focus just on that small part of the project that is actually unique to their “transformation requirement”. All the commodity functionality such as parsing, transport, synchronization, reporting; it is already there. Build for performance and ease-of-use.

Now, my question to you, what other challenges have you seen in your organization, or within your customers’ organization, that a) are common as muck yet still not a commodity when it comes to implementing, b) should be a commidity to implement, and c) are not provided out of the box by the platform.

My reason for asking, as surely I have one, is to validate my understanding of the challenges that SAG users are facing. I have seen many at the 50+ customers I worked with, but I am quite sure there are more and that some of them are not as difficult to resolve as one may think. The end goal is for customers to be able to literally install the webMethods platform and hit the ground running.

To start the list, I am already working (and publishing in the coming weeks) the following:

  • A service load testing and performance benchmarking tool that helps understand the impact of running a service under high load;

  • A policy based Flow code design time analyzer and validator (including CI hooks) that helps ensuring code quality and maintainability, and that highlights performance and technical risks (which is especially helpful when development is outsourced);

Your feedback is very welcome and feature suggestions for any of the above as well.

Many thanks, Chris Schuit
(http://www.centipod.nl)

1 Like
2

This sounds interesting indeed. What will be the license of these tools? And how do you develop them? Do you have your private installation of webMethods?

3

Hello Chris,

I have seen one of my colleague working with http://datafeedr.centipod.nl/. I was lucky enough get a glimpse of what this framework can do. Really awesome…!

The way the framework slices the large files and spawn thread (configurable) is indeed “makes things easier”. :slight_smile:

Looking forward to see the other two frameworks as well…!

Thank you.

4

Hello Chris,

A quick question on Data feeder, does this framework can transfer files to another server as well? Just like Rsync or SCP that can be done in Spring Batch Framework.

I am aware of an use case where the customer has implemented a solution with Spring batch framework to load the data into cache, DB, JMS systems and file transfers as well…!

5

Thanks Christian for posting a framework utility tool, please share us once the trial copy is ready for evaluation and feedback.

In addition to the copy, does it support SFTP/S, FTP/S, File Polling (all OOTB feature of wM integrated), File Sequencing, Duplicate file processing detection, Scheduled transmission of files to target system, Digitally signing file while sending to target system. Encryption/Decryption of files while reception and transmission?

Does it cover all the capabilities of MFT or more than MFT?

6

Thanks for the response. To answer all questions:

Q: What will be the license of these tools?
A: The information on the commercial licensing is available at http://datafeedr.centipod.nl under “Licensing options”

Q: And how do you develop them?
A: The framework is built in Java and wrapped in an IS package. To use it, configure a task and create a simple data handler flow service. Demo and templates are provided as part of the package.

Q: Do you have your private installation of webMethods?
A: I do. Currently looking into cloud options as well but when using extremely large files that only makes sense if all your endpoint systems are in the cloud as well.

Q: does this framework can transfer files to another server as well? Just like Rsync or SCP
A: Not yet. The current version supports local and network file systems but it is build as a pluggable framework and I am working on adapters to read files remotely over FTP, SFTP, FTPS, Rsync, HTTP, HTTPS, SCP and the list will keep growing.

Q: does it support SFTP/S, FTP/S
A: See previous question

Q: File Polling (all OOTB feature of wM integrated)
A: The framework comes with its own file discovery (polling) feature that uses OS and JVM level events.

Q: File Sequencing
A: Currently it does support sequencing files (using notifications on start and completion/failure) but processes all records within one file with random sequence. I am working on a bit of logic that allows record sequencing as well while maintaining minimal memory footprint.

Q: Duplicate file processing detection
A: The framework will always process a file once, even in clustered mode, but when you provide the same file twice, it will consider these both as unique files.

Q: Scheduled transmission of files to target system
A: The framework comes with an API that allows you to schedule file transactions using the IS scheduler or any other scheduling framework

Q: Digitally signing file while sending to target system.
A: This is not supported as, even though it can, the framework is not meant for pure file transfer. It is not an MFT solution. This is on the roadmap though.

Q: Encryption/Decryption of files while reception and transmission
A: Not yet but as with the remote file transfer, this can be added to one of the pluggable modules.

I will also add these questions to the FAQ on the website.

Thanks for your feedback. Much appreciated!

1 Like
7

The FAQ is now available at: http://datafeedr.centipod.nl/#/faq

Chris

8

Good work Christian that you did. Thanks for sharing FAQ’s as well. Once the tool is ready for free trial, we can play around and share you the feedback on the tool. Awaiting your further updates.

Thansk,