Email attachment parsing using the SMTP listener in wM 60

, ,
1

I have wM 6.0 and need to parse a mail attachemnt which is
a .CSV.

HAs any one done this?

Thanks.

Vishal

2

Vishal,

You start by defining an email listener “port” using the IS
Admin tool.

When the port receives an email from the mailbox you
specify, it will attempt to invoke the flow service
specified in the subject line for each attachment after
first invoking the appropriate content handler. Specify the
service in the form folder:service, for example,
“EmailTest:receive”.

In prior releases you had to create custom content handlers
for CSV files, but I believe that CSV files are now handled
by the default content handler.

To process your CSV file you would create a “handling
service” that has as its input an object called ffdata which
is the output field created by the default content handler.

You can use the pub.flatFile:convertToValues built-in
function to convert the contents of your CSV file into a
record structure (IData object) using a flat-file schema
that defines the file structure. You can optionally
validate the structure of this file against a document or
record structure. See the documentation on Flat File
Handling for instructions on how to create flat file schemas
and flat file processing services.

Hope this helps,

Mark

3

Mark,

Will try this and let you know.

Thanks a Ton.

Vishal

4

Vishal,

One other thing is necessary in order to make this work.
You should set the content-type of your email to
“application/x-wmflatfile” in order to invoke the flat file
content handler.

I’m not sure, but I think if your partner was not able to
configure their email program to use this content-type, a
workaround would be to edit the mime.types file located in
your IS server’s \lib folder to add a line that associates
files with the csv extension with the
application/x-wmflatfile mime type.

I did finally locate the documentation for the flat file
handler functionality. It can be found in the Flat File
Schema Developer’s Guide, which is located in the
\Developer\plugins\FlatFilePlugin\doc folder. Chapter 5 of
this document addresses invoking the flat file handler by
submitting a file via email.

Mark

5

I had follow the Flat File Schema Developer’s Guide to try submit via ftp. And I write a flow service invoke savepipeline. I put a flatfile to the service then restorepipeline. I found a string type name ffdata in the pipeline. Is it a object type name ffdata ?

6

the savePipeline and restorePipeline services have some limitation on the types that they can ‘clone’. If the ffdata does not support one of the interfaces that those services rely on, it will be turned into a String but the savePipeline. The Builtin Services Guide should describe the supported interfaces/classes.

7

If the ffdata was not a object type then I can not convert it to IData by convertToValues services.

I had add the csv extension with the application/x-wmflatfile mime type.

I don’t know what I lost.

8

You need to create a flat file schema first before you can parse the file with pub.flatfile:convertToValues.

The input to the service is

ffData - String
ffSchema - fully qualified NS path to flat file schema

Other switches you may want are:

validate = true
returnErrors = both (or AsArray)
maxErrors = -1 (get 'em all)

Hope this helps.

9

Mark,

I have some problem while receving a flatfile via email Attachment to IS.

Attachment file is in .dat format

So I updated the IS/lib/mime.types file and included the line
application/x-wmflatfile dat

and restarted the ISServer.

The inbound mail is invoking a globalservice (testemailReceive) which has input set to *ffdata. (we put savepipe/restorepipe and nothing has been logged in that)and also used getTransportInfo service and we got all the header details of email but not content.

We are seeing error in the Server log:
ISC.0076.0007W] XMLCoder decode invalid data type: java.io.ByteArrayInputStream

Pls help us,its urgent…

TIA.

10

RMG,

(BTW, when are you going to register your WM Users account, anyway!)

Can you put a debugLog statement in the service that is identified in the subject line of your email to confirm that it is being invoked?
What client are you using to send the email for your test?

Mark

11

RMG,

What are the settings for your email listener? What values do you have for

  • “Invoke service for each part of multipart message” [*]“Include email headers when passing message to content handler”

Mark

12

Thanks Mark for your response,

We have configured the email listner and selected the options as:

“Invoke service for each part of multipart message” —YES
“Include email headers when passing message to content handler” --NO

Hope this is correct and we are not still fully success to receive ffdata object in the pipeline instead we are getting (ffdata as String with value java.io.ArrayInputStream)

And we opened an SR with tech support

TIA.

13

We are using LotusNotes as email client and this will be used by our end users too…

14

I think the error you are seeing in the logs is actually from the pub.flow:savePipelineToFile statement itself.

I created a testEmailReceive service that accepted one input, an object called ffdata (no asterisk in front of it). I converted that from a stream to a byte array using pub.io:streamToBytes. I then converted the byte array to a string for debugging purposes using a map statement with a bytesToString transformer.

Once I saw that my string contained the data from the file attached to my email, I called pub.flatFile.stringToValues mapping my string variable to its “ffData” input variable.

I dropped the ffdata object that was my original input variable then invoked pub.flow:savePipelineToFile. The record that was created from parsing my flat file was in the pipeline as expected. No XMLCoder errors appeared in the server’s log.

I sent my test email from another flow service and named the attachment with a .DAT extension.

HTH,

Mark

15

you are correct pub.flow:savePipelineToFile is causing the error and showing in the serverlog.

Then we tried with pub.flow:savePipeline/restorepipe which gave us the
(ffdata as String with value java.io.ArrayInputStream) with no error in the serverlog.

Wondering still what is causing the problem.

And tech support people suggested that to install Developer6.01 service pack1 on our machine.So we are planning this test on monday.

TIA.

16

Mark,

I sent my test email from another flow service and named the attachment with a .DAT extension.

Did you send test email from outlook/lotusnotes or via smtp built service?

I believe client is not the problem some thing wrong in the server side itself.

17

The attached example includes two services. One uses pub.client:smtp to send an email with an attachment containing a flat file. A sample flat file “contacts.dat” is located in the \pub folder.

The second service receives the attachment and processes it using a flat file schema based on a flat file dictionary. The schema and dictionary are included and can be used to successfully parse the simple contacts file.

See the “readme.txt” in the \pub folder for additional details. You will need to create a valid email listener (port) using the Admin tool in your IS server.

Mark
ConnevaEmailExample.zip (16.1 KB)

18

Thanks for the sample provided.

we did created valid email listener and the global service is being invoked by the email client (LotusNotes) but only issue here is we are seeing (ffdata string which has value of java.io.ArrayInputStream) instead of stream object in the pipeline though no errors shown in the serverlog.

we tried attachment as .txt as well as .dat

I will update you if we are success.

19

Mark,

Problem got resolved,and we are getting ffdata stream object in the pipe.

Actually if we put savepipeline as a first step that is making ffdata as a string we checked this in restorepipe step.

So we moved savepipeline as a last step under the StreamToBytes,bytesToString and saved the flatfile string in a file which wrote the file data as expected.

So may be we need to install developer servicepack1 to view ffdata as a object instead of string and move the savepipeline to first step as it should be.

Thanks for your concern during the weekend time.

HTH.