SFTP Remote folder, and pub.client.sftp:get by older modified date

Nezha_Abrabri · January 25, 2016, 11:06am

Hi community,

please i need your help , what we want to do is :

1-get all files frome remote folder using sftp
2-put those file in my local folder
3-loop files in my local folder by date and pattern (example : Loop–get files which name like yyyy-dd-mm_name.txt and yyyy-dd-mm correspond to the older date–)
4-get data from file and execute some services
5-remove files frome remote folder
6-move files frome local folder to archive

[b]Any suggestion or help how to proceed would be greatly apprecaited.

Thanks, [/b]

Holger_von_Thomsen · January 25, 2016, 11:32am

Hi Abrabri,

see following code snippet (partially Pseudo-Code (might be incomplete regarding StreamToString transfortmations etc.)):


pub.client.sftp:login
pub.client.sftp:ls
LOOP Start
pub.client.sftp:get
pub.file:writeToFile
pub.client.sftp:rm
LOOP End
pub.client.sftp:logout

Get File-List according to Pattern
LOOP Start
pub.file:getFile
execute logic
pub.file:delete
LOOP End

Remember to update WmPublic/config/fileAccessControl.cnf with the path to your localFolder and reload the Packages to use the pub.file:*-Services.

Regards,
Holger

Nezha_Abrabri · January 25, 2016, 11:41am

thank you Holger for yr reply,
but i dont have pub.client.sftp:list in my built services??

i think by doing * in remode folder can load all files but in my local folder how procceding them according to Pattern and by the older date???

Regards,
abrabri

MR_as173d · January 25, 2016, 11:48am

Abrabri – You can ls command to list the files, redirect the file names to some file and loop over it. Give a try.

Thanks,

Holger_von_Thomsen · January 25, 2016, 11:50am

Hi Abrabri,

which Version of wM are you using?

The mentioned SFTP-Services are available from wM 9.5 onwards.

Previous versions of wM do not support SFTP out-of-the-box, only FTPS.
FTPS can be handled by pub.client.ftp-Services.

For selecting files by Pattern you can use JZooWildcard-Filter from JavaZoo.
It is public domain open source.

Regards,
Holger

Nezha_Abrabri · January 25, 2016, 11:56am

thank you MR as173d i didn’t understand “redirect the file names to some file”,

think you Holger : the version of my WM is 9.7 attatched all services i have under sftp,there is no pub.client.sftp:list .

Regards

MR_as173d · January 25, 2016, 12:02pm

what I mean is when you use ls command you will get file names in that directory ( Use ls to get files of that pattern) and re-direct to some file, see below one.

ls > FileNames.txt

Now you have all file details in FileNames.txt. Now loop over this file and perform your business logic.

Thanks,

Nezha_Abrabri · January 25, 2016, 12:19pm

thank you,
what i want is :

ls all files thas respect a specific pattern
and move returned files(respecting pattern) of ls command to my local folder, and then get one by one using theire date of modification.

and finnaly use the content of each file : i dont have proble with this step.

any suggestion.
Regards

MR_as173d · January 25, 2016, 12:36pm

Use the below code:

ls <file_name>_yyyy_mm_dd> > FileNames.txt
use java streams to read above file line by line
Move each file to your local
As you have file names in FileNames.txt and files in your local, loop over FileNames.txt using java streams and perform your logic.

Thanks,

Nezha_Abrabri · January 25, 2016, 12:39pm

thank you MR as173d ,
i will give it a try and tell you if it works or not;
Regards,

MR_as173d · January 25, 2016, 12:48pm

Sure, I gave only pseudo code, you should implement it.

Thanks,

Holger_von_Thomsen · January 25, 2016, 4:35pm

Hi,

here is the description from the IS Build-In-Services Guide for pub.client.sftp:ls:

pub.client.sftp:ls
WmPublic. Retrieves the remote directory listing of the specified path. If path is not
specified, the pub.client.sftp:ls service retrieves the file listing of the current remote
directory. The pub.client.sftp:ls service also retrieves additional details such as permissions
and ownership information.

Input Parameters
sessionKey String Unique key for the current SFTP session. The sessionKey
parameter is returned by the pub.client.sftp:login service.
path String Optional. Absolute or relative path of the remote directory. If
no path is specified, the pub.client.sftp:ls service retrieves the directory
listing of the current remote directory.
You can use the wildcard characters asterisk (*) and question mark
(?) after the last slash mark (/) to view all remote directories that
match the specified path.

Output Parameters
returnCode String Standard SFTP protocol return code.
returnMsg String Text message describing the return code.
dirList String List List of directories matching the pattern specified in the
path parameter.

Usage Notes
You cannot execute SFTP commands in parallel using the same session key.

There is no need to redirect the Output of ls to a new file, the service will return a list of matching files in parameter dirList.
You can use this List for your loop.

Regards,
Holger

Nezha_Abrabri · January 25, 2016, 4:42pm

hi,
thank you for yr reply,

but i need to use the content of each file returned by ls, the content dosnt returned by ls???

and copy every filed returned by ls to local folder

ps: idon’t find yet how to use ls with pattern date , when i do namefile* it work good but its not what i need, what i need is to do like shell command : ls namefile which respect general pattern name_yyyy-mm-dd.txt

Regards,

Holger_von_Thomsen · January 25, 2016, 4:53pm

Hi Abrabri,

please check the pseudo code from my first post (meanwhile corrected with ls instead of list).

What will be the value of yyyy-mm-dd when executing ls?

If this cannot be provided to ls during runtime (i.e. 2016-01-25) you will have to do a branch inside the loop to see if the current filename from the list matches the pattern or not. If not just skip it, if yes read from remote and write to local.

Regards,
Holger

Nezha_Abrabri · January 25, 2016, 5:04pm

hi Holger von Thomsen thank you for yr reply,

thats exactly what i did for the first step but i dont know what to don in remotefile of get,
for the pattern could you please show me how to use branche to test if the pattern match or not,
attatched my code
Regards,

Nezha_Abrabri · January 25, 2016, 5:05pm

hi Holger von Thomsen thank you for yr reply,

thats exactly what i did for the first step but i dont know what to don in remotefile of get,
for the pattern could you please show me how to use branche to test if the pattern match or not,

attatched my code.
Regards,