Need to retrieve .CSV files via FTP into PF

[kcmmk5]

Re: Need to retrieve .CSV files via FTP into PF

Just to wind back a fraction. How often are the files being created in the "landing" folder?

If they can appear at any time then doing an MGET MDEL may either grab files already processed, or delete files not processed yet.

You could first do an LS command from the FTP session to a local file, then read the file and compare the file names against a PF that lists files processed and status flags etc.

Then you only need to GET files that have not yet been processed, and only delete files that have been processed ok.

New files could be created once every 15 minutes.
I've thought about the the option of doing the LS command to a file. I actually have a program that I wrote for transferring SAVF's from one system to another that does exactly that. I just wondered if there was a "cleaner" way to do it rather than initiating one ftpsession to do the LS command and another ftp session to do the get and delete. If that is the cleanest way, that's probably how I will do it. Thanks.

[kcmmk5]

Re: Need to retrieve .CSV files via FTP into PF

There is also some code here: look at the top two examples

http://www.code400.com/inside.php?category=OPERATIONS

jamie

Thanks. I had looked at those examples and I am not familiar with the code that they are written in. Everything that I have done up till now has been in CL and RPGIII. I'm finally dabbling in ILE RPG a little now.

[gcraill]

Re: Need to retrieve .CSV files via FTP into PF

New files could be created once every 15 minutes.
I've thought about the the option of doing the LS command to a file. I actually have a program that I wrote for transferring SAVF's from one system to another that does exactly that. I just wondered if there was a "cleaner" way to do it rather than initiating one ftpsession to do the LS command and another ftp session to do the get and delete. If that is the cleanest way, that's probably how I will do it. Thanks.

Or do a blind MGET of everything into a repository on the iSeries, then have the smarts local that check if the file needs to be processed.

But then you still have the requirement to delete the processed files from the remote server, thereby initiating more FTP sessions.

I would do the LS first, then build 2 lists (Files to GET / Files to DLT) and then process those lists. Then "If" any ne file appeared while all that is processing it gets ignored until the next iteration of the process.

Guarantees you don't miss any files, just set the recheck interval to something appropriate to the business requirements.

[kcmmk5]

Re: Need to retrieve .CSV files via FTP into PF

We had to sideline this project for a while. I am just now getting back to working with it. Since then, a few things have changed. One thing that has changed is the file format. It is no longer a CSV file but, instead it is a flat text file (space delimited).

I figured that I would still be able to use the QSH command to combine the files but, I am getting unexpected results.

The command I am using is: QSH CMD('touch -C 1252 /DTFTP/MERGEIN/DT850.TXT | find '/DTFTP/IN/' -name '*.850FF' -exec sort -n {} >/DTFTP/MERGEIN/DT850.TXT \;')

When I run this command, it does combine the files together in to one but, it also sorts the records. I need the command to simply combine the files together without sorting the records.

Does anyone have any ideas? Thanks.

[Scott Klement]

Re: Need to retrieve .CSV files via FTP into PF

I guess you could do something like this:

QSH CMD('touch -C 1252 /DTFTP/MERGEIN/DT850.TXT && cat /DTFTP/IN/*.850FF > /DTFTP/MERGEIN/DT850.TXT')

Though, I'd probably use variables for the pathnames rather than hardcoding it all. It's hard to create a test environment to test the stuff when all the names are hard-coded in the program.

Your code example is a little confusing. You're using a pipe where there's no stdout/stdin to connect, which is weird. And you're using 'find' where a simple wildcard would suffice... unless I'm misreading your intention?

[kcmmk5]

Re: Need to retrieve .CSV files via FTP into PF

I guess you could do something like this:

QSH CMD('touch -C 1252 /DTFTP/MERGEIN/DT850.TXT && cat /DTFTP/IN/*.850FF > /DTFTP/MERGEIN/DT850.TXT')

Though, I'd probably use variables for the pathnames rather than hardcoding it all. It's hard to create a test environment to test the stuff when all the names are hard-coded in the program.

Your code example is a little confusing. You're using a pipe where there's no stdout/stdin to connect, which is weird. And you're using 'find' where a simple wildcard would suffice... unless I'm misreading your intention?

Thank you. That works perfect. And thanks for the tip on using variables for pathnames.

[kcmmk5]

Re: Need to retrieve .CSV files via FTP into PF

Update - I found an issue with that command but, I figured out a work around. The QSH command errors if there are no files in /DTFTP/IN to process. I was able to work around the issue by adding an additional touch statement to create a blank file to process. Here is the updated command:

QSH CMD('touch -C 1252 /DTFTP/IN/blank.850FF && touch -C 1252 /DTFTP/MERGEIN/DT850.TXT && cat /DTFTP/IN/*.850FF > /DTFTP/MERGEIN/DT850.TXT')

Thanks again.