How does LSOUTPUT initially get created?

[Viking]

Hi,

Please help refresh my memory and/or figure this out...

A while back, I created a process that includes a CLLE program (called from an RPG that first calls it with parm "LIST" and later repeatedly with "GET" for each file) that runs an FTP script that goes out to a (non IBM i) FTP server to first get a list of files and then pulls them down and processes each one. It's been running great twice a day for a couple of years. It does this:

             PGM        PARM(&FUNC)
             DCL        VAR(&FUNC) TYPE(*CHAR) LEN(4)

/*-- Set up FTP script to execute based on incoming function parm */
             IF         COND(&FUNC *EQ 'LIST') THEN(CALLSUBR SUBR(LIST))
             IF         COND(&FUNC *EQ 'GET ') THEN(CALLSUBR SUBR(GET))

/*-- Execute FTP script */
             STRTCPFTP  RMTSYS('xx.xxx.xxx.xx') CCSID(*DFT) PORT(21) +
                          TBLFTPOUT(QASCII)

/*-- Delete overrides */
             DLTOVR     FILE(*ALL) LVL(*JOB)

/*------------------------------------------------------------------*/
/* LIST: Get the list of files to be processed from FTP server      */
/*------------------------------------------------------------------*/
             SUBR       SUBR(LIST)

                CHGCURLIB  CURLIB(MYLIB)

/*-- Specify the PF to receive the list of file names and get it ready */
                OVRDBF     FILE(LSOUTPUT) TOFILE(MYLIB/FECLIST) +
                             MBR(FECLIST) OVRSCOPE(*JOB)

                CLRPFM     FILE(MYLIB/FECLIST)

/*-- Set up the FTP script and FTP log file to be used to list files */
                OVRDBF     FILE(INPUT) TOFILE(MYLIB/MYSRCF) +
                             MBR(FTPECIMP01) OVRSCOPE(*JOB)
                OVRDBF     FILE(OUTPUT) TOFILE(MYLIB/FTPECLOG) +
                             MBR(ECIMP01) OVRSCOPE(*JOB)

/*-- Clear FTP log file  */
                CLRPFM     FILE(MYLIB/FTPECLOG) MBR(ECIMP01)

             ENDSUBR

/*-------------------------------------------------------------------*/
/* GET: Pick up the order file from the remote FTP server    */
/*-------------------------------------------------------------------*/
             SUBR       SUBR(GET)

/*-- Set up the FTP script and FTP log file to be used to pick up files */
                OVRDBF     FILE(INPUT) TOFILE(MYLIB/MYSRCF) +
                             MBR(FTPECIMP02) OVRSCOPE(*JOB)
                OVRDBF     FILE(OUTPUT) TOFILE(MYLIB/FTPECLOG) +
                             MBR(ECIMP02) OVRSCOPE(*JOB)

/*-- Clear FTP log file  */
                CLRPFM     FILE(MYLIB/FTPECLOG) MBR(ECIMP02)


             ENDSUBR
/*-------------------------------------------------------------------*/

 QUIT:       ENDPGM

The FTP script "FTPECIMP01" that gets the list of files looks like this:

OPEN 'xx.xxx.xxx.xx'                                                                                     
USER myusername mypassword                                                                             
ASCII                                                                                                                      
Namefmt 1                                                                                                         
LS (Disk                                                                                                           
CLOSE                                                                                                                           
QUIT

This morning, it suddenly isn't pulling down the list of files, but is instead timing out on the host server. Today the job takes a long time and then ends, and when I look in the FTP log after my LS (DISK command, it says first "229 Extended Passive mode OK (|||31770|)" and then "No response from remote host; all connections closed." At first I thought maybe it is a problem with the host FTP server, but I am able to FTP manually with FileZilla and see the files there. I also noticed in my program's joblog that I get the message that file LSOUTPUT does not exist, and sure enough I do not see LSOUTPUT on my system. Not sure if that message is usually there or not or if that's the problem or not? I override it to another file FECLIST anyway, so I'm not actually even populating LSOUTPUT.

Question: How does LSOUTPUT initially get created? I don't recall manually creating it way back when but maybe I did? Or should it get created automatically when I do "LS (DISK" ?

I then tried creating LSOUTPUT manually with CRTPF FILE(LSOUTPUT) RCDLEN(256) MAXMBRS(*NOMAX) and then rerunning the process, but although I don't get the message in the joblog about LSOUTPUT not existing anymore, it still takes a long time and I do not get the listing.

Not sure if I need to contact the server people at the place where I'm getting the files, or if the problem is mine.

Any insight would be appreciated. Thanks.

[Scott M]

Re: How does LSOUTPUT initially get created?

I would first try a command line ftp session and execute the same commands in your script and see what happens on the LS command. Maybe the remote server isn't responding to the LS anymore.

[Scott Klement]

Re: How does LSOUTPUT initially get created?

Each time you attempt to transfer either a file or a directory listing, a new network connection is opened for the transfer. (This wouldn't be a good design in today's world -- but it's how FTP works. FTP is from 1971.)

This message "229 Extended Passive mode OK (|||31770|)" means that the server is functioning normally. It has opened up port 31770 for your client to connect to so that it can download the directory listing (or a file, if this occurred on a GET operation). However, the client is failing to connect, eventually timing out. Most likely that means that this port is being blocked by a firewall or similar device.

The fact that it works from FileZilla means that either the server happened to pick a different port (one that's allowed through the firewall) or that your PC is going through a different firewall than the IBM i is, or something like that.

[Scott Klement]

Re: How does LSOUTPUT initially get created?

To add to what I just posted... there's no way this could be related to the LSOUTPUT file existing or not existing. That would not affect whether it could make a connection to an FTP server -- if that were the problem, there'd be a completely different error.

[Viking]

Re: How does LSOUTPUT initially get created?

Hi again, and thanks for the responses.

"...there's no way this could be related to the LSOUTPUT file existing or not existing"

Thanks for confirming that. So I guess I can just delete the one I created manually when I was trying to troubleshoot this. I actually created a member too just to see what would happen, but I see that the system removes that again anyway.

I had my ISP on the phone while I ran my program and trace the packets to see if any firewall settings were suddenly blocking the transfer of the LS data, but he can also just see it sitting there and waiting until it finally times out (it just sits on the "LS (Disk " command.) He started asking me about extended passive mode (which mine uses) vs. regular passive (which my FileZilla is using) and active, etc., and I just can't remember the ramifications of all that without reading up on it again. He's not familiar with the IBM i. But as I reminded him, this thing has been working steadily every day for a long time, so let's focus on what's changed! He admitted there had been a firewall firmware upgrade since yesterday, but can't see any relevant changes to what we're seeing...

Both the IBM i and I (with Filezilla) both go through the same final firewall since we connect to a collocation facility where everything is, and then from there FTP out to the webserver which is somewhere else.

Anyway, still not resolved as of now and I had to manually bring the files down and process them today. Will need to pick this up again tomorrow.

[tomliotta]

Re: How does LSOUTPUT initially get created?

I don't use FTP enough to be an "expert". It'll be interesting to learn what the resolution is if you learn what's happening. Unfortunately, it might simply start working and we'll never find out.

He started asking me about extended passive mode (which mine uses) vs. regular passive (which my FileZilla is using) and active, etc., and I just can't remember the ramifications of all that without reading up on it again. He's not familiar with the IBM i. But as I reminded him, this thing has been working steadily every day for a long time, so let's focus on what's changed! He admitted there had been a firewall firmware upgrade since yesterday, but can't see any relevant changes to what we're seeing...

Both the IBM i and I (with Filezilla) both go through the same final firewall since we connect to a collocation facility where everything is, and then from there FTP out to the webserver which is somewhere else.

Extended Passive mode isn't specifically for IBM i. It's been part of a FTP RFC for maybe 15 years. Changes to firewalls and some network security practices made 'passive' more and more common, and 'extended passive' later was intended to let FTP work over IPv6 along with IPv4. Since some segments along a route might handle only IPv4, and another might handle only IPv6, while others could handle both (and other protocols), FTP needed to be extended. (And Nat was a big enough problem by itself.)

You probably have been using 'passive' mode because various segments on the network have required it by disallowing 'active' connections with firewall restrictions. Apparently, though, 'passive' FTP works across the entire route since you use it from your PC from the same local network segment. (If that's the case, why use 'extended passive' from your i system?) It's not clear to me why 'passive' works when 'extended passive' doesn't. To me, that'd be a major clue to the source of the problem. I'd verify anything along the entire route that relates to IPv6.

Be aware that your ISP might have no control over any of this. The collocation site (or maybe its ISP) might be where the issue is. Your ISP quite possibly couldn't care less if you use 'active', 'passive' or 'extended passive' FTP if you have a business account.

It's definitely suspicious, though, that there was a "firewall firmware upgrade since yesterday" for your ISP.

[Scott Klement]

Re: How does LSOUTPUT initially get created?

You can add 'SENDEPSV 0' to your FTP process if you don't want it to use the extended passive mode. That seems like a long shot to me -- but maybe this new firewall firmware has a bug with regard to extended passive mode.

The difference between extended passive mode and the old/original passive mode is really that the original mode required the IP address to be sent along with the port number, which often caused problems for NAT (because NAT changes the IP address of the packets, so now the IP address that you're sending to the server will be wrong unless NAT changes not only the packets, but the data in your FTP conversation as well.) Extended passive mode fixed this by making the IP address optional, so that most of the time you'd only need to send the port number... this was, IMHO, a huge improvement that was long past due.

It is strange that your ISP would have any involvement or knowledge of your firewall at all. Normally, the firewall would be something run by your network guy, and would be in between your network and the ISP. And yeah, if the firewall is blocking your connection attempt, then you'd have the exact symptoms that your ISP said they were seeing... (No visible packets sent to the new port number.)

[Viking]

Re: How does LSOUTPUT initially get created?

And the winner is... (drum roll please)... Scott! (big surprise).

Yes, the answer is that the firewall firmware upgrade must have caused the firewalls to now have trouble with extended passive mode. By adding SENDEPSV 0 as per your "long shot" suggestion, the problem was resolved and things are humming along as usual again. I've now changed all my scripts to include this. Thanks for this suggestion and also for the explanation of the differences.

The reason that our ISP would be involved is that our ISP is also our "network guy"... they manage all our networking equipment.

So anyway, as Tom pointed out

"It's definitely suspicious, though, that there was a "firewall firmware upgrade since yesterday" for your ISP. "

... and that was indeed the answer.

Thanks for the input.