Read big data file in rpgle

[Iceberg]

Hi Team,
We have a job (RPGLE pgm) in as400 that reads history files having lots of data which take lots of time to process. Current program is just using rpg i/o operation to read current day records in history file. Is there a way to optimize this program to reduce time like using sqlrpgle or other way.

Thanks

[DAG0000]

Without knowing how your process works I would check several things:

1. Is your process calling other programs multiple times. This is a huge performance hit if a called program is constantly going through its start up process is (opening files and such). If it is you want to leave those programs open in your process. You also should check those programs to see if they are calling anything else. If this is the case, fixing this alone may give you a big enough performance boost you don't have to do anything else.

2. Does the file in question have an index/logical on it? Do you have the correct Index/logicals necessary for what you are doing.

3. Look at the logic your process is going through and see if there is anything that could be streamlined. Depending on how old it is there may be sections that are better off being rewritten or replaced. Be sure you know what your doing before rewriting anything.

Those are some of the basics.

[Peder Udesen]

We need a little more information but assuming you are reading the file sequentially you can increase the buffer
where the records are loaded. Default is 4KB but you can increase it to 256KB ( V7R3 ) using this command

OVRDBF FILE(MYFILE) SEQONLY(*YES *BUF256KB)

The same can be done when writing sequentially to a file.
When writing to a file it must NOT have an index with a unique key.
This causes SEQONLY to be set to *NO.

[TedHolt]

Years ago I was working in a S/38 shop. We had a program that ran during the night and built a report from a history file. The RPG program did all the record selection. It took an hour to run.

I changed the calling CL to run OPNQRYF and run time went from one hour to 30 seconds. I didn't change the RPG program at all.

These days I would change the program to use SQL to read the data. Maybe that approach would work for you.