QCMDEXC -> STRQSH -> QSH cURL, too slow

[OLDMAN25]

I'm in a deep deep problem right now. As usual, I've tried to solve myself for 14 hours first, before asking.

Context:
Trying to run cURL n times. Running cURL alone in QSHELL environment runs it at 0.3 seconds. But when running from an RPGLE program via QCMDEXC -> STRQSH -> QSH cURL, I get 10 seconds. And since I'm trying to run this 100k+ times, it becomes impossible to complete in a day.

I'm looking into the following:
1. QzshSystem to hopefully speed up the cURL call
2. Generate QSH script via ECHO ECHO ECHO, then run that script containing a batch cURL commands. Then run that shell script from the RPGLE, then process the output via loop of DATA-INTO.
3. Eat broken glass and try to make AxisC work again, which never have worked for me before. And pray to the heavens that it's somehow faster.
4. HTTPAPI, but I have to check each and every program that used the old version of HTTPAPI and update it.

[JonBoy]

#4 - and you don't _have_ to check and change every current program. Just place the latest version in a different library and make sure to use a binding directory that specifies the specific library you want rather than use the library list.

#5 option might be to use SQL depending on the type of web service you are connecting to. But that is not likely to be as fast as HTTPAPI.

[UserName10]

Why have an RPGLE program call QCMDEXC call STRQSH to run the QSH script? Why not run the QSH script and then run the RPGLE? If you need to build the script dynamically have a separate program build the script and put the whole thing in a cl pgm.

call scriptbuilder
call qsh
call your rpg pgm

[OLDMAN25]

#4 - and you don't _have_ to check and change every current program. Just place the latest version in a different library and make sure to use a binding directory that specifies the specific library you want rather than use the library list.

#5 option might be to use SQL depending on the type of web service you are connecting to. But that is not likely to be as fast as HTTPAPI.

Thanks Joy. I saw a post on HTTPAPI speed that was dated back in 2013. I think the best it can do is what a postman can do. It makes sense, the curl command I'm using also achieves the postman speed. what i want to try and do now is to remove the overhead from STRQSH

[OLDMAN25]

Why have an RPGLE program call QCMDEXC call STRQSH to run the QSH script? Why not run the QSH script and then run the RPGLE? If you need to build the script dynamically have a separate program build the script and put the whole thing in a cl pgm.

call scriptbuilder
call qsh
call your rpg pgm

I'm doing exactly that now. Currently looking for the best way to write to an IFS file so I can CURL & CURL & CURL & and run that thing to oblivion.

[OLDMAN25]

Solving my own problem here...

By running them via QSHELL in parallel, the absurd X hours become 10 seconds~.

So I wrote a procedure that writes to a flat file the cURL command and appends a '&' at the end. and ';' at the last command.

() - command grouping
& - runs the commands in parallel in qshell
; - Sequential, this waits for the parallel process to complete before running the useless command 'false' and return to program.

Sample code:

(curl -skLX POST "some link" .... &
curl -skLX POST "some link" .... &
curl -skLX POST "some link" .... );
false

Then copy this collected commands in an IFS file, then pipe to SH file.

CPYTOIMPF FROMFILE(HANDSOME/FLATFILE)                            

          TOSTMF('/home/somescript.sh')    

          MBROPT(*REPLACE)                                  

          RCDDLM(*CRLF)                                    

          STRDLM(*NONE)                                    

          RMVBLANK(*TRAILING)  

          ORDERBY(*ARRIVAL)

Then simply run that SH file via QCMDEXC STRQSH .....

For 50 transactions, there's a 97% drop in processing time.

Then just another loop to

Monitor ;
Data-Into ... ;
On-Error ;
Endmon  ;

[Scott Klement]

4. HTTPAPI, but I have to check each and every program that used the old version of HTTPAPI and update it.

Normally people simply update HTTPAPI. There's no need to update any programs that call it... just replace the old HTTPAPI with a new one, and you're all set. Can you explain what problems you ran into that you feel its necessary to change the programs that call it?

[OLDMAN25]

Normally people simply update HTTPAPI. There's no need to update any programs that call it... just replace the old HTTPAPI with a new one, and you're all set. Can you explain what problems you ran into that you feel its necessary to change the programs that call it?

For real.. I'm scared to replace it if it was already there for awhile. They don't have a mirrored environment that runs exactly like the PROD so I'm generally afraid to mess with anything there. Another factor is I haven't dealt with HTTPAPI before so there's the fear-factor of the unknown. I apologize if that's a lame reason but I am really scared to mess with prod

[Scott Klement]

I agree that its good to proceed cautiously, but... eventually, you're going to need to be able to update software, right? You can't just keep what you have forever.

Perhaps a good compromise is a little safety net... back it up before you replace it. Here's a simple way to do that (this assumes the library is called LIBHTTP, if it is something else, substitute that library name below)

CRTSAVF FILE(QGPL/HTTPBAK)
SAVLIB LIB(LIBHTTP) DEV(*SAVF) SAVF(QGPL/HTTPBAK)

Then do the upgrade and try some of the programs. They should work fine. If they don't, just restore the save file.

[JonBoy]

ANy reason not to rebuild with a different library name Scott and use that for the new routine? If he is that worried about disrupting production (which I agree is highly unlikely) then that might be asafer option than throwing in a new version and falling back if a problem arises.

[Scott Klement]

ANy reason not to rebuild with a different library name Scott and use that for the new routine? If he is that worried about disrupting production (which I agree is highly unlikely) then that might be asafer option than throwing in a new version and falling back if a problem arises.

That would work, too, then just adjust the library list if you want to try the application with the new version.

[OLDMAN25]

Scott Klement JonBoy
Thank you both. This is my nth post here and each one is solved based on your advice. The project has moved past me changing back to HTTPAPI at this point, but I must learn this for the next proj if it involves calling APIs again. I'm currently working 16 hours daily 7 days a week so I can't find the room to play around at the moment..

I'll start playing around in another server with HTTPAPI and get some confidence. Scott Klement have you ever faced a scenario that you have to call HTTPAPI to consume an API 1million times and having to reduce that run-time? Changing HTTPAPI from sequential to parallel.

[Scott Klement]

have you ever faced a scenario that you have to call HTTPAPI to consume an API 1million times and having to reduce that run-time? Changing HTTPAPI from sequential to parallel.

It may depend on the API, how it works, etc.

Many APIs allow you to submit multiple items to work on in a single request. This tends to greatly improve performance. For example, when working with IBM Watson's Language Translator, you can submit a single phrase to be translated if you want, or you can submit multiple phrases at the same time by sending an array of phrases, and it will return an array of translated phrases. If you're working with a document containing thousands of phrases, being able to submit a hundred or two at a time will make a huge difference in performance. Look in the docs for the API you're calling and see if it's possible to send multiple items in a request, and then break your million transactions down into, for example, 1000 transactions with 1000 items in each one.

If the API doesn't allow that, then it may allow you to use a persistent HTTP connection. One unique thing about HTTPAPI (that you won't find in a command-line tool like 'curl') is that you can choose to use the persistent request interface. What that means is that you make one TCP connection, and then you use that single connection to submit multiple requests. For example, if you are making 100 requests without persistent connections, that means it has to look up the address for the host name 100 times, make a network connection, 100 times, etc. With persistent, you can run 100 requests with a single network connection, removing the overhead of the repeated connections. The coding can be a bit more advanced, but it does improve performance (if the server allows it.)

Finally, of course, you can split up your workload into multiple jobs that run simultaneously. For example, maybe have 10 jobs, each handling 100,000 requests. You'd have to experiment to determine what the optimal number of jobs and optimal number of requests per job should be. This also complicates the programming quite a bit, but is possible.

All of these are things to consider...

[OLDMAN25]

Thank you!, I read an old email/forum thread of yours referencing that technique on persistent connection. That's something I'll definitely look at. And yes, I'm getting substantial delay from the cURL figuring out which IP to use every time, I found that the server where the API is hosted has some load balancing algorithms that redirects the request and points the cURL to different IPs each time, I wonder what happens if I do a persistent on that...

I did something absurd wherein I spawned 1 QSH job per transaction at a thousand transactions at a time. So I'm facing a TIMA and SELW statuses on those child jobs. It gets the job done but I'm not sure what the Admin will think about it. He provided us a *NOMAX jobq though XD

Thank you again!