HTTP Apache And Parallel Processing Performance

[TheZenbudda]

Based on some recent statistics I've been running on our system with trying to build web services, it seems I may have inaccurate understanding of how:

1. OS and/or Apache "builds new jobs" for an Apache HTTP server
2. OS and/or Apache keeps programs in memory and how activation groups help
3. OS and/or Apache scales with a parallel/concurrent processing
4. OS and/or Apache jobs utilize system resources to run "efficiently" and "play nice" with other jobs running on the system

I have a simple RPG program that does not open any files, leaves on LR, runs in a named activation group. This program grabs a start time, end time, inserts a row into a table with those values, and leaves. The apache config is set to prestart 50 jobs, and when each job starts it calls this simple program (with the assumption that it will be resident in memory and ready-to-go). When I call this service in a queue (1 call, 10 times, in a queue), the one http job that processes this requests processes each one within 200 ms. When I call this service in parallel, the single request execution time starts to scale upwards. When i go from 1x10 requests to 5x10 requests to 10x10 requests up to 50x10, each job starts to slow down well beyond 5 seconds per request. This is the opposite of what i thought the iseries does for you when it comes to job/resource management. I would expect each request to slow down, but not by 5 seconds.

I do realize there are a lot of factors to take into consideration, and I am hoping to learn more about what all of those factors are. One thing that comes to mind is configuring this specific http server to run in it's own *POOL so that it does not interfere with all of the other resources. Before I take this approach, I wanted to make sure I wasn't overlooking something.

[jtaylor___]

Is there a reason you're setting on LR?
When processing in parallel, are different jobs handling the requests?
Are the different jobs waiting on shared resources?
Have been able to narrow down the actual steps in the program that are causing the delays?

[jtaylor___]

Also, you're not running in the BASE pool are you?

[TheZenbudda]

1. not setting on lr
2. yes, 50+ jobs are prestarted and preloaded with the exact program being invoked
3. not sure what you mean there....if you mean things like record locks etc, then no. simple sql inserts are being performed and that's it
4. i have not yet narrowed down an actual line of code. i really do believe its a system or apache config that is causing this. cpu jumps up to 100%
5. i AM running in the BASE pool (hence why i thought we could configure it to get out of there lol).

[jtaylor___]

2. Is it actually using those different jobs, or are the requests all being funneled thru a much smaller set of jobs?
4. I think that narrowing down the actual code will help find the root cause.

[TheZenbudda]

2. definitely using all available jobs concurrently.
3. so i have a question about that. when i run the test, it takes a few seconds for all concurrent requests to start running. i can clearly see quick responses initially but then they start to increase in completion time as the cpu spikes. doesn't this point more to a resource problem than a code problem? i tried my best to stick statistics around the barebones code i have and nothing sticks out in terms of cpu hogging.

[jtaylor___]

I think that identifying the operation(s) in question could help to identify the likely resource bottleneck.
You'd want to capture your stats during the performance slow down.

[TheZenbudda]

jtyalor, Thanks for your continued responses.

I have 2 programs I am logging: the RPG program (PGMA) that is "servicing" web requests which calls an RPG program that "does stuff" (PGMB).

I'm current logging several things:
1. PGMA get time (PGMA start)
2. PGMA get time before calling PGMB (call PGMB start)
3. PGMA calls PGMB with a unique batch id
4. PGMB get time (PGMB start)
5. PGMB generates random number 10 times
6. PGMB get time (PGMB end)
7. PGMB inserts record into log file with start/stop times and batch id and ends (without setting on LR) and returns start/stop times
8. PGMA get time (call PGMB end)
9. PGMA get time (PGMA end)
10. PGMA inserts records into log file with start/stop times and batch id and ends (without setting on LR) and retruns start/stop times to client

The table i am writing to also has a timestamp field in which i'm always using current_timestamp.

When the requests start to ramp up, the CPU spikes to 100% and things slow down. The PGMB execution time is usually in the millisecond second range and sometimes in the microsecond range. The PGMA (restful service provider) elapsed times and the calls to PGMB tend to be suspiciously high in my eyes.

But still, doesn't allowing the CPU to spike to 100% slows these things down? I almost feel like these jobs start out swimming downstream until there are too many jobs in the river, and some of them start swimming side stream and upstream until they can swim down stream again. (sorry for the cheesy analogy).