Batch jobs taking extreme cpu usages

[RSK]

What can I do? I have had several Batch jobs in the last two-three weeks that have been taking over 50% cpu utilization, We are getting worried because it is starting to take the CPU over 100%, and we are getting worried that with this performance issue we are going to crash the I-series and that wouldn't be good. I believe that the the priority on the jobs are at 50.

[tomholden]

Re: Batch jobs taking extreme cpu usages

i've never seen a system go down due to maxxed CPU usage (and i've seen it hundreds of times). the performance will degrade due to the high utilization but it's highly unlikely you'll ever see the system go down due to this issue. if this is something that has just started recently and these jobs have been running for a long time prior to the problem i would check to see if any changes have been made to any of the programs, etc running in those jobs. if that's not the case then check to make sure that no system changes have occurred in that time frame. if these have been running without these issues prior to the last 2-3 weeks then something has had to have changed. another possibility is that your system is running in a degraded state. do a WRKDSKSTS and then F11 to see if the state is DEGRADED, if so most likely you have a cache battery that is either dead or dying. (in either of those cases warning message have likely been in your QSYSOPR messages and have been ignored). but honestly just the CPU utilization will not shut down the system but you should find out what was changed around that time frame and fix that issue.

[B.Hauser]

Re: Batch jobs taking extreme cpu usages

I'd also check if these jobs are using SQL and if perhaps temporary indexes are created.

Birgitta

[MichaelCatalani]

Re: Batch jobs taking extreme cpu usages

Did someone mention the words "crash" and "i-series" in the same sentence? Hath hell frozen over?

The only thing you really need to be concerned with is that the batch jobs make the disk drives so busy hat it affects the performance of the interactive jobs. As long as your memory is configured correctly, it's not a problem. It certainly wont crash the system. I have run a system at 100%, and was so interactive heavy that batch jobs received almost no cpu time betweem 7am and 6pm. It ran this way 6 days a week for years, and only cleared up for a few weeks after each machine upgrade. Your situation is different, in that your batch jobs are filling in the gap.

A good way to look at it is that if your CPU is not utilized at 100%, then you are wasting processing time that is available to use. As long as the batch jobs dont screw too much with the disk drives, and as long as you subsystem pools are correctly configured, then there is no problem at all.

[BenThurley]

Re: Batch jobs taking extreme cpu usages

I think the system IPL's ("crashes") if disk usage hits 100% doesn't it?

The CPU usage has always seemed a bit pointless to me though. I've seen high CPU usage where the system is coping perfectly with decent response times. On other occasions performance has not been so good and the CPU usage has only been about 60%. Performance and CPU usage don't appear to be as tightly related as you would think.

[tomholden]

Re: Batch jobs taking extreme cpu usages

i've never seen that either Ben. the system OS is pretty darn smart and will "cache" transactions for disk I/O but if you're hitting 100% disk busy at any time you have a worse problem than with CPU percentage. and you are correct with the performance/CPU to a certain extent. if CPU usage is over 100% (which is usually due to background system jobs for the DB, etc.) then there is generally an application issue somewhere. but CPU usage can impact performance. if you have a CPU eating query it can and will impact interactive response times and your users will start complaining. i've checked the disk status during some of those instances and disk I/O isn't even close to 100% but response time is "degraded" somewhat. so both disk I/O and CPU intensive apps can affect performance. but as i said generally applications are the culprit when it comes to CPU spikes like those described above. (same goes for disk I/O in a lot of cases). i have seen times where a cross-reference file with less than 100 records with a total record length is like 20 characters long (2 10 char fields!) where the program chains the file on every iteration. i generally change these to load an array and search the array for matches rather than chain and lo and below the app runs much faster and thousands of Disk I/O removed. programmers can be dangerous with a little knowledge...

[BenThurley]

Re: Batch jobs taking extreme cpu usages

By disk usage I meant if you fill the disk up, rather than disk activity such as reads and writes. I'm fairly sure it will IPL in this case but if you fill the disk up then you have to expect some kind of disaster. I remember once we had a rogue program that was filling the disk with spool files. We noticed this when the disk got up to about 98%. It was a bit of a close shave because if it had been left overnight it would have caused an IPL.

[tomholden]

Re: Batch jobs taking extreme cpu usages

ok now i see...yep if you hit max usage it'll IPL and you'll have tons of fun cleaning up...which is why as a general rule of thumb i keep my threshold set to 90% so i have time to actually do something before it gets close to crashing. i never really understood why anyone would set the threshold at a higher percentage. but you are correct. i misread/misinterpreted your statement.