Current Unprotect

[ChrisL]

Hi everyone,

I'm having an issue where one of our iSeries has a constantly growing figure within the Current Unprotect field on WRKSYSSTS. This isn't a recent occurrence and has been growing for months and months. We notice a performance impact when this figure starts to get too high, at which point we normally recommend that an IPL is done to clear this figure down. This then clears down the issue.

As we speak the current unprotect value is 393,361 Mb (on the 19th November this value was 109,394).

The iSeries is fully PTF'd, and is running V5R4. Is there anyway of identifying where this memory is being held? Someone told me there may be tools available at a higher release, but a quick google didn't come up with anything.

Any pointers or help will be gratefully received!

[jj_dahlheimer]

Re: Current Unprotect

I am going through this right now with IBM, highly suggest you contact them as it is quite an intensive process. The tools they are having me look at are pex http://www-01.ibm.com/support/docvie...d=nas8N1010467 (though I think this is more related to our issue of arms utilization spiking), collection services http://www-01.ibm.com/support/docvie...d=nas8N1012204, and looking at the sql cache plan http://www-01.ibm.com/support/docvie...d=nas8N1000021

There are other things, they had me update QMGTOOLS which they use, and they have been on our system collecting info.

As a general rule of thumb IBM said it should be no more than 5% of system asp. But they have seen where even a couple of percent also can cause issues.

[tomliotta]

Re: Current Unprotect

Near 400GB does seem excessive, though it's hard to be completely sure without knowing general system details such as total system sizes and how long the system has gone without IPL. A basic text screen-shot of a DSPSYSSTS (not WRKSYSSTS) panel could help with additional questions. And the referenced size on Nov 19th -- Is that an initial size immediately after an IPL? Or was it a size that was captured after the system was running for a few days/weeks?

A second, perhaps the biggest, problem is that support for V5R4 ended a year and a half ago; so unless Extended Support was contracted, there will be difficulties with help from IBM. (Was this problem evident before Support ended?)

The links from jj_dahlheimer are good places to start. Any active collection services, including database monitors/caches, can build up a lot of data. The data need to be used or disposed of. If there's no intention nor need for any use, there's no need to run services that are optional. Also, 'temporary' DB2 joins/indexes/etc. will contribute to the 'unprotected/temporary' value.

Since this space involves "temporary" data, it's commonly connected with jobs/tasks that are running (or have ended but have internal objects associated with the job structures). An IPL requires the ending of jobs/tasks and releases spaces associated with them. For at least some of these jobs, you can use this command:

WRKSYSACT SEQ(*STGNET)

I don't recall if V5R4 provided WRKSYSACT as a standard command or if a Performance Tools product was required. At the worst, it might be temporarily available through an evaluation install. Also, a problem with its output is that it displays jobs/tasks that have had activity since it started querying the system. One or more of the largest offenders might be dormant for some time before showing up. You might dedicate a session to it and hit F10=Update list after a good part of a day (or maybe even the next day).

For this question, review Temporary Space Growth Problem Debug and maybe IBM i Recommended Fixes - Temporary Disk Space. (The PTF lists are long, but most are in cume packages and should already be applied. Also, be sure to understand the difference between V5R4M5 LIC and V5R4 OS PTFs if necessary.)

[ChrisL]

Re: Current Unprotect

Thanks JJ and Tom - and sorry for the delay in responding.

I'll start with some of the questions from Tom..

Here is the DSPSYSSTS screen - please note the box was IPL'd on Sunday so the current unprotect will have dropped significantly.

                             Display System Status                      
                                                             24/02/15  10:39:10
 % CPU used . . . . . . . :       97.1    Auxiliary storage:                   
 % DB capability  . . . . :      106.3      System ASP . . . . . . :     8371 G
 Elapsed time . . . . . . :   00:00:02      % system ASP used  . . :    53.1857
 Jobs in system . . . . . :       5718      Total  . . . . . . . . :     8371 G
 % perm addresses . . . . :       .077      Current unprotect used :    34809 M
 % temp addresses . . . . :      7.380      Maximum unprotect  . . :    35717 M
                                                                               
 System    Pool    Reserved    Max   -----DB-----  ---Non-DB---                
  Pool   Size (M)  Size (M)  Active  Fault  Pages  Fault  Pages                
    1      7443.2    4088.3   +++++     .0     .0     .3     .3                
    2    189448.3       6.0    2404  352.0   1474  790.1  887.4                
    3     27482.1       1.3     372  125.8   1678   23.2  196.1                
    4     25497.0        .3    5610     .3    7.5     .0     .0                
    5      2549.6        .0      21     .0     .0     .0     .0                
    6      2549.6        .2      20     .0     .0     .0     .0

I have a process which checks the system and logs various performance figures every 15 minutes. I've used these figures to give the data I showed above. I only keep 90 days of detailed data so the 19th November was the furthest back I could go. However, the box was previously IPL'd on 2nd November, so the starting figure was after the box had been running for 2 weeks.

This problem has been around for as long as I can remember, we normally monitor the current unprotect and when it gets to a certain level we start noticing performance issues. As an example, I have an hourly report job that runs (using SQL RPGLE), this runs interactively in 4 minutes - before IPL these jobs were taking hours to run.

I'll take a look through the links provided and see if they help any.

Thanks again guys, your help is much appreciated.

[jj_dahlheimer]

Re: Current Unprotect

If you take IBM generic 5% limit once you hit the 400 gig you would start to notice performance issues. We have been focusing on qtemp and sql usage. Though it appears those are not the biggest issue we have. If you use a lot of qtemp and untuned sql queries you will generate a lot more temp storage as that is where this stuff is stored.

[tomliotta]

Re: Current Unprotect

@jj_dahlheimer: Note that most QTEMP objects that we purposely create are not part of the 'Current unprotect' value. E.g., if you simply copy a large file to QTEMP with CPYF, the size of that new file isn't included. However, if you copy a couple large files and then do a join query over them that creates a big temporary index (or other similar temporary SQL objects), the internal temporaries will be counted. That's true regardless of whether the work is in QTEMP or not.