Using grep to search all files in the directory for a string

[JustinWithoutAnE]

Hi All,

I am trying to write a program that will search all the files in a directory(&PATH) for a string(&SEARCH). The program is supposed to return the name of all the files that contain the string. Here's what I have so far. I tried running the program but STDOUT turns out empty even when I know there is at least one file with the string I am looking for. Am I missing something?

PGM (&PATH &SEARCH &ERROR)                                     

DCL &PATH *CHAR LEN(21)                                        
DCL &SEARCH *CHAR LEN(24)                                      
DCL &ERROR *CHAR LEN(10)                                       
DCL &CMD *CHAR LEN(100)                                        
DCL &QUOTE *CHAR LEN(1) VALUE('''')                            

CHGVAR &ERROR VALUE(' ')                                       

CD DIR(&PATH)                                                  
MONMSG MSGID(CPFA09C) EXEC(DO)                                 
  CHGVAR &ERROR VALUE('NOT_AUTH')                              
  RETURN                                                       
ENDDO                                                          

CHGVAR &CMD VALUE('grep -l -i ' !! &QUOTE !! &SEARCH !! &QUOTE)

CLRPFM QTEMP/STDOUT                  

MONMSG MSGID(CPF0000)                                         

OVRDBF FILE(STDOUT) TOFILE(QTEMP/STDOUT) OVRSCOPE(*CALLLVL)   

QSH CMD(&CMD)                                                 

DLTOVR FILE(STDOUT) LVL(*)                                    

ENDPGM

[Scott Klement]

In order to grep to search the whole directory, you need to specify the -R switch.

[JustinWithoutAnE]

It works now! Thanks! Is there a way to make it search faster? The directories that need to be search can contain up to 10,000 files

I plan to have this program called by another batch job which processes a lot of records. I can't have my program slowing down the calling program.

Here's the new code:

 
 PGM (&PATH &SEARCH &ERROR)                                       DCL &PATH *CHAR LEN(21)                                         DCL &SEARCH *CHAR LEN(24)                                       DCL &ERROR *CHAR LEN(10)                                        DCL &CMD *CHAR LEN(100)                                         DCL &QUOTE *CHAR LEN(1) VALUE('''')                              CHGVAR &ERROR VALUE(' ')                                         CD DIR(&PATH)                                                   MONMSG MSGID(CPFA09C) EXEC(DO)                                    CHGVAR &ERROR VALUE('NOT_AUTH')                                 RETURN                                                        ENDDO                                                            CHGVAR &CMD VALUE('grep -F -l -R -i ' !! &QUOTE !! &SEARCH !! &QUOTE+                !! ' ' !! &PATH)  CLRPFM QTEMP/STDOUT                    MONMSG MSGID(CPF0000)                                           OVRDBF FILE(STDOUT) TOFILE(QTEMP/STDOUT) OVRSCOPE(*CALLLVL)     QSH CMD(&CMD)                                                   DLTOVR FILE(STDOUT) LVL(*)                                      ENDPGM

grep: 001-2103 Error found getting information for file or directory -l. No such path or directory.
grep: 001-2107 Error found processing pattern file -l.

[Scott Klement]

I don't know any way to make it faster. You could try using different grep tools (such as the one from Linux) to see if it works faster than the one IBM provides with QShell, maybe that'd help?

Having said that, I would expect that a search of tens of thousands of files would indeed take a lot of time. When organizations want to search a large amount of data, they don't typically use a tool like grep, but instead put the data into an indexed database designed for searching. For example, Google does not download everyone's web pages and then use grep on them, this would be very slow. It instead uses complex algorithms to get the key details from each page, put them in a searchable database, updates the indexes periodically, and uses the databases for searching.

Of course... there's a downside to that approach, too. It requires more expertise to set up, and there's a lag between the time the document is updated and the time the search indexes are updated. So searching is much faster, but at the expense of paying engineers to maintain it and being willing to wait until the updates occur in the database. And, of course, your data set size would never be nearly as large as Google's.

[JustinWithoutAnE]

Is there a way to code a program in RPG that will search all files in the IFS for the string? Would that be faster?

[Scott Klement]

Is it possible? Of course.

I don't know if it'd be faster. It depends on how well its written vs. how well the grep program is written....

[jtaylor___]

Is there a way to code a program in RPG that will search all files in the IFS for the string? Would that be faster?

Working with the IFS in RPG IV

https://www.scottklement.com/rpg/ifs_ebook/