Limiting the number of subshells spawned

Question

I'm trying to limit the amount of subshells that are spawned in a script that I'm using to sniff our internal network to audit Linux servers in our network. The script works as intended, but due to the way I'm nesting the for loop, it spawns 255 Sub Shells for each network, so therefore it kills the CPU due to the fact that there are over 1000 processes spawned off. I need to be able to limit the amount of processes, and since variables lose their value when in a Sub Shell, I can't figure out a way to make this work. Again, the script works, it just spawns a ton a processes - I need to limit it to, say 10 Processes max:

#!/bin/bash

FILE=/root/ats_net_final

for network in `cat $FILE`;do
                for ip in $network.{1..255};do
                        (
                                SYSNAME=`snmpwalk -v2c -c public -t1 -r1 $ip sysName.0 2>/dev/null | awk '{ print $NF }'`
                                SYSTYPE=`snmpwalk -v2c -c public -t1 -r1 $ip sysDescr.0 2>/dev/null | grep -o Linux`
                                if [ $? -eq 0 ];then
                                        echo "$SYSNAME"
                                        exit 0;
                                else
                                        echo "Processed $ip"
                                        exit 0
                                fi
                        ) &
                done
done

This solution I found that works, but not in my case, because no matter what, it still will spawn the processes before the logic of limiting the processes. I think that maybe I've just been looking at the code too long and it's just a simple logic issue and I'm placing things in the wrong area, or in the wrong order.

---

Answer Accepted:

I've accepted the answer from huitseeker. He was able to provide me with direction of how the logic works, allowing me to get it to work. Final script:

#!/bin/bash

FILE=/root/ats_net_final

for network in `cat $FILE`;do
        #for ip in $network.{1..255};do
        for ip in {1..255};do
            (
                ip=$network.$ip
                SYSNAME=`snmpwalk -v2c -c public -t1 -r1 $ip sysName.0 2>/dev/null | awk '{ print $NF }'`
                SYSTYPE=`snmpwalk -v2c -c public -t1 -r1 $ip sysDescr.0 2>/dev/null | grep -o Linux`
                if [ $? -eq 0 ];then
                    echo "$SYSNAME"
                    exit 0;
                else
                    echo "Processed $ip"
                    exit 0
                fi
            ) &
            if (( $ip % 10 == 0 )); then wait; fi
        done
        wait
done

Answer 1

An easy way to limit the number of concurrent subshells to 10 is:

for ip in $(seq 1 255);do
  (<whatever you did with $ip in the subshell here, do with $network.$ip instead >) &
  if (( $ip % 10 == 0 )); then wait; fi
done
wait

With the last wait being useful not to let the subshells of the last round of the inner loop overlap with those created in first round of the next outer run.

Answer 2

I think I have found a better solution. I implemented using make:

#!/usr/bin/make -f

IPFILE := ipfile
IPS:=$(foreach ip,$(shell cat $(IPFILE)),$(foreach sub,$(shell seq 1 1 254),$(ip).$(sub)))

all: $(IPS)

$(IPS):
        SYSNAME=`snmpwalk -v2c -c public -t1 -r1 $@ sysName.0 2>/dev/null | awk '{ print $$NF }'`; \
        SYSTYPE=`snmpwalk -v2c -c public -t1 -r1 $@ sysDescr.0 2>/dev/null | grep -o Linux`; \
        if [ $$? -eq 0 ]; then \
          echo "$$SYSNAME"; \
        else \
          echo "Processed $@"; \
        fi

The first part of the IPs (e.g. 192.168.1) should be placed to ipfile. Then it generates all the IP addresses into the variable IPS (Like 192.168.1.1 ... 192.168.1.254 ...).

These lines can be copied to e.g. test.mak and add execute rights to the file. If one run it as ./test.mak then it will process the IPs in IPS one by one. But if it is run as ./test.mak -j 10 then it will process 10 IPs at once. Also it can be run as ./test.mak -j 10 -l 0.5. It will run maximum 10 processes or until the system load reaches 0.5.