C Linux Bandwidth Throttling of Application

Question

What are some ways I can try to throttle back a send/sendto() function inside a loop. I am creating a port scanner for my network and I tried two methods but they only seem to work locally (they work when I test them on my home machine but when I try to test them on another machine it doesn't want to create appropriate throttles).

method 1

I was originally parsing /proc/net/dev and reading in the "bytes sent" attribute and basing my sleep time off that. That worked locally (the sleep delay was adjusting to adjust the flow of bandwidth) but as soon as I tried it on another server also with /proc/net/dev it didn't seem to be adjusting data right. I ran dstat on a machine I was locally scanning and it was outputting to much data to fast.

method 2

I then tried to keep track of how many bytes total I was sending and adding it to a total_sent variable which my bandwidth thread would read and compute a sleep timer for. This also worked on my local machine but when I tried it on a server it was saying that it was only sending 1-2 packets each time my bandwidth thread would check total_sent making my bandwidth thread reduce sleep to 0, but even at 0 the total_sent variable did not increase due to the reduced sleep time but instead stayed the same.

Overall I am wanting a way to monitor bandwidth of the Linux computer and calculate a sleep time I can pass into usleep() before or after each of my send/sendto() socket calls to throttle back the bandwidth.

Edit: some other things I forgot to mention is that I do have a speedtest function that calculates upload speed of the machine and I have 2 threads. 1 thread adjusts a global sleep timer based on bandwidth usage and thread 2 sends the packets to the ports on a remote machine to test if they are open and to fingerprint them (right now I am just using udp packets with a sendto() to test this all).

How can I implement bandwidth throttling for a send/sendto() call using usleep().

Edit: Here is the code for my bandwidth monitoring thread. Don't concern yourself about the structure stuff, its just my way of passing data to a thread.

void *bandwidthmonitor_cmd(void *param)
{
  int i = 0;
  double prevbytes = 0, elapsedbytes = 0, byteusage = 0, maxthrottle = 0;

  //recreating my param struct i passed to the thread
  command_struct bandwidth = *((command_struct *)param);
  free(param);

  //set SLEEP (global variable) to a base time in case it was edited and not reset
  SLEEP = 5000;

  //find the maximum throttle speed in kb/s (takes the global var UPLOAD_SPEED
  //which is in kb/s and times it by how much bandwidth % you want to use
  //and devides by 100 to find the maximum in kb/s
  //ex: UPLOAD_SPEED = 60, throttle = 90, maxthrottle = 54
  maxthrottle = (UPLOAD_SPEED * bandwidth.throttle) / 100;
  printf("max throttle: %.1f\n", maxthrottle);

  while(1)
  {
      //find out how many bytes elapsed since last polling of the thread
      elapsedbytes = TOTAL_BYTES_SEND - prevbytes;
      printf("elapsedbytes: %.1f\n", elapsedbytes);

      //set prevbytes to our current bytes so we can have results next loop
      prevbytes = TOTAL_BYTES_SEND;

      //convert our bytes to kb/s
      byteusage = 8 * (elapsedbytes / 1024);

      //throttle control to make it adjust sleep 20 times every 30~ 
      //iterations of the loop
      if(i & 0x40)
      {
           //adjust SLEEP by 1.1 gain
           SLEEP += (maxthrottle - byteusage) * -1.1;//;


           if(SLEEP < 0){
               SLEEP = 0;
           }
           printf("sleep:%.1f\n\n", SLEEP);
      }

      //sleep the thread for a short bit then start the process over
      usleep(25000);
      //increment variable i for our iteration throttling
      i++;
  }

}

My sending thread is just a simple sendto() routine in a while(1) loop sending udp packets for testing. sock is my sockfd, buff is a 64 byte character array filled with "A" and sin is my sockaddr_in.

  while(1)
  {
    TOTAL_BYTES_SEND += 64;

    sendto(sock, buff, strlen(buff), 0, (struct sockaddr *) &sin, sizeof(sin))


    usleep(SLEEP);
  }

I know my socket functions work because I can see the usage in dstat on my local machine and the remote machine. This bandwidth code works on my local system (all the variables change as they should) but on the server I tried testing on elapsed bytes does not change (is always 64/128 per iteration of the thread) and results in SLEEP throttling down to 0 which should in theory make the machine send packets faster but even with SLEEP equating to 0 elapsedbytes remain 64/128. I've also encoded the sendto() function inside a if statement checking for the function returning -1 and alerting me by printf-ing the error code but there hasn't been one in the tests I've done.

Answer 1

It seems like this could be most directly solved by calculating the throttle sleep time in the send thread. I'm not sure I see the benefit of another thread to do this work.

Here is one way to do this:

Select a time window in which you will measure your send rate. Based on your target bandwidth this will give you a byte maximum for that amount of time. You can then check to see if you have sent that many bytes after each sendto(). If you do exceed the byte threshold then sleep until the end of the window in order to perform the throttling.

Here is some untested code showing the idea. Sorry that clock_gettime and struct timespec add some complexity. Google has some nice code snippets for doing more complete comparisons, addition, and subtraction with struct timespec.

#define MAX_BYTES_PER_SECOND (128L * 1024L)
#define TIME_WINDOW_MS 50L
#define MAX_BYTES_PER_WINDOW ((MAX_BYTES_PER_SECOND * TIME_WINDOW_MS) / 1000L)

#include <time.h>
#include <stdlib.h>

int foo(void) {
  struct timespec window_start_time;

  size_t bytes_sent_in_window = 0;
  clock_gettime(CLOCK_REALTIME, &window_start_time);

  while (1) {
    size_t bytes_sent = sendto(sock, buff, strlen(buff), 0, (struct sockaddr *) &sin, sizeof(sin));
    if (bytes_sent < 0) {
      // error handling
    } else {
      bytes_sent_in_window += bytes_sent;

      if (bytes_sent_in_window >= MAX_BYTES_PER_WINDOW) {
        struct timespec now;
        struct timespec thresh;

        // Calculate the end of the window
        thresh.tv_sec = window_start_time.tv_sec;
        thresh.tv_nsec = window_start_time.tv_nsec;
        thresh.tv_nsec += TIME_WINDOW_MS * 1000000;
        if (thresh.tv_nsec > 1000000000L) {
          thresh.tv_sec += 1;
          thresh.tv_nsec -= 1000000000L;
        }

        // get the current time
        clock_gettime(CLOCK_REALTIME, &now);

        // if we have not gotten to the end of the window yet
        if (now.tv_sec < thresh.tv_sec ||
            (now.tv_sec == thresh.tv_sec && now.tv_nsec < thresh.tv_nsec)) {

          struct timespec remaining;

          // calculate the time remaining in the window
          //  - See google for more complete timespec subtract algorithm
          remaining.tv_sec = thresh.tv_sec - now.tv_sec;
          if (thresh.tv_nsec >= now.tv_nsec) {
            remaining.tv_nsec = thresh.tv_nsec - now.tv_nsec;
          } else {
            remaining.tv_nsec = 1000000000L + thresh.tv_nsec - now.tv_nsec;
            remaining.tv_sec -= 1;
          }

          // Sleep to end of window
          nanosleep(&remaining, NULL);
        }

        // Reset counters and timestamp for next window
        bytes_sent_in_window = 0;
        clock_gettime(CLOCK_REALTIME, &window_start_time);
      }
    }
  }
}

Answer 2

If you'd like to do this at the application level, you could use a utility such as trickle to limit or shape the socket transfer rates available to the application.

For instance,

trickle -s -d 50 -w 100 firefox

would start firefox with a max download rate of 50KB/s and a peak detection window of 100KB. Changing these values may produce something suitable for your application testing.