Linux, C, epoll(), read() data incompleted?

Question

Linux, C. Below issue only happens by using epoll(). If I use select() on server socket, there is no data loss.

=============================

Update: I received errno =11 (Try again) in read(). Do I need to continue, or break the while loop?

=============================

I have client side, send 1280 K data in 10 times (each time, I send 128K data);

I have server side, use epoll() to watch incoming data.

Once I got notified, I use below code to read data:

nbytes = Nread(current_fd, buffer, bytes_to_be_read);

int Nread(int fd, char *buffer, size_t count)
  {
          ssize_t r;
          size_t left = count;
          printf("===>\n");
          while (left > 0){
                  r = read(fd, buffer, left);
                  printf("data: %ld\n", r);
                  if (r < 1) {
                          printf("errno: %d\n", errno);
                          break;  //I do received 2 errors of Try Again. How to try again?
                  }

                  left -= r;
                  buffer += r;
          }
          printf("=> done, %ld\n", total - left);
          return count - left;
  }

In Nread, I expected to read bytes_to_be_read (65536) bytes and return.

But, when run the tool, I found server side just receive something like this. Look like I did not read all data from the buffer in some cases. Why?

1===>
data: 65536
=> done, 65536
2===>
data: 65536
=> done, 65536
3===>
data: 60734
data: 4802
=> done, 65536
4===>
data: 55934
data: -1
errno: 11
=> done, 55934  //data is not enough. the missing part should come soon. but why I am not notified?
5===>
data: 60736
data: 4800
=> done, 65536
6===>
data: 65536
=> done, 65536
7===>
data: 65536
=> done, 65536
8===>
data: 65536
=> done, 65536
9===>
data: 65536
=> done, 65536
10===>
data: 65536
=> done, 65536
11===>
data: 65536
=> done, 65536
12===>
data: 65536
=> done, 65536
13===>
data: 65536
=> done, 65536
14===>
data: 65536
=> done, 65536
15===>
data: 65536
=> done, 65536
16===>
data: 65536
=> done, 65536
17===>
data: 65536
=> done, 65536
18===>
data: 65536
=> done, 65536
19===>
data: 65536
=> done, 65536

===============

update:

if I use this code to read all data, then there is no bytes loss. but the performance is bad:

 36 int read_all(int fd, char *buffer)
 37 {
 38         ssize_t count = 0, bytes = 0;
 39
 40         while(1){
 41                 bytes = read(fd, buffer, sizeof(buffer));
 42                 if (bytes < 1)
 43                         break;
 44                 count += bytes;
 45         }
 46         return count;
 47 }

===================================

EPOLL()

efd = epoll_create1 (0);
event.data.fd = listener_fd;
event.events = EPOLLIN | EPOLLET;
epoll_ctl(efd, EPOLL_CTL_ADD, listener_fd, &event);

while (1){
    n_fds = epoll_wait (efd, events, MAX_EPOLL_EVENTS, -1);
    for (i = 0; i < n_fds; i++){
                ...........
       if (islistener) {
             .....
            //handle new connections
       }
      else{
           bzero(buffer, recv_buf_size);
           if ((n= Nread(fd, buffer, recv_buf_size)) <= 0) {
                  //error
           }
           else
           {
                   //add bytes to total:
                    __sync_fetch_and_add(&(mythreads->total_bytes_transferred), n)    ;
           }

}

Answer 1

 event.events = EPOLLIN | EPOLLET;

You are doing edge triggered polling. Which means your last read is probably not reading all the available data. It stops after reading 64k data, even if there is more data available. But the poll will not trigger again due to the edge trigger. Suggest removing EPOLLET.

Answer 2

        break;  //I do received 2 errors of Try Again. How to try again?

By going back to your epoll loop.

        data: 60734  //??? why I am not able to read all 65536 bytes?

Because they hadn't been received yet.

I think you may miss the big picture on how you do non-blocking I/O with epoll.

1. epoll tells you when there's data.
2. You read as much as you can at that time.
3. You go back to step 1.