A Tail of Woe: The Trouble with fstat Caching

tl;dr Subsequent calls to fstat may return stale file info such as size, last accessed time, etc.. no matter what file system you are using.  Always open and close a file in order to poll its size.  e.g. code

In order to prep for a coding interview, I was re-implementing the *nix tail command.  As a refresher, the tail command prints the last 10 lines of a file and, if passed the -f flag, will continue to watch the file, printing to stdout anything written to it.

To account for the -f flag, we poll on file on the size of the file, calling fstat periodically in order to determine the files current size.  i.e.

   
//fd is our file descriptor and follow is a bool telling us to follow the file
if (follow) {
      int readr;
      while (1) {        
        struct stat sbuf2;
        fstat(fd, &sbuf2);
        if (remsize < sbuf2.st_size) {           
          diff = sbuf2.st_size - remsize;           
          lseek(fd, --remsize, SEEK_SET);           
          while (diff > 0) {
            incr = (diff < BUFSIZE) ? diff : BUFSIZE;
            int r = read(fd, buf, incr);
            diff -= incr;
            write(1, buf, incr);
          }
          remsize = sbuf2.st_size;
        } 
        sleep(LOOPSLEEP); 
      }
    }

While the idea behind this code is fine, it contains a subtle problem: fstat may cache the file attributes and return the same values on subsequent calls even though the underlying file has been modified.

I first thought that this only applied to NFS: http://cow.physics.wisc.edu/~craigm/idl/archive/msg00399.html

I then went on to test it locally.  My local machine is running Ubuntu 12.04 with the ext4 filesystem. Locally too, i experienced the same problem.

The solution to this issue is simple: close and then open the file again in order to force the os to reload the file attribute cache:

   if (follow) {
      int readr;
      while (1) {
        //http://cow.physics.wisc.edu/~craigm/idl/archive/msg00399.html
        // - seems to be an issue with many file systems; local ext3 on ubuntu 12.04 also caches fstat
        // - fstat may not return newest file size in files contained in NFS
        // - teach me to dev on ec2
        // no dice fchown(fd, sbuf.st_uid, sbuf.st_gid);
        // no dice fcntl
        if (fd > 0) {
          close(fd);
          fd = open(filepath, O_RDONLY, 0);
        }

        struct stat sbuf2;
        fstat(fd, &sbuf2);
        if (remsize < sbuf2.st_size) {           
          diff = sbuf2.st_size - remsize;           
          lseek(fd, --remsize, SEEK_SET);           
          while (diff > 0) {
            incr = (diff < BUFSIZE) ? diff : BUFSIZE;
            int r = read(fd, buf, incr);
            diff -= incr;
            write(1, buf, incr);
          }
          remsize = sbuf2.st_size;
        } 
        sleep(LOOPSLEEP); 
      }
    }

I did not find that fcntl and fchown necessarily forced the file attributes to be reloaded.  It is safer to close and open the file.  Others have concluded similarly: http://irccrew.org/~cras/nfs-coding-howto.html

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>