Truncating the first 100MB of a file in linux

问题

I am referring to How can you concatenate two huge files with very little spare disk space?

I'm in the midst of implementing the following:

Allocate a sparse file of the combined size.
Copy 100Mb from the end of the second file to the end of the new file.
Truncate 100Mb of the end of the second file
Loop 2&3 till you finish the second file (With 2. modified to the correct place in the destination file).
Do 2&3&4 but with the first file.

I would like to know if is there anyone there who are able to "truncate" a given file in linux? The truncation is by file size, for example if the file is 10GB, I would like to truncate the first 100MB of the file and leave the file with remaining 9.9GB. Anyone could help in this?

Thanks

回答1:

Answer, now this is reality with Linux kernel v3.15 (ext4/xfs)

Read here http://man7.org/linux/man-pages/man2/fallocate.2.html

Testing code

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdlib.h>
#include <fcntl.h>

#ifndef FALLOC_FL_COLLAPSE_RANGE
#define FALLOC_FL_COLLAPSE_RANGE        0x08
#endif

int main(int argc, const char * argv[])
{
    int ret;
    char * page = malloc(4096);
    int fd = open("test.txt", O_CREAT | O_TRUNC | O_RDWR, 0644);

    if (fd == -1) {
        free(page);
        return (-1);
    }

    // Page A
    printf("Write page A\n");
    memset(page, 'A', 4096);
    write(fd, page, 4096);

    // Page B
    printf("Write page B\n");
    memset(page, 'B', 4096);
    write(fd, page, 4096);

    // Remove page A
    ret = fallocate(fd, FALLOC_FL_COLLAPSE_RANGE, 0, 4096);
    printf("Page A should be removed, ret = %d\n", ret);

    close(fd);
    free(page);

    return (0);
}

回答2:

Chopping off the beginning of a file is not possible with most file systems and there's no general API to do it; for example the truncate function only modifies the ending of a file.

You may be able to do it with some file systems though. For example the ext4 file system recently got an ioctl that you may find useful: http://lwn.net/Articles/556136/

Update: About a year after this answer was written, support for removing blocks from beginning and middle of files on ext4 and xfs file systems was added to the fallocate function, by way of the FALLOC_FL_COLLAPSE_RANGE mode. It's more convenient than using the low level iotcl's yourself.

There's also a command line utility with the same name as the C function. Assuming your file is on a supported file system, this will delete the first 100MB:

fallocate -c -o 0 -l 100MB yourfile

回答3:

Please read a good Linux programming book, e.g. Advanced Linux Programming.

You need to use Linux kernel syscalls, see syscalls(2)

In particular truncate(2) (both for truncation, and for extending a sparse file on file systems supporting it), and stat(2) to notably get the file size.

There is no (portable, or filesystem neutral) way to remove bytes from the start (or in the middle) of a file, you can truncate a file only at its end.

回答4:

If you can work with ASCII lines and not bytes, then removing the first n lines of a file is easy. For example to remove the first 100 lines:

sed -i 1,100d /path/to/file

回答5:

This is a pretty old question by now, but here is my take on it. Excluding the requirement for it to be done with limited space available, I would use something similar to the following to truncate the first 100mb of a file:

$ tail --bytes=$(expr $(wc -c < logfile.log) - 104857600) logfile.log > logfile.log.tmp
$ mv logfile.log.tmp logfile.log

Explanation:

This outputs the last nn bytes of the file (tail --bytes).
The number of bytes in the file to output is calculated as the size of the file (wc -c < logfile.log) minus 100Mb (expr $( ... ) - 104857600). This would leave us with 100Mb less than the size of the file to take the tail of (eg. 9.9Gb)
This is then output to a temp file and then moved back to the original file name to leave the truncated file.

回答6:

Remove all but the last 10,000 lines from a file.

sed -i 1,$( ( $(wc -l < path/to/file) -10000 ) )d path/to/file

来源：https://stackoverflow.com/questions/18072180/truncating-the-first-100mb-of-a-file-in-linux

标签

Linux

file

merge

truncate