memory exhausted : for large files using diff

随声附和 提交于 2020-07-06 09:38:41

问题


I am trying to create a patch using two large size folders (~7GB).

Here is how I'm doing it :

$ diff -Naurbw . ../other-folder > file.patch

But maybe due to file sizes, patch is not getting created and giving an error:

diff: memory exhausted

I tried making space more than 15 GB but still the issue persists. Could someone help me out with the flags that I should use?


回答1:


Recently I came across this too when I needed to diff two large files (>5Gb each).

I tried to use 'diff' with different options, but even the --speed-large-files had no effect. Other methods like splitting the files into smaller ones, using xdelta or sorting the files as per this suggestion didn't help either. I even got my hands around a very powerful VM (> 72Gb RAM), but still got this memory exhausted error.

I finally got to work by adding the following parameter to sysctl.conf (sudo vim /etc/sysctl.conf):

vm.overcommit_memory=1

vm.overcommit_memory has three values (0,1,2) and sets the kernel virtual memory accounting mode. From the proc(5) man page:

0: heuristic overcommit (this is the default)
1: always overcommit, never check
2: always check, never overcommit

To make sure that the parameter is indeed applied you can run

sudo sysctl -p

Don't forget to change this parameter back when you finish!




回答2:


bsdiff is slow & requires large memory, xdelta is create large delta for large files.

Try HDiffPatch for large files: https://github.com/sisong/HDiffPatch

  • support diff between large binary files or directories;
  • can run on: Windows, macos, Linux, Android
  • diff & patch both support run with limit memory;

Usage example:

  • Creating a patch: hdiffz -s-256 [-c-lzma2] old_path new_path out_delta_file
  • Applying a patch: hpatchz old_path delta_file out_new_path



回答3:


Try sdiff It's pre-built tool in linux Distributions.

sdiff a.txt b.txt --output=c.txt

will show the files to be Modified.

This worked perfectely for me .............



来源:https://stackoverflow.com/questions/15264062/memory-exhausted-for-large-files-using-diff

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!