Parameter expansion slow for large data sets

Question

If I take the first 1,000 bytes from a file, Bash can replace some characters pretty quick

$ cut -b-1000 get_video_i         


        

Answer 1

Originally, older shells and other utilities imposed LINE_MAX = 2048 on file input for this kind of reason. For huge variables bash has no problem parking them in memory. But substitution requires at least two concurrent copies. And lots of thrashing: as groups of characters are removed whole strings get rewritten. Over and over and over.

There are tools meant for this - sed is a premiere choice. bash is a distant second choice. sed works on streams, bash works on memory blocks.

Another choice: bash is extensible - your can write custom C code to stuff stuff well when bash was not meant to do it.

CFA Johnson has good articles on how to do that:

Some ready to load builtins:

http://cfajohnson.com/shell/bash/loadables/

DIY builtins explained:

http://cfajohnson.com/shell/articles/dynamically-loadable/