Fast and simple binary concatenate files in Powershell

Question

What\'s the best way of concatenating binary files using Powershell? I\'d prefer a one-liner that simple to remember and fast to execute.

The best I\'ve come up with

Answer 1

Performance is very much dependent on the buffer size used. Those are fairly small by default. Concatenating 2x2GB files I'd take a buffersize of about 256kb. Going larger might sometimes fail, smaller and you'll get less throughput than your drive is capable of.

With gc that'd be with -ReadCount not simply -Read (PowerShell 5.0):

gc -ReadCount 256KB -Path $infile -Encoding Byte | ...

Plus I found Add-Content to be better and going file-by-file for a lot of small files, because piping only a moderate amount of data (200MB) I found my computer going oom, PowerShell freezing and CPU at full.

Although Add-Content randomly fails a few times for a few hundred files with an error about the destination file being in use, so I added a while loop and a try catch:

# Empty the file first
sc -Path "$path\video.ts" -Value @() -Encoding Byte 
$tsfiles | foreach {    
    while ($true) {
        try { # I had -ReadCount 0 because the files are smaller than 256KB
            gc -ReadCount 0 -Path "$path\$_" -Encoding Byte | `
                Add-Content -Path "$path\video.ts" -Encoding Byte -ErrorAction Stop
            break;
        } catch {
        }
    }
}

Using a file stream is much faster still. You cannot specify a buffer size with [System.IO.File]::Open but you can with new [System.IO.FileStream] like so:

# $path = "C:\"
$ins = @("a.ts", "b.ts")
$outfile = "$path\out.mp4"
$out = New-Object -TypeName "System.IO.FileStream" -ArgumentList @(
    $outfile, 
    [System.IO.FileMode]::Create,
    [System.IO.FileAccess]::Write,
    [System.IO.FileShare]::None,
    256KB,
    [System.IO.FileOptions]::None)
try {
    foreach ($in in $ins) {
        $fs = New-Object -TypeName "System.IO.FileStream" -ArgumentList @(
            "$path\$in", 
            [System.IO.FileMode]::Open,
            [System.IO.FileAccess]::Read,
            [System.IO.FileShare]::Read,
            256KB,
            [System.IO.FileOptions]::SequentialScan)
        try {
            $fs.CopyTo($out)
        } finally {
            $fs.Dispose()
        }
    }
} finally {
    $out.Dispose()
}