Using PowerShell v3's Invoke-RestMethod to PUT/POST X Mb of a binary file

Question

I have been doing a bit of work with PowerShell v3 (CTP2 from here) and its new Invoke-RestMethod like so:

Invoke-RestMethod -Uri $dest -method PUT -Credential $cred -InFile $file

However, I'd like to use this for pushing very large binary objects, and therefore like be able to push a range of bytes from a large binary file.

For example, if I have a 20Gb VHD, I would like to break it up into chunks of say, 5Gb each (without splitting and saving the individual chunks first) and PUT/POST these to BLOB storage like S3, Rackspace, Azure, etc. I am also assuming the chunk size is larger than the available memory.

I've read Get-Content does not work very efficiently on large binary files, but this doesn't seem like an obscure requirement. Does anyone have any appraoches which could be used for this, particularly in conjunction with PowerShell's new Invoke-RestMethod?

Answer 1

I believe the Invoke-RestMethod parameter you're looking for is

-TransferEncoding Chunked

but there is no control over the chunk or buffer size. Someone can correct me if I am wrong, but I think the chunk size is 4KB. Each chunk gets loaded into memory and then sent, so you memory doesn't fill up with the file you are sending.

Answer 2

To retrieve sections (chunks) of a file, you can create a System.IO.BinaryReader which has a handy dandy Read( [Byte[]] buffer, [int] offset, [int] length) method. Here's a function to make it easy:

function Read-Bytes {
    [CmdletBinding()]
    param (
          [Parameter(Mandatory = $true, Position = 0)]
          [string] $Path
        , [Parameter(Mandatory = $true, Position = 1)]
          [int] $Offset
        , [Parameter(Mandatory = $true, Position = 2)]
          [int] $Size
    )

    if (!(Test-Path -Path $Path)) {
        throw ('Could not locate file: {0}' -f $Path);
    }

    # Initialize a byte array to hold the buffer
    $Buffer = [Byte[]]@(0)*$Size;

    # Get a reference to the file
    $FileStream = (Get-Item -Path $Path).OpenRead();

    if ($Offset -lt $FileStream.Length) {
        $FileStream.Position = $Offset;
        Write-Debug -Message ('Set FileStream position to {0}' -f $Offset);
    }
    else {
        throw ('Failed to set $FileStream offset to {0}' -f $Offset);
    }

    $ReadResult = $FileStream.Read($Buffer, 0, $Size);
    $FileStream.Close();

    # Write buffer to PowerShell pipeline
    Write-Output -InputObject $Buffer;

}

Read-Bytes -Path C:\Windows\System32\KBDIT142.DLL -Size 10 -Offset 90;