Upload BIG files via HTTP

Question

I\'m trying to upload really big VM Images (5-15 Gb size) to an HTTP server using PowerShell.

I tried to use for that few methods (here links to script with net.WebC

Answer 1

By default HttpWebRequest is buffering data in memory. Just set HttpWebRequest.AllowWriteStreamBuffering property to false and you would be able to upload files with almost any size. See more details at msdn

Answer 2

For uploading to Sonatype Nexus3 I used the code below. Took me some time to figure it out, working with uploads and reponse from Nexus3 and uploading and downloading large files (larger than 2GB). We have Apache in front of Nexus3 taking care of the https connections. Using basic authentication made sure that Nexus was responding right, when Apache is in front of it. Sending pre-authentication via HEAD and using chunked upload for large files fixed that uploading large files did not end prematurely.

Downloading large files via Invoke-WebRequest would fail too with different errors. Now I use WebRequest via .Net and it works, see Download-File.

And when the code was run via an automated process (from System Center Orchestrator), https would fail. So we force TLS 1.2 when scheme https is detected.

function New-HttpWebRequest
{
    <#
    .SYNOPSIS
    Creates a new [System.Net.HttpWebRequest] ready for file transmission.

    .DESCRIPTION
    Creates a new [System.Net.HttpWebRequest] ready for file transmission.
    The method will be Put. If the filesize is larger than the buffersize,
    the HttpWebRequest will be configured for chunked transfer.

    .PARAMETER Url
    Url to connect to.

    .PARAMETER Credential
    Credential for authentication at the Url resource.

    .EXAMPLE
    An example
    #>
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [pscredential]$Credential,

        [Parameter(Mandatory=$true)]
        [long]$FileSize,

        [Parameter(Mandatory=$true)]
        [long]$BufferSize
    )

    $webRequest = [System.Net.HttpWebRequest]::Create($Url)
    $webRequest.Timeout = 600 * 1000;
    $webRequest.ReadWriteTimeout = 600 * 1000;
    $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11;
    $webRequest.Method = "PUT";
    $webRequest.ContentType = "application/octet-stream";
    $webRequest.KeepAlive = $true;
    $webRequest.UserAgent = "<I use a specific UserAgent>";
    #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)';
    $webRequest.PreAuthenticate = $true;
    $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
    $webRequest.Headers["Authorization"] = "Basic $auth"

    if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize)
    {
        Write-Verbose "FileSize is greater than BufferSize, using chunked transfer.";
        $webRequest.AllowWriteStreamBuffering = $false;
        $webRequest.SendChunked = $true;
    }
    else
    {
        # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write.
        # Chunked cannot be used in this case.
        $webRequest.AllowWriteStreamBuffering = $true;
        $webRequest.SendChunked = $false;
        $webRequest.ContentLength = $FileSize;
    }

    return $webRequest;
}

function Get-BufferSize
{
    <#
    .SYNOPSIS
    Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size.

    .DESCRIPTION
    Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default.

    .PARAMETER ByteLength
    Length of the bytes for which to calculate a valid buffer size.

    .PARAMETER AtLeast
    The minimum required buffer size, default 1MB.

    .EXAMPLE
    Get-BufferSize 4283304773

    Returns 42991616 which is 41MB.

    .EXAMPLE
    Get-BufferSize 4283304

    Returns 1048576 which is 1MB.

    .EXAMPLE
    Get-BufferSize 4283304 5MB

    Returns 5242880 which is 5MB.
    #>
    param(
        [Parameter(Mandatory=$true)]
        [long]$ByteLength,

        [long]$AtLeast = 1MB
    )

    [long]$size = $ByteLength / 100;
    if ($size -lt $AtLeast)
    {
        $size = $AtLeast;
    }
    else
    {
        $size = [Math]::Round($size / 1MB) * 1MB;
    }

    return $size;
}

function Get-UseChunkedUpload
{
    param(
        [Parameter(Mandatory=$true)]
        [long]$FileSize,

        [Parameter(Mandatory=$true)]
        [long]$BufferSize
    )

    return $FileSize -gt $BufferSize;
}

function Configure-Tls
{
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url
    )

    [System.Uri]$uri = $Url;
    if ($uri.Scheme -eq "https")
    {
        Write-Verbose "Using TLS 1.2";
        [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;
    }
}

function Send-PreAuthenticate
{
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [pscredential]$Credential
    )

    $response = $null;
    try
    {
        [System.Uri]$uri = $Url;
        $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/');
        Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority";
        $wr = [System.Net.WebRequest]::Create($repositoryAuthority);
        $wr.Method = "HEAD";
        $wr.PreAuthenticate = $true;
        $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
        $wr.Headers["Authorization"] = "Basic $auth"
        $response = $wr.GetResponse();
    }
    finally
    {
        if ($response)
        {
            $response.Close();
            $response.Dispose();
            $response = $null;
        }
    }
}

function Upload-File
{
    <#
    .SYNOPSIS
    Uploads a file to the Nexus repository.

    .DESCRIPTION
    Uploads a file to the Nexus repository.
    If the file was uploaded successfully, the url via which the resource can be downloaded is returned.

    .PARAMETER Url
    The Url where the resource should be created.
    Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload.

    .PARAMETER File
    The file that should be uploaded.

    .PARAMETER Credential
    Credential used for authentication at the Nexus repository.

    .EXAMPLE
    Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential)

    .OUTPUTS
    If the file was uploaded successfully, the url via which the resource can be downloaded.
    #>
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [System.IO.FileInfo]$File,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [pscredential]$Credential
    )

    Write-Verbose "Upload-File Url:$Url"

    Configure-Tls -Url $Url;

    $fileSizeBytes = $File.Length;
    #$bufSize = Get-BufferSize $fileSizeBytes;
    $bufSize = 4 * 1MB;
    Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB));
    if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize)
    {
        Write-Verbose "Using chunked upload. Send pre-auth first.";
        Send-PreAuthenticate -Url $Url -Credential $Credential;
    }

    $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length);
    $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize;
    $chunk = New-Object byte[] $bufSize;
    $bytesWritten = 0;
    $fileStream = [System.IO.File]::OpenRead($File.FullName);
    $requestStream = $WebRequest.GetRequestStream();
    try
    {
        while($bytesRead = $fileStream.Read($chunk,0,$bufSize))
        {
            $requestStream.Write($chunk, 0, $bytesRead);
            $requestStream.Flush();
            $bytesWritten += $bytesRead;
            $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten / 1MB));
            Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100);
        }
    }
    catch
    {
        throw;
    }
    finally
    {
        if ($fileStream)
        {
            $fileStream.Close();
        }
        if ($requestStream)
        {
            $requestStream.Close();
            $requestStream.Dispose();
            $requestStream = $null;
        }
        Write-Progress -Activity $progressActivityMessage -Completed;
    }

    # Read the response.
    $response = $null;
    try
    {
        $response = $webRequest.GetResponse();
        Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri);
        return $response.ResponseUri;
    }
    catch
    {
        if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*"))
        {
            throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.")
        }

        throw;
    }
    finally
    {
        if ($response)
        {
            $response.Close();
            $response.Dispose();
            $response = $null;
        }
        if ($webRequest)
        {
            $webRequest = $null;
        }
    }
}

function Download-File
{
    param(
        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$Url,

        [Parameter(Mandatory=$true)]
        [ValidateNotNullOrEmpty()]
        [string]$FileName
    )

    $SDXDownloadType = @"
    using System.IO;
    using System.Net;

    public class SDXDownload
    {
        static public void DownloadFile(string Uri, string Filename)
        {
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri);
            webRequest.Method = "GET";
            using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse())
            using (Stream fileStream = File.OpenWrite(Filename))
            using (Stream streamResponse = myHttpWebResponse.GetResponseStream())
            {
                int bufSize = 64 * 1024;
                byte[] readBuff = new byte[bufSize];
                int bytesRead = streamResponse.Read(readBuff, 0, bufSize);
                while (bytesRead > 0)
                {
                    fileStream.Write(readBuff, 0, bytesRead);
                    bytesRead = streamResponse.Read(readBuff, 0, 256);
                }
            }
        }
    }
"@

    Configure-Tls -Url $Url;

    Add-Type -TypeDefinition $SDXDownloadType;
    [SDXDownload]::DownloadFile($Url, $FileName);
}

Answer 3

Thank you @Stoune, it was last thing that helped to receive finally working solution.

One more, it is need to organize stream file reading and writing to the webrequest buffer. And it possibly to do with that piece of code:

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

And final script look like this:

$user = "admin"
$pass = "admin123"
$dir = "C:\Virtual Hard Disks"
$fileName = "win2012r2std.vhdx"
$file = "$dir/$fileName"
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"
$Timeout=10000000
$bufSize=10000

$cred = New-Object System.Net.NetworkCredential($user, $pass)

$webRequest = [System.Net.HttpWebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
$webRequest.AllowWriteStreamBuffering=$false
$webRequest.SendChunked=$true # needed by previous line
$webRequest.Credentials = $cred

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

$responceStream = $webRequest.getresponse()
#$status = $webRequest.statuscode

$FileStream.Close()
$requestStream.Close()
$responceStream.Close()

$responceStream
$responceStream.GetResponseHeader("Content-Length") 
$responceStream.StatusCode
#$status