Check out git-lfs files on VSTF build

Question

I've a repository hosted on VSTS, containig a file stored through git-lfs. If I just let VSTS build checkout the repository it just downloads the git-lfs metadata file containing the ID of the file.

Here's the output how VSTS gets its source:

Syncing repository: MyRepo (Git)
Checking out c84ef2f2bbad4fa3dc70dbd4100534390b9c8f18 to d:\work\73\s
Checked out branch refs/heads/develop for repository MyRepo at commit c84ef2f2bbad4fa3dc70dbd4100534390b9c8f18

What do I need to do to checkout the real file?

Edit: I assume I need to manually call git lfs fetch after VSTS checked out the source. But how can I handle authentication (which is required by VSTS) in this case?

Answer 1

The process has been updated yet again (March 2017). This time you need to edit the "Get Sources" part of your build definition. Enable the "Advanced Settings" option on the top right and check the option to "Checkout files from LFS".

Answer 2

Update

VSTS now supports git LFS out of the box. It's just a matter of activating the option Repository / Checkout files from LFS in the build definition. It is much simpler than the solution below.

---

I tried Pascal's Enable Git Remote Access build task but I was not able to make it work. Calling git-lfs.exe does not crash but it does not convert the LFS files to the real files.

Here is how I was able to make it work. I first had to enable the Allow Scripts to Access OAuth Token option in my build definition. I then created a PowerShell script that pulls the LFS dependencies:

# Inspired from here: http://ss64.com/ps/syntax-set-eol.html
function Set-UnixLineEndings([string]$file)
{
    # Replace CR+LF with LF
    $text = [IO.File]::ReadAllText($file) -replace "`r`n", "`n"
    [IO.File]::WriteAllText($file, $text)

    # Replace CR with LF
    $text = [IO.File]::ReadAllText($file) -replace "`r", "`n"
    [IO.File]::WriteAllText($file, $text)
}

if ((Test-Path env:SYSTEM_ACCESSTOKEN) -eq $false)
{
    throw "OAuth token not available. Make sure that you select the option 'Allow Scripts to Access OAuth Token' in build 'Options' pane."
}

# git lfs needs the credentials of the git repository. When running
# under VSTS, these credentials are transfered to the git-lfs.exe
# application using the oauth token provided by VSTS. These
# credentials are stored in a file so that git lfs can find them.

$pwPath = Join-Path $PSScriptRoot pw.txt
$gitPwPath = $pwPath.Replace('\', '/')    # Needs to be in unix format.

$repoUri = New-Object Uri $env:BUILD_REPOSITORY_URI

git config credential.helper "store --file=$gitPwPath"
@"
https://OAuth:$env:SYSTEM_ACCESSTOKEN@$($repoUri.Host)
"@ | Set-Content $pwPath

# Again, needs to be in unix format... sigh...
Set-UnixLineEndings -file $pwPath

& ".\git-lfs.exe" pull
if ($LASTEXITCODE -ne 0)
{
    throw 'Failed to pull LFS files.'
}

This obviously assumes that you have stored git-lfs.exe in your git repository AND that this file is not tracked by LFS.

Answer 3

Update

I confirm the process changed, please ignore the following answer.

---

I have to say I just find that :

In your build definition, select Repository tab, and check the option Checkout files from LFS

It can't be easier.

Answer 4

For TFS Build 2015 Update 4 (the self-hosted flavor of this thing, as opposed to VSTS, the cloud service), the "Checkout files from LFS" option does not exist, and as such, we had to do this:

• Check Allow Scripts to Access OAuth Token on the Options tab.

• Commit this batch file into the repository somewhere:

  REM This script is intended to fetch large (LFS) files during the TFS Build process.
  
  REM Solution derived from https://github.com/Microsoft/vsts-agent/issues/1134
  
  git config --unset-all http.extraheader
  git config --add http.extraheader "AUTHORIZATION: bearer %SYSTEM_ACCESSTOKEN%"
  git lfs fetch
  git lfs checkout
  git config --unset http.extraheader        
  

• Add a build step of a Batch Script, and run the above script.

Answer 5

VSTS has now an Allow Scripts to Access OAuth Token option. With this option set on the build definition the OAuth is available to build scripts.

I've created an extension containing build tasks which change the remote Url to use the OAuth token to access the remote repository.