Is it possible to get SQL Server to log into an SFTP and upload a file just by using T-SQL commands, no SSIS?

Question

I have an SSIS package which logs into my client's SFTP site and uploads a file using commands sent to WinSCP.

This package fails quite often and is difficult to work with. Visual Studio crashes so frequently that my team refuses to use it any more. We are migrating all automated tasks to Stored Procedures and SQL Agent Jobs, as they work a lot better than SSIS does.

However, this leaves us with no good solution for automating FTP uploads of report files produced on SQL Server.

I have been doing it using PSFTP and batch files called from VB scripts. The script checks a particular folder to see if a file is there and uploads it if it is. This works, but is inelegant and I have to schedule these scripts to look for a file in the folder every 15 minutes, rather than just call an "upload" command from my stored procedure on SQL Server. There's potential for things to go horribly wrong if someone renames a file in that folder, or puts two files in there, etc.

I can easily create a .CSV file as part of a stored procedure into a particular folder using BCP, e.g.:

--Create Client Update .CSV File if there are any rows in the CLIENT_UPDATE table
IF EXISTS(SELECT 1 FROM CLIENT_UPDATE)
BEGIN
DECLARE @ColumnHeader VARCHAR(8000)
    SET @FileName = @DataPath + '\Output File.csv'
 SELECT @ColumnHeader = COALESCE(@ColumnHeader+',' ,'')+''''+column_name+'''' FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='CLIENT_UPDATE'
    SET @BCPCommand = 'bcp "SELECT '+ @ColumnHeader +' UNION ALL SELECT * FROM CLIENT_UPDATE" queryout "' + @FileName + '" -c -t , -r \n  -S . -T'
   EXEC master..xp_cmdshell @bcpCommand
END

I'd like the next step in that stored procedure to be "Upload .CSV file to client SFTP site". How would I accomplish this?

Answer 1

Success!

I created a stored proc on my database named sp_UploadFileToSFTP, which uses PSFTP.EXE to send files.

-- ================================================================================================================================
-- Description: Upload File to SFTP
--              - Creates "SFTPUploadScript.bat" and "SFTPUploadScript.txt" scripting files to control psftp.exe, then executes the batch file
--              - Requires a local @ScriptFolder on the SQL Server to create files inside, and which contains psftp.exe
--              - "SFTPUploadScript.bat" needs to be run once manually from the SQL Server desktop to capture the keypair of the SFTP site
--                After this, the script will work automatically
-- ================================================================================================================================
CREATE PROCEDURE sp_UploadFileToSFTP 

    @FilePathToUpload varchar(1000),
    @SFTPSiteAddress varchar(1000),
    @SFTPLogin varchar(1000),
    @SFTPPassword varchar(1000),
    @SFTPRemoteFolder varchar(1000)

AS
BEGIN
SET NOCOUNT ON;

--Declare Variables
 DECLARE @ScriptFolder varchar(1000)
     SET @ScriptFolder = 'C:\SFTPBatchFiles\'
 DECLARE @CommandString AS varchar(8000)

--Delete Existing files if they exist
     SET @CommandString = 'del "'+ @ScriptFolder + 'SFTPUploadScript.txt"'
    EXEC master..xp_cmdshell @CommandString
     SET @CommandString = 'del "'+ @ScriptFolder + 'SFTPUploadScript.bat"'
    EXEC master..xp_cmdshell @CommandString

--Create Batch fle with login credentials
     SET @CommandString = 'echo "'+ @ScriptFolder +'psftp.exe" ' + @SFTPSiteAddress + ' -l ' + @SFTPLogin + ' -pw ' + @SFTPPassword + ' -b "' + @ScriptFolder + 'SFTPUploadScript.txt" > "' + @ScriptFolder + 'SFTPUploadScript.bat"'
    EXEC master..xp_cmdshell @CommandString

--Create SFTP upload script file
     SET @CommandString = 'echo cd "' + @SFTPRemoteFolder + '" > "' + @ScriptFolder + 'SFTPUploadScript.txt"'
    EXEC master..xp_cmdshell @CommandString
     SET @CommandString = 'echo put "' + @FilePathToUpload + '" >> "' + @ScriptFolder + 'SFTPUploadScript.txt"'
    EXEC master..xp_cmdshell @CommandString

--Run the Batch File
     SET @CommandString = @ScriptFolder + 'SFTPUploadScript.bat'
    EXEC master..xp_cmdshell @CommandString

END
GO

I can then call it from inside another stored proc like so:

sp_UploadFileToSFTP 'C:\FileToUpload\Test.txt','sftp.mysite.com','Login@domain.com','Password1234','UploadFolder/In here/'

All works perfectly, takes about a quarter of a second to run, a very neat and stable solution.

Compared to what I was doing before which was using SQL Server to drop a file into a folder, then checking that folder for new files every 15 minutes with my Visual Basic SFTP upload script, it's miles better :)

If you want to try this yourself, all you need is a folder on the C:\ drive of your SQL Server named "SFTPBatchFiles", and in there you put psftp.exe, which is part of PuTTY.

You will also need a trusting server admin who will allow you to run psftp commands and your own batch files using xp_cmdshell, and they'll probably need to open a route in your firewall so you can connect to your remote site directly from the SQL Server.

Finally, you will also need to be able to remote desktop into your SQL Server, since you'll need to run the batch file it creates manually once and press "Y" when psftp asks you to accept the new keypair. After that, it's all plain sailing.

No SSIS, no WINSCP, no Visual Studio, no hours upon hours of crashes and debugging!

The only issue right now is that if the FTP transfer fails for whatever reason, the stored procedure still reports success, I'm not bringing the psftp error messages back into SQL Server.

My next task will be to trap the status and error messages as they get generated from xp_cmdshell and give the user some detailed feedback and generate errors if the process fails.

UPDATE:

Trapping errors and feedback from the remote FTP site

--Run the Batch File
  DROP TABLE IF EXISTS #CommandOutput
CREATE TABLE #CommandOutput (ReadLine varchar(1000))
         SET @CommandString = @ScriptFolder + 'SFTPUploadScript.bat'
      INSERT #CommandOutput
        EXEC master..xp_cmdshell @CommandString

--Check for Invalid Password error  
IF EXISTS (SELECT ReadLine 
            FROM #CommandOutput 
           WHERE ReadLine LIKE '%Invalid Password%')
    BEGIN
          PRINT 'Invalid Password Error!'
      END

All of the output from the batch file (what would normally be shown on the screen if you were to run it manually) is now inserted line-by-line into a temporary table named #CommandOutput as the batch file runs.

After the batch file runs, you can then query #CommandOutput to see what's happened with your transfer.

I've added a simple check for the words "Invalid Password" anywhere in the output, as this is a common error you might have. You could add as many of these checks as you liked, or you could import the entire table into an email which gets sent to you every time the job runs, log the table to an FTP event logging table, or any number of other things.