Posting multiple files using invokehttp to API using nifi

问题

I am having 3 files which i need to post to API end. I am fetching 3 files using FetchHDFS process and i want to pass them to API. How can i pass them.

Input:

3 files in HDFS
Content-Type: multipart/form-data

Error:

invokehttp.response.body
{ "message": "Multipart: Boundary not found (user: 'undefined')", "level": "error", "timestamp": "2019-12-11T09:59:05.464Z" }

Flow tried:

inputPort --> 3 FetchHDFS process to fetch 3 different file --> invokeHttps

curl commnad sample:

curl -X POST "https://xxxxxx/xxxxx" -H "accept: application/json" -H "Content-Type: multipart/form-data" -F "file1=@File1.csv;type=application/vnd.ms-excel" -F "file2=@File2.txt;type=text/plain" -F "file3=@File3.csv;type=application/vnd.ms-excel" -F "format=flat"

回答1:

The idea: Before building multipart from several flow files you need to merge them into one.

For this use the MergeContent processor with Merge Format = TAR.

Then use ExecuteGroovyScript to convert TAR to multipart.

@Grab(group='org.apache.httpcomponents', module='httpmime', version='4.5.9')
@Grab(group='org.apache.commons', module='commons-compress', version='1.19')
import org.apache.commons.compress.archivers.tar.TarArchiveInputStream
import org.apache.http.entity.mime.MultipartEntityBuilder
import org.apache.http.entity.ContentType

def ff = session.get()
if(!ff)return

//delegate inputstream class to be able to set the `delegate` later
@groovy.transform.CompileStatic
class TarContentInputStream extends InputStream{
    @Delegate TarArchiveInputStream delegate
    @Override void close(){
        println "--------- try to close"
        if(!delegate.getNextTarEntry())delegate.close()
    }
}

def multipart = MultipartEntityBuilder.create()
def tarContent = new TarContentInputStream()

//iterate through TAR entries and build multipart
def tarInput=new TarArchiveInputStream(ff.read())
def tarEntry = tarInput.getNextTarEntry()
while (tarEntry != null) {
    //reference tarContent to be used as body
    multipart.addBinaryBody( tarEntry.getName(), tarContent, ContentType.DEFAULT_BINARY, tarEntry.getName() )
    tarEntry = tarInput.getNextTarEntry()
}
tarInput.close()
//write multipart content
ff.write{streamIn, streamOut->
    //set real input stream to be used as tar content
    tarContent.delegate = new TarArchiveInputStream(streamIn)
    assert tarContent.delegate.getNextTarEntry() //move to first entry
    multipart = multipart.build()
    multipart.writeTo(streamOut)
}

ff."mime.type" = multipart.getContentType().getValue()
ff.filename = ff.filename+".multipart"

REL_SUCCESS << ff

Note:

for 3 test files combined into tar the code above produces something like:

--boundary
Content-Disposition: form-data; name="file1.txt"; filename="file1.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

file1 content
--boundary
Content-Disposition: form-data; name="file2.txt"; filename="file2.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

file2 content
--boundary
Content-Disposition: form-data; name="file3.txt"; filename="file3.txt"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

file3 content
--boundary--

Currently code scans input stream two times: 1st - to scan tar entries, 2nd - to build content.

I think it's possible to rewrite the code to convert tar to multipart in one shot...

来源：https://stackoverflow.com/questions/59374728/posting-multiple-files-using-invokehttp-to-api-using-nifi

标签

api

apache-nifi