Difference between Apache NiFi and StreamSets

Question

I am planning to do a class project and was going through few technologies where I can automate or set the flow of data between systems and found that there are couple of th

Answer 1

Both Apache NiFi and StreamSets Data Collector are Apache-licensed open source tools.

Hortonworks does have a commercially supported variant called Hortonworks DataFlow (HDF).

While both have a lot of similarities such as a web-based ui, both are used for ingesting data there are a few key differences. They also both consist of a processors linked together to perform transformations, serialization, etc.

NiFi processors are file-oriented and schemaless. This means that a piece of data is represented by a FlowFile (this could be an actual file on disk, or some blob of data acquired elsewhere). Each processor is responsible for understanding the content of the data in order to operate on it. Thus if one processor understands format A and another only understands format B, you may need to perform a data format conversion in between those two processors.

NiFi can be run standalone, or as a cluster using its own built-in clustering system.

StreamSets Data Collector (SDC) however, takes a record based approach. What this means is that as data enters your pipeline it (whether its JSON, CSV, etc) it is parsed into a common format so that the responsibility of understanding the data format is no longer placed on each individual processor and any processor can be connected to any other processor.

SDC also runs standalone, and also a clustered mode, but it runs atop Spark on YARN/Mesos instead, leveraging existing cluster resources you may have.

NiFi has been around for about the last 10 years (but less than 2 years in the open source community).

StreamSets was released to the open source community a little bit later in 2015. It is vendor agnostic, and as far as Hadoop goes Hortonworks, Cloudera, and MapR are all supported.

Full Disclosure: I am an engineer who works on StreamSets.