apache-nifi

Nifi Encrypt json

由 谁都会走 提交于 2019-12-07 23:51:04
问题 I would like to use nifi to encrypt the attributes in a json but not the keys as I would like to upload the data to a mongodb server. Is there a way to do this? For the project I an using twitter data as a proof of concept. So far I have used the EvaluateJsonPath processor to extract only the text of the tweet, and I can encrypt this text, however the resulting json no longer has a key. Can Nifi recreate a json that attaches a key to this attribute that I extracted? Is there a better way to

How can I pass a flowfile attribute to a controller service?

由 别说谁变了你拦得住时间么 提交于 2019-12-07 22:19:29
问题 Hello Internet Hive Mind! I need to query AWS Athena with nifi, however I need to change the staging directory (the S3 bucket & folder where the results will be saved) for each query sent. But the s3_staging_dir property has to be set on the DBCPConnectionPool Controller Service. How can I change the value of that property for each different flow file? Apparently it can't be fetched by expression language alone. Thanks! 回答1: I'm not sure the nature of your flow where each query depends on a

ExecuteSQL doesn't select table if it having dateTime Offset value?

由 六眼飞鱼酱① 提交于 2019-12-07 15:18:32
I have created table with single column having data type -dateTimeOffset value and inserted some values. create table dto (dto datetimeoffset(7)) insert into dto values (GETDATE()) -- inserts date and time with 0 offset insert into dto values (SYSDATETIMEOFFSET()) -- current date time and offset insert into dto values ('20131114 08:54:00 +10:00') -- manual way In Nifi,i have specified "Select * from dto" query in Execute SQL . It shows below error.., java.lang.IllegalArgumentException: createSchema: Unknown SQL type -155 cannot be converted to Avro type If i change that column into dateTime

How to map the column wise data in flowfile in NiFi?

由 孤人 提交于 2019-12-07 14:33:59
问题 i have csv file which having following structure., Alfreds,Centro,Ernst,Island,Bacchus Germany,Mexico,Austria,UK,Canada 01,02,03,04,05 Now i have to move that data into database like below. Name,City,ID Alfreds,Germay,01 Centro,Mexico,02 Ernst,Austria,03 Island,UK,04 Bacchus,Canda,05 i try to map those colums but i can't able to extract the data in column wise. Here my input data in column wise but i need to insert those in row wise in SQLServer Can anyone suggest way to transfer column wise

DBCPConnectionPool controller service for SQL Server, jdbc exception

由 牧云@^-^@ 提交于 2019-12-07 10:58:57
问题 NiFi 1.1.1 tested on both Windows 7 and RHEL 7. The background thread is here. I have created a DBCPConnectionPool controller service pointing to a SQL Server db, I am able to fetch data from a table and write it to the local disk(ExecuteSQL -> ConvertAvroToJSON -> PutFile). My code: public byte[] getMaxLSN(Connection connection, String containerDB) { String dbMaxLSN = "{? = CALL sys.fn_cdc_get_max_lsn()}"; byte[] maxLSN = null; try (final CallableStatement cstmt = connection.prepareCall

Configuring HTTP POST request from Nifi

由 霸气de小男生 提交于 2019-12-07 07:43:13
问题 I am trying to access a WCF service from a REST client. I am sending a POST request from a REST client to a WCF service. For your reference, the detail is as follows. The Service Contract definition is as follows: [ServiceContract] public interface IBZTsoftsensor_WcfService { [OperationContract] [WebInvoke(Method = "POST", RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json, BodyStyle = WebMessageBodyStyle.Wrapped, UriTemplate = "/data")] string ExecuteModelJson

NiFi : Regular Expression in ExtractText gets CSV header instead of data

由 无人久伴 提交于 2019-12-07 06:45:49
问题 I'm working on a flow where I get CSV files. I want to put the records into different directories based on the first field in the CSV record. For ex, the CSV file would look like this country,firstname,lastname,ssn,mob_num US,xxxx,xxxxx,xxxxx,xxxx UK,xxxx,xxxxx,xxxxx,xxxx US,xxxx,xxxxx,xxxxx,xxxx JP,xxxx,xxxxx,xxxxx,xxxx JP,xxxx,xxxxx,xxxxx,xxxx I want to get the field value of the first field i.e, country. Put those records into a particular directory. US records goes to US directory, UK

Nifi processor is not parsing JSON correctly

由 自作多情 提交于 2019-12-07 05:12:28
I am using EvaluateJsonPath to extract one particular value from JSON. I am using the follwoing JSONPath expression: $.data[?(@.containerType == 'SOURCE' && @.path == 'SOURCE_KYLO_DATALAKE')].id This is the JSON document I'm calling the JSONPath on : {"data":[{"id":"dc18bf87-c5a6-4600-9584-e79fb988b1d0","path":["@Rakesh.Prasad@diageo.com"],"tag":"0","type":"CONTAINER","containerType":"HOME"},{"id":"42e52055-4deb-4d5d-942f-4e1c4e48c35e","path":["BPM"],"tag":"3","type":"CONTAINER","containerType":"SPACE"},{"id":"49e3d118-e4f9-41ef-ad97-6b2745c75c4f","path":["DATABRICKS_USAGE_REPORT"],"tag":"0",

How to clear NiFi queues?

由 吃可爱长大的小学妹 提交于 2019-12-07 05:09:53
问题 We are creating some flows in NiFi and there might be some cases where the queues are being build up but due to some reason the flow doesn't work as expected. At the end of the day, i would like to clear the queues and somehow would like to automate it. The question is how can we delete the queues from backend? Is there any way we can achieve that? 回答1: In addition to the explicit "Drop Queue" function Bryan mentioned, a couple other features you may be interested are the "Back Pressure" and

Is it possible to remove white spaces from the CSV files header name in NiFi?

由 拥有回忆 提交于 2019-12-07 04:56:19
问题 I have a CSV file in which some column name have white spaces in it and some column names are without the white space between characters. I want to remove the white spaces from all the header names that has white space in it. Please help. Thank you! Attaching screenshot for reference. Example: 'First Name' I want 'FirstName' I am using ReplaceText processor in which under Search value I have passes \s to search just the header row white spaces and replacement value as Empty string. Also my
订阅 apache-nifi