ConsumeAzureEventHub

Description:

Receives messages from Azure Event Hubs, writing the contents of the message to the content of the FlowFile.

Tags:

azure, microsoft, cloud, eventhub, events, streaming, streams

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

Name	Default Value	Allowable Values	Description
Event Hub Namespace			The namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns. Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub Name			The name of the event hub to pull messages from. Supports Expression Language: true (will be evaluated using variable registry only)
Service Bus Endpoint	.servicebus.windows.net	Azure Azure China Azure Germany Azure US Government	To support namespaces not in the default windows.net domain.
Shared Access Policy Name			The name of the shared access policy. This policy must have Listen claims. Supports Expression Language: true (will be evaluated using variable registry only)
Shared Access Policy Primary Key			The primary key of the shared access policy Sensitive Property: true
Use Azure Managed Identity	false	true false	Choose whether or not to use the managed identity of Azure VM/VMSS
Consumer Group	$Default		The name of the consumer group to use. Supports Expression Language: true (will be evaluated using variable registry only)
Consumer Hostname			The hostname of this event hub consumer instance. If not specified, an unique identifier is generated in 'nifi-<UUID>' format. Supports Expression Language: true (will be evaluated using variable registry only)
Record Reader		Controller Service API: RecordReaderFactory Implementations: JsonPathReader AvroReader XMLReader WindowsEventLogReader ReaderLookup Syslog5424Reader GrokReader ScriptedReader CSVReader SyslogReader ParquetReader JsonTreeReader CEFReader	The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
Record Writer		Controller Service API: RecordSetWriterFactory Implementations: AvroRecordSetWriter ScriptedRecordSetWriter JsonRecordSetWriter ParquetRecordSetWriter RecordSetWriterLookup FreeFormTextRecordSetWriter XMLRecordSetWriter CSVRecordSetWriter	The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.
Initial Offset	end-of-stream	Start of stream End of stream	Specify where to start receiving messages if offset is not stored in Azure Storage.
Prefetch Count	300		No Description Provided. Supports Expression Language: true (will be evaluated using variable registry only)
Batch Size	10		The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent. Supports Expression Language: true (will be evaluated using variable registry only)
Message Receive Timeout	1 min		The amount of time this consumer should wait to receive the Prefetch Count before returning. Supports Expression Language: true (will be evaluated using variable registry only)
Storage Account Name			Name of the Azure Storage account to store event hub consumer group state. Supports Expression Language: true (will be evaluated using variable registry only)
Storage Account Key			The Azure Storage account key to store event hub consumer group state. Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only)
Storage SAS Token			The Azure Storage SAS token to store Event Hub consumer group state. Always starts with a ? character. Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only)
Storage Container Name			Name of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used. Supports Expression Language: true (will be evaluated using variable registry only)

Relationships:

Name	Description
success	FlowFiles received from Event Hub.
parse.failure	If a message from event hub cannot be parsed using the configured Record Reader or failed to be written by the configured Record Writer, the contents of the message will be routed to this Relationship as its own individual FlowFile.

Reads Attributes:

None specified.

Writes Attributes:

Name	Description
eventhub.enqueued.timestamp	The time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub
eventhub.offset	The offset into the partition at which the message was stored
eventhub.sequence	The sequence number associated with the message
eventhub.name	The name of the event hub from which the message was pulled
eventhub.partition	The name of the partition from which the message was pulled
eventhub.property.*	The application properties of this message. IE: 'application' would be 'eventhub.property.application'

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

None specified.