GrokReader

Description:

Provides a mechanism for reading unstructured text data, such as log files, and structuring the data so that it can be processed. The service is configured using Grok patterns. The service reads from a stream of data and splits each message that it finds into a separate Record, each containing the fields that are configured. If a line in the input does not match the expected message pattern, the line of text is either considered to be part of the previous message or is skipped, depending on the configuration, with the exception of stack traces. A stack trace that is found at the end of a log message is considered to be part of the previous message but is added to the 'stackTrace' field of the Record. If a record has no stack trace, it will have a NULL value for the stackTrace field (assuming that the schema does in fact include a stackTrace field of type String). Assuming that the schema includes a '_raw' field of type String, the raw message will be included in the Record.

Additional Details...

Tags:

grok, logs, logfiles, parse, unstructured, text, record, reader, regex, pattern, logstash

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Schema Access Strategystring-fields-from-grok-expression
  • Use String Fields From Grok Expression The schema will be derived by using the field names present in the Grok Expression. All fields will be assumed to be of type String. Additionally, a field will be included with a name of 'stackTrace' and a type of String.
  • Use 'Schema Name' Property The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service.
  • Use 'Schema Text' Property The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions.
  • HWX Schema Reference Attributes The FlowFile contains 3 Attributes that will be used to lookup a Schema from the configured Schema Registry: 'schema.identifier', 'schema.version', and 'schema.protocol.version'
  • HWX Content-Encoded Schema Reference The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, found at https://github.com/hortonworks/registry
  • Confluent Content-Encoded Schema Reference The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This is based on version 3.2.x of the Confluent Schema Registry.
Specifies how to obtain the schema that is to be used for interpreting the data.
Schema RegistryController Service API:
SchemaRegistry
Implementations: HortonworksSchemaRegistry
AvroSchemaRegistry
ConfluentSchemaRegistry
Specifies the Controller Service to use for the Schema Registry

This Property is only considered if the <Schema Access Strategy> Property is set to one of the following values: "confluent-encoded", "schema-name", "hwx-schema-ref-attributes", "hwx-content-encoded-schema"
Schema Name${schema.name}Specifies the name of the schema to lookup in the Schema Registry property
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the <Schema Access Strategy> Property has a value of "schema-name".
Schema VersionSpecifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the <Schema Access Strategy> Property has a value of "schema-name".
Schema BranchSpecifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the <Schema Access Strategy> Property has a value of "schema-name".
Schema Text${avro.schema}The text of an Avro-formatted Schema
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the <Schema Access Strategy> Property has a value of "schema-text-property".
Grok Pattern FilePath to a file that contains Grok Patterns to use for parsing logs. If not specified, a built-in default Pattern file will be used. If specified, all patterns in the given pattern file will override the default patterns. See the Controller Service's Additional Details for a list of pre-defined patterns.

This property requires exactly one file to be provided..

Supports Expression Language: true (will be evaluated using variable registry only)
Grok ExpressionSpecifies the format of a log line in Grok format. This allows the Record Reader to understand how to parse each log line. If a line in the log file does not match this pattern, the line will be assumed to belong to the previous log message.If other Grok expressions are referenced by this expression, they need to be supplied in the Grok Pattern File
No Match Behaviorappend-to-previous-message
  • Append to Previous Message The line of text that does not match the Grok Expression will be appended to the last field of the prior message.
  • Skip Line The line of text that does not match the Grok Expression will be skipped.
  • Raw Line The line of text that does not match the Grok Expression will only be added to the _raw field.
If a line of text is encountered and it does not match the given Grok Expression, and it is not part of a stack trace, this property specifies how the text should be processed.

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified.