FetchS3Object

Description:

Retrieves the contents of an S3 Object and writes it to the content of a FlowFile

Tags:

Amazon, S3, AWS, Get, Fetch

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
BucketNo Description Provided.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Object Key${filename}No Description Provided.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Regionus-west-2
  • AWS GovCloud (US) AWS Region Code : us-gov-west-1
  • AWS GovCloud (US-East) AWS Region Code : us-gov-east-1
  • US East (N. Virginia) AWS Region Code : us-east-1
  • US East (Ohio) AWS Region Code : us-east-2
  • US West (N. California) AWS Region Code : us-west-1
  • US West (Oregon) AWS Region Code : us-west-2
  • EU (Ireland) AWS Region Code : eu-west-1
  • EU (London) AWS Region Code : eu-west-2
  • EU (Paris) AWS Region Code : eu-west-3
  • EU (Frankfurt) AWS Region Code : eu-central-1
  • EU (Stockholm) AWS Region Code : eu-north-1
  • EU (Milan) AWS Region Code : eu-south-1
  • Asia Pacific (Hong Kong) AWS Region Code : ap-east-1
  • Asia Pacific (Mumbai) AWS Region Code : ap-south-1
  • Asia Pacific (Singapore) AWS Region Code : ap-southeast-1
  • Asia Pacific (Sydney) AWS Region Code : ap-southeast-2
  • Asia Pacific (Jakarta) AWS Region Code : ap-southeast-3
  • Asia Pacific (Tokyo) AWS Region Code : ap-northeast-1
  • Asia Pacific (Seoul) AWS Region Code : ap-northeast-2
  • Asia Pacific (Osaka) AWS Region Code : ap-northeast-3
  • South America (Sao Paulo) AWS Region Code : sa-east-1
  • China (Beijing) AWS Region Code : cn-north-1
  • China (Ningxia) AWS Region Code : cn-northwest-1
  • Canada (Central) AWS Region Code : ca-central-1
  • Middle East (Bahrain) AWS Region Code : me-south-1
  • Africa (Cape Town) AWS Region Code : af-south-1
  • US ISO East AWS Region Code : us-iso-east-1
  • US ISOB East (Ohio) AWS Region Code : us-isob-east-1
  • US ISO West AWS Region Code : us-iso-west-1
No Description Provided.
Access Key IDNo Description Provided.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Secret Access KeyNo Description Provided.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Credentials FilePath to a file containing AWS access key and secret key in properties file format.

This property requires exactly one file to be provided..
AWS Credentials Provider ServiceController Service API:
AWSCredentialsProviderService
Implementation: AWSCredentialsProviderControllerService		The Controller Service that is used to obtain aws credentials provider
Communications Timeout30 secsNo Description Provided.
VersionThe Version of the Object to download
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SSL Context ServiceController Service API:
SSLContextService
Implementations: StandardSSLContextService
StandardRestrictedSSLContextService		Specifies an optional SSL Context Service that, if provided, will be used to create connections
Endpoint Override URLEndpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints.
Supports Expression Language: true (will be evaluated using variable registry only)
Signer OverrideDefault Signature
  • Default Signature
  • Signature v4
  • Signature v2
The AWS libraries use the default signer but this property allows you to specify a custom signer to support older S3-compatible services.
Encryption ServiceController Service API:
AmazonS3EncryptionService
Implementation: StandardS3EncryptionService		Specifies the Encryption Service Controller used to configure requests. PutS3Object: For backward compatibility, this value is ignored when 'Server Side Encryption' is set. FetchS3Object: Only needs to be configured in case of Server-side Customer Key, Client-side KMS and Client-side Customer Key encryptions.
Proxy Configuration ServiceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService		Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN
Proxy HostProxy host name or IP
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy Host PortProxy host port
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy UsernameProxy username
Supports Expression Language: true (undefined scope)
Proxy PasswordProxy password
Sensitive Property: true
Supports Expression Language: true (undefined scope)
Requester Paysfalse
  • True Indicates that the requester consents to pay any charges associated with retrieving objects from the S3 bucket.
  • False Does not consent to pay requester charges for retrieving objects from the S3 bucket.
If true, indicates that the requester consents to pay any charges associated with retrieving objects from the S3 bucket. This sets the 'x-amz-request-payer' header to 'requester'.
Range StartThe byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Range LengthThe number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

NameDescription
successFlowFiles are routed to success relationship
failureFlowFiles are routed to failure relationship

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
s3.bucketThe name of the S3 bucket
pathThe path of the file
absolute.pathThe path of the file
filenameThe name of the file
hash.valueThe MD5 sum of the file
hash.algorithmMD5
mime.typeIf S3 provides the content type/MIME type, this attribute will hold that file
s3.etagThe ETag that can be used to see if the file has changed
s3.exceptionThe class name of the exception thrown during processor execution
s3.additionalDetailsThe S3 supplied detail from the failed operation
s3.statusCodeThe HTTP error code (if available) from the failed operation
s3.errorCodeThe S3 moniker of the failed operation
s3.errorMessageThe S3 exception message from the failed operation
s3.expirationTimeIf the file has an expiration date, this attribute will be set, containing the milliseconds since epoch in UTC time
s3.expirationTimeRuleIdThe ID of the rule that dictates this object's expiration time
s3.sseAlgorithmThe server side encryption algorithm of the object
s3.versionThe version of the S3 object
s3.encryptionStrategyThe name of the encryption strategy that was used to store the S3 object (if it is encrypted)

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

See Also:

PutS3Object, DeleteS3Object, ListS3