Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.
google cloud, google, storage, gcs, fetch
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
| Name | Default Value | Allowable Values | Description |
|---|---|---|---|
| Bucket | ${gcs.bucket} | Bucket of the object. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | |
| Name | ${filename} | Name of the object. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | |
| Project ID | Google Cloud Project ID Supports Expression Language: true (will be evaluated using variable registry only) | ||
| GCP Credentials Provider Service | Controller Service API: GCPCredentialsService Implementation: GCPCredentialsControllerService | The Controller Service used to obtain Google Cloud Platform credentials. | |
| Number of retries | 6 | How many retry attempts should be made before routing to the failure relationship. | |
| Proxy host | IP or hostname of the proxy to be used.
You might need to set the following properties in bootstrap for https proxy usage:
-Djdk.http.auth.tunneling.disabledSchemes=
-Djdk.http.auth.proxying.disabledSchemes= Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Proxy port | Proxy port number Supports Expression Language: true (will be evaluated using variable registry only) | ||
| HTTP Proxy Username | HTTP Proxy Username Supports Expression Language: true (will be evaluated using variable registry only) | ||
| HTTP Proxy Password | HTTP Proxy Password Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Proxy Configuration Service | Controller Service API: ProxyConfigurationService Implementation: StandardProxyConfigurationService | Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN | |
| Object Generation | The generation of the Object to download. If not set, the latest generation will be downloaded. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | ||
| Server Side Encryption Key | An AES256 Key (encoded in base64) which the object has been encrypted in. Sensitive Property: true Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | ||
| Range Start | The byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) | ||
| Range Length | The number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object. Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Name | Description |
|---|---|
| success | FlowFiles are routed to this relationship after a successful Google Cloud Storage operation. |
| failure | FlowFiles are routed to this relationship if the Google Cloud Storage operation fails. |
| Name | Description |
|---|---|
| filename | The name of the file, parsed if possible from the Content-Disposition response header |
| gcs.bucket | Bucket of the object. |
| gcs.key | Name of the object. |
| gcs.size | Size of the object. |
| gcs.cache.control | Data cache control of the object. |
| gcs.component.count | The number of components which make up the object. |
| gcs.content.disposition | The data content disposition of the object. |
| gcs.content.encoding | The content encoding of the object. |
| gcs.content.language | The content language of the object. |
| mime.type | The MIME/Content-Type of the object |
| gcs.crc32c | The CRC32C checksum of object's data, encoded in base64 in big-endian order. |
| gcs.create.time | The creation time of the object (milliseconds) |
| gcs.update.time | The last modification time of the object (milliseconds) |
| gcs.encryption.algorithm | The algorithm used to encrypt the object. |
| gcs.encryption.sha256 | The SHA256 hash of the key used to encrypt the object |
| gcs.etag | The HTTP 1.1 Entity tag for the object. |
| gcs.generated.id | The service-generated for the object |
| gcs.generation | The data generation of the object. |
| gcs.md5 | The MD5 hash of the object's data encoded in base64. |
| gcs.media.link | The media download link to the object. |
| gcs.metageneration | The metageneration of the object. |
| gcs.owner | The owner (uploader) of the object. |
| gcs.owner.type | The ACL entity type of the uploader of the object. |
| gcs.uri | The URI of the object as a string. |