Fetches files from an SFTP Server and creates FlowFiles from them
sftp, get, retrieve, files, fetch, remote, ingest, source, input
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.
| Name | Default Value | Allowable Values | Description |
|---|---|---|---|
| Hostname | The fully qualified hostname or IP address of the remote system Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Port | 22 | The port that the remote system is listening on for file transfers Supports Expression Language: true (will be evaluated using variable registry only) | |
| Username | Username Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Password | Password for the user account Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Private Key Path | The fully qualified path to the Private Key file This property requires exactly one file to be provided.. Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Private Key Passphrase | Password for the private key Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Remote Path | The path on the remote system from which to pull or push files Supports Expression Language: true (will be evaluated using variable registry only) | ||
| File Filter Regex | Provides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched | ||
| Path Filter Regex | When Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned | ||
| Polling Interval | 60 sec | Determines how long to wait between fetching the listing for new files | |
| Search Recursively | false |
| If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories |
| Follow symlink | false |
| If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories |
| Ignore Dotted Files | true |
| If true, files whose names begin with a dot (".") will be ignored |
| Delete Original | true |
| Determines whether or not the file is deleted from the remote system after it has been successfully transferred |
| Connection Timeout | 30 sec | Amount of time to wait before timing out while creating a connection | |
| Data Timeout | 30 sec | When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems | |
| Host Key File | If supplied, the given file will be used as the Host Key; otherwise, if 'Strict Host Key Checking' property is applied (set to true) then uses the 'known_hosts' and 'known_hosts2' files from ~/.ssh directory else no host key file will be used This property requires exactly one file to be provided.. | ||
| Max Selects | 100 | The maximum number of files to pull in a single connection | |
| Remote Poll Batch Size | 5000 | The value specifies how many file paths to find in a given directory on the remote system when doing a file listing. This value in general should not need to be modified but when polling against a remote system with a tremendous number of files this value can be critical. Setting this value too high can result very poor performance and setting it too low can cause the flow to be slower than normal. | |
| Strict Host Key Checking | false |
| Indicates whether or not strict enforcement of hosts keys should be applied |
| Send Keep Alive On Timeout | true |
| Send a Keep Alive message every 5 seconds up to 5 times for an overall timeout of 25 seconds. |
| Use Compression | false |
| Indicates whether or not ZLIB compression should be used when transferring files |
| Use Natural Ordering | false |
| If true, will pull files in the order in which they are naturally listed; otherwise, the order in which the files will be pulled is not defined |
| Proxy Configuration Service | Controller Service API: ProxyConfigurationService Implementation: StandardProxyConfigurationService | Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS + AuthN | |
| Proxy Type | DIRECT |
| Proxy type used for file transfers |
| Proxy Host | The fully qualified hostname or IP address of the proxy server Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Proxy Port | The port of the proxy server Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Http Proxy Username | Http Proxy Username Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Http Proxy Password | Http Proxy Password Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Ciphers Allowed | A comma-separated list of Ciphers allowed for SFTP connections. Leave unset to allow all. Available options are: 3des-cbc, 3des-ctr, aes128-cbc, aes128-ctr, aes128-gcm@openssh.com, aes192-cbc, aes192-ctr, aes256-cbc, aes256-ctr, aes256-gcm@openssh.com, arcfour, arcfour128, arcfour256, blowfish-cbc, blowfish-ctr, cast128-cbc, cast128-ctr, chacha20-poly1305@openssh.com, idea-cbc, idea-ctr, serpent128-cbc, serpent128-ctr, serpent192-cbc, serpent192-ctr, serpent256-cbc, serpent256-ctr, twofish-cbc, twofish128-cbc, twofish128-ctr, twofish192-cbc, twofish192-ctr, twofish256-cbc, twofish256-ctr Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Key Algorithms Allowed | A comma-separated list of Key Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: ecdsa-sha2-nistp256, ecdsa-sha2-nistp256-cert-v01@openssh.com, ecdsa-sha2-nistp384, ecdsa-sha2-nistp384-cert-v01@openssh.com, ecdsa-sha2-nistp521, ecdsa-sha2-nistp521-cert-v01@openssh.com, rsa-sha2-256, rsa-sha2-512, ssh-dss, ssh-dss-cert-v01@openssh.com, ssh-ed25519, ssh-ed25519-cert-v01@openssh.com, ssh-rsa, ssh-rsa-cert-v01@openssh.com Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Key Exchange Algorithms Allowed | A comma-separated list of Key Exchange Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: curve25519-sha256, curve25519-sha256@libssh.org, diffie-hellman-group-exchange-sha1, diffie-hellman-group-exchange-sha256, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256, diffie-hellman-group14-sha256@ssh.com, diffie-hellman-group15-sha256, diffie-hellman-group15-sha256@ssh.com, diffie-hellman-group15-sha384@ssh.com, diffie-hellman-group15-sha512, diffie-hellman-group16-sha256, diffie-hellman-group16-sha384@ssh.com, diffie-hellman-group16-sha512, diffie-hellman-group16-sha512@ssh.com, diffie-hellman-group17-sha512, diffie-hellman-group18-sha512, diffie-hellman-group18-sha512@ssh.com, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, ext-info-c Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Message Authentication Codes Allowed | A comma-separated list of Message Authentication Codes allowed for SFTP connections. Leave unset to allow all. Available options are: hmac-md5, hmac-md5-96, hmac-md5-96-etm@openssh.com, hmac-md5-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-96, hmac-ripemd160-etm@openssh.com, hmac-ripemd160@openssh.com, hmac-sha1, hmac-sha1-96, hmac-sha1-96@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-etm@openssh.com Supports Expression Language: true (will be evaluated using variable registry only) |
| Name | Description |
|---|---|
| success | All FlowFiles that are received are routed to success |
| Name | Description |
|---|---|
| filename | The filename is set to the name of the file on the remote server |
| path | The path is set to the path of the file's directory on the remote server. For example, if the <Remote Path> property is set to /tmp, files picked up from /tmp will have the path attribute set to /tmp. If the <Search Recursively> property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to /tmp/abc/1/2/3 |
| file.lastModifiedTime | The date and time that the source file was last modified |
| file.owner | The numeric owner id of the source file |
| file.group | The numeric group id of the source file |
| file.permissions | The read/write/execute permissions of the source file |
| absolute.path | The full/absolute path from where a file was picked up. The current 'path' attribute is still populated, but may be a relative path |