## KnoxToken Configuration ### Introduction --- The Knox Token Service enables the ability for clients to acquire the same JWT token that is used for KnoxSSO with WebSSO flows for UIs to be used for accessing REST APIs. By acquiring the token and setting it as a Bearer token on a request, a client is able to access REST APIs that are protected with the JWTProvider federation provider. This section describes the overall setup requirements and options for KnoxToken service. ### KnoxToken service The Knox Token Service configuration can be configured in any descriptor/topology, tailored to issue tokens to authenticated users, and constrain the usage of the tokens in a number of ways. "services": [ { "name": "KNOXTOKEN", "params": { "knox.token.ttl": "36000000", "knox.token.audiences": "tokenbased", "knox.token.target.url": "https://localhost:8443/gateway/tokenbased", "knox.token.exp.server-managed": "false", "knox.token.renewer.whitelist": "admin", "knox.token.exp.renew-interval": "86400000", "knox.token.exp.max-lifetime": "604800000" } } ] #### KnoxToken Configuration Parameters Parameter | Description | Default | -------------------------------- |------------ |----------- | knox.token.ttl | This indicates the lifespan (milliseconds) of the token. Once it expires a new token must be acquired from KnoxToken service. The 36000000 in the topology above gives you 10 hrs. | 30000 (30 seconds) | knox.token.audiences | This is a comma-separated list of audiences to add to the JWT token. This is used to ensure that a token received by a participating application knows that the token was intended for use with that application. It is optional. In the event that an endpoint has expected audiences and they are not present the token must be rejected. In the event where the token has audiences and the endpoint has none expected then the token is accepted.| empty | knox.token.target.url | This is an optional configuration parameter to indicate the intended endpoint for which the token may be used. The KnoxShell token credential collector can pull this URL from a knoxtokencache file to be used in scripts. This eliminates the need to prompt for or hardcode endpoints in your scripts. | n/a | knox.token.exp.server-managed | This is an optional configuration parameter to enable/disable server-managed token state, to support the associated token renewal and revocation APIs. | false | knox.token.renewer.whitelist | This is an optional configuration parameter to authorize the comma-separated list of users to invoke the associated token renewal and revocation APIs. | | knox.token.exp.renew-interval | This is an optional configuration parameter to specify the amount of time (milliseconds) to be added to a token's TTL when a renewal request is approved. | 86400000 (24 hours) | knox.token.exp.max-lifetime | This is an optional configuration parameter to specify the maximum allowed lifetime (milliseconds) of a token, after which renewal will not be permitted. | 604800000 (7 days) | Note that server-managed token state can be configured for all KnoxToken service deployments in gateway-site (see [gateway.knox.token.exp.server-managed](#Gateway+Server+Configuration)). If it is configured at the gateway level, then the associated service parameter, if configured, will override the gateway configuration. Adding the KnoxToken configuration shown above to a topology that is protected with the ShrioProvider is a very simple and effective way to expose an endpoint from which a Knox token can be requested. Once it is acquired it may be used to access resources at intended endpoints until it expires. The following curl command can be used to acquire a token from the Knox Token service as configured in the sandbox topology: curl -ivku guest:guest-password https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token Resulting in a JSON response that contains the token, the expiration and the optional target endpoint: `{"access_token":"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNDg5OTQyMTg4fQ.bcqSK7zMnABEM_HVsm3oWNDrQ_ei7PcMI4AtZEERY9LaPo9dzugOg3PA5JH2BRF-lXM3tuEYuZPaZVf8PenzjtBbuQsCg9VVImuu2r1YNVJlcTQ7OV-eW50L6OTI0uZfyrFwX6C7jVhf7d7YR1NNxs4eVbXpS1TZ5fDIRSfU3MU","target_url":"https://localhost:8443/gateway/tokenbased","token_type":"Bearer ","expires_in":1489942188233}` The following curl example shows how to add a bearer token to an Authorization header: curl -ivk -H "Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNDg5OTQyMTg4fQ.bcqSK7zMnABEM_HVsm3oWNDrQ_ei7PcMI4AtZEERY9LaPo9dzugOg3PA5JH2BRF-lXM3tuEYuZPaZVf8PenzjtBbuQsCg9VVImuu2r1YNVJlcTQ7OV-eW50L6OTI0uZfyrFwX6C7jVhf7d7YR1NNxs4eVbXpS1TZ5fDIRSfU3MU" https://localhost:8443/gateway/tokenbased/webhdfs/v1/tmp?op=LISTSTATUS #### KnoxToken Renewal and Revocation The KnoxToken service supports the renewal and explicit revocation of tokens it has issued. Support for both requires server-managed token state to be enabled with at least one renewer white-listed. ##### Renewal curl -ivku admin:admin-password -X POST -d $TOKEN 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token/renew' The JSON responses include a flag indicating success or failure. A successful result includes the updated expiration time. { "renewed": "true", "expires": "1584278311658" } Error results include a message describing the reason for failure. Invalid token { "renewed": "false", "error": "Unknown token: 9caf743e-1e0d-4708-a9ac-a684a576067c" } Unauthorized caller { "renewed": "false", "error": "Caller (guest) not authorized to renew tokens." } ##### Revocation curl -ivku admin:admin-password -X POST -d $TOKEN 'https://localhost:8443/gateway/sandbox/knoxtoken/api/v1/token/revoke' The JSON responses include a flag indicating success or failure. { "revoked": "true" } Error results include a message describing the reason for the failure. Invalid token { "revoked": "false", "error": "Unknown token: 9caf743e-1e0d-4708-a9ac-a684a576067c" } Unauthorized caller { "revoked": "false", "error": "Caller (guest) not authorized to revoke tokens." } See documentation in Client Details for KnoxShell init, list and destroy for commands that leverage this token service for CLI sessions. #### Token Generation/Management UIs ##### Overview In Apache Knox v1.6.0 the team added two new UIs that are directly accessible from the Knox Home page: * Token Generation * Token Management By default, the `homepage` topology comes with the `KNOXTOKEN` service enabled with the following attributes: * token TTL is set to 120 days * token service is enabled (default to keystore-based token state service) * the admin user is allowed to renew/revoke tokens In this topology, homepage, two new applications were added in order to display the above-listed UIs: * `tokengen`: this is an old-style JSP UI, with a relatively simple JS code included. The source is located in the [gateway-applications](https://github.com/apache/knox/tree/v1.6.0/gateway-applications/src/main/resources/applications/tokengen) Maven sub-module. * `token-management`: this is an Angular UI. The source is located in its own [knox-token-management-ui](https://github.com/apache/knox/tree/v1.6.0/knox-token-management-ui) Maven sub-module. On the Knox Home page, you will see a new town in the General Proxy Information table like this: ![](knoxtokenmanagement_homepage.png) However, the _Integration Token_ links are disabled by default, because token integration requires a gateway-level alias - called `knox.token.hash.key` - being created and without that alias, it [does not make sense to show those links](https://github.com/apache/knox/pull/512). ##### Creating the token hash key As explained, if you would like to use Knox's token generation features, you will have to create a gateway-level alias with a 256, 384, or 512-bit length JWK. You can do it in - at least - two different ways: 1. You generate your own MAC (using [this online tool](https://8gwifi.org/jwkfunctions.jsp) for instance) and save it as an alias using Knox CLI. 2. You do it running the following Knox CLI command: `generate-jwk --saveAlias knox.token.hash.key` The second option involves a newly created Knox CLI command called `generate-jwk`: ##### Token state service implementations There was an important step the Knox team made to provide more flexibility for our end-users: there are some internal service implementations in Knox that were hard-coded in the Java source code. One of those services is the `Token State` service implementation which you can change in gateway-site.xml going forward by setting the `gateway.service.tokenstate.impl` property to any of: 1. `org.apache.knox.gateway.services.token.impl.DefaultTokenStateService` - keeps all token information in memory, therefore all of this information is lost when Knox is shut down 2. `org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService` - token information is stored in the gateway credential store. This is a durable option, but not suitable for HA deployments 3. `org.apache.knox.gateway.services.token.impl.JournalBasedTokenStateService` - token information is stored in plain files within `$KNOX_DATA_DIR/security/token-state` folder. This option also provides a durable persistence layer for tokens and it might be good for HA scenarios too (in case of KNOX_DATA_DIR is on a shared drive), but the token data is written out in plain text (i.e. not encrypted) so it's less secure. 4. `org.apache.knox.gateway.services.token.impl.ZookeeperTokenStateService` - this is an extension of the keystore-based approach. In this case, token information is stored in Zookeeper using Knox aliases. The token's alias name equals to its generated token ID. 5. `org.apache.knox.gateway.services.token.impl.JDBCTokenStateService` - stores token information in relational databases. It's not only durable, but it's perfectly fine with HA deployments. Currently, PostgreSQL and MySQL databases are supported. By default, the `AliasBasedTokenStateService` implementation is used. ##### Configuring the JDBC token state service If you want to use the newly implemented database token management, you’ve to set `gateway.service.tokenstate.impl` in _gateway-site.xml_ to `org.apache.knox.gateway.services.token.impl.JDBCTokenStateService`. Now, that you have configured your token state backend, you need to configure a valid database in _gateway-site.xml_. There are two ways to do that: 1. You either declare database connection properties one-by-one: `gateway.database.type` - should be set to `postgresql` or `mysql` `gateway.database.host` - the host where your DB server is running `gateway.database.port` - the port that your DB server is listening on `gateway.database.name` - the name of the database you are connecting to 2. Or you declare an all-in-one JDBC connection string called `gateway.database.connection.url`. The following value will show you how to connect to an SSL enabled PostgreSQL server: jdbc:postgresql://$myPostgresServerHost:5432/postgres?user=postgres&ssl=true&sslmode=verify-full&sslrootcert=/usr/local/var/postgresql@10/data/root.crt If your database requires user/password authentication, the following aliases must be saved into the Knox Gateway’s credential store (__gateway-credentials.jceks): * `gateway_database_user` - the username * `gateway_database_password` - the password ###### Database design ![](JDBC_TSS_DB_Design.png) As you can see, there are only 2 tables: * `KNOXTOKENS` contains basic information about the generated token * `KNOX_TOKEN_METADATA` contains an arbitrary number of metadata information for the generated token. At the time of this document being written the following metadata exist: * `passcode` - this is the BASE-64 encoded value of the generated passcode token MAC. That is, the BASE-64 decoded value is a generated MAC. * `userName` - the logged-in user who generated the token * `enabled` - this is a boolean flag indicating that the given token is enabled or not (a _disabled_ token cannot be used for authentication purposes) * `comment` - this is optional metadata, saved only if the user enters something in the _Comment_ input field on the _Token Generation_ page (see below) ##### Generating a token Once you configured the `knox.token.hash.key` alias and optionally customized your token state service, you are all set to generate Knox tokens using the new Token Generation UI: ![](knoxtokenmanagement_token_generation_ui-1.png) The following sections are displayed on the page: * status bar: here you can see an informative message on the configured Token State backend. There are 3 different statuses: * ERROR: shown in red. This indicates a problem with the service backend which makes the feature not work. Usually, this is visible when end-users configure JDBC token state service, but they make a mistake in their DB settings * WARN: displayed in yellow (see above picture). This indicates that the feature is enabled and working, but there are some limitations * INFO: displayed in green. This indicates when the token management backend is properly configured for HA and production deployments * there is an information label explaining the purpose of the token generation page * comment: this is an _optional_ input field that allows end-users to add meaningful comments (mnemonics) to their generated tokens. The maximum length is 255 characters. * the `Configured maximum lifetime` informs the clients about the `knox.token.ttl` property set in the `homepage` topology (defaults to 120 days). If that property is not set (e.g. someone removes it from he homepage topology), Knox uses a hard-coded value of 30 seconds (aka. default Knox token TTL) * Custom token lifetime can be set by adjusting the days/hours/minutes spinners. The default configuration will yield one hour. * Clicking the Generate Token button will try to create a token for you. ##### About the generated token TTL Out of the box, Knox will display the custom lifetime spinners on the Token Generation page. However, they can be hidden by setting the `knox.token.lifespan.input.enabled` property to `false` in the `homepage` topology. Given that possibility and the configured maximum lifetime the generated token can have the following TTL value: * there is no configured token TTL and lifespan inputs are disabled -> the default TTL is used (30 seconds) * there is configured TTL and lifespan inputs are disabled -> the configured TTL is used * there is configured TTL and lifespan inputs are enabled and lifespan inputs result in a value that is less than or equal to the configured TTL -> the lifespan query param is used * there is configured TTL and lifespan inputs are enabled and lifespan inputs result in a value that is greater than the configured TTL -> the configured TTL is used ##### Successful token generation ![](knoxtokenmanagement_token_generation_ui-successful.png) On the resulting page there is two sensitive information that you can use in Knox to authenticate your request: 1. **JWT token** - this is the serialized JWT and is fully compatible with the old-style Bearer authorization method. Clicking the `JWT Token` label on the page will copy the value into the clipboard. You might want to use it as the ‘Token’ user: `$ curl -ku Token:eyJqa3UiOiJodHRwczpcL1wvbG9jYWxob3N0Ojg0NDNcL2dhdGV3YXlcL2hvbWVwYWdlXC9rbm94dG9rZW5cL2FwaVwvdjFcL2p3a3MuanNvbiIsImtpZCI6IkdsOTZfYTM2MTJCZWFsS2tURFRaOTZfVkVsLVhNRVRFRmZuNTRMQ1A2UDQiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImprdSI6Imh0dHBzOlwvXC9sb2NhbGhvc3Q6ODQ0M1wvZ2F0ZXdheVwvaG9tZXBhZ2VcL2tub3h0b2tlblwvYXBpXC92MVwvandrcy5qc29uIiwia2lkIjoiR2w5Nl9hMzYxMkJlYWxLa1REVFo5Nl9WRWwtWE1FVEVGZm41NExDUDZQNCIsImlzcyI6IktOT1hTU08iLCJleHAiOjE2MzY2MjU3MTAsIm1hbmFnZWQudG9rZW4iOiJ0cnVlIiwia25veC5pZCI6ImQxNjFjYWMxLWY5M2UtNDIyOS1hMGRkLTNhNzdhYjkxNDg3MSJ9.e_BNPf_G1iBrU0m3hul5VmmSbpw0w1pUAXl3czOcuxFOQ0Tki-Gq76fCBFUNdKt4QwLpNXxM321cH1TeMG4IhL-92QORSIZgRxY4OUtUgERzcU7-27VNYOzJbaRCjrx-Vb4bSriRJJDwbbXyAoEw_bjiP8EzFFJTPmGcctEzrOLWFk57cLO-2QLd2nbrNd4qmrRR6sEfP81Jg8UL-Ptp66vH_xalJJWuoyoNgGRmH8IMdLVwBgeLeVHiI7NmokuhO-vbctoEwV3Rt4pMpA0VSWGFN0MI4WtU0crjXXHg8U9xSZyOeyT3fMZBXctvBomhGlWaAvuT5AxQGyMMP3VLGw https:/localhost:8443/gateway/sandbox/webhdfs/v1?op=LISTSTATUS` `{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16386,"group":"supergroup","length":0,"modificationTime":1621238405734,"owner":"hdfs","pathSuffix":"tmp","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1621238326078,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}` 2. **Passcode token** - this is the serialized passcode token, which you can use as the ‘Passcode’ user (Clicking the `Passcode Token` label on the page will copy the value into the clipboard): `$ curl -ku Passcode:WkRFMk1XTmhZekV0WmprelpTMDBNakk1TFdFd1pHUXRNMkUzTjJGaU9URTBPRGN4OjpPVEV5Tm1KbFltUXROVEUyWkMwME9HSTBMVGd4TTJZdE1HRmxaalJrWlRVNFpXRTA= https://localhost:8443/gateway/sandbox/webhdfs/v1?op=LISTSTATUS` `{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16386,"group":"supergroup","length":0,"modificationTime":1621238405734,"owner":"hdfs","pathSuffix":"tmp","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1621238326078,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}` The reason, we needed to support the shorter `Passcode token`, is that there are 3rd party tools where the long JWT exceeds input fields limitations so we need to address this issue with shorter token values. The rest of the fields are complementary information such as the expiration date/time of the generated token or the user who created it. ##### Token generation failed If there was an error during token generation, you will see a failure right under the input field boxes (above the Generate Token button): ![](knoxtokenmanagement_token_generation_ui-fail.png) The above error message indicates a failure that the admin user already generated more tokens than they are allowed to. This limitation is configurable in the `gateway-site.xml`: - `gateway.knox.token.limit.per.user` - indicates the maximum number of tokens a user can manage at the same time. `-1` means that users are allowed to create/manage as many tokens as they want. This configuration only applies when the server-managed token state is enabled either in `gateway-site` or at the `topology` level. Defaults to 10. ##### Token Management In addition to the token generation UI, Knox comes with a Token Management UI where logged-in users can see all the active tokens that they generated before. That is, if a token got expired and was removed from the underlying token store, it won't be displayed here. ![](knoxtokenmanagement_token_management_ui-1.png) On this page, you will see basic information about your generated token(s) and you can execute the following actions: 1. Enable/Disable - based on the current status, you can temporarily enable/disable a token. Please note that disabled tokens are not allowed to use for authentication purposes. 2. Revoke - you can remove the token from the persistent store. Please note this action cannot be undone, once you revoked a token Knox will delete it from the in-memory cache as well as the underlying persistent token storage In order to refresh the table, you can use the `Refresh icon` above the table (if you generated tokens on another tab for instance).