Delegations for content trust

Estimated reading time: 14 minutes

Delegations in Docker Content Trust (DCT) allow you to control who can and cannot sign an image tag. A delegation will have a pair of private and public delegation keys. A delegation could contain multiple pairs of keys and contributors in order to a) allow multiple users to be part of a delegation, and b) to support key rotation.

The most important delegation within Docker Content Trust is targets/releases. This is seen as the canonical source of a trusted image tag, and without a contributor’s key being under this delegation, they will be unable to sign a tag.

Fortunately when using the $ docker trust commands, we will automatically initialize a repository, manage the repository keys, and add a collaborator’s key to the targets/releases delegation via docker trust signer add.

Configuring the Docker Client

By default, the $ docker trust commands expect the notary server URL to be the same as the registry URL specified in the image tag (following a similar logic to $ docker push). When using Docker Hub or DTR, the notary server URL is the same as the registry URL. However, for self-hosted environments or 3rd party registries, you will need to specify an alternative URL for the notary server. This is done with:

export DOCKER_CONTENT_TRUST_SERVER=https://<URL>:<PORT>

If you do not export this variable in self-hosted environments, you may see errors such as:

$ docker trust signer add --key cert.pem jeff dtr.example.com/admin/demo
Adding signer "jeff" to dtr.example.com/admin/demo...
[...]
Error: trust data missing for remote repository dtr.example.com/admin/demo or remote repository not found: timestamp key trust data unavailable.  Has a notary repository been initialized?

$ docker trust inspect dtr.example.com/admin/demo --pretty
WARN[0000] Error while downloading remote metadata, using cached timestamp - this might not be the latest version available remotely
[...]

If you have enabled authentication for your notary server, or are using DTR, you will need to log in before you can push data to the notary server.

$ docker login dtr.example.com/user/repo
Username: admin
Password:

Login Succeeded

$ docker trust signer add --key cert.pem jeff dtr.example.com/user/repo
Adding signer "jeff" to dtr.example.com/user/repo...
Initializing signed repository for dtr.example.com/user/repo...
Successfully initialized "dtr.example.com/user/repo"
Successfully added signer: jeff to dtr.example.com/user/repo

If you do not log in, you will see:

$ docker trust signer add --key cert.pem jeff dtr.example.com/user/repo
Adding signer "jeff" to dtr.example.com/user/repo...
Initializing signed repository for dtr.example.com/user/repo...
you are not authorized to perform this operation: server returned 401.

Failed to add signer to: dtr.example.com/user/repo

Configuring the Notary Client

Some of the more advanced features of DCT require the Notary CLI. To install and configure the Notary CLI:

1) Download the client and ensure that it is available on your path.

2) Create a configuration file at ~/.notary/config.json with the following content:

{
  "trust_dir" : "~/.docker/trust",
  "remote_server": {
    "url": "https://dtr.example.com",
    "root_ca": "../.docker/ca.pem"
  }
}

The newly created configuration file contains information about the location of your local Docker trust data and the notary server URL.

For more detailed information about how to use notary outside of the Docker Content Trust use cases, refer to the Notary CLI documentation here

Creating Delegation Keys

A prerequisite to adding your first contributor is a pair of delegation keys. These keys can either be generated locally using $ docker trust, generated by a certificate authority.

Using Docker Trust to Generate Keys

Docker trust has a built-in generator for a delegation key pair, $ docker trust generate <name>. Running this command will automatically load the delegation private key in to the local Docker trust store.

$ docker trust key generate jeff

Generating key for jeff...
Enter passphrase for new jeff key with ID 9deed25: 
Repeat passphrase for new jeff key with ID 9deed25: 
Successfully generated and loaded private key. Corresponding public key available: /home/ubuntu/Documents/mytrustdir/jeff.pub

Manually Generating Keys

If you need to manually generate a private key (either RSA or ECDSA) and a x509 certificate containing the public key, you can use local tools like openssl or cfssl along with a local or company-wide Certificate Authority.

Here is an example of how to generate a 2048-bit RSA portion key (all RSA keys must be at least 2048 bits):

$ openssl genrsa -out delegation.key 2048

Generating RSA private key, 2048 bit long modulus
....................................................+++
............+++
e is 65537 (0x10001)

They should keep delegation.key private because it is used to sign tags.

Then they need to generate an x509 certificate containing the public key, which is what you need from them. Here is the command to generate a CSR (certificate signing request):

$ openssl req -new -sha256 -key delegation.key -out delegation.csr

Then they can send it to whichever CA you trust to sign certificates, or they can self-sign the certificate (in this example, creating a certificate that is valid for 1 year):

$ openssl x509 -req -sha256 -days 365 -in delegation.csr -signkey delegation.key -out delegation.crt

Then they need to give you delegation.crt, whether it is self-signed or signed by a CA.

Finally you will need to add the private key into your local Docker trust store.

$ docker trust key load delegation.key --name jeff

Loading key from "delegation.key"...
Enter passphrase for new jeff key with ID 8ae710e: 
Repeat passphrase for new jeff key with ID 8ae710e: 
Successfully imported key from delegation.key

Viewing local Delegation keys

To list the keys that have been imported in to the local Docker trust store we can use the Notary CLI.

$ notary key list

ROLE       GUN                          KEY ID                                                              LOCATION
----       ---                          ------                                                              --------
root                                    f6c6a4b00fefd8751f86194c7d87a3bede444540eb3378c4a11ce10852ab1f96    /home/ubuntu/.docker/trust/private
jeff                                    9deed251daa1aa6f9d5f9b752847647cf8d705da0763aa5467650d0987ed5306    /home/ubuntu/.docker/trust/private

Managing Delegations in a Notary Server

When the first Delegation is added to the Notary Server using $ docker trust, we automatically initiate trust data for the repository. This includes creating the notary target and snapshots keys, and rotating the snapshot key to be managed by the notary server. More information on these keys can be found here

When initiating a repository, you will need the key and the passphrase of a local Notary Canonical Root Key. If you have not initiated a repository before, and therefore don’t have a Notary root key, $ docker trust will create one for you.

Be sure to protect and back up your Notary Canonical Root Key

Initiating the Repository

To upload the first key to a delegation, at the same time initiating a repository, you can use the $ docker trust signer add command. This will add the contributor’s public key to the targets/releases delegation, and create a second targets/<name> delegation.

For DCT the name of the second delegation, in the below example jeff, is there to help you keep track of the owner of the keys. In more advanced use cases of Notary additional delegations are used for hierarchy.

$ docker trust signer add --key cert.pem jeff dtr.example.com/admin/demo

Adding signer "jeff" to dtr.example.com/admin/demo...
Initializing signed repository for dtr.example.com/admin/demo...
Enter passphrase for root key with ID f6c6a4b: 
Enter passphrase for new repository key with ID b0014f8: 
Repeat passphrase for new repository key with ID b0014f8: 
Successfully initialized "dtr.example.com/admin/demo"
Successfully added signer: jeff to dtr.example.com/admin/demo

You can see which keys have been pushed to the Notary server for each repository with the $ docker trust inspect command.

$ docker trust inspect --pretty dtr.example.com/admin/demo

No signatures for dtr.example.com/admin/demo


List of signers and their keys for dtr.example.com/admin/demo

SIGNER              KEYS
jeff                1091060d7bfd

Administrative keys for dtr.example.com/admin/demo

  Repository Key:	b0014f8e4863df2d028095b74efcb05d872c3591de0af06652944e310d96598d
  Root Key:	64d147e59e44870311dd2d80b9f7840039115ef3dfa5008127d769a5f657a5d7

You could also use the Notary CLI to list delegations and keys. Here you can clearly see the keys were attached to targets/releases and targets/jeff.

$ notary delegation list dtr.example.com/admin/demo

ROLE                PATHS             KEY IDS                                                             THRESHOLD
----                -----             -------                                                             ---------
targets/jeff        "" <all paths>    1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1    1
                                          
targets/releases    "" <all paths>    1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1    1

Adding Additional Signers

Docker Trust allows you to configure multiple delegations per repository, allowing you to manage the lifecycle of delegations. When adding additional delegations with $ docker trust the collaborators key is once again added to the targets/release role.

Note you will need the passphrase for the repository key; this would have been configured when you first initiated the repository.

$ docker trust signer add --key ben.pub ben dtr.example.com/admin/demo

Adding signer "ben" to dtr.example.com/admin/demo...
Enter passphrase for repository key with ID b0014f8: 
Successfully added signer: ben to dtr.example.com/admin/demo

Check to prove that there are now 2 delegations (Signer).

$ docker trust inspect --pretty dtr.example.com/admin/demo

No signatures for dtr.example.com/admin/demo

List of signers and their keys for dtr.example.com/admin/demo

SIGNER              KEYS
ben                 afa404703b25
jeff                1091060d7bfd

Administrative keys for dtr.example.com/admin/demo

  Repository Key:	b0014f8e4863df2d028095b74efcb05d872c3591de0af06652944e310d96598d
  Root Key:	64d147e59e44870311dd2d80b9f7840039115ef3dfa5008127d769a5f657a5d7

Adding Keys to an Existing Delegation

To support things like key rotation and expiring / retiring keys you can publish multiple contributor keys per delegation. The only prerequisite here is to make sure you use the same the delegation name, in this case jeff. Docker trust will automatically handle adding this new key to targets/releases.

Note you will need the passphrase for the repository key; this would have been configured when you first initiated the repository.

$ docker trust signer add --key cert2.pem jeff dtr.example.com/admin/demo

Adding signer "jeff" to dtr.example.com/admin/demo...
Enter passphrase for repository key with ID b0014f8: 
Successfully added signer: jeff to dtr.example.com/admin/demo

Check to prove that the delegation (Signer) now contains multiple Key IDs.

$ docker trust inspect --pretty dtr.example.com/admin/demo

No signatures for dtr.example.com/admin/demo


List of signers and their keys for dtr.example.com/admin/demo

SIGNER              KEYS
jeff                1091060d7bfd, 5570b88df073

Administrative keys for dtr.example.com/admin/demo

  Repository Key:	b0014f8e4863df2d028095b74efcb05d872c3591de0af06652944e310d96598d
  Root Key:	64d147e59e44870311dd2d80b9f7840039115ef3dfa5008127d769a5f657a5d7

Removing a Delegation

If you need to remove a delegation, including the contributor keys that are attached to the targets/releases role, you can use the $ docker trust signer remove command.

Note tags that were signed by the removed delegation will need to be resigned by an active delegation

$ docker trust signer remove dtr.example.com/admin/demo
Removing signer "ben" from dtr.example.com/admin/demo...
Enter passphrase for repository key with ID b0014f8: 
Successfully removed ben from dtr.example.com/admin/demo

Troubleshooting

1) If you see an error that there are no usable keys in targets/releases, you will need to add additional delegations using docker trust signer add before resigning images.

WARN[0000] role targets/releases has fewer keys than its threshold of 1; it will not be usable until keys are added to it

2) If you have added additional delegations already and are seeing an error message that there are no valid signatures in targest/releases, you will need to resign the targets/releases delegation file with the Notary CLI.

WARN[0000] Error getting targets/releases: valid signatures did not meet threshold for targets/releases

Resigning the delegation file is done with the $ notary witness command

$ notary witness dtr.example.com/admin/demo targets/releases --publish

More information on the $ notary witness command can be found here

Removing a Contributor’s Key from a Delegation

As part of rotating keys for a delegation, you may want to remove an individual key but retain the delegation. This can be done with the Notary CLI.

Remember you will have to remove the key from both the targets/releases role and the role specific to that signer targets/<name>.

1) We will need to grab the Key ID from the Notary Server

$ notary delegation list dtr.example.com/admin/demo

ROLE                PATHS             KEY IDS                                                             THRESHOLD
----                -----             -------                                                             ---------
targets/jeff        "" <all paths>    8fb597cbaf196f0781628b2f52bff6b3912e4e8075720378fda60d17232bbcf9    1
                                      1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1    
targets/releases    "" <all paths>    8fb597cbaf196f0781628b2f52bff6b3912e4e8075720378fda60d17232bbcf9    1
                                      1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1

2) Remove from the targets/releases delegation

$ notary delegation remove dtr.example.com/admin/demo targets/targets 1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1 --publish

Auto-publishing changes to dtr.example.com/admin/demo
Enter username: admin
Enter password: 
Enter passphrase for targets key with ID b0014f8: 
Successfully published changes for repository dtr.example.com/admin/demo

3) Remove from the targets/<name> delegation

$ notary delegation remove dtr.example.com/admin/demo targets/jeff 1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1 --publish

Removal of delegation role targets/jeff with keys [5570b88df0736c468493247a07e235e35cf3641270c944d0e9e8899922fc6f99], to repository "dtr.example.com/admin/demo" staged for next publish.

Auto-publishing changes to dtr.example.com/admin/demo
Enter username: admin    
Enter password: 
Enter passphrase for targets key with ID b0014f8: 
Successfully published changes for repository dtr.example.com/admin/demo

4) Check the remaining delegation list

$ notary delegation list dtr.example.com/admin/demo

ROLE                PATHS             KEY IDS                                                             THRESHOLD
----                -----             -------                                                             ---------
targets/jeff        "" <all paths>    8fb597cbaf196f0781628b2f52bff6b3912e4e8075720378fda60d17232bbcf9    1    
targets/releases    "" <all paths>    8fb597cbaf196f0781628b2f52bff6b3912e4e8075720378fda60d17232bbcf9    1

Removing a local Delegation Private Key

As part of rotating delegation keys, you may need to remove a local delegation key from the local Docker trust store. This is done with the Notary CLI, using the $ notary key remove command.

1) We will need to get the Key ID from the local Docker Trust store

$ notary key list

ROLE       GUN                          KEY ID                                                              LOCATION
----       ---                          ------                                                              --------
root                                    f6c6a4b00fefd8751f86194c7d87a3bede444540eb3378c4a11ce10852ab1f96    /home/ubuntu/.docker/trust/private
admin                                   8fb597cbaf196f0781628b2f52bff6b3912e4e8075720378fda60d17232bbcf9    /home/ubuntu/.docker/trust/private
jeff                                    1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1    /home/ubuntu/.docker/trust/private
targets    ...example.com/admin/demo    c819f2eda8fba2810ec6a7f95f051c90276c87fddfc3039058856fad061c009d    /home/ubuntu/.docker/trust/private

2) Remove the key from the local Docker Trust store

$ notary key remove 1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1

Are you sure you want to remove 1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1 (role jeff) from /home/ubuntu/.docker/trust/private?  (yes/no)  y

Deleted 1091060d7bfd938dfa5be703fa057974f9322a4faef6f580334f3d6df44c02d1 (role jeff) from /home/ubuntu/.docker/trust/private.

Removing all trust data from a Repository

You can remove all trust data from a repository, including repository, target, snapshot and all delegations keys using the Notary CLI.

This is often required by a container registry before a particular repository can be deleted.

$ notary delete dtr.example.com/admin/demo --remote

Deleting trust data for repository dtr.example.com/admin/demo
Enter username: admin
Enter password: 
Successfully deleted local and remote trust data for repository dtr.example.com/admin/demo

$ docker trust inspect --pretty dtr.example.com/admin/demo

No signatures or cannot access dtr.example.com/admin/demo
trust, security, delegations, keys, repository