This means we are not limited to pre-signing requests to GetCallerIdentity and can create requests to any action of the STS API. This is verified using the ServiceAccount GCP API which requires the iam.serviceAccounts.get permission in the project hosting the service account. Now that I can retrieve secrets from Vault, I can set my GIT_TOKEN and DATADOG_TOKEN.

This is a few years later, but this seems to work for me: bound_iam_principal_arn=arn:aws:iam::383372584635:role/auth-example-role20190626175122610800000005,arn:aws:iam::822220783772:role/AppRoleTestJLE. This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. However, this operational simplicity is only possible because of hidden complexity in the AWS iam auth method. The only requirement is that the attacker knows the name of an privileged AWS role in the target Vault server. You will need to use the Vault CLI.

How does the /v1/auth/aws/login API endpoint actually work and is there a way a unauthenticated attacker can impersonate a random AWS IAM role? At this point I was slowly getting frustrated and decided to take a look at Vaults response parsing code: is called on every response received from STS as long as the status code is 200. Note that all calls except the logins are done on the Vault server using a privileged token.

Enter "dev" as the policy name and paste the text from dev-policy.hcl into the policy field. for a simple proof-of-concept script that takes care of most of the details.

Finding a reflected parameter that is not constrained to alpha-numeric characters turns out to be tricky. With those tokens set, I can setup my .netrc file so that I can checkout my playbooks and roles from my private GitHub repositories and pass secrets to my Ansible. Third-party services cant easily verify pre-signed requests and AWS IAM doesnt offer any standard signing primitives that could be used to implement certificate based authentication or JWTs. token_meta_client_user_id AROAJTX274TFBJ6ZFA3YC, You can authenticate from a qa instance with: token_renewable true This part of my projected worked just fine, but it left me with another problem. At this point I was slowly getting frustrated and decided to take a look at Vaults response parsing code: func parseGetCallerIdentityResponse(response string) (GetCallerIdentityResponse, error) {, decoder := xml.NewDecoder(strings.NewReader(response)), XMLName xml.Name `xml:"GetCallerIdentityResponse"`, GetCallerIdentityResult []GetCallerIdentityResult `xml:"GetCallerIdentityResult"`, ResponseMetadata []ResponseMetadata `xml:"ResponseMetadata"`. Now simply use the token to sign-in to Vault: curl --request POST --data '{"role": "my-gce-role", "jwt" : "."}' http://vault:8200/v1/auth/gcp/login. As its name implies, GetCallerIdentity returns details about the IAM role or user whose credentials were used to call the API. token_meta_auth_type iam By enabling and configuring the aws auth method, you can create a mapping between certain IAM users or roles to Vault roles. We will illustrate both the ec2 and iam methods. The ec2 method uses the EC2 Metadata Service which enables each instance to fetch its unique AWS Instance Identity Document using its PKCS#7 signature. ranging from static credentials, LDAP or Radius, to full integration into third-party OpenID Connect (OIDC) providers or Cloud Identity Access Management (IAM) platforms. Serialize a request to it while including an Accept: application/json header and put an arbitrary GetCallerIdentityResponse XML blob into the reflected payload. "iam_http_request_method": "POST", "iam_request_body": "encoded-body", , "iam_request_headers" : "encoded-headers", "iam_request_url" : "encoded-url"}', {"request_id":"59b09a0b-f5d5-f4c4-8ed0-af86a2c1f5d4","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":["TTL, of \"768h\" exceeded the effective max_ttl of \"500h\"; TTL value is capped.

However, tricks like embedding a fake userinfo (https://sts.amazonaws.com/:foo@example.com/test) and similar ideas do not work against the robust Go URL parser. using a service account private key under their control or with the projects.serviceAccounts.signJwt IAM API method. If the attacker impersonated. . auth method for deployments on Google Cloud. This token can now be used to fetch the database secret from Vault.

{'iss': 'https://oidc-test-wrbvvljkzwtfpiikylvpckxgafdkxfba.s3.amazonaws.com/'. 1. 2. This means that calling parseGetCallerIdentityResponse with a (JSON encoded) server response such as {abc : xzy} will succeed and return an (empty) CallerIdentityResponse structure. In addition to the normal JWT claims (sub, aud, iat, exp), the tokens returned from the metadata server also contains a special compute_engine claim that lists details about the instance, which are processed as part of the auth process: "google":{"compute_engine":{"instance_creation_timestamp":1594641932,"instance_id":"671398237781058X, XXX","instance_name":"vault","project_id":"fwilhelm-testing-XXXX","project_number":950612XXXX,"zone":"europe-west3-c"}}. Even though Vault will always create a HTTPS request pointing at the hardcoded endpoint, the attacker has full control over the Host http header (request.Host = parsedUrl.Host). read access requires that somebody know that path of the secret they want to pull. For instance, add "AWS": "arn:aws:iam::362381645759:root" to the Trust Relationship of the VaultAccess role in the other account. out of the token header and tries to find a google-wide oAuth key with the same identifier.

auth is built on top of an AWS API method called.

What went wrong here?

For Vault, this turns into a security issue due to a somewhat surprising feature of the Go XML decoder: The decoder silently ignores non XML content before and after the expected XML root. Next, we need to create a policy to limit what the instance has access to. For gce, the client is expected to run on an authorized GCE VM. You could use point a Vault client against your server after setting VAULT_ADDR and VAULT_TOKEN or just run the following commands on your Vault server after SSHing to it.

If the AWS ARN/UserID in our fake GetCallerIdentityResponse has privileges on the Vault server we get a valid session token back, which we can use to interact with the Vault server to fetch some secrets. Connect to your instance with SSH. It fetches a signed token by sending a request to the, of the GCP metadata server. While the OIDC provider setup adds some complexity, we end up with a nice authentication bypass for arbitrary AWS enabled roles. Both iam and gce are built on top of JWT. JWT has a number of design choices that make it very prone to implementation errors (see this blog post by securitum for an overview about typical issues), so I decided to spend a day on reviewing Vaults token processing. Here are the commands you should run on the Vault server for this example: You can authenticate from a dev instance to Vault with:

Repeat the above steps for the path qa/secrets. Knowing the key id and subject of the token, Vault fetches the public key used for signing using the service account GCP API: "Unable to get Google-wide OAuth2 Key, trying service-account public key", "unable to get public key %q for JWT subject %q: {{err}}", If verification succeeds, Vault fills out the, struct that is later used to grant or deny access. For #2, what Jason said is true. This means that calling, with a (JSON encoded) server response such as, {abc : xzy}, This brings us really close to our goal of spoofing an arbitrary caller identity: We just need to find a STS action that reflects attacker controlled text as part of its API response. parseGetCallerIdentityResponse is called on every response received from STS as long as the status code is 200. A popular example use case is to give clients the ability to upload a file to S3 without giving them access to credentials with write permissions. STS supports 8 different actions, but none gives us the ability to completely control the response. While STS responses are XML encoded by default, it also supports JSON encoding for clients that send an Accept: application/json HTTP header. You signed in with another tab or window. The function uses the Golang standard XML library to decode an XML response into a GetCallerIdentityResponse structure and returns an error if decoding fails. Create a service account in a GCP project you control and generate a private key using gcloud: gcloud iam service-accounts keys create key.json --iam-account, sa-name@project-id.iam.gserviceaccount.com, Sign a JWT with a fake compute_engine claim describing an existing and privileged VM.

token_accessor 30b2fd84-73c1-6eef-8f96-bc99791cc2ae With vault configured to allow ec2 instances to authenticate and pull secrets, we now need to write a user_data script to take advantage of the access. the disallow_reauthentication is set to true so that the instance can only authenticate once. For valid tokens, this claim contains the email address of the signing service account. 'sub': 'arn:aws:iam::superprivileged-aws-accountXYZ'. When doing this, set the tag's key to "dev_role" or "qa_role" and set the tag's value to the value of tag_value that was returned when you created the dev_role or qa_role role tag in the previous section, depending on whether you you want the EC2 instance to have the dev or qa roles.. on an AWS account they own and sign arbitrary tokens with their own keys. This is verified using the GCE API and requires an attacker to impersonate an actively running VM. One way to achieve this is to manipulate the Vault server into sending a request to a host we control, bypassing the hardcoded endpoint host. i limit the default session time to 15 minutes (and that still may be longer than it needs to be) and usee the bound_* parameters to limit authentication to specific regions in specific acounts. Since we already have ansible roles for configuring both our common instance settings (Vault SSH, motd, etc) and DataDog, I decided to figure out how to run that Ansible via user_data. Just find the instance, select it, select the Tags tab in the bottom portion of the window, click the "Add/Edit Tags" button, click the "Create Tag" button, add the correct tag, and then click the Save button. This can be done with the Vault UI or CLI. Other than that, the roles just serve as tags on the instances so that dev and qa instances will end up with the right Vault roles and only be able to read dev and qa secrets respectively. enabling the backend is straightforward. token_accessor 6ad0daa4-e8f5-062d-ed93-3ff126f9290c Note that all the steps could have also been done with the Vault HTTP API using curl commands. the attacker gets a valid session token back. I want to be alerted if somebody logs into a production server. While the content in a GCE metadata token is trustworthy and controlled by Google, service account tokens are completely controlled by the owner of the service account and can therefore contain arbitrary claims. token_meta_canonical_arn arn:aws:iam::362381645759:role/chef-qa Instead of attaching some form of authentication token or credential to API requests, AWS requires clients to. You can enable and test Vault's AWS auth method following the steps below. This method supports two ways that EC2 instances can authenticate themselves to Vault: ec2 and iam. Recently I began working on a project to change how we log into our instances in AWS. You should run the command shown from a directory containing your SSH key or provide a path to the key. token_policies [default dev] When the lambda function executes, it authenticates to Vault by sending a request to the, API endpoint. The Vault server sends the pre-signed requests to the STS host and extracts the AWS IAM information out of the result. token_meta_inferred_aws_region us-east-1 , part of the AWS Security Token Service (STS). Login to the UI with a root token or other token that can create any policies. Follow these steps to authenticate against Vault with an EC2 instance in the dev role and read the dev secrets you created above.

By enabling and configuring the. can now be used to grant the dbclient role access to the database secret. We want to create some secrets for members of the dev and qa teams. If the attacker impersonated, a GCE instance with the right attributes (project, label, zones..) everything works out well and. can only be used to authenticate virtual machines running on Google Compute Engine. This brings us really close to our goal of spoofing an arbitrary caller identity: We just need to find a STS action that reflects attacker controlled text as part of its API response.

For infrastructure that runs on a supported cloud provider, using the provider's IAM platform for authentication is a logical choice. However, the complete lack of validation for URL path, query, POST body and HTTP headers still looks like a promising attack surface. Vault

token_meta_inferred_entity_type n/a We can now use our OIDP to sign a JWT that contains an arbitrary GetCallerIdentityResponse as part of its subject claim. Still, an attacker can just create their own OIDC Identity Provider (IdP), register it on an AWS account they own and sign arbitrary tokens with their own keys. In the end, Hashicorp fixed the vulnerability by enforcing an allowlist of HTTP headers, restricting requests to the GetCallerIdentity action and stronger validation of the STS response, which is hopefully enough to protect against unexpected changes to the STS implementation or HTTP parser differences between STS and Golang. This will work for GCE metadata tokens, but not for tokens signed by a service account: func (b *GcpAuthBackend) getSigningKey() (interface{}, error) {, b.Logger().Debug("Getting signing Key for JWT"), return nil, errors.New("expected token to have exactly one header"), b.Logger().Debug("kid found for JWT", "kid", kid), k, gErr := gcputil.OAuth2RSAPublicKey(ctx, kid), b.Logger().Debug("Found Google OAuth2 provider key", "kid", kid). The function extracts HTTP method, URL, body and headers out of the supplied request body which is stored in data. curl -X POST "https://vault-server/v1/auth/aws/login" -d '{"role":"dev-role-iam".

"SubjectFromWebIdentityToken":"arn:aws:iam::superprivileged-aws-accountXYZ"}. "arn:aws:iam::111111111111:role/vaultawsauth", "arn:aws:iam::222222222222:role/vaultawsauth", path "acg/globals/data/*" { header and put an arbitrary GetCallerIdentityResponse XML blob into the reflected payload. As mentioned above, all of this code is shared between the iam and gce auth methods. Vaults aws auth method supports two different authentication mechanisms internally: iam and ec2. If verification succeeds, Vault fills out the loginInfo struct that is later used to grant or deny access. If authentication succeeds, Vault returns a short-lived API token for the, role back to the lambda function. If the token contains a, "expected JWT to have non-empty 'sub' claim", "expected JWT to have claims with GCE metadata", As mentioned above, all of this code is shared between the, auth methods. the second statement allows vault to use sts to do the same validation in other accounts. token_meta_inferred_entity_id n/a That way, instances that have them will be able to authenticate against Vault. Assign the vault-aws-authentication policy shown above to both roles. This becomes even more difficult as the security depends on implementation details of the Security Token Service, which might change at any point in the future. Of course, a normal OIDC provider wont sign a JWT with an XML payload in the subject field. The next section describes how GCP authentication for Vault is implemented and how a simple logic flaw can lead to an authentication bypass in many configurations. Vault will deserialize the request, send it to STS and misinterpret the response. token_meta_canonical_arn arn:aws:iam::362381645759:role/roger20180321221209238500000001 See here for a simple proof-of-concept script that takes care of most of the details. token_meta_inferred_entity_id i-07b2ff4cc60049f47.

If authentication succeeds, Vault returns a short-lived API token for the dbclient role back to the lambda function. I have a bunch of lambdas in different accounts that would like to auth against this role. A decoded example token could look like this: match the details specified in the step 2. contains our spoofed response, identifying us as the AWS IAM account, 'https://oidc-test-wrbvvljkzwtfpiikylvpckxgafdkxfba.s3.amazonaws.com/', 'arn:aws:iam::superprivileged-aws-accountXYZ', We can test if everything is setup correctly by sending a direct request to the STS AssumeRoleWithWebIdentity action using the (signed) token from step 3 and the, 'https://sts.amazonaws.com/?DurationSeconds=900&Action=AssumeRoleWithWebIdentity&Version=2011-06-15&RoleSessionName=web-identity-federation&RoleArn=, "arn:aws:iam::XZY::YOUR-OIDC-ROLE/web-identity-federation", "AROATQ4R7PP5JJNLOF5P6:web-identity-federation", "arn:aws:iam::242434931706:oidc-provider/oidc-test-wrbvvljkzwtfpiikylvpckxgafdkxfba.s3.amazonaws.com/", arn:aws:iam::superprivileged-aws-accountXYZ, The final step is to convert this request into the form expected by Vault (e.g base64 encoding all required headers, the url and an empty post body) and to send it to the target Vault server as a login request on. The only auth restriction that cant be bypassed is a hardcoded service account name, as this value will be equal to the attacker account and not the expected VM account name. When the lambda function executes, it authenticates to Vault by sending a request to the /v1/auth/aws/login API endpoint. token_meta_auth_type iam Install the Vault binary by running these commands: Move vault to a location in your path such as /usr/local/bin with a command like, Determine your instance's pkcs#7 certificate by running, You can now use that certificate to authenticate to Vault with the command.