Prometheus EC2 SD with Multiple AWS Accounts

Recently I needed to setup a Prometheus instance in AWS that could scrape exporters in other AWS accounts and use SD to discover them. While on the surface this is simple, I couldn’t find a direct example on the Internet.

First, you will need to setup an IAM role in each account you wish to discover in. This role will need a trust setup with the account in which your Prometheus instance lives. You will also need to grant EC2 read only privileges so that this role may be used for service discovery. An example CloudFormation statement is below:

PrometheusAssumeRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            AWS:
              - "arn:aws:iam::<MAIN-ACCOUNT-ID>:root"
          Action:
            - "sts:AssumeRole"
    ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess'

Next you need to update your Prometheus role to allow it privileges to AssumeRole. Again an example CF statement is below:

PrometheusRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - "ec2.amazonaws.com"
          Action:
            - "sts:AssumeRole"
    ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess'
    Policies:
      - PolicyName: PrometheusAssumeRole
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
              - "sts:AssumeRole"
            Resource: "*"

Finally you will need to define which role to use when setting up the SD. An example Prometheus job definition in prometheus.yml:

- job_name: node
  ec2_sd_configs:
    # Discover instances in the same account
    - port: 9100
    # Discover instances in account 111111111111
    - port: 9100
      role_arn: arn:aws:iam::111111111111:role/prometheus-assume-role-PrometheusAssumeRole-KJ4TK9KJBPU7 (note this is the ARN of the role created in the other account, yours may be different especially if you do not use CloudFormation)
    # Continue as many entries as you have other accounts/regions
    - port:9100
      role_arn: arn:aws:iam::222222222222:role/prometheus-assume-role-PrometheusAssumeRole-KJ4TK9KJBPU7
      region: us-west-2

And that is it. You can use a single role in your parent account to assume roles to other accounts and keep permissions contained and simple. You may want to lock down permissions more than I have shown here. For example you may not want to allow Prometheus to assume any role. You may not also want the Prometheus role to just discover any and all EC2 instances. These are limits you can apply in the role policy statements.