Detecting Manual AWS Console Actions

Nov 12th, 2019 | 20 minute read

UPDATE 2/18/24: Check out the update to this post 🙂

In this post I’ll describe a set of AWS Cloudtrail alerting rules that let you detect when someone makes a manual change in your AWS Console. This has been one of the highest signal / lowest noise alerts we created in our organization - it lets us know when engineers do things like, i.e., manually add new security group ingress rules through the AWS Console: alert

Motivation

It’s not a controversial opinion that making infrastructure changes through the AWS Console will never scale beyond the smallest organizations and projects. Engineers make temporary changes and forget to revert them, waste money with test infrastructure that never gets spun down, and step on each other’s toes with conflicting changes. It leaves your AWS account in an ill-defined, unreproducible state.

A better approach is practicing infrastructure as code, using tools like AWS Cloudformation, Terraform, or Cloud Development Kit (CDK). Using these tools all infrastructure can go through code review, maintain a change management history, and be searchable & auditable.

At the same time it is undeniably true that certain actions are simply easier through the AWS Console - engineers might need to test some functionality quickly which would otherwise be cumbersome through, i.e. Terraform. As security practitioners this is unfortunate for us since manual changes are much more likely to unintentionally expose something to the internet or cause other problems. It would be very easy for us “solve” this issue by blocking engineering access to the AWS Console, but since security is an enabling function I want engineers to have the access they need to do their jobs and iterate quickly. Thus at my organization we grant all engineers full access to the AWS Console, and instead alert whenever they make a manual change. Together with our other AWS security controls, this strikes a good balance between usability and safety.

Detecting AWS Console changes

This sounds great in theory, but in practice it turns out detecting manual AWS Console changes is not trivial.

Amazon does not expose any field in Cloudtrail logs indicating the origin of the action, so our only option is to key off of the user agent used to make the request (AWS uses custom user agents like S3Console when you use the S3 dashboard, for instance). However the user agents are not consistent - other service dashboards besides S3 use a different user agent, some service dashboards pass through the user agent of your browser, etc.

As a result we’ve built up and tuned the query below over time, based on the false positives and negatives we were seeing. It works great for the services we use (mostly EC2, ECS, S3, Lambda, RDS, Route53, and CloudWatch). If you use additional services you might need to tweak it over time as well, but it should still serve as a strong base to start from.

In pseudocode, we’re alerting whenever the following conditions are true:

at least one of the following is true:

userAgent is one of:

["console.amazonaws.com", "Coral/Jakarta", "Coral/Netty4", "AWS CloudWatch Console"]

or, userAgent starts with one of these prefixes:
```
["S3Console/", "[S3Console", "Mozilla/"]
```

or, userAgent matches one of these wildcard patterns:

["console.*.amazonaws.com", "aws-internal*AWSLambdaConsole/*"]

and, none of the following are true:

eventName starts with one of these prefixes:
```
["Get", "Describe", "List", "Head"]
```

or, eventName is one of:

["DownloadDBLogFilePortion", "TestScheduleExpression", "TestEventPattern", "LookupEvents", "listDnssec", "Decrypt", "REST.GET.OBJECT_LOCK_CONFIGURATION", "ConsoleLogin"]

or, userIdentity.invokedBy is AWS Internal
or, all the following conditions are true:
- eventName is AssumeRole
- and, userAgent is Coral/Netty4
- and, userIdentity.invokedBy is one of
```
["ecs-tasks.amazonaws.com", "ec2.amazonaws.com", "monitoring.rds.amazonaws.com", "lambda.amazonaws.com"]
```

If you’re using an ELK stack for logging like we do, you can also copy our raw elasticsearch query syntax:

Click to expand query

"query": {
  "bool": {
    "must": [
      {
        "range": {
          "@timestamp": {
            "gte": "now/m-1m",
            "lt": "now/m"
          }
        }
      },
      {
        "bool": {
          "should": [
            {
              "term": {
                "userAgent": "console.amazonaws.com"
              }
            },
            {
              "term": {
                "userAgent": "Coral/Jakarta"
              }
            },
            {
              "term": {
                "userAgent": "Coral/Netty4"
              }
            },
            {
              "term": {
                "userAgent": "AWS CloudWatch Console"
              }
            },
            {
              "prefix": {
                "userAgent": "S3Console/"
              }
            },
            {
              "prefix": {
                "userAgent": "[S3Console/"
              }
            },
            {
              "prefix": {
                "userAgent": "Mozilla/"
              }
            },
            {
              "wildcard": {
                "userAgent": "console.*.amazonaws.com"
              }
            },
            {
              "wildcard": {
                "userAgent": "aws-internal*AWSLambdaConsole/*"
              }
            }
          ]
        }
      }
    ],
    "must_not": [
      {
        "prefix": {
          "eventName": "Get"
        }
      },
      {
        "prefix": {
          "eventName": "Describe"
        }
      },
      {
        "prefix": {
          "eventName": "List"
        }
      },
      {
        "prefix": {
          "eventName": "Head"
        }
      },
      {
        "term": {
          "eventName": "DownloadDBLogFilePortion"
        }
      },
      {
        "term": {
          "eventName": "TestScheduleExpression"
        }
      },
      {
        "term": {
          "eventName": "TestEventPattern"
        }
      },
      {
        "term": {
          "eventName": "LookupEvents"
        }
      },
      {
        "term": {
          "eventName": "listDnssec"
        }
      },
      {
        "term": {
          "eventName": "Decrypt"
        }
      },
      {
        "term": {
          "eventName": "REST.GET.OBJECT_LOCK_CONFIGURATION"
        }
      },
      {
        "term": {
          "eventName": "ConsoleLogin"
        }
      },
      {
        "term": {
          "userIdentity.invokedBy.keyword": "AWS Internal"
        }
      },
      {
        "bool": {
          "must": [
            {
              "term": {
                "eventName": "AssumeRole"
              }
            },
            {
              "term": {
                "userAgent": "Coral/Netty4"
              }
            },
            {
              "bool": {
                "should": [
                  {
                    "term": {
                      "userIdentity.invokedBy.keyword": "ecs-tasks.amazonaws.com"
                    }
                  },
                  {
                    "term": {
                      "userIdentity.invokedBy.keyword": "ec2.amazonaws.com"
                    }
                  },
                  {
                    "term": {
                      "userIdentity.invokedBy.keyword": "monitoring.rds.amazonaws.com"
                    }
                  },
                  {
                    "term": {
                      "userIdentity.invokedBy.keyword": "lambda.amazonaws.com"
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  }
}

Conclusion

This alerting has been very successful for our organization - it lets us know when engineers are doing something dangerous, and even for safer actions it’s a good way to stay up-to-date on what engineers are shipping. If anyone from AWS is reading this, it would be a huge help if Cloudtrail indicated when events were initiated via the AWS Console.

P.S. If you enjoy this kind of content feel free to follow me on Twitter: @arkadiyt

Arkadiy Tetelman A security blog

Detecting Manual AWS Console Actions

Motivation

Detecting AWS Console changes

Conclusion