aws | ClickOops

If you’ve been working in AWS for long enough, you will know that nothing good comes from configuring resources in the console aka ClickOps. That being said, having a hard and fast rule that everybody should only have ReadOnly access in the console is also not great. I wanted something that would trigger when people are taking manual actions in the console and alert the team to investigate why this was done and what needs to be done to get our IaC deployment in sync with these changes.

For this reason, I’ve created ClickOops, a simple Lambda that monitors your CloudTrail log files to find manual actions taken in your accounts.

Filtering which events should be classified as a manual action is more involved than you might think. This post by Arkadiy Tetelman goes into detail about what to look for in CloudTrail Events and a sample python implementation can found in Towards Data Science’s blog.

Deploying ClickOops

Everything your need to deploy this can be on my Github page.

We use AWS ControlTower (CT) to manage our multi-account environment and guardrails. CT sets up a Log Archiving account where all CloudTrail and Config logs are forwarded to (for all accounts and regions in your organisation). This gives you a central place to process these logs and without having to worry about deploying (and protecting) these types of solutions across many accounts.

If you are not using CT, you can deploy this for a single account and point it to your CloudTrail logging bucket in the account. Ensure that the key of the files in S3 following this pattern : `*/{{account_id}}/CloudTrail/*/*.json.gz`

You will also need a Slack App configured with Incoming Webhooks. I prefer to configure incoming webhooks to channels that will only be used for one specific purpose to prevent messages from getting lost in busy channels.

After deploying the solution you need to update the SSM parameter containing the Slack Webhook Url. Ironically this is a ClickOps action and you will be notified about this.

ClickOops messages will contain a summary of the IAM action and AWS identity (which should always be a human) which triggered the event. The full CloudTrail event will also be included in the message. An example of that can be seen below.

Future work

As AWS released new services we will need to update the list of CloudTrail events that are allowed to be taken in the console and filter our false positives as I find them. It might also be handy to group accounts and sent messages related to them to a separate Slack Channel.

I’d like to get your feedback if you are using something like this in your environments to monitor how users are interacting with AWS or if you found this helpful.

Updates

[2022–03–15] We’ve created a standalone Terraform module (https://github.com/cloudandthings/terraform-aws-clickops-notifier) for this that is easier to consume and has a few bonus features 🤩