Server-less Slack Bot Token Rotation

When you setup a new Slack bot, you have the option to configure it with expiring tokens used for communicating with Slack. This can be desirable if you do not want your Bot tokens to sit around aging and potentially leaking/getting compromised in some way. Slack recommends configuring this for “developers building on or for security-minded organizations”.

This generally provides an extra layer of security for your access tokens that can be used to do anything your Bot can do in any Slack instance it is installed.

Recently with the “server-less” Slack bot deployment I’ve been working on, I found it difficult to find documentation online for standing up infrastructure in AWS that would refresh my bot token reliably before it expired. Slack’s expiring tokens have a lifetime of 12 hours in which they need to be refreshed, but I found that freshing them slightly more often increased reliability of storing them in Secrets Manager.

I discovered my solution to this issue using a 2018 article from AWS on rotating Twitter API keys as my template. In this article, the author suggests using AWS Secrets Manager with a customized Lambda function that runs all the token refresh calls against Twitter’s API to facilitate the rotation and to replace the stored secret in AWS. I made mine do the same against Slack’s API.

However, Secrets Manager has a limitation in that you can only choose 1 day as the minimum rotation period for a secret (and this 1 day rotation period can be as long as 48 hours if you’re unlucky with the randomization). I don’t think AWS expected this tool to be used for things like API keys that require shorter-lived credentials. To get around this limitation, I thought to use a scheduled CloudWatch event rule to trigger this rotation Lambda function within the 12 hour lifespan of the token. Conveniently, these CloudWatch event rules can be scheduled like a cron job on a Unix system.

In my deployment, I tried a few different intervals for freshing the token. Initially, I tested it for a couple weeks refreshing it every 10 minutes to simulate months of refresh cycles and it never got out of sync. In order to reduce the amount of time my Lambda functions were running, I extended my CloudWatch rule to run every 10 hours. With this single token refresh in the token lifetime, I did run into syncing issues where the token would expire if the refresh didn’t go exactly right.

I ended up with a happy medium where the refresh happens every 4 hours (3 opportunities in the token’s lifetime) and I have not had any syncing issues and use of expiring token has been stable.

Although the Bolt library can help do this refresh process auto-magically, I’ve found it beneficial to have full control over my tokens and to understand the mechanisms behind the secret update process in Secrets Manager. I’ve also been trying to make my Lambda functions as lightweight and fast as possible, so I decided to stick with the Python slack_sdk library rather than Bolt in the hopes that my functions will be faster and lighter.

This guide was written and created with Python version 3.8. If you are using a different version for your bot, you will need to change all version references to the version you are using.

AWS Console Guide #

Secrets Manager #

First you will need to create the Slack bot secret in AWS Secrets Manager.

1. Go to AWS Secrets Manager in your console and click “Store a new secret”.
2. Select “Other type of secrets” for the secret type and input your Slack secrets as shown below (you won’t yet have a Slack refresh token if you haven’t exchanged your non-expiring token for an expiring one).
3. Input the name of the secret e.g. “slack-bot-secrets”.
4. Automatic secret rotation will be disabled for now.
5. Scroll down and click “Store” on the last page after confirming all your configurations.
6. Click the created secret and note down the secret ARN. You will need it in later steps.

Lambda Function Layer #

Second, you will need the slack_sdk layer setup for the Lambda function. For my own deployment, I am using the slack_sdk library in multiple Lambda functions and the layer allows the dependency to easily be shared across multiple functions and allows for easier maintenance/troubleshooting.

1. Create the required slack_sdk zip file for uploading to the Lambda Layer. Run the following commands (for Linux/Mac, they won’t be very different on Windows with a similar zip utility):

$ pip install --target ./python/lib/python3.8/site-packages slack_sdk
$ zip -r slack_sdk.zip python

2. Within the AWS Lambda service in your AWS console, go to Layers on the left hand side and click “Create Layer”.
3. Enter the name “slack_sdk” for the layer, upload the zip created in step 1, select Python 3.8 for the compatible runtime, and click “Create”.
4. This Layer will now be available for any Lambda function that needs it.

Main Lambda Function #

Third, you need to create the Lambda functions that will be doing the Slack bot token rotation.

1. You need to setup your first Lambda function for doing the rotation - slack-bot-rotation.py below has the code that will be used by Secrets Manager to rotate the token. Make sure “Create a new role with basic Lambda permissions” is selected.
2. Paste in the code from slack-bot-rotation.py and click deploy.
3. In the same Code tab of the Lambda function, scroll down to the bottom to add the layer you created in the second phase.
4. Select Custom layers and select the slack_sdk layer from the dropdown, along with the version. Click “Add” to add it to the Lambda function.
5. After you’ve confirmed the layer has been added, go back to the Lambda settings. Click on Configuration -> Permissions and click the role that was created with the Lambda function.
6. You will be brought to the role in the IAM interface. Click “Attach policies”.
7. A custom policy to allow the Lambda to rotate the secret is required. Click “Create policy”.
8. Click the JSON tab to enter the custom policy. This is the JSON to copy into the policy with your secret ARN from phase 1:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLambdaToStoreAndRetrieveSecret",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:DescribeSecret",
                "secretsmanager:GetSecretValue",
                "secretsmanager:PutSecretValue",
                "secretsmanager:UpdateSecretVersionStage"
            ],
            "Resource": "your-secret-arn"
        },
        {
            "Sid": "AllowLambdaToListAllSecrets",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
        }
    ]
}

9. Enter the name and click “Create policy”.
10. Go back to your IAM role tab with the Lambda role and refresh. You should now be able to attach the created policy by searching for “slack” in the attach policy page.
11. Now go back to your Lambda function tab and refresh to confirm the function now has access to the secret create in phase 1.
12. You will now need to allow Secrets Manager service to invoke this function on its own. Under Configuration -> Permissions, go to the “Resource-based policy” section and click “Add permissions”.
13. In the policy statement select “AWS Service” and then Secrets Manager from the dropdown. Then choose “lambda:InvokeFunction” as the Action and give it a Statement ID.

Your rotation Lambda function is now ready to be added Secrets Manager for automatic rotation.

CloudWatch event rule #

Fourth, the CloudWatch event rule needs to be created for rotating the Slack bot secret more often than every day.

1. First the Lambda function used by the CloudWatch rule needs to be created. Go to the Lambda service in your console and click “Create function” again.
2. Name the function and click “Create function”.
3. Paste in the slack-bot-rotation-trigger.py code from below and click “Deploy”.
4. Again we need to edit the created Lambda function role to include the ability to interact with our secret. Click the role under Configuration -> Permissions.
5. Attach and create a policy with the following JSON with ** your secret ARN**:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLambdaToRotateSecret",
            "Effect": "Allow",
            "Action": ["secretsmanager:RotateSecret"],
            "Resource": "your-secret-arn"
        },
        {
            "Sid": "AllowLambdaToListAllSecrets",
            "Effect": "Allow",
            "Action": ["secretsmanager:ListSecrets"],
            "Resource": "*"
        }
    ]
}

6. Make sure the created policy is attached.
7. Now go to the CloudWatch service in your console and select “Rules” under the “Events” option on the left side. Click “Create rule”.
8. Create a rule with the following configuration (running every 4 hours and running the Lambda function created above):
9. Name and create the rule.
10. With the rule enabled, the Slack bot token will be rotated every 4 hours and stored in Secrets Manager.

Slack Bot Rotation Lambda Function #

slack-bot-rotation.py #

CloudWatch Event Rule Lambda Function #

slack-bot-rotation-trigger.py #

While this post is an explainer of what I’ve learned on the topic, it could be a great demonstration of Cunningham’s Law…

The best way to get the right answer on the internet is not to ask a question; it’s to post the wrong answer.

Do let me know in the comments if you’ve found a better way of managing Slack bot token rotation with server-less infrastructure in AWS.

Terraform #

If you use Terraform for managing your AWS infrastructure, I have the configuration for mine below. This is everything I went through in the AWS console above, including the required IAM roles/permissions.

Using the above Terraform code, you would use add this object to your tfvars file including your Slack secrets.

slack_secrets = {
  client_id            = "XXXXXXXXXX"
  client_secret        = "XXXXXXXXXX"
  slack_bot_token      = "xoxb-XXXXXXX"
  refresh_token        = ""
  slack_signing_secret = "XXXXXXXXXX"
}