Streamlit deployment on AWS ECS with Pulumi on a custom domain
Guide and snippets
Streamlit is a powerful tool for building interactive data apps in Python. This guide will show you how to deploy a Streamlit app on AWS ECS using Pulumi, with a Docker image from DockerHub, an Application Load Balancer (ALB), a custom domain, and AWS WAF rules to block unwanted crawling.
Setting up docker access on DockerHub
To start deploying your Streamlit app on AWS ECS, you’ll first need to set up Docker access on DockerHub. This involves creating a secret in AWS Secrets Manager to securely store your DockerHub credentials. Below is the Pulumi code snippet for this step:
import json
import pulumi_aws as aws
docker_hub_secret = aws.secretsmanager.Secret(
"dockerhubCredsSecret",
name="dockerhubCredsSecret",
description="Docker Hub Secret",
)
aws.secretsmanager.SecretVersion(
"dockerhubCredsSecretVersion",
secret_id=docker_hub_secret.id,
secret_string=json.dumps(
{
"username": DOCKERHUB_USERNAME,
"password": DOCKERHUB_TOKEN,
},
),
)In this snippet, we create a secret in AWS Secrets Manager to store your DockerHub credentials. Replace DOCKERHUB_USERNAME and DOCKERHUB_TOKEN with your actual DockerHub username and token. This secret will be used later to authenticate and pull your Docker image.
Creating roles and permissions
To deploy your Streamlit app on ECS, you need to create an IAM role with the necessary permissions. Below is the Pulumi code snippet for setting up the role and attaching the required policies.
import json
import pulumi_aws as aws
from pulumi_aws.iam import Role, Policy, RolePolicyAttachment
# Create IAM role
role = Role(
"EcsServiceRole",
assume_role_policy="""{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": [
"ecs-tasks.amazonaws.com",
"lambda.amazonaws.com"
]
},
"Effect": "Allow",
"Sid": ""
}
]
}""",
)
# Define necessary actions
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface",
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"s3:GetObject",
"kms:Decrypt",
"secretsmanager:GetSecretValue"
]
# Create IAM policy
policy = Policy(
"EcsServicePolicy",
description="Policy for ECS to access necessary services",
policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Action": actions,
"Resource": "*",
"Effect": "Allow",
},
],
},
),
)
# Attach policy to role
RolePolicyAttachment(
"EcsServicePolicyAttachment",
policy_arn=policy.arn,
role=role.name,
)In this snippet, we create an IAM role and attach a policy that grants the necessary permissions for ECS to function with Streamlit. The actions include permissions for logging, network interfaces, and accessing ECR, S3, KMS, and Secrets Manager
Creating docker image
Next, you’ll need to create a Docker image for your Streamlit app. Below is the Pulumi code snippet to build and push the Docker image to a registry.
import pulumi_docker as docker
docker.Image(
STACK,
build=docker.DockerBuildArgs(
args={
"BUILDKIT_INLINE_CACHE": "1",
},
builder_version=docker.BuilderVersion.BUILDER_BUILD_KIT,
context=".",
dockerfile="src/<your_project>/Dockerfile",
),
image_name=IMAGE_NAME,
)In this snippet, we use Pulumi’s Docker provider to build the Docker image. The dockerfile parameter should point to the location of your Dockerfile, and image_name should be the desired name of your image. The BUILDKIT_INLINE_CACHE argument helps optimize the build process by using Docker's BuildKit.
Setting up ECS and ALB
To deploy your Streamlit app on ECS with an Application Load Balancer (ALB), you’ll need to create an ECS task definition, set up the ECS cluster, security groups, ALB, and target group. Below are the Pulumi code snippets for these steps.
ECS Task definition:
import pulumi_aws as aws
import pulumi
def create_ecs_task(role_arn, docker_hub_secret_arn, container_environment):
container_definition_output = pulumi.Output.all(container_environment, docker_hub_secret_arn).apply(
lambda args: json.dumps(
[
{
"name": f"{STACK}-cockpit-container",
"image": IMAGE_NAME,
"cpu": 256,
"portMappings": [
{"containerPort": 8501, "hostPort": 8501, "protocol": "tcp"},
],
"healthCheck": {
"command": ["CMD-SHELL", "curl -f http://localhost:8501/ || exit 1"],
"interval": 10,
"timeout": 10,
"retries": 10,
"startPeriod": 15,
},
"repositoryCredentials": {
"credentialsParameter": args[1],
},
"environment": [{"name": key, "value": value} for key, value in args[0].items()],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "streamlit",
"awslogs-region": REGION,
"awslogs-stream-prefix": STACK,
},
},
},
],
),
)
return aws.ecs.TaskDefinition(
"cockpitTask",
family="cockpit",
cpu="1024" if STACK == "main" else "256",
memory="2048" if STACK == "main" else "512",
network_mode="awsvpc",
requires_compatibilities=["FARGATE"],
execution_role_arn=role_arn,
task_role_arn=role_arn,
container_definitions=container_definition_output,
runtime_platform={"operatingSystemFamily": "LINUX", "cpuArchitecture": "X86_64"},
)ECS Cluster and security group
def create_security_group():
return aws.ec2.SecurityGroup(
"securityGroup",
vpc_id=os.environ["VPC_ID_EU_WEST_1"],
ingress=[
{
"protocol": "-1",
"from_port": 0,
"to_port": 0,
"cidr_blocks": ["0.0.0.0/0"],
},
],
egress=[
{
"protocol": "-1",
"from_port": 0,
"to_port": 0,
"cidr_blocks": ["0.0.0.0/0"],
},
],
)ALB and Target group
def create_alb(subnet1_id, subnet2_id, security_group_id):
alb = aws.lb.LoadBalancer(
"alb",
subnets=[subnet1_id, subnet2_id],
security_groups=[security_group_id],
load_balancer_type="application",
)
target_group = aws.lb.TargetGroup(
"tg",
port=8501,
protocol="HTTP",
vpc_id=os.environ["VPC_ID_EU_WEST_1"],
target_type="ip",
)
return alb, target_groupECS Service with ALB
def create_ecs_service_with_alb(cluster_arn, task_definition_arn, subnet1_id, subnet2_id, security_group_id, target_group_arn, alb_arn):
service = aws.ecs.Service(
"service",
cluster=cluster_arn,
desired_count=1,
launch_type="FARGATE",
enable_execute_command=False if STACK == "main" else True,
task_definition=task_definition_arn,
network_configuration={
"assign_public_ip": True,
"subnets": [subnet1_id, subnet2_id],
"security_groups": [security_group_id],
},
load_balancers=[
{
"target_group_arn": target_group_arn,
"container_name": "container",
"container_port": 8501,
},
],
)
aws.lb.Listener(
"httpsListener",
load_balancer_arn=alb_arn,
default_actions=[{"type": "forward", "target_group_arn": target_group_arn}],
port=443,
protocol="HTTPS",
certificate_arn=COCKPIT_CERTIFICATE_ARN,
)
aws.lb.Listener(
"httpListener",
load_balancer_arn=alb_arn,
default_actions=[
{
"type": "redirect",
"redirect": {
"port": "443",
"protocol": "HTTPS",
"status_code": "HTTP_301",
},
},
],
port=80,
protocol="HTTP",
)Setting up a firewall with WAF
To enhance the security of your Streamlit app, we’ll set up AWS Web Application Firewall (WAF). AWS WAF helps protect your application from common web exploits and bots that can affect availability, compromise security, or consume excessive resources. Below are the Pulumi code snippets for configuring WAF rules to block unwanted traffic.
def create_waf_rules(alb_arn):
rules = [
create_geo_rule(),
create_bot_control_rule(),
create_rate_based_rule(),
]
web_acl = create_web_acl(alb_arn, rules)
log_group = aws.cloudwatch.LogGroup(
f"aws-waf-logs-aclLog",
retention_in_days=14,
skip_destroy=False,
)
aws.wafv2.WebAclLoggingConfiguration(
"aclLogConfig",
resource_arn=web_acl.arn,
log_destination_configs=[log_group.arn],
)
def create_geo_rule():
geo_match_statement = aws.wafv2.WebAclRuleStatementGeoMatchStatementArgs(country_codes=["DE", "NL", "PY"])
return aws.wafv2.WebAclRuleArgs(
name="NonSpecificCountriesRuleArgs",
priority=1,
statement=aws.wafv2.WebAclRuleStatementArgs(
not_statement=aws.wafv2.WebAclRuleStatementNotStatementArgs(
statements=[aws.wafv2.WebAclRuleStatementArgs(geo_match_statement=geo_match_statement)],
),
),
visibility_config=aws.wafv2.WebAclRuleVisibilityConfigArgs(
cloudwatch_metrics_enabled=True,
metric_name="NonSpecificCountriesRuleMetric",
sampled_requests_enabled=True,
),
action=aws.wafv2.WebAclRuleActionArgs(block=aws.wafv2.WebAclRuleActionBlockArgs()),
)
def create_bot_control_rule():
statement = aws.wafv2.WebAclRuleStatementArgs(
managed_rule_group_statement=aws.wafv2.WebAclRuleStatementManagedRuleGroupStatementArgs(
name="AWSManagedRulesBotControlRuleSet",
vendor_name="AWS",
),
)
return aws.wafv2.WebAclRuleArgs(
name="botControlRule",
priority=2,
override_action=aws.wafv2.WebAclRuleOverrideActionArgs(
none=aws.wafv2.WebAclRuleOverrideActionNoneArgs(),
),
statement=statement,
visibility_config=aws.wafv2.WebAclRuleVisibilityConfigArgs(
cloudwatch_metrics_enabled=True,
metric_name="botControlMetric",
sampled_requests_enabled=True,
),
)
def create_rate_based_rule():
return aws.wafv2.WebAclRuleArgs(
name="rateBasedRuleArgs",
priority=3,
statement=aws.wafv2.WebAclRuleStatementArgs(
rate_based_statement=aws.wafv2.WebAclRuleStatementRateBasedStatementArgs(
limit=100,
aggregate_key_type="IP",
),
),
visibility_config=aws.wafv2.WebAclRuleVisibilityConfigArgs(
cloudwatch_metrics_enabled=True,
metric_name="rateBasedRuleMetric",
sampled_requests_enabled=True,
),
action=aws.wafv2.WebAclRuleActionArgs(block=aws.wafv2.WebAclRuleActionBlockArgs()),
)
def create_web_acl(alb_arn, rules):
web_acl = aws.wafv2.WebAcl(
"webAcl",
default_action=aws.wafv2.WebAclDefaultActionArgs(
allow=aws.wafv2.WebAclDefaultActionAllowArgs(),
),
scope="REGIONAL",
visibility_config=aws.wafv2.WebAclVisibilityConfigArgs(
cloudwatch_metrics_enabled=True,
metric_name="webAcl",
sampled_requests_enabled=True,
),
rules=rules,
tags={
"Name": "webAcl",
},
)
aws.wafv2.WebAclAssociation("webAclAssociation", resource_arn=alb_arn, web_acl_arn=web_acl.arn)
return web_aclConclusion
Deploying a Streamlit app on AWS ECS with Pulumi provides a scalable and manageable solution for running interactive data applications. This guide walked you through setting up Docker access, creating roles and permissions, building a Docker image, configuring ECS and ALB, setting up a WAF, and using Route 53 for a custom domain. Additionally, we covered an alternative configuration for testing environments without ALB.
By following these steps, you can ensure that your Streamlit app is secure, reliable, and easy to access.
This article was written by Artem Kozlov and Thorin Schiffer.Setting up a custom domain with Route53
To make your Streamlit app easily accessible, you’ll want to set up a custom domain. Using Amazon Route 53, we can create a DNS record that points to your application’s Application Load Balancer (ALB). Below is the Pulumi code snippet to configure the custom domain.
import pulumi_aws as aws
import pulumi
record = aws.route53.Record(
"cockpitCName",
zone_id=HOSTED_ZONE_ID,
name=f"{get_prefix()}.cockpit.{DOMAIN_NAME}",
type="CNAME",
ttl=300,
records=[alb.dns_name],
)
pulumi.export("DOMAIN_NAME", record.fqdn)In this snippet, we create a CNAME record in Route 53, pointing to the DNS name of the ALB. Replace HOSTED_ZONE_ID and DOMAIN_NAME with your actual hosted zone ID and desired domain name.
Alternative configuration without ALB and domain
For testing environments, you might not need the overhead of an Application Load Balancer (ALB) and a custom domain. Instead, you can deploy your Streamlit app on ECS with a simpler configuration. Below is the Pulumi code snippet for setting up ECS without ALB.
import pulumi_aws as aws
def create_ecs_without_alb():
task_definition = create_ecs_task(role.arn, docker_hub_secret.arn, container_environment)
ecs_cluster = aws.ecs.Cluster("cluster")
subnet1 = aws.ec2.Subnet.get("euWest1A", id=SUBNET_ID_EU_WEST_1A)
subnet2 = aws.ec2.Subnet.get("euWest1B", id=SUBNET_ID_EU_WEST_1B)
security_group = create_security_group()
service = aws.ecs.Service(
"service",
cluster=ecs_cluster.arn,
desired_count=1,
launch_type="FARGATE",
task_definition=task_definition.arn,
network_configuration={
"assign_public_ip": True,
"subnets": [subnet1.id, subnet2.id],
"security_groups": [security_group.id],
},
deployment_minimum_healthy_percent=0,
deployment_maximum_percent=100,
)In this snippet, we create an ECS service without an ALB. This setup is simpler and suitable for testing environments. It includes creating the ECS cluster, defining the task, and configuring the network settings.
Conclusion
Deploying a Streamlit app on AWS ECS with Pulumi provides a scalable and manageable solution for running interactive data applications. This guide walked you through setting up Docker access, creating roles and permissions, building a Docker image, configuring ECS and ALB, setting up a WAF, and using Route 53 for a custom domain. Additionally, we covered an alternative configuration for testing environments without ALB.
By following these steps, you can ensure that your Streamlit app is secure, reliable, and easy to access.
This article was written by Artem Kozlov and Thorin Schiffer.
