Scraping ETF KPIs with AWS Lambda, AWS Fargate, and Alpha Vantage & Yahoo Finance APIs

Public Subnets

In this setup, the ECS Fargate task that runs the containerized application is placed in public subnets, which allows the application code to directly access the internet via the internet gateway. The table below summarizes the key components from the template:

The diagram below depicts the infrastructure in the us-east-1 region:

Private Subnets

In this stack, the ECS Fargate task is placed in private subnets. The table below summarizes the key components from the template:

This infrastructure can be visualized as follows:

Which Subnet Setup Should We Choose?

When deciding between public and private subnets, consider the following definitions from the official documentation:

• Public Subnet: The subnet has a direct route to an internet gateway. Resources in a public subnet can access the public internet.
• Private Subnet: The subnet does not have a direct route to an internet gateway. Resources in a private subnet require a NAT device to access the public internet.

As we shall see, the ETF scraper code only needs outbound traffic to make API calls, and no inbound traffic is expected. Therefore, the two setups are functionally equivalent for this project. Regardless of whether the ECS Fargate task is deployed in public subnets or private subnets, we can specify a security group with rules that prevent any inbound traffic and allows only outbound traffic.

Still, while the public subnets setup may seem simpler, it increases the risk of misconfiguration at the security group level if any inbound traffic is in fact required. The best practice, in general, is to deploy in private subnets and use NAT gateways to access the internet.

Lambda Execution Role

The Lambda execution role allows Lambda to interact with other AWS services.

• Role Name: ${AWS::StackName}-lambda-execution-role
• Policies:
  • LambdaLogPolicy: Allows Lambda to write logs to CloudWatch.
  • LambdaECSPolicy: Allows Lambda to run ECS tasks.
  • LambdaIAMPolicy: Allows Lambda to pass the ECS execution role and task role to ECS; this policy is useful for restricting the Lambda function to only pass specified roles to ECS.

ECS Execution Role

The ECS execution role allows ECS to interact with other AWS services.

• Role Name: ${AWS::StackName}-ecs-execution-role
• Policies:
  • ECSExecutionPolicy: Allows ECS to log into and pull images from ECR, write logs to CloudWatch, get environment files from S3.

ECS Task Role

The ECS task role allows the Fargate task to interact with S3, enabling the application code to upload the scraped data. The task role should contain all permissions required by the application code running in the container. It is separate from the ECS execution role, which is used by ECS to manage the task and not by the task itself.

• Role Name: ${AWS::StackName}-ecs-task-role
• Policies:
  • ECSTaskPolicy: Allows the Fargate task to upload and get objects from S3.

GitHub Actions Role

We enable workflows to authenticate with AWS through Github’s OIDC provider, facilitating secure and direct interactions with AWS services without needing to store long-term credentials as secrets.

• Role Name: ${AWS::StackName}-github-actions-role
• Trust Relationship:
  • Establish a trust relationship with GitHub’s OIDC provider, allowing it to assume this role when authenticated via OIDC.
  • Access is restricted to actions triggered by push to the main branch of the specified GitHub repository, ensuring that only authorized code changes can initiate AWS actions.
• Policies:
  • GithubActionsPolicy: Allows the workflows that assume this role to update the Lambda function, push Docker images to ECR, and interact with S3.

Outputs

The template outputs the ARNs of the roles and the secrets for the GitHub Actions user, which can then be accessed from the console.

• LambdaExecutionRoleArn: ARN of the Lambda execution role.
• ECSExecutionRoleArn: ARN of the ECS execution role.
• ECSTaskRoleArn: ARN of the ECS task role.
• GithubActionsRoleArn: ARN of the GitHub Actions role.

Cluster

An ECS Fargate task is typically run in a cluster, which is a logical grouping of tasks. The template linked above creates an ECS cluster with the following properties:

• ClusterSettings: Enables container insights for the cluster, which automatically collects usage metrics for CPU, memory, disk, and network.
• CapacityProviders: Specifies FARGATE and FARGATE_SPOT (i.e., interruption tolerant tasks at discounted rate relative to on-demand) as capacity providers to optimize cost and availability.
• DefaultCapacityProviderStrategy: Distributes tasks evenly between FARGATE and FARGATE_SPOT.
• Configuration: Enables ExecuteCommandConfiguration with DEFAULT logging using awslogs, which uses the logging configurations defined in the container definition.

Task & Container Definitions

The task definition specifies the IAM roles and compute resources for the task, while the container definition specifies the Docker image and environment variable file locations, and logging configuration for the container.

Important: ECS Fargate requires the awsvpc network mode, providing each task with an elastic network interface. This ensures that each task has its own network interface, improving isolation and security. In our Lambda function code (link), we use the boto3 library to run the ECS Fargate task, specifying the subnets and security group to attach to the network interface.

Outputs

The template outputs three values:

• ECSFargateClusterName: The name of the ECS cluster.
• ECSFargateTaskDefinitionFamily: The name of the task definition family.
• ECSFargateContainerName: The name of the container within the task definition.

All the above will be used as environment variables in the Lambda function to properly trigger the ECS Fargate task.

Lambda Function

The Lambda function is responsible for invoking the ECS Fargate task, via boto3, which runs the ETF scraper application code. The following properties are important to note:

• Handler: The handler method within the Lambda function’s code. For this project, it is lambda_function.lambda_handler. In general, it should match the file name and the method name in the source code.
• Runtime: The runtime environment for the Lambda function, set to python3.11 to match with the python version specified in pyproject.toml.
• Code: The location in S3 where the Lambda function’s deployment package (ZIP file) is stored.

VPC

NAT Gateway:

• Gateway usage: \(730\) hours/month x \(0.045 = \$32.85\)
• Data processing: \(3\) GB/month x \(0.045 = \$0.14\) (This may vary depending on the data processed)
• Total NAT Gateway cost: \(\$32.99\)/month

Note: This cost may be avoided by using a public subnet setup.

Public IPv4 Address:

• \(1\) address x \(730\) hours/month x \(0.005 = \$3.65\)
• Total Public IPv4 Address cost: \(\$3.65\)/month

Fargate

Assuming \(21\) trading days per month, the cost of running the Fargate task for \(21\) days with the following resources:

• vCPU hours:
  • \(21\) tasks x \(1\) vCPU x \(0.67\) hours x \(0.04048\)/hour = \(\$0.57\)
• GB hours:
  • \(21\) tasks x \(2.00\) GB x \(0.67\) hours x \(0.004445\)/GB/hour = \(\$0.13\)
• Ephemeral storage:
  • \(20\) GB (no additional charge)
• Total Fargate cost: \(\$0.70\)/month

Lambda

Assuming the Lambda function is triggered \(21\) times per month, the cost of running the Lambda function falls within the free tier:

• Memory allocation:
  • \(128\) MB (\(0.125\) GB)
• Ephemeral storage:
  • \(512\) MB (\(0.5\) GB)
• Compute time:
  • \(21\) requests x \(2,000\) ms = \(42\) seconds (\(5.25\) GBs)
• Free tier:
  • \(400,000\) GB-s and \(1,000,000\) requests
• Billable GB-s and requests:
  • \(0\) GB-s, \(0\) requests
• Total Lambda cost: \(\$0.00\)/month

S3 and ECR

For this project, the size of the scraped data are in the thousands and the size of the files are in order of KBs. The cost of storing the data in S3 per month is negligible:

• Storage:
  • \(1\) GB x \(0.023 = \$0.02\)
• Total S3 cost: \(\$0.02\)/month

Similarly, the size of the docker image for this project is \(\sim 142\) MB:

• Storage:
  • \(142.16\) MB/month x \(0.0009765625\) GB/MB = \(0.138828125\) GB/month
• Total ECR cost: \(\$0.0139\)/month

EventBridge Scheduler

• Invocations:
  • \(21\) invocations (first \(14,000,000\) free)
• Total EventBridge cost: \(\$0.00\)/month

CloudWatch

Assuming we are only storing logs for the Lambda function and Fargate task, the cost of CloudWatch is very much negligible:

• Total CloudWatch cost: \(\$0.00\)/month

Install AWS CLI

The AWS command line interface is a tool for managing AWS services from the command line. Follow the installation instruction here to install the tool for any given operating system. Verify the installation by running the following commands:

$ which aws
$ aws --version

There are several ways to configure the AWS CLI and credentials in an enterprise setting to enhance security. For this project, however, we will use the long-term credentials approach on our personal machines.

1. Log in to the AWS Management Console:
   • Use the user with administrator access since we need to create a new IAM user.
2. Create a New IAM User With Programmatic Access:
   • Navigate to the IAM console and create a new user with programmatic access.
   • Attach the AdministratorAccess policy (documentation) to this user directly or to the user group to which this user belongs.
   • Securely store the access key id and secret access key provided.
3. Configure the AWS CLI:
   • Run the following command to configure the AWS CLI:

   $ aws configure
   AWS Access Key ID [None]: aws_access_key_id
   AWS Secret Access Key [None]: aws_secret_access_key
   Default region name [None]: aws_region
   Default output format [None]: json

   • Replace aws_access_key_id and aws_secret_access_key with the values provided when creating the IAM user.
4. Note: While this method is sufficient for this project, it is not recommended for enterprise settings. Even for non-enterprise settings, it is advisable to:
   • Rotate the access keys regularly via the IAM console of the user with administrator access.
   • Delete any unused access keys to reduce potential security risks.
   • Avoid adding these keys to any public repositories or share them with unauthorized users.

The steps above ensure that boto3 can interact with AWS services using the configured credentials files.

Run the Python Script

The deploy_stack.py python script requires the following command line arguments:

• --template_file: The path to the CloudFormation template file.
• --parameters_file: The path to the JSON file containing stack parameters.
• --save_outputs: An optional flag to save the stack outputs to a JSON file.

An example parameters file is provided here.

# Activate the virtual environment
$ poetry shell
# Navigate to the project root directory
$ cd path_to_project_root
# Set the PYTHONPATH to the current directory
$ export PYTHONPATH=$PWD

We can then deploy each stack as follows (e.g. using the VPC stack):

# Run the script and save output to json
$ poetry run python3 src/deploy_stack.py --template_file cloudformation_templates/vpc-private.yaml --parameters_file cloudformation_templates/stack_parameters.json --save_outputs

The script outputs will be saved in the outputs directory, assuming the --save_outputs flag is used. These outputs can be utilized to populate the parameters for subsequent stack deployments.

The steps to deploying the resources are the same as those outlined in the AWS Console section above.

1. VPC stack
2. S3 & ECR stack
3. IAM stack
   • Add the secrets to GitHub: AWS_GITHUB_ACTIONS_ROLE_ARN, AWS_REGION, ECR_REPOSITORY
   • Upload environment file and Lambda function code to S3 either manually or via the shell scripts
   • Build and push the Docker image to ECR via the GitHub Actions workflow
4. Lambda & EventBridge stack
   • Add the secrets to GitHub: S3_BUCKET, LAMBDA_FUNCTION
5. ECS Fargate stack
   • Add the environment variables to the Lambda function

Tips for troubleshooting:

• Ensure that the paths to the template and parameters files are correct.
• Verify that the AWS credentials are configured correctly and that the necessary permissions are attached to deploy CloudFormation stacks, i.e., administrator access.
• Check the JSON parameters file stack_parameters.json (link) for any syntax errors or missing values that could affect the deployment.