Setting Up OTP Deployment Servers on Amazon Web Services (AWS)

This document describes how to:

  1. Create an OTP UI server (AWS S3 and CloudFront)
  2. Create an OTP backend load balancer (AWS EC2)
  3. Delegate access to Data Tools for resources in a third party AWS account via an IAM role

Assumptions for IBI-hosted deployments:

  1. You are logged into an IBI Group AWS environment.
  2. You have selected a region (e.g. US East (N. Virginia)).
  3. A Virtual Private Cloud (VPC) with two subnets exists in that AWS environment.

UI Server: S3 Bucket and CloudFront

The OTP user interface is delivered using a plain HTTP file server that does not perform any computations. The file server consists of one S3 bucket. For fast high-bandwidth file delivery, we mirror the S3 bucket using CloudFront.

Create an S3 Bucket

  1. From AWS S3, click Create Bucket. Each deployment uses its own bucket.
  2. Specify a name (write down the name for use in Data Tools later).
  3. When specifying options, uncheck Block All Public Access. Do not grant additional access from the bucket's Permissions tab.

Create a CloudFront instance

  1. From AWS CloudFront Home, click Create Distribution (direct link).
  2. Select Web Distribution, then click Next.
  3. Under Origin Settings > Origin Domain Name, select the DNS name of the S3 bucket you created above.
  4. Under Default Cache Behavior Settings:
  5. Select Redirect HTTP to HTTPS.
  6. Select Yes to Compress Objects Automatically (this reduces download sizes by up to 60-70%).
  7. Under Distribution Settings, select Custom SSL certificate, and select the *.ibi-transit.com certificate.
  8. Under Distribution Settings, set the Default Root Object value to be index.html.
  9. (Optional) Enter a comment to make the distribution easy to search. Leave other parameters as is, and click Create Distribution.
  10. Open the properties of this CloudFront instance, and copy the Domain Name value (e.g. abcdef0123456789.cloudfront.net).

Create a CNAME (i.e. subdomain) for CloudFront

  1. Go to the AWS hosted zone for ibi-transit.com.
    (It should be under AWS Route 53, Hosted Zones.)
  2. Select Create Record Set. Fill in the subdmain (e.g. otp-mod-ui.ibi-transit.com). The OTP UI will be available at this URL. Set the Record Type to CNAME.
  3. In the value field, paste the Domain Name value of the CloudFront instance above.
  4. Click Create.
  5. Return to AWS CloudFront Home.
  6. Edit the properties of the CloudFront instance you created:
  7. In Alternate Domain Names (CNAMEs), paste the CNAME created above.

Upload files for OTP UI

Upload the files referenced here to the S3 bucket created above.

If updating the UI files, remember to invalidate the CloudFront instance (this forces an update of the files on CloudFront).

Backend server: Load balancer

We recommend using an elastic load balancer (ELB) for deploying/upgrading an OTP server instance without interrupting the current one. In doing so, a new server is set up in the background, and when ready (preprocessing done), the new server is added to the load balancer, and the old server removed and destroyed.

The load balancer also allows instantiating multiple OTP servers on large deployments. (You can start, add, or remove more than one OTP server to the load balancer.)

Create the load balancer in AWS

  1. Go to Create Application Load Balancer
    (Under AWS EC2 Load Balancers view, click Create Load Balancer then Application Load Balancer.)
  2. Enter a name, add a listener for HTTPS (443), pick a VPC with two subnets/availability zones available, and select two of the subnets. (Leave other params as is, the HTTP(80) listener should be there by default). Click Next.
  3. Choose a certificate in ACM, pick the IBI Group certificate. (Leave other params as is). Click Next.
  4. Create a new security group, or use an existing one that supports HTTP, HTTPS, SSH. (Leave other params as is). Click Next.
  5. Create a new target group (pick a name). Click Next.
  6. Do not register any targets. Click Finish.
  7. From the Load Balancer view, select the row corresponding to your new load balancer.
  8. Under the Listeners tab, there should be two listeners, one for HTTP and one for HTTPS. Add the HTTPS listener if it is not there.
  9. Open the load balancer properties, and, under the Description tab, copy the load balancer's DNS name (e.g. ibi-dev-otp-1234567890.us-east-1.elb.amazonaws.com).

Create a CNAME (i.e. subdomain) for the load balancer

  1. Go to the AWS hosted zone for ibi-transit.com.\ (It should be under AWS Route 53, Hosted Zones.)
  2. Select Create Record Set. Fill in the subdmain (e.g. otp-mod-dev.ibi-transit.com). Set the Record Type to CNAME.
  3. In the value field, paste the DNS Name of the load balancer instance above.
  4. Click Create.

Delegate Third Party Account Access

For Data Tools to access AWS resources (e.g., S3 and EC2) in third party AWS accounts, additional setup is required. The steps provided in this AWS tutorial detail the process to delegate access across AWS accounts. A more tailored, shorthand version of this process is provided below, but if more background information is needed about, please follow the tutorial link above.

Steps to Give Data Tools Access to Third Party Account

  1. Log into third party (i.e., other organization's) AWS account.
  2. Open the Roles view in the IAM dashboard and create a new role for Another AWS Account (direct link).
  3. Enter IBI Group account ID (or whichever account Data Tools is running in) and click Next: Permissions. Note: the additional security options (e.g., MFA) can be left unchecked.
  4. Select the appropriate permissions needed by Data Tools for OTP deployment (see below for full sample JSON). These should include:
  5. read/write/list permissions to any S3 buckets needed,
  6. full permissions to EC2, and
  7. IAM permissions in order to read and pass IAM roles to EC2 instances.
  8. Finish creating the role.
  9. Log out of the third party account and into the IBI Group account.
  10. Find the role under which the Data Tools application is running and add the following permission to allow Data Tools to assume the new role (be sure to replace the full ARN with your new role ARN, including the third party account ID):
    {
      "Version": "2012-10-17",
      "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::123456789012:role/your-role-name"
      }
    }
    
    1. You should be ready to deploy OTP servers in the third party AWS account!

Sample Permissions for Third Party AWS Role

A sample JSON string with the permissions needed for the role created in the third party account is reproduced below. Note: this is also the base default permissions needed by any Data Tools instance application running on AWS.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListObjectsInBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::your-bucket"]
    },
    {
      "Sid": "AllObjectActions",
      "Effect": "Allow",
      "Action": "s3:*Object",
      "Resource": ["arn:aws:s3:::your-bucket/*"]
    },
    {
      "Action": "ec2:*",
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:Get*",
        "iam:List*",
        "iam:PassRole"
      ],
      "Resource": "*"
    }
  ]
}