Introduction

This is the second part of a two part series on leveraging serverless architecture. In the first part, we built an Extract, Transform, and Load (ETL) pipeline to ingest building permit data from Excel spreadsheets by the City of San Francisco. After applying the raw data to our ETL pipeline, we now have the data available to access within a DynamoDB database. For part two, we will create an Application Program Interface (API) to access and query the data via a Representational State Transfer (REST) API. The REST API will allow authenticated users to access the data over HTTP, which could be leveraged by a web application, mobile applications, command line tools, etc.

 

Requirements
AWS Services

 

To build the REST API, we will leverage DynamoDB, Lambda, API Gateway, Cognito, and SES. The DynamoDB table (permits) created in part one will be the database used to feed our REST API. Lambda functions will be used to query the records from the permits table and be triggered by the API Gateway. The API Gateway will create the HTTP endpoints used to make requests to the REST API and return data based on the request queries. For authentication, Amazon’s Cognito will provide us with user management and the ability to invite users to use the API. Amazon’s Simple Email Service (SES) will be used to send emails to users with the API Keys.

 

setting Up Email

To email users their API keys, we will need to register an email address with Amazon’s SES service so amazon can send an email on our behalf. For this guide, using your personal email will work and you will ONLY be able to send emails to verified accounts, but in a “production” environment, a designated address and service limit increase will need to be requested. Navigate to the SES service in the AWS Console and click Email Addresses on the left side under Identity Management. Then click Verify a New Email Address, enter youremail@example.com in the Email Address field, and click Verify This Email Address.

 

user_64683_58f43744ae719.png

 

Amazon will then send the email address a verification email to allow the SES service send emails on your behalf. Click on the verification link sent by Amazon and the address Status will be verified.

 

user_64683_58f437aa1d5dc.png

 

***NOTE*** you will not be able to send emails to unverified email addresses until you request a service limit increase. From the SES dashboard, click Sending Statistics on the left and click Request a Sending Limit Increase.

user_64683_58f437f4bfeaa.png

 

exercise dataset

For the following guide, we will be using the building permits we loaded into our DynamoDB table permits from the previous guide. The loaded records within the table will allow us to query by partition key (application_number) and/or by attributes associated with the records. The REST API will return subsets of Javascript Object Notation (JSON) data when authenticated queries are requested to the API endpoints.

REST API Planning

The goal of part two is to make the permit data available and queryable to users through a series of REST API endpoints. Authenticated users should be able to find records from the permits table by application_number and find subsets of records based on attribute filters. The portion of our serverless application will need to have publicly accessible HTTP endpoints. Each endpoint will need to authenticate requests to only allow our users access, parse the HTTP requests for query parameters, query the table based on the request parameters, and return the correct subset of data records.

Authentication flow

For this guide, we will create a system to invite users by email and allow the access with API Keys to use the REST API endpoints. This will be a very simple process to get you familiar with Cognito User Pools, sending emails with SES, creating keys with the API Gateway, and using Lambda to interact with these services. We will only be using Cognito to invite users and manage users, but getting familiar with the service will give you an idea of how more robust systems could manage users, groups, and roles.

Creating Users

In this guide, we will not be creating a web page to allow Cognito admins or users to connect to their accounts, but it allows for future development of a web application to manage account information. Go to the Cognito page in the AWS Console and click Manage Your Resource Pools and click Create a User Pool.

user_64683_58f4399649cf2.png

 

The first section is Name, and we will name the new user pool permit-users in the Pool name field and click Step Through Settings.

 

 

user_64683_58f43a0ed788d.png

 

The Attributes section is where to select the required attributes needed for authentication. For this example, we will use the default email attribute and create a custom attribute apiKey to describe a user. Check the email box. Then click Create Custom Attribute, choose Type string, enter Name apiKey, and leave the default Min length and Max length. Finally, click Next step at the bottom of the screen.

 

user_64683_58f43abaaac5a.png

 

Now we will configure how users will sign up and password requirements within the Policies section. Select the radio button Only allow administrators to create users and leave the rest of the parameters to the defaults. Click Next step.

 

user_64683_58f43b441df03.png

 

From the Verifications sections, we will define how users could be verified. Cognito provides the options for Multi-Factor Authentication, email verification, and SMS text message verification. Since this guide will not cover how to allow users to verify their accounts, the only authentication attribute will be email to allow us to automatically verify the invited users, set the Multi-Factor Authentication radio button to Off and check the Email checkbox. The default permitsusers-SMS-Role to send SMS messages will not be created since we do not enable Multi-Factor authentication. Click Next step.

 

user_64683_58f43c6b0cd85.png

 

Next, Message Customization is where you can customize the notifications for email or SMS verification. Since we will automatically verify our invited users, leave the messages to the default values and click Next step.

The next section is Tags and we will not create any tags for this user pool. Click Next step.

The Devices section allows us to remember user devices. Since the REST API will be using bearer tokens in each request, will will select No and click Next step.

user_64683_58f43ce6845e8.png

 

The Apps section allows us to generate keys or authentication methods for third party applications. For the example, we will not have any third party apps, so click Next step.

Triggers section allows you to invoke Lambda functions based on Cognito events like sign-up, authentication, confirmation, etc. We will add a signup trigger later, so do not create a trigger now, and click Next step. Finally, the Review section shows the permits-users user pool configuration. Click Create pool. The user pool is now created and will be applied to our API Gateway.

user_64683_58f43d543fc29.png_800.jpg

 

Now that we have a user pool, we will navigate to the API Gateway in the AWS Console to create a Usage Plan for our API to set rate limits and create keys to apply to the future permits API. Click Usage Plans in the left panel and then click Create. Enter the Name as permits, add a summary within the Description field, set the Rate to 5 requests/second, set the Burst to 10 requests/second, set the Enable Quota to 1000 requests/day, and click Next.

user_64683_58f43de7e0ce2.png

 

The next page is Associated API Stages , but we will not create different stages of our API for this example. Do not add a stage, then click Skip. The proceeding page is the Usage Plan API Keys where keys can be created and displayed. To create API keys, a Cognito trigger will invoke a lambda function when a new user is created. Leave the page blank and click Done.

user_64683_58f43e748585d.png_800.jpg

 

Before we can create any users, a lambda function will auto confirm users, create an API key for them, and then email the key to the new user. Navigate to the Lambda service in the AWS console and click Create a Lambda Function. Then click Blank Function. Next is the Configure Trigger page. Leave the trigger blank since we will assign our Lambda function trigger in the Cognito user pool triggers. Now we will Configure function in the next section. Fill in the field Name as auto-confirm-permit-users, the Description as Autoconfim permit user and send api key, and set the Runtime to Python 2.7.

user_64683_58f43f5cebcad.png

 

Now, we will write the code for the Lambda function code using the Code entry type as Edit code inline. In addition to your verified email address, make sure to get API usage plan ID from the permits Usage Plan we created, and assign them the the variables email_address and usage_plan_id, respectively.

 

import boto3

apigateway = boto3.client(“apigateway”)
ses = boto3.client(“ses”)

## enter ID from permits usage plan
usage_plan_id = “xxxxxx”

## enter your verified email address
email_address = “youremail@example.com”

def lambda_handler(event, context):
username = event[“userName”]
user_email = event[“request”][“userAttributes”][“email”]

api_response = apigateway.create_api_key(
name=username,
description=”API key for permits”,
enabled=True,
generateDistinctId=True)

api_plan_response = apigateway.create_usage_plan_key(
usagePlanId=usage_plan_id,
keyId=api_response[“id”],
keyType=”API_KEY”)

ses.send_email(
Source=email_address,
Destination={“ToAddresses”: [user_email]},
Message={“Subject”: {“Data”: “Your API Key for Permits”},
“Body”: {“Text”: {
“Data”: “Your API Key is {0}”.format(api_response[“value”])
}}})

event[“response”][“autoConfirmUser”] = True
event[“request”][“userAttributes”][“apiKey”] = api_response[“value”]

return event

 

The preceding code will create an API Key for the API Gateway, add the key to the Usage Plan, email the user the key, and then auto confirm the user in Cognito. Scroll down the configuration page, set Existing Role to our serverless-power-user role created in the previous guide, leave the rest of the fields to the defaults, and click Next at the bottom. Review the function configuration and click Create function.

The last step before we can create our first user is to assign the newly created auto-confirm-permit-users function to our Cognito user pool. Navigate to Cognito in the console, select Manage Your User Pools, and choose the permits-users pool. Click Triggers on the left side and select auto-confirm-permit-users from the Pre sign-up dropdown.

user_64683_58f440b9a735e.png

 

To create a user, choose the Users and groups section on the left and click Create User. Enter test_user in the Username field. Uncheck the box for Send an invitation to the new user, uncheck the box Mark phone number as verified, enter a verified email address for testing (Unless sending limit increase has been approved) in the Email field, and click Create user. On create user, the lambda function will be invoked and the API Key will be sent to the user email.

user_64683_58f4412ba5941.png

Creating API Gateway Endpoint

Navigate to the Amazon API Gateway in the AWS Console and click Create API. Select the New API radio button, add permits to the API name, add A REST API for permits data to the Description, and click Create API.

 

user_64683_58f4424bd5bcd.png

 

Let’s create our first API resource! The API Gateway will handle authentication and then invoke a Lambda function to query the permits table. The first resource method will retrieve records by aplication_number through a GET request and the url path will look like /permits/{application_number}. Select Resources under the permits API, then click the Actions dropdown and select Create Resource. Enter the Resource Name as permits and the Resource Path as permits.

user_64683_58f442c7c981a.png_800.jpg

 

Next, select the newly created /permits resource so we can create a child resource that represents the application number parameter. Click the Actions dropdown and then click Create Resource again. This time, the Resource Name will be called application number and the Resource Path will be {proxy+}. Check the Configure as proxy resource and the CORS (Cross Origin Resource Sharing) box, and click Create Resource.

*Note* Using {proxy+} within a resource path allows the service to handle that resource as a parameter.

 

user_64683_58f44391480f0.png

 

Before we can add the GET method to our new resource to retrieve permits by application_number, we will create the Lambda function which will query the table.

Navigate to the Lambda service, click Create a Lambda function, and click Blank function. Fill in the Name as get-permit, enter the Description as Get permit record by application number, and select the Runtime as Python 2.7.

 

user_64683_58f443fa1adcf.png_800.jpg

 

We will leave the Lambda function code as the default and update the code from the command line momentarily. Scroll down the configuration page and set Existing Role to our serverless-power-user role. Update the Advance Settings to 512mb of Memory with a 10 second Timeout to account for possibility of larger queries.

 

Leave the rest of the fields as their defaults, and click Next at the bottom. Review the function configuration and click Create function.

From part one of the guide, we were using pynamodb for a Python interface with Dynamodb. Our models.py constructs a Permits class to interact with the permits table, and its methods can create, load, and query the table. For a quick refresh, here is the models.py file.

 

# models.py

from pynamodb.models import Model
from pynamodb.attributes import (
UnicodeAttribute, NumberAttribute, UTCDateTimeAttribute
)

def PermitsModel(db_region=”, db_host=’http://localhost:8000′):

class PermitClass(Model):
class Meta:
table_name = ‘permits’
read_capacity_units = 5
write_capacity_units = 5
region = db_region
host = db_host

record_id = NumberAttribute(range_key=True)
application_number = UnicodeAttribute(hash_key=True)
status = UnicodeAttribute(default=’issued’)
status_date = UTCDateTimeAttribute(null=True)
file_date = UTCDateTimeAttribute(null=True)
expiration_date = UTCDateTimeAttribute(null=True)
estimated_cost = NumberAttribute(default=0)
revised_cost = NumberAttribute(default=0)
existing_use = UnicodeAttribute(default=”)
proposed_use = UnicodeAttribute(default=”)
description = UnicodeAttribute(default=”)
address = UnicodeAttribute(default=”)
load_date = UTCDateTimeAttribute(null=True)

def __iter__(self):
for name, attr in self._get_attributes().items():
yield name, attr.serialize(getattr(self, name))

return PermitClass

 

Now we will write the function the API invokes when querying a permit by its application number. Our Lambda function is expecting a file named lambda_function.py with the handler function lambda_handler.

 


# lambda_function.py

import json
from models import PermitsModel

Permits = PermitsModel(‘us-west-2’, ‘https://dynamodb.us-west-2.amazonaws.com’)

## A function to format the response
def respond(err, res=None, statusCode=’200′):
return {
‘statusCode’: ‘400’ if err else statusCode,
‘body’: err.message if err else json.dumps(res),
‘headers’: {
‘Content-Type’: ‘application/json’,
},
}

## A function to query permits by partition key (application_number)
def get_items(application_number):
items = []
app_num = ‘#’ + str(application_number)

for item in Permits.query(app_num):
items.append(dict(item))

return items

## Function executed when API is called and returns results
def lambda_handler(event, context):
httpMethod = event[‘requestContext’][‘httpMethod’]
application_number = event[‘pathParameters’][‘proxy’]

## Only allow GET method
if httpMethod != ‘GET’:
return respond(None, {‘message’: httpMethod}, ‘405’)

try:
items = get_items(application_number)
return respond(None, items)
except Exception as e:
return respond(e)

 

We will create a zip file of our code and third party package dependencies to create a deployment package for Lambda to execute.

From the command line, create a new package.zip with our code files.

(venv)$> zip -9 package.zip lambda_function.py models.py

 

Next, we will navigate to our virtualenv site-packages directory and add them to our package.zip.

// cd into virtualenv site-packages directory
(venv)$> cd $VIRTUAL_ENV/lib/python2.7/site-packages

// recursively add all site packages
(venv)$> zip -r9 ../../../../package.zip *

// cd back into the first level of our project directory
(venv)$> cd ../../../../

 

Finally, we will upload our package to S3 and update our Lambda function code

// Copy package to S3
(venv)$> aws s3 cp package.zip \
 s3://serverless-example-record-storage/get-permit-records/package.zip

// Update the lambda function with new package code
(venv)$> aws lambda update-function-code \
–function-name get-permit \
–s3-bucket serverless-example-record-storage
–s3-key get-permit-records/package.zip

 

The function is now up-to-date and ready to be run!

Now lets return to the API Gateway and apply our Lambda function to the resource /permits/{proxy+}. Click on the Any method below the resource path /permits/{proxy+}. Select Lambda Function as the Integration type, choose the Lambda Region where you created the get-permit function, and assign that function to the Lambda Function field. Then click Save.

 

After the method’s creation, click the Any method again to view its Method Execution flow. Now click the Method Request, set the Authorization Settings API Key Required to true, and click the circular checkmark.

 

user_64683_58f4464974c0c.png_800.jpg

 

We will now deploy the API, and then create an API Stage so we can apply our Usage Plan and its associated API Keys to the API deployment and stage.

Click the Resources section under our permits API, click the Action dropdown, and select Deploy API.

 

user_64683_58f446b3a2c44.png

 

Select [New Stage] from the Deployment Stage dropdown. Enter development into the Stage Name and enter permits API development to the Stage description and Deployment description. Click Deploy.

user_64683_58f4470953cc7.png

Return to our permits Usage Plan and click Add API Stage. Select permits under the API field, select development under Stage, and click the circular checkmark to the right. The API is ready to test!

 

Using the REST API

Everything is connected and deployed. The API is ready to query the permits table by application number. We will use curl for the simplest way to start testing the API.

 

// The following request queries an application number from the December 2016 Issued Permits http://sfdbi.org/file/10166
$> curl -X GET --header "x-api-key: <API Key>" <api-gateway-url>/permits/201612013997
//Response
[{
"status": "issued",
"application_number": "#201612013997",
"description": "Replacing aluminum windows with paintable wood windows with acrylic clad. U-FACTOR: 0.32 SHGC: 0.25; SAME SIZE SAME LOCATION.",
"estimated_cost": "9399",
"expiration_date": "2017-06-21T00:00:00.000000+0000",
"status_date": "2017-03-05T02:45:54.883587+0000",
"file_date": "2016-12-01T00:00:00.000000+0000",
"load_date": "2017-02-02T04:33:24.321347+0000",
"revised_cost": "9399",
"proposed_use": "1 family dwelling",
"address": "30800 Bella vista Wy San Francisco, CA",
"record_id": "1556",
"existing_use": "1 family dwelling"
}]
What Else To Do

This is a basic guide for creating a bespoke authenticated REST API. To become better familiar with the services, here are some additional “ToDo” exercises to extend your serverless architecture.

  1. Add GET method to the /permits resource that handles url query string parameters to filter the table by attributes using conditional operators like less than, greater than, between, and equals.
  2. Create API methods to log in to admin and user account
  3. Create API methods for admin to manage users and API keys
  4. Create web pages to log in to accounts and manage keys and users
  5. When admin deletes a user, the user’s API key will be deleted from the API Key group.
Final Thoughts

Creating this simple ETL pipeline and REST API using serverless architecture can be low cost, versatile, and scalable. However, building this via the AWS Console is unsustainable and prone for errors as more complexity is introduced. Don’t worry, though. There are a variety of projects and companies creating tools and frameworks to make your system secure and sustainable across multiple cloud providers. Some notable projects are Serverless, Apex, Chalice, Gordon, ClaudiaJS, etc…

Additional Resources

 

 

Comments are disabled for this guide.