AWS Architecture for an IPL Cricket Contest Website
A client wants to build a cricket contest website and you are asked to design an architecture based on AWS.
The client informed that during IPL matches, over a million users will participate in this contest. The website should be highly available and hackers should not be able to access the user data. This website should be geographically accessible only for Indian users. The client has a budget of $10,000 and a time frame of one month.
Let’s look at some prioritization techniques. These techniques will allow us to arrive at a common starting point.
- Iron Triangle
This is really useful when you need to re-analyze the goals of the project. For example, if you want to get something done faster, then it is going to cost more, or you need to drop some features. Let’s assume a new feature of some reporting dashboard is to be added. Then it will affect the other two sides of the triangle — either Cost or Time.
2. Moscow Method
This is a feature prioritization technique used to reach a common starting point.
M — MUST HAVE — Required for Minimum Viable product — For example Zero Downtime.
O
S — SHOULD HAVE — Requirements that are must for a viable product — Encryption.
C — COULD HAVE — Least important — Monitoring.
O
W — WON’T HAVE — These requirements are either dropped or re-considered — Reporting Dashboard.
Architecture
A Cloud Architect comes up with the below design. However, in this design, we are not using any EC2 instances and using most of the AWS managed services. Hence, the management of the OS and other responsibilities are with AWS under the shared responsibility model. However, since we are using Lambda for serving over a million requests, we need to make sure that a pre-warm-up is done before requesting the Lambda function. And we also need to manage scalability and high availability.
The above design uses the AWS Lambda function which is Serverless and requires custom functions written in one’s preferred language as long as it is supported by Lambda.
The ideal architecture for this kind of requirement can be simplified as done below.
Let’s build this on AWS.
AWS Cognito Identity Pool
Amazon Cognito identity pools provide temporary AWS credentials for users who are guests (unauthenticated) and for users who have been authenticated and received a token. An identity pool is a store of user identity data specific to your account. We’ll create an AWS Cognito identity pool.
Identity Pool Name: Enter the name of the identity pool here.
Make sure you select the “Enable Access to unauthenticated identities” option.
IAM Role for Unauthenticated user and Authenticated users should be as shown below:
In the next step, you can select the platform for which you want to use for integration with Identity Pool. I’ve selected JavaScript as a platform.
So now we have our Cognito identity pool Ready, let’s create the Kinesis Firehose Stream.
Step 1:
Enter Delivery Stream Name: IPLCricketContestStream
Select Direct PUT or other sources since we need to send records directly to the delivery stream.
Enable server-side encryption option and use AWS-owned CMK.
Step 2:
Keep the below options disabled as we won’t need these right now.
Step 3:
Choose Destination as Amazon S3 as we need our data to be dumped over in S3.
Choose Bucket:
Step 4:
Set Buffer Size=1MB and keep an interval of 60 seconds so that every 60 seconds, data will be moved to S3. This is basically used for batch processing of data. Let’s say you want to process 20MB of data every 300 seconds. Then it will work accordingly.
Select S3 Compression as GZIP to save data consumption on S3.
Next, select S3 Encryption and choose KMS Master Key.
You also need to create a new IAM role as shown below:
Step 5:
It will show up a review page where you can review and check all settings. If all is good, then click on the Create Delivery Stream button.
Once the stream is created, it will showup on the dashboard as below:
You can also test this stream using Test with demo data button by selecting Stream from the list.
Static Website:
Now let’s create our static website which will use Cognito identity pool to authenticate users for streaming data to the Kinesis Firehose.
// Initialize the Amazon Cognito credentials provider
AWS.config.region = ‘ap-south-1’; // Region
AWS.config.credentials = new AWS.CognitoIdentityCredentials({
IdentityPoolId: ‘ap-south-1:923993223-ed74–4109–9599–6e2aa9be55f8’,
});AWS.config.credentials.get(function(err){
if(err){
alert(“error retrieving credentials from identity pool”);
console.error(err);
return;
}
//for kinesis service object
//dont change api version
var kinesis = new AWS.Kinesis({
apiVersion:’2013–12–02'
});
var firehose = new AWS.Firehose();
submitEntry=document.getElementById(“submitEntry”);
submitEntry.addEventListener(‘click’, function(event){
var params = {
DeliveryStreamName: ‘IPLCricketContestStream’,
Record: {
Data: JSON.stringify({
first_name:document.getElementById(“first_name”).value,
last_name:document.getElementById(“last_name”).value,
phone:document.getElementById(“phone”).value,
email:document.getElementById(“email”).value,
gender:document.getElementById(“gender”).value,
ip_address: document.getElementById(“ip_address”).value,
time: new Date()
})
}
};
firehose.putRecord(params, function(err, data){
if(err)
console.log(err, err.stack);
else{
console.log(data);
alert(“Your entry has been submitted successfully. All the best !!”);
}
});
});
sendBatch = document.getElementById(‘sendBatch’);
sendBatch.addEventListener(‘click’, function(event){
alert(“sendBatch!!”);
var recordData=[];
for(var i=0;i<100;i++)
{
var record ={
Data: JSON.stringify({
first_name:data[i].first_name,
last_name:data[i].last_name,
phone:data[i].phone,
email:data[i].email,
gender:data[i].gender,
ip_address:event.identity.sourceIP
})
};
recordData.push(record);
}
var params = {
DeliveryStreamName: ‘IPLCricketContestStream’,
Records:recordData
};
firehose.putRecordBatch(params, function(err, data) {
if(err)
console.log(err,err.stack);
else{
alert(“Batch 100 records sending complete”);
console.log(data);
}
});
});
});
Key Points:
IdentityPoolId: Replace the value for this with Cognito Identity ID from your Cognito dashboard.
apiVersion: Don't change this value. Keep the default JavaScript value.
AWS.config.region: Region where identity pool is created.
DeliveryStreamName: Delivery Stream name of your Firehose stream.
Please note that the below script needs to be included in your HTML file.
<script src=”https://sdk.amazonaws.com/js/aws-sdk-2.283.1.min.js"/>
The complete code can be downloaded using the link mentioned at the end of this post.
Once our code is configured, you can open index.html in the browser.
Fill in the form and click on Submit button, you should see a confirmation message on your browser.
Now, open up the Kinesis Firehose Cloud watch metrics. You should see some entries for today’s date:
Next, open up the S3 bucket that is configured in Firehose and navigate to the bucket. As you can see in the below screenshot, a new file has been created with .gz extension.
Download this file and open it with Atom.
As you can see in the above screenshot, all entries are correctly appearing which we inserted from HTML in JSON format. So, in this way, we are storing data in S3 using AWS-managed infrastructure.
Later, you can create lifecycle rules in S3 to move data in Glacier which can save significant costs for data storage.
Above is the UML user interaction diagram where multiple services interaction is shown.
Common issues:
You will get the above error if your Cognito unauthenticated user policy is not allowing PutRecord action. In order to resolve this, please edit unauthticated user role in IAM policy as below. Add one more action as:
“firehose:PutRecord” — this will ensure that the user is authorized to perform firehose:PutRecord on Firehose Stream.
I will publish an article that shows how to read data and use SQL queries using Kinesis Analytics.
The complete code can be downloaded using the link below:
If the article has given you some value, or you know someone who might need it, please share it on the Internet. Don’t let him idle on his blog and waste his potential.
I would also be pleased if you write a comment.
Feel free to share your comments and you can contact me at ujjwalsoni85@gmail.com.
More content at plainenglish.io
