Attendence System - Genese Software Solution

AWS Lambda Use Case

In this Career Genese project we have used AWS Lambda to run our own code on Amazon’s servers, so we don’t need to host the code by ourselves. The best part of using it as we don’t have to pay a dime when our Lambda function isn’t being triggered and only need to pay for computation time.

AWS Lambda Use Cases in this project:

  • Security Updates
  • Automated Backups
  • Serverless Website
  • File Conversion
  • Face Recognition

Attendance System Architecture Diagram Lambda

Description of type of serverless microservice

As there are different microservice available in AWS, we have used AWS Lambda and API gateway as a serverless microservices. The API of a serverless microservice is the central entry point for all client requests. The application logic hides behind a set of programmatic interfaces, typically at REST web services API. This API accepts and processes calls from clients and might implement functionality such as traffic management, request filtering, routing, caching, and authentication and authorization.
AWS Lambda let us run code without provisioning or managing servers. Lambda is highly integrated with API Gateway. The possibility of making a synchronous call from API Gateway to AWS Lambda enables the creation of fully serverless applications and it is described in our documentation.

Total estimated number of events per month:

AWS Lambda triggers are merely actions caused by specific events that will further trigger the lambda function. Any work that triggers the lambda function is an event. And our Lambda functions have approximately 10 thousand events that generate per month. For example, daily user login, Leave requested by the users, daily attendance in entry and exit time, etc.

Description of any challenge or customer objection overcome to drive a lambda adoption:

When we introduce AWS Lambda to our customer they were quite impressed by the service but they told us that they will not be using this service because Lambda is quite complex for the developers who are not familiar with AWS.

The number of Lambda functions deployed.

In this task, we have a sum of 7 lambda capacities, which continue with various activities.

Following are the various list of our function that are used to create this application :

  • carer_genese_question_collectionLambdaFunction
  • SlackIntegrationLambdaFunction
  • InternAttendance
  • CompanyRequestHandler
  • UploadPicture
  • EmployeeRequestLambdaFunction
  • AdminPreSignUpLambdaFunction

Deployment Patterns: At least 5 functions, Use of best practice deployment application model (SAM)

We had to manage diverse kind of AWS administrations which is recorded beneath:
Lambda function, DynamoDb,API Gateway, Rekognition, Cognito Service, Simple Storage Service, Simple Email Service.Additionally Serverless. Moreover, AWS Serverless Application Model (AWS SAM) is utilized which is an open-source system we have to make utilization of it to manufacture and convey AWS serverless applications. Code Commit, Code Build, Code pipeline are utilized to mechanize programming.

Workload

W

Workload 1: Connection with at least 3 other AWS Services

AWS Lambda functions act as a backend to the application. This application utilizes other AWS services and resources that are organized and manipulated by lambda functions. An application uses DynamoDB tables for storing data and records. Lambda is responsible for reads and writes of data from and to the database.

For storing static files, AWS storage service S3 is used and all the GET and PUT operation is done by Lambda functions. Amazon Rekognition is implemented in Lambda function codes to do image processing tasks. Cognito is used for signup and sign in whereas Amazon SES is used to notify the administrator if an event is triggered in Lambda function.

W

Workload 2: Use of servers or containers only in supporting roles

Since this application is absolutely serverless, all the application rationale live inside lambda improve the performance, minimize the complexity and avoid of using the recursive code of our function. There is no requirement for any servers or holders in any sort of jobs or assignments in the application.

W

Workload 3: Serverless workload should use AWS Lambda to handle and process the core data flow for the system

Every single backend rationale that the application depends on is facilitated in AWS Lambda. All the logic behind DynamoDB table read and writes, S3 GET and PUT, Image processing tasks with Rekognition, Cognito Admin User creation and so on resides inside lambda functions we have used.

W

Workload 4: Automating Infrastructures provisioning and managing is not serverless

AWS Lambda serves client side with appropriate data queries, user registration, and many other application tasks. Automation of Infrastructure provisioning and management is not applicable in this use case.

Solution Characteristics 1: Load/Performance Testing

We utilized a bundle of the hub (npm) named load test to test the API ask for and Lambda Invocations.

  • Install-Package Locally with a command: npm install -g loadtest
  • Run load test without a header:
    Command: ‘loadtest -c 10 –rps 200 https://ginyod5w77.execute-api.ap-south-1.amazonaws.com/prod/openapi?addNewIntern=SendCredentials
  • Run loadtest with HTTP header:
    Command: ‘loadtest -c 10 -H token:eyJraWQiOiI4dEY4a1o3Y01pMjZCenRTenY1dGo5b2RtdWZFSU1RcTZXRml0U251cFRNPSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI0YTE5N2FkNi00MjRhLTQyZTUtYWViYy0yMDMyOTg5Yjk5YTUiLCJhdWQiOiI3aW8wcms2Nzl1Z3Y3cDk2dGRrcGVoYms4ZCIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJldmVudF9pZCI6ImE3NTU3NTg0LWRiNWMtMTFlOC05NDY1LWJmNjBkZGUwZmMzMSIsInRva2VuX3VzZSI6ImlkIiwiYXV0aF90aW1lIjoxNTQwODA1MTYyLCJpc3MiOiJodHRwczpcL1wvY29nbml0by1pZHAuYXAtc291dGgtMS5hbWF6b25hd3MuY29tXC9hcC1zb3V0aC0xXzZ0b2pkYjVtNCIsIm5hbWUiOiJBc2hsZXNoYSBJbnRlcm5hdGlvbmFsIiwiY29nbml0bzp1c2VybmFtZSI6ImFzaGxlc2hhIiwiZXhwIjoxNTQwODA4NzYyLCJpYXQiOjE1NDA4MDUxNjIsImVtYWlsIjoiYWNoYXJ5YWRlbW9jcmF0NTdAZ21haWwuY29tIn0.N6n_Le7MYTP7dF1MEcvbHHLE5R1cNvNxVcaG7mJph2CKog43LgLzMSXsOLbip1m64kbO52ovaQM7ATxDzCnb0GqeYdPxZQ09RumoYCqjka3q2sP3Xh4BqMw_zU2AT-Jj7giblklVmQsaXAETmGV3HjyehK2ssEEIurSaklf1EcLgHD0MHoC6Sp41XqFBvS3u4ANOblVjOiZ6eoGs76Aphm0Db4uQ7oXGdSbB6Xj-BCj6mCZ4dIwGBjw62R4JAMGl3pmLTbyARqCn57dn49s6bV_wSfXcjYbKjJSzeJWoeq39ClKYNUV44VkVc6Wkie_bj2N9Af0ESjNLZ6wqBIRZZg –rps 200 https://ginyod5w77.execute-api.ap-south-1.amazonaws.com/prod/companyrequesthandler?param1=Monthly&param2=October 2018&param3=ashlesha’

 

Solution Characteristics 2: IAM Policy Definition:

In this policy, we have utilized diverse IAM approaches to connect it with the IAM Roles to execute necessary tasks in Lambda functions. Using the permissions policy associated with this role, we can grant our Lambda function the permissions that it needs. Different Policies Used are:

  • AttendanceCognitoPolicy(Inline Policy)
  • AttendaceDynamodbPolicy(Inline Policy)
  • ses-full-access(Inline Policy)
  • AWSS3DFullAccess(AWS Managed Policy)
  • AttendaceRecognitionPolicy(Inline Policy)
  • AttendaceS3Policy(Inline Policy)
  • AttendaceSESPolicy(Inline Policy)

Solution Characteristics 3: Managing Failed Executions:

A Lambda capacity can fall flat for any of the accompanying reasons:

  • The capacity times out while endeavoring to achieve an endpoint.
  • The function fails to successfully parse input data.

We have used python boto3 library in our code. How the exemption is taken care of relies on how the Lambda work was invoked. In this task, some of these event sources are set up to invoke a Lambda function synchronously and others invoke it asynchronously. In specific cases, possibly the info parameters might not be right and code rationale may not consent to the information. In such case, lambda distinguishes the special case and returns a fitting message to the customer side.

Solution Characteristics 4: Management of Multiple Microservice functions

  • The API of a microservice is the central entry point for all client requests. The application logic hides behind a set of programmatic interfaces, typically a RESTful web services API.
  • Since clients of a microservice are served from the closest edge location and get responses either from a cache or a proxy server with optimized connections to the origin, latencies can be significantly reduced. However, microservices running close to each other don’t benefit from a CDN. In some cases, this approach might even add more latency. It is a best practice to implement other caching mechanisms to minimize the latencies.
  • Amazon DynamoDB is used to create a database table that can store and retrieve any amount of data and serve any level of request traffic.
  • The Server Side Logic is written in Lambda functions, which are independent microservice functions of AWS.

Solution Characteristics 5: VPC vs No-VPC and concurrency

Lambda capacities are not put in any VPC’s as the application doesn’t have to interface to any servers that require VPC Secure Environment. Regarding concurrency limits, the default limit of 1000 concurrent executions is enough for a current use case. We might need a limit increase in the future for Higher Performing Microservices.

Solution Characteristics 6: Monitoring of Alarms and Metrics:

  • For observing certain measurements for the majority of our Lambda capacities consequently. We have checked diverse CloudWatch measurements of Lambda by the Name of Function. For every serverless service we run, we care about Errors and Throttles we also want to know if our code is failing for any reason, whether errors in our code or too many concurrent Lambda invocations in our account. We can handle both of these problems with CloudWatch custom metrics. Using the AWS SDK, you can make a put_metric_data() call with a CloudWatch client.
  • Name of Different Metrics that we monitored for our Lambda includes:
    • Invocation per lambda function
    • Duration of execution of lambda function
    • Error in a lambda function
    • Throttles to manage the flow of execution

Dynamo DB Use Case

Application Description:

This Attendance System Project is hosted as a serverless in the AWS platform. We have opted Dynamodb as the data storage because dynamodb is NoSQL database service provided by AWS, and it was quite efficient for the use-case of our client.

The tables we have created during the course of our project development are listed below:

  • Company_details
  • Employee
  • Employee_details
  • Employee_presignup
  • Events_records
  • Leave_requests

1. Description of Primary Key design for the Tables and Indexes:

Table 1: Comany_details

Main Table:

Hash Key: Username

This table stores the admin panel data details for the admin dashboard to the Attendance Web Application. Username has been used as a hash key which is efficient to perform query operation in the main table and also this table has a minimal number of data stored.

Table 2: Employee

Main Table:

Hash Key: Name

Sort Key: Date

This table stores a single information record of a particular user for a particular date. “Name” is used as ‘Hash Key’ and “Date” as ‘Sort Key’, thus, it is efficient to perform query operation in the main table using them.

Global Secondary Index:

Hash Key: Name

Sort Key: monthvalue

Dynamodb data items can be accessed in two ways like :

  • Scan Operation
  • Query Operation

The scan operation on a Dynamodb table with a large number of item count results in ineffective utilization of provisioned throughput where we seek the aid of Global Secondary index to define a new primary key combination of “Name” as Hash Key and “monthvalue” as sort Key. While doing so, the data retrieval of records of a user for a particular month from a query operation is easy and have been efficient;

Table 3: Employee_details

Main Table:

Hash Key: Username

This table stores the details of every user of the application. Each user is made identical with each other with the username attribute. Thus, the best option for a primary hash key is to user username itself.

Table 4: Employee_presignup

Main Table:

All the employee upon their account signup by the admin are primarily stored at this table until the users themselves go through the link and change their password. This table stores data temporarily, thus there is no need to use the hash or any sort key in particular.

Table 5: Event_records

Main table:

Hash Key: companyusername

Sort Key: date

Different Companies have their different Events that take place during their office duration. A particular item in the table includes event records for events of a company for a particular date. Hence, optimum efficiency is achieved by making the “companyusername” as the Hash Key and “date” as Sort Key.

Global Secondary Index:

Hash Key: monthvalue

An application requires frequent queries regarding the events for a particular month, While doing so, the implementation of the “monthvalue” attribute as the primary Hask Key is efficient and quick.

Table 6: leave_requests

Main table:

Hash Key: leaveid

For every leave request, we generate a 32 digit unique UUID to serve as the Primary Hash Key for the table.

Global Secondary Index:

Hash Key:

  • employeeusername (employeeusername-index) 
    The application queries every leave requests for a particular user. So for the smooth and effective query actions, we have set “employeeusername” as the Primary Key.

 

  • Managerusername (managerusername-index) 
    Every new request for leave will be notified to the manager. To query the leave requests from a manager’s perspective, this index takes “managerusername”as the Primary Hash Key.

2. Monitoring and alerting for read and write capacity:

Cloudwatch Alarms are created to notify the system (us) about:

Insufficiency caused in provisioned throughput capacity.

3. Enable Continuous Backup and Data Encryption

Continuous backup, which is enabled in our system, provides point-in-time recovery Backup for a continuous data backup until it is explicitly turned off. By default, dynamodb maintains the continuous backups of the table for the last 35 days. We have enabled the encryption in our database for the security purpose.

4. Workload

The workload for our current system in Dynamodb are as follows:

  1. Write Capacity Units: Range between 20,000 to 25,000 WCU-Hrs
  2. Read Capacity Units: Range between 20,000 to 25,000 RCU-Hrs
  3. Total data storage: less than 5 GB till now
  4. Global deployment: In only one region (ap-south-1)

5. Data Access patterns and Transaction Volumes:

Our application may require to scan or query the table item to get the stored data. Thus the data access patterns primarily depend upon the usage of the indexes upon which the database has been designed. Generally, we have used the Global secondary indexes on which hash keys and the sort keys have been defined.

6. Deployment Pattern:

Since the use of the database is new to any of the system we implement on. The applications use the dynamodb database as their new primary database which is easy to use and apply for its no SQL feature implementation.

7. Description of any challenge or customer objection overcome to drive a DynamoDB adoption:

As this service was quite suitable with the use case of our client project there were not many obstacles created by the client side. The problem only was if the service could work perfectly after the integration with their service.

API Gateway Use Case

Attendance System Architecture Diagram of API Gateway

1. Type of API Gateway solution being delivered:

An application uses Rest API’s created in API Gateway to facilitate client-server communications. Client-side sends a request to the API Endpoint which then triggers related lambda function and returns an appropriate response to the client side.

2. The number of expected requests per second (RPS) in API gateway infrastructure:

We have expected 1 request per second in our API gateway infrastructure.

3. Guidance provided to the customer for effective monitoring solution and cloud watch metrics:

Suggestions to enable the CloudWatch logs and its metrics detailed option to track the information or errors of every API call and view the detailed information about API calls, latency, integration latency, 400 errors, and 500 errors.

4. API Versioning:

Since we needed to keep the development and production environment separate, we used the Stage variables to separate different versions of API.

5. Proficiency:

  • Under the API Gateway, we have used mapping template for handling the request and response from the client.
  • For Authentication, we have used Cognito User Pool Authorizers. Client-side sends a request to the API endpoint with a token from Cognito. API Gateway verifies the token using the authorizer and grants further access to the lambda function.

CloudFront Use Case

Architecture Diagram of CloudFront

TCO Analysis:

TCO analysis helps us to know the cost that we can save by using the use of Amazon Services. As we can see in the figure 73% of a yearly cost has been saved by moving our System in the AWS.

After moving in the AWS we still saw the way to reduce the cost and we moved towards the serverless service of the AWS, through which we successfully saved 140 dollars more by integrating with the serverless of AWS.

Detailed reporting with Cloudwatch Reporting and analytics dashboard:

We have web distribution URL added and then we can see the different metrics like a chart for the total number of requests for all HTTPS code, another chart is about the data transfer which shows bytes downloads and uploads by the user and another chart shows the request by the HTTPS status code.

We have created Cloudwatch Alarms to notify us about CloudFront metric data whenever it exceeded the threshold that we have defined. When Cloudwatch triggers an alarm and it sends a notification to the email list.

Signed URLs Cookies:

The system captures images of employees daily. Those images are displayed in the root account. So to deny public access and allow only one user to access those images, we have configured the Signed URLs for those images.

Performance testing:

Since real user metrics are not available because the application being tested is not receiving real web traffic, we evaluate the performance of our CloudFront distribution by using Synthetic Monitoring approach for performance testing for CDN. For this, we mimic actual web traffic by simulating requests from locations worldwide.

For our application, we use Apache Bench Tool from a command line to test heavy loads on the CloudFront.