Utilizing modern technologies and advanced database optimization techniques to implement a backend solution with RESTful API.
- Perform an extract, transform, and load (ETL) process for 30+ million records
- Design RESTful API server to handle front end application requests
- Simplify web scaling through Docker containers
- Implement server side caching with Redis to improve latency, and throughput
- Performance testing and fine tuning throughout the development process
npm install
npm start
Ensure that the following modules are installed before running npm install
- Node v10.13.10 or higher
The ETL process was done programmatically, utilizing Node.js file system module and csv-parser module. 6 CSV files containing over 30 million rows of data was buffered, streamed, and transformed in this format.
| Request Type | Endpoint | Returns | Status |
|---|---|---|---|
| GET | /products | An array containing all products by id in ascending order. Number of products returned defaults to 5 or can be modified using the count and page parameters. | 200 |
| GET | /products/:product_id | An object containing full product details of specific product. | 200 |
| GET | /products/:product_id/related | An array of product ids for all related products for a particular product. | 200 |
| GET | /products/:product_id/styles | An object containing the product id and a results property containing all of the styles data for that product. | 200 |
Load testing was conducted with Artillery.io in local development and then Loader.io in the 2nd phase in deployment on AWS EC2 (t2.micro) instances.
The minimum requirements were as follows:
| Stress Load | Latency | Error Rate | |
|---|---|---|---|
| Minimum Requirements | 100 RPS | < 2000ms | < 1% |
Initial testing for all endpoints passed these requirements with the latency measuring between 2-3ms. Latency and throughout remained under 50ms until around 600 requests per second.
After Redis Caching was implemented, server performance increased significantly for all endpoints as shown in the examples below:
| Route | Stress Load | Latency | Error Rate | Success Rate |
|---|---|---|---|---|
| /products/:product_id | 1000 RPS | 1935ms | 0% | 73% |
| Route | Stress Load | Latency | Error Rate | Success Rate |
|---|---|---|---|---|
| /products/:product_id | 1000 RPS | 12ms | 0% | 100% |
Final Results - A Redis cache decreased the latency from 1935ms to 13ms, a 99.4% decrease, with no errors. The Redis cache is a great solution that provides efficient server side optimization through an optimized database caching system optimized for speed.
 Warren Wong |