How fast you can export and import 1 million data
Setup New Serverless Project
Before we start you need to have Serverless on your machine. You can install a serverless standalone binary version, or with npm.
MacOS/Linux
Run this command in your terminal:
curl -o- -L https://slss.io/install | bashWindows
Install with Chocolatey:
choco install serverlessVia npm
Note: If you don’t already have Node on your machine, you’ll need to install it first. We suggest using the latest LTS version of NodeJS.
Install the Serverless CLI:
npm install -g serverlessInitial setup
To create a new serverless project, run below command and follow the prompts
serverlessFor more detail visit:
https://www.serverless.com/framework/docs/getting-started/
Database Design
Setup NestJS Project
Run the following command to install NestJS
npm i -g @nestjs/cliTo create a new NestJS project, run following command and follow the prompts
nest new your-project-nameMore details visit: https://docs.nestjs.com
Clone the project
https://github.com/sofyan-ahmad/export-and-import-large-dataset
Testing Method
- max
Download Result
Downloading 1,000
Downloading 100,000
Downloading 1,000,000
Upload Result
Uploading 1,000
Uploading 100,000
Uploading 1,000,000
> Turns out it was an error
Let’s try to upload 500,000
Uploading 500,000
Uploading 250,000
Also error
Set Higher Heap Size 2048
Still Error
Conclusion
Limit memory size v8
