I have a nodejs application which adds a raw file in an S3 bucket and also processes it. These two actions are independent but used to run in series. In order to improve the performance, I wanted to have these run in parallel (process the file while the call to add the raw file to S3 is pending). After making the change (call S3 upload method before processing logic but await the result after the processing logic), it seems that the S3's underlying http PUT call is not sent until the process logic is done. I am not sure why that is. We use AWS Node sdk V2.
Pseudo code for when the two ran in series:
const s3Response = await s3Client.upload(params).promise()
// start of fetcher.process_page trace
const processResults = processFile(rawFile)
// end of fetcher.process_page trace
Before- S3 upload and processing happen in series
Pseudo code for when the two ran in parallel:
const s3Promise = s3Client.upload(params).promise()
// start of fetcher.process_page trace
const processResults = processFile(rawFile)
// end of fetcher.process_page trace
const s3Response = await s3Promise
After making the change- processing and S3 upload happen in parallel
