Getting ConnectTimeout long after timeout period with async programming

Question

I have a container that runs in an infinite loop and gets messages from SQS and then handles each message in a new coroutine with concurrency of 100 (protected with asyncio semaphore). In the lifetime of a request, i make numerous calls to external api's, using httpx.AsyncClient.

I see a really strange behavior - im performing an API request using the AsyncClient and set global timeout of 90 seconds, however i get ConnectTimeout errors after few minutes (for example after 3 minutes).

What can cause this type of behavior? My concern is that a request that supposed to take less than a minute actually takes few minutes (and fails).

try:
    self._async_client = AsyncClient(base_url=base_url, timeout=90)
    logging.info('starting')
    response = await self._async_client.post(self._SEARCH_ROUTE,
                                             json=request.dict(),
                                             params=self._get_common_params(account_name, request_id))
    logging.info('finished')
except httpx.ConnectTimeout as ex:
    print(f'I get here after 4 minutes')

I locally stress tested the API im calling, doing 3500 concurrent requests. i get low (0-5) number of failed requests with ConnectTimeout issue (but set it to around 20 seconds, which is also awfully alot)