How to filter an annotated queryset by - Window function - without changing the value of the annotated field

Question

I have a queryset of users, after annotating the rank of each user using Django Window function, I want to query for a user without modifying the rank value

users_points_query = users.order_by(
            '-total_points'
        ).annotate(
            rank=Window(
                expression=Rank(),
                order_by=[
                    F('total_points').desc()
                ],
            )
        )

this works fine, but when filtering on users_points_query query, the rank is calculated again, so the first user will get a rank of 1, which is based on the first row and so on.