SelectMany when used this way to create mouse movements behaves in confusing ways

Question

I'm trying to create a simple observable that would produce two points after each mouse up and down pair.

//in view's code-behind in constructor:
var mouseDowns = Observable.FromEventPattern<MouseButtonEventHandler, MouseButtonEventArgs>(
                    h => canvas.MouseLeftButtonDown += h, h => canvas.MouseLeftButtonDown -= h)
    .Select(e => e.EventArgs.GetPosition(canvas));
var mouseUps = Observable.FromEventPattern<MouseButtonEventHandler, MouseButtonEventArgs>(
                    h => canvas.MouseLeftButtonUp += h, h => canvas.MouseLeftButtonUp -= h)
    .Select(e => e.EventArgs.GetPosition(canvas));

var mouseMovement = mouseDowns
    .Do(mm => Debug.WriteLine($"downs {mm}"))
    .SelectMany(startPoint => mouseUps
        .Select(e => new { StartPoint = startPoint, EndPoint = e })
);

mouseMovement.Subscribe(mm => Debug.WriteLine($"Start: {mm.StartPoint}, End: {mm.EndPoint}"));

//XAML inside grid: <Canvas x:Name="canvas" Background="Transparent" />

This is the result after 3 drags and drops:

downs 50,60
Start: 50,60, End: 52,223
downs 177,91
Start: 50,60, End: 184,260
Start: 177,91, End: 184,260
downs 338,76
Start: 50,60, End: 357,323
Start: 177,91, End: 357,323
Start: 338,76, End: 357,323

And it grows after each mouse click. Now, what is confusing is that the start point gets multiplied, although it is the mouseUps that is inside the SelectMany. I would expect this to be the other way, with StartPoint being the same, and EndPoint cycling trough the all of history.

Can somebody explain why is this happening this way?

Compare this to duplicated = mouseDowns.SelectMany(e => Observable.Return(e).Repeat(2)) which works just fine (it just duplicates each click), and it does not produce 2 events on first click, 4 events on second, etc. What is the difference between this and my first example? Is it because here, the inner observable terminates, unlike the mouseUps? Or is it because mouseUps is a hot observable (is it?), and this one is not?

The original idea was to create a simple observable from MouseDown and MouseUp events. I know it can be done many other ways, for example, with CombineLatest, or with this:

        var drag = mouseDowns
            .TakeUntil(mouseUps)
            .CombineLatest(mouseUps
                .Take(1), (p1, p2) => new MouseMovement { StartPoint = p1, EndPoint = p2 })
            .Repeat();

So, I'm not looking for a solution, I'm looking for an explanation of why SelectMany behaves the way it does.