Unity Netcode for Gameobjects - client does not move when connected to dedicated server [ServerRPC ClientRPC]

Question

I have setup of two scenes in my project. I want my movement to be server authoritative.

Scene A: Object with SceneManager Object with Preloader script

The preloader script is supposed to start the Netcode instance as server when passed "-server" argument from the commandline, then switching to scene B (Map):

using System;
using Unity.Netcode;
using UnityEngine;
using UnityEngine.SceneManagement;

public class Preloader : MonoBehaviour
{
    private void Start()
    {
        // starting as server
        string[] args = Environment.GetCommandLineArgs();
        for (int i = 0; i < args.Length; i++)
        {
            if (args[i] == "-server")
            {
                NetworkManager.Singleton.StartServer();
                SceneManager.LoadScene("Map");
            }
        }
    }
}

When switched to B scene (Map), the player prefab is supposed to spawn. This is the prefab component setup:

Network Object:

Server Player Movement (script):

using Unity.Netcode;
using UnityEngine;

public class ServerPlayerMovement : NetworkBehaviour
{
    [SerializeField]
    private float speed;
    [SerializeField]
    private CharacterController controller;
    [SerializeField]
    private PlayerInput input;

    void Start()
    {
        input = new PlayerInput();
        input.Enable();
    }

    private void Update()
    {
        // read movement
        Vector2 movement = input.Player.Movement.ReadValue<Vector2>();
        // determine player or server
        if (IsServer && IsLocalPlayer)
        {
            // move if server
            Move(movement);
        }
        else if (IsClient && IsLocalPlayer)
        {
            // move if player
            MoveServerRpc(movement);
        }
    }

    // move player
    private void Move(Vector2 movement)
    {
        Vector3 calculatedMove = movement.x * transform.right + movement.y * transform.forward;
        controller.Move(calculatedMove * speed * Time.deltaTime);
    }

    // request player to move
    [ServerRpc]
    private void MoveServerRpc(Vector2 movement)
    {
        Move(movement);
    }
}

Character Controller:

Network Transform:

I created two builds: Windows desktop 64 bit development build and Windows dedicated server 64 bit development build.

When I start the desktop with "-server" from command line, other desktop build instance can connect to it and move with player.

But when I launch the dedicated server build from command line with "-server", the desktop build can connect but the player can not move.

Is there something I am missing?