Accessing AWS SecretsManager from inside a VPC

Question

I have a Lambda function that need to be in a VPC and it needs to access a Secret from SecretsManager. While the code works when the Lambda is outside the VPC it does not work when inside the VPC. The problem is that SecretsManager is not accessible from the VPC. What should be the rules for the Security Group to allow access to SecretsManager ?The current CloudFormation template is the following:

  CrossadVpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 192.168.0.0/21
      EnableDnsHostnames: true
      EnableDnsSupport: true
      InstanceTenancy: default

  VpcEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: true
      SecurityGroupIds:
        - !Ref LambdaSecurityGroup
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.secretsmanager'
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref LambdaSubnet
      VpcId: !Ref CrossadVpc

  LambdaSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: 192.168.0.0/24
      VpcId: !Ref CrossadVpc

  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: LambdaSecurityGroup
      GroupName: LambdaSecurityGroup
      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0
          Description: AllowAllOutbound
          IpProtocol: -1
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          Description: AlllowAllInboundTraffic
          IpProtocol: -1
      VpcId: !Ref CrossadVpc
   
ProxyLambda:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - arm64
      CodeUri: proxy
      Handler: com.github.somegroup.crossad.proxy.Proxy::handleRequest
      MemorySize: 1798
      PackageType: Zip
      Role: !GetAtt LambdaRole.Arn
      Runtime: java17
      Timeout: 600
      Tracing: Active
      Environment:
        Variables:
          ALLOWED_ORIGIN: "*"
          DEFAULT_HOST: !Ref CrossAdHost
          CROSSAD_CREDENTIALS_SECRET_NAME: !Ref CrossAdCredentialsSecretName
      VpcConfig:
        SecurityGroupIds:
          - !GetAtt LambdaSecurityGroup.GroupId
        SubnetIds:
          - !GetAtt LambdaSubnet.SubnetId
      Events:
        GetQuery:
          Type: Api
          Properties:
            Auth:
              Authorizer: CognitoAuth
              Scopes: ["https://crossad.loremipsum.com/scopes/backend"]
            Path: /{proxy+}
            Method: get
            RestApiId: !Ref ApiGateway
        PostQuery:
          Type: Api
          Properties:
            Auth:
              Authorizer: CognitoAuth
              Scopes: [ "https://crossad.loremipsum.com/scopes/backend" ]
            Path: /{proxy+}
            Method: post
            RestApiId: !Ref ApiGateway