D3D11 Create Shader: Encoded Vertex Shader size doesn't match specified size

Question

I am working on a game engine using DirectX 11 and am having trouble getting shaders to encode properly. I am precompiling shaders to .csh files and creating shaders with the byte codes.

I get this error when I try to create any shader, but for this example I will use my PassThrough vertex shader.

D3D11 ERROR: ID3D11Device::CreateVertexShader: Encoded Vertex Shader size doesn't match specified size. [ STATE_CREATION ERROR #166: CREATEVERTEXSHADER_INVALIDSHADERBYTECODE]

The Shader:

#include "../VertexLayouts.hlsli"
// structs in included file
struct PASS_THROUGH_VS
{
    float3 pos : POSITION;
    float2 texCoord : TEXCOORD;
};

struct PASS_THROUGH_PS
{
    float4 pos : SV_POSITION;
    float2 texCoord : TEXCOORD;
};


PASS_THROUGH_PS main( PASS_THROUGH_VS input )
{
    PASS_THROUGH_PS output = (PASS_THROUGH_PS)(0);
    output.pos = float4(input.pos, 1);
    output.texCoord = input.texCoord;
    return output;
}

with these settings: PassThrough_PS properties

In Renderer.h

#include "Vertex Shaders\PassThrough_VS.csh"

In Renderer.cpp

HRESULT hrReturn;

    hrReturn = CreateVertexShader(&PassThrough_VS, sizeof(PassThrough_VS), Pass_Through_VS);
    if (FAILED(hrReturn)) {}
        //return hrReturn;

HRESULT CRenderer::CreateVertexShader(const void* ptrByteCode, SIZE_T szByteCodeLength, eVertexShaderType type)
{
    HRESULT hrReturn;
    tVertShader newShader = {};
    hrReturn = D3Device->CreateVertexShader(&ptrByteCode, szByteCodeLength, nullptr, &(newShader.m_id3dShader)); // WHERE ERROR OCCURS
    if (FAILED(hrReturn))
        return hrReturn;
    newShader.m_ptrByteCode = ptrByteCode;
    newShader.m_szByteCodeLength = szByteCodeLength;
    D3VertexShaders[type] = newShader;
    return hrReturn;
}

Answer 1

The problem is you are passing a pointer-to-a-pointer, so you aren't actually passing the shader blob data to Direct3D.

HRESULT CRenderer::CreateVertexShader(const void* ptrByteCode, SIZE_T szByteCodeLength, eVertexShaderType type)
{
    HRESULT hrReturn;
    tVertShader newShader = {};
    hrReturn = D3Device->CreateVertexShader(&ptrByteCode, szByteCodeLength,
        nullptr, &(newShader.m_id3dShader));
    ...    

The correct code is:

HRESULT CRenderer::CreateVertexShader(const void* ptrByteCode, SIZE_T szByteCodeLength, eVertexShaderType type)
{
    HRESULT hrReturn;
    tVertShader newShader = {};
    hrReturn = D3Device->CreateVertexShader(ptrByteCode, szByteCodeLength,
        nullptr, &(newShader.m_id3dShader));
    ...    

You should take a look at DirectX Tool Kit and the tutorials in particular.

You didn't include any details about the tVertShader type, but you could well have reference counting problems. Consider using Microsoft::WRL::ComPtr instead of raw pointers for managing the lifetimes of Direct3D COM objects. See this article.