What is best way to use wavefront material file color parameter Kd to set vertex color in MetalKit submeshes?

Question

I am developing a MacOS program using MacOS MetalKit and ModelIO. The ultimate goal is to use the 3D model to analyze the acoustic properties of a performance space. Initially I'm just creating a visual image of the space (stage, walls, seating, etc). I create a wavefront file (.obj and accompanying .mtl file).

I can define the vertex color for every vertex in the .obj file. But I would like to use the Kd properties of the mtl file to set the color so that specific colors are associated with specific named materials in the mtl file.

I use ModelIO to create an asset and then extract both the modelio mesh and metal mesh. Here is the bit of Swift code:

// Extract both the MetalKit meshes and the original ModelIO meshes
    var meshes: (modelIOMeshes: [MDLMesh], metalKitMeshes: [MTKMesh])
    meshes = try MTKMesh.newMeshes(asset: asset, device: device)

I see all of the mtl properties for all of the 147 submeshes in the imported modelio data.

If I specify the vertex colors in the obj file then I see all of those colors in the 147 metalKitMeshes submeshes. But if I do not specify all of the vertex colors in the obj file then all colors in the metalKitMeshes submeshes are Black (0,0,0). The colors specified in the mtl file are ignored.

The primary question: Is there a way to use the material colors in the mtl file to automatically set the metal submesh colors?

Secondary question: In the more general case, what is the best way to convey all of the material parameters to the shader?

Thanks

I asked this on the Apple Developer forum, but got no response.

I can import my wavefront file into Blender and all of the objects in the scene are rendered with the correct colors from the materials file. So the structures of my .obj and .mtl files seem to be correct.

warrenm warrenm · Accepted Answer · 2018-12-19T05:45:06

This is an interesting question because it highlights the impedance mismatches between Model I/O and MetalKit. In particular, it illustrates how MetalKit handles materials, which is to say: not at all.

In the interest of brevity, I'm going to assume that the only material property you care about is the base color (what the MTL specification calls diffuse reflectivity, corresponding to the Kd keyword), and that every material in the .mtl file is uniquely named and has a Kd property.

Our task is three-fold. First, we need to extract the material properties we care about from the MDLSubmeshes contained in the asset. Second, we need to associate these materials with their corresponding MTKSubmeshes so we know which material to use when drawing. Third, we need to adapt our shader to accept this material information on a per-submesh basis rather than per-vertex basis.

Since our MTKMesh doesn't have enough context to know the materials associated with its corresponding MDLMesh, we apply the fundamental theorem of software engineering and devise our our mesh type that has an array of materials, one for each material in the MDLMesh, provided the corresponding submesh has a material with a diffuse color:

struct MyMesh {
    var mtkMesh: MTKMesh
    var materials = [Material?]()
}

For now, our material struct is nothing but a wrapper around our base color:

struct Material {
    var baseColor: float3
}

When loading the meshes from our asset, we need to iterate over all submeshes in each mesh and build a collection of our meshes, each with its array of materials:

var myMeshes = [MyMesh]()
for (mdlMesh, mtkMesh) in zip(mdlMeshes, mtkMeshes) {
    guard let mdlSubmeshes = mdlMesh.submeshes as? [MDLSubmesh] else { continue }
    var materials = [Material?]()
    for mdlSubmesh in mdlSubmeshes {
        if let mdlMaterial = mdlSubmesh.material, let mdlBaseColor = mdlMaterial.property(with: .baseColor) {
            let baseColor = mdlBaseColor.float3Value
            let material = Material(baseColor: baseColor)
            materials.append(material)
        } else {
            materials.append(nil)
        }
    }
    let myMesh = MyMesh(mtkMesh: mtkMesh, materials: materials)
    myMeshes.append(myMesh)
}

When drawing, we enumerate over the submeshes, looking up the submesh's material and binding it to a buffer argument:

for (idx, submesh) in mesh.mtkMesh.submeshes.enumerated() {

    if var material = mesh.materials[idx] {
        renderEncoder.setVertexBytes(&material, length: MemoryLayout<Material>.size, index: 3)
    }

    // ... draw ...

Finally, over in our shader, we create a corresponding material struct:

struct Material {
    packed_float3 baseColor;
};

and accept a parameter of this material type, passing along the material's base color as our vertex color for further shading:

vertex VertexOut vertexShader(VertexIn in [[stage_in]],
                               constant InstanceConstants &instanceConstants [[buffer(2)]],
                               constant Material &material [[buffer(3)]])
{
    VertexOut out;
    out.position = instanceConstants.modelViewProjectionMatrix * float4(in.position, 1);
    out.color = float4(material.baseColor, 1);
    out.texCoord = in.texCoord;
    return out;
}

Note that I've chosen arbitrary binding points for the arguments; you can use whichever slots aren't being used by your existing vertex attributes and buffer parameters, as longs as the indices in your Swift code and your shaders agree.

Also note that I'm being really cavalier with struct layout. If Swift decides that it doesn't want to lay out the Material struct such that its address can be type-punned to a pointer to packed_float3, you'll get weird behavior. This is especially something to be aware of as you add additional material properties.

What is best way to use wavefront material file color parameter Kd to set vertex color in MetalKit submeshes?

1 Answers