Assimp: Manipulate bones of rigged mesh manually

Question

I am working on a project which has following goal:

Load rigged 3D mesh (e.g. a human skeleton) with Assimp.NET
Manipulate bones of mesh so it fits your own body (with Microsoft Kinect v2)
Perform vertex skinning

Loading the rigged mesh and extracting bone information works (hopefully) without any problems (based on this tutorial: http://www.richardssoftware.net/2013/10/skinned-models-in-directx-11-with.html). Each bone (class "ModelBone") consists of following information:

Assimp.Matrix4x4 LocalTransform
Assimp.Matrix4x4 GlobalTransform
Assimp.Matrix4x4 Offset

LocalTransform is directly extracted from assimp node (node.Transform).

GlobalTransform includes own LocalTransform and all parent's LocalTransform (see code snipped calculateGlobalTransformation()).

Offset is directly extracted from assimp bone (bone.OffsetMatrix).

At the moment I don't have GPU vertex skinning implemented, but I iterate over each vertex and manipulate it's position and normal vector.

        foreach (Vertex vertex in this.Vertices)
        {
            Vector3D newPosition = new Vector3D();
            Vector3D newNormal = new Vector3D();

            for (int i=0; i < vertex.boneIndices.Length; i++)
            {
                int boneIndex = vertex.boneIndices[i];
                float boneWeight = vertex.boneWeights[i];

                ModelBone bone = this.BoneHierarchy.Bones[boneIndex];

                Matrix4x4 finalTransform = bone.GlobalTransform * bone.Offset;

                // Calculate new vertex position and normal
                newPosition += boneWeight * (finalTransform * vertex.originalPosition);
                newNormal += boneWeight * (finalTransform * vertex.originalNormal);
            }

            // Apply new vertex position and normal
            vertex.position = newPosition;
            vertex.normal = newNormal;
        }

Like I already said, I want to manipulate bones with a Kinect v2 sensor, so I won't have to use animations (e.g. interpolating keyframes, ...)! But for the beginning I want to be able to manipulate bones manually (e.g. rotate torso of mesh by 90 degrees). Therefore I create a 4x4 rotation matrix (90 degrees around x-axis) by calling Assimp.Matrix4x4.FromRotationX(1.5708f);. Then I replace the bone's LocalTransform with this rotation matrix:

Assimp.Matrix4x4 rotation = Assimp.Matrix4x4.FromRotationX(1.5708f);
bone.LocalTransform = rotation;

UpdateTransformations(bone);

After the bone manipulation I use following code to calculate the new GlobalTransform of the bone and it's child bones:

    public void UpdateTransformations(ModelBone bone)
    {
        this.calculateGlobalTransformation(bone);

        foreach (var child in bone.Children)
        {
            UpdateTransformations(child);
        }
    }

    private void calculateGlobalTransformation(ModelBone bone)
    {
        // Global transformation includes own local transformation ...
        bone.GlobalTransform = bone.LocalTransform;

        ModelBone parent = bone.Parent;

        while (parent != null)
        {
            // ... and all local transformations of the parent bones (recursively)
            bone.GlobalTransform = parent.LocalTransform * bone.GlobalTransform;
            parent = parent.Parent;
        }
    }

This approach results in this image. The transformation seems to be applied correctly to all child bones, but the manipulated bone rotates around the world space origin and not around its own local space :( I already tried to include the GlobalTransform translation (last row of GlobalTransform) into the rotation matrix before set it as the LocalTransform, but without success...

I hope somebody can help me with this problem!

Thanks in advance!

wombat wombat · Accepted Answer · 2016-02-03T17:34:56

Finally I found the solution :) All calculations were correct except:

Matrix4x4 finalTransform = bone.GlobalTransform * bone.Offset;

The correct calculation for me is:

Matrix4x4 finalTransform = bone.GlobalTransform * bone.Offset;
finalTransform.transpose();

So it seems to be a row-major / column-major problem. My final CPU vertex skinning code is:

    public void PerformSmoothVertexSkinning()
    {
        // Precompute final transformation matrix for each bone
        List<Matrix4x4> FinalTransforms = new List<Matrix4x4>();

        foreach (ModelBone bone in this.BoneHierarchy.Bones)
        {
            // Multiplying a vector (e.g. vertex position/normal) by finalTransform will (from right to left):
            //      1. transform the vector from mesh space to bone space (by bone.Offset)
            //      2. transform the vector from bone space to world space (by bone.GlobalTransform)
            Matrix4x4 finalTransform = bone.GlobalTransform * bone.Offset;
            finalTransform.Transpose();

            FinalTransforms.Add(finalTransform);
        }

        foreach (Submesh submesh in this.Submeshes)
        {
            foreach (Vertex vertex in submesh.Vertices)
            {
                Vector3D newPosition = new Vector3D();
                Vector3D newNormal = new Vector3D();

                for (int i = 0; i < vertex.BoneIndices.Length; i++)
                {
                    int boneIndex = vertex.BoneIndices[i];
                    float boneWeight = vertex.BoneWeights[i];

                    // Get final transformation matrix to transform each vertex position
                    Matrix4x4 finalVertexTransform = FinalTransforms[boneIndex];

                    // Get final transformation matrix to transform each vertex normal (has to be inverted and transposed!)
                    Matrix4x4 finalNormalTransform = FinalTransforms[boneIndex];
                    finalNormalTransform.Inverse();
                    finalNormalTransform.Transpose();

                    // Calculate new vertex position and normal (average of influencing bones)
                    // Formula: newPosition += boneWeight * (finalVertexTransform * vertex.OriginalPosition);
                    //                      += boneWeight * (bone.GlobalTransform * bone.Offset * vertex.OriginalPosition);
                    // From right to left:
                    //      1. Transform vertex position from mesh space to bone space (by bone.Offset)
                    //      2. Transform vertex position from bone space to world space (by bone.GlobalTransform)
                    //      3. Apply bone weight
                    newPosition += boneWeight * (finalVertexTransform * vertex.OriginalPosition);
                    newNormal += boneWeight * (finalNormalTransform * vertex.OriginalNormal);
                }

                // Apply new vertex position and normal
                vertex.Position = newPosition;
                vertex.Normal = newNormal;
            }
        }
    }

Hopefully this thread is helpful for other people. Thanks for your help Sergey!

Assimp: Manipulate bones of rigged mesh manually

2 Answers