How to lower the memory usage of my (unorthodox) Qt5/QML-based software?

Question

CONTEXT:

I am developing a research prototype for a novel interaction concept and computational desktop environment, I currently call Sketchable Interaction (SI). Currently, SI works only on Debian-based linuxes. In a nutshell, SI allows users to draw interactive regions on their desktop which carry effects.

Once two or more regions overlap, regions which's effects are compatible to each other apply their effects to each other as well. In this way, graphical representations and data of files, etc. can be set, modified or deleted.

Here are some screenshots to give a visual example:

Showing Desktop Environment:

Drawing region (blue one) for Opening Folders/Files:

Finished drawing of blue region:

Opended Desktop-Folder by overlapping it with the blue region and drew a Preview File region:

Moved image file (png) with the cat out of the folder:

Overlapped image file with the cat with the green region to show a preview of the image:

TECHNICAL STATUS QUO OF SI

SI is written in C++ with the current Qt5 and QML versions. SI-Plugins which represent the effects you saw in the screenshots, are written in python3.7+, with the use of Boost.Python and do not use PyQt5.

SI opens a MainWindow and every region drawing (everything you see in the screenshots is a region, including the mouse cursor) is a QWidget which is a borderless child of that MainWindow.

In order to do any styling e.g. display textures, like the folder icon, SI uses QML files, represented as QQuickWidgets which is a borderless child of that QWidget (I am aware of the stacking order problem, but we can ignore that for this question!)

I am able to change QML styling from within SI-Python-Plugins at runtime. This internally uses QMetaObject to pass a QMap<qstr, QVariant> to a function in the container component.

QMetaObject::invokeMethod(reinterpret_cast<QObject *>(d_view->rootObject()), "updateData", QGenericReturnArgument(), Q_ARG(QVariant, region->data()));

I tested this with signals/slots as well, yet was unable to get it working as I intended, the above method does work as intended. Apparently, this is due to initializing exactly one QQmlEngine, instead of one per QQuickWidget. This single QQmlEngine has CppOwnership.

engine = new QQmlEngine(this);
engine->setObjectOwnership(engine, QQmlEngine::CppOwnership);

THE PROBLEM

For testing purposes and performance benchmarking I intend to spawn thousands of regions: The following screenshots shows 1009 regions (1000 in the center).

This one is with all QML related code deactivated

This yields, according to the tool htop, roughly 200 MB memory consumption.

This one is with all QML related code activated

This yields roughly 4900 MB memory consumption.

The texture used in the yellow regions in the example with QML is a 64x64 px 32-bit RGBA image. This memory difference really strikes me as odd.

The memory required for all images equals 1000 (number of regions) * 64 * 64 (number of pixels) * 4 (number of bytes if 4 channels with 8 bit) = 16,384,000 bytes which are ~16.5 MB. Of course there should be some further overhead per image, yet not 4.8 GB of overhead.

I found out via other questions here or other sources that QML apparently needs a lot memory (some call it a memory hog).

E.g.: QML memory usage on big grid

Yet, this high difference could stem from my unorthodox usage of Qt5 and QML.

QUESTION/S

Is there a way to lower this memory consumption, given the current state of the SI software? Are their alternative approaches I did not come up with? Is their a flag in Qt5/QML docs I missed which trivializes the problem?

Sorry for the lengthy post and thanks in advance for your help.

Edit: Typos, Addition of potential critical or suspected code as requested.

Suspected Code: This is the constructor of a QWidget which contains a QQmlQuickWidget and represents a region

RegionRepresentation::RegionRepresentation(QWidget *parent, QQmlEngine* engine, const std::shared_ptr<Region>& region):
    d_color(QColor(region->color().r, region->color().g, region->color().b, region->color().a)),
    d_qml_path(region->qml_path()),
    d_view(new QQuickWidget(engine, this)),
    d_type(region->type()),
    d_uuid(region->uuid()),
    d_name(region->name())
{
    if(!d_qml_path.empty())
        d_view->setSource(QUrl::fromLocalFile(QString(d_qml_path.c_str())));

    d_view->setGeometry(0, 0, region->aabb()[3].x - region->aabb()[0].x, region->aabb()[1].y - region->aabb()[0].y);
    d_view->setParent(this);
    d_view->setAttribute(Qt::WA_AlwaysStackOnTop);
    d_view->setAttribute(Qt::WA_NoSystemBackground);
    d_view->setClearColor(Qt::transparent);

    setParent(parent);
    setGeometry(region->aabb()[0].x, region->aabb()[0].y, region->aabb()[3].x - region->aabb()[0].x, region->aabb()[1].y - region->aabb()[0].y);

    if(region->effect()->has_data_changed())
        QMetaObject::invokeMethod(reinterpret_cast<QObject *>(d_view->rootObject()), "updateData", QGenericReturnArgument(), Q_ARG(QVariant, region->data()));

    d_fill.moveTo(region->contour()[0].x - region->aabb()[0].x, region->contour()[0].y - region->aabb()[0].y);

    std::for_each(region->contour().begin() + 1, region->contour().end(), [&](auto& point)
    {
        d_fill.lineTo(point.x - region->aabb()[0].x, point.y - region->aabb()[0].y);
    });

    show();
}

I can acess and set data in the QQmlQuickWidget from a plugin (python) in that way:

self.set_QML_data(<key for QMap as str>, <value for key as QVariant>, <datatype constant>)

Every region has such a QMap and when it is updated in any way, this is called by RegionRepresentation:

if(region->effect()->has_data_changed())
        QMetaObject::invokeMethod(reinterpret_cast<QObject *>(d_view->rootObject()), "updateData", QGenericReturnArgument(), Q_ARG(QVariant, region->data()));

Populating the QMap is done in this way:

  QVariant qv;

    switch (type)
    {
        case SI_DATA_TYPE_INT:
            d_data[QString(key.c_str())] = QVariant( bp::extract<int>(value))
            d_data_changed = true;
            break;

        case SI_DATA_TYPE_FLOAT:
            d_data[QString(key.c_str())] = QVariant(bp::extract<float>(value));
            d_data_changed = true;
            break;

        case SI_DATA_TYPE_STRING:
            d_data[QString(key.c_str())] = QVariant(QString(bp::extract<char*>(value)));
            d_data_changed = true;
            break;

        case SI_DATA_TYPE_BOOL:
            d_data[QString(key.c_str())] = QVariant(bp::extract<bool>(value));
            d_data_changed = true;
            break;

        case SI_DATA_TYPE_IMAGE_AS_BYTES:
            int img_width = bp::extract<int>(kwargs["width"]);
            int img_height = bp::extract<int>(kwargs["height"]);

            QImage img(img_width, img_height, QImage::Format::Format_RGBA8888);

            if(!value.is_none())
            {
                const bp::list& bytes = bp::list(value);

                int len = bp::len(bytes);
                uint8_t buf[len];

                for(int i = 0; i < len; ++i)
                    buf[i] = (uint8_t) bp::extract<int>(value[i]);

                img.fromData(buf, len, "PNG");

                d_data[QString(key.c_str())] = QVariant(img);
            }
            else
            {
                d_data[QString(key.c_str())] = QVariant(QImage());
            }
            d_data_changed = true;
            break;
    }

In QML this QMap is used that way:

// data is QMap<QString, QVariant>
function updateData(data)
{
    // assume that data has key "width" assigned from python as shown in above code snippet
    qmlcomponent.width = data.width;
}

Here is the typical layout of QML files which are used for styling regions/effects:

Item
{
    function updateData(data)
    {
        texture.width = data.icon_width;
        texture.height = data.icon_height;
        texture.source = data.img_path;
    }

    id: container
    visible: true

   Item {
       id: iconcontainer
       visible: true

       Image {
           id: texture
           anchors.left: parent.left
           anchors.top: parent.top

           visible: true
       }
    }
}

One of the central ideas is, that users of the system can create custom styling for regions and effect and address that styling dynamically at runtime via the associated plugins.

Answer 1

While this is not an answer to your question, i think it might be a valuable info for you, and since i do not have enough reputation points to comment, i'm posting it as an answer.

The memory issue you are seeing looks like a bug and not Qt/QML related. Below is a simple example of how to display a bunch of images in QML, and what to expect regarding memory consumption.

Code below displaying 1040 images with QML consumes under 30 MB of memory (with 64x64 px 32-bit RGBA source image, but it doesn't change much when using larger images). The displayed images are scaled down to 20x20 px, but even if you had enough of screen real estate to show them as 64x64 px and in worst case scenario if the memory consumption would increase linearly, it should be around 10x increase and nowhere near 4.8 GB. I hope this helps, and this is the code i've used:

import QtQuick 2.12
import QtQuick.Window 2.12
import QtQuick.Layouts 1.3


Window {
    visible: true
    width: 1200
    height: 1000
    color: "#00000000"

    ColumnLayout {
        anchors.fill: parent
        anchors.margins: 20
        Repeater {
            model: 26
            RowLayout {
                Repeater {
                    model: 40
                    Image {
                        Layout.preferredWidth: 20
                        Layout.preferredHeight: 20
                        source: "qrc:/tile.png"
                    }
                }
            }
        }
    }
}

And the memory consumption:

Answer 2

First of all, I would suggest to not ask StackOverflow, but ask a profiler what is using our memory. Try heaptrack for instance.

However, I can tell you that the way you are using QQuickWidget is not as it is designed to use. It seems likely that this is where your memory is being used. I would suggest you change your design to use a single QQuickWidget or even use a QGraphicsArea instead of instantiating a new QQuickWidget per item.

Then on the use of QMetaObject::invokeMethod: please don't. It's an anti-pattern to try to poke into your QML. Instead, expose whatever you want to get into QML as a property or a QAbstractItemModel, and bind to that from your QML.