The components don't work together because components are overriding one to another: CUI.rte.ui.ToolkitRegistry.register("cui", ExperienceAEM.ToolkitImpl);
To achieve what you want:
Remove the custom EAEMCuiToolbarBuilder and EAEMDialogManager classes, instead extending CUI.rte.ui.cui.CuiToolbarBuilder
and CUI.rte.ui.cui.CuiDialogManager
directly e.g. CUI.rte.ui.cui.CuiDialogManager = new Class({ extend: CUI.rte.ui.cui.CuiDialogManager, ... });
Remove the EAEMToolkitImpl class and the line CUI.rte.ui.ToolkitRegistry.register("cui", EAEMToolkitImpl);
as it seems no longer necessary
Place the colorpicker and image insert plugins into different GROUP names, so that they dont collide when CUI.rte.plugins.PluginRegistry.register(GROUP, EAEMColorPickerPlugin);
is called.
Change calls of this.superClass._getUISettings(options)
and this.superClass.create.call(this, dialogId, config)
to this.inherited(arguments)
Reference: http://experience-aem.blogspot.com.uy/2015/09/aem-61-touch-ui-rich-text-editor-rte-browse-insert-image.html