Good question. In theory, you ought to be able to produce your own raster tileset following the same Terrain RGB format like this:
map.addSource('mapbox-dem', {
'type': 'raster-dem',
'url': 'mapbox://yourusername.yourtilesetid',
'tileSize': 512,
'maxzoom': 14
});
map.setTerrain({ 'source': 'mapbox-dem', 'exaggeration': 1.5 });
However, the documentation does say:
Only supports Mapbox Terrain RGB (mapbox://mapbox.terrain-rgb):
It's unclear if they mean it only supports that format (seems more likely), or is somehow hardcoded to only support data from that exact data source (which seems a very strange choice).
Based on my very basic testing, Mapbox GL JS does attempt to fetch tiles from whatever source you provide, and doesn't give any weird error messages, but I haven't gone as far as actually producing such a tileset.