The documentations says this:
Iterating over Layers by Index: The IFPLayerManager class is also an
interface class that manages scene layers in 3ds Max. This class
however is a function published interface, and thus is quite different
from the previously discussed layer manager. This class is the C++
interfaces to the MAXScript LayerManager class. This class has a
useful method fro returning a layer by integer index. Thus you can
iterate over scene layers without knowing the names of the layers. To
get a pointer to this object you call the GetCOREInterface() function.
IFPLayerManager* layer_manager = static_cast< IFPLayerManager*> GetCOREInterface( LAYERMANAGER_INTERFACE);
The LAYERMANAGER_INTERFACE identifies the IFPLayerManager interface. With the resulting pointer you can get a scene layer by integer index.
ILayerProperties* hLayer = layer_manager->getLayer(i);
This IFPLayerManager::getLayer() method returns a ILayerProperties
pointer which has much less functionality than ILayer.
To get an ILayer pointer you can get a pointer to any object on that
layer, and extract the ILayer pointer from it.
The ILayerProperties class has functions for child\parent layers:
virtual ILayerProperties* getChildLayerProperties ( int n ) const
pure virtual
Remarks
Returns a pointer to the Nth child Layer properties for this layer.
Parameters:
int n
The index of the child layer properties.
Returns
Returns a pointer to the Nth child layer properties.
virtual ILayerProperties* getParentLayerProperties ( ) const
pure virtual
Remarks Returns the parent Layer. This could be NULL meaning the layer
is at the top level.
Which means you can loop through all by index, then check if it's a parent layer, if it is then loop through all it's child layers, then move on to the next one.
These functions are documented in the 'ILayerProperties Class Reference' in the 3dsmax SDK documentation.