I assume you already know about the --skip-pull-image option that allows you to avoid pulling the container image on rebuild.
Otherwise, looking in the source code and the documentation, there does not appear to be any support for not rebuilding the dependencies.
Note that in the DESIGN.md is suggested how a user could perhaps write their own "plugin" to implement a different "workflow":
... each language-specific piece acts almost like a plugin, its its own self contained directory. Someone could develop their own "package" with the structure
This could help to find a workaround.