I've observed similar behavior. This seems to happen when your go.mod file includes large packages that need to be fetched during deployment. To streamline deployments make sure, you can vendor your dependencies (go mod tidy, go mod vendor).
Note that if you your deployed files include a go.mod file, the vendor directory will be ignored. If you deploy your functions using the gcloud CLI, create a .gcloudignore file in your project's directory and add go.mod and go.sum to that file.
For example:
.gcloudignore
.gcloudignore
.git
.gitignore
go.mod
go.sum
In my case it reduced deployment time up to 4 times.