You can scope the permission to impersonate service accounts at the service account level. If you're using Terraform, the resource would be https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_service_account_iam