Wondering if you can securely connect from Gitlab runner to GCP without generating service account keys? In this short post I’ll try to show you how to do it
All methods I found for now about connecting from Gitlab runner to gcloud via CLI or API, included the step of generating Service Account key and using it in job through CI/CD variables. For a few months, we are able to use Workload Identity Federation for making this connection more secure.
Workload Identity Federation
As you might know (or not?) generating service account keys is not a very secure practice if keys are not rotated. If a key like that is leaked you will need a block service account which may cause even production environments to stop working and then do rotation of keys.
Recently you might have heard about the leak of over 1 billion records of private citizens data because a key was leaked into a blogpost.
Workload Identity Federation (WIF) basically allows you to make secure connections between Gitlab (or other supported providers e.g. through OIDC) and Google Cloud.
Instead of generating access tokens from Service Account key, a short lived JWT token is used. This helps to increase security because even if a token is leaked it can’t be used in a short moment to get access to resources.
The same for generated json files for Workload Identity Federation. File can be leaked as it is not storing any secure information by itself. Without an option to sign a request to STS (Security Token Service) on Google Cloud, an attacker cannot acquire JWT token to access resources on your project.
From where Google Cloud STS knows that request to it is valid and signed? Gitlab sign request with its keys and it can be validated using published Gitlab OIDC configuration there https://gitlab.com/.well-known/openid-configuration.
Of course we should remember to use least privilege security principle and Service Account which we connect to WIF should have only required IAM permissions.
Modified graphic from blog post
Feature described below was working with Github as it has support for OIDC tokens. Gitlab enabled this feature too and using CI_JOB_JWT_V2
you can leverage Workload Identity Federation for secure connection.
The script
Script below is based on the GCP Blogpost tutorial on how to connect to Github.
Important step
Important change to blogpost code, is to allow Gitlab to be used as an audience in the Workload Identity Federation. Now you can set this too by hand when creating Pool from Google Cloud Console.
Important Step
Very important is to remember to set the audience to limit access to Identity Pool only to for instance given Gitlab project ID. It can be email or any other parameter from request. To do that, set the env variable and run the script. SUB=/attribute.project_id/29450825
#First, set PROJECT_ID example export PROJECT_ID=my-project
#Second, set so called SUB to make this connection work only with given Gitlab ID project.
#Example of sub export SUB=/attribute.project_id/29450825
#Last step, set Service Account name which will be used as a connection from Gitlab. export SA_NAME="my-service-account"
if [ -z ${PROJECT_ID} ];
then echo ">> PROJECT_ID not set"; exit 1
else echo "PROJECT_ID set";
fi
if [ -z ${SUB} ];
then echo ">> SUB not set, setting to *"; SUB="/*" ; exit 1
else echo "SUB set";
fi
if [ -z ${SA_NAME} ];
then echo ">> Service account name (SA_NAME) not set"; exit 1
else echo "SA_NAME set";
fi
SERVICE_ACCOUNT_NAME="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
WORKLOAD_IDENTITY_POOL_ID="gitlab-identity-pool"
gcloud config set project ${PROJECT_ID}
echo ">> Creating Workload Identity Pool..."
gcloud iam workload-identity-pools create "gitlab-identity-pool" \
--project="${PROJECT_ID}" \
--location="global" \
--display-name="Gitlab identity pool"
echo ">> Done. Creating OIDC provider in pool..."
gcloud iam workload-identity-pools providers create-oidc "gitlab" \
--project="${PROJECT_ID}" \
--location="global" \
--workload-identity-pool="${WORKLOAD_IDENTITY_POOL_ID}" \
--display-name="Gitlab OIDC provider" \
--attribute-mapping="google.subject=assertion.sub,attribute.aud=assertion.aud,attribute.env=assertion.environment,attribute.project_id=assertion.project_id" \
--issuer-uri="https://gitlab.com"
echo ">> Done. Enabling gitlab as audience..."
gcloud iam workload-identity-pools providers update-oidc gitlab --allowed-audiences=https://gitlab.com --location global --workload-identity-pool $WORKLOAD_IDENTITY_POOL_ID
echo ">> Done. Setting service account permissions..."
gcloud iam service-accounts add-iam-policy-binding $SERVICE_ACCOUNT_NAME \
--project="${PROJECT_ID}" \
--role="roles/iam.workloadIdentityUser" \
--member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WORKLOAD_IDENTITY_POOL_ID}${SUB}"
echo ">> Done. Generating json configuration file..."
gcloud iam workload-identity-pools create-cred-config \
projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$WORKLOAD_IDENTITY_POOL_ID/providers/gitlab \
--service-account=$SERVICE_ACCOUNT_NAME \
--output-file=gitlab_token.json \
--credential-source-file=gitlab_token \
--credential-source-type=text
echo ">> Done. Use below configuration file in Gitlab project"
cat gitlab_token.json
You’ll get a file named gitlab_token.json which you can safely use and share in your project CI/CD variables. Example usage with gcloud CLI is
#If gitlab_token set as variable in Gitlab
- echo $CICD_SA_JSON > ${CI_PROJECT_DIR}/service_key.json
- export GOOGLE_APPLICATION_CREDENTIALS=${CI_PROJECT_DIR}/service_key.json
- echo $CI_JOB_JWT_V2 > gitlab_token
- gcloud auth login --cred-file=${CI_PROJECT_DIR}/service_key.json --activate --project $PROJECT_ID
#If token set as File tyle in Gitlab
- export GOOGLE_APPLICATION_CREDENTIALS=${CICD_SA_JSON}
- echo $CI_JOB_JWT_V2 > gitlab_token
- gcloud auth login --cred-file=${CICD_SA_JSON} --activate --project $PROJECT_ID
Job script can be different from case to case so feel free to ask me in comments or reach me on Twitter with questions.
I hope this will help you to make your pipelines safer. If you have any comments or improvement ideas, please share it :)