{"id":115,"date":"2021-10-04T10:28:12","date_gmt":"2021-10-04T10:28:12","guid":{"rendered":"https:\/\/www.kirazcloud.com\/?p=115"},"modified":"2021-10-06T13:46:56","modified_gmt":"2021-10-06T13:46:56","slug":"cloud-sql-and-gke-connection-with-sql-proxy-and-workload-identitypart1","status":"publish","type":"post","link":"https:\/\/www.kirazcloud.com\/index.php\/2021\/10\/04\/cloud-sql-and-gke-connection-with-sql-proxy-and-workload-identitypart1\/","title":{"rendered":"Cloud Sql and GKE connection"},"content":{"rendered":"\n<p class=\"has-medium-font-size wp-block-paragraph\"><strong>Cloud Sql and GKE connection with Sql Proxy and Workload Identity(Part1)<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To keep db outside a kubernetes cluster has some benefits such as less complexity and building stateless application. To achieve that between your application workload inside the cluster and exterior db we need secure, reliable, modular connection.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In our case study our cluster is in Google GKE and db is Postgres Cloud SQL. To access a Cloud SQL instance from an application running in Google Kubernetes Engine, you can use either the Cloud SQL Proxy (with public or private IP), or connect directly using a private IP address.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The Cloud SQL Proxy is the recommended way to connect to Cloud SQL, even when using private IP. This is because the proxy provides strong encryption and authentication using IAM, which can help keep your database secure.Cloud sql proxy<\/p>\n\n\n\n<h3 class=\"has-small-font-size wp-block-heading\" id=\"Workload-Identity\">Workload Identity<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">With Workload Identity, you can configure a&nbsp;<a href=\"https:\/\/kubernetes.io\/docs\/tasks\/configure-pod-container\/configure-service-account\/\">Kubernetes service account<\/a>&nbsp;to act as a&nbsp;<a href=\"https:\/\/cloud.google.com\/iam\/docs\/understanding-service-accounts\">Google service account<\/a>. Any application running as the Kubernetes service account automatically authenticates as the Google service account when accessing Google Cloud APIs. This enables you to assign fine-grained identity and authorization for applications in your cluster.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To accomplish a secure mapping between Kubernetes service accounts and Google service accounts, Workload Identity introduces the concept of a cluster&#8217;s&nbsp;<em>workload identity pool<\/em>, which allows Identity and Access Management (IAM) to trust and understand Kubernetes service account credentials.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When you\u00a0<a href=\"https:\/\/cloud.google.com\/kubernetes-engine\/docs\/how-to\/workload-identity#enable_on_cluster\">enable Workload Identity<\/a>\u00a0on your GKE cluster, the cluster&#8217;s workload identity pool will be set to\u00a0<code>PROJECT_ID.svc.id.goog<\/code> This lets IAM authenticate Kubernetes service accounts as the following member name:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE\/KSA_NAME]<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">In this member name:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><code>PROJECT_ID.svc.id.goog<\/code>&nbsp;is the workload identity pool set on the cluster.<\/li><li><code>KSA_NAME<\/code>&nbsp;is the name of the Kubernetes service account making the request.<\/li><li><code>K8S_NAMESPACE<\/code>&nbsp;is the Kubernetes namespace where the Kubernetes service account is defined.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">There is only one fixed workload identity pool per Google Cloud project,\u00a0<code>PROJECT_ID.svc.id.goog<\/code>, and it is automatically created for you.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>In addition to enabling Workload Identity at the cluster level,<\/code><br><code>the GKE Metadata Server must be enabled on each node pool<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">To enable Workload Identity on an existing cluster, modify the cluster with the following command:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>gcloud container clusters update bora-playground-301612-dev <\/code><br><code>--workload-pool=bora-playground-301612.svc.id.goog --zone europe-west2-b<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">We need <code>GSA<\/code> and <code>KSA<\/code> will binding each other but here we need to find out which project has own GSA for sql admin , client\u00a0 and editor roles so here is the explanation for GKE compute instance and SQL instance which are existing in a different projects<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>If the Compute Engine instance is in the same project as the Cloud SQL instance, <\/code><br><code>the default service account for the Compute Engine instance has the necessary <\/code><br><code>permissions for authenticating the proxy. If the two instances are in different <\/code><br><code>projects, you must add the Compute Engine instance's service account to the project <\/code><br><code>containing the Cloud SQL instance.<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">and here is the first trying command for project GKE running with <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"mailto:provisioning-first@bora-playground-301612.iam.gserviceaccount.com\">provisioning-first@bora-playground-301612.iam.gserviceaccount.com<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>gcloud iam service-accounts add-iam-policy-binding <\/code><br><code>--role roles\/iam.workloadIdentityUser <\/code><br><code>--member \"serviceAccount:bora-playground-301612.svc.id.goog[keycloak\/keycloak-bora]\" googleborakeycloak@bora-playground-301612.iam.gserviceaccount.com<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">and it should have:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To manage access to a project, you need a role that includes the following permissions:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>resourcemanager.projects.get<\/li><li>resourcemanager.projects.getIamPolicy<\/li><li>resourcemanager.projects.setIamPolicy<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">To gain these permissions while following the principle of least privilege, ask your administrator to grant you one of the following roles:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Project IAM Admin (roles\/resourcemanager.projectIamAdmin)<\/li><li>Security Admin (roles\/iam.securityAdmin)<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">For <code>kunets-internal<\/code> service account<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>gcloud iam service-accounts add-iam-policy-binding <\/code><br><code>--role roles\/iam.workloadIdentityUser <\/code><br><code>--member \"serviceAccount:bora-playground-301612.svc.id.goog[keycloak\/keycloak-bora]\" googleborakeycloak@kunets-internal.iam.gserviceaccount.com --project kunets-internal<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\"><br>Add an annotation to\u00a0<strong>YOUR-KSA-NAME<\/strong>\u00a0to complete the binding:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>kubectl annotate serviceaccount \\ <\/code><br><code>keycloak-bora \\ <\/code><br><code>iam.gke.io\/gcp-service-account=googleborakeycloak@bora-playground-301612.iam.gserviceaccount.com \\ <\/code><br><code>-n keycloak<\/code><\/p><\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"Migrate-applications-to-Workload-Identity\">Migrate applications to Workload Identity<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Select the migration strategy that is ideal for your environment. Node pools can be migrated in place or you can create new node pools with Workload Identity enabled. We recommend creating new node pools if you also need to modify your application to be compatible with this feature. But prefering creating a new node pool is better or even if you create a new cluster if you enable.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"Option-1:-Node-pool-creation-with-Workload-Identity-(recommended)\">Option 1: Node pool creation with Workload Identity (recommended)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Add a new node pool to the cluster with Workload Identity enabled and manually migrate workloads to that pool. This succeeds only if Workload Identity is enabled on the cluster.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>gcloud container node-pools update production <\/code><br><code>--cluster=bora-playground-301612-dev --workload-metadata=GKE_METADATA<\/code><\/p><\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"Option-2:-Node-pool-modification\">Option 2: Node pool modification<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Modify an existing node pool to enable\u00a0GKE_METADATA. This update succeeds only if Workload Identity is enabled on the cluster. It immediately enables Workload Identity for workloads deployed to the node pool. This change will prevent workloads from using the\u00a0<a href=\"https:\/\/cloud.google.com\/compute\/docs\/access\/service-accounts\">Compute Engine service account<\/a>\u00a0and must be carefully rolled out.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>gcloud container node-pools update production <\/code><br><code>--cluster=bora-playground-301612-dev --workload-metadata=GKE_METADATA<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">This action requires&nbsp;container.nodes.update&nbsp;permissions on the project.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"Caution\">Caution<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">I discover some instability when I update existence node-pool when enable <code>GKE_METADATA<\/code> , one of my pod(external-dns) is starting to not reach the google dns zone because of lack of permission or missing credentials, so please check if your default service account is still can be active or not.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>time=\"2021-02-10T21:59:12Z\" level=error msg=\"googleapi: Error 401: Request had <\/code><br><code>invalid authentication credentials. Expected OAuth 2 access token, login cookie <\/code><br><code>or other valid authentication credential. <\/code><br><code>See https:\/\/developers.google.com\/identity\/sign-in\/web\/devconsole-project.\\n <\/code><br><code>More details:\\nReason: authError, Message: Invalid Credentials\\n\"<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">And after some work around how solving this problem, external dns is using default service account which<br>cluster has using so it doesn\u2019t have nodes missing gcp service api authentication, so we have to repeat same process above with separate google service account and kubernetes service account<code>1<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>gcloud iam service-accounts add-iam-policy-binding --role <\/code><br><code>roles\/iam.workloadIdentityUser --member <\/code><br><code>\"serviceAccount:bora-playground-305316.svc.id.goog[ingress-basic\/external-dns]\" <\/code><br><code>external-dns@bora-playground-305316.iam.gserviceaccount.com <\/code><\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>kubectl annotate serviceaccount external-dns <\/code><br><code>iam.gke.io\/gcp-service-account= <\/code><br><code>external-dns@bora-playground-305316.iam.gserviceaccount.com -n ingress-basic<\/code><\/p><\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"Test-the-connections\">Test the connections<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the\u00a0cloud-sdk\u00a0container image, and connecting to it with an interactive session.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>kubectl run -it --image google\/cloud-sdk:slim --serviceaccount keycloak-bora --namespace keycloak workload-identity-test<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">The&nbsp;google\/cloud-sdk&nbsp;image includes the&nbsp;gcloud&nbsp;command-line tool which is a convenient way to consume Google Cloud APIs. It may take some time to download the image.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This action requires create Pods RBAC permission within the namespace.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You are now connected to an interactive shell within the created Pod. Run the following command inside the Pod:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>gcloud auth list<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account&#8217;s authority when calling Google Cloud APIs.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><code>to a Cloud SQL instance on the same VPC using private IP without the proxy.<\/code><\/p><\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Cloud Sql and GKE connection with Sql Proxy and Workload Identity(Part1) To keep db outside a kubernetes cluster has some benefits such as less complexity and building stateless application. To achieve that between your application workload inside the cluster and exterior db we need secure, reliable, modular connection. In our case study our cluster is [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-115","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/posts\/115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/comments?post=115"}],"version-history":[{"count":4,"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/posts\/115\/revisions"}],"predecessor-version":[{"id":163,"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/posts\/115\/revisions\/163"}],"wp:attachment":[{"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/media?parent=115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/categories?post=115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kirazcloud.com\/index.php\/wp-json\/wp\/v2\/tags?post=115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}