K8S Web Operator - Part 1 (Git)

In previous post, I've mentioned a plan to build K8S Based Web Management system with git-based deployment model in place. Something similar to how Heroku works.

If you haven't read about the plan, it's mentioned in the end of this article:

Move from OVH Bare-Metal to Hetzner Cloud

For more than 7 years I was hosting various solutions on the bare-metal servers from OVH - one of the biggest providers in Europe. Recently, I’ve decided to do it more cloud-native way, change current deployment model and run a highly available K8S - on Hetzner Cloud. Why move? Yes,

xor22h.{dev}Julius Lisauskas

K8S Objects

The requested state is stored in a few kubernetes custom objects:

• Customer - Defines customer profile properties like name, email, and authorized ssh-keys
• Webspace - Defines the runtime (php/nodejs/etc..), domain name, replicaCount, and which customer it belongs
• Runtime - Defines a standard Dockerfile template for specific technology stack.

The Operator is responsible for handling these k8s resources and creating required deployments, services & ingress'es. Each webspace also gets a git-repository for editing.

Git Server

The core part of this - is to have a git repository, which are easy to modify and interact from our backend.

I found a nice Go library called wish. It allowed me to create simple SSH based server. Even more - it had a module for git repositories, so starting up the SSH-Based git server is as simple as this:

s, err := wish.NewServer(
  wish.WithAddress(fmt.Sprintf("%s:%d", host, port)),
  wish.WithHostKeyPath(".ssh/git_server_ed25519"),
  wish.WithPasswordAuth(func(ctx ssh.Context, password string) bool {
    return false
  }),
  // allow any SSH key, we will verify it later
  wish.WithPublicKeyAuth(func(ctx ssh.Context, key ssh.PublicKey) bool {
    return true
  }),
  wish.WithMiddleware(gm.Middleware(repoBasePath, a))
)

In my case, auth is handled based on K8S Custom resource. Each customer - can have multiple allowed SSH Keys.

func (a gitAuthHandler) checkAccess(repoName string, pk ssh.PublicKey) (string, gm.AccessLevel) {
	log.Printf("Check acess for repo: %s", repoName)
    
	// first find webspace by git-repo name
	webspace, err := a.apiClient.GetWebspace(repoName)
	if err != nil {
		log.Printf("Failed to get webspace: %v", err)
		return "", gm.NoAccess
	}
    
	// then find the related customer
	customer, err := a.apiClient.GetCustomer(webspace.Spec.Customer)
	if err != nil {
		log.Printf("Failed to get customer: %v", err)
		return webspace.Spec.Customer, gm.NoAccess
	}
    
	// and check all his authorized keys in a loop
	for i, k := range customer.Spec.SSHKeys {
		acceptedKey, _, _, _, _ := ssh.ParseAuthorizedKey([]byte(k))
		if ssh.KeysEqual(pk, acceptedKey) {
			log.Printf("access to repo %v accepted for customer %v with key[%v]", repoName, webspace.Spec.Customer, i)
			return webspace.Spec.Customer, gm.ReadWriteAccess
		}
	}

	log.Printf("Access denied for repo: %v", repoName)
	return "unknown", gm.NoAccess
}

Build on git-push

Pre-receive hook was more complex part. Idea was to build new docker-image on git-push, and deploy it if build succeeds.

A simple bash script was not enough here, so I've created another Go binary which builds the repository. This binary is automatically configured at git-server above as a pre-receive hook.

The flow has two scenarios, one if Dockerfile exists at the root of repository, and one if not. In this case we are generating Dockerfile based on template for specified runtime.

The hook binary connects to remote docker over tcp, builds the image, and pushes it to docker registry.

If everything succeeds - it sends the notification for operator, to deploy the new image, if not - just rejects the push.

Moving forward

Next steps - is to bring support for multiple branches, and allow to have multiple versions of same repo deployed. Something like https://{branch}.{project}.com