Self-hosted Bluesky PDS on Kubernetes

2024-12-01

Intro

A few months ago, I created a simple Helm chart to deploy a Bluesky PDS on Kubernetes. I wanted to extend that a bit and show how it can be deployed.

What’s a PDS?

A PDS or Personal Data Server is a component of the AT Protocol network. It’s where your user data resides on the AT Protocol. Due to the open and federated nature of the AT Protocol, it’s possible to self-host a PDS on your own server and use Bluesky with it.

Requirements

Before deploying our PDS, we will need the following :

Domain name
SMTP Relay service. (e.g. resend works well.)
- Not mandatory to use the PDS, but you’ll need it to verify user accounts.
Kubernetes cluster with :
- Ingress NGINX controller (Should™️ also work with a different Ingress Controller)
- cert-manager with HTTP01 and DNS01 challenge solvers configured
- Long-term storage
Bluesky PDS Helm chart

Note that I purely focus on the PDS deployment. The elements above will not be covered in details. For this experiment, I have deployed Ingress NGINX with a default configuration. The DNS01 challenge solver with cert-manager might be tricky depending on your registrar.

Let’s deploy our PDS

If you take a look at the default values, you will notice there are sensitive values that need to be passed to the chart. As we are good kids, we will pre-create a secret with those values and reference it with the existingSecret.

 1...
 2pds:
 3  config:
 4    hostname: "pds.lab.nerkho.ch"
 5    secrets:
 6      # -- Output of `openssl rand --hex 16`
 7      jwtSecret: ""
 8      # -- Output of `openssl ecparam --name secp256k1 --genkey --noout --outform DER | tail --bytes=+8 | head --bytes=32 | xxd --plain --cols 32`
 9      adminPassword: ""
10      # -- Output of `openssl rand --hex 16`
11      plcRotationKey: ""
12      # -- Example: `smtps://user:password@smtp.example.com:465/`
13      emailSmtpUrl: ""
14      # -- Set the value for existingSecret to use a pre-created secret
15      #existingSecret : ""
16...

We generate each value as indicated (save those somewhere safe) :

1openssl rand --hex 16
2<jwtSecret>
3
4openssl ecparam --name secp256k1 --genkey --noout --outform DER | tail --bytes=+8 | head --bytes=32 | xxd --plain --cols 32
5<plcRotationKey>
6
7openssl rand --hex 16
8<adminPassword>

And we create the secret :

1kubectl create secret generic --namespace app-bluesky bluesky-secret \
2    --from-literal=adminPassword=<adminPassword> \
3    --from-literal=jwtSecret=<jwtSecret> \
4    --from-literal=plcRotationKey=<plcRotationKey>
5    --from-literal=emailSmtpUrl=smtps://resend:<your api key here>@smtp.resend.com:465/

Now, we can reference that secrets in our value file.

1...
2pds:
3  config:
4    hostname: "pds.lab.nerkho.ch"
5    secrets:
6      existingSecret : "bluesky-secret"
7...

Additionally, we need to set the pdsEmailFromAddress value. This is the email address that will be used by the PDS for verification and such.

1pds:
2  config:
3    hostname: "pds.lab.nerkho.ch"
4    secrets:
5      existingSecret: bluesky-secret
6    pdsEmailFromAddress: "pds@nerkho.ch"

For the storage, we probably want to define our storageClass. I’m using Longhorn, so I’ll set it here.

1...
2pds:
3  config:
4    hostname: "pds.lab.nerkho.ch"
5    secrets:
6      existingSecret : "bluesky-secret"
7    dataStorage:
8        storageClass: "longhorn"
9...

We aim to add 2 hosts. One for the PDS and a wildcard for each user. This is required as on the AT Protocol, each user is kind of its own website with its own subdomain.

We also need to add the wildcard domain to our TLS hosts. As mentioned earlier, this will require a DNS01 challenge solvers to be configured on cert-manager.

If this is not possible in your scenario, you could just list each subdomain for each user of your PDS instead of using a wildcard.

 1    ingress:
 2      enabled: true
 3      className: "nginx"
 4      annotations:
 5        cert-manager.io/cluster-issuer: letsencrypt-prod
 6        kubernetes.io/tls-acme: "true"
 7      hosts:
 8        - host: pds.lab.nerkho.ch
 9          paths:
10            - path: /
11              pathType: Prefix
12        - host: "*.pds.lab.nerkho.ch"
13          paths:
14            - path: /
15              pathType: Prefix
16      tls:
17        - secretName: bluesky-pds-tls
18          hosts:
19            - pds.lab.nerkho.ch
20            - "*.pds.lab.nerkho.ch"

With this, we can deploy the PDS. I, personally, use Flux CD on my cluster, but straight helm install ... will work fine as well. If everything goes as planned, you should be able to reach your PDS with curl and see something like this :

1curl https://pds.lab.nerkho.ch/
2This is an AT Protocol Personal Data Server (PDS): https://github.com/bluesky-social/atproto
3
4Most API routes are under /xrpc/⏎

We did not cover it, but I also recommend you to configure the podSecurityContext and securityContext. Commenting out the default value should work just fine.

Create a user and start using Bluesky

We can now create our first user. The first step is to create an invite code.

Note: The pdsadmin scripts expect that you are on the same machine as your PDS. I chose to run the commands myself. Another option is to use this pdsadmin go binary from @lhaig.haigmail.com

1export PDS_ADMIN_PASSWORD=<adminPassword>
2export PDS_HOSTNAME=pds.lab.nerkho.ch
3
4curl --silent --show-error --request POST --header "Content-Type: application/json" "$@" \
5    --user "admin:${PDS_ADMIN_PASSWORD}" \
6    --data '{"useCount": 1}' \
7    "https://${PDS_HOSTNAME}/xrpc/com.atproto.server.createInviteCode" | jq --raw-output '.code'
8
9<invite-code>

With the invite code, we can now create the user. We need to create an input JSON file with the user data.

1{
2  "email": "<username@example.com>",
3  "handle": "<username>.pds.lab.nerkho.ch",
4  "password": "<user-password>",
5  "inviteCode": "<invite-code>"
6}

With this, we can create the user. If this works fine, you will get JSON output indicating the user has been successfully created.

1curl --silent --show-error --request POST --header "Content-Type: application/json" \
2    -d @input.json \
3    "https://${PDS_HOSTNAME}/xrpc/com.atproto.server.createAccount" | jq

At this point, we can now login on https://bsky.app with our new user. We need to specify the URL of our PDS in the first field and login with the defined above.

bluesky-login

Et voilà! You are now using Bluesky with a self-hosted PDS. If you’d prefer to use a different domain than your PDS for your handle, you can do so and set a custom domain as your handle.

Troubleshooting invalid handle warning

If your account shows a warning with invalid handle you can debug that with the Bluesky Handle Debug Page. This should help to find where the issue lies.

bluesky-login

You don’t need both method DNS and HTTP. Just one suffice.

If it’s green there, often times, just updating the handle in the Account settings help.

To go further

Currently, I only use my PDS for experimenting. If you plan to go productive with it, you will probably want to look into monitoring.

For this, I recommend taking a look at this video from @justingarrison.com where he shows how to run a PDS on an RPI and also touches about monitoring.

#kubernetes #pds #bluesky #atproto #self-host

Reply to this post by email ↪