# [[Spice Runner]] ![[Spice Runner.svg]] [site](https://nvdh.dev/spice) Spice Runner is a simple game I created for my presentation [[The Spice Must Flow - The Fremen Guide to Sustainable Observability]] at Øredev in Malmö, Sweden, in 2025. The app was primarily built to showcase ways to observe and reduce the environmental cost of apps running on [[Kubernetes]]. [Here is more information](https://nicolevanderhoeven.com/blog/20251105-the-spice-must-flow/) about the presentation itself. On this page, I try to summarize how I put together the app and what I learned from it. ## About the app ### Architecture #### App stack - `runner.js` contains the game logic (vanilla [[JavaScript]] application, no framework used) - [[Nginx]] is used as a web server #### Leaderboard stack - `leaderboard-client.js` captures the player score and name - `leaderboard.html` is the web page to display the high scores and names - `leaderboard-api` is a [[Go|Golang]] API service - Scores are stored in [[PostgreSQL]] and cached with [[Redis]] #### Observability stack - Frontend - [[Grafana Faro]] for [[Real User Monitoring|RUM]] metrics - Backend - Nginx exporter to collect Nginx specific metrics - [[Grafana Alloy|Alloy]] as a [[Telemetry collector]] - [[Grafana Loki|Loki]] for logs - [[Grafana Tempo|Tempo]] for traces - [[Prometheus]] for metrics - [[Grafana]] for visualization of telemetry - Testing - [[k6 (tool)|k6]] - Deployment - [[Kubernetes]] - [[KEDA]] for horizontal scaling - [[Google Cloud Platform|GCP]] for hosting and vertical scaling ### Code The code was forked from [this repo](https://github.com/wayou/t-rex-runner) containing the original T-rex themed Chromium game. ## What I learned ### [[Grafana Faro]] This was the first time I used Faro in a demo app, and I was able to use it to instrument things that occurred on the frontend, such as: - number of times the user presses jump - game start, stop, and durations - score achieved in game - obstacles encountered Faro has an SDK that I imported, and then I still wrote custom code to instrument the events above that I wanted to be recorded and reported. ### Debugging frontend I ran into an issue where when playing the game myself, I would "die" but not visibly see an obstacle that had killed my character. To debug this, I learned to: - increase logging - temporarily add a box around every obstacle for visual debugging, which still yielded results faster than just logging. Sometimes it's just easier to play through it yourself and make it very obvious as to what's happening. - Add collision boxes to highlight areas where the game believed there had been a collision between my character and an obstacle. ### Persistent Volumes [[Persistent Volume Claim]]s were necessary because in the process of debugging and building the app, I had to restart the entire stack multiple times. When I did that, I realised that all the telemetry was also wiped. To combat that, I created PVCs for Loki, Prometheus, and Tempo so that the data survives restarts. Persistent volumes are [[Block storage]] attached like a disk, and even if pods are restarted, the new ones can be reattached to the volumes, picking up where the old ones left off. [[How to create a persistent volume in Kubernetes]] ### Environmental sustainability ## Resources %% # Excalidraw Data ## Text Elements ## Embedded Files 7d2bf8bdef0e4531cfa9a52da04e955a9d521cab: [[The Spice Must Flow - The Fremen Guide to Sustainable Observability.svg]] ## Drawing ```compressed-json N4KAkARALgngDgUwgLgAQQQDwMYEMA2AlgCYBOuA7hADTgQBuCpAzoQPYB2KqATLZMzYBXUtiRoIACyhQ4zZAHoFAc0JRJQgEYA6bGwC2CgF7N6hbEcK4OCtptbErHALRY8RMpWdx8Q1TdIEfARcZgRmBShcZQUebQBGAE5tAAYaOiCEfQQOKGZuAG1wMFAwMogSbggAdgBmAE1iABEjegBJdLLIWEQqwn1opH5yzG5nHgBWFOHIGDHaxJTtABZl gA4JnmrtlPXa+JmIChJ1bgA2M9rtRJ4UtcS15drqicPJBEJlaW4Js7XtDbxNb3eLxKY8A7FSDWZTBbjTKEQZhQUhsADWCAAwmx8GxSFUUdZmHBcIFcp1yppcNg0cpUUIOMRsbj8RJCRxiaSclAKZAAGaEfD4ADKsDhEkk1I0gV5SJR6IQAHUTpJuJCunLURjRTBxehBB5ZfSvhxwvk0OrymwSdg1HMLSkERr6YzTcxzagOEIhYcwghiNxlikzpND owWOwuBa1mGmKxOAA5ThiNUTeK1Hi1NYpDOHIRwYi4KD+tXVM5lnPxFKJeI8ZaHQjMJqZYsBtB8ghhQ504RwNrED0FAC6h00wkZAFFgtlcoOR4iiBw0dwvT6F2waSX252EIc+eRsv2V978G9QgAVLBQAAyhCX3A7+C7iIFwSPEmqxB4mj5a00xAQPkUgQZYJn2bAO0SXBNkLXYEESCYJlwRJiE2eI8E0WVmHccRUCKLowEtMp4ihecNWwVE4GPNc NTvADRgkXA0mKABfcAyIgXA4DgUUi1w0punebIqiIL4eWGBhCAQCgACFqVpF0mRxPEqgAYj5DTNIpCBsBEMkoDaYt9FFLUsWU1l0FU+IEGs6ztN00h9MMrI5JpHsGSUlkCXIDkSX0+y9O5Zz9AAMUFEUxVwpEcUqCSHKcoyTIVZViFONA+GKHTAtyYKku1SKqgNWLMvioKjIAJWEE0zTVOLsoMoyAHkbTtNVHTqxyyqyELOCgELcH0QV7VQesSvq 4KetyYVCCMXDbg6hKskvTAoAAQVEqN0GCPlxLGzqcsSqJSDWxy2Aod5cDbT0TwWrr9AnRlVtO86QiuzjnoC/aGqyJ7UQoc94CixTPsW0KDwQSq9VXU9Muw1EhQADW4bZkimM54jOUCpmqNZawkuGcXwepAymbREOBZ57mDUNMqMNgDG4ATIHoAghFwkiyhY26DqySqPLdD0IGBiS6RIabZu4ebMtF4hRQQKi0FeaXSBIABZNgAIe3BNGCK7H2fco ZeZFS0CZiAZJxN7SGUKkAAoIWqaheHiR3nddpYJgASllcqEGUb1SSqa27czaZeFqMPQ6dj3vdY7moDyhBmqgSMPWhiT9wGiGcgAlWOGURnERybXde4FE2cObAiAV1By93REOCzsvSArxFhCgRdcLrw4BlIDFSATJu0G7xFe/7rWda3WuW4QOPMrsAArBBsDyYVG7gdXNcbye9Z3CTqRTxhz3p/BC41HoorCYIV8jWVdORAwAd6NB0/XTdd6feuNX 3AxhUyG/OAPj3guUIa0b5HxPtRfAc9yiOGYCXMyuQryqxyEIIBn9wBczoK+cIjM2IsSAA=== ``` %%