# [[Lessons from a CI CD Supply Chain Attack at Grafana Labs]]
![[Lessons from a CI CD Supply Chain Attack at Grafana Labs.svg]]
speakers:: "[[Nick Moore]], [[David Andersson]]"
- Nick: Principal Security Engineer, Grafana Detection & Response Engineering (DRE)
- David: Director of Security Engineering
- All GitHub secrets had been exfiltrated by an attacker... but there was no customer impact. Complete CI/CD compromise
- `pull_request` vs `pull_request_target`
- In GH Actions both are hooks that you can use when you create a workflow
- `pull_request` is a safe default. No access to your environment or secrets
- `pull_request_target` gives people access to your secrets. Designed for maintenance
- Problem: When you combine `pull_request_target` with something a user can control, all of a sudden the user's code is forked code-- safe.
- Attacker used something called Gato-X, which scans and tries to exploit vulnerabilities across multiple companies
- Script was added as a branch name. It was executed.
- GItHub gives secrets to any scripts asking for it in your workflow.
- We'd moved some repo secrets to Hashicorp Vault, but not all. Not grafana/grafana
- Security researcher reported something the issue to them FIRST, 7h after the vulnerability. 10h after vuln was the attack.
- Tools Grafana security used:
- IRM
- Loki
- Zizmor: tool to automatically verify whether your workflows are vulnerable
- Trufflehog: Are there any secrets in repositories?
- Gato-X, same thing the attacker uses
- IRM: cornerstone of the response process.
- communication, alerts, keeping all the context together
- Loki
- all key logs from GitHub written to GitHub (because attackers can delete those and logs can also expire) and Loki
- Zizmor
- It flagged `pull_request_target` as something that is almost always used incorrectly
- detects supply chain attacks, injection vulnerabilities, etc
- Now this lives in our CI as a mandatory part of our control
- We also contribute to this now, as an immediate outcome of this incident
- Trufflehog
- Also used by the attacker because it finds secrets and verifies it against live services.
- One of the tokens was a canary, and that's what triggered the alert
- Gato-X
- Also scans for vulnerabilities like Zizmor, but then it also tries to exploit them
- Canary tokens
- You create tokens that can't do anything. They just have access to an empty project.
- But when anyone uses them to authenticate, an alert is sent to the security team. That's distributed throughout the entire infrastructure
- Lessons
- Preparation beats reaction: canaries, static analysis, secret hygiene
- observability is not just for production. It's also for CI/CD
- Move all GitHub secrets to Vault
- Implemented mandatory Zizmor and Trufflehog scans
- Broadened canary token coverage (you can always have more)
- reduced access scope of widely used GitHub apps
- Did user education around GitHub workflow vulnerabilities
- This has happened again! Aqua Trivy and Axios were hacked. We were touched by them but not hit by them. Prep worked!
-
- All GitHub secrets had been exfiltrated by an attacker... but there was no customer impact. Complete CI/CD compromise
- `pull_request` vs `pull_request_target`
- In GH Actions both are hooks that you can use when you create a workflow
- `pull_request` is a safe default. No access to your environment or secrets
- `pull_request_target` gives people access to your secrets. Designed for maintenance
- Problem: When you combine `pull_request_target` with something a user can control, all of a sudden the user's code is forked code-- safe.
- Attacker used something called GatoX, which scans and tries to exploit vulnerabilities across multiple companies
- Script was added as a branch name. It was executed.
- GItHub gives secrets to any scripts asking for it in your workflow.
- We'd moved some repo se
%%
# Excalidraw Data
## Text Elements
## Drawing
```compressed-json
N4KAkARALgngDgUwgLgAQQQDwMYEMA2AlgCYBOuA7hADTgQBuCpAzoQPYB2KqATLZMzYBXUtiRoIACyhQ4zZAHoFAc0JRJQgEYA6bGwC2CgF7N6hbEcK4OCtptbErHALRY8RMpWdx8Q1TdIEfARcZgRmBShcZQUebR44gAYaOiCEfQQOKGZuAG0AXX4IXDg4AGUoqHFUUDBIdXTqiCJlaRS6hkIECgAhXGwAa2VSYQ5iAGE2fDZSbggAYgAzZZX2
yGwRQKyASSr9CpGBhEnp2Yl5gEYEK6u1iA3SLahd9L7B4dGJqZm5qHIOZhwXBPO4PJ4vfQAMUI+HwFRgwTmgg8oM2mWeewObCOAHUSOpuHxwOs0TtMX9sQh4YiJMiSKjHuiIQAlYStDjhHJoC78EmMsnpADyQOwahg3AuiUSvPupIx6UhnCgkNw+hh4rQAFYZWCmXtFVkyoQjNUeNLibL+fL9AAVLBQACCLS4EmCiygDPB5OBjsebAokhCxG4HCE
sJ1cohAFExg6/QGg3NgSMqBGrRC4ymbfAmiMhGNPXqFeQMqyaahQ+GLcxsCNYQANbgAZgALHEeC3EgBOLsAdibFxbLc1A5lNbr+AAmtwAGxdxLaAAcTZnbc1vcXFx42otRjYBm4tQ69AIQmqF2JAF80170qz88QOcwueg8wWZcMSEaTYTzR1P8QFQIHA3A7v+pAkAAsmwxAIDGuCaMEwZoIsBBhB+EFfKch7EpAPRTMhr7KJouAABQ8BcvbULwlH
URRVGoAumoAJRrJAzIIMoYbAnMpDEWRPBNtKvBCXRomMdoLEQFeN7oliRzClA7AAiGYb4DKiwlggHFjEwhAcMoOF1JAmQIUh3B/GevLrEQIFoJZCDWRAHCqtUDlOcIUBEByFmkFZuEQPowJHKQAByrm+f5xmBcFTDwYhCCEQ5MkWnYABWCDYNkZQuXA0GwfF5koWhjkWv0SmMDa+74EZHSwIgSJpFlylsbKzBQAY2YNWglbqQF0yDIl3Cofg6EBZ
pBhlE1SmcMNJVOfgoSOs1lXVapsIpR0jjMGZxxKvakGZEIc2jaVHSaPpjgGQACoEixMJkYhzKZCXBjKhDMD0l36cohVDfZflnZAH2QSQcBsPpUA5SUcA/X9SWAzKmhsJgU3BM1nD5U0ehZLg+nSWA150IsMLhIel4gJeQA==
```
%%