Securing Runtime of the L2 Base Ethereum Nodes
The Problem
So you are running an L2 node and have a lot of money in it. Like most of us, we “hope” it is secure. In this, let’s unpack some of the challenges that I ran into while trying to secure the “Runtime” for L2 Nodes.
I am not an Ethereum or an L2 Node expert. I am a regular guy who wants to run an L2 node and has started questioning the Runtime’s security posture.
I am doing this experiment on k8s.
The JWT Secret
Let’s start with something simple, such as securing the “JWT” used to authenticate the OP Node to the Nethermind API. What can go wrong if that gets compromised? Money can be lost.
So how do we secure the JWT secret?
For starters, we could use something like Vault or AWS Secrets Manager.
The secrets are now loaded into a Volume on the Node(Linux machine).
The goal is to restrict access to op-node, Nethermind, and runc, and nothing else. Not even the root user.
Our requirement is that only these three processes (as of now) should be able to access it. We want guarantees from the kernel with eBPF.
What is runc?
It is the container runtime. It is the process that runs the container. It is the process that starts OP Node and Nethermind. It needs access because it is the process that talks to the kernel and sets up everything. So it needs access to set things up.
How does the secret get written to the file? Are we storing the secrets on the disk? The files are stored on disk using “tmpfs”
What is “tmpfs”?
It is a temporary file system. It is a file system that is stored in memory. It is a file system that is not persisted to the disk. It is a file system that is deleted when the container is deleted. Another way to say it is an in-memory file system.
The secret from the AWS Secrets Manager or Vault is written to the tmpfs by the k8s runtime.
Why kernel?
Following the principles of zero Trust from crypto, how do we know that an elevated privileged process didn’t read the keys? So that’s why you use the bottom of the stack to ensure it hasn’t been tampered with.
Why eBPF?
We don’t want new kernel modules. eBPF is the new cool kid that solves this problem. Now, the next question is: what if another eBPF allows when our program tries to stop? Can’t the malicious user do it? So that they steal the JWT Token. Yes, we tested that, and it’s not possible. You can read more about this at https://substack.bomfather.dev/p/how-we-secured-our-ebpf-from-ebpf.
Our requirements
The security agent (eBPF) running in the kernel should not only protect the JWT Secret, but also protect the executable that has read and write access to it. Why? Because if the attacker can compromise the executable with access, they have circumvented the problem with a Supply Chain Attack.
Because the eBPF agent runs in kernel space, not in user space, even administrators cannot bypass it. It enforces the policies at the kernel level before user-space-level executables can interfere.
We also want to protect against insider threats, such as system administrators not being able to read secrets. Also, what happens if the administrator stops the security agent? Then it’s game over.
We thought about and built a solution so that the agent cannot be killed. The agent can be killed with a PKI and Nonce-based solution. Similar to the platform the crypto is built on (we copied). https://substack.bomfather.dev/p/stopping-kill-signals-against-your
The LSM Hook
eBPF didn’t invent security from scratch. It plugs into Linux Security Modules (LSM), which is the same framework that powers SELinux and AppArmor.
Here’s what that means.
LSM is the kernel’s security checkpoint. Every time any process tries to open a file, read memory, make a network connection, or create a process, the kernel fires an LSM hook. This happens before the operation completes.
eBPF attaches to these hooks. When Nethermind calls open(”/secrets/jwt.hex”), the LSM hook fires, our eBPF program runs, checks the policy, and decides whether to allow or deny the request.
Why this matters:
Runs in kernel space - Can’t be bypassed by userspace tricks
Happens before the syscall - Blocks the operation, doesn’t just log it
Write Access To The JWT Secret
In our example, we used k3d (a simple Kubernetes cluster).
When we ran our tests with Deny all access to the JWT. We realized we need to provide access:
/bin/k3s
/bin/containerd-shim-runc-v2
/bin/runc
OK. Now that we have figured this out, it is much easier to choose which of these executables we need to secure.
But what happens if someone modifies these executables?
Immutable Executables
We want a security policy that prevents k3s, containerd-shim-runc-v2, and runc from being modified. We are trying to do what an immutable kernel would be like, Fedora.
We want to add these executables to the policy, where we can define an immutable executable that no one can modify.
The security agent would ensure that all these executables aren’t modified while the agent is running. This prevents someone from backdooring runc, for example, and performing malicious actions.
GitHub Gist For Security Policy
What We’re Protecting
Now, I know we’ve been focused on the JWT secret, but here’s the thing: this same eBPF-based approach protects way more than just that one file.
What else are we protecting from the kernel?
Cryptographic Keys:
Nethermind keystore (/data/keystore) - Your validator signing keys. If compromised? Sign malicious blocks, steal funds.
Nethermind P2P node key (/data/nodeKey) - Network identity. Compromise this? Eclipse attacks, network manipulation.
OP Node P2P private key (/data/opnode_p2p_priv.txt) - P2P identity theft territory.
OP Node discovery secret (/data/opnode_discovery_secret.txt) - Network manipulation potential.
Application Data:
Blockchain database, state, receipts at /data
Temporary storage at /tmp/nethermind (in-memory tmpfs)
Same principle as the JWT secret:
Only the specific executable that needs access gets access. Everything else? Denied at the kernel level. So when we say “secure the JWT secret,” we’re really talking about an entire security model that protects all your sensitive data from unauthorized access.
But wait. These are container paths, the view from inside the container. In k8s, /data is mounted from somewhere on the host, like /tmp/base-node/nethermind-data. What stops someone from just SSHing into the host and reading that path directly? Nothing. Yet. More on that later.
The Attack Vector You Didn’t See Coming
OK, so we’ve locked down file access at the kernel level. Only Nethermind and op-node can read the JWT secret.
What if I told you that, even though we’ve restricted which executables can access the secret, ensured it is not backdoored, and verified the SHA256 digest, an attacker can still steal it by hijacking the Nethermind or op-node themselves?
How? One word: LD_PRELOAD.
It’s an environment variable that lets you load your own shared library before any other library loads.
You can intercept every single function call that Nethermind makes. Reading a file? Intercepted. Making a network call? Intercepted. That JWT secret Nethermind just read from /secrets/jwt.hex? Yeah, we can grab that too.
But wait, can the eBPF agent block unauthorized file access, right?
Sure does. But here’s the thing: Nethermind is authorized. So when Nethermind reads the JWT secret, the eBPF agent says, “yep, you’re good buddy.” But what if Nethermind isn’t really Nethermind anymore? What if it’s been... augmented?
Think about it. We protected the file. We protected the executable. But did we protect the execution environment?
And if they can, what else can they intercept? Just the JWT secret? Or every cryptographic operation Nethermind performs?
I’ll tell you what keeps me up at night: We spent all this time securing the file, but forgot about the process that reads it.
Do real-world processes really use LD_PRELOAD? You might think I’m making this up. But nope. It’s real. Remember k3d? It uses LD_PRELOAD to instrument the container runtime for:
/bin/containerd-shim-runc-v2
/bin/k3s
/bin/cni
/bin/aux/xtables-nft-multi
I’ve written a complete technical breakdown with working exploit code here: LD_PRELOAD: The Invisible Key Theft. https://substack.bomfather.dev/p/ld_preload-the-invisible-key-theft
How do we defend against this?
We use eBPF to stop this. Though there is a TOCTU Vulnerability in the eBPF solution. We track when a process starts, capture its environment variables, and use that information to apply policy rules, such as which processes are allowed to use LD_PRELOAD.
What We Learned
Security isn’t about protecting a single thing, like JWT keys in AWS Secret Manager.
You secure the file. Attacker targets the executable.
You secure the executable. Attacker hijacks the environment.
You secure the environment. Attacker finds another vector.
That’s why we keep going back to the kernel. eBPF lets us enforce security at every single checkpoint.
Where Do We Go From Here?
We haven’t covered all of the Runtime vectors. But for now, if you are running an L2 node, you should be asking: who has access to the JWT Secret? What’s protecting my executables? Can someone dump the memory, ptrace it, or attach a debugger to change things? More on that later.