Sandboxing is great, but sandboxing is a pain to use… The issue is that there is almost always a time when multiple executables have to share a machine. When there are too many moving parts, we need to change the way we approach security.

Our solution is kind of an anti-sandbox: everything can access everything, except for a couple of protected resources.

How it works

I think the easiest way to explain this is with an example.

So, in this example, we have a couple of executables, the server, test.sh, and support.sh.

The server accesses Postgres, which in turn accesses the DB.

test.sh can access the GPU and push data to the server.

And support.sh is a helper for test.sh.

Note that in this example, the server accesses Postgres over IP and test.sh accesses the server over IP.

Using VMs

We can start by trying to secure this with VMs to sandbox the executables.

So, the server would get its own VM.

Postgres and the DB would get their own VM.

And, the test.sh, the GPU, and the support.sh would all be in a VM.

Each VM would access the others by poking a hole in its firewall and allowing IP access.

The issue is that the way this is implemented would violate the whole reason of using VMs. Since test.sh needs to access support.sh, both executables are running in a single VM.

Having two executables in a single VM negates the entire reason for using VMs in the first place, since it no longer provides sandboxing.

One real world example of this is Postgres itself. When it runs in the background, it mostly has a script that launches Postgres as a daemon on your system. This is just an example, but you will always have executables on your system that interact with each other (that’s what makes it a system).

Our Solution

Our solution is to protect only a few specific resources rather than protecting everything.

So, in this example, the DB and the GPU would be protected resources.

Then only Postgres can access the DB, and Postgres can only be accessed by the server.

The GPU can be accessed only by test.sh, and test.sh can access only the servers IP endpoint.

And, support.sh can access anyone since it can’t really access any protected resources.

Conclusion

In the end, sandboxing is great, but as I said before, it’s a pain. That’s why we came up with our current solution.

If you want to do sandboxing, go for it! We just didn’t want to deal with the issues with sandboxing.

What is the point of software supply chain security? Software supply chain security primarily focuses on ensuring we upgrade our dependencies to versions that have been patched for known vulnerabilities.

If a malicious actor uses a vulnerability that affects us, they can do three things. One, they can exfiltrate secrets, user data, and proprietary data. Two, they can encrypt your databases (etc) and ransom them back to you. Third, they could inject a malicious script into your code.

The issue with software supply chain security is that we might miss a vulnerability or two. A vulnerability might not be reported, or we might be a bit slow on implementing a patch/dependency update. That is why we created Bomfather as a last line of defense.

Up until now, Bomfather has had solutions for malicious actors encrypting your databases to ransom or injecting malicious scripts into your code, but we didn’t know how to deal with a malicious actor exfiltrating data. That is why we have implemented our IP protection.

The Solution

Imagine that our secrets are stored in a database and that every executable except for a couple of trusted ones is malicious. How can we make sure that only our trusted executables can access the external database?

To solve this, we decided to implement IP protection. This limits which IPs/DNSs an executable can access, and which executables can access an IP/DNS. If a malicious actor can’t access your DB, then they can’t steal your secrets.

All security provided by Bomfather is implemented using eBPF. We secure IPs by hooking into lsm/socket_connect. We restrict both IPs and DNS addresses. For a specific executable, we can restrict what IPs it can access. And we can stop every other executable from accessing the IP.

The Difference between this and a Firewall

Traditional firewalls usually enforce traffic rules at the host or network boundary. If multiple executables run on the same host, they often share the same network policy. Instead, this solution is restricted to the executable level.

An Example of How this Works

We have been talking very theoretically, so this might be easier to understand with an example.

Imagine that we have a KMS, and only a signer should be able to access it. We can attach two restrictions to the signer. Only the signer can access the KMS. And, the signer isn’t allowed to access any IP other than the KMS.

This works by allowing the signer to access the KMS and ensuring that no one else can.

We can take this example up a notch. If a malicious process modifies the signer then it can access the KMS, and exfiltrate data. To solve this, we can use fs-verity to sign the binary and only allow the unmodified signer to access the KMS (Read a bit more about this in our blog post about securing builds with fs-verity, https://open.substack.com/pub/bomfather/p/how-we-secure-builds-with-fs-verity?utm_campaign=post-expanded-share&utm_medium=web).

Conclusion

In the end, this is a huge step forward in supply chain security since there is always the chance we miss a vulnerability, and if we do so, we need a way to make sure it can’t steal anything valuable!

With our builds:

• If you unplug the network cable from our build VM, the build still finishes.

• If someone modifies the Go compiler, the kernel refuses to execute it.

• If a process outside the approved build chain attempts to read or write to protected build paths, a security eBPF program blocks the bad action at the kernel level.

We aren’t saying this is the only way to build software safely (it may be overkill). What we are saying is this is the level of control we wanted and needed.

Why We Went This Far

We have read Ken Thompson’s paper “Reflections on Trusting Trust” many times and understand the importance of the problem it describes.

We wanted to know how to verify that the exact toolchain used to produce a kernel-facing artifact was based on the source code that was committed. How do we verify that the toolchain was not modified between installation and use?

That led us to GCP VMs with custom images. We made sure not to use hosted runners since the most basic step in this journey was controlling the kernel, the boot chain, and all the tools used to create the artifact.

Why Not Github Actions

First of all, they are stuffed with bloatware, which is really nice when you don’t want to install dependencies manually, but really, really bad when you want any level of security. https://substack.bomfather.dev/p/githubs-ubuntu-latest-runners-have

Furthermore, we needed to pin and boot specific kernels (5.18 and 6.18), set boot flags such as lsm=bpf, enable fs-verity on the root filesystem, and manage the lifecycle of signing material during image construction.

All of this is awkward or impossible on generic hosted runners, where the kernel and boot chain are not under your control.

Because of this, we run on custom VMs built from Packer images, ensuring everything remains under our control.

What The Pipeline Actually Does

It has three stages, each with very different costs. First, we prebake a hardened image using packer with a pinned kernel and a signed toolchain with fs-verity enabled. Then we vendor dependencies with go work vendor. Finally, we build and test in a short-lived VM with GOPROXY=off (which tells go not to pull in dependencies or upgrades), and we are currently working on turning off internet access on the entire build machine.

Why We Use fs-verity Instead of Just Checksums

Checksums give us confidence that the binary is good at a certain point in time. With checksums, we can verify what we download. What it does not tell us is whether the binary we downloaded is the same as the one we execute at runtime.

fs-verity gives us that confidence, since it enforces that any binary that it monitors cannot be opened (and therefore executed) if the hash changes from boot. This is guaranteed by the Linux kernel and ensures that binaries are not modified after download and before execution.

This means that if a bad actor compromises your Golang executable, after your image is created, it can’t execute, and the whole build would fail.

What We Sign

We sign binaries that can influence build outputs, including the go toolchain internals, clang/LLVM, some supporting binutils, the protobuf toolchain, and other core build utilities.

This sounds really good, but in theory, it really isn’t. Whenever we want to upgrade anything, we need to do it manually. We need to rebuild our images, sign them, validate them, test that our builds pass, and ensure none of our security steps fail. While GitHub Actions may not be secure, it’s much easier to upgrade dependencies, utilities, and other components.

Dogfooding our eBPF Agent, by making it secure our Builds

This is the part we care about most, since running our security agent against our builds lets us constantly test and verify all our features in the real world.

We use the same eBPF LSM (Linux Security Modules) policy agent we ship to secure its own build. We have a couple of polices that allow us to significantly bump our security posture. The primary goal of our agents is to enforce expected behavior, so that bad binaries and processes cannot access resources they shouldn’t.

• We first secure our own agent and make sure that nobody can tamper with its maps or where it stores its policies (https://substack.bomfather.dev/p/attacking-and-securing-ebpf-maps). We also make sure that nobody can shut down our agent, even if they have root privileges (https://substack.bomfather.dev/p/stopping-kill-signals-against-your-agent)

• We restrict access to the source code directory so that only one binary is allowed to write to it, and another is allowed only to read from it. These are programs with simple, limited behaviors, so they can’t be compromised to do something bad. The best way to build a secure system is to use dumb, single-purpose components.

Our Next Steps to Integrate fs-verity with our eBPF Agent

fs-verity is a great tool that allows us to guarantee that the binaries we download are not tampered with, but we can do even more with it.

We are currently working on integrating fs-verity with our eBPF agent https://docs.ebpf.io/linux/kfuncs/bpf_get_fsverity_digest/. This would allow us to avoid relying on the filesystem for our policies and instead use hashes to verify them.

So if we had a policy that said “/app/server can access /app/db/databaseconnection.yaml”, now we could associate a hash with the server, allowing us to really confirm that our agent thinks that /app/server is the same /app/server we are talking about. Since Linux files can be mounted, moved, and hidden in bizarre ways, being able to associate a hash with them allows us to verify that we are granting access to legitimate processes.

Cost

Doing this is not fun or easy. Every single upgrade we have to do takes a really long time, and sometimes our upgrades break because a security policy triggers when we haven’t done every step right.

Frankly, if we were shipping a normal SaaS backend, this would be overkill.

But we aren’t. We are shipping kernel-level security software, and we’d rather spend extra build time than even risk a compromise.

But, it has an issue… Research has shown that Confidential Computing adds a significant performance overhead on GPUs.

This post walks through these overheads and tries to explain them in a simple manner.

How does Confidential Computing work on GPUs?

At a high level, confidential computing locks down the GPU by encrypting GPU memory and ensuring that all data moved in or out of the GPU is encrypted and authenticated. It even protects GPU to GPU communication.

This is great for security, but it fundamentally changes the way GPUs behave. GPUs are built to be fast when they do a lot of math, and they are fast when they can freely move data at high frequencies.

The big issue is that confidential computing changes how GPUs operate, forcing them to work in ways they weren’t designed for.

The Two Different Patterns of GPU Overhead

1. Memory Heavy AI workloads

Most modern AI workloads (especially LLMs) don’t just compute, but move data at high frequencies.

So, model weights are swapped in and out, attention caches keep being moved between CPUs and GPUs, and memory pressure forces frequent transfers.

But, with confidential computing, all the swaps must be encrypted and decrypted. So, the CPU becomes a bottleneck, leaving the GPU idle while it waits for data.

In one paper, latency was measured to peak at 36.2%, 52.8%, and 88.2% depending on model size and memory pressure.

https://arxiv.org/abs/2411.03357

And in another paper, confidential computing added around 54% to 903.9% (Wow…) slowdown when inferencing on a single GPU and around 10% to 455% overhead when training on a single GPU!

https://arxiv.org/abs/2410.15240

2. Training with Multiple GPUs

This is where things go crazy!

When training with multiple GPUs, GPUs frequently share small updates with each other to stay in sync. Additionally, each exchange is broken up into multiple small messages.

When using confidential computing, every single GPU message must be encrypted and decrypted, making lots of small messages extremely expensive.

Researchers measured an average of a 768% overhead and a maximum of 4060% overhead for the training workloads running on multiple GPUs.

https://arxiv.org/abs/2501.11771

When Should You Use Confidential Computing?

Given the significant overheads of using confidential computing, when should we use it?

Confidential computing is a great idea when a workload is very computation heavy and not very communication heavy. What we see across all the papers is that when running confidential computing, high communication between the GPU and CPU leads to significant slowdowns. So, if a workload spends most of its time doing math in the GPU, the slowdowns can become negligible.

You could also use systems designed around confidential computing. So, one example could be PipeLLM (https://arxiv.org/abs/2411.03357), which anticipates which data the GPU will require and encrypts them at the same time as the computation to reduce confidential computing overhead.

Conclusion

Before reading these papers, I didn’t really know the overhead of using confidential computing on GPUs. I mean, I knew there was some overhead, but never thought it would/could be this crazy. My surprise is what prompted me to write this blog post.

Confidential computing is great for security, but because it fundamentally changes how GPUs operate, it adds a significant performance overhead. Before deciding on confidential computing as a default solution, it is important to know when it is a good fit.

Whenever one creates runtime security policies in eBPF, it usually looks something like this:

When a certain binary runs and tries to access a resource, we check it against an eBPF policy map, enforce the rules, and call it a day—right?

Not really.

Consider the following example:

• sender.sh is launched to move some files from machine A to machine B.

• To do that, sender.sh starts readandsend.sh as a child process and passes it the files

  

In this case, when readandsend.sh is called by sender.sh, it should be allowed to read the provided files and connect to machine B’s IP.

But according to the current (static) policy stored in the eBPF map… it can’t.

So why not simply assign those file and network permissions to readandsend.sh as well?

The problem is that when permissions are assigned across different entities in a system, each entity should ideally have the minimal set of permissions required to function.

The primary reason for this is security: if a given entity is compromised, the attacker gains only a limited set of “powers.”

In the example above, we don’t want to grant readandsend.sh access to all files. Instead, we want it to inherit permissions from its parent only for the duration of the operation.

This may not be a common pattern, but supporting it would allow us to create policies that block certain operations by default and only unlock them when an authorized parent process invokes a child. In other words, capabilities remain restricted unless the correct process chain activates them.

So how hard to implement can that be?

And obviously, we can’t just invent a completely new structure. Instead, to make this possible, the eBPF map that holds the policy should:

• Support taking the union of two map entries (for inheriting permissions)

• Perform that union quickly (ideally O(1), or close to it)

• Fit within the eBPF verifier’s limitations

So… this should be easy, right?

The “Easy” Solution

To achieve this, we represent our policy entries as a bitmask, where each bit corresponds to a resource the executable is allowed to access.

Instead of storing the resource identifiers directly, we store them like this:

Instead of a single eBPF map, we use two eBPF maps:

• A map that assigns each resource an index (bit position)

• A map that stores policies as bitmasks

But why?

When checking access for sender.sh, we:

• Fetch the resource’s index

• Retrieve the binary’s policy bitmask

• Allow access if the bit at that index is set

But when sender.sh calls readandsend.sh, we can simply OR their policy bitmasks to create a combined policy and apply the same checks.

In other words, we dynamically adjust permissions based on who called the binary—while still keeping least privilege.

Here’s a small pseudocode snippet to wrap it up:

I hope you find this resource interesting. Keep an eye out for more updates and developments in eBPF in next week’s newsletter.

Until then, keep 🐝-ing!

Warm regards, Teodor and Neil

This is even more true for tracking processes, which is quite simple with eBPF: you just hook into execve, and you are good to go.

But if you want precise information, exact paths to the binaries that were started, you will need to hook into something like bprm_check_security, a BPF-LSM hook, that allows you to get a little more information on a process.

This is what we do at Bomfather, and it worked perfectly… Until it did not.

Interpreted Scripts

When we ran regular executables, we got a log output that looked something like this (We were logging when a process accessed certain protected files)

But when we ran a shell script, we got

Where did the shell script go?

Well, the reason this happened was because of how the kernel runs these files with a #! (Shebang), which includes shell scripts, or any interpreted language (Python, Perl, Ruby, etc).

When a shell script (or any interpreted script) is executed, the kernel adds an extra step. First execve() is called, and that call hits our eBPF hook. The data we get looks something like this.

But then the kernel reads the first two bytes of the file. If it sees #!, then the Linux kernel will call execve() again!

What happens is the kernel sees the shebang, which tells it not to execute the file; instead, it should pass the file to an interpreter, which will then actually do something. The key here is that the shell script doesn’t do anything by itself at all.

Our eBPF hook triggers on this second execve(), which leads us to the root of our problem.

The second execve() is for the same process, but the args are totally different. Since there is no executable, the only thing “doing” anything in this process is the interpreter. So when our eBPF code writes this data to our storage (an eBPF map), it rewrites over the data that is already stored.

So after the first execve(), we get:

And after the second execve(), we get

To solve this issue, all we have to do is check whether the path for that process (which we may have recorded before) is not empty; if it is, we don’t rewrite it.

This doesn’t solve everything

While the solution above allows us to capture the path of a shell script and any interpreted script, we still have one more catch for shell scripts.

You see, most of the operations you do in a shell script are going to be the calling of other executables. What I mean is that, for example, when you want to read a file in a shells script, you use cat the executable, to do this job.

But what this does is spawn a new process, so if we monitored a shell script and knew it opened file.txt, we expect an output like this.

But instead, we’d get something that looks more like this:

This is because the shell script never opens any file; instead, it delegates that job to another executable by starting a child process.

So if we want to accurately monitor shell scripts, we also have to track their children and treat them as a single process tree in our monitoring logic.

We originally developed on Ubuntu 24.04 on an arm64 chipset (because I develop on MacOS and run my eBPF code in a VM). We were sure our eBPF code worked on a 6.8 kernel running on an arm64 system.

We knew in theory that our code should run on all kernel versions >= 5.18. We knew our code only worked on these kernel versions, since we used bpf_loop.

bpf_loop is a helper function we use in our eBPF code that allows us to set higher loop limits, enabling us to accurately read filenames. Without bpf_loop, we would be restricted to reading the first 256 characters of a filename, which isn’t that good.

Newer Kernels Can Break eBPF

But practice is quite different from theory. When we tried to run our code on a 6.18 kernel, our eBPF code failed to pass the verifier, and we were stumped. Our eBPF code used CO-RE helpers, so why wasn’t it passing the verifier in this new kernel version?

We realized that it wasn’t that our code would not work on newer kernels. Instead, it was the verifier itself that improved, and because of that, the new improved verifier did not approve one of our LSM hooks on bprm_check_security.

What happened was that our return value in this specific hook could not be verified by the verifier after the kernel dynamically optimized our eBPF code. To fix this issue, we used barrier_var to prevent the kernel from optimizing our return value, allowing the verifier to ensure it was safe.

Older Kernels Can Break Stuff Too!

Once we did that, we also ran our code on the lowest kernel version we supported, 5.18, and it didn’t work either! This was the same problem we had with running on 6.18, but in reverse.

The kernel was older than the one we normally ran, so some optimizations that happened on newer kernels wouldn’t apply to these older ones.

In our case, we had to fix a couple of things.

1. We were computing a bitwise operation between two elements and directly assigning the result to a variable; instead, we had to split up the operations, first write to the variable, and then compute our bitwise operations

   

2. We were returning values from inline helper functions, which couldn’t be verified. But when we changed from returning values directly to passing a buffer to the inline helpers and having them write their values to the buffer instead of returning them, the verifier approved it.

   

Chipsets also break things

It turns out our eBPF code didn’t work on x86 either! Now I am pretty sure the reason for this is that x86 has a better verifier and may also optimize our eBPF code better.

Now I don’t have an example of what happened in our eBPF code, nor do I know how we fixed it, since I can’t locate the commit that did. Needless to say, it was quite irritating to fix.

Fixing verifier messages

Now these problems aren’t exactly easy to diagnose. In our codebase, since most of our code was in inline helper functions, the errors would just point to the failing hook and the register that was the problem, which are quite opaque on the surface.

These errors, at least for me, are quite hard to understand, so I would say the best way to understand them is to look them up, find out what the registers mean, and how the eBPF verifier behaves in different scenarios. Many times, you can’t find an example online, so to write good eBPF programs, you will have to learn what these errors mean, but in the past couple of years, debugging verifier errors has become a whole lot easier because of LLMs.

While I still don’t think LLMs can write good eBPF code (It may be able to write somthing that complies, but once you read it, you will be horrified by what it writes), LLMs are surprisingly good at actually tracking where the errors are coming from.

Now LLMs are quite bad at fixing these errors, and there is also a good chance that they totally hallucinate what the real problem is, but if you can get them to help you understand which register correlates to what variable, and what the error message could mean in your context (This works a whole lot better, if the LLM has a websearch option) debugging these errors becomes a whole lot easier.

Testing on multiple kernels

After we realized all of this, we set up some automated testing jobs to run our code on the latest and earliest kernels we supported. We did this by running our own self-hosted runners (not github runners) with 5.18 and the latest stable kernel (6.18). We do this by deploying two custom images, made with Packer.

We don’t use GitHub runners because LSM cannot be enabled on them, and our eBPF code is primarily built on LSM hooks. Github runners also come with a lot of bloat and a lot of vulnerabilities, so running our own VMs made more sense to us, for security’s sake https://substack.bomfather.dev/p/githubs-ubuntu-latest-runners-have.

In this post, let’s explore how we could have hacked this and how we could protect it from being hacked with eBPF Linux Security Modules to secure the runtime. We don’t know how the private keys were compromised, these are our assumptions about how someone could have compromised them.

A quick caveat, we at Bomfather do have an eBPF security solution that implements everything talked about in this blog post, but I wrote this blog post so that anyone can write their own eBPF code to implement runtime security!

Why use eBPF and LSM?

Why are we using eBPF and LSM? Trusting the kernel is better than the user space process, and LSM eliminates TOCTTOU vulnerabilities. eBPF lets you hook into the kernel without writing a kernel module

Why this approach?

Crypto is built on the assumption that everyone is trying to steal your money. You don’t trust servers, you don’t trust exchanges, you don’t trust anyone. You trust keys and signatures.

We’re doing the same thing, but for the runtime. Don’t trust userspace or root. Trust the kernel and trust cryptographic verification.

If we have a security agent running. To shut it down we should need a signed challenge response based on a nonce. To update policies around the agent, we should do the same thing. We copied this directly from how crypto works because it makes sense.

And LSM hooks solve a problem that’s been bugging security people forever TOCTTOU. You check if something is safe, then use it, but it has changed in the meantime. With LSM, the check happens at the exact moment of access. So, no gap.

Crypto was built assuming everyone on the network is trying to rob you. We’re just applying that same paranoia to the server.

For this, we assume we are running on a Linux machine and this approach works on everything, whether they use or don’t use containers.

How does YAML config turn into kernel enforcement?

The userspace agent reads your policy file, parses it, and writes the rules into eBPF maps. Maps are shared memory between userspace and kernel. When an LSM hook fires, the kernel side eBPF program looks up the policy in the map and decides to allow or deny. The YAML is for humans and the map is for the kernel.

Attack Vector 1: Just read the key file.

Attack: cat /path/to/hot-wallet-key.json

If you shell access (SSH, container escape, etc.), this is step one. The compromise is easy and it just works. I wish it wasn’t this easy.

Defense: An eBPF LSM hook denies all read and write by default, and only the “signer” gets read access. So, no one can read the private keys (not even with root). So you can provide root access to your employees without worrying about them accessing the keys.

This LSM policy is the easiest way to avoid a random user/process from reading the Private Keys.

 - container_path: “ns:pod:container” 
# The format is Kubernetes namespace, pod name, and container name. If you’re not running k8s, leave it empty, and the policy applies to the host. 
 executables:
 - path: “/app/signer”
 directories:
 - path: “/hotwallet”
 permission: “read”

SEC(”lsm/file_open”)
int BPF_PROG(lsm_file_open, struct file *file) {
 /*
 * Default deny file access to protected paths.
 * Only explicitly allowed executables can read protected files.
 * Even root is denied.
 */

 // Get the file path being opened
 // Get the current process’s executable path
 // Check if the file is in a protected directory
 // Check if this executable is allowed to access this directory
 // If not allowed, return -EPERM
}

Attack Vector 2: Backdoor the Binary.

Suppose we had an insider attack, which is a lot easier. Think about it, if we can bribe an insider with 5% of the stolen assets and he can move to a country that wouldn’t extradite prisoners, then it is millions of dollars’ worth of risk that someone could be motivated to take. The insider can replace the binary and then leave the country the moment the tokens are stolen.

Attack: Replace the signing binary with a modified version that exfiltrates keys.

Defense: Immutable executables and SHA256 verification. eBPF LSM can prevent binaries from being modified. We should not only protect the “signer” of the transaction, which has access to the private keys, but also the platform’s critical system binaries.

# Critical executables that cannot be modified at runtime
immutable_executables:
 # Application Binaries
 - path: “/app/signer”
  
 # Container Runtime Infrastructure
 - path: “/bin/runc”
 - path: “/bin/k3s”
 - path: “/bin/containerd-shim-runc-v2”
  
 # Network Infrastructure
 - path: “/bin/cni”
 - path: “/bin/aux/xtables-nft-multi”
 - path: “/bin/aux/nft”

This is the solution for vectors 2,6, and 7.

SEC(”lsm/file_open”)
int BPF_PROG(lsm_file_open, struct file *file) {
 /*
 * Block writes to immutable executables and directories.
 */

 // If open mode is write
 // Check if path is a protected executable
 // Check if path is under a protected directory
 // If protected and write attempted, return -EPERM
}

SEC(”lsm/path_rename”)
int BPF_PROG(lsm_path_rename, struct path *old_dir, struct dentry *old_dentry, 
 struct path *new_dir, struct dentry *new_dentry) {
 /*
 * Block rename of protected files.
 */

 // Check if source or destination is protected
 // If either is protected, return -EPERM
}

SEC(”lsm/path_unlink”)
int BPF_PROG(lsm_path_unlink, struct path *dir, struct dentry *dentry) {
 /*
 * Block deletion of protected files.
 */

 // Check if file is protected
 // If protected, return -EPERM
}

The next question would be: how do we know the above executables weren’t compromised before the eBPF security agent started enforcing the policies? When the eBPF agent monitors, it would deny any delete, modify, or symlink to these protected executable files.

To ensure the executables weren’t compromised, we’d need to use kernel SHASUM validation.

Fsverity handles this. Fsverity is a kernel feature that computes the Merkle tree hash of a file’s contents. If anything changes, the kernel invalidates the digest.

Then, at execution time, the eBPF hook checks bpf_get_fsverity_digest() https://docs.ebpf.io/linux/kfuncs/bpf_get_fsverity_digest/ against your expected hash.

If there is a mismatch, then we can deny the execution of the binary. This function works only on LSM and not on tracepoints. There is a fantastic talk about this from the Linux Plumbers Conference BPF_LSM + fsverity for BinaryAuthorization - Song Liu, Boris Burkov.

Attack Vector 3: Memory dump/ptrace

“But we don’t store keys on disk. We use a KMS.” Cool. But the key still has to exist in memory to sign a transaction.

Wherever it came from, file, KMS, HSM API, or remote vault at some point, the signing process has the key material in RAM. That’s the target.

Here is a problem, How do we provide administrators with access to maintain the systems without allowing them to siphon funds from the crypto wallet? The solution is to make sure that they can’t be able to ptrace the security agent or the signer or replace the signer binary.

Here’s the problem, you can’t trust admins. Not because they’re bad people, but because compromised accounts are a thing. An attacker with root can use ptrace to attach to any process and dump its memory. That’s how you’d pull keys out of RAM. So we have a conundrum. We need to give people root to maintain the system. But we can’t let them ptrace the signer or the security agent.

Attack: Attach a debugger to the signer process, dump the heap, and extract key material.

Again, this is where eBPF comes to the rescue.

SEC(”lsm/ptrace_access_check”)
int BPF_PROG(lsm_ptrace_access_check, struct task_struct *child, unsigned int mode) {
 /*
 * Block ALL ptrace operations on protected processes:
 *  - GDB/debugger attachment
 *  - strace system call tracing
 *  - /proc/<pid>/mem access
 *  - process_vm_{readv,writev} syscalls
 *  - Any other ptrace-based inspection
 *
 * prevents debugging.
 */

 // Check if ptrace blocking is disabled
 // Check if the target PID is part of the protected processes list
 // Check if the security agent is being ptrace’d. If we compromise the security agent, then we compromise everything.
 // if any of these checks fail, return -EPERM
}

Attack Vector 4: Kill the security agent

All of this depends on the eBPF security agent. What if a malicious user with sudo privileges kills the agent? Then it’s game over.

Attack- sudo kill -9 <agent-pid>Then, retry vectors 1-3.

Defense: The agent blocks its own kill signals. You can’t kill it without a Public Key Infrastructure challenge response with a nonce. No Key, no shutdown.

SEC(”lsm/task_kill”)
int BPF_PROG(lsm_task_kill, struct task_struct *p, struct kernel_siginfo *info, 
 int sig, const struct cred *cred) {
 /*
 * Block kill signals to the security agent.
 */

 // Check if target process is the security agent
 // If yes, return -EPERM
}

Here is a detailed write up and a video on how to prevent this. https://substack.bomfather.dev/p/stopping-kill-signals-against-your?r=2bngj5&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

Attack Vector 5: Update Policy as Root (eBPF Maps)

Most eBPF security agents use eBPF maps to communicate security policies to the kernel. The maps are globally writable by any privileged user.

Attack - sudo bpftool update eBPF map policy to disable the security. Then retry vectors 1-3.

Defense: The agent would have to write an eBPF LSM policy to prevent the maps from being updated. It can be updated only by the user process that starts the eBPF agent. We can make it better by using the PKI mentioned above with a nonce to update the policies, so we know it’s trusted.

SEC(”lsm/bpf_map”) 
int BPF_PROG(lsm_bpf_map, struct bpf_map *map, fmode_t fmode) {
 /*
 * Block unauthorized modifications to security policies.
 * Only the security agent can update its own policy maps.
 */

 // Check if this is a security policy map
 // Check if the calling process is the security agent
 // If not the agent, return -EPERM
}

We wrote about how to do this in depth and also included a video to explain this https://substack.bomfather.dev/p/attacking-and-securing-ebpf-maps

Attack Vector 6: Write to sensitive directories

Now you can’t read the keys or backdoor the signer. But what if you drop a malicious binary into a directory that’s in the PATH or LD_LIBRARY_PATH? Or write a cron job? Or modify a config file?

Attack: echo “malicious script”> /etc/cron.d/exfil or cp /tmp/evil /usr/local/bin/signer-helper

Defense: Have an eBPF LSM Policy that will prevent any writes to any of these directories.

immutable_directories: 
 - path: “/private-keys”
 - path: “/etc/LD_LIBRARY_PATH”

Attack Vector 7: Replace system executables

Similar to Vector 2, but broader. Why target just the signer? Replace ls, cat, bash - any tool an admin might run. Wait for someone to interact with the keys.

Attack: cp /tmp/evil-bash /bin/bash

Defense: Global readonly executables. Not just an immutable signer binary. All system binaries are protected from modification.

immutable_executables:
  - /bin/ls

Attack Vector 8: LD_PRELOAD hijacking

The “signer” is legit and passed fsverity, only process allowed to read the keys.

Attack: Use the LD_PRELOAD env variable. LD_PRELOAD=/tmp/evil.so ./signer

The injected library intercepts every function call. When the signer reads the private key, the injected malicious code sees it too. LSM allows the read because it’s the authorized process, but it has been hijacked. LD_PRELOAD is normally a good process which is used in a lot of places but it can be used maliciously.

Defense: Use eBPF to track which processes are allowed to use the LD_PRELOAD environment variable. You can track this at the kernel when a process starts.

We aren’t using the LSM hook here, and there is a possibility of TOCTTOU. There is a tracepoint where we capture the ENV data and secure it in the LSM hook.

SEC(”tracepoint/syscalls/sys_enter_execve”)
int trace_execve(struct trace_event_raw_sys_enter* ctx) {
 /*
 * Capture environment variables at process start.
 * Flag processes that have LD_* variables set.
 */

 // Read environment variables from execve args
 // If any starts with “LD_”, flag this process
}

SEC(”lsm/bprm_check_security”)
int BPF_PROG(lsm_bprm_check_security, struct linux_binprm *bprm, int ret) {
 /*
 * Block unauthorized LD_PRELOAD.
 * Only allowlisted executables can use LD_* environment variables.
 */

 // Get executable path
 // If process was flagged for LD_* usage
 //   Check if this executable is allowed to use LD_*
 //   If not allowed, return -EPERM
}

Turns out real world processes actually use it. For example, k3s and cni (cloud native Infrastructure) use this. We wrote about this and provided an actual example https://substack.bomfather.dev/p/ld_preload-the-invisible-key-theft?r=2bngj5&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

Attack Vector 9: Compromise before boot.

All of this assumes the eBPF agent is running and was started on a clean system. But how do we trust that? What if the attacker loaded a rootkit in the bootloader? What happens if they modified the kernel.

Attack: Modify the bootloader or kernel image, load malicious code before the eBPF agent starts. Defense: The system should be configured to use a Secure Boot chain and a Signed bootloader, signed kernel. The system won’t boot if anything in the chain is tampered with. By the time userspace starts and the eBPF agent loads, you have cryptographic proof that nothing was modified.

What’s left?

• Hardware implants.

• Side-channel attacks.

• Compromised chips from the factory.

I can’t help you there. Different threat model. Different budget. If someone is decapping your CPU, you have other problems.

Most of the time, digital access is way easier than physical. Physical attacks cost real money, you need people, equipment, and you need to be somewhere. Compromising one employee or finding one vulnerability? That’s a laptop and patience.

So it gets stolen because cat /secrets/key.json worked. Or someone attached gdb and dumped memory. Or an insider replaced a binary and flew to a country without an extradition treaty.

Upbit lost $37M. We don’t know which attack vector it was. But vectors 1-9? All stoppable with eBPF LSM. No HSM. No air-gapped signing setup. Just kernel enforcement on regular Linux.

You don’t have to stop North Korea. You just have to be harder to rob than the next exchange!

Up until recently we didn’t know that you could get cloud credits without a VC, so we decided to write about this for anyone who wasn’t aware of this!

We reached out to a few cloud providers to reduce our cloud costs. We were successful with Cloudflare, they gave us $5000 in credits. This was good, but we needed bare bones Linux VMs and k8s nodes for our runtime and GPU security, and Cloudflare didn’t have one. But we do use Cloudflare for things like Tunnels and DDoS protection.

We got angel funding from and used Brex as our banking partner. We dug into Brex Perks (https://www.brex.com/support/brex-partner-perks) and realized they offered “Google Cloud: You can have Google Cloud and Firebase costs covered up to $200,000 USD over 2 years.”

The Brex “AWS Activate” was only $5,000. We knew that wouldn’t be enough for what we are building. So we didn’t bother with AWS Activate.

We were already talking to GCP before this, and we shared that we were banking with Brex and wanted to use this option for cloud credits. The GCP team said we’d need funding to get credits. It doesn’t have to be VC, angel or SAFEs work.

The GCP team reached out to AngelList to confirm that Balaji funded us, and once that was confirmed, we got $25,000 in cloud credits. The perks page says $200k, but GCP gave us $25k. Not sure why the gap, but we’ll take it.

This process took more than 8 weeks, with back and forth emails and verification delays. So if you want to do this be patient.

If you’re a Brex customer with angel funding, here’s exactly what to do:

1. Fill out the form https://dashboard.brex.com/p/rewards/perks-and-discounts?categories=cloud-computing&drawer=perk%7EGoogle+Cloud

2. Email cloudstartupsupport@google.com

3. The cloud startup support team usually asks for the Angel’s email address to validate the funding.

Look, $25k isn’t life changing, but it’s at least 6 months of runway we didn’t have to pay for. If you’re bootstrapping or angel funded, every dollar counts.

The process is bureaucratic and slow, but definitely worth it.

If you hit any roadblocks, hit me up at nathan@bomfather.dev, and I’ll try to help!

We know that this blog post sounds kind of sponsored but we aren’t sponsored by Brex. We are just sharing this because it was useful for us!

So you are running an L2 node and have a lot of money in it. Like most of us, we “hope” it is secure. In this, let’s unpack some of the challenges that I ran into while trying to secure the “Runtime” for L2 Nodes.

I am not an Ethereum or an L2 Node expert. I am a regular guy who wants to run an L2 node and has started questioning the Runtime’s security posture.

I am doing this experiment on k8s.

The JWT Secret

Let’s start with something simple, such as securing the “JWT” used to authenticate the OP Node to the Nethermind API. What can go wrong if that gets compromised? Money can be lost.

So how do we secure the JWT secret?

For starters, we could use something like Vault or AWS Secrets Manager.

The secrets are now loaded into a Volume on the Node(Linux machine).

The goal is to restrict access to op-node, Nethermind, and runc, and nothing else. Not even the root user.

Our requirement is that only these three processes (as of now) should be able to access it. We want guarantees from the kernel with eBPF.

What is runc?

It is the container runtime. It is the process that runs the container. It is the process that starts OP Node and Nethermind. It needs access because it is the process that talks to the kernel and sets up everything. So it needs access to set things up.

How does the secret get written to the file? Are we storing the secrets on the disk? The files are stored on disk using “tmpfs”

What is “tmpfs”?

It is a temporary file system. It is a file system that is stored in memory. It is a file system that is not persisted to the disk. It is a file system that is deleted when the container is deleted. Another way to say it is an in-memory file system.

The secret from the AWS Secrets Manager or Vault is written to the tmpfs by the k8s runtime.

Why kernel?

Following the principles of zero Trust from crypto, how do we know that an elevated privileged process didn’t read the keys? So that’s why you use the bottom of the stack to ensure it hasn’t been tampered with.

Why eBPF?

We don’t want new kernel modules. eBPF is the new cool kid that solves this problem. Now, the next question is: what if another eBPF allows when our program tries to stop? Can’t the malicious user do it? So that they steal the JWT Token. Yes, we tested that, and it’s not possible. You can read more about this at https://substack.bomfather.dev/p/how-we-secured-our-ebpf-from-ebpf.

Our requirements

The security agent (eBPF) running in the kernel should not only protect the JWT Secret, but also protect the executable that has read and write access to it. Why? Because if the attacker can compromise the executable with access, they have circumvented the problem with a Supply Chain Attack.

Because the eBPF agent runs in kernel space, not in user space, even administrators cannot bypass it. It enforces the policies at the kernel level before user-space-level executables can interfere.

We also want to protect against insider threats, such as system administrators not being able to read secrets. Also, what happens if the administrator stops the security agent? Then it’s game over.

We thought about and built a solution so that the agent cannot be killed. The agent can be killed with a PKI and Nonce-based solution. Similar to the platform the crypto is built on (we copied). https://substack.bomfather.dev/p/stopping-kill-signals-against-your

The LSM Hook

eBPF didn’t invent security from scratch. It plugs into Linux Security Modules (LSM), which is the same framework that powers SELinux and AppArmor.

Here’s what that means.

LSM is the kernel’s security checkpoint. Every time any process tries to open a file, read memory, make a network connection, or create a process, the kernel fires an LSM hook. This happens before the operation completes.

eBPF attaches to these hooks. When Nethermind calls open(”/secrets/jwt.hex”), the LSM hook fires, our eBPF program runs, checks the policy, and decides whether to allow or deny the request.

Why this matters:

• Runs in kernel space - Can’t be bypassed by userspace tricks

• Happens before the syscall - Blocks the operation, doesn’t just log it

Write Access To The JWT Secret

In our example, we used k3d (a simple Kubernetes cluster).

When we ran our tests with Deny all access to the JWT. We realized we need to provide access:

• /bin/k3s

• /bin/containerd-shim-runc-v2

• /bin/runc

OK. Now that we have figured this out, it is much easier to choose which of these executables we need to secure.

But what happens if someone modifies these executables?

Immutable Executables

We want a security policy that prevents k3s, containerd-shim-runc-v2, and runc from being modified. We are trying to do what an immutable kernel would be like, Fedora.

We want to add these executables to the policy, where we can define an immutable executable that no one can modify.

The security agent would ensure that all these executables aren’t modified while the agent is running. This prevents someone from backdooring runc, for example, and performing malicious actions.

GitHub Gist For Security Policy

What We’re Protecting

Now, I know we’ve been focused on the JWT secret, but here’s the thing: this same eBPF-based approach protects way more than just that one file.

What else are we protecting from the kernel?

Cryptographic Keys:

• Nethermind keystore (/data/keystore) - Your validator signing keys. If compromised? Sign malicious blocks, steal funds.

• Nethermind P2P node key (/data/nodeKey) - Network identity. Compromise this? Eclipse attacks, network manipulation.

• OP Node P2P private key (/data/opnode_p2p_priv.txt) - P2P identity theft territory.

• OP Node discovery secret (/data/opnode_discovery_secret.txt) - Network manipulation potential.

Application Data:

• Blockchain database, state, receipts at /data

• Temporary storage at /tmp/nethermind (in-memory tmpfs)

Same principle as the JWT secret:

Only the specific executable that needs access gets access. Everything else? Denied at the kernel level. So when we say “secure the JWT secret,” we’re really talking about an entire security model that protects all your sensitive data from unauthorized access.

But wait. These are container paths, the view from inside the container. In k8s, /data is mounted from somewhere on the host, like /tmp/base-node/nethermind-data. What stops someone from just SSHing into the host and reading that path directly? Nothing. Yet. More on that later.

The Attack Vector You Didn’t See Coming

OK, so we’ve locked down file access at the kernel level. Only Nethermind and op-node can read the JWT secret.

What if I told you that, even though we’ve restricted which executables can access the secret, ensured it is not backdoored, and verified the SHA256 digest, an attacker can still steal it by hijacking the Nethermind or op-node themselves?

How? One word: LD_PRELOAD.

It’s an environment variable that lets you load your own shared library before any other library loads.

You can intercept every single function call that Nethermind makes. Reading a file? Intercepted. Making a network call? Intercepted. That JWT secret Nethermind just read from /secrets/jwt.hex? Yeah, we can grab that too.

But wait, can the eBPF agent block unauthorized file access, right?

Sure does. But here’s the thing: Nethermind is authorized. So when Nethermind reads the JWT secret, the eBPF agent says, “yep, you’re good buddy.” But what if Nethermind isn’t really Nethermind anymore? What if it’s been... augmented?

Think about it. We protected the file. We protected the executable. But did we protect the execution environment?

And if they can, what else can they intercept? Just the JWT secret? Or every cryptographic operation Nethermind performs?

I’ll tell you what keeps me up at night: We spent all this time securing the file, but forgot about the process that reads it.

Do real-world processes really use LD_PRELOAD? You might think I’m making this up. But nope. It’s real. Remember k3d? It uses LD_PRELOAD to instrument the container runtime for:

• /bin/containerd-shim-runc-v2

• /bin/k3s

• /bin/cni

• /bin/aux/xtables-nft-multi

I’ve written a complete technical breakdown with working exploit code here: LD_PRELOAD: The Invisible Key Theft. https://substack.bomfather.dev/p/ld_preload-the-invisible-key-theft

How do we defend against this?

We use eBPF to stop this. Though there is a TOCTU Vulnerability in the eBPF solution. We track when a process starts, capture its environment variables, and use that information to apply policy rules, such as which processes are allowed to use LD_PRELOAD.

What We Learned

Security isn’t about protecting a single thing, like JWT keys in AWS Secret Manager.

You secure the file. Attacker targets the executable.

You secure the executable. Attacker hijacks the environment.

You secure the environment. Attacker finds another vector.

That’s why we keep going back to the kernel. eBPF lets us enforce security at every single checkpoint.

Where Do We Go From Here?

We haven’t covered all of the Runtime vectors. But for now, if you are running an L2 node, you should be asking: who has access to the JWT Secret? What’s protecting my executables? Can someone dump the memory, ptrace it, or attach a debugger to change things? More on that later.

What’s even happening?

Essentially, an attacker wants to gain elevated access on the machine, but is only an unprivileged user. This vulnerability allows the attacker to obtain root on host , crash the kernel, and disable Linux Security Module policies. In a nutshell, pretty bad stuff.

The root of it all

Essentially, the root cause of all three of these vulnerabilities is a Time-of-Check to Time-of-Use (TOCTU) vulnerability in runc’s mounting logic.

runc just wants to mount normal directories into a container, but an attacker can sabotage this process, to make runc mount some protected directories into the container, the most useful of them being /proc/* dirs.

Attackers want to do this, because as an unprivileged process, they can’t access these protected knobs to control the whole machine, and the kernel.

The attack is quite simple, runc opens a file/dir/mount validates the path, and then performs the mount. In this small time window between mount and read, the attacker is replacing a path with a symlink during a race window, so that runc mounts the attacker’s target instead.

All three CVEs follow this pattern, just in different places in runc.

Attacking masking paths: CVE-2025-31133

Typically, runc masks specific directories, so that sensitive paths are not exposed to containers. It does this by mounting /dev/null over those paths.

What an attacker can do is use that TOCTU attack to make runc mount a sensitive directory instead of /dev/null. In the CVE, the examples directories were /proc/sys/kernel/core_pattern and /proc/sysrq-trigger

Both of these are really, really bad, essentially game-over scenarios, if mounted in the container.

Mounting /proc/sys/kernel/core_pattern

If the attacker chose /proc/sys/kernel/core_pattern, they could basically run any arbitrary code they wanted on the system as root.

This is because core_pattern runs whenever a program crashes. And if an attacker has access to core_pattern, they can configure its path/arguments to run whatever bad script they want. They can also prefix core_pattern with a |, which effectively allows their command to run as a root-level process within the host’s namespace.

So an attacker can write any script they want to core_pattern, prefix it with |, and essentially run it across the whole system as root, doing whatever they want.

Mounting /proc/sysrq-trigger

If they have access to /proc/sysrq-trigger, they can essentially control sysrq, which allows you to perform low-level kernel actions immediately.

You could cause the kernel to panic, shut down processes, and do a lot of bad things.

Attacking the console: CVE-2025-52565

This is very similar to the masking attack; runc needs to give the container a console, so it mounts /dev/pts/$N to /dev/console. Now, an attacker can symlink /dev/pts/$N to a protected dir like core_pattern or sysrq-trigger, so we can write to the console and thereby write to these protected dirs.

Disable Linux Security Module (LSM) polices: CVE‑2025‑52881

When runc needs to write an LSM policy, it needs to write it to /proc/self/attr/* to set LSM labels for the container that is being started. An attacker can execute a very similar TOCTU attack on this write operation.

runc wants to open /proc/self/attr/*, but during that small time window, if an attacker symlinks the file to another location, the LSM labels are not written, so security policies are not enforced. What could be even worse is that those writes could be piped to other sensitive files, such as core_pattern or sysrq-trigger.

Caveats

This is harder for the attacker to execute, since they must be able to symlink the file where the LSM policy attribute path is stored. This also does not affect something like BPF-LSM, since this is only a way for userspace applications to write policies to the kernel.

How can we stop this?

Upgrade to the patched versions of runc as soon as possible, which are versions 1.2.8, 1.3.3, 1.4.0-rc.3. This should remove the immediate threat vector.

As for more proactive steps to take against a similar attack, you could:

Run most workloads as rootless. It

may take extra effort in some cases, but it immediately improves your security posture.

Another significant step we could take is to prevent symlink creation in specific directories.

If we can forbid symlinks on proc/sys and /proc/*/attr/*, you should be able to prevent anyone from directly symlinking to one of these protected dirs, even if a root process (like runc!) is compromised.

To enforce something like this, using something like BPF-LSM, it should be pretty easy to implement policy by writing a hook on something like security_inode_symlink and security_file_open (I don’t have much experience with AppArmor and SELinux, so they may not work, since they rely on LSM labels on files, unlike BPF-LSM, which doesn’t rely on that)

PS: Bomfather can stop this, we aren’t trying to plug ourselves, so use whatever works for you best :)

A Caveat

I’m in no way an Ethereum expert, nor have I ever run an Ethereum node before. I am just a regular guy who wants to run a node for fun and solve my itch.

Then Why Rebuild?

Because I am paranoid and security obsessed and want fewer moving parts in the build and runtime. Especially when it involves (to the moon baby!).

The Problem

Remember the 2021 CodeCov Bash Uploader compromise? A modified script exfiltrated secrets from thousands of build environments. The original geth image in base/node uses the same kind of risky pattern: RUN curl -sSfL ‘<https://just.systems/install.sh>’ | bash -s -- --to /usr/local/bin. This pipes an untrusted script directly to bash without checksum verification. This is a classic supply chain attack.

So what is wrong with the existing image for geth?

• It has a shell, package manager, and unnecessary binaries.

• It is not a static binary build.

• It does not use pinned base images with SHA256 digests. Without the images being pinned, they could have been tampered with and you wouldn’t have a clue what you’re building.

• It is a root user. A root use can do anything.

• It exposes all ports, not just the necessary ones. Its like leaving your house with all the doors and windows open.

• It isn’t a reproducible build.

  

The Solution

How did I secure the build for geth?

• Pinned base images with SHA256 digests. This ensures the image can’t be tampered with and the build is reproducible.

• Static Binary Build. CGO_ENABLED=0 So, no libc dependencies and no dynamic linking.

• Being paranoid I am explicitly setting the user to nonroot even though I know the distroless is already nonroot.

• Expose only the necessary ports. I am used to clicking the lock button twice on my car to ensure the doors are locked even though they were already locked on the first press of the button. I don’t want all the ports open on my docker image.

• The final image is a single binary, there are no other tools or dependencies in the image. Less patching means fewer vulnerabilities and fewer dreaded Dependabot PRs.

• If there is a zero day in the binary, it will be harder to exploit since we don’t have these additional tools. Less code means fewer vulnerabilities (the famous nocode

• philosophy: no code = no bugs).

The Build

FROM golang:1.23@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS build

WORKDIR /app

# Build arguments for version info
ARG OP_NODE_REPO=https://github.com/ethereum-optimism/optimism.git
ARG OP_NODE_TAG=op-node/v1.14.1
ARG OP_NODE_COMMIT=c1081e3ad0004590435e3179e583cdfdbdd6bc61

RUN git clone $OP_NODE_REPO --branch $OP_NODE_TAG --single-branch . && \\
    git switch -c branch-$OP_NODE_TAG && \\
    [ “$(git rev-parse HEAD)” = “$OP_NODE_COMMIT” ]

RUN cd op-node && \\
    GITCOMMIT=$(git rev-parse HEAD) && \\
    GITDATE=$(git show -s --format=%cI HEAD) && \\
    CGO_ENABLED=0 GOOS=linux go build -v \\
      -ldflags “-s -w -X main.GitCommit=${GITCOMMIT} -X main.GitDate=${GITDATE} -X github.com/ethereum-optimism/optimism/op-node/version.Version=${OP_NODE_TAG} -X github.com/ethereum-optimism/optimism/op-node/version.Meta=stable” \\
      -o /bin/op-node \\
      ./cmd

FROM gcr.io/distroless/static:nonroot@sha256:e8a4044e0b4ae4257efa45fc026c0bc30ad320d43bd4c1a7d5271bd241e386d0

WORKDIR /app

COPY --from=build /bin/op-node ./

USER nonroot

EXPOSE 9545 9222 9222/udp 7300

ENTRYPOINT [”./op-node”]

What This Doesn’t Solve

So is geth secure? Kind of.

How do you trust the build environment? Does your build environment have any vulnerabilities? https://substack.bomfather.dev/p/githubs-ubuntu-latest-runners-have ;)

What happens to runtime security like access to the secrets and keys? Do you trust the runtime environment?

Crypto was built to have zero trust, but it has to run somewhere, so a new problem might be centered around whether your k8s secure? How about your cloud provider, are they secure? Or your hardware, do you know if it is secure?

What about the insider threat?

All of this is for another day.

Was It Worth It?

Is this overkill for a testnet node? Yes.

Would I do it again? Also yes.

Will this stop nation-state actors? LMAO no.

Will it stop some script kiddie running automated scans? Maybe!

For a production mainnet node handling real funds? Absolutely.

My next project: convincing myself that my hardware isn’t already compromised from the factory. (Spoiler: it probably is, but ignorance is bliss.)

To the moon! 🚀 (But securely.)

What is the usual way of doing this

Most eBPF agents run as daemons, so they can’t be shut down, only allowing authorized actors to shut them down. But if a malicious process gets elevated privileges, it could shut down our eBPF agent. Now, most security tools would have to stop here, but eBPF does not, we can make our agents even more resilient.

If we want to do this, we need a way to stop malicious processes from killing our agents, no matter their access to the system.

The easy steps to stop death itself

That seems complicated, right? How can we stop any process from killing our agent, no matter their system privileges? With eBPF, we can hook into security_task_killand stop shutdown signals from reaching our agent.

If you write an lsm/task_kill hook, you can stop the death of any process, so if we write a hook like this, you can prevent the death of your process, no matter what happens.

SEC(”lsm/task_kill”)
int BPF_PROG(lsm_task_kill, struct task_struct *p, struct kernel_siginfo *info, int sig, const struct cred *cred) {
  u32 pid = BPF_CORE_READ(p, pid);
  if (pid == my_userspace_agent_pid) {
    return -EPERM;
  }
  return 0;
}

But now I want it to die, really, really badly

Ok, great, now it is impossible to kill, the job is done.

But now, your boss just added a new requirement to your eBPF agent, so you build it and release a new version to use. But now the question is, how do we upgrade our current running eBPF agent?

If we can’t kill the process, we can’t shut it down and start an upgraded version. At this point, nothing can stop it. Your agent used to be useful, but now it’s like a wart that won’t go away.

At the end, to solve this issue, you restart every single machine in your infra, angering your boss (he was never happy, but now he really is pissed) and messing up everyone’s day (or week). You must remove the shutdown security from your eBPF agent, leaving it wide open for any bad actor to kill it.

How did things go so wrong, so fast?

How to tell something that it needs to die

The problem is that we need a way to tell the agent whether a good guy or a bad guy is killing it. But according to our requirements, we can’t even trust root(sudo). What do we do?

We could send a signed message to the eBPF program’s user space agent. If an authorized user signs it, the eBPF program kills itself. However, even this has a problem.

We may be able to keep our key secret, but can we keep that signed message secret? Someone could perform a replay attack and sabotage our system.

If we killed the agent once, and somebody got that signed message, now they could send the message again, at their own discretion.

To solve this issue, we could use a nonce. When an authorized (or unauthorized) user requests to kill the eBPF agent, the agent sends back a random string to be signed. The user must send this signed nonce back to the agent, which, if valid, will shut down the program. This requires the same authorized user to re-sign a new message for each shutdown, ensuring it is not an attacker with an old, used message.

How are keys more secure

Keys are more secure since we can keep them on separate, more secure systems. Whenever the agent needs to be shut down, you could send a request from the agent to another machine, which could verify if they want to shut down the agent manually and then sign the message. The key could even be a hardware key. Using a key lets us move our security to a more secure system (the most secure being hardware keys).

Do I really need to do this?

Well, you don’t exactly have to do this, but then again, somebody could just shut down your program, neutralizing your whole eBPF agent, without even finding a loophole in its logic.

Now, this doesn’t mean that when you start writing and deploying eBPF agents, you immediately need to do this, but without some sort of shutdown security, writing a security tool with eBPF without it kind of defeats the purpose of kernel-level security.

If you want your eBPF agent to be truly secure, securing against shutdowns is not enough. Attackers can manipulate your eBPF maps, disabling policies and destroying logs. If you want to be secure, you should read this blog post: https://substack.bomfather.dev/p/attacking-and-securing-ebpf-maps

Then we run our builds on GitHub’s ubuntu-latest runners, which have 1,681 packages.

That’s absurd. We won’t run production with all this cruft, so why would we build with it? Supply chain attacks targeting build environments are increasingly common and devastating, compromising one dependency can silently infect every artifact we produce. Our build environment can access our source code, secrets, and signing keys. If anything, it should be more locked down than production, not less.

Last week, we inventoried what actually ships on ubuntu-latest. The results convinced us to move away from GitHub runners entirely.

The numbers

We ran an inventory script on a fresh ubuntu-latest runner, and the total package count is 1,681.

Breaking it down, 214 APT system packages, 162 Python packages, 173 Ruby gems, 119 Conda packages, and 13 NPM packages.

We build Go and C code. We need the Go and C compiler, git, and maybe 15 other tools. About 20 packages total.

The runner gives us 1,681 packages. That includes entire language ecosystems we never touch (Python, Ruby, Node, Conda). All sitting there, increasing our attack surface.

The vulnerabilities

We scanned 348 of those packages using OSV.dev. We could only scan the Ruby, Python, and NPM packages because APT and Condaaren’t supported by the scanner yet.

Of the 348 packages scanned, 16 packages have known vulnerabilities, with a total of 63 vulnerabilities across those 16 packages. 9 vulnerabilities were rated high, 38 moderate, 6 low, and 10 unknown.

The Python cryptography package version 41.0.7 has 6 high severity vulnerabilities.

The setuptools package version 68.1.2 has 3 high severity vulnerabilities. CVE-2024-6345 is a path traversal bug where an attacker can write arbitrary files anywhere on the filesystem. That means someone can inject malicious code directly into your build artifacts.

We don’t use Python, and we definitely don’t import cryptography or setuptools. But they’re there in our build environment.

Would we run production with known high severity vulnerabilities? No.

Would we ship a container with cryptography 41.0.7 knowing it has 6 high CVEs? Never.

So why is it okay in our build environment?

The full list

For transparency, here are all 16 vulnerable packages we found:

Jinja2 has 10 vulnerabilities. The cryptography package has 10 with 6 being high. Twisted has 6. Setuptools has 6 with 3 being high. Then certifi, idna, requests, urllib3, and python-apt each have 4. Configobj, cgi, webrick, and net-imap each have 2. Finally rdoc, resolv, and uri each have 1.

These are just the packages we could scan. We couldn’t scan the 1,214 APT packages or the 119 Conda packages. The actual vulnerability count is almost certainly higher.

The code for getting the vulnerabilities is here, bomfather/tools.

Attack surface

Every package is a potential attack vector. We know this, it’s why we minimize our production containers.

A production Go service might have 15 to 25 packages. The Go runtime, some system libraries, a few utilities.

The ubuntu-latest runner has 1,681 packages. That’s somewhere between 50 and 100 times more than our production containers. Each one has the potential to have vulnerabilities.

The math is brutal. If 4.6% of scanned packages have CVEs, and we extrapolate that to all 1,681 packages, we’re looking at roughly 77 vulnerable packages in the full environment. Even if the real number is half that, it’s still 38 packages with known exploits.

In production, one CVE gets immediate attention. We patch, test, deploy. Having 38 or 77 known vulnerabilities would be a critical incident.

The big problem

We treat production extremely delicately with tightly audited layers, and a fast reflex for patching a critical CVE. Our builds have access to source, secrets, and signing keys, so they should deserve equal or stricter treatment.

Here’s what really bothers us. Those 9 high severity vulnerabilities are known, they’re in a public CVE database (published by security researchers).

Yet, they’re still there on the runner.

Maybe GitHub will update the image next week, maybe next month, or maybe they already did, and we tested an older image. The point is, we don’t control the timing, we can’t force an update, and we can’t even see what changed without diffing the entire environment ourselves.

In production, we’d be on those vulnerabilities immediately, patching, testing, and deploying the same day if possible. But for builds? We just accept them, run our CI on whatever GitHub gives us, and hope the vulnerabilities don’t matter.

How we collected this data

We ran this inventory on an actual ubuntu-latest runner using GitHub Actions. Package counts came from the native package managers. APT packages from dpkg, Python from pip, Ruby from gem, and so on.

Vulnerability scanning used the OSV.dev API (Google’s Open Source Vulnerability database). The scan covered Python, Ruby, and NPMpackages, but couldn’t handle APT or Conda packages due to scanner limitations.

Total scan time was about 4 seconds. Out of 348 packages scanned, 16 had CVEs. Totally, we found 63 vulnerabilities.

We only scanned 348 of the 1,681 packages. The remaining 1,333 packages might be perfectly clean, or they might have dozens more vulnerabilities. We can’t tell because the scanner doesn’t support them, and we’re too lazy to write our own scanner for APT packages.

We want control

Currently, we run our builds with GitHub runners since we initially created our product with them, but moving away from them is a priority.

We aren’t moving away because they’re bad, but because they’re wrong for us. We’re security focused and we want a minimum footprint. We want to know exactly what’s in our environment. We want control over updates and we want to keep the same standards for builds that we have for production.

GitHub runners optimize for convenience. They give you everything so you don’t have to think about what you need, which is great for getting started quickly but terrible for security.

You can build your own images, it takes more work, but the security benefits are massive.

If you wouldn’t run 1,681 packages in production, don’t build on it.

Our build environment is production. We’re treating it that way.

We utilize extended Berkeley Packet Filter (eBPF) Linux Security Module (LSM) hooks to secure builds and the GPU (We know that there are a lot of acronyms, but these are important since we will be using these two throughout the blog post).

The issue is that even though we utilize LSM hooks, a malicious actor could write their own LSM hooks to overwrite ours.

What Would an Attacker Do?

We will go over a couple of attacks that an attacker could use to access data in the GPU.

Note that in Unix, returning 0 signifies an OK. Keep that in mind when reading the following code snippets.

Always allow ptrace

A malicious user could attempt to hook lsm/ptrace_access_check and allow all ptrace.

They would use this to enable ptrace on protected processes to dump memory, inject code, etc.

#include “vmlinux.h”
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

SEC(”lsm/ptrace_access_check”)
int BPF_PROG(malicious_ptrace, struct task_struct *child, unsigned int mode) {
    return 0;
}

char LICENSE[] SEC(”license”) = “GPL”;

Always Allow Signals

An attacker could also try to hook lsm/task_kill and allow the killing of protected processes. For example, they could try to kill the Bomfather agent or protected executables via SIGTERM/SIGKILL.

#include “vmlinux.h”
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

SEC(”lsm/task_kill”)
int BPF_PROG(malicious_kill, struct task_struct *p, struct kernel_siginfo *info, int sig, const struct cred *cred) {
    return 0;
}

char LICENSE[] SEC(”license”) = “GPL”;

Beating Up the Bullies on the Playground

Our solution to these two attacks are based on a single idea that we leverage.

The kernel’s LSM caller stops on the first non zero return code. So, if any LSM hooks return a negative error, the kernel immediately denies the operation even if an earlier hook allowed it. You can see this in the Linux Kernel code: https://elixir.bootlin.com/linux/v6.17.1/source/security/security.c#L3722-L3738.

We have our own lsm/ptrace_access_check and lsm/task_kill, which will deny based on a policy provided to Bomfather.

So, if a malicious user were to try using these LSM hooks to inject code or kill the Bomfather process, they wouldn’t be allowed to, since denials take priority over allows in Linux.

This would also happen even if the malicious process was started before Bomfather started running. So, suppose a malicious process allowed all access via lsm/ptrace_access_check and lsm/task_kill before starting Bomfather. In that case, once Bomfather starts, it will deny access because of how Linux is designed.

Conclusion

In this example by placing Bomfather’s LSM hooks that return explicit denies for lsm/ptrace_access_check and lsm/task_kill, we force the kernel to respect our policy even if a malicious LSM came first.

Would the ingress controller restart cleanly? Would the cert manager remember to renew our certificates? Would our LoadBalancer IP suddenly change and take everything down? We weren’t engineering anymore. We were becoming YAML janitors.

Even after moving to Pulumi and writing our infrastructure in Go, we were still wrestling with the complexity of the underlying networking stack.

So we tried something radical. We ripped out our entire ingress stack and replaced it with Cloudflare Tunnels. No more LoadBalancers, no more ingress controllers, no more cert manager. And somehow, our infrastructure is already more reliable.

Here’s why we made this decision.

The Breaking Point

Our setup looked like every other small engineering team’s Kubernetes cluster. We had a LoadBalancer that cost us fifteen bucks a month, pointing to an Nginx ingress controller. Behind that sat the cert-manager, which had one job, keeping our SSL certificates valid, and it failed at this very often. Also, the CRDs are a nightmare.

The architecture was textbook:

Every component in red was a potential failure point. And they failed. Often.

The Radical Simplification

Here’s the thing about Kubernetes networking, we treat it like it’s necessary complexity. But what if it isn’t?

What if, instead of exposing our services to the internet and managing all the security and routing ourselves, we just didn’t expose them at all?

That’s the core insight behind Cloudflare Tunnels. Your services stay completely private. No public IPs. No open ports. Instead, you create an encrypted tunnel from your cluster to Cloudflare’s edge network:

Traffic comes in through Cloudflare’s massive global network, gets routed through your tunnel, and reaches your services. But from the internet’s perspective, your services don’t exist. They don’t allow scanning, resist direct DDoS attacks, and remain reachable only through Cloudflare.

So, in the end, Cloudflare tunnels made our infrastructure simpler and it significantly increased our security posture.

The Implementation

I was skeptical it could really be this simple. It was.

The irony here is that we actually use Pulumi and write our infrastructure in Go, not YAML. Part of escaping YAML hell was adopting real code for infrastructure. So our tunnel configuration looks very similar to:

tunnel, err := cloudflare.NewTunnel(ctx, “api-tunnel”, &cloudflare.TunnelArgs{
    AccountId: pulumi.String(accountId),
    Name:      pulumi.String(”production-tunnel”),
    Secret:    pulumi.String(base64.StdEncoding.EncodeToString(tunnelSecret)),
})

_, err = cloudflare.NewTunnelConfig(ctx, “tunnel-config”, &cloudflare.TunnelConfigArgs{
    TunnelId:  tunnel.ID(),
    Config: &cloudflare.TunnelConfigConfigArgs{
        Ingress: Cloudflare.TunnelConfigIngressArray{
            &cloudflare.TunnelConfigIngressArgs{
                Hostname: pulumi.String(”api.yourcompany.com”),
                Path:     pulumi.String(”/api/*”),
                Service:  pulumi.String(”<http://your-api-service.default.svc.cluster.local:8080>”),
            },
            // ... other routes
        },
    },
})

The concept is simple. You create a tunnel, configure routing, and deploy the daemon. The daemon creates an outbound connection to Cloudflare, and suddenly your services are available globally with HTTPS, DDoS protection, and a CDN.

No LoadBalancer to provision. No ingress controller. No cert manager to babysit.

The Human Cost of Complexity

Look, saving $25 a month on infrastructure wasn’t the point. That’s coffee money.

The real cost is human attention. As an ultra small team, we can’t afford to lose half a day to networking nonsense. Every hour spent debugging why the ingress controller is returning 502s is an hour not spent talking to users or shipping features.

We did the math, If each of us loses just four hours a month to networking issues, that’s 144 hours a year. That’s almost a month of engineering time, just gone.

With Cloudflare tunnels, those problems vanish. New services just work. Certificates renew themselves (or rather, Cloudflare handles it). Kubernetes upgrades don’t break our networking.

We’re getting that month back. We’re spending it on shipping features.

What We’re Expecting

Based on our testing and research, here’s what we’re anticipating:

• Better uptime (Cloudflare’s infrastructure is probably more reliable than our single-node ingress controller)

• Zero certificate renewal headaches

• Faster deployments (no waiting for LoadBalancer provisioning)

• Built-in DDoS protection and global CDN

Most importantly, we expect to stop thinking about networking. It should become invisible infrastructure, the way it’s meant to be. When we need a new service exposed, we add three lines to a config file. When we need to change routing, we update that same file. Everything else just happens.

The Gotchas

This isn’t a silver bullet. There are a few things to watch out for:

WebSockets need some attention. Cloudflare supports them, but there are connection limits on the free tier, and some features require paid plans. We made sure to understand these limits before migrating, and we recommend you do the same.

The per request pricing might add up if you’re pushing serious traffic and hundreds of millions of requests.

You’re coupling yourself to Cloudflare. If they have an outage, you have an outage. Honestly, though, their uptime is probably better than our Kubernetes cluster’s.

Some enterprise features, like SAML authentication or complex traffic policies, might require their higher-tier plans. Again, do your research.

Should You Do This?

If you’re a small to medium team drowning in Kubernetes networking complexity, you should definitely try this. Set up a tunnel for a non-critical service. Spend an hour with it. See how it feels to deploy something without thinking about ingress controllers.

If you’re at a massive scale or have unique networking requirements, you probably need the control that traditional infrastructure provides. But even then, consider it for your development environments.

The question isn’t whether this approach is perfect. It’s whether it’s better than what you’re doing now.

A Note on Transparency

Full disclosure, Cloudflare provides us with credits for its services through its startup program. But we made this architectural decision before any partnership existed, and we’d make the same choice if we didn’t have credits. The time saved not debugging ingress controllers is worth far more than what we’d spend on tunnels.

The Bottom Line

We’re replacing our entire Kubernetes ingress stack with a few lines of Go code in Pulumi. Our infrastructure is getting simpler, faster, more reliable, and more secure. We’re spending our time building features instead of debugging networking.

Sometimes the best solution isn’t to manage complexity better. It’s to eliminate it entirely.

If you are an attacker (who does illegal stuff for a living) and you are reading this blog post... sorry for making your life just a bit harder ;)

Link to demo source code: https://github.com/bomfather/tools/tree/main/ebpf-map-attacker

Why Are We Even Doing Any of This?

eBPF maps are used to almost all eBPF programs because they are essentially the only way for data to be moved from the kernel space to the userspace, and the main way to store information between eBPF hooks.

If a bad actor can read or write data in these maps, it completely compromises the security posture of the entire eBPF tool.

Since the userspace and kernel space must interact with these eBPF maps, they are shared objects, and any elevated-privilege user (sudo, CAP_SYS_ADMIN, etc.) can access these maps.

The Easy Way (Not Really)

It’s pretty easy in theory to attack these maps, with a big caveat. You must have elevated permissions, at a minimum CAP_BPF. Once you have that, just hit the BPF map syscall, and you can edit any map you want. Simple.

Okay, not simple at all. After you get elevated permissions, yes, it’s simple, but getting them is hard.

However, we’ve been seeing a decent number of software vulnerabilities leading to sudo access for bad actors, like NvidiaScape and even sudo itself! So, we can’t assume these attacks can’t happen to our eBPF maps.

If your system contains enough valuable data and you have a kernel-level security solution, it should ideally be able to protect against elevated-privilege attackers.

How Do We Stop Sudo?

Stopping this root-level actor may seem impossible since sudo is… sudo. But eBPF can stop this, we can secure our own eBPF program with eBPF.

We can write an LSM hook(lsm/bpf_map) in our ebpf code to block access to the bpf map open syscall and only allow our userspace program to access it. With this, nobody can call these syscalls and attack us, not even sudo.

We Still Aren’t Safe

Okay, we’ve stopped access to the bpf_map syscall using eBPF, so they shouldn’t be able to access our maps, right?

Well, bad actors can still attack our eBPF maps in a much more roundabout, limited, and sneaky way, one that’s a lot cooler than the first attack. They can attack a single, very useful type of eBPF map: BPF_F_MMAPABLE maps.

What are BPF_F_MMAPABLE maps

Most eBPF maps are stuck behind bpf syscalls, but not all. Sometimes eBPF syscalls take too long.

For example, if we are reading a lot of data off an eBPF map from the userspace, every single time we want to read data we hit a bpf syscall. Since the userspace cannot directly read kernel memory (our eBPF map), the bpf syscall copies that data from the kernel space to the userspace, which is expensive.

Instead, we can use a BPF_F_MMAPABLE map, which allows us to bypass all that. This map allows us to mmap with that memory region without bpf syscalls, removing the heavy copying operation. That means less overhead and less context switching. For large amounts of data, it’s worth it.

Now, did you catch it? We can access the map without a BPF syscall. That means these maps, while very useful for high-performance eBPF programs, also open a new attack surface.

Sneaky Stuff with ptrace

Now that we know we can read and modify BPF_F_MMAPABLE maps, the only thing we need is a file descriptor. Without it, we can’t mmapthe bpf map.

But the only way to get this file descriptor is to hit the bpf syscall, and if access to that syscall is blocked, how can a bad actor get a file descriptor? Instead of calling a bpf syscall to get the file descriptor, they can steal it.

A malicious actor can ptrace our userspace process (the eBPF agent) and steal the file descriptor from our benign program. Once they have it, they can dump the map’s contents and do whatever they want with it.

These BPF_F_MMAPABLE maps are usually used for logging, but if attackers can deactivate logging, you essentially become blind to their intrusion. If you don’t know about it, you can’t stop them.

When It Doesn’t Work the First Time, Get a Bigger Hammer

We already blocked off the BPF syscall, but that won’t stop this attack, and since we’re an eBPF program, we can just block all ptraceattempts on us. Now, nobody can directly access our maps or steal the file descriptors.

Okay, Cool… But I’m Lazy. Now What?

At a minimum, you should secure the BPF syscall. It’s essentially three lines of code, something like this, and without it, your code becomes a (very fat) sitting duck:

SEC(”lsm/bpf_map”)
int BPF_PROG(lsm_bpf_map, struct bpf_map *map, fmode_t fmode) {
	u32 pid = bp f_get_current_pid_tgid() >> 32;
	if (pid == mypid) {
		return 0;
	}
	return -EPERM;
}

You don’t have to stop ptrace if you don’t use BPF_F_MMAPABLE maps (though ChatGPTing a simple eBPF hook can’t be too hard). You don’t really need BPF_F_MMAPABLE maps if you aren’t logging tons of data from kernel to user space.

We may have secured these maps, but an attacker can still just kill our eBPF agent’s userspace process, which would automatically stop all our security. If your interested, our next post will be about how we can secure our agent from death itself?

- /bin/containerd-shim-runc-v2

- /bin/k3s

- /bin/cni

- /bin/aux/xtables-nft-multi

The Threat

LD_PRELOAD is an environment variable that allows you to load a shared library before the program starts. This is a powerful feature that can be used to hook system calls and intercept file operations. There are other similar variables like LD_LIBRARY_PATH.

This is not ENV variables alone. There are things like /etc/ld.so.preload that can be used to load a shared library before the program starts.

Linux built this feature to allow developers to load shared libraries before the program starts so that they can debug and test their code.

Our implementation of this exploitation code is in our Github Repo.

Why Solana?

This is not just a Solana problem. This is a problem for any application that loads credentials from a file. The insider threat is real and a problem for any organization. This is another way to steal private keys. There is too much at stake not to be careful.

The Vulnerability

Now, most of us write userspace applications, and use libraries like glibc to handle the file operations, so we never really think about this issue. glibc does the heavy lifting for us, but what if we hook into the file operations and intercept them before glibc does its thing? This is what LD_PRELOAD allows us to do. It is a way to hook into a library call.

In our case, we did this by hooking into the file operations and intercepting them before glibc did its thing. When the validator read its keypair files from disk, we intercepted the close() call and made a copy before the file descriptor closed.

We wrote a malicious shared library that hooks into the file operations. We also call the real close() function, so the validator continues normally, the user has no idea that this is happening.

How the Attack Works

Our malicious library does something clever, it hooks the close() system call.

Why close() and not open() or read()? Because when a file is closed, we know the application is done with it. We can check what file was just accessed by looking at /proc/self/fd/{fd} before the file descriptor is closed.

The hook is surprisingly simple - about 30 lines of C. Here’s the core concept:

int close(int fd) {
    // 1. Get pointer to real close() using dlsym(RTLD_NEXT, “close”)
    // 2. Read /proc/self/fd/{fd} to see what file this is
    // 3. If path contains “solana-run”, copy the file to /tmp/stolen-validator-data
    // 4. Call real close() so the validator continues normally
}

That’s it. No privilege escalation, kernel exploits, or complex techniques. Just intercepting a standard library call that every program uses.

The key trick is using /proc/self/fd/{fd}, a Linux feature that lets you see what file a file descriptor points to. Before the file descriptor closes, we check if it’s one of the Solana keypair files. If it is, we make a copy.

A More In-Depth View

Both of the following are based on our attack code https://github.com/bomfather/tools/tree/main/ld-preload.

Method 1: Environment Variable (LD_PRELOAD)

1. Compile malicious.so and inject it into the validator container

2. Set LD_PRELOAD=/tmp/malicious.so when starting the validator

3. When the validator opens keypair files, our library is already loaded

4. Every time close() is called, we intercept it

5. Check if the file path contains “solana-run” (the ledger directory)

6. If yes, copy the file to our exfiltration directory

7. Call the real close() so the validator continues normally

Method 2: Persistent (/etc/ld.so.preload)

1. Compile malicious.so inside the container

2. Write /tmp/malicious.so to /etc/ld.so.preload

3. Start the validator (library loads automatically)

4. Same interception and exfiltration process

5. Affects ALL processes in the container, not just the validator

Deploying the Attack

Environment variable method:

LD_PRELOAD=/path/to/malicious.so ./program

Persistent method:

echo “/path/to/malicious.so” > /etc/ld.so.preload
./program  # Library loads automatically

It is as simple as that.

Why This Matters: Scope and Implications

Can containers be exploited?

Yes. Containers don’t protect against LD_PRELOAD attacks because:

• The environment variable is set within the container’s namespace

• /etc/ld.so.preload is a file inside the container

• The process inside the container runs the library inside the container

Container isolation doesn’t help when the attack comes from inside the container.

Do I need to be root?

It depends on the method:

LD_PRELOAD environment variable:

• No root needed for your own processes

• Can be set by any user for processes they start

• That is the scary part

/etc/ld.so.preload file:

• Requires root/privileged access to modify the file

• But once set, it affects ALL processes system-wide

• More dangerous, but requires privilege escalation first

Conclusion

Remember the scenario at the beginning? The drained validator with everything “in order”? This is how it happens, with a single environment variable, no root access needed, and no alerts triggered. Just silent exfiltration of private keys while the validator runs normally.

The scary part isn’t the complexity of the attack but the simplicity. LD_PRELOAD is a legitimate debugging feature. File access by the validator process is expected behavior.

Check if your EDR agent is handling things like this.

Complete source code and demo available at https://github.com/bomfather/tools/tree/main/ld-preload. The steps to run it are in the README.md.

Disclaimer: This tool is for educational and authorized security testing only. Unauthorized access to computer systems is illegal. See LICENSE for complete terms.