After using Qubes-Whonix as a daily driver for a while (and feeling "reasonably secure"), I decided to experiment with using it as a security research lab. Apparently this is not a common use, based on the limited discussion on forums or elsewhere. This made me wonder if it was an obviously bad idea and I was just missing somethin, or whether it is actually a viable setup. So I built a Qubes lab in order to find out.

This is not a how-to so much as a post about architectural decisions, tradeoffs, and unexpected behaviors specific to using Qubes as a lab. Primarily for reverse engineering malware, but flexible and extensible enough to be pointed at whatever.

Different people will have different needs and uses, necessitating unique design choices, configurations, and tooling.
Do what you want. In doing so, hopefully some of this might be useful to you.

Qubes is a desktop operating system that uses isolated Xen-based virtual machines to achieve security by compartmentalization. One of the project's tenets is to "distrust the infrastructure and be free". Being free is cool, but in this context, the more important dictate is to distrust the infrastructure, meaning anything and everything external to the host machine. Off-the-shelf testing distributions like Kali, Parrot, or REMnux, prioritize curating toolsets first and user security/anonymity/privacy (i.e., safety) second, at least at a level required for riskier work. Maybe the fairer statement is that Qubes appears to take user safety more seriously than other comparable OSes. To be even fairer, this is just a hunch (for now).

In any case, my design priorties were, in order: 1. isolation, 2. repeatability, 3. extensibility, and 4. scalability.
For color, that means 1. collect and observe dangerous software without impacting my own environment, 2. set up and tear down ad nauseum without friction or fatigue; automate as much boring stuff as possible, 3. build or install new capabilities easily; perform updates and upgrades easily; remove unwanted software easily, and 4. accomodate a large enough sample size to do interesting research, efficiently. Unlike traditional malware labs that often only provide one or two layers of isolation, i.e., between host(s) and virtual machine(s), without much well-defined scoping, I attempted to maximize isolation in a pipeline with six separate application (AppVM) Qubes. These, in turn, are based on one of three templates that act as golden images for configuring an operating system and installing software. Persistent filesystems only occur in AppVMs. Execution takes place in ephemeral, disposable Qubes (burn after use). For performance and reducing attack surface, I wanted each Qube - whether template, appVM, or disposable - to be as minimal as possible. Only two qubes have access to networking: lab-ingest for acquiring samples and lab-net-dvm for dynamic analysis.

Storage Planning and Management turned out to need some trial and error. First, a 20GB archive from VX Underground failed to download because of insufficient disk space configured in the lab-ingest AppVM. Easy enough to change, but there was a cascading effect across Qubes after downloading and extracting several archives: lab-ingestcan and should be purged regularly to preserve space, but the allocations needed for lab-samples,lab-evidence, and lab-collector may vary. For instance, for a given behavior or file type or research question, I may need an arbitrary range of space between 300GB-1TB in lab-samples alone to scan across a large volume of files.

Tool Installation requires temporary networking for templates, which is not recommended. Calls from a package manager or downloads from Tor Browser could (should?) have been done in a disposable Qube, with the software then transferred via qvm-copy or qvm-move. This might be a change I make later, for the sake of attack surface, the next time I grab more tooling.

Some of this is just plain old GNU/Linux/UNIX stuff that comes with the territory. Some is a matter of preference, taste, or aeshethics. I will not dive into too much, but will mention in passing.

Symlinks may be needed here and there to make a program available as a global command. For example, symlink /usr/local/bin/ghidra to /opt/ghidra/ghidraRun allows loading Ghidra without needing to type its path. Small things, but worth the effort.

Dependency Resolution is always a thing: whichever versions of tools you want may be incompatible with stable releases of whatever distro(s) you are using in Qubes. It is what it is.

Telemetry: auditd requires targeted rules, since defaults generate huge amounts of noise. eBPF is cool but bpftrace is probably okay enough at live syscall tracing and process monitoring for the vast majority of cases. Zeek, raw PCAP, or both - who's to say, really?

Automation
Most everything so far has been manual effort, for love of the game. Essentially, all of it could be automated. Cron jobs in dom0 could orchestrate downstream cron jobs in the AppVMs to perform any of the pipeline functions, including triage, reporting, and generating signatures. The disassembly and debugging is the fun part, though, so I may keep that for some old fashioned human effort.