Mayhem for Code can run on the cloud, but, when you're testing critical/sensitive/confidential code, you can make it more difficult for malicious actors to access Mayhem's findings: you can deploy Mayhem on-prem.
In earlier days, to install Mayhem for Code on-prem, we co-ordinated with the IT staff of our customers to ensure that they have the hardware, OS, and networking set up for Mayhem.
These days, to get Mayhem for Code on your premises:
we can provision hardware for you, if you'd like.
we can provision OS for you, if you'd like.
In our experience, RedHat Enterprise Linux - and its derivatives like Oracle Linux - are quite welcome in the IT environment of our customers. So, we looked into bundling Oracle Linux (OL) as we distribute Mayhem. (Okay, we initially started with CentOS, but have since moved on to Oracle).
This "OS family" can be installed using DVD ISOs that include the Anaconda-installer. Anaconda-installer can be automated using kickstart. We can kickstart the OS and networking set-up. We can also (after adding Mayhem bits into the ISO image) use kickstart to automatically install Mayhem for Code.
The resulting ISO image was 15 GB. 10 GB comes from the "full ISO" to install Oracle Linux (8.5 at the time of this writing), which we use as a base. This "full ISO" carries enough packages to install LibreOffice, video conferencing software, etc. While these are useful software, we don't need to install these on machines set apart for Mayhem.
Can we trim the size of the ISO image?
In the "full ISO" for OL 8.5, we see that the AppStream directory takes up 8.4 GB. If these were "normal" files & directories, we would "normally" try rm [-r]. Can we do that with ISO images?
We can, with xorriso ("X/Open Rock Ridge ISO" tool). We can tell it to -rm $PATH ... and/or -rm_r $PATH ....
At the time of this writing, we can install xorriso using yum in OL and apt-get in Debian/Ubuntu without adding additional package archives/repositories. We can also install it in macOS with Home-brew. We can also use xorriso in a Docker container (using, for example, oraclelinux or ubuntu images), but we noticed that disk I/O performance is significantly slower than using xorriso outside the container.
This informs xorriso to use OracleLinux-R8-U5-x86_64-dvd.iso as "input drive" and sets "output drive" to out.iso.
-boot_image any replay tells xorriso to also copy the bits that make the "full ISO" boot-able.
If you're following along, at this point, we should have an out.iso that is basically equal to the input.
Before telling xorriso to start removing files/folders, we first need to figure out what we need.
We can do this by manually going through the installation process. We should, however, hold off from setting up the network connection: going through an "off-line" installation will force Anaconda-installer to only use RPM files from the DVD (instead of fetching new/updated ones from the internet).
After a successful installation of the OS, Anaconda-installer creates a /root/anaconda-ks.cfg file containing kickstart configuration. We can use this file to automate what we just manually installed. More about this file in a bit.
We'll also need the list of RPM packages that we have just installed. We can get this list from yum list --installed. Here's a sample output:
we first rm -f out.iso to remove any existing out.iso, or else xorriso will "append" to the ISO image, instead of creating a new ISO image.
-rm_r takes a list of folders. We tell xorriso that we're done with the list with --, before giving additional command (-map in the snippet above).
we use -map to copy local files/folders into the ISO image.
At this point, we should have a much smaller ISO image, but we have yet to automate the installation. We have /root/anaconda-ks.cfg. We can edit this file to further adjust how we want to automate Anaconda.
We then need to:
copy this kickstart config file to the ISO image, and
edit the boot menus, to tell Anaconda installer where it can find the kickstart config file.
The "full ISO" can be used to boot (older) BIOS and (newer) EFI systems. We'll need to edit the boot menu for both. Assuming we add the kickstart config file into the ISO image at /anaconda-ks.cfg, we'll need append inst.ks=hd:LABEL=OL-8-5-0-BaseOS-x86_64:/anaconda-ks.cfg to the "kernel command line" (which the Linux kernel will eventually pass to Anaconda-installer).
The edited BIOS boot menu at /isolinux/isolinux.cfg should end up looking like:
. . .
menu label ^Install Oracle Linux 8.5.0
append initrd=initrd.img inst.stage2=hd:LABEL=OL-8-5-0-BaseOS-x86_64 quiet inst.ks=hd:LABEL=OL-8-5-0-BaseOS-x86_64:/anaconda-ks.cfg
. . .
The edited EFI boot menu at /EFI/BOOT/grub.cfg should end up looking like:
-update edited_isolinux.cfg /isolinux/isolinux.cfg will replace/overwrite /isolinux/isolinux.cfg in the ISO image with the edited_isolinux.cfg we supplied. We similarly update grub.cfg.
While we're at it, we also changed the volume ID (-volid) of the ISO image from OL-8-5-0-BaseOS-x86_64 to MAYHEMDRIVE. And, yes, we'll have to update the BIOS and EFI boot menu *.cfg files accordingly.
We use xorriso to do a few more things for us:
We use a few more packages that are not included in the "full ISO". So we used createrepo to create additional yum repositories and had xorriso -map it to the ISO image.
We also added Mayhem to the ISO image.
Some readers might notice that some of what we've done so far is quite similar to mkksiso. And that's partly true: mkksiso uses xorriso under the hood (in combination with other tools) to:
edit boot menus to append kernel command-lines with kickstart config, and
add more content into the ISO image.
But mkksiso doesn't trim the ISO image. Also, mkksiso (part of lorax) is not as easy to install as xorriso in Debian/Ubuntu or macOS.
There are still other ways to create Anaconda-kickstart bootable-DVDs. Notably, Fedora (on which RHEL is based) uses pungi.
We would understand if some readers consider what we're doing with xorriso as a "hack". But it works for us. We started with a 15 GB ISO image. We were able to trim it down to ~4 GB for Mayhem for Code 1.17. So, these days, we've got an ISO image that can automatically install and configure OS and Mayhem, so our customers can start finding and fixing vulnerabilities on-prem, making it that more difficult for malicious actors.