Bryan Drewery

The best way to sandbox a web application is in a FreeBSD jail. Taking this a step further and placing a caching nginx reverse proxy in front of it can increase performance. The backend application server does not need to be on the same server as the internet-facing application server.

To simplify server management I create “meta” packages in FreeBSD ports that can generate a package with only dependencies on other packages. This allows to me to just install this 1 package on the target server and have it pull in all of the packages that I want on there. I assign each server specific “roles” and only install 1 or 2 packages per server depending on which roles they fulfill. The roles may be one of “dev”, “web”, “ports-dev”, “jail”, etc. This ensures that all servers fulfilling specific roles will always have the proper packages installed. For some applications, I use a dedicated jail with a meta package that only pulls in the required dependencies for that application to run. For instance, on a PHP application jail, the meta package may pull in nginx, php, eaccelerator, git, etc.

For additional security layers and separation, I run my web applications inside of dedicated jails. This has been an ongoing progression for me. I will layout where I started, where I progressed, and how I do it now.

For years I have ran into this error while running make installworld:

# make installworld
[...]
===> sys/boot/i386/boot2 (install)
cc -Os  -fno-guess-branch-probability  -fomit-frame-pointer  -fno-unit-at-a-time  -mno-align-long-strings  -mrtd  -mregparm=3  -DUSE_XREAD  -DUFS1_AND_UFS2  -DFLAGS=0x80  -DSIOPRT=0x3f8  -DSIOFMT=0x3  -DSIOSPD=9600  -I/usr/src/sys/boot/i386/boot2/../../common  -I/usr/src/sys/boot/i386/boot2/../btx/lib -I.  -Wall -Waggregate-return -Wbad-function-cast -Wcast-align  -Wmissing-declarations -Wmissing-prototypes -Wnested-externs  -Wpointer-arith -Wshadow -Wstrict-prototypes -Wwrite-strings  -Winline --param max-inline-insns-single=100   -march=i386 -ffreestanding -mpreferred-stack-boundary=2  -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -msoft-float -m32 -std=gnu99    -S -o boot2.s.tmp /usr/src/sys/boot/i386/boot2/boot2.c
sed -e '/align/d' -e '/nop/d' < boot2.s.tmp > boot2.s
rm -f boot2.s.tmp
cc  -m32 -c boot2.s
ld -static -N --gc-sections -nostdlib -m elf_i386_fbsd -Ttext 0x2000 -o boot2.out /usr/obj/usr/src/sys/boot/i386/boot2/../btx/lib/crt0.o boot2.o sio.o
objcopy -S -O binary boot2.out boot2.bin
btxld -v -E 0x2000 -f bin -b /usr/obj/usr/src/sys/boot/i386/boot2/../btx/btx/btx -l boot2.ldr  -o boot2.ld -P 1 boot2.bin
btxld: No such file or directory
*** [boot2.ld] Error code 1

Stop in /usr/src/sys/boot/i386/boot2.
*** [realinstall] Error code 1

This has come, up, before.

Most of the posts mention bad timestamps or incorrect date. I’ve always been running ntpd though and running make buildworld before make installworld.

My longterm workaround was to make -C /usr/src/sys/boot/i386 before running make installworld.

Recently this workaround stopped working for me. Looking into it more, I realized that I was applying custom patches to the src tree and then removing them before make installworld. This was changing timestamps of source files, causing a rebuild during make installworld. Changing my scripts to leave the patch applied until after everything is installed solves it.