Sandboxing PHP part 2

The best way to sandbox a web application is in a FreeBSD jail. Taking this a step further and placing a caching nginx reverse proxy in front of it can increase performance. The backend application server does not need to be on the same server as the internet-facing application server.

A typical setup is:

        | Frontend server     |
        | nginx reverse proxy |
|           Application server           |
|        |
|  +---------------+ +----------------+  |
|  |  Jail: Blog   | |  Jail: Webmail |  |
|  |---------------| |----------------|  |
|  |  php-fpm      | |  php-fpm       |  |
|  |  nginx        | |  nginx         |  |
|  +---------------+ +----------------+  |

This setup allows having the frontend server cache static content to lessen the load on the backend application servers and avoids loading content dynamically through the scripting or CGI interfaces. It also allows segregation for the application servers from the WAN. Each jail can optionally be given WAN access or can be kept as LAN only.

The frontend server and backend application server can be combined if wanted:

|           Application server           |
|        +---------+-----------+         |
|        | nginx reverse proxy |         |
|        +---------+-----------+         |
|                 / \                    |
|        |
|  +---------------+ +----------------+  |
|  |  Jail: Blog   | |  Jail: Webmail |  |
|  |---------------| |----------------|  |
|  |  php-fpm      | |  php-fpm       |  |
|  |  nginx        | |  nginx         |  |
|  +---------------+ +----------------+  |

Reverse proxy / Frontend server

The frontend server is the only one that needs WAN access. It will only need nginx installed which will forward all requests to the backend application servers. A typical configuration for the reverse proxy is:

# /usr/local/etc/nginx/vhosts/blog.example.com

server {
    server_name  blog.example.com;

    # Cache all static content for 2 days
    location ~* ^.+.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$ {
        proxy_cache_valid 200 201 302 120m;
        expires 2d;
        proxy_cache one;

    location / {
        proxy_read_timeout 40;

This will forward all requests to the backend server for blog.example.com on It will also cache all static content on the frontend server for 2 days which will lessen the load on the backend application server.

Application Server

Each application that needs to be setup will have its own jail and its own nginx process inside of that jail if needed. Some applications have their own network daemons that will not require using nginx for FastCGI.

First the jail IPs need to be added to the host. Add them to the interface's address list in /etc/rc.conf:

# /etc/rc.conf

Then restart networking:

host# service netif restart && service routing restart

Jail setup

The sysutils/ezjail port is the easiest utility for setting up the jail on FreeBSD. The jail needs to be created from the host and then populated with packages from inside of the jail. More details for this can be found on the ezjail website.

# Create the base jail
host# ezjail-admin update -i

# Create each application jail
host# ezjail-admin create -c zfs -r /tank/jails/blog blog
host# ezjail-admin create -c zfs -r /tank/jails/webmail webmail

# Start the jails
host# ezjail-admin start blog
host# ezjail-admin start webmail

# Enable ezjail for next boot
host# echo ezjail_enable=YES >> /etc/rc.conf

Next packages can be installed using pkg from the host system. This assumes that meta packages have been setup on the remote repository as described in managing FreeBSD servers with meta packages. This also assumes that the meta package includes all needed dependencies including www/nginx.

host# pkg -j blog install local/blog
host# pkg -j webmail install local/webmail

Application setup

Enter the jail from the host with ezjail-admin console blog. From there nginx, PHP-FPM and the application can all be setup.

This example assumes that PHP-FPM will be used with a PHP application.


nginx needs to be setup to use FastCGI.

# /usr/local/etc/nginx/vhosts/blog.example.com

server {
    server_name  blog.example.com;

    location / {
        alias /usr/local/www/blog/;
        index index.php index.html index.htm;

    location ~ /(.*\.php)$ {
        root /usr/local/www/blog/;

        # Filter out arbitrary code execution
        try_files $uri = 404;
        location ~ \..*/.*\.php$ {return 404;}

        fastcgi_pass   unix:/var/run/php-fpm-www.sock;
        include        fastcgi_params;

# /usr/local/etc/nginx/fastcgi_params

fastcgi_index  index.php;

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

fastcgi_param  PATH_INFO          $fastcgi_path_info;
fastcgi_param  PATH_TRANSLATED    $document_root$fastcgi_path_info;

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

# /etc/rc.conf

Start nginx with service nginx start.


PHP-FPM will start a daemon to listen for local connections from nginx. All that needs to be done for PHP-FPM is to configure it to listen on a UNIX socket and to enable it on boot.

# /usr/local/etc/php-fpm.conf
# Replace listen line with:
listen = /var/run/php-fpm-$pool.sock

# /etc/rc.conf

Start PHP-FPM with service php-fpm start.


Configure the application itself as needed.

Updating jails

Occasionally the jail's package can be updated from the host:

host# pkg -j blog upgrade

Wrap up

By moving each application into its own jail, security and performance can both be improved greatly. mod_php with Apache could be used but is much more heavyweight in each jail.


Managing Role Based FreeBSD servers using meta packages and Poudriere

To simplify server management I create "meta" packages in FreeBSD ports that can generate a package with only dependencies on other packages. This allows to me to just install this 1 package on the target server and have it pull in all of the packages that I want on there. I assign each server specific "roles" and only install 1 or 2 packages per server depending on which roles they fulfill. The roles may be one of "dev", "web", "ports-dev", "jail", etc. This ensures that all servers fulfilling specific roles will always have the proper packages installed. For some applications, I use a dedicated jail with a meta package that only pulls in the required dependencies for that application to run. For instance, on a PHP application jail, the meta package may pull in nginx, php, eaccelerator, git, etc.

Packages are created from ports

To create meta packages, define a port that requires the actual ports that should be installed. Then build packages for those meta ports.

dev-meta port


This meta port will install git, cscope and vim. Which dependencies those pull in do not matter here.

PORTNAME=	local-dev-meta
CATEGORIES=     local

MAINTAINER=	local@localdomain.com
COMMENT=	Meta port for dev packages


RUN_DEPENDS+=	git>0:${PORTSDIR}/devel/git
RUN_DEPENDS+=	cscope>0:${PORTSDIR}/devel/cscope
RUN_DEPENDS+=	vim-lite>0:${PORTSDIR}/editors/vim-lite

do-install: build

.include <bsd.port.mk>

Note the RUNDEPENDS_ line is depending on package names, not binary names. Any version will satisfy the dependency.


Development meta port


The local category must be defined.


Building packages with Poudriere

Poudriere is a tool to build and test packages for FreeBSD. There is a detailed guide on creating pkgng repositories on the poudriere site, so I will only cover it briefly here.

Install poudriere on your build machine:

build# make -C /usr/ports/ports-mgmt/poudriere install clean

Configure poudriere:

build# cat >> /usr/local/etc/poudriere.conf
# Directory where the CCACHE_DIR is in the host
# Directory to store distfiles on the host
build# mkdir /usr/local/etc/poudriere.d
build# cat >> /usr/local/etc/poudriere.d/make.conf

Create a jail and import your existing /usr/ports tree as system:

# Create jail
build# poudriere jail -c -j 83amd64 -v 8.3-RELEASE -a amd64
# Add system's /usr/ports into poudriere
build# poudriere ports -c -F -f none -M /usr/ports -p system

Pick options for your meta package and dependencies:

build# poudriere options -p system local/dev-meta

Now build the packages from the meta port using the system ports tree:

build# poudriere bulk -j 83amd64 -p system local/dev-meta
====>> Creating the reference jail... done
====>> Mounting system devices for 83amd64-system
====>> Mounting ports/packages/distfiles
====>> Mounting ccache from: /usr/ccache
====>> Mounting packages from: /poudriere/data/packages/83amd64-system
====>> Mounting /var/db/ports from: /usr/local/etc/poudriere.d/options
====>> Logs: /poudriere/data/logs/bulk/83amd64-system/2013-07-21_14h14m27s
====>> Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
/etc/resolv.conf -> /poudriere/data/build/83amd64-system/ref/etc/resolv.conf
====>> Starting jail 83amd64-system
====>> Calculating ports order and dependencies
====>> pkg package missing, skipping sanity
====>> Cleaning the build queue
====>> Building 53 packages using 14 builders
====>> Starting/Cloning builders
====>> [01] Starting build of ports-mgmt/pkg
====>> Creating pkgng repository
Generating repository catalog in /packages: done!
====>> Cleaning up
====>> Umounting file systems
====>> Built ports: ports-mgmt/pkg devel/ccache textproc/xmlcatmgr archivers/unzip lang/perl5.14 net/p5-Socket textproc/iso8879 textproc/xmlcharent converters/libiconv devel/gettext devel/m4 devel/libtool net/p5-IO-Socket-IP security/libgpg-error security/p5-Net-SSLeay textproc/docbook-410 textproc/docbook-420 textproc/docbook-430 textproc/docbook-440 textproc/docbook-450 textproc/docbook-500 textproc/docbook-sk textproc/docbook-xml textproc/docbook-xml-430 textproc/docbook-xml-440 devel/bison devel/boehm-gc devel/gmake textproc/docbook-xml-450 devel/pkgconf security/ca_root_nss security/p5-IO-Socket-SSL misc/getopt print/libpaper security/libgcrypt shells/bash textproc/docbook textproc/docbook-xsl textproc/libxml2 textproc/libxslt www/w3m ftp/curl lang/p5-Error lang/python27 mail/p5-Net-SMTP-SSL textproc/asciidoc textproc/expat2 textproc/xmlto devel/cscope devel/cvsps devel/git editors/vim-lite local/dev-meta

====>> [83amd64-system] 53 packages built, 0 failures, 0 ignored, 0 skipped
====>> Logs: /poudriere/data/logs/bulk/83amd64-system/2013-07-21_14h14m27s

The /poudriere/data/packages/83amd64-system directory now contains the pkgng repository that needs to be served. This can be done over NFS, Samba, HTTP, FTP, etc. It is best to serve the /poudriere/data/packages directory and create symlinks of the ABI name to the target. The ABI is a pkgng feature defined as OS:REL:ARCH:BITS. For instance, this build would be freebsd:8:x86:64.

build# ln -s 83amd64-system /poudriere/data/packages/freebsd:8:x86:64

The repository is now ready for use on the target servers.

Role based servers with packages

On the target server, the appropriate meta packages just need to be installed now.

First bootstrap the system with pkg if needed.

dev# mkdir -p /usr/local/etc
dev# echo 'PACKAGESITE=http://packages.domain.com/${ABI}' > /usr/local/etc/pkg.conf
dev# pkg -v
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg please wait
Installing pkg-1.1.4... done
If you are upgrading from the old package format, first run:

  # pkg2ng

Now the local/dev-meta package can be installed:

dev# pkg install local/dev-meta
digests.txz                                                        100%   57KB  57.1KB/s  57.1KB/s   00:00
packagesite.txz                                                    100%  323KB 323.3KB/s 323.3KB/s   00:00
Incremental update completed, 0 packages processed:
0 packages updated, 0 removed and 53 added.
The following 42 packages will be installed:

	Installing libiconv: 1.14_1
	Installing xproto: 7.0.24
	Installing renderproto: 0.11.1
	Installing libXdmcp: 1.1.1
	Installing libXau: 1.0.8
	Installing pkgconf: 0.9.2_1
	Installing libpthread-stubs: 0.3_3
	Installing kbproto: 1.0.6
	Installing expat: 2.0.1_2
	Installing freetype2: 2.4.12_1
	Installing tcl: 8.5.14_1
	Installing openssl: 1.0.1_8
	Installing db42: 4.2.52_5
	Installing perl: 5.14.4
	Installing pcre: 8.33
	Installing libssh2: 1.4.3_1,2
	Installing ca_root_nss: 3.15.1
	Installing p5-Net-SMTP-SSL: 1.01_1
	Installing p5-Error: 0.17020
	Installing curl: 7.31.0
	Installing sqlite3: 3.7.17_1
	Installing p5-Term-ReadKey: 2.30
	Installing cvsps: 2.1_1
	Installing cscope: 15.8a
	Installing gettext: 0.18.3
	Installing libxml2: 2.8.0_2
	Installing fontconfig: 2.9.0,1
	Installing gdbm: 1.10
	Installing python27: 2.7.5_1
	Installing vim-lite: 7.3.1314_2
	Installing libxcb: 1.9.1
	Installing libX11: 1.6.0,1
	Installing apr:
	Installing apache22-worker-mpm: 2.2.25
	Installing libXrender: 0.9.8
	Installing libXft: 2.3.1
	Installing serf: 1.2.1_1
	Installing subversion: 1.8.0_3
	Installing p5-subversion: 1.8.0_3
	Installing tk: 8.5.14_1
	Installing git:
	Installing local-dev-meta: 20130719

The installation will require 433 MB more space

53 MB to be downloaded

Proceed with installing packages [y/N]: y

Only 2 packages were directly installed, so only those 2 show as non-automatic and will not be removed by pkg autoremove:

dev# pkg query -e '%a = 0' '%o

This simplifies maintenance of the server and ensures all servers using this meta package will have the same packages installed on them. When needing to add or remove a dependency from the meta package, just update the /usr/ports/local/dev-meta/Makefile on the build server, bump the PORTREVISION or PORTVERSION and then rebuild with poudriere bulk. Once that is completed, run pkg upgrade and pkg autoremove on the target servers. This will install new dependencies, upgrading existing, and then remove any that are no longer needed on on that server.


Sandboxing PHP part 1

For additional security layers and separation, I run my web applications inside of dedicated jails. This has been an ongoing progression for me. I will layout where I started, where I progressed, and how I do it now.


Originally, I would run all applications under www user using apache+mod_php. This was the classic LAMP approach and the most simple approach to hosting a web application. It is also the most insecure if you are hosting more than 1 user or application. If you are running apache as root then you have effectively given root to the world.

The biggest problem with this is that every application is executing code as the user that apache is running as. So that "secure" config.php file with some user or application's private db credentials in can easily be read by another. This creates a very easy attack vector for taking over another site on a shared system. Just sign-up and read in the file. You now can grant yourself administrative rights on their application by connecting directly to the DB.

Even if you are not doing "shared hosting", this makes every application on your system as weak as the weakest application running under www.

A better approach is to create a dedicated user for every application, or to run an application under the user whos public_html it is in.


An option which was mostly viable up until 2009 was suPHP. It allows executing an application using a setuid wrapper using the php-cli interface. Due to its EoL status, maintenance history, requiring setuid root, and poor performance, I would not recommend using this for new projects.


The next step for me was primarily focused on improving the performance of suphp. This resulted in using mod_fcgi to spawn a CGI process for the application and interact with that. This avoided startup overhead. It's still poor though as it requires the setuid binary and is much more complex since it involved mod_fcgi and an extra wrapper script.


This setup works very well in terms of security and performance. A separate php-fpm instance is spawned for each site. I assign each application a dedicated user. The php-fpm processes run as this user and create a CGI interface for nginx to connect to. This can be taken 1 step further with each application in its own jail, to further protect the host system. The jails are configured without WAN access; they only have LAN access and only in a small subnet dedicated for nginx to connect to the jail with.

In the next part I will cover exactly how this is setup.


btxld: No such file or directory

For years I have ran into this error while running make installworld:

# make installworld
===> sys/boot/i386/boot2 (install)
cc -Os  -fno-guess-branch-probability  -fomit-frame-pointer  -fno-unit-at-a-time  -mno-align-long-strings  -mrtd  -mregparm=3  -DUSE_XREAD  -DUFS1_AND_UFS2  -DFLAGS=0x80  -DSIOPRT=0x3f8  -DSIOFMT=0x3  -DSIOSPD=9600  -I/usr/src/sys/boot/i386/boot2/../../common  -I/usr/src/sys/boot/i386/boot2/../btx/lib -I.  -Wall -Waggregate-return -Wbad-function-cast -Wcast-align  -Wmissing-declarations -Wmissing-prototypes -Wnested-externs  -Wpointer-arith -Wshadow -Wstrict-prototypes -Wwrite-strings  -Winline --param max-inline-insns-single=100   -march=i386 -ffreestanding -mpreferred-stack-boundary=2  -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -msoft-float -m32 -std=gnu99    -S -o boot2.s.tmp /usr/src/sys/boot/i386/boot2/boot2.c
sed -e '/align/d' -e '/nop/d' < boot2.s.tmp > boot2.s
rm -f boot2.s.tmp
cc  -m32 -c boot2.s
ld -static -N --gc-sections -nostdlib -m elf_i386_fbsd -Ttext 0x2000 -o boot2.out /usr/obj/usr/src/sys/boot/i386/boot2/../btx/lib/crt0.o boot2.o sio.o
objcopy -S -O binary boot2.out boot2.bin
btxld -v -E 0x2000 -f bin -b /usr/obj/usr/src/sys/boot/i386/boot2/../btx/btx/btx -l boot2.ldr  -o boot2.ld -P 1 boot2.bin
btxld: No such file or directory
*** [boot2.ld] Error code 1

Stop in /usr/src/sys/boot/i386/boot2.
*** [realinstall] Error code 1

This has come, up, before.

Most of the posts mention bad timestamps or incorrect date. I've always been running ntpd though and running make buildworld before make installworld.

My longterm workaround was to make -C /usr/src/sys/boot/i386 before running make installworld.

Recently this workaround stopped working for me. Looking into it more, I realized that I was applying custom patches to the src tree and then removing them before make installworld. This was changing timestamps of source files, causing a rebuild during make installworld. Changing my scripts to leave the patch applied until after everything is installed solves it.

Pages : 1