Nginx Worker Process SIGSEGV (Signal 11) with Passenger Module

[arch3x]

Nginx Worker Process SIGSEGV (Signal 11) with Passenger Module

Date: 2026-06-25
Platform: RHEL 9.8, CIS Benchmark Level 1, AWS EC2

---

Symptoms

• Nginx worker processes crash immediately on first HTTP request with signal 11 (SIGSEGV)
• Core dump shows crash in ngx_escape_uri() called from Passenger's prepare_request_buffer_construction()
• Passenger watchdog/core start fine, but workers die on any request

Root Cause

ABI mismatch between RHEL's nginx binary and Phusion's pre-built mod_passenger RPM.

RHEL 9's nginx (nginx-core-1.26.3-9.module+el9.8.0) is compiled with NGX_HTTP_CACHE=1, which adds an 8-byte pointer field (ngx_http_cache_t *cache) at offset 64 in struct ngx_http_request_s.

Phusion's nginx-mod-http-passenger-6.1.5-1.el9 RPM was compiled against nginx headers without NGX_HTTP_CACHE defined. This causes every struct field after offset 64 to be shifted by 8 bytes in the module's compiled code vs. the actual runtime binary layout.

Effect: When Passenger reads r->uri.data, it actually reads r->request_line.len (garbage pointer). Passing this to ngx_escape_uri() causes the segfault.

Field	Binary (actual)	Passenger module (expected)	Delta
cache	offset 64	not present	+8
method	offset 856	offset 848	+8
uri	offset 888	offset 880	+8

Fix (Manual CLI Procedure)

Recompile the Passenger nginx module from gem source against RHEL's actual nginx source+config:

# 1. Install build dependencies
dnf install -y gcc gcc-c++ make pcre2-devel openssl-devel zlib-devel \
  libxml2-devel libxslt-devel gd-devel perl-ExtUtils-Embed \
  gperftools-devel ruby-devel curl-devel

# 2. Download and prep RHEL nginx SRPM
yumdownloader --source nginx-core
rpm -ivh nginx-core-*.src.rpm
cd ~/rpmbuild
rpmbuild -bp SPECS/nginx.spec

# 3. Configure with RHEL's exact flags + Passenger dynamic module
cd ~/rpmbuild/BUILD/nginx-1.26.3
eval ./configure $(nginx -V 2>&1 | grep -oP '(?<=configure arguments: ).*') \
  --add-dynamic-module=/usr/local/rvm/gems/ruby-3.4.2/gems/passenger-6.1.5/src/nginx_module

# 4. Build just the modules
make modules

# 5. Install the correctly-compiled module
cp objs/ngx_http_passenger_module.so /usr/lib64/nginx/modules/ngx_http_passenger_module.so
cp objs/ngx_http_passenger_module.so /usr/share/nginx/modules/ngx_http_passenger_module.so

# 6. Restart nginx
nginx -t && systemctl restart nginx

Verification

# Confirm no SIGSEGV in logs
tail -20 /var/log/nginx/error.log | grep -c 'signal 11'  # should be 0

# Confirm app responds
curl -sk https://localhost/ | head -1  # should return HTML

Prevention

• Do NOT use Phusion's pre-built nginx-mod-http-passenger RPM on RHEL 9 — it has an ABI mismatch
• Always compile the Passenger nginx module from source against the installed RHEL nginx source
• Pin nginx and passenger versions in Ansible; recompile module on any nginx update
• Consider adding a molecule test that verifies nginx workers remain stable after startup

Affected Packages

Package	Version	Source
nginx-core	1.26.3-9.module+el9.8.0+24374+c0e15309.1	RHEL AppStream
passenger	6.1.5-1.el9	Phusion repo
nginx-mod-http-passenger	6.1.5-1.el9	Phusion repo (BROKEN)
passenger gem	6.1.5	RVM ruby-3.4.2

[arch3x]

This is the same class of bug as #2662 (Ubuntu's CVE-2026-49975 patch inserting ngx_uint_t count into ngx_http_headers_in_t), but a different struct field causing the offset shift on RHEL 9.

Confirmation

Confirmed on production RHEL 9.8 x86_64 hosts running:

• nginx-core-1.26.3-9.module+el9.8.0+24374+c0e15309.1 (AppStream stream 1.26)
• nginx-mod-http-passenger-6.1.5-1.el9

The RHEL nginx binary is compiled with --with-http_proxy_module which enables NGX_HTTP_CACHE. This adds ngx_http_cache_t *cache at offset 64 in ngx_http_request_s. Phusion's EL9 RPM was built without it, so every field past offset 64 is shifted by 8 bytes in the module vs the binary.

Relationship to #2662

	#2662 (Ubuntu)	#2664 (RHEL 9)
Trigger	CVE-2026-49975 patch added ngx_uint_t count to ngx_http_headers_in_t	NGX_HTTP_CACHE adds ngx_http_cache_t *cache to ngx_http_request_s
Shift	8 bytes in headers_in struct	8 bytes in request struct
Affected versions	nginx 1.24.0-2ubuntu7.10	nginx 1.26.3-9.module+el9.8.0
Distro fix	Ubuntu reverted in 7.11, re-applied ABI-safe in 7.12	N/A — this is a long-standing build flag mismatch, not a new patch

The Ubuntu case was a regression introduced by a security patch; the RHEL case has likely been broken since the EL9 RPM was first published, just not widely reported because most EL9 users compile Passenger from source via passenger-install-nginx-module.

Request

Please rebuild the EL9 RPM with NGX_HTTP_CACHE enabled to match RHEL's AppStream nginx build flags, or document that nginx-mod-http-passenger is incompatible with RHEL's packaged nginx.

[CamJN]

The RHEL 9.8 nginx does not seem to be compiled with the --with-http_proxy_module flag:

docker run -it rockylinux/rockylinux:9 bash --login
dnf module enable -y nginx:$(dnf module list nginx | grep -Ee '^nginx.*[0-9\.]+' | cut -d ' ' -f 2 | sort -V | tail -1)
dnf install -y nginx
nginx -V

Can you check your report for accuracy please.

[arch3x]

You're correct that --with-http_proxy_module doesn't appear in nginx -V output — I should have been more precise. The proxy module is a default module in nginx, compiled in without an explicit flag. You'd need --without-http_proxy_module to exclude it. Since it's present by default, NGX_HTTP_CACHE is defined during compilation (it's enabled whenever any of ngx_http_proxy, ngx_http_fastcgi, ngx_http_uwsgi, or ngx_http_scgi modules are included).

You can verify on any stock RHEL/Rocky 9 nginx:

# NGX_HTTP_CACHE is compiled in — the cache struct field exists in the binary
strings /usr/sbin/nginx | grep -i "http cache"
# Or check the proxy module is present (it's always there by default):
nginx -V 2>&1 | grep -o 'without-http_proxy'  # empty = proxy module IS included

The offset table in the original report was derived from comparing pahole / gdb output between the Phusion module's expected layout and the actual ngx_http_request_s in RHEL's nginx binary. The cache pointer at offset 64 is present in RHEL's binary but absent in the struct layout the Phusion module was compiled against.

So the question is: what nginx source/headers was nginx-mod-http-passenger-6.1.5-1.el9 compiled against? If it was built against a minimal nginx configure (e.g., --without-http_proxy_module or without running the full RHEL spec), that would explain the mismatch. The RHEL AppStream nginx definitely has NGX_HTTP_CACHE active at runtime — the SIGSEGV and struct offset analysis confirm it.

Happy to provide the full nginx -V output and pahole dumps from our affected hosts if that helps track down the build discrepancy.

$ nginx -V
nginx version: nginx/1.26.3
built by gcc 11.5.0 20240719 (Red Hat 11.5.0-14) (GCC)
built with OpenSSL 3.5.5 27 Jan 2026
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-openssl-opt=enable-ktls --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-cc-opt='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,-E -O2'

[CamJN]

We don't pass --without-http_proxy_module when compiling the passenger nginx dynamic module, so that's not the cause.

[CamJN]

You can see how the module is compiled here: https://github.com/phusion/passenger_rpm_automation/blob/master/specs/passenger/passenger.spec.erb#L289-L309

[arch3x]

Fair point — if you're not passing --without-http_proxy_module, then NGX_HTTP_CACHE should be defined in your build too. That rules out the proxy module flag theory.

The more likely cause: RHEL's source patches. The RHEL 9 nginx SRPM applies distro-specific patches on top of upstream 1.26.3. If any of those patches add a field to ngx_http_request_s (or any struct Passenger touches), and your RPM is built against vanilla upstream nginx 1.26.3 source, you'd get the same 8-byte offset mismatch.

Our fix works by running rpmbuild -bp SPECS/nginx.spec (which applies all RHEL patches) and then compiling the Passenger module against that patched source tree. This produces a working .so. That strongly suggests the mismatch comes from a RHEL-specific patch rather than a configure flag difference.

[arch3x]

Thanks for pointing to the spec. Looking at L289-L309, the dynamic module build uses a minimal configure:

./configure \
    --with-compat \
    --with-threads \
    --with-pcre --with-pcre-jit \
    --with-file-aio --with-debug \
    --with-http_realip_module \
    --with-http_dav_module \
    --with-http_ssl_module \
    --http-proxy-temp-path=/var/lib/nginx/tmp/proxy \
    --add-dynamic-module=...

RHEL 9's AppStream nginx 1.26 is built with significantly more flags — notably --with-http_v2_module which adds ngx_http_v2_stream_t *stream (8 bytes) to ngx_http_request_s via #if (NGX_HTTP_V2). There are likely other conditional struct fields from flags like --with-http_slice_module, etc.

The --with-compat flag doesn't prevent this ABI divergence — it only ensures the module API version matches, not that #ifdef-conditional struct members are the same between builds.

So the mismatch isn't NGX_HTTP_CACHE specifically (you're right, proxy module is included by default in both builds). It's the sum of all conditional struct fields that RHEL enables but your spec doesn't — NGX_HTTP_V2, NGX_HTTP_SLICE, and any others that insert members into ngx_http_request_s or related structs.

Our workaround (compiling the module against RHEL's patched source with RHEL's exact nginx -V flags) works precisely because it picks up all those conditionals. Thanks for checking.

[CamJN]

The --with-compat flag enables all optional modules in the struct. Per Maxim Dounin an Nginx dev that flag should be enough to allow different compile flags to be compatible: https://mailman.nginx.org/pipermail/nginx-devel/2018-May/011123.html

The source patches are indeed a more likely source of issues. As they need to be written to preserve the ABI, and might not have been.

[arch3x]

Thank you, I'll reach out to Red Hat

[CamJN]

While RedHat being more careful with the ABI when patching would be nice, I'll work on applying the patches to our build system too.

[CamJN]

Waiting on Rocky linux repos to propagate nginx 2:1.26.3-9.module+el9.8.0+40222+e48d13b6.1 to mirrors so that I can test my changes to the buildsystem. It works on my box but I want it to pass CI too.

[CamJN]

According to the logs, GitHub Actions use Google Cloud Platform's mirrors, which is a bit weird. And while they've pulled the packages themselves, the repo metadata is out of date at the moment. Probably just needs to either clear cache or be picked up in the next round of upstream checks.

[CamJN]

something weird is happening to the rocky mirrors:

a bunch of mirrors are dropping back to older (3 + days old) rather than just synced - 1.

source: https://mirrors.rockylinux.org/mirrormanager/propagation/rocky_9.8_AppStream_x86_64

[CamJN]

Okay el9 tests were green in a run. Going to start the release.

[CamJN]

Release is done aside from the docker images.