Good call, we run an NFS client attached to the internal NFS server of a
gluster setup.  I tested with a document root on local disk and it works
fine even with 700 perms.  Root does seem to have both read and write
access to the remote disk though.

Comparing the old servers to the new one, the old servers running apache
2.2.22 have multiple apache processes running as root, the rest as UIDs of
the users they're accessing.  The new server running apache 2.4.7 only has
1 apache process running as root, the rest as the www-data user.  Does this
sound right?  The old servers are configured to run as www-data even though
no processes are actually running as that.  If I change the config in the
new server to set User as root it won't start stating I need to recompile
with '-DBIG_SECURITY_HOLE'

> Comparing the old servers to the new one, the old servers running apache
> 2.2.22 have multiple apache processes running as root, the rest as UIDs of
> the users they're accessing.  The new server running apache 2.4.7 only has
> 1 apache process running as root, the rest as the www-data user.  Does this
> sound right?

It's a bit of an illusion. They do have the setuid/setgid capabilities
(although those are restricted somewhat by seccomp), so they are in a sense
closer to root than www-data, even though the latter is what's displayed.

> The old servers are configured to run as www-data even though
> no processes are actually running as that.

For mpm-itk under Apache 2.2.x, the User/Group setting only means the default
uid/gid for when you don't set AssignUserID for a vhost.

Gotcha, well I've traced the error to a call to ap_run_open_htaccess inside
apache itself.  getuid() returns 33 (www-data) right before this call, on
the local filesystem this call returns APR_ENOTDIR and on the remote
filesystem it returns APR_EACCES.  The document root directory has 700
permissions and is owned by uid 43959 in both cases so I'm not sure how uid
33 would be able to access it locally.  I'm guessing something else is
going on but this was as far as I was able to trace it, I can't find the
definition for ap_run_open_htaccess() anywhere in the source

> getuid() returns 33 (www-data) right before this call, on
> the local filesystem this call returns APR_ENOTDIR and on the remote
> filesystem it returns APR_EACCES.  The document root directory has 700
> permissions and is owned by uid 43959 in both cases so I'm not sure how uid
> 33 would be able to access it locally.

Again, it has extra capabilities that getuid wouldn't pick up.
In particular, CAP_DAC_READ_SEARCH. I don't know if these would work well
over NFS or not, but from what you're saying, probably not.

> I'm guessing something else is
> going on but this was as far as I was able to trace it, I can't find the
> definition for ap_run_open_htaccess() anywhere in the source

It's in Apache since 2.4.7, if I recall correctly.