I am looking to upgrade machines from ubuntu 12.04 to 14.04.  We use
mpm-itk to isolate sites by permissions.  This worked fine in 12.04 but in
14.04 we are now seeing the error 'pcfg_openfile: unable to check htaccess
file' returning a 403 when trying to access any sites.

The document root directories are chmod 700 to prevent outside access to
them.  If I chmod 755 the sites work.  They also work if I change the
ownership to the www-data user.  I pulled the latest 2.4.7 module from the
archive link but this didn't seem to fix it.

Any ideas?

[config ok?]

yes AssignUserId  #43959 is set in the virtualhost I am using for testing,
and if I print posix_getuid() in the php file I am accessing I
get 43959 back so it appears to be running as the correct user when
accessing the php file.  In this case there is no .htaccess file, it is
just having problems looking for the .htaccess file because the document
root directory is chmod 700.

[nfs and rootsquash?]

Good call, we run an NFS client attached to the internal NFS server of a
gluster setup.  I tested with a document root on local disk and it works
fine even with 700 perms.  Root does seem to have both read and write
access to the remote disk though.

Comparing the old servers to the new one, the old servers running apache
2.2.22 have multiple apache processes running as root, the rest as UIDs of
the users they're accessing.  The new server running apache 2.4.7 only has
1 apache process running as root, the rest as the www-data user.  Does this
sound right?  The old servers are configured to run as www-data even though
no processes are actually running as that.  If I change the config in the
new server to set User as root it won't start stating I need to recompile
with '-DBIG_SECURITY_HOLE'

> Comparing the old servers to the new one, the old servers running apache
> 2.2.22 have multiple apache processes running as root, the rest as UIDs of
> the users they're accessing.  The new server running apache 2.4.7 only has
> 1 apache process running as root, the rest as the www-data user.  Does this
> sound right?

It's a bit of an illusion. They do have the setuid/setgid capabilities
(although those are restricted somewhat by seccomp), so they are in a sense
closer to root than www-data, even though the latter is what's displayed.

> The old servers are configured to run as www-data even though
> no processes are actually running as that.

For mpm-itk under Apache 2.2.x, the User/Group setting only means the default
uid/gid for when you don't set AssignUserID for a vhost.

Gotcha, well I've traced the error to a call to ap_run_open_htaccess inside
apache itself.  getuid() returns 33 (www-data) right before this call, on
the local filesystem this call returns APR_ENOTDIR and on the remote
filesystem it returns APR_EACCES.  The document root directory has 700
permissions and is owned by uid 43959 in both cases so I'm not sure how uid
33 would be able to access it locally.  I'm guessing something else is
going on but this was as far as I was able to trace it, I can't find the
definition for ap_run_open_htaccess() anywhere in the source

> getuid() returns 33 (www-data) right before this call, on
> the local filesystem this call returns APR_ENOTDIR and on the remote
> filesystem it returns APR_EACCES.  The document root directory has 700
> permissions and is owned by uid 43959 in both cases so I'm not sure how uid
> 33 would be able to access it locally.

Again, it has extra capabilities that getuid wouldn't pick up.
In particular, CAP_DAC_READ_SEARCH. I don't know if these would work well
over NFS or not, but from what you're saying, probably not.

> I'm guessing something else is
> going on but this was as far as I was able to trace it, I can't find the
> definition for ap_run_open_htaccess() anywhere in the source

It's in Apache since 2.4.7, if I recall correctly.