Purpose

You want to use JAAS in TomEE with custom (or OpenEJB) LoginModules.

Solution

TomEE tries to keep as possible as it is Tomcat so simply configure your JAAS LoginModule as in Tomcat.

Note: only the first one will be used.

Configuration

Add to your CATALINA_OPTS the java.security.auth.login.config system property:

-Djava.security.auth.login.config=$CATALINA_BASE/conf/login.config

Configure your realm in server.xml file

<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.tomee.loader.OpenEJBListener" />
  <Listener className="org.apache.catalina.security.SecurityListener" />

  <Service name="Catalina">
    <Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <!-- here is the magic -->
      <Realm className="org.apache.catalina.realm.JAASRealm" appName="PropertiesLoginModule"
          userClassNames="org.apache.openejb.core.security.AbstractSecurityService$User"
          roleClassNames="org.apache.openejb.core.security.AbstractSecurityService$Group">
      </Realm>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true" />
    </Engine>
  </Service>
</Server>

Configure your login.config file

PropertiesLogin {
    org.apache.openejb.core.security.jaas.PropertiesLoginModule required
    Debug=false
    UsersFile="users.properties"
    GroupsFile="groups.properties";
};

Configure your login module specifically (users.properties for snippets of this page for instance)