By default, nested groups (nested LDAP groups) are not enabled in IBM Connections. What this means, for example, is that if you add the parent group as member of a restricted community, users belonging to children groups are not automatically Community members.
To enable the nested group you have to customize the scope of member attributes of Group attribute definition in your federated repository.
There are some small differences depending on the type of LDAP repository used:
Change only the scope of member attribute="member" to nested as shown below:
Tivoli Directory Server and Domino
Customize the name of group membership attributes in Person record (ibm-allgroups for TDS and dominoAccessGroups for Domino) and set the scope to nested:
Check the right Object Class used for member attributes (groupOfUniqueNames for TDS and DominoGroup for Domino)
Save the configuration and restart all components!