Missy,
I believe the following may get you what you want. I will try to be clear with the text below.
The LIMIT setting is basically the maximum number of instances that can exist for a document. This appears in the upper part of the screen above the settings for USERS and GROUPS and it will override any settings for USERS or GROUPS.
Therefore, I think what you should do is increase the LIMIT for the document you are working with to a much larger number, perhaps 1000. Then, set individual limits for USERS or GROUPS that are less than or equal to the LIMIT, perhaps 20 or 25. The total of all the limits set for USERS and GROUPS should be equal to or less than the LIMIT setting.
I ran a test for this scenario for a specific document by setting the LIMIT to 100 for this document and then set a limit for one user to 10 and a second user to 100. This will permit 100 instances to exist for the document, not 110. Even though the limits for the two users added together is 110, the maximum limit is 100. I believe the oldest instances will be deleted first. It is possible for the second user to get to 100 instances but that would require no other user creating instances for this document. Once the total number of instances reaches 100, the two users will be competing for space.
Please be aware that RECURRING SCHEDULEs do not count as an instance.
Hopefully this makes sense and gets you closer to what you are trying to do.
Thanks!
Mickey