There is really no very simple way to fulfill such specific positioning requirements for the sub menus. Basically, you would need to use the API and test the current sub menu’s position and dimensions against the page elements you like and re-position it if needed. You can find a code sample for a similar request from another user here:
http://www.smartmenus.org/forums/topic/2nd-level-within-parent-container-borders
Also note that the vertical scrolling for long sub menus can only work within the browser viewport (since the UL elements are truly nested in the parent LI’s and we cannot use CSS clipping because we would cut further sub levels.) So you cannot make that feature work for a specific container on the page instead of the whole viewport.
Hopefully, that helps.