There really aren't enough unused Structural brushes in Foy to make a noticeable difference in visdata. Like I said, the majority of the visdata size is from the portals themselves, not unused buildings/walls.

On a sidenote, just because I can't stand having to pull up menus to show/hide things, here are a couple of useful shortcuts if you didn't already know:

ctrl+D hides/shows all Detail brushes
alt+H hides everything but what is selected
ctrl+P hides/shows Portals
alt+7 hides/shows all Clip types
shift+A selects all of the selected type (meaning all brushes/meshes using the selected texture, all misc_models, all script_models, etc..)