Don't use a vest, I use a Tamarac belt system which serves to the same end. It works for me because I use M4/3 and not a larger system. Probably would not be so convenient with a larger camera system.
I have four lens pouches, the camera pouch, and three smaller pouches. Currently I am carrying a GH2, 14-140 on camera, 100-300, 20, 9-18, and 90 macro. In the smaller pouches I have remote releases, bubble levels, flash light, extra batteries, filters and misc do dads. So I am ready for most anything. Curious, I just weighed the system and it is 6lbs 12oz. It is really comfortable enough that I don't mind wearing it even if I do look like a dork. I use the optional shoulder harness as it makes for more comfort.
It sits on the couch next to my computer, always ready to go. I look out the window frequently and if it looks like a good sunrise or sunset I can be out of the house in five minutes confident that I can handle almost any situation that comes up. Tripod always in the car.
The trouble with backpacks is that you set them on the ground, dig around for what you want, setting things on the ground next to the pack as you search. Sometimes these things don't get put back in the pack. I've almost lost some things this way even though I try to be very careful. With the belt system everything has a home so I know where it is and can make switches without setting anything on the ground. A vest seems like it might be too hot in the summer and does not offer as much protection to the gear as the belt system.
I have additional camera and lens pouches and can change configurations for special needs. For the local Sunday quarter mile dirt track races (car), I substitute a camera pouch for a lens pouch, and carry the 14-45 on one camera and the other with one of the long lenses. All lens/camera combos listed above fit in the camera pouch.
Works great for the way I work. Lots of short local photo excursions, never far from the car. Not too much hiking, I am over 70 and can't handle that anymore.