This is how we usually build an 8/4/16 Ohm speaker with adjustable Volume Control. It can handle up to 15W speakers without any issues.
100V Line Audio Volume Control with Bypass Option
This is a 100V line speaker volume control with an Emergency bypass. It requires DC 24V to bypass the audio level control.
