Regarding driver fitness, leg and core strength are essential; however, flexibility relative to what your swing looks like is just as vital.
Put your driver parallel to you on your chest. Leave your hips still and rotate your shoulders back and forth. Try rotating your hips and not moving your shoulders. When you swing, there is much more separation between your shoulders and hips.
If you are limited in shoulder mobility and if you move your shoulders and your hips turn with them, everything has to stay more together. Being more flexible will help your swing.
Practice turning your hips, isolating them from your shoulders, and do the same with your shoulders.