Work done by a spring & its potential energy

According to work - mechanical energy theorem ,
W = K(final) - K(initial) + U(final) - U(initial) . . . . (1)
as we define Potential energy as negative of work done by conservative force and assuming that the only force in this situation is Spring force then ,
W(spring) = K(final) - K(initial)
As work done is calculated by finding component of spring force in direction of displacement. How can we say that U(final) - U(initial) applies for all possible conditions of extension of spring as displacement may not be in direction of force ?
Spring force = 0.5kx²