To the degree that the leaves are dropping at an inappropriate time that's clearly a sign of damage to the tree. Moreover, significantly leaf loss at critical times for photosynthesis can seriously stress the tree. It is possible that either the oil or the soap is applied to viscously and is blocking respiration; they can also be too strong and have chemical effects.
Also, the leaf drop could be caused by the aphid infestation itself so if it is too severe, and only seems connected to the spring because you notice the aphids when they are severe, and spray, just before the leaves drop.

But if it is just aphids that one is worried about, I would be inclined to use a biological control like lacewing or ladybug larvae; by establishing a healthy predator community, the aphids can be kept under control, the tree won't be damaged by the control, and the overall health of the surrounding local ecosystem is improved.