Linear and logistic regression are much easier (and less error prone) to implement from scratch than neural network training with backpropagation.

That way you can still follow the progression I suggested: implement those regressions by hand using numpy -> compare against (and appreciate) sklearn -> implement SVMs by hand using cvxpy -> appreciate sklearn again.

If you get the hang of "classical" ML, then deep learning becomes easy as it's still machine learning, just with more complicated models and no closed-form solutions.