Kind of a tangent at this point, but there is a very good reason that that couldn't be the case: according to the shell theorem , nowhere in the interior of a spherical shell of matter experiences any net gravitational force -- wherever you are inside the sphere, the forces counteract exactly.

Otherwise, though, the metric expansion of space is different from typical movement, and it isn't correct to say that things or being pushed or pulled. Rather, the distance between every pair of points in the universe increases over time, with that rate of increase proportional to the points' distance. Importantly, for very distant points, the distance can increse faster than the speed of light, which would be disallowed by any model which describes the expansion in terms of objects moving in a traditional sense.

The Universe probably wouldn't fit in a sphere of any size and it very well may be infinite.

But looking at a very large spherical region like the visible universe from our perspective here on Earth, everything is moving away from us. If objects were being pulled towards one particular edge of our imaginary sphere, it would look very different. We'd see a clear drift in that direction, but it all looks pretty even across every direction we look.

You are right that things would still look like we're accelerating away from us, even if we were actually contracting.

Interesting hypothesis! How do we investigate?

What could we expect from a large central gravitational point? We should have other signs of the gravity well:

We would expect a point that we contract towards (and that seems ill fitting, as we see the expansion moves as the observer (including earth) moves), we would expect some kind of mass or similar effect, which would also have a size to fit it in (we know that gravity works different when you're inside the mass, and we would be able to see it, much like black holes or dark matter), we would expect things to orbit the gravity well (which we know that at least our galaxy doesn't orbit us).

You might want to actually check on these things to make sure they apply and are true, but at least at first glance it seems the expansion is better explained without a central gravity.

Not unless we were directly in the center of it, in which case everything would seem to move away from us. But even then, if I'm not mistaken, physicists and astronomers have also proven that other objects are also moving away from one another too.

We can tell how something is moving based on the shift of its light, like the Doppler effect. The further something is the more its light is shifted, which was how we came to know this phenomenon in general of increasing size of space between all things.