The atmosphere stores negligible heat (only weather, not climate), but the ocean has a much greater capacity than the atmosphere, for both heat and CO2 (mainly in the form of HCO3-), and it takes a long time (centuries - millenia) to fully mix the ocean. Also it takes ages for icecaps to melt. If you really stop adding CO2, concentration in the atmosphere will go down slowly as it mixes into deeper ocean, but not back to preindustrial, the surface temperature will likewise go down slowly and partially after a slight lag, but ice will keep melting (-> sea-level rises) for a while. Other gases and aerosols make short term response more complex.
There's no rule of thumb that summarises it, but I made an interactive model - here.