Afaik Youtube is doing that, not Pulseaudio, and there's nothing that can be done about it.

I've never seen this "just" happen, but have seen it during events like switching from headphones to speakers and such.

You may also have your app volumes linked to your master channels, meaning when you lower the sound on your master with something like a key combo, then it lowers the individual app volumes as well, which is generally not something you'd want enabled.

Apps at full, and using PCM/Master channel for general volume is pretty much the "default".

The first thing that came to mind after reading the title was, "uninstall PulseAudio".
Jokes aside, it could be some setting of the DE you are using. If the audio output device is detected as headphones, there tend to be settings that reduce the master volume on certain events.
I remember KDE Plasma used to do that. Every time I switched the output device (or was it when I unplugged and re-plugged it?), it set the headphone volume to 60%.

Im not sure if this is the same thing but i noticed a similar problem in vivaldi but its specific to youtube, or at least it doesnt affect peertube the same way. It seems to be based on the volume of the video, my guess its a "feature" that tries to normalize volume between different videos, but ive never figured out how to disable it.

maybe ff syncs the youtube video volume to the pa slider?

Do you use any pulseaudio modules?

Do you use Discord, by any chance? I remember having this problem when I had audio ducking enabled.