Someone finds an organism and considers it to be "different enough" from all other organisms that have been described in the literature. This person will collect (or, in the past, sometimes illustrate) the organism and store it somewhere such as in a museum - this is the "holotype". The person will then write a paper with the description of the organism, compare it to some of the most similar known members, and make an argument for calling this holotype a member of a new species. If the species has particularly unique traits, or substantially different genetics, the author can argue for the description of a new genus - or even a higher rank.

But... The line is indeed extremely blurry. There is no universal agreement about where to draw the lines. The description of a new taxon is an argument, and experts disagree continuously. The tree is being continuously shuffled and it is not uncommon to see different publications using different scientific names for the same species.

They do, in many cases the choice to assign two populations to different species rather than one is not clearcut. All sorts of weird stuff can happen, like ring species around lone mountains :

https://en.m.wikipedia.org/wiki/Ring_species

Within that species there are populations that cannot interbreed, yet there's still gene flow between them.

Humans like putting things into neat little boxes. Nature is under no obligation to oblige.

In bacteria iirc these concepts still exist - mostly - though there is definitely a Ship of Theseus argument that could be made there.

For viruses though, woah! Literal chimeras exist with like the head from one source, the body from another, and perhaps the tail from still another. After all, why not? They are just functional nanoparticles, and on an evolutionary timescale of billions of years, pretty much anything that can be done will have been tried, multiple times.

These concepts are still practically useful though - well, as much as any concept is, given how the entire field of biology exists in a more or less fluid state. Literally the only law there is Virchow's stating that all cells come from previously-existing cells (by division), and that one even entirely on its face has to be false b/c what about "the first cell"?

Likewise the central dogma that information flows from DNA through RNA to protein has been proven false so many times it is ridiculous. Nowadays we know that it sometimes goes backwards, or skips steps, or both - e.g. proteins straight-up manufacture DNA from scratch (telomerases), or like proteins can make other proteins (not just prion conversion!).

Biology is... weird. It is definitely not just physics, or even chemistry, enacted on a larger scale - it's on a WHOLE. NOTHER. LEVEL!:-P