Python Performance: Why 'if not list' is 2x Faster Than Using len() (blog.codingconfessions.com)

submitted 5 days ago by jnovinger@programming.dev to c/python@programming.dev

52 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[-] Dark_Arc@social.packetloss.gg 11 points 4 days ago* (last edited 4 days ago)

I don't know about others ... but I'm not using Python for execution speed.

Typically the biggest problem in a program is not 100 million calls of len(x) == 0. If it was, the interpreter could just translate that expression during parsing.

[-] FizzyOrange@programming.dev 6 points 4 days ago

I mean, nobody uses Python for execution speed precisely because it is so slow.

[-] sugar_in_your_tea@sh.itjust.works 6 points 4 days ago* (last edited 4 days ago)

Even so, not x is a pretty nice-to-read pattern, and it's nice that it's faster than the less nice len(x) == 0. I generally do not care to distinguish whether a list is None or empty, I just want to know if there's something there. If I care, then I'll typically separate those checks anyway: if x is None: ..., if len(x) == 0: ....

[-] Dark_Arc@social.packetloss.gg 4 points 4 days ago

I prefer the clarity of len(x) == 0 personally; it's a pretty low syntactic penalty to clarify intent.

Obviously, if your variable names are good and you're consistent about your usage not list can be fine too, but it is a very JavaScript-y thing to me to be that vague and/or rely on "truthiness."

[-] sugar_in_your_tea@sh.itjust.works 4 points 4 days ago

The notion of truthiness is defined by the language.

Here are most of the built-in objects considered false:

constants defined to be false: None and False

zero of any numeric type: 0, 0.0, 0j, Decimal(0), Fraction(0, 1)

empty sequences and collections: '', (), [], {}, set(), range(0)

It's not something that happens to work, it's literally defined that way.

if not x is the common way to tell if you have data or not, and in most cases, the difference between None and empty data ([], {}, etc) isn't important.

len(x) == 0 will raise an exception if you give it None, and that's usually not what you want. So I guess the verbose way to do that is if x is None or len(x) == 0:, but that's exactly equivalent to if not x, with the small caveat that it doesn't check if the value has __len__ implemented. If you're getting wonky types thrown around (i.e. getting a class instance when expecting a list), you have bigger problems.

I use type hinting on pretty much all "public" methods and functions, and many of my "private" methods and functions as well. As such, sending the wrong types is incredibly unlikely, so not x is more than sufficient and clearly indicates intent (do I have data?).

[-] Dark_Arc@social.packetloss.gg 2 points 4 days ago* (last edited 4 days ago)

I did not say it's not semantically well defined.

https://en.wikipedia.org/wiki/Brainfuck#Hello_World! -- this is semantically well defined, but it's still vague. Vagueness is a property of how well the syntax is conveying intent.

[-] sugar_in_your_tea@sh.itjust.works 2 points 4 days ago

It's only vague if coming from a language where it's invalid or vague semantically. For example:

Javascript - [] is truthy for whatever reason
C - int x[] = {}; evaluates to true because it's a pointer; C only evaluates to false if something is 0
Rust - invalid because you cannot convert a vec -> bool directly, and there's no concept of null (same w/ Go, but Go has nil, but requires explicit checks)
Lua - empty tables, zero, and empty strings are truthy; basically, it's truthy unless it's nil or false

The only surprising one here is Javascript. I argue Lua and Python make sense for the same reason, Lua just decided to evaluate truthiness based on whether the variable is set, whereas Python decided to evaluate it based on the contents of the variable. I prefer the Python approach here, but I prefer Lua as a language generally (love the simplicity).

[-] AnyOldName3@lemmy.world 3 points 4 days ago

The interpreter can't make the replacement until it's about to execute the line as __bool__ and __len__ are both (Python's equivalent of) virtual functions, so it's got to know the runtime type to know if the substitution is valid. Once it's that late, it'll often be faster to execute the requested function than work out if it could execute something faster instead. Even with type hints, it's possible that a subclass with overridden methods could be passed in, so it's not safe to do anything until the real runtime type is known.

Once there's a JIT involved, there's an opportunity to detect the common types passed to a function and call specialised implementations, but I don't think Python's JIT is clever enough for this. LuaJIT definitely does this kind of optimisation, though.

[-] Dark_Arc@social.packetloss.gg 1 points 4 days ago* (last edited 4 days ago)

Hm... I'll admit I wasn't awkward of the .__len__ function. ~~However, does this mean it's possible to write a len(x) == 0 that's diverges from not x?~~

~~If not, then the substitution is still valid (and not presumably also considers the same fundamental. If so, that's kind of silly.~~

EDIT: I missed the part of your comment about .__bool__ ... so yeah in theory you could have something where these two operations are not equivalent.

Arguably, you could just say that's pathological and invalid. Then, still have an optimized path to prefer not .__bool__() if .__len__() == 0 is the comparison you'd be making. Even with the extra interpreter check during evaluation, that would quite possibly be faster if the overhead is truly high.

EDIT 2: you'd probably need a little bit more overhead than a straight substitution anyways because you need to maintain the semantic of "if this thing is None it's not valid if the syntax was originally len(x)."

[-] Reptorian@programming.dev 1 points 4 days ago

This. I rarely boot up Python for the tasks I need to do, and if they are, they are one of the following:

Assistant code for other coding language
Throwaway script
Prototype before using a faster language

[-] logging_strict@programming.dev 1 points 3 days ago

Assuming an equivalent package is produced, what's the maintenance cost (factoring in coder availability) difference between the Python vs faster language implementations?

^^ therein lies the rub

Reminds of the expression, premature optimization is the root of all evil

if not swimming in funding, might be a darwinic move to choose the faster language and have to code everything yourself from scratch