34
How much flakiness do you tolerate in end to end tests?
(programming.dev)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 30 Aug 2023
34 points (100.0% liked)
Experienced Devs
3978 readers
1 users here now
A community for discussion amongst professional software developers.
Posts should be relevant to those well into their careers.
For those looking to break into the industry, are hustling for their first job, or have just started their career and are looking for advice, check out:
- Logo base by Delapouite under CC BY 3.0 with modifications to add a gradient
founded 1 year ago
MODERATORS
My team has just decided to make working smokes a mandatory part of merging a PR. If the smokes don't work on your branch, it doesn't merge to main. I'm somewhat conflicted - on one hand, we had frequent breaks in the smokes that developers didn't fix, including ones that represented real production issues. On the other, smokes can fail for no reason and are time consuming to run.
We use playwright, running on github actions. The default free tier runner has been awful, and we're moving to larger runners on the platform. We have a retry policy on any smokes that need to run in a step by step order, and we aggressively prune and remove smokes that frequently fail or don't test for real issues.
Shouldn't you find out, why the smoke tests keep failing? It's not a good sign, if you can't even guarantee stability in a controlled environment.
It's not a fully controlled environment, that is the point of smokes.
How do you not fully control the environment your PRs are tested in?
dependencies.