34
How much flakiness do you tolerate in end to end tests?
(programming.dev)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 30 Aug 2023
34 points (100.0% liked)
Experienced Devs
3978 readers
1 users here now
A community for discussion amongst professional software developers.
Posts should be relevant to those well into their careers.
For those looking to break into the industry, are hustling for their first job, or have just started their career and are looking for advice, check out:
- Logo base by Delapouite under CC BY 3.0 with modifications to add a gradient
founded 1 year ago
MODERATORS
My org has issues with e2e, but we keep them because they usually inform that something, somewhere isn't quite right.
Our CI/CD pipeline is configured to automatically re-run a build if a test in the e2e suite fails. If it fails a second time, then it sends up the usual alerts and a human has to get involved.
In addition to that, we track transient failures on the main branch and have stats on which ones are the noisiest. Someone is always peeling the noisiest one off the stack to address why it's failing (usually time zones or browser async issues that are easy to fix).
It's imperfect, and still results in a lot of developers just spam retrying builds to "get stuff done", but we've decided the signal to noise ratio is good enough that we want to keep things the way they are.