I haven't watched or read it, and don't plan to, but this is the gist of other people's criticisms:

1. The main character, Kazuya Kinoshita, is an unlikeable asshole who the series tries to paint as relatable, but which only a menace to society could relate to.
2. The series' author is obsessed with Kazuya's love interest Chizuru to a degree that many consider weird (especially since the author is married). Many believe this ties into:
3. The series' current length is around 380 chapters, which is comparable to Berserk, the original Dragon Ball, The Prince of Tennis, and a few volumes short of the infamous long-runner Glass Mask. The series' most recent plot development is, after 40 or so volumes, Chizuru has understandably rejected Kazuya. This is not technically the end of the series, but I do feel like continuing after this is a very unsafe form of masochism.
4. The only way the author can think of to have the love interests fall for his protagonist is to have him literally save their lives. Speaking of that:
5. Rika, a character in Kazuya's pseudo-harem, has a condition that slows down her heart, and so searches for love in Kazuya to make her heart beat faster. Several people with this condition have equated the actual sensation described by the sudden boost to be more akin to feeling like you're dying.
6. As you can expect from point 3, the pacing is generally agreed to be awful.
7. In the series, Kazuya is turned on by imagining Chizuru with other men, to the point where Chapter 218, where he has an erection in a pool over it, has risen to legendary memetic status. Now, I'm pretty sex-positive, so I don't care that this plot point exists; however, from what I've seen, it seems that not even actual cuckoldry fetishists like the writing of these scenes. People also tend to draw connections between this and point 2, insinuating that the series is accidentally a work of metafiction in which the author is more important than the main character.