The oversimplified explanation is he had to appear tough on communism to avoid be criticized domestically amd abroad. He was a staunch anti-communist, worried that the US failing to shore up the South against the North would cause other "developing democracies" (Western friendly war lords and despots) to question American guarantees and look the the Soviet Union as a sponsor instead. The goal was to help South Vietnam reach a point where they could deal with the Viet Cong without much US assistance. But that was essentially an impossible task as every action to eliminate opposition further delegitamized the government in Saigon.

Morally it was absolutely the wrong play, and politically i would say it hindered his domestic ambitions as Vietnam quickly spiraled out of control under his administration. He was put between a rock and a hard place and had to answer for Truman and Eisenhower's failed choice to support the French reoccupation of Indochina after WW2. The US had committed to the path in 1954 and LBJ's memoirs indicate that, while he resented having to escalate in Vietnam, his ideology offered him no other choice. I don't think that absolves him of the ocean of blood on his hands, but I am sympathetic to the impossible choice.