For the sake of sharing... I took your idea and found this worked just fine:

TroubleMode = "workspace_diagnostics"

local function cycle_trouble_mode()
	local trouble = require("trouble")

	local modes = {
		"document_diagnostics",
		"workspace_diagnostics",
		"lsp_references",
		"lsp_definitions",
		"lsp_type_definitions",
		"quickfix",
		"loclist",
	}

	if trouble.is_open() then
		local function get_next_mode(mode)
			local next_index = 1
			for i = 1, #modes do
				if modes[i] == mode then
					next_index = i == #modes and 1 or i + 1
				end
			end
			return modes[next_index]
		end

		TroubleMode = get_next_mode(TroubleMode)
		trouble.toggle(TroubleMode)
	end
end

I did end up looking at the code a bit and couldn't find what I wanted. But what you suggest is a good idea. I may give that a try.

I tried after writing the function I posted below. This works to flip between workspace and document mode but I can't get to all of the modes like this. Thank you though.