The Constitutional Court, ruling in a case brought by a royalist lawyer, said a controversial 10-point call for reforms of the institution by three student protest leaders in August last year was designed to topple the monarchy.
"The actions have hidden intentions to overthrow the constitutional monarchy and were not a call for reform," a court judge said.
The court was ruling on the constitutionality of their reform call and imposed no penalty but ordered them and their groups "to cease further action in these matters".
The role of the monarchy is a taboo topic in Thailand, where the palace is officially above politics and constitutionally enshrined to be held in "revered worship".
Thailand has one of the world's harshest "lese majeste" laws, setting jail terms of up to 15 years for anyone convicted of defaming, insulting or threatening King Maha Vajiralongkorn and his closest family.
During a universal periodic review on Wednesday by a working group of the UN Human Rights Council, Thailand was urged to amend its lese majeste law by some member states who said it restricted freedom of expression.
Thai officials, however, argued it protects the monarch and therefore national security, and that royal insult cases were carefully handled.
Protest movement
The court case concerned a speech by Panusaya "Rung" Sithijirawattanakul calling for amendments to crown property laws, reducing the royal family's budget allocation and seeking the abolition of the lese majeste law.
Two other protesters, human rights lawyer Arnon Nampa, 37, and Panupong "Mike" Jadnok, 24, also spoke at the same rally.
A group of protesters gathered near the court on Wednesday, among them Panusaya, who said overthrowing the monarchy was not her goal but that she respected the ruling.
Arnon and Panupong are currently in prison in pre-trial detention on other charges and their lawyer, Kritsadang Nutcharat, said they too had no desire to topple the monarchy.
"The ruling could impact future calls for reform," Kritsadang added.