An unexpectedly large protest rally in Kuala Lumpur over the weekend—and the government’s tough response—appears to have galvanized Malaysia’s opposition movement, putting new pressure on Prime Minister Najib Razak as he wrestles over when to call new elections in this racially charged Southeast Asian nation.

Police fired tear gas and water cannons at the estimated 20,000 or more protesters that gathered downtown on Saturday—despite stern warnings to stay home—to press for changes to Malaysia’s electoral system, which they say unfairly favors a ruling coalition that has controlled the country since its independence from Britain in 1957.

Police charged on the demonstrators with batons and detained more than 1,600 people, including some pro-government protesters. Although all the detainees were released by Sunday, the response triggered sharp criticism from human-rights groups such as Amnesty International, which called it “the worst campaign of repression we’ve seen in the country for years.” One person died of a heart attack during the rally.

Authorities moved quickly on Sunday to try to contain the fallout. Home Minister Hishammuddin Hussein criticized protesters for refusing to gather at an alternate site sanctioned by the government, and praised police for preventing more serious incidents “despite being challenged and provoked by the demonstrators,” the national news agency, Bernama, reported.

“The opposition wants Malaysia to be seen as unstable and this, in the end, will give rise to various implications, including to the country’s economy,” he said.

Mr. Najib, meanwhile, said on Saturday that the protesters only represent a minority of Malaysians, and that he still enjoys the support of most people.

Government officials have long maintained the rally organizers aren’t interested in electoral overhauls so much as embarrassing the government and boosting opposition parties ahead of a national election that must be held by spring 2013—a charge they deny.

Even so, political analysts described the rally as a potential turning point for Malaysia’s myriad opposition groups, which have struggled at times to gain traction since Mr. Najib took over the premiership in 2009.

Protest organizers pointed out that even more people would have attended, if the government hadn’t sealed off roads and closed train stations downtown in a bid to keep people away. By late Sunday, more than 83,000 people had joined a Facebook page calling for Mr. Najib’s resignation.

Long considered one of the most stable and peaceful countries in Asia, the majority-Muslim Malay country has become increasingly fractured along racial lines in recent years, as ethnic Chinese, Indian and other residents demand changes to boost their stake in the economy. Other groups, ranging from conservative Islamist Malays to secular progressives, have also sought overhauls to give them a bigger say in government and to rein in corruption, which many Malaysian residents say they believe has gotten out of control.

But after a surprisingly strong turnout in elections in 2008 that denied the ruling National Front coalition a two-thirds majority in Parliament for the first time in decades, the opposition had more recently bogged down amid reports of infighting and other problems.

Mr. Najib, meanwhile, was able to win back some support by backing modest economic overhauls and launching a “1Malaysia” campaign aimed at celebrating the country’s racial diversity and bridging some of its ethnic divides. He benefited, too, from the continuing troubles for Malaysia’s best-known opposition leader, Anwar Ibrahim, who authorities put on trial for allegedly breaking Malaysia’s strict sodomy laws, which he denies.

All that, combined with a robust recovery from the global economic downturn, led many analysts to predict Mr. Najib would call national elections later this year rather than wait until next year or early 2013, when some analysts believe the economy may be weaker.

Saturday’s rally, though, points to a set of challenges for Mr. Najib, largely because it appears to be backed by a wider range of critics than Malaysia’s longstanding opposition coalition headed by Mr. Anwar.Saturday’s rally was “a potential game changer, which is why the government pulled out all the stops to prevent it from taking place,” added Lim Teck Ghee, director of the Center for Policy Initiatives in Kuala Lumpur. “Apart from the fillip it has provided to the opposition, the government’s handling has also alienated further the urban middle class and the youth vote.” If opposition groups maintain their new momentum and grab more votes from the ruling coalition, “we may see a new government” whenever elections are held, he said.

Known as Bersih, or the Coalition for Clean and Fair Elections, it is guided mainly by representatives from nongovernmental organizations and activist groups, some of whom say they have no intention of ever running for political office. Their only objective on Saturday, they say, was to press for electoral-system changes such as equal media coverage for all election candidates, and stronger measures to curb fraud, including the past practice of people voting multiple times—overhauls they say enjoy widespread support.

Bridget Welsh, a professor at the School of Social Sciences at Singapore Management University, said Saturday’s protest was notable for being more multiethnic, and having more middle-class voters, than a previous Bersih rally in 2007—a troubling prospect for Mr. Najib as he debates when to call the election.

“The opposition before, their performance was kind of lackluster,” but now the rally has “really pulled them together,” she said.

Leaders of the rally said on Sunday that while they don’t intend to schedule another major protest soon, they plan to keep up the pressure by other means, including continuing to organize opponents of the government in areas outside of Kuala Lumpur.

The tensions that boiled over on Saturday had been building for days, as Bersih organizers publicized their plans.

Authorities arrested more than 200 activists in the days leading up to the protest, leading to a rare intervention by Malaysia’s constitutional monarch, Tuanku Mizan Zainal Abidin, who issued a statement asking both sides to step back from the brink of a broader conflict. The tensions appeared to ease for a while, as protest organizers negotiated with authorities to schedule a sanctioned rally at a site approved by the government.

But the talks broke down by week’s end, and government officials began warning residents it would tolerate no public demonstration at all, blocking off roads and closing train stations downtown.

Authorities on Saturday fired their water cannons and tear gas to scatter the crowds as they tried to mass near the city’s famous Merdeka Stadium, while helicopters buzzed overhead. Witnesses said police charged on demonstrators with batons, while other protesters shouted “Reform!” and “Long live the people!”

“The system must change,” said one protester from the northern state of Kedah, Akashnan Ahmad, whose glasses were smashed in a confrontation with police. “Look around, see the young faces—they want clean elections.”

“We have proven a point despite all the attacks on the people,” said another protester, S. Arutchelvan, who is secretary-general of the Socialist Party of Malaysia.