The UN General Assembly voted by 94 countries to 15 that The Hague should examine the legal status of the Chagos Islands.
The former British colony used to be part of Mauritius but was detached in 1965 and is now home to a US airbase.
The Foreign Office said it would be an “inappropriate” use of the ICJ.
“This is a disappointing outcome,” a Foreign Office spokesman said: “Sovereignty of the British Indian Ocean Territory is clearly a matter for the UK and Mauritius to resolve ourselves.
“Taking this dispute to the International Court of Justice is an inappropriate use of the ICJ mechanism.”
Mauritius, which gained independence from Britain in 1968, argues that the UK broke international law when it separated off the islands before granting Mauritius its independence.
Diego Garcia, the largest of the group of islands, was leased to the US in 1966.
Families were forced to leave the Chagos Islands in the 1960s and 1970s to make way for a US Air Force base on Diego Garcia, which is leased until 2036.
The Foreign Office said it did not recognise Mauritius’s claim to sovereignty over the islands – but that it would return the islands when they were no longer needed for defence.
“We have committed to cede them to Mauritius when the territory is no longer required for defence purposes,” the spokesman said.
“At present it plays an important role in regional and global security, helping to keep the UK, US and other allies, including Mauritius, safe.”
The government said it would “robustly defend” its position ahead of the ICJ’s decision, which would not be legally binding.
Most EU countries abstained from the vote, which BBC diplomatic correspondent James Landale described as an “embarrassing diplomatic defeat” for the UK.
He said it signalled that Britain’s diplomatic clout had waned after the vote for Brexit.
In 2015, the UK Supreme Court denied a legal challenge by former islanders to return to Chagos after being removed more than 40 years ago.
The court rejected claims that islanders suffered a “significant injustice” by being forcibly removed from their homeland.