He was speaking soon after taking the oath of office at the Amahoro National Stadium in the capital, Kigali.
The ceremony was attended by dozens of African leaders and dignitaries.
Mr Kagame took 98.8% of the vote in the 4 August election, which has been criticised by rights groups.
New York-based Human Rights Watch said the election “took place in a context of very limited free speech or open political space”.
Speaking on Friday, Mr Kagame said that “every attempt that was made whether from within and especially from outside to denigrate the process and glorify the old politics of division only made Rwandans more defiant and more determined to express ourselves through the vote.
“Our experience is that we will be vilified anyway, no matter what. So, we might as well do what we know is right for our people because the results are much better and the costs are much lower.”
The president also thanked his opponents saying they had “created a positive environment where no vote was cast against anyone, but rather all were cast for Rwanda”.
He was challenged by Frank Habineza, from the Democratic Green Party, and Philippe Mpayimana, an independent.
Paul Kagame at a glance:
- Seen as military genius
- His rebel forces helped end 1994 genocide – he has been in power ever since
- Twice invaded much larger neighbour, DR Congo
- Accused of suppressing the opposition and ordering assassination of critics
- Sees Singapore and South Korea as models – economy growing at 7% a year
- Champions women’s rights; most MPs are women
Mr Kagame, 59, came to power in 1994, when his Tutsi rebel group took control of the capital, Kigali, ending the genocide in which some 800,000 ethnic Tutsis and moderate Hutus were slaughtered.
Rwanda’s constitution was amended in 2015, following a referendum, enabling Mr Kagame to run for a third term.
Beyond this, he could run again, potentially staying in power until 2034.
His two challengers have complained that their supporters have been intimidated, which they say explains the low turnout at their rallies.
The candidates have also accused some local authorities of undermining their campaigns.
The ruling party denies any accusations of wrongdoing.