The euro fell to 20-month lows against the dollar, with markets worried that instability in the euro zone's third largest economy could reignite a dormant financial crisis and deal a hammer blow to Italy's fragile banking sector.
Renzi's resignation could open the door to early elections next year and to the possibility of an anti-euro party, the opposition 5-Star Movement, gaining power in the heart of the single currency. 5-Star campaigned hard for a 'No' vote.
"I take full responsibility for the defeat," Renzi said in a televised address to the nation, saying he would hand in his formal resignation to President Sergio Mattarella on December 5.
Mattarella will have to embark on a round of consultations with party leaders before naming a new prime minister -- Italy's fifth in as many years -- who will be tasked with drawing up a new electoral law.
Early projections said Renzi managed to win little more than 40 percent of the vote on December 4 following months of bitter campaigning that pitted him against all major opposition parties, including the anti-system 5-Star Movement.
Italy's parties will now have to work together on the new electoral law, with the 5-Star urging a swift deal to open the way for elections in early 2017, a year ahead of schedule.
Opinion polls show Renzi's Democratic Party (PD) is neck-and-neck with the 5-Star, which has called for a referendum on Italy's membership of the euro currency.