The agreement covers 600 million people and almost a third of the global economy. It's also a major endorsement of a global trading system that is under increasing threat from protectionism.
It will remove tariffs on European exports such as cheese and wine. Japanese automakers and electronics firms will face fewer barriers in the European Union.
The dismantling of trade barriers stands in stark contrast to the approach taken by President Donald Trump, who has imposed tariffs on a range of foreign goods and is threatening more action.
Donald Tusk, president of the European Council, hailed the agreement as the "largest bilateral trade deal ever."
"Relations between the European Union and Japan have never been stronger," he said in a written statement. "Geographically, we are far apart. But politically and economically we could hardly be any closer."
Japan and the European Union traded roughly €129 billion ($152 billion) of goods last year, according to EU data.
Baker McKenzie partner Ross Denton said the deal signed Tuesday sends "a very strong signal to the US Administration that the EU and Japan, two major trade partners of the US, both see the benefits of removing barriers and reducing, not increasing tariffs."
EU trade commissioner Cecilia Malmström said last month that Europe was willing to lower some of its tariffs and cooperate with the United States. But the Trump administration "closed the door" on the talks and subsequently slapped tariffs on EU steel and aluminum. New American tariffs on European cars could follow soon.
Trump also pulled the United States out of the Trans-Pacific Partnership at the start of his presidency, another trade deal that lowered tariffs and trade barriers for the 11 remaining signatories.
Average global tariffs are near record lows. EU products currently face an average tariff of 1.6% when they arrive in Japan, while Japanese products face tariffs of 2.9% in the European Union, according to the World Trade Organization.
Still, the European Union said the tariffs cost its companies up to €1 billion ($1.2 billion) per year.
The trade deal is expected to come into force in 2019 after being approved by lawmakers on both sides.