Company Chairman Steve Kean said he would scrap plans to nearly triple the capacity of the Trans Mountain pipeline, which takes crude from Alberta's oil sands to a facility in the Pacific province of British Columbia, unless the various legal challenges could be resolved by May 31.
The announcement was a blow to the Liberal government of Prime Minister Justin Trudeau, which approved the project and says it is in the national interest.
The project - considered crucial for an oil industry hit by transportation bottlenecks - is fiercely opposed by British Columbia's left-leaning New Democratic government, many municipalities, some aboriginal groups, and environmental activists concerned about possible oil spills.
"We will be judicious in our use of shareholder funds. In keeping with that commitment, we have determined that in the current environment, we will not put KML shareholders at risk on the remaining project spend," Kean said in a statement.
The firm is currently carrying out preliminary work and has not started construction.
Many in the energy industry are concerned about whether any new pipelines can be built in Canada, which sits on the world's third largest proven reserves of crude and is the single largest exporter of energy to the United States.
Although the federal government has the power to approve major pipelines, the 10 provinces enjoy broad responsibility for resource development. That can result in deadlock when a province opposes a decision made by Ottawa.
"This is not good. I think the key point is it shows a lack of confidence in our political and regulatory system," said Tim Pickering, president of Auspice Capital in Calgary, Alberta.
Last October, TransCanada Corp abandoned its C$15.7 billion Energy East pipeline from Alberta to the Atlantic Coast amid mounting regulatory hurdles.
The inability to get oil to ports for export means Canadian crude has been trading at a discount of up to $25 a barrel this year, costing Alberta's heavy oil producers C$30 million to C$40 million in revenue per day.
'THIS PIPELINE WILL BE BUILT'
Federal Natural Resources Minister Jim Carr said in a statement that "with all our partners, we continue to consider all available options."
Alberta Premier Rachel Notley said she was very optimistic the project would go ahead and repeated calls on Trudeau's government to do more to ensure it would be built.
"Alberta is prepared to be an investor in the pipeline. If we take that step, we will be a significantly more determined investor - this pipeline will be built," she told reporters.
Trudeau visited the oil sands on Friday and reiterated the government's determination that the expansion go ahead.
British Columbia's government proposed environmental rules this year that were seen as a direct attack on the project.
It later edged back from a pledge to block increased shipments of crude oil through the province, which had sparked a trade war with Alberta.
"British Columbians expect their government to stand up for their interests and our coast, and to do everything we can to protect our land and waters, our coastal communities and our local economies," Premier John Horgan said in a statement.
Mike Hudema, climate campaigner with Greenpeace Canada, said: "The writing is on the wall, and even Kinder Morgan can read it. Investors should note that the opposition to this project is strong, deep and gets bigger by the day."
By David Ljunggren