The European commission has dropped plans to legally limit a pervasive but naturally occurring chemical found in food, that is linked to cancer, just days after lobbying by industry, the Guardian has learned.

Campaigners say that leaked documents revealing the legislative retreat show “undue influence” by the food industry over EU law-making and a “permanent scandal”, although the issue is complex.

Acrylamide is a hazardous substance found in the browned and burned parts of common starchy foods that have been fried, roasted or oven-cooked at temperatures higher than 248 degrees C.

Crisps, potato chips, breakfast cereals and instant coffee all contain high levels of the substance, as do baby foods, biscuits and rusks.

Scientists are still trying to quantify the health risks posed, but acrylamide has been judged an “extremely hazardous substance” by the US Environmental Protection Agency.

Last year, the European Food Safety Authority (Efsa) found that acrylamide “potentially increases the risk of developing cancer in consumers of all ages” and recommended that exposure be kept as low as possible, as a safe limit has not yet been determined.

“Since any level of exposure to a genotoxic substance could potentially damage DNA and lead to cancer, Efsa’s scientists conclude that they cannot set a tolerable daily intake (TDI) of acrylamide in food,” the authority said.

Acrylamide levels can be reduced by using different ingredients and additives, or by changing storage methods and the temperature at which food is cooked.

But this could affect food industry practice, costs and product tastes, and EU legislation in the area has so far been confined to voluntary codes of practice.

An EU regulation this year had been expected to firm these codes up into more robust public health protections and a draft version produced in late June seen by the Guardian took steps in that direction.

It called on the food industry to “provide evidence of regular testing of their products to ensure that the application of the code of practice is effective in keeping acrylamide levels as low as reasonably achievable and at least below the indicative levels referred to in Annex 3.”

This annex set benchmarks for acrylamide levels in a range of foods, including: crisps, crackers, soft bread, breakfast cereals, biscuits, wafers, gingerbread, coffees, biscuits and baby foods.

But the paper was shared with an industry association, Food Drink Europe, which immediately complained to the commission in a letter seen by the Guardian, saying that “the terminology ‘at least below the indicative value’ could be interpreted as meaning that the indicative values are maximum limits”.

Within days, the offending reference had been removed from a second draft of the law, which was then sent out for broader public consultation.

The European commission has updated the proposed acrylamide law in recent weeks, to at least include a triannual review of the voluntary framework.

Some commission sources say that bolder, more binding measures will be threatened in a final draft, if the food industry does not take the acrylamide issue more seriously.

But Camille Perrin, the senior food policy officer for the European consumer organisation, BEUC, said that an impact assessment would be needed for that to happen, because of the Jean-Claude Juncker commission’s “better regulation” agenda. She suggested that the original proposal had been misinterpreted.

More than 150,000 people have so far signed a petition calling for the EU to set legally binding acrylamide limits, and a social media campaign is also underway.