The Broad Gauge was a controversial choice, requiring as it did entirely new rolling stock. It also meant that where the railway met another line which was narrow gauge, passengers had to change trains to contine their journey. This was called a gauge break.
However it had advantages. In 1845 a Firefly class Broad gauge locomotive was tested against two new standard gauge engines. Pulling the same load over the same distance, the Firefly was faster and used less fuel despite being significantly older. The demonstration brought the Broad Gauge time.
Despite this success, Parliament passed the Gauge Act in 1846, stating that all new railways had to be narrow gauge. The Great Western effectively ignored them, laying the South Devon Railway (1849) and Cornwall Railway (1859) and more.
With GWR already owning most of the Broad Gauge rail in the country, it was ironically a business consideration by their management that sowed the seeds for Broad gauge's eventual end. Permission for a new line was received which would overlap with another railway's routes. To avoid a gauge break, the new line was laid as Dual Gauge.
Dual gauge was a system of three rails, one on one side, but two on the other, allowing the track to take both narrow and broad gauge trains. While technically the best of both worlds, this made points and junctions complex and expensive to maintain.
At the same time, GWR bought a number of smaller narrow gauge railways. By 1861 they had a mix of all three types of rail and rolling stock. The system needed to be standardised.
Perhaps most importantly for the future of the broad gauge, Brunel had died in 1859. He was no longer around to champion his creation. However it would be decades before his influence would fade enough for the last broad gauge to be removed. Daniel Gould, his successer and former chief locomotive designer was also a defender of broad gauge.
A Fuller History
The years of the gauge wars are too detailed to cover here. For a fuller view, this article covers them in depth:
A History of the Great Western Railway