Built on its rock facing south, on a bend of the river Cesse, Agel is characterised by narrow streets surrounding a medieval castle.
Many elements show that man has occupied this area since prehistoric times. The spring water collected at the foot of the "Pech" mountain and which feeds the fountain, dates back to the Romans.

Of the old castle, which was a bit further to the west than the  current one, there remains only the old stone gateway and the name "Fort Vieux" at the square which has replaced it.

The present castle dates from the 12th century and its origins are little known. From its configuration, it must be one of the fortified castles which prolonged the resistance of the nobles of the Count of Toulouse at the time of the crusade of the people of Albi.