The Surat castle is one of the ancient monuments of 16th century existing in the city and bears a significant relevance to its history. However, such a great fortification built to provide the citizens of Surat with an adequate defence against the attacks of the invaders seems to have been forgotten from the minds of the present generation. The Ahmedabad king Sultan Mahmood-III (1538-1554), who was very much annoyed by these frequent destructions of Surat, ordered for building a very strong castle and entrusted the work to Safi Agha, a Turkish soldier who had been ennobled with the title of Khudawand Khan. The work of building the castle was completed in 1546. After the capture of Surat by the emperor Akbar (1573) the fortress remained in the charge of commandants appointed from Delhi till it was seized by the Sidhi admiral of the Mugal fleet in 1751. The Sidhi did not hold the castle for long period, as it was captured by the English in 1759 with rest of the city. Though from the first practically independent, the English held the castle nominally under the Mugal. In token of this divided command, two flags waved from the castle walls, the English ensign on the south-west, and the Moorish standard on the south-east bastion. This practice was continued till, in 1842, on the death of the last of the nawabs of Surat, the English fleet was removed from the Tapi, and the Moorish standard taken down from the castle walls. Though, as a defence against any well-equipped enemy, they have long been useless, the castle buildings initially were being kept in repair, and until the year 1862, were garrisoned by a small body of European and native troops. In that year, as no longer required, the force was withdrawn, and the vacated rooms were made over for the accommodation of the various offices connected with the revenue and police departments, in whose occupation the castle has since remained.
Though in context of the identity of Surat in mediaeval times many views have been expressed by different historians, in all the historical narratives Surat has emerged as one of the major port of international importance on the map of the world trade. A Portuguese traveller named Barbosa during his visit to Gujarat in 1514 has described Surat as a city of great trade in all classes of merchandise, a very important seaport yielding a large revenue to the king, and frequented by many ships from Malabar and many other ports.
Shortly before Barbosa was in Gujarat, Surat is said to have been burnt by the Portuguese in 1512. Surat reportedly suffered from a wholly unprovoked, and piratical raid, in 1530, a second time by the Portuguese under the leadership of Antonio da Silvaria. Though the assailants were opposed by a guard of 300 horses and 10000 foot, but at the first charge the defenders fled, and the town was taken and burnt. As they were still at the war with the Gujarat King, the Portuguese again burnt Surat in the next year, 1531. The Ahmedabad king Sultan Mahmood-III (1538-1554), who was very much annoyed by these frequent destructions of Surat, ordered for building a very strong castle and entrusted the work to Safi Agha, a Turkish soldier who had been ennobled with the title of Khudawand Khan. He was provided with substantial budget and was ordered to plan and build a very strong castle. Khudawand Khan initially selected three alternative sites for building the castle
out of which the last alternative i.e. bank of the river was selected and finalised by the king. It has been reported that during construction stage the Portuguese made several attempts to prevent Khudawand Khan from completing the work both by bribery as well as by force failing which, they came to attack with several vessels armed with cannons, but could not succeed in preventing him from building the castle. He completed the construction of this castle in the year 1546.