The construction of the prestigious architectural complex, residence of the noble Ferretti family, headquarters of the Museum since 1958, dates back to the mid-16th century. The external facade is attributed to Antonio da San Gallo the Younger, a civil and military architect active in Ancona in the 1540s.

The decoration of the main floor, commissioned by Count Angelo di Girolamo Ferretti, was executed by the Mannerist painter Pellegrino Tibaldi and his assistants shortly after 1560; elaborate wooden ceilings were created, the walls of the main hall and several connected rooms were frescoed: the room of emblems, the room of myths, the labyrinth room. The 3rd floor features frescoed ceilings with grotesques, fantastic landscapes, monsters, mythological figures, and masks in the late 16th-century Roman style (Federico Zuccari, Luzio Luzi?).

In 1759, the original building was renovated, probably based on a design by Luigi Vanvitelli, with the expansion of the facade, the construction of the balcony, the central portal, the grand staircase, and the hanging terrace, with a portico and overlying loggias. This expansion was enriched with statues and busts probably from the Ancona workshop of Gioacchino Varlè (1731-1806). Also from the 18th century are the exquisite vaults with grotesques in some small drawing rooms on the 3rd floor.

The palace has undergone several restoration interventions: in the years 1928-1931 on behalf of the Ferretti family, who entrusted the work to the Milanese architect Tornaghi; in the early 1950s, following the air bombings; finally, in the 1970s-1980s, after the Ancona earthquake of 1972.