The first houses were built around the Gothic church of St Michael at Mala banda on the northern side of the bay.
Another group of houses developed around the Baroque church of St John on another rock.
This church, of polygonal layout, is decorated as an example of beautiful stonework, on the picturesque town square, and surrounded by the Renaissance and Baroque houses of the boat owners of Jelsa.
St. Mary’s church dates back to 1331. The reconstruction of the church occurred in 1535, and later in 1573 two years after the successful defence of  Jelsa in which the church-tower played such a decisive role. The facade and the bell-tower were re shaped in the 19th century. The brotherhood of Sts Fabian and Sebastian ordered a painting by Veronese in 1576 for the high altar, which was sadly destroyed later in the fire caused by a thunderbolt on December 26, 1771., together with the miracle statue of Our Lady ("Gospa Stomorena"), which was later restored. Another valuable artefact is the altar polyptych by Urban de Surgge from Bavaria from 1636-45. The sacristy houses a collection of church vestments, liturgical vessels, valuable crucifixes, etc. The painting of the Madonna with St. Fabian and St Sebastian in the large southern chapel is a work of the Flemish-Venetian painter Pieter de Costera (1612/14— 1704).

The picturesque peninsula of Gradina is the location where an Augustinian hermitage was established in 1599, and replaced by the cemetery in 1807. An ancient castrum stood there in Roman times.

Tor - a Greek reconnaissance tower built of megalithic (Cyclopean) blocks in the 4th and 3rd centuy BC, overlooks the whole area of the islands of Hvar and Brač as far as Šolta and the Makarska littoral. V. Pribojević devoted it a few sentences in his famous ode to Hvar. Demetrius of Hvar, reigned here for some ten years until the Roman occupation. The fortified "Town" (Grad) was built one kilometre to the east on the hill above the very ancient road leading to the eastern part of the island.

According to tradition, this fortification - called also Galešnik - was the base of the mighty family Slavogosti, who, in an alliance with the people of Omiš, and under the leadership of Galeša (The Black), rebelled in 1310, against the Venetians who destroyed this town. Galešnik is situated slightly more to the east on an isolated cliff, and is the ruins of an old fortress. The base of this structure is Illyrian and the superstructure Roman. In the early Middle Ages the fortress was still inhabited. Its present name — like that of the islet at the approach to Hvar harbour — is believed to derive from the nobleman Galeša Slavogosti who led an unsuccessful rebellion against Venice in 1310 using this fortification as his stronghold, and was forced to flee. The road leading to Grad is an old Roman road overlaying an older, Illyrian road.