Its actual name is 船島 Funashima. It has been called Ganryū's island since about 1612, after the man who died on it.
It sits in the strait between Honshū and Kyūshū, and you can go: ferries run from Shimonoseki and from Moji. There are two statues on it, mid-fight, permanently.
What is reasonably firm. A duel took place, on or near that island, in or around 1612, between Musashi and a swordsman of standing known as Ganryū — sufficiently established that the encounter required official permission and drew an audience. Musashi won and Ganryū died.
That is the whole of it. Roughly two sentences.
Everything else is later, contested, or both — the oar, the lateness, the laundry-pole blade, the swallow reversal, the youth, the beauty, and the name Sasaki Kojirō itself, which does not appear in the earliest account.
And the direction of the accretion is the interesting part. Look at what three centuries added: a name, a face, a signature technique, a tragic overreach. The tradition did not embellish Musashi. It embellished his opponent. The winner's legend was built by making the loser bigger.