What do the warning buoys say? If they say swim area you can not go inside of them. If they are navigational you can. My bet is that they are the outside limit of the swim area as one requirement is that the areas be marked with the orange/white buoys.

If the buoys are taken in then it's no longer designated as a permitted swim area as a permit requires that the buoys be present to be considered a swim area.