Since the Bluffs are located in the far east of Toronto, they are relatively easy to get to. If you have a car, it's not hard to get there. The thing to be clear about is where exactly you want to go, as there are many different parks and a handful of different car parks to choose from (e.g. Bluffers Park Parking and Bluffers Beach Parking).
The drive to the Bluffs from the west end, north end or downtown Toronto takes between 20 and 40 minutes - depending on where exactly you are coming from. The Bluffs are about 15 kilometres from downtown by car, so it's not far.
You can take the path along Queen Street East to The Beaches (a neighbourhood) and then turn left to follow Kingston Road to Brimley Road South.
Look at this list of Escorts in Toronto
Alternatively, you can take Danforth Road east, but will eventually have to turn south to get to the coast. If you don't have a rental car, you could also take an Uber or a taxi.
From Toronto to Scarborough by public transport
You don't have to drive to the Bluffs to enjoy them. There are a handful of combinations that will get you to the Bluffs by public transit.
The short version is: you can either take the GO train to Scarborough GO station (from Union Station) or the TTC (Toronto underground) to Kennedy Station or Victoria Park Station.
From these closest stops, you can take a bus to Kingston Road. Whichever route you choose, you'll end up getting off somewhere along Kingston Road and walking along Scarboro Crescent Street or Brimley Road S towards the water.
If you're going to be in Toronto for a while (or living there), you should get a Presto Card for public transport. It will make your tickets a little cheaper. Of course, you can also buy a paper ticket (for the GO train).