Your basic premise, that a website needs to stand out, is correct in my opinion. There are a lot more ways to screw it up than there are to get it right which is why there are so many dismal websites and websites that just use templates.
Your question is a little hard to answer, however, because you haven't laid the groundwork upon which an answer can be formed.
You first need to identify an audience—a wedding site will be much different than a fine-art photography site, which will be much different than a commercial photography site.
You also need to understand and be able to articulate your brand. Who are you? Why should the audience in the above question care? Answering questions like these will often lead to better answers to questions like background color and typeface choice. Also, the better answer you have to this question, the less you'll want to use a template.
Having said that, I've noticed some common threads among good websites:
1.The editing is ruthless. You won't see good websites with multiple versions of the same image, like a color and sepia next to each other. You might think it demonstrates flexibility, but really it screams, "I don't know what I want." Editing can be painful because we are inevitably attached to certain images for very personal reasons that are irrelevant to the audience. It can help to have a trusted third party give you candid advice. One poor image can throw off the whole thing and is much more damaging than one less image.
3. Quick. Not only the downloading but also the interface. This is especially true if your audience includes art buyers and photo editors. They want to quickly scan your work and get a sense of who you are photographically. Don't put up road blocks and require lots of clicks to see images. If you need to write instructions about how to use the interface, you're doing it wrong.
4. Search engine friendly. Don't display text as images unless absolutely necessary. Be mindful of page titles, descriptions, etc..
5. Obvious things: no music, no faux borders, don't let the interface distract from the message. Above all, you need to demonstrate good taste.
Good luck. I personally think developing a website is torture. It's a long process that, as far as I can tell, never really ends.