Pages: [1]   Go Down

Author Topic: Problems with layer mask in CS3...  (Read 6117 times)

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« on: April 17, 2010, 07:15:12 pm »

Hi!

In this image, I want to selectively darken the 2 leftmost trees (in the background), which drown in the blown sky behind them.

[attachment=21566:Layer_mask.jpg]

My plan was to
1-   create a layer mask from a selection of the blown highlights,
2-   then use the unselected part of the mask as a base for a new mask by painting black on the rest of the layer where  I do not want to darken the image; this would mean confining the selection to the sky area between the rock wall (left) and the left tree in the middle ground.
3-   apply a tone curve to the latter mask.

I thought I had found the right tool when I read (Real World Photoshop CS3, chapter 9: The Select Menu > Layer Masks, p 522 in my edition): "Note that you can make a selection before clicking the [Add Layer Mask] icon. In this case, the program "paints in" the unselected areas with black for you (on the layer mask)."

But I got stuck already in step 2.
My PS does this painting-black only on the layer mask thumbnail in the layer icon in the palette. The image only shows the marching ants.

When I paint in the image on the mask layer manually, the painting becomes invisible outside the selection, as exspected. But a tone curve clipped to the layer makes no difference as long as the clipping mask is enabled. If it is released, the curve affects the whole underlying image.

A second problem:
When I started trying, I selected the *blown* highlights only, i.e. the sky. In later attempts, I was unable to re-do this, and chose Highlights from the Select menu inside the Color Range dialog box. Now, when I try Select > Color Range, I am presented with a b/w version of the image, based on a selection of the highlights as defined by the Color Range dialog, not only the blown ones.

Layers are, from top to bottom:

Hue/Sat
   Curves 2, clipped to
Layer 2 with layer mask based on highlights
   Curves 1, clipped to
Luminosity mask
Background

Is there something I have misunderstood or overlooked? (The layer IS visible).

Or is it a no-longer-working-under-Snow-Leopard?

Or could it have something to do with this:
Recently, when I open an image in PS using the context menu, there are 2 entrances of PS:
1- (1...2:09:30:00 cutoff; r branch]))
2- (1...16:39:00 cutoff; m branch]))

This has persisted through an un-install and re-install of PS.

Hoping for your help ...   Hening.

Greg Barnett

  • Newbie
  • *
  • Offline Offline
  • Posts: 43
    • http://
Problems with layer mask in CS3...
« Reply #1 on: April 17, 2010, 10:24:31 pm »

Quote from: Hening Bettermann
Or could it have something to do with this:
Recently, when I open an image in PS using the context menu, there are 2 entrances of PS:
1- (1...2:09:30:00 cutoff; r branch]))
2- (1...16:39:00 cutoff; m branch]))

This has persisted through an un-install and re-install of PS.


Those branch messages would appear to indicate that you're running prerelease versions of the application. But considering that you're using CS3, that seems unlikely since they should have long since expired. Where did you get your installer? Are you sure you've got a legitimate copy?
Logged

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #2 on: April 18, 2010, 06:01:12 am »

Thank you, Greg!
Yes I'm sure I have a legitimate copy of PSCS3. I installed from the CD, then updated from 10.0 to 10.0.1 over Internet. - Could it be that the current Adobe Updater recognizes this as a non-current version and calls that pre-release?
« Last Edit: April 18, 2010, 06:33:11 am by Hening Bettermann »
Logged

stamper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5882
Problems with layer mask in CS3...
« Reply #3 on: April 18, 2010, 06:17:28 am »

A couple of suggestions. These might not be 100% correct. Could you not try it without the hue/sat layer. Either flatten it or merge visible. There could be a problem with the layer order. Also when I do this type of thing I make the selection - it doesn't matter how it is achieved - and then Ctrl  + J ( I am using windows) and put the selection on a new layer with the mask. The most obvious thing is to ensure that the mask is being used. I find sometimes when I activate a new layer and tick the box - are you using this - for Create  Clipping Mask then it sometimes applies the change to the whole image. I think you have to ensure that you are working on the mask at all times and not the layer.

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #4 on: April 18, 2010, 06:25:03 am »

[quote
A second problem:
When I started trying, I selected the *blown* highlights only, i.e. the sky. In later attempts, I was unable to re-do this, and chose Highlights from the Select menu inside the Color Range dialog box. Now, when I try Select > Color Range, I am presented with a b/w version of the image, based on a selection of the highlights as defined by the Color Range dialog, not only the blown ones.
/quote]

Ooops - I found the answer to THAT part: "When you open Color Range, Photoshop creates a selection based on your foreground color."

Hening

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #5 on: April 18, 2010, 06:34:44 am »

Thank you, stamper. I'll try to look closer into this. This is my first "advanced" PS edit...

stamper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5882
Problems with layer mask in CS3...
« Reply #6 on: April 18, 2010, 06:48:38 am »

Quote from: Hening Bettermann
Thank you, stamper. I'll try to look closer into this. This is my first "advanced" PS edit...

I have just had another look at your post. You have two layers with selections. It could be that PS doesn't know which layer to clip to? You might need to re arrange the order of the layers so that you clip to the correct selection??????? Again not 100% sure but I might be!

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #7 on: April 18, 2010, 12:44:18 pm »

I tried to select first and then use cmd-J, to put the selection on a new layer, but got the error message
"Could not make a new layer from the selection because the selected area is empty".
Makes sense in a way, since the selected pixels are all blown.
I have no problems keeping track of that I'm working on the layer mask rather than the layer. Also, making the underlying layers invisible makes no difference.

Hmm...

stamper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5882
Problems with layer mask in CS3...
« Reply #8 on: April 19, 2010, 04:10:46 am »

Quote from: Hening Bettermann
I tried to select first and then use cmd-J, to put the selection on a new layer, but got the error message
"Could not make a new layer from the selection because the selected area is empty".
Makes sense in a way, since the selected pixels are all blown.
I have no problems keeping track of that I'm working on the layer mask rather than the layer. Also, making the underlying layers invisible makes no difference.

Hmm...

Again not 100% sure but if you mask the blown areas in Quick mask it becomes an alpha channel then you should be able to cmd + J to a new layer?

stamper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5882
Problems with layer mask in CS3...
« Reply #9 on: April 19, 2010, 04:35:04 am »

I don't know if this is helpful? Scroll down to the bottom of the page.

http://www.earthboundlight.com/phototips.html

hsmeets

  • Full Member
  • ***
  • Offline Offline
  • Posts: 184
Problems with layer mask in CS3...
« Reply #10 on: April 19, 2010, 09:02:41 am »

1) load the file into PS
2) make the selection any way you like
3) go to the layer tool window and at the bottom there is an icon to create a new adjustment layer, click that icon
4) select the type of adjustment layer you would like to create
5) you have now the dialog of the adjustment choosen, do adjustement and click OK

in the layer window a 2nd layer should be visible with a mask (black/white) attached, the mask is as the selection made.

TIP:
Most often you'll see hard edges on the local adjustments you did, to have a smooth transition from full adjustment to no adjustment at all do the following between step 2 and 3 (just one of many possible ways to do this):
2a) after the selection has been made: switch on (what I call) rubylith mode, you'll get a red mask overlaying the image those parts you not selected.
2b) select the gaussian blur filter and apply it depending on selection made and subject with a certain pixel radius
2c) the gaussian blur is not applied to the image but the red mask giving the edges of the mask a smooth transition (gradient)
2d) now swtich red mask off again. The mask is converted back into a selection with the edges having a "feather"
continue wihth step 3.

You'll now see that, depending on the gaussian blur, the adjustment between not selected and selected part of the image goes smoothly from full adjustmust to no adjustment at all. The mask, although often hard to see also has a gradiant from white to black via greys.

A layer mask is not just a black/white = none/full adjustment but all greys between black and white are proportional to the effect of the adjustment.





Logged
Cheers,

Huib

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #11 on: April 19, 2010, 08:59:16 pm »

Thank you both for your concern.

> Again not 100% sure but if you mask the blown areas in Quick mask it becomes an alpha channel then you should be able to cmd + J to a new layer?

To handle problems one at a time, I have now made a copy of the image without the luminosity mask and adjustment layers.

Now if I select the highlights, change into Quick Mask mode, leave the default choice of Color Indicates: Masked Areas, set the color to black and the opacity to 100%, I can see what  I had intended.  But when I then press cmd-J, a new layer is created, but it seems to be a dup of the Background. No selection nor mask is visible neither in the Background nor the new layer, neither viewed one at a time nor both. When the new layer is selected in the palette, the thumbnail icon has the dashed border, I don't recall and can not retrieve on the fly what that means.

When I delete the new layer, the Background has lost the selection, and in the Select menu, Color Range is dimmed!
When I close the image, save changes, and re-open it, the Background is the Quick Mask! Meaning I see a 1-bit b/w image. The original image is lost!
Looks to me like PS has swapped the Background and the new layer.
Is this intended behavior??

>
1) load the file into PS
2) make the selection any way you like
3) go to the layer tool window and at the bottom there is an icon to create a new adjustment layer, click that icon
4) select the type of adjustment layer you would like to create
5) you have now the dialog of the adjustment choosen, do adjustement and click OK
>

Yes I can make that work.
But before I make (curve) adjustments, I want to change the mask: paint in it to expand the masked areas (most of the sky in the areas other than the one between the rock wall at left and the leftmost tree in the middle ground; then invert the selection for the area that is left, so that the 2 fade trees in the background and their branches are selected and the sky in this area de-selected; and then apply a darkening tone curve which in this way will only act on the 2 trees in the background and their branches; which may to some degree cure the overblown look of these branches, darken these 2 trees and leave the sky blown (to preserve the sparkle; the sky is blown, there is no rescue).
 
So I need to have the mask in the whole image size to paint in it.

This passage from your link, stamper, did the trick:

"If instead you Alt/Option-click on the mask rectangle [of the adjustment layer] you will be able to see the mask in black and white in the main window in place of the image itself."
Bingo!
I'm sure this is to be read in the book, but...
Now the only thing I lack is the command to invert the mask or the selection. I think  I have seen something like this somewhere, but can not retrieve it now.
And the mystique behavior of the Quick Mask might also be nice to have cleared up, even though I do not seem to need it right now.
BTW the 2 branches of PS showing in the Context Menu refer to version 10.0 and 10.0.1 respectively. But they should be merged into 1 PS, shouldn't they?
Thanks again! - Hening.

Greg Barnett

  • Newbie
  • *
  • Offline Offline
  • Posts: 43
    • http://
Problems with layer mask in CS3...
« Reply #12 on: April 19, 2010, 10:08:16 pm »

Quote from: Hening Bettermann
BTW the 2 branches of PS showing in the Context Menu refer to version 10.0 and 10.0.1 respectively. But they should be merged into 1 PS, shouldn't they?

Hening,

Sorry for the false alarm on the branch info, it took me a while to figure out what you were referring to as the Context Menu and where it was visible. As to why you're seeing both, I'm guessing that the patch just appends the info rather than replacing it.

 
Logged

stamper

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 5882
Problems with layer mask in CS3...
« Reply #13 on: April 20, 2010, 04:12:02 am »

Now the only thing I lack is the command to invert the mask or the selection. I think I have seen something like this somewhere, but can not retrieve it now.

Unquote

In Windows Ctrl + I inverts the mask from black to white. Did you look at the channel palette to see if an alpha channel had been created in the process? If so loading it onto a layer means that it can be edited. Added to or subtracted. IMO colour range isn't the best tool to make a mask. I would concentrate on the Quick mask. I think it is easier to work worth. BTW all selections should be saved so that the alpha channel shows up under channels.

hsmeets

  • Full Member
  • ***
  • Offline Offline
  • Posts: 184
Problems with layer mask in CS3...
« Reply #14 on: April 20, 2010, 04:22:06 am »

Quote from: Hening Bettermann
Yes I can make that work.
But before I make (curve) adjustments, I want to change the mask: paint in it to expand the masked areas (most of the sky in the areas other than the one between the rock wall at left and the leftmost tree in the middle ground; then invert the selection for the area that is left, so that the 2 fade trees in the background and their branches are selected and the sky in this area de-selected; and then apply a darkening tone curve which in this way will only act on the 2 trees in the background and their branches; which may to some degree cure the overblown look of these branches, darken these 2 trees and leave the sky blown (to preserve the sparkle; the sky is blown, there is no rescue).
 
So I need to have the mask in the whole image size to paint in it.

To adjust a selection before the adjustment layer is createf by painting a mask

- Make a 1st rough selection of the 2 trees in the background you want to apply an adjustment on.
- Switch on Quick Mask Mode
- Paint in the mask with the regular painting tools: brush, eraser, etc to alter/finetune the mask
- Switch of Quick Mask Mode, the mask is now converted to a selection again.
- Create adjustment layer, do adjustment.

Logged
Cheers,

Huib

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #15 on: April 20, 2010, 05:34:37 pm »

Hi again!

stamper
> colour range isn't the best tool to make a mask.

hsmeets
> Make a 1st rough selection of the 2 trees in the background

Well I tried the Select > Color Range because I find it impossible to manually select all those small branches, yes and even the trees and the rock wall. There must be a more elegant method.

As I fiddle with this, I'm becoming aware of, that what I had in mind might be spelled out like this:

1-To isolate the background trees AND the sky around them, I would select the sky by the color range. Then, with a medium broad brush,  paint over the edges of the rock wall and the leftmost tree in the middle ground in a way, that the selection automatically stopped my painting at the edge of these 2 objects.

2-Then use an even broader brush to paint over the rest of the image except the part between the rock wall and the leftmost middle ground tree, adding this painting to the first one, resulting in a contiguous mask.

3-Then, inside the remaining area, (= the background trees + sky) I would again select the sky by the color range, probably this time with a different fuzziness, and mask if off.

4-THEN I would apply a tone curve to the rest = the background trees and their branches.

Is it possible to do this??

If you have an answer at hand, I'd of course love to hear it. But don't use much time on it. Because I start wondering if I should try to learn PhotoLine rather than Photoshop? I get a feeling that by the time I have learned PS, CS3 will no longer run under the Mac OS that may be current at that time. Already now, it is no longer officially supported. And I will not spend 700 $ every time the Mac OS is upgraded.

Best regards - Hening.

hsmeets

  • Full Member
  • ***
  • Offline Offline
  • Posts: 184
Problems with layer mask in CS3...
« Reply #16 on: April 21, 2010, 04:04:06 am »

Quote from: Hening Bettermann
As I fiddle with this, I'm becoming aware of, that what I had in mind might be spelled out like this:
1-To isolate the background trees AND the sky around them, I would select the sky by the color range. Then, with a medium broad brush,  paint over the edges of the rock wall and the leftmost tree in the middle ground in a way, that the selection automatically stopped my painting at the edge of these 2 objects.
2-Then use an even broader brush to paint over the rest of the image except the part between the rock wall and the leftmost middle ground tree, adding this painting to the first one, resulting in a contiguous mask.
3-Then, inside the remaining area, (= the background trees + sky) I would again select the sky by the color range, probably this time with a different fuzziness, and mask if off.
4-THEN I would apply a tone curve to the rest = the background trees and their branches.
Is it possible to do this??

Step by step.

1) Load file into PS
2) Turn on quick mask mode
3) I'm not 100% but you should see a red mask overlaying the image. If not: select a big brush and paint the mask over the entire image.
4) select the eraser tool and adjust size and hardness to the area you want to alter (those trees and air).
5) erase the mask in those areas of the image where you want the adjustment to happen, repeat 4 and 5 as needed.
6) switch quickmask mode off, you'll see now that a selection is made on the area you want to adjust
7) add an adjustment layer (while the selection is active to have that selection as layer mask), do the adjustment.


Logged
Cheers,

Huib

Hening Bettermann

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 945
    • landshape.net
Problems with layer mask in CS3...
« Reply #17 on: April 21, 2010, 06:56:23 am »

Hi Huib!

Thank you very much for your concern. However I'm afraid your method is missing my point. Regardless if I paint or erase (your #5), I will have to manually hit the border between the rockwall/tree and the sky. This is what I hoped to avoid. But if it can't be done otherwise, I may end up doing it your way.

Thanks again! - Hening.

hsmeets

  • Full Member
  • ***
  • Offline Offline
  • Posts: 184
Problems with layer mask in CS3...
« Reply #18 on: April 21, 2010, 11:34:40 am »

Quote from: Hening Bettermann
Hi Huib!

Thank you very much for your concern. However I'm afraid your method is missing my point. Regardless if I paint or erase (your #5), I will have to manually hit the border between the rockwall/tree and the sky. This is what I hoped to avoid. But if it can't be done otherwise, I may end up doing it your way.

Thanks again! - Hening.

Zoom in for detail work :-)
Logged
Cheers,

Huib
Pages: [1]   Go Up