Free resources Go back to Introduction page tutorials
Introduction Tutorials Brushes Other

Masking 2/3: Vector mask

or; how to cut out stuff; part two.

This tutorial (in three parts) shows three different masking options in Photoshop - starting with the easiest. If you already are comfortable with one method, you can jump to the one you need:

  1. Layer mask
  2. Vector mask (this)
  3. Clipping mask

The file I will use in this tutorial can be found HERE - if you want to use it.

General tips:

Vector mask

A vector mask uses a 'path' as a mask. It will really help if you have some practise with the pen tool from e.g. Illustrator.

A "vector mask" is a lot better then a "layer mask" when it comes to sharp outlines and smooth curves. Use it as much as you can. It can even be combined with a layer mask (yes, a layer can have two masks) to get the best of both. If you need to e.g. blur a vector mask you can turn it into a layer mask by right-clicking it in the layers palette and choose "rasterize layer mask".

Note: The first 6 steps are identical to the "layer mask" tutorial.

  1. Bring your image into Photoshop (file -> open).
    In your layers palette it should only show a layer named Background
  2. Right-click (ctrl-click if you only have one button on a mac) and choose "Duplicate layer...".
  3. You now have two layers. Hide the background layer. This is just so that if you damage the image you can still bring the original background layer back.
  4. Now, click the "create new fill or adjustment layer"-button in your layers palette and choose "Solid Color..." This will create a layer with a single colour. We will use this as our background while masking.
  5. Choose a colour that differs from the subject in our image. I picked a green.
    Note: You can change the colour at any point by double-clicking the coloured icon on the layer.
  6. You now have a new color fill layer (with a white mask). If it is on top and covers your image, drag it down under the "Background copy" layer.
  7. I prefer to make the path before i apply it as a mask. To do that, select the pen tool and make sure the 2nd option from the left ('path') is selected and not the first one ('shape layer').
  8. Zoom in to e.g. 200% (use ctrl +/- on a pc, or apple +/- on a mac)
  9. To make a path, click-and-drag with the pen tool. This will create an anchor point, and two direction points. The line between the anchor point and the direction point IS NOT your path, it's just showing you in what direction the path will be heading. The line you see will actualy be a tangent to the curve at the anchor point.
  10. Now click and drag to make your second anchor point. A curve is created (instead of a straight line) between the two points. The shape of this curve is determined by the position of the direction points. Don't worry, practice and you will get the hang of this.
    RULE OF THUMB: The direction points should stretch 1/3 of the distance between the anchor points for a smooth curve. (i this image they are a bit longer)
  11. Now, continue to trace along the smooth edge until you git a "corner". Sometimes you might have to zoom in some more to see what you're doing.
  12. To create a "corner point" just click once instead of clicking and dragging. This will create an anchor point without those direction lines.
  13. Now continue to trace around the image, if you do something wrong you can UNDO (go back in the history palette). If you switch to another tool, you will have to "pick up" the path to continue it. You do this by clicking (or click-dragging) on the last point you made.
    I will now quickly show you the main tools for working with a path:
  14. Here are a couple of illustrations on how (not) to do it:
  15. Now, when you've traced around the object you must 'close' the path. You do this by clicking on the first point of your path. You have now created a path around your object. It may not be perfect, but since this is a path you can always change it later with the above tools.
    This is what my final path looked like (25% zoom). It's very detailed, so you might go for something a bit more simplified.
  16. Take the "Path Selection Tool" and select your path (click on the path once). We will now finally turn this into a mask - and see the fruit of our hard work.
  17. With your top layer, and the path, still selected. Go to Layer > Vector Mask > Current Path.
    This will take our current path, and apply it as a Vector Mask on the current layer.
  18. Your image should now have a vector mask. The thin line you see around the hand in my image is the path. It is invisible (think of it as a guide). To see how it really looks simply deselect the mask (click on it once in the layers palette) or select another layer.
  19. You're now done and can move the hand around, place it on a new background, whatever...
  20. My final .PSD file can be found HERE (4,5 MB) if you want to have a closer look.

The more observant of you may notice that i forgot to cut out the "hole" in the mask between two of my fingers. Oops! Don't worry, just make more paths, you can have them add/subtract to/from the resulting mask (or intersect). Just make sure you put them both into the mask. Boolean operations (add/subtract/etc) can be chosen on your toolbar when two or more paths are selected.