"Feather" sliders in the left part of the dialog window let you soften the mask edges.
Since this adjustment is utterly non-destructive, later on you will be able to restore the mask sharpness without any loss of quality.
- So, select a masked layer in the Layers pane.
- Open the "Masks" dialog.
- To blur edges of the pixel mask, move the "Feather" slider in the "Pixel Mask" panel.
- To soften the vector mask's edges, use the "Feather" slider in the "Vector Mask" area (screenshot).
- To feather the smart filter's mask, use the respective slider in the "Filter Mask" panel.
- If you need to restore the mask sharpness, open the "Masks" dialog once again and reset the "Feather" value to 100%.
NB Unfortunately, the Windows version of PSE doesn't update the image while the script is running.
You can preview the adjustment result on the document thumbnail in the Project Bin (see screenshot).
To update image in the document window you have to close the "Masks" dialog.