There is nothing to fix.
This issue is occurring because you are masking the layer with some other layer. And using Alpha Over and Invert to create a revealing mask.
Can you share the files, I am confused there must be three layers or you are doing something different?
You could replace the outline with an advanced outline (Layer > Make Advanced Outline) and use the width points to exclude that area with zero width or cusps.
Why set the ‘Outline’ layer to straight onto? The green line is going along the same path as the black one, so you could delete the black outline, then duplicate the green, then change the duplicates colour to black, then alter it to have the (stylistic) appearance of the one you deleted.
The black is my goal outline.the green is mask for black.
my goal outline with multi width and the advanced outline have only two width handle.
You can add more width points. 