We need "clear" definition what values should be used as result outside of the mask.

Al least, this is why cv::compare() doesn't have mask support.

P.S. If your mask is a rectangle - then you should use ROI approach instead.

We need "clear" definition what values should be used as result outside of the mask.

Al least, this is why cv::compare() doesn't have mask support.

Yes, that's what I thought when I started working on the feature.
My most useful usecase is to leave outliers untouched. This assumes that the dst image was pre-existing, otherwise they will be undefined pixel values after being allocated by cv::threshold()
I use such a feature to apply different binarizations to different (non rectangular) parts of an image according to some criteria.

@chacha21 Could you list use-cases for the PR?
The proposal was discussed on weekly core team meeting and the team has doubt if we need the change. Alexander's concern is on of reasons.

@chacha21 Could you list use-cases for the PR? The proposal was discussed on weekly core team meeting and the team has doubt if we need the change. Alexander's concern is on of reasons.

Ok. First of all, no problem to decline the PR, I won't try to force push it.

Here are my two use cases :

case 1

• A gray-level image with different areas (not rectangular) having different lighting conditions
• not a single threshold will binarize the whole image, there must be one threshold (from OTSU) per area
• currently, I have to use multiple masks and combinations to build to final image, and computing the OTSU threshold for a non rectangular area is a problem because of outliers pixels

case 2

• a gray-level image is displayed to the user
• the user can draw non-rectangular ROIs (e.g. RotatedRect areas) and want to binarize inside those areas in order to preview how binarization will highlight information of interest

Hello @chacha21 Sorry for delay. We discussed to the PR on the Core team meeting and we are ready to merge the PR with several fixes:

• The new function dst should be defined as InputoutputArray and the function should check that the output buffer is pre-allocated and has enough size. Document the behaviour.
• The dst value with mask==0 should not be modified as you in your proposal. The behaviour is documented.
• HAL is not needed here. The case is too exotic.
• We recommend to use another name like thresholdWithMask to reduce chance of API collision in Python and Java.
• Add python test for the new function to ensure it's wrapped correctly.

The fixes are proposed to get rid of undefined behaviour for the case where mask==0 and wrap things to Python, Java, etc correctly.

Please also rebase and fix merge conflicts.

• Add python test for the new function to ensure it's wrapped correctly.

I am sorry I have 0 skills in Python