Hi!
We are experiencing intermittent issues with the Remove Background API (/v1/segment) where the foreground detection produces inconsistent results for similar images, leading to scaling and positioning problems in our processed outputs.
Our Use Case
We operate an re-commerce photo processing service that uses the PhotoRoom Remove Background API to process product images. We process thousands of images daily using the following parameters:
Endpoint: https://sdk.photoroom.com/v1/segment
Parameters:
- scale: “85%”
- format: “png”
- size: “auto”
After receiving the response, we perform post-processing that relies on the foreground detection accuracy to crop, scale, and center the subject appropriately.
Case 1: Unsuccessful Result (Poor Detection)
{
“x-uncertainty-score”: “0.16845624923706054”,
“x-foreground-top”: “0.6451271”,
“x-foreground-left”: “0.32265624”,
“x-foreground-height”: “0.14194918”,
“x-foreground-width”: “0.38203123”,
“x-selected-endpoint”: “10.0.16.156:80”
}
Issue: The detected foreground is only 14% of image height, positioned at 64.5% down the image. Despite this clearly incorrect detection (the actual product occupies much more of the frame), the uncertainty score is relatively low (0.168).
Case 2: Successful Result (Good Detection)
{
“x-uncertainty-score”: “0.0010350193828344346”,
“x-foreground-top”: “0.2512755”,
“x-foreground-left”: “0.1875”,
“x-foreground-height”: “0.47066328”,
“x-foreground-width”: “0.61742425”,
“x-selected-endpoint”: “10.0.7.109:80”
}
This is the expected behavior: foreground occupies ~47% height, positioned appropriately.
Case 3: Valid Small Item (Correct Detection)
{
“x-uncertainty-score”: “0.00287881076335907”,
“x-foreground-top”: “0.67665815”,
“x-foreground-left”: “0.3219697”,
“x-foreground-height”: “0.108418405”,
“x-foreground-width”: “0.34848487”,
“x-selected-endpoint”: “10.0.66.248:80”
}
This is also valid: A legitimately small product with very low uncertainty (0.003).
The Challenge
We cannot reliably distinguish between:
Case 1: Poor detection (small foreground due to model error)
Case 3: Good detection (small foreground because the product is genuinely small)
The uncertainty score alone is not sufficient because:
Case 1 has uncertainty 0.168 (poor detection)
Case 3 has uncertainty 0.003 (good detection of small item)
Both have small x-foreground-height values, but only Case 1 is problematic.
Current Mitigation
We implemented retry logic that retries requests when x-uncertainty-score >= 0.5. However, this does not catch cases like Case 1 where the uncertainty is below our threshold but the detection is still incorrect. And this is also getting costlier for us.
# Current retry logic
if uncertainty_score >= 0.5 and attempt < max_retries:
Retry the request
continue
Given the above context, I would like to ask the following questions:
1. Is there a recommended way to detect poor foreground detection beyond using the uncertainty score? Are there other response headers or metrics we should consider?
2. Why does the model sometimes return low uncertainty scores for clearly incorrect detections? Is this a known issue with certain image types or conditions?
3. Would using segmentation.prompt help improve consistency? For example, providing hints like “product, item, main object”?
4. Is there variance between different backend endpoints? We notice different x-selected-endpoint values in responses - could this explain inconsistency?
5. Are there any best practices for handling edge cases where the foreground detection may be unreliable?
6. Is there a way to request a “confidence” metric that better reflects detection quality rather than just model certainty?
Currently this poor detection is causing real production disruption for us. This is out peak season and we getting bogged down by photo processing issue.
For your reference, one wrongly processed images:
and correctly processed photos:
Looking forward to hear from you!