Skip to main content

Content Moderation

Related docs: Hosting Infrastructure · Design

1. Overview​

All image uploads on Pakashop — product images, shop logos, shop banners, and KYC documents — pass through an asynchronous content moderation pipeline powered by Sightengine.

The moderation service runs as an independent microservice (moderation) deployed on the same EC2 instance.

2. Architecture​

3. Moderation Service API Contract​

The moderation service provides a specialized endpoint for evaluating uploaded assets.

POST /moderate​

Interface: The endpoint accepts an image source URL and metadata identifying the asset type and its unique platform ID.
Response: Returns a boolean approved status and a set of confidence scores for various safety categories (NSFW, violence, etc.).

The Express API triggers this evaluation asynchronously upon every successful image upload. If the moderation service is temporarily unavailable, the failure is logged and the asset is queued for a background retry, ensuring that high-throughput upload operations remain unblocked.

4. Sightengine Configuration​

The platform utilizes specialized computer vision models to maintain community standards. Key evaluation criteria include:

  • NSFW Detection: Automated flagging of adult or suggestive content.
  • Violence/Gore: Detection of violent imagery or hate symbols.
  • Threshold Policy: Confidence scores below platform safety thresholds trigger an immediate manual review state.

Security: All communication with the moderation provider is secured via platform-level API credentials managed through centralized environment configuration.

5. Violation Actions​

SeverityAction
Flagged (automated)Asset hidden from public display; ModerationFlag record created; admin notified
Reviewed & approved (admin)Asset restored; ModerationFlag.status = APPROVED
Reviewed & rejected (admin)Asset permanently removed from Cloudinary; product/shop suspended
KYC document flaggedSeller application paused pending manual review

6. Retry Logic​

If the Sightengine API is unavailable:

  1. The moderation service retries up to 3 times with exponential back-off (1 s, 2 s, 4 s).
  2. If all retries fail, the asset is marked moderationStatus = PENDING and queued for a scheduled re-check job (runs every 15 minutes).
  3. Assets in PENDING status are visible by default (optimistic approval) unless the asset type is KYC_DOCUMENT, which defaults to hidden until explicitly approved.

7. Admin Review Dashboard​

Flagged assets are visible in the admin dashboard under Content Moderation → Flagged Items. Admins can:

  • View the flagged image alongside the Sightengine confidence scores.
  • Approve — removes the flag; asset becomes public.
  • Reject — deletes the asset from Cloudinary; associated product/shop is suspended.
  • Escalate — marks for senior review without taking action.

All actions are logged in ModerationFlag.reviewedAt, reviewedBy, and reviewAction.