Skip to content
This is NOT an official site of the City of New York. Click here for official AI reports.

NYC Office of Technology and Innovation

NYC - DTPR AI Register

AI-Assisted Video Captioning for Social Media

What it is for

Accessibility
  • Adds captions to social media videos produced by the NYC Comptroller's office, making video content accessible to people who are deaf or hard of hearing.

NYC AI Use Cases Register 2025

Inform
  • Supports information presentation by ensuring that social media videos from the Comptroller's office carry accurate caption text before publication.

NYC AI Use Cases Register 2025

Run by
Office of the Comptroller
Where
No fixed location
Kept
Not stated by the NYC Office of Technology and Innovation.
Shared with
Accountable organization, Vendor

What it is for

AccessibilityInform

Mirage is a captioning tool used by the NYC Office of the Comptroller to automatically add captions to social media videos. Staff upload videos and the tool generates caption text that staff then review before publishing. The tool is used approximately once a week and human staff monitor its outputs before any video is made public.

What it collects and what happens to it

Data taken in

Biometric
Anonymized data
  • Video taken for social media, which may contain audio of voices and visual depictions of people. The AI processes these signals to generate caption text but does not perform identity recognition.

NYC AI Use Cases Register 2025

Processing

Speech & Audio
  • Converts spoken words in the social media video into caption text (speech-to-text transcription). The tool uses audio signals from the video to produce accurate captions.

NYC AI Use Cases Register 2025

What it does

Sensing (Perceptive AI)
Human decides
  • The system processes video input to extract and transcribe the spoken or textual content, converting audio/visual signals into structured caption text. Staff review and approve captions before publishing.

NYC AI Use Cases Register 2025

Creating (Generative AI)
Human decides
  • The tool generates caption text to be overlaid on the video, creating new textual content that did not previously exist as a separate artifact. Human staff review all generated captions before publication.

NYC AI Use Cases Register 2025

Outputs

Generated content
Anonymized data
  • Video output with captions added to the input video, ready for social media publishing after human staff review. The captions are new text generated by the AI tool.

NYC AI Use Cases Register 2025

Run by

Office of the Comptroller
  • The NYC Office of the Comptroller's Communications Department deploys Mirage to caption social media videos produced by the Comptroller's office.

NYC AI Use Cases Register 2025

Built by

Mirage
  • Mirage is the off-the-shelf vendor whose captioning product is licensed and used by the Comptroller's Communications Department. Training data is proprietary to the vendor.

NYC AI Use Cases Register 2025

Kept for

Not stated by the NYC Office of Technology and Innovation.

Shared with

Available to the accountable organization
  • The captioned video output is available to the NYC Office of the Comptroller's Communications Department, which reviews and publishes the final video on social media.

NYC AI Use Cases Register 2025

Available to vendor
  • Training data is proprietary to the vendor. The extent to which the vendor accesses processed video or caption data during tool operation is not specified in the register entry.

NYC AI Use Cases Register 2025

Stored

Not stated by the NYC Office of Technology and Innovation.

How to read the colours

Can it identify you?

Anonymized data
Data about people with the link to who is broken. Stripped of identifiers, blurred, aggregated, or noised so this system can’t reasonably tie a record back to an individual.
Pseudonymous data
Each person’s data is tied to a token (hash, ID, template) that lets this system recognise the same person across events, but the token itself doesn’t reveal a name. Reidentification is possible with extra information.
Identifiable data
The data either contains a direct identifier (name, address, account name, recognisable face or voice, plate number) or carries a token this system uses to look up legal identity during processing.

Who completes the loop?

Human decides
This mode suggests; a person decides what to do next. The AI is always advisory — a human is in the loop on every decision. Example: a triage tool ranks cases for a clinician who chooses which to see first.
Human executes
This mode decides; a person carries out the result. Example: an optimizer plans the day’s trash-collection routes, and drivers run them.
Autonomous
This mode decides and acts on its own. No person reviews each decision or carries out the resulting action.

Definitions from the DTPR standard. Amber is about your data, violet about who decides. The fuller the shape and the deeper the colour, the more identifying the data or the less a person is involved.

What you can do

Ask about this system

Questions go to the NYC Office of Technology and Innovation, not the vendor.

Your rights

  • Right to Be Informed of AI UseMembers of the public should be aware that the NYC Office of the Comptroller uses an AI captioning tool (Mirage) to add captions to its social media videos. This disclosure is available through the NYC AI Use Cases Register.
  • Right to Algorithmic TransparencyThe NYC Office of the Comptroller has disclosed that Mirage is an off-the-shelf captioning tool whose training data is proprietary to the vendor. Details about the underlying model and algorithm are not publicly available. Members of the public may contact the Comptroller's office for further information.

Risks and safeguards

  • Societal & cultural harmInaccurate automated captions could spread misinformation or misrepresent official statements from the Comptroller's office.Safeguard: The tool is described as 'monitored' with human staff reviewing outputs before publication, approximately once a week, to catch and correct errors before any video is made public.