AI-Assisted Photo-to-Service Request Matching for 311

What it is for

Inform

Helps 311 mobile app users identify the correct service request type for a problem they have photographed, and directs them to a pre-filled form or knowledge article.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Run by: Office of Technology and Innovation (OTI)
Where: No fixed location
Kept: Retained as required by Freedom of Information Law
Shared with: Not stated by the NYC Office of Technology and Innovation.
Your copy: You cannot see the data it holds about you. What you can do

What it is for

Inform

This tool lets New York City residents upload a photo of a problem through the 311 mobile app, and then suggests the most relevant service request type to help them file a report. The AI does not make decisions on behalf of the city — it only offers suggestions that the user can accept or ignore. No personally identifying information is collected or stored as part of this process.

What it collects and what happens to it

Data taken in

About a place

Anonymized data

Photos of physical problems in New York City (e.g. potholes, graffiti, broken infrastructure) uploaded by app users. Images are virus scanned and stored to comply with Freedom of Information Law requirements. No identifying information about the person submitting the photo is collected.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Processing

Computer Vision

Uses Microsoft Azure AI Foundry and Azure AI Search to analyze uploaded images and classify them against a catalog of 311 service request types. Training data is proprietary to Microsoft.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Classification & Prediction

Assigns ranked category labels (service request types) to the uploaded image. The output is a ranked list of suggestions, not a definitive determination.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

What it does

Sensing (Perceptive AI)

Human decides

The system interprets a photo uploaded by the user and converts the visual content into structured service request category suggestions. The user decides which suggestion, if any, to accept.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Deciding (Analytical AI)

Human decides

Classifies the visual content of the uploaded photo into one or more ranked service request categories. The system is informative — it does not file the report automatically; the user confirms and selects the appropriate category.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Outputs

A recommendation or prediction

Anonymized data

Suggested matching service request types presented to the user. The user selects the best match and is directed to a pre-filled form or knowledge article. No output is personally identifiable.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Run by

Office of Technology and Innovation (OTI)

The NYC Office of Technology and Innovation (OTI), Applications department, is the city agency responsible for deploying and operating the 311 Mobile App AI Smart Select feature.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Built by

Microsoft

Microsoft provides the off-the-shelf AI platform underlying this feature, specifically Microsoft Azure AI Foundry and Microsoft Azure AI Search. The training data is proprietary to Microsoft.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

IBM

IBM is the professional services vendor that supported the development of the 311 mobile application incorporating this AI feature.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Kept for

Retained as required by Freedom of Information Law

Uploaded images are virus scanned and stored to comply with New York State Freedom of Information Law (FOIL) requirements. No personally identifying information is stored.
Duration: as required by Freedom of Information Law

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Shared with

Not available to me

No identifying information is collected from users. Individuals cannot request their own data because none is linked to them.

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

Stored

Stored on 3rd Party Cloud

Uploaded images are processed and stored on Microsoft Azure cloud infrastructure (Azure AI Foundry and Azure AI Search) in compliance with FOIL requirements.
Duration: as required by Freedom of Information Law

NYC Algorithmic Tools Compliance Report — 311 Mobile App AI Smart Select

How to read the colours

Can it identify you?

Anonymized data: Data about people with the link to who is broken. Stripped of identifiers, blurred, aggregated, or noised so this system can’t reasonably tie a record back to an individual.
Pseudonymous data: Each person’s data is tied to a token (hash, ID, template) that lets this system recognise the same person across events, but the token itself doesn’t reveal a name. Reidentification is possible with extra information.
Identifiable data: The data either contains a direct identifier (name, address, account name, recognisable face or voice, plate number) or carries a token this system uses to look up legal identity during processing.

Who completes the loop?

Human decides: This mode suggests; a person decides what to do next. The AI is always advisory — a human is in the loop on every decision. Example: a triage tool ranks cases for a clinician who chooses which to see first.
Human executes: This mode decides; a person carries out the result. Example: an optimizer plans the day’s trash-collection routes, and drivers run them.
Autonomous: This mode decides and acts on its own. No person reviews each decision or carries out the resulting action.

Definitions from the DTPR standard. Amber is about your data, violet about who decides. The fuller the shape and the deeper the colour, the more identifying the data or the less a person is involved.

What you can do

Ask about this system

Questions go to the NYC Office of Technology and Innovation, not the vendor.

Your rights

Right to Be Informed of AI UseUsers are implicitly informed that the 311 app uses AI to suggest service request types when they use the Smart Select photo upload feature. The register entry is publicly available at the NYC Open Data portal.
Right to Algorithmic TransparencyThe tool is disclosed in the NYC Algorithmic Tools Compliance Report, publicly accessible via NYC Open Data. The system uses Microsoft Azure AI Foundry and Azure AI Search; training data is proprietary to Microsoft.

Risks and safeguards

Reputational harmmisclassification of a photo could route a user to the wrong service type, causing their complaint to be misfiled or ignored.Safeguard: the system is purely advisory — users select from suggested options and can override or ignore the AI suggestion. No automated decision is made on behalf of the user or the city.