Classification

This page details the endpoint for sending images for classification through Foyer Insight. An SDK is provided at https://www.npmjs.com/package/@foyer-inc/insight-sdk

There is a 10MB limit on images sent to the classify endpoint.

Classify a single image

POST https://api.foyer.ai/Media/classify

This endpoint runs your image through classification and object detection. Accepts a single image with the parameters below, or an array of images.

Headers

Name	Type	Description
Authorization	string	Authorization token.

Request Body

Name	Type	Description
file	string	An image encoded as a base64 string
files	array	An array of images encoded as base64 strings
url	string	A URL to an image
urls	array	An array of URLs to images
force	boolean	A boolean representing whether to force a reclassification of an image
includeSegmentations	boolean	A boolean representing whether to return full segmentation masks for detections
includeTagpoints	string	A boolean representing whether to return a tagpoint attribute for each detection
detectionsRequested	array	An array of detection names as strings to be returned from the endpoint

200 Image successfully classified. All decimals are percents. Indoor and outdoor classifications will always be ranked last, regardless of confidence.

{ 
    "classifications": [
        { 
            "name": "living_room", 
            "confidence": 0.9996274709701538, 
            "rank": 1 
        },
        { 
            "name": "outdoor", 
            "confidence": 0.7036548946876468, 
            "rank": 2 
        }
    ],
    "detections": [
        { 
            "class": "floor",
            "area":  0.1321029663085938, 
            "boundingBox": [
                            0,
                            0.091796875,
                            0.998046875,
                            0.896484375
                            ],
            "confidence": 0.4213932454586029,
            "attributes": [
                {
                "name": "floor_type",
                "value": "hardwood",
                "confidence": 0.923857
                }
            ],
            "segmentation": {'size': [512, 512], 'counts': 'YWY62[`1OdoN0ZXd1'}
        }
    ],
    "metadata": {
        "md5": "6897fa4c4cb0cc682ac75d90f83eafb5",
        "width": 1937,
        "height": 2583
    }
}

401 Invalid authentication token.

{
  "statusCode": 401,
  "error": "Unauthorized",
  "message": "Invalid credentials",
  "attributes": {
    "error": "Invalid credentials"
  }
}

403 No associated account found.

<Response body is empty>

Only one of file, files, url or urls may be specified.

Example request body

{
    "url": "https://upload.wikimedia.org/wikipedia/commons/6/6d/Down_House.jpg",
    "force": true
}

Possible classifications

aerial_view	indoor	outdoor
bathroom	bedroom	community
gym	diagram	dining_room
front_of_structure	garage	kitchen
laundry	living_room	office
other	pool	closet

Possible object detections

Features

fireplace	stairway	floor
ceiling	window	wall
stairs	cabinet	sink
counter	closet	toilet
kitchen_island	bar	countertop
bannister	booth	chandelier
shelf	stove	railing
refrigerator	column	bathtub
blind	escalator	door
fountain	stage	buffet_counter
conveyor	canopy	washing_machine
swimming_pool	oven	step
tank	microwave	dishwasher
hood	sconce	shower
radiator

Furniture

table	curtain	chair
bench	couch	chest
mirror	rug	armchair
seat	desk	stand
coffee_table	pool_table	grandstand
display_case	pillow	bookcase
ottoman	swivel_chair	stool
cradle	trashcan	bulletin
bed

Items

poster	painting	lamp
computer	cushion	box
sign	screen	book
toy	arcade_machine	towel
television	apparel	light
pole	bottle	basket
ball	barrel	bag
food	pot	projection_screen
vase	sculpture	blanket
tray	fan	crt
plate	monitor	drinking_glass
clock	flag

Outdoor

sky	tree	road
building	grass	sidewalk
ground	mountain	plant
car	water	house
sea	field	fence
stone	sand	skyscraper
path	runway	river
bridge	flower	hill
palm	boat	hut
truck	tower	bus
awning	streetlight	airplane
dirt_track	land	van
tent	waterfall	ship
motorbike	bicycle	lake
traffic_light	pier

Attributes

Attributes are more specific features gathered from the objects detected in the images. For the segmentations that have them, the attributes will give more robust information about the image. Attributes will always have a name field and a value field. Currently, the possible attributes are:

• tagpoint: the tagpoint attribute's value is an array in the shape of [x, y] where x and y are percentage-based. They form a point that is guaranteed to be on the segmentation for tagging/display purposes.

• is_stainless: A boolean. This attribute appears on refrigerator, stove, oven, washing_machine, hood, and dishwasher segmentations.

• floor_type: A string of one of the possible flooring classifications (defined below). An additional float field, confidence, exists on this attribute.

Possible flooring classifications

A floor segmentation will automatically have a flooring type within its attributes.

carpet	stone	marble
hardwood	tile	other

Classify images from a RESO object

POST https://api.foyer.ai/Media/classifyReso

This endpoint runs multiple images through classification and object detection from a valid RESO payload.

Headers

Name	Type	Description
Authorization	string	Authorization token.

Request Body

Name	Type	Description
value	array	A single RESO object or an array of multiple. Objects must contain a Media Resource as defined by the RESO standard.

200 Images successfully classified. Classifier information will be added to the ImageOf field of each Media Resource.

{
  "@odata.context": "http://odata.listhub.moveaws.com/odata/$metadata#Property",
  "value": [
    {
      "@odata.id": "http://odata.listhub.moveaws.com/odata/Property('3yd-A2SELL-281884')",
      "ListingKey": "3yd-A2SELL-281884",
      "ListingId": "281884",
      "PropertyType": "Land",
      "PropertySubType": "Other",
      "ListPrice": 21900,
      ...
      "Media": [
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/1",
          "MediaKey": "2483763c0011d2a36483fb3f28ca8c00564abb4d",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Diagram"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/2",
          "MediaKey": "192607ff01a04478df871a6146b5e44e9683deb2",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/3",
          "MediaKey": "748444298599641bba9ae7278f777759779afe9f",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/4",
          "MediaKey": "a763fdf6f243a9ade93a3695af5eaf2c7233829a",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/5",
          "MediaKey": "7d2531b1ddc50deb7c256c2c67c88a399a5b9b68",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/6",
          "MediaKey": "19683d3d369aaade3551c0c192b5c186de2c563b",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/7",
          "MediaKey": "4652e1f0e0a5400747b4f5f8f1eaa74d65edf943",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        },
        {
          "MediaCategory": "Photo",
          "MediaURL": "http://photos.listhub.net/A2SELL/281884/8",
          "MediaKey": "cd4a82af4d64d5549031b1cbfbeee24a0103c00a",
          "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
          "ImageOf": [
            "Outdoor"
          ]
        }
      ]
    }
  ],
  "@odata.nextLink": "https://odata.listhub.moveaws.com/odata/Property?%24top=1&%24skiptoken=3yd-A2SELL-281884"
}

401 Invalid authentication token.

{
  "statusCode": 401,
  "error": "Unauthorized",
  "message": "Invalid credentials",
  "attributes": {
    "error": "Invalid credentials"
  }
}

403 No associated account found.

<Response body is empty>

This function accepts a RESO payload and returns inference data from all provided Media objects. This returns the RESO data with additional classification and segmentation data added to it. To fit RESO guidelines where possible, output classifications are contained in the ImageOf field.

The classifyRESO endpoint does not return mask data. Mask data for images on a full listing can make the resulting payload many times larger. We can provide bulk inference with mask data included, but logistics for the large payloads must first be arranged.

See the RESO definition for a Media Resource. We require the MediaURL field to be defined for image classification.

Example request body

{
    "@odata.id": "http://odata.listhub.moveaws.com/odata/Property('3yd-A2SELL-281884')",
    "ListingKey": "3yd-A2SELL-281884",
    "ListingId": "281884",
    "PropertyType": "Land",
    "PropertySubType": "Other",
    "ListPrice": 21900,
    "Media": [
    {
        "MediaCategory": "Photo",
        "MediaURL": "http://photos.listhub.net/A2SELL/281884/1",
        "MediaKey": "2483763c0011d2a36483fb3f28ca8c00564abb4d",
        "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z"
    }
    ]
}

Example request return

{
    "@odata.id": "http://odata.listhub.moveaws.com/odata/Property('3yd-A2SELL-281884')",
    "ListingKey": "3yd-A2SELL-281884",
    "ListingId": "281884",
    "PropertyType": "Land",
    "PropertySubType": "Other",
    "ListPrice": 21900,
    "Media": [
        {
            "MediaCategory": "Photo",
            "MediaURL": "http://photos.listhub.net/A2SELL/281884/1",
            "MediaKey": "2483763c0011d2a36483fb3f28ca8c00564abb4d",
            "MediaModificationTimestamp": "2019-10-03T09:14:00.000Z",
            "ImageOf": [
                "diagram",
                "plat_map",
                "outdoor"
            ],
            "Segmap": [
                {
                    "class": "wall",
                    "attributes": [],
                    "area": 0.00493621826171875,
                    "boundingBox": [
                        0.408203125,
                        0.169921875,
                        0.974609375,
                        0.490234375
                    ],
                    "confidence": 0.4345703125
                },
                {
                    "class": "sky",
                    "attributes": [],
                    "area": 0.9121208190917969,
                    "boundingBox": [
                        0,
                        0,
                        0.998046875,
                        0.998046875
                    ],
                    "confidence": 0.93798828125
                },
                {
                    "class": "fence",
                    "attributes": [],
                    "area": 0.0113372802734375,
                    "boundingBox": [
                        0.546875,
                        0.248046875,
                        0.96875,
                        0.37890625
                    ],
                    "confidence": 0.47607421875
                }
            ]
        }
    ]
}