Use Visual Question Answering (VQA) to get image information

Console

1. In the Google Cloud console, open the Vertex AI Studio > Vision tab in the Vertex AI dashboard.

   Go to the Vertex AI Studio tab

2. In the lower menu, click Visual Q & A.

3. Click Upload image to select your local image to caption.

4. In the Parameters panel, choose your Number of captions and Language.

5. In the prompt field, enter a question about your uploaded image.

6. Click play_arrowSubmit.

REST

For more information about imagetext model requests, see the imagetext model API reference.

Before using any of the request data, make the following replacements:

• PROJECT_ID: Your Google Cloud project ID.
• LOCATION: Your project's region. For example, us-central1, europe-west2, or asia-northeast3. For a list of available regions, see Generative AI on Vertex AI locations.
• VQA_PROMPT: The question you want to get answered about your image.
  • What color is this shoe?
  • What type of sleeves are on the shirt?
• B64_IMAGE: The image to get captions for. The image must be specified as a base64-encoded byte string. Size limit: 10 MB.
• RESPONSE_COUNT: The number of answers you want to generate. Accepted integer values: 1-3.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagetext:predict

Request JSON body:

{
  "instances": [
    {
      "prompt": "VQA_PROMPT",
      "image": {
          "bytesBase64Encoded": "B64_IMAGE"
      }
    }
  ],
  "parameters": {
    "sampleCount": RESPONSE_COUNT
  }
}

To send your request, choose one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagetext:predict"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/imagetext:predict" | Select-Object -Expand Content

{
  "predictions": [
    "cappuccino",
    "coffee"
  ]
}

Python

Before trying this sample, follow the Python setup instructions in the Vertex AI quickstart using client libraries. For more information, see the Vertex AI Python API reference documentation.

To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

In this sample you use the load_from_file method to reference a local file as the base Image to get information about. After you specify the base image, you use the ask_question method on the ImageTextModel and print the answers.

import vertexai
from vertexai.preview.vision_models import Image, ImageTextModel

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"
# input_file = "my-input.png"
# question = "" # The question about the contents of the image.

vertexai.init(project=project_id, location="us-central1")

model = ImageTextModel.from_pretrained("imagetext@001")
source_img = Image.load_from_file(location=input_file)

answers = model.ask_question(
    image=source_img,
    question=question,
    # Optional parameters
    number_of_results=1,
)

print(answers)