Understanding the Issue with Uploading Pictures in Google Assistant Actions on iPhone XR and iPhone 11

The recent behavior of Google Assistant actions not working as expected when trying to upload pictures on iPhone XR and iPhone 11 has caused frustration among developers. In this article, we will delve into the technical details behind this issue and explore possible solutions.

What is Dialog Flow?

Dialog Flow is a service provided by Google that allows developers to build conversational interfaces for their applications. It provides a platform for creating voice-activated interfaces that can understand natural language inputs and respond accordingly. In the context of this article, we are specifically looking at how Dialog Flow handles Basic Card responses, which include images and buttons.

What is a Basic Card?

A Basic Card is a type of response used in Dialog Flow to present users with information in a concise manner. It typically includes a title, subtitle, text, image, and buttons. The Basic Card is rendered as a card interface that can be displayed on various devices, including smartphones and tablets.

Understanding the Issue

When we try to upload pictures using Google Assistant actions on iPhone XR and iPhone 11, the app closes abruptly, and we are taken back to the App screen. This behavior is observed for other actions that attempt to upload pictures as well. The issue seems specific to Dialog Flow actions that return Basic Card responses.

Testing Environment

To recreate this issue, we can follow these steps:

1. Create a simple Dialog Flow action with a Basic Card response.
2. Configure the Basic Card to include an image and buttons for uploading pictures.
3. When the Basic Card is displayed in Google Assistant, click on the “Upload Google Photo” button.
4. Ideally, you should see options to upload a picture, such as “Take Photo or Video”, “Photo Library”, and “Browse”.

Observations

Upon closer inspection, we can observe that the issue seems specific to iPhone XR and iPhone 11 running iOS 13.7. The same behavior is observed when using other actions that attempt to upload pictures.

Interestingly, the same link works as expected on these devices when used in a browser like Safari or Chrome, but not within Google Assistant.

Technical Background

To understand why this issue occurs, we need to dive into some technical details.

When Dialog Flow renders a Basic Card response, it uses the display property to determine how the card should be displayed. In this case, the display is set to “CROPPED”, which suggests that the image should be cropped to fit within the available space.

However, on iPhone XR and iPhone 11 running iOS 13.7, there seems to be an issue with rendering the Basic Card response in the Google Assistant interface. This could be due to a problem with the way Dialog Flow handles images or a limitation in the iOS 13.7 implementation.

Possible Solutions

To resolve this issue, we can try the following solutions:

1. Use a different image format: Try using a different image format, such as PNG or JPEG, instead of the recommended actionsresources/logo_assistant_2x_64dp.png image.
2. Adjust the display property: Experiment with different values for the display property, such as “FULL”, to see if that resolves the issue.
3. Use a different dialog flow feature: Consider using a different Dialog Flow feature, such as a CarouselCard, which may be more suitable for displaying images and buttons.

Code Example

Here’s an updated code example that uses a CarouselCard instead of a Basic Card:

conv.ask(new CarouselCard({
  items: [
    {
      title: 'Upload Google Photo',
      image: new Image({
        url: 'https://photos.google.com/',
        alt: 'Image alternate text'
      })
    },
    {
      title: 'Take Photo or Video',
      image: new Image({
        url: 'https://storage.googleapis.com/actionsresources/logo_assistant_2x_64dp.png',
        alt: 'Image alternate text'
      })
    }
  ]
}))

Conclusion

The issue with uploading pictures in Google Assistant actions on iPhone XR and iPhone 11 is a complex one that requires further investigation. By understanding the technical background behind Dialog Flow’s Basic Card responses and exploring possible solutions, we can work towards resolving this issue.

In future articles, we will delve deeper into the world of Dialog Flow and explore more advanced features for building conversational interfaces.

Last modified on 2023-12-16