Google action response with an image (basic card)

#1

Hello
I use google-action on my node-red machine.
That works fine.
Now, I want to add some image in the response. So, I think the good way is to use basic card or table card.
I found the code in the google action console
So, I have inserted this code in the function wich creates the anwser :slight_smile:

{
  "payload": {
    "google": {
      "expectUserResponse": true,
      "richResponse": {
        "items": [
          {
            "simpleResponse": {
              "textToSpeech": "This is a basic card example."
            }
          },
          {
            "basicCard": {
              "title": "Title: this is a title",
              "subtitle": "This is a subtitle",
              "formattedText": "This is a basic card.  Text in a basic card can include \"quotes\" and\n        most other unicode characters including emoji 📱.  Basic cards also support\n        some markdown formatting like *emphasis* or _italics_, **strong** or\n        __bold__, and ***bold itallic*** or ___strong emphasis___ as well as other\n        things like line  \nbreaks",
              "image": {
                "url": "https://example.com/image.png",
                "accessibilityText": "Image alternate text"
              },
              "buttons": [
                {
                  "title": "This is a button",
                  "openUrlAction": {
                    "url": "https://assistant.google.com/"
                  }
                }
              ],
              "imageDisplayOptions": "CROPPED"
            }
          }
        ]
      }
    }
  }
}

This code should display an image, a title and a button.
But i only get a return message saying : [object Object]

I need to know if the syntax I chose is the correct one (dialogflow Json)?
I also tried with an other syntax : ACTIONS SDK JSON. But I get the same result.

With the link above, can you tell me is what is written there is useable with the google-action version 1.2.1?

Here is the debug result of the google action console :slight_smile:

{
  "response": "[object Object]",
  "expectUserResponse": true,
  "conversationToken": "EroCS2w1Tm...",
  "audioResponse": "//NExAAAAA...",
  "ssmlMarkList": [],
  "debugInfo": {
    "assistantToAgentDebug": {
      "curlCommand": "curl -v https://88.176.64.72:8081/ -H 'Content-Type: application/json;charset=UTF-8' -H 'Google-Actions-API-Version: 2' -H 'Authorization: eyJhbGciOiJSUzI1NiIsImtpZCI6IjM3ODJkM2YwYmM4OTAwOGQ5ZDJjMDE3MzBmNzY1Y2ZiMTlkM2I3MGUiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJhdWQiOiJteS10ZXN0LWFwcC1kMTNkZSIsIm5iZiI6MTU1NTI3NzYxNywiaWF0IjoxNTU1Mjc3OTE3LCJleHAiOjE1NTUyNzgwMzcsImp0aSI6IjgwMTI5MTBkNmExYjIwOWQzZjk3NjI0YTcwYTFlZDBkYWM0MTBmM2MifQ.hODIBx-Geau2JwJAAD0TwHm3tsl210o6JRWTWDGnwFgDwpvCZJt_1UQ-o49NtYx-KApfHxSU2i_DcnHca3MX8kkdz51B475w3X4advT5sU69MgZI82O7SKaYsJIsKSq30yvO43m81I6_Vcx6IrPWlu5mn6YGC2bw5Lc_dPjyUQXNtW7YfOnxK2OE2MOBHN2kjKe7ulcSAWYjVRzKW2EVus1i_Y8uwF5ieSMBYYSo0uvFS0qIQjeOULwE-xKp_fZ3P3cLAhlYQH76EIZSxI6OQfxYFQkVfTYkUDxoXzjELQuQbNMZ3FcjrvQ4FKhL14JevjB78nBhm2yOkZbRkq1akQ'  -A 'Mozilla/5.0 (compatible; Google-Cloud-Functions/2.1; +http://www.google.com/bot.html)' -X POST -d '{\"user\":{\"userId\":\"ABwppHGZMM6CHA-JcPkrCzVpkgGv953hFvVdAGAEGOWQSETGxFO18zXyIbXrhHAlw63M9Gz7dKcFxn3fIGKd2sw\",\"locale\":\"fr-CA\",\"lastSeen\":\"2019-04-14T21:27:04Z\",\"userStorage\":\"{\\\"data\\\":{}}\"},\"conversation\":{\"conversationId\":\"ABwppHHYGJV-sVySdFxJJtWlUtpt3qL6HyB3fIi0N-ose5aUNUagVlhJsKIAcY3cmofXWo8nfTeDQLLMp0Ji-AU\",\"type\":\"ACTIVE\",\"conversationToken\":\"{}\"},\"inputs\":[{\"intent\":\"actions.intent.TEXT\",\"rawInputs\":[{\"inputType\":\"KEYBOARD\",\"query\":\"image\"}],\"arguments\":[{\"name\":\"text\",\"rawText\":\"image\",\"textValue\":\"image\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.ACCOUNT_LINKING\"},{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]}],\"requestType\":\"SIMULATOR\"}'",
      "assistantToAgentJson": "{\"user\":{\"userId\":\"ABwppHGZMM6CHA-JcPkrCzVpkgGv953hFvVdAGAEGOWQSETGxFO18zXyIbXrhHAlw63M9Gz7dKcFxn3fIGKd2sw\",\"locale\":\"fr-CA\",\"lastSeen\":\"2019-04-14T21:27:04Z\",\"userStorage\":\"{\\\"data\\\":{}}\"},\"conversation\":{\"conversationId\":\"ABwppHHYGJV-sVySdFxJJtWlUtpt3qL6HyB3fIi0N-ose5aUNUagVlhJsKIAcY3cmofXWo8nfTeDQLLMp0Ji-AU\",\"type\":\"ACTIVE\",\"conversationToken\":\"{}\"},\"inputs\":[{\"intent\":\"actions.intent.TEXT\",\"rawInputs\":[{\"inputType\":\"KEYBOARD\",\"query\":\"image\"}],\"arguments\":[{\"name\":\"text\",\"rawText\":\"image\",\"textValue\":\"image\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.ACCOUNT_LINKING\"},{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]}],\"requestType\":\"SIMULATOR\"}",
      "delegatedRequest": {
        "delegatedRequest": ""
      }
    },
    "agentToAssistantDebug": {
      "agentToAssistantJson": "{\"conversationToken\":\"{}\",\"expectUserResponse\":true,\"expectedInputs\":[{\"inputPrompt\":{\"initialPrompts\":[{\"textToSpeech\":\"[object Object]\"}],\"noInputPrompts\":[]},\"possibleIntents\":[{\"intent\":\"actions.intent.TEXT\"}]}]}",
      "delegatedResponse": {
        "delegatedResponse": ""
      }
    },
    "sharedDebugInfoList": []
  },
  "visualResponse": {
    "visualElementsList": [
      {
        "displayText": {
          "content": "[object Object]"
        }
      }
    ],
    "suggestionsList": [],
    "agentLogoUrl": "https://www.gstatic.com/voice/opa/partner_icons/generic_3p_avatar.png",
    "agentStyle": {
      "primaryColor": "",
      "fontFamily": "",
      "borderRadius": 0,
      "backgroundColor": "",
      "backgroundImageUrl": ""
    }
  },
  "clientError": 0,
  "is3pResponse": true,
  "clientOperationList": [
    {
      "operationType": 1,
      "micUpdatePayLoad": {
        "micMode": 1
      }
    }
  ],
  "projectName": ""
}

Can anybody help me?

0 Likes

#2

Hi.
Nobody can help me on this problem?
Isn't the description of my problem clear enough?
Do i have to add some new elements?
Thanks

0 Likes

#3

It could well be that no one has done what you are trying to do. You might just be a ground breaker.

0 Likes

#4

I'm not familiar with node-RED, but the [object Object] part suggests that you're not actually sending the JSON, but rather sending the toString() version of the JavaScript containing the reply. The JSON itself isn't the problem - it is how you're sending it. Without seeing the code that you're using to send it, there isn't much more I can suggest.

0 Likes

#5

Look under V

0 Likes

#6

look under V

0 Likes

#7

Hello

Here is the Flow ise to send the
response. (attched fle). The code is inside the 'image' node.

But I don't know what other part of
code you need to look at the problem.

Can anyone tell me what other part of
code would be usefull to find the problem, and where i can find it?

I also attach a wireshark capture, maybe
this can help. And also the screenshot of the iPhone.

Here are the steps :

  • 1 : I connect to the action, with
    'parler avec l'automate'.

  • 2 : Then, I tell 'image' so that the
    node 'image' is triggered

  • 3 Then, I tell 'numero' so that I
    can receive a random number

  • 4 finally, I terminate the conversation
    by saying 'au revoir'.

I suppose you will be able to find these
french words in the debug and wireshark too.

FlowAndDebugForBasicCard.odt.txt (30.3 KB)

wireshark_capture_basic_card.pcapng.txt (47.3 KB)

0 Likes