About launching android env with home screen

[llv22]

Dear colleagues:

I'm now trying to use android_env for one research topic. Instead of monitoring one specific application, I need to check how users operate on the emulator (esp. click, navigation, input) and leverage adb logging or interceptor to understand what happened behind for API calling.

Is it possible to do this via textproto setting? launching to the home and accessing multiple apps to simulate a more flexible user scenario?

I saw @kenjitoyama mentioned in #19, you're mainly relying on accessibility events to access the third party apps or default apps lacking source code.

"We haven't trained explicitly for YouTube, it was just an example. However, we did train agents on apps for which we don't have the source code such as the Clock or System Settings by reusing accessibility events. Please see https://github.com/deepmind/android_env/blob/main/docs/example_tasks.md#accessibility-forwarder. "

I don't know whether there are more possibilities to do android apps monitoring, beside accessibility events?

[kenjitoyama]

Hi @llv22, what limitations do you see from using accessibility events / trees? They are very comprehensive and allow you listen for events such as clicks, swipes, keyboard inputs, apps being opened/closed and a bunch of other things. Please see https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent for details.

The reason we use Accessibility events is that they're already supported by the Android ecosystem, many apps provide support with additional information like labels for buttons which are super useful. On top of that, this is already unit-tested and implemented in on every modern Android phone, without requiring additional binary pushes. If you do anything custom, you will have a problem to deploy such solution and to make it work across different devices, Android versions and so on.

Let me know what you think.

Thanks!

[llv22]

@kenjitoyama I worked out an accessibility service on an Android device and found gesture movement capture via AccessbilityEvent can't be captured generally in the Android system. If you check https://developer.android.com/reference/android/view/accessibility/AccessibilityEvent documentation, the supported gesture events are under the model of touch exploration model, which is different from the normal gesture that we talked. For instance, if I figure-wipe the bottom of the screen, can we detect this event as a whole? or only detect mouse events, then compose them into the figure-wipe event?

So just like mentioned in https://github.com/deepmind/android_env/blob/c53f779b344497640ce9899b3717408e81b49848/docs/environment.md#action-space, if we try to detect gesture like

Touch & Hold | Flick Left | Flick Right | Scroll (H) | Scroll (V) | Drag & Drop

, we have to develop our own wrapper? could you confirm?

[kenjitoyama]

We had lots of problems with Touch Exploration mode in the AccessibilityForwarder and have disabled it.

Android doesn't have a native notion of swipe, fling, drag or anything like that. These higher level concepts are left to app developers who may or may not use existing libraries to detect them. For example, Android provides one such library here: https://developer.android.com/reference/android/view/GestureDetector. In our experience it works well and you just have to adjust some constants depending on your personal preferences.