Agentic QA 1.0
AndroidWorld Benchmark
A standardized evaluation suite for measuring how effectively AI agents perform real-world tasks within the Android operating system.
How we achieved it? 94.8%
Success Rate
116
Total Tasks
110
Success
6
Failed
Benchmark Results
AudioRecorderRecordAudio
✓
AudioRecorderRecordAudioWithFileName
✓
BrowserDraw
✓
BrowserMaze
✓
BrowserMultiply
✓
CameraTakePhoto
✓
CameraTakeVideo
✓
ClockStopWatchPausedVerify
✓
ClockStopWatchRunning
✓
ClockTimerEntry
✓
ContactsAddContact
✓
ContactsNewContactDraft
✓
ExpenseAddMultiple
✓
ExpenseAddMultipleFromGallery
✓
ExpenseAddMultipleFromMarkor
✓
ExpenseAddSingle
✓
ExpenseDeleteDuplicates
✓
ExpenseDeleteDuplicates2
✓
ExpenseDeleteMultiple
✓
ExpenseDeleteMultiple2
✓
ExpenseDeleteSingle
✓
FilesDeleteFile
✓
FilesMoveFile
✓
MarkorCreateFolder
✓
MarkorCreateNote
✓
MarkorCreateNoteAndSms
✓
MarkorCreateNoteFromClipboard
✓
MarkorDeleteAllNotes
✓
MarkorDeleteNewestNote
✓
MarkorDeleteNote
✓
MarkorEditNote
✓
MarkorMoveNote
✓
MarkorTranscribeReceipt
✓
NotesIsTodo
✓
NotesMeetingAttendeeCount
✓
NotesRecipeIngredientCount
✓
NotesTodoItemCount
✓
OpenAppTaskEval
✓
OsmAndFavorite
✓
OsmAndMarker
✓
OsmAndTrack
✓
RecipeAddMultipleRecipes
✓
RecipeAddMultipleRecipesFromImage
✓
RecipeAddMultipleRecipesFromMarkor
✓
RecipeAddMultipleRecipesFromMarkor2
✓
RecipeAddSingleRecipe
✓
RecipeDeleteDuplicateRecipes
✓
RecipeDeleteMultipleRecipes
✓
RecipeDeleteMultipleRecipesWithConstraint
✓
RecipeDeleteMultipleRecipesWithNoise
✓
RecipeDeleteSingleRecipe
✓
RecipeDeleteSingleWithRecipeWithNoise
✓
RetroCreatePlaylist
✓
RetroPlayingQueue
✓
RetroPlaylistDuration
✓
RetroSavePlaylist
✓
SaveCopyOfReceiptTaskEval
✓
SimpleCalendarAddOneEvent
✓
SimpleCalendarAddOneEventInTwoWeeks
✓
SimpleCalendarAddOneEventRelativeDay
✓
SimpleCalendarAddOneEventTomorrow
✓
SimpleCalendarAddRepeatingEvent
✓
SimpleCalendarAnyEventsOnDate
✓
SimpleCalendarDeleteEvents
✓
SimpleCalendarDeleteEventsOnRelativeDay
✓
SimpleCalendarDeleteOneEvent
✓
SimpleCalendarEventOnDateAtTime
✓
SimpleCalendarEventsInNextWeek
✓
SimpleCalendarEventsInTimeRange
✓
SimpleCalendarEventsOnDate
✓
SimpleCalendarFirstEventAfterStartTime
✓
SimpleCalendarLocationOfEvent
✓
SimpleCalendarNextEvent
✓
SimpleCalendarNextMeetingWithPerson
✓
SimpleDrawProCreateDrawing
✓
SimpleSmsReply
✓
SimpleSmsReplyMostRecent
✓
SimpleSmsResend
✓
SimpleSmsSend
✓
SimpleSmsSendClipboardContent
✓
SimpleSmsSendReceivedAddress
✓
SportsTrackerActivitiesCountForWeek
✓
SportsTrackerActivitiesOnDate
✓
SportsTrackerActivityDuration
✓
SportsTrackerLongestDistanceActivity
✓
SportsTrackerTotalDistanceForCategoryOverInterval
✓
SportsTrackerTotalDurationForCategoryThisWeek
✓
SystemBluetoothTurnOff
✓
SystemBluetoothTurnOffVerify
✓
SystemBluetoothTurnOn
✓
SystemBluetoothTurnOnVerify
✓
SystemBrightnessMax
✓
SystemBrightnessMaxVerify
✓
SystemBrightnessMin
✓
SystemBrightnessMinVerify
✓
SystemCopyToClipboard
✓
SystemWifiTurnOff
✓
SystemWifiTurnOffVerify
✓
SystemWifiTurnOn
✓
SystemWifiTurnOnVerify
✓
TasksCompletedTasksForDate
✓
TasksDueNextWeek
✓
TasksDueOnDate
✓
TasksHighPriorityTasks
✓
TasksHighPriorityTasksDueOnDate
✓
TasksIncompleteTasksOnDate
✓
TurnOffWifiAndTurnOnBluetooth
✓
TurnOnWifiAndOpenApp
✓
VlcCreatePlaylist
✓
VlcCreateTwoPlaylists
✓
MarkorAddNoteHeader
✗
MarkorChangeNoteContent
✗
MarkorMergeNotes
✗
MarkorTranscribeVideo
✗
RecipeDeleteDuplicateRecipes2
✗
RecipeDeleteDuplicateRecipes3
✗