Blue Prism Accessibility Mode
Accessibility mode is an interface available in Application Modeller that can be used for many Windows and Browser based applications.
The purpose of this document is to act as a useful source of advice to Blue Prism developers who are using the Accessibility Mode interface when building their solutions.
It is a per-requisite of using this guide that the Foundation Training Course is completed, as some of the terms and techniques discussed require a basic understanding of Object Studio and Application Modeller.
This guide includes instructions for the following:
- How to switch to Accessibility mode in Application Modeller
- Advice on which attributes to select in Application Modeller when using Active Accessibility
- Information about some of the features unique to Active Accessibility Mode
- Why some screen elements might be difficult to identify using Application Navigator
- Why the performance of Active Accessibility mode might be slow for some applications and the techniques you can implement to improve performance.
What is Accessibility Mode?
Accessibility Mode uses Microsoft Active Accessibility (AA) which is a Component Object Model (COM) technology built into Object Studio that is available to interact with many application types running in a Microsoft Windows environment. It is an interface originally created for screen reading and speaking applications to assist Windows application users with disabilities.
Accessibility mode works well with most modern Windows and Browser applications, but there is no guarantee that it will work for all applications or all elements within an application as it is dependent upon how the app is built.
Switching to Accessibility mode
To use AA mode in Application Modeller, press the Identify button as normal to spy a new element, then simply press the "ALT" key when in the Identify (or spy) mode to change the interface being used. Keep pressing the "ALT" key until the spy tool turns blue.
If Blue Prism crashes or give an error when you press the "ALT" key to change to Accessibility mode, it is likely that the incorrect Application Modeller mode is being used for the application you want to interface with. To fix this re-run the Application Modeller wizard and experiment with the different 32bit/64bit modes available.
Accessibility Mode attributes
Once you identify a new element in Application Modeller using Accessibility Mode you will find that the element has a very large number of attributes available for you to select, many of them ticked by default.
Untick most attributes
Every application is different, so the correct attributes to tick will be different for each system and will initially require some experimentation.
By default, Accessibility mode ticks a lot of attributes for any new element identified, most of these should be unticked to make Blue Prisms search for the correct element more efficient.
For most applications the following can be unticked:
- Any elements with a blank value
- All 'Parent' attributes unless required to make the element a unique match
The 'Visible' and 'Enabled' attributes usually have no impact if left ticked or unticked, so can often be unticked
- Untick any attribute that might not be persistent, where it could be prone to changing (such as Element Count if the element is in a list, or any elements in a browser application that contain the URL).
Once most of the attributes have been unticked, you can start experimenting with ticking just the handful of attributes that are required to uniquely identify the element.
Copy Element Mask
When you have decided upon the best attributes to leave ticked for an element, it is likely that this will be the same for most elements within the application. To save time ticking the correct attributes for other new elements you can use the Copy Element Mask and Paste Element Mask options within Application Modeller to copy the attribute ticks from one element to others:
To copy attribute ticks from one element to another simply right click on the element you want to copy from in the Application Model tree view and select the 'Copy Element Mask' context menu option. Then right click on the element you want to have the same ticks and select the 'Paste Element Mask 'context menu option.
Once you have copied an element’s mask you simply right click on the element you want to copy the ticks to and select the Past Element Mask option. With this feature only the ticked attributes will be updated while the values of the element will remain unchanged.
Match Index and Match Reverse attributes
With Accessibility mode the underlying Application Model that Blue Prism needs to search through to find elements can be a lot larger than for other interface types. Because of this it is likely that the Accessibility interface can be slow to find elements.
To speed up the interface the Match Index attribute should always be used for Accessibility mode elements. This works by stopping Blue Prism from trawling through the application model for duplicate elements once one matching element is found.
Depending on where the element is within the application model, sometimes just Match Index needs to be ticked, sometimes Match Reverse will also need to be ticked. You will need to experiment to find the best combination.
Invisible attribute
For some applications, the underlying Application Model that Blue Prism needs to search through to find elements can be very large, and it can include elements that are not actually displayed on any current screens. This can cause duplicate elements to be found, where a screen element that is not visible has all the same attributes as a screen element you want to interact with.
To cater for this issue, it is recommended that you experiment with the 'Invisible', 'Visible', and 'Enabled' attributes.
For many applications the 'Invisible' attribute seems to be the best attribute to try ticking first. 'Invisible' seems to work in more applications than the Visible or Enabled attributes as a way of ensuring your element is within the current screen of your application.
Accessibility Mode features
This section outlines some information to help when using your elements in Object studio.
Activating the screen before using Global clicks
For some Accessibility Mode elements, the only option for clicking an elements, the only option for clicking an element such as a button or link may be to send a Global Mouse Click, or a Global Mouse Click center.
When using a Global Mouse Click (or any global action) you must first ensure that the element you wish to use is visually displayed on desktop in front of any other applications. This is done by:
- First, identify a Windows mode element for the main window that the element you want to click is in.
- Add a Navigate stage to your flow that uses your main window element and the 'Activate Application' action.
Your final flow before a global click will look something like this:
Usually, activating the window can be done once at the start of each action that contains global interactions rather than needing to activate before all global actions.
When debugging an action that uses global clicks or keystrokes stepping through your flow may give the Blue Prism application itself focus, taking it away from the application you are interfacing with. Because of this loss of focus it is often best to run to breakpoints when testing actions that contain global clicks or global send keys.
Default Action
Rather than using Global Clicks for Active Accessibility elements, an option that is sometimes usable is to use the default action for the element instead.
For Accessibility Mode elements identified in Application Modeller there is an attribute called 'Default Action'. If this attribute has a value Blue Prism can use that Default Action within a navigate stage.
To be able to use the Default Action option the Element Type shown for the element in Application Modeller needs to be set to 'Active Accessibility'. If element type has been automatically changed to a different type (i.e. Button (AA) than it can be changed, but this will mean that some Button specific actions will be no longer available within the navigate stage.
This screenshot shows the Default Action attribute, and the Active Accessibility Element Type in Application Modeller.
To use the Default Action you simply select Default as the action for the element in a navigate stage as shown in this screenshot: The most common default actions are 'Press' which presses a button or element, and 'Jump' which selects a link in a browser application.
Focus
For Accessibility elements a 'Focus' action is available in Navigate stages that can sometimes be used to ensure that element is visible within the screen before you use it. This option can be useful for scrolling list views or web pages where the element might not be on the screen but requires a global interaction (such as global clicks) to be able to interface with it.
The Focus action does not work for all applications. Where this option does not work for you other global options such as clicking a scroll bar or using the Down keystroke will be required to scroll to the item you want (please refer to the Surface Automation training).
Get Table
Accessibility mode has a similar 'Get Table' action available in the Read stage that is available with some other interfaces. The main things to know about using Get Table are:
- You usually don't need to identify the entire table in Application Modeller, Get Table when used on an element within the table (such as a header value)
- The success of using the Get Table action depends upon the structure of the table you are trying to read values from. Some tables may actually be a mixture of individual bespoke elements rather than one large table element.
- For some tables Get Table will only return individual rows. Where this occurs logic would need to be built to us the Get Table read on every row of the table and build up the data in a collection. This could be done by using a dynamic attribute for row number incrementing.
- If Get Table does not work for an element than other options are:
- Read individual table elements separately rather than reading the entire table at once.
- Using Surface Automation techniques to read rows or columns showing in the table
Get Windows Attribute
For all interfaces the Read stage has an option to read the value of any attribute related to that element by using the 'Get Windows Attribute' option.
This option is often useful for Accessibility mode elements, because there are more attributes available that may contain the data you wish to read.
For example, the value you want to read from an element may be held in the Value or Parent Value attribute, but the only option available within the read stage is 'Get Text' or 'Get Name'. In a scenario like this Get Windows Attribute can be used.
To use 'Get Windows Attribute' simply select that option in the Data drop down in a Read stage and set the 'Attribute Name' input parameter to be the name of the attribute as it is displayed in Application Modeller.
For example, this screenshot shows a read stage being used to read the 'Parent Name' attribute value for an Accessibility mode element:
Accessibility Mode Common Problems
Sometimes elements cannot be identified in Application Modeller because of the structure of the application screen you are interfacing makes it impossible for the spy tool to highlight the element you want to identify.
For example, the screen may have been developed with a frame or border element on top of the edit box you want to identify. Blue Prism can only identify the larger frame element using its spy tool rather than the element you want to use.
Application Navigator
Application Modeller contains a feature called Application Navigator which provides a tree-view of all the Accessibility Elements available within the application so that they can be easily found and selected.
To use Application Modeller the following steps should be taken:
Instead of clicking the Identify button click the Open Application Navigator option with the button menu.
Please note depending upon the size of the Accessibility Mode it may take Blue Prism a long time to generate the Application Navigator treeview. If you find this is taking too long for your application than this may not be a viable option for you to use.
Once the application model treeview has been generated you will see the following screen:
As you click on different elements within the Application Model treeview the element will highlight in the application that you are interfacing with.
Once the element you want is found and selected in the treeview you can press the OK button to use the element in Application Modeller.
The search bar at the top of the tree view can be used to find an element quickly based upon the text within it.
Microsoft accessibility testing tools
Where Application Navigator is too slow to use or does not display an element you think it should, use a Microsoft accessibility testing tool to interrogate screen elements and find the attributes related to different elements.
AccExplorer.exe, UISpy.exe, and Inspect.exe are tools that have been available within different versions of Microsoft's Windows Software Development Kit (SDK) for different versions of Windows. These tools are not distributed by Blue Prism; to try to use them to evaluate your application please look for them directly from Microsoft's support web sites.
Using a Microsoft tool to evaluate an elements attributes allows you to potentially modify an Active Accessibility element within Application Modeller so that it can be used. This is done by modifying the attribute values to match the values shown in Microsoft accessibility testing tool.
Applications that are not accessible
Some applications are not accessible using Windows Active Accessibility interface. This may be because they are built using a platform that is not compatible with AA, or the element was not built in an accessible way; or it could be that accessibility functionality does exist but has been disabled in some other way.
Where accessibility mode does not work your options are:
Find out what platform the application was built using and if that has a different interface that can be used. For example, Java uses Java Access Bridge, SAP uses SAP GUI Scripting, etc.
Find out from the system vendor if there is an option to enable accessibility within the application. Some applications or platforms support this explicitly.
Where no other interface is available, Surface Automation techniques will need to be used instead of Accessibility mode.
----------------------------------------------------------------------------------------------------------------------------
Contact me if you need any RPA project development or Support
----------------------------------------------------------------------------------------------------------------------------
Contact me if you need a Responsive website/ web-application
----------------------------------------------------------------------------------------------------------------------------
