macOS accessibility API on WebKit applications with AXTextMarker

Question

I need to access data from webkit applications such as Safari, Mail and maybe others. I can see in the Accessibility Inspector there is :AXTextMarker and AXTextMarkerForRange.

I tried the usual way to get this info :

AXUIElementRef systemWideElement = AXUIElementCreateSystemWide();    //creating system wide element
AXUIElementRef focussedElement = NULL;

AXError error = AXUIElementCopyAttributeValue(systemWideElement, kAXFocusedUIElementAttribute, (CFTypeRef *)&focussedElement);                //Copy the focused
if (error != kAXErrorSuccess){
    NSLog(@"Could not get focused element");
}else{
    AXValueRef marker = NULL;
    AXError getTextValueError = AXUIElementCopyAttributeValue(focussedElement, kAXMarkerUIElementsAttribute , (CFTypeRef *)&marker);        
}

kAXMarkerUIElementsAttribute is the only thing I can see with Marker but everything is empty each time.

I guess for security reasons, I cannot access them? Is there any way possible. I am developing an app for people with difficulties reading and it could really help.

Thanks

Answer 1

Tips and tricks:

Ask focussedElement for its supported attributes. Use the functions:

AXError AXUIElementCopyAttributeNames(AXUIElementRef element, CFArrayRef  _Nullable *names);

Returns a list of all the attributes supported by the specified accessibility object.

and

AXError AXUIElementCopyParameterizedAttributeNames(AXUIElementRef element, CFArrayRef  _Nullable *names);

Returns a list of all the parameterized attributes supported by the specified accessibility object.

Most undocumented attributes are self explanatory.

For example get the selected text as attributed string:

CFTypeRef markerRange = NULL;
AXError error = AXUIElementCopyAttributeValue(focussedElement, (CFStringRef)@"AXSelectedTextMarkerRange", &markerRange);
CFTypeRef result = NULL;
error = AXUIElementCopyParameterizedAttributeValue(focussedElement, (CFStringRef)@"AXAttributedStringForTextMarkerRange", markerRange, &result);

Answer 2

Try this, I haven't tested this but I found some research on the API and I think this may solve your problem.

It seems for the accessibility API to fully work the application needs to be trusted or basically have root access. Is the application running in root?

I found some apple script from a past project. seemed relevant

NSDictionary *errorInfo = [NSDictionary new];
NSString *script =  [NSString stringWithFormat:@"do shell script \"%@ %@\" with administrator privileges", @"/usr/sbin/lsof",@"-c filecoord"];
NSLog(@"Running... %@",script);
NSAppleScript *appleScript = [[NSAppleScript new] initWithSource:script];
NSAppleEventDescriptor * eventResult = [appleScript executeAndReturnError:&errorInfo];

This apple script isn't excatily what you need but its a start. I'm not sure how to fully request root access to an application from inside the application its self, But for testing, you can always just run your application as root and continue your debugging

Please let me know if this helped you in any way or not

Answer 3

Accessibility apps need to be added in System Preferences in the following section:

System Preferences -> Security & Privacy -> Privacy tab -> Accessibility (on the left)

Note that during development you need to include Xcode in this list if you are developing/running within Xcode.

I didn't test out your exact code, but I tested some comparable swift code and that worked fine provided Xcode was added in that spot in system preferences.