Fwd: Aural MathJax Tool Demo from Benetech

[Antranig Basman]

Thanks James! This is an interesting link. I mentioned some of the pitfalls and numerous unsolved problems 
in this area in the call we had the other week. As far as I can see, most of the github project consists of 
a bolting together of two other projects, MathJax itself, and speak.js, a port of a C++ speech synthesizer 
using the wacky JS compilation system "emscripten". As a brief aside, I should mention that such a thing is 
far from genuinely wacky any more since as of quite recently, the performance characteristics of JavaScript 
code on certainly regular kinds of numerical algorithm when using Google's V8 engine has pulled ahead of the 
equivalent code in Java. This is a quite incredible result that I think noone would have predicted even 3 
years ago.

The main "guts" of the conversion system appears to lie within this file, and a few others clustered near it:

https://github.com/gcapiel/mathjax/blob/master/unpackedlanguagebuffer/extensions/jax2MathSpeak.js

I think the system is a good start, but doesn't do anything to attack the kinds of issues that can make 
formulas very hard to understand when spoken, which particularly relate to helping the hearer to match 
together matching pairs of delimiters or nesting. Here is an example of a formula which I think will be very 
hard to tackle -

http://en.wikipedia.org/wiki/Lagrange%27s_trigonometric_identities#Other_sums_of_trigonometric_functions

This is a relatively basic formula which could well appear at the high school level. The difficulty is with 
clearly expressing the multiple levels of nesting in the sin functions in the expression on the right hand 
side. That said, we should actually put the system to the test! But these are the kinds of things we will 
need to deal with in order not to end up with a system which is just unintelligible, no matter how carefully 
people listen and how often they repeat the text. I'm also concerned that using MathML as the input format, 
some important semantic information may have been lost already - I have a feeling, although little evidence 
for it, that processing TeX expressions rather than MathML would lead to a better possibility of preserving 
semantically important information such as common nesting level.

I found a link to the system, AsTeR, which I referred to during the talk - it mentions explicitly these 
difficult issues I'm describing and presents some solutions to them - e.g. "vary audio state along a 
dimension that is orthogonal to (independent of ) the dimension used to convey sub-expressions" - this could 
of course only be vindicated using a real user study but at least it is addressing the issue:

http://www.cs.cornell.edu/home/raman/aster/demo.html

Not unexpectedly, this system was put together a while ago by our revered friend T.V. Raman.

Cheers,
Antranig

On 06/09/2012 13:08, James W. Yoon wrote:
> Lisa from ISKME sent along a link to a pretty neat notation(MathML)-to-speech library. What do folks think
> of this? (Antranig, I'm especially interested in your thoughts.)
>
> James
>
> Begin forwarded message:
>
>> *Resent-From: *<jyoon at ocadu.ca <mailto:jyoon at ocadu.ca>>
>> *From: *Lisa McLaughlin <lisam at iskme.org <mailto:lisam at iskme.org>>
>> *Subject: **Aural MathJax Tool Demo from Benetech*
>> *Date: *4 September, 2012 5:19:31 PM EDT
>> *To: *"James W. Yoon" <jyoon at ocadu.ca <mailto:jyoon at ocadu.ca>>
>>
>> James,
>>
>> I thought you might be interested in checking this out, benetech sent this to us
>>
>>
>> Demo of the work in progress in having MathJax generate aural output of MathML:
>> http://gcapiel.github.com/mathjax/
>>
>> Let me know if you think it would make sense to consider adding this to open author.
>>
>> Thanks and best, Lisa
>>
>> --
>> Lisa McLaughlin
>> Director of Open Knowledge Networks
>> OER Commons Manager
>> Institute for the Study of Knowledge
>> Management in Education (ISKME)
>> (734) 730-3747
>> www.oercommons.org <http://www.oercommons.org/>
>> www.iskme.org <http://www.iskme.org/>
>> www.twitter.com/lmclaug <http://www.twitter.com/lmclaug>
>>
>