I’ve mentioned before that I’m currently working as Localization Director for Waha, which is an app and Bible Study curriculum that we are working towards translating into hundreds of different languages. Many of the languages on our roadmap are low-resource languages, meaning that they have very few materials and tools available to help us and our partners in the translation process.

Part of my role as Localization Director involves designing workflows that make the best use of the limited time of our volunteer translators. To that end, I’ve been striving to keep up to date with advancements in Natural Language Processing (NLP) for low-resource languages, as these tools can be really helpful at easing the labor burden of localization projects, when they’re used effectively.

This week, I realized that one of those recent advancements could help me build a project that has been a nerdy dream of mine for a long time!

💡The Idea

Years ago, when I was living in Türkiye and learning the Turkish language, there were a couple practices that I found especially effective and enjoyable for improving my vocabulary and my understanding of the spoken language.

The first of these was studying live translation. There were a number of events (cultural seminars, international church services, etc.) where someone on stage would say something in in English, and that sentence would be immediately translated into Turkish. (Or, vise versa: Turkish first, and then English)

I found these to be great opportunities for language study! When I heard the English sentence, I would immediately try to translate it into Turkish in my head, before hearing it translated by a native speaker right after. This allowed me to compare how I thought it should be translated, with how someone with mastery of the language would correctly translate it. If I got it right, it was a nice hit of dopamine. If I got it wrong, it gave me a chance to see why I was wrong, and make the correction in the future. And occasionally, I would hear the English, and realize I had NO IDEA how to translate that into Turkish. In those instances, I could write down the sentence and save it for later studying.

The second practice I especially enjoyed was listening to Audio Bibles in my target language. Because I’m pretty familiar with large chunks of the scripture in English, listening the Turkish audio bible (especially after I had a basic conversational understanding), was a really good source of Comprehensible Input. I would listen in Turkish to passages that I knew fairly well, and when I didn’t understand something that was said, I could use the context I knew from English to help me “fill in the gaps”.

(I’ve actually also done this with Hebrew, and with Koine Greek as I’ve mentioned before. 🫣 )

The one downside with the audio bible technique is that, when working with a language that I’m less proficient in (like Hebrew or Greek), it can be really easy to lose track of what is being said. I find I can follow along fairly well when I’m understanding 60-70% of what’s being said… but if I don’t understand anything for a full sentence or two, it’s very easy to lose track of what’s happening in the audio, and to not be able to follow it at all going forward.

And if you’re not comprehending the input, it’s not really Comprehensible Input, is it? ¯\_(ツ)_/¯

Because of that, I had the thought a few times while listening to foreign language audio bibles that I wish I could combine both of those above experiences.

I wish I could hear the English of a verse of scripture, followed immediately by that same verse in the language I’m studying.

Basically, I was wishing there was such thing as an Interlinear Audio Bible.

If you’ve never used a normal Interlinear text before, they’re kinda awesome! They show you the same text in two different languages, aligned so you can read both sentences one after the other. In the case of interlinear bibles, there is also often extra grammatical information that is also shown (example).

My thought was that it would be awesome to have something like that in audio, too! Where I could hear a verse in English, and then hear that same verse my target language. Then, if I don’t understand a few sentences of the target language, I could still follow the narrative effectively, and pick up my understanding of the target language a few verses down the line.

However, if you’ve ever done any audio editing, you know how insanely long and time consuming of a process it would be to make something like this manually. You would need to have the text of both languages, and the audio of both languages open, and go through verse-by-verse, stitching them together in an audio editor, making a new file from scratch.

I imagine it would take me a little less than an hour to make a single chapter like this manually. The whole 1189 chapters of the Protestant canon would take MONTHS.

That time investment definitely wasn’t worth it to me.

So, I decided to leave that idea as just an idea, among a long list of other ideas. 🙃

I didn’t really end up thinking about it again for probably 5 years.

🤖 The Bots

Then, recently, while doing some workflow design research for Waha, someone showed me the open source NLP models that Meta has been publishing. Specifically, they showed me the Massively Multilingual Speech (MMS) project, which offers Text-to-Speech and Speech-to-Text functionality to over 1,100 different languages.

🤯🤯🤯

That’s a lot!

While digging into it, I discovered that their open source MMS model and code also included support for something called “Forced Alignment and Segmentation” in all those languages! Forced alignment is a functionality where, if you have the text that’s spoken for a given audio file, you can find out where in that audio each piece of the text starts and stops. In the case of an audio bible, I could find the start-stop times of all the verses in a given chapter, assuming I had the text and the audio available.

You could then use that to automatically cut out those verses from the main audio bible chapter, and save them as individual files.

This is where things start to get interesting!

If I can programmatically take a chapter of an audio bible and split it into audio files of each individual verse, I could easily reorganize those verse audio files however I wanted!

I could, say, run this process on to audio bibles from two different languages, and then organize the resulting files such that they alternate verses between languages!

Then I could then use ffmpeg to stitch those files back together into one single file!

That would automate the most time-consuming parts of the process of creating an Interlinear Audio Bible, like I’d dreamed of having years ago!

Well, I gave it a try with the Berean Audio Bible and the La Biblia en Español Sencillo.

Here’s what came out!

I’m really happy with it! Instead of taking an hour or more to make a single chapter, I was able to put it together in just a couple minutes (after I cracked the much longer and more complicated process of figuring out get the MMS model working on my laptop, of course 😅). I could now make another chapter in just a few more minutes! Or, with maybe an hour or two of additional work, I could write a script to do this automatically for all 1,189 chapters in the bible, and generate a full Interlinear Audio Bible!

I might end up doing that if I can find some free time this week.

I listened to this file on-repeat while making some Banana Bread last night, and in just the first couple plays-through I was able to internalize some Spanish vocabulary and grammar that I hadn’t yet been able to crack, even after more than a year of studying Spanish on Duolingo!

And I wasn’t even “studying”!

I just had the audio playing in the background while I was working.

I’m a fan of that!

So, I’m looking forward to making more of these!

🤔 P.S. The hardest part about this

Something that’s worth mentioning is that the hardest part of this project, ironically, wasn’t getting the AI model set up and running on my laptop!

It was, instead finding two Audio Bibles with license permissions that allowed me to manipulate and redistribute them like I’ve done above.

The VAST majority of audio bible licenses out there would not have allowed this ☝🏻, meaning I could have been subject to a DMCA takedown from their owners.

So, my thanks goes out to the people behind the Berean Bible and AudioBiblia.com for making their translations and recordings available in the public domain (Berean Bible) and Creative Commons (La Biblia en Español Sencillo).

I wish more organizations would follow their lead!

Copy.Church has some great explanations and advocacy about why this should be a much wider practice.