MIR for Early Music Documents: Technologies behind the Semi-Automatic Encoding of Mensural Sources

well first thank faan for inviting me I am really happy to be here um and talking about one of my um favorite subjects so Mir or music information retrieval for early music documents in particular I'm going to focus on the Technologies behind the semi-automatic encoding of menal sources into symbolic scores um so again this is my new um affiliation the University of nois but it what I'm presenting here is mostly my work during my PhD so I'm including my affiliation to Mill University so uh what is behind this long title technology is behind the semi-automatic encoding of menstrual sources into symbolic scores um first I want to clarify what menstral sources are and what are symbolic scor and why do we want to obtain the symbolic scores from these particular sources um so going into the first question what are menstral sources these are music sources either manuscripts or printed books written in menstrual notation so of course the next question would be what is menal notation um so it's a notation system using the Middle Ages late Middle Ages and the Renaissance for notating polyphonic music so music where each of the voice es like you can see in there has their own particular Melody H it is the immediate predecessor of our current notation system the common western music notation and as such already has certain similarities you notate peit in the same way you have a staff lines as you can see and at the very beginning of each of those States you can see that you have Cliffs um so p is clearly notated and in the same way as we notate it nowadays you already have different no shapes however the no shape is not enough to convey the duration of of a note in this type of notation uh so what do I mean by this it is common to have the same note even next to each other and in that case even with the same pitch but they will have a different duration so this is the modern transcription of this passage starting from those two notes the first note has a ternary volley which they call perfect and I'm going to talk about later and the second note has a binary value so the first note is a dotted half note so three quter notes ternary volue the second one is just a half note so it has just a binary volum and another example here with these two semi briefs which are those diamond shapes over there and the first one has a regular quarter node value but the following one has is twice as long it's a half note so what what is going on so in menstrual notation the duration depends on two factors the first one is what we call a mensuration which is basically the meter of the piece H which can be duple meter or triple meter in duple meter the noes have a binary value by default in triple meter they have a ternary value by default which is what was happening in these pieces in this first example the first brive has a turn value by default so what's happening with the next brief what's going on there that's when the other concept comes into play which is the context in triple meter the duration of the individual not symbols is not absolute it's not just that default value but rather depends on the context the context refers to the notes preceding or following and it can change the value of a note so um this piece I don't know I am not going to tell you why we know it's in triple meter but it's in triple meter and you have the three four here in the the time signature to prove it so it's in triple meter and that means that all the briefs all those Square notes are supposed to have a tary value by default um so again that's what happened in this particular note it's a perfecta again what I call perfect is theary value and the other one is not because the context change its value the context refers to that following semi brief which magically change its value from that perfect note into an imperfect one that's what they call when they lose one bit and becomes an imperfect or binary note uh and this is a common modification based on context called imperfection ER there are two common modifications that happens because of context in tripometer the first is Perfection the other one is alteration and that's what happened in this second example this note is a a regular semi Brea is a regular note and then the other one that the one that follows it's actually alterate and this alteration means that a note doubles its value so those are the two kind of modifications um so this is just to give you a perspective of what's happening in menal annotation and the things that we have to deal with when working with it um which is clearly notated node shapes do not explicitly convey the value of a note if we're in this triple meter H their default value is this perfect tary value but that can be changed by the context in these two ways by an imperfection where the node becomes imperfect or binary or by an alteration which makes the node twice as long um so there are rules about when to apply the these H context dependent modifications when to apply imperfection when to apply alteration when to apply both of them which is what's happening in the second example we have an imperfection at the very beginning here this note is imperfecta and we have an alteration here in the red note or a want to apply none of them so that's the first part this is just to tell you it was like a uh me notation 101 just an introduction and um so now that we talk about menstral sources why do we want of them symbolic scores from their from them and what are symbolic scores so H usually menstral not menstral sources are written in separate parts so either a having the parts in a different area of the page or the book opening or have them in completely separate books like part books um what we want is rather to have them in here in this separate parts layout to obtain them in this kind of layout in a score layout where all the notes are vertically aligned why would we want that so there is an advantage on having them in the score layout if you have them in the original layout you can only see the melodic information for each of the voices if you have them in the score layout you can in addition to see that melodic information you have access to the vertical sonorities you can see which notes are being sung at the same time um so that's the advantage of having them as a scores H but we don't want just any score we want a symbolic score which is a score that is encoded in a symbolic or machine readable format like Mii which I'm going to mention later and that has also advantages in itself because if the machine is able to read the score it can um process it for example it can play it back it can maybe converting to Modern values if you have a script for that and it can do much more um but of course to obtain this H we have to deal with the interpretation of the noes duration with all these imperfections and alterations and this kind of H context dependent modifications so um now let's move into that we have talked about symbolic scores let's move into symbolic formats that we can use to encode this type of notation so first of all a a symbolic format in general terms is a machine rable format that incode symbols of a music document um I included music XML because it's like the most common example and maybe the one that you're most familiar with and you can export it in finales CIO music score and other graphical editors for music notation um it's typically the way that a you use it as an interchange format maybe you have just C values and you want to pass your file to someone that is using finale of course you cannot send your file and because they won't be able to open it so you can export it to music XML and the other person can import it in Finale and actually see H the music containing there uh but of course music XML and all of these graphical editors are just for common music notation and we want to use something that has aort from notation so these two are uh the main formats that have support for M notation Mii and Home Room so Mei stands for music en coding initiative um you might have heard about it in this lecture series or will hear about it later on as well um me refers to the scholar community that works on defining the best practices to incl a wide variety of Music documents and also to the format that they are investing their time in creating um so as I said it can include a wide variet of Music documents and that includes documents in early music notation so documents that have Noms also tablet or menstral notation um for the people that are not familiar with Mii it looks like this like the example on the right you basically have tags and each of these tags represents an element so in this case I have three elements in myi a beam and within that I have order to Children Elements which would be these two notes and this is representing what it's in in this image over here so a beam with two notes um it follows this kind of hierarchical structure so this beam element has two child notes note elements and each element can have attributes that Define a some characteristics for that element so in this case the first one has a p name attribute which represents the pitch name which would be f it has a knock attribute for the octave so it's an F4 H that's uh this particular F over here H the door for the duration is an eighth note as you can see in the image and the stand direction is up so it's basically just defining characteristics of that particular element and the next note element of course is the same but the pitch name is C rather than f um and what we're using to render the Mei so to display the music that is actually encoded in Mii it's Verio um which is a C++ Library developed by luran pan so as I said this has support for menal notation and you can see it here um so this is the same example that I um showed you before so we have here um one of the briefs as you can see it has a perfect alary value and here we have another one that has an imperfect value and they are both encoded in the Mei file you can see that they have the right Peach and octave and both of them are bries that's the no shape but the one at the beginning of the passage is imperfect the quality imperfect and the one at the end of the passage has the quality perfect so all of that can be encoded in Mei similar with what's happening here for the semi briefs this one has a regular H value and this one has an alterate value and you can see that in the Mii encoding H this is just dur semi brev and the other one it has an additional D quality attribute that says that it's altered um so one of the formats to encode my strong notation the other one would be hrum uh so this is a symbolic format defined by a spine so by this columns basically each of these columns is one of the staves that is present in the image so here you have your notes for the soprano the notes for the alto the tenor and the base so if you basically rotate this like 90° you will have the score that is over here um so the home format for common western music notation is Kern which you can see here at the very beginning of the of the file in line nine H this is encoding common music notation because here you have the text that says current for menstral notation is men's and the other advantage of hrum is that you have already a lot of music analysis tools they can only be used with C so with the common music notation but you have even tools for renaissance music you just have to have a modern transcription of the Renaissance piece so as I said support from inst rotation this is the same uh passage included in Mii like I showed before and also in hrum um so you can see it has the same features U the first note has c c to represent the note C in octave 5 H and then you have here the other notes that are un OED below b a g and d and the first letter in each of these entries is basically the not ship so capital S is for a breev you can see it in the first and last note H and then a lowercase s is for the semi briefs so for the diamond shapes um and for all the Perfecto imperfecta and alra you can also encod it there you can see the characters in color the I for imperfecta the Plus for alra and the P for perfect so to great for in code menst Nation so now that we have covered what menstrual sources are H what are symbolic scores H I'm going to focus on a real life example of a project that involves the semi-automatic encoding of menstrual sources into symbolic scores in particular about the Mir Technologies so this is a project that I conducted for my dissertation the goal was to preserve preserve and increase access to a set of polyic qu books in Guatemala why Guatemala because that's my my country so I wanted to apply to to those sources um so preserve an increased access to this by using digitization and music and coding Technologies so I obtained permission for a a pilot project with a book of masses from guala and the first stage of course was to obtain the digital images because in that case there were a high resolution color digital images but that's another topic and we had to even do a DIY or do a yourself book scanner and everything because we didn't have the equipment for that but again that's not music information retrieval that's before so H what I'm going to focus on is how from these sources with the different um voices we obtain this Mi this Mii Mi score sorry with interpreted durations um and again the idea of this course is to increase access to the carpus because a you can ask a um bovio which I mentioned before to actually play back the music H if you write an script you can actually H perform automatic transcription of these in to Modern values because you already have the durations and everything is lined up and so the first will make it accessible to the general public the second the automatic transcription into modern values would make it accessible to Modern musicians and the third point is that well this music came to Guatemala from Spain so you have a lot of a concordances with the European sources so if both are encoded the European Source the peace in European source and you have encoded a piece in the Guatemalan Source you can um do comparisons of concordant sources easily using software so that's also interesting for experts because they don't have to focus their time on transcribing the music into a score but Focus your time in music analysis so H for this project this was the three step process digitization of course is H basically photographing the the folio the pages of the manuscripts then um Optical music recognition so Optical music recognition is similar to optical character recognition or OCR so that's when you have a an image of a text document and you use OCR to recognize the text characters in that document and in this sense you can make the image machine readable and searchable so you can do the exact same thing with omr or Optical music recognition so it will give you a symbolic file an MI M eii sorry an Mii Parts file that encodes the symbols in each of the parts or voices that are in the original manuscript and but as you know we are dealing with menal annotation and encoding the symbols just the pitch and the no ship is not enough because of the previous explanation so to provide a full information regarding the node duration and to obtain the score proper line up H we also need to encode all the imperfections and alterations so we need this uh third step uh that gives us the score so this third step is actually um consist of two parts the first one is something that interprets the duration of the notes and aligns them in a score in a score layout so the automatic voice alignment or automatic scoring up of the parts and then um the other um part would be the tutorial correction because um to properly line line up the piece you also need to correct the scal errors are kind of messing up the alignment for you so this part will also correct a the errors and incode both readings the original and the corrected one U so that's the workflow um these are the tools behind the scenes as I told you we have to build our own DIY book scanner for this and I have slides like extra slides for this if you want to see them but it's not relevant to this particular talk um but yeah we built our own book scanner for that particular manuscript um so I'm going to focus on the music information retrieval Technologies so the first one is Moret a muret is the toer we use for omr and it's H developed in the University of Alicante by David R um and it has support for obal music recognition of both common music notation and M notation both handwritten and printed so um the interface looks like this this is the overview interface you can see all the pages of a particular piece that's actually from the guala manuscript um and you have three icons on on on top of each of these images and they represent three different interfaces uh corresponding to each of the steps to complete the omr process in what so the first of those interfaces um is the document analysis or document segmentation so this is a common pre-processing stage in omr so H basically in document analysis or document segmentation you segment your documents into the pieces that you will use afterwards in this particular case in modet you ask mod to automatically detect all the stuff regions for you because those are the regions that have the music symbols that you want to eventually encode so that's the first interface the second one represented by this guitar on the top if you can see it this one is where um you assign the voices so one sure not this step is performed automatically you ask Moret to do this for you it has been trained to do that um this other step the assignation of the voices that's a manual step it's really simple you basically do a multiple selection of the staff regions and assign them to a particular part a particular voice um the next um the next interface the last one actually performs two processes that are common in omr the first one is the music symbol recognition process in this case you select any of the stuff regions of the pages the ones that have already been recognized during the document analysis so the first stage and you as Mur to recognize the symbols for you so you as Mur to recognize the symbols for you and it will a show you in this middle panel the symbols that it has identified and um it basically has two types of information for each of these symbols the type of symbol uh so if it's a GFF uh here it looks more like modern so a whole note a half note or these kind of things and the position in the staff where it is in which line or space it lies so you have the interface to correct it you can move the peach up or down or select a line or space that denote is lying on and you can also correct the symbols using this panel on the left just going to drink some water um okay so that's the music symbol recognition part H so it tells you the class of symbols that it finds and the position in the staff where they lie the lineer space and again you can correct the all this information in that interface um the second thing that it does um it's the music notation recognition step of omr this is another automatic process so now that you have the classes of symbols and that you have the line or space where it lies you can H basically deduce a the peaches for for example by using the C that is here so it's trying to not only recognize the symbol but also kind of reconstruct the notation so this is uh what in mored they call the the difference between what they say it's an agnostic representation so something that is agnostic to any meaning of the music symbols and what they call a semantic representation so something that already understands a little bit more about music semantics and music structure so a in here you will have information about Peach already uh and you can correct all of this information using this panel on the left this panel on the left is actually home drum format is men's the format that I mentioned you could use for encoding mensor annotation but in this case I don't know if you will be able to see it but in this case h it's encoding Peach and noap but it is not providing anything about perfect imperfect or alterate quality that was not a what they wanted to do um for this interface um if you want to encode that you will have to manually add the right characters for this in here so add the P the I or the plus sign for for all this type of information so at the end when you're happy with your results you can select a couple of pages or the whole piece and Export it into Mei and that's a the last stage of any omr process which is the en coding of the piece so we have the symbols we have the omr done we can move to the third um part of this workflow the one where we get the score so for this I use the measuring polyphony editor this was being developed while I was H doing the dissertation the sorry the principal investigator for this project is Professor Karen Desman and that's her old affiliation Brand's University when she was working on this um now she's in men I think um and the developers of these are actually from Mill we have Juliet Reginal the lead developer and you have me which I implemented this voice alignment functionality this interpretation of the note values um and Consultants like lauran pan from perio craiga from hrum and Andrew Hinson from triple so uh the measuring polyon editor or MP editor looks like this it can be used with or without omr so H the NP editor can be used from scratch by uploading a manuscript and entering all the notes of the piece by typing them in your keyboard um you can load a man from Gala and ecodis I think or you can upload an Mii file that has already all the notes encoded for you like the one we generated with the omr and this is the way that we are going to use it and that I use it for this project so H you have all the symbols coming from the omr um again you're free to enter all these yourselves like you create each of these staff regions and then type in the symbols but in our case they come from the omr file so let's move to the next step of this editor which is the score editor so we continue into the score editor and that automatically lines up the notes for you in this format that's the part that I work on and you have some options below to make it more a human readable and you can switch to Modern Cs and you can bar by different node values H so here we're barring by the semi Bri that's how the piece looks when it's bar by the semi Bri so it shows a little bit better the perfect groupings or the triple meter of the piece and you can update the voice alignment when correcting notes so H you can select notes you can move them up and down to change the pitch and you can change the note shape you can insert notes and every time that I insert a new note you will see how the voice alignment um updates H here I'm going to insert a d of division between those mins to sorry here I inserted it here there's no dot to change the perfect groupings um so you can do all of these modifications here and in case you want to record both readings the original and what you modified like in case of scribal Errors you can do that by moving into the editorial mode over here H so in that case you will have whatever was there before and your correction encoded in the Mii file um and when you're happy with that you can download your Mei score so this is the general workflow I use it for um for encoding these pieces from the particular manuscript um of course there was a lot of work that had to be done for example for making these two tools that already existed interoperable so the muret the omr files that were exported were actually useful for the measuring polyphony project so we made them a so that they follow certain measuring polyon specifications H and I guess you already know this or you already to doce this we did that because H we wanted to make the press as automatic as possible right so when we export something in Mei for a within Moret we can actually obtain the score here if we want but you will see that that score is not really line up you can see here it's not really lining up and that's because all the perfect imperfect and alterate values are not encoded if you want to encode it again as I said you will have to manually enter it here in the men's file um so we don't want to do that the omr is capable of recognizing the symbols and that that's really well done in here so let's use that Advantage um that automatic part and as you know you have very nice interface to correct every symbol that is wrong um so let's use that and then um let's export it to the measuring polyon editor for so that that takes care of the other process that you will have to actually do manually here um so I made a lot of work with u the two developers of the tools uh of these two tools uh so then of course you can export that into the MP editor as soon as you click that you get transported into the measuring polyon editor and you can load the file in here and as I said you have all all of this information already coming from the omr so you are avoiding this H manual task of entering all that yourself and just focusing on the automatic test and correcting that so trying to get him the best of both Tools in here so as I said a I work with the developers of these two tools David R and jul Reginal to make them as um interoperable as possible so we have a paper on that the other thing that we did is um so I have talked about the automatic voice alignment in that part of the measuring polyon editor but I haven't talked to much about the editorial correction part so H what we wanted to know is if there was any way to facilitate the correction of your file once you have it score if there was any way to facilitate the detection of these scribal errors so for that I work with a Craig app who works on hrum and er we try to use this music analysis tools from hrum hrum has a nice disant filter that what it does is that it labels the disan as on a piece for you H so we integrated that into the measuring Pony editor you can actually activate in here and when you activate that what will happen with your score is that it will display some labels below some in blue some in Orange the ones in blue are the like wellknown or let's call them legal dissonances so for example here have a few passing tones you can see here a passing tone between these two notes or between these two notes detor we have here a lower neighbor we have here a suspension and here it's resolving the suspension and the agent of the suspension is marked here for example so all of that is marked in blue like let's call them legal dissonances and everything other dissonances that are unclassified they are marking orange and these are basically seconds and sevenths that are a that don't seem to fit any functionality and also for Force above the base and there are different types of unclassified dissonances so for example unclassified dissonances in parallel accompaniment and all these kind of subclasses of unclassified resonances so those are marking orang so the idea was to test whether making these illegal resonances H obvious to the user help in any way Inc correcting the score um and we evaluated that with a subset of the pieces in another publication and discovered that yes indeed marking the legal lances reduce the correction time H because rather than going through this whole piece and find out why uh through all this piece uh what is messing up the alignment and why the the notes don't end at the same time you can just uh Focus your search ER on basically all the notes preceding the first orange label Yeah so basically this this reduces your region to look for errors to just this part um and um let me see so that it reduce the amount of time for your search for errors and in addition to that it also increases the accuracy of your Corrections because um going back to the same example H you can see that the Cadence following that first orange label uh if you look at the Cadence you can see that it reveals that ER this voice the alto has to be cut by a minim it's a minim too long so the reason why this can increase the accuracy using these labels is because if you test different solutions you can see what happens with the orange label so for example if I take this rest and I cut it by a minim I will see that the labels following the orange labels following disappear but also a new one appears so that's probably not the right solution we can experiment with another thing I already know the resolution so I'm going to point it out if you cut that semi brief by a minim then you can see what happens stor labels disappear the Cadence lines up so that at least you gives you some hint you can test your different solutions and that increases the accuracy of finding the right solution and other thing that we notice was that surprisingly um marking illegal resonances also helped detecting omr errors that went unnoticed on the previous stage because we already had conducted the omr we have already corrected and there were a few errors that went there and we were able to detect them with this so this is the work I did with a craiga from H drum and Julet Reginal from the measuring polyon project oh by the way the disent filter was developed both by Craig app and Alexander Morgan attribution is on a previous slide so H all in all the outcomes of this project we're kind of still working on this project my dissertation is done we're we're still working um I have a musicologist now check all the final pieces but the outcomes consist on the digital images which hopefully will be in D the menstrual Mii files including both the separate parts and the scores they are on GitHub but we are still checking them and also the workflow he this workflow presented here allows for the semi-automatic transcription of menal music from digital images of manuscripts into into a scores in menal notation H it's applic to other menstral music sources don't have to be only the Guatemala Corpus um so the workflow can be reused by other communities around the globe er um even the ones who have collections in need of preservation but have limited resources because I was trying for all of these tools to be as accessible as possible too so these are DIY tools like the DIY book scanner that I mentioned before and the software is free online and open um so um the other thing is that I want to emphasize that this project was trying to H contribute to the development of research project that already exists rather to implement a monolithic tool that as it all um so it was contributing to each individual project and to their interoperability and people always feel good when you reach out to them and say oh I'm interested in your tool it's working pretty well and I want to do this and maybe uh put it together with this other one so people get excited and getting invested if they can so and it's better than just constructing another tool that might eventually just stop working if you don't follow through um so it's an example of bringing projects and researchers together for a common goal um I wanted to show one of the pieces just so that you can see the outcome um let me see so this is one of the triple meter pieces yeah this is just the Sanctus of this Misa is actually the Sona movement of the sanus of this Misa the title is a little bit misleading because it's not the whole Mass we don't oh thank you so here is I I just uploaded it in this is um this is Verio that is rendering it's Mii viewer so you can go to that URL and upload your piece here uh so I just upload one of the pieces and um Veria can play it for you so let's hope you can hear it okay so uh that was just a one movement of a of the sanct of this piece let me change to my slides now okay good um so what I wanted to do with this presentation was to present you an overview of the tools that exist from mation and how to put them together for a particular goal I am there are tools that I haven't mentioned uh so the ones that I mentioned here were two file formats the music en coding initiative the H drum men's format ER I also mentioned the Vero library for rendering uh these files rendering a music ER from rendering Mei and I mentioned some Optical music recognition tools I mentioned muret there is another one arus I didn't use her speak because this is for printed notation it's also developed by lauran pan who developed B A also mentioned the measuring polyphony editor as an online Editor to enter and process M for annotation I use it for its voice alignment or scoring up automatic functionality and and it support for editorial Corrections there's another tool this is more recent H Merit H which also deals with interpretation of duration of the notes in motation but more from a analytical point of view um it's developed by Anna flaxin and David Lewis um and I also mentioned the use of Music analysis tools like a the disant filter from the hom Live library uh which is one to from hrum um I want to acknowledge all the institutions and all the people that helped me in this large project of the digitization of the guala quir book and I want to thank you all for your attention these are my old email still works and my new email so thank you [Music]

2024-12-23 16:21

Other news