I was so close to finishing this tonight. What you're looking at is this monster error log from my data flattener that I wrote out to process the musecore corpus. 240,000ish pieces of sheet music.
The process here is really interesting! First, I take the index csv and dup it into a SQLite database, where I create two tables. One for meta data about the pieces that exists nowhere else, like category. And the other for the fingerprinting system I wrote that measures the similarity between any two pieces of music on the system.
At the end of the process, it compares the sonic signatures of everything, to everything else, because of course it does. And then, when there's a match between any two or more things, it adds a record to the SQLite database in the relations table notating a similar piece.
Still with me?
Assembling the training record
Third step of the process is reading through the compositions themselves, figuring out this batty file system in the corpus they gave me, and stringing all the data from the disparate sources together.
Once I have a pretty little line of data that includes the musical composition, the particular pieces of meta we want to train, and the related pieces, it generates 240ish thousand perfectly structured completely pristine markdown files that we're going to throw at Claude by means of API, so we can transform them to get musical variations, and calculate out some simple mathematical notation that musecore doesn't have.
Markdown in, markdown out.
Then we QA with ChatGPT 1o, to check for mistakes, obvious copyright infringement, and hallucinations.
Then we're ready to train our first Lora. Mwhahahaha.
Or at least... that's how it would go... if it was behaving.
Why do I have to run into bugs five minutes before bed?
Anyway, I love you all.
Night.