LOAD ZX 40: "esxDOS - Portuguese Technology" - Miguel Guerreiro
This next talk makes me really proud, because Miguel has a very big importance, and in my opinion, he is not very well known. Therefore having him here with us today, knowing that he was with us also in 2019, when we made the opening of the first exhibition, is a great satisfaction. So, thank you Miguel. Miguel Guerreiro, a ZX Spectrum enthusiast since the age of 5, this computer has accompanied him and, in some cases, shaped his life. In more recent years we have to thank Miguel for the the memory cards connected to ZX Spectrum clones or peripherals like divMMC and others.
By the way, Miguel is co-creator of the divMMC specification, together with Alessandro Dorigatti and Mario Prato. Miguel also has a very large history of things that he donated to the museum, with a great historical significance, and I am remembering one of the first divMMC that, with Mário Prato's permission, was given to us. All these systems use a system called esxDOS, which was created by Miguel in 2005, that is, if today we use memory cards in some clones or most of these peripherals, it is due to a Portuguese that we should thank and that is very important to me. In addition to his interest in new hardware and operating system programming, Miguel is also involved in what is called the ZX Spectrum Demoscene since 1995, having accumulated several international contacts. He also participated as a programmer in some introductions, demos and disc mags, i.e. digital magazines, and tonight, at that gathering we are going to have at the museum, Miguel is also going to demonstrate a little bit what this Demoscene is all about.
Filipe (Cruz) is also present, who I know usually also collaborates with Miguel. And so, without further delay... Miguel, please join us for your talk. Good morning. My name is Miguel Guerreiro.
And as João said, I've been a Spectrum geek since I was five years old. Technology has evolved, but I always kept the Spectrum in my life, one way or another, and I tried to make the Spectrum evolve and more or less keep up with the evolution of technology. In 1998 I was still using a Spectrum +3 with three-inch floppy disks and those floppy disks crashed a lot and it was becoming difficult to find floppy disks for sale. So I created at that time an operating system called SpectrumOS, a very original name, in that I had a Spectrum connected to a PC with a serial cable. I apologize, but this is going to be quite technical, I have no other way to talk about these things.
On the PC side, I made a program in Pascal that received commands from Spectrum and returned directory listings or file contents and that was it, I could load programs like that from the PC. The downside is that it did not support any kind of TAP files or snapshots, I had to convert all the software to that specific format. Here is a picture of what this operating system looked like.
Basically it showed the PC drives, there is the F drive. It was some kind of drive that I had on my PC at the time. In 2000, I put together a group of programmers in England that was called Raww, with two w's, and made a new version of the operating system.
I was able to increase the speed to 38,400 bits per second, and with that I was able to make a graphical interface, rudimentary but it still a graphical interface and added on the PC side a command that returned the mouse position. So it allowed me to share the PC mouse with the Spectrum. I would move the mouse on the PC and it would move also on the Spectrum. After a short time, someone offered me a standard floppy disk interface, three and a half inches and then I completely abandoned this type of system. It was like this, it had a desktop image, was already able to upload snapshots and the save screen saved an image and ran the same executables as the other version, which I had created in 98. In 2001...
I have a very good friend of mine in England, Neil Laws, who has also been involved with the Spectrum for many years. He works on hardware, graphics, and music and we were doing a project that was to take a motherboard from a Spectrum +3 and modify it by putting more RAM in it, put an IDE interface, mouse, etc. To have a more modern Spectrum, suitable to our needs at the time, if you can say that. And it was here that some concepts that are in the current esxDOS began to emerge. This project never saw the light of day, because I'm not very good at hardware and then Neil became disinterested and the thing died there. People here may not know, but in the former Soviet Union there were many Spectrum clones that used a system called TR-DOS, and at the time, from '92 until maybe 2002, they made a lot of software, They even made ZIP and RAR decompressors.
They made a lot of software, but it only ran on that system, the TR-DOS. So at that time I started to make a TR-DOS system emulator for the interface +D, which was the one I had, which read three and a half inch floppy disks. It didn't run all kinds of software because that required traps at the hardware level, but it already runned a lot of software and this was the foundation of the TR-DOS emulator that exists in today's esxDOS. Here is the image, it should appear on the left side, the file names, etc.
But that's all I could find because a lot also I have already lost over the years. In 2005, a guy from former Czechoslovakia (I think) had created an IDE interface for the Spectrum called divIDE. This interface was simple, inexpensive, Anyone or any company that wanted to, could do it and sell it. And all Spectrum users around the world started using it.
It became a standard. Neil, once again, sent me a divIDE, he offered it to me, and I immediately started programming for it, with the goal of starting to use disks and abandoning floppy disks. He also offered me a device - that I was just talking about to someone outside - in DivIDE, the code runs in a ROM and to test a new iteration of the code, you have to flash the ROM and restart it in order to test and that would take some time.
So, he offered me also an emulator of the ROM, which was a device that I fitted into the divIDE interface and connected a parallel port to the PC, compiled the code on the PC, I would press reset and it would start automatically with the new version of the code. Without this, I believe that things would not have evolved so quickly. This was really the magic that allowed us to move forward very quickly.
With that device, and at the time I had also quit my job because of burnout and I stayed at home for six months, so I took advantage of that and got a beta version 0.5 ready in short time, that could bear to read filesystems FAT12/16/32, and supported writing only in FAT16. This version, as I did not consider it (because it had bugs) suitable for public consumption, was only shown at an event in England and it wasn't released to the public. Because of this many people started to say that it didn't exist, that I was bragging and saying that I had done it, but it wasn't true because I didn't release it, so that meant that it did not exist - This version had a problem with the files, they used a proprietary header and it was necessary to convert the files on the PC before you could use them on the Spectrum. Here is a picture of this version, the first that was released to the public.
The other versions, 0.1, 0.2, 0.3, 0.4 were internal, I'm the only one who saw them. In 2009, after a few years, a version finally came out to the public, the version 0.7.3, which at the time had fewer bugs, already supported writing in FAT16 and FAT32, without corrupting anything. The file headers were the same ones that the Spectrum +3 used, which were the +3DOS headers, because this way it was more compatible and any tool that worked with Spectrum +3 files could convert them. This is where the complaints about the esxDOS not supporting Long File Names (LFN) started.
As some of you know, in the old days before Windows 95, files could only be eight characters long and three for the extension, and my DOS to this day still only supports eight characters and three for the extension, and there started to be a lot of complaints, Some even almost evolved into physical violence. I'm joking of course. At this point the Slovakian programmer UB880D, to this day I don't know his name, I have been with him several times, I only know this nickname. He contacted me because he liked the operating system and wanted to start programming commands and extensions for the operating system. Here is an image, this one its actually a photo that someone sent me, because I think I didn't have any. In 2012, two versions were released and I can't remember why I released 0.7.4 on April Fool's Day, which only fixed two or three bugs.
I think it was just for people to say: "The guy had three years, didn't release anything, now he only releases this? He doesn't do anything." On April 23rd, ten years ago today, on the Spectrum's anniversary, the version 0.8.0 was released which was possibly the version that had more fixes and new features, including what people had asked for, a menu that when the button NMI was pressed would allow you to graphically choose files, etc. This was UB880D that created this menu. I work in Unix system administration and I only see the console and commands, but for the rest of the people is not quite like this, so he created a menu. Complaints about the lack of LFN support slowly began to turn into threats.
There was a guy who was after me sending mails and calling me names, saying that it was unthinkable that the Spectrum didn't support long file names (LFN). From this version foward it also has the logo up there on the left side, which is a waste of ROM, but that's what it is. In 2013, Alessandro Dorigatti from Italy, contacted me because he had created a coreFPGA, which replicated a Spectrum 128K perfectly for the V6Z80P+ board. But that board only supported SD cards, it did not support IDE disks or Compact Flash cards, which is what divIDE supported. He had no operating system ready for that core. What I told Alessandro was: "My system is 100% geared towards the divIDE memory mapping model and I can't get around it, but if you can implement this model of memory mapping and if you give me the ports to access the SD cards, I should be able to make a version for that pretty quickly." This is where the modularity of the esxDOS showed its worth, because all it took was replacing the IDE driver to an SD or MMC card driver and it worked right away.
It was a one-week thing. The version 0.8.5 was released in June 2013 and already had support for the divMMC interface, which is what we called it, because was a mixture of a divIDE and MMC, which is the original name for SD cards, and for the Betadisk/TR-DOS emulator, which I had started years ago at KGB. Here Mario Prato produced the first divMMC interfaces, I personally was against it because I wanted to add some things to the specification that were not added, because he then released that and it was no longer possible to change the specification. One more image.
The Spectrum Next, in 2016. I don't remember when the Kickstarter started, does anyone know for sure if it was that year? If it wasn't during the Kickstarter, it was as soon as the Kickstarter ended. I think it was during the Kickstarter.
They contacted me, Henrique and Victor Trucco, they wanted to use this DOS as the official Spectrum Next operating system and they sent me two prototypes, i think. Yes, there is a recognition I want to make. We have something in the museum, which I didn't even know existed and never thought of ever owning, that Miguel gave us, precisely because of this story. We have a prototype of the Next, Next is a Brazilian project, comes initially from a system by Fabio Belavenuto and Victor Trucco, which then evolved into what came to be Next. It had its first crowdfunding and is now in its second, that they will deliver next year. But in between, there were prototypes and there is in the museum a prototype (that I think is called issue 0, but I'm not sure), of the Next, which was Miguel who one day gave it to me, something I never imagined existed and it has been on display at the museum for quite some time now.
Thank you. So they sent that prototype, then they sent the final version, which would become the Spectrum Next. There is also a Czech guy who is Velesoft, again, I don't know his name, I only know him as Velesoft, and we developed the divMMC architecture as I intended and sent it to the Next team, for them to implement, but then they decided to use another solution.
The first time I tested the DMA (direct memory access) on my DOS was with the Next, but they had a bug - for every byte that was read, 4 came back. And they couldn't solve it at the time, and I then stopped using DMA. The problem was that they talked to me at a time when I didn't have much free time, and they were asking for new versions every week and I didn't have time for that, so they ended up choosing another operating system for the Spectrum Next.
A few years later, I got in touch with some people related to the Spectrum in Brazil, that now I don't remember the name of, because they have some interfaces that also run TR-DOS which are CAS and CBI, and from time to time I like to try to implement these things and I was able to implement support for those interfaces using the same TR-DOS emulator that is used in the former Soviet Union. In 2019, Jan Kučera from Czech Republic created the MBO3+ Ultimate interface. This interface has a lot more capacity that the Spectrum, it has 16MB of RAM, has HDMI output, has everything and more. But it's too expensive.
Unless I'm mistaken, that's 300 something euros or 400, something like that. He offered me one, I didn't complain, and made a version of esxDOS optimized for this interface. Right now, the best interface for running the esxDOS is the MBO3+ Ultimate. The versions 0.8.7 and 0.8.8 were mostly optimizations for this interface, including the DMA, which works as it should and allows transfers in the Spectrum 48K, of almost one megabyte per second.
Which is very good. for a 48K. Here it is.
The logo even has the name of the interface there. 2021, is when I released the latest version. It was about a year ago. The TR-DOS emulation was not complete because the FORMAT and MOVE commands were missing, because I didn't have the space to put them, but I was able to optimize the commands from the original ROM and put them in. And also support for images in the SCL format, which take up less space than the TRD, which are the normal Betadisk and TR-DOS images.
Unfortunately, in this version I exhausted the resources of the original divIDE interface that were 8K ROM and 32K RAM. And with this I'm able to do paging, I can read things from the disk and change them. This is to try to support the users who still have this older version. So I only have two options now, either I stop supporting divIDE and support just divMMC, which has 128K to 512K RAM, or else I have to do a fork of the code, which is what I do from time to time and think it through again. Basically refactoring the code, that has already been done from the 0.5 to the 0.7
and from the 0.7 to the 0.8. And this is what I started to do for the version 0.9, which is completely different from what currently exists. In this version I already have people collaborating. UB880D and another colleague of his, are doing some parts of the code because it's already too much work for just one person. But this version already supports RAM from 32K to 512K, with malloc/free allocation to be able to load things and then free up memory if necessary. The drivers and the filesystems
were already modules, but they were static modules. They were in the ROM and it wasn't possible to load them from a disk. From now on, modules can be loaded from a SD card or a CompactFlash card, it now supports Ethernet and Wi-Fi devices, there are already a few. Finally with the memory with malloc, it will support long file names (LFN), the FAT. This is more or less already implemented.
The syscalls are reentrant... This doesn't mean much, but what it means is that theoretically it is possible to make a multitasking operating system on top of this, because you can call the same syscall in a re-entrant way. Another thing that changes, which is a thing I never liked about the current version, is that in the current version, to access drives and devices, it operates differently from how to access files. The UNIX philosophy ("everything is a file"), I have been thinking for months and months about the best way to do this, and I succeeded. So now both filesystems and files are all accessed in the same way. It is very advanced, as you can see, it has the logo, the memory and the version.
Maybe in 2024 this version can come out. Some trivia. There are many more, but these were the ones I thought I should point out. Throughout all this time,
as a person who gets very tired of the operating system I use, this has been through Windows, Linux, BeOS, MorphOS and currently macOS, because that's what I use at work and got used to it. The first code I created to read FAT, while the interface was still in the mail, it was tested on a Spectrum +2 with floppy disks. I was trying to read the floppy disks in the FAT12 format.
The versions with the most innovations (0.8.0 and 0.8.5), were programmed when I was sick at home, one with malaria, there were two weeks in which I couldn't go to work, but I was able to type, and another with a flu here in Portugal, very complicated, so two weeks that I was able to make a lot of progress. The current version has about 10,000 lines of code, in assembly and the assembler used is "sjasmplus". UNIX, Amiga and BSDOS, which is an operating system used in Czechoslovakia (Czechoslovakia no longer exists), in the Czech Republic and Slovakia, were the sources of inspiration for me.
The source code for the version 0.5 was lost in a disk crash, Everything was lost and I had no backups, I had to rewrite everything for the version 0.7. And finally, this latest version, in one year, has been downloaded 3247 times, from the official website. And that's it. This is what I had to say.
I don't know if anyone has any questions or understood anything I said. Let's move on to the questions. I was going to ask you a question, but you blew it, first you scared me but then you said you already announced the long file names (LFN), so I won't bother you with that anymore. You have also said that by 2024 the version 0.9 comes out.
Therefore, We are 2 minutes late, I saw two people raising their arm. Even more? Four, five. It's impossible. Let's respect the first 2 that I saw and then you can talk to Miguel, naturally he will be at the museum tonight. So this gentleman, and Vasco.
I'll try to be quick. I know Miguel before the divIDE project. I know Miguel because we have a partnership within our companies.
He went to my store one day, and I showed him a board that connected to the Spectrum, it had an SD Card, and he turns to me and says this: "I'm the one who develops the software for that." Later I talked to you at a restaurant table and I said this exactly, and you were a bit confused, I noticed that you don't put your name in the credits. What is the reason? I don't put my name, I put an alias. Because... I don't know... It's funny, We have a friend in common, Mário Viegas. It was through him that I spoke to you the first time and I have an identical story with Miguel. When I called Miguel the first time
and talked with him, he told me he was at the opening of our first exhibition - where you were also Vasco - in 2019, and I was speechless, he did this phenomenal thing, that I didn't know it was a Portuguese person who developed it. He was in Cantanhede, I didn't met him, and so it's fascinating. Mario asked me to ask you this question as well, (in addition to the one about the LFN) precisely that, about being a person who does not seek to have any kind of exposure and Vasco's question is along these lines, but you've already answered it, it's an option. It's so that I don't have the fans running after me on the street. Let's see if they don't ask for your autograph. There was one more question here.
Let's get to that one to wrap up the talk please. Good morning. Just a curiosity. In the midst of all these advances. What was that point that you really think it was that technological advancement, not as an operating system. What is the routine? What is the driver? What was the thing that, for you, was the greatest innovation? I think the thing that gave me the most pleasure programming in the middle of all this was, to make the TR-DOS emulator I had to make an emulator of the chip that controls the floppy drive, the pulls, the whole thing, and that in a system that has no timers, it is very difficult to do. And I think that's probably one of the hardest things I've had to to do, much more difficult than the LFN, for example.
Very good. Miguel, thank you very much! Thank you all!
2022-05-23 13:40
