Here is a high-level overview of how the Spyglass system works.
First off, what is it? Well it is a technology that allows one to click through an Apple II (and later on, Macintosh) disk image and get meaningful representations of the files therein without using an emulator or actual hardware.
This is a pretty simple thing to describe but it takes quite a lot of code to pull it all together.
The first technology is the foundation for Spyglass: Format Converter. I envisioned this technology in 2010 and I finally put it in code this spring and summer of 2013. Its name describes what it does: it converts between various file formats and it will access the contents of disk images and compression archives such as ShrinkIt and StuffIt.
Format Converter does not rely on any file extensions to determine the file type. Instead it relies on well-defined magic numbers combined with some statistical analysis (that is, files of format X typically have byte sequence Y).
Once the file type is determined, the proper converter is called. For example if the file starts with \nGL, that's a Binary II archive and the Binary II converter will be activated.
Format Converter supports compression such as Gzip or Bzip2 as well as recursion. For example, a Gzip file that has a BinHex encoded file that contains a StuffIt 1.5.1 archive. Format Converter is able to drill down the stack to get to the heart of the matter.
Up until now, the only applications of Format Converter have been the enhanced file catalogs in Vault, the selection of disk images for the additional screenshots, and downloading binaries from comp.binaries.apple2.
Integrating Format Converter with these areas of Mac GUI has been very straightforward and simple. It is easy in the small tasks, but is it easy with the larger tasks?
Enter Spyglass. Now here is the heart of this article. It is the largest application of Format Converter so far, and it does show that integration is easy with the larger tasks.
Spyglass takes the existing Format Converter features that had been developed this summer, and placed additional demands: it must recognize the Apple II file types such as lo-res graphics screens, Teach documents, and machine language binaries. It must also be able to extract individual files from the disk images.
Spending some more time with the Apple II DOS and ProDOS reference manuals and tech notes allowed me to add the code for these new abilities.
Now the last step is to put it all together into an application to click through on the web browser.
I decided that I wanted Spyglass to work from three starting points:
1.) Files uploaded from the browser
2.) Files entered from a URL to an FTP or HTTP site
3.) Files selected from the Vault Downloads collection
All of these starting points are called "resources," since that is a generic term. We start by fetching the file either from Vault or a URL, or from the browser. To prevent abuse, file size must be less than 1 MB. This limit may be increased in the future. The file is copied to the Mac GUI server and some tests are run on it. We compute an MD5 hash of the file contents and save that in order to check for duplicates. We also determine the file type using Format Converter and reject types that don't yet work in Spyglass (all the Macintosh types, for instance).
There is a provision for file expiry. Files uploaded by the web browser will expire after 30 minutes and will be deleted from the system.
Each resource is assigned a resource ID that will be its point of reference in the future. This ID is globally unique and can be used in URLs to Spyglass. If it should happen that the resource file is deleted from the Mac GUI server, but its database entry still remains, it can be downloaded again from its URL.
Once acquired, the resource will remain available to everyone until it expires. All files and directories are accessible via static URLs. This means that someone could post a disk image, then email someone a link to a text file within that disk image! This also means that search crawlers such as Googlebot or Bingbot could index the disk contents.
Several common Apple II file types are converted to meaningful formats in the web browser:
- Clicking on a lo-res or high-res screen will show the image as PNG
- Teach, AppleWorks word processor, ORCA, Merlin, and most other generic text files are displayed as ASCII
- machine language programs in 6502 or 65C02 are disassembled
- ShrinkIt files will show their catalogs (and later will be fully clickable too)
- Integer BASIC and Applesoft programs are detokenized in the LIST format
- unknown types, or optionally, any file at all, can also be shown in a combination hex/ASCII data dump
Any file can be downloaded in raw form to the computer. This means that you could pull a ShrinkIt file out of a disk image then use Shrink-FitX on Mac OS X to extract its contents.