ADFFS 2.86 public beta

[JonAbbott]

This update adds HFE support, missing sector creation, JFD/ADF auto-flush and the ability to write an unprotected floppy image back to a physical floppy.

I strongly advise taking a copy of any "Auto flush" floppy images before using them, just in case I've missed an issue and they end up corrupt.

Note that although JFD's imaged or Auto flushed by ADFFS 2.86 are technically backward compatible with ADFFS 2.85 and earlier; older versions of ADFFS will report the image requires ADFFS 2.86.

To improve the speed and reduce the memory required by ZLib, Save and Flush now compress using less compression on ARM2/3 machines, requiring around 32KB of RMA to work. RiscPC and newer will use max compression, which requires around 400KB of RMA.

Auto-flush
Auto-flush will write-back changes to a mounted JFD image file 3 seconds after the last write. To work, JFD must be imaged with 2.86 (or resaved via Save as), and Auto flush enabled when the floppy was imaged, as shown in the image below (or forced via *ADFFlush On):

Auto flush.png (31.89 KiB) Viewed 23049 times

JFD modified by previous 2.86 betas (pre 15th Feb 2025) should be deleted as they are likely corrupt and will probably crash ADFFS when loading/modifying. There were two issues which are now resolved: JFD not natively imaged with 2.86 were becoming corrupt on the 1st auto-flush when it auto-updated the file structure.

I also noticed an issue in the imager which was adding a rogue word after the sector table, which in turn caused a buffer overrun when a missing sector was added. Given every JFD to date has this problem, I've added a check to the JFD handler to correct it.

Older JFD are no longer auto-updated to 2.86 as they were with earlier builds. For auto-flush to work you will need to either reimage the JFD, or use Save As to resave it (I've not tested this to any extent). This is a temporary measure until I figure out the best way to auto-update JFD to 2.86 when they're written too for the first time.

Auto-flush works by appending all delta changes to the end of the JFD file. For example a blank JFD that's had 800KB written to it will compress and append 800KB at each flush. As Auto-flush JFD's are primarily for game saves, a flush will usually only consist of a few sectors worth of data.
Although its not yet implemented, it's possible to rollback a JFD to its original state.

If a flush fails due to lack of RMA or a disc/compression error, it will fail silently as it occurs in the background. I might adjust it to flash the screen border on older machines, where its more likely to be an issue.

Missing sector creation
This is a long standing item in the development list that I couldn't implement until delta-writes were implemented (part of auto-flush). When a JFD is created via a recording file, it images only the sectors that are touched by the game. ie files and protection sectors. That leaves gaps across the image which to date would trigger a "missing sector" error if read/written to. For read only floppy images the missing sector are not an issue, but games that use the floppy for settings/highscores/saves etc its obviously a problem.

ADFFS will now seamlessly add those missing sectors when they're accessed, flagging them as delta changes so they're auto-flushed back to the JFD if appropriate.

It has however highlighted a potential problem in that some protection might rely on the "missing sector" error. To ensure backward compatibility, JFD not updated to 2.86 will continue to generate the "missing sector" and will not add missing sectors.

Some games will need reimaging with 2.86 so the imager captures unformatted protection sectors. I don't have a full list yet, but a few I'm aware of are Minotaur, Formula Two Thousand and Pandora's Box. Essentially if the floppy is RW, flagged for auto-flush and relies on unformatted sectors it needs reimaging.

HFE support
As I do not have a Greaseweazle or any known good HFE to test with, I've been working off HFE I've converted from Kryoflux images, so consider this "early access".

HFEv1/v2 and v3 files are supported, although fuzzy bits will not work correctly on v2/v3 due to the way the HFE is parsed through a 1772 as it's loaded - not when accessed.

De-duped recording files
The recording files bundled with ADFFS were all as captured by myself and contained a lot of duplicated reads and non-merged consecutive reads. I've now de-duped and merged consecutive reads to reduce their size down from 874KB to 55KB! If the resulting recording file ended up imaging the whole floppy and it had no protection sectors, the recording file has now been removed - the imager will fall-back to full disc imaging.

I've tested reimaging a few games at random, but as there's 583 recordings I can't test them all. Let me know if you notice any problems.

I have also flagged obvious known protection sectors to avoid them showing as "bad". As I've done this programmatically, there's going to be some games that still show protection sectors as "bad", so if you see a track marked as bad when imaging, let me know and I'll check if it's protection or not and correct accordingly.

Write to ADFS::0
This will write the mounted image to a physical floppy. Disk protection will not be written, so its only useful for unprotected floppies.

Write to ADFS.png (17.65 KiB) Viewed 20991 times

Changes since 2.85

• Debug overlay is now destroyed when the Module exits instead of when the JIT shuts down, to avoid a hard-lock when a JIT Abort occurs
• Blitter was corrupting the stack if it exited early due to the overlay not existing (fixes Stunt Racer 2000)
• reduced the minimum overlay size from 32x32 to 4x4
• OS_ReadDynamicArea was returning an invalid size (fixes Rockfall)
• If GraphicsV fails to map an overlay, its address is now zero'd so the blitter exits early
• JIT UpCall 6 is no longer passed onto hypervised tasks
• JIT IRQ handler wasn't storing IRQ R13 on return from the OS, which may have been reset by RTSupport
• Updated JFD imager to create JFD's suitable for delta writes
• Flush wasn't exiting early if the floppy hadn't changed
• ADFOpen/Mount wasn't reporting ZLib errors
• OS_Byte 161 now returns with the boot flag set, if the currently mounted floppy is flagged as needing Shift-Break to run
• Updated ZLib to v0.05
• Delta writes now implemented
• Now displays a % Hourglass while saving / flushing
• Added "Auto flush" to the ADFFS Filer info page
• Save / Flush will now use "Best Speed" instead of "Best Compression" when saving on ARM2/3 machines
• Auto flush now performed when swapping or ejecting mounted images
• Removed restrictions on path/filename lengths by moving the mounted filename to the RMA
• ADFFS$RecordingPath is no longer supported when creating recordings
• Early versions of Window Manager may have Aborted when imaging floppies
• Updated JoyMod to 5.02 and modified it to fallback to Joystick_Read if RTFM isn't detected
• Added explicit "Sector not found" bit to sector table
• When flushing a JFD created prior to 2.86 for the first time, it was not clearing the delta change flag from the sector table correctly
• replaced all DIV routines
• Removed Debugger as a protected Module
• ADFCRC with no parameters will now display the last CRC value
• JFD imager was adding an unnecessary word to the end of the sector table
• Missing sector behaviour changed (on JFD's created with 2.86):
  - DiscOp 0 (Verify) will return zeroed sector(s) but not add them
  - DiscOp 1 (Read) will add and return zeroed sector(s)
  - DiscOp 2 (write) will add the missing sector(s)
• When adding a missing track, the extra word the imager was appending to the sector table (pre 9th Feb 2025 builds) caused the tracks sector list to become corrupt
• Filer now displays Flush and Write-protect status for all filetypes
• Added a sanity check to the 1772 emulator to ensure sector sizes are <= 8192 bytes (fixes Nosher HFE)
• Recording files: De-duped and merged consecutive reads in recording files to reduce their size
• Recording files: Marked reads that don't align with sectors as protection
• *ScreenSize was failing to reduce the screen memory on IOC
• Added HFEv1/v2/v3 support (with the caveat that fuzzy bits won't work correctly)
• Added an HFE version check and suitable error
• Recording files: Removed all full-disc recordings and corrected the discrec in the fall-back ADFS F recording
• JFD imager wasn't always showing known protection sectors
• JFD imager sector de-dupe wasn't preferring protection sectors
• Added generic ADFS L format to the JFD imager
• Filer now displays the JFD/HFE structure version
• Added "Write to ADFS::0"
• ADF are now auto-flushed
• Changed a lot of internal variables sizes from word to byte
• Added ADFFS_Action 3 to temporarily disable missing sector creation
• Refactored all SWI code to use relative addressing for variables where possible
• The RISC OS 5/6 compatibility icons were swapped in the Filer display
• When loading a file it wasn't clearing the header information, which could cause the Filer to misreport the OS/CPU compatibility
• Adjusted MEMC sound DMA to more closely match physical and it now buffers further ahead to avoid audio glitches (fixes audio in Zelanites)

Known issues

• Overlay builds trigger a spinlock in SDIODriver and BCMSupport on the Pi (ROOL post here)
• Champions: Jahangir Khan World Championship Squash (1991) (Krisalis Software) - Joystick doesn't work when Boot Floppy is used on an A3010. The same issue will apply to other games that use JoyMod, which are mostly Krisalis titles (hopefully fixed - I don't have an A3010 to test) (fix implemented, but not confirmed working)
• Zelanites has no sound on a RiscPC and the sound effects are poor quality on a Pi
• Games that set the video DMA directly throw a JIT Abort (bug introduced with 2.86o)

Resolved issues

• If auto-flush occurs on a JFD created pre 2.86, it will overwrite the original sectors on the first flush. Temporary workarounds are to either reimage the floppy, or save it via the Filer so it's update to a 2.86 compatible JFD before mounting it (fixed by preventing auto-flush on pre 2.86)
• Mounting a JFD created with a really early version of ADFFS and then saving it will result in a corrupt JFD (the imager was adding an extra word to the sector table, that was causing a buffer overrun)
• If a mounted JFD is written to and "Save as" then used to create a copy, the sector changes may not be merged correctly
• Loading a Utility under the JIT triggers an Abort (bug introduced with 2.86h)
• JIT the check for VProtect could cause a crash (bug introduced with 2.86m)
• Sound_Config will corrupt a random block of memory (bug introduced with 2.86m)

Script changes

• F1026101 Nevryon (1990) (The Fourth Dimension) - Switch to MODE 0 before running the game to work around an in-game issue
• F1034101 Spobbleoid (1994) (Cybernation Entertainment Systems) - Added HD install instructions
• F1034201 Spobbleoid Fantasy (1994) (Cybernation Entertainment Systems) - Added HD install instructions
• F1066501 WorldScape (1990) (Eclipse) - Now patches the game back to RO3
• F1036801 Time Machine, The (1994) (The Fourth Dimension) - Added HD install instructions
• F1027401 Pandora's Box (1991) (The Fourth Dimension) - Now patches the game back to RO3. Modified to run from HD, with saves going to Choices:Pandora
• F1035101 Stunt Racer 2000 (1993) (The Fourth Dimension) - Now patches the game back to RO3 so it will run from HD
• F1023201 Lotus Turbo Challenge 2 (1992) (Krisalis Software) - Modified to handle IRQ re-entrancy
• F1001801 Arcade 3 (1988) (Clares Micro Supplies) - Now patches the game back to RO3. Added HD install instructions
• F1049001 Champions: Jahangir Khan World Championship Squash (1991) (Krisalis Software) - correct Joystick detection
• F1021001 Jahangir Khan World Championship Squash (1991) (Krisalis Software) - correct Joystick detection
• F1020001 Holed Out!! Designer (1989) (The Fourth Dimension) - Added HD install instructions
• F1017201 Gods (1992) (Krisalis Software) - Adjusted clip ratio so the scrolling hints are displayed
• F1054501 Gods [RPC version] (2001) (R-Comp Interactive) - Adjusted clip ratio so the scrolling hints are displayed
• F1040301 Zelanites - The Onslaught (1991) (MicroPower) - Now patches the game back to RO3 so it exits to the desktop and saves the highscores after they're entered
• F1048701 Play It Again Sam 3 (1993) (Superior Software) - Now patches Zelanites back to RO3 so it exits to the desktop and saves the highscores after they're entered
• F1048701/02 Play It Again Sam 3 (1993) (Superior Software) - SFS is now made 32bit compatible and added HD install instructions
• F1035101 Stunt Racer 2000 (1993) (The Fourth Dimension) - Added delays to menus and no longer removes RAM: when it exits
• F1045401/03 Chocks Away Compendium [Mark 2 version] (1991) (The Fourth Dimension) - Patches to run from any filesystem and switched to use JoyMod
• F1059801 Lemmings 2: The Tribes [SA version] (2001) (R-Comp Interactive) - Load QTM before starting the JIT to avoid audio issues

NOTE: For auto-flush and missing sector creation to work, JFD's will need updating. This can be done by either reimaging the JFD or using Save As. Ensure you keep the original JFD as this is a beta and there might be further changes to the 2.86 JFD file layout.

[wmd]

Saving still not working on Zarkon Invasion or Mazey Man (Arcade 3) for me. I have zeroed the high score files using your BASIC script from the other thread. Three seconds after entering a new score in either game the hourglass appears for a few seconds then disappears. I would have expected the hourglass to appear longer as the JFD was made with ADFFS 2.85. When I reboot the machine and check the Modified counter, it has not changed. JFD is not write protected and has Auto Flush set to Y.

[JonAbbott]

Saving still not working on Zarkon Invasion or Mazey Man (Arcade 3) for me

Try reimaging the JFD, it should all be green.

EDIT: I've just checked and the JFD was updated with my highscore after a reboot

[wmd]

Tried re-imaging (from the 2.85 JFD) - there are significantly less red blocks, but there are still 3 reds once the imaging has completed.

[JonAbbott]

Tried re-imaging (from the 2.85 JFD) - there are significantly less red blocks, but there are still 3 reds once the imaging has completed.

I'd image from your original APD just to be sure, but if you're seeing red, the floppy image is corrupt.

EDIT: I've just noticed a missing sector for Zarkon Invasion, which is causing an abort. I'll get it fixed and let you know once I've updated the download in the OP

[JonAbbott]

Tried re-imaging (from the 2.85 JFD) - there are significantly less red blocks, but there are still 3 reds once the imaging has completed.

I'd image from your original APD just to be sure, but if you're seeing red, the floppy image is corrupt.

EDIT: I've just noticed a missing sector for Zarkon Invasion, which is causing an abort. I'll get it fixed and let you know once I've updated the download in the OP

I've updated the OP and double-checked Zarkon works and does auto-flush back to the floppy okay. Please try reimaging from your original APD, you should see only one red block, which is the bit-rot towards the end of the image.

[wmd]

When I image from my APD file I get 3 red blocks:

[JonAbbott]

When I image from my APD file I get 3 red blocks:

I don't think you've not updated from the OP as those first two aren't imaged with the latest !ADFFS.imaging.JFDrecord

[wmd]

Yes, you are right, I missed that part of your post(s). I'll test it now.

[wmd]

APD to JFD worked as it should this time, with just 1 red block. Ran the score clear script, and Eject was almost instant. Checked the score files in !Edit to make sure they had been cleared - they had. Modified counter went up to 2. However, after getting a high score (only tested in Mazey Man) the egg timer does not appear after 3 seconds. After waiting more than 3 seconds, did a reboot and score was not saved, Modified counter not incremented. JFD is not write protected and has Auto flush set to Y.