Use the libretro VFS interface in libretro builds

[white-axe]

• Closes libretro: make rom access use the libretro VFS api for using SAF files served by a document provider in android #21087

Allows, for example, loading games from the Storage Access Framework when using the libretro core in the RetroArch Android app.

I removed the duplicate header files in the libretro directory in this repository that are already present in libretro-common, by the way, because this pull request relies on libretro-common which in turn relies on libretro-common's versions of the duplicated header files. I see no reason to have two copies of the libretro-common header files anyways.

[i30817]

That looks more complicated than I was expecting.

Don't mind me just posting to get subscribed to notifications in the mobile GitHub interface

[white-axe]

It's because the original code depends on library functions that themselves use stdio functions in a few places. I rewrote them to be more agnostic of what file system API is being used.

• ZipFileReader and the zip-related functions in Core/Loaders.cpp are changed to use libzip's functions that take IO callback functions instead of the equivalent ones that operate directly on stdio.
• Ditto for the PNG image writer in pngSave().
• The gzip file writer in SymbolMap.cpp now manually calls the low-level deflate() function and manually writes the deflated data bit-by-bit to the output file instead of using gzopen()/gzdopen().

The file_stream_transforms.h header file that redirects the remaining stdio calls made from directly inside of the PPSSPP codebase also doesn't like it when you write std::FILE, std::fclose, etc. so I removed the std:: from the beginning.

Also, I've consolidated the various platform-dependent methods of seeking files into File::Fseek(), File::Fseektell() and File::Ftell() helper functions so that I could add a method for seeking files on libretro only without having to insert ifdefs everywhere.

[hrydgard]

Pretty impressive! Yeah, this is probably really the only way to go about it.

I kicked off the CI. Will give it some light testing later before merge.

[hrydgard]

As you can see in the CI reports, there are some build issues for non-libretro builds.

[hrydgard]

Triggered a new build. Since this is your first PR, CI needs manual triggering. I'll try to keep up reasonably.

[hrydgard]

Thanks for the fixes! Will do a final review tomorrow, I think we'll be able to get this in shortly.

[hrydgard]

Running on Android, there seems to be some pointer truncation problem in the new libpng code:

01-05 15:14:55.188 13705 13705 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
01-05 15:14:55.188 13705 13705 F DEBUG   : Abort message: 'Pointer tag for 0x72a8d2d2c8 was truncated, see 'https://source.android.com/devices/tech/debug/tagged-pointers'.'
01-05 15:14:55.188 13705 13705 F DEBUG   :     x0  0000000000000000  x1  0000000000003573  x2  0000000000000006  x3  0000007288543170
01-05 15:14:55.188 13705 13705 F DEBUG   :     x4  7260761f37623163  x5  7260761f37623163  x6  7260761f37623163  x7  7f7f7f7f7f7f7f7f
01-05 15:14:55.188 13705 13705 F DEBUG   :     x8  00000000000000f0  x9  0000007625ced498  x10 ffffff80fffffbdf  x11 0000000000000000
01-05 15:14:55.188 13705 13705 F DEBUG   :     x12 00000000012e4022  x13 0000000276aefd5c  x14 00331a828343861a  x15 0000000026762762
01-05 15:14:55.188 13705 13705 F DEBUG   :     x16 0000007625d99fe0  x17 0000007625d85fc0  x18 0000007284ae0000  x19 0000000000003535
01-05 15:14:55.188 13705 13705 F DEBUG   :     x20 0000000000003573  x21 00000000ffffffff  x22 0000000000003535  x23 0000000000003535
01-05 15:14:55.188 13705 13705 F DEBUG   :     x24 00000072885445f0  x25 00000072885445f0  x26 0000007288544930  x27 0000007650fa8940
01-05 15:14:55.188 13705 13705 F DEBUG   :     x28 000000728844c000  x29 00000072885431f0
01-05 15:14:55.188 13705 13705 F DEBUG   :     lr  0000007625d24e34  sp  0000007288543150  pc  0000007625d24e64  pst 0000000000001000
01-05 15:14:55.188 13705 13705 F DEBUG   : 25 total frames
01-05 15:14:55.188 13705 13705 F DEBUG   : backtrace:
01-05 15:14:55.188 13705 13705 F DEBUG   :       #00 pc 000000000005ae64  /apex/com.android.runtime/lib64/bionic/libc.so (abort+168)
01-05 15:14:55.188 13705 13705 F DEBUG   :       #01 pc 0000000000045430  /apex/com.android.runtime/lib64/bionic/libc.so (free+104)
01-05 15:14:55.188 13705 13705 F DEBUG   :       #02 pc 0000000001901cb4  /arm64/libppsspp_jni.so (pngLoadPtr(unsigned char const*, unsigned long, int*, int*, unsigned char**)+304)
01-05 15:14:55.188 13705 13705 F DEBUG   :       #03 pc 0000000001d622b0  /arm64/libppsspp_jni.so (Image::LoadPNG(char const*)+124)
01-05 15:14:55.188 13705 13705 F DEBUG   :       #04 pc 0000000001d4eaf4  /arm64/libppsspp_jni.so
01-05 15:14:55.188 13705 13705 F DEBUG   :       #05 pc 0000000001d4d8d4  /arm64/libppsspp_jni.so (GenerateUIAtlas(Draw::DrawContext*, Atlas*, float, bool)+300)
01-05 15:14:55.188 13705 13705 F DEBUG   :       #06 pc 0000000001d4f3e4  /arm64/libppsspp_jni.so (AtlasProvider(Draw::DrawContext*, AtlasChoice, float, bool)+144)
01-05 15:14:55.188 13705 13705 F DEBUG   :       #07 pc 0000000001a6803c  /arm64/libppsspp_jni.so
01-05 15:14:55.189 13705 13705 F DEBUG   :       #08 pc 0000000001a67fd0  /arm64/libppsspp_jni.so (AtlasData std::__ndk1::__invoke_void_return_wrapper<AtlasData, false>::__call[abi:ne210000]<AtlasData (*&)(Draw::DrawContext*, AtlasChoice, float, bool), Draw::DrawContext*, AtlasChoice, float, bool>(AtlasData (*&)(Draw::DrawContext*, AtlasChoice, float, bool), Draw::DrawContext*&&, AtlasChoice&&, float&&, bool&&)+52)
01-0

[hrydgard]

Actually, that's not the root cause. It just fails to load all pngs from assets on Android. Assets are read from the APK as a zip file, and something is wrong with that.

pngLoadPtr does need *image_data_ptr = nullptr; at the start to avoid this misleading error though.

[hrydgard]

Yeah, the zipfilereader is not functioning correctly, it reads garbled data. Not sure why.

[hrydgard]

Same happens on standalone in Windows - the zipfilereader is busted.

On Windows, we don't use the zipfilereader for assets, so the app starts. But if you try to install something from the homebrew store, it blows up.

It's clear you didn't test this.

[white-axe]

Should be fixed for installing games from the homebrew store, but the Android app is still broken. I'm working on fixing that now.

[white-axe]

Okay, the non-libretro Android builds should work again.

[hrydgard]

Hm. PPSSPP never really has any need to access 4GB+ files (as PSP ISOs are a lot smaller than that, even the largest "combined" ISOs like the Final Fantasy Type-0 translation).

Combining fseek and lseek like that looks super scary, and I'm not sure if we shouldn't just do a plain old fseek there instead.

[white-axe]

Alright, I've changed that code path to use normal fseek and ftell.

[hrydgard]

Did a quick check on Windows and Android again, things seem fine, and the code looks alright.

Let's get it in.

[i30817]

Yo. I'm loving the feature but I think there might be a specific bug report in the future you might want to either ignore or fix now.

Hm. PPSSPP never really has any need to access 4GB+ files (as PSP ISOs are a lot smaller than that, even the largest "combined" ISOs like the Final Fantasy Type-0 translation).

Combining fseek and lseek like that looks super scary, and I'm not sure if we shouldn't just do a plain old fseek there instead.

From what I read fseek only actually supports 2^31 (2147483648) not 2^32 (4294967296) because of the sign bit of normal long.

The FF type zero hack\translation is 3158917120 bytes

I think you can see the problem. I think this game in particular is the exact reason this code was done weird like this.

Now I'm not saying you should support a weird hack that should have limited itself to not merge CDs (does it even RUN in a PSP?) but I think you should be aware if you're breaking it on purpose.

Now the internet may be completely wrong and/or modern fseek actually has the full range unsigned long so this is something you should verify (somehow).

[hrydgard]

As far as I know, FF: Type-0 (including the hack) runs fine in PPSSPP standalone as-is. Let me know if that's not the case.

[i30817]

No no that was a auto corrupt typo. I was asking if the translation hack works on a psp with however you load homebrew there because it's larger than 2^31.

I suspect that right now in PPSSPP it will break sometime when cd 2 files are addressed in the single merged disc but I don't have time to play it until cd 2 areas.

I did try to load it with the recent code (on WebDAV in RETROARCH) so I know it loads and goes to the tutorial in game with this code, just worried about the later areas.

[i30817]

Generally if this wasn't the only translation of FF type 0 that isn't a official one on the PS4 or Xbox one I wouldnt even bother saying anything because supporting this kind of feels like a hack if the original psp wouldn't be able to address a 3gb game in a 3gb UMD if one existed.

If only the translator had the good sense of releasing a version in two discs too, I wouldn't say anything either.

[i30817]

There are a bunch of saves in this page: https://gamefaqs.gamespot.com/psp/951986-final-fantasy-type-0/saves

I'll try to load one of the last chapter ones in retroarch.

[i30817]

Ok I tested out the last chapter start save until the end credits and some of the videos in the "history" (log) of the game.

No problems

Im in android and the game is being served from Apache WebDAV in a 32 bits Linux computer so I think that until further notice of someone opening a real bug in type 0 (or another weird large game) it's fine.