[Draft] Full filestore refactor w/ SQLite3 #35

Katilith · 2022-08-22T23:37:36Z

work in progress

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>

Bump minimist from 1.2.5 to 1.2.6

…ons of the same variables, name, and nameHash within the index

…und there

…n local files will be used

…ile store support. Also removed a lot of redundant old code, added in proper todos for stubbed methods, added interface file names to the JS5 archive config, and made everything just a bit cleaner.

…actoring all of the indexing code to support the two differing formats Also adding in all of the cache file names+hashes for Jag format caches

… + hunt.idx

…elds

…ntity reading from the database.

Also adds the skeleton for JS5 game item file decoding

…JS5 builds

… to speed up index load times and only pull/store blobs on-demand

…ig route to the js5 http server

… JAG indexing bugs

Jameskmonger · 2022-09-05T23:25:44Z

src/file-system/jag/content/archives/interfaces/jag-interface-archive.ts

+ return inter;
+ }
+
+ async decodeAll(): Promise<void> {


as this will be a bit of a pattern across all the different archive types, I wonder if these two decodeAll functions might be better as static factory functions

i.e. rather than

async decodeAll(): Promise<void> {

You could have

static async decodeAll(jagStore: JagFileStore): Promise<JagInterfaceArchive>

Jameskmonger · 2022-09-05T23:29:04Z

src/file-system/jag/content/archives/interfaces/jag-interface-archive.ts

+ inter.unknownServerAttribute2 = data.get('byte', 'unsigned') === 1;
+ }
+
+ if (type === 2) {


It may be worth pulling these type values into a set of constants rather than magic numbers

Jameskmonger · 2022-09-05T23:32:29Z

src/file-system/jag/content/archives/interfaces/jag-interface-archive.ts

+ // hoveredPopup = u_short, but only a single u_byte is written if there is no hovered popup
+ // use u_smart_short ?
+ inter.hoveredPopup = data.get('byte', 'unsigned');
+ if (inter.hoveredPopup !== 0) {


where are you getting this from?

I don't see it in the 435 client, which seems to read 2 bytes regardless

https://github.com/runejs/refactored-client-435/blob/28e607739b41bb4d1ea0894aa52259fa0fedd5fb/src/main/java/org/runejs/client/cache/media/gameInterface/GameInterface.java#L1535

Jameskmonger · 2022-09-05T23:36:24Z

src/file-system/js5/content/config/js5-game-items.ts

+ } else if (o === 6) {
+ item.modelRotationY = data.get('short', 'u');
+ } else if (o === 7) {
+ // client subtracts 65536 if the value is over 32767


Yes, I think so

I suspect that they have custom obfuscation to switch around signed/unsigned method calls in order to do the signing with the result

Jameskmonger · 2022-09-05T23:37:21Z

src/file-system/js5/content/config/js5-game-items.ts

+ } else if (o === 26) {
+ item.femaleModelId2 = data.get('short', 'u');
+ } else if (o >= 30 && o < 35) {
+ // cases 27-29 missing - what were they (if anything)?


they were missing in #194 so I suspect we'll never know

https://github.com/thedaneeffect/RuneScape-Beta-Public/blob/master/src/main/java/rs/data/ObjType.java#L172

Jameskmonger · 2022-09-05T23:38:27Z

src/file-system/js5/content/config/js5-game-items.ts

+ // so this should be a SIGNED short then, right?...
+ item.modelOffsetY = data.get('short', 'u');
+ } else if (o === 10) {
+ // case 9 missing - what was it (if anything)?


oddly enough 9 is missing in #194 too but was in the code and empty (unlike some of the others here such as 17)

https://github.com/thedaneeffect/RuneScape-Beta-Public/blob/master/src/main/java/rs/data/ObjType.java#L153

Jameskmonger · 2022-09-05T23:43:01Z

src/file-system/js5/content/config/js5-game-items.ts

+ } else if (o === 91) {
+ item.femaleDialogueModelId1 = data.get('short', 'u');
+ } else if (o === 92) {
+ item.maleDialogueModelId2 = data.get('short', 'u');


I have the dialogue model IDs down in 317 as:

90: maleDialogueModelId

91: femaleDialogueModelId

92: maleDialogueHatModelId

93: femaleDialogueHatModelId

This is from a few years back so I'm not 100% sure on them now

I have just spent a little while investigating these and they both seem to be used to construct the dialogue model for that item.

Generally 90 and 91 represent the main model while 92 and 93 are additions

Quite often, 90/91 will be a basic human head model and 92/93 will be the head item itself

Sometimes the human head model is bundled into the 90 and 91 models, for example the 'dragon med helm' item:

92: model id 57 (human head with helmet on)

94: model id 71 (horns from d med helm)

whereas other times they are separate for example the 'leather cowl' item:

90: model id 32 (cowl only)

92: model is 63 (human head)

Jameskmonger · 2022-09-06T00:13:38Z

src/file-system/js5/js5-group.ts

+ async getFile(fileKey: number): Promise<Js5File | null>;
+ async getFile(fileName: string): Promise<Js5File | null>;
+ async getFile(fileIdentifier: number | string): Promise<Js5File | null>;
+ async getFile(fileIdentifier: number | string): Promise<Js5File | null> {


A lot of this code seems very similar to the getArchive code in the Js5FileStore, possibly you can abstract this away

Jameskmonger · 2022-09-06T00:14:18Z

src/file-system/js5/js5.ts

+ const archiveDetails = archive.index;
+
+ if (archiveDetails.key === 255) {
+ return;


does this need a louder error? (I'm not sure what key of 255 means)

Jameskmonger · 2022-09-06T00:15:17Z

src/scripts/name-hasher.ts

+
+ const INT_MAX = 2147483648;
+
+ // Emulate Java's INT overflow-wrapping


I made an iadd function, but it was mostly syntactic sugar

You can just do (a + b) | 0 to emulate the int32 behaviour from Java

Katilith and others added 30 commits March 14, 2022 14:17

Adding support for archive formats 6 and 7

e598fe1

Fixing bugs with the new formats

83d2106

Adding support for archives 16-28 (until game build 582)

cc2f61e

Adding archive 29 support and fixing the readme

2e0721a

Adding support for configurable config group names

fde2f98

Merge pull request #1 from runejs/dependabot/npm_and_yarn/minimist-1.2.6

2562060

Bump minimist from 1.2.5 to 1.2.6

Working to implement a simpler file-system/store approach

0e58044

Implementing js5 main index encoding and changing some variable names

2520636

Removing old filestore-system files and reorganizing the new package

df2a688

Changing the packed/ dir to js5/ and the indexes/ dir to index/

108a798

Getting the basic js5 indexing system working and writing to SQLite DB

9fbe8eb

Implementing filestore index loading and fixing some JS5 decode issues

abf8198

Adding validation for checksum, shaDigest, fileSize, compressed versi…

8cebf95

…ons of the same variables, name, and nameHash within the index

Fixing childCount database column name

95e7e9e

Working on adding support for Jag Cache stores (builds 234-399)

9818569

Adding the ability to pull data/caches from OpenRS2.org if they're fo…

4b13554

…und there

XTEA keys will now be fetched from OpenRS2.org, if none are found the…

12e244b

…n local files will be used

Porting over JS5 archive and group encoding in preparation for flat f…

ddd98bf

…ile store support. Also removed a lot of redundant old code, added in proper todos for stubbed methods, added interface file names to the JS5 archive config, and made everything just a bit cleaner.

Working on JAG format cache decoding and indexing - about 50% completed

f70e4ad

Moving Jag format index entities to their own table structure and ref…

74969f2

…actoring all of the indexing code to support the two differing formats Also adding in all of the cache file names+hashes for Jag format caches

Adding missing file name/namehashes for caches 274 and 289 - hunt.dat…

ba0bccc

… + hunt.idx

Finishing up JAG cache indexing by adding the remaining missing DB fi…

d073b3d

…elds

Prototyping out some parsers & decoders for JAG format files

2a1943e

Stubbing out some JAG game interface decoder code

4c5728a

Prototyping JAG interface decoding. Also adds support for JAG index e…

c1037f7

…ntity reading from the database.

Implementing JAG interface decoding and saving within JAG databases

e1b775f

Reimplementing Js5 http server support for the webclient project

3a33c22

Updating to the latest /common 3.0.0 beta version and applying fixes

072eeb9

Also adds the skeleton for JS5 game item file decoding

Improving the HTTP server using decorators with full support for all …

c8a75e7

…JS5 builds

Katilith added 3 commits August 29, 2022 13:00

Moving JS5 uncompressed and compressed file data into separate tables…

907d7ec

… to speed up index load times and only pull/store blobs on-demand

Merging uncompressed and compressed data tables + adding archive-conf…

1e0b45e

…ig route to the js5 http server

Moving JAG file data out to another table like JS5 and fixing tons of…

d63f842

… JAG indexing bugs

Jameskmonger reviewed Sep 6, 2022

View reviewed changes

Jameskmonger mentioned this pull request Jan 16, 2023

chore: use @runejs/store instead of filestore to remove canvas dependency runejs/server#389

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Full filestore refactor w/ SQLite3 #35

[Draft] Full filestore refactor w/ SQLite3 #35

Katilith commented Aug 22, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 5, 2022

Jameskmonger Sep 6, 2022

Jameskmonger Sep 6, 2022

Jameskmonger Sep 6, 2022

Jameskmonger Sep 6, 2022


		const INT_MAX = 2147483648;

		// Emulate Java's INT overflow-wrapping

[Draft] Full filestore refactor w/ SQLite3 #35

Are you sure you want to change the base?

[Draft] Full filestore refactor w/ SQLite3 #35

Conversation

Katilith commented Aug 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment