TrailMyx's Advanced File System

[TrailMyx]

Code: [Select]

;============================================================
; Script Name: TrailMyx's Advanced File System Handling Subs
; Author: TrailMyx
; Version: 1.3
; Shard OSI / FS: Any
; Revision Date: 10/20/2007
; Purpose:
;     Simplify the saving of variables to a file.  Also to speed up the process
;   by minimizing the number times calls to the shell are issued.  A maximum of
;   2000 bytes are assigned to each shell call, with an unlimited amount of calls
;   possible.  It's easy to store 1000's of variables in a relatively short period
;   of time.
;
;   This set of functions will also handle any namespace (local, global, immediate)!
;   Also, this file is ready to be 'called'.
;
; Limitations:
;     You cannot store spaces to a file due to the way DOS handles the command line.
;   So if a space is found, it will be converted to an underscore '_'
;
; Functions:
;   TM_FileSystem_CreateFileHandle
;      -- ex. gosub TM_FileSystem_CreateFileHandle handle1
;   TM_FileSystem_SaveVariable
;      -- ex. gosub TM_FileSystem_SaveVariable local std handle1 testval ; note not to include the '%' or '!' for variable
;   TM_FileSystem_SaveFile
;      -- ex. gosub TM_FileSystem_SaveFile handle1 c:\bigtest.txt
;   TM_FileSystem_LoadFile
;      -- ex. gosub TM_FileSystem_LoadFile c:\bigtest.txt
;   TM_FileSystem_SaveArray
;      -- ex. gosub TM_FileSystem_SaveArray local ns1 handle ns1_test 10 20
;
; Support functions:
;   TM_ReadVariables
;   AddUnderscore
;   AddSpace
;
;============================================================
; Revision History:
;   v1.0 - Initial release.
;   v1.1 - 5/16/2007 - Added support for array saving TM_SaveArray.
;   v1.2 - Much faster load times.
;   v1.3 - Much faster save preparation, fixed a bug in load pointer, added call handler
;=================================================================



Introduction:

I wanted to be able to save a very large number of variables quickly.  I wrote these subs to be able to handle 1000s of variables, keeping the saving time down to a very reasonable amount.  In fact you can save nearly 100 variables with a single DOS cmd.exe execution!  I use a similar file format in my rail engine, but this is meant to be generic.

You may also store information about different namespaces within the same output text file.

New to version 1.1:

• Array Support:  Now you can handle arrays of any size using TM_FileSystem_SaveArray.
  

Theory:

Basically, you are building a very long string of variables that will be saved.  In order to write a bunch of variables to a file you do the following:

1) Grab a file handle using TM_FileSystem_CreateFileHandle
2) Add variables to this handle using TM_FileSystem_SaveVariable
3) Add as many variables as you want to this file handle, note these have not been saved to the file yet...
4) Once you have added all the variables you want to save to the file handle, you must use TM_FileSystem_SaveFile to actually write the file string containing all your variables to disk.  During the saving process, the very long string containing all your variables will be cut up into strings of less than 2000 bytes.  These sub-strings will be saved to specified file.

Loading this file using TM_FileSystem_LoadFile will re-assemble the sub strings contained in the save file into your large string containing all your saved variables.  Then this string will be parced and the appropriate values will be written to the specified variables.


Here are some examples:

Code: [Select]

set #LPC 10000
for %i 0 1000
  set %test . %i %i * 100
 
gosub TM_FileSystem_CreateFileHandle handle1 ; clear handle named 'handle1'
for %i 0 1000
{
  gosub TM_FileSystem_SaveVariable local std handle1 test , %i ; store %test0, %test1, %test2, etc.
}

gosub TM_FileSystem_SaveFile handle1 c:\bigtest.txt ; save the file
stop

This test generates 1000 variables ranging from %test0 to %test1000 and assigns values from 0 100000.  The corresponding file looks like this:

Code: [Select]

set !fileout0 localstdtest00localstdtest1100localstdtest2200localstdtest3300localstdtest4400localstdtest5500localstdtest6600localstdtest7700localstdtest8800localstdtest9900localstdtest101000localstdtest111100localstdtest121200localstdtest131300localstdtest141400localstdtest151500localstdtest161600localstdtest171700localstdtest181800localstdtest191900localstdtest202000localstdtest212100localstdtest222200localstdtest232300localstdtest242400localstdtest252500localstdtest262600localstdtest272700localstdtest282800localstdtest292900localstdtest303000localstdtest313100localstdtest323200localstdtest333300localstdtest343400localstdtest353500localstdtest363600localstdtest373700localstdtest383800localstdtest393900localstdtest404000localstdtest414100localstdtest424200localstdtest434300localstdtest444400localstdtest454500localstdtest464600localstdtest474700localstdtest484800localstdtest494900localstdtest505000localstdtest515100localstdtest525200localstdtest535300localstdtest545400localstdtest555500localstdtest565600localstdtest575700localstdtest585800localstdtest595900localstdtest606000localstdtest616100localstdtest626200localstdtest636300localstdtest646400localstdtest656500localstdtest666600localstdtest676700localstdtest686800localstdtest696900localstdtest707000localstdtest717100localstdtest727200localstdtest737300localstdtest747400localstdtest757500localstdtest767600localstdtest777700localstdtest787800localstdtest797900localstdtest808000localstdtest818100localstdtest828200localstdtest838300localstdtest848400localstdtest858500localstdtest868600localstdtest878700localstdtest888800localstdtest898900localstdtest909000localstdtest91910
set !fileout1 0localstdtest929200localstdtest939300localstdtest949400localstdtest959500localstdtest969600localstdtest979700localstdtest989800localstdtest999900localstdtest10010000localstdtest10110100localstdtest10210200localstdtest10310300localstdtest10410400localstdtest10510500localstdtest10610600localstdtest10710700localstdtest10810800localstdtest10910900localstdtest11011000localstdtest11111100localstdtest11211200localstdtest11311300localstdtest11411400localstdtest11511500localstdtest11611600localstdtest11711700localstdtest11811800localstdtest11911900localstdtest12012000localstdtest12112100localstdtest12212200localstdtest12312300localstdtest12412400localstdtest12512500localstdtest12612600localstdtest12712700localstdtest12812800localstdtest12912900localstdtest13013000localstdtest13113100localstdtest13213200localstdtest13313300localstdtest13413400localstdtest13513500localstdtest13613600localstdtest13713700localstdtest13813800localstdtest13913900localstdtest14014000localstdtest14114100localstdtest14214200localstdtest14314300localstdtest14414400localstdtest14514500localstdtest14614600localstdtest14714700localstdtest14814800localstdtest14914900localstdtest15015000localstdtest15115100localstdtest15215200localstdtest15315300localstdtest15415400localstdtest15515500localstdtest15615600localstdtest15715700localstdtest15815800localstdtest15915900localstdtest16016000localstdtest16116100localstdtest16216200localstdtest16316300localstdtest16416400localstdtest16516500localstdtest16616600localstdtest16716700localstdtest16816800localstdtest16916900localstdtest17017000localstdtest17117100localstdtest17217200localstdtest17317300localstdtest17417400localstdtest1751750
set !fileout2 << there's more here that I deleted (it whas 27Kbytes) >>
.
.
.

Finally, if you want to load this file, just do:

Code: [Select]

gosub TM_FileSystem_LoadFile c:\bigtest.txt

Here's an example that stores information to 2 different namespaces:

Code: [Select]

gosub TM_FileSystem_CreateFileHandle handle1
set %testval 12345
gosub TM_FileSystem_SaveVariable local std handle1 testval  ; don't add the '%' or '!'
namespace push
namespace local NS1
set !ns1val 56789
gosub TM_FileSystem_SaveVariable local NS1 handle1 ns1val   ; don't add the '%' or '!'
gosub TM_FileSystem_SaveFile handle1 c:\filesystem_test.txt

If you load the file, the data will automatically be written to the approperate namespaces

Code: [Select]

gosub TM_FileSystem_LoadFile c:\filesystem_test.txt

so you'll find:
%testval = 12345 in local:std
!ns1val = 56789 in local:NS1


Note in order to start a clean file, you need to call TM_FileSystem_CreateFileHandle to make sure the TM_FileSystem namespace variable doesn't have old data.

Note:

Remember that you can't store spaces using this filesystem.  It automatically convertes spaces to '_'(underscores) so beware.

Scripts known to use this filesystem:

• WHQuester beta2.00 - Save/Restore user preferences, artifact list, etc.
• TrailMyx's Complete Looting Assistant (CLAw)  - Save/Restore user setup.
• TrailMyx's Recall Beetle Logging Machine

If you use this filesystem, please PM me so I can add your script to the above list!  Thanks, TrailMyx

[OMGBurgers]

I'm having trouble getting this to work ;*(

Whenever I run the example code of:

Code: [Select]

gosub TM_FileSystem_CreateFileHandle NULL handle1
set %testval 12345
gosub TM_FileSystem_SaveVariable NULL local std handle1 testval  ; don't add the '%' or '!'
namespace push
namespace local NS1
set !ns1val 56789
gosub TM_FileSystem_SaveVariable NULL local NS1 handle1 ns1val   ; don't add the '%' or '!'
gosub TM_FileSystem_SaveFile NULL handle1 c:\filesystem_test.txt

my file named "handle1" only contains:

Code: [Select]

set !fileout0 

Am I doing something wrong?  I'm so confused lol.

[OMGBurgers]

Edit:  I opened your beetlejacker as I remember using it and it had save/load ability and found the subs and compared them.  Seems they are different.  I replaced the ones in this thread with the ones from the beetlejacker and it's working 100% now.  I'm guessing this needs to be updated?  Anyhow just wanted to let you know hope you don't mind   You can delete these posts once your read to keep this thread uncluttered.

[TrailMyx]

Hmm, I thought I had updated these.  Oh well, I'll do that.  I remember change them, but I can't remember what I had actually changed.  Thanks for pointing this out.  Don't worry about clutter.  Even my house is a mess!

[TrailMyx]

Oh, actually I know why the test code doesn't work.  The 1.3 version of the file system takes advantage of the call/gosub option.  So you might try that sample code again, but remove the "NULL" references.  It seems I'm using the older version in my code still.  lol!  damn cut/paste.

[OMGBurgers]

Ohhh sweet!

I'll switch it around once I get this current sub hammered out and working properly.  These file saving routines are a life saver.  I'm coding a crafting training engine cause I hate all the ones out there but wanted one to be universal.  So it needs the ability to load/save multiple data tables into easyuo so it can know how to train tailoring, and how to train blacksmithing, but all using the same script!  Hopefully I can post up something for players to mess around with within a few days.  I'd like it to train my tailoring to 120 while I sleep tonight

[Khameleon]

nice I can't wait to see some more of your work

[TrailMyx]

I'll switch it around once I get this current sub hammered out and working properly.  These file saving routines are a life saver.  I'm coding a crafting training engine cause I hate all the ones out there but wanted one to be universal.  So it needs the ability to load/save multiple data tables into easyuo so it can know how to train tailoring, and how to train blacksmithing, but all using the same script!  Hopefully I can post up something for players to mess around with within a few days.  I'd like it to train my tailoring to 120 while I sleep tonight

That sounds great!  Definitely look forward to something like that.  These days, the scripts that are out there are just sooooo old.  Mmm, crafting!

Yeh, these file save/load subs really turned into one of the staples I rely on constantly.  They are really fast and since they also work with namespaces, you can completely memorize the operational state of a script, including arrays and namespaces.  Best thing is they don't pollute your registry and you can take the saved files and transfer them to other users.  I have a couple scripts where I've pre-processed a whole bunch of data that I just import when the script starts.  I guess the best thing is file save speed.  Sure it takes a bit of time when the save file gets large, but think about how long that'd take if you saved each variable at a time.  I remember those days.  Yuck!

[Khameleon]

ya, I remember a long while back I don't remember who wrote the file, but I used to run a batch file called SaveVarsToFile.bat man that was the only way to save a txt file back then. good thing we have evolved.

[OMGBurgers]

They are really fast and since they also work with namespaces, you can completely memorize the operational state of a script, including arrays and namespaces.

At first it was working like really good, the load was almost instant but it took 8 saves to save my data which confused me.  Then later I was looking at my code and I kept making it save everytime during the for statement.  It resulted in the same end file, but took longer.  Once I realized my error by mistake (Expanding more options to the script) it saved in one prompt like I expected it to.  I knew I was doing something wrong just never felt like finding it yet since I just had one table I made and it was solid.  But then I started editing it and all haha.

Best thing is they don't pollute your registry and you can take the saved files and transfer them to other users.  I have a couple scripts where I've pre-processed a whole bunch of data that I just import when the script starts.  I guess the best thing is file save speed.  Sure it takes a bit of time when the save file gets large, but think about how long that'd take if you saved each variable at a time.  I remember those days.  Yuck!

Ain't that the truth. I used to save everything to the registery and have to write a sub to handle it every time... talk about annoying!  Now I'll make my saves universal and the best part is, all my variables all are updated instantly I don't have to handle it .  And things like rails or in this case tables a lot of people will use I don't have to bother hard coding all my defaults, I can just attach them.  I hated editing bulk default data like rails that I had hard coded :/  Never again!

I sooooo didn't want to write this script because of the save/load, but really wanted to finish blacksmith/tailoring so I can soulstone it when I'm done burning what I have.  Then this morning I remembered you had save/load sniplets here and they were perfect lol.  Gota give you a major thanks on these!

[TrailMyx]

Hey, I'll take major thanks! 

That's why I have so many scripting support routines out.  If I can take one of those road blocks away, it helps promote people to start doing cool things without having to slog through the mundane basics like file saving or list handling.

I can't wait to see what you're working on!

[TrailMyx]

BTW, the rail engine pretty much uses the same method for compressing/saving.  I guess I could just use this filesystem directly, but I had written the rail engine before I made it "generic" and useful for any data storage requirements.  I might get around to making the rail engine generic someday.

You have access to the Unicorn tamer.  In that one, I have examples of the array saving/loading.

Edit:  never mind.  I forget what I was using the arrays in.  Wasn't the tamer.  Thank goodness I can search it quickly with SUO. 

[OMGBurgers]

Haha yes SUO made it so helpful so far jumping around between all my subs!  It's like I don't even need these 5 line blocks of ####'s I used so I could visually see the start of a new sub from using easyuo lol.  God that literally gave me a headache scrolling/paging up and down while editing scripts!

The funny thing is I was looking at the "old" version of easyuo that you gota use with the menu designer they have posted and I can't believe that it actually HAS options and their "new" version dont.  I thought that was pretty embarassing haha.

[Khameleon]

lol, look at that.. I was MIA during that development stage.

[VicVega]

I think this can be very useful in order to change scripts. You can save variables like time and skill gaining in the files in order to know if your code changing has improve the timmings or hasn't.

 I will try it later on. Thanks.