fixed rare memory leak in parser; updated changelog and readmes for v…

…ersion 0.8.0

README.md

@@ -1,6 +1,6 @@
 # **slippc** - A Slippi replay (.slp) parser, compressor, JSON converter, and basic analysis program written in C++
 
-Supports replays between versions 0.1.0 and 3.12.0. Last updated 2022-02-18. [View Change Log](./changelog.md)
+Supports replays between versions 0.1.0 and 3.12.0. Last updated 2022-02-19. [View Change Log](./changelog.md)
 
 ## Requirements
   * _make_ and _g++_, for building _slippc_
@@ -15,8 +15,8 @@ Supports replays between versions 0.1.0 and 3.12.0. Last updated 2022-02-18. [Vi
     -f        When used with -j <jsonfile>, write full frame info (instead of just frame deltas)
     -x        Compress or decompress a replay
     -X        Set output file name for compression
-    -d           Run at debug level <debuglevel> (show debug output)
-    -h           Show this help message
+    -d        Run at debug level <debuglevel> (show debug output)
+    -h        Show this help message
 ```
 
 ## Basic Overview
@@ -44,11 +44,11 @@ Passing the -x option to _slippc_ will compress an input .slp file specified wit
 
 _slippc_ validates all compressed files by decompressing them in memory and verifying the decoded file matches the original file. If for whatever reason this decode fails, no .zlp file will be created. As an additional failsafe, _slippc_ will never delete any original files, and will refuse to overwrite existing files if there is a filename conflict.
 
-Compression currently works for all replays between version 0.1.0 and 3.12.0, thought it cannot and will not compress corrupt replay files. Typical compression rates range from 93-97% for most normal replays. Compressed .zlp files may be loaded through _slippc_ for parsed JSON and analysis JSON output.
+Compression should work for all replays between version 0.1.0 and 3.12.0, thought it cannot and will not compress corrupt replay files (if you have a non-corrupt replay that won't compress, please create an issue with the replay attached). Typical compression rates range from 93-97% for most normal replays. Compressed .zlp files may be loaded through _slippc_ for parsed JSON and analysis JSON output.
 
 ## JSON Output
 
-Passing the -j option to _slippc_ will output the .slp file specified with -i as a .json file, which may be opened in any text editor and inspected directly, or further parsed and analyzed using any JSON parser. Most data is presented in integer or float format, as stored in the .slp file. Major additions include the "game\_start\_raw" field, which is a base64 encoding of Melee's internal structure for initializing a new game, and the "parser\_version" field, which describes the semantic versioning version number of the _slippc_ parser used to generate the file. By default, to keep file sizes down, _slippc_ only records deltas between frames (i.e., fields that change) for each player; by passing the -f option, _slippc_ will output a .json with all data at each frame intact, including unchanged fields. The top-level "frame_count" field specifies the total number of frames in each player's "frames" field, with "first\_frame" designating Melee's internal frame counter for the first frame (always -123), and "last\_frame" designating the final frame of the game.
+Passing the -j option to _slippc_ will output the .slp file specified with -i as a .json file, which may be opened in any text editor and inspected directly, or further parsed and analyzed using any JSON parser. Most data is presented in integer or float format, as stored in the .slp file. Major additions include the "game\_start\_raw" field, which is a base64 encoding of Melee's internal structure for initializing a new game, and the "parser\_version" field, which describes the semantic versioning version number of the _slippc_ parser used to generate the file. By default, to keep file sizes down, _slippc_ only records deltas between frames (i.e., fields that change) for each player; by passing the -f option, _slippc_ will output a .json with all data at each frame intact, including unchanged fields. The top-level "frame_count" field specifies the total number of frames in each player's "frames" field, with "first\_frame" designating Melee's internal frame counter for the first frame (should always be -123), and "last\_frame" designating the final frame of the game.
 
 ## Analysis
 
@@ -66,6 +66,8 @@ By passing a directory as the input file with the -i flag, _slippc_ will operate
   * -a : _input_-analysis.json
   * -X : _input_.zlp
 
+In directory mode, any errors during compression are written to an _\_errors.txt_ file in the directory specified with -X. Due to logistical overhead for parsing directories containing both raw and slippc-compressed files, directory mode currently does not have functionality to decompress all compressed files in a directory.
+
 ### Neutral Interactions
   The following are considered neutral states; frame counts should be identical for both players:
 

changelog.md

@@ -1,11 +1,27 @@
+### 2022-02-19
+  * Added support for parsing, analyzing, and compressing replays up to 3.12.0
+  * Added support for parsing, analyzing, and compressing replays down to 0.x.x
+  * Added support for parsing and analyzing (not compressing) partially corrupt replays
+  * Added support for entire-directory inputs for parsing, analyzing, and compression (not decompression yet)
+  * Added constraint on output filenames during compression ensuring all compressed replays end with .zlp and all decompressed replays end with .slp
+  * Added a testing program, test suites, and test replay files to ensure backwards compatibility, forwards compatibility, and data integrity
+  * Integrated cbartsch's changes into parser and analyzer
+  * Improved the way the compression algorithm handles items for better compression (most noticeable for Yoshi's Story games)
+  * Improved compatibility for compressing replays with lots of items or rollback'd frames that previously could not be compressed
+  * Improved resource management during compression to reduce memory footprint and increase compression speed
+  * Improved debug, warning, and error outputs
+  * Fixed several memory leaks so everything runs cleanly through valgrind
+  * Errors compressing files in directory mode are now written to a log file
+  * Bumped parser, analyzer, and compression versions to 0.8.0
+
 ### 2021-03-21
   * Added a bunch of missing game start block info to parser output
   * Added a bunch of missing player start block info to parser output
   * Added documentation to all SlippiReplay struct attributes in replay.h
   * Added original input file name to output of parser and analyzer
   * Merged cbartsch's fix for item counting
   * Renamed and reorganized a few fields in output for parser and analyzer
-  * Bumper parser and analyzer versions to 0.7.0
+  * Bumped parser and analyzer versions to 0.7.0
 
 ### 2021-03-13
   * Added new outputs to parser:

makefile-win.mk

@@ -13,6 +13,7 @@ src/analysis.h \
 src/compressor.h \
 src/enums.h \
 src/schema.h \
+src/gecko-legacy.h \
 src/util.h
 
 OBJS += \

src/compressor.cpp

@@ -813,20 +813,18 @@ namespace slip {
     unsigned start_fp           = 0;  //Frame pointer to next start frame
     unsigned end_fp             = 0;  //Frame pointer to next end frame
 
-    // DOCUMENT
-    const int RB_SIZE = 4;
-    char* dupe_frames = new char[RB_SIZE]{0};
+    //Initialize data for frame deferral counting
     char* defer_pre[8];
     char* defer_post[8];
     for (unsigned i = 0; i < 8; ++i) {
       defer_pre[i]  = new char[RB_SIZE]{0};
       defer_post[i] = new char[RB_SIZE]{0};
     }
+    char* dupe_frames = new char[RB_SIZE]{0};
     int max_frame          = -125;
     unsigned max_item_seen = 0;
     int modframe           = 0;
     uint8_t defer          = 0;
-    // DOCUMENT
 
     //Rearrange memory
     uint8_t pid; //Temporary variable for player id
@@ -847,7 +845,8 @@ namespace slip {
                 cur_frame        = readBE4S(&_rb[b+O_FRAME]);
             }
 
-            // DOCUMENT
+            // If we're shuffling, we need to keep track of deferred frames
+            //   once we unshuffle events back into order
             if(!unshuffle) {
               // Determine the number of times each recent frame has been duplicated
               modframe = ((cur_frame+256)%RB_SIZE);
@@ -863,7 +862,6 @@ namespace slip {
                 dupe_frames[modframe] += 1;
               }
             }
-            // DOCUMENT
 
             frame_counter[start_fp] = cur_frame;
             ++start_fp;
@@ -1250,6 +1248,17 @@ namespace slip {
     }
 
     //Free memory
+
+    delete[] ev_counts;
+    delete[] ev_max;
+    delete[] ev_size;
+
+    for (unsigned i = 0; i < 8; ++i) {
+      delete[] defer_pre[i];
+      delete[] defer_post[i];
+    }
+    delete[] dupe_frames;
+
     for (unsigned i = 0; i < 20; ++i) {
         delete[] ev_buf[i];
     }

src/compressor.h

@@ -28,8 +28,9 @@ const uint32_t DEFER_ITEM_BITS       = 0xC000;      //Bitmask for storing deferr
 
 const uint32_t ITEM_SLOTS            = 256;         //Max number of items we expect to track at once
 const uint32_t MESSAGE_SIZE          = 517;         //Size of Message Splitter event
-const uint32_t MAX_ROLLBACK          = 128;         //Max number of frames game can roll back
 const uint32_t MAX_ITEMS_C           = 2048;        //Max number of items to track initially
+const uint32_t MAX_ROLLBACK          = 128;         //Max number of frames game can roll back
+const int      RB_SIZE               = 4;           //Size of circular queue for tracking repeated frames
 
 const int      FRAME_ENC_DELTA       = 1;           //Delta when predicting and encoding next frame
 const int      ALLOC_EVENTS          = 100000;      //Number of events to initially allocate space for shuffling

src/parser.cpp

@@ -531,17 +531,16 @@ namespace slip {
 
     uint32_t id    = readBE4U(&_rb[_bp+O_ITEM_ID]);
 
-    if (id > MAX_ITEMS) {
+    if (id >= MAX_ITEMS) {
       return true;  //We can't output this many items (TODO: what's with item ID 3039053192???)
     }
 
-    int32_t f    = _replay.item[id].num_frames;
-
-    _replay.item[id].spawn_id          = id;
+    int32_t f = _replay.item[id].num_frames;
+    _replay.item[id].spawn_id = id;
+    if (_replay.item[id].frame == nullptr) {
+      _replay.item[id].frame = new SlippiItemFrame[MAX_ITEM_LIFE];
+    }
     if (id >= _replay.num_items) {
-      for(unsigned i = _replay.num_items; i <= id; ++i) {
-        _replay.item[i].frame = new SlippiItemFrame[MAX_ITEM_LIFE];
-      }
       _replay.num_items = id + 1;
     }
 

src/tests.cpp

@@ -109,25 +109,13 @@ int testTestFiles() {
   return 0;
 }
 
-int sanityCheck(slip::Parser *p, std::string name, std::string path, bool corrupt = false) {
+int sanityCheck(slip::Parser *p, std::string name, std::string path) {
   ASSERT(name+" Exists",access( path.c_str(), F_OK ) == 0,
         name << " does not exist");
   BAILONFAIL(1);
-  if (corrupt) {
-    bool didparse = p->load(path.c_str());
-    if(didparse) {
-      SUGGEST(name+" Attempted to Parse (and succeeds)",true,
-        name << " does not parse");
-    } else {
-      SUGGEST(name+" Attempted to Parse (and failed)",true,
-        name << " does not parse");
-      return 0;
-    }
-  } else {
-    ASSERT(name+" Parses",p->load(path.c_str()),
-      name << " does not parse");
-      BAILONFAIL(1);
-  }
+  ASSERT(name+" Parses",p->load(path.c_str()),
+    name << " does not parse");
+    BAILONFAIL(1);
 
   const SlippiReplay* r = p->replay();
   ASSERT("  First frame is -123",r->first_frame == -123,
@@ -209,7 +197,7 @@ int testCorruptFiles() {
       std::string name = entry.path().stem().string();
     // for(unsigned i = 0; i < ncomps; ++i) {
       slip::Parser *p = new slip::Parser(_debug);
-      if(sanityCheck(p,name,path, true) != 0) {
+      if(sanityCheck(p,name,path) != 0) {
         ++errors;
       }
       delete p;
@@ -230,7 +218,7 @@ int testConsistencySanity() {
       std::string name = entry.path().stem().string();
     // for(unsigned i = 0; i < ncomps; ++i) {
       slip::Parser *p = new slip::Parser(_debug);
-      if(sanityCheck(p,name,path, true) != 0) {
+      if(sanityCheck(p,name,path) != 0) {
         ++errors;
       }
       delete p;
@@ -410,7 +398,7 @@ int testCompressionBackcompat() {
       delete c;
 
       std::string test_md5_r = md5file(tmpunzlp.c_str());
-      ASSERT("MD5 of restored file for "+name+" matches original = "+test_md5_o,test_md5_r.compare(test_md5_o) == 0,
+      ASSERT("MD5 of restored file for "+name+" is "+test_md5_o,test_md5_r.compare(test_md5_o) == 0,
         "MD5 of restored file for " << name << " is " << test_md5_r);
     }
     return 0;
@@ -463,7 +451,7 @@ int testCompressionVersions() {
       delete c;
 
       std::string test_md5_r = md5file(tmpunzlp.c_str());
-      ASSERT("MD5 of restored file for "+name+" matches original = "+test_md5_o,test_md5_r.compare(test_md5_o) == 0,
+      ASSERT("MD5 of restored file for "+name+" is "+test_md5_o,test_md5_r.compare(test_md5_o) == 0,
         "MD5 of restored file for " << name << " is " << test_md5_r);
     }
   return 0;
@@ -493,7 +481,7 @@ int runtests(int argc, char** argv) {
     }
   }
 
-  // testTestFiles();
+  testTestFiles();
   testKnownFiles();
   testCorruptFiles();
   testCompressionBackcompat();