[GUIDE] [1.20.0.7] Editing game files using a hex editor

Forum rules
SCS as a company do not wish to have paid mods on this forum. While we understand that not all paid mods use the Intellectual Property of other companies/people, it is very hard to moderate what is and isn't acceptable when money is involved. There are also concerns that it could look unfavorable to potential work partners going forward if SCS allow mods that may potentially use unlicensed branding.
Posting in the Mods forum (ATS and ETS2) is restricted to sharing free-to-the-public mods and providing support for mods. For more details, please check chapters [4] and [5] of Forum Rules.
User avatar
kouladi
Posts: 53
Joined: 04 Dec 2014 13:19
Location: Greece

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#11 Post by kouladi » 08 Nov 2015 15:09

Great info with excellent analysis and presentation! Much appreciated Cadde! Thank you very much!

User avatar
Cadde
Posts: 9535
Joined: 24 Apr 2013 18:08
Location: Have no fear, i am from the internets!

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#12 Post by Cadde » 11 Nov 2015 15:13

Excellent input mwl4, thank you!

I am on a hiatus from ETS 2 related stuff so those with the knowhow will most likely make good use of what you shared here.
Knowing, for example, how to make 010 templates or similar in other software (as well as those knowing programming ofc) makes it easy to work with these file formats.

If i haven't mentioned it before. Blender2SCS also has a lot of information related to PMG/PMD/TOBJ etc formats. As it's open/readable source people can use that to figure out some more stuff.
On extended hiatus.

User avatar
mwl4
Posts: 104
Joined: 25 Dec 2012 16:06
Location: Poland
Contact:

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#13 Post by mwl4 » 15 Mar 2016 13:42

Code: Select all

/*****************************************************************************
 * ets2_files - Structure definitions for the .pmd file formats used in ETS2 and other games by SCSSoft
 *  
 * Copyright (c) 2016
 *
 *****************************************************************************
 * Revision History:
 *  15/03/2016 - mwl4
 */

typedef quad                    s64;
typedef unsigned quad           u64;
typedef long                    s32;
typedef unsigned long           u32;
typedef short                   s16;
typedef unsigned short          u16;
typedef byte                    s8;
typedef unsigned byte           u8;

typedef u64                     token_t;

struct pmd_attrib_link_t
{
   s32 m_from;                      // +0
   s32 m_to;                        // +4
};

struct pmd_attrib_def_t
{
   token_t m_name;                  // +0
   s32 m_type;                      // +8 ; 0 = INT
   s32 m_offset;                    // +12 ; offset in value block
};

struct pmd_header_t // sizeof(64)
{
   u32 m_version;                   // +0

   u32 m_material_count;            // +4
   u32 m_look_count;                // +8
   u32 m_piece_count;               // +12
   u32 m_variant_count;             // +16
   u32 m_part_count;                // +20
   u32 m_attribs_count;             // +24

   u32 m_attribs_values_size;       // +28
   u32 m_material_block_size;       // +32

   u32 m_look_offset;               // +36
   u32 m_variant_offset;            // +40
   u32 m_part_attribs_offset;       // +44
   u32 m_attribs_value_offset;      // +48
   u32 m_attribs_offset;            // +52
   u32 m_material_offset;           // +56
   u32 m_material_data_offset;      // +60
};

struct pmd
{
   struct pmd_header_t header;
   token_t looks[header.m_look_count];
   token_t variants[header.m_variant_count];
   struct pmd_attrib_link_t attribs_link[header.m_part_count];
   struct pmd_attrib_def_t attribs_def[header.m_attribs_count];
   u8 attribs_value_block[header.m_variant_count * header.m_attribs_values_size];
   u32 materials_offset[header.m_look_count * header.m_material_count];
   u8 materials_data[header.m_material_block_size]; 
};

pmd file;

/* eof */

Code: Select all

/*****************************************************************************
 * ets2_files - Structure definitions for the .pmg (0x13 version)
 * file format used in ETS2 and other games by SCSSoft
 *  
 * Copyright (c) mwl4 2016
 *
 *****************************************************************************/

typedef quad                    s64;
typedef unsigned quad           u64;
typedef long                    s32;
typedef unsigned long           u32;
typedef short                   s16;
typedef unsigned short          u16;
typedef byte                    s8;
typedef unsigned byte           u8;

typedef s64                     i64;
typedef s32                     i32;
typedef s16                     i16;
typedef s8                      i8;

typedef u64                     token_t;
typedef i8                      bone_id;

struct float2 // sizeof(8)
{
    float x;
    float y;
};

struct float3 // sizeof(12)
{
    float x;
    float y;
    float z;
};

struct quat_t // sizeof(16)
{
    float w;
    float x;
    float y;
    float z;
};

struct float4x4 // sizeof(64)
{
    float m[4*4];
};

struct pmg_vert_coord_t // sizeof(12)
{
    struct float3 coord;    
};

struct pmg_vert_normal_t // sizeof(12)
{
    struct float3 normal;
};

struct pmg_vert_uv_t // sizeof(8)
{
    struct float2 uv;   
};

struct pmg_header_t // sizeof(116)
{
    u8 m_version;                               // +0
    u8 m_signature[3];                          // +1
    i32 m_piece_count;                          // +4
    i32 m_part_count;                           // +8
    i32 m_bone_count;                           // +12
    i32 m_locator_count;                        // +16
    float3 m_bb_center;                         // +20
    float m_bb_diagonal_size;                   // +32
    float3 m_bb_coord_1;                        // +36
    float3 m_bb_coord_2;                        // +48
    i32 m_bone_offset;                          // +60
    i32 m_part_offset;                          // +64
    i32 m_locator_offset;                       // +68
    i32 m_piece_offset;                         // +72
    i32 m_locator_name_offset;                  // +76
    i32 m_locators_name_size;                   // +80
    i32 m_anims_binds_offset;                   // +84
    i32 m_anims_binds_size;                     // +88
    i32 m_geometry_offset;                      // +92
    i32 m_geometry_size;                        // +96
    i32 m_uv_offset;                            // +100
    i32 m_uv_size;                              // +104
    i32 m_triangle_offset;                      // +108
    i32 m_triangles_size;                       // +112
}; /* ENSURE_SIZE(pmg_header, 116); */

struct pmg_bone_t // sizeof(200)
{
    token_t m_name;                             // +0
    float4x4 m_transformation;                  // +8
    float4x4 m_transformation_reversed;         // +72
    quat_t m_stretch;                           // +136
    quat_t m_rotation;                          // +152
    float3 m_translation;                       // +168
    float3 m_scale;                             // +180
    float m_sign_of_determinant_of_matrix;      // +192
    bone_id m_parent;                           // +196
    u8 m_pad[3];                                // +197
}; /* ENSURE_SIZE(pmg_bone, 200); */

struct pmg_uv_mask_t
{
    enum <i8> channel_t { NO_CHANNEL = 15 };
    channel_t m_texcoord0 : 4; channel_t m_texcoord1 : 4;
    channel_t m_texcoord2 : 4; channel_t m_texcoord3 : 4; 
    channel_t m_texcoord4 : 4; channel_t m_texcoord5 : 4;
    channel_t m_texcoord6 : 4; channel_t m_texcoord7 : 4;
};

struct pmg_piece_t // sizeof(104)
{
    u32 m_triangles;                            // +0 ; edges
    u32 m_verts;                                // +4
    struct pmg_uv_mask_t m_uv_mask;             // +8
    i32 m_uv_channels;                          // +12
    i32 m_bone_count;                           // +16
    i32 m_material;                             // +20
    float3 m_bb_center;                         // +24
    float m_bb_diagonal_size;                   // +36
    float3 m_bb_coord1;                         // +40
    float3 m_bb_coord2;                         // +52
    i32 m_vert_position_offset;                 // +64
    i32 m_vert_normal_offset;                   // +68
    i32 m_vert_uv_offset;                       // +72
    i32 m_vert_color_offset;                    // +76
    i32 m_vert_color2_offset;                   // +80
    i32 m_vert_tangent_offset;                  // +84
    i32 m_triangle_offset;                      // +88
    i32 m_anim_bind_offset;                     // +92
    i32 m_anim_bind_bones_offset;               // +96
    i32 m_anim_bind_bones_weight_offset;        // +100
}; /* ENSURE_SIZE(pmg_piece, 104); */

struct pmg_part_t // sizeof(24)
{
    token_t m_name;                             // +0
    i32 m_piece_count;                          // +8
    i32 m_pieces_idx;                           // +12
    i32 m_locator_count;                        // +16
    i32 m_locators_idx;                         // +20
}; /* ENSURE_SIZE(pmg_part, 24); */

struct pmg_locator_t // sizeof(44)
{
    token_t m_name;                             // +0
    float3 m_position;                          // +8
    float m_scale;                              // +20
    quat_t m_rotation;                          // +24
    i32 m_name_block_offset;                    // +40
}; /* ENSURE_SIZE(pmg_locator, 44); */

struct pmg_vert_color_t // sizeof(4)
{
    u8 m_r;                                     // +0
    u8 m_g;                                     // +1
    u8 m_b;                                     // +2
    u8 m_a;                                     // +3
}; /* ENSURE_SIZE(pmg_vert_color, 4); */

struct pmg_vert_tangent_t // sizeof(16)
{
    float w;                                    // +0
    float x;                                    // +4
    float y;                                    // +8
    float z;                                    // +12
}; /* ENSURE_SIZE(pmg_vert_tangent, 16); */

struct pmg_triangle_t // sizeof(6)
{
    u16 a[3];                                   // +0
}; /* ENSURE_SIZE(pmg_triangle, 6); */

struct pmg
{
    struct pmg_header_t     header;
    u8                      padding[header.m_bone_offset - 116 /* sizeof(pmg_header_t) */];
    struct pmg_bone_t       bones[header.m_bone_count];
    struct pmg_part_t       parts[header.m_part_count];
    struct pmg_locator_t    locators[header.m_locator_count];
    struct pmg_piece_t      pieces[header.m_piece_count];
};

pmg file;

/* eof */

Adjusted to using in 010 editor .pmd and .pmg structure.

User avatar
Cadde
Posts: 9535
Joined: 24 Apr 2013 18:08
Location: Have no fear, i am from the internets!

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#14 Post by Cadde » 15 Mar 2016 19:00

Thanks!
On extended hiatus.

User avatar
mwl4
Posts: 104
Joined: 25 Dec 2012 16:06
Location: Poland
Contact:

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#15 Post by mwl4 » 19 Mar 2016 14:10

Code: Select all

/*****************************************************************************
 * ets2_files - Structure definitions for the .tobj file formats used in ETS2 and other games by SCSSoft
 *  
 * Copyright (c) 2016
 *
 *****************************************************************************
 * Revision History:
 *  19/03/2016 - mwl4
 */

typedef quad                    s64;
typedef unsigned quad           u64;
typedef long                    s32;
typedef unsigned long           u32;
typedef short                   s16;
typedef unsigned short          u16;
typedef byte                    s8;
typedef unsigned byte           u8;

typedef s64                     i64;
typedef s32                     i32;
typedef s16                     i16;
typedef s8                      i8;

typedef unsigned quad           token_t;

enum <u8> type_t {
    map_1d = 1, 
    map_2d = 2, 
    map_3d = 3, 
    map_cube = 5 
};

enum <u8> filter_t {
    nearest = 0, 
    linear = 1, 
    unknown = 2, 
    _default_ = 3 
};

enum <u8> addr_t {
    repeat = 0, 
    clamp = 1, 
    clamp_to_edge = 2, 
    clamp_to_border = 3, 
    mirror = 4, 
    mirror_clamp = 5, 
    mirror_clamp_to_edge = 6 
};

struct tobj_header_t
{
    u32 m_magic;                // 1890650625
    u32 m_unkn0;
    u32 m_unkn1;
    u32 m_unkn2;
    u32 m_unkn3;
    u16 m_unkn4;
    u8 m_bias;
    u8 m_unkn4_0;
    enum type_t m_type;
    u8 m_unkn5;
    enum filter_t m_mag_filter;
    enum filter_t m_min_filter;
    enum filter_t m_mip_filter;
    u8 m_unkn6;
    enum addr_t m_addr_u;
    enum addr_t m_addr_v;
    enum addr_t m_addr_w;
    u8 m_ui;                    // boolean
    u8 m_unkn7;
    u8 m_unkn8;
    u8 m_unkn9;
    u8 m_unkn10;
    u8 m_tsnormal;              // boolean
    u8 m_unkn11;
}; // ENSURE_SIZE(tobj_header, 40);

struct tobj_texture_t
{
    u32 m_length;
    u32 m_unknown;
    char m_data[m_length];
}; // ENSURE_SIZE(tobj_texture, 8);

struct tobj_t
{
    tobj_header_t header;
    tobj_texture_t tex0;
    // for cube type there are 6 textures
};

tobj_t file;
.tobj structure.

User avatar
Cadde
Posts: 9535
Joined: 24 Apr 2013 18:08
Location: Have no fear, i am from the internets!

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#16 Post by Cadde » 19 Mar 2016 14:45

Nice work! Really awesome reversing skills!
On extended hiatus.

User avatar
mwl4
Posts: 104
Joined: 25 Dec 2012 16:06
Location: Poland
Contact:

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#17 Post by mwl4 » 19 Mar 2016 18:40

Code: Select all

/*****************************************************************************
 * ets2_files - Structure definitions for the .ppd (0x15 version)
 * file format used in ETS2 and other games by SCS Software
 *  
 * Copyright (c) mwl4 2016
 *
 *****************************************************************************/

typedef quad                    s64;
typedef unsigned quad           u64;
typedef long                    s32;
typedef unsigned long           u32;
typedef short                   s16;
typedef unsigned short          u16;
typedef byte                    s8;
typedef unsigned byte           u8;

typedef s64                     i64;
typedef s32                     i32;
typedef s16                     i16;
typedef s8                      i8;

typedef u64                     token_t;

struct float3
{
    float x;
    float y;
    float z;
};

struct float4
{
    float w;
    float x;
    float y;
    float z;
};

struct quat_t
{
    float w;
    float x;
    float y;
    float z;
};

struct placement_t
{
    struct float3 pos;
    struct quat_t rot;
};

struct ppd_node_t // sizeof(104)
{
    u32 m_terrain_point_idx;                
    u32 m_terrain_point_count;  
    u32 m_variant_idx;          
    u32 m_variant_count;                    
    struct float3 m_pos;            
    struct float3 m_dir;            
    i32 m_input_lines[8];           
    i32 m_output_lines[8];
};

struct ppd_curve_t // sizeof(128)
{
    token_t m_name;
    u32 m_flags;
    u32 m_leads_to_nodes;
    struct float3 m_start_pos;
    struct float3 m_end_pos;
    struct quat_t m_start_rot;
    struct quat_t m_end_rot;
    float m_length;
    i32 m_next_lines[4];
    i32 m_prev_lines[4];
    u32 m_count_next;
    u32 m_count_prev;
    i32 m_semaphore_id;
    token_t m_traffic_rule;
};

struct ppd_sign_t // sizeof(52)
{
    token_t m_name;
    struct float3 m_position;
    struct quat_t m_rotation;
    token_t m_model;
    token_t m_part;
};

struct ppd_semaphore_t // sizeof(68)
{
    struct float3 m_position;
    struct quat_t m_rotation;
    u32 m_type;
    u32 m_semaphore_id;
    struct float4 m_intervals;
    float m_cycle;
    token_t m_profile;
    u32 m_unknown;
};

struct ppd_spawn_point_t // sizeof(32)
{   
    struct float3 m_position;
    struct quat_t m_rotation;
    u32 m_type;
};

struct ppd_map_point_t // sizeof(48)
{
    u32 m_map_visual_flags;
    u32 m_map_nav_flags;
    struct float3 m_position;
    i32 m_neighbours[6];
    u32 m_neighbour_count;
};

struct ppd_terrain_point_variant_t
{
    u32 m_attach0;
    u32 m_attach1;
};

struct ppd_trigger_point_t // sizeof(48)
{
    u32 m_trigger_id;
    token_t m_trigger_action;
    float m_trigger_range;
    float m_trigger_reset_delay;
    float m_trigger_reset_dist;
    u32 m_flags;
    struct float3 m_position;
    s32 m_neighbours[2];
};

struct ppd_intersection_t // sizeof(16)
{
    u32 m_inter_curve_id;
    float m_inter_position;
    float m_inter_radius;
    u32 m_flags;
};

struct ppd_header_t
{
    u32 m_version;                      // +0 ; 21(0x15) supported only

    u32 m_node_count;                   // +4
    u32 m_nav_curve_count;              // +8
    u32 m_sign_count;                   // +12
    u32 m_semaphore_count;              // +16
    u32 m_spawn_point_count;            // +20
    u32 m_terrain_point_count;          // +24
    u32 m_terrain_point_variant_count;  // +28
    u32 m_map_point_count;              // +32
    u32 m_trigger_point_count;          // +36
    u32 m_intersection_count;           // +40

    u32 m_node_offset;                  // +44
    u32 m_nav_curve_offset;             // +48
    u32 m_sign_offset;                  // +52
    u32 m_semaphore_offset;             // +56
    u32 m_spawn_point_offset;           // +60
    u32 m_terrain_point_pos_offset;     // +64
    u32 m_terrain_point_normal_offset;  // +72
    u32 m_terrain_point_variant_offset; // +68
    u32 m_map_point_offset;             // +76
    u32 m_trigger_point_offset;         // +80
    u32 m_intersection_offset;          // +84
};

struct ppd
{
    struct ppd_header_t                 header;
    struct ppd_node_t                   node[header.m_node_count];
    struct ppd_curve_t                  curve[header.m_nav_curve_count];
    struct ppd_sign_t                   sign[header.m_sign_count];
    struct ppd_semaphore_t              semaphore[header.m_semaphore_count];
    struct ppd_spawn_point_t            spawn_point[header.m_spawn_point_count];
    struct float3                       terrain_point_position[header.m_terrain_point_count];
    struct float3                       terrain_point_normal[header.m_terrain_point_count];
    struct ppd_terrain_point_variant_t  terrain_point_variant[header.m_terrain_point_variant_count];
    struct ppd_map_point_t              map_point[header.m_map_point_count];
    struct ppd_trigger_point_t          trigger_point[header.m_trigger_point_count];
    struct ppd_intersection_t           intersection[header.m_intersection_count];
};

ppd file;

/* eof */
.ppd structure.

hydraulics
Posts: 34
Joined: 20 Nov 2013 23:06

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#18 Post by hydraulics » 16 Apr 2016 23:42

Great post man, well planned out!

I have done a bit of file reverse engineering in the past, using the method you suggested.
I was using an old hex editor and didn't have anywhere near the tool chain the one you suggest. Bookmarks and embedded code looks like great features.

I used to test the old Haulin games at the time, then Zmodeler released its SDK, which had all the header file data. So it was no fun after that.

Then Simon released his SCSBlender tool, again source was there.

I used to use c# and the binary stream reader, which was handy for a READ_ONLY app that I was creating.

I noticed your post tonight and had a bit of a mad notion idea, maybe I could create a READ_ONLY app in the browser(javascript).

To test proof of concept more than anything else really. I have hit a bit of a snag however, maybe someone could offer suggestions?!

The idea was just to get something up really quickly.
The local file is read into an arrayBuffer, the first 24bytes I can read without a problem, however when I try to read the next 2bytes(Uint8) I get an undefined error.

Here is the very ugly/early code, content getting passed to the function is an array buffer from the file stream.

Code: Select all

var parseTOBJ = function(content){
    
    var header = [];

    var Uint32 = new Uint32Array(content.slice(0, 4*6));
    
    header.push({
      'magic'     : Uint32[0],  // 1890650625
      'unknown0'  : Uint32[1],  // 0
      'unknown1'  : Uint32[2],  // 0
      'unknown2'  : Uint32[3],  // 0
      'unknown3'  : Uint32[4],  // 0
      'unknown4'  : Uint32[5]   // I guess this is a U16int ?? 
    });
    

    var Uint8   = new Uint8Array(content.slice(24, 2));
    
    header.push({
      'm_bias'    : Uint8[0],   //undefined
      'm_unkn4_0' : Uint8[1]    //undefined
    });
    
    console.log(header);
  }


I will post the rest of the code, just in case anyone else wants to test this out

Code: Select all

var handleLocalFile = function(e){
    
    var local  = e.currentTarget.files[0];
    var reader = new FileReader();
    
    reader.onload = (function(file){
      return function(e){
        
        var content = e.target.result;
        var type    = file.name.split('.');
        
        switch(type[1]){
          case 'sii':
            /**
             * This needs to be handled differently
             * using the readAsBinaryString function
             */
            var signature = /^ScsC/;
            
            if(type[0] !== "game" && signature.test(content)){
              return; // not a game file, and not decrypted, bail ???
            }
            
          break;
          case 'tobj':
            parseTOBJ(content);
          break;
          default : 
            return;
        }
        
      };
    })(local); 
    
    try{
      reader.readAsArrayBuffer(local);
    }catch(err){ console.error(err); }
    
  };
  
  $('#scsfile').on('change', $.proxy(handleLocalFile, this));

User avatar
Cadde
Posts: 9535
Joined: 24 Apr 2013 18:08
Location: Have no fear, i am from the internets!

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#19 Post by Cadde » 17 Apr 2016 00:28

Sorry, i don't do JavaScript so i can't really help you much on that front. But even to me, i see no reason as to why it should fail.
What happens if you attempt to read the two bytes without first reading your array of 32 bit values? Could it be that either the header or the reader ends up in a locked state?

As i said, i haven't touched much JavaScript at all in my time. But if your code would work as it think it does it just doesn't make sense.

Oh, and do try reading both your array of UInt32's and your array of UInt8's before the header.push. Push them all into the header in one go?
On extended hiatus.

hydraulics
Posts: 34
Joined: 20 Nov 2013 23:06

Re: [GUIDE] [1.20.0.7] Editing game files using a hex editor

#20 Post by hydraulics » 17 Apr 2016 01:44

That is the difficulty.

As its an Array buffer and not a stream(ie: no file pointer) the indexes need to be precise.
It's not the best method, but again I am just playing around with it. This logic would of course be very silly for a much larger dynamic file such as a .pmg

I am only working on the header structure you guys came up with. 4*6(4 bytes(Uint32) x 5) I believe is what it was, I may stand corrected!

Post Reply

Return to “Modding Guides”

Who is online

Users browsing this forum: No registered users and 1 guest