BlenderTools - Conversion Helper design

User avatar
50keda
SCS Software
Posts: 1954
Joined: 07 Dec 2012 11:59

BlenderTools - Conversion Helper design

#1 Post by 50keda » 08 Jan 2016 15:32

Hi guys,

as I am in the middle of creating shortend "pipepline" for mod creation, I would like to know what do you think about this:
Image

...automatic search for "mod" folder:
Image


Now we finally have magic button: "PACK CONVERTED DATA" one click for: export, convert, pack, copy to mod folder (or any other specified under the first field of "Mod Packing")


Thoughts, ideas are welcome!


Take care,
50k

User avatar
ohaha
Global moderator
Posts: 16881
Joined: 07 Dec 2012 06:31
Location: Watchin' y'all
Contact:

Re: BlenderTools - Conversion Helper design

#2 Post by ohaha » 08 Jan 2016 15:40

I need some time to chew all this.
But, since we're talking about exporting directly to the mod, there are a couple of questions / suggestions.

Whenever I edit a material, on an already existing model, it gets exported with a new name / path. But the old material remains in automat. With time (and several edits), I end up with unused files (mats) in automat folder.
Question is, is there a way to track all these "leftovers" and remove them?

Secondly, if I use non-packed mods (in folder state), can we have a checkbox to skip the archiving process, and just copy to a folder, instead?

Thanks :)
ImageImage
ImageImage

User avatar
galimim
Posts: 838
Joined: 26 Jan 2013 22:11
Location: Slovenija
Contact:

Re: BlenderTools - Conversion Helper design

#3 Post by galimim » 08 Jan 2016 16:38

Would it be possible for the conversion tools to look only at the folders used by the game? Example if I have folder named "Test" in my mod folder, it would not touch it, not copy or convert.

User avatar
50keda
SCS Software
Posts: 1954
Joined: 07 Dec 2012 11:59

Re: BlenderTools - Conversion Helper design

#4 Post by 50keda » 08 Jan 2016 18:04

ohaha wrote:Whenever I edit a material, on an already existing model, it gets exported with a new name / path. But the old material remains in automat. With time (and several edits), I end up with unused files (mats) in automat folder.
Question is, is there a way to track all these "leftovers" and remove them?
What are you talking about is simple action of cleaning "rsrc" folder before doing final "build/convert".
Old conversion helper has "Clean 'rsrc'" option and new one by current design has two options for that:
1. "CLEAN RSRC" operator, which directly removes everything that is currently in conversion tools 'rsrc' folder -> cleans it
2. "Auto Clean" option, which will simply again remove everything from 'rsrc' before exporting/converting/packing
ohaha wrote: Secondly, if I use non-packed mods (in folder state), can we have a checkbox to skip the archiving process, and just copy to a folder, instead?
Well this one is tricky, so easy answer is no as you can simply use "No Compression" and also it's not "just skip archiving" :D as I am not copying any converted sources anywhere, instead I am creating zip file directly.
Secondly what you are implying is editing data after conversion, which was never a good/safe solution. If you need to edit sth then edit it in your source mod data. As you will have this "magic" button now, I really don't see why you would have to have everything in "mod" folder...
galimim wrote:Would it be possible for the conversion tools to look only at the folders used by the game? Example if I have folder named "Test" in my mod folder, it would not touch it, not copy or convert.
And who is saying that someone else won't use Test directory for doing a mod out of it? There is no restriction on directories in our game (of course game searches for def files and some resources on particular "hardcoded" places, but it doesn't restrict user to use only directories you get from extracted base.scs)

On the other hand you can do that for yourself, open "<conversion_tools/bin/win_x86/tools/resconvert.sii" and see how rules are created, you can simply exclude directories you want by creating your own rules ;)

And there is other more suitable way for you, by creating proper hierarchy:
1. You need dir structure like this:

Code: Select all

-- MyModFolder
 |
 |- base -> put anything you don't want to be converted in here, make sure this folder is really named "base" as it serves as sibling SCS game hierarchy
 |- my_mod_base -> folder where you put your exported models, sii files... that you need in your mod
2. when using BT, set SCS Project Base Path to -> MyModFolder/my_mod_base
3. now you can load textures in BT from "base" and they will be treated the same as they would be inside "my_mod_base"
4. when you will use conversion "MyModFolder/base" will be ignored as you current SCS Project Base Path points to "MyModFolder/my_mod_base"
5. you just got what you wanted in the first place, ignoring any resources you don't want. So in your case "Test" folder would have to be in "base" not in "my_mod_base".

This behaviour is very usefull if you are using any resources from base.scs and you don't want to export or include them in your mod, hance look at the structure on my MapEdit github -> https://github.com/50k-Customs/ETS2-MapEdit

User avatar
ohaha
Global moderator
Posts: 16881
Joined: 07 Dec 2012 06:31
Location: Watchin' y'all
Contact:

Re: BlenderTools - Conversion Helper design

#5 Post by ohaha » 08 Jan 2016 18:25

Well, it's a bit more complicated...
Currently I export to a dedicated base, for each mod. I really don't want to mix my mods, for obvious reasons.
After export and conversion, I copy this new data into the mod, overwriting the old data (which is main reason I keep mods in unpacked state). This mostly includes automats, pmd+pmg and, in rare cases tobj's+dds's.
Clearing rsrc folder doesn't help much, because the obsolete materials are in my mod's automat folder - copied at the moment they were used.
Further more, I cannot use "clear rsrc" if I set conversion tools target to my mod, because I am not exporting the whole mod, but only parts. Clearing the mod folder would remove everything, including data which doesn't get updated, basically destroying the mod.

Anyway, after thinking about it, the conclusion is that keeping track of stuff which was exported previously is rather impossible. So, I'll stick to paying attention to what gets changed, after each conversion.

Thanks :)
ImageImage
ImageImage

User avatar
SiSL
Beta tester
Posts: 6697
Joined: 27 Oct 2013 06:23

Re: BlenderTools - Conversion Helper design

#6 Post by SiSL » 08 Jan 2016 19:08

I say IMO ofcourse, best way is to have 3 seperate folder

1. RSCS (to be cleaned everytime)
2. Includes Folder (Defs, other stuff people want to keep)
3. Final (which will me under ETS2 mods folder)

Bascially after compile in clean RSCS, copy files into Final with Others folder...

So that Final mod will always keep "cleaned automat" , + def files and other stuff people want to keep in the mod...

Also keeping Sample Base intact but starting a new folder is the slower down too...

User avatar
50keda
SCS Software
Posts: 1954
Joined: 07 Dec 2012 11:59

Re: BlenderTools - Conversion Helper design

#7 Post by 50keda » 08 Jan 2016 22:23

Well we ended up where I did not want: discussing different personal moders pipelines :D

If you want me to understand your points, you will really have to give me an example of your "good" practice with what you stated.

So as author of all this I will start with what I imagine to be a good practice:

Code: Select all

1. I decide to create a mod for a simple map model or truck accessory
2. I create "my_perfect_billboard_mod" and decide that I will create my own billboard with parts of existing texture and my new one called "perfect_billboard.tga"
3. I had check base.scs and happily noted that I will need some texture from it: "model/billboard.dds", so I create "my_perfect_billboard_mod/base" and create/copy texture to it: "my_perfect_billboard_mod/base/model/billboard.dds". Why place it into "base"? Because I know that this texture is already in game and I don't need to convert/pack it, I just need it for usage in BT.
4. After that I also created my own texture which needs to be converted and packed in my mod, so I will create another folder: "my_perfect_billboard_mod/upgraded_billboard"
5. Now I will place newly created texture inside it: "my_perfect_billboard_mod/upgraded_billboard/model/50k_perfect_billboard/new_billboard.tga"
6. As I have all resources I needed, I run Blender and firstly set SCS Project Base Path to: "my_perfect_billboard_mod/upgraded_billboard/" as this is where my mod base with new content is and will be exported.
7. Now create my magical billboard, featuring two materials. 
First one uses old texture, so I select "my_perfect_billboard_mod/base/model/billboard.dds" with Blender file browser.
Second material uses new texture and I select "my_perfect_billboard_mod/upgraded_billboard/model/50k_perfect_billboard/new_billboard.tga" with Blender file browser.
8. For export to take place I create SCS Root with name "billboard" and set Default Export Path to: "model/50k_perfect_billboard/"
9. Now I want my model in game so "EXPORT SCENE", "CONVERT CURRENT SCS PROJECT"
10. So far so got, now I got my sources somewhere on PC but I don't have a clue where.....hmm okay let's see ...
11. Oh I can use Mod Packing panel aswell :roll: 
12. So let's specify folder where I want my mod to be put and name of the mod under Mod Packing input fields.
13. Now finally hit "PACK CONVERTED DATA", which will find that sources somewhere on PC and pack it into mod instead of me.
15. Is that it?
16. Oh no! I forgot definition for my new billboard, okay :/ So I will create "my_perfect_billboard_mod/upgraded_billboard/def/world/model.50k.sii" and define path in it for my model.
NOTE: Step 16. is crucial, I don't fix converted/packed data, I add sii file in my "source" directory!
17. Oh so now I should convert & pack again :/ Waaaah, way I just don't use magic button eh? So I tick "Auto Convert" and press "PACK CONVERTED DATA"
18. Done, you got updated working perfect mod inside "mod" folder packed as zip already, so run the game try it out.
19. Now any change to model in Blender is not a problem I change it, tick "Auto Export", tick "Auto Convert" and press "PACK CONVERTED DATA", here you go mod exported & converted & packed & placed to the mod dir.
So what is it that I am missing here with this "pipeline"?

PS: if you will take a look at my GitHub for MapEdit you will see exactly the steps from above, I use "base" directory where I put already existing resources and then I create other derivative folders beside for any particular standalone "modules", in my case that is small prefab and map. Now as prefab can be shipped standalone, I created separated mod folder and put it there so I can convert it anytime without need to also convert map itself. Still with "Custom Paths" I can convert both at the same time, which I do when I am packing "50k_MapEdit.scs"

Now go one and share your examples so I can get ideas for this topic: "design of conversion helper"

User avatar
SiSL
Beta tester
Posts: 6697
Joined: 27 Oct 2013 06:23

Re: BlenderTools - Conversion Helper design

#8 Post by SiSL » 09 Jan 2016 03:00

Well, here I go :) But need to remind, that's only for single item modeller like me :)

Code: Select all

Folders:
1. SampleBase folder (no need to explain this one I guess), 
2. ETS2Base folder (Entire extracted of base.scs)
3. Mod folder (Final round up folder) in Documents\ETS2\mods folder eg. name "Test" 
4. Includes Folder (Stuff that will always be included but never changed by convertion tools, I place def file required in this place)

Progress:
1. Entire design progress is like currently is, do your model to your SampleBase folder... 
2. Convertion tools convert to RSCS folder
3. After convertion is done, RSCS folder files (optionally cleans itself) and IncludesFolder(#4) files combined together into Mod Folder #3 (And optionally cleans #3 Mod Folder before copying anything) , basically #3 (real mod folder) will be always "clean item" 
4. I get into game, add "Test"( #3) folder as mod and try my mod, since def files already included. 

After I feel done (which is not needed for this tool), I rename my SampleBase folder to item name and zip it for my archieve :)
Currently I manually do Progress steps after RSCS folder.

User avatar
50keda
SCS Software
Posts: 1954
Joined: 07 Dec 2012 11:59

Re: BlenderTools - Conversion Helper design

#9 Post by 50keda » 09 Jan 2016 13:08

@SiSl: I am not sure if this is your "workflow proposal" or you are doing it like this all the time? Because there is one thing that scares me:
4. Includes Folder (Stuff that will always be included but never changed by convertion tools, I place def file required in this place)
Conversion tools DO NOT change folder structure or any files which are not in "middle format", meaning ".sii", ".ogg", ".sui", ".mat" files won't be changed by conversion in any matter, they will be only copied to 'rsrc'. Meaning you don't need any includes folder, because you can put all sii files already in your "SampleBase" mod folder, moreover you won't need to do any merging before packing.

Secondly I strongly discourage this:
3. Mod folder (Final round up folder) in Documents\ETS2\mods folder eg. name "Test"
You should have your sources somewhere else and only copy final mod package inside "mods" folder.

I think that whole workflow presented with BT & conversion tools is not clear to many of the users and it is my fault as I obviously didn't present it well. So I will create presentation image to illustrate how everything is intended to be used.

User avatar
50keda
SCS Software
Posts: 1954
Joined: 07 Dec 2012 11:59

Re: BlenderTools - Conversion Helper design

#10 Post by 50keda » 12 Jan 2016 11:41

Here we go image representation of intended workflow using BT and conversion tools:

Comparison of current workflow against new one:
Image

One automated with cleaning 'rsrc':
Image

One automated using custom paths (multiple sources into one mod):
Image

PS: when new conversion helper will be released with some BT improvements, I will also put this to github, so users will have guideline on whole process ;)

Post Reply

Return to “Public research”

Who is online

Users browsing this forum: No registered users and 2 guests