Go Back
11 Votes

Traditional Synergy in Visual Studio Repository Project Should Support Multiple Repository Input Files


Delivered

Our workflow consists of the need to insert over 2,300 individual files which compose our repository.

Currently the Repository Project only properly supports the ability to insert from a single source file into the repository.

We desire the ability to support an arbitrary number of files.

20 Comments | Posted by Ace Olszowka to Visual Studio Integration on 4/3/2017 10:04 PM
Andrew Thornton
Yes, same for us.  We've got over 2000 structures and currently keep them separate.  Moving to Traditional repository in Visual studio will be a significant downgrade.  I was wondering if we could keep the files separate in another project and use a pre-build action to merge - not what I want to do but it seems the only option for a project with as many structures as ours.

6/16/2017 7:08 PM   1  
Bill Ferris
This will be a problem for us a well.  We have 824 separate structure files.  There's no way I'm merging them.

7/6/2017 6:27 PM   1  
Stuart Cranston
After I make a change to my repository I want the changes to export to the appropriate schema be it a template / file / format or structure. I currently have over 4000 seperate schema files one for each individual element.
Another consideration is the schemas that are created should not have datetime information as I would not want to commit date changes into my source control system.
 

7/21/2017 8:23 AM   0  
Gordon Ireland
On a current project I am involved with, we hold all the schema files - one per structure - in a folder.  We use a BAT file to merge all of the schema files in the folder into a single, temporary file and then use command line instructions to load that single file into the repository.  That BAT file can be included as a command within VS.

It only takes a minute or so to merge and load the hundreds of structures/schemas in the project, and we do not have to change the BAT file as it picks up all files with a  given extension in the schema folder  

8/7/2017 8:48 AM   0  
Richard Morris
The reason for the request to support multiple scheme files is two fold.  Firstly the current RPS project does not support pre-build scripting so the creation of the single schema file from a number of schema files using a batch script has to be performed manually outside of VS.  This means it's not part of the "build" process, and so could be missed.  This capability is currently being added I believe.  The second is to be ability to identify changes to a single "structure" so that only code that references that "structure" is rebuilt, thus not requiring a complete rebuild when a structure in a single schema file changes.

8/8/2017 9:51 AM   1  
Bill Ferris
If a rebuild is going to be performed every time a structure is changed then there will be a lot of unnecessary rebuilds -- even if the rebuild is limited to only the code that references the structure.  You might end up rebuilding hundreds of routines when only several are really affected.  We're not using VS, but currently when we add a new field to a structure (taking from an existing filler area) we rarely rebuild everything that uses that structure.  We rebuild the routines that have been modified to use the new field.

8/8/2017 1:33 PM   0  
Andrew Thornton
At present a complete rebuild occurs on all things, OLB.ELB, mainlines every time the repository changes.
Our current strategy (non-VS) is also that when we add a field to a structure we don't rebuild everything. I've had discussions with Synergex support about this and the only way we currently seem to be able to mimic this in VS is by not having dependencies between projects, or not including projects to build.
We do full builds only overnight as it takes some time.
Reducing the number of routines that rebuild when there is a change as suggested by Richard does need to be done, but also we need to be able to go further and not build any dependencies if the repository changes, unless a rebuild has been requested.

8/8/2017 1:43 PM   0  
Ace Olszowka
We're willing to accept a complete rebuild of the world when the repository is changed; in our experience it causes far fewer problems and is much simpler to implement and far more reliable.

8/8/2017 2:12 PM   1  
Richard Morris
The rebuild after an RPS change is due to the fact that it's a single schema.  The project needs to be clever - handle multiple schema files and only cause something to build if a structure/field has changed (not keys, relations, files, etc.).  Ideally if the field change affects the compiler. When we get a windows script project that should rebuild things when toolkit related attributes are changed.

Once that happens then only the routines that use the changed structure will be built, which is a lot safer that cherry-picking the routines.

The build doing a rebuild is (I agree with ace) the safest option. As long as only the routines that are affected are rebuilt.  
 

8/8/2017 2:21 PM   0  
Andrew Thornton
I don't have a spare half hour to rebuild the entire system every time I add/change a field on a repository structure.  Well, I would if that's what actually happened of course, but productivity would drop through the floor and people might just get a bit annoyed.
There's actually a similar problem with mainlines and ELBs.  Our mainlines are normally small programs that xcall a subroutine in an ELB.  we don't currently re-compile mainlines (again except overnight)

8/8/2017 2:28 PM   0  
Richard Morris
But is this not the same in your non-Synergy/C# environments?  Make some sort of change in a library routine and everything above it rebuilds - even if that change has no effect on the program/class/routine?

8/8/2017 2:42 PM   0  
Andrew Thornton
I'm not comparing anything but SynergyDE (non-VS) with SynergyDE (VS), as that's the migration we're trying to make
I might still be able to get everything into Visual Studio so our overnight builds can be done with VS and msbuild, and I can then eventually try to move people across to using VS instead of our manual procedure at present, but if productivity is that low as a result it won't of course happen.
So the fact remains, at the moment we don't rebuild anything at all when we change the repository.  We don't rebuild mainlines when we change something in an ELB.

8/8/2017 2:50 PM   1  
Ace Olszowka
@Andrew/Richard
We encountered the same thing internally here at CU and there has been much discussion on the proper way to fix it.

The clearest/most concise description of the problem (and the differences) between C# and Synergy I have been able to come up with is this:
  • In the C# World we are in a thought process of having multiple, small, specialized Libraries/DLLs (what we would consider ELB's in Synergy)
  • In the Synergy world there are few, large, and comprehensive libraries. This has happened for a variety of reasons and isn't necessarily wrong (for example the ELB linking limit), but simply a different way of approaching the problem
In the past this did not make a lick of difference because developers "knew" enough about these libraries to understand when a rebuild was needed of certain programs. However there is no (easy) way for the compiler/build system to have this knowledge or be told about it without splitting up the libraries.

To give a concrete example we have an ELB called DataMapperLibrary; as the name suggests this is a library which contains all of our data mappers to give us Synergy Structures from our Database tables. This ELB is quite large because it contains EVERYTHING. However for 95% of the applications they really don't care that a structure has changed in an unrelated portion that they don't use (like for example the format of Purchase Orders might have changed, whereas the Customer Management program could care less about PO's).

One solution (and the one I have strongly pushed for) is to enable more specialization of these libraries; therefore instead of one massive DataMapperLibrary ELB you have multiple more specialized DataMapperLibrary ELB's (for example DataMapperLibrary.AccountsPayable, DataMapperLibrary.Payroll, DataMapperLibrary.CylinderControl, etc) this means that areas of the code that DO care about changes to these libraries are properly rebuilt; while those that do not are simply not linked to these libraries and therefore are not rebuilt.

Unfortunately like most things in software development this means massive refactoring, which involves lots of time, money, and most importantly RISK in breaking out a system which "works". This is where powerful refactoring tooling and code discovery that are given to us by Visual Studio comes into play. We're relying heavily on these tools to give us the information we need to make the best possible decisions on how to properly split these out with minimal risk.

In addition unless you fully commit to Visual Studio's build system splitting up a single ELB into 10's if not 100's of smaller ELB's does not sound attractive. However again there is a fundamental difference between Visual Studio and "Traditional"/"Notepad"/"CLI" Synergy Development, in the VS world multiple dependencies and projects are very low maintenance cost as the logic to determine the build order and if it needs to be built are handled by the system.

Again unfortunately this means that is going to get A LOT worse before it gets A LOT better.

I have no silver bullet; but I'm open to suggestions. Again we've had HUGE and spirited debates about this internally due to the fundamental design differences between C# and Synergy Developers.

8/8/2017 2:58 PM   0  
Richard Morris
I think we may be getting a little off topic here.  I think it's right that an RPS change causes all code that references the modified element is rebuilt.  I guess it would make sense to add an a new idea along the lines of "don't rebuilt dependent projects unless routine signatures change".  Maybe that would speed things up?

8/8/2017 3:05 PM   1  
Bill Ferris
"then only the routines that use the changed structure will be built, which is a lot safer that cherry-picking the routines."

1) I would hardly call rebuilding only those routines that are affected by a new field borne out of filler to be "cherry picking" -- since by definition they will be getting changed, and therefore rebuilt, anyway.

2) It's only "safer" if one has absolutely no knowledge of the overall application and what affects it.

3) That having been said, VS really has no real knowledge of the application, why the repository change is being made or what the overall project change is about.  It is not smarter than  the programmer (or at least shouldn't be).  I agree with Andrew's suggestion that manual control of a rebuild relating to repository changes would be a desirable option.

8/8/2017 3:07 PM   2  
Ace Olszowka
@Richard
I agree; I think its a discussion that needs to be filed as a separate idea and the deep discussion moved over there.

I did want to make sure that that ELB was addressed here though, as it is something we've also run into internally (so Andrew you're not alone) I'm open to discussion on how best to resolve that (on another thread if need be).

The problem with the Repository still stands; and is a specialized case of the ELB/OLB linking simply due to its global nature.

8/8/2017 3:08 PM   0  
Gordon Ireland
This post covers a lot of ground, and is not just about traditional synergy projects in VS.   The main thing I would like to see in a repository project is the ability to have multiple schema files in a project.  You can add them, but they get ignored.

11/7/2017 3:38 PM   0  
Ace Olszowka
This Issue blocks this https://synergexresourcecenter.force.com/apex/SiteIdea?id=0870d000000XjPxAAK

8/13/2018 5:49 PM   0  
Synergex
This is now available in version 11.1. 

10/2/2019 7:58 PM   0  
Ace Olszowka
If anyone is successful in using this new functionality in 11 with a large, comprehensive test case?

10/3/2019 1:23 PM   0  
Please log in to comment on this idea.