Go Back

Understanding Harmony Core, and dealing with header and line objects

I am posting this so we can have a discussion on what was presented at conference.  Some of the questions may be simple but it doesn't matter.  The key thing is that we all end up with a good understanding of how to apply this to the real world of our applications.

First of all, I am assuming this builds on the work of Symphony Framework which code generates file and data classes from repository entries.

So Harmony core ends up with an object or collection of objects representing a given file/record type.

It then converts this to JSON and exposes these as restful web services.

$ODATA can then be applied to the results exposed by the web services.

When a create or an update is applied, it works in reverse - the object on the client side is converted to JSON, and Harmony Core creates an object from this and then updates the associated file.

Please correct me if I have misunderstood anything along the way.

I can see how the above can deal with extending the data classes to include additional derived fields, and $ODATA and the client does not and cannot know what is a real field and what is a derived field, and nor does it need to.

My question is around what happens if you want to extend an order class to include a collection of detail line objects.  Can you expose the whole order with its detail lines via web services ?  (I'm guessing you can).

What do $ODATA queries make of this - could you select all orders which include Product ABC123 ?

I'm sure if the web service recveived such an order object, the custom code could then pick out the detail lines and decide how to update the file. 


 

7 Answers
1   | Posted by Gordon Ireland to Other on 10/17/2018 12:07 PM
Steve Ives
Hi Gordon,

First of all, thanks for taking the time to write this, and to kicking off the discussion. Your understanding of how the Harmony Core environment works is broadly correct, but I think it might help to make a few clarifications. So here goes.

Yes, Harmony Core builds on many of the original ideas that went into the Symphony Harmony environment, and many of the concepts, CodeGen template code, and code in the Harmony Core libraries came directly from Symphony Harmony. That's why we selected the name Harmony Core, as a homage if you will to recognize that earlier work, combines with the fact that Harmony Core is developed using .NET Core.

As you state, CodeGen templates are used to generate various data and logic classes based on your repository structure (record) definitions in a repository. Those classes are then added to various projects and compiled into assemblies that when executed, expose RESTful web services that represent and expose the underlying data structures. These web services can be accessed, via the HTTP protocol, to perform CRUD (create, read, update and delete) operations on the underlying data.

First of all I want to clarify something. While it is true that RESTful web services almost always involve transmitting and receiving data over the wire in JSON (JavaScript Object Notation) format, the developer never really has to deal with this. It is not Harmony Core that is dealing with serializing (converting objects into JSON) the data; that happens at a much lower level in ASP.NET Web API.

Developers simply expose methods and those methods have parameters. Those parameters are based on simple types (int, string, boolean, etc.), complex types (data objects), and collections of types (arrays, collections, etc.). The signature of any given method ultimately determines the “shape” of the JSON data that is expected, or returned.

And you are correct, when a CREATE, UPDATE or PATCH (partial update) operations occur the process happens in reverse; ASP.NET Web API de-serializes the data (converts the received JSON into the types required by the method being called, hopefully), and our methods receive that data.
When we’re talking about web services we tend to refer to these methods as endpoints. Here is an example of an endpoint that returns all of a certain type of entity, in this case customers (click it, it should work):

https://harmonycoreapp.azurewebsites.net/odata/Customers

Any by default you can also access a single entity, by primary key, like this:

https://harmonycoreapp.azurewebsites.net/odata/Customers(1)

In addition, if you chose to do so, Harmony Core can also expose endpoints that allow you to access your data based on alternate keys. For example:

https://harmonycoreapp.azurewebsites.net/odata/Customers(State=’CA’)

These are all simple queries that result in accessing the data via Synergy SELECT class operations. But as you know, Harmony Core exposes OData endpoints, which means that in addition using the endpoints as shown above, consumers may use optional OData “queries” to refine and shape the data that they receive. Consider this example:

https://harmonycoreapp.azurewebsites.net/odata/Customers(1)?$select=CustomerNumber,Name

In this case the consumer has chosen to access a single customer, but indicated that they only require the customer number and name fields to be returned.

OData queries allow consumers to use the following keywords to construct queries:
  • $select                 Specify properties to return
  • $filter                   Constrain records to return (WHERE)
  • $orderby              Sort the results
  • $top                      Limit the number of results returned
  • $skip                     Skip records before returning results (paging)
By the way, these operations can be combined in order to create some pretty complex expressions.

Getting back to your original question, you stated that “OData can then be applied to the results exposed by the web services”. While you can think of it that way in terms of the final result, an OData query doesn’t operate ON the result set, rather it is used to FORM the result set.

Let’s consider a simple example:

https://harmonycoreapp.azurewebsites.net/odata/Customers?$filter=CreditLimit%20ge%209000&$orderby=CreditLimit%20desc

This one is a little more difficult to read, because OData uses spaces in some queries, but in a URL the spaces are represented by the characters %20. So the OData query in this case is:
 
$filter=CreditLimit ge 9000&$orderby=CreditLimit desc

The web service code doesn’t access all customers and then apply the OData query to filter the results in order to determine what data is returned to the consumer. Rather the OData query was used to access the data that way in the first place.

Harmony Core uses the new Synergy Entity Framework provider, which in turn transforms the OData query from the URL into appropriate Synergy SELECT, FROM, WHERE and ORDERBY operations. The OData query literally forms the Synergy SELECT statement that is used to obtain the requested data. So in the previous example, a WHERE object was used to filter the rows returned to only contain customers with a credit limit of $9000 or higher, and an ORDERBY object was used to sort the results so that the customers with the highest credit limits are returned first.

Now, while OData queries like $select, $filter and $orderby are very cool, and very powerful, one of the most cool and powerful features is $expand. This allows queries to follow relationships. Think JOIN in the SQL world; that’s what $expand allows you to do.

So, when you ask “what happens if you want to extend an order class to include a collection of detail line objects?  Can you expose the whole order with its detail lines via web services?”. The answer is that you don’t need to extend anything, OData is aware of the relationship (assuming you define the relationship in your repository, and enable expand support in Harmony Core) and allows you to use the $expand keyword to drill directly through into that data as you describe. Try this:

https://harmonycoreapp.azurewebsites.net/odata/Orders(3)?$expand=REL_OrderItems

As you can see, the order header record is returned, and has a “navigation property” called REL_OrderItems. Because our query included the keywords $expand=REL_OrderItems, the property was populated with the items for the order!

By the way, it is possible to drill down through several levels of data in this way, and OData also allows you to select fields, filter rows, and sort data in any way you want, at all levels. Now try this:

https://harmonycoreapp.azurewebsites.net/odata/Customers(8)?$select=CustomerNumber,Name&$expand=REL_Orders($select=OrderNumber,DateOrdered;$expand=REL_OrderItems($select=ItemNumber,ItemOrdered,QuantityOrdered,UnitPrice;$expand=REL_ItemOrdered($select=CommonName,Size;$expand=REL_VendorNumber($select=Name))))

This example returns selected data from five ISAM files. Pretty cool huh? And this functionality is all driven off a single web service endpoint called “Customers”, no custom code is required to use advanced features like this!

But speaking of custom code … while in some cases exposing CRUD operations might satisfy the requirements of some client applications, in many cases is will be necessary to expose additional endpoints that have custom signatures (parameters) and execute custom business logic.
In your original question you site one example of this; an order and its order line items. While the generated OData endpoints allow you to query hierarchical data like this, it doesn’t allow you to create new hierarchical data in a single operation, so to create an order you would need to first create the order, and then create the order lines. But as a developer you could add a custom endpoint that does both of these things in a single operation. Indeed this was the example that was used in the post conference workshop.

Harmony Core supports several mechanisms for exposing custom endpoints, but I won’t get into the technical detail about that right now. But I will mention that at least one of those mechanisms supports executing traditional Synergy subroutines and functions on all supported traditional Synergy platforms; so integrating code on Windows, UNIX, Linux and OpenVMS back-end systems is no problem. But for now let’s get back to the discussion about how RESTful web services and in particular, OData services work.

There is one part of your question that is still a work in progress. You asked "could you select all orders which include Product ABC123?". The answer is that OData supports that type of query, via the use of ANY and ALL features, but we're still working on adding support for those features. But watch this space, we have a design in place and we hope to get it added just as soon as possible.

I hope that this addresses at least some of your questions, but feel free to post again if you need to know more.

By the way, I would add that we are actively working on building out more documentation and examples for Harmony Core, and you can find all of that at https://github.com/Synergex/HarmonyCore/wiki.

10/17/2018 7:40 PM   2  
Gordon Ireland
Thanks Steve

Thanks for clarifying.  Even though I proudly worked through the workshop, I reverted to how I would extend an order class to add detail lines for a WPF application.  That involves a fair bit of customisation, whereas with in the $expand approach in the web services, that is not necessary.

If you add an order header, the result will contain the order number, which can then be assigned to the set of detail lines you would then pass back

I didn't realise the selection and filter criteria were passed to the select class.  While it looked clever in the workshop, I thought the code generated method exposed the full data set, and then the query picked what it wanted from the result set.  Thankfully I was wrong, and its even cleverer than I thought.  If you only want 3 customers out of 100,000, then only those 3 customers are exposed via the web service.

Going back to the "create order" sceanrio, the customisation would allow existing code to be callled to assign an order number (as you demonstrated), update outstanding order value, update committed stock figures, trigger back orders, email a confirmation etc


 

10/17/2018 7:57 PM   1  
Steve Ives
Your create order scenario is a perfect example of when it makes sense to use custom code, that way the code can do anything you need it to to. Here's the example custom code we used for Create Order in the workshop:
 
public method CreateNewOrder, int
    required in aOrder, @Order
    required in aOrderItems, @List<OrderItem>
proc
    ;;Validate inbound data (we're not a controller so we can't use ModelState!)

    if (aOrderItems.Count<1)
        throw new ApplicationException("No items were provided!")

    ;TODO: Need more data validation
    ;;Customer ID needs to be valid
    ;;Item ID's need to be valid
    ;;And more

    ;;Allocate the next available order number to the new order
    disposable data keyFactory = (@IPrimaryKeyFactory)mServiceProvider.GetService(^typeof(IPrimaryKeyFactory))
    keyFactory.AssignPrimaryKey(aOrder)

    ;;Propagate the new order number to the order items, and polulate line item numbers
    data item, @OrderItem
    data itemNumber = 0
    foreach item in aOrderItems
    begin
        item.OrderNumber = aOrder.OrderNumber
        item.ItemNumber = (itemNumber+=1)
    end

    ;;Save the new order
    mDbContext.Orders.Add(aOrder)
    mDbContext.OrderItems.AddRange(aOrderItems)
    mDbContext.SaveChanges(keyFactory)

    ;TODO: What happens if something fails?

    mreturn aOrder.OrderNumber

endmethod

As you can see, our method accepts an Order object and any number of OrderItem objects. We do some simple validation (more is needed) then we use our "primary key factory" to determine the next available order number, although you could do that any way you want. Having done that we ensure that the order number is saved to the Order and OrderItems objects, and we assign line item numbers to the OrderItem objects. Then we use the Harmony Core Entity Framework Provider provider to add the data to the order files; the advantage of doing this is that all of the data is part of a "transaction", all the STORE's either happen or don't happen, but you would also save the data any way you want to.
 

10/17/2018 8:13 PM   1  
Steve Ives
Oh, by the way, the JSON that we POST to the endpoint associated with this method looks like this:

 
{
	"aOrder":{
		"CustomerNumber": 8,
		"PlacedBy": "Max Henry",
		"CustomerReference": "NEWORD1",
		"PaymentTermsCode": "04",
		"DateOrdered": "2018-03-07T00:00:00Z"
	},
	"aOrderItems": [
		{
			"ItemOrdered": 20,
			"QuantityOrdered": 25,
			"UnitPrice": 1.49
		},
		{
			"ItemOrdered": 5,
			"QuantityOrdered": 25,
			"UnitPrice": 1.49
		}
	]
}

10/17/2018 8:15 PM   1  
Gordon Ireland
I was speaking to Jeff Greene and he kindly put some code together to show how a Harmony Core web service call could return a set of customers and their associated orders, and then turn this into an async list of customer objects, with each customer object containing a list of order objects.

I know we created the web service to provide the information, but didn't know who the result would be turned into something which could be processed at the client side.

The answer turned out to be 4 lines of code :
 - create an HTTP client 
- define security protocol
- make a call to the harmony core web service
- create a list of objects from the results returned 


           disposable data client = new HttpClient()
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
            data resultString = await client.GetStringAsync("https://harmonycoreapp.azurewebsites.net/odata/Customers?$expand=REL_Orders")
            data customers, @ODataCustomers, JsonConvert.DeserializeObject<ODataCustomers>(resultString)

Pretty neat

12/14/2018 3:51 PM   0  
Steve Ives
Yep, that's one of the coolest things about RESTful web services; they are called via simple HTTP requests, and that is something that is easy to do from pretty much any language with just a few lines of code. By the way, here's an example call from traditional Synergy:
 
data uri, string, "https://harmonycoreapp.azurewebsites.net/odata/Customers(1)?$expand=REL_Orders"
data caFile, string, "C:\path\to\TrustedRootCA.pem"
data responseString, string
data errorText, string
data sendHeaders, [#]string, new string[#] {"Host: harmonycoreapp.azurewebsites.net"}
data recvHeaders, [#]string
data httpError, int
httpError = %http_get(uri,5,responseString,errorText,sendHeaders,recvHeaders,,,,,caFile,,"1.0")

A couple of notes about this:
  1. Notice that we're explicitly using HTTP 1.0 for this call. This is because if we use HTTP 1.1 the WebAPI service will take advantage of an HTTP 1.1 feature called Chunked Transfer Encoding. which isn't currently supported by the Synergy HTTP API. Support will be added in Synergy 11.
  2. This code doesn't deserialize the response to a collection of objects like Jeff's code does. Here the responseString variable is literally a string containing the JSON response from the service, but you can use Jeff's very cool traditional Synergy JSON Parser class which is available in the Harmony Core source code repository.
Thanks for your continued interest in Harmony Core. We are continuing to work on the project, and will have more exciting news to share in the near future.

12/14/2018 10:32 PM   0  
Richard Morris

I feel I should clarify a few aspects of this thread.  The Symphony Framework has been developed over the last 8 years and provides the ability to expose Synergy data and logic to a wide number of client architectures.  These include in-process WPF desktop applications, enhancements to UI Toolkit programs and via a number of network endpoints including TCP, HTTP(s) and Microsoft’s Service Bus Relay.  Symphony Conductor is at the core of the Symphony Framework providing data mapping between Synergy proprietary data types and industry standard data types.  Symphony Harmony provides data access and remote logic execution capabilities that fully utilize the Symphony Conductor data models and the Synergy Select class.  Symphony Bridge is a Windows Service that provides self-hosted network endpoints that allow access through Symphony Harmony to Synergy ISAM data and existing application logic.  Symphony Bridge “out-of-the-box” provides network endpoints for TCP and HTTP(s) as well as Service Bus Relay.  It has, for some 3 years, provided a fully functional, standard RESTful API to allow access to the same Synergy ISAM data and existing application logic using standard URI syntax.  You can perform all the standard tasks (selecting data with full filtering capabilities, updating or creating data, etc.) using simple URI syntax.

Harmony Core is not an extension or evolution of the Symphony Framework.  Harmony Core has taken some of the ideas of the Symphony Framework and a small number of code concepts but is a complete rewrite.  Harmony Core is not compatible with anything currently available within the Symphony Framework.

The RESTful endpoint provided by Symphony Bridge provides powerful data access to individual ISAM files by marshalling the URI query to the Synergy select class and optimizing data access by performing sparse select queries.  This same endpoint provides seamless access to existing application logic using the concept of stored procedures without having to extend or write additional controller code.  Symphony Bridge protects the execution of this existing logic by using AppDomain’s to ensure that common/global data, hard-wired channel numbers and all none thread-safe code is completely isolated.  Stored procedures can be used to return complex, multi-layered collections of data objects or process complex inbound structures to store data into any supported database.

With nearly 30 years’ experience of working with Synergy ISAM files I am acutely aware of the problems encountered when trying to “join” files together and attempting to present them as a “relational” data model. Most applications today already have the complex logic required to “join” related files together to “load” for example a sales order.  Through Symphony Bridge you can expose the sales order object as a collection of Symphony Data Objects and provide access using your tried and tested logic.  This prevents to need to try to construct a complex URI query on each individual client to retrieve the same data.  The code is in one place – the server. 

Symphony Harmony also allows you to define Composite fields.  These enable you to define read-only non-repository fields that can return joined or calculated data.  When extending the structure based data object with a composite field you declare the fields that Symphony Harmony must also load to enable correct processing of your composite field.  For example you could define a composite field of SupplierName on your Order Header.  The SupplierName field requires the SupplierCode field to be loaded from the order header file.  If you select records from your order header file and include the SupplierName field, Symphony Harmony knows it must also read the SupplierCode field from the data file before executing the code within the SupplierName property where you load and return the required supplier information.

Harmony Core is an alternative to the Symphony Framework with, as presented at DevPartner 2018, less functionality.  Symphony Framework is now owned and developed by RCP Consultants.


1/7/2019 1:32 PM   0  
Please log in to comment or answer this question.