System of Systems Engineering is the idea of connecting disparate systems together to bring forward some emergent behavior. In Health Information Technology (HIT) this can mean connecting a Laboratory Information Management System (LIMS) to an EHR (acute or ambulatory) or maybe an EHR to a HIE to a Population Health solution. Or maybe it is the connection of multiple EHR solutions to allow for a seamless exchange within a geographic community. In Healthcare the number of connections is endless, so lets explore what it takes to test something like this.

Start with the Pop Health example

As a hospital care coordinator or a research physician I am trying to identify groups of patients who need specialized care or have specialized conditions. It might be that I want to actively manage my diabetic population to meet regulatory requirements or better yet to actively improve patient’s health. Or maybe I need to generate lists for Care Plans, or identify Multi-Condition Comorbidity, (MCC) cohorts. I might need to use the EHR data to identify cohorts of patients who are triggering an emerging pharmacovigilance state or novel disease cohort identification.

All of these things require significant collaboration across the entire solution. But more importantly, they are SoS engagements that require an understanding of the solutions from not only the individual solution perspective but the overarching emergent solution. Just as the engineering must consider the cross-product engagement, so must the testing of those solutions.

What do you need:

You need everything you need to develop a SoS solution that you need for a single solution – just more of it – and somethings that are optional in a single solution become required in a SoS solution. We will talk about: Requirements, Environments, Automation, Data, Usage Analytics, Test Data Crafting, Telemetry… 

Requirements

To test a solution you of course need to the requirements for the SoS solution. They are either already defined, or you will need to begin with a reverse engineering engagement to define them yourself. You must establish the current state of the source systems schema, enumerations, encoding, workflows, and common usage. Then, you must establish the same for the intermediary repositories. Finally, you must establish how the end analytics solution meet those requirements. As an aside, the more intermediate repositories you have in your data the harder it will be to conduct your analytics. Getting your data from the source may not be possible when you are dealing with a geographic community but keeping data acquisition as close to the source as possible is ideal. 

You must also generate requirements for your emergent behaviors, set the acceptance criteria and hold the contributing solutions to those criteria. This will mean that the source systems may need to change their behavior to align with the new need. Just because a feature/function/schema/dictionary worked for a limited workflow does not mean that it will work for the new emergent need. This also means that you will have to maintain cross product development registry and track to it. Note that your acceptance criteria must include the input, intermediation, and outputs — and realize that you may have an input acceptance criteria error that is generated two or three systems prior. I am not going to talk about Project Management in this note but it is essential to the success of the project. Perhaps that is another article.

Environment

Requirements out of the way, you then need to establish your test and development environment. I put test first but, in all seriousness, if you don’t have a SoS development environment you will eagerly fail to achieve success. What does the test environment look like?

It looks an awful lot like your live environment. It starts with the systems engineering. You need to have instances of the solution in this environment – connected to all the other components of the Solution. All of your EHRs, connected to your solution intermediaries, connected to your analytics solution, connected to the universe of your solutions. It is a big task, but it is required, and if you don’t have it you cannot succeed. Let me restate this: If you don’t have a representative environment for your development and testing your project will fail – full stop.

You could throw this up into a cloud based solution so you can scale it as you need, and turn it off when you don’t need to reduce costs. But, these costs are far less then a complete failure in your solution so do not be penny wise and pound foolish.

Next, you will need configurations and representative solution conditions. If you have a couple customers who will be your guinea pigs, then you should extract their environment for an overlay into your Dev/Test environment. If their solutions are highly customized, then you will also need a standard version of the solution in your environment. If you are worried about the number of versions of solutions you have – get unworried, you need them to make this thing succeed.

Automation

You will need automation for this solution. You are going to repeat the same tests multiple times, and you are also going to want to extend the complexity of your efforts over time. This means that baseline functions should be passed back to automation for regression batteries. Remember, if you are going to do something more than 7-9 times, you probably should consider automating it. You also need to consider that your automation plan may not be a single plan. If you have multiple architectures, you might need multiple automation solutions. It would be better to have one solution but that might not be possible – but consider how one automation script might trigger a script in another solution.

You should also consider automation oracles, functions that verify the results of your automation and check for a multitude of errors. They should be based on your failure taxonomy as well as your success criteria. If you can get there you are dramatically increasing your success probabilities. You are also dramatically reducing the cost of ownership, cost of support, and improving reliability. 

Customer Usage Analytics

You should also do analysis on the customer usage such that you understand how their usage will impact your solutions. You will find that their medicine is not the medicine that you think it is. They have used it in completely different ways then you intended. Some of this usage will impact the solutions downstream and you will need to consider them in your solutions. Other usages may be bad workflows that will negatively impact your customer’s success. They need to fix the bad workflows, but you need to consider the valid ones and react accordingly.

This is one of those areas where an existing solution may need to change in order to participate in the larger solution. A solution that allows text allergy entry should probably disallow such entries, or the receiving solution should allow for that text to be consumed as a note, either way you need to look at that data because you will find bad workflows. A solution that has artificial restrictions on the number of allergy reactions should probably rework their schema. And, your encoding/dictionaries/enumerations will probably need to map and align to some degree (a lot more than some). Comparing the customer usages across input solutions will help you identify these issues.

As an example, we analyzed multiple acute and ambulatory EHRs for cross solution exchange of clinical data. When we looked at this data we found misuse of dictionary items, use of text fields to ‘extend’ domains (allergy), mal-alignment of common industry structures, and many other anomalies. Many of these issues were between multiple instances of the same platform

Creating the Data

You are going to have to create data to exercise this solution. De-identified data is the best core data to use as it will reflect a customer’s usage, but you will also need to craft data for deterministic validation. Again, your automation plan is critical here; if you don’t have one you will spend man years creating data. You need to mine the customer data for unusual cases and incorporate them into your crafted solution data. In addition to that, you will also need to craft cases that will exercise specific functionality. They are usually multi-variate test cases around a disease, set of diseases, or regulatory conditions and I would suggest using a intelligent naming convention so that your automation can load them by the hundreds, thousands, or millions.

Intelligent naming conventions are essentially a test case that is repeated multiple times. Anderson, MedAllCatAAA Blue and Anderson MedAllCatAAB Blue are the same case, but iterations of that case. That case being the insertion of a set of Medication Allergies from EHR Blue. While MedAllCatAAA Green would be from EHR Green. If you need to have them work together then their EHR reference would be BlueGreen instead of Blue or Green. If you craft these cases such that they are easily identifiable and recognizable you will save your external testing team a world of grief. Creating a dictionary for these test cases is an easy way to assign specific cases to specific testers across the solution.

Telemetry

You should have telemetry built into your solution so you are looking at usage of the solution – across the entirety of the solution. You MUST also develop telemetry that tells you the solution is alive, awake, and working. Multi-solution solutions are difficult to understand – they get very complex – so having telemetry that tells you where something has failed will allow you to route issues to the correct individual or team. This telemetry should be designed not only for test, but for production as you will always want to know, or have support check to see if the solution is operational.

Team configuration

The composition of your teams are important. You must have people who know their areas and the next area down the pipe. Better yet, they should know the entire SoS solution – perhaps from only one aspect, but as much of it as possible. In the end you must have a complete skillset on your team. A skillset that encompasses all the major components of the entire solution – and they must be working together. Silos will not work here.

For example, you may have a clinician who can craft the medication order tests, and validate their exchange. You could (should) have several individuals with database skills who can at the very least walk the solution, database to database to see the transitioning of the data. You will need individuals who can see the configuration of your solutions, preferably across all the solutions. You will need individuals who can understand and use the interfaces, including how those interfaces may be configurable. You will need people who can analyze the data that is sourced, exchanged, are received for both verification and validation. Just a small set of examples of the team that you will want and need.

Some Cargo Cult and Mad Cow testing

Lastly, if a group tests in isolation they have not tested the solution. If a group doesn’t have a QA organization – it hasn’t tested its solution. If a group mocks or stubs the next solution out and never actually hooks up to that solution – it hasn’t tested the solution. Do not expect that these situations will achieve success – instead, they will fail. If you want a SoS solution to be successful, you must employ SoS principles. It is complicated, it is difficult, but the only way that you can succeed is to engage this level of engineering.

Conclusion

System of system testing requires the solution be present and connected. It requires extra work that requires extra teams who are dedicated to that effort. It requires data that reflects the source systems data and then is run through the solution to its end purpose. It requires looking at your source system data to identify errors, omissions, and oddities. And remember that quality is fitness to purpose and use — that sometimes means the source system must conform to a new standard of quality.