Handling startup of Integration Platform
During startup the Integration Platform will no longer try to load all queued and suspended jobs in one big batch witch could take minutes. Instead it will load them in batches of 1000 at a time.
Handling shutdown of Integration Platform
Queued jobs are no longer changed to status suspended during shutdown, which could take minutes if there are a lot of them. The suspended status is now only set on started jobs during shutdown of LIP.
Release date:
Fix for telemetry timestamp
Previously, the time for telemetry was set as the time when telemetry was sent to AppInsights, not the time when the event actually occurred. This is now fixed.
Filtering on time for logs in AppInsight
Browsing speed for JobExecutions is improved.
Export sales to Cloud
As a part of exporting sales from OnPrem Reporting to Azure. A new job is created for ReportingCloudIntegration package.
New job Blobingestion will copy Blobtype ReportingSales to a separate container.
State of new integration packages
Added information log when trying to change state of a package that is not successfully loaded by job host.
Role for start/restart of packages
You can start/stop/restart packages with just the PackageOperator-role.
Additional:
- There is a default "Home"-page, that does not load any data (to more easily switch tenants).
- Renamed and re-ordered the menu elements: Added "Manage packages" menu element (previously under "Configure package > Management", Fixed layout of "Configure package > Documentation"-tab by renaming "Documentation"-field to "Operations notes", moved the Save-button up, and made it clear that the version fields are not editable.
Version 3.0.13
Release date:
Track metrics for CPU usage by integration package
It is possible to see the total CPU usage that a LIP package has used since startup.
This is logged every 1 minute and sent to AppInsights.
Fix for logging
Fixed an issue that caused the last few seconds of logs not to be sent to ApplicationInsights when a package was stopped or upgraded.
Notifications
If an exception happened when sending notification from an integration package to the Chain Shell the exception was logged as an Integration Platform error and was not shown in the integration package. This change also adds the information so that we know which package the exception happened in so that the responsible team can see that the exception happened in their package instead of the Integration Platform team getting the exception.
Better exeption handling
Fixed an issue that occurred when re-trying status updates in Integration database that sometimes caused exceptions.
Stability fix
Fixed an issue where the Integration platform crashed and restarted during startup if a job creates a new job execution and run it before all integration packages has started. The exception is handled and logged in AppInsights/Event viewer.
Retry in GenericMessageForwarder package
Integration Platform 3.0.11 support new Service Bus library which has built in retry on connection errors. This is the first package that uses this new feature and this should make this job more stable.
Will be released after IP 3.0.11 is released to both on prem and cloud.
Release date:
Handling of problem/timeout situations
Major refactoring of how the Integration Platform use the database to better handle timeout and intermitten network issues.
Release date:
Filtering in BatchTransformOrchestration
Data that is loaded from the data source can be filtered and exported it to a single file, multiple files or both.
For example:
If you have 3 stores in POS Master with prices and you configure that you want to export store 1 and 2 it will export prices for store 1 and 2 in a file per store if you configure that you want them in separate files. You can also choose to export stores 1 and 2 in one file, or you can configure that you want to export in one file and files per store (both). Prices for store number 3 will not be exported at all when this filter is set.
Version 3.0.2
Fix in restart loop
(Work ID: 173040)
Fixed an issue that would lead to a package entering a restart loop after being enabled for the first time.
Improved performance when restarting
(Work ID: 174317, 174410)
Sometimes jobs fail to start when the Integration Platform is restarted due to the amount of data in the database. We have added an index that will improve the startup-time when starting integration packages. Also the code is optimized.
Cleaned up error messages
(Work ID: 174374)
Cleaned up and simplified error messages related to SignalR losing it's token making the Integration platform console unresponsive.
Added a retry for restarting the Integration Platform JobHost
(Work ID: 175127, 177549)
A retry has been added for resuming of job orchestrations after a restart to prevent the orchestration from timing out.
We have also doubled the timeout on resume orchestration after a JobHost restart.
Fixed multiple orchestrations after restart
(Work ID: 175226)
Fixed an issue that causes multiple orchestrations to run at the same time after the LIPHost was restarted.
Logging of jobs completed with warnings
(Work ID: 175234)
JobExecutions with status CompletedWithWarnings send requests to AppInsights.
Failed package upgrade
(Work ID: 176337)
Fixed an issue that caused the entire JobHost to go into a restart loop if a package upgrade failed.
Telemetry for integration package
(Work ID: 176631)
Fixed a bug where telemetry was not updated with information about the integration package and job the telemetry was sent from.
Updated .Net CORE compatibility
(Work ID: 177348)
The JobHost is compatible with packages that are running .Net CORE 3.
Resuming of orchestrations
(Work ID: 177615)
Fixed an issue with resuming of orchestrations after restart/package restart that would cause triggers to crash for certain jobs due to a timeout while resuming orchestrations.
Version 3.0.1
Support for automatic upgrades of Integration Platform packages in EG Cloud
(Cloud only)
#131980, #172575, #159029
To be able to upgrade packages quickly and with minimal risk we have created a new way of upgrading Integration Platform packages without having to deploy the platform each time.
When releasing a new integration package we will upload it to a repository blob storage in Azure. Each Integration Platform installation is connected to this repository and a user can choose which package he/she wants to install for the given tenant. When a new version of an integration package is uploaded each Integration Platform installation will automatically download the new package and upgrade it.
This feature includes changes in the Integration Platform Database (New tables), Integration Platform Manager (New user interface for selecting packages) and Integration Platform Host (Autoupgrader and Watchdog)
Support logging to Application Insights for Integration Platform Host and Integration Platform Manager (Cloud only)
#144161, #162056, #162867, #163132, #166634, #171254
When running Integration Platform in Cloud it is possible to configure the Integration Platform to send logs to Azure Application Insights (Instead of the Integration Platform database). This is enabled for all tenants in EG Cloud, both in Test and Production.
One drawback of this is that it will take some time ( between 1 second and 5 minutes) before the log is shown in Application Insights. If the user wants to see live logs from Integration Platform and Integration packages we have the new and improved streaming console in Integration Platform manager.
Saving logs in Application Insights have many advantages:
We don’t need to clean up logs our self any more, making the cleanup job much faster.
We can now configure automatic alerts in Azure for if an integration job fails of if there are a lot of jobs queued.
Application Insights has built-in dashboard giving us a quick overview of performance, failures etc.
Our Integration Platform dashboard will load a lot faster.
Since Integration Platform database will become smaller (for one tenant we went from 90GB to 90MB) all other databases could receive a little increase in performance.
Integration Platform environment for EG Cloud Administration
#165874, #170722
Added support for Integration Platform running in an environment context without tenants. This is only used by internal jobs needed for creating new tenants and deploying new cloud applications.
Minor new features
#144002 - Added logging of start, stop and restart commands. Added handler for cleaning up buffered telemetry client when appdomain is unloaded.
#162447 – Added warning if a job execution has more than 500 log lines due to maximum entries returned from Application Insights.
#162065 – Added log status “completedWithWarnings” for jobs where one or more error occurs but the job didn’t fail. Example: If 1 out of 100 POSLogs fails and the rest was processed correctly.
#162928 – Removed start/end times and replaced with created, started after and duration in Integration Platform Manager dashboard.
#142280, #162928, #145856 – Removed unused indexes and added two new indexes to speed up Integration Platform dashboard. Also added query hint to speed up cleanup job.
#169999 - Changed date and time to local time instead of UTC when streaming logs in console.
#163130 – Better formatting of exception messages in the user interface
#148324 – Added defragmentation of indexes during cleanup.
#163114, #168795, #162928, #165082, #162928 - Refactored SignalR hub and SignalR connection. Made multiple changes to the streaming console. Added filtering on all jobs in a package in the streaming console.
Improvements
#145856 – Fixed an issue where Integration Platform created multiple database connections to the database when only one was needed. Also added retry connection to Integration Platform database for up to 1 hour.
#156887 – Fixed an encoding issue in JsonLinesBatchSerlializer
#144002 – Fixed various issues that prevents jobs from being shut down
#158359 – Added exception handling for integration package startup
#156887 – Fixed an issue in jsonline encoding where it added UTF8-BOM on each line.
#160890 – Added check if we are running in Azure or on premise when connecting to database to be able to support Backoffice Integration subscribers database connect strings.
#144002 – Fixed an issue where renewing access token for Integration Platform Management failed causing loss of connection to the streaming console (SignalR)
#162839 – An issue where suspended and queued job executions would not be resumed at job startup. After installing Integration Platform 3.0 queued and suspended jobs created in Integration Platform 3.0 will be resumed during startup but not jobs created in Integration Platform 2.2.x.
#162867 - Serialized job execution context are verified when saved to database
#163173 – Fixed an issue where log details was missing when viewing logs in Integration Platform dashboard.
#163165 – Fixed an issue where Integration Platform jobs continued to read form Service Bus when the database was offline causing us to loose messages. Also added retry if database is offline.
#162928 - Fixed a reconnect to correct tenant when switching tenant. Fixed shared number of consoles/jobs connected between tenants.
#162867 – Fixed an issue where resuming job executions failed when deserializing the stored execution context in database failed. Also added logging when this happens.
#145680 – Fixed bug in the scheduler that caused Integration Platform packages running twice or more. Also added better logging.
#165054 – Fixed an issue when running transforming in parallel caused a racing condition.
#149426 – Sometimes a job can get status “completed” when it actually failed. This was fixed in this PBI and status should now be “CompletedWithWarnings”
#162928 – Multiple SignalR fixes.
#165519 – Improved error handling when stopping packages during upgrade. Prior to this fix the job execution would not start again when the package restarts.
#148157 – Fixed wrong execution state when Integration Platform is restarted and a job is running.
#162065 – Removed duplicates in log
#173063 - Fixed issue where job executions got status completed even if they failed.
#173734 – Integration Platform will now resume and complete all queued jobs after startup before starting new jobs.
Other technical/minor changes
Copied serialization type and status from old batchEventPublisher
Multiple NuGet updates
Migrated to GIT
Made JobHost internal, since we need to expose as little as possible from core.
Improved documentation on how to use the batch client
Converted the project to new csproj.
Warnings for runtimeredirects in WebJobHost
Warnings for missing references in database project
Warning for incorrect System.Net.Http version
Made SQL warnings as errors.
Fixed casing errors in a sql script.
Fixed error where new/disabled packages do not get their job config initalized. Also fixed error where conflict dlls are not archived.
Converted to asp.net core 3.0 app (and project)
Fixed error if jobhost starts without any existing packages.
Added machine hostname to Integration Platform logs
Fixed an exception when showing exceptions