Asynchronous Batch Process Solution Revisited – part 4

In the previous post, I walked through the ABP (Asynchronous Batch Process) Aggregative Query Scenario. In this post, last in this series, I’ll go through the External Action scenario.

Prerequisites

1. Download the Asynchronous Batch Process Solution, import into Microsoft Dynamics CRM 2015 on-premise/Online organization
2. Go to Settings – > Solutions and Open the ABP solution. Go to the Batch Process entity definition and check the Settings checkbox in the ‘Areas that display this entity’ section
3. Save and publish the solution

As always, I advise against publishing any external solution on your production environment without testing it first.

External Action scenario

This scenario describes a scheduled action, not necessarily related to Microsoft Dynamics CRM records, hence external.
The target records FetchXML query can be used but it is optional.
This action is usually a wrapper for Custom Workflow Activity performing some magic which the native process tools can not.

Some sample business requirements of this type:

– Retrieve users data from AD (Active Directory) on daily basis and update Microsoft Dynamics CRM users data accordingly
– Import/export data to/from Microsoft Dynamics CRM to an external application over night
– Generate and email SSRS report to a mailing list every month

Following are the steps to implement the daily AD users data synchronization. For this scenario I assume you already registered a Custom Workflow Activity with some capability to retrieve data from AD and update System User records.

1. Create a new Action targeting ‘None (global)’ and name it ‘Synchronize AD users’
2. Add a step to activate the AD synchronization Custom Workflow Activity component
3. Save and activate the process

4. Create a new Batch Process record (Settings –> Batch Processes)
5. Name the Batch Process: ‘Daily AD users synchronization’ (or other meaningful name)
6. Set the Activation Frequency to ‘Daily’
7. Set the Process field value to ‘Synchronize AD users’ process
8. Set the Next Activation to any future date and time. I suggest selecting the application’s least busy time
9. Set the Status Reason to ‘Scheduled’
10. Click Save to schedule the Batch Process

You are done! The Batch Process is set and waiting for the next activation date and time. If all goes well, the AD users
synchronization will execute as defined and the Batch Process will reschedule to the same time next day.

If you would like to deactivate the Batch Process scheduling, change the Status Reason value to ‘Suspended’ and save. Since there may already be a Workflow instance scheduled and waiting, go to the Background Processes for the Batch Process record and cancel waiting instances.
This also applies if you change the Next Activation date/time once the Batch Process record is already created and scheduled.

I hope you find the scenarios I have described in the series as useful to you. If you have development skills or resources, you can customize the ABAP solution as you like, the source code is available on CodePlex.
I will keep adding features to this component and upgrade along with Microsoft Dynamics CRM versions until MS supplies a similar built in solution.

Advertisement

Asynchronous Batch Process Solution Revisited – part 3

In the previous post, I walked through the ABP Target Records Scenario. In this post, I’ll go through the Aggregative Query Scenario.

Prerequisites

1. Download the Asynchronous Batch Process Solution, import into Microsoft Dynamics CRM 2015 on-premise/Online organization
2. Go to Settings – > Solutions and Open the ABP solution. Go to the Batch Process entity definition and check the Settings checkbox in the ‘Areas that display this entity’ section
3. Save and publish the solution

As always, I advise against publishing any external solution on your production environment without testing it first.

Aggregative Query Scenario

This scenario describes a scheduled aggregative calculation on Microsoft Dynamics CRM records, followed by performing some business logic operation with the calculation result. Due to the aggregative calculation result nature, the business logic operation is usually an email notification or record creation/update.
Note that grouping is unsupported by this version of the ABP.

Some sample business requirements of this type:

– Send daily email message to Service Manager with total number of daily resolved cases
– Each month, create a Note for each Sales Person user, recording total estimated revenue of opened Opportunities
– Send monthly report to an external application with sum of total closed Orders

Unfortunately, it is not yet possible to define aggregative queries using the Advanced Find editor, but there are useful tools such as FetchXMLBuilder (part of the XRMToolBox  suite) that can help you define FetchXML syntax for such queries.
Also, the Microsoft Dynamics CRM SDK contains useful samples for aggregative queries which can be easily tweaked to suit your purpose.

Following are the steps to implement the daily resolved cases requirement:

1. Create a new Action targeting ‘None (global)’ and name it ‘Send daily resolved cases total notification’
2. Uncheck all ‘Start when’ trigger checkboxes to prevent automatic triggering of the process
3. Add an input parameter of type string named aggregationResult. This specific name is critical for for the ABP operation.
4. Add a step to send an email notification to the Service Manager, including the required text embedding the aggregationResult parameter value
5. Save and activate the process

6. Define aggregative query. I used the FetchXMLBuilder too available in the XRMToolBox  suite and copy the query text to the clipboard

7. Create a new Batch Process record (Settings –> Batch Processes)
8. Paste the aggregative FetchXML query into the Target Records field. Note that the alias is used as as part of the aggregation result, so a meaningful name is preferred
9. Name the Batch Process: ‘Daily resolved cases total notification’ (or other meaningful name)
10. Set the Activation Frequency to ‘Daily’
11. Set the Process field value to ‘Send daily resolved cases total notification’ process
12. Set the Next Activation to any future date and time. I suggest selecting the application’s least busy time
13. Set the Status Reason to ‘Scheduled’
14. Click Save to schedule the Batch Process

You are done! The Batch Process is set and waiting for the next activation date and time. If all goes well, an email notification will be sent as defined and the Batch Process will reschedule to the same time next day.

If you would like to deactivate the Batch Process scheduling, change the Status Reason value to ‘Suspended’ and save. Since there may already be a Workflow instance scheduled and waiting, go to the Background Processes for the Batch Process record and cancel waiting instances.
This also applies if you change the Next Activation date/time once the Batch Process record is already created and scheduled.

In the next post, I’ll walkthrough the External Action scenario. Stay tuned.

Asynchronous Batch Process Solution Revisited – part 2

In my previous post, I described some of the Asynchronous Batch Process Solution (ABP) version 2 internals. In the next posts, I would like to go through different business scenarios and explain how to use the ABP solution, step by step.

Prerequisites

1. Download the Asynchronous Batch Process Solution, import into Microsoft Dynamics CRM 2015 on-premise/Online organization
2. Go to Settings – > Solutions and Open the ABP solution. Go to the Batch Process entity definition and check the Settings checkbox in the ‘Areas that display this entity’ section
3. Save and publish the solution

As always, I advise against publishing any external solution on your production environment without testing it first.

Target Records Query Scenario

This scenario describes a business logic operation (Action) which is automatically applied to a collection of Microsoft Dynamics CRM records (target records), either once or repeatedly.

Some sample business requirements of this type:

– Send birthday greeting to target Contacts on their birthday date
– Send monthly newsletter to target customers
– Weekly Leads Evaluation – once a week, disqualify all Leads that are more than 5 days old and are not rated ‘Hot’
– Update large collection of records by overriding the 250 records limit (max. grid page size)

Following are the steps to implement the Weekly Leads Evaluation requirement:

1. Create a new Process (Workflow Rule (real-time/background) or Action) targeting the Lead entity and name it ‘Inactive Lead Disqualify’
2. Uncheck all ‘Start when’ trigger checkboxes to prevent automatic triggering of the process
3. If you created a Workflow Rule, check the ‘As an on-demand process’ checkbox and select the correct Scope
4. Add a Check Condition step to test if the Lead record is in an Open status. Add another step to change Lead Status to Cancelled (Disqualified state)
5. Save and activate the process

6. Open Advanced Find and define a new query to retrieve all Lead records where Created On date is older than 5 days and are not rated ‘Hot’
7. Click to Edit Columns button and remove all columns to maximize the query efficiency
8. Click the Download Fetch XML button and open the resulting file with some text editor
9. Copy the FetchXML query text to the clipboard

10. Create a new Batch Process record (Settings –> Batch Processes)
11. Paste the FetchXML query test into the Target Records field
12. Name the Batch Process: Weekly Leads Evaluation Process (or other meaningful name)
13. Set the Activation Frequency to ‘Weekly’
14. Set the Next Activation to any future date and time. I suggest selecting the application’s least busy time
15. Set the Status Reason to ‘Scheduled’
16. Click Save to schedule the Batch Process.

You are done! The Batch Process is set and waiting for the next activation date and time.

Looking at the Background Processes tab for the Batch Process record, you should notice one waiting Workflow instance which is waiting to retrieve all matching Lead records and to apply the Inactive Lead Disqualify process to each resulting record.

Once the Batch Process was executed once, you will notice that the Next Activation date was rescheduled for one week after the last activation.
If all went right, the Last Activation Completion Status is Success and the Activation Counter is set to 1.

 

 

If you created a background Workflow Rule, you can view a list of the records that were affected by the batch process execution. To do that, navigate to the Inactive Lead Disqualify workflow rule and click the Process Sessions tab.

If you would like to deactivate the Batch Process scheduling, change the Status Reason value to ‘Suspended’ and save. Since there may already be a Workflow instance scheduled and waiting, go to the Background Processes for the Batch Process record and cancel waiting instances.
This also applies if you change the Next Activation date/time once the Batch Process record is already created and scheduled.

In the next post, I’ll walkthrough the Aggregative Query Scenario.

Asynchronous Batch Process Solution Revisited – part 1

I had some free time and decided to review the Asynchronous Batch Process Pattern Solution. On January 2013 I promised to publish a revised version and didn’t manage to do so. To redeem myself, I have decided to publish this version on CodePlex, so it can be customized by anyone who needs to.
If you don’t know what ABPP is all about and what’s in it for you, I suggest you start here.

In the first part of this post I’ll describe the new features, architecture considerations and implementation details. In the next part, I’ll walkthrough the different scenarios and use cases.

You can download an unmanaged solution as well as source code here.
As always, I advise not to publish an external solution on your production environment without testing it first. I tested this solution with Online (2015 RU 0.1) and on-premise 2015 deployments.

 

Following are some of the features included in the revised solution (version 2.0):

1. Microsoft Dynamics CRM 2015/Online supported solution
2. Action support – allow mapping to Action/Workflow Rule (sync./a-sync.) process as an Action component
3. Integration scenarios support – activate Action process without specifying target records. This allows scheduling activation of any external action (like posting/retrieving data to/from an external application or data source) which can be wrapped with Custom Workflow Activity component
4. Aggregative FetchXML query support – schedule execution of a an aggregative query (multiple aggregation support) and map aggregation result(s) to target Action as an input parameter
5. Hourly scheduling  frequency support
6. Diagnostics – update batch process record with last activation result (success/failure) and failure reason.
7. Fetchxml 5000 records limit override – use conversion to Query Expression in order to handle queries that exceed 5000 records
8. Activation counter – additional attribute that automatically increment with each successful Batch Process activation. This allows adding a workflow step or Rule that will suspend the Batch Process scheduling after X activations
9. Zombie process prevention – scheduling activation only for future dates to prevent spawning of Workflow instances for past dates that will never occur

I have skipped some features that I could not design without breaking the ABPP paradigm, which generally dictates a complete separation between the Schedule component, the Action component and the target business records.
These features need some more thinking and if you have any suggested architecture, tell me about it.
The features I skipped:

1. Centralized error handling – since the Action component is oblivious to its activator, execution errors (especially with a-sync. process Action handlers) cannot be easily related to the process activation
2. Detailed Tracing – similarly to error handling, recording detailed tracing information such as total number of failure/success cases or processed records ids is a challenge due to the ABPP paradigm

Now, let’s get down to the details.

First, a small addition to the ERD. The new Process Activation entity purpose is to document each activation details.
In the future, I’ll add option to document the ids of the records which were returned by the Target Records Query.
This will allow tracing of the specific records that were affected by each ABP activation.

Currently, the Process Activation entity contains these custom attributes:

1. Parent Batch Process id
2. Activation start date
3. Activation end date
4. Total number of processed business records

Next, the main algorithm diagram:

Asynchronous Batch Process Solution Revisited – part 1

I had some free time and decided to review the Asynchronous Batch Process Pattern Solution. On January 2013 I promised to publish a revised version and didn’t manage to do so. To redeem myself, I have decided to publish this version on CodePlex, so it can be customized by anyone who needs to.
If you don’t know what ABPP is all about and what’s in it for you, I suggest you start here.

In the first part of this post I’ll describe the new features, architecture considerations and implementation details. In the next part, I’ll walkthrough the different scenarios and use cases.

You can download an unmanaged solution as well as source code here.
As always, I advise not to publish an external solution on your production environment without testing it first. I tested this solution with Online (2015 RU 0.1) and on-premise 2015 deployments.

Following are some of the features included in the revised solution (version 2.0):

1. Microsoft Dynamics CRM 2015/Online supported solution
2. Action support – allow mapping to Action/Workflow Rule (sync./a-sync.) process as an Action component
3. Integration scenarios support – activate Action process without specifying target records. This allows scheduling activation of any external action (like posting/retrieving data to/from an external application or data source) which can be wrapped with Custom Workflow Activity component
4. Aggregative FetchXML query support – schedule execution of a an aggregative query (multiple aggregation support) and map aggregation result(s) to target Action as an input parameter
5. Hourly scheduling  frequency support
6. Diagnostics – update batch process record with last activation result (success/failure) and failure reason.
7. Fetchxml 5000 records limit override – use conversion to Query Expression in order to handle queries that exceed 5000 records
8. Activation counter – additional attribute that automatically increment with each successful Batch Process activation. This allows adding a workflow step or Rule that will suspend the Batch Process scheduling after X activations
9. Zombie process prevention – scheduling activation only for future dates to prevent spawning of Workflow instances for past dates that will never occur

I have skipped some features that I could not design without breaking the ABPP paradigm, which generally dictates a complete separation between the Schedule component, the Action component and the target business records.
These features need some more thinking and if you have any suggested architecture, tell me about it.
The features I skipped:

1. Centralized error handling – since the Action component is oblivious to its activator, execution errors (especially with a-sync. process Action handlers) cannot be easily related to the process activation
2. Detailed Tracing – similarly to error handling, recording detailed tracing information such as total number of failure/success cases or processed records ids is a challenge due to the ABPP paradigm

Now, let’s get down to the details.

First, a small addition to the ERD. The new Process Activation entity purpose is to document each activation details.
In the future, I’ll add option to document the ids of the records which were returned by the Target Records Query.
This will allow tracing of the specific records that were affected by each ABP activation.

Currently, the Process Activation entity contains these custom attributes:

1. Parent Batch Process id
2. Activation start date
3. Activation end date
4. Total number of processed business records

Next, the main algorithm diagram:

Asynchronous Batch Process Pattern – Leveraging December 2012 Service Update (Polaris) Features

December 2012 Service Update features two new interesting capabilities:

• Custom Activities support for Microsoft Dynamics CRM Online
• Executing multiple requests with the ExecuteMultipleRequest class  

These capabilities allow important improvements to the Asynchronous Batch Process Pattern solution described here.

So what’s new?

The lack of Custom Activities support in Online deployment required the usage of a Plugin component to perform the Target Records query and implementing the Action Workflow to each result record.  
With December 2012 Service Update Custom Activities support for Microsoft Dynamics CRM Online, a Custom Activity can now replace the Plugin component. This implementation renders the Asynchronous Batch Process Pattern solution simple and robust. 

The new ExecuteMultipleRequest class allows sending a bulk of requests to the Organization service in one request, instead of a stream of separate requests. This feature is now used in the Asynchronous Batch Process Pattern solution In order to apply the Action Workflow Rule to multiple Target Records in a single request and to improve the solution’s performance. 

I will make the new and improved solution available for download soon.