sf-cog

# Data Management

We deal with a few things about data in Salesforce in this chapter.

We love to deal with complex data.
-- No One

# Batch Data Processing

The typical way to get data in your salesforce org is for the users to type them in 😃. Unfortunately that is not always possible -

  1. There are other systems in the enterprise which collect data in hundreds or thousands of records
  2. You may be setting up Salesforce for the first time and importing data from an existing system
  3. Your company may be acquiring another company that has its own data set to be available for them to use the system

In all those cases (and probably hundreds more), you should consider -

  1. using the batch data processing utilities.
  2. integrating external systems using Salesforce provided APIs to import/export data

As part of admin course, we will explore a bit on the batch data utilities available within Salesforce in this section -

  1. Data Loader application (independent program supplied by Salesforce)
  2. Data import wizard (available within Salesforce app)
  3. Overview of third party batch data processing options

# Data Loader

Data loader is a desktop tool to batch import, update or export data to/from salesforce. You use files to source data when using data loader.

salesforce-data-loader-1

# Considerations for using Data Loader

Here are the typical considerations while doing any export/import of data-

  • Identify the need for bulk operations
  • Use Data Loader to-
    • Create field mapping
    • Work with fields and attachments
  • Use data import wizard (within app) and Windows-based Data Loader tool
  • Work with basic SOQL/SOSL
  • Effect of Apex rules, workflow, assignment rules on Data Loader

# Installing Data Loader

Data loader is an independent program to batch process data. Refer the Data Loader installation instructions to install the program for your operating system. It essentially consists of two steps-

  1. Install OpenJDK
  2. Go to Setup > Integration > Data Loader. Download Data Loader and set it up on your device

# Using Data Loader

To use Data Loader, you first create mappings to map columns in your file to the fields in Salesforce.

  • Admins (or super users) can create mappings within Salesforce for Data Import
  • Data files (including relationships ) need to identify a Salesforce id, or external id for upserts
    • Can be created manually or using data manipulation routines outside Salesforce
  • Data Loader tool is typically used by admin / support team – has advantages over the data import wizard

salesforce-data-loader-entities-fields.png

Exporting data will include following steps -

salesforce-data-loader-export-steps

Import data process will look like this -

salesforce-data-loader-import-steps

The development life cycle consists of -

  1. Design backing up routines for data that will get impacted
  2. Design data transformations external to data loader
    • Change values or structure of data in the file
    • Third party utilities like Informatica ETL/Cloud help streamline these processes for larger enterprises
  3. Design data map - map fields to columns
  4. Design test cases - automated checks and manual tests. Include process testing as part of data testing (e.g. check whether order management goes through just fine for the products being imported)
  5. Import in sandbox. Conduct system tests for data loads and for backup routines
  6. Import in production

Data Loader can process records in bulk and you should set a good enough number for you.

Note that data loader by itself cannot be scheduled to run at a specific time or to repeat job executions, but you can use the Command Line arguments to run the data loader and schedule the program using external schedulers available for your operating system. (typically this is done on a server machine.)

# Data Import Wizard

Data import wizard is available through the Salesforce app and should be your go-to tool for simple batch processing routines.

Access data import wizard by going to Setup > Integrations > Data Import Wizard.

salesforce-data-import-wizard

Click on Launch Wizard for a guided screens to get started.

  • Choose entities
  • Choose your operation - insert, update, insert & update
  • Choose your file
  • Create/edit mappings between the input file and Salesforce fields
  • Start import!

salesforce-data-import-wizard-choose-entities-file

As you see, this is simpler than using data loader.

  • No need to install anything, and the tool is available within Salesforce app
  • You can schedule export
  • You can get alerted on data duplicates and choose to ignore duplicate records while importing data

On the other hand -

  1. It supports only record sets up to 50,000 records
  2. Although data import wizard supports all custom objects, only a few Salesforce default objects are supported (e.g. Accounts, Contacts, Leads). Data Loader supports all custom and Salesforce provided objects (excluding users)
  3. You cannot delete records
  4. No export facility available (Data Loader supports exports)

# Third Party Data Processing Utilities

Data loader provides an easy solution to import data, but is not always effective for all use cases. For example -

  1. You need complex data transformation before loading data to Salesforce
  2. You need custom roll-back logic for treating data exceptions
  3. Establish entity relationships for inbound data using custom keys or using complex logic
  4. You need custom error handling for sensitive data

Large enterprises with complex data needs often consider external tools to data integration.

The third party data processing utilities also call Salesforce batch data API to load data (similar to Data loader), but provide additional functions to make your life easier (at a cost of course). Common functions provided by external programs -

  • Manipulate data before feeding into salesforce (or even generating files for data loader)
  • Provide better ways to create relationships with less manual intervention
  • Custom error handling and better control on rollback

Your org-wide API limits apply for any data processing operations using these third party utilities.

Examples -

  • Jitterbit Data Loader for Salesforce
  • Apsona app
  • Mulesoft (part of Salesforce in Q2 '18, but is additional cost)
  • Informatica Cloud / On Premise solution with Salesforce Connector
  • Salesforce Connector in Heroku
  • Talend & other solutions that manipulate data structure and values before files are processed by Salesforce Data Loader

# More on Data Management

# Duplicate Rules

Duplicate rules enable you to define criteria that will signal potential duplicates.

Using duplicate rules, you can -

  1. Provide guidance to user viewing information when there are duplicates
  2. Define possible actions when the record being created is a potential duplicate
  3. Can limit searching for duplicates in the records that user has access to or can span the entire data set

Duplicate rules play together with 'matching rules' to do their work. Salesforce provides standard rules for accounts, person accounts, contacts and leads.

To create a duplication rule -

  1. Go to Setup > Home tab
  2. Find for Duplicate Rules or locate that under Data > Duplicate Management
  3. Click on New drop-down and select the object where you want the rule to be created
  4. Allow or block creation of potential duplicates by selecting values from picklist
  5. Enforce or bypass sharing rules by selecting the relevant options
  6. Select a matching rule against the object (it is a good time to remind you to create one & activate it if you don't have that yet)
  7. Select additional filter criteria to apply to matching rules (for e.g. do not consider accounts of status 'Long Dead')

Matching rules provide you a way to select one or more fields to compare in order to identify duplicates.

Note that -

  1. There can be more than one duplicate rule. If a potential duplicate rule is triggered by a rule, subsequent rules are skipped
  2. Max 5 duplicate rules can be created against an object
  3. Duplicate rules are not triggered in special circumstances -
    • when records are being created in 'weird ways' customer self-registration, or when using quick create
    • when restoring record from recycle bin
    • when record is being manually merged
  4. Data being created/updated through APIs or from batch routines can ignore duplicate rules and trigger their own logic

Duplicate rules enable a simple but powerful data control in organisations that allow anyone to create accounts/contacts/leads or believe in The Chaos Theory.

# Workshop

No. Type Description Time (hrs)
1 Do Explore data wizard 0.5
2 Do Explore data loader 0.5
3 Do Explore mass data changes 0.5
4 Do Explore dataloader.io/other 0.5

# Explore data wizard

  1. Export contacts
  2. Import new contacts using the contact template

# Explore data loader

  1. Export visits
  2. Create new patients if necessary
  3. Import visits using the visit template, associate visits with Doctors, patients
  4. Import bed assignment data for patients

# Explore mass data changes

  1. Change status of ‘Dormant’ customers to active if there is a visit in the last one week

# Explore Dataloader.io / Other

  1. Explore third party tools to export/import data

# Teaching Aids

# Presentation

# 1. Data Management: sf-cog

More info

← TRUST - Security Revisited →