PUBLIC
SAP HANA Platform 2.0 SPS 04
Document Version: 1.1 – 2019-10-31
SAP HANA System Replication
© 2019 SAP SE or an SAP aliate company. All rights reserved.
THE BEST RUN
Content
1 SAP HANA System Replication..................................................6
2 SAP HANA System Replication: Basic Concepts.....................................8
2.1 Introduction.................................................................9
2.2 Replication Modes for SAP HANA System Replication...................................12
2.3 Operation Modes for SAP HANA System Replication....................................14
2.4 Data Transferred to the Secondary System..........................................16
2.5 Resync Optimization..........................................................19
Data Retention........................................................... 20
Log Retention............................................................20
2.6 Network Recommendations.....................................................27
Distance Between Data Centers............................................... 27
Network Throughput.......................................................28
Data and Log Compression...................................................30
Data and Log Volume Encryption...............................................31
2.7 Secondary System Usage...................................................... 31
3 SAP HANA System Replication: Conguration .....................................34
3.1 General Prerequisites for Conguring SAP HANA System Replication....................... 35
3.2 Conguring SAP HANA System Replication ......................................... 38
Congure System Replication with the SAP HANA Cockpit............................ 39
Congure SAP HANA System Replication with hdbnsutil..............................51
Congure SAP HANA System Replication with the SAP HANA Studio.....................55
3.3 Initializing the Secondary.......................................................57
Initialize the Secondary with Storage Copy from Primary..............................58
3.4 Full Sync Option for SAP HANA System Replication....................................58
3.5 Add and Remove Hosts in SAP HANA System Replication................................60
3.6 Changing the Operation Mode...................................................61
3.7 Changing the Replication Mode..................................................62
3.8 SAP HANA System Replication Conguration Parameters............................... 62
3.9 SAP HANA System Replication Command Line Reference................................71
3.10 Disabling SAP HANA System Replication............................................74
Disable SAP HANA System Replication with the SAP HANA Cockpit......................74
Disable SAP HANA System Replication with hdbnsutil................................80
Disable SAP HANA System Replication with the SAP HANA Studio.......................81
3.11 SAP HANA System Replication Setup for XS Advanced Runtime...........................81
3.12 SAP HANA System Replication with Tenant Databases..................................83
2
P U B L I C
SAP HANA System Replication
Content
4 SAP HANA System Replication: Takeover and Failback...............................85
4.1 Takeover..................................................................86
Perform a Takeover with the SAP HANA Cockpit....................................87
Perform a Takeover with hdbnsutil..............................................91
Perform a Takeover with the SAP HANA Studio.....................................92
Client Connection Recovery After Takeover....................................... 92
Invisible Takeover and Restart.................................................97
Takeover with Handshake....................................................99
4.2 Failback..................................................................100
Perform a Failback with the SAP HANA Cockpit....................................101
Perform a Failback with the SAP HANA Studio.................................... 104
Perform a Failback with hdbnsutil............................................. 105
5 SAP HANA System Replication: Secondary Time Travel..............................107
5.1 Secondary Time Travel........................................................107
5.2 Execute Secondary Time Travel .................................................108
5.3 Execute Secondary Time Travel While Replication Continues.............................109
5.4 Conguration Parameters .....................................................110
5.5 Monitoring Secondary Time Travel................................................112
6 SAP HANA System Replication with Active/Active (Read Enabled).....................113
6.1 Active/Active (Read Enabled) System Replication.....................................114
6.2 Generic Conditions for Active/Active (Read Enabled).................................. 116
6.3 Conguring an Active/Active (Read Enabled) System Replication..........................117
Conguration Parameters...................................................118
Checking the Active/Active (Read Enabled) Conguration............................ 118
6.4 Connection Types........................................................... 119
Client Requirements For A Takeover............................................122
Hint-Based Statement Routing for Active/Active (Read Enabled)....................... 123
6.5 Memory Management........................................................124
6.6 Virtual IP Address Handling.................................................... 125
6.7 Authentication Methods...................................................... 126
6.8 Monitoring Active/Active (Read Enabled) .......................................... 127
7 SAP HANA System Replication Setups..........................................128
7.1 SAP HANA Multitier System Replication ...........................................128
Conguring SAP HANA Multitier System Replication ................................129
Performing a Takeover and a Failback in SAP HANA Multitier System Replication ............139
7.2 SAP HANA Multitarget System Replication..........................................142
Example: Congure a SAP HANA Multitarget System Replication....................... 144
Disaster Recovery Scenarios for Multitarget System Replication........................145
8 SAP HANA System Replication: Operation and Maintenance..........................147
SAP HANA System Replication
Content
P U B L I C 3
8.1 SAP HANA System Replication Details ............................................148
8.2 Alerts....................................................................152
SAP HANA System Replication Alerts...........................................153
Monitoring Secondary Systems...............................................155
Monitoring and Replicating INI File Parameter Changes..............................158
8.3 Checking the SAP HANA System Replication Status...................................159
Checking the Status with landscapeHostConguration.py............................160
Checking the Status with systemReplicationStatus.py...............................163
Checking the Status with getTakeoverRecommendation.py...........................165
Example: Checking the Status on the Primary and Secondary Systems...................166
Checking the Status with the HDB Console ...................................... 170
Checking the Status with Predened SQL Statements...............................172
8.4 Monitoring System Replication..................................................175
Monitoring SAP HANA System Replication with the SAP HANA Cockpit...................175
Monitoring SAP HANA System Replication with hdbnsutil............................ 183
Monitoring SAP HANA System Replication with the SAP HANA Studio................... 185
Monitoring SAP HANA System Replication with SQL query............................186
8.5 System Replication Network Connection...........................................187
Secure Conguration of the Network Connection.................................. 188
Encryption of the Connection.................................................191
Monitoring the Network Connection............................................192
Monitoring the Network Latency..............................................192
8.6 Copy or Move Tenants Within System Replication.....................................194
8.7 Copying a System using System Replication ........................................195
Copy a System using System Replication ........................................196
8.8 Updating SAP HANA Systems with SAP HANA System Replication........................ 196
Update SAP HANA Systems Running in a System Replication Setup.....................197
Use SAP HANA System Replication for Near Zero Downtime Upgrades...................199
9 Troubleshoot System Replication..............................................209
9.1 I/O Related Root Causes and Solutions............................................210
Analyzing I/O Throughput and Latency..........................................212
Savepoint Performance.....................................................214
9.2 Replication Performance Problems...............................................216
9.3 Setup and Initial Conguration Problems...........................................221
9.4 Intermittent Connectivity Problems..............................................225
9.5 LogReplay: Managing the Size of the Log File........................................227
9.6 SAP HANA System Replication Communication Problems..............................230
9.7 Stress Test with NIPING.......................................................231
9.8 Reference: Alerts........................................................... 232
10 Security Aspects for SAP HANA System Replication................................251
4
P U B L I C
SAP HANA System Replication
Content
10.1 Secure Internal Communication.................................................252
10.2 Secure Internal Communication Between Sites in System Replication Scenarios...............256
10.3 Legacy Conguration of Secure Internal Communication............................... 257
10.4 Congure Secure Communication (TLS/SSL) Between Primary and Secondary Sites...........259
10.5 Communication Channels.....................................................261
10.6 Network Security...........................................................263
10.7 Internal Application Encryption Service............................................266
11 SQL and System View Reference.............................................. 269
11.1 M_SERVICE_REPLICATION System View.......................................... 269
11.2 M_SYSTEM_REPLICATION System View...........................................274
11.3 M_SYSTEM_AVAILABILITY System View...........................................276
11.4 M_SYSTEM_REPLICATION_MVCC_HISTORY System View..............................278
11.5 M_SYSTEM_REPLICATION_TAKEOVER_HISTORY System View...........................279
11.6 M_LOG_SEGMENTS System View................................................281
11.7 M_SNAPSHOTS System View.................................................. 284
SAP HANA System Replication
Content
P U B L I C 5
1 SAP HANA System Replication
The SAP HANA System Replication guide is the entry point for all information related to SAP HANA system
replication.
Why should I use SAP HANA system replication?
SAP HANA system replication is a mechanism ensuring the high availability of your SAP HANA system. System
replication is SAP's recommended conguration for addressing SAP HANA outage reduction due to planned
maintenance, faults, and disasters. It supports a recovery point objective (RPO) of 0 seconds and a recovery
time objective (RTO) measured in minutes.
What can I learn about SAP HANA system replication in this guide?
Before conguring your system replication landscape, it's important to learn about basic concepts.
You will have to take decisions related to the used operation and replication modes. The selected operation
mode determines what types of data packages are sent to the secondary system. The operation mode
determines also which technique (data retention or log retention) is used to achieve a resync whenever system
replication is out of sync. The network connection between the primary and the secondary systems is also
important, because it impacts the overall performance of the systems involved in a system replication
landscape. Finally, in system replication landscapes you can run also other systems (for example, DEV, QA
systems, or even productive systems) on the secondary system’s hardware while the primary system is in
production. For information on these aspects, see SAP HANA System Replication: Basic Concepts.
After checking all the necessary prerequisites for conguring system replication, you can use the SAP HANA
cockpit, the SAP HANA studio, or the hdbnsutil command line tool to congure system replication. At this
point, it's also useful to know how to initialize the secondary system, how to change the chosen operation or
replication mode, or how to add or remove hosts. You can enable the full sync option in a synchronous system
replication to reach a true RPO=0. For information on these aspects, see SAP HANA System Replication:
Conguration. This chapter also provides an overview of the conguration parameters, a command line
reference, as well as information about SAP HANA tenant database systems in a system replication
conguration, and the system replication setup for XS advanced.
Learn next how to perform a takeover and a failback for planned or unplanned downtimes of the primary
system. You can perform a standard takeover, a takeover with handshake, or an invisible takeover. For
information, see SAP HANA System Replication: Takeover and Failback.
To quickly access again data, which was deleted in the original system, you can prepare your system replication
landscape for secondary time travel. Secondary time travel allows you to start the secondary system or the log
replay at a previous point in time. For information, see SAP HANA System Replication: Secondary Time Travel.
To support read access on the secondary system, you can congure an Active/Active (read enabled) system
replication. Active/Active (read enabled) reduces the load on the primary system, but does not double the
6
P U B L I C
SAP HANA System Replication
SAP HANA System Replication
capacity; it simply extends read capabilities. In an Active/Active (read enabled) system replication
conguration, the SQL ports on the secondary system are open for read access. This makes it possible to use
the secondary system for read-intense tasks and to have a better balance of workloads improving the overall
performance of the SAP HANA database. For more information about Active/Active (read enabled), see SAP
HANA System Replication with Active/Active (Read Enabled).
Besides the standard setup, in which a primary system ships all the data to the secondary system, you can also
congure a multitier or a multitarget system replication. In a multititer system replication, a tier 2 system
replication setup can be used as the source for adding further tiers in a chain. In a multitarget system
replication, the primary system can replicate data changes to more than one secondary system. To learn how
to congure the dierent setups and how to handle in dierent disaster recovery scenarios, see SAP HANA
System Replication Setups.
There are multiple ways to monitor system replication and to verify if the primary and secondary systems are in
sync and are running correctly:
● Alerts on the primary and secondary systems warn you of potential problems.
● To ensure rapid takeover in the event of planned or unplanned downtime, you can check the status of the
replication between the primary and the secondary systems.
● You can monitor system replication using the SAP HANA cockpit, the SAP HANA studio, the hdbnsutil
command line tool, or SQL queries.
For more information, see SAP HANA System Replication: Operation and Maintenance. This chapter also
includes information about the network connection, how to copy or move tenants in a system replication
conguration, or how to use SAP HANA system replication to update your SAP HANA systems.
Troubleshoot System Replication describes how to analyze, avoid, and solve problems related to system
replication.
Communication between systems in a system replication scenario is always authenticated. In addition, it is
possible to secure internal network communication between primary and secondary systems using TLS/SSL.
For more information, see Security Aspects for SAP HANA System Replication.
Finally, use the SQL and System View Reference chapter in this guide to have a look at the system views
relevant for system replication (for example, M_SERVICE_REPLICATION, M_SYSTEM_REPLICATION).
Related Information
SAP HANA System Replication: Basic Concepts [page 8]
SAP HANA System Replication: Conguration [page 34]
SAP HANA System Replication: Takeover and Failback [page 85]
Secondary Time Travel [page 107]
SAP HANA System Replication with Active/Active (Read Enabled) [page 113]
SAP HANA System Replication Setups [page 128]
SAP HANA System Replication: Operation and Maintenance [page 147]
Troubleshoot System Replication [page 209]
Security Aspects for SAP HANA System Replication [page 251]
SQL and System View Reference [page 269]
SAP HANA System Replication
SAP HANA System Replication
P U B L I C 7
2 SAP HANA System Replication: Basic
Concepts
Before conguring your system replication landscape, learn about basic concepts and possible setups.
What do I need to know before conguring system replication?
Learn which operation and replication modes are supported. The selected operation mode determines what
types of data packages are sent to the secondary system. The operation mode determines also which
technique (data retention or log retention) is used to achieve a resync whenever system replication is out of
sync.
The network connection between the primary and the secondary systems is also important, because it impacts
the overall performance of the systems involved in a system replication landscape. While the network
throughput requirements of the communication channel used in SAP HANA system replication are inuenced
by the used operation modes, the selected replication modes impact the network latency requirements.
Finally, in system replication landscapes you can run also other systems (for example, DEV, QA systems, or
even productive systems) on the secondary system’s hardware while the primary system is in production.
Where can I nd more information?
The following SAP Notes are relevant for a full understanding of the basic concepts described in this chapter:
SAP Notes
SAP Note Title
1999880
FAQ: SAP HANA System Replication
1969700
SQL Statement Collection for SAP HANA
1681092
Multiple SAP HANA DBMSs (SIDs) on one SAP HANA system
2211663
The license changes in an SAP HANA database after the deregistration of the
secondary site from a system replication setting
2447994
SAP HANA Dynamic Tiering Support for SAP HANA System Replication
8 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Related Information
Introduction [page 9]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
Data Transferred to the Secondary System [page 16]
Resync Optimization [page 19]
Network Recommendations [page 27]
Secondary System Usage [page 31]
2.1 Introduction
SAP HANA system replication is a mechanism for ensuring the high availability of your SAP HANA system.
What is system replication?
System replication is SAP's recommended conguration for addressing SAP HANA outage reduction due to
planned maintenance, faults, and disasters. It supports a recovery point objective (RPO) of 0 seconds and a
recovery time objective (RTO) measured in minutes.
System replication is set up so that a secondary system is congured as an exact copy of the active primary
system, with the same number of active hosts in each system. The number of standby hosts need not be
identical. Furthermore, it requires a reliable link between the primary and secondary systems.
Each service of the primary system communicates pairwise with a counterpart in the secondary system. The
main dierence to the primary system is that the secondary system does not accept requests or queries. The
secondary system can accept queries only in an Active/Active (read enabled) conguration. For more
information, see SAP HANA System Replication with Active/Active (Read Enabled).
The secondary system can be located near the primary system to serve as a rapid failover solution for planned
downtime, or to handle storage corruption or other local faults. Alternatively or additionally, a secondary
system can be installed in a remote data center for disaster recovery. The instances in the secondary system
operate in live replication mode. In this mode all secondary system services constantly communicate with their
primary counterparts, replicate and persist data and logs, and typically load data to memory. The log and data
can be compressed before shipping. For more information, see Data and Log Compression.
How does system replication work?
Once SAP HANA system replication is enabled, each server process on the secondary system establishes a
connection with its primary system counterpart and requests a snapshot of the data. From then on, all logged
changes in the primary system are replicated continuously. Whenever logs are persisted (meaning they are
written to the log volumes of each service) in the primary system, they are also sent to the secondary system.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 9
The following graphic illustrates the general system replication processes:
Note
To prevent unauthorized access to the SAP HANA database, the internal communication channels between
the primary site and the secondary site in a system replication scenario need to be protected. This may
include ltering access to the relevant ports and channels by rewalls, implementing network separation,
or applying additional protection at the network level (for example, VPN, IPSec). We recommend routing
the connection between the two sites over a special site-to-site high-speed network, which typically already
implements security measures such as separation from other network access and encryption or
authentication between sites. The details of security measures and implementation of additional network
security measures depend on your specic environment. For more information about network and security
aspects, see the SAP HANA Master Guide and the SAP HANA Security Guide.
A transaction in the primary system is not committed before the redo logs are replicated. This is determined by
the selected replication mode when setting up system replication. For a detailed description of each replication
mode, see Replication Modes for SAP HANA System Replication.
If the connection to the secondary system is lost or if the secondary system crashes, the primary system (after
a brief, congurable timeout) resumes operations. The way the received logs on the secondary system are
handled depends on the selected operation mode. For a detailed description of each operation mode, see
Operation Modes for SAP HANA System Replication.
While system replication is running, the secondary system congured identically to the primray system is on
standy until a takeover takes place.
In the event of a failure that justies a full system takeover, you switch the secondary system from the live
replication mode to a full operation mode. The secondary system, which already preloaded the same column
data as the primary system and possibly is already read enabled, becomes the primary system by replaying the
last transaction logs and then starts to accept queries.When the original system can be restored to service, it
can be congured as the new secondary system or reverted to the original conguration.
10
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Which other setups are possible?
Besides the above presented standard setup, in which a primary system ships all the data to the secondary
system, you can also congure a multitier or a multitarget system replication.
In a multititer system replication, a tier 2 system replication setup can be used as the source for replication in a
chained setup of primary system, tier 2 secondary system, and tier 3 secondary system. The primary system is
always on tier 1. The replication source for the tier 2 secondary system is the primary system, while the
replication source for the tier 3 secondary system is the tier 2 secondary. For more information, see SAP HANA
Multitier System Replication.
In a multitarget system replication, the primary system can replicate data changes to more than one secondary
system. For more information, see SAP HANA Multitarget System Replication.
What about supported endianness?
There is no support for running one site on a little-endian hardware and the other site on a big-endian system in
a system replication landscape. In SAP HANA the following endianness is supported in the corresponding SAP
HANA and OS versions:
Supported Endianness
SAP HANA Version Linux OS Version Endianness
SPS11 & SPS12 Linux Intel SLES 11 little-endian
Linux Power SLES 11 big-endian
SAP HANA 2.0 SPS00 Linux Intel SLES 12 little-endian
Linux Power SLES 12
Furthermore, system replication is supported between Intel little-endian (SAP HANA SPS12 or SAP HANA 2.0
SPS 00) and Power little-endian (SAP HANA 2.0 SPS 00).
What about license validity?
The primary system automatically replicates relevant license information to the secondary system. No
additional license needs to be installed, since the primary and secondary system have the same SID. For more
information about licensing in SAP HANA system replication, see SAP Note 2211663.
When using an Active/Active (read enabled) system replication conguration, the secondary system is subject
to licensing. For more information, see SAP HANA Feature Scope Description.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 11
Related Information
SAP HANA System Replication with Active/Active (Read Enabled) [page 113]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
Data and Log Compression [page 30]
SAP HANA Multitier System Replication [page 128]
SAP HANA Multitarget System Replication [page 142]
SAP Note 2211663
2.2 Replication Modes for SAP HANA System Replication
While registering the secondary system, you need to decide which replication mode to use.
SAP HANA oers dierent modes for the replication of the redo log:
Replication modes
Log Replication Mode
Description
Synchronous in-mem
ory (SYNCMEM)
The primary system commits the transaction after it gets a reply that the log was received by the
secondary system and stored in memory. The delay for the transaction in the primary system is
smaller, because it only includes the time for transmitting the data. The disk I/O speed on the sec
ondary system doesn't inuence the primary's performance.
When the connection to the secondary system is lost, the primary system continues the transac
tion processing and writes the changes only to the local disk.
Data loss can occur in the following situations:
● When the primary and the secondary system fail at the same time while the secondary sys
tem is connected. The data is not written to disk – neither on the primary nor on the secon
dary system.
● When a takeover is executed while the secondary system is unavailable. The data that arrived
on the secondary is outdated compared to the data on the primary.
This option provides better performance because it is not necessary to wait for disk I/O on the
secondary system, but it is more vulnerable to data loss.
12
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Log Replication Mode Description
Synchronous on disk
(SYNC)
The primary system waits with committing the transaction until it gets a reply that the log is per
sisted in the secondary system. This option guarantees immediate consistency between both sys
tems, at a cost of delaying the transaction by the time for data transmission and persisting in the
secondary system.
When the connection to the secondary system is lost, the primary system continues the transac
tion processing and writes the changes only to the local disk. No data loss occurs in this scenario
as long as the secondary system is connected. Data loss can occur, when a takeover is executed
while the secondary system is disconnected.
Additionally, this replication mode can run with a full sync option. This means that log write is suc
cessful when the log buer has been written to the log le of the primary and the secondary sys
tem. When the secondary system is disconnected (for example, because of network failure), the
primary system suspends the transaction processing until the connection to the secondary sys
tem is reestablished. No data loss occurs in this scenario. You can set the full sync option for sys
tem replication with the parameter
[system_replication]/enable_full_sync.
Note
If SAP HANA system replication runs in the SYNC replication mode with the full sync option
enabled, and if the connection to the secondary site is interrupted, no write operations on the
primary site are possible. The operation of creating a tenant database, for example, will wait
until the connection to the secondary is reestablished or the SQL statement times out.
For more information about how to enable the full sync option, see Full Sync Option for System
Replication.
Asynchronous
(ASYNC)
The primary system commits the transaction after sending the log without waiting for a response.
Here we have no delay because the data transmission is asynchronous to the transaction in the
primary system.
This option provides better performance because it is not necessary to wait for log I/O on the sec
ondary system. Database consistency across all services on the secondary system is guaranteed.
However, it is more vulnerable to data loss. Data changes may be lost during takeover.
Note
If you plan to add SAP HANA dynamic tiering to your landscape in the future, please check supported
replication modes in SAP Note 2447994 before you enable SAP HANA system replication.
The replication mode can be changed without going through a full data shipping from the primary system to
the secondary system afterwards. For more information, see Changing the Replication Mode.
Related Information
Full Sync Option for SAP HANA System Replication [page 58]
Changing the Replication Mode [page 62]
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 13
2.3 Operation Modes for SAP HANA System Replication
While registering the secondary system, you need to decide in which operation mode to run SAP HANA system
replication.
System replication can be run in three operation modes: delta_datashipping, logreplay or
logreplay_readaccess. Depending on the congured operation mode, the database sends dierent types
of data packages to the secondary system. For more information, see
Data Transferred to the Secondary
System.
Operation Mode Description
delta_datashipping
This mode establishes a system replication where occasionally (per default every 10 minutes) a
delta data shipping takes place in addition to the continuous log shipping.
The secondary system persists the received log entries but it does not replay them until it has to
take over. To shorten the log replay time, data snapshots are transmitted from time to time from
the primary to the secondary system. The data snapshots are transferred asynchronously as dif
ferential backups (data backup deltas) triggered by the secondary system, which asks for a data
backup delta with changes since the last one. During takeover the redo log needs to be replayed
up to the last arrived delta data shipment.
logreplay
In this operation mode, a redo log shipping is done after system replication was initially cong
ured with one full data shipping.
The redo log is continuously replayed on the secondary system immediately after arrival making
this step superuous during takeover. Since the log is continuously replayed, the secondary sys
tem can take over immediately, if the primary system fails. With continuous log replay, the log en
tries are sent from the redo log buers in memory. When the secondary system is temporarily
disconnected, the primary system must not claim and overwrite the log segments that have not
been replicated yet. This is achieved by retaining these log segments up to a congurable maxi
mum retention size. When the maximum retention size is reached, the log segments are re
claimed and overwritten with new log to prevent a standstill of the primary system. After such a
situation, a full data snapshot needs to be transferred again, when the secondary system is con
nected again.
Because this operation mode does not require delta data shippings, the amount of data that
needs to be transferred to the secondary system is reduced.
logreplay_readaccess
This mode is required for replication to an Active/Active (read enabled) secondary system.
Using this operation mode while conguring your system replication, read access becomes possi
ble on the secondary system by establishing a direct connection to the secondary system or by
providing a SELECT statement from the primary system with a HINT. For more information , see
also Client Support for Active/Active (Read Enabled) and SAP HANA SQL Reference Guide.
This operation mode is similar to the logreplay operation mode regarding the continuous log
shipping, the redo log replay on the secondary system, as well as the required initial full data ship
ping and the takeover. As with the logreplay operation mode, the redo log is replicated to the sec
ondary system and continuously replayed to keep the secondary system synchronized.
14 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Note
In a multitier or multitarget system replication it is not possible to combine the logreplay and
delta_datashipping operation modes.
Limitations
Before you begin preparing a replication strategy for an SAP HANA system, consider the following important
aspects regarding the operation modes logreplay and logreplay_readaccess:
● Registering a secondary with operation mode logreplay against a primary running on an SAP HANA
revision less than or equal to SPS10 will not work, because the primary does not yet support this feature.
Furthermore, for operation mode logreplay_readaccess the primary must be running on a revision
SAP HANA 2 SPS00 or higher.
● Only the operation mode delta_datashipping will work when registering the original primary (failback)
after upgrade of the secondary during a near zero downtime upgrade from an SAP HANA revision less than
or equal to SPS 10 to SPS 11, because the former primary’s version does not yet support logreplay.
● If the connection to the secondary is not available, the primary system will keep writing the redo log
segments in the online log area to be prepared for the delta log shipping after the connection is
reestablished. These log segments are marked as RetainedFree until the secondary is in sync again. In this
case there is a risk that the log volume may run full. To prevent this:
○ If a secondary is not used anymore, it must be unregistered (sr_unregister).
○ If a takeover to the secondary was done, the former primary should be disabled (sr_disable).
For more information, see How to Avoid Log Full Situations in LogReplay: Managing the Size of the Log File.
● The logreplay operation modes do not support history tables.
Note
If you plan to add SAP HANA dynamic tiering to your landscape in the future, please check supported
operation modes in SAP Note 2447994 before you enable SAP HANA system replication.
Note
When selecting an operation mode, keep in mind that the selected operation mode impacts the network
throughput requirements of the communication channel used in SAP HANA system replication. For more
information about this, see Network Recommendations.
For information about how to change the operation mode, see Changing the Operation Mode.
Related Information
Data Transferred to the Secondary System [page 16]
Active/Active (Read Enabled)
SAP HANA System Replication Command Line Reference [page 71]
LogReplay: Managing the Size of the Log File [page 227]
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 15
Network Recommendations [page 27]
Changing the Operation Mode [page 61]
2.4 Data Transferred to the Secondary System
The selected operation mode determines what types of data packages are sent to the secondary system.
When system replication is congured, the following types of data packages can be sent to the secondary
system:
● Initial full data shipping
When system replication is congured, a full set of data created as an SAP HANA in-place snapshot on the
disk of the primary system is initially sent.
● Delta data shipping
When using the delta_datashipping operation mode, the data that has changed since the last full or
the last delta data shipping is transported from the data area of the primary system to the data area of the
secondary system. The default time is every 10 minutes.
When using logreplay and logreplay_readaccess, delta data shippings are not required.
● Continous redo log shipping
Every committing write transaction on the primary system generates redo log buers, which are
continuously sent to the secondary system.
The following graphic illustrates this trac on the transportation channel between the primary and the
secondary system for the delta_datashipping operation mode:
16
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
The following graphic illustrates this trac on the transportation channel between the primary and the
secondary system for the logreplay and logreplay_readaccess operation modes:
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 17
Note
With the ini le parameter datashipping_parallel_channels (default 4), the full and the delta data
shipping are done using parallel network channels. You can change it on the secondary system in the
global.ini section [system replication]. For more information about conguration parameters,
see SAP HANA System Replication Conguration Parameters.
Related Information
SAP HANA System Replication Conguration Parameters [page 62]
Troubleshoot System Replication [page 209]
System Replication Network Connection [page 187]
18
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
2.5 Resync Optimization
Whenever the primary and the secondary system are disconnected, SAP HANA system replication is out of
sync. To get in sync again, a shipping of the missing data is initiated.
The system tries to avoid a full data shipping and to achieve a resync with a delta data or a log shipping. To get
the primary and the secondary system in sync again, their persistencies (that is, the data and log volumes)
must be compatible. The system that is to be registered as the secondary system checks if its persistence is
compatible with the primary system. If this check succeeds, a delta shipping can be carried out instead of
requesting a full data shipping from the primary system.
A maximum of three checks are executed by the secondary system in the following order:
1. Check if the newest savepoint is compatible.
The to-be secondary system checks if its newest savepoint is compatible. This check most likely succeeds
if the secondary system has just been shut down for a short time.
2. Check if the newest replication snapshot is compatible:
Replication snapshots are written on the system replication primary and secondary system while the
replication is up and running. The replication snapshots are created on the secondary system each time a
savepoint is written. On the primary system, the replication snapshots are created periodically (time and
volume-based) to preserve a state that is known to be shipped to the secondary system. As the snapshot
verication takes some time, a replication snapshot that is not yet veried to be shipped may have been
created on the primary system.
This check most likely succeeds after a test takeover on the secondary system because this state has to be
available also on the primary system.
3. Check if the active replication snapshot is compatible:
The active replication snapshot is a special replication snapshot created on the primary system and
veried to be shipped to the secondary system. This check most likely succeeds during a failback
operation because it's created on the old primary and the snapshot is veried to be shipped.
The rst savepoint or snapshot that is compatible with the primary system will be used for delta data shipping.
If none of the three savepoints or snapshots are compatible, then a full data shipping will automatically be
carried out.
Note
If system replication is out of sync and you need to register again the initial secondary system, use the
command hdbnsutil –sr_register. It is not needed to unregister the secondary system before
registering it again. Unregistering the initial secondary system before registering it would hinder an
optimized resync and would trigger a full data shipping.
Depending on the chosen operation mode, two dierent techniques are in place to achieve a resync: data
retention and log retention. For more information, see Data Retention or Log Retention
Related Information
Data Retention [page 20]
Log Retention [page 20]
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 19
2.5.1 Data Retention
Data retention is the technique used to resync the disconnected systems when using the
delta_datashipping operation mode.
With delta_datashipping, the availability of the last snapshot, that was successfully sent to the secondary
system, impacts the type of data sent by the primary system:
● If the last snapshot successfully sent to the secondary system is still available, the primary system sends
the incremental data to get in sync again.
● If it is no longer available, a full data shipping is necessary to get in sync again.
The datashipping_snapshot_max_retention_time parameter with a default of 300 minutes species for
how long the primary system will keep the snapshot. For more information about system replication
parameters, see SAP HANA System Replication Conguration Parameters.
Note
If the connection to your secondary system is temporarily lost for a longer time than the default setting of
the datashipping_snapshot_max_retention_time parameter, a delta data shipping is still possible.
In the following two use cases this parameter determines whether a full data shipping is necessary or a delta
data shipping will be sucient:
● The failback scenario
After a takeover of the secondary system, the primary system is registered as a new secondary.
● The re-register scenario
After a takeover of the seondary system, this system is re-registered as a secondary to the original system.
Related Information
SAP HANA System Replication: Takeover and Failback [page 85]
SAP HANA System Replication Conguration Parameters [page 62]
2.5.2 Log Retention
With the logreplay and logreplay_readaccess operation modes, log segments can be marked as
retained so that they can sync a secondary system after being disconnected.
With continuous log replay, delta data shipping cannot be used to sync a secondary system anymore. This is
because although the primary's and secondary's persistence are logically compatible, they are no longer
physically compatible. This means the data that is contained in the persistence is the same, but the layout of
the data on pages can be dierent on the secondary system. Therefore, a secondary system can sync via delta
log shipping only. This happens, for example, in the following use cases:
● The secondary system has been disconnected for some time (for example, because of a network problem
or temporary shutdown of the secondary system).
20
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
● A former primary system has been registered for failback.
The secondary system only uses the log of the online log area of the primary system for re-syncing. The log
must be retained for a longer time period than in the delta_datashipping mode to be able to sync the
secondary system. If getting in sync again doesn't work with delta log shipping (for example, because the log
has been reused), a full data shipping becomes necessary. To avoid this, the concept of log retention has been
introduced.
For more information on log retention in dierent scenarios, see Log Retention for Secondary Disconnect, Log
Retention for Failback, and Log Retention and Multitarget System Replication.
The following parameters are signicant for log retention:
● Use the enable_log_retention parameter to enable or disable log retention.
● Use the logshipping_max_retention_size parameter to specify how the system behaves when many
log segments of the type RetainedFree are created.
For a full description of the parameters, see SAP HANA System Replication Conguration Parameters.
Related Information
Log Retention for Secondary Disconnect [page 21]
Log Retention for Failback [page 22]
Estimating the Maximum Retention Time [page 23]
Log Retention and Multitarget System Replication [page 25]
SAP HANA System Replication Conguration Parameters [page 62]
2.5.2.1 Log Retention for Secondary Disconnect
When a secondary system congured with the operation mode logreplay or logreplay_readaccess is
disconnected, the primary system will not reuse the log segments in the online log area that are required to
sync the secondary system using delta log shipping.
These log segments are marked as RetainedFree until the secondary has successfully synced again. If a
secondary system is stopped, the log volume will grow on the primary system until the log volume has lled up
with log segments. Once the secondary system reconnects and has synced the missing log, these log
segments are then set to Free and can be reused after that. This behavior is automatically turned on, if a
secondary system with the operation mode logreplay or logreplay_readaccess is registered.
Log segments are retained on the primary as long as the secondary system is registered, but not connected to
the primary system.
Note
Therefore, if a secondary system is shut down and not used for a longer period of time, unregister it rst, to
prevent log volumes from accumulating on the primary system. However, in such a case a full data shipping
will be necessary when the system reconnects.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 21
Note
Log full means that no more log segments can be created in the log volume, because the log segment
directory is full. Currently the number of log segments is roughly 10000. Disk full means that the disk is full,
which is not necessarily the case in the log full situation. Log retention usually deals with disk full situations,
because with 10000 log segments having each a size of 1 GB you can create 10 TB of log segments.
To understand for how long the SAP HANA system replication landscape will survive before running into a log
full situation, see Estimating the Maximum Retention Time.
The logshipping_max_retention_size parameter determines if a full log volume can be prevented at the
price of a possibly necessary full data shipping when the system reconnects. The value of this parameter
(default is 1 TB per log volume) should not exceed the size of the le system reserved for all log volumes.
Related Information
Estimating the Maximum Retention Time [page 23]
2.5.2.2 Log Retention for Failback
On the secondary system, log retention is required to do a failback with optimized data transport.
The primary system periodically creates persistence snapshots during replication and provides the log position
information to the secondary system. After takeover, when the old primary is started again as a secondary, the
most recent snapshot is opened on the old primary system and the missing log is requested from the new
primary system.
Log retention can occur in two situations:
● While replication is active
During replication, the secondary system keeps all log starting from the last snapshot position provided by
the primary system. The old log is automatically released after a new snapshot has been created on the
primary system. This behavior is turned on by default and it ensures that during replication only a few
RetainedFree segments are kept online. They are needed to ll the gap between the primary snapshot
and the current potential takeover log position.
● After a takeover
After takeover, the new primary has to keep the log until a new secondary system is registered and has
synced the missing log. Because syncing can take some time, this behavior has to be turned on explicitly
on the new primray system as follows:
global.ini/[system_replication]/enable_log_retention = on
Recommendation
If you have a setup in which there will be frequent failbacks between two systems, we recommend that
you set the following parameter on both system to simplify the conguration: global.ini/
[system_replication]/enable_log_retention = on
22
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Note
If the old primary system will not be reused as the new secondary system (failback), it should be
disabled after the takeover using hdbnsutil -sr_disable to prevent log volumes from
accumulating on the new primary system. You can disable it with SAP HANA cockpit, SAP HANA
studio, or using the command line.
2.5.2.3 Estimating the Maximum Retention Time
How long will your SAP HANA system replication landscape survive in a disconnect scenario before running
into a log full situation, when logreplay or logreplay_readaccess is congured?
You can use the SAP HANA cockpit to get an estimation of the maximum retention time.
After opening the system replication tile from the system database overview page, the System Replication
Overview section provides information either about the Estimated Log Full Time or the Estimated Log Retention
Time.
The Estimated Log Full Time value shows you how much time is left before the primary system runs into a log
full situation.
The Estimated Log Retention Time value shows you how much time is left before the primary system starts to
overwrite the RetainedFree marked log segments making a full data shipping necessary for resync.
Example
The screenshot below shows you where to nd the relevant values on the System Replication Overview:
The smallest calculated value from the Estimated Log Retention Time and Estimated Log Full Time columns
will be displayed. In the screenshot above, you see an estimated log full time of 564 days. This means that
the indexserver of the MB1 tenant will run into a log full in 564 days.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 23
For more information, open the LOG REPLAY tab and look at the Estimated Log Full Time and Estimated Log
Retention Time columns. In this services list, one row is shown for each service that is relevant for replication,
because it has its own persistence consisting of data and log volume. The calculation is based on the value of
the logshipping_max_retention_size parameter, the write load on the primary system (taken from the
log backup size), and the available disk space. The columns provide the log retention details only for the log
volumes.
For a full description of each column, see the table below:
Column Name Description
Volume ID The volume ID of the service
Volume Sub Path The last two directories of the le system path where the log volume of this service is located
Host The host where the service is running
Service The relevant service for system replication (for example, indexserver, nameserver, xsengine).
Only services that have their own persistence are relevant for replication.
Port The port number
Database Name The database name
Estimated Log Retention
Time:
Maximum number of days or hours on which the logs can be retained to resync the primary
and the secondary after the replication was interrupted without requiring a full data shipping
Estimated Log Full Time Maximum number of days or hours until the log volume runs full when created redo logs can
no longer be reused
Congured Log Retention
Size (GB)
Determined by the value of the logshipping_max_retention_size parameter that is
valid for this service. For example, the setting of this parameter in the indexserver.ini on the
host on which the service is running overwrites the setting of this parameter in global.ini.
Log Backup Size per Day
(GB)
The maximum log backup size per day indicating which replication relevant write load occurred
for this service.
Disk ID Internal Device ID (optional information)
Free Disk Space (GB) Optional information
24 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
2.5.2.4 Log Retention and Multitarget System Replication
When the primary system replicates data changes to more than one secondary system, you should use force
log retention and log retention propagation to reach an optimized re-sync and avoid a full data shipping after
takeover or other disconnect situations.
Force log retention
Force log retention is used on a system to retain log until it's actively disabled. To use force log retention, enter
the value force_on_takeover for the enable_log_retention conguration parameter.
If enable_log_retention = force_on_takeover is congured, the log will be retained during replication
for all direct secondaries until a takeover is executed. During takeover, the parameter is set to
force. This
means the log will be retained independently of any secondary system.
Example
A typical scenario is described in the following steps:
1. Congure all systems with [system_replication]/enable_log_retention =
force_on_takeover
2. During takeover on a secondary system, if force_on_takeover is set, the value is changed to
enable_log_retention = force. This means that starting from the takeover, the log is retained
until it's explicitly disabled.
3. Re-register all required systems until the landscape is fully functional again.
4. Reset [system_replication]/enable_log_retention = force_on_takeover on the system
on which takeover has been executed before re-establishing the original conguration.
The conguration must be done manually (for example, by the administrator or using setup scripts) because
the SAP HANA system doesn't know when the system landscape has been completely recongured.
Log retention propagation
Log retention propagation is used to retain the log based on the smallest savepoint log position in the whole
system replication landscape. Log retention propagation should be enabled if you want to re-order your
systems in a complex system replication setup.
This can be done by setting the following parameter in global.ini: [system_replication]/
propagate_log_retention = on. If you want to propagate log retention in a system replication landscape
between all systems, this parameter should be set on all systems in the landscape.
When you set this parameter on a system, it behaves as follows:
● The system sends to its source system the minimum log position of its own savepoint and the retained log
position it gets from all direct secondaries as retained position.
● The system sends the minimum log position of its own savepoint and the retained log position it gets from
all direct secondaries to the secondaries as retained log position.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 25
● The system uses the minmum log position it gets from all direct secondaries and its source system (if not
primary) as own retained log position.
Example
To explain these concepts we are using the setup described in Multitarget System Replication. In this setup,
primary system A replicates data changes to secondary system B located in the same data center. Primary
system A also replicates data changes to the secondary system C located in another data center. Secondary
system C is a source system for a further secondary system D located in the same data center with system C.
For a quick overview, use the graphic below:
If there is a takeover on secondary system B, you must register system C to B and system A to B to recreate the
original conguration. To avoid a full data shipping for both systems, system B must retain all the log until
systems A and C have synced again. This can't be accomplished by setting global.ini/
[system_replication]/enable_log_retention = on because system B doesn't know how many
systems must be re-attached until the landscape is back in its functional state.
Force log retention should be used on system B until systems A and C are registered again and synced.
If you want to re-order your systems, enable log retention propagation. Log retention without propagation only
aects the direct neighbors. For example, if system D is stopped in this setup, system C retains log for D, but
not for A and B. If system D is re-attached to systems B or A and propagation is not turned on, log could be
missing because systems A and B do not retain their log with respect to D.
26
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Related Information
SAP HANA Multitarget System Replication [page 142]
Disaster Recovery Scenarios for Multitarget System Replication [page 145]
2.6 Network Recommendations
The network connectionn between the primary and the secondary system impacts the overall performance of
the SAP HANA systems involved in an SAP HANA system replication landscape.
The network throughput requirements of the communication channel used in SAP HANA system replication
are inuenced by the used operation modes. For more information, see Network Throughput.
The network latency requirements are inuenced by the used replication modes. For example, for a
synchronous replication mode the performance of the primary system is inuenced by the time it takes until
the acknowledgement from the secondary system arrives. For more information, see Distance Between Data
Centers.
For more information about monitoring and conguring the network connection between the primary and the
secondary systems for SAP HANA system replication, see System Replication Network Connection.
Related Information
Distance Between Data Centers [page 27]
Network Throughput [page 28]
Data and Log Compression [page 30]
Data and Log Volume Encryption [page 31]
System Replication Network Connection [page 187]
2.6.1 Distance Between Data Centers
The network latency is inuenced by the selected replication mode.
If the distance between your sites is less than 100 km, use the synchronous replication modes: SYNC or
SYNCMEM.
If the data centers are more than 100 km apart, the asynchronous replication mode ASYNC is recommended.
For more information, see the Check Network Conguration (Long Distance) section in Replication Performance
Problems.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 27
Related Information
Replication Performance Problems [page 216]
Replication Modes for SAP HANA System Replication [page 12]
Monitoring the Network Latency [page 192]
2.6.2 Network Throughput
The requirements regarding the network throughput are inuenced by the selected operation mode.
The selected operation mode inuences the size of the shipped data over the network. To estimate the required
throughput, you need to know the size of the data and log that are generated during your daily workload. You
can gain this information using one of the SQL statements contained in the zip le attached to the SAP Note
1969700 as follows:
1. Open the HANA_Replication_SystemReplication_Bandwidth.txt from the SQL Statements.zip le.
2. Copy the statement to the SQL Console.
3. Change the Modication section and execute.
Example
You can change this section so that the results are displayed per day.
The displayed results provide the following information:
28
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Tab Name Description
SNAPSHOT_TIME Time slot for which the results are valid
HOST Host name
PERSISTENCE_GB (Current) persistence data size (GB)
DATA_SIZE_GB Total size of data written to disk (GB)
LOG_SIZE_GB Total size of logs generated (GB)
TOTAL_SIZE_GB Total amount of data and logs generated (GB)
LOG_PCT Percentage of log compared to total size (%)
AVG_BANDWIDTH_MBIT Average required network bandwidth to replication site (Mbit). It is only available for
certain TIME_AGGREGATE_BY values.
SIMPLE_BANDWITH_MBIT Simple network bandwidth calculation (Mbit) based on the formula that it should be
possible to ship the persistence once per day.
As mentioned above, the requirements regarding the network throughput depend on the selected operation
mode. PERSISTENCE_GB and LOG_SIZE_GB are the most important values in this context. The following
overview distinguishes between the operation modes making use of these values:
Operation mode Throughput requirements
delta_datashipping It must be possible to transport the size of the persistently stored data within one day
from the primary system to the secondary system. The size of the persistently stored
data can be obtained from the above-mentioned PERSISTENCE_GB column.
Example
Given: 4,3 TB of persistently stored data
Throughput: 4,3 TB/day <-> ~ 50 MByte/s => ~ 0,5 GBit/s connection required
logreplay & logreplay_readac
cess
It must be possible to transport the size of the log backups of one day from the primary
system to the secondary system within one day.
With logreplay and logreplay_readaccess a pure log shipping is done, since
no delta data shippings are necessary. The network throughput requirements depend
mainly on the transactional workload on your primary system.
The LOG_SIZE_GB column indicates the log size that was created per day. During a nor
mal system replication operation, your network channel must be capable of handling
this amount of data per day.
For more information about operation modes, see Operation Modes for SAP HANA System Replication.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 29
Related Information
SAP Note 1969700
Operation Modes for SAP HANA System Replication [page 14]
2.6.3 Data and Log Compression
Data and log compression can be congured to reduce the amount of trac between systems especially over
long distances (for exmaple, when using the ASYNC replication mode).
Data and log compression can be used for the initial full data shipping, the sub sequential delta data shipping,
as well as for the continuous log shipping.The following types of compression for log and data shipping are
supported:
● Log
○ Log buer tail compression
○ Log buer content compression
● Data
○ Data page compression
Log Buer Tail Compression
All log buers are aligned to 4kb boundaries by a ller entry. With log buer tail compression the ller entry is
cut o from the buer before sending it over the network and added again when the buer has reached the
secondary system. So, only the net buer size is transferred to the secondary system.
The size of the ller entry is less than 4kb, this is the maximum size reduction per sent log buer. If the size of
the log buers is large, the compression ratio is limited. Log buer tail compression is turned on by default.
Log Buer and Page Content Compression
Log buers and data pages shipped to the secondary system can be compressed using a lossless compression
algorithm (lz4). By default content compression is turned o. You can turn it on by setting the following
conguration parameters on the secondary system in the system_replication section of the global.ini le:
● enable_log_compression = true
● enable_data_compression = true
Log buer content compression works also in combination with log buer tail compression. So, only the
content part of the log buer is compressed, without considering the ller entry.
30
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Related Information
External link to LZ4
2.6.4 Data and Log Volume Encryption
Depending on the version of your SAP HANA database, you must enable data volume encryption dierently.
Before SAP HANA 2.0 SPS02, data volume encryption was enabled by setting the ini parameter global.ini/
persistence/data_encryption=true.
Depending on the conguration of the inile checker, this change was either automatically replicated to the
secondary (for the setting globel.ini/[inifile_checker]=replicate) or you had to set it manually
there as well. For more information, see Monitoring and Replicationg INI File Parameter Changes.
As of SAP HANA 2.0 SPS02, you must change the root keys used for data volume encryption and log volume
encryption only in the primary system. The new keys will be propagated to all secondary systems. Enable or
disable the data and log volume encryption in the primary system only. The setting will be propagated to all
secondary systems as well. For details on enabling data and log volume encryption, see
Enable Data and Log
Volume Encryption in an Existing SAP HANA Database.
Related Information
Monitoring and Replicating INI File Parameter Changes [page 158]
Enable Data and Log Volume Encryption in an Existing SAP HANA Database
2.7 Secondary System Usage
In system replication landscapes, you can run other systems (for example, DEV, QA systems, or even
productive systems) on the secondary system’s hardware while the primary system is in production.
Recommendations
When running other systems on the secondary system’s hardware, keep in mind the following
recommendations:
● We recommend using a separate storage infrastructure for each system. Since the secondary system
requires the same I/O capacity as the primary, the additional systems could have a negative impact on the
secondary’s I/O.
● The SIDs and instance numbers used for the additional systems running on the secondary hardware must
be dierent from the system replication SID.
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 31
● The <instance number>+1 of the productive system must not be used and must be free on both sites,
because this port range is used for the system replication communication.
● To save memory resources, switch o the preload of tables on the secondary system using global.ini/
[system_replication]-> preload_column_tables=false.
The takeover process will take longer as no data is preloaded to memory on the secondary system.
● The available memory on the secondary system’s hardware must be shared among the systems running
there. However, the secondary system should receive the amount of memory it needs. Only the remaining
memory can be shared among the DEV or QA systems.
To allocate memory for every system running on the secondary’s hardware, set the global allocation limit
for each system using global.ini/[memorymanager]-> global_allocation_limit.
● If there are not enough resources to handle the load of all systems, the additional systems running on the
secondary need to be shut down in case of a takeover.
● You can change the global.ini on the secondary accordingly and then activate the change with hdbnsutil
–reconfig because no SQL is possible in this state.
Resources required on the secondary system
When planning to run other systems on the secondary system, you need to consider the available hardware
resources, the table preload option (preload_column_tables), and the memory
(global_allocation_limit) needed by the secondary system.
The tables below describe these requirements for each combination of table preload and operation mode:
Operation Mode: delta_datashipping
Preload Memory needed for the secondary system (global_allocation_limit)
On
Set the global_allocation_limit to the same value as the memory available on the primary system.
O minimum 64 GB
or
row store size + 20 GB (if this sum is higher)
Determine the row store size with the following statement:
select host, round (sum(page_size*USED_BLOCK_COUNT)/1024/1024/1024,2) as
"RowStore Size GB" from m_data_volume_page_statistics where page_sizeclass
= '16k-RowStore' group by host;
If this limit is not set, the SAP HANA database on the secondary system uses as much memory as it can get and
possibly takes it away from the DEV/QA systems, which could run into out-of-memory.
Note
If the row store size of the primary system grows during operation, it might become necessary to increase
the global_allocation_limit on the secondary system.
32
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
Operation Mode: logreplay
Preload Memory needed for the secondary system (global_allocation_limit)
On
Set the global_allocation_limit to the same value as the memory available on the primary system.
O size of loaded column tables (in-memory) + row store size + 50 GB
Determine the size of the loaded column tables (in-memory) with this SQL statement:
select round(sum(memory_size_in_total)/1024/1024/1024) size_GB from
m_cs_tables;
Note
If the row store size of the primary system grows during operation, it might become necessary to increase
the global_allocation_limit on the secondary system.
Operation Mode: logreplay_readaccess
Preload Memory needed for the secondary system (global_allocation_limit)
On
Set the global_allocation_limit to the same value as the memory available on the primary system.
Note
Read access on the Active/Active (read enabled) secondary system requires additional CPU capacity.
O
Set the global_allocation_limit to the same value as the memory available on the primary system.
Note
This load option reduces the required memory size for the Active/Active (read enabled) secondary thanks
to tables which are used read-only in the primary system. However, read access on the Active/Active (read
enabled) secondary system requires additional CPU capacity.
All resources that are not needed by the secondary system can be used to run further systems on the
secondary. These resources must be granted to them by explicitly setting the global allocation limit.
For more information on multiple SAP HANA systems running productively on one host, see SAP Note 1681092.
The note also provides information about the cross replication possibility of two systems with two hosts.
Related Information
SAP Note 1681092
SAP HANA System Replication
SAP HANA System Replication: Basic Concepts
P U B L I C 33
3 SAP HANA System Replication:
Conguration
After checking all the necessary prerequisites for conguring system replication, use the SAP HANA cockpit,
the SAP HANA studio, or the hdbnsutil command line tool to congure system replication.
How can I congure and disable system replication?
After checking all the necessary prerequisites, you can use the SAP HANA cockpit, the SAP HANA studio, or
the hdbnsutil command line tool to congure system replication. This section describes how to congure
and disable system replication. Generally, you have to perform the following steps:
● Perform an initial data backup or a storage snapshot.
● Enable the primary system for system replication.
● Establish a connection between the secondary and the primary systems.
● Initiate a full data replication by conguring system replication on the secondary and starting it. Thereafter,
incremental data replication (only in delta_datashipping operation mode) and continuous redo log
replication (in all operation modes) start automatically.
● Disable system replication on the secondary system.
● Disable system replication on the primary system.
This chapter describes also how to initialize the secondary system, how to change the chosen operation or
replication mode, or how to add and remove hosts. Furthermore, it shows you how to enable the full sync
option in a synchronous system replication to reach a true RPO=0. This chapter also provides an overview of
the conguration parameters, a command line reference, as well as information about SAP HANA tenant
database systems in a system replication conguration, and the system replication setup for XS advanced.
Where can I nd more information?
The following SAP Notes are relevant for a full understanding of the basic concepts described in this chapter:
SAP Notes
SAP Note Title
2211663
The license changes in an SAP HANA database after the deregistration of the
secondary site from a system replication setting
2369981
Required conguration steps for authentication with HANA System Replica
tion
34 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
SAP Note Title
2300936
Host Auto-Failover & System Replication Setup with SAP HANA extended
application services, advanced model
2447994
SAP HANA Dynamic Tiering Support for SAP HANA System Replication
611361
Hostnames of SAP servers
1945676
Correct Usage of hdbnsutil - sr_unregister
2300936
Host Auto-Failover & System Replication Setup with SAP HANA extended
application services, advanced model
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Conguring SAP HANA System Replication [page 38]
Initializing the Secondary [page 57]
Full Sync Option for SAP HANA System Replication [page 58]
Add and Remove Hosts in SAP HANA System Replication [page 60]
Changing the Operation Mode [page 61]
Changing the Replication Mode [page 62]
SAP HANA System Replication Conguration Parameters [page 62]
SAP HANA System Replication Command Line Reference [page 71]
Disabling SAP HANA System Replication [page 74]
SAP HANA System Replication Setup for XS Advanced Runtime [page 81]
SAP HANA System Replication with Tenant Databases [page 83]
3.1 General Prerequisites for Conguring SAP HANA
System Replication
Before you congure SAP HANA system replication, several prerequisites must be fullled.
● The primary and secondary system are both installed and congured. You have veried that both are
independently up and running.
● SAP HANA systems can only be replicated as the whole system. This means that the system database and
all tenant databases are part of the system replication. A takeover can only be performed as a whole
system. A takeover on the level of a single tenant database is not possible.
● The conguration of active hosts in the primary and secondary system must be the same. This means that
the number of active hosts, the names of the host roles, failover groups, and worker groups must be
identical in both systems. This implies that if there is a standby host on the primary system it need not be
available on the secondary system and vice versa.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 35
● Check that the host names in the primary system are dierent to the host names used in the secondary
system. You can see the SAP HANA host name for each host in the environment variable
SAP_RETRIEVAL_PATH (/usr/sap/<SID>/HDB<InstNo>/<hostname>) and with the python script
landscapeHostConfiguration.py. For more information, see Host Name Resolution for System
Replication and Checking the Status with landscapeHostConguration.py.
If the host names of the primary and the secondary system are the same (for example, because two
systems are used that have identical host names), change the host names used on the secondary system.
For more information, see Rename an SAP HANA System Host.
● System replication between two systems on the same host is not supported.
● Both systems should run on the same endianness platform.
● You are logged on to both systems as the operating system user (user <sid>adm) or you have provided its
credentials when prompted.
● You need the operating system user to set up a system replication landscape, to perform a takeover and a
failback, as well as to disable system replication with the SAP HANA cockpit. For more information, see
Operating System User <sid>adm and Connect to a Resource using Database Credentials.
● The secondary system must have the same SAP system ID (<SID>) and instance number as the
primary system.
Note
The primary system replicates all relevant license information to the secondary system. An additional
license is not required. For more information, see SAP Note 2211663.
● All conguration steps have to be executed on the master name server node only.
● The .ini le conguration must be similar for both systems. Any changes made manually, or by SQL
commands on one system should be manually duplicated on the other system.
Automatic conguration parameter checks will alert you to conguration dierences between the two
systems.
Note
To keep the ini le conguration similar on both systems, the INI parameter checker is per default
congured to check for dierences. Additionally, it can be congured to replicate parameter changes
from the primary system to the secondary system.
● Ensure that log_mode is set to normal in the persistence section of the global.ini le. This mode means
that the log segments are backed up.
● During an upgrade of the system replication landscape, the software version of the current secondary
system must be equal or newer to the version of the current primary system.
Note
During a failback, the roles of your systems in the system replication landscape switch. Make sure in
this case that your primary system does not have a newer software version than the secondary system.
Note
For Active/Active (read enabled) setups, the SAP HANA versions must be the same on the primary and
the secondary system. Use this setup mainly during the upgrade process of the system replication
landscape.
36
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
● To secure the system replication communication channel between the primary and the secondary system,
congure the ini parameters [system_replication_communication] / listeninterface and
allowed_sender as described in Host Name Resolution for System Replication.
Note
If you plan to add SAP HANA dynamic tiering to your landscape in the future, please seeSAP Note
2447994 before you enable HANA system replication. SAP HANA dynamic tiering requires certain
communication ports, operation modes, and replication modes.
● If a new tenant database is created in a running SAP HANA system replication, it must be backed up to
participate in the replication. Afterward, the initial data shipping is started automatically for this tenant
database. If a takeover is done while the initial data shipping is running and not nished, this new tenant
database will not be operational after takeover and will have to be recovered with backup and recovery (see
the SAP HANA Database Backup and Recovery section of the SAP HANA Administration Guide).
● Before you congure SAP HANA system replication, you must copy the system PKI SSFS .key and the .dat
le from the primary system to the secondary system:
/usr/sap/<SID>/SYS/global/security/rsecssfs/data/SSFS_<SID>.DAT
/usr/sap/<SID>/SYS/global/security/rsecssfs/key/SSFS_<SID>.KEY
For more information, see SAP Note 2369981 Required conguration steps for authentication with HANA
System Replication.
If you installed XS advanced, you must also copy the XSA SSFS .key and the .dat le from the primary
system to the secondary system in the following directories:
/usr/sap/<SID>/SYS/global/xsa/security/ssfs/data/SSFS_<SID>.DAT
/usr/sap/<SID>/SYS/global/xsa/security/ssfs/key/SSFS_<SID>.KEY
For more information, see SAP Note 2300936 Host Auto-Failover & System Replication Setup with SAP
HANA extended application services, advanced model.
The copied les will become active during system restart. Therefore, it is recommended to copy them when
the secondary system is oine (for example, before registration).
● For SAP HANA system replication, a port oset value of 100 is congured to reserve ports for system
replication communication. The port number of the replication port is calculated by adding the value for
this replication port oset to the internal port number of the corresponding service. Thus, although the
same <instance number> is used for primary and secondary systems, the <instance number>+1 is
reserved for both systems, because this port range is needed for system replication communication. For
SAP HANA systems, this port oset is set to 10000 shifting the ports from the 3
<instance number>00
to the 4<instance number>00 port range for the services. This is necessary in SAP HANA system
replication with SAP HANA systems, because after 3<instance number>99 is reached new tenant
databases allocate port numbers of the next higher instance number.
Note
To avoid interference with ephemeral ports it might be necessary to adjust the OS port range when
using SAP HANA system replication in combination with SAP HANA tenant databases. On Linux this
can be accomplished with the following command in the system startup script: echo "50000 65535"
> /proc/sys/net/ipv4/ip_local_port_range.
● In preparation for maintenance tasks (for example, near zero downtime upgrades), congure a user in the
local userstore under the SRTAKEOVER key. For more information, see Congure a User Under the
SRTAKEOVER Key.
● SAP HANA dynamic tiering is not supported with multitarget system replication. For more information
about SAP HANA system replication with SAP HANA dynamic tiering, see SAP Note 2447994.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 37
Related Information
Host Name Resolution for System Replication
Rename an SAP HANA System Host
Secure Internal Communication Between Sites in System Replication Scenarios [page 256]
SAP HANA Database Backup and Recovery
Copying and Moving Tenant Databases
Congure a User Under the SRTAKEOVER Key [page 201]
Connect to a Resource using Database Credentials
Operating System User sidadm
Checking the Status with landscapeHostConguration.py [page 160]
SAP Note 2211663
SAP Note 2369981
SAP Note 2300936
SAP Note 2447994
3.2 Conguring SAP HANA System Replication
You can congure system replication using SAP HANA cockpit, SAP HANA studio, or hdbnsutil.
Note
To congure system replication, the primary system must have been backed up at least once. For more
information, see Backup and Recovery.
You can congure system replication using the following tools:
● SAP HANA cockpit
For more information, see Congure System Replication with the SAP HANA Cockpit.
● hdbnsutil
For more information, see Congure System Replication with hdbnsutil.
● SAP HANA studio
For more information, see Congure System Replication with the SAP HANA Studio.
Note
To congure SAP HANA system replication for tenant databases, all conguration steps must be done on
the system database. However, the data backups must be created for the system database as well as for all
tenant databases. After a new tenant database was created in an SAP HANA tenant database system
running with SAP HANA system replication, a backup of this new tenant database is necessary to
automatically start its replication. Otherwise the replication for this new tenant database will not start.
38
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Related Information
SAP HANA Database Backup and Recovery
Congure System Replication with the SAP HANA Cockpit [page 39]
Congure SAP HANA System Replication with hdbnsutil [page 51]
Congure SAP HANA System Replication with the SAP HANA Studio [page 55]
3.2.1 Congure System Replication with the SAP HANA
Cockpit
Use the SAP HANA cockpit to congure system replication between two SAP HANA systems.
Prerequisites
● You have considered all the general prerequisites needed to set up system replication. For more
information, see General Prerequisites for Setting Up SAP HANA System Replication.
● You have added both systems in the SAP HANA cockpit.
● You need the operating system user to set up a system replication landscape with the SAP HANA cockpit.
For more information, see Operating System User <sid>adm and Connect to a Resource using Database
Credentials.
Context
Use the System Replication tile on the system overview page to congure system replication. Once the
conguration is done, the tile displays information on the operation mode, the replication mode, the
conguration type, and the status of system replication. For more information, see System Replication Tile.
To congure system replication in the SAP HANA cockpit, rst enable system replication on the primary
system and then register the secondary system.
You can congure system replication in two ways in the SAP HANA cockpit:
● From the primary system. For more information, see Congure System Replication from the Primary
System.
You can use this method for tier 2 and tier 3 setups. Even though this method is easier, it doesn't work to
congure multitarget or more than tier 3 setups directly from the primary system.
● From the primary and secondary systems. For more information, see Congure System Replication from
the Primary and the Secondary Systems.
You can use this method to congure any system replication setups you want.
To congure system replication in the SAP HANA cockpit, rst enable system replication on the primary
system and then register the secondary system.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 39
To change the conguration of an existing system replication, you can register again a previously stopped
secondary system when a full data shipping is needed or when you want to change the operation mode. For
more information, see Reinitialize the Secondary System.
For an example of a system replication conguration, see Example: Congure SAP HANA System Replication
with the SAP HANA Cockpit.
Related Information
System Replication Tile [page 40]
Congure SAP HANA System Replication from the Primary System [page 43]
Congure SAP HANA System Replication from the Primary and the Secondary Systems [page 45]
Reinitialize the Secondary System [page 46]
Example: Congure SAP HANA System Replication with the SAP HANA Cockpit [page 47]
3.2.1.1 System Replication Tile
The System Replication tile on the system overview page displays information about the operation mode, the
replication mode, the conguration type, and the status of system replication.
For example, you can see on the System Replication tile when system replication is not congured:
After conguring system replication, the tile looks dierently for the primary and the secondary system.
On the primary system's tile you can see the role of the system, the replication mode, the operation mode, and
whether read access is enabled for Active/Active (read enabled) congurations.
40
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
This is an example of a system replication tile for the primary system:
On the secondary system's tile you can see the role of the system and the replication mode.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 41
This an example of a system replication tile for the secondary system:
In an Active/Active (read enabled) conguration, the secondary system's tile displays also the
logreplay_readaccess operation mode:
42
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
The system replication tile of your secondary system indicates when the services are oine (for example, when
you stop the secondary system to disable system replication):
3.2.1.2 Congure SAP HANA System Replication from the
Primary System
To congure SAP HANA system replication, rst enable system replication on the primary system and then
register the secondary system. Use the SAP HANA cockpit to execute these separate steps in one
conguration step from the primary system.
Prerequisites
You have considered all the general prerequisites needed to set up system replication. For more information,
see General Prerequisites for Setting Up SAP HANA System Replication.
Context
This topic describes how to congure system replication from the primary system in SAP HANA cockpit in one
conguration step. You can use this method for 2-tier and 3-tier setups.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 43
Note
If you plan to add SAP HANA dynamic tiering to your landscape in the future, see SAP Note 2447994 before
you enable HANA system replication. SAP HANA dynamic tiering requires certain communication ports,
operation modes, and replication modes.
Procedure
1. On the Overview page of the future primary system, choose the System Replication tile.
If you never congured system replication before, this tile displays the message System replication is not
yet enabled for this system.
The System Replication page opens. If you performed a data backup before enabling system replication,
this page displays the last data backup on the top left and the Congure System Replication button on the
top right.
2. Choose Congure System Replication.
The System Replication Conguration dialog opens, allowing you to run the conguration in background.
3. Enter the logical name used to represent the primary system in the Tier 1 System Details screen area.
4. Enter the logical name used to represent the secondary system in the Tier 2 System Details screen area.
Keep in mind that the secondary system must have the same SAP system ID (<SID>) and instance number
as the primary system so that they are identied as secondaries.
5. Select the secondary system host and mark the checkbox below this area to stop the system.
6. Select a replication mode. For more information on the available replication modes, see Replication Modes
for SAP HANA System Replication.
7. Select an operation mode. For more information on the available operation modes, see Operation Modes
for SAP HANA System Replication.
8. Decide whether to initiate a full data shipping or not.
9. Check Start Secondary after Registration.
10. Optional: To add a third tier to your system replication landscape conguration, click Add Tier 3 System on
the bottom left.
11. Choose Congure.
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
SAP Note 2447994
44
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
3.2.1.3 Congure SAP HANA System Replication from the
Primary and the Secondary Systems
To set up SAP HANA system replication, rst enable system replication on the primary system and then
register the secondary system. Use the SAP HANA cockpit to execute these conguration steps on the primary
system and separately on the secondary system.
Prerequisites
You have considered all the general prerequisites needed to set up system replication. For more information,
see General Prerequisites for Setting Up SAP HANA System Replication.
Context
This topic describes how to enable system replication on the primary system and then register the secondary
system using the SAP HANA cockpit. You can use this method to congure any system replication setups you
want.
Note
If you plan to add SAP HANA dynamic tiering to your landscape in the future, see SAP Note 2447994 before
you enable HANA system replication. SAP HANA dynamic tiering requires certain communication ports,
operation modes, and replication modes.
Procedure
1. On the Overview page of the future primary system, choose the System Replication tile.
If you never congured system replication before, this tile displays the message System replication is not
yet enabled for this system.
The System Replication page opens. If you performed a data backup before enabling system replication,
this page displays overview information on the primary system on the top left and the Enable This System
as Primary link on the top right.
2. Enter the logical name used to represent the primary system and choose Congure on the bottom right.
3. On the Overview of the future secondary system, choose the Overall Database Status tile.
4. Choose Stop System on the bottom right, because the system has to be oine in order to be registered as
a secondary system.
Back on the Overview of the future secondary system, the Overall Database Status tile displays the status
Stopped.
5. On the Overview page of the secondary system, choose the System Replication tile.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 45
The System Replication page opens, displaying overview information about the secondary system on the
top left and the Register Secondary System button on the top right.
6. Choose Register Secondary System.
The System Replication Conguration page opens.
7. On the System Replication Conguration page enter the logical name used to represent the secondary
system.
8. On the System Replication Conguration page select a replication mode. For more information on the
available replication modes, see Replication Modes for SAP HANA System Replication.
9. Select an operation mode. For more information on the available operation modes, see Operation Modes
for SAP HANA System Replication.
10. Enter the host of the source system.
Note
If you are operating a distributed system on multiple hosts, enter the name of the host on which the
master name server is running.
11. Check Start Secondary after Registration.
12. Review the congured information and choose Congure on the bottom right.
The System Replication Conguration dialog opens. After the conguration is complete, the System
Replication Overview page displays information on the congured systems.
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
SAP Note 2447994
3.2.1.4 Reinitialize the Secondary System
You can register again a previously stopped secondary system using the SAP HANA cockpit.
Context
You can register again a previously stopped secondary system. You must do this when a full data shipping is
needed or when you want to change the operation mode.
46
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Procedure
1. On the Overview page of the stopped secondary system, choose the System Replication tile.
2. On the System Replication Overview, choose Reinitialize Secondary System on the top right.
3. On the System Replication Conguration page, you can now change the conguration.
Change the operation mode or resync the persistencies using the Initiate full data shipping option.
The secondary system is up and running again.
3.2.1.5 Example: Congure SAP HANA System Replication
with the SAP HANA Cockpit
Learn how to congure system replication with the SAP HANA cockpit from the primary system.
Context
Note
To congure system replication, the primary system must have been backed up at least once. This step is
included in the procedure described below. Skip this step, if the system was backed up before. For more
information, see Backup and Recovery.
Procedure
1. Create a full data backup for the system database as well as for all tenant databases.
a. Choose the system database from the Resource Directory. This is the future primary system.
b. Choose Manage database backups on the system overview page of the primary system.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 47
c. Choose Create Backup on the top right.
d. Create a complete backup, choose a destination type, enter a prex and the backup destination, then
nally choose Back Up.
For example, the backup settings can look like this:
48
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
You can see the running backup progress for each service followed by the backup details once the
backup is completed:
To see the data backups contained in the backup catalog, go back to the previous page using the arrow
on top left.
e. To back up also the tenant database, select it from the Resource Directory and proceed as described in
the previous steps.
This is an example of the system database and the tenant database of the future primary in the
resource directory:
Note
Keep in mind that you need to copy the system PKI SSFS key and data le from the primary system
to the secondary before registering the secondary system. The corresponding les can be found
on the primary:
/usr/sap/<SID>/SYS/global/security/rsecssfs/data/SSFS_<SID>.DAT
/usr/sap/<SID>/SYS/global/security/rsecssfs/key/SSFS_<SID>.KEY
2. To congure system replication, proceed as follows:
a. Choose the system database that is going to be your primary system from the Resource Directory.
b. On the System Overview page of the system database, choose the System Replication tile to congure
the primary system.
c. Choose Congure System Replication on the top right to congure system replication without having to
switch to the future secondary system.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 49
d. Enter the required information on the System Replication Conguration page.
Note
Keep in mind that the secondary system must have the same SAP system ID (<SID>) and instance
number as the primary system so that they are identied as secondaries.
This is an example of possible conguration settings:
Note
To congure a third tier, choose Add Tier 3 System and follow the instructions in Example:
Congure SAP HANA Multitier System Replication with the SAP HANA Cockpit.
50
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
The overview shows now that the SAP HANA system replication conguration was successful:
Related Information
Example: Congure SAP HANA Multitier System Replication with SAP HANA Cockpit [page 133]
Backup and Recovery
3.2.2 Congure SAP HANA System Replication with
hdbnsutil
You can congure SAP HANA system replication with the hdbnsutil command line tool.
Prerequisites
You have considered all the general prerequisities needed to congure system replication. For more
information, see General Prerequisites for Setting Up SAP HANA System Replication.
Note
For SAP HANA tenant database systems all databases must be backed up using hdbnsutil via the database
name option:
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 51
● for the system database -d SystemDB
● for the tenant databases -d <tenantDBName>
Procedure
1. Enable system replication on the primary system as follows:
a. In the Administration editor of SAP HANA studio, choose the Conguration tab and ensure that
log_mode is set to normal in the persistence section of the global.ini le.
Log mode normal means that log segments must be backed up. Log mode overwrite means that log
segments are freed by the savepoint (therefore only useful for test installations without backup and
recovery).
b. Do an initial data backup or create a storage snapshot. In multiple-container systems, the system
database and all tenant databases must be backed up. For more information, see Create Data Backups
and Delta Backups.
c. As <sid>adm on the command line enable the primary for system replication and give it a logical name
with the following command. The primary system must be online at this time:
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_enable --name=<siteName>
Option Name
Value Description
--name <primary_alias> Alias used to represent your primary
system and assign it as the primary
system for system replication
To check if the system has been successfully enabled for system replication with hdbnsutil run:
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_state
2. Stop the secondary system:
sapcontrol -nr <instance_number> –function StopSystem HDB
If you are running with HANA 2.0 you will need to copy the system PKI SSFS key and data le from the
primary system to the secondary before registering the secondary system. The corresponding les can be
found on the primary:
/usr/sap/<SID>/SYS/global/security/rsecssfs/data/SSFS_<SID>.DAT
/usr/sap/<SID>/SYS/global/security/rsecssfs/key/SSFS_<SID>.KEY
3. Register the secondary system as follows:
52
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
a. Enable system replication on the secondary system as user <sid>adm with the following command:
hdbnsutil -sr_register --name=<secondarySiteName>
--remoteHost=<primary_host> --remoteInstance=<primary_systemnr>
--replicationMode=[sync|syncmem|async]--operationMode=[delta_datashipping|
logreplay|logreplay_readaccess]
hdbnsutil -sr_register Call Options
Option Name
Value Description
--name <secondarySiteName> Alias used to represent the secon
dary system
--remoteHost <primary_host> Name of the primary host that the
secondary registers with
--remoteInstance <primary_instancenr> Instance number of primary
--replicationMode [sync|syncmem|async] Log replication modes
--operationMode [delta_datashipping|logreplay|logre
play_readaccess]
Log operation mode
--online N/A If the system is running you can use
this parameter to automatically per
form a system restart. Not relevant if
the system is shut down.
--force_full_replica
N/A Use this parameter to initiate a full
data shipping. Otherwise a delta
data shipping is attempted
To check if the system has been successfully enabled for system replication with hdbnsutil run:
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_state
b. Start the secondary system to reinitialize it with the following command:
As <sid>adm:
/usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function
StartSystem HDB
Once the secondary system is started, the replication process will start automatically.
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Create Data Backups and Delta Backups
Rename an SAP HANA System Host
Host Name Resolution for System Replication
Create Data Backups and Delta Backups (SAP HANA Studio)
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 53
3.2.2.1 Example: Congure SAP HANA System Replication
This example shows you how to congure system replication with a single host system.
Context
To congure system replication with two hosts, you may have to change the host names.
In this example a single host system is used. In multi-host systems all hosts have to be renamed.
Note
To rename hosts in a production system replication landscape, system replication must be rst
deactivated. This means you have to rst unregister and disable the secondary system before renaming
any hosts. Once you have renamed the hosts then you can enable recovery mode again and register the
secondary system with the primary system to re-activate system replication.
Procedure
1. Enable system replication on the primary system, with the hostname ej11.
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_enable --name=dcsite1
2. Stop the secondary system. The primary system can stay online.
As <sid>adm
/usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function StopSystem
HDB
3. Register the secondary system with the following command:
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_register
--name=dcsite2
--remoteHost=ej11
--remoteInstance=50
--mode=sync
--operationMode=logreplay
Also see SAP Note 611361 Hostnames of SAP servers
4. Start the secondary system. This initiates the initial data transfer.
As <sid>adm
/usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function StartSystem
HDB
54
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Related Information
Rename an SAP HANA System Host
SAP Note 611361
3.2.3 Congure SAP HANA System Replication with the SAP
HANA Studio
To congure SAP HANA system replication between two identical SAP HANA systems, you must rst enable
system replication on the primary system and then register the secondary system.
Prerequisites
● You have considered all the general prerequisites needed to set up system replication. For more
information, see General Prerequisites for Setting Up SAP HANA System Replication.
● You have added both systems in the SAP HANA studio.
Procedure
1. Enable system replication on the primary system, which has to be online, as follows:
a. In the Systems view, right-click the primary system and choose Conguration and Monitoring
Congure System Replication .
The Congure System Replication dialog opens. The Enable System Replication option is selected by
default.
Note
You can also access the Congure System Replication dialog from the Landscape System
Replication tab.
b. Choose Next.
c. Enter the logical name used to represent the primary system and choose Next.
d. Review the congured information and choose Finish.
e. Stop the secondary with right-click on the secondary system and choosing Conguration and
Monitoring Stop System .
2. Register the secondary system as follows:
a. Stop the secondary system if it is still running. Right-click the secondary system and choose
Conguration and Monitoring Stop System
b. In the Systems view, right-click the secondary system and choose Conguration and Monitoring
Congure System Replication .
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 55
The Congure System Replication dialog opens.
c. Choose Register Secondary System and then Next.
d. Enter the required system information and the logical name used to represent the secondary system.
Note
If you are operating a distributed system on multiple hosts, you enter the name of the host on
which the master name server is running.
e. Specify the log replication mode. For more information on the available replication modes, see
Replication Modes for SAP HANA System Replication.
f. Specify the operation mode. For more information, see Operation Modes for SAP HANA System
Replication. The logreplay_readaccess operation mode is not available in SAP HANA studio.
g. Review the congured information and choose Finish.
3. Optional: Congure the parameters in the system_replication section of the global.ini le.
These parameters determine for example the size and frequency of data and log shipping requests. All
parameters have a default conguration.
4. If necessary, start the secondary system.
Note
The secondary system is started automatically unless you deselected the corresponding option during
conguration (step 2).
The secondary system requests an initial full data replica from the primary system.
Results
You have enabled system replication and registered the secondary system with the primary system. The
secondary system operates in recovery mode. All secondary system services constantly communicate with
their primary counterparts, replicate and persist data and logs, and load data to memory. However, the
secondary system does not accept SQL connections.
In the Systems view, the primary and secondary systems appear as operational ( ). If the secondary system
is not open for read access, it appears as operational ( ) but with an error ( ) indicating that no connection
to the database is available. For more information, see Generic Conditions for Active/Active (Read Enabled).
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
Add an SAP HANA System
Stop a System
SAP HANA System Replication Conguration Parameters [page 62]
56
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Create Data Backups and Delta Backups (SAP HANA Studio)
Rename an SAP HANA System Host
Enable Data and Log Volume Encryption in an Existing SAP HANA Database
Data and Log Volume Encryption
Encryption Key Management
Generic Conditions for Active/Active (Read Enabled) [page 116]
SAP Note 611361
3.3 Initializing the Secondary
Whenever the secondary is registered with the primary system, the goal is to get the persistence (that is, the
data and log volumes) on the secondary system into a consistent state to the primary system.
After initially conguring system replication, a full data shipping takes place. This happens automatically, but it
can also be done manually. For more information, see Initialize the Secondary with Storage Copy from Primary.
When initializing the secondary the following two situations can occur:
● The secondary system is completely unrelated to the primary system
If the secondary system is unrelated to the primary system, a full data shipping is done. An in-place
snapshot created on the disk of the primary system is initially sent to the secondary system.
This initial full data shipping can be prevented by manual intervention and the secondary system can be
initialized with a binary storage copy of the primary system’s persistence. For more information, see
Initialize the Secondary with Storage Copy from Primary.
● The secondary system is related to the primary system for one of the following reasons:
○ It was already registered before as a secondary system to this primary and probably shut down for a
time.
○ It is a former primary system, which will become a secondary system though failback switching the
replicating direction.
If the persistence (that is data and log volumes) of the secondary system is related to the primary system
(it actually contains the persistence of the primary at a former time), the newly registered system can be
synced with a delta data or log shipping avoiding a full data shipping.
After a new registration of the secondary system, a delta data or log shipping is always attempted. For more
information, see Resync Optimization.
Related Information
Initialize the Secondary with Storage Copy from Primary [page 58]
Resync Optimization [page 19]
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 57
3.3.1 Initialize the Secondary with Storage Copy from
Primary
The secondary system can be initialized using a binary storage copy from the primary system.
Context
For this procedure copy only the data, not the log.
Procedure
1. Create a consistent binary storage copy from the primary system for the persistence of all services. You
can use the snapshot technology to create an IO consistent persistence copy. Create a full copy of the
persistence using the IO consistent storage snapshots.
If you can't use the method above, create a consistent OS copy of persistence while the primary system is
stopped.
2. Shut down the secondary system.
3. Transfer or mount the full copy on the secondary system.
4. Replace the persistence of the secondary system with the storage copy from the primary system.
5. Register the secondary system without [--force_full_replica].
6. Start the secondary system.
Results
When the secondary system is started after the new registration, the initialization optimizations are carried
out. The system checks if the persistence of the secondary system is compatible with the persistence of the
primary system. The secondary system checks if its persistence is compatible with the persistence of the
primary system. If this check succeeds, the secondary system requests only a delta data shipping.
3.4 Full Sync Option for SAP HANA System Replication
To reach a true RPO=0 for synchronous system replication, the full sync option can be enabled for SYNC
replication mode.
With the activated full sync option, the transaction processing on the primary system blocks when the
secondary is currently not connected and newly created log buers cannot be shipped to the secondary
58
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
system. This behavior ensures that no transaction can be committed on the primary without shipping the log
buers to the secondary system.
The full sync option can be switched on and o using the command:
hdbnsutil -sr_fullsync --enable|--disable
This changes the setting of the parameter enable_full_sync in the [system_replication] section of the
global.ini le accordingly.
Recommendation
When conguring the full sync option, proceed as follows:
1. Congure your system replication with the SYNC replication mode. This replication mode is the
prerequisite for enabling the full sync option.
2. Check that the system replication status is active and in sync for all services.
3. Enable the full sync option with hdbnsutil -sr_fullsync --enable
In the M_SERVICE_REPLICATION system view, the setting of the full sync option can be viewed in the column
FULL_SYNC. It can have the following values:
● DISABLED: Full sync is not congured at all.
The parameter enable_full_sync = false in the system_replication section of the global.ini le.
● ENABLED: Full sync is congured, but it is not yet active.
Transactions do not block in this state. To become active, the secondary has to connect and the replication
status has to be ACTIVE.
● ACTIVE: Full sync mode is congured and active.
If the network connection to a connected secondary is closed, the transactions on the primary system will
block in this state.
If full sync is enabled when an active secondary is currently connected, the FULL_SYNC column will be
immediately set to ACTIVE.
If the secondary is stopped, disable the full sync option. Otherwise the primary blocks and it is not possible to
stop it.
Note
Use the hdbnsutil command to resolve a blocking situation of the primary system caused by the enabled
full sync option. This is important because a conguration changing command could also block in this
state. You must do this also when you want to shut down the currently blocking primary system. Otherwise
it is not possible to stop it.
The Cluster Manager that is used to operate SAP HANA system replication landscapes could provide a timeout
after which the blocking situation is resolved automatically using the hdbnsutil command and deactivating the
full sync option. However, after the reason for the blocking situation disappears, you must activate the full sync
option again (manually or automatically with the help of the cluster manager tool).
In a multitarget system replication setup, congure the full sync option on the primary system. Enter the site
name used for the secondary system when registering it:
global.ini
[system_replication]
enable_full_sync[<secondary_site_name>] = true
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 59
Note
In a multitarget system replication setup, you can use hdbnsutil -sr_fullsync to turn o the full sync
option.
Related Information
SAP HANA System Replication Command Line Reference [page 71]
3.5 Add and Remove Hosts in SAP HANA System
Replication
You can add a new host to a replicated system with the SAP HANA lifecycle manager.
Context
Note
Hosts must be added equally to both primary and secondary sites.
Note
Do not turn o system replication when adding a host.
Recommendation
It is recommended that a host is added to the secondary site before adding it to the primary site. This
avoids the situation where the new host saves data without rst being in sync.
Procedure
1. Add a host to the secondary site and start it.
2. Add a host to the primary site and start it.
Replication begins automatically.
3. To remove a host, rst remove it from the primary site and then remove the host from the secondary site.
60
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Related Information
Add Hosts Using the Command-Line Interface
Remove Hosts Using the Command-Line Interface
3.6 Changing the Operation Mode
The operation mode can only be changed by stopping and re-registering the secondary system with the
desired operation mode.
You can change operation modes using the hdbnsutil -sr_register command and explicitly setting the
new operation mode with the -operationMode option:
hdbnsutil -sr_register
--remoteHost=<primary hostname>
--remoteInstance=<instance number>
--replicationMode=[sync|syncmem|async]
--operationMode=[delta_datashipping|logreplay|logreplay_readaccess]
--name=<siteName>
To start the replication with the new operation mode, start the secondary system:
sapcontrol -nr <instance_number> -function StartSystem HDB
Note
It is not necessary to unregister the secondary while changing the operation mode. The hdbnsutil -
sr_register command overwrites the previous register conguration. To understand the scenarios in
which you should unregister a secondary system, see SAP Note 1945676.
When changing the operation mode from delta_datashipping to logreplay or logreplay_readaccess,
no full data shipping is necessary. Full data shipping is necessary, however, when switching from logreplay or
logreplay_readaccess back to delta_datashipping.
Related Information
SAP Note 1945676
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 61
3.7 Changing the Replication Mode
The replication mode can be changed without going through a full data shipping from the primary system to
the secondary system afterwards.
To change the replication mode, use the following command on the online or oine secondary system:
hdbnsutil -sr_changemode --mode=sync|syncmem|async
In the M_SERVICE_REPLICATION view you can check whether the replication mode was changed correctly.
The following command provides this information too:
hdbnsutil -sr_state --sapcontrol=1
Related Information
M_SERVICE_REPLICATION System View [page 269]
3.8 SAP HANA System Replication Conguration
Parameters
Several conguration parameters are available for conguring SAP HANA system replication between the
primary and secondary system.
The system replication parameters are dened in the [system_replication] section of the global.ini
le and have the default values shown below. The System column denes whether the parameter can be set on
the primary, the secondary, or both.
Note
preload_column_tables uses the Boolean keywords "true" or "false". Numbers do not work in place of
the keywords.
Parameter
datashipping_min_time_interval
Type
int
Unit
seconds
Default
600 (10 min)
System
Secondary
62 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Parameter
datashipping_min_time_interval
Description
Minimum time interval between two data shipping requests from the secondary system.
If datashipping_logsize_threshold (see next parameter) is reached rst, the data shipping re
quest will be sent before the time interval is elapsed when the log size threshold is reached.
Parameter
datashipping_logsize_threshold
Type
int
Unit
bytes
Default
5*1024*1024*1024 (5 GB)
System
Secondary
Description
Minimum amount of log shipped between two data shipping requests from the secondary system.
If the time dened by datashipping_min_time_interval (see previous parameter) has passed
before reaching this threshold, the data shipping request will be sent before this threshold is reached when
the time interval has elapsed.
Parameter
preload_column_tables
Type
bool
Unit
true/false
Default
true
System
Primary and Secondary
Description
If activated column store tables are preloaded to the main part of memory.
If set on the primary system, the loaded table information is collected and stored in the snapshot that is
shipped.
If set on thesecondary system, this information is evaluated and the tables are actually preloaded there
according to the information received on the loaded tables.
Parameter
datashipping_snapshot_max_retention_time
Type
int
Unit
minutes
Default
300
System
Primary
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 63
Parameter
datashipping_snapshot_max_retention_time
Description
Maximum retention time (in minutes) of the last snapshot that has been completely shipped to the secon
dary system.
Shipped snapshots older than datashipping_snapshot_max_retention_time will be dropped
automatically. Snapshots currently used in data shipping are not aected and are not dropped, if data
shipping takes longer than datashipping_snapshot_max_retention_time. They can be drop
ped if data shipping has been nished. If the parameter is set to 0, snapshots are immediately dropped
after data replication nishes.
When roles are switched between the primary and secondary systems preparing a failback later on, the
secondary can be initialized with a delta replica between this snapshot and the current persistent state on
the new primary after takeover. In order to do this:
● A snapshot has to exist on the new secondary when it starts for the rst time as secondary.
● The snapshot has to be compatible with the persistence of the new primary.
It is veried, if the snapshot has been the source of the primary system before takeover. It cannot be used,
if the secondary is registered with an incompatible primary system. If both conditions are true, the secon
dary can be initialized with a delta replica.
Parameter
logshipping_timeout
Type
int
Unit
seconds
Default
30
System
Primary
Description
Number of seconds the primary waits for the acknowledgment after sending a log buer to the secondary
system.
If the primary does not receive the acknowledgment for a sent log buer within the time dened by
logshipping_timeout, it closes the connection to the secondary system in order to continue data
processing. This is done to prevent the primary system from blocking transaction processing if there is a
hang situation on the connection to the secondary system.
After the timeout period for a send operation has elapsed, transactions are written only on the primary
system until the secondary has reconnected.
The logshipping_timeout does not dene a blocking period for logshipping on the primary system
in general. It is used to close hanging connections on the primary system that are not getting automatically
closed. If the redo log cannot be sent to the secondary system within this time, the connection is tempora
rily closed and the primary writes the redo log locally. This can happen any time, also when the primary is
currently not waiting for acknowledgments from the secondary system.
Use the ful sync option, if the primary system should block whenever the connection to the secondary sys
tem is lost. In this case the primary system will stop.
64
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Parameter
logshipping_async_buffer_size
Type
int
Unit
bytes
Default
67108864 (64MB)
System
Primary
Description
In asynchronous replication mode, the log writer copies the log buers into an intermediate memory buer
rst and continues processing. A separate thread reads log buers from this memory buer and sends
them to the secondary system asynchronously over the network.
This parameter determines how much log can be intermediately buered. This buer is especially useful in
peak times when log is generated faster than it can be sent to the secondary system. If the buer is large, it
can handle peaks for a longer period of time.
The behavior of buer full situations can be controlled by the parameter
logshipping_async_wait_on_buffer_full.
The parameter can be changed online, but will become active the next time the secondary system recon
nects.
Parameter
logshipping_async_wait_on_buffer_full
Type
bool
Unit
true/false
Default
true
System
Primary
Description
This parameter controls the behavior of the primary system in asynchronous log shipping mode when the
log shipping buer is full.
If set to true, the primary system potentially waits until there is enough space in the log shipping buer, so
that the log buer can be copied into it. This can slow down the primary system if there is currently high
load that cannot be handled by the network connection.
If the parameter is set to false, the connection to the secondary system will be temporarily closed to not
impact the primary system. Later, the secondary can reconnect and sync using delta shipping.
Parameter
reconnect_time_interval
Type
int
Unit
seconds
Default
30
System
Secondary
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 65
Parameter
reconnect_time_interval
Description
If a secondary system is disconnected from the primary system because of network problems, the secon
dary system tries to reconnect periodically after the time interval specied in this parameter has passed.
Parameter
enable_full_sync
Type
bool
Unit
true/false
Default
false
System
Primary
Description
If set, system replication operates in full sync mode when the SYNC replication mode is set.
In full sync mode, transaction processing blocks when the secondary is currently not connected and newly
created log buers cannot be shipped to the secondary system. This behavior ensures that no transaction
can be locally committed without shipping to the secondary system.
For more information, see Full Sync Option for SAP HANA System Replication.
Parameter
enable_log_compression
Type
bool
Unit
true/false
Default
false
System
Secondary
Description
If activated, log buers will be compressed before sending them over the network to the secondary sys
tem. The secondary system decompresses the log buers it receives and then writes them to disk. If the
network bandwidth is the bottleneck in the system replication setup, log buer compression can improve
log shipping performance because less data is being sent over the network.
The drawback to sending a compressed log buer to the secondary system is that it requires additional
time and processing power for compression and decompression. This can result in worse log shipping per
formance if turned on in a conguration with a fast network.
The parameter has to be set on the secondary system. It can be changed online, but the secondary system
has to re-connect to the primary system in order to activate the parameter change.
Parameter
enable_data_compression
Type
bool
Unit
true/false
Default
false
66 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Parameter
enable_data_compression
System
Secondary
Description
If activated, data pages will be compressed before sending them over the network to the secondary sys
tem. The secondary system decompresses the data pages it receives and then writes them to disk. If the
network bandwidth is the bottleneck in the system replication setup, data compression can improve log
shipping performance because less data is being sent over the network.
The drawback to sending compressed data pages to the secondary system is that it requires additional
time and processing power for compression and decompression. This can result in worse data shipping
performance if turned on in a conguration with a fast network.
The parameter has to be set on the secondary system. It can be changed online, but the secondary system
has to re-connect to the primary system in order to activate the parameter change.
Parameter
keep_old_style_alert
Type
bool
Unit
true/false
Default
false
System
Primary
Description
Before SPS 09 closed replication connections and conguration parameter mismatches were alerted with
Alert 21.
With SPS 09 two dedicated alerts have been introduced for both error situations. By default old style alert
ing is still oered for backwards compatibility. When setting this parameter to false, the old behavior is
turned o and only new alerts will be generated.
For more information, see SAP HANA System Replication Alerts.
Parameter
operation_mode
Type
enum
Unit
delta_datashipping/logreplay/logreplay_readaccess
Default
logreplay
System
Secondary
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 67
Parameter
operation_mode
Description
Operation mode of the secondary site during replication.
There are three dierent settings for this parameter:
● delta_datashipping
System Replication uses data and log shipping for replication. Log buers received by the secondary
system are just saved to disk, savepoints after intermediate delta data shippings truncate the log. Col
umn table merges are not executed on the secondary system, but merged tables on the primary sys
tem are transported via delta data shippings to the secondary system.
● logreplay
System Replication uses an initial data shipping to initialize the secondary system. After that only log
shipping is done and log buers received by the secondary are replayed there. Savepoints are exe
cuted individually for each service and column table merges are executed on the secondary system.
● logreplay_readaccess
System Replication uses an initial data shipping to initialize the secondary system. After that only log
shipping is done and log buers received by the secondary are replayed there. Savepoints are exe
cuted individually for each service and column table merges are executed on the secondary system.
Furthermore, read only access via SQL is possible to the secondary system.
For more information, see Operation Modes for SAP HANA System Replication.
Parameter
enable_log_retention
Type
enum
Unit
auto/o/on
Default
auto
System
Primary, Secondary
68 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Parameter
enable_log_retention
Description
Enables or disables log retention on a system replication system. Log retention on the primary system is
useful when the secondary should sync with the primary by re-shipping missing log after a network outage
or downtime. If the missing log is not available anymore on the primary system, a data shipping is required
(delta in operation mode
delta_datashipping, full in all other operation modes). Log retention on
the secondary system is needed to keep log for optimized re-sync during failback.
There are three conguration options:
● auto
Log retention is automatically enabled, if the secondary is in logreplay or
logreplay_readaccess operation modes. For the delta_datashipping operation mode it
is disabled by default.
● on
Log retention is enabled.
● o
Log retention is disabled.
When log retention is enabled and the system is congured as primary, the primary will not free log seg
ments when the secondary system is disconnected, but keep them marked as RetainedFree for a po
tential optimized resync.
When setting log retention explicitly to on or o, it should also be set for delta_datashipping opera
tion mode or for failback with delta log shipping optimization. In the latter case after takeover to the secon
dary, the old primary can re-sync via missing log with the new primary system and no full data shipping is
required for initialization.
In a multitarget system replication conguration, if enable_log_retention =
force_on_takeover is congured, the log will be retained during replication for all direct secondaries
until a takeover is executed. During takeover, the parameter is set to force. This means that the log will
be retained independently of any secondary system. For more information, see Log Retention and Multitar
get System Replication in Log Retention.
Parameter
logshipping_max_retention_size
Type
int
Unit
MB
Default
1048576 (1TB)
System
Primary
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 69
Parameter
logshipping_max_retention_size
Description
Sets the maximum amount of log that will be kept for syncing a secondary system. This value only has an
eect if log retention is enabled.
Two situations have to be distinguished here:
If logshipping_max_retention_size has been set to a value other than 0, when no secondary is
connected log segments are not reused even if they are truncated and backed up until the max size limit
has been reached or the system runs into a log full situation.
If the maximum size limit is reached or in log full situations, the segments that are only kept for syncing the
secondary system will be reused. This setting prevents the system from hanging on the primary system
because of too many log segments that are held for syncing the secondary system. With this setting, the
primary keeps running with the drawback that the secondary cannot sync anymore.
If logshipping_max_retention_size is congured to 0, log segments required for syncing the
secondary are not reused and a log full results in a system standstill on the primary system until log writing
can continue. This setting allows you to congure an upper limit up to which redo log segments are kept in
RetainedFree state on the primary system before they are overwritten for syncing with a secondary
system. When the reason for the log full has been resolved, the transaction processing can continue.
Note
The default setting logshipping_max_retention_size = 1048576 (MB) of 1 TB means
that 1 TB of size is congured for every service, which replicates data to a secondary system. That is,
every service owning a persistence in form of data and log volume.
Example
If the services nameserver, two indexservers (for example, two tenant databases) and an xsengine are
running in your SAP HANA system, the total congured log retention size will be 4 TB (4 x 1 TB). With
this setting it can happen that the disk full is reached before the RetainedFree marked log segments
are overwritten.
If you want to change the default value of 1 TB, you can do this in the global.ini. Another option is to set
this parameter in the service ini les individually. For example, if the value is set in the global.ini of the
system database, in the global.ini of a tenant database, and in the indexserver.ini of a tenant database,
the indexserver.ini setting would win and will be taken for log retention of this indexserver.
Parameter
datashipping_parallel_channels
Type
int
Default
4
System
Secondary
70 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Parameter
datashipping_parallel_channels
Description
The parameter denes the number of network channels used by full or delta datashipping. The actual
number of channels for each shipping can be adjusted by the system to reduce overhead depending on the
current amount of data to be shipped.
Higher parallelism can be useful when large amount of data (above several GB at least) needs to be ship
ped and the utilization of network bandwidth by single network stream is low. Please note that the overall
bandwidth is still limited by the I/O bandwidth, because the data needs to be read from the primary sys
tem.
To deactivate the parameter, change the default to 0.
Related Information
Full Sync Option for SAP HANA System Replication [page 58]
Operation Modes for SAP HANA System Replication [page 14]
Log Retention and Multitarget System Replication [page 25]
Change a System Property in SAP HANA Studio
Log Retention [page 20]
3.9 SAP HANA System Replication Command Line
Reference
This topic provides an overview of SAP HANA system replication commands and options.
Command
-sr_enable
Options [--name=<site alias>]
System Primary
Online/Oine Online
Description Enables a system to serve as a system replication source system.
In multitier and multitarget setups the --name= option is mandatory. Use -sr_enable to enable the
source system for any further tier that is added to the system replication landscape.
Command -sr_disable
System Primary
Online/Oine Online
Description Disables system replication capabilities on the primary system.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 71
Command -sr_register
System Secondary
Online/Oine Oine
Description
● --remoteHost=<primary master host>
Registers a system to a source system and creates the replication path for the system replication.
● --remoteInstance=<primary instance id>
● --replicationMode=sync|syncmem|async
Species the replication mode.
● --operationMode=delta_datashipping|logreplay|logreplay_readaccess
Species the operation mode.
● --name=<unique site name>
Species the system name.
● [--online]
If the system is running you can use this parameter to automatically perform a system restart. Not
relevant if the system is shut down.
● [--force_full_replica]
If a parameter is given, a full data shipping is initiated. Otherwise a delta datashipping is attempted.
Command
-sr_unregister
Options [--id=<site id>|--name=<site name>]
System Primary, Secondary
Online/Oine
Secondary oine
Primary online (to remove metadata)
72 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Command -sr_unregister
Description
Unregisters a secondary system from its source.
Use this command on the secondary that needs to be unregistered. When the secondary system is not
available, this command can also run on the primary. In this case use either the –id option or the –name
option to identify the secondary system.
Note
There are three scenarios in which it is necessary to unregister system replication:
● When the secondary system is available, but should be de-coupled permanently
You will be able to use the secondary system as a standard SAP HANA installation afterwards.
● When the secondary system is not available anymore and the primary system needs to be
cleaned up in order to be able to register a new system
This can occur when the secondary system was uninstalled or when it cannot be recovered af
ter a disaster.
● When you want to re-establish the original setup after a takeover in a multitier system replica
tion conguration
For more information, see Restore the Original SAP HANA Multitier System Replication Congu
ration.
To understand how to use the -sr_unregister command correctly, see SAP Note 1945676.
Command
-sr_changemode
Options --mode=sync|syncmem|async
System Secondary
Online/Oine Online and oine
Description Changes the replication mode of a secondary system.
Command -sr_takeover
System Secondary
Online/Oine Online and oine
Description Makes this secondary the new primary system.
Command -sr_state
System Primary and Secondary
Online/Oine Online and oine
Description Shows status information about a system replication system.
Related Information
Example: Restore the Original SAP HANA Multitier System Replication Conguration [page 141]
SAP Note 1945676: Correct usage of hdbnsutil -sr_unregister
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 73
3.10 Disabling SAP HANA System Replication
Remove a system replication conguration when you want to run the two systems separately or if you don't
need this high availability and disaster recovery mechanism anymore.
To remove a system replication conguration, unregister the secondary and disable the primary system. For
more information on the use cases of the –sr_unregister command, see SAP Note 1945676.
You can disable system replication using the following tools:
● SAP HANA cockpit
For more information, see Disable SAP HANA System Replication with the SAP HANA Cockpit.
● SAP HANA studio
For more information, see Disable SAP HANA System Replication with the SAP HANA Studio.
● hdbnsutil
For more information, see Disable SAP HANA System Replication with hdbnsutil.
Related Information
Disable SAP HANA System Replication with the SAP HANA Cockpit [page 74]
Example: Disable SAP HANA System Replication with the SAP HANA Cockpit [page 75]
Disable SAP HANA System Replication with the SAP HANA Studio [page 81]
Disable SAP HANA System Replication with hdbnsutil [page 80]
SAP Note 1945676
3.10.1 Disable SAP HANA System Replication with the SAP
HANA Cockpit
You can disable SAP HANA system replication in an SAP HANA system using the SAP HANA cockpit.
Prerequisites
● The secondary system must be oine.
● You need the operating system user to disable system replication with the SAP HANA cockpit. For more
information, see Operating System User <sid>adm and Connect to a Resource using Database Credentials.
74
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
Procedure
1. Stop the secondary system from the Overall Database Status tile
2. Unregister the secondary system as follows:
a. On the Overview page of the stopped secondary system, choose the System Replication tile.
The System Replication tile opens displaying the System Replication Overview.
b. Choose Unregister this Secondary on the top right.
Depending whether your system should be online or oine after unregistering it, check the Start
system after unregistration option in the conrmation dialog and choose OK. For more information, see
SAP Note 1945676.
3. Disable system replication on the primary system as follows:
a. On the Overview page of the primary system, choose the System Replication tile.
The System Replication tile opens displaying the System Replication Overview.
b. Choose Disable System Replication on the top right and conrm that you want to disable system
replication.
The Ignore the secondary system option allows you to disable the primary system even though the
secondary is still attached. This could be relevant, if the secondary has been uninstalled in the
meantime.
Related Information
Connect to a Resource using Database Credentials
Operating System User sidadm
SAP Note 1945676
3.10.1.1 Example: Disable SAP HANA System Replication with
the SAP HANA Cockpit
Learn how to disable system replication with the SAP HANA cockpit.
Prerequisites
● The secondary system must be oine.
● You need the operating system user to disable system replication with the SAP HANA cockpit. For more
information, see Operating System User <sid>adm and Connect to a Resource using Database Credentials.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 75
Procedure
1. Stop the secondary system from the Overall Database Status tile.
The system replication tile on the overview page of the stopped secondary system indicates that all
services are now oine.
76
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
2. Choose the system replication tile on the overview page of the stopped secondary system.
The system replication overview opens giving the possibility to unregister this secondary system:
3. Choose Unregister this Secondary on the top right.
A conrmation dialog opens.
4. Optional: Depending whether your system should be online or oine after unregistering it, check the Start
system after unregistration option in the conrmation dialog and conrm.
For more information, see SAP Note 1945676.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 77
5. Switch now to the primary system. Choose the system replication tile of the primary system.
This tile indicates that the primary system is not connected to the secondary system anymore:
6. On the system replication overview choose Disable System Replication on the top right.
A conrmation dialog opens.
78
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
7. Conrm that you want to disable system replication.
The Ignore the secondary system option allows you to disable the primary system even though the
secondary is still attached. This could be relevant, if the secondary has been uninstalled in the meantime.
On the overview you can now see that the primary system is disabled and system replication is not
congured.
Related Information
SAP Note 1945676
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 79
3.10.2 Disable SAP HANA System Replication with hdbnsutil
You can disable SAP HANA system replication withhdbnsutil.
Prerequisites
● You are logged on to both systems as the operating system user (user <sid>adm) or are able to enter these
credentials when prompted.
● The secondary system must be oine.
Procedure
1. Stop the secondary system:
sapcontrol –nr <instance_number> -function StopSystem HDB
2. On secondary system unregister the secondary system:
hdbnsutil -sr_unregister --id=<secondarySiteID>
Note
If system replication is out of sync and you need to register again the initial secondary system, use the
command hdbnsutil –sr_register. It is not necessary to unregister the secondary system before
registering it again.
For an overview of hdbnsutil –sr_unregister use cases, see SAP Note 1945676.
3. Disable system replication on the primary system as follows:
hdbnsutil –sr_disable
Related Information
SAP HANA System Replication Command Line Reference [page 71]
SAP Note 1945676
80
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
3.10.3 Disable SAP HANA System Replication with the SAP
HANA Studio
You can disable SAP HANA system replication in an SAP HANA system using the SAP HANA studio.
Prerequisites
● You are logged on to both systems as the operating system user (user <sid>adm) or are able to enter
these credentials when prompted.
● The secondary system must be oine.
Procedure
1. Unregister the secondary system as follows:
a. In the Systems view, right-click the primary system and choose Conguration and Monitoring
Congure System Replication .
The Congure System Replication dialog opens.
Note
You can also access the Congure System Replication dialog from the Landscape System
Replication tab.
b. Choose Unregister secondary system and then Next.
c. Enter the required system information and choose Next.
d. Review the congured information and choose Finish.
2. Disable system replication on the primary system as follows:
a. In the Systems view, right-click the primary system and choose Conguration and Monitoring
Congure System Replication .
b. Choose Disable system replication and choose Next.
c. Review the congured information and choose Finish.
3.11 SAP HANA System Replication Setup for XS Advanced
Runtime
In a system replication setup, all the data – including XS advanced runtime system data and application data –
is replicated to a secondary system.
XS advanced services and applications run only on the currently active system. On the secondary system, XS
advanced services are in an idle state until the takeover takes place.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 81
After the takeover, all XS advanced services are started which in turn brings up all XS advanced applications on
the secondary system. Moreover, XS advanced services and applications use the same domains and
certicates that were present in the primary system before the takeover started.
For this to work, the central point for XS advanced requests must be the same on the primary and the
secondary systems. This is established by using a failover router similar to the host auto-failover setup. For
more information about the host auto-failover setup, see Host Auto-Failover Setup with XS Advanced Runtime.
SAP HANA System Replication Setup for XS advanced
In case the failover router terminates SSL, the same rules apply as described in Host Auto-Failover Setup with
XS Advanced Runtime.
For more information, see SAP Note 2300936.
Related Information
Host Auto-Failover Setup with XS Advanced Run Time
SAP Note 2300936
82
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
3.12 SAP HANA System Replication with Tenant Databases
The usual SAP HANA system replication principles apply for tenant database systems.
SAP HANA supports system replication for tenant databases on the system level, this means the tenant
database system as a whole including all tenant databases. An SAP HANA system installed in multiple-
container mode always has exactly one system database and any number of tenant databases (including zero).
Before you begin preparing a replication strategy for an SAP HANA system, you should be aware of the
following important points:
● SAP HANA systems can only be replicated as the whole system. This means that the system database and
all tenant databases are part of the system replication. A takeover can only be performed as a whole
system. A takeover on the level of a single tenant database is not possible.
● If a new tenant database is created in a congured SAP HANA system replication, it must be backed up to
participate in the replication. Afterwards, the initial data shipping is started automatically for this tenant
database. If a takeover is done while the initial data shipping is running and not nished, this new tenant
database will not be operational after takeover and will have to be recovered with backup and recovery (see
the SAP HANA Database Backup and Recovery section of the SAP HANA Administration Guide).
● If an active tenant database is stopped in a running SAP HANA system replication, it is stopped on the
secondary site as well. If a takeover is done while tenant databases (which were part of the system
replication) are stopped, they will be in the same state after takeover as they were on the primary site when
they were stopped. The tenant databases must be started to complete the takeover.
● If SAP HANA system replication runs in replication mode SYNC with the full sync option enabled, and if the
connection to the secondary site is interrupted, no write operations on the primary site are possible. The
operation of creating a tenant database, for example, will wait until the connection to the secondary is
reestablished or the SQL statement times out.
● With SAP HANA systems, the services needed are generated automatically in the tenant databases.
● For SAP HANA system replication, a port oset value of 100 is congured to reserve ports for system
replication communication. The port number of the replication port is calculated by adding the value for
this replication port oset to the internal port number of the corresponding service. Thus, although the
same <instance number> is used for primary and secondary systems, the <instance number>+1 is
reserved for both systems, because this port range is needed for system replication communication. For
SAP HANA systems, this port oset is set to 10000 shifting the ports from the 3<instance number>00
to the 4<instance number>00 port range for the services. This is necessary in SAP HANA system
replication with SAP HANA systems, because after 3<instance number>99 is reached new tenant
databases allocate port numbers of the next higher instance number.
Note
To avoid interference with ephemeral ports it might be necessary to adjust the OS port range when
using SAP HANA system replication in combination with SAP HANA tenant databases. On Linux this
can be accomplished with the following command in the system startup script: echo "50000 65535"
> /proc/sys/net/ipv4/ip_local_port_range.
● It is possible to copy or move tenant databases between SAP HANA systems. However, you can only use
this feature if system replication is not enabled for high availability purposes on either the source or target
system for the entire duration of the copy or move process. For more information, see Copying and Moving
Tenant Databases Between Systems.
SAP HANA System Replication
SAP HANA System Replication: Conguration
P U B L I C 83
Note
When copying or moving a tenant to the primary system of a system replication landscape, the data
shipping for this tenant starts immediately. While this initial data shipping is running, a takeover will
cause a loss of data for this tenant on the new primary.
● For SAP HANA systems running with the HIGH isolation level, the system PKI SSFS data and key le must
be copied from the primary system to the same location on the secondary system(s). For more
information, see Increase the System Isolation Level in the SAP HANA Administration Guide.
For more information on the individual points, see the Availability and Scalability section of the SAP HANA
Administration Guide
.
Related Information
Availability and Scalability
SAP HANA Database Backup and Recovery
Copying and Moving Tenant Databases
Increase the System Isolation Level
84
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Conguration
4 SAP HANA System Replication: Takeover
and Failback
Learn how to perform a takeover and a failback for planned or unplanned downtimes of the primary system.
How can I perform a takeover and a failover?
This section describes how to perform a takeover and a failover with three dierent tools. Generally, you have
to perform the following steps:
● Switch your active system from the current primary system to the secondary system
● Register the former primary as the secondary with the now active primary system. The roles are switched
compared to the original setup.
What types of takeover are available?
You can perform a standard takeover, a takeover with handshake, or an invisible takeover.
Perform a standard takeover when there are unplanned downtimes of the primary system. This can happen
when the primary system has a serious problem and is not available. You can also perform a standard takeover
for planned downtimes such as near zero downtime upgrades (NZDU).
Perform a takeover with handshake for a safe planned takeover while the primary is still running (for example,
in NZDU). All new writing transactions on the primary system will be suspended and the takeover is rst
executed when all redo log is available on the secondary system. For more information, see Takeover with
Handshake.
Perform an invisible takeover to achieve an automatic recovery of your sessions after takeover to your new
primary. For dedicated client applications this takeover will be invisible. For more information, see Invisible
Takeover.
Where can I nd more information?
The following SAP Notes are relevant for a full understanding of the basic concepts described in this chapter:
SAP Notes
SAP Note Title
2063657
SAP HANA System Replication Takeover Decision Guideline
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 85
SAP Note Title
2053504
System replication: Hanging client processes after a takeover
Related Information
Takeover [page 86]
Client Connection Recovery After Takeover [page 92]
Invisible Takeover and Restart [page 97]
Takeover with Handshake [page 99]
Failback [page 100]
Checking the SAP HANA System Replication Status [page 159]
4.1 Takeover
The takeover process is the name for the task of switching your active system from the current primary system
to the secondary system.
If your primary data center is not available (for example, because of a disaster or because of planned
downtime) and a decision has been made to take over to the secondary data center, you can perform a
takeover on your secondary system. To help you decide if a takeover is advisable, see the decision tree in SAP
Note 2063657.
We recommend to use third-party external tools to check if hosts, the network, and data center are still
available. In addition, you can use python scripts to decide when a takeover should be carried out. For a
detailed description of the available python scripts, see Checking the System Replication Status.
Once the takeover command runs, the former secondary system becomes the new primary system – more
correctly, it becomes your new actively running system.
Note
The takeover does not include stopping the former primary system. It will still run, if you don't stop it.
You can perform a takeover using the following tools:
● SAP HANA cockpit
For more information, see Perform a Takeover with the SAP HANA Cockpit.
● hdbnsutil
For more information, see Perform a Takeover with hdbnsutil.
● SAP HANA studio
For more information, see Perform a Takeover with the SAP HANA Studio.
86
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
Related Information
Checking the SAP HANA System Replication Status [page 159]
Perform a Takeover with the SAP HANA Cockpit [page 87]
Example: Perform a Takeover with the SAP HANA Cockpit [page 88]
Perform a Takeover with hdbnsutil [page 91]
Perform a Takeover with the SAP HANA Studio [page 92]
Client Connection Recovery After Takeover [page 92]
Invisible Takeover and Restart [page 97]
Takeover with Handshake [page 99]
SAP Note 2063657
4.1.1 Perform a Takeover with the SAP HANA Cockpit
You can perform a takeover on your secondary system using the SAP HANA cockpit.
Prerequisites
● It is recommended to stop the primary system before starting a takeover.
Note
If you are performing a takeover as part of a planned downtime, you should rst make sure that the
primary system has been fully stopped before performing a takeover to the secondary system.
● The secondary system must be fully initialized.
● You need the operating system user to perform a takeover with the SAP HANA cockpit. For more
information, see Operating System User <sid>adm and Connect to a Resource using Database Credentials.
● The takeover command can be executed both when the secondary system is oine or online.
Procedure
1. On the Overview page of the secondary system meant to perform the takeover, choose the System
Replication tile.
The System Replication tile opens displaying the System Replication Overview.
2. Choose Take Over on the top right.
3. To start the takeover, click Ok in the Takeover dialog.
You can also start a takeover with handshake by choosing to fully synchronize the secondary system. For
more information about the takeover with handshake, see Takeover with Handshake.
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 87
4. Stop the primary system from the Overall Database Status.
Related Information
Connect to a Resource using Database Credentials
Operating System User sidadm
Takeover with Handshake [page 99]
4.1.1.1 Example: Perform a Takeover with the SAP HANA
Cockpit
Learn how to perform a takeover with the SAP HANA cockpit.
Prerequisites
You need the operating system user to perform a takeover with the SAP HANA cockpit. For more information,
see Operating System User <sid>adm and Connect to a Resource using Database Credentials.
Context
This examples shows how to perform a takeover from the primary system Site A (Host Id2131) to the
secondary system Site B2 (Host Id2132).
For a planned takeover (for example, within the process of a near zero downtime upgrade) stop the primary
system (Site A) from the Overall Database Status tile.
88
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
On the Manage Services page, you should see that the primary system (Site A) was stopped.
Procedure
1. Choose the system replication tile on the secondary system's (Site B2) overview page.
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 89
The System Replication Overview opens.
2. Choose Take Over on the top right.
You can also choose to start a takeover with handshake. For more information, see Takeover with
Handshake.
After the takeover is nished, this system is not functioning as a secondary system anymore. It becomes
the new primary.
90
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
4.1.2 Perform a Takeover with hdbnsutil
You can perform a takeover on your secondary system with the hdbnsutil command line tool.
Prerequisites
● The secondary system must be fully initialized.
You can check this in M_SERVICE_REPLICATION or in the SAP HANA studio Administration Console
Landscape System Replication . The secondary system is ready for takeover if all services display
REPLICATION_STATUS ACTIVE.
● The takeover command can be executed both when the secondary system is oine or online.
Note
If you are performing a takeover as part of a planned downtime, you should rst make sure that the primary
system has been fully stopped before performing a takeover to the secondary system.
Procedure
As <sid>adm enter the following command on the secondary system to enable the secondary system to take
over and become the primary system:
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_takeover [--comment="Your Comment"]
Use the --comment to add a reason for the takeover (for example, to distinguish between a planned or
unplanned takeover). This comment is displayed in the M_SYSTEM_REPLICATION_TAKEOVER_HISTORY
monitoring view in the COMMENTS column.
If the system is oine, the takeover is actually carried out when the system is started again.
Related Information
Stop a System
Monitoring SAP HANA Systems During Stop and Start
M_SERVICE_REPLICATION System View [page 269]
M_SYSTEM_REPLICATION_TAKEOVER_HISTORY System View [page 279]
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 91
4.1.3 Perform a Takeover with the SAP HANA Studio
You can perform a takeover on your secondary system using the SAP HANA studio.
Prerequisites
● The secondary system must be fully initialized.
● The takeover command can be executed both when the secondary system is oine or online.
● You are logged on to the secondary system as the operating system user (user <sid>adm) or can enter
these credentials when prompted.
Note
If you are performing a takeover as part of a planned downtime, you should rst make sure that the primary
system has been fully stopped before performing a takeover to the secondary system.
Procedure
1. In the Systems view, right-click the secondary system and choose Conguration and Monitoring
Congure System Replication .
2. Choose Perform Takeover from the actions list.
3. Enter the required system information and choose Next.
4. Review the information and choose Finish.
Results
The secondary system is now the production system. If the system is already running, it comes out of recovery
mode and becomes fully operational immediately: it replays the last transaction logs and starts to accept
queries. If the system is oine, it takes over production operation when you start it.
4.1.4 Client Connection Recovery After Takeover
Connection recovery after a takeover can be done with network-based IP redirection or network-based DNS
redirection.
After a takeover, the client or the application server need to be able to continuously reach the SAP HANA
system, no matter which system is currently the primary system after takeover.
After a takeover, the new primary database server is not aware of previous connections which existed between
clients and the former primary server. If the client application does not issue a new request and keeps waiting
92
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
for a reply from the server, it will not receive an explicit request to close these connections from either of the
servers and will keep waiting indenitely. To prevent this, the SAP HANA client library supports the TCP
keepalive feature provided by the operating system. This feature will lead the client to abort the invalid
connection on its end and to trigger a reconnect after a specied period during which the former primary
server is not reachable.
However, the default keepalive settings for the operating system (2 hours) may lead the client processes to wait
for a long time before they abort the connection on their end and trigger a reconnect with the new primary
system. For example, the default Linux settings leave the clients waiting for more than two hours before
aborting the connection. For more information, see SAP Note 2053504. For instructions on how to congure
the keepalive settings to match your needs, see the corresponding documentation for your operating system.
There are dierent possibilities for enabling client connection recovery:
● IP redirection
A virtual IP address is assigned to the virtual host name. In case of a takeover, the virtual IP will unbind from
the network adapter of the primary system and bind to the adapter on the secondary system.
● DNS redirection
In this scenario the IP for the host name in the DNS will be changed from the address of the primary to the
address of the secondary system.
Both methods have their advantages. If there are no existing constraints, the IP redirection has the clear
benet of being faster to process in a script rather than synchronizing changes of DNS entries over a global
network.
Network-based IP Redirection
The principle of IP redirection is to dene an additional "logical" host name (hana1, in the diagram below) with
its own separate logical IP address (10.68.105.51), and then map this initially to the MAC address of the original
host in the primary system (by binding it to one of the host's interfaces). As part of the takeover procedure, a
script is executed which re-maps the unchanged logical IP address to the corresponding takeover host in the
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 93
secondary system. This must be done pair-wise, for each host in the primary system. The remapping aects
the L2 (OSI layer 2: data link ) switching, as can be seen in step 4 of the following diagram:
IP redirection can be implemented using a number of actual techniques, for instance with the use of Linux
commands which aect the network ARP tables (ip addr add/del…), by conguring L2 network switches
directly, or by using cluster management software. Following the IP redirection conguration, the ARP caches
should be ushed, to provide an almost instantaneous recovery experience to clients.
94
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
IP redirection requires that both the primary and failover host(s) are on the same L2 network. If the standby
system is in a completely separate L3 network, then DNS redirection is the preferred alternative solution.
Network-based DNS Redirection
DNS redirection is an alternative to IP redirection. DNS is a binding from a logical domain name to an IP
address. Clients contact a DNS server to obtain the IP address of the SAP HANA host (step 1 below) they wish
to reach. As part of the takeover procedure, a script is executed that changes the DNS name-to-IP mapping
from the primary host to the corresponding host in the secondary system (pair-wise for all hosts in the
system). From that point in time, clients are redirected to the failover hosts, as in step 2 of the following
diagram:
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 95
This solution shares the advantage with IP redirection that there are no clientspecic congurations. Further, it
supports disaster recovery congurations where the primary and secondary standby systems may be in two
completely dierent network domains (separated by routers). One drawback of this solution is that modifying
DNS mappings requires a vendor-proprietary solution. Further, due to DNS caching in nodes (both clients and
intermediate network equipment), it may take a while (up to hours) until the DNS changes are propagated,
causing clients to experience downtime despite the recovery of the system.
HA/DR providers can inform external entities about activities inside SAP HANA scale-out (such as Host Auto-
Failover) and SAP HANA system replication setups. In a Python script actions can be dened which should be
executed before or after certain activities (for example, startup, shutdown, failover, takeover, connection
changed, service state changed). One example for these so-called hooks is moving virtual IP addresses after
takeover in SAP HANA system replication. Additionally, external cluster management software can be used to
perform the client reconnect after takeover. For more information, see Implementing HA/DR Providers.
Related Information
Implementing a HA/DR Provider
Connecting Using Active/Active (Read Enabled)
SAP Note 2053504
96
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
4.1.5 Invisible Takeover and Restart
During an invisible takeover or a restart, the session's state needs to be recovered and restored to the new
primary system.
During a standard takeover you switch your active system from the current primary system to the secondary
system. After a standard takeover, the primary system loses all connections to the client. Moreover, the
secondary system is not aware of the previous connections, which existed between the client and the primary
system. This is dierent in an invisible takeover.
You can perform an invisible takeover to achieve an automatic recovery of your sessions after takeover to your
new primary system. For dedicated client applications this takeover will be invisible. Dierently from a standard
takeover, an invisible takeover ensures that the client reconnects to the primary system and the sessions are
restored to the secondary system.
This seamless recovery is possible also when restarting the system (for example, after a system crashes).
The session's state needs to be recovered and restored to the new primary system in an invisible takeover
scenario or to the new system in a restart scenario. The cross-layer between the session and the client library
makes the seamless recovery possible. This cross-layer feature called transparent session recovery recovers
the current session's state and the physical connection.
Note
The transparent session recovery is supported by SQLDBC for SAP HANA 2.0.
Use the graphic below to better understand this process in an invisible takeover or a restart scenario after the
active system crashed:
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 97
Conguration
The enable_session_recovery parameter in the session section of the ini le controls the session recovery.
The default value is true recovering all session variables and restoring the client connections from the primary
system to the secondary system. This parameter is congurable online, but the changes can be applied only to
the connections established after making the changes.
Limitations
● Sessions which have created or updated a global temporary table with any DDL or DML commands won’t
be recovered. However, sessions which have created a local temporary table will be recovered without the
table recovery.
● Ongoing write transaction will be rolled back with an error and the session can be recovered when an
application restarts the failed transaction with no explicit reconnect trial from the application
● Almost all session variables from the current session context are recovered.
● When a response for a request is not successfully sent from the client to the server, the session is not
recovered. However, sessions are still recovered when a SQL command is not sent from the client to the
server.
98
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
4.1.6 Takeover with Handshake
The takeover with handshake ensures that all the sent redo log is written to disk on the secondary system.
During a planned takeover, it is important to ensure that no data gets lost (all primary updates must be
available on the secondary system) and the former primary system is isolated to avoid a split-brain situation
with multiple active primary systems.
The takeover with handshake is ideal for a safe planned takeover while the primary is still running. All new
writing transactions on the primary system are suspended and the takeover is only executed when all redo log
is available on the secondary system. When performing a takeover with handshake, it is not needed to check
the replication status or to stop the old primary before the takeover.
In a nutshell, a takeover with handshake avoids:
● Data loss, because the log is available on the secondary system before the takeover is triggered.
● Split-brain situations, because the former primary will be suspended.
Note
The takeover with handshake will only be performed if the two previously mentioned conditions are
guaranteed. Otherwise, the takeover will be aborted and the primary resumed.
Trigger a takeover with handshake using hdbnsutil -sr_takeover -–suspendPrimary on the secondary
system.
If a primary service cannot be accessed or a service replication is not active or in sync, the takeover will be
aborted and reported as an error. In this case, there is no impact on the system and the replication remains as
it was. The suspended primary can be unblocked using the -sr_register hdnsutil command.
Note
The following limitations apply:
● It is supported only on the second tier.
● It is not supported with Dynamic Tiering services.
Related Information
Use SAP HANA System Replication for Near Zero Downtime Upgrades [page 199]
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 99
4.2 Failback
The failback process is the name for the task of registering the former primary system as a new secondary
when it becomes available again.
After a takeover has been carried out, the roles between primary and secondary can be switched over. In the
case of a failback, the former primary has to be registered as the secondary with the now active primary
system. The roles are switched compared to the original setup.
This is the same procedure as for setting up a normal secondary described in Conguring SAP HANA System
Replication. However, in this scenario when the new secondary is registered with the new primary, it checks if a
delta shipping is possible to resync the two sites rather than carrying out a full data shipping. If a delta shipping
is possible, it only ships the delta, which signicantly reduces the initialization time during registration of the
new secondary. When the new secondary starts, it checks rst if there is a local snapshot available from the
time when the system was the primary system. If a snapshot is available, the system then checks if it is
compatible with the new primary. When both checks are positive, the new secondary can be initialized with a
delta replica from the new primary.
You can perform a failback using the following tools:
● SAP HANA cockpit
For more information, see Perform a Failback with the SAP HANA Cockpit.
● hdbnsutil
For more information, see Perform a Failback with hdbnsutil.
● SAP HANA studio
For more information, see Perform a Failback with the SAP HANA Studio.
Related Information
Perform a Failback with the SAP HANA Cockpit [page 101]
Example: Perform a Failback with the SAP HANA Cockpit [page 102]
Perform a Failback with the SAP HANA Studio [page 104]
Perform a Failback with hdbnsutil [page 105]
Virtual IP Address Handling [page 125]
100
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
4.2.1 Perform a Failback with the SAP HANA Cockpit
To perform a failback in the SAP HANA cockpit, register the former primary system as secondary to the current
primary.
Prerequisites
● You need the operating system user to perform a failback with the SAP HANA cockpit. For more
information, see Operating System User <sid>adm and Connect to a Resource using Database Credentials.
● The former primary system is not running.
● The current primary system is running.
Context
In the SAP HANA cockpit you can perform a failback either from the current primary system or from the former
primary system (this is the future secondary). The conguration steps are the same as described in Congure
System Replication from the Primary System or Congure System Replication from the Primary and the
Secondary System.
This procedure describes how to register the former primary as a new secondary. Use the System Replication
card on the System Overview page of the former stopped primary to register this system as a new secondary.
Procedure
1. Register the secondary system as follows:
a. On the System Overview page of the former primary system, choose the System Replication tile.
The System Replication page opens.
b. Choose Register as Secondary on the top right.
The Register Secondary System page opens.
c. On the Register Secondary System page enter the logical name used to represent the secondary
system.
d. Select a replication mode. For more information on the available replication modes, see Replication
Modes for SAP HANA System Replication.
e. Select an operation mode. For more information on the available operation modes, see Operation
Modes for SAP HANA System Replication.
f. Enter the host of the source system.
Note
If you are operating a distributed system on multiple hosts, enter the name of the host on which
the master name server is running.
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 101
g. Check Start Secondary after Registration.
2. Review the congured information and choose Congure on the bottom right.
Results
The original primary system is now registered as the secondary system with the current primary system (that
is, the original secondary system). The secondary system is getting in sync again with the primary system. As
such, it is attempting to avoid a full data shipping.
Verify that the secondary system replication status is All services are active and in sync.
Related Information
Perform a Failback with the SAP HANA Cockpit [page 101]
Congure SAP HANA System Replication from the Primary and the Secondary Systems [page 45]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
Connect to a Resource using Database Credentials
Operating System User sidadm
4.2.1.1 Example: Perform a Failback with the SAP HANA
Cockpit
Learn how to perform a failback with the SAP HANA cockpit.
Prerequisites
You need the operating system user to perform a failback with the SAP HANA cockpit. For more information,
see Operating System User <sid>adm and Connect to a Resource using Database Credentials.
Context
This example shows how to perform a failback by registering the former primary (Site A) as a secondary
system to the new primary (Site B2).
102
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
Procedure
1. Choose the system replication tile of the stopped primary system (Site A) and choose Register as
Secondary on the overview page.
This will register the primary system (Site A) as a secondary system.
2. On the conguration page, add the name of the former primary system (Site A), the replication and
operation modes, as well as the master host name of the former secondary system (Id2132). Finally,
choose Congure.
The conguration can look like this:
After the conguration is completed, the roles switch: the former primary (Site A) runs as a secondary
system to the new primary (Site B2). This new primary (Site B2) is the former secondary system. You can
verify the switch on the overview page.
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 103
4.2.2 Perform a Failback with the SAP HANA Studio
You can perform a failback using the SAP HANA studio.
Prerequisites
● You are logged on to both systems as the operating system user (user <sid>adm) or are able to enter these
credentials when prompted.
● The former primary system is not running.
● The current primary system is running.
Context
To fail back, you must attach your former primary system as new secondary system to the current primary
system.
Procedure
1. Register the former primary system as the new secondary system as follows:
a. In the Systems view, right-click the primary system and choose Conguration and Monitoring
Congure System Replication .
The Congure System Replication dialog opens.
Note
You can also access the Congure System Replication dialog from the Landscape System
Replication tab.
b. Choose Register Secondary System and then Next.
c. Enter the required system information and the logical name used to represent the system.
Note
If you are operating a distributed system on multiple hosts, you enter the name of the host on
which the master name server is running.
d. Specify the log replication mode. For more information on the available replication modes, see
Replication Modes for SAP HANA System Replication.
e. Specify the operation mode. For more information on the available operation modes, see Operation
Modes for SAP HANA System Replication.
f. Review the congured information and choose Finish.
g. If necessary, start the former primary system.
104
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
Note
The former primary system is started automatically unless you deselected the corresponding
option during conguration.
The former primary system is now registered as the secondary system with the current primary system
(that is, the former secondary system). As the data that is already available in the former primary system
cannot be reused, a complete initialization is carried out. This means that a full data replication takes place
until the former primary system is fully in sync.
2. Verify that the secondary system replication status is All services are active and in sync.
You can see this status in the Administration editor on the Overview tab.
Results
The former primary system and the former secondary system switched their roles.
Related Information
Conguring SAP HANA System Replication [page 38]
Replication Modes for SAP HANA System Replication [page 12]
Operation Modes for SAP HANA System Replication [page 14]
4.2.3 Perform a Failback with hdbnsutil
You can perform a failback using the hdbnsutil command line tool.
Prerequisites
● You need the operating system user to perform a failback. For more information, see Operating System
User <sid>adm.
● The former primary system is not running.
● The current primary system is running.
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
P U B L I C 105
Context
This is the same procedure as is used for setting up a normal secondary described in Congure SAP HANA
System Replication with hdbnsutil. To fail back, attach your original primary system as new secondary system
to the current primary system. .
Procedure
When the former primary is available again and oine, it can be registered as the new secondary:
hdbnsutil -sr_register --remoteHost=<new primary hostname>
--remoteInstance=<instance number>
--replicationMode=<sync/syncmem/async>
--operationMode=<delta_datashipping|logreplay|logreplay_readaccess>
--name=<siteName>
Related Information
Operating System User sidadm
Congure SAP HANA System Replication with hdbnsutil [page 51]
106
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Takeover and Failback
5 SAP HANA System Replication:
Secondary Time Travel
Learn how to quickly access again data, which was deleted in the original system.
Secondary time travel allows you to start the secondary system or the log replay at a previous point in time.
Related Information
Secondary Time Travel [page 107]
Execute Secondary Time Travel [page 108]
Execute Secondary Time Travel While Replication Continues [page 109]
Conguration Parameters [page 110]
Monitoring Secondary Time Travel [page 112]
5.1 Secondary Time Travel
You can start the secondary system or the log replay at a previous point in time.
Secondary time travel can be used:
● To quickly access again data, which was deleted in the original system. For more information, see Execute
Secondary Time Travel.
● To intentionally keep the secondary system's log replay delayed. This can be used to read older data from
the secondary system, while the secondary keeps replicating. For more information, see Execute
Secondary Time Travel While Replication Continues.
To prepare the secondary system for time travel, snapshots are kept on the secondary system for a dened
time travel period. These snapshots can be used later to start the system at an older point in time. Additional
log will be retained on the secondary system starting from the oldest time travel snapshot. After opening the
old snapshot, the additional log has to be replayed to reach the requested point in time.
Note
You can use secondary time travel only with the operation modes logreplay or
logreplay_readaccess. Conguring the SYNC replication mode with the full sync option, does not lead
to a freeze of the primary system upon secondary time travel.
To prepare the system for time travel, the global.ini/[system_replication]/
timetravel_max_retention_time parameter must be congured on the secondary system. This
parameter denes the time period to which the secondary system can be brought back in the past. Optionally,
SAP HANA System Replication
SAP HANA System Replication: Secondary Time Travel
P U B L I C 107
the global.ini/[system_replication]/timetravel_snapshot_creation_interval parameter can
be adapted to adjust the secondary's snapshot creation. After setting these parameters, the secondary starts
retaining log and keeping created snapshots.
Related Information
Execute Secondary Time Travel [page 108]
Conguration Parameters [page 110]
Execute Secondary Time Travel While Replication Continues [page 109]
Monitoring Secondary Time Travel [page 112]
5.2 Execute Secondary Time Travel
You can start the secondary system in online mode at a previous point in time to access again data, which was
deleted in the original system.
Prerequisites
● Set the requiered parameters to dene the point in time to which the secondary system can be brought
back in the past. For a full list of available parameters for secondary time travel, see Conguration
Parameters.
Note
Set the parameters carefully to avoid log full or disk full situations. For time travel to work, log and
snapshots are kept online in the data area. Because of this, log and data will grow on the secondary
system when time travel is turned on. The system workload determines how much data is needed.
● Use the logreplay or the logreplay_readaccess operation mode in your system replication setup.
Procedure
1. Stop the secondary system.
2. Execute hdbnsutil -sr_timetravel --startTime=<startTime> [--callTakeoverHooks=on|
off] [--comment="Your Comment"]
For startTime use the following format specied in UTC: dd.mm.yyyy - hh.mm.ss
You can specify if takeover hooks should be called. If the timetravel_call_takeover_hooks
parameter is not explicitly specied, takeover hooks won't be called. For more information on takeover
hooks, see Implementing a HA/DR Provider.
108
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Secondary Time Travel
Use the --comment to add a reason for the time travel. This comment is displayed in the
M_SYSTEM_REPLICATION_TAKEOVER_HISTORY monitoring view in the COMMENTS column.
3. Start the secondary system.
The secondary system will enter in online mode at the specied point in time during restart. After restart,
the other services read the requested point in time and open their persistence using this information. If the
requested point in time cannot be reached, then time travel will be aborted. A check ensures that there are
time travel snapshots older than the start time for each service.
Related Information
Conguration Parameters [page 110]
M_SYSTEM_REPLICATION_TAKEOVER_HISTORY System View [page 279]
Implementing a HA/DR Provider
5.3 Execute Secondary Time Travel While Replication
Continues
You can start the log replay at a previous point in time to read older data from the secondary system, while the
secondary keeps replicating.
Prerequisites
● Set the required parameters to dene the point in time to which the secondary system can be brought
back in the past. For a full list of available parameters for secondary time travel, see Conguration
Parameters.
Note
Set the parameters carefully to avoid log full or disk full situations. For time travel to work, log and
snapshots are kept online in the data area. Because of this, log and data will grow on the secondary
system when time travel is turned on. The system workload determines how much data is needed.
● Use the logreplay or logreplay_readaccess operation modes.
Procedure
1. Stop the secondary system.
2. Execute hdbnsutil -sr_timetravel --startTime=<startTime> --startMode=replicate
SAP HANA System Replication
SAP HANA System Replication: Secondary Time Travel
P U B L I C 109
For startTime use the following format specied in UTC: dd.mm.yyyy - hh.mm.ss
3. Start the secondary system.
After the system has started, the persistence has been opened on the specied point in time, it is
replicating log, and log replay is not running.
4. Optional: Trigger the log replay manually with hdbnsutil -sr_recoveruntil {--
endTime=<timestamp>|max} [--nowait]
For <timestamp> use the following format specied in UTC: dd.mm.yyyy - hh.mm.ss
Use max to trigger the log replay up to the newest possible point in time. In this case, the target timestamp
is automatically determined by checking the valid time travel range for each service.
Use --nowait to specify if the command should be executed asynchronously.
5. Optional: Stop the manual replay mode by setting the timetravel_logreplay_mode parameter back to
auto or using hdbnsutil -sr_replaymode --mode={auto|manual}
Related Information
Conguration Parameters [page 110]
5.4 Conguration Parameters
Use the following parameters to prepare your system for secondary time travel.
The parameters are dened in the system_replication section of the INI le. All parameters are set on the
secondary system.
Parame
ter timetravel_max_retention_time
Type integer
Unit minutes
Default 0
Descrip
tion
If set to 0, secondary time travel is turned o. If this parameter is set to a value dierent from 0, the secondary
system can be brought online up to the dened point in the past.
Parame
ter timetravel_snapshot_creation_interval
Type integer
110 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Secondary Time Travel
Parame
ter timetravel_snapshot_creation_interval
Unit minutes
Default 1440
Descrip
tion
Denes how frequently snapshots are created for secondary time travel. Time travel snapshots are kept until
they get older than the dened
timetravel_max_retention_time parameter. If time travel needs to be
done on an older point in time, the snapshot that best ts the requested point in time will be opened and the
remaining changes are applied via log replay.
A new snapshot is created when the point in time dened in this parameter has passed since the last snapshot
creation. Snapshots older than the point in time dened in
time_travel_max_retention_time are
dropped.
Parame
ter timetravel_call_takeover_hooks
Type bool
Values true, false
Default false
Descrip
tion
Indicates if takeover hooks should be called during secondary time travel.
Parame
ter timetravel_logreplay_mode
Type enum
Values auto, manual
Default auto
Descrip
tion
Denes how the log replay is executed on the secondary system.
The following settings are allowed:
● Auto
The log replay is done automatically up to the newest possible log position.
● Manual
You must manually trigger the log replay up to the requested timestamp using the -sr_recoveruntil
hdbnsutil command.
SAP HANA System Replication
SAP HANA System Replication: Secondary Time Travel
P U B L I C 111
5.5 Monitoring Secondary Time Travel
You can monitor the retaining log and the created snapshots.
To monitor secondary time travel, the secondary system must be online. The current time travel range cannot
be determined, when the secondary is oine.
You can determine the valid range for which time travel can be executed in two ways:
● Using hdbnsutil -sr_timetravel --printRange
This command provides a range for each service in which time travel can be executed:
Value
Description
START_TIME Contains the oldest possible point in time for which timetravel can be executed.
As time travel is done for all services, the intersection of all ranges have to be checked to make
sure, all services can reach the specied timestamp.
END_TIME
Contains the last possible point in time for which timetravel can be executed.
For slave services, this timestamp can be some time back without being outdated, if there was
no more activity on this slave for some time. To ignore the slave volumes, only the lines for
transaction master can be considered. Those are marked as MASTER in the COORDINA
TOR_TYPE column.
● Using SQL on the primary system via the
_SYS_DATABASES_SR_SITE_<sitename>.M_SYSTEM_REPLICATION_TIMETRAVEL secondary proxy view.
The start time or the log position of the system can be monitored using
M_SYSTEM_REPLICATION_TAKEOVER_HISTORY.
112
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Secondary Time Travel
6 SAP HANA System Replication with
Active/Active (Read Enabled)
Active/Active (read enabled) enables SAP HANA system replication to support read access on the secondary
system.
What can I learn about Active/Active (Read Enabled) in this section?
After checking all the necessary prerequisites, learn how to congure and monitor an Active/Active (read
enabled) system replication. This chapter also provides information about memory management, virtual IP
address handling, and authentication methods.
Where can I nd more information?
The following SAP Notes are relevant for Active/Active (Read Enabled):
SAP Notes
SAP Note Title
2447994
SAP HANA Dynamic Tiering Support for SAP HANA System Replication
1681092
Multiple SAP HANA DBMSs (SIDs) on one SAP HANA system
2116157
FAQ: SAP HANA Consistency Checks and Corruptions
2391079
Access restrictions in Active/Active (read enabled) system setup
Related Information
Active/Active (Read Enabled) System Replication [page 114]
Generic Conditions for Active/Active (Read Enabled) [page 116]
Connection Types [page 119]
Conguring an Active/Active (Read Enabled) System Replication [page 117]
Memory Management [page 124]
Virtual IP Address Handling [page 125]
Authentication Methods [page 126]
Monitoring Active/Active (Read Enabled) [page 127]
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 113
6.1 Active/Active (Read Enabled) System Replication
Active/Active (read enabled) enables SAP HANA system replication to support read access on the secondary
system and is supported by all client APIs.
Active/Active (Read Enabled)
Active/Active (read enabled) reduces the load on the primary system but does not double the capacity; it
simply extends read capabilities. In an Active/Active (read enabled) system replication conguration, the SQL
ports on the secondary system are open for read access. This makes it possible to use the secondary system
for read-intense tasks and to have a better balance of workloads improving the overall performance of the SAP
HANA database.
Note
In case of an outage, all functions concentrate on the secondary system. Because of this, the sizing of the
secondary system is important to ensure the right performance in disaster scenarios.
Active/Active (read enabled) is integrated into the SAP HANA system replication solution and gets activated
with the operation mode logreplay_readaccess. This operation mode provides fast takeovers, reduced
need for bandwidth in continuous operation, and support for replication modes such as SYNC (with or without
the full sync option), SYNCMEM, and ASYNC. The redo log replay runs asynchronously to the primary
operations.
Note
To use the logreplay_readaccess operation mode, the primary and the secondary systems must have
the same SAP HANA version. For this reason, read-only access to the secondary system is not possible
during a rolling upgrade until both versions are the same again.
The following graphic focuses on the Active/Active (read enabled) conguration. The primary system is fully
active and supports reading and writing, while the secondary system is enabled for read queries.For a detailed
illustration of the general system replication processes, see Introduction.
114
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
In a multitier setup, read access on the secondary is only supported for tier 2. In a 3-tier system using Active/
Active (read enabled), the logreplay_readaccess mode is required between the primary and the active
secondary systems, while the logreplay mode is required between the other (tier 2 and tier 3) secondary
systems. In a multitarget setup, multiple secondary systems with logreplay_readaccess are supported.
License Management
In an Active/Active (read enabled) conguration, the secondary system is operated automatically with the
license key of the primary system. Changes of the license key are done on the primary system and replicated to
the secondary system. For more information, see Managing SAP HANA Licenses.
Related Information
Introduction [page 9]
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 115
Conguring SAP HANA System Replication
Operation Modes for SAP HANA System Replication [page 14]
Client Support for Active/Active (Read Enabled)
SAP HANA SQL Reference
Managing SAP HANA Licenses
https://www.youtube.com/embed/ZRtVkuLuLuM
SAP Note 2391079
6.2 Generic Conditions for Active/Active (Read Enabled)
When using the secondary system for read access, several aspects need to be considered.
Points to Consider
● The processors in the primary and secondary systems must be both either Intel-based or IBM Power-
based with the same byte ordering. A platform mixture is not supported.
● The secondary system allows read access if the primary system runs the same SAP HANA version. A
dierent version leads to prohibiting the read access until the same software version is used.
● The redo log replay runs as an asynchronous process on the secondary system. The secondary system
provides statement level snapshot isolation with potentially delayed view on the data and no minimum
delay guarantee.
● The secondary system gets its own virtual IP addresses or host names representing the secondary
function.
● The query execution in the secondary system is rejected if it needs background migrations requiring redo
log writes (e.g. L2-Delta migration).
Recommendation
Perform the migration in the primary system (for example, load table) and wait until it is replicated and
replayed in the secondary system.
● Internal processes for operations like ColumnStore delta merges take place on the secondary system.
● DML execution on an Active/Active (read enabled) secondary system is possible for row store no-logging
retention tables and global temporary tables. Their Explain Plan is also available on the secondary system.
For more information, see Data Manipulation Statements.
Limitations
● Active/Active (read enabled) is supported in a multitier SAP HANA replication system. However, read
access is limited to tier 2. The logreplay operation mode is required between tier 2 and the further tier
level and no read access connections can be opened to the further tier level.
116
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
● If Active/Active (read enabled) is used with Dynamic Tiering services, there is no read access to Dynamic
Tiering data on the secondary system.
● The export of tables is possible with CSV as target. However, binary exports on the secondary system are
not supported.
● The use of workload classes is not supported on the secondary system.
Support for Multiple SAP HANA Databases
It is possible to use the read-enabled secondary system for other SAP HANA systems such as development or
QA environments. In this case the following sizing conditions apply:
● The secondary hardware must oer the same CPU and memory capacities as those oered by the primary
system plus the resources for the additional system.
● After a takeover, the system must be capable of handling both the primary’s writing load and the
secondary’s reporting load.
For more information about this scenario, see also SAP Note 1681092 Multiple SAP HANA DBMSs (SIDs) on
one SAP HANA system.
Related Information
Data Manipulation Statements
SAP Note 1681092
SAP Note 2447994
6.3 Conguring an Active/Active (Read Enabled) System
Replication
Active/Active (read enabled) is integrated into the SAP HANA system replication solution and gets activated
with the operation mode logreplay_readaccess.
To congure an Active/Active (read enabled) system replication, follow the procedure described for conguring
system replication and select the operation mode logreplay_readaccess.
For more information about checking the Active/Active (read enabled) conguration, see Check the Active/
Active (Read Enabled) Conguration.
Related Information
Conguration Parameters [page 118]
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 117
Conguring SAP HANA System Replication
Checking the Active/Active (Read Enabled) Conguration [page 118]
https://www.youtube.com/embed/jR4GBWak1n0
6.3.1 Conguration Parameters
Several parameters are available for conguring Active/Active (read enabled).
Parameter: operation_mode
Values: logreplay_readaccess
System: Secondary
Description: System Replication uses an initial data shipping to initialize the secondary system. After that, only log ship
ping is done and the log buers received by the secondary system are being replayed there. Savepoints are
executed individually for each service. Column table merges are executed on the secondary system. Addi
tionally, read access via SQL is provided to the secondary system.
Relevant for Active/Active (read enabled) are also enable_log_retention and
logshipping_max_retention_size. For more information about these parameters, see SAP HANA System
Replication Conguration Parameters.
Related Information
SAP HANA System Replication Conguration Parameters [page 62]
6.3.2 Checking the Active/Active (Read Enabled)
Conguration
You can check if your system replication is congured as an Active/Active (read enabled) system.
SAP HANA Cockpit
On the System Replication tile on the primary system, the logreplay_readaccess operation mode indicates
that your system is an Active/Active (read enabled) system. Additionally, an enabled secondary read access
informs you that the SQL ports are open for reading on the Active/Active (read enabled) secondary system.
118
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
On the system overview page of the secondary system, the Mode: read-only indicates that your system is an
Active/Active (read enabled) system. Additionally, the Delay in ms is shown on top indicating how far behind is
the consistent view on the data of this secondary system compared to the current data of the primary system.
For examples and more information, see System Replication Tile and Example: Monitoring SAP HANA System
Replication with SAP HANA Cockpit.
SAP HANA Studio
On the primary system, select M_SYSTEM_REPLICATION from the monitoring view. To nd out if your system
is an Active/Active (read enabled) system, verify the columns OPERATION_MODE and
SECONDARY_READ_ACCESS_STATUS.
Command Line
As <sid>adm run one of the following commands and then look for the operation mode
logreplay_readaccess:
python
$DIR_INSTANCE/exe/python_support/systemReplicationStatus.py --
sapcontrol=1 | grep OPERATION_MODE
service/ld4144/30207/OPERATION_MODE=logreplay_readaccess
service/ld4144/30201/OPERATION_MODE=logreplay_readaccess
service/ld4144/30203/OPERATION_MODE=logreplay_readaccess
or
hdbnsutil -sr_state | grep "operation mode"
operation mode: logreplay_readaccess
Related Information
System Replication Tile [page 40]
Example: Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 177]
6.4 Connection Types
Connecting to an Active/Active (read enabled) system allows you to take advantage of a secondary system for
better overall performance.
There are two ways to access the read-enabled secondary system::
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 119
● Opening an explicit connection to the secondary system
● Executing a SQL statement on the primary system which is redirected to the secondary system according
to a hint
Explicit read-only connection to the secondary system
The application opens an explicit connection to the secondary system. There is no session property sharing.
The following graphic illustrates an explicit read-only connection to the secondary system:
Implicit hint-based statement routing
The application connects directly to the primary system. On this connection, SQL statements with system
replicationspecic hints are prepared and executed. They are then automatically routed to and executed by
the secondary system. For more information about hints, see the SAP HANA SQL and System Views Reference
Guide.
120
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
Note
Hint-based statement routing is supported by SAP HANA ODBC, SQLDBC, ADO.Net, JDBC drivers for SAP
HANA 2.0 and the SAP HANA Node.js.
The SAP HANA client opens an additional connection to the secondary system according to the host
information returned by the primary system.
This connection type unfolds as follows:
1. The SAP HANA client sends the statement-prepare with hint to the primary system.
2. The primary system decides where to execute the statement and returns the result to the SAP HANA
client.
3. The SAP HANA client sends the statement execution call to the secondary system. The session property
changes are handed over to the secondary system via the SAP HANA client. If the secondary system
cannot execute the statement, it returns an error and the SAP HANA client sends the statement to the
primary system.
The following graphic illustrates an implicit hint-based statement routing:
For more information about hint-based statement routing, see Hint-Based Routing for Active/Active (Read
Enabled).
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 121
To cancel long-running sessions on a read enabled secondary system, use: ALTER SYSTEM CANCEL
SESSION.
Limitations:
● The SAP HANA client has an open working connection to the primary system.
● Hint-based statement routing is not supported for CALL procedures.
● Temporary tables are not supported.
● Hint-based statement routing is not supported for write transactions.
Related Information
SAP HANA SQL and System Views Reference
Client Requirements For A Takeover [page 122]
Hint-Based Statement Routing for Active/Active (Read Enabled) [page 123]
Client Support for Active/Active (Read Enabled)
6.4.1 Client Requirements For A Takeover
Congure the client connection so that the client connects to the correct system after failover occurs.
Failover is when a secondary system takes over from a primary system that is oine. Use the following
conguration to ensure that the client connects to the correct system when a failover occurs.
● If you are using connection distribution, then you must use the siteType connection property to specify
whether the connection is made to the PRIMARY or SECONDARY system. The siteType connection
property is required because connection distribution adds server locations from cached topology, which
may be out of date because it contains data from before the takeover occurred.
● If you are not using connection distribution, then specify one or more virtual IP addresses (if available) that
always point to the desired type of system. For example, if you are connecting to the primary system, then
these virtual IP addresses must map to the new primary system (the old secondary system that took over
as the primary system) after a failover so that the client can reliably connect to the correct system.
● If you are not using connection distribution and there are no virtual IP addresses, then you must use the
siteType connection property to specify whether the connection is made to the PRIMARY or SECONDARY
system. To reliably connect both before and after a failover, you must specify one or more locations for
both systems (the system that was initially the primary and the system that was initially the secondary).
For example, to connect to the current primary system using an ODBC-style connection string, use
siteType=PRIMARY;serverNode=<site1host>:<port>,<site2host>:<port>, where
<site1host>:<port> is a location for the site that was initially the primary system, and
<site2host>:<port> is a location for the site that was initially the secondary system.
Failure to follow these guidelines can result in connecting to the wrong system after a failover (for example,
connecting to the secondary system instead of the primary system).
122
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
Related Information
Client Connection Recovery After Takeover [page 92]
Virtual IP Address Handling [page 125]
6.4.2 Hint-Based Statement Routing for Active/Active (Read
Enabled)
Connections to a primary system can use hint-based statement routing statement execution to a secondary
system on a per-statement basis. This reduces the load on the primary system and increases overall
performance.
To indicate that a statement should be hint-based routed to the secondary system, add the hint text WITH
HINT(RESULT_LAG('hana_sr')) to the end of the SQL SELECT statement. For example:
SELECT C1, C2 FROM T1 WHERE C3 = 'constant value' WITH
HINT(RESULT_LAG('hana_sr'))
Queries that are executed directly (not prepared) are not hint-based routed even if they contain a hint. To take
advantage of hint-based statement routing, there must be separate prepare and execute operations at the
SQLDBC or JDBC level. In some cases, applications or interfaces that use SQLDBC or JDBC (such as SAP
HANA Studio, SAP HANA Cockpit, ABAP, or PyDBAPI) can perform a separate prepare and execute without the
user's knowledge.
Routing Conditions
In the following cases, the execution transparently falls back to the primary system, even if the statement
contains a hint to route to the secondary system:
● The connection's isolation level is set to repeatable read or serializable.
● Connection to the secondary system is not possible (for example, there is a secondary system outage, a
networking issue, and so on).
● The connection currently has a write transaction (uncommitted insert, update, or delete) in progress.
● The query references temporary tables.
If the routed connection is dropped while fetching from a hint-routed query result set, an error may be returned
to the application.
Note
Hint-based statement routing is only applied to SELECT statements.
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 123
Fallback Routing
In the following cases, a statement that has been routed to the secondary system gets re-routed to the primary
system:
● The hint contains a maximum delay time parameter and the secondary system is delayed by more than
that amount.
● The secondary system is near its maximum memory usage.
● The statement prepared in the primary system does not detect any access to the Dynamic Tiering data,
but the statement execution in the secondary system requires Dynamic Tiering data.
Timeout
If the previous hint-based routed statement execution falls back to the primary system due to a connection or
communication error, then future hint-based statement routing does not attempt to re-connect to the
secondary system for several seconds. This avoids the performance cost of retrying the connection to the
secondary system frequently when it is likely to fail.
In this case, the time between reconnection attempts to the secondary system is between ve seconds and ve
minutes from the last reconnection attempt. The time between reconnection attempts automatically increases
if reconnection attempts continue to fail.
In a multitier system replication system, hint-based statement routing always routes from the primary to the
secondary system.
Related Information
HINT Details
Connection Types [page 119]
6.5 Memory Management
Several parameters can be used to set the memory limit for read accesses on the secondary system.
The total statement memory is limited to 50 % of the global allocation limit, because 50% of the storage is
reserved for log replay. Log replay should not fail because of memory limitations.
124
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
Use the parameters below to set the memory limit for read accesses on the secondary system:
Parame
ter: sr_total_statement_memory_limit
Type: int (GB)
Default: (empty)
Section memorymanager
Descrip
tion:
Memory limit in GB:
● (empty): 50% of global allocation limit
● 0: disable the feature
● N: set the value as a limit
Parame
ter: sr_enable_tracking
Type: bool
Default: on
Section resource_tracking
Descrip
tion:
Main switch for resource tracking used on the system replication secondary system.
Parame
ter: sr_memory_tracking
Type: bool
Default: on
Section resource_tracking
Descrip
tion:
Enables or disables memory tracking on the secondary system.
6.6 Virtual IP Address Handling
In an Active/Active (read enabled) conguration, a second virtual IP address for the read access on the
secondary system is needed.
Since in an Active/Active (read enabled) conguration both systems are open for SQL access, a second virtual
IP address for read access on the secondary system is needed.
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 125
During takeover you can keep the virtual IP address of the secondary system. This virtual IP address will be
used for read access until a reconnect occurs. The former virtual IP address of the primary system is also
rebound to access the former secondary system, which is the now active system. In this situation, two virtual IP
addresses are available for accessing the former secondary system after takeover. For more information, see
Client Connection Recovery After Takeover.
Note
Make sure that the now active system is capable to handle the workload of the former primary system and
the read-access secondary system.
During failback the system replication systems switch their roles and the virtual IP addresses switch their
locations too.
Related Information
Client Connection Recovery After Takeover [page 92]
Connecting Using Active/Active (Read Enabled)
Failback [page 100]
6.7 Authentication Methods
There are several authentication methods supported for an Active/Active (read enabled) system replication.
The following authentication methods are supported for the primary system:
● Basic (User Name/Password)
● Kerberos
● SAML
● Session Cookies
The secondary system delegates the authentication phase to the primary system using the existing
communication channel from the secondary system to the primary system. Remote authentication tickets or
credentials are sent over the data centers.
126
P U B L I C
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
6.8 Monitoring Active/Active (Read Enabled)
You can monitor the Active/Active (read enabled) solution using proxy views or the SAP HANA Cockpit.
Proxy views
The embedded statistics server runs in the primary system and collects data from the secondary system
providing them in the corresponding proxy schema. For more information, see Monitoring Secondary Systems.
SAP HANA Cockpit
The monitoring functionality in the SAP HANA cockpit supports access to the read-enabled system.
Additionally, it provides information about the delay of the currently available consistent view. For more
information, see Monitoring SAP HANA System Replication with the SAP HANA Cockpit.
Related Information
Monitoring Secondary Systems [page 155]
Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 175]
SAP HANA System Replication
SAP HANA System Replication with Active/Active (Read Enabled)
P U B L I C 127
7 SAP HANA System Replication Setups
Learn how to congure a multitier or a multitarget system replication.
Which setups are available for system replication?
Besides the standard setup, in which a primary system ships all the data to the secondary system, you can also
congure a multitier or a multitarget system replication.
In a multititer system replication, a tier 2 system replication setup can be used as the source for adding further
tiers in a chain. The primary system is always on tier 1. The replication source for the tier 2 secondary system is
the primary system, while the replication source for the tier 3 secondary system is the tier 2 secondary. For
more information, see SAP HANA Multitier System Replication.
In a multitarget system replication, the primary system can replicate data changes to more than one secondary
system. For more information, see SAP HANA Multitarget System Replication.
Related Information
SAP HANA Multitier System Replication [page 128]
Conguring SAP HANA Multitier System Replication [page 129]
Performing a Takeover and a Failback in SAP HANA Multitier System Replication [page 139]
SAP HANA Multitarget System Replication [page 142]
Example: Congure a SAP HANA Multitarget System Replication [page 144]
Disaster Recovery Scenarios for Multitarget System Replication [page 145]
7.1 SAP HANA Multitier System Replication
To oer higher levels of availability, you can link multiple systems in a SAP HANA multitier system replication
landscape.
You can congure system replication to support geo-clustering, that is multitier system replication between a
primary data center and other geographically remote data centers to form a single highly available system.
Related Information
128
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
Conguring SAP HANA Multitier System Replication [page 129]
Performing a Takeover and a Failback in SAP HANA Multitier System Replication [page 139]
7.1.1 Conguring SAP HANA Multitier System Replication
With multitier system replication, a tier 2 system replication setup can be used as the source for replication in a
chained setup of primary system, tier 2 secondary system, and tier 3 secondary system.
After conguring a basic system replication scenario, you must add a third system to provide another level of
redundancy. In a multitier setup, the primary system is always on tier 1, a tier 2 secondary has a primary
system as its replication source, and a tier 3 secondary has the tier 2 secondary as its replication source.
The operation mode must be the same for all systems. However, if logreplay_readaccess is used between
tier 1 and tier 2, as an exception from this rule only the logreplay operation mode can be used between tier 2
and tier 3. Furthermore, it is not possible to combine the
logreplay and delta_datashipping operation
modes.
Multitier system replication supports various replication mode combinations. For more information, see
Supported Replication Modes Between Systems.
You can congure multitier system replication using the following tools:
● SAP HANA cockpit
For more information, see Example: Congure SAP HANA Multitier System Replication with the SAP HANA
Cockpit.
● SAP HANA studio
For more information, see Example: Congure SAP HANA Multitier System Replication with the SAP HANA
Studio.
● hdbnsutil
For more information, see Example: Congure SAP HANA Multitier System Replication with hdbnsutil.
Related Information
Supported Replication Modes Between Systems [page 130]
Example: Congure SAP HANA Multitier System Replication with SAP HANA Cockpit [page 133]
Example: Congure SAP HANA Multitier System Replication with hdbnsutil [page 136]
Example: Congure SAP HANA Multitier System Replication with SAP HANA Studio [page 138]
Operation Modes for SAP HANA System Replication [page 14]
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 129
7.1.1.1 Supported Replication Modes Between Systems
In a multitier system replication scenario, the following replication mode combinations are supported.
Replication Mode Combinations
Tier1 to Tier 2 Tier 2 to Tier 3 Description Use Case
SYNC SYNC In this setup, tier 1, tier 2,
and tier 3 are coupled
with SYNC replication
mode.
Tier 2 sends the
acknowledgment to tier 1
after the log buer has
been received and
written to disk, and after
the log buer has also
been received and
written by tier 3.
When primary has
received the
acknowledge, the buer
has been persisted by all
the tiers.
Tier 1 and tier 2 are
located in a local data
center for fast
takeover.
Tier 3 is used for
disaster recovery in a
second close data
center.
SYNC
SYNCMEM Tier 2 sends the
acknowledge to tier 1
after the log buer has
been received, written to
disk and it has been also
received by tier 3.
When the primary
receives
acknowledgment, it is not
clear that also tier 3 has
persisted the buer to
disk, but disk IO on tier 3
has been triggered.
Tier 1 and tier 2 are
located in a local data
center for fast
takeover.
Tier 3 is used for
disaster recovery in a
second close data
center.
SYNC
ASYNC Tier 1 and tier 2 are
closely coupled with
replication mode SYNC,
while tier 3 is decoupled
by using ASYNC.
Tier 1 and tier 2 are
located in a local data
center for fast
takeover.
130 P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
Tier1 to Tier 2 Tier 2 to Tier 3 Description Use Case
Tier 2 acknowledges the
arrival of the redo log
buers in-memory and
on disk to tier 1, while it
only hands over the redo
log buer to the network
without awaiting an
acknowledgment from
tier 3.
If the connection to tier 3
is too slow and the
ASYNC replication buer
(an intermediate
memory buer) is
running full, ASYNC
replication to tier 3 can
have an impact on the
primary.
Tier 3 is used for
disaster recovery in a
far distant data center.
SYNCMEM SYNC In this synchronous
setup tier 1 and tier 2 are
closely coupled with
replication mode
SYNCMEM, while tier 3 is
closely coupled with
SYNC.
Tier 2 sends the
acknowledgment to tier 1
after the log buer has
been received in
memory. IO is triggered
asynchronously. The
asynchronous IOalso
triggers the send
operation to tier 3. The
log write on tier 2 is
conrmed, when also tier
3 has written the log
buer.
When the primary
receives the
acknowledge, it is
unclear, if tier 3 has
already received and
persisted the log buer.
Tier 1 and tier 2 are
located in a local data
center for fast
takeover.
Tier 3 is used for
disaster recovery in a
second close data
center.
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 131
Tier1 to Tier 2 Tier 2 to Tier 3 Description Use Case
SYNCMEM SYNCMEM In this setup tier 1, tier 2,
and tier 3 are coupled
with replication mode
SYNCMEM.
Tier 2 sends the
acknowledgment to tier 1
after the log buer has
been received in
memory. IO is triggered
asynchronously. The
asynchronous IO also
triggers the send
operation to tier 3. The
log write on tier 2 is
conrmed, when tier 3
has received the log
buer in memory.
When the primary
receives the
acknowledge, it is
unclear, if tier 3 has
already received and
persisted the log buer.
Tier 1 and tier 2 are
located in a local data
center for fast
takeover.
Tier 3 is used for
disaster recovery in a
second close data
center.
SYNCMEM
ASYNC Tier 1 and tier 2 are
closely coupled with
replication mode
SYNCMEM, while tier-3 is
decoupled with ASYNC
replication.
Tier 2 acknowledges the
arrival of the redo log
buers in-memory to tier
1, while it only hands over
the redo log buer to the
network without awaiting
an acknowledgment from
tier 3.
If the connection to tier 3
is too slow and the
ASYNC replication buer
(an intermediate
memory buer) is
running full, ASYNC
Primary and tier 2 are
used in a local data
center for fast
takeover.
Tier 3 is used for
disaster recovery in a
far distant data center.
132 P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
Tier1 to Tier 2 Tier 2 to Tier 3 Description Use Case
replication can have an
impact on the primary.
ASYNC ASYNC With these asynchronous
replication modes there
is no wait for
acknowledgments
between tiers (no
acknowledge
propagation).
A replication backlog for
tier 2 and tier 3 is
possible.
Information about the
replication status on tier
1 and tier 2 is available in
the ASYNC replication
buer (an intermediate
memory buer). This
buer running full could
cause a minimal impact
on the performance of
the primary.
Tier 1 performance is
most important as well
as a disaster recovery
capability. For best
performance of tier 1
decouple tier 2 and
tier 3.
Data loss on tier 2 and
tier 3 is possible to
some extent, but
performance is more
critical.
7.1.1.2 Example: Congure SAP HANA Multitier System
Replication with SAP HANA Cockpit
Learn how to congure a multitier system replication landscape with the SAP HANA cockpit from the primary
system.
Prerequisites
● You have considered all the general prerequisities needed to congure system replication. For more
information, see General Prerequisites for Conguring SAP HANA System Replication.
● All three systems A,B, and C are registered in the SAP HANA cockpit.
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 133
Procedure
1. On the Resource Directory page, choose Site A or the SAP HANA system that will be the primary system.
2. On the System Overview page, choose the system replication tile.
3. Choose Congure System Replication on the top right to congure system replication.
4. Enter the required information on the System Replication Conguration page.
134
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
5. Choose Add 3 tier system and enter the requiered information for the tier 3 secondary system.
6. Choose Congure on the bottom right.
On the system replication overview, you should now see the tier 3 conguration.
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 135
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Supported Replication Modes Between Systems [page 130]
Operation Modes for SAP HANA System Replication [page 14]
Example: Congure SAP HANA System Replication with the SAP HANA Cockpit [page 47]
7.1.1.3 Example: Congure SAP HANA Multitier System
Replication with hdbnsutil
Learn how to congure SAP HANA multitier system replication with the hdbnsutil command line.
Prerequisites
● You have considered all the general prerequisities needed to congure system replication. For more
information, see General Prerequisites for Conguring SAP HANA System Replication.
● You have installed and congured three identical, independently operational SAP HANA systems – a
primary system, a tier 2 secondary system, and a tier 3 secondary system.
Context
The following steps show how to congure such a system. In this scenario there are three SAP HANA systems:
A, B, and C, named SiteA, SiteB, and SiteC. Furthermore, in this scenario multitier system replication supports
a tier 2 secondary with sync replication mode and a tier 3 secondary with async replication mode. The
operation mode is logreplay.
Procedure
1. [A] Start the SAP HANA database.
2. [A] Create a data backup or storage snapshot. In multiple-container systems, the system database and all
tenant databases must be backed up.
3. [A] Enable system replication and give the system a logical name. As <sid>adm:
cd /usr/sap/<sid>/HDB<instancenr>/exe
./hdbnsutil -sr_enable --name=SiteA
4. Stop the tier 2 secondary.
136
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
As <sid>adm run the SAPControl program to shut down the system:
/usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function StopSystem
HDB
5. [B] On the stopped tier 2 secondary, register site B with Site A as <sid>adm:
hdbnsutil -sr_register --replicationMode=sync --operationMode=logreplay --
name=SiteB
--remoteInstance=<instId> --remoteHost=<hostname_of_A>
6. [B] Start the tier 2 secondary system.
As <sid>adm run the SAPControl program to start the system:
/usr/sap/hostctrl/exe/sapcontrol sapcontrol -nr <system number> -function
StartSystem HDB
7. [B] Enable this site as the source for a tier 3 secondary system:
As <sid>adm on the tier 2 secondary run hdbnsutil -sr_enable
8. [C] Stop the tier 3 secondary system.
As <sid>adm run the SAPControl program to shut down the system:
/usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function StopSystem
HDB
9. [C] On the stopped system, register siteC as a tier 3 secondary system as <sid>adm:
hdbnsutil -sr_register --replicationMode=async --operationMode=logreplay --
name=SiteC
--remoteInstance=<instId> --remoteHost=<hostname_of_B>
10. [C] Start the SAP HANA database on the tier 3 secondary.
As <sid>adm run the SAPControl program to start the system:
/usr/sap/hostctrl/exe/sapcontrol sapcontrol -nr <system number> -function
StartSystem HDB
11. Check the replication status with systemReplicationStatus.py on command line or in the
M_SERVICE_REPLICATION system view.
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Supported Replication Modes Between Systems [page 130]
Operation Modes for SAP HANA System Replication [page 14]
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 137
7.1.1.4 Example: Congure SAP HANA Multitier System
Replication with SAP HANA Studio
You can congure SAP HANA multitier system replication using the SAP HANA studio.
Prerequisites
● You have considered all the general prerequisities needed to set up system replication. For more
information, see General Prerequisites for Conguring SAP HANA System Replication.
● You have added the systems in the SAP HANA studio.
● You have veried that the log_mode parameter in the persistence section of the global.ini le is set
to
normal for the systems.
You can do this in the Administration editor (Conguration tab) of the SAP HANA studio.
● You have stopped the tier 3 secondary system.
Context
The following example describes how to add a tier 3 secondary with a synchronously running tier 2 system
replication.
Procedure
1. Enable system replication on the tier 2 secondary, which has to be online, as follows:
a. In the Systems view right click the tier 2 secondary system, choose Conguration and Monitoring
Congure System Replication
The Congure System Replication dialog opens. The Enable System Replication option is selected by
default. The site name is already known from the topology metadata.
b. Choose Next.
c. Review the congured information and choose Finish.
2. Register the tier 3 secondary system as follows:
a. You have installed and congured three identical, independently operationalStop the tier 3 secondary
system if it is still running. Right-click the tier 3 secondary system and choose Conguration and
Monitoring
Stop System
b. In the Systems view, right-click the tier 3 secondary system and choose Conguration and
Monitoring Congure System Replication .
The Congure System Replication dialog opens.
c. Choose Register Secondary System and then Next.
d. Enter the required system information and the logical name used to represent the tier 3 secondary
system.
138
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
Note
If you are operating a distributed system on multiple hosts, you enter the name of the host on
which the master name server is running.
e. Specify the replication mode and enter the tier 2 secondary system's host name.
For more information, see Supported Replication Modes Between Systems.
f. Review the congured information and choose Finish.
Results
The secondary system is automatically started and the replication process to the tier 3 secondary then starts
automatically.
Related Information
General Prerequisites for Conguring SAP HANA System Replication [page 35]
Supported Replication Modes Between Systems [page 130]
7.1.2 Performing a Takeover and a Failback in SAP HANA
Multitier System Replication
You can perform a takeover and a failback also in a SAP HANA multitier system replication.
If the primary system fails, you can perform a takeover to the tier 2 secondary system.
Once your failed system is operational again, you can attach it as a tier 3 secondary system or you can restore
the original multitier system replication conguration. To learn how to perform these steps with the hdbnsutil
command line, see Attach the Original Primary System as a New Tier 3 Secondary System and Restore the
Original SAP HANA Multitier System Replication Conguration.
Related Information
Example: Attach the Original Primary System as a New Tier 3 Secondary System [page 140]
Example: Restore the Original SAP HANA Multitier System Replication Conguration [page 141]
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 139
7.1.2.1 Example: Attach the Original Primary System as a
New Tier 3 Secondary System
Once your failed system is operational again, you can attach it as a tier 3 secondary system.
Context
The steps below show how to set up multitier system replication again after a takeover. In these scenarios there
are three SAP HANA systems A, B and C, named SiteA, SiteB, and SiteC. Furthermore, in this scenario multitier
system replication supports a tier 2 secondary with sync replication mode and a tier 3 secondary with async
replication mode.
Procedure
SiteA failed, SiteB has taken over and now you attach SiteA as the tier 3 secondary.
1. [C] Change the replication mode of the new tier 2 secondary:
cd /usr/sap/<sid>/HDB<instance_number>/exe
./hdbnsutil -sr_changemode --replicationMode=sync
Multitier system replication supports various replication mode combinations. For more information, see
Supported Replication Modes between Sites.
2. [C] Enable SiteC as the replication source:
hdbnsutil -sr_enable
3. [A] Make sure that the SAP HANA database is stopped. This should be the case as a takeover was already
carried out otherwise you can stop it with the following command:
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StopSystem
HDB
4. [A] Register SiteA as a new tier 3 secondary.
hdbnsutil -sr_register --replicationMode=async --name=SiteA --
remoteInstance=<instId> --remoteHost=<hostname_of_C>
5. [A] Start the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StartSystem
HDB
6. [B] Check in M_SERVICE_REPLICATION that sync system replication is ACTIVE from SiteB to SiteC and
that async replication is ACTIVE from SiteC to SiteA.
140
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
Related Information
Supported Replication Modes Between Systems [page 130]
7.1.2.2 Example: Restore the Original SAP HANA Multitier
System Replication Conguration
Once your failed system is operational again, you can restore the original SAP HANA multitier system
replication conguration.
Context
The steps below show how to set up multitier system replication again after a takeover. In these scenarios there
are three SAP HANA systems A, B and C, named SiteA, SiteB, and SiteC. Furthermore, in this scenario multitier
system replication supports a tier 2 secondary with sync replication mode and a tier 3 secondary with async
replication mode.
Procedure
You want to restore the original multitier setup:
1. [C] Stop the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StopSystem
HDB
2. [C] Unregister SiteC from SiteB:
hdbnsutil -sr_unregister
3. [A] Stop the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StopSystem
HDB
4. [A] Register as secondary:
hdbnsutil -sr_register --replicationMode=sync --name=SiteA --
remoteInstance=<instId> --remoteHost=<hostname_of_B>
5. [A] Start the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StartSystem
HDB
6. [B] Check in M_SERVICE_REPLICATION that sync system replication is ACTIVE from SiteB to SiteA.
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 141
7. [A] SiteA takes over as the primary system:
hdbnsutil -sr_takeover
8. [B] Stop the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StopSystem
HDB
9. [A] Enable system replication:
hdbnsutil -sr_enable --name=SiteA
10. [B] Register SiteB as the tier 2 secondary of SiteA.
hdbnsutil -sr_register --replicationMode=sync --name=SiteB --
remoteInstance=<instId> --remoteHost=<hostname_of_A>
11. [B] Start the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StartSystem
HDB
12. [B] Enable SiteB as a replication source system:
hdbnsutil -sr_enable
13. [C] Register SiteC as a tier 3 secondary in the multitier system replication scenario:
hdbnsutil -sr_register --replicationMode=async --name=SiteC --
remoteInstance=<instId> --remoteHost=<hostname_of_B>
14. [C] Start the SAP HANA database
/usr/sap/hostctrl/exe/sapcontrol -no <instance_number> –function StartSystem
HDB
15. [B] Check in M_SERVICE_REPLICATION that sync replication is ACTIVE from SiteA to SiteB and that async
replication is ACTIVE from SiteB to SiteC.
7.2 SAP HANA Multitarget System Replication
In a multitarget system replication, the primary system can replicate data changes to more than one secondary
system.
Multitarget system replication brings advantages for several use cases:
● Update scenarios
● Rearrangements of system replication multitier chains
● Reaching higher availability (before stopping existing structures, new structures can be build and
established)
The graphic shows a possible setup for multitarget system replication.
142
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
Primary system A in data center 1 replicates data changes to secondary system B in the same data center.
Primary system A also replicates data changes to secondary system C in data center 2. Secondary system C is
a source system for a further secondary system D located in the same data center with system C.
To understand how to handle in dierent disaster recovery scenarios, see Disaster Recovery Scenarios for
Multitarget System Replication. In a multitarget system replication, the secondary systems can be congured to
automatically re-register to a new source system when the original source system unavailable.
In a multitarget system replication setup, you can congure multiple secondaries as Active/Active (read
enabled). Only one of these secondaries can be accessed via hint-based statement routing; the others must be
accessed via direct connection. For more information on Active/Active (read enabled), see Active/Active (Read
Enabled).
Related Information
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 143
Example: Congure a SAP HANA Multitarget System Replication [page 144]
Operation Modes for SAP HANA System Replication [page 14]
Disaster Recovery Scenarios for Multitarget System Replication [page 145]
Full Sync Option for SAP HANA System Replication [page 58]
Log Retention [page 20]
Log Retention and Multitarget System Replication [page 25]
Congure Secure Communication (TLS/SSL) Between Primary and Secondary Sites [page 259]
Use Multitarget System Replication for Near Zero Downtime Upgrades [page 203]
Active/Active (Read Enabled)
7.2.1 Example: Congure a SAP HANA Multitarget System
Replication
You can congure a multitarget system replication in which a primary system replicates data changes to more
than one secondary system.
Context
In this example, primary system A in data center 1 replicates data changes to secondary system B in the same
data center. Primary system A also replicates data changes to secondary system C in data center 2. Secondary
system C is a source system for a further secondary system D located in the same data center with system C.
Procedure
1. On primary system A in data center 1:
a. Create backups.
b. Enable system replication.
2. On the local secondary system B in data center 1:
a. Stop the system.
b. Register it to A.
c. Start the system.
3. On the remote secondary system C in data center 2:
a. Stop the system.
b. Register it to A.
c. Start the system.
4. On the remote secondary system D in data center 2:
a. Stop the system.
b. Register it to C.
144
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
c. Start the system.
7.2.2 Disaster Recovery Scenarios for Multitarget System
Replication
Several solutions are available when the systems involved in a multitarget system replication conguration fail.
We are using the setup described in Multitarget System Replication to exemplify the procedure. In this setup,
primary system A replicates data changes to secondary system B located in the same data center. Primary
system A also replicates data changes to the secondary system C located in data center 2. Secondary system
C is a source system for a further secondary system D located in the same data center with system C.
Failure on Primary System A
When primary system A fails, proceed as follows:
1. Take over on secondary system B in data center 1.
2. Register secondary system C in data center 2 to the new primary system B in data center 1. Then, register
secondary system D in data center 2 to secondary system C.
3. After the failure on the previous primary system A is solved, register it to the new primary system B in data
center 1.
Alternatively, you can set the global.ini/[system_replication]/
register_secondaries_on_takeover paramater to true and take over on secondary system B in data
SAP HANA System Replication
SAP HANA System Replication Setups
P U B L I C 145
center 1. As a result, secondary system C in data center 2 will register automatically to the new primary system
B in data center, while secondary system D in data center 2 will register automatically to secondary system C.
After the failure on the previous primary system A is solved, register it to the new primary system B in data
center 1.
Failure of Data Center 1
When all the systems in data center 1 fail, proceed as follows:
1. Take over on secondary system C in data center 2.
2. After the failure on the previous primary system is solved, register system A to the new primary system C
in data center 2.
3. Register secondary system B as tier 3 to system A in data center 1.
For more information about takeover and failback, see Performing a Takeover and Performing a Failback.
Related Information
Performing a Takeover
Performing a Failback
Full Sync Option for SAP HANA System Replication [page 58]
Log Retention [page 20]
146
P U B L I C
SAP HANA System Replication
SAP HANA System Replication Setups
8 SAP HANA System Replication: Operation
and Maintenance
Learn how to monitor SAP HANA system replication.
How can I monitor system replication?
There are multiple ways to monitor system replication and to verify if the primary and secondary systems are in
sync and are running correctly:
● Alerts on the primary and secondary systems warn you of potential problems.
● To ensure rapid takeover in the event of planned or unplanned downtime, you can check the status of the
replication between the primary and the secondary system.
● You can monitor system replication using the SAP HANA cockpit, the SAP HANA studio, the hdbnsutil
command line tool, or SQL queries.
This chapter also includes information about the network connection, how to copy or move tenants in a system
replication conguration, or how to use SAP HANA system replication to update your SAP HANA systems.
Where can I nd more information?
The following SAP Notes are relevant for a full understanding of the concepts described in this chapter:
SAP Notes
SAP Note Title
1917938
Migration of the statistics server for Revision 74 or higher
2369981
Required conguration steps for authentication with HANA System Replica
tion
2300936
Host Auto-Failover & System Replication Setup with SAP HANA extended
application services, advanced model
1999880
FAQ: SAP HANA System Replication
1969700
SQL Statement Collection for SAP HANA
2494079
Near-Zero-Downtime-Upgrade to HANA 2 SPS02 or above when internal
communicationSSL is used
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 147
SAP Note Title
2081065
Troubleshooting SAP HANA Network
1984882
Using HANA System Replication for Hardware Exchange with minimum/zero
Downtime
1984882
How-To Guides & Whitepapers For SAP HANA High Availability
2386973
Near Zero Downtime Upgrades for HANA Database 3-tier System Replica
tion
Related Information
SAP HANA System Replication Details [page 148]
Alerts [page 152]
Checking the SAP HANA System Replication Status [page 159]
Monitoring System Replication [page 175]
System Replication Network Connection [page 187]
Copy or Move Tenants Within System Replication [page 194]
Copying a System using System Replication [page 195]
Updating SAP HANA Systems with SAP HANA System Replication [page 196]
SAP HANA SQL and System Views Reference
8.1 SAP HANA System Replication Details
Detailed information from the M_SERVICE_REPLICATION and the M_SYSTEM_REPLICATION monitoring views
about system replication.
General Overview
Column
Description
Site ID 1 Generated ID of the primary site
Secondary Site ID 2 Generated ID of the secondary site
Service Name of the service
Volume ID Persistence volume ID
Operation Mode
● LOGREPLAY
● LOGREPLAY_READACCESS
● DELTA DATA SHIPPING
148 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Column Description
Replication Mode Congured replication mode:
● SYNC: synchronous replication with acknowledgement
when buer has been written to disk on the secondary
system
● SYNCMEM: synchronous replication with acknowledge
ment when buer arrived in memory on the secondary
system
● ASYNC: asynchronous replication where the primary
does not wait for the acknowledgement
● UNKNOWN: is set if replication mode could not be deter
mined (this might be the case, for example. if there are
communication errors when getting status information
from a service).
Replication Status
Current status of replication:
● UNKNOWN: secondary did not connect to primary since
last restart of the primary
● INITIALIZING: initial data transfer is running, in this
state, the secondary is rst usable when this is nished
● SYNCING: secondary is syncing again (for example, after
a temporary connection loss or restart of the secondary)
● ACTIVE: initialization or sync with primary is complete
and secondary is continuously replicating. If crash oc
curs, no data loss will occur in SYNC replication mode.
● ERROR: replication cannot take place because the sec
ondary system is not accessible (details can be found in
Replication Details)
Replication Details
Additional information for Replication Status, for example,
the error text if status is ERROR.
Full Sync Indicates if the service is currently operating in sync replica
tion mode with the full sync option set.
If full sync is enabled in a running system, full sync might not
be active immediately. This is done to prevent the system
from blocking transactions immediately when setting the pa
rameter to true. Instead, in a rst step, full sync has to be en
abled. In a second step it is internally activated, when the
secondary is connected and becomes ACTIVE.
● DISABLED: full sync is not congured at all
● ENABLED: full sync is congured, but it is not yet active,
so transactions do not block in this state. To become ac
tive the secondary has to connect and Replication Sta
tus has to be ACTIVE.
● ACTIVE: full sync mode is congured and active. If a con
nection of a connected secondary is getting closed,
transactions on the primary side will block in this state.
If full sync is enabled when an active secondary is currently
connected, the FULL_SYNC will be immediately set to AC
TIVE.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 149
Column Description
Secondary Fully Recoverable TRUE: No full data backup is needed after takeover on secon
dary. Backups created on the primary and local log segments
enable a full database recovery.
FALSE: Log segments needed for a full database recovery are
missing. After takeover a full data backup has to be executed
before a full recovery up to the most recent time point can be
executed.
Secondary Active
Status of the secondary node (also see ACTIVE_STATUS in
M_SERVICES)
Secondary Connect Time Timestamp the secondary connected to the primary. If there
are reconnects from the secondary side, this eld contains
the last connect time.
Secondary Reconnect Count Number of reconnects from secondary side for this service.
Secondary Failover Count Number of failovers for this service on secondary side.
Buer Full count Number of times, the asynchronous replication buer was
full since last service restart (only relevant for replication
mode async; 0 for replication modes sync/syncmem).
Log Positions
Column
Description
Last Log Position Last known log position on primary
Last Log Position Time Timestamp of last known log position
Replayed Log Position Log end position of the last known replayed log buer on
secondary site
Replayed Log Position Time Timestamp of the last known replayed log buer on the sec
ondary site
Last Shipped Log Position Time Timestamp of last log position being shipped to secondary
Shipped Log Buer Count Number of log buers shipped to secondary
Shipped Log Buers Total Size (Bytes) Size of all log buers shipped to secondary
Shipped Log Buers Total Time (µs) Time taken to ship all the log buers to the secondary.
● SYNC/SYNCMEM: total round trip time to send the log
buers and receive the acknowledgment.
● ASYNC: start time when sending the log buers, end
time when the OS reports that the log buers were sent
(and the log shipping buer space was freed). This
could be shorter than the SYNC/SYNCMEM duration
Time delay (ms) Time delay between the last shipped log position time and
the replayed log position time on the secondary
Size delay (Bytes) Size delay between the last shipped log position size and the
replayed log position size on the secondary (1 log position =
64 bytes)
150 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Savepoints
Column Description
Last Savepoint Version Last savepoint version on primary
Last Savepoint Log Position Log position of current savepoint
Last Savepoint Start Time Timestamp of current savepoint
Last Shipped Savepoint Version Last savepoint version shipped to secondary
Last Shipped Savepoint Log Position Log position of last shipped savepoint
Last Shipped Savepoint Time Timestamp of last shipped savepoint
Full Data Replica
Column
Description
Full Data Replica Shipped Count Number of full data replicas shipped to secondary
Full Data Replica Shipped Total Size (Bytes) Total size of all full data replica shipped to secondary
Full Data Replica Shipping Total Time (µs) Duration for shipping all full data replica
Last Full Data Replica Shipped Size (Bytes) Size of last full data replica shipped to secondary
Start Time of Last Full Data Replica Start time of last full data replica
End Time of Last Full Data Replica End time of last full data replica
Delta Data Replica
This information is only displayed if the operation mode is delta_datashipping.
Column
Description
Delta Data Replica Shipped Count Number of delta data replicas shipped to secondary
Delta Data Replica Shipped Total Size (Bytes) Total size of all delta data replicas shipped to secondary
Delta Data Replica Shipped Total Time (µs) Duration for shipping of all delta data replicas
Size of Last Delta Data Replica (Bytes) Size of last delta data replica
Start Time of Last Delta Data Replica Start time of last data delta replica
End Time of Last Delta Data Replica End time of last data delta replica
Log Shipping Backlog
Column
Description
Current Replication Backlog Size (Bytes) Current replication backlog in bytes, this means, size of all
log buers that have been created on primary site, but not
yet sent to the secondary site.
Even in replication modes sync/syncmem this column can
have a value dierent from 0.
Here it represents the size of log buers that are in the local
send queue (max number of those buers is the number
congured log buers on primary site).
Max Replication Backlog Size (Bytes)
Max replication backlog in bytes (max value of BACK
LOG_SIZE since system start).
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 151
Column Description
Current Replication Backlog Time (µs) Current replication backlog in microseconds. This is time dif
ference between time of the last sent log buer and the cur
rent log buer.
Even in replication modes sync/syncmem this column can
have a value dierent from 0, because log buers are still in
the send queue (max number of these buers is the number
of log buers congured on primary site).
Max Replication Backlog Time (µs)
Max replication backlog in microseconds (max value of
BACKLOG_TIME since system startup).
Log Replay
This information is only displayed if the operation mode is logreplay and logreplay_readaccess.
Column
Description
Replay Backlog Size (Bytes) Species the size of all log buers that have been shipped to
the secondary site but have not yet been replayed on the
secondary site.
Max Replay Backlog Size (Bytes) Species the maximum value of the REPLAY_BACK
LOG_SIZE since the system startup.
Replay Backlog Time (µs) Species the time dierence between the time of the last
shipped log buer and the last replayed log buer on the
secondary site.
Max Replay Backlog Time (µs) Species the maximum value of REPLAY_BACKLOG_TIME
since the system startup.
Related Information
SQL and System View Reference [page 269]
8.2 Alerts
Alerts on the primary and secondary systems warn you of potential problems.
For an overview of the alerts issued by the primary system, see SAP HANA System Replication Alerts.
On the system overview page of SAP HANA cockpit, choose Show All on the Alerts tile to see all alerts. Alerts
occurring on the secondary system are reported on the primary system. In the SAP HANA cockpit, they are
displayed as alerts and associated with the host where they occurred.
152
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
This is an example of an alert in the SAP HANA cockpit which occured on the secondary system (ld4127) and is
reported on the primary system (ld4126):
In the SAP HANA studio, you can choose the Alerts tab to see all generated alerts.
If you want to receive an e-mail when an alert condition for all or specic checks is fullled, follow the steps
described in Congure E-Mail Notications for Alerts for SAP HANA studio or Set Up E-mail Notication for SAP
HANA cockpit.
An alerting for the secondary systems was also established with the introduction of the so-called proxy views.
Alerts occurring on the secondary hosts are shown on the primary system as alerts and associated with the
host where they occurred. For more information, see Monitoring Secondary Systems.
Related Information
SAP HANA System Replication Alerts [page 153]
Monitoring Secondary Systems [page 155]
Congure E-Mail Notications for Alerts
Set Up E-Mail Notication
Proxy Schemas and Views [page 156]
Monitoring and Replicating INI File Parameter Changes [page 158]
8.2.1 SAP HANA System Replication Alerts
This is an overview of all alerts issued for the primary system.
Alert ID 78
Connection Closed
Description
Alert 78 is raised when a system replication connection is closed.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 153
Alert ID 79 Conguration Parameter Mismatch
Description
Alert 79 is raised when there is a conguration parameter mismatch between the primary and
the secondary system.
Alert ID 94 Logreplay Backlog
Description
Alert 94 is raised when the system replication logreplay backlog increases. A delayed log replay
on the secondary system causes a longer takeover time.
The alert has a dierent priority based on the size of the redo log that was not yet replayed:
● Low: 10 GB < logreplay backlog < 50 GB
● Medium: 50 GB <= logreplay backlog < 500 GB
● High: logreplay backlog >= 500 GB
To identify the reason for the increased system replication logreplay backlog, check the state of
the services on the secondary system. To get more information, monitor the secondary system.
Possible causes for the increased system replication logreplay backlog can be, for example, a
slow or not functioning log replay, or a non-running service on the secondary system.
Alert ID 104
Increased Log Shipping Backlog
Description
Alert 104 is raised when the system replication log shipping is delayed or does not work properly
causing data loss on the secondary system, if a takeover is done.
The alert has a dierent priority based on the threshold reached:
● Low: 1 GB < log shipping backlog < 10 GB
● Medium: 10 GB <= log shipping backlog < 50 GB
● High: log shipping backlog >= 50 GB
To identify the reason for the increased system replication log shipping backlog, check the sta
tus of the secondary system. Possible causes for the increased system replication log shipping
backlog can be a slow network performance, connection problems, or other internal issues (for
example, in SYNC or SYNCMEM replication modes).
Alert ID 106
ASYNC Replication In-Memory Buer Overow
Description
Alert 106 is raised when the local in-memory buer in the ASYNC replication mode is running
full indicating possible network issues with the connection to the secondary system.
The alert has a dierent priority based on the threshold reached:
● Medium: if buer runs full once within 24 hours
● High: if buer runs full more than once within 24 hours
To identify the reason for the local in-memory buer running full, check the buer size, the
network, the IO on the secondary system, or look for peak loads.
The alert depends on the setting of the
logshipping_async_wait_on_buffer_full parameter. For more information
about this parameter, see SAP HANA System Replication Conguration Parameters.
154 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Alert ID 107 Inconsistent fallback snapshot
Description
Alert 107 is raised for the primary and the secondary systems when there are broken fallback
snapshots. For more information about fallback snapshots, see Create a Fallback Snapshot.
Note
Before SAP HANA 1.0 SPS 09 there was one alert categorized as "Internal Event" (Alert 21) which covered
alerts 78 and 79. Both situations were covered by one event type and could only be distinguished by the
information text provided. Since SAP HANA 1.0 SPS 11 old style alerts based on alert 21 are not created
anymore as a default.
You can create them by setting the conguration parameter keep_old_style alert to true. These
alerts can be required to keep the existing monitoring infrastructure working. If activated, new alerts and
old style alerts are created in parallel. Old alerting can be disabled by setting the keep_old_style alert
conguration parameter to false in global.ini le.
The new alerts require that you have migrated to the embedded statistics server. For more information, see
SAP Note 1917938.
Related Information
Monitoring Secondary Systems [page 155]
SAP HANA System Replication Conguration Parameters [page 62]
Create a Fallback Snapshot
SAP Note 1917938
8.2.2 Monitoring Secondary Systems
Remote SQL access on the primary system allows monitoring and reporting of the secondary system statistics.
There is a possibility to monitor the secondary system through proxy schemas and views. Proxy schemas and
views are provided on the primary system. They extract the corresponding information from the monitoring
views on the secondary system. The retrieval of statistics is unaected by the replication or operation mode
and is available for a two system replication setup as well as for multitier landscapes.
A new schema is created on the primary system for each registered secondary system. This schema follows
the naming convention _SYS_SR_SITE_<siteName>, where <siteName> is the case-sensitive name given at
the registration time of the secondary system. This schema contains a selected subset of monitoring views (for
example, M_VOLUME_IO_TOTAL_STATISTICS), which proxies the statistics from the secondary system. These
proxy views have the same column denitions as the equivalently named public synonyms already available for
the primary system. When a secondary system is unregistered, the corresponding schema will be dropped.
Note
If system replication is congured as an Active/Active (read enabled) system with the
logreplay_readaccess operation mode, then more data is available from the secondary system in the
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 155
_SYS_SR_SITE_<siteName> proxy schema and more monitoring views of the secondary system can be
accessed using virtual tables.
Based on these views and tables available in the proxy schema, the statistics server is able to generate alerts
on the secondary systems of a system replication landscape. Alerts issued by the secondary systems are
displayed in the Alerts tile of the SAP HANA cockpit.
SYS_DATABASES_SR_SITE <secondary_site_name> is a second proxy schema containing proxy views, which
can be queried to get information from the corresponding view in the SYS_DATABASES schema on the
secondary system. These proxy views simplify the monitoring of secondary systems from the primary system.
Limitations
● Monitoring view access is only possible if the primary and secondary systems run with exactly the same
software version.
● When such a proxy view is queried and the secondary system is not started, no results are shown without
the report of an SQL error.
● Querying against multitenant landscapes is limited to single tenant databases or the system database,
meaning there are no views unifying all tenants on the system database similar to the SYS_DATABSES
schema.
Related Information
Alert Details
Proxy Schemas and Views [page 156]
8.2.2.1 Proxy Schemas and Views
You can monitor the secondary system through proxy schemas and views.
To see the proxy views of the secondary system's monitoring views, you can use the SAP HANA cockpit. On the
system overview page of the primary system, choose Execute SQL to open the SAP HANA Database Explorer.
156
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Then open the primary’s catalog and go to the corresponding schema. This is a schema example:
To see the virtual tables available in the proxy schema of the secondary system, open the proxy schema for the
primary system in the SAP HANA Database Explorer for the primary system and choose Tables. A long list of
accessible monitoring views from the secondary will be available.
Example
The virtual tables can look as follows:
Any of these proxy views or virtual tables can be accessed using SQL from the primary system by providing the
correct secondary’s schema name. For example:
select * from "_SYS_SR_SITE_SiteB"."M_HOST_INFORMATION";
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 157
8.2.3 Monitoring and Replicating INI File Parameter Changes
INI le parameters should be the same on each site of a system replication landscape and are checked
automatically.
The conguration parameter checker reports any dierences between primary, secondary, and further
secondary systems.
Some parameters may have dierent settings on the primary and the secondary system on purpose. One
example is the global_allocation_limit parameter where the secondary is used for other systems. By
adding those parameters to the exclusion list below, they are excluded from checking and replication.
Activate ini parameter replication so that changes made on the primary are automatically replicated to the
secondary systems. Otherwise, changes should be manually duplicated on the other systems.
The checks of the conguration parameter checker:
● Are done every hour by default.
● Generate alerts.
The alerts are visible in the SAP HANA cockpit, SAP HANA studio, and the M_EVENTS system view.
To view the alerts in the SAP HANA cockpit, look for the alerts tile on the system overview page of the SAP
HANA cockpit and choose Show All. You can see all alerts including the ones created because of parameter
mismatch.
● Are optimized for the most recently changed parameters.
Enable and disable the parameter check on the primary site with [inifile_checker]/enable = true |
false. The parameter checker is on by default.
Enable and disable the parameter replication on the primary site with [inifile_checker]/replicate =
true | false. The parameter replication is o by default.
The ini le parameter replication follows these rules:
Parameter set on
Activity
Primary System Secondary System
yes no Copy parameter to the secondary system.
no yes Delete parameter on the secondary system.
set to value x set to value y Copy value x to the secondary system.
The parameter changes on the secondary system are applied dierently for each parameter:
● Online changeable parameters become active after the ALTER SYSTEM command or by editing the .ini le
followed by the automatically triggered hdbnsutil –reconfig command.
● Oine changeable parameters become active after a restart. When changing such a parameter, it is
necessary to restart the primary and secondary systems before a takeover. For more information about
conguration parameters, see Conguration Parameter Reference.
To prevent parameters from generating alerts and eventually getting replicated, it is possible to create
exclusions. In the following example, dierent global allocation limits (GAL) on primary and secondary systems
can be set without being overwritten by the parameter replication.
158
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Example
If for example you intend to use your secondary system for DEV/QA systems and set the global allocation
limit to its minimal value (as described above), you may exclude this global_allocation_limit
parameter from these checks as follows:
[inifile_checker]
enable = true|false
interval = 3600
exclusion_global.ini/SYSTEM = memorymanager/global_allocation_limit
The exclusion rules are written in the following syntax (comma separated list) and take eect immediately:
exclusion_[inifile name|*][/<LAYER>] = [section with
wildcards|*][/parameter with wildcards|*], ...
<LAYER> := SYSTEM\|HOST\|DATABASE\|\*
Related Information
Conguring SAP HANA System Properties (INI Files)
SAP HANA Conguration Parameter Reference
8.3 Checking the SAP HANA System Replication Status
To ensure rapid takeover in the event of planned or unplanned downtime, you can monitor the status of
replication between the primary system and the secondary system.
There are several ways to gather information about the overall status of the sites and of the system replication:
● With the landscapeHostConfiguration.py script
This script provides information only about the state of one SAP HANA database. A returned error state
(for example, return code 1) could indicate that a takeover to the secondary should be considered.
● With the systemReplicationStatus.py script
This script shows whether the secondary systems are in sync or not. This provides more condence if a
takeover is justied.
● With the getTakeoverRecommendation script
This script shows the takeover recommendation based on the current system state.
Recommendation
Rather than calling both landscapeHostConfiguration.py and systemReplicationStatus.py
manually and calculate the required action based on return codes, you can use the
getTakeoverRecommendation.py script.
● With HDB console
Using the HDB console you can check the system replication status on all hosts and for all services.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 159
● With predened SQL statements
Related Information
Checking the Status with landscapeHostConguration.py [page 160]
Checking the Status with systemReplicationStatus.py [page 163]
Checking the Status with getTakeoverRecommendation.py [page 165]
Example: Checking the Status on the Primary and Secondary Systems [page 166]
Checking the Status with the HDB Console [page 170]
Checking the Status with Predened SQL Statements [page 172]
Monitoring Status and Resource Usage of System Components
8.3.1 Checking the Status with
landscapeHostConguration.py
Check the status of the primary system using landscapeHostConfiguration.py.
Check the overall status of the primary system using as <sid>adm user the
landscapeHostConfiguration.py script located in $DIR_INSTANCE/../exe/python_support.
<sid>adm># python $DIR_INSTANCE/exe/python_support/landscapeHostConfiguration.py
| Host | Host | Host | ... | NameServer | NameServer | ...
| | Active | Status | | Config Role| Actual Role |
| ----- | ------ | ------ | --------- | ---------- | ----------- | ------
| host1 | yes | ok | ... | master 1 | master | ...
| host2 | yes | ok | ... | master 2 | slave | ...
overall host status: ok
Use the --sapcontrol=1 parameter, if you require a reliable and parsable output.
<sid>adm># python $DIR_INSTANCE/exe/python_support/landscapeHostConfiguration.py
--sapcontrol=1
SAPCONTROL-OK: <begin>
host/ld2131/hostActualRoles=worker
host/ld2131/removeStatus=
host/ld2131/nameServerConfigRole=master 1
host/ld2131/failoverStatus=
host/ld2131/hostConfigRoles=worker
host/ld2131/failoverActualGroup=default
host/ld2131/storageConfigPartition=1
host/ld2131/host=ld2131
host/ld2131/indexServerConfigRole=worker
host/ld2131/failoverConfigGroup=default
host/ld2131/storageActualPartition=1
host/ld2131/indexServerActualRole=master
host/ld2131/nameServerActualRole=master
host/ld2131/hostActive=yes
host/ld2131/hostStatus=ok
host/ld2131/storagePartition=0
host/ld2132/hostActualRoles=worker
host/ld2132/removeStatus=
host/ld2132/nameServerConfigRole=master 3
160
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
host/ld2132/failoverStatus=
host/ld2132/hostConfigRoles=worker
host/ld2132/failoverActualGroup=default
host/ld2132/storageConfigPartition=2
host/ld2132/host=ld2132
host/ld2132/indexServerConfigRole=worker
host/ld2132/failoverConfigGroup=default
host/ld2132/storageActualPartition=2
host/ld2132/indexServerActualRole=slave
host/ld2132/nameServerActualRole=slave
host/ld2132/hostActive=yes
host/ld2132/hostStatus=ok
host/ld2133/hostActualRoles=standby
host/ld2133/removeStatus=
host/ld2133/nameServerConfigRole=master 2
host/ld2133/failoverStatus=
host/ld2133/hostConfigRoles=standby
host/ld2133/failoverActualGroup=default
host/ld2133/storageConfigPartition=0
host/ld2133/host=ld2133
host/ld2133/indexServerConfigRole=standby
host/ld2133/failoverConfigGroup=default
host/ld2133/storageActualPartition=0
host/ld2133/indexServerActualRole=standby
host/ld2133/nameServerActualRole=slave
host/ld2133/hostActive=yes
host/ld2133/hostStatus=ignore
overall_status=ok
SAPCONTROL-OK: <end>
Note
This script provides information about the state of one SAP HANA database only. A takeover is only
recommended when the return code from the script is 1 (error).
The return codes of the script are:
Return code
Description
0 Fatal
Internal script error, the state could not be determined
1
Error
2
Warning
3
Info
4
OK
Status Error and Return Code=1:
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> python
landscapeHostConfiguration.py
| Host | Host | Host | Failover | Remove | Storage | Storage | Failover |
Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host |
Worker | Worker |
| | Active | Status | Status | Status | Config | Actual | Config | Actual |
Config | Actual | Config | Actual | Config | Actual | Config | Actual |
| | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role |
Roles | Roles | Groups | Groups |
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 161
| ------ | ------ | ------ | -------- | ------ | --------- | --------- |
-------- | -------- | ---------- | ---------- | ----------- | ----------- |
------- | ------- | ------- | ------- |
| ld2131 | yes | ok | | | 1 | 1 | default | default | master 1 | master | worker
| master | worker | worker | default | default |
| ld2132 | no | error | | | 2 | 2 | default | default | master 2 | slave |
worker | slave | worker | worker | default | default |
| ld2133 | no | ignore | | | 0 | 0 | default | default | master 3 | slave |
standby | standby | standby | standby | default | - |
overall host status: error
Return Code
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> echo $?
1
Status Warning and Return Code=2
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> python
landscapeHostConfiguration.py
| Host | Host | Host | Failover | Remove | Storage | Storage | Failover |
Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host |
Worker | Worker |
| | Active | Status | Status | Status | Config | Actual | Config | Actual |
Config | Actual | Config | Actual | Config | Actual | Config | Actual |
| | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role |
Roles | Roles | Groups | Groups |
| ------ | ------ | ------- | -------------- | ------ | --------- | --------- |
-------- | -------- | ---------- | ---------- | ----------- | ----------- |
------- | ------- | ------- | ------- |
| ld2131 | yes | ok | | | 1 | 1 | default | default | master 1 | master | worker
| master | worker | worker | default | default |
| ld2132 | no | warning | waiting 30 sec | | 2 | 2 | default | default | master
2 | slave | worker | slave | worker | worker | default | default |
| ld2133 | yes | ignore | | | 0 | 0 | default | default | master 3 | slave |
standby | standby | standby | standby | default | - |
overall host status: warning
Return Code
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> echo $?
2
Status Info and Return Code=3:
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> python
landscapeHostConfiguration.py
| Host | Host | Host | Failover | Remove | Storage | Storage | Failover |
Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host |
Worker | Worker |
| | Active | Status | Status | Status | Config | Actual | Config | Actual |
Config | Actual | Config | Actual | Config | Actual | Config | Actual |
| | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role |
Roles | Roles | Groups | Groups |
| ------ | ------ | ------ | -------- | ------ | --------- | --------- |
-------- | -------- | ---------- | ---------- | ----------- | ----------- |
------- | ------- | ------- | ------- |
| ld2131 | yes | ok | | | 1 | 1 | default | default | master 1 | master | worker
| master | worker | worker | default | default |
| ld2132 | yes | info | | | 2 | 0 | default | default | master 2 | slave |
worker | standby | worker | standby | default | - |
| ld2133 | yes | info | | | 0 | 2 | default | default | master 3 | slave |
standby | slave | standby | worker | default | default |
overall host status: info
Return Code
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> echo $?
3
Status OK and Return Code=4
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> python
landscapeHostConfiguration.py
162
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
| Host | Host | Host | Failover | Remove | Storage | Storage | Failover |
Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host |
Worker | Worker |
| | Active | Status | Status | Status | Config | Actual | Config | Actual |
Config | Actual | Config | Actual | Config | Actual | Config | Actual |
| | | | | | Partition | Partition | Group | Group | Role | Role | Role | Role |
Roles | Roles | Groups | Groups |
| ------ | ------ | ------ | -------- | ------ | --------- | --------- |
-------- | -------- | ---------- | ---------- | ----------- | ----------- |
------- | ------- | ------- | ------- |
| ld2131 | yes | ok | | | 1 | 1 | default | default | master 1 | master | worker
| master | worker | worker | default | default |
| ld2132 | yes | ok | | | 2 | 2 | default | default | master 2 | slave | worker
| slave | worker | worker | default | default |
| ld2133 | yes | ignore | | | 0 | 0 | default | default | master 3 | slave |
standby | standby | standby | standby | default | - |
overall host status: ok
Return Code
dx7adm@ld2131:/usr/sap/DX7/HDB07/exe/python_support> echo $?
4
In the event of a network split, a so called split brain scenario, the script cannot tell if the instance in the other
half of the network is fully functional. Therefore, a takeover decision should not be based on this script alone.
Related Information
Example: Checking the Status on the Primary and Secondary Systems [page 166]
8.3.2 Checking the Status with systemReplicationStatus.py
Check the status of system replication using the systemReplicationStatus.py script.
systemReplicationStatus.py shows whether the secondary systems are in sync or not. This provides
more condence if a takeover is justied. If system replication was never in sync or is outdated, unexpected
loss of data might occur.
Check the overall status of the system replication using as <sid>adm user the
systemReplicationStatus.py script located in $DIR_INSTANCE/../exe/python_support.
<sid>adm># python $DIR_INSTANCE/exe/python_support/systemReplicationStatus.py
| Host | Service Name | Site Name | Secondary | ... | Replication |...
| | | | Host | | Status |
| ------ | ---------------- | --------- | --------- | ----- | ----------- |---
| ld7805 | indexserver | WALLDORF | ld8475 | ... | ACTIVE |
| ld8513 | statisticsserver | WALLDORF | ld8476 | | ACTIVE |
| ld8513 | xsengine | WALLDORF | ld8476 | | ACTIVE |
| ld8513 | nameserver | WALLDORF | ld8476 | | ACTIVE |
| ld8513 | indexserver | WALLDORF | ld8476 | | ACTIVE |
| ld8559 | indexserver | WALLDORF | NOT MAPPED | | |
status system replication site "2": ACTIVE
status system replication site "3": ACTIVE
overall system replication status: ACTIVE
Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mode: PRIMARY
site id: 1
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 163
site name: WALLDORF
The return codes of the script are:
Return code Description
10 No System Replication
11
Error
Error occurred on the connection. Additional details on the error can be found in REPLICATION_STA
TUS_DETAILS.
12
Unknown
The secondary system did not connect to primary since last restart of the primary system.
13
Initializing
Initial data transfer is in progress. In this state, the secondary is not usable at all.
14
Syncing
The secondary system is syncing again (for example, after a temporary connection loss or restart of
the secondary).
15
Active
Initialization or sync with the primary is complete and the secondary is continuously replicating. No
data loss will occur in SYNC mode.
Use the following state-transition graphic to understand the return codes of the script.
164
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Related Information
Monitoring and Replicating INI File Parameter Changes [page 158]
M_SERVICE_REPLICATION System View [page 269]
Example: Checking the Status on the Primary and Secondary Systems [page 166]
SAP HANA SQL and System Views Reference
8.3.3 Checking the Status with
getTakeoverRecommendation.py
Check the status of system replication using the getTakeoverRecommendation.py script.
The getTakeoverRecommendation script shows the takeover recommendation based on the current system
state. However, when the primary system faces any error situation, the system replication status cannot be
determined anymore. Thus, the previous state should be saved and compared with the current state.
The script provides the following recommendations as takeover status based on the results of the other two
scripts landscapeHostConfiguration.py and systemReplicationStatus.py. It also provides an
overall status and a return code to match the overall host status.
landscapeHostCongura
tion systemReplicationStatus Takeover Status
Reason for the takeover rec
ommendation
Error/Fatal/Warning NoHsr/Error/ Unknown/Initi
alizing/Syncing/Active
Required Primary system has errors
OK/Info/Ignore NoHsr/Error/ Unknown/Initi
alizing/Syncing
Cannot decide Unknown system replication
status
OK/Info/Ignore Active Possible Primary system is up and
system replication is in sync
This is a sample implementation of a python script that uses getTakeoverRecommendation to act as a
minimalistic cluster manager.
Use the --sapcontrol=1 parameter, if you require a reliable and parsable output.
import time
import subprocess
from getTakeoverRecommendation import TakeoverDecision
def main():
wasInSync = False
while True:
recommendation =
subprocess.call(["python","getTakeoverRecommendation.py","--sapcontrol=1"])
if not wasInSync and recommendation is TakeoverDecision.Required:
print "Primary defect & no sync => NO TAKEOVER"
if wasInSync and recommendation is TakeoverDecision.Required:
print "Primary defect & sync => TAKEOVER"
nowInSync = recommendation is TakeoverDecision.Possible
wasInSync = nowInSync
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 165
The output depends on the previous state with the result of the current call of
getTakeoverRecommendation. If no sync state is reached, a takeover is not advised. But once the systems
are in sync, the next error of the primary system will suggest a takeover. Any subsequent negative return value
will reset the sync state as it is no longer ensured that the replicated data is current.
Related Information
Example: Checking the Status on the Primary and Secondary Systems [page 166]
8.3.4 Example: Checking the Status on the Primary and
Secondary Systems
Learn how to check the status on the primary and secondary systems with
landscapeHostConfiguration.py, systemReplicationStatus.py, and
getTakeoverRecommendation.
The examples below oer an overview of the outputs for the three scripts with and without "--sapcontrol=1".
Use echo $? to receive the return code of the last executed script.
Primary System
m13adm@ld2131:/usr/sap/M13/HDB13/exe/python_support> python
systemReplicationStatus.py
| Database | Host | Port | Service Name | Volume ID | Site ID | Site Name |
Secondary | Secondary | Secondary | Secondary | Secondary | Replication |
Replication | Replication |
| | | | | | | |
Host | Port | Site ID | Site Name | Active Status | Mode |
Status | Status Details |
| -------- | ------ | ----- | ------------ | --------- | ------- | --------- |
--------- | --------- | --------- | --------- | ------------- | ----------- |
----------- | -------------- |
| SYSTEMDB | ld2131 | 31301 | nameserver | 1 | 1 | SiteA |
ld2132 | 31301 | 2 | SiteB2 | YES | SYNCMEM |
ACTIVE | |
| M13 | ld2131 | 31307 | xsengine | 2 | 1 | SiteA |
ld2132 | 31307 | 2 | SiteB2 | YES | SYNCMEM |
ACTIVE | |
| M13 | ld2131 | 31303 | indexserver | 3 | 1 | SiteA |
ld2132 | 31303 | 2 | SiteB2 | YES | SYNCMEM |
ACTIVE | |
status system replication site "2": ACTIVE
overall system replication status: ACTIVE
Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mode: PRIMARY
site id: 1
site name: SiteA
166
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
=================================================================================
================================================
m13adm@ld2131:/usr/sap/M13/HDB13/exe/python_support> python
systemReplicationStatus.py --sapcontrol=1
SAPCONTROL-OK: <begin>
service/ld2131/31301/SHIPPED_LOG_POSITION_TIME=2019-01-04 17:49:03.563462
service/ld2131/31301/LAST_LOG_POSITION_TIME=2019-01-04 17:49:03.563462
service/ld2131/31301/SHIPPED_FULL_REPLICA_DURATION=6191118
service/ld2131/31301/SHIPPED_LAST_DELTA_REPLICA_START_TIME=-
service/ld2131/31301/SHIPPED_FULL_REPLICA_SIZE=1577443328
service/ld2131/31301/SITE_ID=1
service/ld2131/31301/LAST_LOG_POSITION=79676672
service/ld2131/31301/SECONDARY_ACTIVE_STATUS=YES
service/ld2131/31301/LAST_SAVEPOINT_LOG_POSITION=79676418
service/ld2131/31301/FULL_SYNC=DISABLED
service/ld2131/31301/OPERATION_MODE=logreplay_readaccess
service/ld2131/31301/SHIPPED_LAST_FULL_REPLICA_START_TIME=2018-12-18
16:08:45.743753
service/ld2131/31301/LAST_SAVEPOINT_VERSION=6633
...
service/ld2131/31303/SITE_NAME=SiteA
service/ld2131/31303/SECONDARY_SITE_NAME=SiteB2
service/ld2131/31303/REPLAYED_LOG_POSITION_TIME=2019-01-04 17:49:07.122280
service/ld2131/31303/SHIPPED_LAST_FULL_REPLICA_END_TIME=2018-12-18
16:09:07.125423
service/ld2131/31303/CREATION_TIME=2018-12-18 13:37:49.825044
service/ld2131/31303/HOST=ld2131
service/ld2131/31303/SHIPPED_SAVEPOINT_VERSION=71
service/ld2131/31303/SECONDARY_HOST=ld2132
service/ld2131/31303/VOLUME_ID=3
service/ld2131/31303/SHIPPED_LAST_FULL_REPLICA_SIZE=1678131200
service/ld2131/31303/SHIPPED_LOG_BUFFERS_SIZE=11519025152
service/ld2131/31303/REPLICATION_MODE=SYNCMEM
service/ld2131/31303/DATABASE=M13
service/ld2131/31303/REPLAYED_LOG_POSITION=264306432
service/ld2131/31303/SECONDARY_RECONNECT_COUNT=1
service/ld2131/31303/SHIPPED_SAVEPOINT_START_TIME=2018-12-18 16:09:00.543785
service/ld2131/31303/SECONDARY_PORT=31303
service/ld2131/31303/SHIPPED_SAVEPOINT_LOG_POSITION=84321794
service/ld2131/31303/REPLICATION_STATUS=ACTIVE
service/ld2131/31303/SECONDARY_CONNECT_TIME=2018-12-18 16:09:00.513111
service/ld2131/31303/SHIPPED_LOG_BUFFERS_COUNT=2395097
service/ld2131/31303/SECONDARY_SITE_ID=2
site/2/SITE_NAME=SiteB2
site/2/SOURCE_SITE_ID=1
site/2/REPLICATION_MODE=SYNCMEM
site/2/REPLICATION_STATUS=ACTIVE
overall_replication_status=ACTIVE
site/1/REPLICATION_MODE=PRIMARY
site/1/SITE_NAME=SiteA
local_site_id=1
SAPCONTROL-OK: <end>
=================================================================================
================================================
m13adm@ld2131:/usr/sap/M13/HDB13/exe/python_support> echo $?
15
*********************************************************************************
************************************************
*********************************************************************************
************************************************
m13adm@ld2131:/usr/sap/M13/HDB13/exe/python_support> python
landscapeHostConfiguration.py
| Host | Host | Host | Failover | Remove | Storage | Storage |
Failover | Failover | NameServer | NameServer | IndexServer | IndexServer |
Host | Host | Worker | Worker |
| | Active | Status | Status | Status | Config | Actual |
Config | Actual | Config | Actual | Config | Actual |
Config | Actual | Config | Actual |
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 167
| | | | | | Partition | Partition |
Group | Group | Role | Role | Role | Role |
Roles | Roles | Groups | Groups |
| ------ | ------ | ------ | -------- | ------ | --------- | --------- |
-------- | -------- | ---------- | ---------- | ----------- | ----------- |
------ | ------ | ------- | ------- |
| ld2131 | yes | ok | | | 1 | 1 |
default | default | master 1 | master | worker | master |
worker | worker | default | default |
=================================================================================
================================================
overall host status: ok
m13adm@ld2131:/usr/sap/M13/HDB13/exe/python_support> python
landscapeHostConfiguration.py --sapcontrol=1
SAPCONTROL-OK: <begin>
hostActualRoles=worker
removeStatus=
nameServerConfigRole=master 1
failoverStatus=
hostConfigRoles=worker
failoverActualGroup=default
storageConfigPartition=1
host=ld2131
indexServerConfigRole=worker
failoverConfigGroup=default
storageActualPartition=1
indexServerActualRole=master
nameServerActualRole=master
hostActive=yes
workerActualGroups=default
workerConfigGroups=default
hostStatus=ok
storagePartition=1
SAPCONTROL-OK: <end>
=================================================================================
================================================
m13adm@ld2131:/usr/sap/M13/HDB13/exe/python_support> echo $?
4
Secondary System
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> python
systemReplicationStatus.py
this system is either not running or not the primary system replication site
Local System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mode: SYNCMEM
site id: 2
site name: SiteB2
active primary site: 1
primary masters: ld2131
=================================================================================
===============================================
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> python
systemReplicationStatus.py --sapcontrol=1
SAPCONTROL-OK: <begin>
site/2/REPLICATION_MODE=SYNCMEM
site/2/SITE_NAME=SiteB2
site/2/SOURCE_SITE_ID=1
site/2/PRIMARY_MASTERS=ld2131
local_site_id=2
SAPCONTROL-OK: <end>
168
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
=================================================================================
================================================
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> echo $?
12
*********************************************************************************
************************************************
*********************************************************************************
************************************************
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> python
landscapeHostConfiguration.py
| Host | Host | Host | Failover | Remove | Storage | Storage |
Failover | Failover | NameServer | NameServer | IndexServer | IndexServer |
Host | Host | Worker | Worker |
| | Active | Status | Status | Status | Config | Actual |
Config | Actual | Config | Actual | Config | Actual |
Config | Actual | Config | Actual |
| | | | | | Partition | Partition |
Group | Group | Role | Role | Role | Role |
Roles | Roles | Groups | Groups |
| ------ | ------ | ------ | -------- | ------ | --------- | --------- |
-------- | -------- | ---------- | ---------- | ----------- | ----------- |
------ | ------ | ------- | ------- |
| ld2132 | yes | ok | | | 1 | 1 |
default | default | master 1 | master | worker | master |
worker | worker | default | default |
overall host status: ok
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> python
landscapeHostConfiguration.py --sapcontrol=1
SAPCONTROL-OK: <begin>
hostActualRoles=worker
removeStatus=
nameServerConfigRole=master 1
failoverStatus=
hostConfigRoles=worker
failoverActualGroup=default
storageConfigPartition=1
host=ld2132
indexServerConfigRole=worker
failoverConfigGroup=default
storageActualPartition=1
indexServerActualRole=master
nameServerActualRole=master
hostActive=yes
workerActualGroups=default
workerConfigGroups=default
hostStatus=ok
storagePartition=1
SAPCONTROL-OK: <end>
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> echo $?
4
*********************************************************************************
************************************************
*********************************************************************************
************************************************
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> python
getTakeoverRecommendation.py
=================================================================================
================================================
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> python
getTakeoverRecommendation.py --sapcontrol=1
SAPCONTROL-OK: <begin>
landscapeHostConfiguration/code=4
landscapeHostConfiguration/description=ok
getTakeoverRecommendation/code=0
getTakeoverRecommendation/description=cannot decide
SAPCONTROL-OK: <end>
=================================================================================
================================================
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 169
m13adm@ld2132:/usr/sap/M13/HDB13/exe/python_support> echo $?
0
8.3.5 Checking the Status with the HDB Console
Use the HDB console to check the system replication status on all hosts and for all services.
Using the HDB console for a status check can be especially useful in an ASYNC replication. Yon can see
additional information currently not shown by the system view, because in this mode the primary systen does
not wait for acknowledgement upon arrival of the shipped redo log buer. In this case, you have to check the
current log position on the secondary system on each node and for each persistency relevant service. The
command is to be executed per service using the hdbcons -p <servicePID> "replication into"
parameter.
In the following example, you can get information about the used replication and operation modes (mode,
operation_mode). You can also see which IP address is used for data and log transfer (Log connection and Data
connection) and – since this system replication example is running with operation mode logreplay – you can
see how far the log replay is hanging behind the shipped log on this secondary (the delta between
shippedLogPos and replayFinishLogPos ). For all services the replication status should be
ReplicationStatus_Active.
<sid>adm># hdbcons -p 54321 "replication info"
SAP HANA DB Management Client Console (type '\?' to get help for client commands)
Try to open connection to server process 'hdbindexserver' on system 'M19',
instance '19'
SAP HANA DB Management Server Console (type 'help' to get help for server
commands)
Executable: hdbindexserver (PID: 66110)
[OK]
--
listing default statistics for volume 3
System Replication Secondary Information
========================================
System Replication Secondary Configuration
[system_replication] site_id = 2
[system_replication] site_name = SiteA
[system_replication] mode = sync
[system_replication] operation_mode = logreplay
[system_replication] datashipping_min_logsize_threshold = 5368709120
[system_replication] datashipping_min_time_interval = 600
[system_replication] reconnect_time_interval = 30
[system_replication] enable_log_compression = false
[system_replication] preload_column_tables = true
[system_replication] ensure_backup_history = true
[system_replication] enable_ssl = off
[system_replication] keep_old_style_alert = false
[system_replication] enable_log_retention = 1
[system_replication] logshipping_max_retention_size = 1048576
Last Primary Host: ld2133
Last Primary Port: 32003
Log Connection
- ptr : 0x00007fd58931a400
- channel : NetworkChannel FD 25 [0x00007fd5ad064a98] {refCnt=3, idx=2}
10.96.4.20/65117_tcp->10.96.4.22/32003_tcp Connected,[r---]
- mode : ReplicationMode_Synchronous
- logSinceLastBackup : 663552 bytes
- timeSinceLastBackup : 67431655 microseconds
Data Connection
- ptr : 0x00007fd589315000
170
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
- channel : NetworkChannel FD 31 [0x00007fd5ad064c58] {refCnt=2, idx=3}
10.96.4.20/65118_tcp->10.96.4.22/32003_tcp Connected,[----]
Secondary Statistics
- Creation Timestamp : 08.12.2015-14.25.27 (1449584727282603)
- Last Reset Timestamp : 08.12.2015-14.25.27 (1449584727282603)
- Statistic Reset Count : 0
- ReplicationMode : sync
- OperationMode : logreplay
- ReplicationStatus : ReplicationStatus_Active
- ReplicationStatusDetails :
- ReplicationFullSync : DISABLED
- shippedLogPos : 0x641cbb00
- shippedLogPosTimestamp : 08.12.2015-14.59.17 (1449586757965706)
- sentLogPos : 0x0
- sentLogPosTimestamp : 01.01.1970-00.00.00 (0)
- shippedLogBuffersCount : 11241
- shippedLogBuffersSize : 8335585280 bytes
- shippedLogBuffersSizeUsed : 8309875456 bytes (99.69%)
- shippedLogBuffersSizeNet : 8309875456 bytes (99.69%)
- shippedLogBufferDuration : 0 microseconds
- shippedLogBufferDurationMin : 0 microseconds
- shippedLogBufferDurationMax : 0 microseconds
- shippedLogBufferDurationSend : 0 microseconds
- shippedLogBufferDurationComp : 0 microseconds
- shippedLogBufferThroughput : 0.00 bytes/s
- replayFinishLogPos : 0x641cbb00
- replayFinishLogPosTimestamp : 08.12.2015-14.59.17 (1449586757965706)
- replayStartLogPos : 0x641cbb00
- replayPushLogPos : 0x641cbb00
- replayRetentionLogPos : 0x62a66fcb
- replayStepCount : 61709
- replayLogSize : 8335581056 bytes
- replayDuration : 111608005 microseconds
- shippedSavepointVersion : 2252
- shippedSavepointLogPos : 0x5c595f82
- shippedSavepointTimestamp : 08.12.2015-14.25.28 (1449584728678668)
- shippedFullBackupCount : 1
- shippedFullBackupSize : 17884512256 bytes
- shippedFullBackupSizeNet : 17884512256 bytes (100.00%)
- shippedFullBackupDuration : 81098893 microseconds
- shippedFullBackupDurationComp : 0 microseconds
- shippedFullBackupThroughput : 220527205.67 bytes/s
- shippedLastFullBackupSize : 17884512256 bytes
- shippedLastFullBackupSizeNet : 17884512256 bytes (100.00%)
- shippedLastFullBackupStart : 08.12.2015-14.25.28 (1449584728678668)
- shippedLastFullBackupEnd : 08.12.2015-14.26.49 (1449584809777561)
- shippedLastFullBackupDuration : 81098893 microseconds
- shippedDeltaBackupCount : 0
- shippedDeltaBackupSize : 0 bytes
- shippedDeltaBackupSizeNet : 0 bytes (-nan%)
- shippedDeltaBackupDuration : 0 microseconds
- shippedDeltaBackupDurationComp : 0 microsecond
- shippedDeltaBackupThroughput : 0.00 bytes/s
- shippedLastDeltaBackupSize : 0 bytes
- shippedLastDeltaBackupSizeNet : 0 bytes (-nan%)
- shippedLastDeltaBackupStart : not set
- shippedLastDeltaBackupEnd : not set
- shippedLastDeltaBackupDuration : 0 microseconds
- Secondary sync'ed via Log Count : 0
- syncLogCount : 0
- syncLogSize : 0 bytes
- Secondary Backup History : complete
- shippedMissingLogCount : 0
- shippedMissingLogSize : 0 bytes
- backlogSize : 0 bytes
- backlogTime : 0 microseconds
- backlogSizeMax : 0 bytes
- backlogTimeMax : 0 microseconds
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 171
- Secondary Log Connect time : 08.12.2015-14.25.27 (1449584727296916)
- Secondary Data Connect time : 08.12.2015-14.25.27 (1449584727491743)
- Secondary Log Close time : not set
- Secondary Data Close time : not set
- Secondary Log Reconnect Count : 0
- Secondary Log Failover Count : 0
- Secondary Data Reconnect Count : 0
- Secondary Data Failover Count : 0
[OK]
--
[EXIT]
--[BYE]
Related Information
SAP Note 1999880
8.3.6 Checking the Status with Predened SQL Statements
Use predened SQL statements to check the system replication status.
Predened SQL statements are attached to the SAP Note 1969700. You can copy and execute the statements
contained in the text les in the SAP HANA Database Explorer or the SAP HANA studio:
172
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Example
Copy and execute the HANA_Replication_SystemReplication_Bandwidth.text in the SAP HANA Database
Explorer:
In the SAP HANA studio, go for the primary system to the System Information tab and right-click Name
Import SQL Statements :
Import the SQL statements downloaded from the SAP Note and right-click on the statements, then choose
Replication Overview .
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 173
This will open information about system replication and the replication state for each service:
In a synchronous replication, it is interesting to compare Local log buer write throughput (MB/s) with Log
buer shipping throughput (MB/s). If these values dier too much, they could indicate network problems or a
problem with the I/O on the secondary system.
For more information about how to interpret check results, see SAP Note 1999993.
174
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
8.4 Monitoring System Replication
You can monitor system replication using dierent tools.
● SAP HANA cockpit
For more information, see Monitoring System Replication with the SAP HANA Cockpit and Example:
Monitoring System Replication with the SAP HANA Cockpit.
● SAP HANA studio
For more information, see Monitoring System Replication with the SAP HANA Studio.
● hdbnsutil
For more information, see Monitoring System Replication with hdbnsutil.
● SQL query
For more information, see Monitoring System Replication with SQL query.
Related Information
Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 175]
Example: Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 177]
Monitoring SAP HANA System Replication with the SAP HANA Studio [page 185]
Monitoring SAP HANA System Replication with hdbnsutil [page 183]
Monitoring SAP HANA System Replication with SQL query [page 186]
8.4.1 Monitoring SAP HANA System Replication with the SAP
HANA Cockpit
To monitor SAP HANA system replication, you can use the System Replication tile in the SAP HANA cockpit.
To open the System Replication Overview page, click the System Replication tile on the Overview page in the
SAP HANA cockpit.
The System Replication Overview displays a graphical representation of the system replication landscape with
the following information:
● The name and role of the system, as well as the selected operation mode
For the operation modes logreplay and logreplay_readaccess a retention time estimation is also
displayed. The Estimated log retention time is an estimation of the time left before the primary system
starts to overwrite the RetainedFree marked log segments and a full data shipping becomes necessary to
get the primary and secondary systems back in sync after a disconnect situation. The Estimated log full
time is an estimation of the time left before the primary system runs into a log full. The value shown in the
header shows the situation into which the system could run rst: log retention or log full.
● If the SQL ports of the secondary system are open for read access
● The replication mode used between the systems
● The current average redo log shipping time and the average size of shipped redo log buers
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 175
It describes how long it took on average to send redo log buers to the secondary site based on
measurements of the last 24 hours.
Furthermore, detailed information on system replication is provided in the following tabs:
Note
These tabs are displayed only if you congured a system replication before.
Tabs on the System Replication Overview
Tab Name Description
Related Alerts The Related Alerts tab provides a description of any existing alerts, as well as their
priority. This tab is only displayed when system replication related alerts are availa
ble.
Replicated Services The Replicated Services tab provides information on the replication status per site
and service.
Network The Network tab provides information on the time it took to ship the redo log to the
secondary system and to write the redo log to the local log volume on disk.
You can select the network connection that you want to analyze (for example,
Network Site 1 to 2 or Network Site 2 to 3). The graph displayed compares the local
write wait time with the remote write wait time monitored over the last 24 hours.
Log Shipping Backlog The Log Shipping Backlog tab provides a graphical representation on the history of
the log shipping backlog.
Log Replay The Log Replay tab provides a graphical representation on the delay of the secon
dary system. This tab is displayed if the chosen operation mode for the system repli
cation landscape is
logreplay or logreplay_readaccess.
When this tab is activated for a secondary system, the log replay delay is shown for
the last 24 hours.
Furthermore, in this tab you can select to visualize the estimated log retention time
as well as the estimated log full time for all system replication relevant services.
Network Speed Check The Network Speed Check tab provides a way to measure the network speed of the
system replication host-to-host network channel mappings.
Network Security Settings The Network Security Settings tab displays the specic network security details con
gured between the primary and the secondary systems.
176 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
8.4.1.1 Example: Monitoring SAP HANA System Replication
with the SAP HANA Cockpit
Learn how to monitor multiple SAP HANA systems with the SAP HANA cockpit.
In the SAP HANA cockpit, you can monitor system replication using the system replication tile, the system
replication overview, and the tabs on the system replication overview.
System Replication Tile
If system replication is congured, the corresponding tile appears on the main screen of the system overview
page providing information about the type of landscape (tier 2 or tier 3), the replication modes between the
primary and the secondary systems, the operation mode, as well as an overall replication status. For examples,
see System Replication Tile. If all tiers are shown in green and the system replication tile indicates that all
services are active and in sync, your system is doing well. Red tiers would indicate a problem with the
replication.
If the tile does not show up, you have to grant the system replication specic role to the corresponding user.
System Replication Overview
To check the status of replication in detail, open the system replication tile. The application provides an
overview on the system replication conguration and status. The “chain” of systems with their replication
modes is also displayed containing further information about the sites and the network connections between
them.
For example, the system replication overview can look like this:
A graphical representation of your system replication landscape is given. It tells you the chosen system names,
the replication mode used between the systems, as well as how long it took on average to send redo log buers
to the secondary system based on measurements from the last 24 hours. For a synchronous replication, this is
the round trip time for sending the redo log buer and receiving the acknowledgement; for an asynchronous
replication it refers to the time that it takes until the log buer was sent after its creation.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 177
Related Alerts Tab
If a system replication relevant alert occurred, the rst tab is the Related Alerts tab.
If no system replication relevant alert exists, this tab is not shown.
Replicated Services Tab
In the Replicated Services tab, an excerpt of the M_SERVICE_REPLICATION monitoring view is shown. The
displayed table shows the replication state for each system and service.
For example, the tab can look like this:
Recommendation
If you want to look at the trace les of all systems in your system replication landscape, choose the View
trace and diagnostic les link on the primary system's overview page in the SAP HANA cockpit. This makes
all diagnosis les from the trace directories of all systems visible in the browser.
To see the details for the corresponding service grouped thematically, choose one row like in the following
example for one indexserver. Since this information is context aware, you only get the information required for
this system. For example, because this system is running with the logreplay operation mode, no information
on
delta data shipping is shown here. However, the context-sensitive information about the log replay
178
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
delay is displayed. The delta between Last Log Position and Replayed Log Position indicates how far the log
replay is behind on the secondary system:
Network Tab
You can select the network connection that you want to analyze from the drop down menu. This can be
Network Site 1 to 2 or Network Site 2 to 3. The displayed graphic compares the local write wait time (writing
redo log buer into the local log volume) with the remote write wait time (shipping the redo log and receiving
the acknowledgement) monitored over the last 24 hours. You can see if peak times occurred and how the
network connection reacted.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 179
For example:
If the ASYNC replication mode is congured between two systems, you also receive information about the
network performance by selecting the corresponding connection between tier 2 and tier 3. For example:
In this example, the average write wait time in ms describes the time it took from the creation of the redo log
buer (committing a write transaction) until the redo log buer was sent out to the network. This value is an
indicator for peak load phases and could point to network or I/O problems on the secondary system, which can
inuence the primary’s performance as well.
For more information, see Monitoring the Network Latency.
180
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Log Replay Tab
This tab is only visible, if logreplay or logreplay_readaccess is congured for the system replication
landscape. The log replay delay on the secondary system is displayed as size in GB for the last 24 hours. If at
some point in time, the threshold of the corresponding alert (for example, ID 94 System Replication Logreplay
Backlog) was exceeded, this is indicated accordingly. For example:
For more information on how to analyze a high replay backlog, see SAP Note 2409671: High replay backlog on
HANA System Replication Secondary Site.
Additionally, you can select the log retention information for your system. For more information, see Estimating
the Maximum Retention Time.
Network Speed Check
To measure the network speed for system replication, host-to-host mappings from the primary system to the
secondary system (and even to a third tier) can be done using the Network Speed Check (System Replication
Communication) tab.
To measure the network speed (MB/s), choose a package size from the drop-down menu and choose Measure
Network Speed.
In this example, there are standby hosts congured on the primary (ha-test-03) and the secondary (ha-
test-06) system. The network speed measurement cannot be done for standby hosts, because they do not
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 181
have access to the data and log volumes of the SAP HANA database and thus are not relevant for replication in
this state:
You can also measure the network speed for scale-out SAP HANA databases using theNetwork Speed Check
(Internal Communication) tab. It measures the network speed in both directions for each host to each other
host of the scale-out system. To do so, choose a package size from the drop-down menu and choose Measure
Network Speed
. The result is a list representing the network speed for every host in the scale-out system to
every other host of the system. The measurement is done in both directions. Thus, every host-mapping is
displayed twice in this list with switched roles of Sender and Receiver. The fastest connection is displayed rst.
Note
Measuring the network speed of your SAP HANA system replication communication channels or internode
communication channels in your SAP HANA scale-out system will have an impact on your network
performance and thus on your running systems for the time of the measurement.
182
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Network Security Settings
This tab shows the security details congured between the primary and the secondary systems:
Related Information
Estimating the Maximum Retention Time [page 23]
System Replication Tile [page 40]
Monitoring the Network Latency [page 192]
SAP Note 2409671
8.4.2 Monitoring SAP HANA System Replication with
hdbnsutil
You can monitor SAP HANA system replication using hdbnsutil.
Standard System Replication
To view the status of the system replication topology conguration on both systems, execute hdbnsutil –
sr_state on the primary and the secondary:
tedadm@ld2131:/usr/sap/TED/HDB07> hdbnsutil -sr_state
checking for active or inactive nameserver ...
System Replication State
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 183
~~~~~~~~~~~~~~~~~~~~~~~~
mode: primary
site id: 1
site name: SITEA
Host Mappings:
~~~~~~~~~~~~~~
ld2131 ->
[SITEA] ld2131
ld2131 ->
[SITEB] ld2132
done.
Multitier System Replication
For a multitier system replication the mappings of all three systems are displayed:
ut1adm@ld2131:/usr/sap/UT1/HDB01> hdbnsutil -sr_state
checking for active or inactive nameserver ...
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~
mode: primary
site id: 1
site name: SITEA
Host Mappings:
~~~~~~~~~~~~~~
ld2131 ->[SITEA] ld2131
ld2131 ->[SITEC] ld2133
ld2131 ->[SITEB] ld2132
done.
When using the additional option –-sapcontrol=1, the key value pair output can be parsed by a script line by
line.
Here is the output where the -sr_state command was executed on a primary system of a multitier system
replication:
ut1adm@ld2131:/usr/sap/UT1/HDB01> hdbnsutil -sr_state --sapcontrol=1
checking for active or inactive nameserver ...
SAPCONTROL-OK: <begin>
mode=primary
site id=1
site name=SITEA
mapping/ld2131=SITEA/ld2131
mapping/ld2131=SITEC/ld2133
mapping/ld2131=SITEB/ld2132
SAPCONTROL-OK: <end>
Done
Here is the output where the -sr_state command was executed on a tier 2 secondary site of a multitier
system replication:
ut1adm@ld2132:/usr/sap/UT1/HDB01> hdbnsutil -sr_state --sapcontrol=1
checking for active or inactive nameserver ...
SAPCONTROL-OK: <begin>
mode=sync
site id=2
site name=SITEB
active primary site=1
mapping/ld2132=SITEA/ld2131
mapping/ld2132=SiteC/ld2133
184
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
mapping/ld2132=SITEB/ld2132
primary masters=ld2131
SAPCONTROL-OK: <end>
done.
Output Reference
Output
Description
Mode
Can have the values primary, sync, async, and syncmem to represent the mode relevant
on the system where the command is executed.
For example, in a multitier system replication on the primary the mode would be primary,
on the tier 2 secondary it could be either sync or syncmem, and on the tier 3 secondary it is
async.
Site ID
A unique identier of a system which is incremented for each system attached to a SAP HANA
system replication. It is removed, when system replication is disabled.
Site Name The name you give your systems during the enable and register steps of the system replica
tion conguration.
Mapping/<currentHost> Shows which hosts are involved in this SAP HANA system replication together with their sys
tem name. If the SAP HANA database is oine, this host mapping cannot be shown on the
secondaries.
Active primary site Shows the system ID of the currently active system.
Primary masters Shows the host names of the currently active master candidates of the primary.
Note
When running hdbnsutil –sr_state on an oine SAP HANA, no host mapping will be available. For
more information, see SAP Note 2315257.
Related Information
SAP Note 2315257
8.4.3 Monitoring SAP HANA System Replication with the
SAP HANA Studio
You can monitor SAP HANA system replication using the SAP HANA studio.
You can monitor system replication in the administration editor of the primary system as follows:
● The general status is displayed on the Overview tab.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 185
This should be All services are active and in sync.
● Detailed information is available on the Landscape System Replication tab.
Here you can see the system replication status. For all services, the REPLICATION_STATUS should be
ACTIVE. Detailed information about shipped sizes and shipping times are also available.
Since the secondary instance does not accept SQL connections while data replication is active, basic
information about the secondary system is also shown.
For more information about the meaning of the individual elds, see the M_SERVICE_REPLICATION
system view. For more information on the system replication status, see Checking the System Replication
Status.
Related Information
Checking the SAP HANA System Replication Status [page 159]
M_SERVICE_REPLICATION System View [page 269]
8.4.4 Monitoring SAP HANA System Replication with SQL
query
Use SQL to directly get system replication specic information from the system views.
The M_SYSTEM_REPLICATION view provides general system replication relevant information about the whole
system. For example, it gives information on the used replication mode, operation mode, and as which tier a
system is congured.
Example
In this example SiteA with SITE_ID 1 is currently congured as TIER 1 as primary.
On the primary execute:
select * from "SYS"."M_SYSTEM_REPLICATION";
The M_SERVICE_REPLICATION system view is another possibility to get system replication information. The
contents of the M_SERVICE_REPLICATION view are collected by the statistics server every hour. Therefore, the
history of the data and log replication can be viewed in the table.
186
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Example
On the primary, execute the following command to view the data replicated by the index servers (volume 4
in this example) from the primary to the tier 2 secondary:
select * from "_SYS_STATISTICS"."HOST_SERVICE_REPLICATION"
where volume_id=4 and site_id=1;
Related Information
M_SERVICE_REPLICATION System View [page 269]
M_SYSTEM_REPLICATION System View [page 274]
SAP HANA SQL and System Views Reference
8.5 System Replication Network Connection
The replication in a congured system replication uses either a public or a separate network channel between
the involved data centers.
Learn in this section how to congure and monitor the system replication network connection. For detailed
information about network speed chekcs with SAP HANA cockpit, see Example: Monitoring SAP HANA System
Replication with the SAP HANA Cockpit.
For more information about security aspects, see also Security Aspects for SAP HANA System Replication.
For information about the distance between the data centers, network throughput, as well as data and log
compression, see Network Recommendations.
Related Information
Secure Conguration of the Network Connection [page 188]
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 187
Encryption of the Connection [page 191]
Monitoring the Network Connection [page 192]
Monitoring the Network Latency [page 192]
Example: Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 177]
Security Aspects for SAP HANA System Replication [page 251]
Network Recommendations [page 27]
8.5.1 Secure Conguration of the Network Connection
By default, the primary and secondary systems establish communication using the internal host names.
With an IPaddress–virtualHostname mapping on the involved systems, the system replication host name
resolution can be set conguring a separate network for system replication data trac between the primary
and the secondary system.
This is done in the [system_replication_hostname_resolution] section in global.ini, where all hosts of
the primary and the secondary systems must be dened on each site:
global.ini/[system_replication_hostname_resolution]
<ip-address_same_site>=<internal_host_same_site>
<ip-address_other_site>=<internal_host_other_site>
This is also valid for a multitier system replication consisting of three sites (primary, tier 2 secondary, tier 3
secondary or more) because the roles can switch after takeovers and failbacks.
Note
The parameters in the global.ini le must be set prior to registering the secondary system, because the
hdbnsutil -sr_register command uses this mapping. Registration is one step in the process of
conguring the secondary system.
The entries in the [system_replication_hostname_resolution] section are used in combination with
the listeninterface parameter in the [system_replication_communication] section. The following
combinations are possible:
[system_replication_com
munication]
listeninterface
[system_replication_host
name_resolution] Additional Information
.global No mappings specied Default, if nothing is specied. The default network
route is used for system replication communication.
Note
If you use a public network instead of a separate
network, you must secure this connection with
additional measures such as a rewall or a virtual
private network and SSL.
188 P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
[system_replication_com
munication]
listeninterface
[system_replication_host
name_resolution] Additional Information
.global Entries for the primary and secon
dary hosts (for all hosts in multitier
setups)
A separate network is used for system replication
communication.
Recommendation
In this way you can use a separate network for
multitier system replication.
.internal
Entries for the primary and secon
dary hosts
A separate network is used for system replication
communication. The primary hosts listen on the dedi
cated ports of the separate network only and incom
ing requests on the public interfaces are rejected.
Note
In SAP HANA 1.0 SPS 11, network communication
for system replication with
listeninterface=.internal is sup
ported for tier 2 replication, but not for tier 3 set
ups.
There are two ways to activate [system_replication_hostname_resolution] in your system:
● Restart all sites after setting the parameter
● Temporarily resolve the system replication conguration (no restart of the primary is necessary):
1. Stop secondary.
2. Unregister secondary.
3. Disable primary.
4. Enable primary.
5. Register secondary.
6. Start secondary.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 189
Example
Here is an example of the settings for a tier 2 system replication (3 node system) using a separate internal
network per site and a separate connection for the system replication:
If for some reason, no separate network channel was congured for the SAP HANA system replication
communication between the involved systems, the allowed_sender parameter could be used to restrict
communication between primary and secondary to certain hosts. For this, the following settings can be
congured in the global.ini le on the primary system:
global.ini/[system_replication_communication]
Parameter: allowed_sender
Value: <list of IP-addresses of secondary or CIDR-netmasks>
Example: 10.0.1.0/30
The default is "no restriction".
Related Information
Host Name Resolution for System Replication
Security Aspects for SAP HANA System Replication [page 251]
190
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
8.5.2 Encryption of the Connection
The system replication connections for data, redo log, and metadata of the nameserver are secured with SSL
by systemPKI without the need to switch on SSL for the internal communication.
All system replication connections are specied only by the setting
[system_replication_communication]/enable_ssl. This simplies security settings for system
replication.
Note
Before upgrading SAP HANA from a previous version to HANA 2.0 SPS02 or higher using near zero
downtime upgrade (NZDU), see SAP Note 2494079: Near-Zero-Downtime-Upgrade to HANA 2 SPS02 or
above when internal communication SSL is used.
SAP HANA System Replication supports secure network communication (SSL) for data and log shipping to the
secondary system. The following settings can be congured in the global.ini le:
global.ini/[system_replication_communication]
Parameter: enable_ssl
Values:
off: ssl is disabled for source and target replication channels (default)
on: ssl is enabled for source and target replication channels
source: ssl is enabled as source replication channel only
target: ssl is enabled as target replication channel only
In this context, the 4xx06 port was added in system replication communication covering encrypted metadata
communication of the nameserver. The following ports are available:
TCP Port
Service Used For
4xx01 nameserver Log and data shipping (System DB)
4xx02 nameserver Metadata communication (System DB) – unencrypted
4xx06 nameserver Metadata communication (System DB) – encrypted via SSL
4xx40 - 4xx97 indexserver Log and data shipping (tenant DBs)
4xx40 - 4xx97 scriptserver Log and data shipping (optional, tenant DBs)
4xx40 - 4xx97 docstore
Log and data shipping (optional, tenant DBs)
Related Information
Security Aspects for SAP HANA System Replication [page 251]
SAP Note 2494079
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 191
8.5.3 Monitoring the Network Connection
You can monitor the network connection using alerts or the HDB console.
The connection between the primary and the secondary system must be available for replication. If this is not
the case for a certain time, the redo log cannot be shipped to the secondary system, the log segments start
piling up on the primary, and the secondary system is not ready for takeover. For more information, see Log
Retention.
The alert 78 (System Replication Connection Closed) is visible in the SAP HANA cockpit, the SAP HANA studio,
and in the system view M_EVENTS to ensure that the primary system stays operational at all times, even if the
connection is occasionally lost.
Additionally, the replication connection can be checked using the HDB console: hdbcons -p <PID of
replicating service> "replication info" . The PID (process ID) of the replicating service can be
obtained by using HDB proc as <sid>adm. For example: hdbcons -p 12345 "replication info"
The output delivers Log Connection information for the connection used by the provided service. It also shows
errors, if the connection cannot be resolved properly:
…
Log Connection
- ptr : 0x00007fdb6e8e3410
- channel : NetworkChannel FD 158 [0x00007fdb6f1bbc90] {refCnt=3, idx=1}
10.68.91.226/3 0103_tcp->10.68.92.13/49537_tcp Connected,[r---]
...
To check, if the congured connection is actually used, use the OS command: lsof –n –p <indexserver-
pid>.
For a detailed analysis of the network connection used for system replication, see Troubleshoot System
Replication and SAP Note 2081065: Troubleshooting SAP HANA Network.
Related Information
Log Retention [page 20]
SAP HANA System Replication Alerts [page 153]
Troubleshoot System Replication [page 209]
SAP Note 2081065
8.5.4 Monitoring the Network Latency
You can monitor the network latency with the SAP HANA cockpit or the SAP HANA studio.
The latency for the redo log shipping is of interest in a running system replication for the synchronous
replication modes SYNC or SYNCMEM regardless of the used operation mode.
192
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
Note
The redo log shipping wait time for 4 KB log buers must be less than a millisecond or in a low single-digit
millisecond range – depending on the application requirements (relevant for synchronous replication
modes only).
All changes to data are captured in the redo log. The SAP HANA database asynchronously persists the redo log
with I/O orders of 4 KB to 1 MB size into log segment les in the log volume of the primary system. A
transaction writing a commit into the redo log waits until the buer containing the commit has been written to
the log volume. This wait time for 4 KB log buers should be less than a millisecond or in a low single-digit
millisecond range. Once you congured system replication, you can check the local and the remote log write
wait times in the SAP HANA cockpit or collect them in the SAP HANA studio with the SQL statement
HANA_Replication_Overview attached to the SAP Note 1969700.
Network Latency in the SAP HANA Cockpit
In the SAP HANA cockpit you receive an overview of your running system replication displaying the average log
buer size between the primary and the secondary, as well as the average redo log shipping wait time. Opening
the Network tab, you can also have a look at the 24 hour history of these local log write wait times and the
remote log shipping wait times. For more information, see Example: Monitoring SAP HANA System Replication
with the SAP HANA Cockpit.
For example:
Network Latency in the SAP HANA Studio
You can gather these values also with the SAP HANA studio after importing the SQL statement collection
(attached to SAP Note 1969700) to the SAP HANA studio: System Information Import SQL Statements .
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 193
Right-click on the statements under Replication Overview and choose Execute from the context menu.
You will receive information about the system replication landscape and the replication state for each service.
The log write wait times on the “local” system (that is, on the persistence of the primary system) is returned by
the mentioned SQL statement as Average local log buer write time (ms) per service, where the index server is
the one of interest.
Additionally, the mentioned SQL statement returns the redo log write latency for the shipping to the secondary,
which can be slightly higher than the locally measured log write wait time. The returned value Average local log
buer write time (ms) represents the time between enqueueing and nishing a request.
Of interest is also the Local log buer write throughput (MB/s) compared to the Log buer shipping throughput
(MB/s) in synchronous replication. If these two values dier too much, this could be an indication for network
problems in a synchronous replication or a problem with the I/O on the secondary system (for SYNC). For
information about ASYNC replication, see the Check Network Conguration (Long Distance) section in
Replication Performance Problems.
Related Information
Example: Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 177]
Replication Performance Problems [page 216]
SAP Note 1969700
8.6 Copy or Move Tenants Within System Replication
You can copy or move a tenant in a SAP HANA tenant database system into or out of a primary system running
in a system replication conguration.
There are three tenant copy or move scenarios that are applicable in a primary system:
Copy or move a tenant into a primary system
A tenant database is copied or moved into the primary system of a system replication conguration.
The copied or moved tenant starts replicating after the FINALIZE command was executed.
Copy or move a tenant from the primary system to a tenant database system
(≠ secondary)
A tenant database is copied or moved out of the primary system into a target tenant database dierent from
the secondary.
194
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
The copied or moved tenant arrives in the target tenant database system and is runnable after the FINALIZE
command was executed.
Copy or move a tenant within the primary system
A tenant database is copied or moved within the primary system of a system replication.
The “cloned” tenant is created in the primary system and the replication starts after the FINALIZE command
was executed.
Related Information
Copying and Moving Tenant Databases
SAP HANA System Replication with Tenant Databases [page 83]
8.7 Copying a System using System Replication
SAP HANA system replication can be used to create a copy of an SAP HANA database in a quick and simple
way.
You can register another SAP HANA database in one of the two system replication scenarios:
● As a secondary for a standalone SAP HANA database
● As a tier 3 secondary in a tier 2 system replication landscape
System Replication Scenarios
Original Setup
Source Database Target Database
Standalone SAP HANA Database Primary Secondary
Tier 2 System Replication Tier 2 Secondary Tier 3 Secondary
After the replication is active and in sync, a takeover to the newly added tier makes the standalone SAP HANA
database runnable with identical data as the source database.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 195
8.7.1 Copy a System using System Replication
You can use SAP HANA system replication to create a copy of an SAP HANA database.
Prerequisites
● You need an SAP HANA database called source database, which is to be copied.
● You need a separate (virtual) host or more (virtual) hosts for the database copy called target database.
Procedure
1. Install an SAP HANA database of the same or a higher revision as the target database on the separate
(virtual) host or on the (virtual) hosts.
2. Prepare the source database for replication using hdbnsutil -sr_enable [--name=<siteName>]
3. Register the target database either as a secondary or tier 3 secondary depending on your original setup
using hdbnsutil -sr_register --remoteHost=<primary master host> --
remoteInstance=<primary instance id> --replicationMode=[sync|syncmem|async] --
name=<sitename> --operationMode=[delta_datashipping|logreplay]
If the parameter is given, the operation mode is set. The default operation mode is logreplay.
4. Start the newly registered target database.
5. When the system replication is active and in sync, perform a takeover on the target database.
6. After the takeover is done, the target database is running as a copy of the source database.
7. To avoid confusion with the source databases, rename the <SID> and change the instance number using
the tool hdblcm.
8.8 Updating SAP HANA Systems with SAP HANA System
Replication
You can update your SAP HANA systems running in a SAP HANA system replication.
If for some reason you have to stop and restart the primary or the secondary, once the systems are available
they will automatically try to get in sync again. There are no manual steps necessary.
If the system is running with logreplay or logreplay_readaccess, see Resync Optimization to prevent
your system from running full or having to do a full data shipping. This can happen when the time during which
the primary could not replicate gets too long. An optimized resync with delta data or log shipping can be
achieved avoiding a full data shipping, depending on the the time the upgrade is taking and the settings of the
logshipping_max_retention_size and datashipping_snapshot_max_retention_time parameters.
196
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
You must update your SAP HANA systems running in a system replication setup by updating the secondary
system rst and then updating the primary system. For more information, see Update an SAP HANA System
Running in a System Replication Setup.
The secondary system can run with a higher software version than the primary system. For more information,
see Use SAP HANA System Replication for Near Zero Downtime Upgrades.
Note
System Replication with SAP HANA 2.0 requires authentication for data and log shipping channels. The
authentication is done using the certicates in the system PKI SSFS store. Thus, there is an additional
manual setup step required to exchange certicates in the system PKI SSFS store between the primary and
the secondary system when upgrading from SAP HANA 1.0 to SAP HANA 2.0. For more information, see
SAP Note 2369981.
Hardware can also be exchanged with a minimal downtime using SAP HANA system replication. For more
information, see SAP Note 1984882: Using HANA system replication for Hardware Exchange with Minimum
Downtime.
Related Information
Resync Optimization [page 19]
Congure a User Under the SRTAKEOVER Key [page 201]
Update SAP HANA Systems Running in a System Replication Setup [page 197]
Use SAP HANA System Replication for Near Zero Downtime Upgrades [page 199]
Use Multitarget System Replication for Near Zero Downtime Upgrades [page 203]
SAP Note 2369981
SAP Note 1984882
8.8.1 Update SAP HANA Systems Running in a System
Replication Setup
You can update your SAP HANA system with active system replication by updating the secondary and the
primary system one after the other.
Prerequisites
System replication is congured and active between two SAP HANA systems.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 197
Context
You must update your SAP HANA system running in a system replication setup by updating the secondary
system rst and then updating the primary system.
Remember
For system replication setups it is required that the secondary system has the same version as the primary
system or a higher version. As such, the secondary system must always be updated before the primary
system.
Note
Updating one system after the other results in some downtime. If you want to update your system with
reduced downtime, see Use SAP HANA System Replication for Near Zero Downtime Upgrades.
It is possible to reduce the time required to perform an update. For more information, see Prepare an
Update for Flexible System Downtime .
Procedure
1. Upgrade the SAP HANA server software and all installed components on the secondary system.
From your installation directory execute as root or as <sid>adm:
./hdblcm --action=update
2. With the secondary system online, use the SAP HANA lifecycle management tools to upgrade all the other
components to the same revision as the server software.
3. Verify that system replication is active and that all services are in sync.
You can check that the REPLICATION_STATUS column in M_SERVICE_REPLICATION has the value ACTIVE
for all services.
4. Upgrade the SAP HANA server software and all installed components on the primary system.
From your installation directory, execute as root or as <sid>adm:
./hdblcm --action=update
5. With the secondary system online, use the SAP HANA lifecycle management tools to upgrade all other
components to the same revision as the server software.
6. Verify that system replication is active and that all services are in sync.
Related Information
Use SAP HANA System Replication for Near Zero Downtime Upgrades [page 199]
Prepare an Update for Flexible System Downtime
198
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
SAP Note 2407186
SAP Note 2599514
8.8.2 Use SAP HANA System Replication for Near Zero
Downtime Upgrades
You can use SAP HANA system replication to upgrade your SAP HANA systems as the secondary system can
run with a higher software version than the primary system.
Prerequisites
You congured a user in the local userstore under the SRTAKEOVER key. For more information, see Congure a
User Under the SRTAKEOVER Key.
System replication is congured and active between two identical SAP HANA systems:
● The primary system is the production system.
● The secondary system will become the production system after the upgrade.
● The prerequisite is to run both systems with the same endianness.
Context
With system replication active, you can rst upgrade the secondary system to a new revision and have it take
over in the role of primary system. The takeover is carried out in only a few minutes and committed
transactions or data are not lost. You can then do an upgrade on the primary system, which is now in the role of
secondary.
Note
It is possible to reduce the time required to perform an update. For more information, see Prepare an
Update for Flexible System Downtime.
The secondary system can be initially installed with the new software version or upgraded to the new software
version when the replication has already been congured. After the secondary has been upgraded, all data has
to be replicated to the secondary system (already having the new software version). When the secondary
system is ACTIVE (all services have synced), a takeover has to be executed on the secondary system. This step
makes the secondary system the production system running with the new software version.
Note
If you are upgrading from SAP HANA 1.0 to SAP HANA 2.0, copy the system PKI SSFS key and the data le
from the current primary system to the new to-be secondary system:
/usr/sap/<SID>/SYS/global/xsa/security/ssfs/data/SSFS_<SID>.DAT
/usr/sap/<SID>/SYS/global/xsa/security/ssfs/key/SSFS_<SID>.KEY
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 199
For more information, see SAP Note 2369981: Required conguration steps for authentication with HANA
System Replication.
In an Active/Active (read enabled) system replication setup, the version of the primary and the secondary
systems must be identical. For the near zero downtime upgrade to work, the operation mode on the secondary
system is automatically set to logreplay. Like this, the two systems can get back in sync before the takeover
step. To establish again the Active/Active (read enabled) landscape at the end, the logreplay_readaccess
operation mode must be explicitly specied during the former registration of the primary system as a new
secondary system.
For more information about near zero downtime upgrades when using a multitarget system replication setup,
see Use Multitarget System Replication for Near Zero Downtime Upgrades.
Procedure
1. Upgrade the secondary system's SAP HANA server software and all other components.
From your installation directory execute as root:
./hdblcm --action=update
2. Verify that system replication is active and that all services are in sync.
You can check that the column REPLICATION_STATUS in M_SERVICE_REPLICATION has the value ACTIVE
for all services.
3. Stop the primary system.
4. Perform a takeover on the secondary system, switch virtual IP addresses to the secondary system, and
start using it productively.
As <sid>adm perform a takeover:
hdbnsutil -sr_takeover
5. If XS Advanced is being updated as well, update the XS Advanced applications.
./hdblcm --action=update
6. Upgrade the original primary from the installation directory as root user using the --
hdbupd_server_nostart option together with all other components. This is necessary because
otherwise the primary has to be stopped again before it can be registered as the secondary.
./hdblcm --action=update --hdbupd_server_nostart
Note
For a fast synchronization of the sites – after registering again the original primary system – perform
this failback within the time given by the datashipping_snapshot_max_retention_time
parameter (default 300 minutes). Otherwise, a full data shipping will be done. Furthermore, the
optimized resync depends on the availability of the last snapshot. For more information about near
zero downtime upgrades in multitier system replication, see SAP Note 2386973.
200
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
7. Register the original primary as secondary as <sid>adm.
hdbnsutil -sr_register --name=<secondary_alias>
--remoteHost=<primary_host> --remoteInstance=<primary_systemnr>
--replicationMode=[sync|syncmem|async]–-operationMode=[delta_datashipping|
logreplay|logreplay_readaccess]
8. Start the original primary.
Related Information
Congure a User Under the SRTAKEOVER Key [page 201]
Prepare an Update for Flexible System Downtime
Updating the SAP HANA System
Perform a Near-Zero Downtime Update
Use Multitarget System Replication for Near Zero Downtime Upgrades [page 203]
Deploy a Multi-Target Application with Zero-Downtime Maintenance
SAP Note 2369981
SAP Note 1984882
SAP Note 2386973
SAP Note 2494079
SAP Note 2407186
SAP Note 2300936
8.8.2.1 Congure a User Under the SRTAKEOVER Key
In preparation for maintenance tasks, you must congure a user in the local userstore under the SRTAKEOVER
key.
Context
The SRTAKEOVER user requires the necessary privileges to import the repository content of the new version of
the software during the takeover process.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 201
Procedure
1. As <sid>adm congure a user in the local userstore under the SRTAKEOVER key. Use a public host name
to access the corresponding SQL port of the System DB (<SystemDBsqlport>). Execute this command
on the primary and secondary systems:
hdbuserstore SET SRTAKEOVER <publichostname>:<SystemDBsqlport> <myrepouser>
<myrepouser_password>
Note
This conguration step should be performed only in the system database, not in every single tenant.
2. Create a <myrepouser> user with the necessary privileges to import the repository content as follows:
CREATE USER MY_REPO_IMPORT_USER PASSWORD MyRepoUserPW123;
GRANT EXECUTE ON SYS.REPOSITORY_REST TO MY_REPO_IMPORT_USER;
GRANT REPO.READ ON ".REPO_PACKAGE_ROOT" TO MY_REPO_IMPORT_USER;
GRANT REPO.IMPORT TO MY_REPO_IMPORT_USER;
GRANT SELECT ON _SYS_REPO.DELIVERY_UNITS TO MY_REPO_IMPORT_USER;
GRANT REPO.ACTIVATE_IMPORTED_OBJECTS ON ".REPO_PACKAGE_ROOT" TO
MY_REPO_IMPORT_USER;
For example, for public host name "mypublichost" and system number "00", "MY_REPO_IMPORT_USER",
and "MyRepoUserPW123" :
hdbuserstore SET SRTAKEOVER mypublichost:30013 MY_REPO_IMPORT_USER
MyRepoUserPW123
The host name has to be the public host name of the host on which the command is executed and the port
is the SQL port number of the system database.
For more information, see the Secure User Store (hdbuserstore) section in the SAP HANA Security Guide.
Note
In a scale-out conguration, the command has to be executed on all hosts. If the password for the
repository import user is changed, the password saved in the userstore also has to be changed.
Related Information
Use SAP HANA System Replication for Near Zero Downtime Upgrades [page 199]
Secure User Store (hdbuserstore)
202
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
8.8.2.2 Use Multitarget System Replication for Near Zero
Downtime Upgrades
You can upgrade your SAP HANA systems running in a multitarget system replication setup.
Prerequisites
Multitarget system replication is congured and active between identical SAP HANA systems.
Context
We are using the following setup to exemplify the procedure:
Example
Primary system A replicates data changes to secondary system B located in the same data center. Primary
system A also replicates data changes to the secondary system C located in data center 2. Secondary
system C is a source system for a further secondary system D located in the same data center with system
C.
In this setup:
● The primary system is the production system.
● The secondary system located in the same data center as the primary system will become the
production system after the upgrade. Further secondary systems are located in a remote data center.
● There is no replication error.
Procedure
1. Upgrade secondary system B in data center 1 and secondary system D in data center 2.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 203
2. Register secondary system D in data center 2 to secondary system B in data center 1.
204
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
3. Upgrade secondary system C in data center 2. Then, register secondary system C to secondary system D
in data center 2.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 205
4. Take over on secondary system B in data center 1.
After takeover, secondary system B will be the new primary system.
206
P U B L I C
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
5. Upgrade and register the previous primary system A to the new primary system B in data center 1.
SAP HANA System Replication
SAP HANA System Replication: Operation and Maintenance
P U B L I C 207
9 Troubleshoot System Replication
Learn how to analyze, avoid, and solve problems related to system replication.
Which problems can I avoid or solve related to system replication?
This chapter provides information about:
● Solving I/O performance problems related to system replication
● Determining the underlying cause for replication performance problems and solving them
● Solving conguration problems during the initial SAP HANA system replication setup
● Solving sporadic network interruptions causing problems in the SAP HANA system replication mechanism
● Avoiding log full situations and recovering from them
● Solving communication problems when they occur between the replication sites
Furthermore, the chapter also provides information on how to conduct a network stress test with NIPING and
gives you an overview of all SAP HANA alerts.
Where can I nd more information?
The following SAP Notes are relevant for troubleshooting system replication:
SAP Notes
SAP Note Title
1930979
Alert: Sync/Async Read Ratio
1969700
Collection of SQL Statements for SAP HANA
1999880
FAQ: SAP HANA System Replication
2166157
Error: 'failed to open channel ! reason: connection refused' when setting up
Disaster Recovery
2083715
Analyzing log volume full situations
1227116
Creating network traces
1679938
Log Volume is full
500235
Network Diagnosis with NIPING
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 209
Related Information
I/O Related Root Causes and Solutions [page 210]
Analyzing I/O Throughput and Latency [page 212]
Savepoint Performance [page 214]
Replication Performance Problems [page 216]
Setup and Initial Conguration Problems [page 221]
Intermittent Connectivity Problems [page 225]
LogReplay: Managing the Size of the Log File [page 227]
SAP HANA System Replication Communication Problems [page 230]
Stress Test with NIPING [page 231]
Reference: Alerts [page 232]
9.1 I/O Related Root Causes and Solutions
This section covers troubleshooting of I/O performance problems. Although SAP HANA is an in-memory
database, I/O still plays a critical role for the performance of the system.
From an end user perspective, an application or the system as a whole runs slowly, is unresponsive or can even
seem to hang if there are issues with I/O performance. In the Disk Volume Monitor available in the Disk Usage
tile in SAP HANA cockpit you can see the attached volumes and which services use which volumes. For details
of the attached volumes, such as les and I/O statistics, select a row.
In certain scenarios data is read from or written to disk, for example during the transaction commit. Most of the
time this is done asynchronously but at certain points in time synchronous I/O is done. Even during
asynchronous I/O it may be that important data structures are locked.
Examples are included in the following table.
Scenario
Description
Savepoint A savepoint ensures that all changed persistent data since the last savepoint gets written to
disk. The SAP HANA database triggers savepoints in 5 minutes intervals by default. Data is
automatically saved from memory to the data volume located on disk. Depending on the
type of data the block sizes vary between 4 KB and 16 MB. Savepoints run asynchronously
to SAP HANA update operations. Database update transactions only wait at the critical
phase of the savepoint, which is usually taking some microseconds.
Snapshot
The SAP HANA database snapshots are used by certain operations like backup and system
copy. They are created by triggering a system wide consistent savepoint. The system keeps
the blocks belonging to the snapshot at least until the drop of the snapshot. Detailed infor
mation about snapshots can be found in the
SAP HANA Administration Guide.
Delta Merge
The delta merge itself takes place in memory. Updates on column store tables are stored in
the delta storage. During the delta merge these changes are applied to the main storage,
where they are stored read optimized and compressed. Right after the delta merge, the new
main storage is persisted in the data volume, that is, written to disk. The delta merge does
not block parallel read and update transactions.
210 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
Scenario Description
Write Transactions All changes to persistent data are captured in the redo log. SAP HANA asynchronously
writes the redo log with I/O orders of 4 KB to 1 MB size into log segments. Transactions writ
ing a commit into the redo log wait until the buer containing the commit has been written
to the log volume.
Database restart
At database startup the services load their persistence including catalog and row store ta
bles into memory, that is, the persistence is read from the storage. Additionally the redo log
entries written after the last savepoint have to be read from the log volume and replayed in
the data area in memory. When this is nished the database is accessible. The bigger the
row store is, the longer it takes until the system is available for operations again.
Failover (Host Auto-Fail
over)
On the standby host the services are running in idle mode. Upon failover, the data and log
volumes of the failed host are automatically assigned to the standby host, which then has
read and write access to the les of the failed active host. Row as well as column store tables
(the latter on demand) must be loaded into memory. The log entries have to be replayed.
Takeover (System Replica
tion)
The secondary system is already running, that is the services are active but cannot accept
SQL and thus are not usable by the application. Just like in the database restart (see above)
the row store tables need to be loaded into memory from persistent storage. If table preload
is used, then most of the column store tables are already in memory. During takeover the
replicated redo logs that were shipped since the last data transport from primary to secon
dary have to be replayed.
Data Backup
For a data backup the current payload of the data volumes is read and copied to the backup
storage. For writing a data backup it is essential that on the I/O connection there are no col
lisions with other transactional operations running against the database.
Log Backup Log backups store the content of a closed log segment. They are automatically and asyn
chronously created by reading the payload from the log segments and writing them to the
backup area.
Database Recovery The restore of a data backup reads the backup content from the backup device and writes it
to the SAP HANA data volumes. The I/O write orders of the data recovery have a size of 64
MB. Also the redo log can be replayed during a database recovery, that is the log backups
are read from the backup device and the log entries get replayed.
In the following table the I/O operations are listed which are executed by the above-mentioned scenarios,
including the block sizes that are read or written:
I/O pattern
Data Volume Log Volume (redo log) Backup Medium
Savepoint,
Snapshot,
Delta merge
WRITE
4 KB – 16 MB asynchronous
bulk writes, up to 64 MB (clus
tered Row Store super blocks)
Write transactions
WRITE
OLTP – mostly 4 KB log write
I/O performance is relevant
OLAP – writes with larger I/O
order sizes
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 211
I/O pattern Data Volume Log Volume (redo log) Backup Medium
Table load:
DB Restart,
Failover,
Takeover
READ
4 KB – 16 MB blocks, up to 64
MB (clustered Row Store super
blocks)
READ
Data Backup
READ
4 KB – 16 MB blocks, up to 64
MB (clustered Row Store super
blocks) are asynchronously
copied to “[data] backup buf
fer” of 512 MB
WRITE
in up to 64 MB blocks from
“[data] backup buer”
Log Backup
READ
asynchronously copied to
“[data] backup buer” of 128
MB
WRITE
in up to 64 MB blocks from
“[data] backup buer”
Database Recovery
WRITE
4 KB – 16 MB blocks, up to 64
MB (clustered Row Store super
blocks)
READ
Read block sizes from backup
le headers and copy blocks
into “[data] backup buer” of
size 512 MB
READ
Read block sizes from backup
le headers and copy blocks
into “[data] backup buer” of
size 128 MB
9.1.1 Analyzing I/O Throughput and Latency
When analyzing I/O, the focus is on throughput and latency (time taken). A set of system views (with names
beginning M_VOLUME_IO_*) is available to help you analyze throughput and examples are given here to
illustrate how they can be used.
You can use the following example query to read I/O statistics data which will help you to analyze the
throughput of the system (in this example the index server). The result of this query presents a set of columns
including throughput in MB and trigger ratios (the relationship between trigger time and I/O time) for both read
and write operations:
select v.host, v.port, v.service_name, s.type,
round(s.total_read_size / 1024 / 1024, 3) as "Reads in MB",
round(s.total_read_size / case s.total_read_time when 0 then -1 else
s.total_read_time end, 3) as "Read Throughput in MB",
round(s.total_read_time / 1000 / 1000, 3) as "Read Time in Sec",
trigger_read_ratio as "Read Ratio",
round(s.total_write_size / 1024 / 1024, 3) as "Writes in MB",
round(s.total_write_size / case s.total_write_time when 0 then -1 else
s.total_write_time end, 3) as "Write Throughput in MB",
round(s.total_write_time / 1000 / 1000, 3) as "Write Time in Sec" ,
trigger_write_ratio as "Write Ratio"
from "PUBLIC"."M_VOLUME_IO_TOTAL_STATISTICS_RESET" s, PUBLIC.M_VOLUMES v
where s.volume_id = v.volume_id
212
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
and type not in ( 'TRACE' )
and v.volume_id in (select volume_id from m_volumes where service_name =
'indexserver')
order by type, service_name, s.volume_id;
Note that some of the system views for I/O can be used with a resettable counter so that you can gather data
for just the most recent period since the counter was set. This example is based on the
M_VOLUME_IO_TOTAL_STATISTICS system view but uses the ‘reset’ version of the view.
You can reset the statistics counter to analyze the I/O throughput for a certain time frame by running the
following reset command:
alter system reset monitoring view M_VOLUME_IO_TOTAL_STATISTICS_RESET;
Multitier and Replication Scenarios
In a system using replication between primary and secondary sites it is possible to analyze throughput of the
secondary remotely by running these queries on the primary site. This method uses the proxy schema of the
secondary system on the primary and can be used in a 2-tier system replication setup as well as for multitier
landscapes.
The proxy schema follows the naming convention _SYS_SR_SITE_<siteName>, where <siteName> is the name
of the secondary site (case-sensitive). In the FROM clause of the example query given above the schema
PUBLIC is used. In a system replication landscape replace this with the proxy schema as shown in the following
example for a secondary with site name 'SiteB':
from "_SYS_SR_SITE_SiteB"."M_VOLUME_IO_TOTAL_STATISTICS_RESET" s,
"_SYS_SR_SITE_SiteB"."M_VOLUMES" v
Trigger Ratios
I/O calls are executed asynchronously, that is, the thread does not wait for the order to return. The trigger-ratio
of asynchronous reads and writes measures the trigger time divided by the I/O time. A ratio close to 0 shows
good performance; it indicates that the thread does not wait at all. A ratio close to 1 means that the thread
waits until the I/O request is completed.
Refer to SAP Note 1930979 and SAP Notes for Alerts 60 and 61 for more information about the signicance of
the trigger ratio values.
Latency
The latency values are important for LOG devices. To analyze the latency, use the following example query
which returns the log write wait time (for data of type LOG) with various buer sizes written by the index server.
The time values returned are the number of microseconds between enqueueing and nishing a request.
select host, port type,
round(max_io_buffer_size / 1024, 3) "Maximum buffer size in KB",
trigger_async_write_count,
avg_trigger_async_write_time as "Avg Trigger Async Write Time in
Microsecond",
max_trigger_async_write_time as "Max Trigger Async Write Time in
Microsecond",
write_count, avg_write_time as "Avg Write Time in Microsecond",
max_write_time as "Max Write Time in Microsecond"
from "PUBLIC"."M_VOLUME_IO_DETAILED_STATISTICS_RESET"
where type = 'LOG'
and volume_id in (select volume_id from m_volumes where service_name =
'indexserver')
and (write_count <> 0 or avg_trigger_async_write_time <> 0);
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 213
Related Information
SAP Note 1930979
M_VOLUME_IO_TOTAL_STATISTICS_RESET System View
Reference: Alerts [page 232]
9.1.2 Savepoint Performance
To perform a savepoint write operation, SAP HANA needs to take a global database lock. This period is called
the “critical phase” of a savepoint. While SAP HANA was designed to keep this time period as short as possible,
poor I/O performance can extend it to a length that causes a considerable performance impact.
Savepoints are used to implement backup and disaster recovery in SAP HANA. If the state of SAP HANA has to
be recovered, the database log from the last savepoint will be replayed.
You can analyze the savepoint performance with this SQL statement:
select start_time, volume_id,
round(duration / 1000000) as "Duration in Seconds",
round(critical_phase_duration / 1000000) as "Critical Phase Duration in
Seconds",
round(total_size / 1024 / 1024) as "Size in MB",
round(total_size / duration) as "Appro. MB/sec",
round (flushed_rowstore_size / 1024 / 1024) as "Row Store Part MB"
from m_savepoints
where volume_id in ( select volume_id from m_volumes where service_name =
'indexserver') ;
This statement shows how long the last and the current savepoint writes took/are taking. Especially the critical
phase duration, in which savepoints need to take a global database lock, must be observed carefully.
The critical phase duration should not be longer than a second. In the example below the times are signicantly
higher due to I/O problems.
Savepoints
The following SQL shows a histogram on the critical phase duration:
select
214
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
to_char(SERVER_TIMESTAMP,'yyyy.mm.dd') as "time",
sum(case when (critical_phase_duration <= 1000000) then 1 else 0
end) as "<= 1 s",
sum(case when (critical_phase_duration > 1000000 and critical_phase_duration
<=2000000) then 1 else 0
end) as "<= 2 s",
sum(case when (critical_phase_duration > 2000000 and critical_phase_duration
<=3000000) then 1 else 0
end) as "<= 3 s",
sum(case when (critical_phase_duration > 3000000 and critical_phase_duration
<=4000000) then 1 else 0
end) as "<= 4 s",
sum(case when (critical_phase_duration > 4000000 and critical_phase_duration
<=5000000) then 1 else 0
end) as "<= 5 s",
sum(case when (critical_phase_duration > 5000000 and critical_phase_duration
<=10000000) then 1 else 0
end) as "<= 10 s",
sum(case when (critical_phase_duration > 10000000 and critical_phase_duration
<=20000000) then 1 else 0
end) as "<= 20 s",
sum(case when (critical_phase_duration > 20000000 and critical_phase_duration
<=40000000) then 1 else 0
end) as "<= 40 s",
sum(case when (critical_phase_duration > 40000000 and critical_phase_duration
<=60000000) then 1 else 0
end) as "<= 60 s",
sum(case when (critical_phase_duration > 60000000 ) then 1 else 0
end) as "> 60 s",
count(critical_phase_duration) as "ALL"
from "_SYS_STATISTICS"."HOST_SAVEPOINTS"
where volume_id in (select volume_id from m_volumes where service_name =
'indexserver')
and weekday (server_timestamp) not in (5, 6)
group by to_char(SERVER_TIMESTAMP,'yyyy.mm.dd')
order by to_char(SERVER_TIMESTAMP,'yyyy.mm.dd') desc;
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 215
Savepoint Histogram
The performance of the backup can be analyzed with this statement:
select mbc.backup_id,
SECONDS_BETWEEN (mbc.sys_start_time, mbc.sys_end_time) seconds,
round(sum(backup_size) / 1024 / 1024 / 1024,2) size_gb,
round(sum(backup_size) / SECONDS_BETWEEN (mbc.sys_start_time, mbc.sys_end_time) /
1024 / 1024, 2) speed_mbs
from m_backup_catalog_files mbcf , m_backup_catalog mbc
where mbc.entry_type_name = 'complete data backup'
and mbc.state_name = 'successful'
and mbcf.backup_id = mbc.backup_id
group by mbc.backup_id, mbc.sys_end_time, mbc.sys_start_time order by
mbc.sys_start_time
9.2 Replication Performance Problems
If system replication appears to slow down transaction processing, you can check the network and disk I/O on
the secondary site.
A slow-down related to system replication can occur in the following scenarios:
● ASYNC replication mode is congured over long distances;
● multi-tier system replication is congured and a tier 3 system is attached;
● SYNC/SYNCMEM replication mode is congured over short distances.
216
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
The following troubleshooting steps can help you determine and resolve the underlying cause.
Check If Log Can Be Shipped in Time
You can check the system replication KPI values to analyze the problem and verify that it is really related to
system replication:
● check if log shipping is signicantly slower than local log write (SYNC/SYNCMEM)
● check Async Buer Full Count (ASYNC)
You can check system replication KPIs in SAP HANA cockpit (see Monitoring SAP HANA System Replication in
the SAP HANA Administration Guide). You can also get an overview of basic system replication KPIs by running
the query HANA_Replication_SystemReplication_Overview_*_MDC.txt (from SAP KBA 1969700). This query is
based on the system view M_SERVICE_REPLICATION and can be used to compare log shipping time to local
log write time. For synchronous replication the following KPIs are shown:
Example output of SQL Statement HANA_Replication_SystemReplication_Overview.txt
KEY VALUE
Replication mode SYNC
Secondary connect time 2016/10/18 14:02:36
Days since secondary connect time 1.26
Used persistence size (GB) 205.76
Log backup size / day (GB) 83.58
Local log buer write size (MB) 101130.54
Shipped log buer size (MB) 100258.52
Avg. local log buer write size (KB) 6.14
Avg. shipped log buer size (KB) 6.14
Avg. local log buer write time (ms) 0.13
Avg. log buer shipping time (ms) 0.24
Local log buer write throughput (MB/s) 44.68
Log buer shipping throughput (MB/s) 24.99
Initial data shipping size (MB) 0.00
Initial data shipping time (s) 0.00
Last delta data shipping size (MB) 2736.00
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 217
KEY VALUE
Last delta data shipping time (s) 13.00
Delta data shipping size (MB) 758704.00
Delta data shipping time (s) 3538.20
Delta data shipping throughput (MB/s) 214.43
Delta data shipping size / day (MB) n/a
Replication delay (s) 0.00
The following KPIs are of particular importance, the shipping time should not be signicantly higher than the
local write time:
● Avg. local log buer write time (ms)
● Avg. log buer shipping time (ms)
You can see a graphical comparison of these local and shipped values in the cockpit System Replication
Overview (Network tab). The graph displayed compares the local write wait time with the remote write wait
time monitored over the last 24 hours:
For asynchronous replication scenarios the redo log is written into an Asynchronous Log Buer, which
occasionally can run full in case the logs are not shipped in a timely manner to the secondary instance. This can
lead to a performance overhead on the primary site as by default it waits with new COMMITS until there is free
space in the buer. This can be avoided by setting the parameter
logshipping_async_wait_on_buffer_full in the system_replication section of the global.ini le to
FALSE.
218
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
Note
In order to maintain a stable connection during the initial data shipment this parameter should be set to
true. This is recommended because if the log shipping connection is reset for any reason the data shipment
connection is also reset and the initial data shipment has to start again from the beginning. In multi-tier
scenarios a restarted full data shipment from primary to secondary site also results in a completely new full
data shipment to a tertiary site. For the duration of the initial shipment, therefore, you may also increase
the value of the logshipping_timeout parameter on the primary which has a default value of 30
seconds.
The size of the asynchronous log shipping buer on the primary site is normally adequate; the default value of
the logshipping_async_buffer_size parameter is 256MB for the indexserver (in the indexserver.ini) and
64MB for all other services (maintained in the global.ini). However, if additional free memory is available this
value can also be increased for specic services with a high log generation (such as the indexserver). You
should make such changes only in the servicespecic ini les rather than in the global.ini le.
Once the asynchronous replication connection is established you can see how much is in the async buer by
checking the value of BACKLOG_SIZE in the system view M_SERVICE_REPLICATION. If there is no connection
this column shows the number of log entries that have been generated on the primary but which have not yet
reached the secondary. You can also see this information (the backlogSize) by running the following command
on the primary with admin rights:
hdbcons ‘replication info’
- backlogSize : 2781184 bytes
For further details of the most common performance issues caused by system replication and under which
circumstances they occur, please refer to SAP KBA 1999880 - FAQ: SAP HANA System Replication.
Check If Data Load Can Be Handled by Network Link
To estimate the required bandwidth for Data/Log shipping, use
HANA_Replication_SystemReplication_Bandwidth.txt (from SAP Note 1069700), which is based on the I/O
statistics from the primary site. We recommend executing this SQL statement when system replication is
disabled. The data returned will help you to estimate the amount of data/log shipped from the primary site and
compare this to the available bandwidth.
You can also do a network performance test using, for example, the open source IPERF tool or similar, to
measure the real application network performance. The recommended bandwidth is 10 Gbit/s.
If the network bandwidth is not adequate you can activate data and log compression which signicantly
reduces the shipment size by setting the following parameters in the system_replication section of
global.ini:
● enable_log_compression = TRUE
● enable_data_compression = TRUE
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 219
Check Network Conguration (Long Distance)
Increasing the TCP window size can result in better network utilization and higher throughput. If the bandwidth
can handle load, check if the network is shared and whether other applications may be interfering with
performance. Collect network information on bandwidth and latency from the Linux kernel parameters as
described here. For these values refer also to SAP Note 2382421 - Optimizing the Network Conguration on
HANA and OS-Level (Linux Kernel Parameters) :
● Check the network utilization prole for the network link to see if the maximum capacity of the network has
been reached.
● If the network is not fully utilized, check the linux kernel TCP conguration with sysctl –a | egrep
“net.core|net.ipv4.tcp” .
● Check that window scaling is set to the default value of 1. net.ipv4.tcp_window_scaling = 1.
● Check whether the max size can be increased for net.ipv4.tcp_wmem and net.ipv4.tcp_rmem.
● Calculate the Bandwidth Delay Product (BDP): Bandwidth * Latency (for example, BDP = 50ms * 3 Gbps =
19.2 MB). The BDP tells you what TCP window size is needed to use the network link fully.
Check Disk I/O on a Secondary Site
Slow disk I/O on the secondary can postpone releasing log buers on the primary, which results in wait
situations on the primary. You can do the following:
● Use a Disk Performance Test Tool
Execute fsperf on log volume, for example:
$ fsperf /usr/sap/TST/SYS/global/hdb/log/mnt00001/hdb00002
● Check the Monitoring and Administration area
If SQL is not available, use command line tools (this has to be done for each individual service), for
example:
$ hdbcons "statreg print -n M_VOLUME_IO_TOTAL_STATISTICS -h"
A runtime dump also contains I/O statistics, which you can see with: $ hdbcons “runtimedump dump”.
Caution
Technical expertise is required to use hdbcons. To avoid incorrect usage, use hdbcons only with the
guidance of SAP HANA development support.
● Check I/O relevant tables in the proxy schema of the corresponding secondary site, which provide SQL
access on the primary site on statistic views of the secondary. For more information, see Monitoring
Secondary Sites in the SAP HANA Administration Guide.
Related Information
SAP Note 1969700
SAP Note 1999880
SAP Note 2382421
220
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
Monitoring Secondary Systems [page 155]
Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 175]
9.3 Setup and Initial Conguration Problems
This section outlines the analysis steps you need to take in case you face conguration problems during the
initial HANA System Replication Setup.
The initial SAP HANA System Replication Setup steps are as follows:
● enabling the SAP HANA System Replication on the primary site with sr_enable
● registering the secondary system with sr_register
While there are no errors to be expected when you enable the primary site, the registration operation on the
secondary site can fail due to various errors.
Note
If you are in the process of setting up HANA System Replication for the rst time, please make sure you
have met all the prerequisites and performed the necessary preparation steps, outlined in the SAP HANA
Administration Guide.
Pay special attention to the following points:
● Are the primary and secondary sites architecturally identical?
● Are the network interface congurations identical on both sites? (refer to the SCN document How to
Congure Network Settings for HANA System Replication for details).
● Are the ports needed for system replication open and reachable from the primary and the secondary
site?
Wrong Topology Information
Upon registering the secondary site, the following error is raised:
Output Code
> hdbnsutil -sr_register
--remoteHost=primary_host --remoteInstance=<primary_instance_no> --
mode=<replication_mode>
--name=<logical_site_name>
adding site ...
checking for inactive nameserver ...
nameserver primary_host:3xx01 not responding.
collecting information ...
error: source system and target system have overlapping logical hostnames;
each site must have a unique set of logical hostnames.
hdbrename can be used to change names;
failed. trace file nameserver_primary_host.00000.000.trc may contain more
error details.
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 221
The root cause for those issues is usually a wrong topology information. In this case, the secondary site
contained the following landscape denition in the nameserver.ini:
Sample Code
[landscape]
id = <id>
master = <secondary_host>:3xx01
worker = <primary_host>
active_master = <secondary_host>:3xx01
roles_<primary_host> = worker
The worker property contained the hostname of the primary site, which was wrong. Therefore, the registration
failed. The problem should disappear once the correct hosts are maintained in the master and worker (if any)
properties. You need to check on both sites if the information maintained in the nameserver topology is
consistent.
Resyncing the Secondary: Persistence Compatibility Checks
If the primary and secondary systems are disconnected for any reason, they must be resynced. If the
persistencies (that is, the data and log volume snapshots) of the primary and secondary are compatible, it is
possible to achieve a resync with only a delta data shipment or a log shipment; in this case full data shipping is
not necessary. Even if the data snapshots are not compatible, the system will automatically attempt a full data
shipment (Resync Optimization). If necessary, a full data shipment can be triggered manually using the
following command:
hdbnsutil -sr_register --force_full_replica
Trace messages related to persistence which indicate that this is necessary include the following:
● Secondary persistence is not compatible with primary persistence.
● The persistence of at least one service is not initialized correctly.
Communication Problems with the Primary Site
The sr_register command on the secondary site is failing with:
Output Code
> hdbnsutil -sr_register --name=<logical_site_name> --
remoteHost=<primary_host --remoteInstance=<primary_instance_no> --
mode=<replication_mode> --force_full_replica --sapcontrol=1
unable to contact primary site host <primary_host>:3xx02. connection
refused,location=<primary_host>:3xx02
Possible Root Cause 1: Ports Not Open / Blocked by Firewall
This error usually indicates a general communication problem between the primary and secondary site. Mostly,
this is caused by the primary host not listening on the required ports for various reasons. You can check
222
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
whether the required ports 3<instance_number>01 and 3<instance_number>02 (non-MDC scenarios) or
4<instance_number>02 (MDC scenarios) are listening on the required interfaces with the following
command on OS level as privileged user (for example, root):
>netstat –apn | grep 3<instance_no>02
>netstat –apn | grep 4<instance_no>02
If you see that these ports are open and listening on the localhost interface only, you will not be able to reach
them from the secondary site. You need to adjust the settings for listeninterface in the global.ini le
from .local to .global:
Sample Code
[communication]
listeninterface=.global
With this setting, the following interface:port pairs should be visible in netstat:
Note
If the ports are open, check whether they are not ltered by your rewall. Often it is not sucient to check
the connectivity to remote hosts via ping, because ping uses the ICMP protocol for communication. You
can easily verify the accessibility of remote hosts by issuing a telnet call. For example:
>telnet <primary_host> 30001
>telnet <primary_host> 30102
Possible Root Cause 2: SSL-Related Problems
Another cause for this error could be a wrongly implemented SSL conguration.
Note
If you do not secure the HANA network with SSL, do not implement any parameter changes related to SSL.
This can be revealed by activated corresponding traces on the primary site via SAP HANA cockpit:
● Database Explorer Trace Conguration Database Trace Search for “sr_nameserver” Change from
INFO to DEBUG
OK
● Database Explorer Trace Conguration Database Trace Search for “trexnet” Change from ERROR
to INFO
OK
Alternatively, the traces can be activated in the SQL console by issuing the following statements as a SYSTEM
user:
Source Code
alter system alter configuration ('indexserver.ini','SYSTEM') SET
('trace','sr_nameserver')='debug' with reconfigure;
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 223
alter system alter configuration ('indexserver.ini','SYSTEM') SET
('trace','trexnet')='info' with reconfigure;
After the trace activation, the registration problem needs to be reproduced by re-running the sr_register
command on the secondary. The nameserver trace on the primary site would reveal the following errors in the
CommonCrypto Engine:
Output Code
Crypto/SSL/CommonCrypto/Engine.cpp:563: SSL handshake failed: SSL error
[536871970]: Unknown error, General error: 0x20000422 | SAPCRYPTOLIB |
SSL_accept
SSL API error
Version in SSLPlaintext.version field of currently received record differs
from
the one negotiated in the current or currently accomplished handshake.
0xa060023c | SSL | ssl3_accept
Version in SSLPlaintext.version field of currently received record differs
from
the one negotiated in the current or currently accomplished handshake.
0xa060023c | SSL | ssl3_get_record
Version in SSLPlaintext.version field of currently received record differs
from
the one negotiated in the current or currently accomplished handshake.
(ErrCode: 536871970)
Make sure the following parameters are consistent on both sites in the conguration le global.ini:
Sample Code
[communication]
ssl = systempki
..
...
[system_replication_communication]
enable_ssl = on
You need to ensure that the SSFS key and data les are stored on both sites. The following les must exist on
both sites:
$DIR_INSTANCE/../global/security/rsecssfs/data/SSFS_<SID>.DAT
$DIR_INSTANCE/../global/security/rsecssfs/data/SSFS_<SID>.KEY
Possible Root Cause 3: Wrong Conguration of the Internal Hostname Resolution
Parameters
Please check whether the internal hostname resolution information is consistent on both sites. The following
how-to guides are a good source of information:
● How to Congure Network Settings for SAP HANA System Replication
● How to Perform System Replication for SAP HANA.
224
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
Possible Root-Cause 4: Wrong MTU Size Congured
A closer look at the nameserver trace le on the secondary site would reveal:
Output Code
error: unable to contact primary site; to <primary_host_ip> (<primary_host>):
3xx01; original error: timeout occured,location=<primary_host_ip> :3xx02. Was
MTU size set to 1500? (https://css.wdf.sap.corp/sap/support/notes/2142892);
This problem is discussed in full detail in SAP Note 2166157 - Error: 'failed to open channel ! reason: connection
refused' when setting up Disaster Recovery.
Possible Root Cause 5: HANA Service Unavailability
Check the availability of the indexserver / nameserver process on the primary site. Often the services faced an
intermittent restart, crash or reconguration which did not go unrecognized by the secondary site.
Related Information
SAP HANA Administration Guide
Congure Tracing in the SAP HANA Database Explorer
How to Perform System Replication for SAP HANA
How to Congure Network Settings for SAP HANA System Replication
SAP Note 2166157
9.4 Intermittent Connectivity Problems
This section discusses the mitigation strategies for sporadic network interruptions causing problems in the
SAP HANA System Replication mechanism.
A common intermittent error is that the log buer is not shipped in a timely fashion from the primary to the
secondary site.
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 225
Log Shipping Timeout
Possible Root Cause 1: Log Area Is Full on the Secondary Site
If the System Replication Mode is set to SYNC – full sync, the commits on primary are halted as nothing
can be written to the log area on secondary site any longer. On the secondary site, the trace les contain the
following error:
Output Code
i EventHandler LocalFileCallback.cpp(00455) : [DISKFULL] (1st request) [W] ,
buffer= 0x00007f7eef8ae000, offset= 589299712, size= 0/524288, file= "<root>/
logsegment_000_00000508.dat
" ((open, mode= RW, access= rw-rw-r--, flags= DIRECT|LAZY_OPEN), factory=
(root= "/hana/log/<SID>/mnt00001/hdb00003/" (access= rw-rw-r--, flags=
AUTOCREATE_DIRECTORY, usage= LOG, fs= xfs,
config=
(async_write_submit_active=auto,async_write_submit_blocks=new,async_read_submi
t=off,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max
_parallel_io_requests=64,
min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0,
fullRetries= 0 (0/10)}
To quickly mitigate the situation, you can disable the “full sync” option by running the following command:
>hdbnsutil -sr_fullsync --disable
Afterwards, the log area on the secondary site needs to be analyzed with regard to why the log segments are
not freed up. This is usually caused by an erroneous log backup mechanism.
For further details refer to the following SAP Notes:
SAP Note 2083715 - Analyzing log volume full situations
SAP Note 1679938 - Log Volume is full
Possible Root Cause 2: Sporadic Communication Issues on the Network Layer
For more information about how to deal with communication problems between the primary and the
secondary site, see SAP HANA System Replication Communication Problems.
Related Information
SAP Note 2083715
SAP Note 1679938
SAP HANA System Replication Communication Problems [page 230]
226
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
9.5 LogReplay: Managing the Size of the Log File
There is a risk in replication scenarios which use one of the logreplay operation modes of causing a disk full
situation on the primary if the secondary system is not available for any reason; this can potentially lead to a
complete freeze of the database.
The logreplay modes (logreplay introduced in HANA 1.0 SPS10 and logreplay_readaccess introduced in HANA
2) require a log history on the primary so that a secondary system can be resynchronized without the need for
a full data shipment. As long as a secondary system is registered the log le will continue to grow. When the
secondary system synchronizes, then the log is automatically cleared down. However, if the replication
environment changes, if for example, the secondary is separated because of network problems, manual
intervention may be required to manage the log le or, in the worst case scenario, to recover from a disk full
situation. This problem can also happen on a secondary system where a takeover has occurred.
The log replay modes are described in detail in the SAP HANA Administration Guide section System Replication
With Operation Mode Logreplay. This section of the SAP HANA Troubleshooting and Performance Analysis
Guide describes procedures to rstly prevent problems from occurring and secondly to resolve a disk full
situation.
Log File Retention (RetainedFree Status)
If the secondary cannot be synchronized for any reason, then log segments continue to be written but are
marked as RetainedFree. You can check for RetainedFree log segments either in SAP HANA cockpit or from the
command line.
1) To check using SAP HANA cockpit, start from the Disk Usage app and open Monitor Disk Volume. The
graph shows usage for log and data volumes; you can lter the display for a specic volume and server (for
example indexserver). Check the State column of the log les for RetainedFree log segments as shown here:
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 227
2) To check using the command line, execute the following command as <sid>adm for a specic log volume
(hdb00003 in this example – the log volume of one indexserver):
#>hdblogdiag seglist $DIR_INSTANCE/../SYS/global/hdb/log/mnt00001/hdb00003
The result shows details of each log segment including status information. Look for any segments with status
RetainedFree as shown here:
LogSegment[0/2:0xec98740-0xecb6000(0x763000B)/
GUID=759DC14B-00D7-20161122-134436-39A00002ED/
PrevGUID=759DC14B-00D7-20161122-134436-39A00002EC,TS=2016-11-30
06:55:18.008741,Hole=0xec98740/
RetainedFree/0x0]@0x00007f34cb32a010
How to Avoid Log Full Situations
Unregister an Unused Secondary
If the secondary is disconnected for a prolonged period and if it is not to be used as a replication server
anymore, then unregister the secondary site and disable the primary functionality. This will stop the
RetainedFree log entries from being written:
1. Unregister the secondary; this is normally done from the secondary site but can be done from the primary
if the secondary is not available anymore:
hdbnsutil –sr_unregister
2. Disable the primary (from the primary site):
hdbnsutil –sr_disable
3. Execute recongure on the primary site:
hdbnsutil –reconfig
You can use this same procedure for a primary which, after a takeover, will no longer be used for failback.
Set a Maximum Retention Size
Another option to manage the log size is to set a value for the logshipping_max_retention_size
parameter. If the log size reaches this limit, then RetainedFree log entries will be overwritten. Note the following
points in this case:
● If any RetainedFree log entries are lost, then synchronization by logreplay will no longer be possible and a
full data shipment will be necessary to resynchronize the secondary.
● It is not possible to switch back to delta mode to resynchronize - only a full data shipping is possible.
Tip
As a further general precaution, to prevent any disk full situation from arising you can reserve a portion of
the disk with an emergency placeholder le (containing any dummy values), for example, occupying 5 –
10 % of the le system. This le can then be deleted if ever necessary to quickly solve disk full situations.
228
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
How to Recover From Log Full Situations
Secondary Has Been Taken out of Service
If the secondary has been permanently taken out of service, then these log entries will never be required. In this
case the secondary can be unregistered and the log volume cleaned up:
1. Unregister the secondary (same steps as previous subsection: unregister, disable and recongure).
2. Delete the Free marked log segments from the command line for each of the persistent relevant services
(nameserver, indexserver, xsengine, …). To do this run hdbcons with the log release parameter as
<sid>adm. In a multi-database system the -p switch is required with the process ID of the service (such as
indexserver):
hdbcons –p <PID_of_service> “log release”
Secondary Still Required
If the secondary is still required, then restart it and allow it to resynchronize. When this has completed, the
RetainedFree log segments on the primary will be marked as Free, you can then clean up the log as described
above by running hdbcons with the log release parameter.
Log Full Has Caused Complete Database Freeze
If the log full has caused a complete database freeze, you can try to move the log to another linked le system
and replay the log from there. Essentially, this is a three step procedure, refer to SAP Note 1679938 Log Volume
is full for complete details:
● Stop the primary system.
● Mount the log volumes of the primary via symbolic link to another le system.
● Start the primary and the secondary and allow them to resynchronize.
When this has completed, you can then clean up the log by running hdbcons with the log release parameter
as described above.
Related Information
SAP Note 1679938
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 229
9.6 SAP HANA System Replication Communication
Problems
Problems during initial setup of the system replication can be caused by incorrect conguration, incorrect
hostname resolution or wrong denition of the network to be used for the communication between the
replication sites.
Context
System replication environments depend on network bandwidth and stability. In case communication
problems occur between the replication sites (for example, between SITE A and SITE B), the rst indication of a
faulty system replication setup will arise.
Procedure
1. Check the nameserver traceles.
The starting point for the troubleshooting activity are the nameserver traceles. The most common errors
found are:
Sample Code
e TNS TNSClient.cpp(00800) : sendRequest dr_secondaryactivestatus to
<hostname>:<system_replication_port> failed with NetException.
data=(S)host=<hostname>|service=<service_name>|(I)drsender=2|
e sr_nameserver TNSClient.cpp(06787) : error when sending request
'dr_secondaryactivestatus' to <hostname>:<system_replication_port>:
connection broken,location=<hostname>:<system_replication_port>
e TrexNetBuffer BufferedIO.cpp(01151) : erroneous channel ### from #####
to <hostname>:<system_replication_port>: read from channel failed;
resetting buffer
Further errors received from the remote side:
Sample Code
Generic stream error: getsockopt, Event=EPOLLERR - , rc=104: Connection
reset by peer
Generic stream error: getsockopt, Event=EPOLLERR - , rc=110: Connection
timed out
It is important to understand that if those errors suddenly occur in a working system replication
environment, they are often indicators of problems on the network layer. From an SAP HANA perspective,
there is nothing that could be toggled, as it requires further analysis by a network expert. The investigation,
in this case, needs to focus on the TCP trac by recording a tcpdump in order to get a rough
understanding how TCP retransmissions, out-of-order packets or lost packets are contributing to the
overall network trac. How a tcpdump is recorded is described in SAP Note 1227116 - Creating network
230
P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
traces. As these errors are not generated by the SAP HANA server, please consider consulting your in-
house network experts or your hardware vendor before engaging with SAP Product Support.
2. Set the parameter sr_dataaccess to debug.
In the DB Administration area of the SAP HANA cockpit open the Conguration of System Properties
monitor. In the [trace] section of the indexserver.ini le set the parameter sr_dataaccess = debug.
This parameter enables a more detailed trace of the components involved in the system replication
mechanisms. For more information about how to change parameters, see
Memory Information from Logs
and Traces.
Related Information
SAP Note 1227116
Memory Information from Logs and Traces
9.7 Stress Test with NIPING
The SAP NIPING tool is a powerful tool which can be used to perform specic network stability tests.
Prerequisites
You must have OS level access to the SAP HANA host and the client host.
Procedure
Read SAP Note 500235 - Network Diagnosis with NIPING.
A stress test with SAP's NIPING tool may be performed in order to conrm the high network latency (or
bandwidth exhaustion).
Related Information
SAP Note 500235
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 231
9.8 Reference: Alerts
This reference section gives details of all alerts and includes links to any related SAP Notes.
All alerts include a recommended user action and for many alerts a corresponding SAP Note is available. In
column 4 of the following table ('Conguration') an asterisk '*' identies alerts which have multiple
congurable severity thresholds, Info-only alerts are agged by 'Info', and those with only a single congurable
value with '1'.
Alerts
ID Name Description Cfg User Action Category
Further Infor
mation
0 Internal statis
tics server prob
lem
Identies internal statistics
server problem.
Resolve the problem. For
more information, see the
trace les. You may need to
activate tracing rst.
Availability
SAP Note:
1803039
1 Host physical
memory usage
Determines what percentage
of total physical memory
available on the host is used.
All processes consuming
memory are considered, in
cluding non-SAP HANA proc
esses.
*
Investigate memory usage of
processes.
Note
Only relevant in HANA
1.0. Not active in HANA
2.0. See SAP Note:
2757696 Alert 1
shows wrong information
Memory
SAP Note:
1898317 SAP
Note:1840954
2 Disk usage Determines what percentage
of each disk containing data,
log, and trace les is used.
This includes space used by
non-SAP HANA les.
*
Investigate disk usage of
processes. Increase disk
space, for example by shrink
ing volumes, deleting diagno
sis les, or adding additional
storage.
Disk
SAP Note:
1900643
3 Inactive services Identies inactive services. 1 Investigate why the service is
inactive, for example, by
checking the service's trace
les.
Availability
Inactive > 600
seconds. SAP
Note:
1902033
4 Restarted serv
ices
Identies services that have
restarted since the last time
the check was performed.
Investigate why the service
had to restart or be re
started, for example, by
checking the service's trace
les.
Availability
SAP Note:
1909660
5 Host CPU Usage Determines the percentage
CPU idle time on the host
and therefore whether or not
CPU resources are running
low.
*
Investigate CPU usage. CPU SAP Note:
1909670
232 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
10 Delta merge
(mergedog) con
guration
Determines whether or not
the 'active' parameter in the
'mergedog' section of system
conguration le(s) is 'yes'.
mergedog is the system
process that periodically
checks column tables to de
termine whether or not a
delta merge operation needs
to be executed.
Change in SYSTEM layer the
parameter active in sec
tion(s) mergedog to yes
Congura
tion
SAP Note:
1909641
12 Memory usage
of name server
Determines what percentage
of allocated shared memory
is being used by the name
server on a host.
*
Increase the shared memory
size of the name server. In
the 'topology' section of the
nameserver.ini le, increase
the value of the 'size' param
eter.
Memory
SAP Note:
1977101
16 Lock wait time
out congura
tion
Determines whether the
'lock_wait_timeout' parame
ter in the 'transaction' sec
tion of the indexserver.ini le
is between 100,000 and
7,200,000.
In the 'transaction' section of
the indexserver.ini le, set
the 'lock_wait_timeout' pa
rameter to a value between
100,000 and 7,200,000 for
the System layer.
Congura
tion
SAP Note:
1909707
17 Record count of
non-partitioned
column-store ta
bles
Determines the number of
records in non-partitioned
column-store tables. Current
table size is not critical. Parti
tioning need only be consid
ered if tables are expected to
grow rapidly (a non-parti
tioned table cannot contain
more than 2,147,483,648 (2
billion) rows).
Info
Consider partitioning the ta
ble only if you expect it to
grow rapidly.
Memory SAP HANA Ad
ministration
Guide > Table
Partitioning, SAP
Note:
1909763
20
Table growth of
non-partitioned
column-store ta
bles
Determines the growth rate
of non-partitioned columns
tables.
*
Consider partitioning the ta
ble. See also: Guided Answer:
How to reduce the size of a
table.
Memory
SAP HANA Ad
ministration
Guide > Table
Partitioning, SAP
Note:
1910140
21
Internal event Identies internal database
events.
* Resolve the event and then
mark it as resolved by exe
cuting the SQL statement
ALTER SYSTEM SET EVENT
HANDLED '<host>:<port>'
<id>. Note that this is not
necessary for INFO events.
Availability
SAP Note:
1977252
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 233
ID Name Description Cfg User Action Category
Further Infor
mation
22 Notication of all
alerts
Determines whether or not
there have been any alerts
since the last check and if so,
sends a summary e-mail to
specied recipients.
Investigate the alerts.
Availability
23 Notication of
medium and
high priority
alerts
Determines whether or not
there have been any medium
and high priority alerts since
the last check and if so,
sends a summary e-mail to
specied recipients.
Investigate the alerts.
Availability
24 Notication of
high priority
alerts
Determines whether or not
there have been any high pri
ority alerts since the last
check and if so, sends a sum
mary e-mail to specied re
cipients.
Investigate the alerts.
Availability
25 Open connec
tions
Determines what percentage
of the maximum number of
permitted SQL connections
are open. The maximum
number of permitted con
nections is congured in the
"session" section of the in
dexserver.ini le.
*
Investigate why the maxi
mum number of permitted
open connections is being
approached.
Sessions/
Transac
tions
SAP Note:
1910159
26 Unassigned vol
umes
Identies volumes that are
not assigned a service.
Investigate why the volume is
not assigned a service. For
example, the assigned serv
ice is not active, the removal
of a host failed, or the re
moval of a service was per
formed incorrectly.
Congura
tion
SAP Note:
1910169
27 Record count of
column-store ta
ble partitions
Determines the number of
records in the partitions of
column-store tables. A table
partition cannot contain
more than 2,000,000,000 (2
billion) rows.
*
Consider repartitioning the
table.
Memory SAP HANA Ad
ministration
Guide > Table
Partitioning, SAP
Note:
1910188
28
Most recent sa
vepoint opera
tion
Determines how long ago the
last savepoint was dened,
that is, how long ago a com
plete, consistent image of the
database was persisted to
disk.
*
Investigate why there was a
delay dening the last save
point and consider triggering
the operation manually by
executing the SQL statement
ALTER SYSTEM SAVEPOINT.
Disk
SAP Note:
1977291
29 Size of delta
storage of col
umn-store ta
bles
Determines the size of the
delta storage of column ta
bles.
* Investigate the delta merge
history in the monitoring
view M_DELTA_MERGE_STA
TISTICS. Consider merging
the table delta manually.
Memory
SAP Note:
1977314
234 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
30 Check internal
disk full event
Determines whether or not
the disks to which data and
log les are written are full. A
disk-full event causes your
database to stop and must
be resolved.
*
Resolve the disk-full event as
follows: In the Administration
Editor on the Overview tab,
choose the "Disk Full Events"
link and mark the event as
handled. Alternatively, exe
cute the SQL statements AL
TER SYSTEM SET EVENT
ACKNOWLEDGED
'<host>:<port>' <id> and AL
TER SYSTEM SET EVENT
HANDLED '<host>:<port>'
<id>.
Disk
SAP Note:
1898460
31 License expiry Determines how many days
until your license expires.
Once your license expires,
you can no longer use the
system, except to install a
new license.
*
Obtain a valid license and in
stall it. For the exact expira
tion date, see the monitoring
view M_LICENSE.
Availability
Security, Author
ization and Li
censing, SAP
Note:
1899480
32
Log mode LEG
ACY
Determines whether or not
the database is running in log
mode "legacy". Log mode
"legacy" does not support
point-in-recovery and is not
recommended for productive
systems.
If you need point-in-time re
covery, recongure the log
mode of your system to "nor
mal". In the "persistence"
section of the global.ini con
guration le, set the param
eter "log_mode" to "normal"
for the System layer. When
you change the log mode,
you must restart the data
base system to activate the
changes. It is also recom
mended that you perform a
full data backup.
Backup
Conguration
Parameter Is
sues. SAP Note:
1900296
33
Log mode OVER
WRITE
Determines whether or not
the database is running in log
mode "overwrite". Log mode
"overwrite" does not support
point-in-recovery (only recov
ery to a data backup) and is
not recommended for pro
ductive systems.
If you need point-in-time re
covery, recongure the log
mode of your system to "nor
mal". In the "persistence"
section of the global.ini con
guration le, set the param
eter "log_mode" to "normal"
for the System layer. When
you change the log mode,
you must restart the data
base system to activate the
changes. It is also recom
mended that you perform a
full data backup.
Backup
SAP HANA Ad
ministration
Guide > Backing
up and Recover
ing the SAP
HANA Database.
SAP Note:
1900267
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 235
ID Name Description Cfg User Action Category
Further Infor
mation
34 Unavailable vol
umes
Determines whether or not
all volumes are available.
Investigate why the volume is
not available.
Congura
tion
SAP HANA Ad
ministration
Guide > Backing
up and Recover
ing the SAP
HANA Database,
SAP Note:
1900682
35
Existence of
data backup
Determines whether or not a
data backup exists. Without a
data backup, your database
cannot be recovered.
Perform a data backup as
soon as possible.
Backup SAP HANA Ad
ministration
Guide > Backing
up and Recover
ing the SAP
HANA Database,
SAP Note:
1900728
36
Status of most
recent data
backup
Determines whether or not
the most recent data backup
was successful.
Investigate why the last data
backup failed, resolve the
problem, and perform a new
data backup as soon as pos
sible.
Backup
SAP HANA Ad
ministration
Guide > Backing
up and Recover
ing the SAP
HANA Database,
SAP Note:
1900795
37
Age of most re
cent data
backup
Determines the age of the
most recent successful data
backup.
* Perform a data backup as
soon as possible.
Backup SAP HANA Ad
ministration
Guide > Backing
up and Recover
ing the SAP
HANA Database,
SAP Note:
1900730
38
Status of most
recent log back
ups
Determines whether or not
the most recent log backups
for services and volumes
were successful.
Investigate why the log
backup failed and resolve the
problem.
Backup SAP HANA Ad
ministration
Guide > Backing
up and Recover
ing the SAP
HANA Database,
SAP Note:
1900788
39
Long-running
statements
Identies long-running SQL
statements.
* Investigate the statement.
For more information, see
the table _SYS_STATIS
TICS.HOST_LONG_RUN
NING_STATEMENTS.
Sessions/
Transac
tions
SAP Note:
1849392
236 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
40 Total memory
usage of col
umn-store ta
bles
Determines what percentage
of the eective allocation
limit is being consumed by
individual column-store ta
bles as a whole (that is, the
cumulative size of all of a ta
ble's columns and internal
structures)
*
Consider partitioning or re
partitioning the table.
Memory SAP Note:
1977268
41 In-memory Da
taStore activa
tion
Determines whether or not
there is a problem with the
activation of an in-memory
DataStore object.
For more information, see
the table _SYS_STATIS
TICS.GLOBAL_DEC_EX
TRACTOR_STATUS and SAP
Note 1665553.
Availability
SAP Note:
1665553 SAP
Note:1977230
42 Long-running /
idling cursors.
Identies long-running /
idling cursors.
* Close the cursor in the appli
cation, or kill the connection
by executing the SQL state
ment ALTER SYSTEM DIS
CONNECT SESSION <LOGI
CAL_CONNECTION_ID>. For
more information, see the ta
ble HOST_LONG_IDLE_CUR
SOR (_SYS_STATISTICS).
Sessions/
Transac
tions
SAP Note:
1900261
43 Memory usage
of services
Determines what percentage
of its eective allocation limit
a service is using.
* Check for services that con
sume a lot of memory.
Memory SAP Note:
1900257
44 Licensed mem
ory usage
Determines what percentage
of licensed memory is used.
Info Increase licensed amount of
main memory. You can see
the peak memory allocation
since installation in the sys
tem view M_LICENSE (col
umn PRODUCT_USAGE).
Memory
SAP Note:
1899511
45 Memory usage
of main storage
of column-store
tables
Determines what percentage
of the eective allocation
limit is being consumed by
the main storage of individual
column-store tables.
*
Consider partitioning or re
partitioning the table.
Memory SAP Note:
1977269
46 RTEdump les Identies new runtime dump
les (*rtedump*) have been
generated in the trace direc
tory of the system. These
contain information about,
for example, build, loaded
modules, running threads,
CPU, and so on.
Check the contents of the
dump les.
Diagnosis
Files
SAP Note:
1977099
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 237
ID Name Description Cfg User Action Category
Further Infor
mation
47 Long-running se
rializable trans
actions
Identies long-running serial
izable transactions.
* Close the serializable trans
action in the application or
kill the connection by execut
ing the SQL statement AL
TER SYSTEM DISCONNECT
SESSION <LOGICAL_CON
NECTION_ID>. For more in
formation, see the table
HOST_LONG_SERIALIZA
BLE_TRANSACTION
(_SYS_STATISTICS).
Sessions/
Transac
tions
Transactional
Problems
48 Long-running
uncommitted
write transac
tions
Identies long-running un
committed write transac
tions.
* Close the uncommitted
transaction in the application
or kill the connection by exe
cuting the SQL statement
ALTER SYSTEM DISCON
NECT SESSION <LOGI
CAL_CONNECTION_ID>. For
more information, see the ta
ble HOST_UNCOMMIT
TED_WRITE_TRANSACTION
(_SYS_STATISTICS).
Sessions/
Transac
tions
SAP Note:
1977276
49 Long-running
blocking situa
tions
Identies long-running block
ing situations.
* Investigate the blocking and
blocked transactions and if
appropriate cancel one of
them.
Sessions/
Transac
tions
SAP Note:
2079396
50 Number of diag
nosis les
Determines the number of di
agnosis les written by the
system (excluding ziples).
An unusually large number of
les can indicate a problem
with the database (for exam
ple, problem with trace le
rotation or a high number of
crashes).
1
Investigate the diagnosis
les.
Diagnosis
Files
See KBA
1977162, SAP
Note:
1977162
51 Size of diagnosis
les
Identies large diagnosis
les. Unusually large les can
indicate a problem with the
database.
*
Check the diagnosis les in
the SAP HANA studio for de
tails.
Diagnosis
Files
See KBA
1977208, SAP
Note:
1977208
52 Crashdump les Identies new crashdump
les that have been gener
ated in the trace directory of
the system.
Check the contents of the
dump les.
Diagnosis
Files
SAP Note:
1977218
53 Pagedump les Identies new pagedump
les that have been gener
ated in the trace directory of
the system.
Check the contents of the
dump les.
Diagnosis
Files
SAP Note:
1977242
238 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
54 Savepoint dura
tion
Identies long-running save
point operations.
* Check disk I/O performance. Backup CPU Related
Root Causes and
Solutions, I/O
Related Root
Causes and Sol
utions, SAP
Note:
1977220
55
Columnstore un
loads
Determines how many col
umns in columnstore tables
have been unloaded from
memory. This can indicate
performance issues.
*
Check sizing with respect to
data distribution.
Memory SAP Note:
1977207
56 Python trace ac
tivity
Determines whether or not
the python trace is active and
for how long. The python
trace aects system per
formance.
*
If no longer required, deacti
vate the python trace in the
relevant conguration le.
Diagnosis
Files
SAP Note:
1977098
57 Instance secure
store le system
(SSFS) inacces
sible
Determines if the instance
secure store in the le sys
tem (SSFS) of your SAP
HANA system is accessible
to the database.
Check and make sure that
the instance SSFS is accessi
ble to the database.
Security SAP Note:
1977221
58 Plan cache size Determines whether or not
the plan cache is too small.
Info Currently Alert 58 is inactive
and replaced by Alert 91.
Please activate Alert 91 - Plan
Cache Hit Ratio
Memory
SAP Note:
1977253
59 Percentage of
transactions
blocked
Determines the percentage
of transactions that are
blocked.
* Investigate blocking and
blocked transactions and if
appropriate cancel some of
them.
Sessions/
Transac
tions
SAP Note:
2081856
60 Sync/Async
read ratio
Identies a bad trigger asyn
chronous read ratio. This
means that asynchronous
reads are blocking and be
have almost like synchronous
reads. This might have nega
tive impact on SAP HANA I/O
performance in certain sce
narios.
Info
Please refer to SAP note
1930979.
Disk I/O Related Root
Causes and Sol
utions, SAP
Note:
1965379
61
Sync/Async
write ratio
Identies a bad trigger asyn
chronous write ratio. This
means that asynchronous
writes are blocking and be
have almost like synchronous
writes. This might have nega
tive impact on SAP HANA I/O
performance in certain sce
narios.
Info
Please refer to SAP note
1930979.
Disk I/O Related Root
Causes and Sol
utions, SAP
Note:
1965379
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 239
ID Name Description Cfg User Action Category
Further Infor
mation
62 Expiration of da
tabase user
passwords
Identies database users
whose password is due to ex
pire in line with the cong
ured password policy. If the
password expires, the user
will be locked. If the user in
question is a technical user,
this may impact application
availability. It is recom
mended that you disable the
password lifetime check of
technical users so that their
password never expires (AL
TER USER <username> DIS
ABLE PASSWORD LIFE
TIME).
Change the password of the
database user.
Security SAP Note:
2082406
63 Granting of
SAP_INTER
NAL_HANA_SU
PPORT role
Determines if the internal
support role (SAP_INTER
NAL_HANA_SUPPORT) is
currently granted to any da
tabase users.
1
Check if the corresponding
users still need the role. If
not, revoke the role from
them.
Security
SAP Note:
2081857
64 Total memory
usage of table-
based audit log
Determines what percentage
of the eective memory allo
cation limit is being con
sumed by the database table
used for table-based audit
logging. If this table grows
too large, the availability of
the database could be im
pacted.
*
Consider exporting the con
tent of the table and then
truncating the table.
Memory SAP Note:
2081869
65 Runtime of the
log backups cur
rently running
Determines whether or not
the most recent log backup
terminates in the given time.
* Investigate why the log
backup runs for too long, and
resolve the issue.
Backup SAP HANA Ad
ministration
Guide, SAP
Note:
2081845
66
Storage snap
shot is prepared
Determines whether or not
the period, during which the
database is prepared for a
storage snapshot, exceeds a
given threshold.
*
Investigate why the storage
snapshot was not conrmed
or abandoned, and resolve
the issue.
Backup
SAP HANA Ad
ministration
Guide, SAP
Note:
2081405
67
Table growth of
rowstore tables
Determines the growth rate
of rowstore tables
* Try to reduce the size of row
store table by removing un
used data
Memory SAP Note:
2054411
68 Total memory
usage of row
store
Determines the current
memory size of a row store
used by a service
* Investigate memory usage by
row store tables and consider
cleanup of unused data
Memory SAP Note:
2050579
240 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
69 Enablement of
automatic log
backup
Determines whether auto
matic log backup is enabled.
Enable automatic log backup.
For more details please see
SAP HANA Administration
Guide.
Backup
SAP HANA Ad
ministration
Guide, SAP
Note:
2081360
70
Consistency of
internal system
components af
ter system up
grade
Veries the consistency of
schemas and tables in inter
nal system components (for
example, the repository) af
ter a system upgrade.
Contact SAP support.
Availability
71 Row store frag
mentation
Check for fragmentation of
row store.
Implement SAP Note
1813245.
Memory SAP Note:
1813245
72 Number of log
segments
Determines the number of
log segments in the log vol
ume of each serviceCheck
for number of log segments.
*
Make sure that log backups
are being automatically cre
ated and that there is enough
space available for them.
Check whether the system
has been frequently and un
usually restarting services. If
it has, then resolve the root
cause of this issue and create
log backups as soon as pos
sible.
Backup
75 Rowstore ver
sion space skew
Determines whether the row
store version chain is too
long.
* Identify the connection or
transaction that is blocking
version garbage collection.
You can do this in the SAP
HANA studio by executing
"MVCC Blocker Statement"
and "MVCC Blocker Transac
tion" available on the System
Information tab of the Ad
ministration editor. If possi
ble, kill the blocking connec
tion or cancel the blocking
transaction. For your infor
mation, you can nd table in
formation by using query
"SELECT * FROM TABLES
WHERE TABLE_OID = <table
object ID>".
Memory
Transactional
Problems
76 Discrepancy be
tween host
server times
Identies discrepancies be
tween the server times of
hosts in a scale-out system.
* Check operating system time
settings.
Congura
tion
77 Database disk
usage
Determines the total used
disk space of the database.
All data, logs, traces and
backups are considered.
*
Investigate the disk usage of
the database. See system
view M_DISK_USAGE for
more details.
Disk
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 241
ID Name Description Cfg User Action Category
Further Infor
mation
78 Connection be
tween systems
in system repli
cation setup
Identies closed connections
between the primary system
and a secondary system. If
connections are closed, the
primary system is no longer
being replicated.
Investigate why connections
are closed (for example, net
work problem) and resolve
the issue.
Availability
SAP HANA Ad
ministration
Guide
79 Conguration
consistency of
systems in sys
tem replication
setup
Identies conguration pa
rameters that do not have
the same value on the pri
mary system and a secon
dary system. Most congura
tion parameters should have
the same value on both sys
tems because the secondary
system has to take over in
the event of a disaster.
If the identied conguration
parameter(s) should have
the same value in both sys
tems, adjust the congura
tion. If dierent values are
acceptable, add the parame
ter(s) as an exception in
global.ini/[inile_checker].
Congura
tion
SAP HANA Ad
ministration
Guide
80 Availability of ta
ble replication
Monitors error messages re
lated to table replication.
Determine which tables en
countered the table replica
tion error using system view
M_TABLE_REPLICAS, and
then check the correspond
ing indexserver alert traces.
Availability
81 Cached view size Determines how much mem
ory is occupied by cached
view
* Increase the size of the
cached view. In the "re
sult_cache" section of the in
dexserver.ini le, increase the
value of the "total_size" pa
rameter.
Memory
82 Timezone con
version
Compares SAP HANA inter
nal timezone conversion with
Operating System timezone
conversion.
*
Update SAP HANA internal
timezone tables (refer to SAP
note 1932132).
Congura
tion
SAP Note:
1932132
83 Table consis
tency
Identies the number of er
rors and aected tables de
tected by _SYS_STATIS
TICS.Collector_Global_Ta
ble_Consistency.
*
Contact SAP support. Availability
84 Insecure in
stance SSFS en
cryption cong
uration
Determines whether the
master key of the instance
secure store in the le sys
tem (SSFS) of your SAP
HANA system has been
changed. If the SSFS master
key is not changed after in
stallation, it cannot be guar
anteed that the initial key is
unique.
Change the instance SSFS
master key as soon as possi
ble. For more information,
see the SAP HANA Adminis
tration Guide.
Security
SAP HANA Ad
ministration
Guide
242 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
85 Insecure sys
temPKI SSFS
encryption con
guration
Determines whether the
master key of the secure
store in the le system
(SSFS) of your system's in
ternal public key infrastruc
ture (system PKI) has been
changed. If the SSFS master
key is not changed after in
stallation, it cannot be guar
anteed that the initial key is
unique.
Change the system PKI SSFS
master key as soon as possi
ble. For more information,
see the SAP HANA Adminis
tration Guide.
Security
SAP HANA Ad
ministration
Guide
86 Internal commu
nication is con
gured too
openly
Determines whether the
ports used by SAP HANA for
internal communication are
securely congured. If the
"listeninterface" property in
the "communication" section
of the global.ini le does not
have the value ".local" for sin
gle-host systems and ".all" or
".global" for multiple-host
systems, internal communi
cation channels are exter
nally exploitable.
The parameter [communica
tion] listeninterface in
global.ini is not set to a se
cure value. Please refer to
SAP Note 2183363 or the
section on internal host
name resolution in the SAP
HANA Administration Guide.
Security
SAP Note:
2183363
87 Granting of SAP
HANA DI sup
port privileges
Determines if support privi
leges for the SAP HANA De
ployment Infrastructure (DI)
are currently granted to any
database users or roles.
Check if the corresponding
users still need the privileges.
If not, revoke the privileges
from them.
Security
88 Auto merge for
column-store ta
bles
Determines if the delta
merge of a table was exe
cuted successfully or not.
* The delta merge was not exe
cuted successfully for a ta
ble. Check the error descrip
tion in view
M_DELTA_MERGE_STATIS
TICS and also Indexserver
trace.
Memory
89 Missing volume
les
Determines if there is any
volume le missing.
Volume le missing, data
base instance is broken, stop
immediately all operations
on this instance.
Congura
tion
90 Status of HANA
platform lifecy
cle management
conguration
Determines if the system was
not installed/updated with
the SAP HANA Database
Lifecycle Manager
(HDBLCM).
Install/update SAP HANA
Database Lifecycle Manager
(HDBLCM). Implement SAP
note 2078425
Congura
tion
SAP Note:
2078425
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 243
ID Name Description Cfg User Action Category
Further Infor
mation
91 Plan cache hit
ratio
Determines whether or not
the plan cache hit ratio is too
low.
* Increase the size of the plan
cache. In the "sql" section of
the indexserver.ini le, in
crease the value of the
"plan_cache_size" parame
ter.
Memory
92 Root keys of per
sistent services
are not properly
synchronized
Not al services that persist
data could be reached the
last time the root key change
of the data volume encryp
tion service was changed. As
a result, at least one service
is running with an old root
key.
Trigger a savepoint for this
service or ush the SSFS
cache using hdbcons
Security
93 Streaming Li
cense expiry
Determines how many days
until your streaming license
expires. Once your license ex
pires, you can no longer start
streaming projects.
*
Obtain a valid license and in
stall it. For the exact expira
tion date, see the monitoring
view M_LICENSES.
Availability
94 Log replay back
log for system
replication sec
ondary
System Replication secon
dary site has a higher log re
play backlog than expected.
* Investigate on secondary
site, why log replay backlog is
increased
Availability
95 Availability of
Data Quality ref
erence data (di
rectory les)
Determine the Data Quality
reference data expiration
dates.
* Download the latest Data
Quality reference data les
and update the system. (For
more details about updating
the directories, see the In
stallation and Conguration
Guide for SAP HANA Smart
Data Integration and SAP
HANA Smart Data Quality.)
Availability
Installation and
Conguration
Guide for SAP
HANA Smart
Data Integration
and SAP HANA
Smart Data
Quality
96
Long-running
tasks
Identies all long-running
tasks.
* Investigate the long-running
tasks. For more information,
see the task statistics tables
or views in _SYS_TASK
schema and trace log.
97
Granting of SAP
HANA DI con
tainer import
privileges
Determines if the container
import feature of the SAP
HANA Deployment Infra
structure (DI) is enabled and
if import privileges for SAP
HANA DI containers are cur
rently granted to any data
base users or roles.
Check if the identied users
still need the privileges. If
not, revoke the privileges
from them and disable the
SAP HANA DI container im
port feature.
Security
98 LOB garbage
collection activ
ity
Determines whether or not
the lob garbage collection is
activated.
Activate the LOB garbage
collection using the corre
sponding conguration pa
rameters.
Congura
tion
244 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
99 Maintenance
Status
Checks the installed SP ver
sion against the recom
mended SP version.
* Please consider upgrading to
the recommended SP ver
sion.
Congura
tion
100 Unsupported
operating sys
tem in use
Determines if the operating
system of the SAP HANA Da
tabase hosts is supported.
Upgrade the operating sys
tem to a supported version
(see SAP HANA Master
Guide for more information).
Congura
tion
101 SQL access for
SAP HANA DI
technical users
Determines if SQL access
has been enabled for any
SAP HANA DI technical
users. SAP HANA DI techni
cal users are either users
whose names start with
'_SYS_DI' or SAP HANA DI
container technical users
(<container name>, <con
tainer name>DI, <container
name>OO).
Check if the identied users
('_SYS_DI*' users or SAP
HANA DI container technical
users) still need SQL access.
If not, disable SQL access for
these users and deactivate
the users.
Security
102 Existence of sys
tem database
backup
Determines whether or not a
system database backup ex
ists. Without a system data
base backup, your system
cannot be recovered.
Perform a backup of the sys
tem database as soon as
possible.
Backup
103 Usage of depre
cated features
Determines if any deprecated
features were used in the last
interval.
Check the view M_FEA
TURE_USAGE to see which
features were used. Refer to
SAP Note 2425002 for fur
ther information.
104
System replica
tion: increased
log shipping
backlog
Monitors log shipping back
log. Alert is raised when
threshold value is reached
(priority dependent on
threshold values).
*
To identify the reason for the
increased system replication
log shipping backlog, check
the status of the secondary
system. Possible causes for
the increased system replica
tion log shipping backlog can
be a slow network perform
ance, connection problems,
or other internal issues.
105
Total Open
Transactions
Check
Monitors the number of open
transactions per service.
* Double check if the applica
tion is closing the connection
correctly, and whether the
high transaction load on the
system is expected.
Sessions/
Transac
tions
106 ASYNC replica
tion in-memory
buer overow
Checks if local in-memory
buer in ASYNC replication
mode runs full.
Check buer size, peak
loads, network, IO on secon
dary.
Availability
107 Inconsistent fall
back snapshot
Checks for inconsistent fall
back snapshots.
1 Drop the inconsistent snap
shot
Availability
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 245
ID Name Description Cfg User Action Category
Further Infor
mation
108 Old fallback
snapshot
Checks for out of date fall
back snapshots (older than
the dened thresholds).
* Check the age and possibly
drop the snapshot
Availability
109 Backup history
broken
Checks if the backup history
is incomplete or inconsistent
(the log_mode is internally
set to overwrite, it is not en
sured that the service is fully
recoverable via backup).
Perform a data backup as
soon as possible to ensure
that the service is fully recov
erable.
Backup
110 Catalog Consis
tency
An alert is raised if the Cata
log Consistency Check de
tects errors (identies the
number of errors and af
fected objects).
*
Contact SAP support. Availability
111 Replication sta
tus of replication
log
Check whether the status of
replication log is disabled.
* Truncate replication log table
and enable replication log
Availability
112 Missing STO
NITH with
shared storage
Check whether a STONITH
provider is congured in a
scale-out system with shared
basepaths.
1
Implement a STONITH pro
vider.
? See Implement
ing a HA/DR
Provider in the
SAP HANA Sys
tem Administra
tion Guide
.
113
Open le count Determines what percentage
of total open le handles are
in use. All processes are con
sidered, including non-SAP
HANA processes.
*
You can congure the Linux
file-max parameter using
the following commands:
1. Check the current maxi
mum number of allowed le
handles: cat /proc/sys/fs/
lemax
2. Extend maximum number
of le handles in /etc/
sysctl.conf: fs.lemax =
20000000
3. Activate changes for oper
ating system: sysctl -p /etc/
sysctl.conf
Congura
tion
114 Active async IO
count
Determines what percentage
of total asynchronous input/
output requests are in use.
All processes are considered,
including non-SAP HANA
processes.
*
You can increase the size of
the Linux I/O queue (
aio-
max-nr parameter) as de
scribed in SAP Note
1868829.
Congura
tion
SAP Note:
1868829
246 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
115 Timezone envi
ronment varia
ble verication
Determines if the timezone
environment variable TZ can
be interpreted. See M_TIME
ZONE_ALERTS. Otherwise
HANA falls back to a system
call which can cause signi
cant performance issues.
1
Please set the timezone envi
ronment variable TZ to a
valid value according to the
POSIX documentation and
restart the HANA database.
See also SAP Note 2100040.
Congura
tion
SAP Note:
2100040
116 Transparent
huge pages sta
tus
Determines if Transparent
Huge Pages (THP) are acti
vated which can cause issues
for the database.
1
Deactivate THP by setting
the kernel parameter to
"[never]".
SAP Notes:
2031375
118 Port ephemeral
max count
Checks for free local ports by
referring to following kernel
parameters:
net.ipv4.ip_local_p
ort_range
and
net.ipv4.ip_local_r
eserved_ports. The alert
is raised if the number of free
local ports is below the con
gured minimum.
1
Increase the local port range
by setting prole parameters
as described in SAP Note
401162, or free ports which
do not have to be reserved.
The number of free local
ports and ports which should
be reserved for HANA serv
ices can be found in the sys
tem view M_HOST_INFOR
MATION, keys:
net_port_ephemeral_
max_count and
net_port_ranges.
Congura
tion
SAP Note:
401162 . See
also SAP Host
Agent and Linux
kernel parame
ters in the SAP
HANA System
Administration
Guide
.
119
Required local
SAP HANA port
ranges
Checks for local ports which
are required but which have
not been reserved (see Alert
118). If not reserved there is a
risk that these ports could be
automatically assigned to
other applications.
The alert is raised if any ports
in the local ports range are
not reserved.
1
Reserve ports by setting pro
le parameters as described
in SAP Note 401162.
Details of unreserved ports
can be retrieved from the
system view M_HOST_IN
FORMATION key:
net_port_unreserved
_ranges.
Congura
tion
SAP Note:
401162 See
also SAP Host
Agent and Linux
kernel parame
ters in the SAP
HANA System
Administration
Guide
.
128
LDAP Enabled
Users without
SSL
Checks for the vulnerability
where users may be enabled
for LDAP Authentication but
SSL is not enabled.
Congure SSL to reduce risk
of man-in-the-middle attacks
and privacy protection.
Security
129 Check trusted
certicate expi
ration date
Determines if there are any
trusted certicates that will
expire soon or have already
expired.
1
Replace the certicates that
are expiring or are already ex
pired with new ones.
Security
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 247
ID Name Description Cfg User Action Category
Further Infor
mation
130 Check own cer
ticate expira
tion date
Determines if there are any
own or chained certicates
that will expire soon or have
already expired.
*
Replace the certicates that
are expiring or are already ex
pired with new ones.
Security
131 Session re
quests queued
by admission
control
Determines the number of
session requests waiting in
the admission control queue.
This can indicate an issue
with the response time of the
request.
1
Investigate why the session
requests newly queued by
admission control. Refer to
M_ADMISSION_CON
TROL_EVENTS for more in
formation.
Availability
132 Session re
quests rejected
by admission
control
Determines the number of
session requests newly re
jected by admission control.
This can indicate an issue
with the availability of the da
tabase.
1
Investigate why the session
requests newly rejected by
admission control. Refer to
M_ADMISSION_CON
TROL_EVENTS for more in
formation.
Availability
135 Checks congu
ration for SAP
HANA SLD Data
Supplier
For system replication, virtual
and physical databases must
be congured to send the
correct landscape data to the
Landscape Management Da
tabase via the System Land
scape Directory (SLD).
Congure SAP HANA SLD
Data Supplier as docu
mented in SAP HANA System
Administration Guide (see
link in Further Information).
Congura
tion
Conguring SAP
HANA for Sys
tem Replication
Technical Sce
nario in SAP Sol
ution Manager
136
Unsupported
Parameter Val
ues Set
Checks if conguration pa
rameters are set to an un
supported value.
Correct the values of congu
ration parameters as neces
sary.
Congura
tion
137 Restart Required
for Congura
tion Change
Check if a restart is required
for a conguration change to
become eective.
If necessary, restart the sys
tem.
Congura
tion
500 Dbspace usage Checks for the dbspace size
usage.
* Investigate the usage of
dbspace and increase the
size.
Disk
501 Dbspace status Determines whether or not
all dbspaces are available.
Investigate why the dbspace
is not available.
Availability
502 Dbspace le sta
tus
Determines whether or not
all dbspace les are available.
Investigate why the dbspace
le is not available.
Availability
600 Inactive Stream
ing applications
Identies inactive Streaming
applications.
Investigate why the Stream
ing application is inactive, for
example, by checking the
Streaming application's trace
les.
Availability
601 Inactive Stream
ing project man
aged adapters
Identies inactive Streaming
project managed adapters.
Investigate why the Stream
ing project managed adapter
is inactive, for example, by
checking the trace les.
Availability
248 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
ID Name Description Cfg User Action Category
Further Infor
mation
602 Streaming
project physical
memory usage
Determines what percentage
of total physical memory
available on the host is used
for the streaming project.
*
Investigate memory usage of
the streaming project.
Memory
603 Streaming
project CPU us
age
Determines the percentage
CPU usage for a streaming
project on the host and
therefore whether or not CPU
resources are running out.
*
Investigate CPU usage. CPU
604 Number of pub
lishers of
streaming
project
Identify the large publishers
of streaming project. Make
sure that they will not break
the streaming project.
*
Investigate whether these
publishers are created inten
tionally.
Congura
tion
605 Number of sub
scribers of
streaming
project
Identify the large subscribers
of streaming project. Make
sure that they will not break
the streaming project.
*
Investigate whether these
subscribers are created in
tentionally.
Congura
tion
606 Row throughput
of subscriber of
streaming
project
Identify which subscriber of
streaming project has low
throughput measured in rows
per second.
1
Investigate why the sub
scriber works slowly.
Congura
tion
607 Transaction
throughput of
subscriber of
streaming
project
Identify which subscriber of
streaming project has trans
action throughput measured
in transactions per second.
1
Investigate why the sub
scriber works slowly.
Congura
tion
608 Row throughput
of publisher of
streaming
project
Identify which publisher of
streaming project has low
throughput measured in rows
per second.
1
Investigate why the publisher
works slowly.
Congura
tion
609 Transaction
throughput of
publisher of
streaming
project
Identify which publisher of
streaming project has trans
action throughput measured
in transactions per second.
1
Investigate why the publisher
works slowly.
Congura
tion
610 Bad rows of
project managed
adapter
Identify which project man
aged adapter has much rows
with error.
1 Investigate why the adapter
has such much rows with er
ror.
Congura
tion
611 High latency of
project managed
adapter
Identify which project man
aged adapter has high la
tency.
1 Investigate why the adapter
has high latency.
Congura
tion
612 Large queue of
stream of
streaming
project
Identify which stream of
streaming project has large
queue.
1 Investigate why the stream
has large queue.
Congura
tion
SAP HANA System Replication
Troubleshoot System Replication
P U B L I C 249
ID Name Description Cfg User Action Category
Further Infor
mation
613 Large store of
stream of
streaming
project
Identify which stream of
streaming project has large
store.
1 Investigate why the stream
has large store.
Congura
tion
700 Agent availability Determines how many mi
nutes the agent has been in
active.
* Investigate connection of
agent and check if agent is
up and running.
Availability
701 Agent memory
usage
Determines what percentage
of total available memory on
agent is used.
* Investigate which adapter or
processes use a lot of mem
ory.
Memory
710 Remote Sub
scription excep
tion
Checks for recent exceptions
in remote subscriptions and
remote sources.
Investigate the error mes
sage and the error code and
restart the remote subscrip
tion if necessary.
0
250 P U B L I C
SAP HANA System Replication
Troubleshoot System Replication
10 Security Aspects for SAP HANA System
Replication
Learn about the secure operation and conguration of SAP HANA system replication.
Which security aspects are addressed in this guide?
Communication between sites in a system replication scenario is always authenticated. In addition, it is
possible to secure internal network communication between primary and secondary systems using TLS/SSL.
You can learn how to congure TLS/SSL on communication channels between primary and secondary systems
using the system public key infrastructure (PKI).
To integrate SAP HANA securely into your network environment, several general recommendations apply.
Learn how to protect the channels used in a system replication scenario.
Where can I nd more information?
The following SAP Notes are relevant for a full understanding of the concepts described in this chapter:
SAP Notes
SAP Note Title
2175672
Migration steps from manual SSL conguration for internal communication
to automatic conguration using system PKI
2356851
SAP HANA Dynamic Tiering Support for SAP HANA System Replication
2447994
SAP HANA Dynamic Tiering Support for SAP HANA System Replication
Related Information
Secure Internal Communication [page 252]
Secure Internal Communication Between Sites in System Replication Scenarios [page 256]
Legacy Conguration of Secure Internal Communication [page 257]
Congure Secure Communication (TLS/SSL) Between Primary and Secondary Sites [page 259]
Communication Channels [page 261]
Network Security [page 263]
Internal Application Encryption Service [page 266]
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 251
10.1 Secure Internal Communication
All internal SAP HANA communication can be secured using the Transport Layer Security (TLS)/Secure
Sockets Layer (SSL) protocol. A simple public-key infrastructure (PKI) is set up during installation for this
purpose.
Internal Communication Channels
The internal communication channels shown below can be secured using TLS/SSL.
Note
SAP HANA internal communication has sometimes been unocially referred to as TREXNet
communication. However, the term TREXNet is not valid in the context of SAP HANA.
Communication between databases
Communication between the hosts in a multiple-host system and between processes on a
single host
252
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
Communication between the sites in a system with system replication enabled
Communication between the SAP HANA database and additional server components
For example: Extended storage server (SAP HANA dynamic tiering) or a streaming analytics server (SAP HANA
Streaming Analytics).
System Public Key Infrastructure
A dedicated PKI is created for internal communication automatically during system installation. Every host on
which a database server and optional component server is running, as well as every tenant database in the
system, are integrated into this PKI, which uses CommonCryptoLib as the cryptographic library.
Each host and database receive a public and private key pair and a public-key certicate for mutual
authentication. These certicates are all signed by a dedicated trusted certicate authority (CA) that is unique
to the SAP HANA instance. The root personal security environment (PSE) le is stored in the system PKI SSFS
(secure store in the le system). All other PSEs are encrypted with an automatically generated random PIN and
stored in the le system. Certicates are automatically renewed when they expire.
Note
A unique master key that protects the system PKI SSFS is generated during installation or update.
However, if you received your system pre-installed from a hardware or hosting partner, we recommend that
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 253
you change it immediately after handover to ensure that it is not known outside of your organization. For
more information about how to change the SSFS master keys, see the SAP HANA Administration Guide.
To secure internal communication between hosts and sites, you can set up and congure your own PKI, but we
recommend you use the system PKI. The system PKI is always used to secure communication within tenant
databases and communication with optional server components.
Note
If high isolation is congured for tenant databases, the system PKI must also be used to secure
communication between hosts.
For more information about migrating to the system PKI from a manually congured PKI, see SAP Note
2175672.
TLS/SSL Conguration Using System PKI
No interaction is required to set up the system PKI, but you may need to explicitly enable TLS/SSL depending
on the channel as follows:
Communication Channel
Conguration Required to Enable TLS/SSL
Communication between the processes
of individual databases
Congure the system for high isolation.
High isolation requires that the processes of individual databases run under dedi
cated operating system (OS) users in dedicated OS groups. In addition, it enables
certicatebased authentication so that only the processes belonging to the same
database can communicate with each other.
If you also want data communication within databases to be encrypted, you must
change the value of the property [communication] ssl in the
global.ini from off to systemPKI. If the property ssl is not visible (for
example, in the SAP HANA studio), add the key ssl with the value systemPKI
to the section communication.
Remember
Change (or add) the property in the system database in the SYSTEM layer of
the conguration le.
Note
If cross-database access is enabled, communication between congured ten
ant databases is allowed.
For more information about how to congure a system for high isolation, see the
SAP HANA Administration Guide.
254
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
Communication Channel Conguration Required to Enable TLS/SSL
Communication between hosts in a
multiple-host system and localhost
communication
Enable TLS/SSL manually.
In the global.ini conguration le, change the value of the property
[communication] ssl to systemPKI.
This conguration ensures that only hosts belonging to the same system can
communicate with each other and that all data communication between hosts is
encrypted.
Note
In a system that is not congured for high isolation, you can still enable se
cure communication between hosts. Remember you change the property in
the system database in the SYSTEM layer.
Enabling secure communication between hosts automatically enables secure
communication between processes on the same host without any further cong
uration. Note the following:
● If you are operating a single-host and require secure localhost communica
tion, you must still enable TLS/SSL for inter-host communication as descri
bed above.
● If you have enabled TLS/SSL for inter-host communication as described
above, but do not require secure localhost communication, you can change
the value of the property [communication] ssl_local from on to
off.
Communication between sites in a sys
tem with system replication enabled
Several steps are required to enable TLS/SSL for the communication channel
used for system replication. For more information, see Secure Internal Communi
cation Between Sites in System Replication Scenarios
.
Communication between the SAP
HANA database and additional server
components
No conguration required
TLS/SSL is automatically enabled and cannot be disabled.
Related Information
Secure Stores in the File System (SSFS)
Change the SSFS Master Keys
Database Isolation
Increase the System Isolation Level
Server-Side TLS/SSL Conguration Properties for Internal Communication
Secure Internal Communication Between Sites in System Replication Scenarios [page 256]
Legacy Conguration of Secure Internal Communication [page 257]
SAP Note 2175672
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 255
10.2 Secure Internal Communication Between Sites in
System Replication Scenarios
Communication between sites in a system replication scenario is always authenticated. In addition, it is
possible to secure internal network communication between primary and secondary sites using TLS/SSL.
System replication is a mechanism for ensuring the high availability of SAP HANA systems, as well as disaster
recovery. Through the continuous replication of data from a primary to a secondary system (or systems),
including in-memory loading, system replication facilitates rapid failover in the event of a disaster. Production
operations can be resumed with minimal downtime.
Communication between the sites in a system replication landscape must be secured. The system PKI (public
key infrastructure) that is automatically created during system installation is the default and recommended
mechanism for communication. No interaction is required to set up the system PKI. However, you can also set
up and congure your own PKI (see Legacy Conguration of Secure Internal Communication).
Remember
System replication is congured for the system as a whole. This means that the system database and all
tenant databases are part of the system replication.
Conguring Authentication Between Sites
To ensure that only congured systems in a system replication landscape can communicate with each other,
SAP HANA uses certicatebased authentication based on the system PKI. To establish trust between systems,
you must copy the system PKI SSFS data le and key le from the primary system to the same location on the
secondary system(s). These les can be found at the following locations:
● $DIR_INSTANCE/../SYS/global/security/rsecssfs/data/SSFS_<SID>.DAT
● $DIR_INSTANCE/../SYS/global/security/rsecssfs/key/SSFS_<SID>.KEY
Copy the les before you register the secondary system with the primary system.
Conguring TLS/SSL-Secured Communication
In addition to authenticated communication, it is also possible to secure the following communication channels
between primary and secondary systems using TLS/SSL:
● Metadata channel used to transmit metadata (for example, topology information) between the sites
● Data channel used to transmit data between the sites
For more information on how to enable TLS/SSL on these communication channels, see the SAP HANA
Administration Guide.
256
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
Note
On SAP HANA systems with dynamic tiering, the same conguration applies. No additional steps are
required. However, before you congure communication for dynamic tiering, see SAP Note 2356851.
Be aware that you need additional licenses for SAP HANA options and capabilities. For more information,
see Important Disclaimer for Features in SAP HANA.
Additional Network Security
You can further secure communication between sites by conguring SAP HANA to use exclusively a separate
network dedicated to system replication for communication between primary and secondary sites.
It is also recommended that you congure the IP addresses of those hosts that are allowed to connect to the
ports required for system replication.
For more information, see also the section on network security and the section on host name resolution in the
SAP HANA Administration Guide.
Related Information
Legacy Conguration of Secure Internal Communication [page 257]
Server-Side TLS/SSL Conguration Properties for Internal Communication
Congure Secure Communication (TLS/SSL) Between Primary and Secondary Sites [page 259]
Host Name Resolution for System Replication
Network Security [page 263]
Secure Internal Communication [page 252]
SAP Note 2356851
10.3 Legacy Conguration of Secure Internal
Communication
Although it is recommended that you use the system PKI (public key infrastructure) that is automatically
created during system installation to secure internal communication channels, you can set up and congure
your own PKI. This manually congured PKI is also used if system replication is congured for the system.
TLS/SSL Conguration for Communication Between Hosts
Since a host can both initiate a connection with another host (client role) as well as be the target of a
connection initiated by another host (server role), every host in the system requires a public and private key
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 257
pair, and a public-key certicate (server certicate) with which it can identify itself to other hosts. Each host
also needs the certicate or certicates with which it can validate the identity of other hosts. Typically, this is
the root certicate or the certicate of the certication authority (CA) that signed the other hosts' certicates.
Note
SAP HANA dynamic tiering does not support legacy conguration (using a manually congured PKI). If you
are using SAP HANA dynamic tiering, use the system PKI conguration.
Use CommonCryptoLib as the cryptographic library. It is installed by default as part of SAP HANA server
installation.
To manually congure secure communication between hosts:
1. Create a CA for the SAP HANA installation using external tools, for example, the OpenSSL command line
tool.
We recommend that you use a dedicated CA to sign all certicates used. We recommend storing your CA
certicate in $DIR_INSTANCE/ca. This is typically the root certicate.
Recommendation
Create one private CA for each SAP HANA host. Do not use public CA for securing internal SAP HANA
communication.
2. On every host, create the required server certicates.
Every host is veried with its fully qualied domain name (FQDN). The common name (CN) must be the
FQDN of the host you get by reverse DNS look-up. The other elds describe your organization.
3. Sign the certicates with the CA.
4. On every host, create a local keystore named sapsrv_internal.pse in directory $SECUDIR and import
the private key and certicate, and the CA certicate (or root certicate).
In the communication section of the le global.ini, create the property ssl with the value on.
TLS/SSL Conguration for Cross-Site Communication in System Replication
Scenarios
In a system with system replication enabled, communication between sites (metadata and data channels) can
be secured using the same conguration described above. For the data communication, you also need to
enable SSL with the property [system_replication_communication] enable_ssl in the global.ini
conguration le. For more information, see Secure Internal Communication Between Sites in System
Replication Scenarios.
Keystore Conguration
The [communication] sslInternalKeyStore parameter in the global.ini conguration le species
the path to the keystore le that contains the certicates for the following internal communication channels:
● Communication between hosts
258
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
● Communication between sites in system replication scenarios (data communication channel).
The default value is $SECUDIR/sapsrv_internal.pse.
Related Information
Secure Internal Communication Between Sites in System Replication Scenarios [page 256]
10.4 Congure Secure Communication (TLS/SSL) Between
Primary and Secondary Sites
Congure TLS/SSL on communication channels between primary and secondary systems using the system
public key infrastructure (PKI).
Prerequisites
● You have the credentials of the operating system user, <sid>adm.
● You have the system privilege INIFILE ADMIN.
Context
The following communication channels between primary and secondary systems can be secured using TLS/
SSL:
● Metadata channel used to transmit metadata (for example, topology information) between the sites
● Data channel used to transmit data between the sites.
Note
On SAP HANA systems with dynamic tiering, the following steps also enable the system PKI for internal
system replication communication. No additional steps are required. Before you congure communication
for dynamic tiering see SAP Note 2447994 - SAP HANA Dynamic Tiering Support for SAP HANA System
Replication.
Be aware that you need additional licenses for SAP HANA options and capabilities. For more information,
see Important Disclaimer for Features in SAP HANA Platform, Options and Capabilities.
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 259
Procedure
1. Shut down all systems.
If you want to avoid downtime when enabling TLS/SSL, disable system replication. You can enable or
disable TLS/SSL without downtime only if the primary system is not enabled.
2. In the primary and secondary system, enable TLS/SSL for the data channel.
In the global.ini le, congure the property [system_replication_communication]
enable_ssl. The following values are possible:
Value
Description
o (default) TLS/SSL is disabled for replication source and target systems
on TLS/SSL is enabled for replication source and target systems
Note
You must enable SSL for the whole system, that is, in the global.ini le of the system database. Setting
this feature for single tenant databases is not supported.
For a simple system replication scenario involving two systems, it is sucient to set the property to on in
both systems.
For multitier and multitarget system replication scenarios involving three systems, you can apply on in all 3
systems to secure all system replication connections. Alternatively, you can use site_name as index to
secure either only the communication to the tier 3 secondary system or only the communication to the
primary system.
Example
To exclude the communication between the primary and the secondary, and to secure the
communication between all other systems, set the parameter as follows:
siteA ------> siteB ------> siteC
enable_ssl=on enable_ssl=on
enable_ssl=on
enable_ssl[siteB]=off enable_ssl[siteA]=off
Note
To avoid communication failure between systems, TLS/SSL must be enabled on all systems at the
same time. TLS/SSL won't be used unless the secondary system reconnects with the primary. To do
this either restart the primary and secondary systems, or re-setup system replication .
3. As <sid>adm, restart the sapstartsrv service on the secondary system(s):
a. sapcontrol -nr <instance_no> -function StopService
b. /usr/sap/<sid>/HDB<instance_no>/exe/sapstartsrv pf=/usr/sap/<sid>/SYS/
profile/<sid>_HDB<instance_no>_<host> -D -u <sid>adm
4. Restart all systems.
260
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
Related Information
SAP Note 2447994 - SAP HANA Dynamic Tiering Support for SAP HANA System Replication
10.5 Communication Channels
The network communication channels used by SAP HANA can be categorized into those used for database
clients connecting to SAP HANA and those used for internal database communication. SAP recommends using
encrypted communication channels where possible.
The following is an overview of the network communication channels used by SAP HANA.
To support the dierent SAP HANA scenarios and set-ups, SAP HANA has dierent types of network
communication channels:
● Channels used for external access to SAP HANA functionality by end-user clients, administration clients,
application servers, and for data provisioning through SQL or HTTP
● Channels used for SAP HANA internal communication within the database, between hosts in multiple-host
systems, and between systems in system-replication scenarios
Note
SAP HANA internal communication has sometimes been unocially referred to as TREXNet
communication. However, the term TREXNet is not valid in the context of SAP HANA.
The connections between SAP HANA and external components and applications come under these categories:
● Connections for administrative purposes
● Connections for data provisioning
● Connections from database clients that access the SQL/MDX interface of the SAP HANA database
● Connections from HTTP/S clients
● Outbound connections
You can see an example of what these connections look like in the gure below. Network connections are
depicted by dotted arrows. The direction of each arrow indicates which component is the initiator and which
component is the listener. Administrative access to and from SAP HANA is depicted by the green dashed
arrows. Port numbers are shown with a black background. The "xx" in the port numbers stands for the number
of your SAP HANA instance.
The gure below shows all the network channels used by SAP HANA. For the purposes of illustration, a single-
host installation with two tenant databases is depicted. However, the connections shown apply equally to a
distributed scenario.
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 261
Connections Between SAP HANA and External Components
Note
Some components depicted in the gure are supported on Intel-based hardware platforms only (for
example, SAP HANA Streaming Analytics). Refer to the Product Availability Matrix (PAM).
In addition, the dierent components of SAP HANA, as well as the hosts in a distributed scenario,
communicate with each other over internal SAP HANA connections. These connections are also used in
system replication scenarios for communication between a primary site and secondary site(s) to ensure high
availability in the event of a data center failure.
The following gure shows an example of a distributed SAP HANA system with two active hosts and an extra
standby host, both fully replicated to a secondary site to provide full disaster recovery support.
SAP HANA Internal Connections
262
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
Related Information
Securing Data Communication
Network Administration
Product Availability Matrix
10.6 Network Security
To integrate SAP HANA securely into your network environment, several general recommendations apply.
The components of an SAP HANA landscape communicate through dierent network communication
channels. It is recommended security practice to have a welldened network topology to control and limit
network access to SAP HANA to only those communication channels needed for your scenario, and to apply
appropriate additional security measures, such as encryption, where necessary. This can be achieved through
dierent means, such as separate network zones and network rewalls, and through the conguration options
provided by SAP HANA (for example, encryption). The exact setup depends on your environment, your
implementation scenario, and your security requirements and policies.
The detailed network set-up and recommendations are described in network administration section of the SAP
HANA Administration Guide. This section contains some additional security-relevant information.
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 263
Caution
It is strongly recommended that you apply the measures described in this section to protect access to the
SAP HANA database's internal communication channels and to mitigate the risk of unauthorized access to
these services.
Network Zones
● We recommend that you operate the dierent components of the SAP HANA platform in separate network
zones.
To prevent unauthorized access to the SAP HANA environment and the SAP HANA database through the
network, use network rewall technology to create network zones for the dierent components and to
restrictively lter the trac between these zones implementing a "minimum required communication"
approach. The relevant network zones depend on your specic application scenario and your network
infrastructure. For more information about network zones, see the SAP HANA Administration Guide.
● We recommend that you operate SAP HANA in a protected data-center environment.Allow only dedicated
authorized network trac from other network zones (for example, user access from the client network
zone) to follow these rules:
○ Clients accessing external standard database functionality, for example by SQL, have only access to
the database client access port.
○ Clients (for example, browser applications) accessing the SAP HANA environment through the HTTP
access feature of SAP HANA Extended Application Services, classic model (XS classic), for example
SAP HANA UI Toolkit for Info Access, have only access to the SAP HANA XS ports.
○ Some administrative functions (for example, starting and stopping the SAP HANA instance) have
access to the administrative ports.
○ XS classic exposes some administrative applications (for example, administration of Security
Assertion Markup Language (SAML) for user authentication). We recommend using URL ltering (for
example, reverse proxy) to control the exposure of dierent applications to dierent network zones.
Internal Communication
Database internal communication channels are only used for the following:
● Communication within the database
● Communication between hosts in distributed (multiple-host) scenarios
● Communication between multiple sites in system replication (high-availability) scenarios
● Communication between the SAP HANA database and server components, such as extended storage
(SAP HANA dynamic tiering)
Note the following network security considerations for single-host, multiple-host, and system replication (high-
availability) scenarios.
264
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
Single-Host Scenario
In a single-host scenario, access to the network ports for database internal communication from other network
hosts is blocked by default. We recommend that you do not change this setting. The internal communication
ports are bound to localhost.
Note
In single-host scenarios, the same communication channels are used for communication between the
dierent processes on a single host, and the internal IP addresses/ports are by default bound to the
localhost interface. Note that this does not apply to dynamic tiering. The dynamic tiering service is
bound to all interfaces, although the internal communication between SAP HANA database and dynamic
tiering uses the localhost interface.
Multiple-Host Scenario
In a distributed scenario (that is, one instance of SAP HANA on multiple hosts), internal network
communication takes place between the hosts at one site via internal. Certied SAP HANA hosts contain either
dedicated or virtualized network interfaces that are congured as part of a private network using separate IP
addresses and ports.
We recommend operating all hosts in a dedicated sub-network.
To prevent unauthorized access to the database via the internal communication channels in distributed
systems, we recommend that you prevent access to these network channels and ports from outside the
system. There are a number of ways to isolate internal network ports from the client network:
● Using the SAP HANA conguration option to route communication between the hosts of a distributed
environment onto a specied network and binding those internal network services exclusively to the
network interface (recommended option)
For more information about conguring inter-service communication, see the SAP HANA Administration
Guide.
Note
In system replication scenarios, you can use this feature in the presence of a secondary site. However,
note that additional ports used for communication between primary and secondary sites are opened
on the network interface. These ports need to be protected.
● Using operating system commands (for example, iptables), and/or network device conguration
● Using network rewall functions to block access to internal ports in specic network zones
If your setup does not permit isolating internal network communication, consider using encryption to protect
the internal communication. For more information, see the section on securing internal communication.
System Replication Scenario
In a system replication scenario, you can protect the channels used in the following ways:
● Conguring SAP HANA to use exclusively a separate network dedicated to system replication for
communication between primary and secondary site
● Conguring secure communication using the TLS/SSL protocol for encryption and mutual authentication
between sites
● Specifying the IP addresses allowed to connect to system replication ports
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 265
Additional Measures for Securing Internal Communication
We recommend that you protect internal communication further by applying additional mechanisms. This may
include ltering access to the relevant ports and channels by rewalls, implementing network separation, or
applying additional protection at the network level (for example, VPN, IPSec).
We recommend routing the connection between the sites over a special site-to-site high-speed network, which
typically already implements security measures such as separation from other network access and encryption
or authentication between sites. The details of security measures and additional network security measures
needed will depend on your specic environment. For more information about network administration, see the
SAP HANA Administration Guide
SAP HANA Extended Application Services, Advanced Model
Security mechanisms are applied to protect the communication paths used by the SAP HANA XS advanced
server infrastructure. SAP provides network topology recommendations to restrict access at the network level.
For more information, see the section on SAP HANA XS advanced security.
Data Replication Technologies
Additional network congurations may be required depending on the implemented data replication technology.
For more information, see the section on security for SAP HANA replication technologies.
Related Information
Secure Internal Communication [page 252]
Network Administration
Network Zones
Host Name Resolution for System Replication
Conguring SAP HANA Inter-Service Communication
Security for SAP HANA Extended Application Services, Advanced Model
Security for SAP HANA Replication Technologies
10.7 Internal Application Encryption Service
The internal encryption service is used internally by applications requiring data encryption.
Note
In the SAP HANA 1.0 documentation, the internal application encryption service was referred to as the
internal data encryption service.
266
P U B L I C
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
The internal application encryption service is used in the following contexts:
● Secure internal credential store
This service stores credentials required by SAP HANA for outbound connections. It is used, for example,
when data is retrieved from remote data sources using SAP HANA smart data access. It is also used during
HTTP destination calls from SAP HANA XS classic applications.
For more information, see the section on the secure internal credential store.
● Application data requiring encryption
Application developers can maintain values in the SAP HANA secure store for SAP HANA XS advanced
applications or dene secure stores using the SAP HANA XS classic $.security.Store API.
For more information, see:
○ SAP HANA XS advanced: Maintain Values in the SAP HANA Secure Store in the SAP HANA Developer
Guide for SAP HANA XS Advanced Model
○ SAP HANA XS classic:Using the Server-Side JavaScript APIs in the SAP HANA Developer Guide (For
SAP HANA Studio) and Class:Store in the SAP HANA XS JavaScript API Reference.
● Private key store
This service stores the private keys of the SAP HANA server required for secure client-server
communication, if the relevant personal security environment (PSE) is stored in the database. PSEs stored
in the database are called certicate collections.
For more information, see the section on SSL conguration on the SAP HANA server and certicate
management in SAP HANA.
Every consumer of the service has its own system-internal application encryption key. These keys are
generated as follows:
● The application key for the internal credential store is generated randomly during the rst startup.
● Application keys for XS secure stores are created at the same time as the XS secure store.
● The application key for the private key store is created when the rst private key is set for a certicate
collection.
Application encryption keys are encrypted with the application encryption service root key.
SAP HANA generates unique root keys on installation or database creation. However, if you received SAP HANA
from a hardware or hosting partner, we recommend that you change the root key of the internal application
encryption service to ensure it is not known outside your organization. We recommend that you do this
immediately after system installation or handover from your hardware or hosting partner.
Note
In a system-replication conguration, change root keys in the primary system only. New keys will be
propagated to all secondary systems. The secondary systems must be running and replicating.
The system database and all tenant database have their own individual application encryption service root key.
Related Information
Secure Internal Credential Store
Certicate Management in SAP HANA
Maintain Values in the SAP HANA Secure Store
Using the Server-Side JavaScript APIs
SAP HANA System Replication
Security Aspects for SAP HANA System Replication
P U B L I C 267
11 SQL and System View Reference
Use this chapter to nd the system views relevant for system replication.
Related Information
M_SERVICE_REPLICATION System View [page 269]
M_SYSTEM_REPLICATION System View [page 274]
M_SYSTEM_AVAILABILITY System View [page 276]
M_SYSTEM_REPLICATION_MVCC_HISTORY System View [page 278]
M_SYSTEM_REPLICATION_TAKEOVER_HISTORY System View [page 279]
M_SYSTEM_REPLICATION_TIMETRAVEL System View
M_LOG_SEGMENTS System View [page 281]
M_SNAPSHOTS System View [page 284]
11.1 M_SERVICE_REPLICATION System View
Provides information about replicated services.
Structure
Column name Data type Unit Description
HOST VARCHAR(64) Species the host name.
PORT INTEGER(10) Species the internal port.
VOLUME_ID INTEGER Species the volume ID.
SITE_ID INTEGER Species the generated site
ID.
SITE_NAME VARCHAR(256) Species the logical site
name.
SAP HANA System Replication
SQL and System View Reference
P U B L I C 269
Column name Data type Unit Description
SECONDARY_HOST VARCHAR(64) Species the secondary host
name.
SECONDARY_PORT INTEGER Species the secondary port.
SECONDARY_SITE_ID INTEGER Species the generated ID of
the secondary site.
SECONDARY_SITE_NAME VARCHAR(256) Species the secondary logi
cal site name.
SECONDARY_ACTIVE_STA
TUS
VARCHAR(16) Species the secondary ac
tive status.
SECONDARY_CON
NECT_TIME
TIMESTAMP Species the time the con
nection was established from
the secondary site.
SECONDARY_RECON
NECT_COUNT
INTEGER Species the secondary re
connect count.
SECONDARY_FAIL
OVER_COUNT
INTEGER Species the secondary fail
over count.
SECONDARY_FULLY_RE
COVERABLE
VARCHAR(5) Species that the secondary
system can be fully recov
ered with a backup from the
primary system.
If this value is FALSE, then
the backup history is broken.
If there is a takeover at that
time, then start a new data
backup once the takeover is
nished.
REPLICATION_MODE VARCHAR(16) Species the replication
mode.
REPLICATION_STATUS VARCHAR(16) Species the replication sta
tus.
REPLICATION_STATUS_DE
TAILS
VARCHAR(1.024) Species the replication sta
tus details.
270 P U B L I C
SAP HANA System Replication
SQL and System View Reference
Column name Data type Unit Description
FULL_SYNC VARCHAR(16) Species the full sync status:
● DISABLED: the full sync
is not congured
● ENABLED: the full sync
is congured, but it is
not yet active (transac
tions do not block in this
state)
● ACTIVE: the full sync
mode is congured and
active
LAST_LOG_POSITION BIGINT Species the current log po
sition.
LAST_LOG_POSITION_TIME TIMESTAMP Species the current log po
sition timestamp.
LAST_SAVEPOINT_VERSION INTEGER Species the current save
point version.
LAST_SAVEPOINT_LOG_PO
SITION
BIGINT(19) Species the current save
point log position.
LAST_SAVE
POINT_START_TIME
TIMESTAMP Species the current save
point timestamp.
SHIPPED_LOG_POSITION BIGINT Species the shipped log
positon.
SHIPPED_LOG_POSI
TION_TIME
TIMESTAMP Species the shipped log po
sition timestamp.
SHIPPED_LOG_BUF
FERS_COUNT
BIGINT Species the shipped log
buers count.
SHIPPED_LOG_BUF
FERS_SIZE
BIGINT Species the shipped log
buers size in bytes.
SHIPPED_LOG_BUF
FERS_DURATION
BIGINT Species the shipped log
buer duration in microsec
onds.
REPLAYED_LOG_POSITION BIGINT Species the log end position
of the last known replayed
log buer on the secondary
site.
SAP HANA System Replication
SQL and System View Reference
P U B L I C 271
Column name Data type Unit Description
REPLAYED_LOG_POSI
TION_TIME
TIMESTAMP Species the timestamp of
the last known replayed log
buer on the secondary site.
SHIPPED_SAVEPOINT_VER
SION
INTEGER Species the shipped save
point version.
SHIPPED_SAVE
POINT_LOG_POSITION
BIGINT Species the shipped save
point log position.
SHIPPED_SAVE
POINT_START_TIME
TIMESTAMP Species the shipped save
point start time.
SHIPPED_FULL_REP
LICA_COUNT
BIGINT Species the shipped full
replica count.
SHIPPED_FULL_REP
LICA_SIZE
BIGINT Species the shipped full
replica size in bytes.
SHIPPED_FULL_REP
LICA_DURATION
BIGINT Species the shipped full
replica duration in microsec
onds.
SHIPPED_LAST_FULL_REP
LICA_SIZE
BIGINT Species the shipped last full
replica size in bytes.
SHIPPED_LAST_FULL_REP
LICA_START_TIME
TIMESTAMP Species the shipped last full
replica start time.
SHIPPED_LAST_FULL_REP
LICA_END_TIME
TIMESTAMP Species the shipped last full
replica end time.
SHIPPED_DELTA_REP
LICA_COUNT
BIGINT Species the shipped delta
replica count.
SHIPPED_DELTA_REP
LICA_SIZE
BIGINT Species the shipped delta
replica size in bytes.
SHIPPED_DELTA_REP
LICA_DURATION
BIGINT Species the shipped delta
replica duration in microsec
onds.
SHIP
PED_LAST_DELTA_REP
LICA_SIZE
BIGINT Species the shipped last
delta replica size in bytes.
SHIP
PED_LAST_DELTA_REP
LICA_START_TIME
TIMESTAMP Species the shipped last
delta replica start time.
272 P U B L I C
SAP HANA System Replication
SQL and System View Reference
Column name Data type Unit Description
SHIP
PED_LAST_DELTA_REP
LICA_END_TIME
TIMESTAMP Species the shipped last
delta replica end time.
ASYNC_BUF
FER_FULL_COUNT
BIGINT Species the number of
times the asynchronous rep
lication buer was full.
BACKLOG_SIZE BIGINT Species the current replica
tion backlog in bytes.
MAX_BACKLOG_SIZE BIGINT Species the maximum repli
cation backlog in bytes.
BACKLOG_TIME BIGINT Species the current replica
tion backlog in microsec
onds.
MAX_BACKLOG_TIME BIGINT Species the maximum repli
cation backlog in microsec
onds.
REPLAY_BACKLOG_SIZE BIGINT Byte Species the size of all log
buers that have been ship
ped to the secondary site but
have not yet been replayed
on the scondary site.
REPLAY_BACKLOG_TIME BIGINT Microseconds Species the time dierence
between the time of the last
shipped log buer and the
last replayed log buer on
the secondary site.
MAX_REPLAY_BACK
LOG_SIZE
BIGINT Byte Species the maximum value
of the REPLAY_BACK
LOG_SIZE since the system
startup.
MAX_REPLAY_BACK
LOG_TIME
BIGINT Microseconds Species the maximum value
of REPLAY_BACKLOG_TIME
since the system startup.
SAP HANA System Replication
SQL and System View Reference
P U B L I C 273
11.2 M_SYSTEM_REPLICATION System View
Monitors system replication information.
Structure
Column name Data type Description
SITE_ID INTEGER Species the generated site ID.
SITE_NAME VARCHAR(256) Species the site name.
SECONDARY_SITE_ID INTEGER Species the generated site ID of the
secondary site.
SECONDARY_SITE_NAME VARCHAR(256) Species the secondary logical site
name.
REPLICATION_MODE VARCHAR(7) Species the congured replication
mode. This can be specied as:
● SYNC: the synchronous replication
that acknowledges when a buer
has been written to a disk.
● SYNCMEM: the synchronous repli
cation that acknowledges when a
buer has arrived in the memory.
● ASYNC: the asynchronous replica
tion.
● UNKNOWN: set if the replication
mode cannot be determined (for
example, if there is a communica
tion error when getting status in
formation from a service).
REPLICATION_STATUS VARCHAR(12)
Species the replication status.
OPERATION_MODE VARCHAR(32) Species the operation mode.
274 P U B L I C
SAP HANA System Replication
SQL and System View Reference
Column name Data type Description
SECONDARY_READ_ACCESS_STATUS VARCHAR(16) Indicates whether the secondary sys
tem is read-enabled and whether read
access is activated. This column can
have the following values:
● NOT CONFIGURED: species that
an operation mode is used that
does not allow read access.
● STOPPED: species that the sec
ondary is running in operation
mode logreplay_readaccess, but it
is currently stopped.
● VERSION MISMATCH: species
that the secondary system is run
ning in operation mode logre
play_readaccess but read access is
internally disabled on the secon
dary system because it is on a dif
ferent SAP HANA version than the
primary system.
● INITIALIZING: species that the
secondary site is running in opera
tion mode logreplay_readaccess
but read access is not yet com
pletely initialized. SQL connections
to the secondary system fail in this
state.
● ACTIVE: species that the secon
dary system is running in operation
mode logreplay_readaccess and is
initialized for read access. SQL
connections are possible in this
state.
TIER INTEGER Species the tier on which the source
site is located.
SAP HANA System Replication
SQL and System View Reference
P U B L I C 275
11.3 M_SYSTEM_AVAILABILITY System View
Monitors the availability of the system.
Structure
Column name Data type Description
EVENT_TIME TIMESTAMP Time this event was originally traced
GUID NVARCHAR ( 36 ) Event guide
IS_ORIGIN VARCHAR ( 5 ) Original entry
TRACE_HOST VARCHAR ( 64 ) Host the trace le was read from
EVENT_NAME VARCHAR ( 32 ) Event name. Possible values include:
● database_add
● database_remove
● failover_begin
● failover_end
● host_remove_prepare
● host_remove_reorg
● host_remove_abort
● host_remove
● ping
● recovery_begin
● recovery_end
● service_remove
● service_remove_abort
● service_remove_prepare
● service_remove_reorg
● service_started
●
service_starting
● service_stopped
● service_stopping
● service_unknown
EVENT_DETAIL VARCHAR ( 256 ) Additional information
ERROR_MESSAGE VARCHAR ( 256 ) Error message
276 P U B L I C
SAP HANA System Replication
SQL and System View Reference
Column name Data type Description
SYSTEM_ACTIVE VARCHAR ( 16 ) System active status (summary of ac
tive values of all hosts). Possible values
include: no, yes, unknown, starting,
stopping.
SYSTEM_STATUS VARCHAR ( 16 ) System status. Possible values include:
● error
● ignore
● info
● ok
● warning
HOST VARCHAR ( 64 ) Host that traced the event
HOST_ACTIVE VARCHAR ( 16 ) Host active status (summary of active
values of all services on that host)
HOST_STATUS VARCHAR ( 16 ) Host status
DATABASE_NAME VARCHAR ( 256 ) Database name
DATABASE_ACTIVE VARCHAR ( 16 ) Database active status
SERVICE_NAME VARCHAR ( 32 ) Service name
PORT INTEGER Port of service
VOLUME_ID INTEGER Volume ID
SERVICE_ACTIVE VARCHAR ( 16 ) Service active status
HOST_CONFIG_ROLES VARCHAR ( 64 ) Congured host roles
HOST_ACTUAL_ROLES VARCHAR ( 64 ) Actual host roles
STORAGE_CONFIG_PARTITION INTEGER Congured storage partition
STORAGE_ACTUAL_PARTITION INTEGER Actual storage partition
TARGET_HOST VARCHAR ( 64 ) Failover target host
TARGET_HOST_CONFIG_ROLES VARCHAR ( 64 ) Same as TARGET_HOST_CON
FIG_ROLES for failover target
TARGET_HOST_ACTUAL_ROLES VARCHAR ( 64 ) Same as TARGET_HOST_AC
TUAL_ROLES for failover target
TARGET_STORAGE_CONFIG_PARTI
TION
INTEGER Same as TARGET_STORAGE_CON
FIG_PARTITION for failover target
SAP HANA System Replication
SQL and System View Reference
P U B L I C 277
Column name Data type Description
TARGET_STORAGE_ACTUAL_PARTI
TION
INTEGER Same as TARGET_STORAGE_AC
TUAL_PARTITION for failover target
SITE_ID INTEGER Replication site ID
11.4 M_SYSTEM_REPLICATION_MVCC_HISTORY System
View
Displays the global multi-version concurrency control (MVCC) timestamp history in the secondary site for
system replication. The global MVCC timestamp of the secondary site is updated after a chunk of logs from the
primary site is replayed on the secondary site.
Structure
Column name Data type Description
GLOBAL_MVCC_TIMESTAMP BIGINT Species the global MVCC timestamp
SECONDARY_SITE_TIME TIMESTAMP Species the global MVCC timestamp
updated time of the secondary site
PRIMARY_SITE_TIME TIMESTAMP Species the global MVCC updated
time of the primary site
SECONDARY_SITE_UPDATE_DURA
TION
BIGINT Species the global MVCC update dura
tion of the secondary site in millisec
onds
278 P U B L I C
SAP HANA System Replication
SQL and System View Reference
11.5 M_SYSTEM_REPLICATION_TAKEOVER_HISTORY
System View
Provides access to a history of HSR takeover executions.
Structure
Column name Data type Description
TAKEOVER_START_TIME TIMESTAMP Species the start time of the takeover
command. This value matches tenant
takeovers that are executed within the
same system takeover process.
TAKEOVER_END_TIME TIMESTAMP Species the end time of the takeover
command.
EXECUTION_START_TIME TIMESTAMP Species the execution start time for
takeover of the transaction domain.
EXECUTION_END_TIME TIMESTAMP Species the execution end time for
takeover of the transaction domain.
SITE_ID INTEGER Species the generated ID of the secon
dary site at takeover time.
SITE _NAME VARCHAR(64) Species the logical name provided by
the site administrator at takeover time.
MASTER_NAMESERVER_HOST VARCHAR(64) Species the master nameserver host
at takeover time.
VERSION VARCHAR(32) Species the SAP HANA version for the
site that is executing the takeover.
SOURCE_SITE_ID INTEGER Species the generated ID of the source
site at takeover time.
SOURCE_SITE _NAME VARCHAR(64) Species the logical name for the
source site provided by the site admin
istrator at takeover time.
SOURCE_MASTER_NAME
SERVER_HOST
VARCHAR(64) Species the source site master name
server host at takeover time.
SAP HANA System Replication
SQL and System View Reference
P U B L I C 279
Column name Data type Description
SOURCE_VERSION VARCHAR(32) Species the source site SAP HANA
version.
TAKEOVER_TYPE VARCHAR(10) Species how the system went online:
● ONLINE: an online takeover
● OFFLINE: an oine takeover
● TIMETRAVEL: a timetravel take
over
REPLICATION_MODE VARCHAR(16) Species the replication mode at take
over time.
OPERATION_MODE VARCHAR(32) Species the operation mode at take
over time.
REPLICATION_STATUS VARCHAR(16) Species the replication status at take
over time.
LOG_POSITION BIGINT Species the master log position, that
has been reached by takeover.
LOG_POSITION_TIME TIMESTAMP Species the time reached by the take
over.
SHIPPED_LOG_POSITION BIGINT Species the highest master log posi
tion that has been shipped before exe
cuting takeover.
SHIPPED_LOG_POSITION_TIME TIMESTAMP Species the time of the last shipped
log buer before executing takeover.
COMMENTS NVARCHAR(5000) Species a comment for the remote
subscription.
280 P U B L I C
SAP HANA System Replication
SQL and System View Reference
11.6 M_LOG_SEGMENTS System View
Provides log segment statistics.
Structure
Column name Data type Unit Description
HOST VARCHAR(64) Host name
PORT INTEGER Internal port
VOLUME_ID INTEGER Persistence Volume ID
PARTITION_ID BIGINT Log partition ID
SEGMENT_ID BIGINT Log segment ID within parti
tion
FILE_NAME VARCHAR(512) Log segment le name
FILE_OFFSET BIGINT Byte Start position of log segment
in the le
SAP HANA System Replication
SQL and System View Reference
P U B L I C 281
Column name Data type Unit Description
STATE VARCHAR(16) Log segment state. Possible
values are:
● Formatting - The log
segment is being for
matted and not yet
used.
● Preallocated - The log
segment has been preal
located, but never used.
● Writing - The log seg
ment is currently being
written.
● Closed - The log seg
ment is closed, not
backed up and is still re
quired for restart.
● Truncated - The log seg
ment is not required for
restart, but has not been
backed up.
● BackedUp - The log seg
ment has been backed
up, but is still required
for restart.
● RetainedFree - The log
segment has been
backed up and is not re
quired for restart, but is
required to resync the
system replication sites.
● Free - The log segment
has been backed up, it is
not required for restart
and can be reused.
MIN_POSITION BIGINT First position contained in
this log segment
MAX_POSITION BIGINT Position behind the last log
record in this log segment
(closed log segments only)
HOLE_POSITION BIGINT Start position of the log hole
before this log segment
(equal to min position if no
hole)
282 P U B L I C
SAP HANA System Replication
SQL and System View Reference
Column name Data type Unit Description
USED_SIZE BIGINT Byte Used log segment size in
bytes
TOTAL_SIZE BIGINT Byte Total log segment size in
bytes
IN_BACKUP VARCHAR(5) Flag for log segment in
backup: TRUE, FALSE
ENCRYPTION_KEY_HASH VARCHAR(64) Hash of key used for log seg
ment encryption
LAST_COMMIT_TIMESTAMP TIMESTAMP Timestamp of the last com
mit in this log segment
Additional Information description
This view describes each allocated log segment and shows its current state and log position range that is
currently contained in the segment.
This view has a resettable counterpart; you can see the values since the last reset in the
M_LOG_SEGMENTS_RESET system view. To reset the view, execute the following statement: ALTER SYSTEM
RESET MONITORING VIEW SYS.M_LOG_SEGMENTS_RESET;.
Related Information
M_LOG_SEGMENTS_RESET System View
M_LOG_BUFFERS System View
M_LOG_PARTITIONS System View
M_VOLUME_IO_TOTAL_STATISTICS System View
SAP HANA System Replication
SQL and System View Reference
P U B L I C 283
11.7 M_SNAPSHOTS System View
Provides information about existing snapshots.
Structure
Column name Data type Description
HOST VARCHAR(64) Species the host name.
PORT INTEGER Species the internal port.
VOLUME_ID INTEGER Species the persistence volume ID.
ID BIGINT Species the snapshot ID.
TIMESTAMP TIMESTAMP Species the creation time.
PURPOSE VARCHAR(24) Species why the snapshot was exe
cuted.
FOR_BACKUP VARCHAR(5) Species if the snapshot was created
for backup: TRUE or FALSE.
ANCHOR BIGINT Species the anchor.
REDO_LOG_POSITION BIGINT Species the redo log position corre
sponding to the snapshot.
LAST_COMMIT_TIME TIMESTAMP Species the timestamp of the last
commit of the snapshot.
Related Information
Monitoring and Managing Tenant Databases
M_DATABASES System View
284
P U B L I C
SAP HANA System Replication
SQL and System View Reference
Important Disclaimers and Legal Information
Hyperlinks
Some links are classied by an icon and/or a mouseover text. These links provide additional information.
About the icons:
● Links with the icon : You are entering a Web site that is not hosted by SAP. By using such links, you agree (unless expressly stated otherwise in your
agreements with SAP) to this:
● The content of the linked-to site is not SAP documentation. You may not infer any product claims against SAP based on this information.
● SAP does not agree or disagree with the content on the linked-to site, nor does SAP warrant the availability and correctness. SAP shall not be liable for any
damages caused by the use of such content unless damages have been caused by SAP's gross negligence or willful misconduct.
● Links with the icon : You are leaving the documentation for that particular SAP product or service and are entering a SAP-hosted Web site. By using such
links, you agree that (unless expressly stated otherwise in your agreements with SAP) you may not infer any product claims against SAP based on this
information.
Beta and Other Experimental Features
Experimental features are not part of the ocially delivered scope that SAP guarantees for future releases. This means that experimental features may be changed by
SAP at any time for any reason without notice. Experimental features are not for productive use. You may not demonstrate, test, examine, evaluate or otherwise use
the experimental features in a live operating environment or with data that has not been suciently backed up.
The purpose of experimental features is to get feedback early on, allowing customers and partners to inuence the future product accordingly. By providing your
feedback (e.g. in the SAP Community), you accept that intellectual property rights of the contributions or derivative works shall remain the exclusive property of SAP.
Example Code
Any software coding and/or code snippets are examples. They are not for productive use. The example code is only intended to better explain and visualize the syntax
and phrasing rules. SAP does not warrant the correctness and completeness of the example code. SAP shall not be liable for errors or damages caused by the use of
example code unless damages have been caused by SAP's gross negligence or willful misconduct.
Gender-Related Language
We try not to use genderspecic word forms and formulations. As appropriate for context and readability, SAP may use masculine word forms to refer to all genders.
SAP HANA System Replication
Important Disclaimers and Legal Information
P U B L I C 285
www.sap.com/contactsap
© 2019 SAP SE or an SAP aliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form
or for any purpose without the express permission of SAP SE or an SAP
aliate company. The information contained herein may be changed
without prior notice.
Some software products marketed by SAP SE and its distributors
contain proprietary software components of other software vendors.
National product specications may vary.
These materials are provided by SAP SE or an SAP aliate company for
informational purposes only, without representation or warranty of any
kind, and SAP or its aliated companies shall not be liable for errors or
omissions with respect to the materials. The only warranties for SAP or
SAP aliate company products and services are those that are set forth
in the express warranty statements accompanying such products and
services, if any. Nothing herein should be construed as constituting an
additional warranty.
SAP and other SAP products and services mentioned herein as well as
their respective logos are trademarks or registered trademarks of SAP
SE (or an SAP aliate company) in Germany and other countries. All
other product and service names mentioned are the trademarks of their
respective companies.
Please see https://www.sap.com/about/legal/trademark.html for
additional trademark information and notices.
THE BEST RUN
