wikiSynology UC3400 Metadata-Overflow
On the Synology UC3400, a very high number of metadata objects (>10 million) leads to a critical condition in combination with migrations and snapshot load that may cause CPU and RAM overloads and cause migrations to fail.
This article describes symptoms and provides possible solutions.
Initial situation
The UC3400 administrates SAN and LUN structures with the help of a internal metadata database. When there is a very large number of objects, the metadata can grow significantly. In addition, long snapshot chains, numerous copy-on-write operations, and the simultaneous processing of I/O and metadata accesses lead to increased system load.
The situation becomes particularly critical when there is a high number of snapshots, long operation hours without reorganisation, the migration of more extensive LUNs or the high rates of changes within a short time.
Problem description
When there are more than approximately 10 million metadata objects, a significant system overload occurs on the UC3400. This condition is often referred to as Metadata overflow.
Possible symptoms are:
- CPU-/RAM-load permanently close to 100%
- migrations are canceled or "timeouted"
- system reacts very slowly
- storage and snapshot operations block each other
- SAN-services fail temporarily
- risk for inconsistent LUN conditions increases
The following applies to the Synology UC3400:
- < 5 million objects → stable
- 5–10 million objects → marginal
- 10 million objects → critical (overflow risk)
- 50 milion objects → can only be operated effectively with significant optimization
Immediate measures
The following measures can be implemented to restore operational capability quickly:
Note: These measures are intended solely as temporary workarounds and do not constitute a long-term solution for production operations.
- stop migration
- reduce snapshot load
- restart services (only in maintenance window)
Recommended solution strategy
The most sustainable solution is a reorganization of the LUN-structure and a migration into smaller, clearly separated units.
Create new LUN
The most stable method is:
- create new LUN
- transmit data via storage migration or backup restore
- delete old LUN
As a result, all old metadata will be deleted completely.
Alternative: Metadata-Rebuild
| Note: A Metadata-Rebuild should be only performed if there is a current and functional backup of all relevant data! |
This risky procedure is intended solely as a last resort for recovery after all other diagnostic and repair measures have been proven unsuccessful.
Only with complete backup:
- metadata structure is deleted and recreated
- risk of iSCSI-Mapping losses
- only recommended for emergencies
Prevention
To avoid the above mentioned problems, the following measures can be implemented:
- limit snapshot number
- LUN-recreation for long durations on a regular basis
- unload metadata regularly
- divide migrations in smaller sessions
- monitoring of CPU / RAM and SAN metadata load
Recommendation: When migrating large amounts of data, you should involve Synology Support early on. This way, they can assist with the data migration if needed, and you will receive recommendations on best practices to minimize risks.
 |
Translator: Alina Ranzinger Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.
|
