В моем приложении у меня есть запрос, который выполняет поиск в таблице «файлы».
Таблица «файлы» разделена на «f». «Создан» (см. Определение таблицы и имеет ~ 26 миллионов строк для клиента 19 («f». «Cid = 19).
Дело в том, если я сделаю этот запрос:
SELECT "f"."id" AS "FileId"
, "f"."name" AS "FileName"
, "f"."year" AS "Fileyear"
, "f"."cid" AS "clientId"
, "f"."created" AS "FileDate"
, CASE WHEN ("vnVE0"."value" is not null AND "vnVE0"."value" != '')
THEN CAST("vnVE0"."value" AS decimal(28,2))
ELSE 0 END AS "keywordValueCol0_numeric"
FROM files "f"
OUTER APPLY
(
SELECT DISTINCT
VT.[value]
FROM dbo.value_number AS VT
WHERE
VT.id_file = F.id
AND VT.id_field = 260
) AS "vnVE0"
WHERE "grapado" IS NULL AND "masterversion" IS NULL AND ("f"."year" = 2013 OR "f"."year" = 0) AND "f"."cid" = 19
GROUP BY "f"."id", "f"."name", "f"."year", "f"."cid", "f"."created", CASE WHEN ("vnVE0"."value" is not null AND "vnVE0"."value" != '')
THEN CAST("vnVE0"."value" AS decimal(28,2))
ELSE 0 END
ORDER BY (SELECT NULL)
OFFSET 0 ROWS FETCH NEXT 50 ROWS ONLY;
Я получаю результаты в течение 0 секунд со следующим планом выполнения: https://www.brentozar.com/pastetheplan/?id=SkV0-FDcG
Но если я попытаюсь упорядочить по «имени», запрос станет слишком медленным:
SELECT "f"."id" AS "FileId"
, "f"."name" AS "FileName"
, "f"."year" AS "Fileyear"
, "f"."cid" AS "clientId"
, "f"."created" AS "FileDate"
, CASE WHEN ("vnVE0"."value" is not null AND "vnVE0"."value" != '')
THEN CAST("vnVE0"."value" AS decimal(28,2))
ELSE 0 END AS "keywordValueCol0_numeric"
FROM files "f"
OUTER APPLY
(
SELECT DISTINCT
VT.[value]
FROM dbo.value_number AS VT
WHERE
VT.id_file = F.id
AND VT.id_field = 260
) AS "vnVE0"
WHERE "grapado" IS NULL AND "masterversion" IS NULL AND ("f"."year" = 2013 OR "f"."year" = 0) AND "f"."cid" = 19
GROUP BY "f"."id", "f"."name", "f"."year", "f"."cid", "f"."created", CASE WHEN ("vnVE0"."value" is not null AND "vnVE0"."value" != '')
THEN CAST("vnVE0"."value" AS decimal(28,2))
ELSE 0 END
ORDER BY "f"."name"
OFFSET 0 ROWS FETCH NEXT 50 ROWS ONLY;
Этот запрос занимает 11 минут, чтобы вернуть мне результаты со следующим планом выполнения: https://www.brentozar.com/pastetheplan/?id=Sk3Fbtv9M
Также, если я изменю порядок по столбцам, результат будет таким же.
Как видно из плана выполнения, индекс «files_mv» имеет стоимость 61%, это определение индекса:
CREATE NONCLUSTERED INDEX [files_mv] ON [dbo].[files]
(
[masterversion] ASC,
[year] ASC,
[cat_id] ASC,
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[sub_id] ASC,
[tip_id] ASC
)
INCLUDE ( [id],
[name]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
GO
Я использую SQL Server с Azure. В частности, база данных SQL Azure с уровнем цены / модели "S4 Estándar (200 DTU)".
Я получаю много данных, но думаю, что интернет-соединение не является узким местом, потому что в других запросах я тоже получаю много данных, и они быстрее.
Кроме того, я делаю массивную вставку данных в эти таблицы, и через несколько дней у меня будет до ~ 240 миллионов строк в таблице файлов (для одного cid) и ~ 480 миллионов строк в таблице value_number.
Дополнительная информация
Функция разделения "PF_files_partitioning":
CREATE PARTITION FUNCTION PF_files_partitioning (DATETIME2(7))
AS
RANGE LEFT FOR VALUES ( '2013-03-31 23:59:59',
'2013-06-30 23:59:59',
'2013-09-30 23:59:59',
'2013-12-31 23:59:59',
'2014-03-31 23:59:59',
'2014-06-30 23:59:59',
'2014-09-30 23:59:59',
'2014-12-31 23:59:59',
'2015-03-31 23:59:59',
'2015-06-30 23:59:59',
'2015-09-30 23:59:59',
'2015-12-31 23:59:59',
'2016-03-31 23:59:59',
'2016-06-30 23:59:59',
'2016-09-30 23:59:59',
'2016-12-31 23:59:59',
'2017-03-31 23:59:59',
'2017-06-30 23:59:59',
'2017-09-30 23:59:59',
'2017-12-31 23:59:59',
'2018-03-31 23:59:59')
Схема разбиения "PS_files_partitioning":
CREATE PARTITION SCHEME PS_files_partitioning AS PARTITION PF_files_partitioning ALL TO ([PRIMARY]);
** У меня будет около 15 миллионов строк в каждом разделе.
таблица файлов:
CREATE TABLE [dbo].[files](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[cid] [tinyint] NOT NULL,
[eid] [bigint] NOT NULL,
[cat_id] [bigint] NOT NULL,
[tip_id] [bigint] NULL,
[sub_id] [bigint] NULL,
[year] [smallint] NOT NULL,
[caducidad] [smallint] NULL,
[grapadopri] [int] NOT NULL,
[grapado] [bigint] NULL,
[name] [nvarchar](255) NOT NULL,
[extension] [tinyint] NOT NULL,
[size] [bigint] NOT NULL,
[id_doc] [bit] NOT NULL,
[observaciones] [nvarchar](255) NOT NULL,
[indexed] [bit] NOT NULL,
[signed] [bit] NOT NULL,
[created] [datetime2](7) NOT NULL,
[name_lower] [nvarchar](255) NOT NULL,
[modified] [datetime2](7) NULL,
[related] [bit] NOT NULL,
[masterversion] [bigint] NULL,
[versioned] [bit] NOT NULL,
[hwsignature] [tinyint] NOT NULL,
[blockedUserId] [smallint] NULL,
CONSTRAINT [PK_files_id] PRIMARY KEY CLUSTERED
(
[id] ASC,
[created] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created]),
CONSTRAINT [files$estructure_unique] UNIQUE NONCLUSTERED
(
[cat_id] ASC,
[tip_id] ASC,
[sub_id] ASC,
[year] ASC,
[name] ASC,
[grapado] ASC,
[created] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)
таблица значений:
CREATE TABLE [dbo].[value_number](
[id] [bigint] IDENTITY(1,1) NOT NULL,
[id_file] [bigint] NOT NULL DEFAULT ((0)),
[id_field] [bigint] NOT NULL DEFAULT ((0)),
[value] [nvarchar](255) NULL DEFAULT (NULL),
[id_doc] [bigint] NULL DEFAULT (NULL)
CONSTRAINT [PK_value_number_id] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
)
Индексы файлов таблицы
CREATE NONCLUSTERED INDEX [files_clientes] ON [dbo].[files]
(
[cid] ASC
)
INCLUDE ([id]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [files_grapado] ON [dbo].[files]
(
[grapado] ASC
)
INCLUDE ( [id],
[name]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [files_mv] ON [dbo].[files]
(
[masterversion] ASC,
[year] ASC,
[cat_id] ASC,
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[sub_id] ASC,
[tip_id] ASC
)
INCLUDE ( [id],
[name]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [files_ocr] ON [dbo].[files]
(
[cid] ASC,
[grapado] ASC,
[indexed] ASC,
[masterversion] ASC,
[extension] ASC
)
INCLUDE ( [id],
[eid],
[cat_id],
[tip_id],
[sub_id],
[year],
[name]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [files_ocr2] ON [dbo].[files]
(
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[indexed] ASC,
[masterversion] ASC,
[extension] ASC
)
INCLUDE ( [id],
[cat_id],
[tip_id],
[sub_id],
[year],
[name]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [files_ocr3] ON [dbo].[files]
(
[cid] ASC,
[cat_id] ASC,
[grapado] ASC,
[indexed] ASC,
[masterversion] ASC,
[extension] ASC
)
INCLUDE ( [eid],
[tip_id],
[sub_id],
[year],
[name]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [busqueda_name] ON [dbo].[files]
(
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[year] ASC
)
INCLUDE ( [id],
[cat_id],
[tip_id],
[sub_id],
[grapadopri],
[name],
[size],
[id_doc],
[signed],
[created],
[modified],
[related],
[masterversion]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [busqueda2] ON [dbo].[files]
(
[cid] ASC,
[eid] ASC,
[cat_id] ASC,
[grapado] ASC,
[masterversion] ASC,
[year] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [cid] ON [dbo].[files]
(
[cid] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [eid] ON [dbo].[files]
(
[eid] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [extension] ON [dbo].[files]
(
[extension] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [FK_files_archivo] ON [dbo].[files]
(
[grapado] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [FK_files_tipo] ON [dbo].[files]
(
[tip_id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [grapadopri] ON [dbo].[files]
(
[grapadopri] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [index_all] ON [dbo].[files]
(
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[masterversion] ASC
)
INCLUDE ( [cat_id],
[tip_id],
[sub_id],
[year],
[grapadopri],
[name],
[size],
[id_doc],
[signed],
[created],
[modified],
[related],
[versioned]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [missing_index_7_6] ON [dbo].[files]
(
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[name] ASC,
[year] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [ocrCloudClients] ON [dbo].[files]
(
[grapado] ASC,
[indexed] ASC,
[extension] ASC
)
INCLUDE ( [cid],
[eid],
[cat_id],
[tip_id],
[sub_id]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [searchEntity] ON [dbo].[files]
(
[cid] ASC,
[eid] ASC,
[grapado] ASC,
[masterversion] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
CREATE NONCLUSTERED INDEX [sub_id] ON [dbo].[files]
(
[sub_id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON PS_files_partitioning([created])
Индексы таблицы value_number
CREATE NONCLUSTERED INDEX [searchValues] ON [dbo].[value_number]
(
[id_field] ASC
)
INCLUDE ( [id_file],
[value]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
CREATE NONCLUSTERED INDEX [search] ON [dbo].[value_number]
(
[id_file] ASC,
[id_field] ASC
)
INCLUDE ( [value]) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
CREATE NONCLUSTERED INDEX [id_field] ON [dbo].[value_number]
(
[id_field] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
CREATE NONCLUSTERED INDEX [FK_valueesN_documento] ON [dbo].[value_number]
(
[id_doc] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
CREATE NONCLUSTERED INDEX [FK_valueesN_archivo] ON [dbo].[value_number]
(
[id_file] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
Статистика актуальна. Я изменил типы данных для года и других столбцов, и теперь производительность кажется немного лучше, но план выполнения остается прежним. Я пытаюсь исправить оценку кардинальности (изменение индекса), но пока не добился успеха. Согласно документации Azure у меня должно быть 130 уровней совместимости в базе данных, и у меня уже есть 100 с ProductVersion 12.0.